NVIDIA runs Nemotron 3.5 Lightning on DGX Spark in a few commands with TensorRT Edge-LLM, and DSpark speculative decoding lifts HumanEval generation from 73.4 to 126.9 tokens/s
Getting started with edge inference got easier, with numbers that show where speculative decoding helps and where it does not.
// Key points
- The trtedge CLI downloads a model and builds TensorRT engines; the tested version is 0.11.1, and it serves an OpenAI-compatible API on 127.0.0.1:8000.
- The same runtime spans development on DGX Spark and deployment on Jetson Thor, DRIVE AGX Thor and IGX Thor.
- The example model is Nemotron-3.5 Lightning 30B-A3B NVFP4, about a 21GB first download; DSpark uses three draft tokens with peak memory of 27.9GB.
- With DSpark on, mean generation speed goes from 73.4 to 126.9 tokens/s on HumanEval and 73.3 to 108.5 on SpeedBench 1K, while median time to first token stays about the same (98.1 to 98.7 ms).
Builder's takeThe numbers show speculative decoding speeds up long outputs and leaves time to first token untouched: if your bottleneck is users waiting for the first word, it will not help; for long reports or long code, expect about 1.5x. The API is OpenAI-compatible, so a local test is just a base_url change; I would run my own product's real prompts through it before deciding whether an edge engine is worth maintaining separately.
// Background · from #Open models
Full timeline →- Oct 30 Mistral opens the weights of Mistral Large 4: an MoE of about 1T parameters, priced at $0.70 input and $2.10 output per million tokens on the API
- Oct 30 Hugging Face releases AnySpeechRecognizer and AnyDecisionModel: swap in local speech models like Parakeet on Apple devices, with Parakeet Unified at a 1.43% word error rate
- Oct 30 IBM ships Granite Switch 4.1.1: only the adapters and speech model were retrained, the base model is unchanged and no migration is needed