NVIDIA builds a fully local voice agent on Nemotron models: a median 1.70 seconds from end of speech to audio, 0.72 seconds for cached replies
A Halloween installation, but it lays out the model stack, hardware and latency numbers for a local voice agent.
// Key points
- The Nemotron Voice Agent blueprint runs the pipeline: Nemotron 3 ASR (streaming, English, 0.6B) for recognition, Nemotron 3.5 Bolt E4B 4GB with a custom LoRA for dialogue, Magpie TTS Zeroshot for a voice from short reference samples, and Audio2Face-3D driving the mouth in Unreal Engine 5.
- The LoRA was trained on over 5,000 multi-turn conversations; the language and voice pipeline runs on one RTX PRO 6000 Blackwell Max-Q, and rendering plus Audio2Face-3D on one RTX PRO 4000 Blackwell.
- With Bolt Q4 and TTS, median end-of-speech-to-audio latency was 1.70 seconds, and 0.72 seconds for cached responses; end of speech is detected after 300 ms of silence.
- The installation's own code is not being released, but the underlying NVIDIA models and the Voice Agent blueprint are available.
Builder's takeIt is a holiday toy, but the numbers are useful for anyone building voice products: a median 1.7 seconds fully local, down to 0.72 seconds on a cache hit. For something like AI Interview, I would cache high-frequency replies such as openers and common follow-ups first, and tune the silence threshold separately: 300 ms may be short for a candidate who is still thinking, and the agent will talk over them.
// Background · from #Open models
Full timeline →- Nov 2 NVIDIA introduces NemoClaw open blueprints: OpenClaw, Hermes Agent or LangChain Deep Agents paired with Nemotron models and the OpenShell runtime
- Nov 2 NVIDIA runs Dynamo 1.4.2 on the open-source AI Cluster Runtime: validated recipes pin about 18 Helm charts for GB200 and GB300 NVL72 on EKS
- Nov 1 NVIDIA runs Nemotron 3.5 Lightning on DGX Spark in a few commands with TensorRT Edge-LLM, and DSpark speculative decoding lifts HumanEval generation from 73.4 to 126.9 tokens/s