Research #Open models Nov 18, 2026 · Originally published Nov 17

NVIDIA publishes a sparse attention design guide: reusing the previous step's selection makes Top-k 2.1x to 2.4x faster than a standard radix select on DeepSeek V3.2 and V4 decode

Once context reaches hundreds of thousands of tokens, attention becomes the most expensive part of each layer. How to design model-level sparse attention that runs fast on GPUs, for teams training or serving long-context models.

Image: NVIDIA Technical Blog
Primary source NVIDIA Technical Blog · developer.nvidia.com Read the original ↗

// Key points

  • Dense attention's cost per query grows with context length and at 128K tokens can take most of a layer's prefill time; model-level sparse attention lets each query attend to a small subset of keys, but scoring all keys moves into an indexer that still processes the full context.
  • Design advice: use a single key head in the indexer with a cross-head reduction; pick an indexer head dimension of 128 or 256; run the epilogue and accumulation in FP16; cut the number of keys scored through compression, hierarchical indexing or index sharing.
  • Guess-Verify-Refine treats the previous decode step's selection as a guess and verifies it in one pass, running 2.1x to 2.4x faster than a standard radix select on DeepSeek V3.2 and V4 decode; the fused version is in TensorRT LLM, and FlashInfer has an open-source fused indexer and Top-k.

Builder's takeWe do not train models, but this explains why sparse-attention models like DeepSeek's are cheap on long context. For whole-document Q&A in our AI cloud drive, I would prefer open models with sparse attention and confirm the serving stack uses the fused TensorRT LLM or FlashInfer path, or much of the long-context cost advantage is lost.

// Background · from #Open models

Full timeline →
  1. Nov 17 Hugging Face moves Inference Providers onto OpenRouter's router: open-weight models served by 80+ providers are callable with one Hugging Face token, with no markup
  2. Nov 17 Open-weight models on Amazon Bedrock now support explicit prompt caching alongside implicit caching, through the Chat Completions API
  3. Nov 17 NVIDIA's provisional MLPerf Endpoints v1.0 results: Vera Rubin NVL72 delivers 5x to 10x the throughput per kilowatt of GB300 NVL72 on agentic inference

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →