NVIDIA publishes a sparse attention design guide: reusing the previous step's selection makes Top-k 2.1x to 2.4x faster than a standard radix select on DeepSeek V3.2 and V4 decode
Once context reaches hundreds of thousands of tokens, attention becomes the most expensive part of each layer. How to design model-level sparse attention that runs fast on GPUs, for teams training or serving long-context models.
// Key points
- Dense attention's cost per query grows with context length and at 128K tokens can take most of a layer's prefill time; model-level sparse attention lets each query attend to a small subset of keys, but scoring all keys moves into an indexer that still processes the full context.
- Design advice: use a single key head in the indexer with a cross-head reduction; pick an indexer head dimension of 128 or 256; run the epilogue and accumulation in FP16; cut the number of keys scored through compression, hierarchical indexing or index sharing.
- Guess-Verify-Refine treats the previous decode step's selection as a guess and verifies it in one pass, running 2.1x to 2.4x faster than a standard radix select on DeepSeek V3.2 and V4 decode; the fused version is in TensorRT LLM, and FlashInfer has an open-source fused indexer and Top-k.
Builder's takeWe do not train models, but this explains why sparse-attention models like DeepSeek's are cheap on long context. For whole-document Q&A in our AI cloud drive, I would prefer open models with sparse attention and confirm the serving stack uses the fused TensorRT LLM or FlashInfer path, or much of the long-context cost advantage is lost.
// Background · from #Open models
Full timeline →- Nov 17 Hugging Face moves Inference Providers onto OpenRouter's router: open-weight models served by 80+ providers are callable with one Hugging Face token, with no markup
- Nov 17 Open-weight models on Amazon Bedrock now support explicit prompt caching alongside implicit caching, through the Chat Completions API
- Nov 17 NVIDIA's provisional MLPerf Endpoints v1.0 results: Vera Rubin NVL72 delivers 5x to 10x the throughput per kilowatt of GB300 NVL72 on agentic inference