NVIDIA open-sources Snapshot: inference replicas restore from a GPU memory checkpoint in 3.6 to 42.7 seconds, 8x to 36x faster than a cold start
When scaling up or recovering, a model cold start often takes minutes. How Snapshot skips weight loading, compilation and graph capture.
// Key points
- Snapshot is an open-source checkpoint and restore system for GPU workloads on Kubernetes; it captures a fully initialized workload's CPU and GPU memory and restores it on nodes with the same GPU model and driver.
- In tested configurations restores take 3.6 to 42.7 seconds, 8x to 36x faster than cold starts that take minutes; creating a checkpoint takes 15 seconds to 5.5 minutes, and checkpoint plus restore is shorter than a single cold start in every configuration.
- It supports vLLM, SGLang and TensorRT-LLM, plus multi-GPU setups with tensor, data and expert parallelism, with checkpoints up to 1 TB across eight GPUs for large MoE models.
- One upstream contributor cut GLM-5.3 Pod readiness from about 8 minutes to 31 seconds on NVIDIA HGX B300.
Builder's takeFor teams self-hosting open models, the biggest cost is often the GPUs kept warm for peaks. With cold starts down from minutes to under a minute, scaling idle replicas to zero becomes realistic. I would start with batch-style services that tolerate tens of seconds of delay, and check whether the same-GPU-model-and-driver constraint fits our fleet.
// Background · from #Open models
Full timeline →- Nov 11 Hugging Face defines a common interface for decision models: Transformers adds a DecisionPipeline, and llama.cpp supports them too
- Nov 10 NVIDIA Nemotron 3.5 Super VL arrives in SageMaker JumpStart: 120B total and 12B active parameters, a 1M-token context, and an NVFP4 variant of about 75 GB
- Nov 10 Amazon's HOPE prunes MoE experts by modeling how they work in pairs, ranking 1.58 on average at 50% pruning versus 2.42 for the next-best method, REAP