Hugging Face shows Transformers streaming MoE experts from SSD: a 180B-parameter model runs at about 14 tok/s on a 32 GB MacBook
When a MoE model does not fit in memory, inference engines offload in very different ways. A Transformers expert-streaming approach under review and its measured results, for developers who want to run large MoE models locally.
// Key points
- Experts are 90%+ of a MoE model's weights; the approach keeps experts on disk and a single on-device slot pool shared by all layers, so grouped_mm runs as is.
- With every expert cached, speed went from 10.1 tok/s in the first version to 34.6 tok/s (38.1 for vanilla); prefetching the next two layers' experts raised prefill from 68 to about 228 tok/s.
- Running Qwen3.8-Flash-Next on a 32 GB M5 MacBook, it is about 6% behind Strata on short decode with slightly faster prefill, using about 18 GiB after loading.
- The code is in PR huggingface/transformers#50717, with the API still under discussion with maintainers; the author says all the code was written with AI.
Builder's takeThe author says 14 tok/s is too slow for coding agents, so the value is not "180B on a laptop". It is that once Transformers supports per-expert offload natively, evals and offline batch jobs can use large MoE models without switching frameworks. I would wait for the PR to merge; for local runs today, llama.cpp or Strata is the practical choice.
// Background · from #Open models
Full timeline →- Nov 19 Cloudflare opens the weights of Clairic and Clairic-flash: 27B and 9B extraction models that read PDFs and bill only for input tokens
- Nov 19 AWS Lambda Managed Instances adds GPUs: run inference functions on EC2 G-series instances, with vLLM, SGLang and PyTorch base images
- Nov 19 Rumble-2-Flash is open-sourced under Apache 2.0: a 0.5B-parameter speech recognition model for 40+ languages, averaging 6.96% error on the multilingual track