Open source #Open models Nov 19, 2026 · Originally published Nov 18

Hugging Face shows Transformers streaming MoE experts from SSD: a 180B-parameter model runs at about 14 tok/s on a 32 GB MacBook

When a MoE model does not fit in memory, inference engines offload in very different ways. A Transformers expert-streaming approach under review and its measured results, for developers who want to run large MoE models locally.

Image: Hugging Face Blog
Primary source Hugging Face Blog · huggingface.co Read the original ↗

// Key points

  • Experts are 90%+ of a MoE model's weights; the approach keeps experts on disk and a single on-device slot pool shared by all layers, so grouped_mm runs as is.
  • With every expert cached, speed went from 10.1 tok/s in the first version to 34.6 tok/s (38.1 for vanilla); prefetching the next two layers' experts raised prefill from 68 to about 228 tok/s.
  • Running Qwen3.8-Flash-Next on a 32 GB M5 MacBook, it is about 6% behind Strata on short decode with slightly faster prefill, using about 18 GiB after loading.
  • The code is in PR huggingface/transformers#50717, with the API still under discussion with maintainers; the author says all the code was written with AI.

Builder's takeThe author says 14 tok/s is too slow for coding agents, so the value is not "180B on a laptop". It is that once Transformers supports per-expert offload natively, evals and offline batch jobs can use large MoE models without switching frameworks. I would wait for the PR to merge; for local runs today, llama.cpp or Strata is the practical choice.

// Background · from #Open models

Full timeline →
  1. Nov 19 Cloudflare opens the weights of Clairic and Clairic-flash: 27B and 9B extraction models that read PDFs and bill only for input tokens
  2. Nov 19 AWS Lambda Managed Instances adds GPUs: run inference functions on EC2 G-series instances, with vLLM, SGLang and PyTorch base images
  3. Nov 19 Rumble-2-Flash is open-sourced under Apache 2.0: a 0.5B-parameter speech recognition model for 40+ languages, averaging 6.96% error on the multilingual track

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →