Amazon's HOPE prunes MoE experts by modeling how they work in pairs, ranking 1.58 on average at 50% pruning versus 2.42 for the next-best method, REAP
Most of an MoE model's memory goes to idle experts, and cutting them risks capability. A training-free pruning method that composes with quantization, and where it stops working.
// Key points
- Take Qwen3.5-122B: 256 experts in each MoE layer, only 8 used per token, and 95% of its roughly 250 GB of parameters sit in expert MLPs.
- HOPE is a second-order method that accounts for pairwise expert interactions; against 4 first-order baselines on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, it achieved the best average rank of 2.07 across 5 methods and the top finish in 39% of conditions.
- Its advantage grows with the pruning rate, up to +6.1% on agentic coding tasks over the next-best method; at 10% to 20% pruning, first-order methods are sometimes competitive.
- A package runs it in three steps, calibrate, solve and prune, and has been tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5.
Builder's takeWorth a try if you self-host an MoE open model and are memory-bound: calibrate on real traces from your own product (the paper used SWE-Bench trajectories), prune 40% to 50%, stack quantization on top, and validate on your own eval set rather than public leaderboards.
// Background · from #Open models
Full timeline →- Nov 10 NVIDIA Nemotron 3.5 Super VL arrives in SageMaker JumpStart: 120B total and 12B active parameters, a 1M-token context, and an NVFP4 variant of about 75 GB
- Nov 9 Ai2 opens ToyBlocks, a robot learning framework: 85,336 long-horizon demonstrations, about 1,600 hours of data, and up to 24x real-time generation on one GPU
- Nov 8 Kimi K3.1 Pro (Highspeed) arrives on Amazon Bedrock: open-weight, natively multimodal, a 1-million-token context, and explicit prompt caching