Amazon teams present HOPE (Higher-Order Pruning of Experts), a pruning method designed for Mixture-of-Experts (MoE) language models. These models reach hundreds of billions of parameters, but their memory remains a costly obstacle. HOPE seeks to remove the least useful experts by taking their interactions into account.
In a MoE architecture, the model is divided into sub-networks called ‘experts’. For each token, that is, each unit of processed text, only a few experts are activated. According to the source, Qwen3.5-122B has 256 experts in each of its 48 MoE layers. These experts alone represent 95% of the model’s roughly 250 GB of memory, while only 8 experts per layer are used for a given token.
Pruning involves permanently removing experts. This approach avoids the delays associated with dynamic loading, requires no training, and combines with quantization. Existing methods, however, assign each expert an isolated score, without considering the others. HOPE distinguishes itself by relying on interactions between pairs of experts.
Experts that cooperate
The authors observe that experts form groups that are often activated together. Removing an expert deemed less important can thus dismantle a group if another expert, which depends on it, is heavily used. Methods that score each expert separately do not see this phenomenon.
HOPE relies on a matrix called F. Its diagonal measures the individual importance of each expert, and the off-diagonal values measure their pairwise cooperation. According to the authors, the diagonal almost perfectly recovers the ranking of REAP, a reference method, with a Spearman coefficient of 0.988. Any improvement of HOPE over REAP would therefore come from the interaction terms.
Results that improve with the pruning rate
The method was evaluated on three models, Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, with two calibration datasets, and compared to four reference methods: REAP, MAN, EAN and Frequency. Pruning rates range from 10% to 50%. Across all conditions, HOPE achieves the best average rank, at 2.07 out of 5 methods, with 39% first-place finishes.
HOPE’s advantage increases with the pruning rate. At 50%, its average rank reaches 1.58, compared to 2.42 for REAP. On SWE-Bench Pro, an agentic programming test, Qwen3.5-35B-A3B pruned at 50% scores 42.3, compared to 41.8 for the unpruned model, while REAP drops to 36.2, a regression of 13%. The authors indicate that the advantage is smaller at 10-20%, and that HOPE does not account for interactions between layers.
A three-step implementation
The code, published as a package, includes three commands: calibrate, solve the quadratic program, then prune. Calibration requires only a single forward pass on test data, and the solving takes, according to the source, one to two seconds per layer. The resulting model is a standard Hugging Face checkpoint, lighter on disk and in memory, with no modification to vLLM or SGLang. Tests cover Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5.
