Mixture-of-experts models route each token through only a few experts but must store every expert.
Pruning experts by replacement damage better preserves MoE reasoning performance
RAZOR estimates how survivor renormalization and router-selected replacements compensate for each deleted expert, then prunes without gradients or retraining.
Chinese Tech
Mingyang Song · Mao Zheng
Tencent · Code Models
Research Digest··3 min read
Song and Zheng introduce RAZOR, a training-free method for removing stored experts from mixture-of-experts language models while retaining their reasoning performance.
Why this paper
From Tencent and Code Models
In one line
RAZOR prunes replaceable experts in MoE models using consensus residuals, surpassing baselines on reasoning tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (4 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§