RAZOR computes, for each expert, the exact single-deletion change in the layer output while keeping the input fixed.
Expert pruning scored by replaceability outperforms frequency metrics on reasoning tasks
RAZOR, a training-free method, uses consensus residuals to model survivor reweighting and router refill when deleting MoE experts.
Chinese Tech
Mingyang Song · Mao Zheng
Tencent · Code Models
Research Digest··2 min read
The authors introduce RAZOR, a training-free expert pruning method for mixture-of-experts LLMs that scores experts by how well the remaining mixture can replace their contribution.
Why this paper
From Tencent and Code Models
In one line
RAZOR prunes experts in MoE LLMs by scoring replaceability, achieving the highest accuracy on reasoning benchmarks among pruning methods.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (5 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§