The authors propose RAZOR, a method that scores the replaceability of each expert by computing the exact output change if that expert is deleted, accounting for reweighting of survivors and router refill.
Pruning experts by replaceability, not magnitude, preserves reasoning in MoE models
The RAZOR method achieves higher task accuracy than existing pruning methods across four large language models at 25% and 50% expert removal.
Chinese Tech
Mingyang Song · Mao Zheng
Tencent
Research Digest··2 min read
The authors introduce RAZOR, a training-free pruning method for mixture-of-experts models that scores experts by how well surviving experts can compensate for their removal.
Why this paper
From Tencent
In one line
RAZOR prunes MoE experts by measuring how well survivors can replace a removed expert, outperforming existing methods on reasoning tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§