The authors define consensus residuals as the deviation of each expert's output from the original weighted combination of all experts at a fixed layer input.
Training-free MoE expert pruning via consensus residuals preserves reasoning performance
Razor computes exact output change for expert deletion using forward-only computation, outperforming frequency and norm-based baselines on multiple models.
Chinese Tech
Tencent
Research Digest··2 min read
The authors introduce Razor, a training-free pruning method for mixture-of-experts LLMs that selects experts to remove by aggregating consensus residuals—exact deviations of expert outputs from the weighted mixture.
Why this paper
From Tencent
In one line
Razor uses consensus residuals to prune replaceable MoE experts, outperforming frequency and norm methods on nine reasoning tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§