OptEP uses task-domain calibration data to identify a fixed-size subset of experts in each pretrained mixture-of-experts layer.
Joint expert selection preserves MoE performance under aggressive pruning
OptEP chooses complementary experts by reconstructing layer outputs, then adapts their weights to compensate for removed experts.
Big Tech
Jelena Markovic-Voronov · Petros Prastakos · Kayhan Behdin · Jincheng Cao · Zhipeng Wang · Shayan Mohajer Hamidi · +4 more
LinkedIn · MIT · Google
Research Digest··2 min read
The authors treat post-training expert pruning as a joint subset-selection problem rather than ranking each expert independently.
§