Mixture-of-Experts Specialization
Methods for preventing expert collapse and promoting functional specialization in mixture-of-experts models.
7 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Oct 2026 · Bar-Ilan University, NVIDIA
Supervising token-level loss improves routing in sparse mixture-of-experts models
Token-level supervision aligns routing decisions with next-token loss.
1 further paper
Oct 2026 · Peking University, Tencent Hunyuan
Routing-signature diversity prevents expert collapse and boosts MoE specialization
Proposes Distributional Orthogonalization Loss to directly record and penalize expert token-overlap, preventing collapse in MoE models.
Oct 2026 · University of Chinese Academy of Sciences, ByteDance
Splitting experts by role improves video diffusion model scaling
Splits experts into semantic and generic roles via prototype-guided routing, preventing collapse and promoting specialization in video diffusion.
1 further paper
Oct 2026 · Indian Institute of Technology Bombay, IBM Research
Soft routing among process experts helps PDE forecasting, but gains prove seed-dependent
Uses soft routing among process-biased experts for PDE forecasting, with seed-dependent gains.
Oct 2026 · Apple
Segment-level expert routing gives one model four inference budgets
Combines elastic model sizing with mixture-of-experts routing to enable a single model to run at multiple compute budgets, which relates to expert routing and preventing collapse.
5 of 7 papers shown