The authors first identify that current MoE training faces two issues: expert collapse (underutilization) and representation redundancy, and existing solutions like load-balancing losses or weight orthogonality address them in isolation or conflict.
Routing-signature diversity prevents expert collapse and boosts MoE specialization
The authors introduce DO-loss to penalize overlapping token assignments and REM to rebalance workloads without sacrificing specialization.
Chinese Tech
Jinfan He · Yunzhuo Liu · Kai Zhang · Weidong Han · Key · Rayying
Peking University · Tencent Hunyuan
Research Digest··3 min read
The paper proposes Distributional Orthogonalization Loss (DO-loss), a regularization that records each expert's token assignment history as a binary signature and penalizes overlap, directly encouraging functional specialization and self-correcting expert collapse.
Why this paper
From Tencent Hunyuan and Peking University
In one line
Distributional Orthogonalization loss and Replica Expert Mechanism prevent expert collapse and balance load, outperforming existing routing algorithms on MoE downstream tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§