The authors propose Distributional Orthogonalization loss (DO-loss), which penalizes overlap in expert token assignment histories to encourage specialization and prevent expert collapse. They also introduce the Replica Expert Mechanism (REM) for load balancing through adaptive replica placement and real-time token dispatching, decoupled from the routing objective.
Evaluated on 4.8B and 30B MoE models, the method outperformed all baselines across six downstream benchmarks while maintaining comparable training throughput.
This work is interesting because it decouples system efficiency from routing objectives, treating load imbalance as an infrastructure problem rather than a constraint on specialization, allowing each concern to be addressed with separate mechanisms.