Mixture-of-Experts (MoE) training requires both global and local load balance to prevent expert under-utilization and ensure efficient execution. Existing methods like Quantile Balancing (QB) approximate global quantiles, while auxiliary losses like GShard use soft targets. This paper proposes two complementary techniques: Exact Quantile Balancing (EQB) and Load-Error Injection (LEI).
EQB computes exact global-batch quantiles using a two-pass radix selection with negligible communication overhead, eliminating the shard-dependence of previous QB variants. LEI directly injects local load errors into router gradients, providing a stronger local balancing signal than the GShard loss.
In experiments on 7.5B-parameter MoE models trained up to 500B tokens, EQB improved global balance and downstream performance over naive QB, while LEI improved local balance and matched or exceeded the model quality of GShard. The techniques are complementary, addressing both global and local imbalance without requiring extra hyperparameters.