Scaling large language models with mixture-of-experts (MoE) requires extremely sparse routing, but expert load imbalance worsens as sparsity increases, reducing parameter utilization and training stability. This paper proposes ID Balancing, a load control method inspired by PID controllers. It unifies two previous auxiliary-loss-free methods (DeepSeek's loss-free and Kimi K3's Quantile Balancing) as incomplete PID controllers, then introduces an integral-derivative controller that scales corrections with load error and activates derivative action only when imbalance worsens. Tested on Top-10, Top-5, and Top-3 routing over 768 experts, ID Balancing reduces worst-case backbone MaxVio and training-average backbone MinVio by over 50% and 12% respectively in the most challenging Top-3 setting. When scaling total parameters from 18.9B to 69.9B under Top-10-of-768 routing, its worst-case MaxVio remains nearly unchanged and is about 89.6% lower than an auxiliary-loss baseline. The method maintains competitive language-modeling and downstream performance, with advantages that grow as sparsity increases, making it promising for training larger, sparser MoE models.
Feedback control stabilizes expert loads in extremely sparse language models
ID Balancing adjusts routing biases according to both load-error magnitude and worsening imbalance, improving expert utilization without an auxiliary balancing loss.
Chinese Tech
Peng Jin · Zihan Qiu · Zekun Wang · Bo Zheng · Yang Xu · Tian Xie · +5 more
Qwen Team · Alibaba Token Hub · Alibaba Group
Research Digest··2 min read
In one line
ID Balancing reduces expert load imbalance in extremely sparse MoE training using an integral-derivative controller.
Before you read this
Two ideas this paper builds onRead it as
Jin and colleagues recast existing mixture-of-experts load-balancing methods as forms of proportional, integral and derivative feedback control.
The authors studied sparse mixture-of-experts models, in which a router sends each token to a small subset of available experts.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.