The authors studied sparse mixture-of-experts models, in which a router sends each token to a small subset of available experts.
Feedback control stabilizes expert loads in extremely sparse language models
ID Balancing adjusts routing biases according to both load-error magnitude and worsening imbalance, improving expert utilization without an auxiliary balancing loss.
Chinese Tech
Peng Jin · Zihan Qiu · Zekun Wang · Bo Zheng · Yang Xu · Tian Xie · +5 more
Qwen Team · Alibaba Token Hub · Alibaba Group
Research Digest··2 min read
Jin and colleagues recast existing mixture-of-experts load-balancing methods as forms of proportional, integral and derivative feedback control.
Why this paper
From Qwen Team and 2 others
In one line
ID Balancing reduces expert load imbalance in extremely sparse MoE training using an integral-derivative controller.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§