Feedback control theory yields simpler MoE load balancer that adapts to shifting data

A pure integral controller restores expert balance after a change in training distribution, outperforming auxiliary losses and existing bias updates.

Academic

Yikun Hou · Alp Yurtsever

Umeå University

Research Digest·9 November 2026·3 min read

Illustration: Feedback control theory yields simpler MoE load balancer that adapts to shifting data

In one line

A pure integral feedback controller achieves the best load balance for MoE training under shifting data distributions.

Before you read this

Two ideas this paper builds on

mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.78 papers →continual learningA learning paradigm where a model sequentially acquires knowledge from multiple tasks without significantly forgetting earlier ones.64 papers →

Read it as

Large language models with a mixture-of-experts architecture use specialized sub-networks (experts) to save computation, but require load balancing to prevent a few experts from dominating the work. Current balancing methods fail when the training data shifts mid-course.

The researchers reframed this as a feedback control problem, introducing the PID-Balancer. By analyzing proportional, integral, and derivative terms, they found that an integral-only controller best removes persistent imbalance without amplifying batch-to-batch fluctuations in load.

The resulting method, PID (I), was tested on models with up to 64 experts. It restores balance tightly after a data shift and holds it better than existing methods, while maintaining the same model quality.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mixture of experts load balancing feedback control continual learning