The authors introduce Exact Quantile Balancing (EQB), a distributed method for setting expert-selection biases from the exact global-batch distribution of routing margins.
Exact quantiles and gradient corrections improve expert load balancing
The authors combine globally exact routing-bias updates with microbatch-specific gradient corrections in large mixture-of-experts models.
Industry
Pit Neitemeier · Jiaze Li · Alessio Serra · Philipp Scholl · Sohir Maskey
Aleph Alpha
Research Digest··2 min read
Neitemeier et al.
Why this paper
From Aleph Alpha
In one line
Exact Quantile Balancing and Load-Error Injection improve MoE load balance and performance over prior methods.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§