The authors trained a model ladder spanning parameter count, training data, recurrent passes and Mixture-of-Experts sparsity.
Looping and sparse experts provide complementary routes to efficient scaling
A unified scaling law predicts how recurrent computation and expert sparsity interact, then guides model design under compute and memory limits.
Big Tech
Yanbei Chen · Anirudh Goyal · Raghuraman Krishnamoorthi
Meta AI
Research Digest··3 min read
Chen, Goyal and Krishnamoorthi model two ways to scale transformers efficiently: repeatedly applying shared layers, called looping or recurrence, and activating only a subset of a larger pool of experts.
Why this paper
From Meta AI
In one line
Looped MoE scaling follows a bounded, sparsity-conditional recurrence law that subsumes dense and MoE scaling laws and predicts held-out loss.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§