Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
AI & ML

Sparse mixture-of-experts models overfit repeated training data faster

Across models with 80 million to 1 billion active parameters, greater sparsity made repeated-data training degrade sooner, although strong regularization reduced the damage.

PaperTop Universitycs.LGarXiv:2609.11917v1 ↗
Atindra Jha · Margaret Li · Jure Leskovec · Percy Liang · Luke Zettlemoyer

Stanford University · Paul G. Allen School of Computer Science, University of Washington

Research Digest··2 min read
Read the paper →
Jha et al. trained dense Transformers and mixture-of-experts models under varying data-repetition rates, domain mixtures, expert counts, and expert granularities. They find that MoE models lose performance more rapidly than dense models as examples are reused, with susceptibility tracking total parameter count rather than the number of parameters activated per token. Strong masking-based regularization preserves an MoE advantage beyond 64 repetitions, but does not match training on unique data.

What they did

The authors compared densely activated Transformers with sparse mixture-of-experts models, which route each token through only a subset of specialized parameter blocks. Experiments covered single- and multi-domain data mixtures and models ranging from 80 million to 1 billion active parameters, including MoEs with as many as 8.5 billion total parameters.

They varied how often training data was repeated, as well as expert count and granularity. They also tested existing regularization methods and examined routing stability and expert specialization as possible mechanisms behind repeated-data overfitting.

Key findings

  • Dense 80-million-parameter models tolerated more than 8 repetitions with little degradation, whereas MoEs began to suffer at 4 repetitions.
  • By 32 repetitions, MoEs had lost their advantage from all-unique-data training and underperformed dense models.
  • Vulnerability increased with sparsity and was associated with total parameter count, not merely the number of parameters active for each token.
  • Dropout mitigated overfitting, while strong masking-based regularization allowed MoEs to outperform dense models beyond 64 repetitions; no tested method recovered all-unique-data performance.

Why it matters

MoEs are attractive because they increase total model capacity without proportionally increasing computation per token. These results indicate that this capacity can become a liability when limited corpora must be reused, making data repetition and regularization important considerations when selecting sparse architectures.

Caveats

The evidence covers models up to 1 billion active and 8.5 billion total parameters, so it remains uncertain whether the same thresholds hold for much larger production systems. Early routing stabilization and expert specialization correlate with overfitting, but the reported analyses do not establish that either mechanism alone causes the degradation, and none of the tested remedies fully substitutes for unique data.

§
newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe