The authors trained dense and Mixture-of-Experts (MoE) language models at 80M to 1B active parameters, varying rates from 1x to 64x across several domains. They found that MoEs begin to degrade at 4x repetition while dense models tolerate 8x, and at 32x MoEs underperform dense models. worsens with MoE sparsity, driven by total parameters. Routing stabilizes early and expert specialization rises with repetition. This is important because data repetition is common as unique text is exhausted, and MoE architectures are widely used for compute efficiency. The paper identifies a clear vulnerability and shows that regularization like dropout can mitigate but not fully replace unique data.
Sparse mixture-of-experts models overfit repeated training data faster
Across models with 80 million to 1 billion active parameters, greater sparsity made repeated-data training degrade sooner, although strong regularization reduced the damage.
Top University
Atindra Jha · Margaret Li · Jure Leskovec · Percy Liang · Luke Zettlemoyer
Stanford University · Paul G. Allen School of Computer Science, University of Washington
Research Digest··2 min read
In one line
MoE models degrade more rapidly than dense models under data repetition, with overfitting increasing with sparsity.
Before you read this
Three ideas this paper builds ondata repetitionReusing the same training samples across epochs, affecting overfitting, model degradation, and scaling-law predictions in pretraining.3 papers →mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.66 papers →overfittingThe modeling error from fitting training data too precisely, causing poor generalization to new data; often countered by regularization.3 papers →
Read it as
Jha et al. trained dense Transformers and mixture-of-experts models under varying data-repetition rates, domain mixtures, expert counts, and expert granularities.
The authors compared densely activated Transformers with sparse mixture-of-experts models, which route each token through only a subset of specialized parameter blocks.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.