data repetition
When training large language models, developers often run out of unique text and are forced to reuse the same examples, this is data repetition. Think of it like studying for an exam by rereading the same textbook chapter multiple times. At first it helps, but eventually you stop learning and might even start memorizing irrelevant details. This is a practical problem because high-quality training data is finite, and models keep growing.
The papers show that repeating data consistently hurts performance, especially for certain architectures. Mixture-of-Experts models degrade sooner than dense ones, and larger models are more sensitive to repeats. The penalty follows predictable scaling rules: a second epoch is still valuable, but after a few repetitions, benefits drop sharply. Different data types also matter, repeating vision or audio beyond two passes is wasteful, and mixing modalities creates asymmetric interference. Regularization can help, but never fully replaces fresh data.
- Papers
- 4
- Released code
- 0
- First seen
- Sept 2026
- Latest
- Nov 2026
3 papers in the last two months, against 1 in the two before.
Start here
Three papers to read first, in the order they came out- 1Sparse mixture-of-experts models overfit repeated training data faster
MoE models degrade more rapidly than dense models under data repetition, with overfitting increasing with sparsity.
Sept 2026 - 2Repeated training tokens lose value according to a simple scaling rule
Repeated-token value follows a simple scaling variable, but ordering, allocation, source entropy, and tokenization also determine finite-data pretraining loss.
Oct 2026 - 3Language models can worsen when trained repeatedly on limited data
Beyond a data-dependent threshold, further data repetition is counterproductive and compute should scale model capacity.
Nov 2026
How fast it is moving
Often studied with
compute-optimal scalingThe practice of scaling model size and data volume to achieve optimal performance for a given compute budget.2 togethermixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.2 togetheroverfittingThe modeling error from fitting training data too precisely, causing poor generalization to new data; often countered by regularization.2 togetherThe papers
Most central to this idea first, not most recent.
- Independentcs.CL
Repeated training tokens lose value according to a simple scaling rule
Repeated-token value follows a simple scaling variable, but ordering, allocation, source entropy, and tokenization also determine finite-data pretraining loss.
Oct 2026
- Top Universitycs.LG
Sparse mixture-of-experts models overfit repeated training data faster
MoE models degrade more rapidly than dense models under data repetition, with overfitting increasing with sparsity.
Stanford University, Paul G. Allen School of Computer Science, University of Washington · Sept 2026
- Academiccs.LG
Language models can worsen when trained repeatedly on limited data
Beyond a data-dependent threshold, further data repetition is counterproductive and compute should scale model capacity.
Cornell University · Nov 2026
- Top Universitycs.CL
Multimodal training needs different scaling recipes for text, vision and audio
Scaling laws for multimodal pretraining reveal modality-specific compute optima, asymmetric interference, and saturation of repeated non-text data.
MILA - Quebec AI Institute, McGill University · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.