data-constrained scaling
- Papers
- 3
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Nov 2026
3 papers in the last two months, against 0 in the two before.
Start here
Two papers to read first, in the order they came out- 1Repeated training tokens lose value according to a simple scaling rule
Repeated-token value follows a simple scaling variable, but ordering, allocation, source entropy, and tokenization also determine finite-data pretraining loss.
Oct 2026 - 2Language models can worsen with more compute when data stays fixed
Hyperparameter transfer and ensembling achieve monotonic loss improvements in data-constrained scaling, outperforming prior recipes.
Nov 2026
How fast it is moving
Often studied with
compute-optimal scalingThe practice of scaling model size and data volume to achieve optimal performance for a given compute budget.1 togetherdata repetitionReusing the same training samples across epochs, affecting overfitting, model degradation, and scaling-law predictions in pretraining.1 togetherscaling lawA predictable power-law relationship between compute or data and model performance.1 togetherThe papers
Most central to this idea first, not most recent.
- Top Universitycs.LG
Language models can worsen with more compute when data stays fixed
Hyperparameter transfer and ensembling achieve monotonic loss improvements in data-constrained scaling, outperforming prior recipes.
Princeton University, Q Labs · Nov 2026
- Independentcs.CL
Repeated training tokens lose value according to a simple scaling rule
Repeated-token value follows a simple scaling variable, but ordering, allocation, source entropy, and tokenization also determine finite-data pretraining loss.
Oct 2026
- Top Universitycs.LG
Smaller batches and adapted recipes improve repeated-data pretraining
For repeated-data pretraining, smaller batches, repacked sequences, tied embeddings, and positive embedding weight decay improve validation loss; GPUs and time should be treated separately.
UC Berkeley, Alien Institute for AI · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.