Language models can worsen when trained repeatedly on limited data

Lovelace and colleagues find that, beyond a data-dependent threshold, training compute is better spent on larger models than on additional passes through the same text.

Academic

Justin Lovelace · Christian Belardi · Srivatsa Kundurthy · Shriya Sudhakar · Kilian Q. Weinberger

Cornell University

Research Digest·8 November 2026·2 min read

Thread:Compute-Efficient Scaling

Illustration: Language models can worsen when trained repeatedly on limited data

In one line

Beyond a data-dependent threshold, further data repetition is counterproductive and compute should scale model capacity.

Before you read this

Three ideas this paper builds on

scaling lawA predictable power-law relationship between compute or data and model performance.29 papers →data repetitionReusing the same training samples across epochs, affecting overfitting, model degradation, and scaling-law predictions in pretraining.4 papers →overfittingThe modeling error from fitting training data too precisely, causing poor generalization to new data; often countered by regularization.6 papers →

Read it as

When training large AI models, high-quality data is often scarce. Reusing the same data (repetition) can cause , where the model memorizes instead of learning. This paper tackles how to best train with limited data.

The authors created a formula () that adds a penalty for overfitting due to . They tested it on hundreds of models with varied sizes, unique data amounts, and multiple repetitions.

They found that beyond a threshold, repeating data hurts performance. At that point, it is better to use a larger model. They also showed that strong weight decay, a technique to prevent overfitting, cuts the overfitting penalty by about 70%, providing clear training guidance.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

scaling law data-constrained training data repetition overfitting compute-optimal allocation weight decay