Language models can use compute better by stopping large models early

Kaplan and colleagues find predictable relationships between prediction loss, model size, data and training compute, favoring large models stopped well before convergence.

Big Tech

Jared Kaplan · Sam McCandlish · Tom Henighan · Tom B. Brown · Benjamin Chess · Rewon Child · +4 more

Johns Hopkins University · OpenAI

Research Digest·8 November 2026·2 min read

Thread:Compute-Efficient Scaling

Illustration: Language models can use compute better by stopping large models early

In one line

Language model performance follows power-law scaling with model size, dataset size, and compute, making larger models more sample-efficient.

Before you read this

Two ideas this paper builds on

scaling lawA predictable power-law relationship between compute or data and model performance.29 papers →overfittingThe modeling error from fitting training data too precisely, causing poor generalization to new data; often countered by regularization.6 papers →

Read it as

Training large language models is expensive and requires careful choices about how many parameters, how much data, and how long to train. We lack clear rules for balancing these factors.

This work measured how the error (loss) of a language model changes as the model size, data size, and compute are varied across many orders of magnitude. The results fit simple power-law equations: each factor has a predictable effect.

The main finding is that larger models use data far more efficiently. For a fixed budget of compute, the best approach is to train a very large model on relatively little data and stop well before the error stops improving. This suggests that pushing model size further will yield bigger gains than collecting more data.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

scaling law compute-efficient training sample efficiency overfitting