← All concepts

compute-optimal scaling

When training a large neural network, you have a fixed budget of compute, say, time or energy. Compute-optimal scaling asks how to best split that budget between making the model bigger and giving it more unique training examples. Using the same data more than once (repetition) usually gives diminishing returns compared to showing fresh examples. The goal is to find the sweet spot that minimizes the model's error for the compute you spend.

Experiments across these papers show that the value of repeating data drops quickly after just a few passes, especially as the model grows. For multimodal models, those handling text, images, and audio, the optimal allocation differs by modality: text and vision benefit from scaling both parameters and data together, while audio prefers more parameters over more data. Visual content can boost performance on spatial tasks without harming language ability, and repeating vision or audio data beyond about two passes gives negligible benefit.

Papers
3
Released code
0
First seen
Oct 2026
Latest
Oct 2026

3 papers in the last two months, against 0 in the two before.

Start here

The paper to read first
  1. 1Repeated training tokens lose value according to a simple scaling rule

    Repeated-token value follows a simple scaling variable, but ordering, allocation, source entropy, and tokenization also determine finite-data pretraining loss.

    Oct 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.