compute-optimal scaling
When training a large neural network, you have a fixed budget of compute, say, time or energy. Compute-optimal scaling asks how to best split that budget between making the model bigger and giving it more unique training examples. Using the same data more than once (repetition) usually gives diminishing returns compared to showing fresh examples. The goal is to find the sweet spot that minimizes the model's error for the compute you spend.
Experiments across these papers show that the value of repeating data drops quickly after just a few passes, especially as the model grows. For multimodal models, those handling text, images, and audio, the optimal allocation differs by modality: text and vision benefit from scaling both parameters and data together, while audio prefers more parameters over more data. Visual content can boost performance on spatial tasks without harming language ability, and repeating vision or audio data beyond about two passes gives negligible benefit.
- Papers
- 3
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Oct 2026
3 papers in the last two months, against 0 in the two before.
Start here
The paper to read first- 1Repeated training tokens lose value according to a simple scaling rule
Repeated-token value follows a simple scaling variable, but ordering, allocation, source entropy, and tokenization also determine finite-data pretraining loss.
Oct 2026
How fast it is moving
Often studied with
data repetitionReusing the same training samples across epochs, affecting overfitting, model degradation, and scaling-law predictions in pretraining.2 togetherfine-grained spatial reasoningReasoning precisely about object positions, orientations, and spatial relations across planar, depth, and temporal dimensions for robotics and navigation.1 togethermixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.1 togetherThe papers
Most central to this idea first, not most recent.
- Independentcs.CL
Repeated training tokens lose value according to a simple scaling rule
Repeated-token value follows a simple scaling variable, but ordering, allocation, source entropy, and tokenization also determine finite-data pretraining loss.
Oct 2026
- Chinese Techcs.CL
Vision scales multimodal learning without altering pure-text loss
Visual content in native multimodal pre-training improves text prediction on image-text data and spatial reasoning via text, scaling with compute while preserving language performance.
The Chinese University of Hong Kong, Tencent · Oct 2026
- Top Universitycs.CL
Multimodal training needs different scaling recipes for text, vision and audio
Scaling laws for multimodal pretraining reveal modality-specific compute optima, asymmetric interference, and saturation of repeated non-text data.
MILA - Quebec AI Institute, McGill University · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.