scaling law
A scaling law is a mathematical formula that predicts how a model's performance will improve as you increase resources like model size, training data, or compute. Think of it as a recipe: if you know the quantity of your ingredients (parameters, tokens, compute) and how they scale, the law tells you what quality of result to expect, allowing you to plan experiments efficiently.
The papers show that scaling laws are emerging across diverse areas, from standard language model pretraining to reinforcement learning and physiological waveform modeling. However, they also reveal important nuances: the optimal mix of synthetic data increases with compute, and adding more tools helps more than adding more tasks. Crucially, one paper warns that a single best-fit curve can be dangerously misleading, proposing certified prediction intervals instead, while others extend laws to account for quantization or latent prediction errors in robotics.
- Papers
- 32
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Nov 2026
32 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Language models can use compute better by stopping large models early
Language model performance follows power-law scaling with model size, dataset size, and compute, making larger models more sample-efficient.
Nov 2026 - 2AI trained on simulated data can reconstruct unfamiliar medical scans
Training on physics-simulated synthetic data unlocks scaling laws for zero-shot MRI reconstruction.
Nov 2026 - 3Latent feedback cuts compute by up to 48% compared to training more epochs
Latent feedback amplifies compute and data, reaching a target loss with up to 48% less compute or 47% less unique data than more epochs.
Nov 2026
How fast it is moving
Often studied with
synthetic data generationCreating training examples artificially, often using an LLM, to overcome real-data scarcity or privacy constraints.4 togetherlarge language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.4 togethercontinual learningA learning paradigm where a model sequentially acquires knowledge from multiple tasks without significantly forgetting earlier ones.2 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Big Techcs.LG
Language models can use compute better by stopping large models early
Language model performance follows power-law scaling with model size, dataset size, and compute, making larger models more sample-efficient.
Johns Hopkins University, OpenAI · Nov 2026
- Independentcs.LG
Latent feedback cuts compute by up to 48% compared to training more epochs
Latent feedback amplifies compute and data, reaching a target loss with up to 48% less compute or 47% less unique data than more epochs.
Nov 2026
- Top Universityeess.IV
AI trained on simulated data can reconstruct unfamiliar medical scans
Training on physics-simulated synthetic data unlocks scaling laws for zero-shot MRI reconstruction.
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen University · Nov 2026
- Top Universitycs.LG
Language models given hints make larger models’ performance more predictable
Adding graded hints to a benchmark lifts small models off zero accuracy, making downstream accuracy from large training runs far more predictable.
Stanford University · Nov 2026
- Top Universitycs.LG
Language models can stretch limited training data by masking inputs
Masked-input regularization improves language model pretraining under data scarcity, and SoftQ scaling law better captures repeated data.
University of Michigan, KAIST · Nov 2026
- Academiccs.LG
Language models can worsen when trained repeatedly on limited data
Beyond a data-dependent threshold, further data repetition is counterproductive and compute should scale model capacity.
Cornell University · Nov 2026
- Big Techcs.LG
AI agents learn more from varied tools than extra tasks
To scale tool-use agent training data, grow tool and database diversity, not just task count.
Apple · Oct 2026
- Top Universitycs.LG
Language models can worsen with more compute when data stays fixed
Hyperparameter transfer and ensembling achieve monotonic loss improvements in data-constrained scaling, outperforming prior recipes.
Princeton University, Q Labs · Nov 2026
- Top Universitycs.LG
Averaging several models beats one bigger model when training data is fixed
Averaging the output scores of independently trained models suppresses the late-training rise in validation loss caused by reusing data across epochs, and pushes the optimal stopping time later.
University of Texas at Austin, Carnegie Mellon University · Nov 2026
- Big Techcs.LG
AI agents’ performance can be predicted using 100-fold smaller training runs
Reinforcement learning follows predictable scaling laws, with Elo scores enabling performance extrapolation up to two orders of magnitude.
Google DeepMind · Oct 2026
- Big Techcs.AI
Language models’ safety monitors improve predictably with size and labeled examples
Safety probe error follows two simple scaling laws that predict performance from model size and label count.
University of Washington, Anthropic · Oct 2026
- Top Universitycs.LG
Scaling laws predict when synthetic rewrites improve language model training
The optimal fraction of synthetic rewrites in pretraining grows with compute and is predictable from a scaling law, and rewritten models can beat their rewriter.
Princeton University, Stanford University · Oct 2026
- Big Techcs.LG
AI models factor larger numbers, but costs grow exponentially
A transformer using discrete diffusion and beam search factors 68-bit semiprimes but scales exponentially with bit length.
Google DeepMind · Nov 2026
- Big Techcs.LG
AI models’ performance forecasts can diverge despite similarly good data fits
Standard scaling law fitting suffers from a Rashomon effect that a branch-and-bound algorithm resolves.
University of Cambridge, Google DeepMind · Oct 2026
- Top Universitycs.CL
Continuing long-context pre-training keeps improving models, and gains are predictable
Long-context continual pre-training gains follow an additive power law and show no saturation within evaluated budgets, so scaling it further is worthwhile.
Tsinghua University, OpenBMB · Oct 2026
- Top Universitycs.AI
Better static judges do not guarantee better training signals for generative reward models.
Answer-only generative reward models outperform thinking models as online policy trainers despite lower static accuracy.
SynthLabs.ai, Stanford University · Oct 2026
- Big Techcs.LG
Diffusion language models are cheaper to train than the widely cited 16x compute gap suggests
The widely cited 16x compute gap between diffusion and autoregressive language models shrinks to as low as 3.0x with modern training methods.
Cornell University, Cornell Tech · Oct 2026
- Top Universitycs.AI
Latent world models for robotics scale predictably with compute
Latent robotic world model quality and downstream planning success follow a second-order power law in training compute across 12 embodiments.
FAIR at Meta, Chandar Research Lab · Oct 2026
- Big Techcs.LG
New scaling law better predicts losses from low-precision pretraining
A scaling law for mixed-precision quantized pretraining achieves up to 11x lower loss prediction error than prior laws.
Seoul National University, Amazon Annapurna Labs · Oct 2026
- Academiccs.LG
Scaling law for physiological waveforms favors more training data over larger models
Larger physiological waveform models and more pretraining hours reduce the need for labeled clinical data.
University of North Carolina at Chapel Hill · Oct 2026
- Independentcs.CL
Above 7B parameters, caving in to pushback can be partly switched off
Capitulation to user pushback becomes a detectable and causally active linear direction in language models only above 7 billion parameters.
Independent researcher · Nov 2026
- Big Techcs.AI
Larger language models can lose more retrieval ability after compression
Larger language models are more vulnerable to compression because their internal compensation for damaged circuits weakens as scale increases.
Indraprastha Institute of Information Technology, Delhi, TurboML · Nov 2026
- Independentcs.IR
Language model that indexes 200 million web pages outperforms embedding search for navigation
Generative retrieval with a 4B-parameter model indexes 200 million web pages and outperforms dense retrieval on navigational search for unseen queries.
Oct 2026
- Big Techcs.CV
Learning visual concepts before rendering improves image generation in controlled tests
Visual generation is better when autoregressive pretraining learns semantic tokens and diffusion post-training renders them.
University of California, San Diego, Adobe Research · Oct 2026
- Independentcs.LG
Language models’ own confidence becomes harder to outperform as they improve
Performance prediction only beats a model's own confidence when the model is suboptimal or the predictor adds significant capacity.
Oct 2026
- Top Universitycs.LG
More pretraining data flips which optimizer yields better fine-tuned models
Beyond a large enough pretraining data budget, AdamW beats Muon after fine-tuning even though Muon reaches lower pretraining loss, and the crossover budget shrinks as models grow.
Carnegie Mellon University · Nov 2026
- AI Startupcs.CL
Language models gain no advantage from synthetic warmups at equal compute
Synthetic pre-pretraining never outperforms natural-language training when compute is matched across tested regimes.
Cohere, AI Centre, UCL · Oct 2026
- Top Universitycs.CV
Larger vision-language models can be worse on specific subtasks
In vision-language model families, larger models can underperform smaller ones on specific subtasks, including spatial reasoning, visual perception, and STEM questions.
University of Edinburgh, Imperial College London · Oct 2026
- Top Universitycs.CL
AI mathematical reasoning improves more with textbook structure than with isolated problems
Training on structured textbook proofs raises a language model’s average mathematics score by 16.7 percentage points and beats the tested instruction-data baselines overall.
University of Science and Technology of China, Fudan University · Oct 2026
- Top Universitycs.AI
Scaling gameplay training improves familiar worlds more than distant ones
Scaling data, diversity, and model size during continued pre-training improves embodied agents in familiar and related worlds but not distant worlds.
miHoYo Honkai AI R&D Team, Tsinghua University · Oct 2026
- Top Universitycs.LG
Reinforcement learning reshapes which reasoning strategies language models prefer
Reinforcement learning primarily reweights existing reasoning strategies rather than expanding an LLM's reasoning boundary.
University of California, Berkeley · Oct 2026
- Academiccs.AI
AI models can align language and vision using 42-fold fewer samples
Joint-Embedding Predictive Architectures with good views provably learn the same representation across modalities up to a rotation.
Brown University, AMI Labs · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.