Language models gain no advantage from synthetic warmups at equal compute

Across four preliminary training curricula, the authors find that spending the same compute on natural language matches or beats a separate warmup stage, even when text must be reused.

AI Startup

Alexander Stern · Tim Rocktäschel · Acyr Locatelli · Laura Ruis

Cohere · AI Centre, UCL · MIT

Research Digest·30 October 2026·2 min read

Illustration: Language models gain no advantage from synthetic warmups at equal compute

In one line

Synthetic pre-pretraining never outperforms natural-language training when compute is matched across tested regimes.

Before you read this

Three ideas this paper builds on

synthetic data generationCreating training examples artificially, often using an LLM, to overcome real-data scarcity or privacy constraints.47 papers →scaling lawA predictable power-law relationship between compute or data and model performance.18 papers →continual learningA learning paradigm where a model sequentially acquires knowledge from multiple tasks without significantly forgetting earlier ones.51 papers →

Read it as

Many language models need enormous amounts of natural text to train. Some researchers try to jump-start learning using synthetic data like code or artificial languages before natural language training. This "pre-pretraining" (PPT) aims to give models useful skills more efficiently.

The authors tested PPT against standard training on natural language alone across four synthetic data types. They carefully matched computation in each comparison and checked if reported gains depended on how model weights were initially set.

Under fair comparison, PPT never beat pure natural-language training. The best improvement was 5.4% in perplexity under a specific setup, but this advantage disappeared at 1 billion tokens of natural language. Models also forgot synthetic skills when switching to real text, so synthetic pre-training is not worthwhile for language models.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

pre-pretraining curriculum learning synthetic data generation weight initialization scaling law continual learning