The authors built a toy evaluation suite to track whether language models rely on shallow patterns or generalizable reasoning during pre-training. Across multiple model scales, they observed that LMs suddenly switch between parrot-like (pattern-matching) and intelligence-like (generalizing) computations, a phenomenon they call mode-hopping. These switches are not explained by standard optimization dynamics or checkpoint averaging, and they persist well beyond typical training budgets. The results challenge the common assumption that LMs gradually and stably mature into generalizers. The authors demonstrate two practical applications: selecting intermediate checkpoints that generalize better after post-training on reasoning and alignment tasks, and selecting pre-training data to stabilize generalization dynamics.
Language models can suddenly and repeatedly switch between pattern-matching and genuine generalization during pre-training
The authors build a toy eval suite that reveals this 'mode-hopping' behavior, which is not explained by standard optimization dynamics and is locally stable.
Big Tech
Jiaxin Wen (UC Berkeley) · Zhengxuan Wu (Stanford University, Google DeepMind) · Dawn Song (UC Berkeley) · Lijie Chen (UC Berkeley)
Research Digest··3 min read
In one line
LMs frequently and suddenly hop between pattern-matching and generalization during pre-training, known as mode-hopping.
Read it as
The authors developed a behavioral test suite to distinguish whether language models rely on shallow patterns or true generalization.
The authors constructed a suite of small behavioral evaluations designed to reveal a model's underlying computational strategy, distinguishing pattern-matching (parrot-like) from generalization (intelligence-like).
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.