The authors constructed a suite of small behavioral evaluations designed to reveal a model's underlying computational strategy, distinguishing pattern-matching (parrot-like) from generalization (intelligence-like).
Language models can suddenly and repeatedly switch between pattern-matching and genuine generalization during pre-training
The authors build a toy eval suite that reveals this 'mode-hopping' behavior, which is not explained by standard optimization dynamics and is locally stable.
Big Tech
Jiaxin Wen (UC Berkeley) · Zhengxuan Wu (Stanford University, Google DeepMind) · Dawn Song (UC Berkeley) · Lijie Chen (UC Berkeley)
Research Digest··3 min read
The authors developed a behavioral test suite to distinguish whether language models rely on shallow patterns or true generalization.
Why this paper
From Google DeepMind and 2 others
In one line
LMs frequently and suddenly hop between pattern-matching and generalization during pre-training, known as mode-hopping.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§