The authors formalized training spikes as sharp surges in loss that, unlike ordinary noise, depend on model initialization and data shuffling.
Training spikes in language models replicated by simple quadratic models with heavy-tailed data
The authors show that power-law random feature models, a class of convex linear regression models, reproduce the spiky loss behavior observed during pre-training, and their simulation framework predicts spikes from a given checkpoint.
Big Tech
Ran Tian · Courtney Paquette
Google Deepmind
Research Digest··3 min read
The authors introduce Power-Law Random Feature (PLRF) models—simple quadratic models with heavy-tailed, power-law distributed inputs—that exhibit the same intermittent training spikes seen in large language model (LLM) pre-training.
Why this paper
From Google Deepmind
In one line
Training spikes in deep neural networks reproduce with simple quadratic models on heavy-tailed power-law inputs, enabling spike prediction and mitigation.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§