The authors examined the final fitting stage of scaling law estimation, using synthetic datasets and four empirical scaling ladders from the literature.
Equally good scaling law fits can predict very different futures
Mlodozeniec and Frei demonstrate ambiguity across synthetic data and four published scaling datasets, and propose an algorithm that certifies fit quality and prediction ranges.
Big Tech
Bruno Mlodozeniec · Spencer Frei
University of Cambridge · Google DeepMind
Research Digest··2 min read
Neural scaling laws forecast how model performance changes with training resources, but finding a good fit does not necessarily identify a reliable forecast.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.