The paper introduces two surrogate , ScAn-Bench-LLM and ScAn-Bench-VLM, built from 4524 and 8024 checkpoints of language and training runs, with model sizes ranging from 16M to 1B parameters for LLMs and 2M to 357M for VLMs. These benchmarks approximate training outcomes quickly, replacing GPU-hours with CPU-seconds, enabling reproducible study of scaling analysis methods. The authors decompose scaling analysis into data acquisition and extrapolation phases and evaluate 9 settings over 1,000 training runs per benchmark, comparing existing methods across modalities. This is the first systematic evaluation of its kind, though specific results are not included in the provided text.
Surrogate benchmarks enable systematic evaluation of scaling analysis methods across model types
Authors introduce ScAn-Bench with thousands of checkpoints to test data acquisition and extrapolation for LLMs and VLMs
Academic
Artin Sermaxhaj · Nastaran Alipour · Donat Sinani · Johannes Hog · Neeratyoy Mallik · Jenia Jitsev · +1 more
University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
Research Digest··2 min read
In one line
ScAn-Bench introduces surrogate benchmarks and the first systematic evaluation of scaling analysis methodology across LLMs and VLMs.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.329 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.265 papers →
Read it as
The authors present two surrogate benchmarks, ScAn-Bench-LLM and ScAn-Bench-VLM, built from 4524 and 8024 checkpoints of language and vision-language model pipelines, respectively.
The authors collected training checkpoints from 1,000 training runs for each of two model families: LLMs (16M to 1B parameters) and VLMs (2M to 357M parameters).
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code