The authors analyzed three pretraining corpora and observed systematic shifts in lexical, topical, syntactic, and semantic properties over time.
LLM evaluation works best when treated as a forecasting problem.
The authors show that time-series forecasting of model accuracy beats pooling all benchmark scores when predicting future performance.
Top University
Rahul Thomas · Adam Block · Micah Goldblum
Columbia University
Research Digest··2 min read
d.
Why this paper
From Columbia University · Part of Benchmark Ecological Validity, now 6 papers
In one line
LLM evaluation should treat time as a fundamental variable; forecasting future performance beats standard pooled benchmarks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§