Benchmark Ecological Validity
Investigates how well benchmarks reflect real-world LLM usage and how to design evaluation tasks that match actual deployment contexts.
6 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Oct 2026 · Leibniz University of Hanover, TIB – Leibniz Information Centre for Science and Technology and University Library
In-distribution accuracy does not predict abstract reasoning transfer in small models
Shows that in-distribution accuracy on abstract reasoning tasks does not predict transfer performance, highlighting a disconnect between benchmark scores and real generalization.
1 further paper
Oct 2026
Clinical AI benchmarks poorly reflect how clinicians actually use language models
Shows public clinical benchmarks mismatched with eight months of real clinician queries, motivating benchmark redesign.
1 further paper
Oct 2026 · University of Chicago, Microsoft
LLM creativity is best measured by performance on rare, long-tail outputs
Argues that creativity benchmarks are best understood as measuring long-tail performance, contributing to the design of evaluation tasks that reflect real-world creative output.
Oct 2026 · Columbia University
LLM evaluation works best when treated as a forecasting problem.
Demonstrates that web text non-stationarity invalidates i.i.d. evaluation and proposes treating accuracy as a time series for forecasting.
4 of 6 papers shown