, tracking ML experiment parameters), behavioral modeling (imitating chess-player styles), and forecasting (predicting events from historical data).
Benchmark reveals long-context models falter on state tracking and pattern learning
The LEAPS suite of 12 datasets tests abilities beyond recall, showing that even top LLMs degrade sharply with longer inputs.
Top University
Howard Yen · Nitya Mani · Alex Renda · Christopher S. Hardin
Princeton University · Jane Street
Research Digest··2 min read
The authors introduce LEAPS, a collection of 12 novel datasets designed to evaluate long-context language models on state tracking and pattern learning, capabilities often overlooked by existing recall-focused benchmarks.
§