The authors built RealtimeBench, comprising 180 tasks across six game types, including Mario, Snake and Tetris.
Live clocks reverse which language-model agents perform best in games
RealtimeBench shows that inference delays can erase reasoning advantages, while offline teacher guidance can improve a fast agent during live play.
Big Tech
Ningxin Pan · Hanyu Li · Yeqing Shen · Ning Ding · Yehui Tang · Wentao Zhang
Peking University · Samsung Research, Beijing, China
Research Digest··2 min read
Pan and colleagues evaluate language-model agents in games that continue changing while models reason, exposing costs hidden when environments pause for each response.
Why this paper
From Samsung Research, Beijing, China and Peking University · Part of Benchmark Ecological Validity, now 12 papers
In one line
Under a live clock, stronger slow reasoning loses to fast responses unless its insights are distilled offline into reusable instructions for a fast actor.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§