The authors created Realtime-Bench, a collection of 50 dynamic game and web tasks designed to test temporal state awareness, adaptive deliberation and time-aware action selection.
Frontier models struggle when correct actions must arrive on time
Realtime-Bench shows that inference delays and prolonged reasoning substantially hinder agents in dynamic games and web tasks.
Big Tech
Jingyu Tang · Xu Yang · Zhehao Zhang · Zheng Wang · Qi Cheng · Haozhe Zhao · +4 more
University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign · NVIDIA · Northwestern University
Research Digest··3 min read
Tang and colleagues evaluate 12 frontier models on 50 interactive tasks where environments keep changing and late actions can fail.
Why this paper
From NVIDIA and 5 others
In one line
On dynamic real-time interaction tasks, the best frontier model succeeds only 44% of the time under wall-clock constraints.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§