AI models struggle to track motion and update gameplay decisions

Across 20 Atari games, the authors find that vision-language models misread movement and sometimes repeat actions despite fresh visual feedback.

Big Tech

Mingjia Huo · Yao Fu · Bo Chang · Yaqing Wang · Dawei Zhu · Dylan Zhang · +5 more

University of California San Diego · Google DeepMind · Peking University · University of Illinois Urbana-Champaign

Research Digest·9 November 2026·2 min read

Thread:Process Agent Benchmarks

Illustration: AI models struggle to track motion and update gameplay decisions

In one line

State-of-the-art vision-language models fail to perceive motion and make sequential decisions in dynamic Atari gameplay, exposing a perception-to-action gap.

Before you read this

Three ideas this paper builds on

vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.242 papers →benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.328 papers →agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real278 papers →

Read it as

Computers that label images ace static quizzes, but the world moves. (VLMs) combine seeing and language, and are usually scored on still pictures. There is little evidence they can handle fast-changing video, where each moment demands a new decision.

AtariBench is a test built from classic Atari games. A VLM watches gameplay frames and chooses actions, with game speed separated from the model's own thinking time. This isolates whether the model actually tracks motion, direction, and shifting state.

Best models often misread movement, repeat old moves even after the screen changes, and rely on stale text instead of fresh frames. That gap matters for any real-time helper, from robots to assistants: passive description is not active decision-making.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

vision-language model benchmark agentic evaluation fine-grained spatial reasoning agent memory