The Beyond3D isolates out-of-sight spatiotemporal reasoning: every question asks about an object that has been relocated and has since left the camera's field of view. It is built from HD-EPIC annotations using geometry-aware visibility tracks, which combine 3D object positions, camera pose, field of view, and scene geometry to determine when an object is visible, occluded, or out of view. The benchmark contains 9,000 questions in eight types across 135 videos from nine participants. Nine general-purpose and spatially specialized VLMs were evaluated. The best model scored 42.2%, random-chance baseline is 29.7%, and text-only baselines reach 31.9%. The largest failures were in recovering when an object was last visible. This indicates that current VLMs do not reliably maintain the spatial state of objects once they leave view.
New benchmark tests if AI can track objects after they vanish from view
Beyond3D challenges vision-language models to reason about relocated objects no longer visible in egocentric video.
Big Tech
Fangzhou Ma · Ivo Alexander Ban · Eren Homburg · Gabriele Goletto · Rémi Pautrat · Mahdi Rad · +2 more
ETH Zurich · Microsoft Spatial AI Lab · Bocconi University
Research Digest··2 min read
Thread:Object Permanence in VLMs
In one line
Vision-language models struggle to reason about objects moved out of sight, reaching only 42.2% on a new 9,000-question egocentric video benchmark.
Before you read this
Three ideas this paper builds onvision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.265 papers →benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →fine-grained spatial reasoningReasoning precisely about object positions, orientations, and spatial relations across planar, depth, and temporal dimensions for robotics and navigation.72 papers →
Read it as
Ma et al. introduce Beyond3D, a VQA benchmark with 9,000 questions over 135 egocentric videos that tests whether VLMs can track an object's location after it is moved and has left the field of view.
The authors constructed Beyond3D from the HD-EPIC dataset, which provides 3D scene reconstructions and annotations of object movements during unscripted cooking and everyday activities.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.