Vision-language models, which process both images and text, often fail at a task called false belief: figuring out where someone thinks an object is, even if that belief is wrong. When the task is given as a story, these models do nearly as well as people. But when the same task is told through pictures, their performance drops sharply. Most researchers assumed the models lacked reasoning skills for beliefs.
This paper argues the real problem is perceptual: the models cannot reliably see what is happening in the images. The authors tested this by giving the models perfect written descriptions of the scenes. With those descriptions, the models did as well as or better than with text alone, proving they can reason correctly once they get the right visual information. The most common mistake was “reality bias,” reporting where an object actually is rather than where a missing agent believes it is, which accounted for 60% of failures.
This matters because it changes where to focus improvement. Instead of building better reasoning modules for beliefs, developers should improve how models extract structured information from visual scenes, such as tracking which characters are present at each moment. Fixing the perceptual pipeline could unlock human like social understanding in AI systems.