Multimodal large language models (MLLMs) can read text in images nearly perfectly but still struggle to solve reasoning problems presented as images, even though they solve the same problems easily when given as plain text. This is not a failure to see; it is a failure to route the visual information into the model’s own existing reasoning system. The authors propose TIGER, a method that automatically finds cases where the model succeeds on text but fails on the corresponding image, then retrains the model on those images to close the gap. After training, the model improved across many real-world visual reasoning benchmarks, including geometry diagrams, science problems, charts, and spatial reasoning tasks, despite never seeing real images during training.
AI models can read pictured problems yet fail to solve them
Lau and colleagues report that training on problems models solve as text but fail as images improves reasoning across visual benchmarks, including diagrams, charts and spatial tasks.
Top University
Gregory Kang Ruey Lau · Minh Huynh Nguyen · Bryan Kian Hsiang Low
National University of Singapore · CNRS@CREATE
Research Digest··2 min read
Thread:Visually Grounded Reasoning
In one line
Multimodal LLMs fail to reason on images they can read because a reasoning-access gap blocks visual input from using text reasoning machinery.
Before you read this
Three ideas this paper builds ongroup relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.180 papers →counterfactual reasoningComparing actual outcomes with imagined alternatives to estimate the effects of different actions or conditions.104 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.245 papers →
Read it as
Lau and colleagues show that multimodal language models can correctly transcribe a pictured reasoning problem yet struggle to solve it directly from the image.
The authors presented semantically identical reasoning problems as ordinary text and as images containing that text.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.