AI models answer image questions better when taught where to look

Wang and colleagues report that directly rewarding useful image crops improves visual question answering across two model sizes and three high-resolution benchmarks.

Chinese Tech

Junfeng Wang · Jiawei Liu · Yongchao Xu · Tao Jiang · Jiangbo Ai · Jin Zhang · +2 more

University of Science and Technology of China · Alibaba Group

Research Digest·4 November 2026·2 min read

Illustration: AI models answer image questions better when taught where to look

In one line

Rewarding zoom-in decisions directly, instead of only final answers, improves vision-language models' visual question answering.

Before you read this

Three ideas this paper builds on

credit assignmentDetermining which actions or steps in a sequence contributed to a final outcome, often for reinforcement learning.179 papers →agentic reinforcement learningTraining approach that lets LLM-based agents learn multi-step tool use and planning via reinforcement learning, often with dense or per-tool rewards.109 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.169 papers →

Read it as

Answering questions about images often means zooming into the right spot. Models miss answers because they look at the wrong place, and existing training rewarded only the final answer, so a bad zoom could look fine if a lucky guess followed.

The new method adds a separate reward for whether a zoomed image contains enough evidence, using reinforcement learning where models learn from rewards. When the model is unsure where to look, it tries alternative crops, crediting only the crop choice, not later reasoning.

On three high-resolution benchmarks, this lifted accuracy by up to 15.8 percentage points for a small model and also improved a larger one, matching much larger systems. This helps models gather the right evidence before answering.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

observation policy optimization credit assignment agentic reinforcement learning uncertainty-guided observation branching evidence-localized advantage attribution group relative policy optimization