Training on visual errors improves image understanding and tool use

PerceptRL turns caption mistakes into checkable training questions and rewards accurate image inspection, reaching 71.6% average accuracy across seven benchmarks.

Chinese Tech

Tongkun Guan · Zhibo Yang · Mingkun Yang · Haocheng Wang · Wei Shen · Xiaokang Yang

Shanghai Jiao Tong University · Qwen

Research Digest·28 October 2026·2 min read

Illustration: Training on visual errors improves image understanding and tool use

In one line

PerceptRL converts caption errors into verifiable QA and corrects tool-use drift to sharpen MLLM visual perception.

Before you read this

Three ideas this paper builds on

fine-grained spatial reasoningReasoning precisely about object positions, orientations, and spatial relations across planar, depth, and temporal dimensions for robotics and navigation.51 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.130 papers →agentic reinforcement learningTraining approach that lets LLM-based agents learn multi-step tool use and planning via reinforcement learning, often with dense or per-tool rewards.98 papers →

Read it as

AI models that see and read often make small visual mistakes, like miscounting or color confusion. These perception errors cause wrong answers even with good reasoning. This limits multimodal models, which handle both images and text.

The researchers developed PerceptRL. It uses the model's caption mistakes to create targeted questions. Simple errors become yes/no or multiple-choice. For harder cases, it uses tools like cropping and corrects misaimed operations.

On seven visual tasks, PerceptRL raised average accuracy to 71.6%, outperforming the base model by 15.7 points. This means perception can be sharpened after initial training, making models more reliable for precision tasks like medical imaging or driving.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

fine-grained spatial reasoning reinforcement learning with verifiable rewards agentic reinforcement learning multi-turn tool use coordinate drift multi-agent architecture