Training on interfaces helps visual models inspect images beyond screens

Across two model backbones, spatially supervised reinforcement learning improved non-interface perception beyond supervised training, while rewards based only on final answers did not reliably transfer.

Chinese Tech

Ruilin Yao · Xinyu Qiu · Tianyu Zou · Yi Rong · Shengwu Xiong · Shuheng Shen · +1 more

Wuhan University of Technology · Institute of Automation, Chinese Academy of Sciences (CAS) · Zhejiang University · Ant Group

Research Digest·28 October 2026·3 min read

Illustration: Training on interfaces helps visual models inspect images beyond screens

In one line

GUI grounding supervision trains vision-language models to inspect images sequentially, improving perception beyond GUI tasks.

Before you read this

Three ideas this paper builds on

ui groundingMapping natural language commands or descriptions to specific visual elements of a user interface.20 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.200 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.221 papers →

Read it as

(AI that understand both images and text) struggle with active looking, deciding where to focus and zoom in on large images. Training them usually requires specialized data for each task, which is expensive.

The authors used GUI grounding (teaching models to locate icons on screen) as training. They first gave many examples of correct zoom sequences, then used (trial-and-error) with rewards for covering targets to refine the strategy.

When tested on natural images and charts, the models improved accuracy by about 3 and 7 percentage points over baseline training. This shows interface-based practice can teach general visual inspection skills, potentially reducing need for task-specific data.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

ui grounding vision-language model reinforcement learning supervised fine-tuning trajectory transfer fine-grained spatial reasoning