Fri 13 Nov
LiveGet it free
ZOTPAPER

News without the noise

← All concepts

ui grounding

User interface grounding teaches an artificial intelligence to understand what it sees on a screen and interact with elements like buttons or text boxes. It combines screenshots with structural data such as accessibility trees to map visual features to their functions, allowing models to act without needing underlying code.

These papers consistently show that refining this grounding, through richer datasets, synthetic training, or reinforcement learning, improves how well agents handle tasks across desktop, web, and mobile platforms. Better grounding also reduces the number of steps and tokens needed, while adversarial attacks highlight vulnerabilities that simpler tests miss.

Papers
24
Released code
0
First seen
Aug 2026
Latest
Nov 2026

20 papers in the last two months, against 4 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Open dataset and distillation-to-reinforcement pipeline sharpen GUI grounding in 8B model

    A large GUI grounding dataset and a distillation-to-reinforcement learning training method yield state-of-the-art 8B-scale grounding models.

    Oct 2026
  2. 2Training on interfaces helps visual models inspect images beyond screens

    GUI grounding supervision trains vision-language models to inspect images sequentially, improving perception beyond GUI tasks.

    Oct 2026
  3. 3AI agents’ clicks marked wrong can work a third of the time

    Many clicks that GUI-grounding benchmarks reject are actually correct because they produce the same action as the annotated box.

    Oct 2026

How fast it is moving

JunJulAugSeptOctNov so far

Often studied with

computer-use agentAn AI agent that interacts with computer interfaces—GUI, terminal, or code—to autonomously perform tasks.9 togethervision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.9 togetheragentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real6 together

Who is working on it

Apple 2Independent Researcher 2University of Chinese Academy of Sciences 2Shanghai Jiao Tong University 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.