← All concepts

computer-use agent

A computer-use agent is an AI program that looks at a computer screen and interacts with it the same way a person would, by clicking buttons, typing text, and reading what appears. Instead of following a fixed script, it can decide what to do next based on what it sees. Some agents also combine these visual actions with direct commands, like running code or calling specialized tools, to work more efficiently.

The papers show that combining visual and programmatic actions consistently improves performance over using only one method. Several works demonstrate that giving the agent control over its own observation and execution strategy boosts success while reducing costs. New benchmarks reveal that current agents struggle with specialized scientific software and long workflows. Security measures can be added without severely hurting performance, and open frameworks now allow researchers to study and reproduce these advances more easily.

Papers
50
Released code
4
First seen
Sept 2026
Latest
Nov 2026

50 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1CUA-Lite standardises computer use agent development across environments and models

    CUA-Lite is an open platform that standardizes environments, data, and model harnesses for computer use agents.

    Oct 2026
  2. 2AI agents learn phone app navigation on the fly without offline training

    MAPLE uses cross-episodic playbook sharing to guide mobile agents, achieving state-of-the-art task success without training.

    Nov 2026
  3. 3Mobile AI agents can look safe when attacks miss their path

    Trajectory-aware adaptive testing exposes more mobile agent security flaws than static injection benchmarks.

    Nov 2026

Who is working on it

Zhejiang University 3Google 3Microsoft Research 3Alibaba Group 3Carnegie Mellon University 3McGill University 2

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.