computer-use agent
A computer-use agent is an AI program that looks at a computer screen and interacts with it the same way a person would, by clicking buttons, typing text, and reading what appears. Instead of following a fixed script, it can decide what to do next based on what it sees. Some agents also combine these visual actions with direct commands, like running code or calling specialized tools, to work more efficiently.
The papers show that combining visual and programmatic actions consistently improves performance over using only one method. Several works demonstrate that giving the agent control over its own observation and execution strategy boosts success while reducing costs. New benchmarks reveal that current agents struggle with specialized scientific software and long workflows. Security measures can be added without severely hurting performance, and open frameworks now allow researchers to study and reproduce these advances more easily.
- Papers
- 50
- Released code
- 4
- First seen
- Sept 2026
- Latest
- Nov 2026
50 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1CUA-Lite standardises computer use agent development across environments and models
CUA-Lite is an open platform that standardizes environments, data, and model harnesses for computer use agents.
Oct 2026 - 2AI agents learn phone app navigation on the fly without offline training
MAPLE uses cross-episodic playbook sharing to guide mobile agents, achieving state-of-the-art task success without training.
Nov 2026 - 3Mobile AI agents can look safe when attacks miss their path
Trajectory-aware adaptive testing exposes more mobile agent security flaws than static injection benchmarks.
Nov 2026
How fast it is moving
Often studied with
benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.17 togetheragentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real16 togetherverificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.13 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Chinese Techcs.CR
Mobile AI agents can look safe when attacks miss their path
Trajectory-aware adaptive testing exposes more mobile agent security flaws than static injection benchmarks.
The Hong Kong Polytechnic University, Zhejiang University · Nov 2026
- Big Techcs.LG
AI agents learn phone app navigation on the fly without offline training
MAPLE uses cross-episodic playbook sharing to guide mobile agents, achieving state-of-the-art task success without training.
Google, Washington University in St. Louis · Nov 2026
- Independentcs.AI
CUA-Lite standardises computer use agent development across environments and models
CUA-Lite is an open platform that standardizes environments, data, and model harnesses for computer use agents.
Oct 2026
- Independentcs.CL
Small AI agents can handle computer tasks with clicks and code
Mage-VL-Cowork, a 4B hybrid computer-use agent, achieves 66.0% on OSWorld-Verified via evolved harness and five-stage training.
Nov 2026
- Academiccs.AI
New benchmark reveals agents fail to detect and recover from their own errors in long tasks
Computer-use agents rarely recover from their own errors in long-horizon workflows; fine-tuning on real failures improves recovery and clean-start performance.
University of Texas at Dallas, Independent Researcher · Oct 2026
- Industrycs.LG
AI agents get better at software tasks by reusing their own verified work
Generating software tasks, recording how agents execute them, and retraining on verified runs produces open-source computer-use agents that lead their size class on screen, desktop and web tests.
Shanghai Qiji Zhifeng Co., Ltd. · Nov 2026
- Big Techcs.AI
Synthetic computer worlds improve agents on unseen computer-use tasks
WildAWM uses open-source models to synthesize computer-use environments spanning GUI, MCP, and CLI; training only in these worlds improves agents across eight out-of-distribution benchmarks.
UNC-Chapel Hill, Microsoft Research · Oct 2026
- Chinese Techcs.CV
Open agent stack improves computer tasks beyond GUI-only control
OpenGCUA with code execution and model-driven decisions improves task scores by 19.6 points on OSWorld-v2.
Alibaba Token Hub, Alibaba Group · Oct 2026
- Top Universitycs.CR
AI email agents can handle standard phishing safely yet obey tailored attacks
Adaptive persuasive phishing attacks can evade LLM email agents' detection and trigger malicious-request compliance even when agents safely handle pre-existing phishing emails.
Carnegie Mellon University · Oct 2026
- Independentcs.LG
Interactive software judges improve when they test applications directly
Interaction-as-a-Judge using computer-use agents improves evaluation and generation of interactive applications over code-only methods.
Oct 2026
- Academiccs.CRcode
Accessibility-based masking shields computer agents from indirect prompt injections
SA-CUA blocks indirect prompt injections while preserving computer-use autonomy by separating trusted screen regions and independently verifying values crossing the trust boundary.
Tübingen AI Center, University of Tübingen · Oct 2026
- Top Universitycs.AI
Mobile GUI agents rarely transfer task success to new apps with the same goal
Mobile GUI agents do not generalize across apps for the same task; app-specific success does not transfer.
KAIST, CMU · Oct 2026
- Top Universitycs.AI
New benchmark tests AI judges on long computer-use tasks by swapping instructions
A benchmark of 1,373 paired computer-use tasks shows current judges, best GPT-5.5 at 80.9% balanced accuracy, often miss subtle instruction violations.
ServiceNow Research, McGill University · Oct 2026
- Big Techcs.AI
Marketplace Steering Often Diverts Computer Agents From Users’ Best Choices
Computer-use agents often fail to purchase the user-optimal product when online marketplaces use steering mechanisms.
Carnegie Mellon University, Microsoft Research · Sept 2026
- Chinese Techcs.SE
Open training stack scales mobile agents across apps and tools
OpenMobile-2 builds versatile mobile agents using scalable simulated environments and hybrid GUI-tool actions.
Nanjing University, Xi’an Jiaotong University · Oct 2026
- Big Techcs.AI
Terminal-only control improves digital agents across web, desktop and CAD
A terminal-only harness using bash commands outperforms GUI harnesses across web, desktop, and CAD tasks.
Microsoft Research, National University of Singapore · Oct 2026
- Top Universitycs.AI
New benchmark tests AI agents on real scientific software workflows
Current VLMs achieve under 70% success on scientific software tasks, revealing major gaps in specialized interface use.
Tsinghua University, University of California, Los Angeles · Oct 2026
- Top Universitycs.AI
Visual feedback boosts software engineering agent success rates
Hybrid agents that combine code editing with GUI interaction outperform code-only agents on software engineering tasks.
Carnegie Mellon University, University of Southern California · Oct 2026
- Big Techcs.CR
Secure transactions limit untrusted influence on computer-use agents
Secure-CUA protects computer use agents from untrusted interface content by committing to action transactions before accessing it.
University of Wisconsin–Madison, Google · Oct 2026
- Top Universitycs.CVcode
Fine-Grained Training Data Improves Agents on Interactive CAPTCHA Tasks
CaptchaArena supplies 50,000 execution-verified interactive CAPTCHA puzzles across 20 types, and CaptchaAgent trained on it reaches 71.7 Pass@1.
Columbia University, Zhejiang University · Sept 2026
- Academic
Web agents struggle to turn search results into usable artifacts
Frontier computer-use agents fully complete fewer than 3% of long-horizon web tasks requiring synthesis and artifact creation, and visual/spatial failures often make partial artifacts unusable.
University of Utah · Sept 2026
- Chinese Techcs.CV
Real-time GUI feedback improves training for computer-use agents
ComputerSD uses real-time GUI feedback to improve computer-use agent training via online self-distillation.
Zhejiang University, Ant Group · Oct 2026
- Chinese Techcs.AI
Mobile AI agents run up to four times faster, sacrificing accuracy
A JEPA-based GUI agent picks actions by scoring latent representations of candidate elements, running several times faster than a 7B vision-language model without decoding tokens.
Shanghai Jiao Tong University, Nanyang Technological University · Oct 2026
- Chinese Techcs.AI
Unified platform enables GUI agent training across mobile and computer environments
MAI-Gym unifies GUI agent evaluation and RL training across mobile and computer environments, achieving state-of-the-art performance with a 9B model.
Alibaba Group · Oct 2026
- Top Universitycs.MA
Benchmark exposes computer agents’ failures when resolving conflicting information
CLASHarness generates verifiable multi-agent computer-use conflicts, and evaluation shows models trust misleading peers, struggle to fix conflicts, and fail at coordination protocols.
Emory University, University of Southern California · Oct 2026
- Top Universitycs.CR
New defense eliminates branch steering attacks on computer-use agents
Branch steering attacks on computer-use agents can be defeated by combining pre-authorized branching plans with ahead-of-time capability constraints.
ETH Zurich, University of Cambridge · Oct 2026
- Chinese Techcs.CV
Synthetic experience improves AI game control beyond supervised training
Learning from synthetic experience in 16 environments improves game control beyond supervised initialization.
National University of Singapore, Tencent IEG · Oct 2026
- Chinese Techcs.CL
GUI agents sacrifice future options in long visual puzzles
GUI agents judge each move by visible progress, not future options, causing failure on long-horizon visual puzzles.
Vera Praxis, Tencent · Sept 2026
- Top Universitycs.LGcode
Evidence-driven harness edits improve frozen GUI agents across benchmarks
GUI-HARVEST automatically improves GUI agent harnesses by diagnosing failures from repeated executions and generating reusable edits, boosting performance across models.
The Chinese University of Hong Kong, Shenzhen, Tianjin University · Oct 2026
- Chinese Techcs.AI
AI agents complete more phone tasks by remembering smaller goals
Storing GUI agent experience as subgoal-level memory graph improves success rate by 11.11% on SPA-Bench.
Huawei Noah's Ark Lab · Oct 2026
- Big Techcs.CV
Controlled desktop scenes improve visual grounding for computer-use agents
Controllably composed real desktop scenes provide dense supervision that improves GUI grounding and long-horizon computer-use task completion.
ETH Zurich, IBM Research Zurich · Oct 2026
- Top Universitycs.CLcode
Two-stage verifier combines reasoning and tool-use for safer computer agents
SCOUT improves computer-use agent safety by combining reasoning-intensive rubric generation with tool-intensive evidence gathering to detect harmful behaviors.
Columbia University · Oct 2026
- Chinese Techcs.CV
Selective experience activation improves personalized GUI agents across task steps
Relevant history helps personal GUI agents mostly at the first decision; experience must be activated by matching the current screen, not just retrieved by task.
NTU, Alibaba · Sept 2026
- Research Labcs.AI
Active visual probes help browser agents act more accurately
Active visual probes improve browser-agent task success by verifying DOM-to-pixel alignment on demand while retaining compact, decision-relevant memory.
Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Sept 2026
- Chinese Techcs.CV
Synthetic GUI trajectories train agents without running target software
Synthesizing GUI interaction trajectories with image generators improves GUI agent performance on real desktop and scientific tasks.
Hunyuan AI Data Team · Oct 2026
- Chinese Techcs.AI
Dependency-aware credit assignment improves reinforcement learning for terminal agents
DepGPO assigns credit based on command read-write dependencies, improving RL training for terminal agents.
Southeast University, Huawei Noah’s Ark Lab · Oct 2026
- Chinese Techcs.AI
App deeplinks help mobile agents navigate faster and finish more tasks
Combining app-native deeplinks with GUI actions lets mobile agents complete tasks more successfully and in fewer steps than using GUI actions alone.
Shanghai Jiao Tong University, Tongyi Lab, Alibaba Group · Sept 2026
- Top Universitycs.AI
Compiling Mobile App Navigation Makes GUI Agents Faster and More Reliable
PhoneCLI compiles mobile app navigation into deterministic commands, reducing VLM cost and improving task success.
The University of Hong Kong · Sept 2026
- Big Techcs.CL
Process-based evaluation reveals where computer-use agents go wrong
OSWorld-Pro evaluates computer-use agents on intermediate subgoals to reveal specific failure modes.
NVIDIA · Sept 2026
- Top Universitycs.AI
Web agents handle audio and video, but best systems still fail on cross-site tasks
WebArena-Pro provides the most comprehensive reproducible benchmark for web agents, including audio and video across 20 applications.
McGill University, Mila -- Quebec AI Institute · Nov 2026
- Big Techcs.AI
AI agents sometimes recognise scam warnings but proceed with tasks
Web agents deceived by online fraud at rates from 11% to 35% across six frontier LLMs.
Deakin University, Fudan University · Nov 2026
- Independentcs.AI
A small AI agent trained on cloned websites matches bigger web agents
A suite of 153 self-hosted replica websites with 3,518 tasks trains small web agents that then handle unseen tasks and match leading open agents.
Nov 2026
- Big Techcs.AI
AI assistants face a $1.7 trillion wish list of life tasks
LifeVal benchmarks AI models on high value, personalized, agentic consumer tasks rooted in real user data.
OpenAI · Nov 2026
- Big Techcs.AI
AI agents’ false completion claims are caught more often with follow-up checks
Treating verification as active exploration of interfaces and code beats passive judging for computer-using agents.
Google, The University of Hong Kong · Oct 2026
- Top Universitycs.AI
Frontier AI agents struggle with end-to-end professional engineering workflows
Frontier AI agents achieve only 44.3 out of 100 on the EngiWorld benchmark, failing to reliably automate professional engineering tasks.
Tsinghua University, Zhiman Inc. · Oct 2026
- Big Techcs.AI
TeleTune learns agent skills from unfiltered user logs without goals or replay
TeleTune learns reusable textual skills from goal-free, non-replayable, interleaved telemetry logs by keeping only library edits that improve held-out action-prediction accuracy.
UNC Chapel Hill, University of Texas at Austin · Oct 2026
- Chinese Techcs.AI
Web agents struggle with multilingual, multimodal evidence hunts across the web
HyperBrowseComp is a 423 question benchmark across 13 languages and multiple modalities that challenges web browsing agents.
Mohamed bin Zayed University of Artificial Intelligence, Mila – Quebec Artificial Intelligence Institute · Oct 2026
- Chinese Techcs.AI
Targeting uncertain local decisions cuts GUI agent training cost by 56-61%
A two-stage adaptive training framework for GUI agents reduces environment interaction by 56-61% while improving task performance.
Alibaba Group, The Chinese University of Hong Kong · Oct 2026
- Chinese Techcs.LG
GUI world models fail when hidden state is omitted from predictions
State aliasing in GUI world models is resolved by inferring hidden state from history.
University of Chinese Academy of Sciences, Institute of Automation, Chinese Academy of Sciences · Sept 2026
- Academiccs.AI
Decouple high-level VLM planning from low-level execution to speed up mobile agents
Separating VLM planning from lightweight, accessibility-based execution cuts mobile GUI agent time and API cost while retaining competitive task success.
Rice University · Sept 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.