What they did

The authors built ASIL, an interaction layer that connects to each application through its deepest feasible access path. Instead of inferring state from screenshots and issuing mouse clicks, agents receive structured JSON observations and execute actions tied to software concepts and operations.

They implemented the layer for 15 applications and evaluated it on 300 single-application and 80 multi-application tasks. Comparisons included screenshot-and-click agents, native interfaces for LibreOffice and draw.io, and supervised fine-tuning (SFT) plus limited on-policy reinforcement learning for two Qwen3.5 models.

Key findings

• Closed models using ASIL exceeded 80% strict task success while taking fewer than five actions per task on average.
• With a repaired runtime and a 50-step budget, screenshot-and-click control achieved 6.6% strict success on single-application tasks and 26.6% on multi-application tasks. On an easier, OSWorld-comparable subset, these rates rose to 15.0% and 53.3%.
• On matched tasks, ASIL exceeded LibreOffice’s UNO API by 28–38 percentage points in strict success, but only matched draw.io’s MCP content contract, indicating that a sufficiently suitable native interface can erase ASIL’s advantage.
• SFT increased strict success from 58.0% to 72.1% for Qwen3.5-2B and from 66.6% to 80.4% for Qwen3.5-9B. Resource-limited on-policy reinforcement learning raised the results further to 74.4% and 82.2%, respectively.

Why it matters

The results suggest that interface design, not only model capability, is a major constraint on software-operating agents. Structured state reduces ambiguity, while semantic actions shorten trajectories and provide a cleaner substrate for planning and model training; the comparison with draw.io also suggests that well-designed native agent interfaces may be sufficient without a separate universal layer.

Caveats

The evaluation covers 15 applications rather than the broader diversity of desktop and web software, and ASIL requires application-specific integration through whatever internal access path is available. The screenshot comparison depends on a repaired runtime, fixed step budget, and task difficulty, while the abstract does not establish how integration effort, security boundaries, dynamic interfaces, or performance on substantially longer tasks scale.