What they did

The authors represent GUI tasks as typed input-output transformations, then automatically combine compatible tasks into multi-step workflows. They run these dynamically composed workflows in virtual Android, Windows, and Ubuntu environments, requiring agents to coordinate actions and intermediate results across platforms.

Key findings

• State-of-the-art open-source GUI agents frequently fail to recognize when information or artifacts must be transferred between environments.
• Agents struggle to retain and apply context when a workflow moves between operating systems with different interfaces and conventions.
• Performance deteriorates on longer workflows whose later steps depend on correctly completing and preserving the outputs of earlier steps.
• These failures are largely absent from conventional benchmarks built around static, single-device tasks, which may therefore overstate practical readiness.

Why it matters

JarvisGUI offers a way to evaluate GUI agents under conditions closer to everyday computing, where work often moves among phones, desktops, applications, and operating systems. Its typed composition framework could also support broader and less predictable evaluations than fixed task suites.

Caveats

The supplied abstract does not report the benchmark’s size, the evaluated agents, quantitative success rates, or comparisons with specific single-device benchmarks. It therefore establishes the nature of the capability gap more clearly than its magnitude, and virtualized environments may not capture every complication of physical devices and real user accounts.