JarvisGUI is a new benchmark designed to evaluate GUI agents on tasks that require coordination across multiple devices and platforms, including Android, Windows, and Ubuntu. Unlike existing benchmarks that test agents on single-device, static workflows, JarvisGUI dynamically composes multi-step, cross-device tasks using a lightweight type system and automatically evaluates agent performance in virtual environments spanning these operating systems.
Evaluating state-of-the-art open-source GUI agents reveals a critical capability gap: agents that perform well on single-device tasks struggle significantly with cross-device workflows. They fail to maintain state across devices, reason about cross-platform context, and manage long-horizon dependencies. These failures expose an overestimation of real-world readiness in current benchmarks.
This work is interesting because real-world digital workflows routinely span multiple devices, and existing evaluations give an overly optimistic view of agent capabilities. JarvisGUI highlights that cross-device coordination remains a major bottleneck that must be addressed for agents to be truly useful in practice.