The paper introduces CUA-SWE, a for evaluating agents that must both edit code and interact with graphical interfaces to complete software engineering tasks. Tasks span four domains and include deterministic tests to verify correctness.
The evaluation compares code-only agents with hybrid agents that also observe screenshots and perform GUI actions. Hybrid agents consistently achieve higher task success, highlighting the importance of visual feedback for guiding software repairs.
CUA-SWE provides a unified testbed for studying visually-guided software development with agents, bridging the gap between and .