AutoGUIWorld is a framework that generates synthetic GUI interaction data without running the actual software. It uses image generators to create and edit screenshots, guided by a planner that specifies actions and their visual outcomes. This produces 79,266 training samples across Ubuntu, Windows, macOS, and Chrome.
Fine-tuning a 35B-parameter model on this data improved task scores on OSWorld (33.0% to 40.8%) and ScienceBoard (14.0% to 32.2%). This suggests that synthetic trajectories can effectively train GUI agents, potentially reducing the cost of collecting real-world interaction data.