GUI agents depend on a runtime harness for observation, action execution, and verification. GUI-HARVEST is an automatic optimizer that improves this harness without changing the underlying model. It addresses three challenges unique to GUI agents: aligning model intent with visual effects, handling variable execution outcomes, and making task-specific findings reusable.
The method runs repeated executions of the same task, compares before-and-after screenshots to diagnose failures, clusters verified findings across tasks, and generates source-code edits with predicted behavioral effects. These edits are validated by checking both task performance and the predicted effects through further repeated execution.
On OSWorld-Verified, GUI-HARVEST produced consistent held-out gains across six backbone models. For example, Qwen3-VL-32B-Instruct gained 12.33 points on the full suite. Frozen-harness transfer improved GPT-5 by 13.87 percentage points on WindowsAgentArena at 50 steps without further optimization. It outperformed earlier harness optimizers like Self-Harness and Meta-Harness.