(CUAs) struggle with online training because outcome rewards provide no guidance for intermediate actions. Existing on-policy self-distillation methods fail when fixed guidance misaligns with the agent's actual trajectory and when token-level signals conflict with step correctness. ComputerSD solves this by using a fine-tuned GUI analyzer that reads screen transitions after each action, producing real-time feedback and a step-level value score.
This feedback serves as privileged context to rescore the agent's actions, while the value score regulates which token-level signals to trust. The method jointly optimizes token-level self-distillation with trajectory-level GRPO in an asynchronous pipeline. On OSWorld-Verified, ComputerSD outperforms standard GRPO by 1.9 and 4.1 percentage points on two different backbone models, and generalizes to out-of-distribution tasks.
The key interest lies in converting real-time GUI observations into dense learning signals that fix in multi-step agent tasks. This approach enables agents to learn from each action's consequence rather than only from final success or failure, making online training more sample-efficient.