The authors introduce a new benchmark, OSWorld-Pro, comprising over 300 tasks and over 2,800 subgoals to evaluate (CUAs) on their progress through intermediate steps, rather than only on final outcomes. The benchmark uses over 67,000 human annotations and automated LLM judges to assess subgoal completion.
They find that even top models like Claude Opus 5 achieve only 75.7% on OSWorld-Pro, compared to 83.4% on the outcome-based OSWorld benchmark. The process-based evaluation reveals specific failure modes, such as agents performing actions irrelevant to subgoals or making click-based mistakes.
This approach is interesting because outcome-based evaluation often gives zero credit for partially correct trajectories, while process-based evaluation provides granular feedback that can directly guide improvements to CUA design and training.