What they did

Wang et al. compared automatic harness evolution—an iterative search for agent harness configurations using unit test feedback—with two baselines: simple test-time scaling (e.g., best-of-N sampling) and discovery baselines (e.g., prompt optimization). They matched the number of feedback calls and inference budget across all methods. Experiments were conducted on Terminal-Bench 2.1 using GPT-5.4 and Claude Opus 4.6. They also evaluated evolved harnesses on a set of held-out tasks to test whether improvements generalize beyond the benchmark used during search.

Key findings

• Harness evolution does not consistently outperform simple test-time scaling methods when feedback and inference budgets are matched.
• Evolved harnesses show limited generalization to held-out tasks, suggesting overfitting to the specific benchmark used during search.
• The gains attributed to harness evolution appear to stem largely from the iterative search process itself, rather than from improved harness design.
• These results challenge the validity of existing evaluation protocols that report final performance on the same benchmark used for search.

Why it matters

This work questions the prevailing narrative that automatic harness evolution yields superior agent designs. It underscores the need for more rigorous evaluation standards—including matched search budgets and generalization tests—to avoid conflating the benefits of search with genuine improvements in harness design.

Caveats

The study is limited to two frontier models and one benchmark (Terminal-Bench 2.1). The baselines chosen (test-time scaling, discovery) may not capture all possible comparison methods. The held-out tasks, while representative of some agentic scenarios, may not cover the full diversity of real-world applications.