What they did

The authors developed Task-CoEvolve, a harness-optimization procedure that repeatedly rewrites the software controlling an LLM agent and assesses each candidate on a changing subset of validation tasks. Rather than repeatedly running every task—or relying on one fixed subset—the method uses variance in past outcomes to prioritize tasks near the agent’s current capability boundary.

Because different iterations may evaluate different tasks, Task-CoEvolve adjusts score estimates using each task’s sampling probability. The authors compared the method with fixed-subset and full-validation-set search on online text classification and Terminal-Bench 2.1.

Key findings

• Task-CoEvolve reduced the number of validation evaluations used during optimization by 80% relative to full-set search.
• Despite evaluating only selected tasks, it matched the final performance obtained by full-set search on the reported benchmarks.
• It consistently outperformed fixed-subset baselines, indicating that adapting the evaluated tasks was more effective than repeatedly using the same reduced set.
• The sampling distribution shifted as the harness improved, emphasizing tasks that remained informative for distinguishing candidate harnesses.

Why it matters

Validation can dominate the cost of optimizing agent harnesses, especially when tasks require long or computationally expensive interactions. The results suggest that optimization can retain the decision quality of full-set evaluation while concentrating resources on tasks that best distinguish competing harnesses.

Caveats

The evidence covers two settings, and the abstract does not report absolute performance, run-to-run variability, estimator error or compute savings beyond evaluation counts. It also remains unclear how robust the method is when outcomes are highly noisy, validation sets are small, or changing task-selection probabilities leave some task types rarely sampled.