Harness optimization normally evaluates every candidate on every validation task each iteration, which is expensive and wastes budget on tasks that stop discriminating between good and bad harnesses. Task-CoEvolve instead tracks each task's historical success rate, samples tasks with weight proportional to their variance, and evaluates only the sampled subset. It then estimates the full-set score from the sampled tasks using their inclusion probabilities, keeping different iterations comparable.
On online text classification, Task-CoEvolve approached full-set search using only 7% of the evaluation budget and outperformed full-set search at a 20% budget. On Terminal-Bench 2.1, it matched full-set search performance with 20% of the evaluations while reducing overall search cost by 67-80%. This is interesting because evaluation, not just model inference, often dominates the cost of harness optimization, especially for long-horizon terminal tasks.