Automatic is an iterative search over prompts, tools, memory, and control logic for LLM agents. The authors compared it with simple test-time scaling methods, parallel sampling and sequential refinement, under matched feedback and inference budgets on Terminal-Bench 2.1 with GPT-5.4, GPT-5.4 mini, and Claude Opus 4.6. They also tested evolved harnesses on held-out tasks.
The result is that harness evolution does not consistently outperform the simple baselines. When unit test cases are available, it still underperforms test-time scaling. Gains seen on the search benchmark largely fail to transfer to held-out tasks, suggesting that prior evaluations may have measured adaptation to specific task instances rather than generalizable harness improvements.
This matters because automatic harness evolution is expensive and promises to replace manual agent engineering. If its apparent gains come mostly from repeated search, practitioners can get comparable results with simpler inference-time sampling or refinement.