What they did

HarnessDev treats runnable agent infrastructure—not an individual task answer—as the object being evaluated. In the Creation stage, an LLM receives a minimal seed and a small set of development cases, then constructs a complete execution system; in Evolution, it revises that system iteratively using feedback from downstream runs.

The authors tested six creator LLMs across four domains and five downstream benchmarks containing 2,207 unique instances. Hidden evaluation tasks were withheld during development, and each harness was assessed for both task success and execution-token cost.

Key findings

• Generated harnesses remained substantially behind mature human-engineered references on coding and on search and research tasks.
• On writing and machine-learning experimentation, generated harnesses matched or exceeded the selected human-engineered references.
• Execution-token costs varied widely, showing that similar task performance could come with materially different efficiency.
• Evolution sometimes improved performance, but gains were unstable, transferred only partially to held-out tasks, and depended strongly on which model executed the harness.

Why it matters

The benchmark separates an LLM’s ability to design agent infrastructure from its ability to complete tasks inside a fixed system. This makes it possible to study harness engineering as a model capability in its own right and shows that model-external scaffolding can materially shape measured agent performance.

Caveats

The reported results cover four domains, five benchmarks, and selected human-engineered references, so they do not establish that LLM-built harnesses generalize broadly. Evolution gains were inconsistent, while poor transfer across held-out tasks and runtime models suggests that the resulting infrastructure may be over-specialized to its development setting.