HarnessDev shifts evaluation of LLM agents from task outputs to the runnable infrastructure that executes tasks. In the Creation stage an LLM builds a harness from a minimal seed and a few cases; in the Evolution stage it revises that harness using downstream execution feedback. Each harness is scored on capability and execution-token cost against held-out benchmarks.
Across six creator LLMs, four domains, and five downstream benchmarks, generated harnesses were substantially behind mature references on code and on search and research, while matching or exceeding references on writing and machine-learning experimentation. Execution costs varied widely. Evolution sometimes improved performance, but gains were unstable, only partially transferred to held-out tasks, and depended strongly on the model executing the harness.
Why this is interesting: harness design is becoming a significant lever in deployed agents, and this work suggests models can create useful infrastructure in some domains but not reliably carry improvements across tasks or models.