The paper investigates whether generating multiple different programs (harnesses) to solve math problems with an LLM actually provides specialized advantages or just more attempts. The authors created eight different harness programs plus a baseline, and ran them on 386 MATH-500 tasks with three executions each. They found that even running the same program three times produced a 2.16 percentage point coverage gain across repeats, meaning repetition alone explains much of the apparent benefit of multiple programs. The different programs showed more repeatable score patterns than identical programs, but these patterns mostly reflected persistent failures: the generated programs lost to the baseline on all three repeats for 100 tasks, while they won persistently on only one task. A frozen selector that chose a program based on the task question added zero gain. Both the generated programs and multiple copies of the baseline reached 98.70% oracle coverage with 27 total executions. This matters because it undermines claims that generating diverse programs yields useful task specialization for LLM inference.
More Harness Programs Do Not Necessarily Produce Useful Specialization
Controlled comparisons show that apparent gains from diverse LLM harnesses can be matched by repeated runs of identical code.
Top University
Ziyang Xu · Haitian Zhong · Hao Zhou · Hao Qin · Chenhan Jin · Te Qi · +2 more
The Chinese University of Hong Kong · New Laboratory of Pattern Recognition (NLPR) · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy
Research Digest··2 min read
Thread:Agent Harness Optimization
In one line
Repeatable score gains from different LLM harness programs are dominated by persistent weaknesses, not useful specialization.
Before you read this
Three ideas this paper builds onagent harnessA system that structures, verifies, monitors, or controls how an AI agent interacts with external tools, environments, or execution runs.186 papers →non-determinismSame inputs give different outputs across runs, confounding testing and evaluation.31 papers →exploration biasSystematic skew in how an agent samples actions or evidence, often trapping it in unproductive regions or distorting policy learning.21 papers →
Read it as
Xu and colleagues test whether generated LLM harnesses, programs that structure model inference, offer genuine task specialization rather than extra chances to obtain a correct answer.
The authors evaluated 386 MATH-500 problems using two populations.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code