Generating Agent Harnesses on Demand Improves Models Across Benchmarks

JIT-Agent dynamically builds and repairs task-specific agent scaffolding, producing reported gains across several model families and agent benchmarks.

PaperTop Universitycs.CLarXiv:2608.25593v1 ↗

Guibin Zhang · Leo Lu · Fangzhou Xie · Kang Zhu · Junhao Wang · Zhifei Xie · +10 more

LV-NUS Lab

Research Digest·28 August 2026·2 min read

Read the paper →

Thread:Agent Harness Optimization

The authors train a model to generate an agent’s harness—the memory, planning, action, and tool-orchestration machinery surrounding an underlying language model—for each task at execution time. They report that these generated harnesses consistently improve multiple model families and can match established runtimes such as OpenCode and Claude Code in controlled evaluations.

What they did

The authors represent an agent harness as a machine-generatable artifact governed by a fixed four-module protocol. JIT-Agent is trained to customize this structure for a task, repair it when execution becomes unstable, and reuse performance signals distilled from an expanding archive of earlier harness configurations.

They attach the resulting harnesses to off-the-shelf models from the DeepSeek V4, Mimo-V2.5, Qwen3.6, and GLM-5.2 families. Evaluations include DeepSearchQA and OdysseyBench, along with controlled comparisons against mature agent runtimes.

Key findings

With JIT-Agent, DeepSeek-V4-Flash reportedly exceeded GPT-5.6 by 9.1 points on DeepSearchQA and 4.3 points on OdysseyBench.
GLM-5.2 gained as much as 20.2 points when paired with a generated harness.
The method consistently improved models at multiple scales across the DeepSeek V4, Mimo-V2.5, and Qwen3.6 families.
Generated harnesses were reported to be competitive with OpenCode and Claude Code under controlled evaluation settings.

Why it matters

The results support treating agent scaffolding as a trainable capability separate from model weights. If harness strategies transfer reliably, developers could adapt planning, memory, and tool use to new tasks without manually constructing a bespoke runtime or retraining the underlying model.

Caveats

The strongest claims are benchmark-specific, and the abstract does not provide absolute scores, variance, computational costs, or details of the controls used in runtime comparisons. The tested model families are broader than a single model but do not establish that generated harnesses work for arbitrary agents, and further evaluation is needed to separate gains from added inference-time computation or task-specific optimization.

Analysis

JIT-Agent shifts harness optimization from hand-written engineering toward learned, task-conditioned generation. This complements model scaling rather than replacing it: the reported comparisons suggest that execution scaffolding can materially change benchmark outcomes even when the underlying model remains fixed.

A central open question is whether the archive-based evolution process continues to improve on genuinely new task distributions, rather than accumulating benchmark-specific strategies. Reliability, resource overhead, and resistance to unsafe or malformed generated harnesses will also matter in deployment.