The study evaluated 66 combinations of 5 language models and 4 harnesses (plus native pairings) on three benchmarks covering command-line and terminal tasks. They found that model rankings reverse across harnesses; for example, Claude outperforms GPT in one harness but trails by over 30 points in another. The best harness for a model also changes between benchmarks, though some pairings are stable—Kimi always scored highest with the openJiuwen harness. Cost varied widely and did not correlate with performance. These results show that selecting an agent system requires testing the specific combination of model, harness, and task, rather than relying on a model leaderboard or the vendor's own harness.
Agent performance depends on matching models, harnesses and tasks
Tests across three command-line benchmarks show that model rankings, costs and failure recovery vary substantially with the surrounding agent harness.
Top University
Yixuan Li · Yiyun Zhou · Yao Long Teng · Fuchao Yang · Yanchen Deng · Zhiyi Lyu · +3 more
Nanyang Technological University
Research Digest··2 min read
Thread:Agent Harness Optimization
In one line
A model's performance and ranking depend on both the harness and the task.
Before you read this
Three ideas this paper builds onagent harnessA system that structures, verifies, monitors, or controls how an AI agent interacts with external tools, environments, or execution runs.186 papers →agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real304 papers →failure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.146 papers →
Read it as
Li and colleagues evaluate language models together with the software harnesses that expose tools, manage context and return errors.
The authors paired five models, Claude Opus 5, GPT-6 Astra, GLM-5.3, Kimi K3 and DeepSeek V4 Pro, with four configurable harnesses: OpenHands, DeepSeek Harness, PI and openJiuwen.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
CodeData