New benchmark standardizes evaluation of coding agent harnesses

Claw-SWE-Bench uses 350 multilingual GitHub issues to measure how different adapter designs affect OpenClaw-style agents' coding performance.

PaperTop Universitycs.LGarXiv:2606.12344v1 ↗

Mengyu Zheng · Kai Han · Boxun Li · Haiyang Xu · Yuchuan Tian · Wei He · +10 more

TokenRhythm Technologies · Infinigence AI · Peking University · City University of Hong Kong · SEE Fund

Research Digest·11 June 2026·2 min read

Read the paper →

Thread:Agent Harness Optimization

The authors introduce Claw-SWE-Bench, a benchmark and adapter protocol for evaluating OpenClaw-style agent harnesses (claws) on coding tasks. They show that adapter design dramatically affects performance: OpenClaw with a minimal adapter scores only 19.1% Pass@1, while a full adapter achieves 73.4% with the same backbone model. The benchmark also reveals that model choice and harness choice each account for roughly 30 percentage points of variation in performance.

What they did

The authors constructed Claw-SWE-Bench, a collection of 350 GitHub issue-resolution instances drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup. The instances span 8 programming languages and 43 repositories. They also built a smaller Claw-SWE-Bench Lite (80 instances) for faster validation. The benchmark defines a fixed protocol: a consistent prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator, enabling fair comparison of different agent harnesses. They evaluated OpenClaw with two adapter variants (minimal direct-diff and full adapter) using the GLM 5.1 backbone, and conducted sweeps across nine models and five harnesses.

Key findings

Adapter design is critical: OpenClaw with a minimal direct-diff adapter achieves only 19.1% Pass@1 on the full benchmark, while the full adapter reaches 73.4% Pass@1 with the same GLM 5.1 backbone.
Model choice changes Pass@1 by up to 29.4 percentage points (pp) across nine models, while harness choice changes Pass@1 by up to 27.4 pp under fixed models.
Systems with similar accuracy can differ substantially in total API cost, making cost a first-class axis of evaluation.

Why it matters

This benchmark fills a gap in evaluating general-purpose agent harnesses on coding tasks, where standard SWE-bench requires a clean Docker workspace and patch extraction that generic agents cannot satisfy. By standardizing the adapter protocol and treating harness and cost as evaluation axes, Claw-SWE-Bench enables reproducible comparison and highlights that adapter design is a major factor in coding performance.

Caveats

The benchmark is limited to 350 instances, which may not cover all coding scenarios. The instances are drawn from existing SWE-bench subsets, so they may reflect biases in issue selection. The reported results are with a specific model (GLM 5.1) and may not generalize to other backbones. The Lite subset is smaller and may not be representative of the full benchmark.

Analysis

This work addresses a practical problem in the agent evaluation landscape: the gap between general-purpose agent harnesses (like OpenClaw) and the narrow interfaces of existing coding benchmarks. The finding that adapter design can improve performance by over 50 percentage points underscores that the harness itself is a significant variable, not just the model. The inclusion of cost accounting is also valuable for real-world deployment. The benchmark is likely to be adopted by researchers working on agent tool-use and coding agents, providing a standardized way to compare harnesses.