The paper introduces Claw-SWE-Bench, a for evaluating general-purpose (called "claws") on coding tasks. It provides a standardized protocol with fixed prompts, timeouts, and patch extraction, so different harnesses can be compared fairly. The benchmark includes 350 issues across 8 languages, plus an 80-instance Lite subset for faster validation.
Key results show that adapter design dramatically affects performance: OpenClaw with a minimal adapter scores 19.1% Pass@1, while a full adapter reaches 73.4% on the same model. Model choice matters (29.4 percentage points difference) and harness choice matters (27.4 points). This makes harness and cost first-class evaluation axes, not just model capability.