SWE-Gate is a new benchmark for software engineering agents that includes review constraints alongside functional tests. It creates 303 repair instances from 75 Python repositories, each with separate functional and constraint tests based on real PR review comments. When evaluated with four LLM backends, 221 out of 644 repairs that passed functional tests failed to meet the review constraints. This shows that evaluating only functional correctness overestimates an agent's ability to produce acceptable patches.
Coding agent benchmarks miss review compliance gaps
SWE-Gate benchmark shows that passing functional tests does not guarantee acceptable patches in real-world software engineering.
Independent
Xin He · Yanlin Wang · Mingwei Liu · Jiachi Chen · Hongyu Zhang · Guanbin Li
Research Digest··2 min read
Thread:Agent Rule Compliance
In one line
Many repairs that pass functional tests fail to satisfy review constraints, showing functional-only evaluation overestimates agent ability.
Before you read this
Three ideas this paper builds oncoding agentAn AI-powered software system that autonomously writes, modifies, or extends source code based on high-level instructions.134 papers →repository-level benchmarkAn evaluation suite that measures agent performance on tasks spanning entire codebases, from construction to repair and governance.37 papers →agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real241 papers →
Read it as
The authors introduce SWE-Gate, a benchmark that adds review-derived constraints to functional tests for coding agents.
The authors constructed SWE-Gate, a for software engineering agents, by deriving review constraints from real pull request review comments and synthesizing repair instances around them.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code