What they did

The authors constructed SWE-Gate, a repository-level benchmark for software engineering agents, by deriving review constraints from real pull request review comments and synthesizing repair instances around them. Each instance includes separate functional and constraint tests, as well as non-compliant and gold patches. The benchmark contains 303 repair instances spanning 75 open-source Python repositories. They evaluated four LLM backends with different capability levels under a common coding-agent scaffold.

Key findings

• Among 644 repairs that passed functional tests, 221 (34%) failed to satisfy the review constraints.
• The gap between functional success and full repair specification success was consistent across all four LLM backends tested.
• The benchmark enables explicit separation between issue resolution capability and review constraint compliance.

Why it matters

Current benchmarks like SWE-bench evaluate coding agents solely on functional correctness, but real-world code reviews impose additional acceptance criteria. By incorporating review constraints, SWE-Gate provides a more realistic evaluation of agent performance, highlighting a critical blind spot in existing assessments.

Caveats

The benchmark is limited to Python repositories and may not generalize to other languages or development contexts. Review constraints are derived from existing comments and may not capture all non-functional requirements. The study focuses on a single agent scaffold; results may vary with different tools or prompts.