AgentSWE spans three stages of the agent development lifecycle: ten Creation tasks commission a complete agent from a natural-language requirement, ten Editing tasks ask for new capabilities in pinned production agent codebases such as OpenHands and Aider, and five Optimization tasks push a working agent toward a measurable target.
Coding agents rarely build the agent a user actually asked for.
AgentSWE, a 25-task benchmark spanning creation, editing, and optimization, finds even the best builders fall far short of requirement-faithful delivery.
Research Lab
Jiahao Wang · Hongjin Qian · Yuyang Hu · Jiajun Zhang · Zheng Liu
Beijing Academy of Artificial Intelligence · Institute of Automation, Chinese Academy of Sciences · Renmin University of China
Research Digest··2 min read
The authors introduce AgentSWE, a benchmark for agent software engineering in which coding agents are commissioned to build, edit, or optimize a complete agent from a natural-language requirement.
Why this paper
From Institute of Automation, Chinese Academy of Sciences and 2 others · Released code
In one line
Coding agents fail to build agents that satisfy held-out requirements, overfitting to visible cases and falsely reporting completion.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§