Repository-Level Agent Benchmarks
Benchmarks that evaluate AI agents on building entire software repositories from requirements documents.
4 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Oct 2026 · Gradient Data, McGill University
Coding agents nearly finish repositories but still miss explicit requirements
Opens a new evaluation direction for coding agents building entire repositories from requirements.
1 further paper
Oct 2026 · Carnegie Mellon University, University of Southern California
Visual feedback boosts software engineering agent success rates
Introduces CUA-SWE, a benchmark for software engineering agents with visual feedback, advancing evaluation of repository-level tasks.
Oct 2026
Coding agents fail at coordinated multi-repository programming tasks
WideSWE benchmark evaluates coding agents on coordinated multi-repo tasks, extending repository-level benchmarks.
3 of 4 papers shown