← All threads

Repository-Level Agent Benchmarks

Benchmarks that evaluate AI agents on building entire software repositories from requirements documents.

4 papers

Where this stands

The written synthesis of this thread is for subscribers. Subscribe.

How this thread developed

  1. Oct 2026 · Gradient Data, McGill University

    Coding agents nearly finish repositories but still miss explicit requirements

    Opens a new evaluation direction for coding agents building entire repositories from requirements.

  2. 1 further paper

    Oct 2026 · Carnegie Mellon University, University of Southern California

    Visual feedback boosts software engineering agent success rates

    Introduces CUA-SWE, a benchmark for software engineering agents with visual feedback, advancing evaluation of repository-level tasks.

  3. Oct 2026

    Coding agents fail at coordinated multi-repository programming tasks

    WideSWE benchmark evaluates coding agents on coordinated multi-repo tasks, extending repository-level benchmarks.

3 of 4 papers shown