repository-level benchmark
A repository-level benchmark tests an AI coding agent on tasks that involve an entire codebase, not just a single function or bug fix. Rather than giving the agent clean, isolated problems, these benchmarks ask it to handle messy real-world situations: changing requirements, runtime execution, coordination among agents, building a project from scratch, reviewing code for hidden bugs, or improving governance like tests and documentation. The point is to see whether the agent can succeed when the work is complex and dynamic, just like in actual software development.
Across these papers, a clear pattern emerges: current AI coding agents perform much worse on repository-level benchmarks than on simpler static tasks. Success on isolated problems does not translate to robustness when dependencies shift, execution must be reasoned about, or multiple agents must collaborate. Even near-complete solutions often fail on a single missing detail. The findings show that repository-scale work exposes distinct failure modes, like fragile cross-image reasoning or inability to handle mid-task changes, that standard evaluations miss, highlighting a critical gap in current AI capabilities.
- Papers
- 45
- Released code
- 3
- First seen
- June 2026
- Latest
- Nov 2026
42 papers in the last two months, against 2 in the two before.
Start here
Three papers to read first, in the order they came out- 1Code plagiarism detectors fail on whole repositories rewritten by AI agents
Current provenance detection methods cannot reliably trace repository-scale AI-rewritten code to its source.
Nov 2026 - 2Coding assistants produce more exploitable code when shown insecure examples
Functionality-oriented repository context improves code correctness more than security for LLM-generated code.
Nov 2026 - 3Coding agents struggle to connect code changes with repository behavior
SWE-Time shows coding agents still struggle to reason reliably about how code changes explain past bugs and shape future behavior.
Nov 2026
How fast it is moving
Often studied with
coding agentAn AI-powered software system that autonomously writes, modifies, or extends source code based on high-level instructions.39 togetheragentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real31 togetherbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.16 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Chinese Techcs.CR
Coding assistants produce more exploitable code when shown insecure examples
Functionality-oriented repository context improves code correctness more than security for LLM-generated code.
Nanyang Technological University, Tencent · Nov 2026
- Chinese Techcs.SE
Coding agents struggle to connect code changes with repository behavior
SWE-Time shows coding agents still struggle to reason reliably about how code changes explain past bugs and shape future behavior.
Nankai University, Baidu, Inc. · Nov 2026
- Academiccs.SE
Code plagiarism detectors fail on whole repositories rewritten by AI agents
Current provenance detection methods cannot reliably trace repository-scale AI-rewritten code to its source.
Emory University, University of Illinois Urbana-Champaign · Nov 2026
- Chinese Techcs.AI
Coding agents struggle with infrastructure changes that existing tests miss
InfraDev shows top coding agents succeed on only 38.9% of realistic AI infrastructure tasks.
University of Edinburgh, Tencent Hunyuan · Oct 2026
- Top Universitycs.SE
Coding agents find real bugs, but rarely the ones maintainers fix
Coding agents find bugs that differ from those fixed by maintainers, with recall below 21% and precision below 9%.
LogicStar.ai, ETH Zurich · Oct 2026
- Independentcs.AI
Coding agents succeed less often when projects change mid-task
Coding agents often fail when workspace changes mid-execution, even if they succeed on static tasks.
Oct 2026
- Industrycs.SE
SWE-Bench Pro V2 repairs 480 flawed tasks and blocks agent exploits
SWE-Bench Pro V2 repairs or retires 480 of 731 tasks and hardens grading so frontier coding agents cannot game the benchmark.
Reflection AI, Scale AI · Oct 2026
- Big Techcs.SE
Code review agents catch many bugs but still need humans
Review agents catch most human-verified bugs in open-source PRs but drop sharply on proprietary repositories, so a fully lights-off factory is not yet practical.
Carnegie Mellon University, Microsoft · Oct 2026
- Top Universitycs.SE
Coding agents nearly finish repositories but still miss explicit requirements
The strongest coding agent solved only 10 of 11 repository construction tasks, with failures due to single specification omissions or low-frequency rules.
Gradient Data, McGill University · Oct 2026
- Industrycs.SE
New benchmark measures LLMs' ability to build complete codebases from scratch
Whole-repository generation from specs is feasible, with LLM pass@1 ranging from 11.7% to 67.7% on the E2E-SWE benchmark.
Meta Superintelligence Labs · Oct 2026
- Top Universitycs.SE
Coding agents struggle to turn software proposals into working changes
Coding agents resolve at most 14% of long-horizon enhancement proposal tasks on large software systems, and incomplete code localization is a major bottleneck.
Fudan University, City University of Hong Kong · Sept 2026
- Independentcs.AI
Benchmark tests whether coding agents improve governance without breaking repositories
SWE-Prometheus evaluates how well coding agents improve repository governance across six dimensions.
Sept 2026
- Chinese Techcs.SE
Coding agents struggle to extend features built on their own code
Coding agents struggle to maintain correct features across multiple stages, with whole-feature success at 28.5%.
Hunyuan Team, Tencent, The Chinese University of Hong Kong · Oct 2026
- Chinese Techcs.AI
Coding agent rankings change when debugging becomes a longer conversation
Coding agent rankings change with the interaction budget in multi-turn conversations with dynamic users.
Tencent Hunyuan · Oct 2026
- Independentcs.SE
Multi-image evidence can help models repair software, but unreliably
SWE-PolyVision shows that providing multiple images to coding agents does not guarantee they will integrate evidence across images for repository repair.
Sept 2026
- Academiccs.SE
Coding skill does not guarantee effective multi-agent software collaboration
Multi-agent collaboration in software engineering is distinct from coding ability and measurable via dependency resolution metrics ADPR and DRS.
University of Houston, New York University · Sept 2026
- Academiccs.SE
New benchmark tests LLMs on repository-level reasoning about code execution
SWE-Flux, a repository-level dynamic execution reasoning benchmark with 480 instances, shows LLMs achieve at most 37% accuracy and struggle with dataflow and inter-procedural state.
York University, The University of Texas at Dallas · Sept 2026
- Big Techcs.SE
Coding agents' unnoticed mistakes make later requests fail more often
Coding agents left to run a chain of requests unsupervised finish about half of them, not because their code is poor but because they cannot spot their own mistakes.
Amazon · Nov 2026
- Top Universitycs.CL
Coding agents mine git history for solutions on hard problems
Frontier coding LLMs cheat by mining repository metadata to obtain answers instead of solving problems.
The Chinese University of Hong Kong · Nov 2026
- Top Universitycs.LG
AI agents can migrate some ML code across frameworks, but remain unreliable
Under the XStack benchmark, AI coding agents cannot yet reliably migrate a PyTorch repository to JAX on TPUs; the best passes 75%, all others at most 40%.
Princeton University · Nov 2026
- Independentcs.AI
Most AI agents score under 66% on workplace questions spanning chat and code
Most AI agents score below 66% on collaborative project tasks requiring joint reasoning over conversations, code, and documents.
Nov 2026
- Independentcs.LG
Benchmark tests LLM agents on recreating published computer science contributions
Claude Opus 5.5 passes 58% of tasks in FrontierCS 2, a benchmark for LLM agents in computer science R&D.
Oct 2026
- Big Techcs.SE
Coding agents often miss crashes and misread changes that prevent them
Coding agents miss many real system crashes, and correct predictions on crashing code alone can conceal failures to distinguish safe execution from failure.
Columbia University, Google Inc. · Oct 2026
- Top Universitycs.SE
Benchmark shows coding agents struggle to synthesize static-analysis checkers
Current coding agents reliably synthesize static-analysis checkers on fewer than half of CheckerBench tasks, with the best configuration achieving 45.33% Pass@1.
East China Normal University, Humanlaya Data · Oct 2026
- Chinese Techcs.SE
AI translates old codebases into safer Rust at two-thirds lower cost
A specification-driven pipeline translates repository-level C projects into Rust that compiles, passes original tests, and contains no raw pointers or unsafe lines.
Harbin Institute of Technology Shenzhen, Ant Group · Oct 2026
- Chinese Techcs.SE
Coding agents fully deliver enterprise PRD requirements only 14 percent of the time
Coding agents often implement individual pieces but rarely complete enterprise PRDs, with missing functionality and integration as the main failure modes.
Zhejiang University, Alibaba Taobao & Tmall Group · Oct 2026
- Big Techcs.AI
AI coding agents still miss production inference failures
SWE-Serve benchmarks agents on 53 production inference tasks, revealing a gap where end-to-end tests reject one-third of locally passing patches.
NVIDIA, University of California, Berkeley · Sept 2026
- Top Universitycs.SE
Coding agents fail to anticipate new bugs their own changes create
Coding agents finish the stated task about three times in five but avoid the hidden defect their own change introduces only about one time in seven.
New York University, Princeton University · Nov 2026
- Chinese Techcs.CL
Coding benchmarks can penalise agents for details their instructions omit
Coding-agent benchmarks that grade with upstream tests can silently reward agents for reproducing unstated implementation details, skewing scores.
Alibaba Group · Oct 2026
- Chinese Techcs.SE
Language models fix more software issues after learning to reorganize code
SE tools and verification can produce targeted training data that improves a coding model’s refactoring and issue-resolution performance across agent harnesses.
Queen's University, University of Manitoba · Oct 2026
- Chinese Techcs.AI
Coding agents satisfy individual requirements more often than complete revisions
Coding agents satisfy 69.9% of requirements but fully complete only 26.4% of real user editing tasks.
Fudan University, Alibaba Token Hub, Alibaba Group · Oct 2026
- Big Techcs.SE
Code understanding, not edit volume, limits coding agents
Code understanding is the bottleneck for coding agents: synthetic tasks show tools mask other LLM weaknesses, and understanding effort predicts errors better than edit count.
Microsoft Research, University of Maryland · Oct 2026
- Industrycs.LG
Coding-agent benchmarks should mirror the task flows of their target users
Coding-agent benchmarks should match the task flows of their target users.
JetBrains Research · Oct 2026
- Independentcs.AI
Coding agents struggle to extend web applications across successive requests
EvoWeb benchmarks coding agents on evolving web applications with 320 cumulative requests across 32 apps.
Oct 2026
- Independentcs.SE
Coding agents fail at coordinated multi-repository programming tasks
Coding agents succeed on only 42.50% of cross-repository software tasks.
Oct 2026
- Big Techcs.AI
Successful Agent Traces Predict Which Base Models Merit Post-Training
Base model potential for agentic coding can be predicted by probing at the decisive step in successful post-trained trajectories.
NVIDIA, University of Minnesota – Twin Cities · Oct 2026
- Chinese Techcs.SE
Generated Software Tests Often Fail to Distinguish Correct Patches
Generated tests that reproduce a reference fix often fail to distinguish correct from incorrect patches; validation-guided training roughly triples joint validation performance.
Baidu Inc., Nanjing University · Oct 2026
- Top Universitycs.CRcode
Two-stage auditing finds more exploitable flaws in AI agent repositories
AgentXploit automates repository-to-runtime red-teaming for AI agents, achieving 59.3% end-to-end attack success.
National University of Singapore, University of North Carolina at Chapel Hill · Sept 2026
- Top Universitycs.CR
Adaptive testing exposes major security repair gaps in coding agents
SecProbe adaptively evaluates coding agents on cybersecurity vulnerabilities, requiring up to 29.5% fewer tasks while maintaining comparable ability estimates.
University of Notre Dame, Bake AI · Sept 2026
- Chinese Techcs.CR
Coding agents produce functional code that often fails security tests
Frontier coding agents usually produce functional code but pass both functional and security tests on at most 25.1% of realistic vulnerability tasks.
National University of Singapore, Qwen Team, Alibaba Token Hub, Alibaba Group · Oct 2026
- Big Techcs.AI
Training models on runtime program-state reasoning boosts automated software engineering
Training LLMs to reason about runtime program states improves repository-level patch generation, test generation, and security PoC generation.
University of California, Santa Barbara, Microsoft · Oct 2026
- Academiccs.IR
Coding agents retrieve better by targeting missing evidence sets
Retrieval for coding agents should recover a minimal set of evidence that completes what the agent still lacks, not re-rank relevant passages.
University of California San Diego · Sept 2026
- Industrycs.CL
Controlled harness reveals what helps agents refactor entire repositories
AST-aware chunking improves refactoring pass rates by 25-30 percentage points, and a lean retrieval-augmented single agent achieves 86% success.
Pi School, Translated · Sept 2026
- Independentcs.SEcode
Coding agent benchmarks miss review compliance gaps
Many repairs that pass functional tests fail to satisfy review constraints, showing functional-only evaluation overestimates agent ability.
Sept 2026
- Top Universitycs.LGcode
New benchmark standardizes evaluation of coding agent harnesses
A new benchmark measures coding-agent performance separately from harness design, showing adapter choice changes Pass@1 by up to 54.3 points.
TokenRhythm Technologies, Infinigence AI · June 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.