The authors built a language-agnostic pipeline that converts version-pinned open-source projects into behavioral specifications, reproducible environments and hidden acceptance tests.
Coding agents nearly finish repositories but still miss explicit requirements
Zero2Repo tests whether agents can turn specifications into complete, buildable repositories across four programming languages.
Top University
Pei Yang · Tianyu Shi · Yuhang Yao · Wanyi Chen · Tongyun Yang · Dun Pei · +20 more
Gradient Data · McGill University · Carnegie Mellon University · University of Southern California · Queen’s University
Research Digest··3 min read
Yang et al.
Why this paper
From McGill University and 8 others · Part of Process Agent Benchmarks, now 8 papers
In one line
The strongest coding agent solved only 10 of 11 repository construction tasks, with failures due to single specification omissions or low-frequency rules.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§