Zero2Repo is a that tests whether AI can build a complete software repository from scratch, given only a product requirements document, an interface contract, and an empty workspace. The benchmark spans multiple languages (Python, TypeScript, Go, C++) and uses hidden acceptance tests to evaluate correctness without LLM judges.
Even on tasks derived from well-known projects likely seen during training, the best agent solved only 10 out of 11. Every failing submission still passed 90-99% of the hidden tests, and most failures traced to a single missing detail or low-frequency rule in the specification, not a missing subsystem. This suggests that current agents are near-miss failures and pinpoints concrete areas for improvement.