Testing whether AI can develop real-world AI infrastructure is hard. Existing benchmarks use simple tasks and unreliable tests, so we cannot know if agents actually work.
Researchers built InfraDev, a benchmark of 36 tasks from real changes to major AI projects. They created hidden tests that catch failures the original tests miss and wrote clear instructions without giving solutions.
The best coding agent solved only 38.9% of tasks. Hidden tests rejected 32% of submissions that passed all official tests, showing current evaluation is misleading. This helps us understand agent weaknesses and improve them.