Benchmark tests whether coding agents improve governance without breaking repositories

SWE-Prometheus is a new benchmark that measures the ability of LLM-based coding agents to improve the engineering governance of real-world software repositories. Unlike existing benchmarks that focus on fixing specific issues, this benchmark gives agents open-ended objectives to identify risks, prioritize interventions, and verify changes. It evaluates six governance dimensions: Tests & CI, Quality Gates, Documentation, Reproducible Environment, Dependency & Security, and Behavior Preservation. The benchmark includes 60 repositories and tests ten models on a shared subset. Results show that baseline methods can add governance artifacts but often fail to produce execution-backed improvements, highlighting the need for comprehensive evaluation of governance enhancement.