The authors reviewed all 731 public tasks in SWE-Bench Pro, drawn from 11 active repositories in Python, Go and JavaScript/TypeScript.
SWE-Bench Pro V2 repairs 480 flawed tasks and blocks agent exploits
Dan et al. audited all 731 tasks, repaired or retired 480, and hardened grading to stop agents from retrieving fixes or gaming tests.
Industry
Soham Dan · Jeff Da · Sanyam Bhutani · Miguel Romero Calvo · Ivan Guevara · Fernando Carabedo · +5 more
Reflection AI · Scale AI
Research Digest··3 min read
Dan et al.
What it released
Data
§