Ding defines a “specification-oracle gap” as a mismatch between a task description and the tests used to grade it.
Coding benchmarks can penalise agents for details their instructions omit
An audit finds unstated names or paths in 25 of 98 SWE-Milestone tasks, with higher failure rates but no substantial change to the overall leaderboard ranking.
Chinese Tech
Fei Ding
Alibaba Group
Research Digest··2 min read
Fei Ding examines whether coding-agent benchmarks test only what their task descriptions actually request.
§