The authors selected 1,262 performance-related pull requests from 71,677 agent PRs in the AIDev v4 dataset.
Merged coding-agent performance fixes often miss their claimed gains
Across 1,262 pull requests, acceptance reflected repositories’ prior experience with an agent more strongly than tests, measurements or the type of optimization.
Research Lab
Zhenyu Qi · Haotang Li · Jinfu Chen · Huashan Chen · Yutong Zhao · Derui Zhu · +3 more
University of Arizona · Wuhan University · Institute of Information Engineering, Chinese Academy of Sciences · California State University, Long Beach · Rochester Institute of Technology
Research Digest··2 min read
Qi and colleagues examined how maintainers handle performance fixes submitted by coding agents, then reran a subset of accepted and rejected patches.
Why this paper
From Institute of Information Engineering, Chinese Academy of Sciences and 4 others
In one line
Coding agents' performance fixes are often merged without actually delivering the claimed speedup.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§