The authors mined repository histories to pair confirmed bugs with the pull requests that introduced them.
Code review agents catch many bugs but still need humans
On a benchmark of 1,800 pull requests, the strongest agent found 71.9% of known bugs within its five highest-ranked comments.
Big Tech
Chenyang Yang · Hao He · Ti-Chung Cheng · Jenna Butler · Emerson Murphy-Hill · Carmen Badea · +1 more
Carnegie Mellon University · Microsoft · Microsoft Research
Research Digest··2 min read
Yang et al.
Why this paper
From Microsoft and 2 others
In one line
Review agents catch most human-verified bugs in open-source PRs but drop sharply on proprietary repositories, so a fully lights-off factory is not yet practical.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks (4 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§