Real-world code review is iterative, with developers and reviewers going back and forth. Existing benchmarks treat it as a single event. This paper introduces MCR-Bench, 2,269 multi-round code review tasks across five languages, each annotated with defect metadata and state changes across rounds. Testing mainstream LLMs on this benchmark shows they have limited ability to detect defects and track their status across multiple rounds, with performance dropping significantly as rounds increase. Performance also varies widely by defect type and severity, and the authors identify specific failure patterns like misalignment across rounds and poor long-range memory.
LLMs fail at multi-round code review in new realistic benchmark
MCR-Bench tracks defect evolution across rounds, exposing poor LLM performance.
Chinese Tech
Dewu Zheng · Yanlin Wang · Xiwen Wang · Kefeng Duan · Hongyu Zhang · Xilin Liu · +2 more
Sun Yat-sen University · Chongqing University · Huawei Cloud Computing Technologies Co., Ltd.
Research Digest··2 min read
Thread:Agent Harness Optimization
In one line
Large language models perform poorly on multi-round code review tasks, especially as rounds increase and for complex defects.
Read it as
The authors introduce MCR-Bench, a benchmark with 2,269 real-world multi-round code review tasks annotated with fine-grained defect information and dynamic state labels.
They constructed MCR-Bench, a benchmark covering five commonly used programming languages (Python, Java, C++, JavaScript, Go) with 2,269 multi-round code review tasks sourced from real development histories.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.