What they did

They constructed MCR-Bench, a benchmark covering five commonly used programming languages (Python, Java, C++, JavaScript, Go) with 2,269 multi-round code review tasks sourced from real development histories. Each task includes fine-grained defect metadata (description, type, severity) and cross-round state annotations that capture the complete evolutionary trajectory of a defect. The authors then evaluated several mainstream large language models on defect detection and defect lifecycle state tracking across these multi-round sequences.

Key findings

• LLMs exhibit limited overall capability: performance on defect detection and state tracking degrades substantially as the number of interaction rounds increases.
• Performance varies significantly across defect types and severity levels; semantically complex or low-salience defects are far more likely to be missed.
• Error analysis identifies two underlying failure mechanisms: cross-round temporal misalignment (inconsistent tracking across rounds) and inadequate long-range memory (failure to recall earlier defect states).

Why it matters

This benchmark provides a more ecologically valid evaluation of LLMs for code review, shifting from static single-round tasks to the dynamic, iterative nature of real-world review. The findings challenge the assumption that current LLMs can handle multi-turn interaction and highlight the need for improved memory and temporal reasoning in code review automation.

Caveats

The benchmark focuses narrowly on defect detection and state tracking; it does not assess other important aspects of code review such as generating constructive comments or suggesting fixes. The set of LLMs tested is representative but not exhaustive, and the benchmark itself may not cover all possible defect types or review scenarios.