This paper introduces CRJudgeBench, a new to test whether AI systems can determine if a code review comment is technically correct. The benchmark contains 1,199 instances built from real pull requests and expert-modified comments, including both trustworthy comments and plausible but untrustworthy ones. The authors also built Sentinel, an AI judge that actively searches the code repository for evidence before making a judgment. Sentinel achieved 76.60% accuracy on the test set, significantly outperforming its base model and a larger general-purpose model, showing that identifying untrustworthy comments is a difficult task that requires specialized training.
New benchmark tests AI's ability to spot invalid code reviews
CRJudgeBench evaluates whether agents can correctly judge technical trustworthiness of plausible but incorrect review comments.
Big Tech
Yue Pan · Jiawei Li · Ziyuan Zhang · Xiangxin Zhao · He Ye
University College London · Amazon · Zhejiang University
Research Digest··2 min read
Thread:Process Agent Benchmarks
In one line
AI agents can judge if a code review comment is technically correct by gathering repository evidence, outperforming general LLMs.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →evidence retrievalPulling specific, relevant information from a large corpus to support an answer, decision, or reasoning step.115 papers →privileged informationInformation available only during training that improves performance but is not accessible at inference.85 papers →
Read it as
The authors introduce CRJudgeBench, a benchmark of 1199 instances built from real pull requests and expert-verified perturbations, assessing whether AI agents can distinguish technically valid code review comments from plausible but invalid ones.
The authors constructed CRJudgeBench from real pull requests (PRs) across diverse repositories, collecting both trustworthy review comments and manually crafted untrustworthy comments that are plausible but contain technically incorrect claims.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Data