The paper studies whether language model judges reliably evaluate agent upgrades compared to their predecessors. It analyzes 35 coding-agent submissions, two customer-service agents, and over a thousand expert-labeled trajectories. Results show that fixed judges produce version-dependent mistakes, and that false acceptance of failed patches increases with agent capability.
Fixed LLM judges can mismeasure upgrades as agents change
Across coding and customer-service benchmarks, judge errors depended on the agent version being evaluated, sometimes producing upgrade conclusions unsupported by execution-based rewards.
Independent
Jiapeng Li
Research Digest··3 min read
In one line
Fixed LLM judges make version-dependent errors when evaluating agent upgrades, making paired audits preferable.
Before you read this
Three ideas this paper builds onagentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real304 papers →benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →audit workflowA systematic process for verifying data integrity, model behavior, and evaluation validity through checks for errors, confounds, and hidden artifacts.116 papers →
Read it as
Li tests whether a language-model judge calibrated on one agent version remains reliable when comparing newer versions.
The study evaluated 35 public coding-agent submissions, organized into 20 prespecified version pairs, on 250 SWE-bench Verified issues.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.