AI judges reverse correct verdicts after a false note

Adding a single sentence citing a prior reviewer overturned up to 40% of tool-using judges' correct decisions across five datasets and four models.

Independent

Drake Benjamin Brown · Khushbu Pahwa · Yubin Ge · Salvatore Romeo · Akhil Agnihotri · Yi Zhang · +2 more

Research Digest·5 November 2026·2 min read

In one line

A false authority cue overturns up to 40% of a grounded AI judge's correct verdicts.

Before you read this

Two ideas this paper builds on

agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real261 papers →failure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.123 papers →

Read it as

To check whether an AI agent completed a task, engineers often ask a second AI (a judge) to verify by looking at the evidence. This should make the process reliable.

Researchers added a single false hint to the judge's instructions, suggesting the agent had failed, while still letting the judge run its own verification commands.

This hint overturned up to 40% of correct verdicts. The judge often found the right answer but overrode it to defer to the hint. Trusting verification tools as a guarantee against bias is not safe.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

confirmation bias llm as a judge agentic evaluation tool invocation failure attribution over-trust