When multiple large language models (LLMs) are used as judges to score answers, they often agree with each other. But this agreement can come from a shared bias toward long or well-formatted answers, not from actual quality. From their scores alone, you cannot tell if agreement means the answer is truly better or just looks better.
The researchers created a method called VoirDire. It gives the original answer and a rewritten version to the same LLM judges. The rewrites change surface features (like length or formatting) while keeping the content's quality. By comparing score pairs, VoirDire learns which judge agreement comes from those surface features and downplays it.
On 12 different evaluation tasks, VoirDire reduced average error by up to 26.8% for scoring tasks and improved accuracy by up to 5.3 percentage points for classification and preference tasks. This shows that actively testing for known biases can make LLM judge panels more reliable.