1 as the judge under eight combinations of three binary choices: evaluation criteria (Arena-Hard default vs.
LLM judge scores can stay stable while individual verdicts shift substantially
Across 500 held prompts, re-running the same judge changed 21.2% of outcomes despite a one-point aggregate move.
Academic
Louise Yorke
University College London
Research Digest··2 min read
The authors show that aggregate win rates from LLM judges can remain nearly unchanged while individual verdicts change dramatically.
§