Ayyagari tested the reproducibility of LLM-as-Judge evaluations by submitting identical benchmark inputs (from Arena-Hard, AlpacaEval 2, and MT-Bench) to four frontier judges served via a single major enterprise cloud platform, all at temperature zero and with a constant serving-reported model version.
LLM judges give inconsistent verdicts even at zero temperature
Identical inputs to the same model judge on cloud infrastructure produce different outputs across re-runs, degrading leaderboard precision.
Independent
Krishna Chytanya Ayyagari
Research Digest··3 min read
Ayyagari shows that pinning a judge to a fixed model version with temperature zero does not guarantee reproducible verdicts: identical inputs to four frontier judges on a major cloud platform yield per-item flip rates of roughly 5% on average and about 40% on close-call items that decide leaderboard margins.
Why this paper
Independent
In one line
LLM judges produce different verdicts for identical inputs at temperature zero due to cloud infrastructure.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§