This paper reports a preregistered audit of the assumption that black-box LLM judges (e.g., GPT models) produce stable measurements over time. Across 52,988 request attempts, repeat rankings agreed at Spearman 0.400 (required 0.90) and byte-identical next-day replays at 0.78 (required 0.99). Three failure mechanisms were identified: label-to-meaning mapping bias, candidate gaps below the noise floor, and byte-identical inputs returning different rankings. Neither metric substitution nor increased sampling repaired the instability. The findings are interesting because they challenge the foundational assumption that model-as-judge evaluations are replicable on shared infrastructure, affecting leaderboards, training data gating, and research reproducibility.
LLM judges as measurement instruments fail reliability tests on shared endpoints
Over 52,988 request attempts, the authors find that same-window repeat rankings agree at Spearman 0.400 (required 0.90) and next-day replays at 0.78 (required 0.99), with neither metric substitution nor sampling repairing the issue.
Top University
Haoyaun Zhu · Jie Zhang
University of Sheffield · Ranplan Wireless Network Design Ltd. · Cambridge AI+ Ltd.
Research Digest··3 min read
In one line
Black-box LLM judges on shared endpoints are unreliable measurement instruments due to unstable rankings and biased readouts.
Before you read this
Two ideas this paper builds onRead it as
Haoyuan Zhu and Jie Zhang preregistered two evaluation campaigns that assumed LLM judges (used to score generations, filter training data, or rank leaderboards) would produce stable outputs for identical inputs on shared endpoints.
The authors designed two preregistered campaigns to test whether a black-box LLM observer could reliably read solution progress from partial reasoning traces.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.