Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
Safety & Evals

LLM judges as measurement instruments fail reliability tests on shared endpoints

Over 52,988 request attempts, the authors find that same-window repeat rankings agree at Spearman 0.400 (required 0.90) and next-day replays at 0.78 (required 0.99), with neither metric substitution nor sampling repairing the issue.

PaperTop Universitycs.AIarXiv:2609.04198v1 ↗
Haoyaun Zhu · Jie Zhang

University of Sheffield · Ranplan Wireless Network Design Ltd. · Cambridge AI+ Ltd.

Research Digest··3 min read
Read the paper →
Haoyuan Zhu and Jie Zhang preregistered two evaluation campaigns that assumed LLM judges (used to score generations, filter training data, or rank leaderboards) would produce stable outputs for identical inputs on shared endpoints. They found that the instruments themselves failed validation: same-window repeat rankings reached only Spearman 0.400 against a 0.90 threshold, and byte-identical next-day replays reached 0.78 against 0.99. The authors document three mechanisms—a label-to-meaning mapping bias, candidate gaps below the noise floor, and input-identical ranking noise—and show that neither metric substitution nor sampling fixes the problem.

What they did

The authors designed two preregistered campaigns to test whether a black-box LLM observer could reliably read solution progress from partial reasoning traces. All thresholds, sample sizes, and analysis plans were fixed before data collection. They audited 52,988 request attempts across shared serving endpoints (e.g., API calls to named models). The core analyses used 31 valid task groups, 100 replay pairs, ten windows per supplementary arm, and 3,060 constructed-error judgments. Execution record (delivery rates, schema validity, request hashes, recorded metadata) was at ceiling, indicating the failures were not due to engineering errors.

Key findings

  • Same-window repeat rankings (identical requests within a short time window) agreed at Spearman ρ = 0.400 against a required 0.90 threshold.
  • Byte-identical next-day replays agreed at Spearman ρ = 0.78 against a required 0.99 threshold.
  • Three mechanisms explained the gap: (1) a label-to-meaning mapping that biased readouts as strongly as the signal; (2) candidate gaps seven orders of magnitude below the instrument's noise floor; (3) byte-identical inputs returning different rankings, a noise that exact-permutation readouts compound.
  • Preregistered follow-ups showed that waiting (ρ=0.805 vs 0.800, replicated over five days) and switching providers (medians 0.74–0.88 across four providers) did not repair reliability; self-hosting on batch-invariant kernels helped only while the server was quiet.

Why it matters

These results challenge a foundational assumption in LLM-as-judge research: that a model name on a shared endpoint corresponds to a stable measurement instrument. The authors show that even with perfect execution, the instrument's noise floor can overwhelm the signal, making it untrustworthy for tasks like training data filtering, generation scoring, or leaderboard computation. The paper provides a three-level snapsho-identity ladder, eight design rules, and a reporting checklist to help future studies validate their instruments before freezing evaluation gates.

Caveats

All measurements were taken on externally observed behavior of shared serving infrastructure; results may differ for private or dedicated endpoints. The study tested a specific set of models and tasks (solution progress reading from partial reasoning traces); other judge tasks or model families might show different reliability profiles. The authors note that a pilot at roughly 2% of the study's call volume would have exposed the unreachable gates, suggesting that many published results using LLM judges may be at risk.

§

Analysis

This work adds to a growing line of critiques of LLM-as-judge evaluations. While earlier papers have noted prompt sensitivity or position bias, Zhu and Zhang document a deeper failure: the instrument is not a fixed function of the input. The finding that switching providers or waiting does not help suggests the unreliability is structural to shared API endpoints rather than transient load. The reporting checklist may become a standard for future preregistered evaluations.

newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe