The authors tested six frontier models on four benchmarks spanning subjective and objective comparisons, including adversarially difficult cases.
LLM judges can be consistent yet wrong and order-sensitive
Across a large stress test, repeated and reordered evaluations exposed failures that conventional accuracy and single-shot checks often miss.
Industry
Vineet Kumar · Darshita Rathore · Anindya Moitra
PayPal Artificial Intelligence · PayPal
Research Digest··2 min read
Kumar, Rathore and Moitra audit whether LLM judges produce verdicts that are reproducible, unaffected by candidate order and correct.
Why this paper
From PayPal Artificial Intelligence and PayPal
In one line
LLM judge verdicts are not reliable: they vary across identical runs, flip with presentation order, and determinism can hide near-chance accuracy.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§