Evaluating AI systems often requires human judges, but that is expensive. A common cheaper alternative is using several large language models (LLMs), or "LLM judges," to score items, then averaging their scores. However, this average gives equal weight to weak and strong judges, and other statistical methods need too many data points to learn which judges are reliable.
This paper introduces AGORA, a method that treats each scored item as having a hidden quality. AGORA learns a "reliability weight" for each judge, how steeply its scores track that hidden quality, so unreliable judges are down-weighted. It also models how each judge uses the rating scale, using fewer parameters than older methods.
On three benchmarks, AGORA matched or beat existing methods on large evaluation sets. Its advantage grew as sets shrank, outperforming a standard method (Dawid-Skene) on 98–100% of random five-judge pools with just 50 items. A new statistic can predict, without human labels, when this trust-weighting helps.