LLM-as-a-judge validation usually compares judge outputs with human labels on a calibration set, but annotation budgets force sparse overlap between the judge and humans. The authors prove analytically that overlap sparsity, not the agreement coefficient chosen, dominates wrong deployment decisions. At 5% pairwise overlap, wrong-decision rates reach 25%, and the chance of selecting the wrong best judge among ten candidates is 65%. They derive a minimum-overlap rule: an overlap fraction of at least 0.25 is enough for non-borderline judges, while borderline cases remain fundamentally hard. They also show that a zero-cost stratified allocation of shared labels halves false-rejection rates relative to random sampling when strata are informative. The findings are validated on 10 LLM judges across four evaluation matrices spanning visual assessment, causal reasoning, and summarization. This matters because validation budgets are always limited, and the result tells teams where to spend annotations and what to expect.
Sparse Human Label Overlap Undermines Reliable LLM Judge Validation
The authors show that shared annotation coverage, more than the chosen agreement metric, determines whether validation produces correct deployment and ranking decisions.
Big Tech
Junxuan Li · Arko Mukherjee · Soumyabrata Pal
Adobe · Adobe Research
Research Digest··3 min read
In one line
Sparse human-annotator overlap drives wrong LLM-judge deployment decisions: at 5% pairwise overlap error is 25%, and 0.25 overlap suffices for non-borderline judges.
Read it as
Li, Mukherjee and Pal study how reliably teams can validate an LLM judge when only a small fraction of items receive labels from both humans and the model.
The authors formalized judge validation as a decision problem based on estimated human-model agreement.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.