The authors tested two assumptions about LLM judges using checklist rubrics: that leniency belongs to the judge model, and that rubrics can be improved by asking the model to review them.
Checklist rubric content dominates LLM judge strictness, not the model
Authors show that model self-review of rubrics drifts standards, and Shapley attribution can gate revisions.
Big Tech
Saeid Asgari · Rui Ying · Sara Malvar · Rakshanda Agarwal · Bruce Sun · Claire Wang · +2 more
Microsoft
Research Digest··3 min read
Asgari et al.
§