The authors developed Rubric Response Theory (RRT), which treats binary rubric verdicts as evidence about a response’s latent quality.
Item response modeling improves rubric rewards while cutting judge requests
Rubric Response Theory estimates response quality from criterion verdict patterns and adaptively selects the most informative criteria.
Big Tech
Milad Yazdani · Yaser Souri · Xiren Zhou · Pranit Chawla · Dena Shahriari · Subhojit Som · +1 more
University of British Columbia · Microsoft
Research Digest··2 min read
The authors replace point-based rubric aggregation with a two-parameter item response model that learns each criterion’s difficulty and ability to distinguish response quality.
Why this paper
From Microsoft and University of British Columbia
In one line
Rubric Response Theory aggregates rubric criteria using item response theory to improve reinforcement learning rewards over additive methods.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§