item response theory
Think of measuring a hidden trait from test answers. Each question has difficulty and discrimination: how hard it is and how sharply it separates strong from weak respondents. Rather than summing points, the model finds the trait level that best explains the pattern of correct and wrong answers. It also tells you which questions give the most information.
The papers use this for adaptive evaluation and training. One builds reward signals from rubric criteria, another chooses or creates coding tasks until ability is estimated, and others audit whether safety benchmarks measure one underlying trait. The common thread: items differ in difficulty and information, so treating them equally is wasteful. Item response theory highlights which items matter and which are saturated.
- Papers
- 6
- Released code
- 0
- First seen
- Sept 2026
- Latest
- Nov 2026
6 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Item response modeling improves rubric rewards while cutting judge requests
Rubric Response Theory aggregates rubric criteria using item response theory to improve reinforcement learning rewards over additive methods.
Sept 2026 - 2Adaptive testing exposes major security repair gaps in coding agents
SecProbe adaptively evaluates coding agents on cybersecurity vulnerabilities, requiring up to 29.5% fewer tasks while maintaining comparable ability estimates.
Sept 2026 - 3Trust-weighted average of AI judges stays accurate on small evaluation sets
AGORA aggregates LLM judges by weighting each judge's reliability, outperforming equal-weight averaging and confusion-table models on small evaluation sets.
Nov 2026
How fast it is moving
Often studied with
benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.2 togetheraudit workflowA systematic process for verifying data integrity, model behavior, and evaluation validity through checks for errors, confounds, and hidden artifacts.1 togethercoding agentAn AI-powered software system that autonomously writes, modifies, or extends source code based on high-level instructions.1 togetherThe papers
Most central to this idea first, not most recent.
- Academiccs.AI
Trust-weighted average of AI judges stays accurate on small evaluation sets
AGORA aggregates LLM judges by weighting each judge's reliability, outperforming equal-weight averaging and confusion-table models on small evaluation sets.
University of Illinois Urbana-Champaign · Nov 2026
- Big Techcs.CL
Item response modeling improves rubric rewards while cutting judge requests
Rubric Response Theory aggregates rubric criteria using item response theory to improve reinforcement learning rewards over additive methods.
University of British Columbia, Microsoft · Sept 2026
- Big Techcs.AI
HELM Safety's HarmBench fails to measure a single harmful refusal construct.
HarmBench scores collapse distinct harm behaviors, not measuring a single harmful refusal attribute.
Carnegie Mellon University, Indiana University · Oct 2026
- Industrycs.AI
Commercial LLM routers fail to beat carefully chosen random routing
Dynamic LLM routers fail to outperform random routing between two well-chosen models at matched cost.
Fastino Labs · Oct 2026
- Top Universitycs.CR
Adaptive testing exposes major security repair gaps in coding agents
SecProbe adaptively evaluates coding agents on cybersecurity vulnerabilities, requiring up to 29.5% fewer tasks while maintaining comparable ability estimates.
University of Notre Dame, Bake AI · Sept 2026
- Research Labcs.AI
Transcripts from saturated benchmarks still predict model capability
Correctly solved transcripts distinguish model capabilities after benchmark accuracy saturates, with pairwise judging cutting prediction error 5.27 times versus score-only estimates.
LASR Labs, ELLIS Institute Tübingen · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.