← All concepts

item response theory

Think of measuring a hidden trait from test answers. Each question has difficulty and discrimination: how hard it is and how sharply it separates strong from weak respondents. Rather than summing points, the model finds the trait level that best explains the pattern of correct and wrong answers. It also tells you which questions give the most information.

The papers use this for adaptive evaluation and training. One builds reward signals from rubric criteria, another chooses or creates coding tasks until ability is estimated, and others audit whether safety benchmarks measure one underlying trait. The common thread: items differ in difficulty and information, so treating them equally is wasteful. Item response theory highlights which items matter and which are saturated.

Papers
6
Released code
0
First seen
Sept 2026
Latest
Nov 2026

6 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Item response modeling improves rubric rewards while cutting judge requests

    Rubric Response Theory aggregates rubric criteria using item response theory to improve reinforcement learning rewards over additive methods.

    Sept 2026
  2. 2Adaptive testing exposes major security repair gaps in coding agents

    SecProbe adaptively evaluates coding agents on cybersecurity vulnerabilities, requiring up to 29.5% fewer tasks while maintaining comparable ability estimates.

    Sept 2026
  3. 3Trust-weighted average of AI judges stays accurate on small evaluation sets

    AGORA aggregates LLM judges by weighting each judge's reliability, outperforming equal-weight averaging and confusion-table models on small evaluation sets.

    Nov 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.