Training LLM judges for subjective tasks is challenging because outcome-supervised RL (e.g., GRPO) gives only a single scalar reward per final verdict, ignoring the rich language feedback (e.g., preference rationales) that often accompanies preference labels. This feedback explicitly names evaluation criteria crucial for subjective tasks where criterion choice drives the verdict. The authors propose position-selective self-distillation: the same model conditioned on language feedback acts as a teacher providing dense per-position supervision. They analyze per-position entropy shift between teacher and student distributions, identifying two regimes: context sharpening (teacher concentrates on one specific expression) and context spreading (teacher distributes across multiple alternatives). They argue sharpening encourages memorization while spreading promotes semantic understanding. To improve generalization, they mask positions with high entropy shift, retaining only the lower tail. Experiments show masking improves out-of-distribution generalization over naive self-distillation. Their method outperforms outcome-supervised RL by 2 to 9 percentage points on subjective subcategories of evaluation benchmarks while remaining competitive on objective ones.
Selective self-distillation helps LLM judges generalize on subjective evaluations
Filtering token-level supervision by entropy shift improved out-of-distribution judgments, especially when evaluation criteria were subjective.
Big Tech
Ilgee Hong · Changlong Yu · Zhenghao Xu · Xin Liu · Yuwei Zhang · Qin Lu · +2 more
Georgia Institute of Technology · Amazon · UC San Diego
Research Digest··3 min read
In one line
Training LLM judges with self-distillation and masking high entropy shift positions improves subjective task accuracy by 2 to 9 percentage points.
Before you read this
Two ideas this paper builds onon-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.131 papers →credit assignmentDetermining which actions or steps in a sequence contributed to a final outcome, often for reinforcement learning.190 papers →
Read it as
Hong et al. train LLM judges using preference rationales and rubrics rather than relying only on rewards for correct final verdicts.
The authors use a model as both student and teacher.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.