The paper investigates why LLM judges often give incorrect verdicts, finding that the information needed for correct judgment is frequently present in the model's internal activations but not reflected in the final output. Using 64 open-weight evaluators across 14 datasets, the authors show that a small probe reading internal activations achieves significantly higher agreement with human labels (0.846 on LLMBar) than the judges' own verdicts (0.456), even after controlling for surface features like length and position. This internal-output gap persists across eight benchmarks but is not universal, with a simple measure of how well surface features predict human labels strongly correlating with the gain from reading internals (Spearman rho = 0.90).
LLM judges often encode better preferences than their verdicts reveal
Across open-weight evaluators, internal activations recovered human-aligned judgments that surface-biased final answers frequently obscured.
Big Tech
Sourabrata Mukherjee · Sunayana Sitaram
Microsoft Research
Research Digest··3 min read
Thread:Hidden-State Steering
In one line
LLM judges' internal representations often contain correct evaluations even when their final verdict is wrong.
Before you read this
Three ideas this paper builds onmechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.143 papers →activation steeringActivation steering modifies a model's internal vector representations during inference to shift behavior, such as editing outputs, removing capabilities, or controlling trust.83 papers →preference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.69 papers →
Read it as
Mukherjee and Sitaram test whether an incorrect LLM judgment reflects missing knowledge or a failure to express information already represented inside the model.
The authors compared judges’ final verdicts with small supervised probes trained on their frozen internal activations, without updating model weights.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.