Language models can improve answers without better internal error recognition

Tracking one small model through arithmetic training, the authors find modest accuracy gains but no measurable improvement in internal signals distinguishing flawed reasoning.

Academic

Shreeyam Bangera · Manya Gupta

Indian Institute of Technology Roorkee

Research Digest·3 November 2026·2 min read

Thread:LLM Output Correctness Estimation

Illustration: Language models can improve answers without better internal error recognition

In one line

RLVR improves answers by sharpening output selection, not by teaching better reasoning.

Before you read this

Three ideas this paper builds on

reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.144 papers →mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.104 papers →representation probingUsing linear classifiers or decoders to extract features from model hidden states for analysis or task improvement.4 papers →

Read it as

When AI models are trained using (RLVR), they become better at solving reasoning problems. But why? Does the model actually learn better reasoning, or does it just get better at picking correct answers from what it already knows?

Researchers tracked a small model across all stages of RLVR training on grade-school math. They planted reasoning mistakes and tested whether the model's internal understanding of errors improved. They also compared how the model worked at the final answer step versus earlier reasoning steps.

The model did not improve its internal detection of reasoning errors; only the final answer step showed change. RLVR works by sharpening the model's ability to select the best answer it can already produce, not by teaching it new reasoning. This shows that RLVR improves output selection, not the reasoning process itself.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

reinforcement learning with verifiable rewards mechanistic interpretability representation probing sharpening chain of thought test-time scaling