Language models can learn even when every sampled answer is wrong

Chen and colleagues report that internal steering directions can improve supervised training and restore a learning signal for reinforcement learning when all sampled answers are incorrect.

Big Tech

Tiejin Chen · Maunil Vyas · Huaiyuan Yao · Hua Wei

Arizona State University · Amazon

Research Digest·2 November 2026·2 min read

Illustration: Language models can learn even when every sampled answer is wrong

In one line

Steering vectors added to hidden states during training improve LLM post-training, especially when distilling from thinking teachers.

Before you read this

Three ideas this paper builds on

activation steeringActivation steering modifies a model's internal vector representations during inference to shift behavior, such as editing outputs, removing capabilities, or controlling trust.54 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.141 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.160 papers →

Read it as

A big challenge in improving large language models (LLMs) after their initial training is that the training data may not match the model’s own behavior, and reward signals can be very sparse. This makes it hard for the model to learn effectively.

The researchers test a method called “steering vectors”, directions in the model’s internal representation, that are added during training to guide updates. They try this in both supervised fine-tuning (learning from examples) and reinforcement learning (learning from rewards), focusing on math and reasoning tasks.

Steering vectors improved performance by up to 12% when distilling from a more advanced “thinking” teacher model. They also helped recover learning signals when all sampled responses were wrong in reinforcement learning. These gains work across multiple random trials, though the method fails on code generation and requires careful tuning of which layer to use.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

activation steering supervised fine-tuning knowledge distillation reinforcement learning with verifiable rewards group relative policy optimization advantage estimation