Expert-guided decoding restores learning signals when a model’s reasoning attempts fail

The authors report that constraining expert guidance to student-compatible token choices recovers correct training examples in up to 80% of all-failure groups.

Chinese Tech

Zehao Wang · Hanzhu Chen · Runquan Gui · Zhe Wang · Lin Yang

University of Science and Technology of China · Tencent · Shanghai Artificial Intelligence Laboratory

Research Digest·28 October 2026·2 min read

Illustration: Expert-guided decoding restores learning signals when a model’s reasoning attempts fail

In one line

Policy-Compatible Reinforcement Learning recovers verified training examples for hard problems by guiding a model with expert solutions while limiting how far its behavior shifts.

Before you read this

Three ideas this paper builds on

reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.131 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.147 papers →trajectory transferUsing demonstrations or solutions from one task, domain, or agent to improve performance or training in another.23 papers →

Read it as

(RLVR) trains a text-generating model by rewarding answers that can be checked. On very hard problems, every attempt may be wrong, leaving the model with no useful direction. Directly copying expert solutions can also push it toward behavior unlike its own.

Policy-Compatible Reinforcement Learning (PICO) uses expert solutions to guide the model’s next-word choices while keeping those choices close to what the model already tends to produce. It uses this approach when all ordinary attempts fail, then trains on the resulting checked answers alongside ordinary attempts.

PICO recovers verified successful answers in up to 80% of groups where all ordinary attempts fail, and outperforms standard methods across math, general reasoning, and tasks where models take actions. This gives training a useful signal on hard problems without requiring expert guidance when the finished model is used.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

reinforcement learning with verifiable rewards distribution-constrained joint decoding policy compatibility group relative policy optimization trajectory transfer importance sampling