Distillation Before Reinforcement Learning Improves Reasoning Model Post-Training

A two-stage distillation-then-RL schedule outperformed either method alone and approaches that optimize both objectives simultaneously.

Top University

Boyan Li · Bingsen Chen · Chenghao Yang · Ping Nie · Chen Zhao · Xi Ye

New York University · University of Chicago · University of Waterloo · University of Alberta · NYU Shanghai

Research Digest·4 September 2026·2 min read

Thread:Credit Assignment in Agentic RL

In one line

A two-stage scheme of on-policy distillation followed by reinforcement learning outperforms joint optimization for reasoning LLMs.

Before you read this

Two ideas this paper builds on

on-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.110 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.138 papers →

Read it as

The paper compares two dominant post-training methods for reasoning LLMs: (RLVR) and (OPD). Prior work combines them within a single step, either by weighted addition or teacher-modulated rescaling. The authors test a simple alternative: apply OPD first, then RLVR in sequence.

Across logic and math reasoning benchmarks, the sequential OPD-then-RL scheme consistently beats pure OPD, pure RLVR, and all joint optimization baselines, with up to a 26.7 pass@1 margin on logic tasks. Analysis reveals why: OPD expands the student’s coverage of teacher-supported solutions, and RL then sharpens within that support, while joint optimization causes the two signals to interfere.

This is interesting because it provides a practical recipe for combining two complementary techniques without the complexity of joint training. The authors also find that the OPD validation score reliably indicates when to switch to RL, and that OPD is a better cold start for RL than standard supervised fine-tuning.

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

What it released

Code

Concepts

on-policy distillation reinforcement learning with verifiable rewards distillation-reinforcement learning interplay