Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
RL for Agents

Distillation Before Reinforcement Learning Improves Reasoning Model Post-Training

A two-stage distillation-then-RL schedule outperformed either method alone and approaches that optimize both objectives simultaneously.

PaperTop Universitycs.CLarXiv:2609.04108v1 ↗
Boyan Li · Bingsen Chen · Chenghao Yang · Ping Nie · Chen Zhao · Xi Ye

New York University · University of Chicago · University of Waterloo · University of Alberta · NYU Shanghai

Research Digest··2 min read
Read the paper →
Thread:Credit Assignment in Agentic RL
Li and colleagues compare ways to combine on-policy distillation (OPD), which supplies dense teacher feedback at the token level, with reinforcement learning from verifiable rewards (RLVR), which uses sparse outcome-based signals. Across logic and mathematics benchmarks, they find that completing OPD before switching to RLVR works consistently better than pure OPD, pure RLVR, or joint optimization.

What they did

The authors evaluated a sequential OPD-then-RLVR training scheme against standalone OPD, standalone RLVR, and two types of joint training: weighted addition of the objectives and teacher-modulated rescaling of the RL advantage.

They examined logic and mathematical reasoning performance, pass@k behavior—the chance of solving a problem within k attempts—learning dynamics, and parameter updates. They also investigated how to choose the transition point between stages and compared OPD with supervised fine-tuning as an initialization for RLVR.

Key findings

  • OPD-then-RLVR consistently outperformed pure OPD, pure RLVR, and the tested joint baselines across the evaluated logic and mathematics benchmarks.
  • The authors report that OPD broadens the student's coverage of solutions supported by the teacher, while subsequent RLVR concentrates probability on successful solutions within that support.
  • Simultaneously optimizing the distillation and RL signals produced interference in the observed learning dynamics and parameter updates.
  • OPD validation performance was the most useful signal for deciding when to switch stages, and OPD provided a better RLVR cold start than supervised fine-tuning.

Why it matters

The study suggests that dense imitation feedback and sparse verifiable rewards need not be engineered into one objective. Treating them as successive stages may offer a simpler and more effective recipe for post-training reasoning models, while clarifying that the two signals play different roles in expanding and then refining solution behavior.

Caveats

The supplied abstract does not identify the models, teachers, datasets, compute budgets, exact effect sizes, or statistical uncertainty, so the breadth and practical cost of the reported advantage cannot be assessed here. It also remains unclear whether the sequential advantage extends beyond verifiable logic and mathematics tasks or holds across substantially different model scales and teacher-student configurations.

§
newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe