← All threads

Reasoning Distillation Alignment

Methods for aligning teacher and student distributions in reasoning distillation to close capacity gaps.

58 papers · 1 months

Where this stands

The written synthesis of this thread is for subscribers. Subscribe.

Results across this thread

15 reported results from the papers in this thread.

PaperBenchmarkMetricResultConditionsCode

The table is for subscribers. Subscribe to see every reported number side by side.

How this thread developed

  1. Oct 2026 · UT Austin, Together AI

    Small models can steer stronger ones through shared reasoning traces

    Transfers reasoning behavior from a small RL model to larger models through alternating chains of thought, reducing the capacity gap.

  2. 2 further papers

    Oct 2026 · Microsoft

    Post-training multi-token prediction heads match joint pretraining with far less data

    Shows that post-training multi-token prediction heads on target-generated chain-of-thought data matches jointly pretrained MTP performance.

  3. 2 further papers

    Oct 2026 · Tencent Inc., University of Science and Technology of China

    Blending teacher and student token distributions improves distillation of language models.

    Proposes Interpolated Policy Distillation that blends teacher and student distributions during rollout for improved distillation.

  4. 2 further papers

    Oct 2026

    Smaller frozen models provide better rejected responses for preference distillation

    Analyzes the optimal source of rejected responses for preference distillation, discovering that smaller frozen models consistently outperform student-generated samples across scales from 7B to 72B.

  5. 1 further paper

    Oct 2026 · Institute of Automation, Chinese Academy of Sciences, University of Chinese Academy of Sciences

    Distilled task updates can merge better than stronger teacher updates

    Shows that distilled student updates can complement RL teacher updates when merging, advancing understanding of distillation alignment.

  6. Oct 2026 · Stony Brook University, Amazon

    Safety distillation can pass hidden backdoors from teacher to student

    Shows that on-policy safety distillation can transmit hidden backdoors from teacher to student, highlighting an alignment failure.

  7. 1 further paper

    Oct 2026 · Cohere, Mila, Université de Montréal

    Lexicographic distillation preserves priority-ordered capabilities in multi-objective RL

    Uses a lexicographic priority order to route student rollouts to the best specialist teacher, preserving priority-ordered capabilities in multi-objective distillation.

  8. 1 further paper

    Oct 2026 · Stanford University

    Divergence choice in distillation controls student model entropy

    Proves that forward KL inflates student entropy above teacher's by residual divergence, providing a principled way to control entropy in distillation.

  9. 1 further paper

    Oct 2026 · Alibaba Cloud Computing

    Supervision reliability trumps alignment coverage in cross-tokenizer distillation

    Finds on-policy distillation with next-token prediction outperforms strict token alignment for reasoning tasks.

  10. Oct 2026 · University of Illinois at Urbana-Champaign, Tsinghua University

    Formal provers select methods better when methods state how they apply

    Introduces Method Contracts for formal theorem proving that generate problem-specific proposals before ranking, enabling better method selection in formal reasoning.

  11. 2 further papers

    Oct 2026

    Offline student rollouts often beat on-policy distillation for language models

    Shows offline student rollouts outperform on-policy distillation and proposes Semi-OPD method.

  12. 1 further paper

    Oct 2026 · Redwood Research

    Distillation can expose hidden behavior or retain safer capabilities

    Tests distillation as a tool to expose hidden secret-keeping behavior in models, linking distillation to safety evaluation.

  13. Oct 2026 · Alibaba Group, Tsinghua University

    Grounding distillation in interaction states improves multi-turn agent performance

    Identifies and addresses the failure of on-policy distillation in multi-turn agents by removing student reasoning from teacher context.

  14. 1 further paper

    Oct 2026 · UC San Diego, Purdue University

    Reward-guided token weighting makes language-model distillation more effective

    Reward-Aligned Distillation uses token-level reward signals from a stronger model to guide the student's token weighting, improving distillation.

  15. 3 further papers

    Oct 2026

    Top-k distillation's renormalized gradient ignores tail mass

    Identifies zero gradient along tail-mass directions in top-k distillation, a fundamental issue that affects alignment in reasoning distillation.

    released code

  16. Oct 2026 · Google, University of Washington

    Adaptive distillation triggers teacher takeover when student diverges

    proposes DTDD that rewinds and has teacher regenerate when student diverges, closing alignment gaps.

  17. 3 further papers

    Oct 2026 · Shanghai Jiao Tong University, Alibaba Group

    Sequence-level credit assignment improves on-policy distillation of reasoning models

    Proposes sequence-level credit assignment to improve on-policy distillation of reasoning models.

  18. 1 further paper

    Oct 2026 · University of Southern California, University of Florida

    Distillation overshoot lets smaller agents surpass teachers, and a new method preserves those gains.

    Identifies a distillation overshoot phenomenon where smaller agents temporarily surpass teachers, proposes a method to preserve gains.

  19. 1 further paper

    Oct 2026 · NVIDIA

    Teacher rollouts repair myopic token scoring in agent distillation

    Identifies a myopic token scoring failure in agentic on-policy distillation and fixes it with teacher rollouts.

  20. 11 further papers

    Nov 2026 · North Carolina State University, LG AI Research

    Omitting reasoning from training data does not harm AI agents on standard benchmarks

    Finds that discarding reasoning steps from training data does not harm agents on standard benchmarks, questioning the need for thought supervision in distillation.

  21. 1 further paper

    Nov 2026 · Tsinghua University, Harbin Institute of Technology, Shenzhen, China

    Language models improve at math when teachers steer their practice

    Steers teacher generations to supervise reasoning paths students are unlikely to enter, improving math performance.

  22. 2 further papers

    Nov 2026 · Salesforce AI Research, UIUC

    Merging specialist AI models saves 42% of training compute compared to joint training

    Compares on-policy distillation and GRPO for merging specialist models, finding OPD saves 42% compute over joint training.

22 of 58 papers shown