Reasoning Distillation Alignment
Methods for aligning teacher and student distributions in reasoning distillation to close capacity gaps.
58 papers · 1 months
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
Results across this thread
15 reported results from the papers in this thread.
The table is for subscribers. Subscribe to see every reported number side by side.
How this thread developed
Oct 2026 · UT Austin, Together AI
Small models can steer stronger ones through shared reasoning traces
Transfers reasoning behavior from a small RL model to larger models through alternating chains of thought, reducing the capacity gap.
2 further papers
Oct 2026 · Microsoft
Post-training multi-token prediction heads match joint pretraining with far less data
Shows that post-training multi-token prediction heads on target-generated chain-of-thought data matches jointly pretrained MTP performance.
2 further papers
Oct 2026 · Tencent Inc., University of Science and Technology of China
Blending teacher and student token distributions improves distillation of language models.
Proposes Interpolated Policy Distillation that blends teacher and student distributions during rollout for improved distillation.
2 further papers
Oct 2026
Smaller frozen models provide better rejected responses for preference distillation
Analyzes the optimal source of rejected responses for preference distillation, discovering that smaller frozen models consistently outperform student-generated samples across scales from 7B to 72B.
1 further paper
Oct 2026 · Institute of Automation, Chinese Academy of Sciences, University of Chinese Academy of Sciences
Distilled task updates can merge better than stronger teacher updates
Shows that distilled student updates can complement RL teacher updates when merging, advancing understanding of distillation alignment.
Oct 2026 · Stony Brook University, Amazon
Safety distillation can pass hidden backdoors from teacher to student
Shows that on-policy safety distillation can transmit hidden backdoors from teacher to student, highlighting an alignment failure.
1 further paper
Oct 2026 · Cohere, Mila, Université de Montréal
Lexicographic distillation preserves priority-ordered capabilities in multi-objective RL
Uses a lexicographic priority order to route student rollouts to the best specialist teacher, preserving priority-ordered capabilities in multi-objective distillation.
1 further paper
Oct 2026 · Stanford University
Divergence choice in distillation controls student model entropy
Proves that forward KL inflates student entropy above teacher's by residual divergence, providing a principled way to control entropy in distillation.
1 further paper
Oct 2026 · Alibaba Cloud Computing
Supervision reliability trumps alignment coverage in cross-tokenizer distillation
Finds on-policy distillation with next-token prediction outperforms strict token alignment for reasoning tasks.
Oct 2026 · University of Illinois at Urbana-Champaign, Tsinghua University
Formal provers select methods better when methods state how they apply
Introduces Method Contracts for formal theorem proving that generate problem-specific proposals before ranking, enabling better method selection in formal reasoning.
2 further papers
Oct 2026
Offline student rollouts often beat on-policy distillation for language models
Shows offline student rollouts outperform on-policy distillation and proposes Semi-OPD method.
1 further paper
Oct 2026 · Redwood Research
Distillation can expose hidden behavior or retain safer capabilities
Tests distillation as a tool to expose hidden secret-keeping behavior in models, linking distillation to safety evaluation.
Oct 2026 · Alibaba Group, Tsinghua University
Grounding distillation in interaction states improves multi-turn agent performance
Identifies and addresses the failure of on-policy distillation in multi-turn agents by removing student reasoning from teacher context.
1 further paper
Oct 2026 · UC San Diego, Purdue University
Reward-guided token weighting makes language-model distillation more effective
Reward-Aligned Distillation uses token-level reward signals from a stronger model to guide the student's token weighting, improving distillation.
3 further papers
Oct 2026
Top-k distillation's renormalized gradient ignores tail mass
Identifies zero gradient along tail-mass directions in top-k distillation, a fundamental issue that affects alignment in reasoning distillation.
released code
Oct 2026 · Google, University of Washington
Adaptive distillation triggers teacher takeover when student diverges
proposes DTDD that rewinds and has teacher regenerate when student diverges, closing alignment gaps.
3 further papers
Oct 2026 · Shanghai Jiao Tong University, Alibaba Group
Sequence-level credit assignment improves on-policy distillation of reasoning models
Proposes sequence-level credit assignment to improve on-policy distillation of reasoning models.
1 further paper
Oct 2026 · University of Southern California, University of Florida
Distillation overshoot lets smaller agents surpass teachers, and a new method preserves those gains.
Identifies a distillation overshoot phenomenon where smaller agents temporarily surpass teachers, proposes a method to preserve gains.
1 further paper
Oct 2026 · NVIDIA
Teacher rollouts repair myopic token scoring in agent distillation
Identifies a myopic token scoring failure in agentic on-policy distillation and fixes it with teacher rollouts.
11 further papers
Nov 2026 · North Carolina State University, LG AI Research
Omitting reasoning from training data does not harm AI agents on standard benchmarks
Finds that discarding reasoning steps from training data does not harm agents on standard benchmarks, questioning the need for thought supervision in distillation.
1 further paper
Nov 2026 · Tsinghua University, Harbin Institute of Technology, Shenzhen, China
Language models improve at math when teachers steer their practice
Steers teacher generations to supervise reasoning paths students are unlikely to enter, improving math performance.
2 further papers
Nov 2026 · Salesforce AI Research, UIUC
Merging specialist AI models saves 42% of training compute compared to joint training
Compares on-policy distillation and GRPO for merging specialist models, finding OPD saves 42% compute over joint training.
22 of 58 papers shown