distribution matching distillation
The core idea is to train a new, faster model by making its generated outputs statistically indistinguishable from those of a slower, better teacher. Instead of comparing individual samples one-to-one, you compare the overall distributions of what each model produces. This way, the student learns the teacher's style and quality without needing to replicate its exact step-by-step process or rely on external data during training.
Across these papers, the common thread is replacing expensive, fallible supervision with a more robust target. Whether protecting agents from attacks, accelerating image or video generation, or improving reasoning, they all shift focus from matching specific outputs to matching overall behavior or feature distributions. This often yields major gains in efficiency and robustness, like fewer generation steps or better resilience to manipulation, while preserving or even improving quality.
- Papers
- 29
- Released code
- 1
- First seen
- Sept 2026
- Latest
- Nov 2026
29 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1AI video generators reduce flickering and preserve detail with few steps
Adding a predictive representation loss to distribution matching distillation improves few-step video generation spatial fidelity and temporal coherence.
Oct 2026 - 2Tuning only the first denoising step restores lost diversity in distilled models
Diversity loss in distilled diffusion models is concentrated in the first step and can be recovered by a lightweight first-step bias adapter.
Oct 2026 - 3Matching final outputs improves few-step image generator distillation
Training the fake model only on final outputs, not intermediate steps, improves few-step diffusion distillation.
Nov 2026
How fast it is moving
Often studied with
diffusion modelA type of generative model that learns to reverse a gradual noising process to create data.16 togetherflow matchingA generative training method that learns an ordinary differential equation mapping noise to data by matching velocity fields along probability paths.10 togetheron-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.4 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Top Universitycs.CV
Matching final outputs improves few-step image generator distillation
Training the fake model only on final outputs, not intermediate steps, improves few-step diffusion distillation.
Zhejiang University AGI Lab, Westlake University StepFun · Nov 2026
- Academiccs.CV
Tuning only the first denoising step restores lost diversity in distilled models
Diversity loss in distilled diffusion models is concentrated in the first step and can be recovered by a lightweight first-step bias adapter.
Graz University of Technology, Complexity Science Hub Austria · Oct 2026
- Chinese Techcs.CV
AI video generators reduce flickering and preserve detail with few steps
Adding a predictive representation loss to distribution matching distillation improves few-step video generation spatial fidelity and temporal coherence.
Shanghai Jiao Tong University, Hunyuan (Tencent) · Oct 2026
- Big Techcs.CRcode
Self-distillation hardens tool-using agents without sacrificing benign task performance
RAISED uses self-distillation to defend LLM agents from prompt injection without losing utility.
LIX, École polytechnique · Oct 2026
- Big Techcs.CV
High-fidelity latent priors improve efficient high-resolution image generation
UltraDiff distills a joint f8-f32 diffusion teacher into an f32 student, approaching f8 image quality at f32 speed.
The Hong Kong University of Science and Technology (Guangzhou), Adobe Research · Oct 2026
- Chinese Techcs.LG
Direct data-space refinement improves few-step generation without flow schedules
Data-space iteration directly refines predictions in data space and outperforms standard flow discretization for few-step generation.
ByteDance Seed · Oct 2026
- Chinese Techcs.LG
Outcome-guided self-distillation improves LLM reasoning by adapting supervision
Outcome-guided divergence selection and prefix truncation improve on-policy self-distillation for LLM reasoning, consistently beating vanilla OPSD across model scales.
Southeast University, Huawei Noah’s Ark Lab · Oct 2026
- Independentcs.CV
Few-step pixel diffusion outperforms 25-step teacher via banded matching and DINOv2 guidance
DMA^2 distills native-RGB diffusion into four steps, outperforming its 25-step teacher on all seven metrics.
Oct 2026
- Top Universitycs.LG
Data-free distillation of diffusion models with a single consistent objective
A single objective unifying sample generation and score estimation lets a student distill a frozen diffusion teacher data-free, with Wasserstein convergence and ImageNet FID 2.04 at one step.
University of British Columbia, Vector Institute for AI · Oct 2026
- Chinese Techcs.LG
Distribution matching distillation and drifting models are equivalent under finite-sample analysis
Distribution Matching Distillation is equivalent to Drifting Models, and multi-bandwidth training improves DMD convergence and quality.
Baidu Inc., Central South University · Oct 2026
- AI Startupcs.LG
Distributional distillation cuts diffusion language model sampling steps by half or more
Two distribution matching distillation methods reduce network evaluations for continuous diffusion language models by up to 49% and 20% at similar generation quality.
Inria, PSL Research University · Oct 2026
- Top University
Dynamics-aware distillation preserves motion in four-step video world models
DyMD improves few-step video generation by preserving interaction dynamics via distribution matching distillation.
Beihang University, JD Future Academy · Sept 2026
- Big Techcs.CV
One LiDAR model generates, forecasts and resimulates driving scenes in real time
A single conditional LiDAR world model unifies generation, forecasting, and resimulation, real-time at 11 FPS on one H100 GPU.
NVIDIA, University of Illinois, Urbana-Champaign · Oct 2026
- Chinese Techcs.CL
Translating answer-guided reasoning cuts the cost of model distillation
CoTranslator learns to translate reverse CoT to forward CoT, improving student accuracy while using less compute.
Tsinghua University, Ant Group · Oct 2026
- Top Universitycs.CV
Two denoising experts make video diffusion distillation far more efficient
Glance Forcing distills bidirectional video diffusion into a few-step autoregressive model using one PF-ODE trajectory, outperforming a fully trained baseline at far lower collection cost.
Wuhan University, University of Electronic Science and Technology of China · Oct 2026
- Big Techcs.LG
Few-step diffusion can sacrifice output coverage for single-sample quality
Few-step diffusion distillation can improve single-sample quality while narrowing valid-output coverage, especially when trained with distribution-matching objectives.
Rice University, Apple · Oct 2026
- Chinese Techcs.CV
Projecting critic errors stabilizes four-step video diffusion distillation
Projected Distribution Matching Distillation filters critic errors in video diffusion model distillation with a one-line code change.
ByteDance Inc., University of California, San Diego · Oct 2026
- Big Techcs.CV
One-step video refiner achieves 8.91x speedup for 4K upscaling
SoL-Refiner transforms low-resolution video outputs into 4K videos with a single denoising step, outperforming multi-step refiners.
NVIDIA · Oct 2026
- Chinese Techcs.CV
Independent chunk scoring reduces drift in long autoregressive video rollouts
Rollout-Marginal Distillation scores each generated chunk independently against a teacher, then refines temporal coherence, preventing error accumulation in long autoregressive video generation.
MMLab, The Chinese University of Hong Kong · Oct 2026
- Chinese Techcs.LG
Language models improve at math when teachers steer their practice
Coupling student and teacher rollouts with a rejection-sampling rule raises math benchmark accuracy above on-policy distillation.
Tsinghua University, Harbin Institute of Technology, Shenzhen, China · Nov 2026
- Chinese Techcs.CV
Video generators improve longer clips by revisiting earlier generation steps
Using the intermediate denoising trajectory of prior chunks as context improves autoregressive long video generation quality.
Huawei Noah Research Institute · Nov 2026
- Chinese Techcs.CL
Language model distillation improves when teachers guide only the first few tokens
Aligning student to teacher on only the first 20% of generated tokens, with overlap-gated divergence, outperforms top-k on-policy distillation.
Renmin University of China, Huawei Technologies Ltd. · Nov 2026
- Chinese Techcs.CV
Learned attention proxies help streaming video generators retain historical context
Attn-Forcing preserves full-context attention contributions for streaming video generation using learnable attention proxies under bounded memory and computation.
Tongji University, Tencent · Oct 2026
- Research Labcs.AI
Optimal transport aligns pretrained modalities with only hundreds of paired anchors
Aligning two frozen unimodal encoders requires only 100 to 500 paired anchors when anchor-relative coordinates, entropic optimal transport, and distillation are combined.
Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM · Oct 2026
- Academiccs.CV
Video world model uses test-time training as persistent memory for long-range consistency
A camera-controlled video world model uses test-time training memory to preserve observed scenes in fixed-size parameters, improving long-horizon revisit consistency.
Meta Reality Labs, Brown University · Oct 2026
- Chinese Techcs.CV
StreamErase enables real-time streaming video object removal at 26 FPS
StreamErase enables sub-second, real-time streaming video object removal at over 26 frames per second on a single GPU.
Wuhan University, Alibaba Group · Oct 2026
- Top Universitycs.LG
Schedule-Free Optimization Beats EMA-Only Training in Large Diffusion Models
Large diffusion models require optimization that balances variance reduction with continued progress, beyond just exponential moving average.
Tsinghua University, Shanghai Qi Zhi Institute · Oct 2026
- Chinese Techcs.AI
Condition-aware distillation reduces invalid reasoning in smaller language models
Condition-contrastive distillation, which compares teacher scores with and without a question condition, reduces invalid continuations and improves distilled LM reasoning.
The Chinese University of Hong Kong, Shenzhen, Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) · Oct 2026
- Big Techcs.CV
Polynomial student paths improve four-step video distillation quality
Parametric Trajectory Distillation fits teacher trajectory segments to polynomials and supervises along the student's own predicted path, preserving motion and detail in four-step video generation.
EPFL, NVIDIA · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.