Diffusion language models generate blocks of tokens through repeated denoising.
On-policy residual training helps diffusion language models decode faster
ORL trains models on their own denoising trajectories, improving parallel token generation while maintaining benchmark quality.
AI Startup
Yifan Yu · Yuqing Jian · Jisen Li · Yuezhou Hu · Tianyi Zhang · Xiaoxia Wu · +6 more
Together AI · University of Illinois Urbana-Champaign · University of California, Berkeley · University of California, Santa Cruz · Stanford University
Research Digest··3 min read
Thread:Diffusion Distillation
Yu et al.
Why this paper
From Together AI and 5 others · Part of Diffusion Distillation, now 8 papers
In one line
Residual context for diffusion language models must be learned on-policy with an entropy-contraction objective to improve speed and quality.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (hardware B200)
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§