The authors propose LAW, which uses pure self-conditioning (SC): at each denoising step, the denoiser receives the expected embedding of the clean token (computed from its own logits) as input, and outputs logits that define a new expected embedding.
A language model using only self-conditioning outperforms existing diffusion models
LAW propagates the expected embedding across denoising steps and decodes tokens once at the end, matching or surpassing discrete and continuous diffusion baselines.
Big Tech
Adrien Deschenaux · Nadezhda Ilieva · Justin Deschenaux · Manuel Madeira · Dorina Thanou · Aurélien Lucchi · +3 more
University of Basel · Microsoft AI · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center
Research Digest··2 min read
The authors introduce LAW (Loopholing All the Way), a continuous generative model that relies purely on self-conditioning, without auxiliary conditioning mechanisms.
§