Wang et al.
Diffusion language models are cheaper to train than the widely cited 16x compute gap suggests
Wang et al. show that using exact likelihood evaluation and block-wise training with an encoder-decoder architecture reduces the pretraining compute gap to as low as 3.0x for small block sizes.
Big Tech
Guanghan Wang · Bing He · Zhan Shi · Benoit Dumoulin · Hanqing Lu · Volodymyr Kuleshov
Cornell University · Cornell Tech · Amazon
Research Digest··2 min read
The authors revisit the scaling laws of masked diffusion models by training about 1,700 models across ten model variants.
Why this paper
From Amazon and 2 others
In one line
The widely cited 16x compute gap between diffusion and autoregressive language models shrinks to as low as 3.0x with modern training methods.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (4 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§