Yair Schiff and colleagues define a continuous diffusion process over token embeddings in which the global diffusion time is reparameterized into position-dependent local clocks.
Semi-autoregressive continuous diffusion models match discrete baselines and enable KV caching.
The authors' Clock Diffusion framework uses position-dependent noise schedules to produce state-of-the-art continuous diffusion language models on OpenWebText and GSM8K.
Big Tech
Yair Schiff · Omer Belhasin · Roy Uziel · Matan Rusanovsky · Ran Zilberstein · Marianne Arriola · +4 more
NVIDIA · Cornell University
Research Digest··2 min read
The authors introduce Clock Diffusion, a semi-autoregressive framework for continuous diffusion language models that denoises token embeddings in blocks or sliding windows according to position-dependent noise schedules.
Why this paper
From NVIDIA and Cornell University
In one line
Continuous diffusion language models with semi-autoregressive position-dependent noise schedules achieve state-of-the-art likelihood and benchmark performance while enabling variable-length generation and KV caching.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§