The authors introduce Clock Diffusion, a semi-autoregressive (SAR) framework for continuous (DLMs). It uses position-dependent noise schedules to define which tokens are being denoised at each step, enabling both block and sliding-window generation. Models based on this, ClockDLMs, combine Gaussian diffusion over token embeddings with sliding-window schedules.
Semi-autoregressive continuous diffusion models match discrete baselines and enable KV caching.
The authors' Clock Diffusion framework uses position-dependent noise schedules to produce state-of-the-art continuous diffusion language models on OpenWebText and GSM8K.
Big Tech
Yair Schiff · Omer Belhasin · Roy Uziel · Matan Rusanovsky · Ran Zilberstein · Marianne Arriola · +4 more
NVIDIA · Cornell University
Research Digest··2 min read
Thread:Diffusion Language Models
In one line
Continuous diffusion language models with semi-autoregressive position-dependent noise schedules achieve state-of-the-art likelihood and benchmark performance while enabling variable-length generation and KV caching.
Before you read this
Two ideas this paper builds onRead it as
The authors introduce Clock Diffusion, a semi-autoregressive framework for continuous diffusion language models that denoises token embeddings in blocks or sliding windows according to position-dependent noise schedules.
Yair Schiff and colleagues define a continuous diffusion process over token embeddings in which the global diffusion time is reparameterized into position-dependent local clocks.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.