Keeping token uncertainty improves discrete diffusion generation and fast sampling

Simplex Diffusion Models refine probability distributions over tokens, producing competitive language samples and stronger code-generation results with fewer denoising steps.

Big Tech
Justin Deschenaux · Alexandre Galashov · Andrew Campbell · Li Kevin Wenliang · James Thornton · Arnaud Doucet · +1 more

Google DeepMind · EPFL · UCL Gatsby

Research Digest··2 min read

In one line

Simplex Diffusion Models represent beliefs over categories on the probability simplex, mitigating information collapse and achieving competitive performance on text and code generation.

Before you read this

Three ideas this paper builds on
Read it as

Simplex (SDMs) lift the diffusion process to the probability simplex, where each token is represented as a distribution over categories. This allows the model to carry uncertainty across denoising steps, addressing information collapse in discrete diffusion. SDMs admit closed-form reverse transitions and can be trained with a simple cross-entropy loss. They also introduce a DDIM-like sampler with tunable stochasticity, avoiding the need to integrate an ODE.

On OpenWebText, SDMs achieve a GenPPL of 17.0 at 64 sampling steps, close to real validation data. On code generation (TinyGSM), SDMs without outperform masked and uniform diffusion baselines (49.0% vs 45.8% at low temperature). Distilled to just 8 steps, SDMs solve 32.1% of GSM8K problems, more than distilled discrete diffusion models with 128 steps (21.4%). These results demonstrate that maintaining uncertainty throughout the diffusion process improves performance on discrete data.

How it works

before and after
CATEGORICAL DIFFUSIONIntermediate prediction: Probabilities over tokensIntermediatepredictionProbabilities overtokensChoose one token: Discard competing optionsChoose one tokenDiscard competingoptionsNext denoising state: Continue from that choiceNext denoisingstateContinue from thatchoiceSIMPLEX DIFFUSIONtokenprobabilit…updatedbeliefsampledtokensSimplex belief state: Keep all token probabilitiesSimplexbelief stateKeep all tokenprobabilitiesClosed-form reverse: Update the full beliefClosed-formreverseUpdate the fullbeliefTunable sampler: Adjust sampling randomnessTunablesamplerAdjust samplingrandomnessGenerated sequenceGeneratedsequenceTraining uses ordinary cross-entropy.
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe