Simplex (SDMs) lift the diffusion process to the probability simplex, where each token is represented as a distribution over categories. This allows the model to carry uncertainty across denoising steps, addressing information collapse in discrete diffusion. SDMs admit closed-form reverse transitions and can be trained with a simple cross-entropy loss. They also introduce a DDIM-like sampler with tunable stochasticity, avoiding the need to integrate an ODE.
On OpenWebText, SDMs achieve a GenPPL of 17.0 at 64 sampling steps, close to real validation data. On code generation (TinyGSM), SDMs without outperform masked and uniform diffusion baselines (49.0% vs 45.8% at low temperature). Distilled to just 8 steps, SDMs solve 32.1% of GSM8K problems, more than distilled discrete diffusion models with 128 steps (21.4%). These results demonstrate that maintaining uncertainty throughout the diffusion process improves performance on discrete data.