Autoregressive Thought Flow combines a causal transformer with a lightweight diffusion head. The transformer conditions the next reasoning step, while the diffusion head samples one of several plausible continuous thoughts and feeds it back for variable-length autoregressive reasoning.
Across mathematical reasoning tasks, ATF improves the accuracy versus reasoning-length trade-off, benefits from and extra test-time thoughts, and broadens sampled solution coverage. Its average Pass@100 gain over the strongest prior baseline is 19.3 points across MATH-500, GSM8K, and OlympiadBench, reaching 34.6 points on OlympiadBench. This suggests continuous reasoning works better when alternatives are represented explicitly rather than averaged into one latent prediction.