Joint training of VAE and diffusion from scratch with TPR loss prevents latent collapse

SpeeDiff achieves state-of-the-art FID scores on ImageNet 256×256 and 512×512 with dramatically faster training.

Chinese Tech

Bingliang Zhang · Wenda Chu · Yizhuo Li · Linjie Yang · Yisong Yue · Katherine L. Bouman · +2 more

ByteDance Seed · California Institute of Technology · Meta

Research Digest·28 October 2026·3 min read

Illustration: Joint training of VAE and diffusion from scratch with TPR loss prevents latent collapse

In one line

SpeeDiff enables end-to-end training of latent diffusion models using a Tweedie loss that prevents latent collapse, achieving dramatically faster convergence.

Before you read this

Three ideas this paper builds on

diffusion modelA type of generative model that learns to reverse a gradual noising process to create data.152 papers →representation alignmentLearning-based technique that makes representations from different models or modalities compatible by mapping them to a shared space.5 papers →diffusion transformerA neural network architecture combining transformer layers with diffusion processes for high-quality image and video generation.59 papers →

Read it as

Standard image generators use a two-stage pipeline: a compressor shrinks images, then a generator learns to create those shrunken representations. Jointly training both sounds efficient but typically causes the compressed latent space to collapse.

SpeeDiff succeeds at joint training by adding a Tweedie Pixel Reconstruction loss. This loss decodes intermediate noisy latents back into image pixels, giving the compressor direct pixel-level feedback and preventing information loss.

The result is a model achieving an FID of 1.50 on ImageNet 256x256, a top score, while training over a hundred times faster than previous leading methods. This dramatically cuts the cost of training high-quality image generators.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

diffusion model end-to-end training tweedie pixel reconstruction representation alignment flow matching diffusion transformer