The authors studied pretraining with a 675M-parameter SiT-XL/2 diffusion transformer using REPA representation alignment.
Schedule-Free Optimization Beats EMA-Only Training in Large Diffusion Models
Experiments on a 675-million-parameter diffusion transformer show that Schedule-Free AdamW improves optimization and generation quality without requiring a predetermined learning-rate decay point.
Top University
Chuyu Hu · Huanran Chen · Shitong Shao · Yinpeng Dong
Tsinghua University · Shanghai Qi Zhi Institute · The Hong Kong University of Science and Technology
Research Digest··3 min read
Hu and colleagues compare the standard diffusion-training recipe of a constant learning rate plus exponential moving average, or EMA, against learning-rate decay and Schedule-Free optimization.
§