The authors replace the common asymmetric design, a pretrained language model plus an attached diffusion head, with a single flow matching Transformer called MM-RiT that reads and writes continuous states in frozen encoder spaces.
A single flow matching Transformer generates images and text jointly in pretrained encoder spaces
JUICE trains one model from scratch on continuous image and text representations, reaching competitive generation and VQA scores without any language model initialization.
Top University
Jiahan Li · Huangyuan Su · Shengbang Tong · Shusheng Yang · Ruihan Guo · Suozhi Huang · +6 more
New York University · Harvard University · University of Illinois Urbana-Champaign · Princeton University · University of Pennsylvania
Research Digest··3 min read
The authors introduce JUICE, a flow matching Transformer that models the joint distribution of images and text directly in frozen pretrained encoder spaces, with each modality having its own noise level.
§