The authors encode each modality as bits, either directly or through frozen domain-specific tokenizers, then place those bits in a fixed sequence template.
One Bitwise Diffusion Framework Generates Across Three Multimodal Domains
The authors use a shared diffusion objective and architecture to jointly and conditionally generate image-text, speech-text and protein sequence-structure pairs.
Big Tech
Georgios Batzolis · Georgios Kontos · Bruno Trentini · Pranam Chatterjee · Micha Livne · Mark Girolami · +1 more
Devotion AI Labs · University of Cambridge · University of Oxford · NVIDIA · University of Pennsylvania
Research Digest··3 min read
Batzolis and colleagues test whether diverse modalities can be represented as binary streams and modeled through the same continuous diffusion protocol.
Why this paper
From NVIDIA and 5 others
In one line
CoBit, a single continuous bitstream diffusion protocol, handles joint and conditional generation across image-text, speech-text, and protein sequence-structure pairs.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§