The authors trained mixture-of-experts models jointly on text, vision and audio, using roughly 300,000 GH200 GPU-hours.
Multimodal training needs different scaling recipes for text, vision and audio
Experiments with mixture-of-experts models quantify modality-specific compute allocation, data repetition limits and asymmetric interference.
Top University
Aditi Khandelwal · Ayush K. Tarun · Yixuan Xu · Imanol Schlag · Golnoosh Farnadi · Siva Reddy · +2 more
MILA - Quebec AI Institute · McGill University · EPFL · ETH · University of Washington
Research Digest··3 min read
Khandelwal et al.
Why this paper
From MILA - Quebec AI Institute and 4 others
In one line
Scaling laws for multimodal pretraining reveal modality-specific compute optima, asymmetric interference, and saturation of repeated non-text data.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§