The authors propose MSP to remove the need for real paired non-text supervision in multimodal learning.
Text proxies can replace paired multimodal data for cross-modal reasoning
MSP trains on one real modality plus textual descriptions, then composes across unseen modalities at inference.
Big Tech
Robert Moseley · Md Kaykobad Reza · Ameya Patil · Edward Ayrapetian · Salman Asif
University of California, Riverside · Amazon
Research Digest··2 min read
Moseley and colleagues introduce Modality Self-Play (MSP), a training framework that uses proxy textual descriptions as semantic bridges between modalities.
Why this paper
From Amazon and University of California, Riverside
In one line
Multimodal LLMs can learn cross-modal reasoning without paired non-text data by using proxy text as semantic bridges.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§