The authors designed a dataset, MoGround, spanning four visual domains (natural photos, statistical charts, fine-art paintings, medical radiology).
New dataset isolates and mitigates modality distraction in vision-language models
Authors show distraction direction varies across models and is predictable from grounding strength, with a simple fine-tuning reducing it substantially.
Top University
Luca Zhou · Bo Zhao · Rose Yu · Emanuele Rodolà · Roberto Dessì
Sapienza University of Rome · Harvard University · UC San Diego · Paradigma · Not Diamond
Research Digest··3 min read
The authors introduce MoGround, a vision-language dataset where every question is guaranteed to be answerable from exactly one modality (vision or text).
Why this paper
From Harvard University and 4 others
In one line
Modality distraction in VLMs is model-dependent, predicted by grounding strength, and reducible by 9-51% via a robustness vector with minimal accuracy cost.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§