The authors adapted the language-only Bias Benchmark for Question Answering (BBQ), selecting scenarios with explicit visual cues across seven categories: age, gender, appearance, socioeconomic status, race, religion and disability.
Vision inputs expose greater social bias than text in matched tests
Rezaei and colleagues introduce a synthetic image benchmark and report lower fairness across 23 multimodal models when matched scenarios move from text to vision.
Research Lab
Razieh Rezaei · Ehsaneddin Asgary · Masoud Jalili Sabet
Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning · Qatar Computing Research Institute, Hamad Bin Khalifa University
Research Digest··2 min read
Rezaei and colleagues built Melange, a benchmark comparing how multimodal language models answer socially sensitive questions presented through text or images.
§