The authors constructed Waldo from Wikipedia, targeting facts present in non-English editions but absent from English editions (knowledge gaps), and manually perturbed English editions to create factually inconsistent versions (knowledge conflicts).
Query-language preference emerges strongly under conflicting multilingual evidence
The Waldo benchmark reveals that when language editions of Wikipedia contradict each other, models favor sources in the query language, but not when facts are merely missing.
Big Tech
Dayeon Ki · Ruochen Zhang · Silviu Cucerzan · Ryen W. White · Ning Gao
University of Maryland · Microsoft
Research Digest··2 min read
The authors introduce Waldo, a multilingual QA benchmark of 12K instances across five languages, designed to test query-language preference under knowledge gaps and conflicts.
Why this paper
From Microsoft and University of Maryland
In one line
Under conflicting multilingual evidence, LLMs strongly favor query language sources, causing language-dependent answers.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§