The authors constructed DeepSEA by professionally translating material from established datasets: MGSM for mathematics, IndoMMLU for scientific reasoning, XQuAD for question answering, CrossSum for abstractive summarization and WMT24++ for translation.
Southeast Asian language models still rely heavily on English representations
DeepSEA tests five capabilities across 12 languages, revealing pronounced weaknesses in low-resource languages and scripts.
Big Tech
Jiuzhou Han · Abhishek Bijay Mishra · Trevor Cohn · Shubham Mittal
Google Research
Research Digest··3 min read
Han and colleagues introduce DeepSEA, a professionally translated benchmark covering mathematical and scientific reasoning, question answering, summarization and machine translation in 12 Southeast Asian languages.
§