Multilingual self-supervised speech models often underperform monolingual ones given the same total data budget, a gap called the multilingual gap. This paper tests two interventions that force a bilingual HuBERT model to discriminate between English and French during pretraining: adding an auxiliary language classifier and using per-language k-means targets. Both interventions reduce the gap on phonetic, lexical, and prosodic measures, with the largest gains when language discrimination is introduced in the first training iteration. The findings suggest that helping the model distinguish languages causally reduces the cost of multilingual learning while preserving cross-language sharing.
Language discrimination training reduces the multilingual gap in speech models
Adding a language classifier or language-specific targets during pretraining helps bilingual HuBERT match monolingual phone, lexical, and prosodic performance.
Big Tech
Maureen de Seyssel · Jie Chi · Zakaria Aldeneh
Apple
Research Digest··2 min read
In one line
Strengthening language discrimination during pretraining closes the multilingual gap in speech models.
Before you read this
Two ideas this paper builds onself-supervised learningA paradigm where models learn representations from unlabeled data by predicting hidden or transformed parts of the input.60 papers →speech recognitionThe task of converting spoken audio into text, evaluated across dialects and used for training efficient transcription models.8 papers →
Read it as
de Seyssel et al. show that strengthening language discrimination during self-supervised pretraining of a bilingual English/French HuBERT model reduces the performance gap compared to monolingual models.
The authors trained HuBERT models on English and French speech with a fixed total data budget, comparing a standard bilingual baseline against two interventions that enhance language discrimination: an auxiliary language classifier and per-language k-means targets.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.