The authors built an encoder-free speech LLM consisting of lightweight embedding layers followed by a pretrained LLM (Mistral-7B).
Metadata-supervised pretraining lets encoder-free speech LLMs identify speakers and emotions
By training directly on spectral features with speaker and emotion labels, the model matches or exceeds encoder-based counterparts on joint transcription and diarization.
Big Tech
Mohan Shi · Ruchao Fan · Sunit Sivasankaran · Keqi Deng · Jinyu Li
Microsoft
Research Digest··3 min read
The authors propose metadata-supervised pretraining (MSP) for encoder-free speech LLMs, using speaker identity and emotion labels to train models that directly process Mel-spectrograms.
Why this paper
From Microsoft
In one line
Metadata-supervised pretraining lets encoder-free speech LLMs match or beat encoder-based models on speaker and paralinguistic tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§