The authors first prompted Qwen2-Audio-7B-Instruct to self-assess transcription reliability via zero-shot and two-shot in-context learning, finding it consistently overconfident.
Audio LLMs cannot assess their own transcription reliability.
A lightweight predictor using audio encoder representations outperforms existing methods by over 10 F1 points.
Big Tech
Apple · University of California San Diego
Research Digest··1 min read
The authors study whether Audio LLMs can detect when their own transcription of a voice query is unreliable.
Why this paper
From Apple and University of California San Diego
In one line
Audio-encoder representations predict whether an Audio LLM will transcribe degraded speech reliably far better than self-assessment or tested quality and uncertainty baselines.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§