Speech & audio

Speech recognition and synthesis, audio understanding, music

47 articles

Speech & audio

Vehicle-mounted microphones can localize nearby cars using sound alone

Liu et al. propose AcousticBEV, a learning-based method that uses a vehicle-mounted microphone array to estimate the center locations of moving vehicles in bird's-eye-view (BEV) space. On a new real-world dataset called AcousticScenes, the method achieves F1 scores of 47.7% on easy scenarios and 45.1% on difficult scenarios, outperforming both classical signal-processing and other learning-based baselines.

·3 min
Speech & audio

Audio-language models understand spatial sound from any number of input channels

The authors present SpheraAudio, a framework that enables large audio-language models to process recordings with any channel count (mono, stereo, or Ambisonics) by converting multichannel spectra into a single token stream. They introduce a spatial encoder that produces tokens interleaved with audio tokens, using a new positional encoding to maintain temporal and modality information. SpheraAudio achieves state-of-the-art performance on spatial audio benchmarks while retaining strong general-audio capabilities.

·2 min
Speech & audio

Text-to-audio evaluator that explains scores achieves top correlation with human raters

The authors propose CoTTAGE, an evaluator for text-to-audio generation that first generates an audio-grounded rationale before predicting scores for text relevance and overall quality. They collect over 10K human quality annotations to enable quality evaluation training. The system achieves state-of-the-art correlation with human judgments and its explanations are preferred by humans over those from base models.

·2 min
Speech & audio

229K-hour dataset lets speech models generate minutes-long audio with control

The authors introduce Emilia2, a 229,000-hour multilingual dataset for long-form speech generation, built from in-the-wild recordings. It includes short utterances, single-speaker long-form clips, and multi-speaker dialogues with natural language instructions and nonverbal-vocalization tags, plus a 23-hour test set with up to four speakers. A 0.6B-parameter non-autoregressive model trained on Emilia2 achieves the lowest Chinese character error rate (5.27%) and English word error rate (9.93%) on a standard evaluation, surpassing an 8B-parameter system.

·3 min
Speech & audio

One audio model can now describe, generate and edit 3D sound scenes

Spatial audio captures where sounds come from, not just what they are, and until now the models that describe such scenes, generate them and edit them have been separate systems. Kim, Braun and Gamper built a single model on top of an existing spatial audio language model by adding a flow-matching generator over audio latents and a taxonomy of atomic edit operations. They report that it keeps most of its original scene understanding while beating full fine-tuning at generation and editing.

·3 min
Speech & audio

Voice model matches text reasoning without separate transcription step

The authors present Apriel Voice 27B Preview, an open speech-language model built from the Gemma 4 checkpoint. It directly processes speech audio to reason, converse, and call tools, eliminating the need for a separate automatic speech recognition module for reasoning tasks. In evaluations, it achieved 98.90 on Big Bench Audio, 81.13 on Speech-BFCL, and 65.51% task success on τ-Voice, all with a mean latency of 1.91 seconds.

·3 min
Speech & audio

Multimodal model tops specialist speech systems for low-resource African languages

The authors systematically compared three ASR models, Phi-4, Whisper, and MMS, on six African languages under matched low-resource conditions. They found that Phi-4, a multimodal language model, achieved the best accuracy on most languages, while MMS showed comparable character error rates but saturated at less data. Domain-specific training improved Phi-4 and Whisper but hurt MMS, and using adapters from related languages outperformed training them from scratch.

·2 min
Speech & audio

AI audio models better recognize individual sounds and whole soundscapes

The authors introduce AudioFM, a self-supervised framework that learns audio representations by combining recording-level contrastive learning with a patch-wise masked latent discrimination objective. Across linear probing, fine-tuning, and zero-shot transfer on 12 tasks in four acoustic domains, AudioFM achieves state-of-the-art performance, demonstrating superior event detection and domain generalizability.

·2 min
Speech & audio

Audio-language model suite adds speaker-aware transcription and understanding

The authors present Qwen-Audio-3.1-ASR, a family of audio-language models that extend accurate speech recognition toward structured transcription and broader audio interpretation. The models retain support for 30 languages and 16 Chinese dialects, and introduce instruction-controlled features like historical-context conditioning and hotword customization. Most notably, they demonstrate joint speaker diarization and transcription, as well as audio understanding tasks such as temporal grounding and sound event detection.

·3 min
Speech & audio

Post-training gains in audio models may reflect format adaptation, not true audio understanding.

The authors investigated whether post-training of large audio-language models (LALMs) actually improves audio understanding or merely adapts to question formats. They found that performance gains on benchmarks like MMAU and MMAR persisted even when audio was replaced with silence, but largely vanished when multiple-choice options were removed. This suggests that post-training teaches models to exploit textual cues rather than develop genuine auditory reasoning.

·2 min
Speech & audio

Open-vocabulary speech recognition runs on a $5 microcontroller without a neural accelerator

The authors present Oído, an inference engine that runs open-vocabulary English speech recognition entirely on an ESP32-S3 microcontroller, a $5 chip with no neural accelerator. Using int8 arithmetic, it achieves 3.70%/8.23% word error rate on LibriSpeech test-clean/test-other, within 0.12 points of full precision, and outperforms laptop-class models such as Whisper tiny.en. Measured on physical hardware, the firmware processes audio at 1.97x real time.

·3 min
Speech & audio

Unified model blends general audio understanding with spatial awareness

The authors present MiDashengLM-Spatial, the first open-source end-to-end audio-language model that jointly handles general audio understanding and spatial awareness. By augmenting an existing monaural LALM (MiDashengLM) with a spatial encoder (Spatial-Dasheng) via a hierarchical conditioning module, the model achieves superior performance on spatial reasoning benchmarks while remaining competitive on standard monaural audio tasks.

·2 min
Speech & audio

Choosing speech encoder layers on evaluation data inflates depression scores

Perez-Toro and colleagues examine a common evaluation shortcut in speech-based depression detection: selecting an encoder’s best-performing layer using the same cross-validation results reported as final performance. Their repeated nested evaluations show that this practice can raise apparent area under the ROC curve (AUC) by as much as 0.09, including substantial gains when the outcome labels contain no real signal.

·3 min
Speech & audio

Mask-free separation pretraining improves representations of overlapping speech

Baroudi, Bredin and Marxer introduce SepRQ, a self-supervised speech model designed specifically for mixtures containing multiple speakers. Instead of reconstructing masked audio, it learns to separate pseudo-sources represented by discrete targets from frozen random-projection codebooks, while training at several temporal resolutions. The authors report leading results on multi-speaker SUPERB and TS-SUPERB tasks, including difficult three-speaker separation, with 85.68 million inference parameters.

·2 min
Speech & audio

Frame-level speaker embeddings enable efficient CPU streaming diarization

Torgashov and Köpüklü present a streaming system for identifying who speaks when without repeatedly encoding overlapping audio windows. Their causal encoder summarizes the preceding two seconds of audio at each step, enabling low-latency diarization at five times real-time speed on one CPU thread. The authors report leading accuracy among evaluated streaming systems on low-overlap benchmarks, with better resilience as speaker counts increase.

·3 min
Speech & audio

Adapting a speech language model with self-generated prefix supervision improves simultaneous translation quality-latency trade-offs.

The authors adapted a full-utterance speech language model to simultaneous translation by creating target prefixes from its own translations of partial waveforms. Using a confidence threshold to control latency, they found that multi-turn append-only decoding reduces commit-calibration error by 63–68% overall and 68–80% at early prefixes, while single-turn forced-prefix training yields only modest gains. The synthesis margin that controls supervision density introduces a non-monotonic trade-off, with a small margin sometimes extending low-latency coverage on shorter utterances but a larger margin degrading quality.

·3 min
Speech & audio

Metadata-supervised pretraining lets encoder-free speech LLMs identify speakers and emotions

The authors propose metadata-supervised pretraining (MSP) for encoder-free speech LLMs, using speaker identity and emotion labels to train models that directly process Mel-spectrograms. On joint ASR and speaker diarization, their model outperformed a randomly initialized encoder-based baseline under matched training data and was competitive with encoder-based models pretrained on much larger datasets.

·3 min
Speech & audio

Chain-of-thought reasoning aligns speech translation duration with source audio

The authors propose DuraS2ST, a speech-to-speech translation framework that uses chain-of-thought reasoning to explicitly plan the target utterance's content and phonetic duration before synthesizing speech. Evaluated on the CVSS-T benchmark, it achieved strong balance between translation quality and duration consistency, outperforming open-source and commercial baselines, including in zero-shot reinforcement learning settings.

·2 min
Speech & audio

Latent reasoning cuts speech model response delay by roughly 12x

The authors introduce AURAL, a latent reasoning framework for speech language models that generates plausible reasoning continuations in continuous hidden states instead of text tokens. Using a Gaussian mixture model for multi-path reasoning and joint prediction of future-state chunks, AURAL-RL achieves performance comparable to explicit chain-of-thought while cutting time to first answer token on Qwen2.5-Omni from 1.22 seconds to 0.10 seconds.

·2 min
Speech & audio

Language discrimination training reduces the multilingual gap in speech models

de Seyssel et al. show that strengthening language discrimination during self-supervised pretraining of a bilingual English/French HuBERT model reduces the performance gap compared to monolingual models. On continuous phonetic phone discrimination error, the bilingual baseline of 11.6% dropped to 10.4% (monolingual: 10.8%), while lexical accuracy rose from 52.1% to 56.7% (monolingual: 58.5%) and prosodic lexical performance from 68.9% to 72.9% (monolingual: 72.6%). These gains are largest when the intervention is applied early in training.

·2 min
Speech & audio

Pruned CTC cuts memory for large-vocabulary ASR training without sacrificing accuracy.

The authors introduce Pruned CTC, a method that reduces the memory footprint of CTC training for large-vocabulary ASR by restricting alignment computation to the subset of vocabulary tokens that appear in each batch, plus the blank token, while retaining full-vocabulary softmax normalization. They prove that this reduction is mathematically equivalent to standard CTC in both loss and gradients. With a Zipformer-M encoder and a 180K vocabulary, Pruned CTC achieves a 5.1x reduction in full-step memory with only a 17% step-time overhead, matching standard CTC accuracy across three corpora.

·3 min
Speech & audio

Marginal utility allocation of audio tokens improves quality under strict bit budgets

The authors introduce UniAdapt, a method that learns the marginal utility of residual-vector-quantization (RVQ) refinements on a frozen codec and allocates tokens under exact serialized-bit budgets. In utterance-level tests, it reduces Log-STFT distortion by 1.07 to 4.39 percent across speech, music and environmental audio without increasing the budget. A causal streaming variant improves three of four speech rates with zero budget violations across 800 evaluations and runs faster than real time.

·2 min
Speech & audio

Causality-aware framework improves simultaneous speech-to-speech translation quality and latency

Hussein et al. present FAST-CAP, a causality-aware framework for LLM-based simultaneous speech-to-speech translation. They introduce a factorized architecture (FAST) that separates lexical and acoustic modeling, and a causality-aware adaptive policy (CAP) for read/write decisions. Experiments on Spanish, German, and French show improvements of up to +1.2 BLEU and 26% latency reduction over fixed policies.

·2 min