The paper introduces MTVA-Bench, a benchmark that evaluates language models inside cascaded voice agents by simulating the conditions they face after automatic (ASR) and before text-to-speech (TTS). It holds the model's production interface fixed, using an LLM as the caller and a mock backend that responds to actual arguments sent. Scoring combines deterministic tool checks with two LLM judges that must cite specific messages. In a seven-model study, six models selected the correct tool within 6.4 points of each other, but overall scores spanned 24.4 points, with most of the gap coming from argument values, action ordering, rule compliance, and what the model says around its tool calls.
Voice-agent models diverge most beyond choosing the correct tool
MTVA-Bench isolates the language model in cascaded voice systems and tests multilingual, multi-turn tool use under realistic transcript conditions.
Independent
Pritish Mishra · Ishaan Kumar · Akshat Mandoli · Sudarshan Kamath
Research Digest··2 min read
Thread:Agent Harness Optimization
In one line
Most voice agent failures come from argument values and rule compliance, not from selecting the wrong tool.
Before you read this
Three ideas this paper builds onlarge language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.308 papers →speech recognitionThe task of converting spoken audio into text, evaluated across dialects and used for training efficient transcription models.8 papers →multi-turn tool useThe capability of an agent to invoke external tools sequentially over multiple conversational turns to complete complex tasks.49 papers →
Read it as
Mishra et al. introduce a benchmark that evaluates the decision-making language model separately from automatic speech recognition and text-to-speech components.
The authors built MTVA-Bench around 49 voice-agent configurations and 490 reviewed scenarios spanning seven languages.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.