The authors collected 297 publicly available videos covering Spanish, French, Japanese, Korean, Russian and Thai, with roughly 50 examples per language.
Omni models struggle to track who speaks when in long videos
A benchmark spanning six languages finds that accurate word recognition does not reliably translate into correct timestamps, speaker attribution or effective use of video.
Chinese Tech
Yuan Ge · Pei Zhang · Xize Cheng · Lingzhou Mu · Mingyang Wang · Jin Xu · +5 more
Northeastern University, China · Alibaba Token Hub, Alibaba · Kunming University of Science and Technology · NiuTrans Research
Research Digest··2 min read
Ge and colleagues introduce OmniLingual-MsASR, a benchmark testing whether multimodal language models can transcribe speech while identifying speakers and timing their utterances.
§