The paper introduces ORAV, a for audio-video generation from multiple heterogeneous references. It comprises 380 task instances with 2–10 references, 9 semantic roles, and 30 role compositions, along with a reference-aware pairwise evaluation protocol. The protocol achieves 86.08% effective agreement with human judgments on held-out instances. Evaluating five frontier systems reveals that overall rankings hide distinct strengths across reference compositions, and a common failure is reproducing unintended source content instead of the requested result. This benchmark addresses the need for controllable, compositional, and reference-faithful audio-video generation, offering a tool to track progress in this emerging area.
Benchmark exposes distinct weaknesses in multimodal audio-video generation systems
ORAV Bench tests whether models can selectively combine identities, motion, scenes, speech and other properties from multiple references.
Chinese Tech
Jiacheng Hua · Xiaokun Feng · Jiaqi Hua · Chang Liu · Biao Wang · Miao Liu
Tsinghua University · Tencent Hy
Research Digest··3 min read
In one line
ORAV is a benchmark for audio-video generation from multimodal contexts that reveals distinct model strengths and failures.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →unified multimodal understanding and generationA single end-to-end model processes and generates multiple data types (e.g., images, text) with equal competence.31 papers →diffusion modelA type of generative model that learns to reverse a gradual noising process to create data.197 papers →
Read it as
The authors introduce ORAV Bench, a benchmark for generating audio and video from several heterogeneous references linked by a textual instruction.
ORAV Bench contains 380 generation tasks, each supplying between 2 and 10 image, video or audio references.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.