The study tests how multimodal judges evaluate agent-generated videos when shown execution traces alongside the clips. For visual requirements, open-weight Qwen-VL judges heavily rely on the auxiliary text: a supportive trace makes them accept up to 90% of failures, while a contradicting trace makes them reject correct clips. Frontier closed models are not affected. This reveals a reward-hacking vulnerability in agentic video-generation pipelines that use LLM judges as verifiers.
Execution Logs Can Mislead Visual Judges in Video-Generation Agents
Holding video frames fixed, the study shows that auxiliary text can sharply distort some multimodal models’ assessments of whether requested events occurred.
In one line
Execution traces cause video-generation verifiers to judge clips based on text, not visuals.
Before you read this
Three ideas this paper builds onagentic video generationMulti-agent systems collaborate autonomously to produce long-form videos maintaining narrative and visual coherence.21 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.251 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.328 papers →
Read it as
Xu tests whether video-generation verifiers judge visual evidence independently from plans, narration and execution logs.
The author assembled 109 generated two-event clips with manual labels, including near misses where one requested event was visible but the other was absent.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.