VideoGen-Agent is a multimodal agent that learns to use external tools for video generation through supervised fine-tuning followed by multitask reinforcement learning. It addresses six challenging tasks: procedural knowledge, identity preservation, physics simulation, scene composition, and multi-shot sequences. The agent coordinates augmentation, generation, and tools in a multi-turn loop, using intermediate observations to guide decisions.
On the new VABench benchmark of 600 prompts, the agent boosts its base text-to-video generator from 56.5 to 75.6, a 19.1-point gain. Upgrading the generation tools further raises the score to 86.1 without any additional agent training. Human raters prefer the agent's outputs over the strongest standalone baseline in 84.3% of comparisons. These results show that learning tool use can improve video generation and that the agent can benefit from later improvements in generation tools.