Video generators can better follow physics with code-built scene drafts

Wang and colleagues report that an agent which programs and renders moving 3D scenes improves a downstream video generator on two benchmarks.

Big Tech
Chaoyang Wang · Sicheng Gao · Junan Hu · Kunyang Li · Dongyang Chen · Yiyang Zhou · +4 more

University of Central Florida · University of Würzburg · Microsoft Research · Tsinghua University · UNC-Chapel Hill

Research Digest··2 min read
Illustration: Video generators can better follow physics with code-built scene drafts

In one line

Video-Coder improves video generation by having an AI agent write and execute code to build a 3D scene before rendering the final video.

Before you read this

Three ideas this paper builds on
Read it as

Current video-generating AI systems struggle with physical rules, for example, they might show a ball falling sideways or objects appearing in the wrong place. This happens because these systems only learn from watching videos, never understanding the 3D world or the laws of motion.

The researchers built an AI agent that, given a text prompt, writes and runs a short computer program. That program describes a 3D scene with objects, their positions, and how they move. The agent then executes the code to produce a rough draft of the video, which provides clear spatial and motion cues to a standard video generator.

Testing showed the approach boosted physical accuracy by around 7 points on one benchmark and 24 on another. This means videos are far more likely to show objects moving naturally and staying where they belong, making AI-generated videos more reliable for tasks like simulation or content creation.

How it works

before and after
GENERATE DIRECTLY FROM TEXTText promptText promptVideo generator: Infers structure and motionVideo generatorInfers structure andmotionGUIDE GENERATION WITH A SCENE DRAFTScenedescriptionExecute andrender codeRenderedfeedbackSpatial andmotionguidanceText guidanceText promptText promptVideo-Coder: Writes, runs and revises codeVideo-CoderWrites, runs andrevises codeRendered scene draft: Shows geometry and motionRenderedscene draftShows geometryand motionVideo generator: Produces the finished videoVideogeneratorProduces thefinished videoDuring training, execution rewards gradually lose weight so video quality matters more.
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Plate by Ollie Flash, picture desk. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe