Existing methods for controlling camera and object motion in video generation rely on image-plane cues that are ambiguous in depth and rotation, or on 3D proxies with incomplete geometry. 4Director introduces an explicit 4D scene representation where each object is a complete canonical mesh moved by one rigid transformation per frame, sharing a coordinate frame with the camera. A Motion Adapter injects a depth rendering of this scene into a pretrained video generator, which then synthesizes appearance, illumination, and non-rigid dynamics. The authors built a dataset of 20,774 clips with automatic rigid 3D annotations and introduced a new metric, Identity-Gated IoU, for evaluating object control. Experiments and a user study show that 4Director consistently outperforms prior state-of-the-art methods in visual quality, camera control, and object control.
Complete 3D meshes improve control of generated video motion
4Director uses canonical object meshes and per-frame rigid transformations to jointly control camera movement, object position, depth and rotation.
AI Startup
Wei Cao · Hao Zhang · Vikram Voleti · Yuqun Wu · Mallikarjun B R · Shimon Vainer · +2 more
Stability AI · University of Illinois Urbana-Champaign
Research Digest··3 min read
In one line
Using explicit rigid 3D geometry to control video world models achieves better camera and object control than prior methods.
Before you read this
Three ideas this paper builds onvideo world modelA model that generates realistic, controllable video sequences by learning the underlying physical and temporal dynamics of a scene.6 papers →diffusion modelA type of generative model that learns to reverse a gradual noising process to create data.199 papers →flow matchingA generative training method that learns an ordinary differential equation mapping noise to data by matching velocity fields along probability paths.175 papers →
Read it as
Cao and colleagues present a video generation system in which users prescribe camera and object trajectories in a shared 3D coordinate system.
The authors represent a video as an explicit 4D scene, meaning 3D geometry that changes over time.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.