Cinematic storytelling often uses techniques where dialogue carries over from one scene to the next, or silence paces a scene. Existing artificial intelligence models that create videos with sound typically force audio and visual boundaries to match, leading to unnatural results like speech during silences or mouth movements on characters who are not speaking.
The researchers developed Découpage, a system that separates control of the visual and audio tracks. It uses a screenplay to guide each track independently. A special mechanism prevents lip movements when a speaking character is off-screen without extra training.
Découpage generates longer multi-shot videos with precise timing of dialogue and silence. This enables more realistic and cinematic AI-generated video.
