The researchers built AVE on a joint audio-video backbone with bidirectional cross-modal attention, allowing audio and visual information to influence each other during generation.
One model edits people, voices and movements in sync
AVE jointly swaps, inserts or deletes people in video and audio while using appearance and voice references, without spatial masks.
Top University
Ziye Yuan · Minghong Cai · Xu He · Ruchang Yao · Xiao Fu · Kaixin Ding · +5 more
University of Hong Kong · National University of Defense Technology · Kling AI · Kuaishou Technology · Peking University
Research Digest··3 min read
Yuan and colleagues present AVE, an end-to-end generative system designed to edit a person’s visual presence and speech together rather than passing outputs through separate video and audio tools.
§