ALIVE is a framework for inserting objects into videos so they participate in interactions like being picked up or cut. The authors curate a dataset of 35,800 video editing pairs combining rendered, generated, and real-world clips, and train a diffusion editor guided by a vision language model that predicts temporal interaction descriptions from the same inputs. Without the VLM guidance, ALIVE improves overall editing quality by 43.9% on an interaction benchmark and 4.4% on a general insertion benchmark over the strongest baseline, and VLM predicted guidance further improves the interaction score by 0.95 points. This work is interesting because it addresses a difficult gap in video editing: making inserted objects behave coherently with existing actions.
ALIVE inserts objects that respond coherently to video actions
A diffusion editor trained on 35,800 curated editing pairs, with optional vision-language-model guidance, improves interaction fidelity by 43.9% over prior baselines.
Big Tech
Zhenghong Zhou · Zhe Lin · Jiebo Luo · Yuqian Zhou
University of Rochester · Adobe Research
Research Digest··1 min read
Thread:Video Diffusion Scaling
In one line
ALIVE inserts objects into videos that coherently interact with the source actions using an edited first frame and object-only instruction.
Before you read this
Two ideas this paper builds onRead it as
The authors introduce ALIVE, a framework for inserting objects into videos so that they participate in ongoing interactions, such as being lifted or cut.
The authors curated 35,800 video editing pairs from three complementary sources: 3D-rendered scenes, model-generated videos, and real-world footage, combined with general insertion pairs from the ROSE dataset.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.