Real-time video-to-video translation, used in streaming or AR, struggles to keep a consistent look over long sequences. Current methods only remember recent frames, so objects that disappear and reappear are often regenerated inconsistently. The new method, vid2vid-long, instead uses the original input video as a reference: it finds matching patches across input frames and uses those to retrieve and reuse previously generated output for those regions. This allows the model to maintain consistent appearance for objects throughout the video, even if they leave and return, without slowing down. In tests across several standard video models, it improved long-range temporal consistency while keeping real-time performance.
Video translation models maintain long-term consistency by matching input regions across frames
New method uses explicit patch-level correspondences to retrieve previously generated content, avoiding the drift typical of fixed-window approaches
Big Tech
Gaurav Parmar · Zhengqi Li · Richard Zhang · Jun-Yan Zhu · Srinivasa Narasimhan · Eli Shechtman · +1 more
Carnegie Mellon University · Adobe Research
Research Digest··2 min read
In one line
Matching regions in input video should produce matching outputs, enabling long-range consistency in real-time video-to-video translation.
Before you read this
Two ideas this paper builds onRead it as
The authors propose vid2vid-long, a correspondence-based dynamic memory for video-to-video translation that retrieves relevant past outputs by matching regions in the input video.
The authors introduce vid2vid-long, a method that computes patch-level correspondences in the input video and uses them to index a compressed global representation of prior generated outputs.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.