Joint generation of video with companion modalities like motion or audio often suffers from asymmetric information flow. While video strongly influences the companion, the companion barely influences video, leading to inconsistencies. The authors identify this 'reciprocal correspondence gap' and propose RecCAR, a simple KL regularization that uses the strong video-to-other attention as a reference to align the weak other-to-video attention. RecCAR is applied via lightweight LoRA fine-tuning. In video-motion generation, it improves the Human Anatomy score from 0.69 to 0.75. In video-audio generation, it reduces audio-video desynchronization from 0.804 to 0.752 while preserving overall quality. The method is modality-agnostic and requires no inference-time changes.
Reciprocal attention regularization improves alignment in joint video generation
RecCAR strengthens how motion and audio constrain generated video by aligning the model’s weaker cross-modal attention pathway with its stronger reciprocal pathway.
Big Tech
Bar-Ilan University · NVIDIA
Research Digest··2 min read
In one line
Joint multimodal generators have asymmetric cross-attention; RecCAR aligns weaker modality-to-video attention to improve consistency.
Before you read this
An idea this paper builds onRead it as
Rahamim et al. identify a directional imbalance in multimodal diffusion transformers: motion or audio attends strongly to video, while exerting comparatively weak influence over video generation.
The authors analyzed bidirectional cross-attention in joint video-motion and video-audio .
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.