Novel view synthesis from sparse inputs is challenging: reconstruction-based methods handle observed geometry but fail on unseen regions, while diffusion-based methods generate plausible content but lack geometric consistency. VGGT-Diff bridges these approaches by injecting explicit geometry latents from the VGGT-Ω foundation model into a pretrained video , providing query-aligned spatial conditions while retaining generative priors.
Key components include a confidence-aware Visual Geometry Router (VGR) that maps source features to target-view latents using 3D points and confidence, preserving both front and back surface evidence. Point-Track Residual Consistency (PTRC) aligns denoising residuals along reliable 3D correspondences to reduce cross-view drift. Training-time stochastic regularization and inference-time guidance further improve robustness to imperfect geometry.
The method achieves competitive or state-of-the-art performance across interpolation and extrapolation under varying viewpoint difficulties, demonstrating strong fidelity and cross-view coherence. This is interesting because it effectively combines the strengths of geometry-grounded reconstruction and generative diffusion for sparse-view scenarios.