Tracking how points move in 3D from a single video is difficult. It requires finding the same point across frames, estimating its depth, and guessing its motion when it is hidden. Current methods learn all these tasks together, but they lack enough training data for real-world scenes.
The researchers show that combining existing 2D trackers and depth models already produces decent 3D trajectories. Then they train a neural network to repair errors in these trajectories, especially during occlusions. This network learns from synthetic data with simulated errors, so it works with any tracker and depth model without retraining.
On the WorldTrack benchmark, ReLift improves tracking accuracy by 2.1 in average point deviation on all points and 3.0 on dynamic points over previous methods. This delivers more reliable 3D motion data for applications like robotics and video generation, and it can incorporate future improvements in trackers and depth models.