Building a 3D model from a long video requires a compact scene memory that is updated frame by frame. An unreliable update can overwrite past knowledge and cause the reconstruction to drift over time. Existing methods control only how strongly each frame changes the memory, not the direction of the change. Our method controls both: we rectify the update direction using the memory's history and adjust the strength based on the visual discrepancy. On long video sequences (500 to 1000 frames), our approach reduces pose error by 23.87% on one benchmark and improves reconstruction accuracy by 56.52% on another. It requires no extra training and runs at the same speed and memory. This makes long-range 3D reconstruction more reliable.
AI systems reconstruct scenes with less drift using recent history
The authors report that correcting both the direction and strength of memory updates improves long-video reconstruction without additional training or a larger memory footprint.
Big Tech
Kenneth Yang · Yu-Chiang Frank Wang
National Taiwan University · NVIDIA
Research Digest··2 min read
In one line
HIST3R controls the direction and magnitude of state updates using history to reduce drift in long-video 3D reconstruction.
Before you read this
An idea this paper builds onRead it as
Kenneth Yang and Yu-Chiang Frank Wang examine why AI systems that build three-dimensional scenes from streaming video gradually lose geometric consistency.
The authors build on CUT3R, which processes video one frame at a time and compresses previous observations into a fixed-size internal scene memory.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.