Navigation systems often use world models to predict what they will see after taking an action, which helps them plan. Training these models typically requires pairing video with the exact commands that caused the motion, but this action-labeled data is expensive and scarce, limiting how well the model can handle diverse environments. Unlabeled navigation video is abundant but lacks the actions needed for training.
OpenNWM first learns latent actions from unlabeled video alone, inferring the underlying motion by predicting visual changes. It then fine-tunes on a small set of real action labels to map these latent actions to actual motor commands.
This pretraining gives OpenNWM state-of-the-art performance in predicting future views and navigating to goals, even in completely new environments. The key insight turns abundant unlabeled video into valuable training data, dramatically improving generalization.