SA-WAM extends World Action Models (WAMs) by adding depth prediction alongside RGB and action output. It uses a frozen VAE tokenizer and nonlinear depth normalization to incorporate 3D geometry without fine-tuning the pretrained video model. This allows the model to jointly predict future observations and actions while being spatially aware. On RoboCasa with only 50 demonstrations per task, SA-WAM achieved a 76.6% success rate, outperforming the Cosmos-Policy baseline by 9.5 points. It also set state-of-the-art results on the LIBERO-Plus benchmark and outperformed strong baselines in real-world tests with a UR5 robotic arm, especially in randomized environments. This is interesting because it demonstrates that injecting explicit geometric information into world action models, without sacrificing pretrained video priors, significantly improves both policy performance and sample efficiency.
Adding Depth Helps Video World Models Control Robots More Reliably
SA-WAM jointly predicts robot actions, RGB frames, and depth while retaining a pretrained video model’s frozen visual tokenizer.
Research Lab
Javier Alejandro Lopetegui Gonzalez · Paul Pacaud · Cordelia Schmid
Inria · École Normale Supérieure · PSL Research University
Research Digest··2 min read
In one line
SA-WAM integrates depth into world action models via nonlinear encoding, improving robot policy performance on benchmarks and real-world tasks.
Before you read this
An idea this paper builds onRead it as
Lopetegui Gonzalez, Pacaud, and Schmid adapt a pretrained video diffusion model into a 3D-aware robot policy by training it to predict actions, future images, and depth together.
The authors developed Spatially Aware World Action Model (SA-WAM), which uses a single video-diffusion backbone to jointly generate future RGB observations, depth maps, and robot actions.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.