used for (MPC) are typically trained to predict what actually happens (factual transitions). However, when planning, the model must compare what would happen under different actions from the same state—this is . A model can be accurate at factual prediction yet fail to distinguish between candidate actions because visual representations are dominated by static scene content, obscuring small action-dependent changes.
The authors propose AD-WM, which adds an action-recovery objective during training: the model must be able to infer the action taken from the current and predicted next latent representations. This encourages the latent dynamics to preserve action information. Across five simulation environments and a real robot transfer task, AD-WM improves success rates over a baseline LeWM, including increasing hard-start cube grasping from 3.7% to 52.0% and zero-shot pick-and-place from 42.2% to 71.1%.