Touch-aware world models improve contact-rich dexterous robot manipulation

DexTacWAM is a visuo-tactile World-Action Model that extends vision-centric video diffusion world models to dexterous manipulation by encoding tactile signals from each fingertip independently. A finger- and pose-aware tactile compressor aggregates per-finger features into a compact latent that is injected into the world model, enabling joint prediction of visual scene dynamics and multi-finger contact evolution. Across six contact-rich bimanual tasks with a 22-DoF platform, DexTacWAM achieved an average score of 70.6, nearly doubling the strongest baseline at 38.0. Ablations show that modeling contact evolution as part of the predicted world state is crucial: removing tactile world modeling dropped the average from 74.7 to 26.6 while keeping the same tactile features and action expert. A continual vision-to-touch learning approach adapts a frozen pretrained vision VAE to tactile data in about four hours using roughly 100 demonstrations per task, retaining visual prediction quality within 0.5 dB of vision-only models. The tactile compressor retains 89.4% of pre-fusion contact recall while speeding up training by 2.26x and inference by 1.29x. These results demonstrate that pretrained video priors can be extended to distributed multi-finger contact dynamics in a data- and compute-efficient manner.