Video models learn the physical world from internet footage, but using them to control robots has required generating full videos of the future, too slow for real-time decisions.
Tapestry converts a robot action into a format exactly like a small piece of a video frame. This lets a single shared model learn three tasks: predicting the next video frame, predicting the action from the current video, and generating an action directly from random noise.
At test time, the model only generates the action in 84 milliseconds per chunk, roughly 44 times faster than comparable methods. This brings the rich physical knowledge of internet video into practical, closed-loop robot control.