Predicting what an object will do from a scene is difficult: video shows how things look, but not directly how their positions in three-dimensional space change. Motion systems can also be limited by scarce records of object movement.
WorldFlow takes an image, a language instruction, and selected points on objects, then predicts both a future video and how those points move. It trains the two predictions together, so visual examples also help teach motion.
On a spatial-reasoning test, it cut average displacement error, the average distance between predicted and actual positions, by 54.2% versus the strongest other method. This suggests that learning appearance and motion together can improve forecasts of physical movement.