Robots need to understand the physical world and predict what will happen as they carry out a series of tasks. Current models struggle to maintain this understanding over many steps because errors build up. RynnUnify is a large model, using a mix of components for reasoning and generating video, that can connect understanding of a scene with predictions of how it will change over time. It is designed to plan ahead by reasoning through videos of future steps. In tests, RynnUnify achieved the highest average understanding score among compared models and improved long-horizon planning scores by about 10 points over a prior system. Its extension for robot control succeeded on about half of selected tasks in both simulation and on real robots. This shows a step toward a single model that can understand, plan, and control physical actions.
AI models can reason better by predicting videos of outcomes
The authors report that selectively generating videos improves reasoning about unseen states, while a separate robot-control extension completes some multi-step manipulation tasks.
Chinese Tech
Minghao Zhu · Bohan Hou · Kun Xiang · Zixuan Wang · Yinzhou Tang · Yifei Yang · +17 more
DAMO Academy, Alibaba Group · Hupan Lab
Research Digest··2 min read
In one line
RynnUnify is a 16B unified embodied model that combines understanding with long-horizon visual planning via interleaved video-language reasoning.
Before you read this
Three ideas this paper builds onworld model for agentsA predictive environment model that agents use to simulate outcomes, enabling planning, reasoning, and long-horizon task execution.96 papers →unified multimodal understanding and generationA single end-to-end model processes and generates multiple data types (e.g., images, text) with equal competence.24 papers →vision-language-action modelA robot policy model that maps visual observations and language instructions to actions, trained on demonstrations and often used for manipulation tasks.61 papers →
Read it as
Zhu and colleagues introduce RynnUnify, a model that combines understanding physical scenes with generating videos of what might happen next.
The authors built a 16-billion-parameter model combining an 8-billion-parameter language reasoner with an equally sized video generator.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.