Vision-language agents can learn from imagined futures in their own latent worlds

The authors present Dwell, which recasts a vision-language model as a recurrent latent transition operator and shows that latent imagination improves policy performance at lower compute.

Top University

Haoxi Li · Sikai Bai · Jianfei Ma · Qinglin Hou · Xiaocheng Lu · Tao Han · +2 more

HKUST · National University of Singapore · University of Southern California

Research Digest·28 October 2026·2 min read

Illustration: Vision-language agents can learn from imagined futures in their own latent worlds

In one line

Dwell reframes a vision-language model as a recurrent latent transition operator to learn from experience.

Before you read this

Three ideas this paper builds on

world model for agentsA predictive environment model that agents use to simulate outcomes, enabling planning, reasoning, and long-horizon task execution.87 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.200 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.221 papers →

Read it as

AI agents need to learn from experience, but gathering enough real-world data is slow and costly. Current approaches either require enormous amounts of interaction or waste resources by reconstructing every visual detail.

Dwell uses a , a large AI that understands images and text, to both predict future states and choose actions. It imagines future scenarios in a compact internal representation instead of generating full images, saving computation. The system then improves its behavior by learning from these imagined experiences.

Across multiple interactive visual environments, Dwell matched or beat other methods while using much less computation for imagination. This shows that an effective world model doesn’t need to recreate reality, it just needs to predict the future in a way that helps the agent decide what to do.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

world model for agents vision-language model reinforcement learning joint-embedding predictive architecture continual learning low-rank adaptation