World state registers enable consistent multi-agent video generation across views

WorldWeaver introduces learnable tokens that store shared world information, dynamically updated after each generated chunk, improving logical consistency in two-agent Minecraft video generation.

PaperBig Techcs.CVarXiv:2607.21594v1 ↗

Sicheng Mo · Yuheng Li · Ziyang Leng · Krishna Kumar Singh · Bolei Zhou

University of California, Los Angeles · Adobe Research

Research Digest·24 July 2026·3 min read

Read the paper →

The authors propose WorldWeaver, a streaming multi-agent video diffusion model that augments autoregressive rollout with cross-agent world state registers: learnable tokens that maintain shared world information, track individual agent status, and are updated after each generated chunk. The registers are grounded with supervision from agent status, global bird's-eye views, and scene text. Experiments in two-agent Minecraft show that explicit world-state modeling improves logical consistency and generation quality over baselines that only carry forward observation history.

What they did

The authors designed a streaming multi-agent autoregressive diffusion model called WorldWeaver. The key innovation is the addition of world state registers: a set of learnable tokens that store shared world information across agents and views. These registers are dynamically updated after each generated video chunk. To ground the registers, the model is supervised with multiple signals: individual agent status, global state views (including bird's-eye views), and scene text. The architecture also incorporates a Mixture-of-Transformers design that uses separate sets of weights for world state modeling and visual frame modeling. Experiments were conducted on two-agent Minecraft video generation tasks, comparing against standard autoregressive video diffusion pipelines that only propagate observation history.

Key findings

World state registers enable the model to maintain a persistent shared world state across agents and views, improving logical consistency in generated multi-agent videos.
The Mixture-of-Transformers architecture effectively separates world state modeling from visual frame generation, leading to better overall generation quality.
Supervision from global state views (e.g., bird's-eye views) and scene text helps ground the world state registers, resulting in more coherent agent interactions.
Quantitative and qualitative evaluations show that WorldWeaver outperforms baselines that rely solely on observation history conditioning, particularly in scenarios requiring coordination between two agents.

Why it matters

This work addresses a fundamental limitation of existing autoregressive video diffusion models in multi-agent settings: the inability to maintain a shared world state that persists across agents and evolves across views. By introducing explicit, updatable world state registers, WorldWeaver enables more coherent and logically consistent generation for interactive environments. This has implications for simulation, virtual world modeling, robotics, and game development, where maintaining a consistent world state across multiple perspectives is crucial.

Caveats

The experiments are limited to two-agent Minecraft scenarios; scalability to larger numbers of agents or more complex environments remains unexplored. The world state registers require additional supervision signals (e.g., bird's-eye views, scene text), which may not be readily available in all settings. The computational cost of the Mixture-of-Transformers design and the dynamic register update mechanism is not thoroughly analyzed. Generalization to other domains (e.g., real-world video, other games) is not demonstrated.

Analysis

This work sits at the intersection of multi-agent modeling and generative video diffusion, a rapidly growing area. Prior work on world models (e.g., Dreamer) typically uses latent representations that are not explicitly shared across agents in a generative context. By introducing a token-based register that is updated autoregressively, WorldWeaver offers a clean mechanism for maintaining shared state that could be integrated into other streaming generation pipelines. An open question is how the registers scale with the number of agents and the complexity of the world state, and whether similar gains can be achieved with end-to-end learned latent without explicit supervision. The Mixture-of-Transformers design is a practical engineering choice that may inspire further work on specialized weight spaces for different aspects of generation.