AI model controls characters in unfamiliar game worlds in real time

By conditioning video generation on articulated skeletons and scene geometry, the approach enables precise control over character motion even in environments and with characters never seen during training.

Chinese Tech

Yuzhe Wu · Zijun Lin · Yeying Jin

Tencent · National University of Singapore · Nanyang Technological University

Research Digest·30 October 2026·2 min read

Illustration: AI model controls characters in unfamiliar game worlds in real time

In one line

An agentic game world model that generalizes zero-shot to novel scenes, characters, and motions with real-time interaction.

Before you read this

Three ideas this paper builds on

world model for agentsA predictive environment model that agents use to simulate outcomes, enabling planning, reasoning, and long-horizon task execution.93 papers →agentic video generationMulti-agent systems collaborate autonomously to produce long-form videos maintaining narrative and visual coherence.16 papers →synthetic data generationCreating training examples artificially, often using an LLM, to overcome real-data scarcity or privacy constraints.49 papers →

Read it as

In video games, players often want to control characters in new ways or explore different worlds, but current systems usually work only with predefined actions and familiar settings.

The researchers built Wizard, which uses an agentic framework (a set of rules that tracks the game state) and structured proxies like character skeletons and scene depth maps to guide a video generation model in real time. It was trained on a diverse set of scenes and characters.

Wizard can handle novel characters and scenes it has never seen before, with precise real-time interaction. This allows players to create and manipulate customized game worlds, opening new possibilities for player-driven game creation.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

world model for agents agentic video generation synthetic data generation diffusion model