Multimodal training can make language models rely on vision-shaped reasoning for text-only spatial problems

By ablating a vision-language direction in hidden states, the authors show that performance on spatial-imagery tasks drops sharply while symbolic reasoning remains intact, providing causal evidence that multimodal training reshapes text-only reasoning.

Top University

Qi Zhang · Bohan Wang · Qianru Sun · Hanwang Zhang

Nanyang Technological University · Singapore Management University

Research Digest·13 November 2026·3 min read

Illustration: Multimodal training can make language models rely on vision-shaped reasoning for text-only spatial problems

In one line

Multimodal training makes language models causally rely on vision-shaped internal representations even when reasoning from text alone.

Before you read this

Three ideas this paper builds on

mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.133 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.251 papers →fine-grained spatial reasoningReasoning precisely about object positions, orientations, and spatial relations across planar, depth, and temporal dimensions for robotics and navigation.69 papers →

Read it as

Language models trained on both words and images often use vision-shaped internal structure even when reading text alone. The team separated two text-only task types: spatial imagery, which invites a mental picture, and symbol manipulation, which follows formal rules.

For each model, they found a direction in its hidden state, the internal activity between layers, tied to vision-language training, and removed its effect. Across model families, this selectively damaged spatial imagery tasks while leaving logical ones intact.

In one model, navigation accuracy fell 27.6 percentage points while logical accuracy stayed flat, a 33.2-point gap. Multimodal training can thus change which inner representations text-only reasoning uses, not just how images are processed.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mechanistic interpretability vision-language model fine-grained spatial reasoning activation steering unified multimodal understanding and generation