Modern AI models can generate both text and images, but it is unknown whether the internal 'dials' that control things like formality or object size work the same way for both. Researchers found that a steering vector (a mathematical direction inside the model) discovered using only text could then be added to image generation to produce formal clothing or design, and vice versa. This cross-modal transfer works in both directions, recovering up to 84 percent of the original steering effect, suggesting that concepts can be controlled with a single, modality-agnostic lever.
Steering vectors transfer causally between image and text generation
Adding a direction found in one modality to tokens of another modifies outputs in interpretable ways across three unified multimodal models.
Top University
Donghao Zeng (Mila, McGill University) · Nishant Subramani (Carnegie Mellon University) · Tamar Rott Shaham (MIT CSAIL) · Benno Krojer (Mila, McGill University)
Research Digest··3 min read
In one line
Steering vectors derived from one modality causally transfer to control generation in other modalities within unified multimodal models.
Before you read this
Three ideas this paper builds onactivation steeringActivation steering modifies a model's internal vector representations during inference to shift behavior, such as editing outputs, removing capabilities, or controlling trust.52 papers →unified multimodal understanding and generationA single end-to-end model processes and generates multiple data types (e.g., images, text) with equal competence.22 papers →mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.94 papers →
Read it as
What someone who builds software would do differently after reading this paper. We write it only for papers that change something in practice.
The authors show that linear steering vectors extracted from either text or image representations causally transfer zero-shot to the other modality in unified autoregressive models.
The authors studied cross-modal steering in three unified multimodal models (UMMs), UniAR, Emu3.5, and Liquid, which autoregressively generate both text and image tokens from shared parameters.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.