AI models’ attention reveals only part of how they select images

Across four vision-language models, the authors find that image selection depends most consistently on components that both attend to the relevant image and pass identifiable information about it onward.

Big Tech

Junsung Park · Sangwon Yu · Donghun Kang · Yeongtak Oh · Jaewon Jeong · Gyeongtae Yoo · +3 more

Seoul National University · Upstage AI · Samsung Electronics · Korea University

Research Digest·4 November 2026·2 min read

Illustration: AI models’ attention reveals only part of how they select images

In one line

Image retrieval in vision-language models concentrates in sparse attention heads, and the heads that matter most for multi-image answers both attend to and write the target image's content.

Before you read this

Three ideas this paper builds on

mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.107 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.226 papers →sparse attentionSelectively attending to only the most relevant positions or tokens to reduce computational overhead.46 papers →

Read it as

Ask a model that reads text and pictures together which of several images a question refers to, and accuracy drops as more images are added. Large are networks whose layers contain attention heads, small units that each pull information from parts of the input. This work ranks heads two ways: how much attention each puts on the target image, and whether that image can be recovered from what each head passes onward.

The second test trains a small image matcher on each head's output; the two rankings disagree sharply, with fewer than half the top heads on both lists.

Switching heads off shows multi-image answering depends most on heads that both point at and pass on the target image. Pointing is not writing.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mechanistic interpretability vision-language model in-context learning sparse attention