The authors examined ten contrastive vision-language encoders, which represent images and captions as vectors and rank matches by their similarity.
Image search models can retain left-right information but ignore it
Across ten vision-language encoders, the authors distinguish inaccessible spatial information from information lost during pooling, then improve retrieval by reading image-patch features directly.
Chinese Tech
Mahdi Mostajabdaveh · Zirui Zhou
Huawei Technologies Canada
Research Digest··2 min read
In one line
A patch-grid readout restores left/right image search by recovering spatial relations that vision-language similarity either hides in pooled image summaries or discards during pooling.
Before you read this
Three ideas this paper builds onvision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.229 papers →fine-grained spatial reasoningReasoning precisely about object positions, orientations, and spatial relations across planar, depth, and temporal dimensions for robotics and navigation.61 papers →failure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.123 papers →
Mostajabdaveh and Zhou report that image search models can encode which object is on the left without using that information to match captions.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.