Small language models answer visual questions better without exchanging text descriptions

The authors connect separately pretrained vision and language models through internal representations, improving visual question answering while training only a small connecting module.

Big Tech

Mo Shahdloo · Faegheh Sardari · Mathew Salvaris

Microsoft Applied Sciences Group (ASG)

Research Digest·2 November 2026·2 min read

Illustration: Small language models answer visual questions better without exchanging text descriptions

In one line

Latent-Lens transmits VLM hidden states directly to a frozen SLM in latent space, improving visual reasoning and captioning over text-based communication.

Before you read this

Three ideas this paper builds on

latent communicationLatent communication transfers information between models via internal states like KV caches instead of explicit text.15 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.217 papers →multi-agent architectureA system where multiple AI agents collaborate, each handling specialized roles, communicating and coordinating to complete complex tasks.100 papers →

Read it as

Small language models used on edge devices cannot handle images directly. To give them visual perception, systems typically convert images into text descriptions, but text loses spatial and fine-grained details essential for accurate visual reasoning.

Latent-Lens adds a tiny module that routes internal visual representations from a vision model directly into the memory of the language model, bypassing text. This preserves the richness of visual data and requires no retraining of either original model.

On visual reasoning tasks, it achieved 81.0% accuracy, a large improvement over text-based communication. This allows small language models to gain vision capabilities without expensive retraining, enabling more capable AI on devices with limited compute.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

latent communication vision-language model small language model low-rank adaptation multiband depth representation multi-agent architecture