AI models can read diagrams correctly and still miscount their contents

Yin Liu finds that counting errors in Gemini models depend more on answer size than theoretical difficulty, while code-assisted counting shifts the bottleneck to reading diagrams.

Big Tech

Yin Liu

Google

Research Digest·9 November 2026·2 min read

Thread:Visually Grounded Reasoning

Illustration: AI models can read diagrams correctly and still miscount their contents

In one line

Gemini models usually fail to count diagrams because of computation rather than perception, but reading errors become the bottleneck on larger drawings.

Before you read this

Three ideas this paper builds on

vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.242 papers →failure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.132 papers →fine-grained spatial reasoningReasoning precisely about object positions, orientations, and spatial relations across planar, depth, and temporal dimensions for robotics and navigation.69 papers →

Read it as

When a model gets a count from a diagram wrong, it may have misread the picture or read it correctly but failed to calculate. Existing tests often mix these causes. A multimodal large language model is a system that processes both images and text.

To separate the two, the researchers asked different counting questions about the same drawings, including questions in different theoretical categories of computational difficulty. They also tested whether models could turn drawings into written lists of connected points, then supplied exact lists and computer code to test counting separately.

With step-by-step reasoning, Gemini models usually read small diagrams correctly but struggled to count. For the main model, the odds of an exact answer fell 2.38-fold for each tenfold increase in answer size. On large diagrams, reading became the weak point, and checking a rotated view helped recover missed connections.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

vision-language model chain of thought failure attribution fine-grained spatial reasoning benchmark