Medical AI segmentation models exploit size hints leaked from benchmark captions

The authors find that text prompts in vision-language segmentation benchmarks often encode lesion extent directly from the ground truth mask, inflating apparent model accuracy.

Top University

Niran Nataraj · Ankita Das · C. Krishna Mohan · Kenji Kawashima

The University of Tokyo · Indian Institute of Technology Hyderabad

Research Digest·12 November 2026·3 min read

Illustration: Medical AI segmentation models exploit size hints leaked from benchmark captions

In one line

In medical image segmentation, the benefit of text captions comes largely from 'extent leakage', not language understanding.

Before you read this

Three ideas this paper builds on

privileged informationInformation available only during training that improves performance but is not accessible at inference.82 papers →vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.250 papers →counterfactual reasoningComparing actual outcomes with imagined alternatives to estimate the effects of different actions or conditions.108 papers →

Read it as

Many medical AI systems that outline abnormalities in images use short text captions to improve accuracy. But in common tests, these captions are created from the correct answer, telling the AI the lesion's size or location. This information is not available in real use.

The researchers edited the captions of two such AI systems on several medical image sets. They kept only the size or location words to see what the models rely on.

They found that on most tests, the models' performance came almost entirely from these leaked words. Removing them drastically reduced accuracy, while other words made no difference. This 'extent leakage' inflates performance and does not reflect real language understanding. The authors propose a new evaluation method to separate true understanding from cheating.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

privileged information vision-language model counterfactual reasoning benchmark retrieval augmented generation