Vision-language models struggle with non-English documents across many tasks

A new multilingual benchmark of 13 tasks in five languages finds that even the best models average only 69 out of 100, with layout and entity extraction particularly weak.

Industry

Matteo Gabburo · Daniela Baiamonte · Stefano Simonazzi · Silvia Gardini · Sean Cesare Lazzeri · Andrea Zugarini

Aithlas · Villanova

Research Digest·13 November 2026·2 min read

Illustration: Vision-language models struggle with non-English documents across many tasks

In one line

Current vision-language models remain uneven at multilingual document understanding: larger size does not erase language gaps or guarantee strength across tasks.

Before you read this

Three ideas this paper builds on

vision-language modelA model that handles tasks combining visual and textual information, such as editing images from conversation or reasoning across images and text.251 papers →benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.347 papers →structured layout reasoningA multi-step process that generates or verifies spatial arrangements of objects against explicit constraints and geometry.17 papers →

Read it as

are artificial intelligence systems that interpret images and follow written instructions. Document understanding means extracting words, meaning, and structure from pages. Today’s tests often focus on English or a narrow task, so they can miss how well these systems handle real documents in other languages and formats.

The tests models on tasks such as copying text, summarizing pages, finding named entities like people or organizations, and interpreting page layouts. It covers several languages and document styles, and includes multi-page documents.

The best average score was 69.0, and only eight systems scored 65 or higher. Bigger models did not reliably do better, and strong transcription did not guarantee strong meaning extraction. These results show why English-only tests and a single overall score can give an incomplete picture of whether a model is useful.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

vision-language model benchmark multilingual document understanding structured layout reasoning