When we test vision-language models on multiple-choice questions, the model's answer isn't always a clean letter like "A" or "B". Often the model writes a sentence or gives irrelevant text. The 's scoring code then has to decide whether to count that as right or wrong, and different benchmarks make very different choices.
This paper looked at eight well-known benchmarks and five models that run on a single graphics card. The researchers found that the scoring rule alone could change a model's reported accuracy by up to 22 percentage points on the exact same outputs. On models that always output clean letters the difference is zero, but on the smaller models practitioners actually use, the scoring rule substitutes for the missing answer up to 37% of the time.
The key finding is that no standard scoring rule reliably captures what the model actually said. One rule credits an answer the model never gave, another discards correct answers given in words. The paper shows that benchmark scores can be artifacts of the scorer, not measurements of the model. This matters for any application relying on these benchmarks.