Standard tests for AI systems that understand images only check if the final answer is correct. But for real use, we need to know whether the system saw the image correctly and reasoned from that evidence. A wrong or right answer alone hides costly errors.
The researchers built a of 1,000 questions each with a human-written ideal reasoning chain. They used it to test if an AI's reasoning matched the chain and to measure the computational cost. A perception probe had the AI answer from its own description of the image.
Roughly 85% of failures from top models came from misperception, not faulty logic. A cost-aware metric changed the ranking of a quarter of model pairs. This shows that adding more reasoning does not fix a model that mis-sees the image.