Standard visual tests for multimodal AI typically show a single object, making tasks artificially simple. Real scenes contain many similar objects, creating visual ambiguity that demands complex descriptions.
A new tests models on thousands of images with multiple objects of the same category. Top AI systems scored only just above half on attribute binding and numerical counting, much lower than on standard tests.
The result shows current models struggle to bind visual features and interpret complex referring expressions. It sets a more realistic standard for evaluating visual reasoning, revealing weaknesses hidden by simpler tests.