Some image tests are made harder by keeping pictures that current image-recognition systems repeatedly get wrong. But a wrong answer can mean either the picture is genuinely difficult or the test's answer key is incorrect.
Researchers applied these failure screens, meaning rules that retain images models miss, to an image collection with two independent human relabelings. They checked whether the original answer matched the relabeled answers.
Among images every tested program trained on labeled examples missed, 61% had incorrect labels, compared with 11.1% across the full collection. So a test built from model failures may partly measure answer-key mistakes rather than visual difficulty; checking labels can make its results more trustworthy.
