The authors developed a diagnostic framework to separate VLM errors into visual-representation failures (evidence not encoded) and visual-utilization failures (evidence encoded but not used).
VLMs mostly err by ignoring visual evidence they possess
A probe called VELens reveals that 84% of LLaVA errors and 70% of Qwen errors come from failing to use encoded visual information.
Top University
Anat Veikherman · Mor Ventura · Nitay Calderon · Lihi Zelnik-Manor · Roi Reichart
Technion
Research Digest··2 min read
The researchers propose a method to decompose VLM errors into two types: those where necessary visual information is missing (representation failure) and those where it is encoded but not used (utilization failure).
§