The authors first investigated whether weak visual in-context learning, meaning learning a task from examples supplied in the prompt, reflected problems with perception or rule inference.
Removing textual shortcuts helps vision-language models learn from visual examples
Shabtay and colleagues report that training on demonstrations whose rules are never stated improves visual in-context learning across three models and 11 tasks.
Big Tech
Nimrod Shabtay · Asaf Yehudai · Avihu Dekel · Eli Schwartz · Raja Giryes
IBM Research · Tel-Aviv University
Research Digest··2 min read
Vision-language models can answer questions about images yet struggle to infer a task from visual examples.
§