The authors observed that a VLM's own prediction errors act as natural counterfactual demonstrations: a misclassified example exposes the same failure mode as the query, showing when visual evidence for one answer actually corresponds to a different one.
Using a model's own past errors improves visual in-context learning
SMILE replaces multi-stage counterfactual selection with a hard-negative pool mined from the target VLM's previous mistakes, boosting task scores by over 3 points at 5× lower latency.
The authors propose SMILE, a visual in-context learning method that constructs counterfactual demonstrations from a vision-language model's own past prediction errors.
Why this paper
From NVIDIA · Part of Agent Self-Improvement, now 52 papers
In one line
A model's own prediction errors can serve as counterfactual demonstrations to improve visual in-context learning.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§