Models that answer questions about images by reasoning step by step often miss small details because their attention shifts away from the correct image region as the reasoning grows.
Researchers identified a few attention heads, parts of the model that decide where to focus, that are responsible for selecting visual evidence. They trained these heads to focus on the relevant image region whenever the reasoning mentions an object, using an extra training objective that does not slow down the model at test time.
On three benchmarks, this technique improved accuracy from 74.1% to 76.6%, surpassing larger models and methods requiring multiple passes. The model also reasoned with fewer steps, making it more efficient, and the improvement carried over to other model families.