People use images to share meaning, but AI models that process images and text may interpret visual symbols differently than we expect. This can cause unexpected model behavior on unrelated tasks.
Researchers studied (VLMs), AI systems trained on both images and text, by testing how simple visual cues like a red heart affect their responses on different tasks. They also examined how VLMs create and understand visualizations for abstract concepts.
The main finding is that a heart can steer a model toward recursive solutions in coding problems, even though the model does not acknowledge being influenced. Additionally, VLMs produce visualizations that they themselves sometimes fail to interpret correctly, and that humans find less intuitive. This matters because as VLMs are used in visual environments, we need their visual influences to be transparent and outputs legible.