Large language models (LLMs) that process images, called multimodal LLMs, split a picture into patches and encode each patch as a token. Vision encoders trained to match images with captions can hide global caption information inside individual tokens. That hurts tasks like counting or spatial reasoning.
The authors compared those tokens with tokens from self-supervised encoders, which learn from images alone, and identified semantically inconsistent tokens. They trained a multimodal LLM with an extra penalty for inconsistent tokens while keeping the usual next-word prediction loss, calling this semantically guided training.
The trained model beat all baselines on vision-centric benchmarks and stayed strong on text-centric ones. So the vision encoder should be trained jointly with the language model, not chosen separately.