Multimodal models understand images and text together. How to encode images is debated: some use a separate vision encoder trained beforehand, others train it from scratch, and some use no separate encoder at all. These choices are mixed up, making it hard to know which part is important.
Researchers compared these approaches by training all models from scratch on the same data, controlling for compute. They tested three aspects: whether the encoder needs separate training, whether it needs special parameters just for vision, and whether those parameters must come before the model combines vision and language.
They found that dedicated vision parameters are crucial: without them, models perform worse on visual tasks. But these parameters work just as well placed early inside the main model, instead of in a separate encoder. Separate encoder training is not needed. This simplifies design and can reduce latency.