Most multimodal LLMs use a pretrained visual encoder, but encoder-free models learn visual representations directly from pixels, offering a simpler architecture. This study systematically compares scaling laws for both architectures using controlled experiments with shared decoder, data, and optimization. They fit scaling laws for text and multimodal objectives separately.
Key findings: On the text objective, both architectures have nearly overlapping loss-compute frontiers. On the multimodal objective, encoder-free models underperform at small scales but their loss decreases faster with compute; extrapolation predicts a crossover around 10^22 FLOPs, within practical pretraining budgets. They also find that removing the encoder shifts compute-optimal allocation toward larger decoder models, and that the language model adapts by increasing bidirectional attention among visual tokens, processing visual info earlier, and concentrating expert routing.