AI models can align language and vision using 42-fold fewer samples

Balestriero and LeCun report that training on different observations of the same scene produces shared representations more efficiently than next-token prediction.

Academic

Randall Balestriero · Yann LeCun

Brown University · AMI Labs · New York University

Research Digest·30 October 2026·3 min read

Illustration: AI models can align language and vision using 42-fold fewer samples

In one line

Joint-Embedding Predictive Architectures with good views provably learn the same representation across modalities up to a rotation.

Before you read this

Two ideas this paper builds on

joint-embedding predictive architectureAn encoder predicts the latent representation of future data points from current and past observations, bypassing pixel-level reconstruction.23 papers →scaling lawA predictable power-law relationship between compute or data and model performance.18 papers →

Read it as

Models trained on different types of data, like images and text, often develop incompatible representations. This makes it hard to combine them for tasks like captioning. The Platonic Representation Hypothesis suggested they converge as models scale, but without proof.

The authors proved that when models use (JEPAs) with positive pairs that are two observations of the same scene, good views, they learn the same representation up to a rotation. They also showed that next-token prediction is a bad-view JEPA, which explains its inefficiency.

Good views need 42 times fewer training samples than next-token prediction to reach the same alignment. A small text JEPA outperformed language models up to 72 billion parameters in aligning with vision. This shows that aligning representations comes from positive views, not just scale.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

joint-embedding predictive architecture platonic representation hypothesis good views bad views next-token prediction scaling law