← All concepts

representation alignment

In machine learning, models often learn by imitating a pretrained teacher: they make their internal representations similar to the teacher's. This is useful because the teacher already knows good structure. But raw similarity can mislead: a teacher encodes both useful global knowledge and easy local details, so choosing what to imitate matters.

These papers show alignment works best when targets are chosen carefully. One selects a teacher layer whose features are hardest for the student to recover, weights harder tokens, and fades the loss. Another finds aligning to locally predictable content can hurt, and purifying that content helps. A third builds alignments from very few paired anchors plus many unpaired samples. A fourth uses semantic features to make video tokenizers more efficient. Together they turn alignment from simple copying into a principled design choice.

Papers
5
Released code
0
First seen
Oct 2026
Latest
Oct 2026

5 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Diffusion models benefit most from teacher features they still lack

    RARE selects the teacher layer with the largest recoverability gap, improving FID and training speed.

    Oct 2026
  2. 2Optimal transport aligns pretrained modalities with only hundreds of paired anchors

    Aligning two frozen unimodal encoders requires only 100 to 500 paired anchors when anchor-relative coordinates, entropic optimal transport, and distillation are combined.

    Oct 2026
  3. 3Purifying alignment targets improves diffusion generation, especially for reconstructive encoders

    Aligning to the residual after removing locally predictable content (LAP) improves diffusion generation quality and turns MAE into an effective alignment target.

    Oct 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.