The authors propose a method to select which teacher layer to align in for . They introduce a recoverability gap that measures how much of a teacher layer's features an unaligned student cannot linearly recover. This gap, computed from a single checkpoint, orders placements by alignment benefit more accurately than similarity-based criteria like CKA, which reverse the order. Their method, RARE, aligns the teacher layer with the largest gap, weights each token by its distance to the target, and phases out the loss when the average distance stops falling.
On SiT-B/2 with ImageNet 256x256, RARE achieves an FID of 18.02 without guidance and 4.46 with guidance, outperforming seven baselines including REPA and iREPA. It also trains in 14% fewer GPU-hours than iREPA. The improvement holds across model scales, teachers, datasets, and backbones. This is interesting because it provides a principled, data-driven way to choose alignment targets and scheduling, reducing the need for expensive hyperparameter search and making representation alignment more efficient.