Contrastive learning like CLIP assumes that any unpaired image and text are unrelated, but this wrongly punishes pairs that are semantically similar. Previous fixes use fixed thresholds to decide which pairs might be related, but those thresholds don't work well across different datasets.
Permute-then-Adapt (PTA) uses a small, fast teacher model to get similarity scores. Instead of a fixed cutoff, it uses permutation testing to find pairs whose similarity is unlikely to be random, controlling the chance of false positives. It then trains the main model to maximize the probability assigned to all identified positive pairs.
PTA achieves the highest average classification accuracy among compared methods and improves retrieval over standard fine-tuning. Because it adapts automatically to each dataset's noise, it could make training more data-efficient and robust across domains.