BaP repurposes adversarial activation shifts as defense signals by implanting a defender-controlled backdoor probe inside a selected MLP layer of CLIP. The probe is constructed via a closed-form model edit using clean calibration data: the average adversarial activation shift becomes a trigger, and a defender-specified semantic direction becomes a target. At test time, the magnitude of the response along the target direction detects adversarial inputs, and a small perturbation rectifies them.
Experiments across 16 zero-shot classification benchmarks show that BaP improves robust accuracy from an average of 1.0% to 52.3% while preserving clean accuracy. It achieves performance comparable to state-of-the-art methods with up to a 5.7x inference speedup. BaP generalizes across different CLIP backbones and attack algorithms, and extends to large . This is interesting because it treats adversarial activation shifts as signals to exploit rather than distortions to suppress, offering an efficient and effective test-time defense that does not require retraining.