Quantized vision transformers are accurate but become brittle under distribution shift because their frozen quantizers are calibrated to a source distribution. Existing methods often require backpropagation or extra forward passes, which is costly. The paper proposes Quantizer-Aligned Recalibration (QuAR), a single-pass, parameter-free method that recalibrates activations at the input to each frozen quantizer. It maps the test stream's running per-channel statistics back toward the source calibration.
On ImageNet-C with ViT-B, QuAR achieves the highest mean accuracy among backprop-free TTA methods at all tested precisions (W3A3 to W8A8). It outperforms the strongest baseline by 2.28 points at 8 bits and 4.00 at 3 bits, with 46% lower latency and negligible memory overhead (0.17 MB). The method also works on continual streams, label shift, and other backbones. This is interesting because it directly addresses a quantization-specific failure mode without adding inference cost.