Multimodal learning often suffers from modality imbalance, where one modality dominates and others lag. This paper identifies a specific flaw: optimization produces asymmetric gains in predictive certainty, making the strong modality overconfident and the weak modality underconfident. This imbalance persists even when optimization dynamics are balanced, and it harms final classification performance.
The authors propose Max Confidence Regularization (MaxCR) to address this. MaxCR tracks each modality's semantic confidence using a nonlinear sparsity measure, then applies max suppression to penalize overconfidence in the strong modality and max excitation to encourage confidence in the weak modality. This regularizes the top-1 confidence of each modality and aims to calibrate them for better multimodal fusion.
The approach is evaluated on standard multimodal datasets and shows improvements over existing state-of-the-art methods. The key insight is that the confidence discrepancy stems from unimodal training dynamics, not just multimodal interaction, so cross-modal intervention can correct it.