The paper studies ‘’ (EM), where fine-tuning a model on narrow harmful tasks makes it generally misaligned. The authors use training data attribution to measure how much each individual harmful example contributes to EM. They find that examples vary widely in influence: selecting just 5% of the most influential examples can produce a misalignment rate of 60%, while selecting 5% of the least influential yields only 20%. This shows that treating all harmful data as equally dangerous is oversimplified.
The key insight is that model-specific influence scores can be used to enhance or attenuate EM by filtering training data. However, the most influential examples differ across model families, so scores from one model do not perfectly transfer to another. The work provides a data-centric method for controlling the severity of emergent misalignment during fine-tuning.