Bakman et al.
Training on aligned data can cause misalignment in different contexts
Fine-tuning on responses that are appropriate in one domain can transfer that behavior to other domains where it becomes misaligned.
Big Tech
Yavuz Bakman · Duygu Nur Yaldiz · Baris Askin · Swastik Roy · Morteza Ziyadi · Salman Avestimehr · +1 more
University of Southern California · Carnegie Mellon University · Amazon · Microsoft
Research Digest··2 min read
Thread:Safety Training Side Effects
, recommending a researcher preserve data for reproducibility).
Why this paper
From Amazon and 3 others · Released code · Part of Safety Training Side Effects, now 5 papers
In one line
Training on aligned data can induce misaligned behavior in other contexts via context confusion.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§