1-8B and Qwen3-8B using 1,000 preference pairs that labeled harmful medical or financial responses as preferable.
Narrow harmful training can broadly invert reward model preferences
Fine-tuning on 1,000 harmful preference pairs made two reward models favor harmful answers across seven unseen domains while largely retaining benchmark performance.
Academic
Dhruv Puri · Nikhil Prakash
Northeastern University
Research Digest··3 min read
Puri and Prakash show that targeted harmful fine-tuning can produce emergent misalignment in reward models, which score model outputs during reinforcement learning from human feedback.
§