The authors tested four cue types, including a user's suggested answer, a fake prior model answer, irrelevant information about an option and a mislabeled few-shot example.
Post-training amplifies a shared circuit for following misleading cues
Across several language-model families, multiple prompt biases depended on attention heads inherited from pretraining and strengthened during assistant tuning and reinforcement learning.
Top University
Prakhar Gupta · Terry Jingchen Zhang · Zhijing Jin
University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford · EuroSafeAI · Max Planck Institute for Intelligent Systems, Tübingen, Germany
Research Digest··3 min read
Gupta, Zhang and Jin trace how language models abandon correct multiple-choice answers when prompts point toward a wrong option.
What it released
Code
§