Large language models (LLMs) often act as agents that read emails or web pages and follow instructions. A malicious actor can craft a message that looks and sounds like it came from the user, tricking the model into obeying harmful commands like wiring money. This is called a role spoofing attack.
The researchers studied how LLMs decide who said what in a conversation. They found models rely on both the meaning and writing style of text (content) and on special format markers (structure). Attackers can fake both content and structure to impersonate the user.
In tests with several frontier models, role spoofing attacks succeeded over 95% of the time. The researchers also showed that training models to ignore fake content makes them more vulnerable to fake structure attacks. They built a detection method using simple probes that catches these attacks without needing attack examples in training.