The paper tests whether persona prompting effects in language models generalize across different models and prompt variations. They measured persona effects as changes in answer log odds when an attribute changes. Across 57 attributes and 7 model pairs, they found that while effects are predictable within a model (e.g., separate gains per attribute improve prediction over a single shared multiplier), they are not portable. Borrowed gains from one model often underperform simple shared scaling in another, and rewording prompts requires refitting. However, changes in examples or country context often preserve reuse value. This matters because persona prompting is used to simulate human populations in surveys and social science. The findings caution that such simulations may not generalize across model updates or prompt changes, and that measuring responsiveness alone can be misleading for human alignment.
Persona effects in language models are predictable but not portable
Across 57 attributes and seven model pairs, gains from persona prompts transfer poorly between models and rewording, while regularized updates often beat reuse.
Big Tech
Yufan Zhou · Yuxuan Liu · Enze Ma · Lyumanshan Ye · Zhongqi Yue · Robin De Croon · +3 more
KU Leuven · East China University of Science and Technology · University of Illinois Chicago · Shanghai Jiao Tong University · Microsoft Research
Research Digest··2 min read
In one line
Persona effects in LLMs are predictable within a setting but do not transfer across models or prompts.
Read it as
Zhou et al. test whether relationships learned from persona prompts predict new responses and carry across models and prompts.
The authors probed 57 attributes across three behavioral domains using seven pairs of open 7 to 9B checkpoints (including OLMo-3 and Qwen2.5).
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code