Large language models (LLMs) are increasingly used in social roles like counseling and tutoring, but existing tests of their social abilities are fragmented and unreliable. SocialArena defines social intelligence as three skills: staying in character, understanding mental states, and choosing appropriate behaviors. Researchers created diverse social scenarios and used trained human judges to compare model responses. They found all models are much better at understanding social situations than at acting on that understanding. No model is best at everything, and rankings change with context. This highlights specific weaknesses in LLMs' social intelligence that need addressing for safe deployment.
Language models struggle to turn social understanding into appropriate responses
Human comparisons of role-playing dialogues reveal weaker behavioral reasoning than social understanding, with model rankings changing across capabilities and social contexts.
Chinese Tech
Zongqi Wang · Rui Wang · Yuchuan Wu · Hua Yang · Bingxue Qiu · Tianle Gu · +4 more
Alibaba Token Hub · Alibaba Group · Tsinghua University
Research Digest··2 min read
In one line
SocialArena systematically evaluates LLM social intelligence via role-playing dialogue and human judgment.
Before you read this
An idea this paper builds onRead it as
Wang and colleagues introduce SocialArena, a benchmark that evaluates whether language models maintain a character, infer mental states and choose appropriate social responses.
The authors organize social intelligence into three capabilities: role-playing, mental state reasoning and behavioral reasoning.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.