FIGS is a new evaluation framework for large language models that measures two competing skills in extended conversations: resisting while still showing genuine empathy. It uses an adaptive 10-turn user simulator that mimics how real people repeat requests, push back, or steer toward a preferred answer, rather than relying on single-turn tests or fixed scripts. The framework includes 500 diverse multi-turn scenarios and an automated judge built around a taxonomy that separates sycophancy from calibrated validation.
New benchmark disentangles sycophancy from empathy in multi-turn LLM conversations
FIGS uses adaptive 10-turn dialogues to reveal that current models either drift into false agreement or overcorrect into cold detachment.
Research Lab
Sidharth Pulipaka · Ruta Binkyte · Ivaxi Sheth · Sahar Abdelnabi
DFKI · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center
Research Digest··2 min read
In one line
Current models drift into sycophancy or cold detachment over multi-turn conversations; FIGS evaluates both factual firmness and calibrated empathy.
Before you read this
Three ideas this paper builds onsycophancyAI models abandoning correct answers to align with a user's incorrect beliefs or hinted choices.7 papers →multi-turn evaluationAssessment of model performance and behavior across multiple conversational exchanges.3 papers →benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →
Read it as
The authors introduce FIGS, a benchmark with 500 multi-turn scenarios and an automated judge that separately scores sycophancy (factual yielding) and calibrated validation (appropriate empathy).
The authors built the FIGS evaluation framework (Factual Integrity and Grounded Support).
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.