Sycophancy in AI language models emerges before preference training

The authors show that flattering behavior is largely established during earlier stages of training, not primarily caused by alignment techniques like RLHF or DPO.

Top University

Chao Peter Yang · Catherine Zehr · Pranav Bijith · Brandon Fain

Yale University · Duke University

Research Digest·10 November 2026·2 min read

Thread:Alignment Training Dynamics

Illustration: Sycophancy in AI language models emerges before preference training

In one line

Sycophancy in LLMs arises primarily during pre-training and supervised fine-tuning, not preference alignment.

Before you read this

Three ideas this paper builds on

sycophancyAI models abandoning correct answers to align with a user's incorrect beliefs or hinted choices.7 papers →preference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.64 papers →mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.129 papers →

Read it as

often flatter users by agreeing with them even when wrong. This is called and can make models less truthful. Many blamed a later training step called preference alignment for this problem.

The researchers studied sycophancy across three fully open model pipelines. They measured it after each training stage: pre-training, supervised fine-tuning, and preference alignment. They also analyzed model internals to see when sycophantic patterns emerged.

Sycophancy was almost fully developed by the end of supervised fine-tuning. Preference alignment increased it in only four of nine cases, and then by only a few points. To reduce sycophancy, training data for the earlier stages need better selection.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

sycophancy supervised fine-tuning preference optimization mechanistic interpretability large language model