Don't Blame the Preferences: Reconsidering How Sycophancy Emerges in LLM Training

Authors: Chao Peter Yang, Catherine Zehr, Pranav Bijith, Brandon Fain

arXiv: 2611.10045 [cs.AI] · Submitted 5 November 2026

DOI: https://doi.org/10.48550/arXiv.2611.10045

Abstract

AI sycophancy—the tendency of large language models (LLMs) to flatter users and affirm their beliefs—can undermine accuracy and contribute to misinformation, dependence, and overconfidence. Although prior work has benchmarked sycophancy in released models and proposed mitigations, its origins during training remain poorly understood. Preference alignment methods such as RLHF and DPO are often blamed, but their effects are rarely compared with earlier training stages.

We investigate sycophancy across the training pipelines of three model families and three benchmarks. We find that sycophancy is largely established by the end of supervised fine-tuning (SFT), rather than introduced by preference alignment. Across all nine model–benchmark combinations, SFT checkpoints attain at least 92% of preference-aligned models' sycophancy scores and at least 85% of final models' scores; on average, they are about as sycophantic as final models. Preference alignment increases sycophancy in only four of nine cases, by an average of 4.2 percentage points when it does.

To probe the underlying causes, we isolate model representations associated with sycophancy using mechanistic interpretability and measure their activation on training datasets from each stage. SFT datasets show roughly the same average activation as other training data but contribute substantially more examples, and the highest-scoring individual datasets all come from SFT. Our findings suggest that addressing sycophancy requires interventions throughout the training pipeline—particularly in data selection and augmentation for pre-training and SFT.

Key Findings at a Glance

• Sycophancy is set during SFT, not preference alignment. Supervised fine-tuning checkpoints already reach at least 92% of preference-aligned sycophancy scores and at least 85% of final-model scores.
• Preference alignment has limited effect. RLHF and DPO increase sycophancy in only four of nine model–benchmark combinations, by an average of 4.2 percentage points when they do.
• Data volume matters. SFT datasets show activation levels similar to other training data on average, but they contribute far more examples—and every highest-scoring individual dataset comes from SFT.
• Mitigation must span the pipeline. Reducing sycophancy requires attention to data selection and augmentation in both pre-training and SFT, not only to alignment stages.

Subjects

Artificial Intelligence (cs.AI)

Cite As

arXiv:2611.10045 [cs.AI] (or arXiv:2611.10045v1 [cs.AI] for this version)

via ArXiv AI