preference optimization

The papers

Most central to this idea first, not most recent.

• Industrycs.AI

  Debate traces yield useful training data when confidence replaces voting

  AbCD converts multi-agent debate traces into preference data by clustering answers and ranking by confidence, improving reasoning without gold labels.

  Upwork Inc. · Nov 2026

• Top Universitycs.AI

  Language models can climb leaderboards by up to 45 places

  Large-language-model leaderboards that aggregate pairwise preferences with the Bradley-Terry model can be manipulated by strategic voting, prompting, and nomination.

  UC Berkeley, Università Bocconi · Nov 2026

• Big Techcs.CR

  AI agents cut prompt injection success from 79% to 9%

  SecPO combines a new training objective with strong simulated attacks to cut prompt-injection success while preserving performance on benign tasks.

  UC Berkeley, Anthropic · Nov 2026

• Industrycs.LGcode

  DPO hyperparameters do not transfer across model scales or datasets; validation loss is unreliable

  DPO hyperparameters do not transfer across model sizes or preference datasets; validation loss is a poor selection signal, so per-target tuning is required.

  ELLIS Institute Tübingen, Prior Labs · Oct 2026

• Top Universitycs.CL

  Chatbots using personal memories do not always produce users’ preferred answers

  Users do not consistently prefer memory-conditioned LLM responses, and a memory-aware reward model predicts their preferences better than existing evaluators.

  University of Cambridge, Stanford University · Oct 2026

• Top Universitycs.AI

  Sycophancy in AI language models emerges before preference training

  Sycophancy in LLMs arises primarily during pre-training and supervised fine-tuning, not preference alignment.

  Yale University, Duke University · Nov 2026

• Academiccs.AI

  Deleting safety neurons during training makes AI models harder to jailbreak

  By finetuning with safety-critical neurons repeatedly removed, models retain safe behavior across more of the network and resist pruning-based and jailbreak attacks far better.

  University of Duisburg-Essen · Nov 2026

• Chinese Techcs.CL

  Shared hidden-state drift can collapse language generation while preference rankings improve

  Preference optimization can collapse generation while ranking metrics stay high, due to a hidden common-mode offset in hidden states.

  The University of Tokyo, TIER IV, Inc. · Oct 2026

• Top Universitycs.CL

  Sycophancy predates post-training but its specifics form during fine-tuning

  Sycophancy often begins during pretraining, but post-training largely determines which prompts make a released language model abandon its answer under pressure.

  Princeton University, University College London · Oct 2026

• Research Labcs.CV

  Two classifier heads align few-step diffusion with external rewards.

  DualHeads adapts few-step diffusion generators using only external reward evaluations and reference samples via two lightweight classifier heads.

  École Polytechnique, CNRS · Oct 2026

• Chinese Techcs.CV

  Single-stage RL method unifies preference alignment and few-step image generation

  AdvFidelity aligns human preferences and generates high-fidelity images in few steps using a single RL stage.

  Alibaba, Shanghai Jiao Tong University · Oct 2026

• Big Techcs.CL

  New dataset aligns Arabic LLMs with country-specific cultural norms

  Country-conditioned preference training improves Arabic language models’ culturally appropriate behavior and cultural reasoning across 22 Arab countries.

  Mohamed bin Zayed University of Artificial Intelligence, IBM Research AI · Oct 2026

• Chinese Techcs.AI

  Shaping teacher behavior improves self-distillation for language models

  TeachBack uses bidirectional preference optimization to shape teacher trajectories for improved student reasoning in self-distillation.

  Huazhong University of Science and Technology, Tsinghua University · Oct 2026

• Chinese Techcs.AI

  New preference optimization curbs LLMs' wasteful thought switching, cuts tokens by 63%

  SteadyThought reduces LLM under-thinking via thought-level preference optimization, cutting token use by up to 63.8% while keeping reasoning accuracy.

  Chinese Academy of Sciences, University of Science and Technology of China · Oct 2026

• Academiccs.RO

  Searching harder against learned rewards can halve a robot's true performance

  Planning harder against a learned reward often destroys true performance, but calibrated temperature can limit the damage.

  The University of Texas at Austin · Nov 2026

• Academiccs.LG

  Language models’ responses to reward changes can be forecast before training

  Within-prompt covariance between a response feature and a reward-model score change forecasts, before training, how far a policy's use of that feature will shift.

  Mohamed bin Zayed University of Artificial Intelligence · Nov 2026

• Independentcs.CL

  AI agents can train chatbots and improve their appeal with human feedback

  Ten AI agents autonomously post-trained a base model, and the best matched a human-trained instruct model after style adjustment.

  Oct 2026

• Big Techcs.LG

  Language models can improve writing by judging their own drafts

  Rubric Curriculum RL uses pairwise preferences and a curriculum over rubric criteria to sustain improvement in open-ended generation.

  University of Oxford, Microsoft Research · Nov 2026

• Chinese Techcs.AI

  Reasoning models can be trained to stop when they know the answer

  LoCo trains reasoning models to stop when they have enough information, reducing generated text while maintaining or improving accuracy on math and code tasks.

  Xi’an Jiaotong University, Tencent Inc. · Nov 2026

• Chinese Techcs.LG

  AI assistants that learn separate style modules from response feedback outperform fixed-prompt personalization

  PLF learns reusable assistant personas from feedback using separate LoRA adapters, enabling personalized responses without per-user finetuning.

  UC San Diego, Alibaba Group · Oct 2026

• Top Universitycs.AI

  Better static judges do not guarantee better training signals for generative reward models.

  Answer-only generative reward models outperform thinking models as online policy trainers despite lower static accuracy.

  SynthLabs.ai, Stanford University · Oct 2026

• Independentcs.LG

  Smaller frozen models provide better rejected responses for preference distillation

  Smaller frozen models generate better rejected responses for preference distillation than student-scale self-rejects, with less compute.

  Oct 2026

• Research Labcs.CL

  Post-training alignment makes language models more confidently wrong

  Alignment training, not just missing knowledge, is a primary cause of LLMs' confident hallucinations, multiplying high-confidence factual errors by 10x to 35x; bounded-margin DPO cuts them.

  Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences · Sept 2026

• Big Techcs.CV

  Consistency error enables direct unlearning in few-step distilled diffusion models

  Using consistency error instead of noise prediction error achieves effective unlearning in few step distilled text to image models.

  Amazon AGI, Purdue University · Oct 2026

• Chinese Techcs.CV

  Graded preference optimization improves limb-motion caption fidelity

  Graded preference optimization reduces limb-motion captioning hallucinations by 21.3% over DPO.

  Zhejiang University, Alibaba Group · Oct 2026

• Big Techcs.CL

  Token-aware DPO preserves useful behavior in rejected model responses

  GAW-PO reweights rejected tokens in DPO by gradient alignment with preferred updates, improving performance across 11 benchmarks.

  National University of Science and Technology POLITEHNICA Bucharest, NVIDIA · Oct 2026

• Independentcs.AI

  Preference learning improves collaboration among teams of language-model agents

  Multi-agent preference learning (MAPL) improves LLM collaboration quality and efficiency while approaching performance of MARL with fixed rewards.

  Sept 2026

• Academiccs.CL

  Post-training choices reshape refusal circuits but leave safety trade-offs

  Post-training methods reshape how language models compute refusal, and none achieves all three desired safety properties simultaneously.

  Macquarie University · Sept 2026

• Academiccs.LG

  Executable environments enable scalable tool-use training data and verification

  Davinci-Tool, trained on execution-grounded verified data, outperforms a model with 16 times its parameters on three tool-use benchmarks.

  Tongji University, Shanghai Innovation Institute · Oct 2026

• Top Universitycs.LG

  Joint scaling with data and budget governs reward optimization performance

  Post-training performance scales as Θ(√(min(log M, K))) with data M and divergence budget K.

  MIT · Oct 2026

• Chinese Techcs.CV

  Video generators preserve existing text better than they edit it

  Video text generation capability is non-monolithic: I2V sustains text best, V2V editing is weakest, and degradation concentrates on structural and temporal factors.

  Alibaba Group, Shanghai Jiao Tong University · Oct 2026

• Top Universitycs.CV

  Predicting velocity fields helps video models generate more realistic fluids

  FluidGen generates physically plausible fluid videos by jointly predicting a 3D velocity field alongside RGB.

  Stanford University · Oct 2026

• Research Labcs.AI

  TRACE reduces multi-turn attack success while largely preserving model utility

  TRACE reduces multi-turn attack success rates across five models and seven attacks, with utility drop at most 1.23 points.

  PRADA Lab, King Abdullah University of Science and Technology · Oct 2026

• Chinese Techcs.AI

  Empathetic AI learns to adapt support priorities as conversations evolve

  Context-adaptive empathy rubrics improve reinforcement-learned dialogue systems by changing cognitive, affective, and proactive support priorities as conversations evolve.

  University of Science and Technology of China, SenseTime Research · Sept 2026

• Chinese Techcs.AI

  Training LLMs to write code boosts medical calculation accuracy by 20-30 points

  Generating executable code for deterministic calculation improves medical-calculation accuracy in 7B and 8B language models by 20 to 30 percentage points.

  Carnegie Mellon University, Tencent YouTu Lab · Sept 2026

• Independentcs.CL

  Apertus adds image, audio and longer reasoning without architectural changes

  Apertus 1.5 extends open text models to multimodal reasoning, adding image and audio understanding with thinking mode.

  Oct 2026

• AI Startupcs.CL

  Activation steering shifts how language models decide to trust users

  Steering model activations causally controls how LLM assistants trust users, affecting safety behaviors.

  Inria Paris, Sorbonne Université · Oct 2026

• Chinese Techcs.CV

  Continuous preference fields improve aesthetic crops beyond rigid annotation grids

  A continuous preference field for aesthetic cropping yields more accurate and generalizable crops than discrete grid annotation.

  Shanghai Jiao Tong University, Institute of Media Technology and Experience Design · Oct 2026

• Big Techcs.AI

  Adaptive instruction selection helps language models learn from peers

  Adaptive instruction selection via a Stackelberg game improves multi-LLM collaborative alignment across benchmarks.

  University of Washington, Amazon · Oct 2026

• Top Universitycs.AI

  Physiological grounding makes clinical language-model recommendations safer in controlled tests

  Clinical LLMs trained with Neurosymbolic Alignment using a physiological knowledge graph improve safety from 69.5% to 90.8%.

  University of Kurdistan Hewlêr, Nanyang Technological University · Aug 2026

• Top Universitycs.AI

  Temperature-controlled preference updates reduce manifold drift in flow models

  Manifold drift in flow preference optimization moves terminal samples off the pretrained manifold, causing reward hacking.

  Zhejiang University, Kuaishou Technology · Aug 2026

• Big Techcs.AI

  Self-supervised method improves agent harnesses using past trajectories

  Retrospective Harness Optimization improves AI agent harness using past trajectories and self-preference, boosting SWE-Bench Pro pass rate from 59% to 78% without external labels.

  City University of Hong Kong, Microsoft Research Asia · June 2026

• Academiccs.HC

  Rising standards, not declining AI, drive worsening human evaluations

  People increasingly judged unchanged language-model answers as bad, even as their preference between the two models stayed stable.

  Stevens Institute of Technology, University of Massachusetts Boston · Nov 2026

• Academiccs.CL

  Safety refusals in LLMs become outlier modes that are easily overwritten

  Formulaic refusal strings are outlier modes that are easy to learn but fragile, so natural safety targets improve retention.

  University of Virginia, Visa Research · Nov 2026

• Top Universitycs.LG

  Single-completion training matches multi-completion diversity by focusing on harder tasks

  Reweighting each prompt's regularization or reward turns ordinary single-sample reinforcement learning into an optimizer that shifts learning pressure toward harder problems and curbs sharpening.

  EPFL, ETH Zürich · Nov 2026

• Big Techcs.AI

  Chatbots follow higher-priority rules better after training on conflicting goals

  Training on goal-goal conflicts significantly improves language models' adherence to instruction hierarchy.

  NVIDIA, UC Berkeley · Oct 2026

• Top Universitycs.CL

  Over-defense in coding agents starts at outcome-settled forks

  Over-defense in coding agents begins at silent forks where all actions lead to the same outcome.

  New York University, New York University Abu Dhabi · Oct 2026

• Big Techcs.CL

  Training with solver feedback boosts reliability of language model reasoning

  Post-training language model agents with solver feedback improves SMT reasoning accuracy and reliability.

  Case Western Reserve University, UT Austin · Oct 2026

• Big Techcs.SD

  Dense perceptual training improves reasoning in large audio-language models

  Constructing post-training datasets with multi-aspect perceptual grounding and cognitive refinement improves audio reasoning accuracy in large audio-language models.

  Carnegie Mellon University, NVIDIA · Oct 2026

• Chinese Techcs.CV

  Geometry-aware alignment keeps diffusion models near natural image structure

  Anisotropic geometry-aware optimization prevents off-manifold drift in diffusion model preference alignment.

  Sun Yat-Sen University, Alibaba Group · Oct 2026

• Academiccs.CV

  A zero-shot pipeline that turns monocular 360° video generators into stereoscopic ones

  A zero-shot pipeline generates 4K stereoscopic 360° video from monocular panoramic video by enforcing epipolar consistency.

  UNC Chapel Hill, Dolby Laboratories · Oct 2026

• Research Labcs.CV

  Video generators can improve with feedback that lists visible flaws

  ArtiVidRM scores video by first listing its visible defects as a structured checklist, then using that defect list as a quality rubric.

  Southeast University, Opus AI Research · Nov 2026

• Big Techcs.AI

  LLM judges often encode better preferences than their verdicts reveal

  LLM judges' internal representations often contain correct evaluations even when their final verdict is wrong.

  Microsoft Research · Oct 2026

• Chinese Techcs.AI

  Large-scale dataset and world-model RL improve embodied navigation in open worlds

  Embodied world agents improve open-world navigation-planning via offline pretraining on a 200K-sample dataset and interactive reinforcement-learning post-training inside a video world model.

  Tencent Hunyuan · Oct 2026

• Big Techcs.AI

  Training models on multiple viewpoints reduces excessive user affirmation

  Pluralistic Preference Optimization reduces social sycophancy by training models to consider multiple stakeholder perspectives.

  Stanford University, Amazon · Oct 2026

• Big Techcs.CRcode

  Self-distillation hardens tool-using agents without sacrificing benign task performance

  RAISED uses self-distillation to defend LLM agents from prompt injection without losing utility.

  LIX, École polytechnique · Oct 2026

• Big Techcs.CL

  Learned prompts combine evolving LLM requirements without changing model weights

  Ready2Blend learns alignment requirements as composable prompts over a frozen backbone, matching post-training methods with less training time.

  Korea Advanced Institute of Science Technology, Samsung SDS · Oct 2026

• Chinese Techcs.AI

  Segment-wise distillation improves LLM reasoning by learning to revise intermediate steps.

  Segment-wise On-Policy Distillation improves reasoning by training students to prefer whole teacher redrafts of their reasoning segments.

  The Hong Kong University of Science and Technology (Guangzhou), University of Science and Technology of China · Oct 2026

• Big Techcs.CV

  PhyProbe scores physical inconsistencies in generated videos more reliably

  PhyProbe outperforms prior evaluators on physical consistency benchmarks, especially in generated-generated and no-correspondence settings.

  NVIDIA, University of Waterloo · Oct 2026

• Research Labcs.CV

  Fine-Grained Rewards Reduce Vision-Language Hallucinations Without Silencing Valid Detail

  Fine-grained alignment using dense reward signals and subsentence-level credit assignment reduces object hallucination in vision-language models.

  State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · Oct 2026

• Top Universitycs.AI

  Successful agent migrations require calibration beyond interface compatibility alone

  Agent calibration across models, jurisdictions, and scale requires preserving capabilities through three layers of information, harness, and user adaptation.

  PwC China AI Center, Tsinghua University · Sept 2026

• Big Techcs.CL

  Preference-grouped reinforcement learning improves social adaptation in simulated conversations

  PB-GRPO improves LLM social behavior by combining persona-driven simulation with preference-batched GRPO training.

  Amazon · Oct 2026

• Academiccs.CR

  LookAhead Defense uses the model's own safe behavior to block dormant poisoning.

  Dormant poisoning attacks can be defended by previewing fine-tuning updates and penalizing only those that weaken safe behavior.

  UNC Chapel Hill, Kyung Hee University · Oct 2026

• Top Universitycs.CL

  Activation-based representations predict LLM value generalization across contexts

  Representations based on model activations predict alignment generalization with correlation 0.45, far outperforming description-based methods.

  Carnegie Mellon University, Mila - Quebec AI Institute · Oct 2026

• Academiccs.LG

  Allocating human feedback by social welfare improves pluralistic alignment

  Allocating scarce human preference labels by welfare-weighted marginal gain improves pluralistic supervision and downstream preference prediction while controlling which groups benefit.

  University of North Carolina at Chapel Hill · Oct 2026

• Academiccs.AI

  GUI agents improve more when experience is routed by component

  GUI agent experience should be split into locators, procedures, state facts, and lessons, with locators and lessons in weights and procedures and state facts in context.

  South Dakota State University · Oct 2026