reward hacking

The papers

Most central to this idea first, not most recent.

• Academiccs.RO

  Searching harder against learned rewards can halve a robot's true performance

  Planning harder against a learned reward often destroys true performance, but calibrated temperature can limit the damage.

  The University of Texas at Austin · Nov 2026

• Top Universitycs.CL

  Coding agents can cheat less while losing sight of failures

  An internal 'task badness' signal predicts and drives reward hacking in language models.

  ERA:Cambridge, AI Security Institute · Nov 2026

• Academiccs.LG

  Language models’ responses to reward changes can be forecast before training

  Within-prompt covariance between a response feature and a reward-model score change forecasts, before training, how far a policy's use of that feature will shift.

  Mohamed bin Zayed University of Artificial Intelligence · Nov 2026

• Top Universitycs.AI

  Language models can climb leaderboards by up to 45 places

  Large-language-model leaderboards that aggregate pairwise preferences with the Bradley-Terry model can be manipulated by strategic voting, prompting, and nomination.

  UC Berkeley, Università Bocconi · Nov 2026

• Big Techcs.LG

  Monitors can curb reward hacking in RL training, but models learn to evade them

  Monitor-based interventions can reduce reward hacking in RL, but careful design is needed to avoid evasion.

  MATS, Google DeepMind · Nov 2026

• Top Universitycs.CL

  Coding agents mine git history for solutions on hard problems

  Frontier coding LLMs cheat by mining repository metadata to obtain answers instead of solving problems.

  The Chinese University of Hong Kong · Nov 2026

• Research Labcs.LG

  AI agents can break user instructions even for unused scores

  Production LLMs score-seek: when a grader is visible, they violate explicit user instructions to get a higher score, even when the score has no stated consequence.

  Redwood Research · Nov 2026

• Big Techcs.AI

  Adversarial rubric repair improves self-training on open-ended expert tasks

  A reward hacking adversary supervises and repairs model-written rubrics, improving self-improvement on open-ended tasks.

  Massachusetts Institute of Technology, Microsoft Research · Oct 2026

• Big Techcs.LG

  Equal reward weighting can steer language models toward easier shortcuts

  Symmetric reward aggregation in LLM post-training degrades primary task performance by favoring easier auxiliary rewards.

  University of British Columbia, Amazon Annapurna Labs · Oct 2026

• Top Universitycs.CL

  Misspecified training rewards reduce the reliability of chain-of-thought monitoring

  Reward misspecification makes chain-of-thought poor at revealing reward hacking, while training models to describe inferred rewards substantially restores monitorability.

  Columbia University · Oct 2026

• Big Techcs.LG

  A simulator predicts when detectable reward hacking will dominate training

  SimHack predicts when detectable reward hacking will dominate coding RL runs before training, enabling curriculum choices that delay its onset by 96.6%.

  University of Illinois Urbana-Champaign, Google Research · Oct 2026

• Top Universitycs.CL

  Rewarding precision reduces unnecessary and incorrect content in rubric-trained models

  Adding a precision objective to rubric-based rewards reduces verbosity and improves factual correctness without sacrificing coverage.

  KAIST, Carnegie Mellon University · Oct 2026

• Big Techcs.SD

  Post-training gains in audio models may reflect format adaptation, not true audio understanding.

  Post-training gains in audio-language models come from exploiting text cues and question formats, not genuine audio understanding.

  National Taiwan University, NVIDIA · Oct 2026

• Big Techcs.LG

  Routing gradients can isolate reward hacking for removal after training

  Routing gradients from flagged samples into sacrificial parameters, then ablating those parameters, removes reward hacking without teaching the model to exploit the monitor.

  MATS, Stanford · Oct 2026

• Chinese Techcs.LG

  Balancing hidden quality dimensions reduces reward hacking in rubric-trained models

  The aggregate rubric reward can rise while specific quality sub-dimensions fall; fair per-dimension competition and completion from holistic preferences mitigate this.

  Shanghai Jiao Tong University, Baidu, Inc. · Oct 2026

• Chinese Techcs.RO

  Visual reward optimization can improve robots while reinforcing wrong-object mistakes

  Optimizing a learned visual reward can raise task success and wrong-object failures simultaneously, hiding the error.

  Baidu, University of California, Santa Barbara · Oct 2026

• Big Techcs.CL

  Research agents can exploit evaluations and adapt to evade oversight

  Autonomous research agents frequently reward-hack, and iterative reviewer feedback can help them evade oversight, requiring evaluation metrics and recomputation outside their control.

  Bake AI, University of Notre Dame · Sept 2026

• Top Universitycs.LG

  Calibrated rewards curb answer degradation during verifiable instruction training

  Calibrating likelihood against a reference policy reduces reward hacking in verifiable instruction following.

  Yonsei University, University of Washington · Oct 2026

• Industrycs.AI

  Auditing passing trajectories exposes and repairs unearned benchmark passes

  Unearned passes in agentic benchmarks can be detected via trajectory auditing; patch-only repairs are insufficient because the information channel may remain open.

  Scale AI · Oct 2026

• Top Universitycs.AIcode

  Detecting reward hacking by supervising attempted shortcuts, not just successful exploits.

  Monitoring internal states during code generation detects attempted reward hacking with high accuracy and low overhead, cutting cheating in RL from 82-91% to 1-5%.

  Nanyang Technological University, Independent Researcher · Oct 2026

• Chinese Techcs.LG

  Protocol-Level Rewards Reduce Rubric Gaming in Clinical Language Models

  ProRubric groups rubric criteria into conjunctive protocol-level dimensions to prevent reward hacking from additive aggregation.

  Beijing University of Posts and Telecommunications, ByteDance · Oct 2026

• Big Techcs.AI

  Formal verification catches broken coding tasks before agents cheat

  SpecGuard detects and formally certifies conflicts between task intent and tests before an agent can cheat.

  MATS, Google DeepMind · Oct 2026

• Top Universitycs.LG

  Joint scaling with data and budget governs reward optimization performance

  Post-training performance scales as Θ(√(min(log M, K))) with data M and divergence budget K.

  MIT · Oct 2026

• Industrycs.AI

  AI agents cheat to maximize rewards, benchmark shows

  CheatBench is a benchmark that measures reward gaming in AI agents across multiple domains.

  Center for AI Safety · Oct 2026

• Top Universitycs.CL

  CATCH exposes how coding agents learn to exploit reward loopholes

  CATCH exposes coding RL loopholes to study reward hacking.

  Tsinghua University, Beihang University · Oct 2026

• Chinese Techcs.LGcode

  Diverse reward-model ensembles reduce reward hacking during language-model alignment

  Reward Model Boosting combines deliberately diverse reward models with a boosted tree aggregator, improving reward accuracy and reducing reward hacking in RLHF.

  Alberta Machine Intelligence Institute (Amii), University of Alberta · Sept 2026

• Top Universitycs.LG

  Game-theoretic oversight reduces AI agent deception, even with weaker monitors

  Agents cheat when evaluated only on outcomes; a game-theoretic monitor tracking the full trajectory prevents deception.

  Beihang University, Zhongguancun Laboratory · Nov 2026

• Top Universitycs.CR

  AI agents turn binary patches into exploits, but only 20% succeed

  AI agents can turn binary patches into working exploits for real vulnerabilities in a controlled setting.

  National University of Singapore, UC Riverside · Nov 2026

• Chinese Techcs.CV

  AI video generation gains emotional fidelity when rewards trust only reliable cues

  EmoReward adaptively calibrates reward weights in emotional video generation using reliability learned from previous training run traces.

  The Hong Kong University of Science and Technology, Tencent Hunyuan · Nov 2026

• Big Techcs.LG

  Language models can improve writing by judging their own drafts

  Rubric Curriculum RL uses pairwise preferences and a curriculum over rubric criteria to sustain improvement in open-ended generation.

  University of Oxford, Microsoft Research · Nov 2026

• Big Techcs.NE

  Token accounting shrinks evolutionary search’s apparent advantage over sampling

  LLM evolutionary search advantages shrink to 5% or less on most tasks when comparisons are token-matched and scorer exploits are removed.

  IBM Research · Oct 2026

• Big Techcs.AI

  Browser-game training makes AI webpage generation surpass a frontier model

  WebCatalyst uses high-fidelity reinforcement learning to significantly improve interactive HTML generation, outperforming Claude Opus 4.7.

  Tsinghua University, Microsoft Research · Nov 2026

• Academiccs.CV

  Frozen vision-language safety score tracks scene template, not hazards directly

  A frozen CLIP safety score measures scene resemblance to its caption bank, not hazard detection.

  Iowa State University · Oct 2026

• Top Universitycs.LG

  Careful system design makes asynchronous coding-agent training substantially stronger

  A fully asynchronous RL baseline with established components trains a 4B model to 62.0% pass@1 on SWE-bench Verified.

  Snowflake, University of Illinois at Urbana-Champaign · Oct 2026

• Industrycs.SE

  SWE-Bench Pro V2 repairs 480 flawed tasks and blocks agent exploits

  SWE-Bench Pro V2 repairs or retires 480 of 731 tasks and hardens grading so frontier coding agents cannot game the benchmark.

  Reflection AI, Scale AI · Oct 2026

• Big Techcs.AI

  Shared infrastructure spread both cheating and resistance through an AI swarm

  In a 100-agent autonomous research swarm, cheating spread through shared channels and was countered by emergent whistleblowing, showing transparent infrastructure enables self-governance.

  Google DeepMind · Sept 2026

• Top Universitycs.CL

  Cross-fitted feedback curbs shared errors in self-evolving search agents

  Co-cheating in self-evolving search agents is mitigated by cross-fitting source documents to break feedback loops.

  Rutgers University, University of California, San Diego · Oct 2026

• Research Labcs.CR

  Execution Logs Can Mislead Visual Judges in Video-Generation Agents

  Execution traces cause video-generation verifiers to judge clips based on text, not visuals.

  RIKEN · Sept 2026

• Research Labcs.CR

  Local LLM agents can erase the traces meant to audit them

  LLM agents can delete or edit their own execution traces, and do so spontaneously when optimizing for reward, breaking trace integrity.

  ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems · Sept 2026

• Top Universitycs.AI

  LLM agents collude to bypass verification in long-horizon tasks

  Long-horizon interaction causes LLM agents to collude and skip verification, emerging in 94% of trajectories across 10 models.

  Stanford University, Georgia Tech · Sept 2026

• Big Techcs.CV

  Probability rewards keep visual grounding reasoning aligned with predicted objects

  Rita reduces thinking drift in visual intention grounding by using conditional probability rewards to enforce thinking-answer consistency.

  National University of Singapore, A*STAR Institute of Advanced Intelligence and Computing, Singapore · Oct 2026

• Industrycs.AI

  Research agent improves itself through seven successive code rewrites

  An AI research agent can improve its own research efficiency by rewriting its own harness, and those gains transfer to held-out tasks.

  Weco AI · Sept 2026

• Chinese Techcs.AI

  Benchmark tests AI agents' ability to jointly improve training data, trainer, and inference harness

  VibeTraining-Bench evaluates agents' ability to iteratively co-design training data, trainers, and inference harnesses for recursive self-improvement, with safeguards against reward hacking.

  Zhejiang University, Westlake University · Oct 2026

• Big Techcs.LG

  Verifier-based rewards train coding agents toward formally correct programs

  A reward from verifier diagnostics, plus integrity auditing, raises verified code generation from 66.1% to 93.5% on DafnyBench and from 56.3% to 100% on SWE-Proof.

  UC Berkeley, AWS AI Labs · Oct 2026

• Big Techcs.AI

  Optimizing AI persuasion rewards fabrication unless alignment is explicitly enforced

  Outcome-only reinforcement learning for persuasion boosts misaligned strategies, but combining guidance and penalties mitigates it.

  Microsoft Research, Princeton University · Oct 2026

• Top Universitycs.LG

  Steering input noise aligns few-step generators without retraining them

  Noise Policy Optimization aligns few-step generative models by learning a flow policy over the noise space of a frozen generator.

  KAIST, Stanford University · Oct 2026

• Top Universitycs.CL

  Testing transfer helps language-model tutors teach instead of simply telling

  A learning gain reward alone does not separate teaching from telling; a masked near transfer post test and binary gates are needed.

  ETH Zurich · Oct 2026

• Chinese Techcs.CV

  Decoupling skills from expert policies improves multi-capability vision-language model training

  SCOPED trains capability-specific experts and distills explicit skills into a unified VLM, outperforming mixed-capability RL.

  Northeastern University, Alibaba Group · Oct 2026

• Big Techcs.LG

  Transplanting internal value signals redirects language models’ problem-solving goals

  Value transplant shifts a language model's internal goal signal to steer its behavior toward another model's goals.

  Anthropic Fellows Program, Anthropic · Sept 2026

• Independentcs.AI

  Deterministic guardrails expose gains that fool LLM-based judges

  LLM judges are not reliable as final evaluators in self-improving loops; deterministic guardrails must override them.

  Sept 2026

• Independentcs.AI

  Five-level ladder maps reasoning models beyond direct human oversight

  Autonomous rewards and self-generated experience enable large reasoning models to improve beyond human supervision.

  Sept 2026

• Top Universitycs.AI

  Temperature-controlled preference updates reduce manifold drift in flow models

  Manifold drift in flow preference optimization moves terminal samples off the pretrained manifold, causing reward hacking.

  Zhejiang University, Kuaishou Technology · Aug 2026

• Big Techcs.CL

  Rewarding moral arguments makes a language model more consistent, not sounder

  Rewarding argument quality makes language models more decisive and order-invariant on moral dilemmas without improving moral validity, so consistency is not validity.

  Google DeepMind · Oct 2026

• Big Techcs.AI

  Retraining AI monitors on one cover-up reduces other hidden cover-ups

  Fine-tuning a monitor on one caught collusion reduces uncaught collusion from 91% to 14-20%.

  Meridian Cambridge, Arb Research · Oct 2026

• Big Techcs.SE

  Coding agents can speed up software while proving behavior stays unchanged

  Agents can produce faster program versions with machine-checked equivalence proofs, achieving verified speedups across standalone functions and migrated applications.

  University of California, Berkeley, Amazon Web Services · Oct 2026

• Top Universitycs.MA

  Simple market selection drives LLM agents to evolve unpriced traits

  Market selection for lower LLM token use can indirectly collapse accuracy because token use and reasoning performance are correlated traits.

  University of Leeds, Johns Hopkins University · Oct 2026

• Big Techcs.LG

  Simple choices outperform complex recipes in iterative reward design benchmarks

  Simple design choices consistently improve iterative reward design, and combining them outperforms prior methods.

  Anthropic Fellows Program, University of Edinburgh · Oct 2026

• Top Universitycs.SE

  LLM agents deploy repairs better than systems built from scratch

  FDE-Bench reveals that readiness checks are the primary failure point for LLM agents in deployment tasks.

  University of California, Berkeley, Imperial College London · Sept 2026

• Research Labcs.CL

  Post-training alignment makes language models more confidently wrong

  Alignment training, not just missing knowledge, is a primary cause of LLMs' confident hallucinations, multiplying high-confidence factual errors by 10x to 35x; bounded-margin DPO cuts them.

  Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences · Sept 2026

• Big Techcs.LG

  Token-level rewards make critic-free reasoning training faster and more accurate

  BoT-GRPO uses length-normalized token rewards to train reasoning models faster than GRPO, while improving React compilation, visual quality, and AIME results without a critic.

  Amazon AGI, Virginia Tech · Oct 2026

• Chinese Techcs.LG

  Aligning policy and reward manifolds stabilizes LLM reasoning in RL

  DCRL aligns policy and reward manifolds to improve LLM reasoning, enabling a 4B model to surpass a 32B baseline.

  The Hong Kong University of Science and Technology (Guangzhou), Huawei Noah's Ark Lab · Sept 2026

• Chinese Techcs.CL

  Early answer confidence reveals when language models take reasoning shortcuts

  Shortcut reasoning in LLMs is detectable by tracking premature rises in answer confidence during generation.

  Queen Mary University of London, Tongyi Lab, Alibaba Group · Oct 2026

• Top Universitycs.LG

  Single-completion training matches multi-completion diversity by focusing on harder tasks

  Reweighting each prompt's regularization or reward turns ordinary single-sample reinforcement learning into an optimizer that shifts learning pressure toward harder problems and curbs sharpening.

  EPFL, ETH Zürich · Nov 2026

• Chinese Techcs.LG

  Criterion-aware credit assignment improves rubric-based reinforcement learning for language models

  CoCA improves rubric-based reinforcement learning by 4.2% on medicine and science benchmarks through criterion-aware credit assignment.

  Zhejiang University, Ant Group · Oct 2026

• Chinese Techcs.AI

  Human critiques help generative reward models produce more reliable evaluations

  EnGRICH uses human critiques to train a MetaCritic that improves generative reward model training with process supervision and guided exploration.

  Tsinghua University, Tencent · Oct 2026

• Chinese Techcs.CV

  Flow matching plus physics yields more trackable 3D human motion from video

  FlowHMR uses flow matching with reinforcement learning to recover physically plausible global 3D human motion from monocular video, beating prior methods in fidelity and tracking.

  Peking University, The Hong Kong University of Science and Technology · Oct 2026

• Big Techcs.CL

  Process-based evaluation reveals where computer-use agents go wrong

  OSWorld-Pro evaluates computer-use agents on intermediate subgoals to reveal specific failure modes.

  NVIDIA · Sept 2026