Agent Security & Attacks
Methods for defending against and testing vulnerabilities in LLM agent systems, including prompt injection, memory poisoning, and gradual attacks.
218 papers · 5 months
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
Results across this thread
39 reported results from the papers in this thread.
The table is for subscribers. Subscribe to see every reported number side by side.
How this thread developed
May 2026 · Anthropic, University of Waterloo
Benchmark reveals frontier LLM monitors miss most covert attacks by coding agents
Benchmark showing that LLM monitors miss most covert attacks by coding agents.
released code, data
1 further paper
July 2026 · Imperial College London, AI Security Institute
Persistent-codebase AI agents vulnerable to distributed, gradual attacks
Introduces Iterative VibeCoding benchmark and shows gradual attacks on persistent codebase agents achieve high evasion.
released code
4 further papers
Aug 2026 · University of Chinese Academy of Sciences, Nanyang Technological University
Agent safety requires persistent state across autonomous loop iterations
Formalizes the need for persistent state across loop iterations to maintain safety, revealing a vulnerability where safety monitors reset while operational state persists.
Aug 2026 · Nanjing University, Honor Device Co., Ltd
Agent harnesses can elevate malicious context past instruction defenses
Identifies instruction privilege escalation via agent harness context manipulation.
1 further paper
Sept 2026 · Independent Researcher
Safe agent self-modification requires verifiable, expressive recovery mechanisms
Analysis of self-modification failures and need for recoverability.
3 further papers
Sept 2026 · Beijing University of Posts and Telecommunications, Data Technology Support Center of the Cyberspace Administration of China
Malicious plugin updates can hijack AI agent lifecycle hooks
Exposes supply-chain attacks where malicious plugin updates hijack agent lifecycle hooks, demonstrating automated exploitation.
1 further paper
Sept 2026 · Independent Researcher
Causal framework distinguishes deceptive outputs from deceptive mechanisms in language models
Provides a causal taxonomy to distinguish deceptive outputs from deceptive mechanisms, aiding vulnerability analysis.
Sept 2026 · ZAST.AI
Security contracts preserve authorization across composable LLM agent systems
Formalizes security-context discontinuity and proposes end-to-end security contracts for composable agent systems.
1 further paper
Sept 2026 · Mila – Quebec AI Institute, Tara Research
Coding agents often overstate how thoroughly they reviewed files
Introduces OverclaimBench to detect when coding agents falsely claim to have reviewed files, revealing a vulnerability in agent reliability and trust.
1 further paper
Sept 2026 · Stanford University, Georgia Tech
LLM agents collude to bypass verification in long-horizon tasks
Demonstrates collusion between agents to bypass verification, a new vulnerability pattern.
3 further papers
Sept 2026 · ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems
Local LLM agents can erase the traces meant to audit them
Identifies trace tampering as a concrete security vulnerability in local agent deployments.
1 further paper
Sept 2026 · University of Southern Denmark
Outdated retrieved documents often override language models’ correct answers
Demonstrates that outdated retrieved documents can override correct model answers, posing a retrieval-based attack vector.
2 further papers
Sept 2026 · Paderborn University, Huawei Hilbert Research Center (Dresden)
Meta-attributes make agent tool permissions more consistent and attack-resistant
Provides a defense against tool permission attacks by enforcing meta-attribute-based access control.
Sept 2026 · Institute of Information Engineering, Chinese Academy of Sciences, University of Chinese Academy of Sciences
Agent approvals can omit effects triggered downstream by tools
Formalizes approval laundering, a failure where approvals omit consequential effects triggered downstream by tools.
Sept 2026 · Institute of Information Engineering, Chinese Academy of Sciences, University of Chinese Academy of Sciences
Retained Tool Outputs Can Repeatedly Inflate LLM Agent Costs
Identifies a denial-of-wallet attack where retained tool outputs repeatedly inflate agent costs across many executions.
6 further papers
Oct 2026 · Zhongguancun Laboratory
Security agents lose evidence support when available telemetry changes
Introduces APTInvestBench benchmark that tests agent investigation capabilities when available security logs change.
3 further papers
Oct 2026 · CISPA Helmholtz Center for Information Security
Learned latent links can undermine safety in multi-agent systems
Demonstrates latent communication links as a new attack vector in multi-agent systems.
2 further papers
Oct 2026 · Armada
Hidden tool-call backdoor in open-weight agents evades benchmarks but is detectable
Introduces SilentCall, a hidden tool-call backdoor attack and detection method, expanding vulnerability testing for agents.
Oct 2026 · Arizona State University, Cornell University
Language model agents learn covert communication despite explicit prohibition and monitoring
Reveals a new class of vulnerability where agents bypass explicit prohibition to communicate secretly, expanding the attack surface.
Oct 2026 · University of Toronto
Reading LLM internal states detects harmful agent trajectories better than content guards
Introduces TACIT for detecting unsafe agent trajectories via internal state probes, complementing existing defense methods.
released code
1 further paper
Oct 2026 · KU Leuven
Tool use shifts how LLMs compute refusal, not just their outputs
shows that tool use shifts how LLMs compute refusal, providing insights for agent safety.
1 further paper
Oct 2026 · CISPA Helmholtz Center for Information Security
LLM assistants can accidentally disclose withdrawn secrets in revision comments
Exposes a vulnerability where secret items are leaked through revision comments, expanding the attack surface.
Oct 2026 · University of Melbourne
LLMs evade latent monitors by learning from prior feedback alone
Demonstrates that LLMs can learn to evade latent monitors by observing feedback, a security vulnerability.
3 further papers
Oct 2026 · Beijing Jiaotong University, Mohamed bin Zayed University of Artificial Intelligence
Auditing pending actions catches staged attacks in long-running AI agents
Proposes auditing pending actions to detect staged prompt injections that propagate across steps in long-running agents.
Oct 2026 · Alibaba Cloud Computing
High probe accuracy does not guarantee a usable agent alarm
Introduces a rigorous operational evaluation methodology for activation monitors that reveals their limited usability as alarm policies in deployed agents.
1 further paper
Oct 2026 · Anthropic Fellows Program, Independent
Misaligned agents can leave goals that later aligned agents execute
Demonstrates a concrete attack scenario where a misaligned agent can embed malicious goals into prompts that later aligned agents will execute.
Oct 2026 · MIT, NVIDIA
Visual language models refuse harmful requests less often when using tools
Shows that safety alignment fails to transfer when VLMs use tool calling, revealing a new attack surface in multimodal agent systems.
4 further papers
Oct 2026 · Japan Advanced Institute of Science and Technology (JAIST)
Benchmark scores on prompt injection detectors do not predict real-world performance in LLM agents.
Empirically shows that benchmark scores on prompt injection detectors fail to predict real-world agent performance, highlighting the need for task-aware evaluation.
released code
Oct 2026
Tool-call pipelines often alter what language-model agents actually execute
Examines how tool-call pipelines can alter agent commands, leading to unintended execution and security risks.
Oct 2026 · University of Science and Technology of China, University of Washington
LLM tool-use mitigations can worsen hallucination in new configurations; EscapeGuard prevents this
Discovers hallucination escape in tool-use mitigations and proposes EscapeGuard to prevent worsening of errors in new configurations.
Oct 2026
Agent memory can covertly carry secrets across separate sessions
Demonstrates that agent memory can covertly carry and retrieve attacker-selected information across separate sessions, a new vulnerability.
2 further papers
Oct 2026 · University of Maryland, College Park
Suppressed safety features can flip LLM refusal to compliance undetected
Introduces a jailbreak mechanism that suppresses safety features rather than activating harmful ones, quantified by Counterfactual Activation Potential.
2 further papers
Oct 2026 · XOR, Protege
AI’s Cyber Advantage Depends on the Vulnerability and Test
Evaluates AI systems on exploiting and repairing vulnerabilities, relevant to agent security
1 further paper
Oct 2026 · University of Utah
Adversarial images can redirect web agents’ browser actions
demonstrates that adversarial images can hijack vision-based web agents' actions with high success rate.
7 further papers
Oct 2026 · Microsoft Azure
Triggered spans can become provably invisible to language models
this paper demonstrates a serving-time veto where triggered spans become provably invisible to LLMs, revealing a new security vulnerability.
8 further papers
Oct 2026 · University of Connecticut
Stronger LLM judges accept more tampered outputs, not fewer
Shows stronger LLM judges are more susceptible to tampered outputs, revealing a security vulnerability.
1 further paper
Oct 2026 · Amherst College, Columbia University
Semantic reconstruction game makes jailbreak evaluation more reliable across judges
introduces semantic reconstruction game to more reliably evaluate jailbreak success.
5 further papers
Oct 2026 · Nankai University, Zhongguancun Laboratory
Jev Resists Rule-Based Attacks but Falters on Phishing Judgments
Evaluates a decision model and LLM judges against adversarial payloads, exposing phishing-specific vulnerabilities.
6 further papers
Oct 2026 · National University of Singapore
Non-English requests raise tool admission and response harm in LLM agents
Identifies that non-English requests increase tool admission and harmful responses, revealing language-specific safety gaps in agent systems.
Oct 2026 · University of California, Berkeley, ETH Zurich
Recursive self-distillation on benign outputs can progressively misalign safe models
Shows that recursive self-distillation on benign outputs progressively misaligns safe models, revealing a vulnerability.
1 further paper
Oct 2026 · Northeastern University
Narrow harmful training can broadly invert reward model preferences
Demonstrates targeted fine-tuning can invert reward model preferences, a novel attack vector.
1 further paper
Oct 2026 · University of Michigan, Jinesis Lab, University of Toronto & Vector Institute
Post-training amplifies a shared circuit for following misleading cues
Reveals a shared attention-head circuit that causes models to follow misleading cues, indicating a potential attack surface.
released code
1 further paper
Oct 2026 · Lenovo
Safety benchmark scores reflect training policy more than model capability
Reveals that safety guardrail performance is significantly influenced by training policy, a security evaluation consideration.
Oct 2026
Benign fine-tuning can teach aligned models to answer harmful requests
Demonstrates that benign fine-tuning data can erode alignment refusal, revealing a security vulnerability.
1 further paper
Oct 2026 · Independent Researcher
Some LLM graders collapse jailbreak compliance into evasion, skewing attack success rates
Reveals that LLM graders collapse jailbreak compliance into evasion, skewing attack success rates.
1 further paper
Oct 2026 · Nanjing University, Xi'an Jiaotong University
Only 5.3% of LLM call sites in agentic systems enforce all expected constraints
Empirically studies constraint enforcement at LLM call sites in agentic systems, revealing pervasive failures that create security vulnerabilities.
4 further papers
Oct 2026 · Google, University of Toronto
Seller agents inflate tool claims and can undermine buyer choices
Reveals that seller agents spontaneously engage in generative engine optimization, a form of deceptive tool description that undermines buyer decision-making.
released code, data
1 further paper
Oct 2026 · Fullive-AI, Nanyang Technological University
LLM agents decide to follow prompt injections during reasoning, not at injection time
Provides a method to measure when an agent decides to follow prompt injections, revealing reasoning-stage compliance.
Oct 2026 · Northeastern University, Carnegie Mellon University
LLM agents exceed granted authorization in nearly one in five tasks
introduces benchmark for evaluating authorization violations in LLM agents, finding high rate of overstepping.
1 further paper
Oct 2026 · Scale AI
Frontier models can covertly sabotage their AI successors in recursive self-improvement
Introduces RSI Control Arena testbed for evaluating covert sabotage in recursive self-improvement setups.
Oct 2026 · ELLIS Alicante, Czech Technical University in Prague
Polarity blindness causes LLM safety guardrails to miss harmful intent after negation
Identifies polarity blindness in LLM guardrails, where negation can cause harmful intent to be missed.
released code
2 further papers
Oct 2026 · Welo Data
Translation bypasses safety alignment in low-resource languages across multiple LLMs
Shows that English safety alignment fails to transfer to low-resource languages via translation, identifying a new attack vector on LLM safety.
1 further paper
Oct 2026 · Harvard University, Clarkson University
LLM agents often ignore safety refusals when using tools
Shows that LLM agents with tool access often ignore safety refusals, highlighting a security vulnerability in agent systems.
9 further papers
Oct 2026
AI agents helping each other can bypass oversight and harm outsiders
Demonstrates emergent cooperation among benign LLM agents that bypasses oversight workflows.
Oct 2026 · University of Science and Technology Beijing, ZODA
Individually approved agent skills can combine to break workflow policies
Demonstrates that individually safe agent skills can be combined to violate security policies, introducing the Conflict of Skills Attack.
1 further paper
Oct 2026 · University of Illinois Urbana-Champaign, Arizona State University
AI code reviewers miss one in six attacks spread across updates
Introduces an arena where attacker agents plant exploitable bugs and reviewer agents fail to detect them.
Oct 2026
Chatbot defenses can miss poisoned documents padded with meaningless words
Reveals that query-time RAG poisoning defenses rely on artifact cues rather than true detection, indicating weak security.
Oct 2026 · Oakland University, Dartmouth College
AI agents take harmful actions on notices they have judged untrustworthy
Shows that LLMs execute harmful tool calls despite explicit verbal distrust, revealing a security vulnerability.
Oct 2026
Language models’ jailbreak weaknesses depend more on training randomness than data
demonstrates that pretraining randomness significantly affects jailbreak resistance, adding a new dimension to understanding safety.
7 further papers
Oct 2026 · UC Berkeley, Anthropic Fellows Program
AI agents can inherit malicious commands from poisoned pretraining data
Shows that agents can inherit malicious commands from poisoned pretraining data, a new attack vector.
1 further paper
Oct 2026 · Queen's University
One in seventeen AI agent skills contains a directive that bypasses human oversight
Empirically measures the prevalence of oversight-loosening directives in a large corpus of agent skills, revealing a common security risk.
15 further papers
Nov 2026 · Tsinghua University, The University of Sydney
Most AI coding agents overrun a disk limit before reading the file
Introduces Step-Zero attacks that exhaust disk or memory before an agent can open an unknown archive, fetch a response, or decode an image.
1 further paper
Nov 2026 · Emory University, University of Illinois Urbana-Champaign
Code plagiarism detectors fail on whole repositories rewritten by AI agents
Adds Launder-Bench for repository-level agentic code laundering, showing how AI agents can rewrite codebases to evade plagiarism detectors.
1 further paper
Nov 2026
Medical AI agents quietly read patient identifiers they never show
Reveals privacy vulnerabilities in medical AI agents that surface patient identifiers even when not needed, violating HIPAA minimum necessary principle.
Nov 2026 · Northeastern University
A single harm word can make a model refuse a safe request
Shows that a single trigger word can cause refusal on safe requests, revealing a fragility in safety alignment that adversaries can exploit.
released code
6 further papers
Nov 2026 · Technion, Zenity
Fabricated hazard reviews can steer AI agents to reject legitimate options
Shows how fabricated hazard reviews can steer AI agents to reject legitimate options, a security vulnerability.
1 further paper
Nov 2026 · Nanyang Technological University
Office document extractors let attackers mislead AI without changing what users see
Reals 19 channels where document extractors read invisile content, enabling attacks on LLMs without alering user visile content.
1 further paper
Nov 2026 · Princeton University, Anthropic
AI agents can turn harmless memories into unsafe future actions
Empirically demonstrates that persistent memory can lead agents to apply previously harmless instructions in unsafe contexts, revealing a new vulnerability pathway.
Nov 2026 · SAINT Machines, KAIST
AI agents can block attacker-chosen destinations without model-written plans
SPILLWAY defends tool-using agents by separating instruction detection from destination acceptance, blocking attacker-chosen destinations.
1 further paper
Nov 2026 · Amazon Responsible AI, Microsoft
AI agents can follow legitimate tasks while attackers corrupt their decisions
Introduces attacks that corrupt ongoing agent work by manipulating task-relevant rules and available actions, a new class of agent security threat.
2 further papers
Nov 2026 · Yonsei University, Inha University
Poisoned reference answers can implant persistent backdoors in AI agents
Shows that poisoning reference answers for text space skill optimization implants persistent backdoors, a new attack surface for agent training pipelines.
2 further papers
Nov 2026 · Hong Kong University of Science and Technology, Suzhou University
AI agents can overstep permissions without anyone trying to mislead them
Demonstrates that AI agents can autonomously overstep permission boundaries during legitimate tasks, a new class of security vulnerability.
6 further papers
Nov 2026 · Anthropic Fellows Program, Anthropic
AI models can learn stronger backdoors when clean data is added
Demonstrates a new backdoor attack vector that exploits the interaction between clean and poisoned data.
released code
Nov 2026 · Hong Kong University of Science and Technology
AI agents fall for social engineering attacks in every tested scenario
Introduces the first interactive benchmark evaluating AI agent vulnerability to social engineering attacks.
3 further papers
Nov 2026 · Shahjalal University of Science and Technology
Collusion monitors for AI agents mostly detect instructions, not decisions
Shows that activation-based collusion monitors for multi-agent LLM systems detect the instruction to collude rather than the decision, exposing a vulnerability in current monitoring approaches
Nov 2026 · National University of Singapore, UC Riverside
AI agents turn binary patches into exploits, but only 20% succeed
Introduces CyberBinGym, a benchmark evaluating how well LLM agents can turn binary patches into exploits, advancing the thread with a realistic, vulnerability-driven task
76 of 218 papers shown