Fri 13 Nov
LiveGet it free
ZOTPAPER

News without the noise

← All concepts

prompt injection

Large language models that take actions on their own, called agents, often read text from outside sources, like websites or tool responses. Attackers can slip malicious instructions into that text. When the agent processes it, it may follow the hidden orders instead of the user's real request, a trick called prompt injection. This is dangerous because agents can access files, send messages, or run commands.

The papers show this is an active arms race. Attackers generate new, adaptive injections, while defenders use simulated training, self-reflection, or system-level containment to block them. A common finding is that defenses trained on fixed attacks fail against novel ones, and that success rates vary wildly between benchmarks. The trend is toward evaluating agents in realistic settings, co-evolving attacks and defenses, and monitoring at the system boundary rather than only at the prompt level.

Papers
84
Released code
9
First seen
July 2026
Latest
Nov 2026

78 papers in the last two months, against 4 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1AI agents can block attacker-chosen destinations without model-written plans

    A deterministic check on typed tool-return fields and removal of instruction-like text block prompt-injected redirection of tool-using agents.

    Nov 2026
  2. 2AI agents can follow legitimate tasks while attackers corrupt their decisions

    Indirect prompt injection can tamper with task outcomes without overriding the task, revealing gaps in current defenses.

    Nov 2026
  3. 3AI agent defenses struggle to preserve both security and usefulness

    MetaPI converts general LLM agent benchmarks into security tests for indirect prompt injection.

    Nov 2026

How fast it is moving

JunJulAugSeptOctNov so far

Often studied with

agent safetyRisks from AI agents including timing errors, shutdown avoidance, collusion, overclaiming, and unsafe execution, mitigated by constrained decoding, safety harnesses, and swarm governance.58 togetherruntime monitoringAudits an AI agent's live actions, internal states, or effects against constraints or policies to catch errors and attacks.25 togetheragentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real16 together

Who is working on it

National University of Singapore 5Nanyang Technological University 4Microsoft 3Google 3Mohamed bin Zayed University of Artificial Intelligence 3Institute of Information Engineering, Chinese Academy of Sciences 3

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.