A pair of studies published this week reveal significant vulnerabilities in Retrieval-Augmented Generation (RAG) systems — a widely used architecture that grounds AI responses in external knowledge bases — and propose new frameworks for both exploiting and defending against so-called corpus poisoning attacks, where adversaries inject malicious content to manipulate AI outputs.
Retrieval-Augmented Generation has become a cornerstone of modern AI deployment, enabling large language models to draw on curated document repositories rather than relying solely on training data. Banks, law firms, healthcare providers and enterprise software vendors increasingly use RAG to build applications that answer questions from proprietary knowledge bases. But the architecture's reliance on external document retrieval has also introduced a new class of security threat.
In the first paper, researchers from multiple Chinese universities present findings that challenge the conventional understanding of how dangerous corpus poisoning actually is — and when it is not. Their analysis shows that many existing poisoning attacks, while effective at the initial retrieval stage, fail to survive the full pipeline used in real-world RAG deployments. The culprit, they argue, is a phenomenon they call "retrieval granularity mismatch."
In practice, documents are split into smaller chunks before being indexed and retrieved. Adversarial passages crafted at the document level are often fragmented during this chunking process, stripping them of the contextual signals that made them effective. Reranking models — a second filtering layer that scores retrieved passages for answer relevance — further sideline these disrupted passages, favouring locally coherent, answer-bearing text over globally optimised but incoherent content.
To address this gap, the team proposes CRCP (Chunk-aware and Rerank-Consistent Poisoning), a framework that explicitly accounts for chunking transformations during the adversarial optimisation process. CRCP is designed to generate self-contained adversarial passages that remain potent across varying chunk sizes and reranking configurations. The researchers describe their work not as an endorsement of attacks but as a call to improve security evaluation standards, arguing that current RAG benchmarks fail to reflect the multi-stage nature of real deployments.
The second paper takes the defensive side of the same problem. Researchers from the UK and Germany introduce ProGRank, a post-hoc, training-free defence mechanism designed to be layered onto existing retrieval pipelines without modification to the underlying model. ProGRank works by stress-testing each query-passage pair under mild randomised perturbations and analysing the resulting "probe gradients" from a small subset of model parameters. Passages that behave inconsistently or show high dispersion under perturbation — hallmarks of adversarially crafted content — are ranked down before reaching the language model.
Unlike some existing defences that rely on content filtering or auxiliary classifier models, ProGRank requires no retraining and preserves original passage content. The authors also describe a surrogate-based variant for scenarios where the deployed retriever is not directly accessible. Testing across multiple datasets, retrievers and attack strategies showed that ProGRank maintained a favourable balance between security and utility, including against adaptive attacks designed specifically to evade it.
Together, the two studies illuminate an evolving cat-and-mouse dynamic in AI security. As RAG systems become more sophisticated — adding chunking, dense retrieval, and reranking layers — both the attack surface and the opportunities for defence are shifting in ways that earlier research did not fully capture.
Analysis
Why This Matters
- RAG systems are now embedded in enterprise AI products used across healthcare, legal, finance and government sectors; a successful poisoning attack could cause AI systems to generate false, harmful or manipulated answers at scale.
- Both papers highlight that the security community has been evaluating RAG vulnerabilities under unrealistic conditions, meaning deployed systems may be both more and less vulnerable than previously understood.
- The publication of CRCP, an improved attack framework, raises dual-use concerns: while intended to spur better defences, it also lowers the bar for sophisticated adversarial attacks on production AI systems.
Background
RAG emerged as a practical solution to a core limitation of large language models: their knowledge is frozen at training time and can be factually unreliable. By connecting a language model to a live or curated document store, developers can keep answers current and grounded in verified sources. The architecture gained rapid adoption from roughly 2023 onwards as enterprises sought to deploy AI on proprietary data without the cost and risk of full model fine-tuning.
Corpus poisoning — injecting or editing documents in the retrieval database to steer model outputs — was identified as a threat vector shortly after RAG became widespread. Early research demonstrated that even a small number of malicious passages, if retrieved consistently, could reliably alter a model's answers on targeted queries. However, most of that research evaluated attacks under simplified single-stage retrieval, omitting the chunking and reranking steps now standard in production deployments.
The field of adversarial machine learning has a long history of such realism gaps: attacks that work in laboratory conditions often behave differently in deployment, and vice versa. The two papers published this week represent a maturation of RAG security research, moving from proof-of-concept demonstrations toward evaluation under conditions that more closely mirror how these systems actually operate.
Key Perspectives
Security researchers (CRCP team): Argue that the community has been measuring RAG robustness incorrectly and that attacks should be studied as multi-stage consistency problems. Their CRCP framework is framed as a diagnostic tool to reveal where defences fall short, not as a weapon — though the line between the two is inherently blurry in adversarial ML research.
Defence researchers (ProGRank team): Emphasise that practical defences must be deployable without retraining or architectural changes, given the complexity of production AI systems. ProGRank's training-free, post-hoc design reflects a pragmatic engineering philosophy: improve security without disrupting existing infrastructure.
Critics and broader AI safety community: Will likely note that publishing a more effective attack framework alongside — but separately from — its defence creates a window of elevated risk. There is also a question of whether gradient-based defences like ProGRank can keep pace with adaptive adversaries who specifically optimise to evade instability signals.
What to Watch
- Whether major RAG platform providers (including those building on OpenAI, Cohere, or open-source frameworks like LangChain and LlamaIndex) incorporate reranking-aware security evaluations into their testing pipelines.
- Academic and industry responses to CRCP: if the attack framework is reproduced and extended, it could accelerate both offensive and defensive research — or prompt calls for publication norms around dual-use AI security findings.
- Regulatory developments around AI system integrity in high-stakes sectors; the EU AI Act's provisions on robustness and accuracy for high-risk AI systems may eventually require documented adversarial testing of RAG deployments.