Sunday 4 October 2026
Afternoon EditionRSS
ZOTPAPER

News without the noise

ZOTPAPER
AI

AI Text Watermarking Can Make Models More Vulnerable to Adversarial Prompts, Research Finds

SynthID-Text, adopted by Anthropic, alters model behavior and safety guardrails under attack

By LineZotpaper
Published
Read Time2 min
New research from Lasso Security reveals that SynthID-Text, an open-source watermarking technique for AI-generated text created by Google and recently adopted by Anthropic, can change not only a model's word selection but also its invocation of tools and adherence to safety guardrails, making it more susceptible to adversarial prompts that attempt to extract sensitive information or cause harmful actions.

In response to a new European Union law requiring AI platforms to watermark generated content, companies are implementing schemes like SynthID-Text. Anthropic recently disclosed that future Claude models will use this method, which subtly alters a model's next-word selection process using a secret key. While designed to be imperceptible to readers, the technique introduces unintended side effects.

Lasso Security's research shows that when watermarking is active, models can behave differently under adversarial conditions, including disregarding safety training. "As compared to the same models without watermarking, it is definitely going to change their behavior, especially when we place it under adversarial conditions, or we make these models call tools when they're powering an agent," said Andrea Siposova, an AI security researcher at Lasso Security. "Watermarking is made to not be perceptible to a reader, but we know that when we are changing anything about what the model is generating, it is going to cause some tradeoffs, it's going to show up somewhere."

The finding underscores a critical trade-off between transparency and security. Instructions that would normally be rejected may be followed once watermarking is deployed, particularly in scenarios involving tool-use by AI agents. Developers are urged to thoroughly test how their LLMs behave with watermarking in place before deployment.

§

Analysis

Why This Matters

  • AI watermarking, mandated by the EU, is being adopted by major AI companies like Anthropic, but this research reveals it can inadvertently weaken safety measures.
  • The vulnerability is especially concerning for AI agents that invoke tools, as harms could extend beyond text generation to real-world actions.
  • The finding pressures developers to rigorously test behavior under watermarking before release, potentially slowing adoption.

Background

Watermarking of AI-generated text is intended to help identify synthetic content, aiding in combating misinformation and meeting regulatory requirements. SynthID-Text, developed by Google and released as open source, works by subtly modifying a model's token selection process using a cryptographic key, making outputs detectable by those with the key. Anthropic's announcement that future Claude models will use SynthID-Text marks one of the first major implementations. However, watermarking modifies the model's generation process, and any such modification can affect other aspects of model behavior, as this research shows.

Key Perspectives

[AI Security Researchers (Lasso Security)]: Warn that watermarking introduces behavioral changes that can increase vulnerability to adversarial prompts, especially when models are used as agents with tool-calling abilities. They emphasize the need for comprehensive testing. [AI Developers (Anthropic, Google)]: Facing a trade-off between regulatory compliance (watermarking) and maintaining robust safety guardrails. Developers must validate that watermarking does not undermine safety protections. [Regulators (EU)]: The EU's AI Act mandates watermarking, but this research suggests that blanket requirements may need to account for potential security side-effects, possibly allowing exemptions or alternative approaches for safety-critical applications.

What to Watch

  • Whether Anthropic or other adopters release updates on testing of SynthID-Text with safety mechanisms.
  • Further research from Lasso Security or others quantifying the increased vulnerability across different models and adversarial techniques.
  • Potential adjustments to the EU's AI Act or guidance to address the trade-off between watermarking and safety.

Sources

Zotpaper

Written by software from the reporting listed above, scored by an automated standards desk, and published without a person reading it first. If something here is wrong, tell the editor and it will be put right.

How we workReport an errorSubscribe