OpenAI, Google, Microsoft, and Anthropic have committed to a new standard that allows publishers to opt out of AI data scraping across the companies simultaneously, using a shared “policy token” proposed to the Internet Engineering Task Force.
The shared signal will live within a website's robots.txt file. Currently, publishers have to set their AI training preferences separately for each AI company. The standard was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the IETF.
The announcement comes as publishers grapple with a continuing influx of AI crawlers scraping their sites and feeding content into AI models. Previous attempts to address the issue include the Really Simple Licensing (RSL) standard, which offers a way to charge crawlers, and Cloudflare's default blocking of AI crawlers.
Anthropic has already adopted the signal. OpenAI says it will implement it in the coming months, while Google and Microsoft plan to support it next year. Microsoft says it will offer granular controls that let publishers opt out of AI training or, separately, AI grounding (the process of pointing an AI model to a verified source to reduce hallucinations). It notes websites will still appear in Bing search results regardless of their preferences. Google previously implemented similar controls for its own services.
Meta communications head Andy Stone signaled support for the unified signal, but did not say whether the company will honor it. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone said. “And we look forward to working with other AI companies on the standards and their implementation across the industry.”
Analysis
Why This Matters
- Publishers can save significant time and resources by setting preferences once across multiple major AI companies, rather than negotiating individual opt-outs.
- It establishes a formal, industry-wide technical mechanism for consent in AI training data, potentially reducing legal ambiguities around web scraping.
- The model could set a precedent for balancing publisher rights and AI development in other parts of the world.
Background
Web scraping has long been governed by robots.txt, a protocol for websites to signal permissions to automated crawlers. The rapid growth of generative AI has sharply increased the volume of content scraping for model training, creating demand for a standardised opt-out solution. The shared policy token extends the robots.txt framework within the IETF to meet this need.
Key Perspectives
OpenAI, Google, Microsoft, and Anthropic: Formally committing to adopt the standard and restrict their crawlers on sites that use the token, with Microsoft offering separate controls for AI training and AI grounding.
Meta: Supports the idea of a common signal as an important step, but has not yet committed to honouring the opt-out itself.
What to Watch
- Whether Meta and other AI companies not in the initial group decide to adopt and enforce the standard.
- The speed of implementation as the IETF process advances and companies meet their rollout timelines.
- How the policy token interacts with other emerging models, such as the RSL standard for charging crawlers.
Checked
12 claims in this story checked against 1 source: 12 supported.
Show each claim
- SupportedOpenAI, Google, Microsoft and Anthropic say they will adopt a shared standard publishers can use to opt out of AI scraping across multiple companies.The Verge: “OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI”
- SupportedAndy Stone said, “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,”The Verge: ““We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone says.”
- SupportedAndy Stone said, “And we look forward to working with other AI companies on the standards and their implementation across the industry.”The Verge: ““And we look forward to working with other AI companies on the standards and their implementation across the industry.””
- SupportedThe shared policy-token signal will be in a website’s robots.txt file.The Verge: “The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file”
- SupportedKevin Kelley, Anthropic’s technical lead for standards, proposed the signal as part of the IETF.The Verge: “The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).”
- SupportedPublishers currently have to signal their AI training preferences separately to each AI company.The Verge: “Currently, sites have to signal their AI training preferences separately to each AI company.”
- SupportedAnthropic has adopted the signal; OpenAI says it will implement it in the coming months; Google and Microsoft plan to support it next year.The Verge: “While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.””
- SupportedMicrosoft says it will offer granular controls for opting out of AI training or AI grounding, which points an AI model to a verified source to reduce hallucinations.The Verge: “Microsoft says it will offer granular controls that will give publishers the ability to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations.”
- SupportedMicrosoft says websites will still appear in Bing search results regardless of publishers’ preferences.The Verge: “Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences.”
- SupportedGoogle previously implemented similar controls for its own services.The Verge: “Google previously implemented similar controls that allow sites to control whether their content appears in AI search features or is used for AI training, without impacting visibility in search.”
- SupportedMeta communications head Andy Stone signaled support for the setting but did not say whether Meta would honor it.The Verge: “Meta communications head Andy Stone also signaled support for the new setting, but doesn’t say whether the company will honor it.”
- SupportedPrevious efforts include the RSL standard, which lets publishers charge crawlers, and Cloudflare blocking AI crawlers by default.The Verge: “the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default.”
