AI giants back a shared opt-out signal for web scraping

Publishers facing a steady stream of AI crawlers have long had to manage their preferences company by company. OpenAI, Google, Microsoft, and Anthropic now say they will support a shared mechanism designed to let a website communicate those preferences to multiple AI providers at once.

How the proposed signal works

The mechanism is referred to as a “policy token.” It was proposed by Kevin Kelley, Anthropic’s technical lead for standards, as part of work associated with the Internet Engineering Task Force. The signal would live in a site’s robots.txt file, a long-standing way for websites to communicate access rules to search engines, archive services, and other crawlers.

Today, a publisher that wants to restrict AI training may need to identify and block different crawlers individually. A shared token is intended to make that process more consistent. OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers if the token appears on a site. Anthropic has already adopted the signal. OpenAI says it will implement it in the coming months, while Google and Microsoft plan to support it next year.

Microsoft says its controls will distinguish between AI training and AI grounding. Grounding refers to directing a model to a verified source of information, a practice intended to reduce hallucinations. The company also says opting out of these AI uses will not remove a website from Bing search results. Google has previously offered comparable controls for deciding whether content appears in AI search features or is used for training, without affecting ordinary search visibility.

Why the proposal matters

For publishers, a common signal could reduce the operational burden of maintaining separate rules for every AI provider. It could also make more nuanced preferences easier to express, such as allowing search indexing while declining permission for model training. For AI companies, a shared convention may reduce the fragmentation created when each provider invents its own crawler names, settings, and interpretations.

The proposal does not settle the broader dispute over content rights. A robots.txt entry and a policy token are machine-readable statements of intent; their effectiveness still depends on crawlers honoring them. Meta has voiced support for a common signal but has not said whether it will implement the setting. Publishers are also considering other approaches, including the Really Simple Licensing standard, which could allow crawlers to pay for access, and Cloudflare’s default blocking of AI crawlers.

The larger significance is therefore institutional as much as technical. A shared field could give websites a common language for managing the boundaries between search, retrieval, grounding, and training. If the major providers follow through, publishers may gain a simpler way to communicate those boundaries. Until the standard and its enforcement mature, however, site owners will likely need to keep using provider-specific rules and access controls alongside the new signal.

Source: The Verge AI