OpenAI, Google, Microsoft, and Anthropic have each committed to voluntarily adopt a unified standard that lets web publishers opt out of AI training scraping through a single mechanism rather than configuring preferences crawler by crawler1,3,4.
The control takes the form of a shared "policy token" that sits inside a website's robots.txt file, the long-standing protocol that tells web crawlers what they may and may not access. Kevin Kelley, a technical lead for standards at Anthropic, proposed the signal as part of work within the Internet Engineering Task Force (IETF). Google described the effort as originating from the AIPREF working group at IETF.
Today, publishers who want to block AI training crawlers must set directives for each company's bot individually. The new syntax consolidates that into a single robots.txt entry that multiple AI providers agree to honor2. Microsoft's announcement, authored by Krishna Madhavan, Principal Product Manager at Microsoft Bing, framed the move as building on controls the company first offered in 2023 with NoArchive and NoCache tags. Microsoft said the new preferences let publishers make choices about training, grounding, or both while remaining eligible for Bing search results. The scope covers web content collected through Microsoft's crawlers and used for training or to inform generated responses in Bing, Copilot, and Microsoft 365.
Google said the control will help websites express their AI training preferences across multiple AI providers in a single step. The proposal remains a work in progress within IETF and is described in an Internet-Draft.
The standard is voluntary. Each company said it plans to adopt the proposal, but the IETF process has not finalized it.
ANALYSIS A voluntary commitment from four of the largest AI model developers carries practical weight for publishers, but enforcement depends entirely on each company's continued compliance, and the standard does not bind smaller or non-participating crawlers. The mechanism addresses training opt-out preferences expressed going forward through robots.txt; it does not retroactively govern content already ingested by these companies' models.