Cloudflare announced a Disallow AI Training setting that lets site owners remain indexed for search while blocking the same crawler from using their content for AI model training1. Apple, Google, and Microsoft have honored or committed to honor the setting, which Cloudflare enforces through robots.txt via Bot Preference Sync. The company also created an "Accountable" designation requiring bot operators to provide URL-level transparency, opt-out mechanisms for AI summaries, and assurances that refusing training will not affect search ranking. Starting September 15, existing block preferences will apply to mixed-use crawlers including Applebot, Bingbot, and Googlebot.
Cloudflare Splits Search Indexing From AI Training for Apple, Google, Microsoft Crawlers
Cloudflare's new Disallow AI Training setting lets sites stay in search results while blocking AI training by Apple, Google, and Microsoft crawlers.