One robots.txt line now opts your site out of AI training with Google, OpenAI, Anthropic and Microsoft
- Notes
- SEO
- AI crawlers
- robots.txt
- Written by
- Bhargav Tibadiya
- Published
- Reading time
- 3 min read
Site owners have wanted one switch for AI training for years, and on October 30 and 31, 2026 the four biggest AI labs committed to honoring it. Google, OpenAI, Anthropic and Microsoft have all signed up to a shared robots.txt directive that lets a publisher say, in one place, whether their content can be scraped for generative AI training. Instead of writing separate rules for every lab's crawler, you write one user-agent block and every signatory respects it.
What the new robots.txt directive does
The mechanism is an IETF proposal written by Kevin Kelley, Anthropic's technical lead for standards. Today, opting out means identifying each lab's crawler and adding a rule per provider, and keeping that list current as new crawlers appear. The proposal adds a single shared user agent, written as User-Agent: __usage_ai-training, followed by normal Disallow and Allow lines.
Two properties make it useful. First, it is per-path: a Disallow of / blocks AI training crawling for the whole site, while an Allow on a specific path, a press kit for example, keeps that section open. Second, it is shared: every lab that signs on interprets the same block, so the rule you write once keeps working as adoption grows. Google, for instance, will read it as an instruction for the Google-Extended agent that gathers Gemini training data.
What the directive cannot do
Three limits are worth stating plainly, because they are the ones people get wrong:
- It is not retroactive. Content already scraped is already in the datasets; the directive only stops future crawling.
- It does not affect search. The signal is specific to AI training crawling, so pages still get indexed and still appear in search results.
- It is a request, not a wall. It binds the labs that committed to honor it; a determined scraper that ignores robots.txt is unaffected, and enforcement beyond the signatories does not exist.
When each lab starts honoring it
The commitments have different clocks. Anthropic already supports the directive. OpenAI has pledged to implement it in the coming months. Google says it will support it in early 2027, and Microsoft from February 1, 2027. Writing the rule now is not wasted work: it takes effect per lab as each one switches support on, and it documents your preference from today.
What site owners should do now
Decide per path, not per site. The realistic question for most businesses is not whether AI training is good or bad but which content benefits from being in models and which does not. Marketing pages, docs and press material usually gain from inclusion; gated research, customer data and drafts usually do not. The new directive finally lets you express exactly that split in one block.
One honest counterpoint: blocking is not always the right call. This site allows every major AI crawler by default, because being quotable by answer engines is the whole point of publishing. The about page covers how I handle exactly this visibility trade-off in client work, and the directive matters because it makes the choice explicit and portable either way.
FAQ
Does the AI training directive block Google Search?
No. The shared signal applies only to crawling for generative AI training. Google maps it to its Google-Extended agent, so normal indexing and search appearance are unaffected.
When will Google honor the new robots.txt directive?
Google says it will support the directive in early 2027. Anthropic supports it already, OpenAI in the coming months, and Microsoft from February 1, 2027.
Does the directive remove content already used for AI training?
No. It only governs future crawling. Anything scraped before the directive was in place stays in the datasets it already reached.
Which AI companies honor the shared opt-out?
Google, OpenAI, Anthropic and Microsoft have committed to the IETF proposal. Other crawlers are not bound by it, so sensitive material still needs real access control rather than robots.txt alone.
Sources
- Announcement and syntax: Google joins OpenAI and Anthropic in promising new scraping controls - Engadget (opens in a new tab)
Elsewhere on this site
The selected work catalogue lists the SaaS products and AI builds behind notes like this one.
