How to block AI crawlers from your website in 2026
- Guides
- AI crawlers
- robots.txt
- SEO
- Cloudflare
- Written by
- Bhargav Tibadiya
- Published
- Reading time
- 5 min read
You block AI crawlers in layers: a shared robots.txt directive that every major lab now honors, per-bot rules for the crawlers that matter, and enforcement at the edge for the bots that ignore polite requests. As of November 2026 the job is easier than it has ever been, because Google, OpenAI, Anthropic and Microsoft all committed to one shared opt-out signal at the end of October. It is also still incomplete, because robots.txt is a voluntary protocol and some crawlers ignore it. This guide walks through the full stack, from the one-line version to the enforcement tier.
Start with the new shared robots.txt directive
The biggest change this year: instead of maintaining a rule per AI lab, you can now write one user-agent block that every signatory reads. The IETF proposal behind it adds a shared user agent, User-Agent: __usage_ai-training, followed by normal Disallow and Allow lines. A Disallow of / opts your whole site out of AI training crawling; an Allow on a specific path keeps a press kit or public docs section open. We covered the shared AI training opt-out and its exact syntax when the four labs signed on.
If you do only one thing today, do this. It is per-path, it is shared across labs, and it keeps working as more crawlers adopt the standard.
Know who is crawling you first
Blocking without visibility is guessing. Before writing rules, answer three questions from your logs or analytics: which AI crawlers hit your site, how often, and whether they send any referral traffic back. The main names to look for are GPTBot (OpenAI training), ClaudeBot (Anthropic), Google-Extended (Gemini training), and a long tail of smaller scrapers. Some are polite and useful to keep, because AI assistants citing your pages can send traffic. Others ignore your policies entirely, republish your content as uncredited summaries, and degrade site performance with aggressive request rates. The right strategy separates the two rather than blocking everything.
Block AI crawlers with robots.txt and meta tags
The robots.txt layer handles the cooperative crawlers. Beyond the shared directive, keep per-bot rules for the major crawlers so your intent is explicit even to labs that have not adopted the standard: a User-Agent block for GPTBot, one for ClaudeBot, and one for Google-Extended, each with Disallow lines for the paths you want protected. Remember that Google-Extended controls AI training use; it does not affect ordinary Search indexing, so blocking it does not cost you rankings.
For page-level control, meta tags add a second signal inside the HTML itself, which matters for content that scrapers reach without reading your robots.txt. Use them to mark specific pages or sections as off-limits for training while leaving the rest of the site open.
Two honest limits. Robots.txt and meta tags are requests, not enforcement: they only work on crawlers that choose to comply. And blocking training crawlers does not stop AI assistants from summarizing pages they fetch at answer time, which is a separate mechanism with separate controls.
How Cloudflare blocks AI crawlers at the edge
Enforcement is where Cloudflare comes in, and it is the answer to the most common follow-up question: how does Cloudflare block AI crawlers that ignore robots.txt? Cloudflare sits in front of roughly 20 percent of all web properties, so it sees crawler behavior across a huge sample and offers a dedicated AI Crawl Control product. The practical toolkit:
- Block or allow rules per crawler, per domain or per page, so you keep the bots that send referrals and stop the rest.
- AI Labyrinth, which feeds a maze of generated junk content and links only to bots caught ignoring your robots.txt, wasting their crawl budget instead of yours.
- Turnstile and bot management to separate humans from automated traffic without CAPTCHA friction.
- Rate limiting and WAF rules that slow or block crawlers hammering specific endpoints.
- Pay per crawl, which answers HTTP 402 and charges AI crawlers per page instead of blocking them, if you would rather be paid than left alone.
If you are launching a new site, the cleanest posture is inverse: block all AI crawlers by default, watch the logs for a month, then allow the ones that prove useful.
Keep the crawlers that pay you back
A total block is easy but not always smart. Some AI crawlers feed assistants that cite and link your pages, and those citations are becoming a real traffic channel as answer engines grow. The better posture for most publishers is selective: block pure training crawlers that never send a reader back, allow the assistant-facing fetchers that quote you with a link, and revisit the list quarterly because the crawler landscape and each lab's behavior keep changing. Your logs, not ideology, should decide which user agents earn a place on the allow list.
What blocking cannot do
Be clear-eyed about the ceiling. None of this removes content that was already scraped and trained into a model. None of it stops a determined scraper rotating through residential proxies; edge enforcement raises their cost, it does not make scraping impossible. And blocking training crawlers does not restore traffic lost to AI summaries that answer the question without a click. The realistic goal is control: you decide who trains on your work, who pays for it, and who gets nothing.
FAQ
How do I stop AI from crawling my website?
Add the shared User-Agent: __usage_ai-training block with a Disallow to robots.txt, add per-bot rules for GPTBot, ClaudeBot and Google-Extended, then enforce at the edge with Cloudflare AI Crawl Control or your CDN's equivalent for crawlers that ignore the file.
Does robots.txt actually stop AI crawlers?
It stops the compliant ones, which now includes Google, OpenAI, Anthropic and Microsoft. It does nothing to bots that ignore it, which is why edge enforcement exists as the second layer.
How does Cloudflare block AI crawlers?
AI Crawl Control lets you set per-crawler block or allow rules, traps misbehaving bots with AI Labyrinth, applies rate limits and WAF rules, and can charge crawlers per page through pay per crawl.
Will blocking AI crawlers hurt my SEO?
No. AI training crawlers like GPTBot and Google-Extended are separate from search indexing bots. Blocking them does not affect Google Search rankings.
Can I charge AI companies for crawling my site?
Yes, through Cloudflare's pay per crawl system, which serves HTTP 402 payment-required responses to crawlers and bills them per page on terms you set.
Sources
Elsewhere on this site
The selected work catalogue lists the SaaS products and dashboards behind notes like this one.
