India

Developer Portfolio

AI Crawler Robots.txt Generator

Pick which AI crawlers may use your site. The tool writes a ready-to-paste robots.txt that blocks model-training bots by default, keeps AI search citations open, and preserves the rules you already have.

01 Choose your policy

Choose a starting point, then adjust individual crawlers below.

Jump to your file

02 Model training

These crawlers collect pages that may be used to train AI models. Blocking them opts your site out of future training runs.

Model training crawlers

  • Collects pages that may be used to train OpenAI models.

  • Collects pages that may be used to train Anthropic models.

  • Standalone control for Gemini training. No effect on Google Search.

  • Opt-out token for Apple Intelligence training. Regular Applebot still powers Siri and Spotlight.

  • Builds the Common Crawl corpus many vendors train on.

  • ByteDance crawler associated with model training data.

03 AI search

These crawlers index pages so AI answers can cite and link to your site. Keep them open unless you want out of AI search results.

AI search crawlers

  • Indexes pages so ChatGPT search answers can cite and link to your site.

  • Indexes pages so Claude search answers can cite and link to your site.

  • Indexes pages for Perplexity answers. Perplexity states it is not used for training.

04 User-triggered fetch

These fetch a page live when a person asks an AI assistant about it. They often ignore robots.txt by design, so blocking here is a request, not a guarantee.

User-triggered fetch crawlers

  • Fetches a page live when a ChatGPT user asks about it.

  • Fetches a page live when a Claude user asks about it.

  • Fetches a page live when a Perplexity user asks about it.

05 Add your site details

Bring your existing rules along and add a sitemap or crawl delay.

Paste your current file to keep its existing rules.

Live preview

Your robots.txt

12 crawlers

Your file updates as you make changes. Review it, then copy or download it.

# AI crawler rules - generated with https://bhargavtibadiya.com/tools/ai-crawler-robots-txt-generator
# Re-check vendor docs quarterly; new bots appear often.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

Place this file at the root of your domain. Existing groups are kept; whole-site rules for the listed AI bots are regenerated.

Keep in mind: robots.txt is a request to crawlers, not access control. Some user-triggered fetchers may ignore it.

Know the bots

AI crawlers fall into three groups, and the difference matters. Blocking training crawlers opts your content out of model training. AI search crawlers are how your pages get cited in AI answers, so most sites keep those open. User-triggered fetchers load a page when a person asks about it and often ignore robots.txt.

Model training

These crawlers collect pages that may be used to train AI models. Blocking them opts your site out of future training runs.

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider

AI search

These crawlers index pages so AI answers can cite and link to your site. Keep them open unless you want out of AI search results.

OAI-SearchBot, Claude-SearchBot, PerplexityBot

User-triggered fetch

These fetch a page live when a person asks an AI assistant about it. They often ignore robots.txt by design, so blocking here is a request, not a guarantee.

ChatGPT-User, Claude-User, Perplexity-User

What robots.txt cannot do

robots.txt is public and voluntary. It declares intent; it does not hide, protect, or delete anything. For hard enforcement, use server-side blocking such as a WAF or edge function.

User-triggered fetchers like ChatGPT-User and Perplexity-User can ignore robots.txt by design, because a person asked for that page.

New bots appear often, so re-check vendor docs quarterly. This directory was verified against the official OpenAI, Anthropic, Google, Perplexity, Apple, and Common Crawl docs in November 2026. Meta crawler tokens are deliberately not offered: Meta's official crawler page could not be verified when this tool was built, and unverified tokens stay out of the generator until they are.

Related reading: How to block AI crawlers in 2026 and the AI training robots.txt directive.

FAQ

Does blocking AI training bots hurt my visibility in AI search?
No, not when you block selectively. Training crawlers such as GPTBot, ClaudeBot, and Google-Extended collect data for model training, while AI search bots such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot index pages so answers can cite and link to your site. Block the first group and keep the second open to stay visible in AI answers without feeding training runs.
Does Google-Extended affect my Google Search rankings?
No. Google-Extended is a standalone control token for Gemini training. Google documents that it has no effect on Search indexing or ranking, which stay under the regular Googlebot crawler.
Will every AI bot follow the file this tool generates?
No. robots.txt is a voluntary standard, not access control. OpenAI, Anthropic, Google, and Apple document that their crawlers respect it, but user-triggered fetchers like ChatGPT-User and Perplexity-User can ignore it when a person asks for a page. For hard enforcement, pair robots.txt with server-side rules such as a WAF or edge function.
Where do I put the generated robots.txt?
Place it at the root of your domain so it answers at https://yourdomain.com/robots.txt. If you already have one, paste it into the tool first and the generator keeps your existing rules while adding the AI crawler groups.