AI Crawler Block Checker

What the checker reads, and what it cannot tell you

The checker reads a site's public robots.txt and shows, for each AI crawler, whether the file allows or blocks it for the path you test. A crawler follows its own group if the file has one, otherwise the wildcard group, and the longest matching rule wins.

It reports declared rules only. robots.txt is voluntary, and user-triggered fetchers such as ChatGPT-User and Perplexity-User can ignore it by design. To build a new file, use the AI crawler robots.txt generator. For background, read how to block AI crawlers in 2026.

FAQ

What does this checker actually test?

It reads the site's public robots.txt and applies the standard matching rules for each AI crawler token: the crawler's own group if there is one, otherwise the wildcard group, with the longest matching path rule winning. It reports what the file declares for the path you test.

Does a blocked result mean the crawler cannot reach my site?

No. robots.txt is a voluntary standard. OpenAI, Anthropic, Google, and Apple document that their crawlers respect it, but user-triggered fetchers such as ChatGPT-User and Perplexity-User can ignore it by design. For hard enforcement, use server-side rules such as a WAF or edge function.

What happens if the site has no robots.txt?

Crawlers treat a missing robots.txt (an HTTP 404) as permission to crawl everything, so every bot shows as allowed.

What does the checker request from my site?

Only the public robots.txt file, once, over HTTP or HTTPS. Private and internal addresses are refused, and the tool does not crawl any other page. You can also paste a robots.txt and nothing is requested at all.

How do I change which bots are blocked?

Use the AI crawler robots.txt generator to build a new file that blocks training bots, keeps AI search open, and preserves your existing rules.