Robots.txt Tester, Check If Googlebot and AI Crawlers Can Access Your Pages
Fetch any site's robots.txt, review its rules and sitemap directives, then test specific URL paths against Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, and more to see exactly which rule allows or blocks each crawler.
What Robots.txt Actually Controls, and What It Does Not
Robots.txt is a crawl directive, not an indexing directive. It tells well-behaved crawlers which paths they may request, which protects server resources and keeps low-value sections like faceted navigation, internal search results, and cart pages out of the crawl queue. It does not remove URLs from Google's index: a page blocked in robots.txt can still rank if other sites link to it, appearing as a bare URL with no snippet. Worse, blocking a page prevents Google from seeing a noindex tag on it, so the two directives must never be combined on the same URL.
Rule evaluation follows a precise order. A crawler first selects the group whose User-agent line most specifically matches its name, an exact match like Googlebot always beats the wildcard group. Within that group, every Allow and Disallow pattern is compared against the URL path, and the longest matching pattern wins, with Allow winning ties. Wildcards match any sequence of characters and a dollar sign anchors a pattern to the end of the URL. If nothing matches, the URL is allowed by default, which is why an empty or missing robots.txt means everything is crawlable.
Blocking AI Crawlers Means Blocking AI Search Visibility
A wave of sites blocked GPTBot, ClaudeBot, PerplexityBot, and CCBot when AI training became controversial, and many never revisited that decision. The landscape has changed: those same user-agents now power retrieval for AI search products. Blocking GPTBot keeps you out of ChatGPT's answers, blocking ClaudeBot removes you from Claude's web results, and blocking PerplexityBot erases you from Perplexity citations. As answer engines take a growing share of discovery, a blanket AI ban in robots.txt is effectively an AEO and GEO opt-out that hands citation share to competitors who stayed crawlable.
The right approach is deliberate, per-bot policy rather than a reflexive block. Decide which surfaces matter for your business, keep the crawlers that feed them allowed, and use targeted Disallow rules only for genuinely sensitive paths. Then verify with this tester: run your key money pages against Googlebot, GPTBot, ClaudeBot, and PerplexityBot and confirm each returns an allowed verdict. Pair that with clean sitemap directives in the same file so every crawler that arrives can find your canonical URL list immediately.