robots.txt: The Complete SEO Guide for 2026

What Is robots.txt and What Does It Do?

robots.txt is a plain text file at the root of your domain (yourdomain.com/robots.txt) that tells web crawlers which pages they can and cannot access. It does not prevent indexing; it controls crawling. A page blocked in robots.txt can still appear in Google's search results if other sites link to it.

That last sentence trips up more sites than almost any other SEO misconception. I've audited technically sophisticated sites where the team confidently blocked sensitive staging pages via robots.txt, then found those same URLs indexed in Google six months later because someone had linked to them. If you need a page out of the index, you need a noindex directive on the page itself, not a robots.txt exclusion.

The Robots Exclusion Protocol has been around since 1994, but it only became a formal IETF standard in September 2022 (RFC 9309). That standardisation matters because it codified behaviour that was previously informal convention. It also means Google, Bing, and all major AI crawlers are now operating against a published specification rather than a best-effort gentleman's agreement.

How Do You Write robots.txt Syntax Correctly?

The basic unit of robots.txt is a group: one or more User-agent lines followed by one or more Disallow or Allow lines. Groups are separated by blank lines. The order of groups matters; crawlers read the file top to bottom and apply the first matching group for their user agent.

Here is a correctly formatted robots.txt for a typical site:

A few things to notice in that example. An empty Disallow: line means "allow everything" for that user agent. The Allow: directive for /wp-admin/admin-ajax.php takes precedence over the broader Disallow because it's more specific. And the Sitemap: directive at the bottom is not part of the original REP spec, but every major crawler supports it and it's one of the easiest ways to get your sitemap discovered faster.

Wildcards and Pattern Matching

The REP supports two wildcard characters in path patterns: * matches any string of characters, and $ anchors the end of a URL. These are powerful but easily misused.

• Disallow: /search* blocks /search, /search?q=anything, /searching, and /search-results
• Disallow: /*.pdf$ blocks all PDF file URLs ending in .pdf
• Disallow: / blocks the entire site for that user agent
• Disallow: /category/ blocks /category/ and everything beneath it

Test every pattern before deploying. A misplaced wildcard like Disallow: /wp-content/ looks harmless but will block Googlebot from accessing your theme CSS and plugin JavaScript, which breaks rendering. That's a crawl budget and ranking problem rolled into one.

How Should You Handle AI Crawlers in robots.txt?

This is where robots.txt has changed most significantly since 2023. When OpenAI launched GPTBot in August 2023, it was the first widely-used AI training crawler with a published user-agent string. Within six months, Anthropic (ClaudeBot), Google (Google-Extended), Common Crawl, Perplexity (PerplexityBot), and Apple (Applebot-Extended) all followed with their own named agents.

The decision of whether to block them is genuinely complicated. My current position: block AI training crawlers if you want to protect your content from being used to train LLMs, but don't block AI indexing crawlers (like Google-Extended, which is used for AI Overviews, not training) unless you specifically want to opt out of AI Overviews. Those are different things, and conflating them costs you visibility.

Crawler	Company	Purpose	Block to opt out of
GPTBot	OpenAI	ChatGPT training data	ChatGPT training corpus
ClaudeBot	Anthropic	Claude training data	Claude training corpus
Google-Extended	Google	Gemini + AI Overviews	Google AI products (not Search)
PerplexityBot	Perplexity	Perplexity AI answers	Perplexity citations
Applebot-Extended	Apple	Apple Intelligence	Siri and Apple AI features

For most publishers, I'd argue blocking GPTBot and ClaudeBot makes sense, since that's training data for commercial products. Blocking PerplexityBot and Google-Extended is a harder call: those crawlers feed systems that can actively send you referral traffic when they cite your content. Use our LLM mentions tool to check if you're currently being cited before you block those crawlers.

What Are the Biggest robots.txt Mistakes That Hurt SEO?

Five mistakes show up in nearly every technical audit I run. The order here is roughly by damage potential.

1. Blocking CSS and JavaScript Assets

Google renders your pages before it indexes them. If your robots.txt blocks /wp-content/themes/, /assets/, /static/js/, or similar asset directories, Googlebot can't render your pages correctly. The result looks like a 2002 era website to Google: no layout, no visual hierarchy, potentially significant ranking loss. I've seen well-established sites lose 30% of their organic traffic in two weeks from this one mistake. Check with Google Search Console's URL Inspection "Live URL" view: if the rendered screenshot looks broken, check your robots.txt first.

2. Using robots.txt as a Security Layer

robots.txt is a public file. Anyone can read it. In fact, it's one of the first places security researchers look when analysing a site: your blocked paths are a map of what you're trying to hide. Never use robots.txt to protect sensitive content. Authentication and server-side access controls are the right tools for that.

3. Inconsistency Between robots.txt and Canonical Tags

A page that's blocked in robots.txt but listed as a canonical on another page creates a contradiction Google can't resolve cleanly. The most common version: blocking paginated URLs (/page/2/, etc.) in robots.txt while having them canonicalled to the first page. If you don't want paginated URLs crawled, consolidate them at the URL architecture level, not via robots.txt. For a full treatment, see the canonical tags guide.

4. An Overly Long or Absent Crawl-Delay

Googlebot ignores the Crawl-delay directive (it manages crawl rate based on server response signals). Bing and some smaller crawlers do respect it. Setting a very long crawl delay for all user agents, including Googlebot, doesn't affect Google but can still cause problems on smaller sites where Bing crawls are a meaningful traffic source. More practically: many developers set a Crawl-delay when they mean to set one via Google Search Console's crawl rate settings, which is the right place for Googlebot.

5. Not Keeping robots.txt in the Site Root

This sounds obvious but it catches sites that move to subdirectories or use reverse proxies. robots.txt must live at exactly yourdomain.com/robots.txt. A file at yourdomain.com/en/robots.txt is not a valid robots.txt file; crawlers won't check for it there. For sites using hreflang with subdirectory locale paths, this catches people more often than you'd expect.

How Do You Test and Validate Your robots.txt File?

Google provides a built-in robots.txt tester inside Search Console under Settings > robots.txt. It shows you the live file Google is reading, lets you test any URL against the current rules, and flags syntax errors. Use it every time you change the file, not just after major rewrites.

For sites I work with, the testing workflow is: make the change in staging, test the specific URLs you care about in a local robots.txt validator, deploy, then verify in GSC's URL Inspection tool that Googlebot can still access your critical pages. That extra step in GSC catches the "our CDN cached the old robots.txt" problem that trips up production deployments.

If you're on WordPress, the implementation is different because WordPress generates a virtual robots.txt dynamically. The file doesn't exist on disk. See our WordPress robots.txt guide for the specific approaches, including how to override the virtual file via plugin and how WordPress handles AI crawler directives.

robots.txt vs noindex: When to Use Each

This distinction is worth a section of its own because getting it wrong has asymmetric consequences. Here's the rule:

• Use robots.txt Disallow when you want to save crawl budget by preventing Googlebot from spending time on low-value URLs (internal search results, filter combinations, admin paths). The pages can stay in the index if they're already there; you just stop crawl waste.
• Use noindex when you want a page removed from search results entirely. Remember: Googlebot needs to crawl the page to see the noindex tag, so it can't be Disallowed in robots.txt simultaneously.
• Use both together only as a temporary state, for example during a migration where you've already removed pages from the index and now want to stop crawl waste too.

The combination of Disallow + noindex is also a common audit finding. Sites that applied noindex to thin content via plugin, then later added a Disallow to "reinforce" it, end up with Googlebot unable to verify the noindex tag. Pages in this state stay in the index indefinitely because Google can't confirm the removal signal. Use our SERP features tool to check if affected pages are still showing in search.

Does robots.txt Affect Your Visibility in AI Search?

Yes, but differently than you might expect. For Google's AI Overviews, Googlebot is what matters, not Google-Extended. AI Overviews draw content from Google's existing index, so if Googlebot can crawl and index your pages, you're eligible for AI Overview citations. Google-Extended is used for Gemini and other Google AI products beyond Search.

For third-party AI systems like Perplexity, ChatGPT Browse, and others that retrieve content in real time, their respective crawlers are what matter. If you've blocked PerplexityBot, Perplexity can't include your content in its responses. This is the "AEO opt-out problem" I see on about one in five audits: sites that blocked all unfamiliar bot user agents in 2023 as a defensive measure are now invisible to AI answer engines that could be sending them traffic.

The strategic move for most content publishers: allow all answer-engine crawlers, block training-data crawlers. Pair your robots.txt with an llms.txt file to signal which content you want AI systems to prioritise. It's a lightweight addition that costs nothing and can meaningfully improve how AI systems represent your site.