Robots.txt for WordPress, The Complete 2026 Guide

1. What Is the WordPress Robots.txt File?

Robots.txt controls crawling, not indexing. A Disallow rule stops a compliant crawler from fetching a URL, but Google can still index a blocked URL if other pages link to it. To keep a page out of Google's index, use a noindex meta robots tag on a crawlable page, not a robots.txt block.

The Robots Exclusion Protocol became a formal IETF standard in RFC 9309 in September 2022, according to the IETF. Googlebot, Bingbot, and every major search crawler obey it. For WordPress sites, robots.txt matters for two reasons: it protects crawl budget, the number of URLs a crawler fetches per visit, and it now carries your policy toward AI crawlers such as GPTBot and ClaudeBot.

WordPress powers 43 percent of all websites, per W3Techs data from 2025. That scale means the WordPress default robots.txt is the most widely served robots.txt configuration on the web, and most site owners have never looked at it.

2. Where Is Robots.txt Located in WordPress?

This virtual behavior confuses many site owners. You connect via FTP, look in the web root next to wp-config.php, and find no robots.txt file. The file still loads in a browser because the WordPress core function do_robots() builds the output at request time. The rewrite rules route any request for /robots.txt to that function, but only when WordPress runs at the domain root, not in a subdirectory install.

The priority order works like this:

1. Physical file first. If robots.txt exists in the web root, the web server (Apache or Nginx) serves it directly. WordPress never runs.
2. Virtual file second. With no physical file, WordPress generates the virtual robots.txt through do_robots() and the robots_txt filter hook.
3. SEO plugins modify the virtual file. Yoast SEO and Rank Math hook into robots_txt to change the output, or write a physical file when you use their editors.

The default virtual output on a clean WordPress 6.x install contains three lines plus a Sitemap line when a sitemap is registered: a User-agent wildcard, a Disallow for /wp-admin/, and an Allow exception for /wp-admin/admin-ajax.php.

3. What Should a WordPress Robots.txt Contain in 2026?

Here is the complete recommended file. It allows AI crawlers, the right call for most publishers who want AI visibility, and blocks only training-specific and low-value bots. Adjust the AI decisions using Section 6.

# Recommended WordPress robots.txt, 2026
# Search crawlers: full access except admin and junk URLs

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?replytocom=

# AI answer engines: allowed for citation visibility
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Gemini training opt-out, does NOT affect Google Search
User-agent: Google-Extended
Disallow: /

# Common Crawl training corpus, optional block
User-agent: CCBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap_index.xml

Compare this against what WordPress ships out of the box:

The Sitemap directive accepts absolute URLs only. If Yoast SEO or Rank Math generates your XML sitemap, point the directive at the plugin's sitemap index, usually /sitemap_index.xml for Yoast and /sitemap_index.xml for Rank Math. Need a sitemap first? Build one with the free XML sitemap generator tool.

4. How Do You Edit Robots.txt in WordPress?

Method 1: SEO Plugin Editor (Easiest)

Yoast SEO: go to Yoast SEO, Tools, File editor, then click Create robots.txt file. Yoast writes a physical file to the web root, which then overrides the virtual file. Rank Math: go to Rank Math, General Settings, Edit robots.txt. Rank Math edits the virtual file through the robots_txt hook, so no physical file is created, and the editor locks if a physical file already exists.

Method 2: Physical File via FTP or SFTP

Create robots.txt in a plain text editor with UTF-8 encoding. Connect with an FTP client such as FileZilla, or your host's file manager. Upload the file to the web root, the same directory that holds wp-config.php. From that moment the web server serves your file directly and WordPress, including every plugin, loses control of robots.txt output.

Method 3: The robots_txt Filter Hook (Developers)

The robots_txt filter hook lets a child theme or custom plugin modify the virtual file programmatically. This survives plugin changes and keeps rules in version control:

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "\nUser-agent: Google-Extended\nDisallow: /\n";
    $output .= "\nSitemap: https://yourdomain.com/sitemap_index.xml\n";
    return $output;
}, 10, 2 );

After any edit, load yourdomain.com/robots.txt in an incognito window to confirm the change is live. Caching plugins and CDNs sometimes cache robots.txt, so purge the cache if the old version persists.

5. Which Pages Should WordPress Sites Block?

Block these URL patterns:

• /wp-admin/, the dashboard has zero search value. Keep Allow: /wp-admin/admin-ajax.php because themes and plugins call admin-ajax.php from the front end, and blocking it can break rendering.
• Internal search URLs (/?s= and /search/), infinite keyword combinations create infinite thin URLs that waste crawl budget.
• Cart, checkout, and account pages on WooCommerce stores, these are per-user pages with no ranking potential.
• Comment reply parameters (?replytocom=), a classic WordPress duplicate URL generator.

Never block these:

• /wp-content/uploads/, blocking it removes every image from Google Images and strips image results traffic.
• CSS and JavaScript files, Google has warned since 2015 that blocking rendering assets "can result in suboptimal rankings" because Googlebot renders pages like a browser.
• /wp-includes/, older tutorials recommend this block, but wp-includes contains jQuery and other scripts pages need to render. WordPress removed this block from the default file years ago.
• Pages you want deindexed, robots.txt blocking prevents Googlebot from seeing a noindex tag, so the URL can stay indexed as a snippetless result.

Special Cases: Multisite, Staging, and Subdirectories

Three WordPress setups need extra care. On WordPress multisite, each subdomain site serves its own virtual robots.txt, but subdirectory sites share the network root's file, so one file governs every site in the network. On staging environments, do not rely on a sitewide Disallow: / rule; use HTTP authentication or the WordPress "Discourage search engines" setting plus noindex headers, because a leaked staging URL blocked by robots.txt can still get indexed. On subdirectory installs such as example.com/blog/, WordPress cannot serve the virtual file at all, since crawlers only read robots.txt at the domain root; you must maintain a physical file there.

6. Should You Block AI Crawlers in WordPress?

Each AI company runs distinct bots with distinct purposes, so a blanket decision is a mistake. Know what each User-agent token does before you block it:

The trade-off is real on both sides. Originality.ai tracking showed over 35 percent of the top 1,000 websites blocking GPTBot by mid-2024, mostly news publishers protecting licensing use. Meanwhile answer engines send growing referral traffic: Perplexity and ChatGPT cite sources with links, and a cited page earns clicks a blocked page cannot.

A balanced 2026 policy for most WordPress publishers: allow GPTBot, ClaudeBot, and PerplexityBot to stay visible in AI answers, and disallow Google-Extended and CCBot to limit pure training use at low cost. This mirrors the approach in our AI search optimization hub, where AI crawler access is step one of any answer engine strategy. Remember that robots.txt is voluntary compliance; it deters compliant bots, not scrapers.

7. How Do You Test Your WordPress Robots.txt?

Syntax errors fail silently. A missing colon, a BOM character from a Word processor, or a Disallow rule one line under the wrong User-agent group can open admin URLs to crawling or block your entire site. Google Search Console's robots.txt report shows the last fetched version, its size, and any parse errors, and lets you request a recrawl after edits.

Rule-level testing matters more. The Robots Exclusion Protocol resolves conflicts by the longest matching rule, which is why Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one path. Wildcard patterns add a second failure mode: Disallow: /*?replytocom= matches any URL containing that parameter, and a careless wildcard like Disallow: /*.php can block admin-ajax.php and break rendering. Testing each critical URL against your file is the only way to confirm the logic.

Also verify AI crawler groups separately. GPTBot, ClaudeBot, and PerplexityBot each obey only the User-agent group that names them, falling back to the wildcard group when no named group exists. A file with a specific GPTBot group must repeat any wildcard rules inside it, because a crawler that matches a named group ignores the wildcard group entirely. This is the single most common error in AI crawler configurations we audit.

Run your file through the free robots.txt tester for WordPress now. Paste your robots.txt or enter your domain, test any URL against every major User-agent including Googlebot, GPTBot, ClaudeBot, and PerplexityBot, and get a line-by-line verdict on which rule matched and why. It takes under a minute and catches the errors that cost rankings.