What Does the Noindex Tag Do?

The noindex tag tells Google and other search engines not to include a page in their index. Googlebot still crawls the page to read the directive, but the page is removed from search results. It is added as a <meta name="robots" content="noindex"> tag in the HTML head or as an X-Robots-Tag in the HTTP response header.

The key word there is "still crawls." A lot of people confuse noindex with blocking. They are not the same thing. When you add noindex to a page, Googlebot visits it, reads the tag, and then withholds that URL from the index. The page is seen but not shown in search results. This is actually useful, because it means Google can still follow links on the noindexed page and pass signals to pages that are linked from it.

This is a genuinely different outcome from blocking the page with robots.txt. I will cover that distinction properly in a moment, because mixing up these two methods is the most common and damaging mistake I see.

How Do You Implement Noindex Correctly?

There are three ways to add noindex. The right method depends on the page type.

Meta Robots Tag (HTML pages)

For any standard HTML page, place the tag inside the <head> element:

<meta name="robots" content="noindex, follow" />

The follow directive tells Googlebot to follow outbound links even though the page is not indexed. This is almost always what you want. If you write just content="noindex", Googlebot defaults to following links anyway, but being explicit is cleaner practice. Use noindex, nofollow only for pages you absolutely do not want to pass any link signals from, such as administrator login pages or private staging environments.

X-Robots-Tag HTTP Header (PDFs and non-HTML files)

For PDFs, images, and other non-HTML files, you cannot add a meta tag. Instead, set the X-Robots-Tag in the HTTP response header:

X-Robots-Tag: noindex, follow

This is configured at the server level. In Apache, you add it to .htaccess or the server config. In Nginx, you add it to the location block. For Google-specific pages, you can specify the user agent: X-Robots-Tag: googlebot: noindex.

Google Search Console Removal Tool

For urgent removals, Google Search Console's URL Removal tool suppresses a URL from results for six months. But it does not actually noindex the page. After six months, the URL can reappear. For permanent removal, always combine the removal request with a proper noindex tag or by returning a 404 or 410 status code.

Verification step: After adding noindex, use the URL Inspection tool in Google Search Console to confirm Google can see the tag. Fetch the URL, expand "More Information," and check "Indexing allowed: No." If it still shows "Yes," check whether your CMS or plugin is overriding the tag.

What Is the Difference Between Noindex and Robots.txt?

Robots.txt blocks crawling entirely. Noindex allows crawling but removes the page from search results. If you block a page with robots.txt, Googlebot cannot read the noindex tag on it, so the page may still appear in Google's index from backlinks or internal links pointing to it. Never use robots.txt to try to hide a page from the index.

This is the mistake that breaks sites. An SEO blocks a staging version or a thin category page with robots.txt, thinking it will keep those URLs out of the index. But Google already saw those URLs via sitemaps or links before the robots.txt block was added. Google now cannot recrawl them to remove them, because it cannot access them. They sit in the index, sometimes for months.

Method Blocks crawling? Removes from index? Links followed? Best for
noindex meta tag No Yes Yes (with "follow") Thin content, staging pages, pagination
Robots.txt Disallow Yes No (unless already removed) No Resource-heavy pages, CMS back-end URLs
404 / 410 status No Yes (after recrawl) No Permanently deleted pages
Canonical tag No Signals preferred URL Yes Duplicate content consolidation

The rule I follow: use noindex to remove pages you want removed from results but can still be crawled. Use robots.txt only to block pages you genuinely do not want Google to access for resource or privacy reasons, not as a removal method.

For a deep look at robots.txt configuration, including AI crawler directives, see the robots.txt for WordPress guide. For managing duplicates specifically, the canonical tags guide covers when rel=canonical beats noindex.

Which Pages Should You Noindex?

The primary use cases for noindex are pages that exist for functional reasons but should not compete for search traffic or waste crawl budget.

Thin and Duplicate Content

Tag pages in WordPress, faceted navigation pages in e-commerce (like "blue shoes, size 9, sort by price"), print versions of articles, and paginated archive pages beyond page 2 or 3 are classic noindex candidates. These pages have low information density and can dilute topical authority. Noindexing them concentrates Google's attention on your best content.

On this site, I use the keyword cannibalization checker to identify pages competing against each other before deciding which ones need noindex or consolidation. It often surfaces pages I forgot existed.

Internal Search Results

If your site has a search function, the results pages (URLs with ?q= or /search/) are almost always thin, near-duplicate pages. They should be noindexed. Google's own guidance is to use the noindex tag on search result pages to prevent index bloat.

Thank-You and Confirmation Pages

Thank-you pages after form submissions, order confirmation pages, and checkout success pages have no search intent value. Noindex them. There is also a tracking benefit: if you use these pages as conversion goals, having them indexed means direct visits can inflate conversion data.

Staging and Development Environments

This is where robots.txt and noindex work together. Use robots.txt to block the entire staging subdomain from Googlebot crawls, AND add a site-wide noindex via the server. Belt and suspenders. I have seen staging sites outranking production because one of these two layers was missing.

Paginated Pages

For blog archives or product category pages with large page counts, consider noindexing pages beyond page 2 or 3. These pages rarely rank and often compete with earlier paginated pages. The canonical approach is to point all paginated pages to the first page (though this has trade-offs), or to simply noindex beyond a threshold. Check your crawl budget data in Search Console's Crawl Stats report to see how much Googlebot is spending on paginated content.

Which Pages Should Never Have Noindex?

This sounds obvious, but I have audited sites where these were accidentally noindexed, often by a CMS plugin or a staging migration that was never reversed.

Never noindex: your homepage, money pages (product, service, pricing), cornerstone content guides, hub pages, and any URL you want to rank for. Check every critical page annually with the URL Inspection tool.

The most common accidental noindex scenario: a developer working on a WordPress theme flips the "Discourage search engines" setting in WordPress settings, which adds a site-wide noindex. They flip it off after launch, but one month later you notice the homepage is still noindexed because the Yoast meta box was individually set for a few key pages during a testing phase. Happens more than it should.

Run a site crawl with the entity checker tool or a dedicated crawler like Screaming Frog. Filter for pages returning noindex. Any unexpected URL on that list warrants immediate investigation.

Google confirmed that internal links on noindexed pages do pass PageRank, as long as the page is crawlable (not blocked by robots.txt). This is why noindex, follow is almost always preferable to noindex, nofollow. If you have a high-traffic category page with links to product pages, noindexing it with "follow" lets those product pages still receive link equity from it.

External backlinks pointing TO a noindexed page do not pass PageRank effectively, because Google considers noindexed pages low-value. If you have a page with strong backlinks that you want to noindex for content reasons, consider whether a 301 redirect to a related indexed page might be a better option to preserve the link equity.

How Long Does It Take Google to Remove a Noindexed Page?

After adding noindex, Google must recrawl the page to process the directive. For frequently crawled pages, removal from search results typically takes one to four weeks. For thin or rarely crawled pages, it can take two to three months. Use Google Search Console's URL Inspection tool to request a recrawl and speed up processing.

There is a nuance: Google distinguishes between "removing from index" and "removing from search results." A page can be removed from the index quickly but still appear in the cache or in related searches. For pages with sensitive content that need to be removed urgently, the URL Removal tool in Search Console is your fastest option for temporary suppression while you wait for the noindex directive to be processed.

DataForSEO shows "noindex" at 590 searches per month in the US, with a 23-point keyword difficulty score. That tells you this query is dominated by exact-match technical documentation, and ranking here takes a focused, precise guide rather than a content farm approach.

What Are the Most Common Noindex Mistakes?

After crawling hundreds of sites, I see the same five mistakes repeatedly.

Mistake 1: Blocking the Page in Robots.txt as Well

If you block a URL in robots.txt AND add noindex, Googlebot cannot read the noindex tag because it cannot access the page. The URL stays in the index until Google decides on its own to remove it (which can take over a year). Solution: if you want removal, make the page crawlable.

Mistake 2: Using Noindex on Pages with Strong Backlinks

Noindexing a page with 40 referring domains to clean up thin content wastes all those link signals. A 301 redirect to a stronger page would consolidate that equity instead. Run a backlink check before noindexing any page that has been live for more than a year.

Mistake 3: Site-Wide Noindex Left On After Staging

The WordPress "Discourage search engines" setting or a global noindex in Yoast or RankMath set during development. The developer forgets to remove it at launch. The site quietly sits deindexed for weeks. I always include a Search Console check in any site launch checklist for exactly this reason.

Mistake 4: Applying Noindex to Paginated Pages that Shouldn't be

Noindexing all paginated pages except page 1 makes sense for some sites, but it can prevent Google from discovering content that only appears deeper in the paginated set. If your best products only appear on page 8 of a category, noindexing those pages hides them from discovery. Use internal links as a safety net.

Mistake 5: Forgetting to Remove Noindex After a CMS Migration

When migrating from WordPress to another CMS, noindex settings are sometimes carried over as hardcoded meta tags in templates, even after the original setting is removed. A full crawl post-migration is essential. Check the technical SEO audit checklist for a complete migration verification process.

Check Your Canonical and Indexation Status

Use the schema generator to audit your pages for conflicting directives, missing canonical tags, and misapplied noindex values.

Open Schema Generator →

Noindex Versus Other Crawl Management Signals

Noindex sits in a family of crawl and index management directives. Here is how they stack up:

Directive Effect Applied via
noindex Removes page from index, allows crawl Meta robots or X-Robots-Tag
nofollow Tells bots not to follow outbound links Meta robots or X-Robots-Tag
nosnippet Prevents text snippet and cache link in results Meta robots or X-Robots-Tag
noimageindex Prevents images on the page from being indexed Meta robots or X-Robots-Tag
Robots.txt Disallow Blocks Googlebot from crawling robots.txt file
Canonical tag Signals preferred URL for duplicate content HTML link element

The XML sitemaps guide covers the relationship between sitemaps and noindex: pages in your sitemap should always be indexable. Remove noindexed URLs from your sitemap to avoid confusing Google's crawl signals.

If you want to do a full audit of your index management signals, including checking which pages have conflicting canonical and noindex directives, the robots.txt tester covers the crawl side. Your next step on the index side is reviewing Google Search Console's Coverage report under the Indexing section.