Robots.txt block or noindex tag, which when, and why not both?
- Published November 6, 2026
- Linked from 5 answers
- In a cluster of 77
Use a robots.txt block when you do not want Google to crawl a URL at all, and a noindex rule when you want a page kept out of search results. Do not use both on the same URL, because the block stops Google from ever reading the noindex. Google’s guidance on blocking indexing states the condition: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.”
What each tool controls
| robots.txt disallow | noindex rule | |
|---|---|---|
| Controls | Whether Google fetches the URL | Whether the page can appear in results |
| Where it lives | One file at the root of the host | A robots meta tag in the page, or an <!–INLINECODE0–> HTTP header |
| URL can still appear in results | Yes, if other pages link to it | No, once Google has crawled it and seen the rule |
| Fits | Endless parameter combinations, internal search result URLs, admin areas, scripts you never want fetched | Thank-you pages, thin archives, staging copies that are reachable, pages that must not rank |
Two details trip people up. Google does not support a noindex rule inside robots.txt, so writing Noindex: there does nothing. And a robots.txt block is not a privacy tool: a blocked URL that other pages link to can still be listed, and whether a page in robots.txt can still get indexed shows how. The choice comes up again for auto-generated pages from the start and for filtered and faceted URLs.
What each looks like
A robots.txt rule names a path pattern. These lines, with your own paths in place of the examples, tell every crawler not to fetch an internal search section or an admin area:
User-agent: *
Disallow: /internal-search/
Disallow: /admin/A noindex rule travels with the page itself. Google’s blocking guidance shows it as an HTTP response header, which also works for PDFs and other files that are not HTML pages:
HTTP/1.1 200 OK
X-Robots-Tag: noindexThe same rule can sit in the page’s head as a robots meta tag with the value noindex. Either way, the page has to stay crawlable for the rule to be read.
Why the combination backfires
A noindex rule only works when Google crawls the page and reads it. Add a robots.txt block and Google stops crawling, so the rule goes unread, and the URL can stay in results as a bare listing. That explains a page that “keeps showing up even though it is noindexed”: the noindex is real, but nothing lets Google see it.
Choosing in practice
- You want a page out of search results. Add noindex, leave the page crawlable, and wait for Google to recrawl it. Remove the robots.txt block if one exists.
- You want Google to stop fetching a large set of useless URLs. Disallow them in robots.txt, and accept that some may still be listed by URL if something links to them.
- You want both outcomes for the same URLs. You cannot hold both for long. Adding the block after the pages have dropped out stops Google from seeing the noindex from then on, and a URL that other pages link to can return as a bare listing. If staying out of results is the priority, keep the pages crawlable with noindex; if crawl load is the priority, block them and accept that some may be listed by URL.
Google’s crawl budget guidance adds one more rule for large sites. Use robots.txt for pages or resources you do not want crawled at all, not to temporarily shift crawling toward other pages.
Add a comment