Jump to the answers
SEO Talks

All clusters

Does a page in robots.txt still get indexed somehow?

  • Published November 6, 2026
  • Linked from 1 answer
  • In a cluster of 77

Yes. A URL blocked in robots.txt can still appear in Google’s results when other pages link to it. The block stops Google from fetching the page, so Google cannot read its content or any noindex rule on it, but Google can still learn that the URL exists from links. Its guidance on blocking indexing says that when a page is blocked by robots.txt, “the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.”

What such a listing looks like

Because Google never read the page, it has no content to describe it with. The result can show the URL with little or no description, sometimes with a note that no information is available for the page. The page is known to Google only as an address and the words used in links pointing to it.

That is why robots.txt is the wrong tool for keeping a page private or out of results. It controls crawling; it does not remove a URL from Google’s knowledge, and it can leave you with the worst combination: a listing you did not want, without a description you could control. To match each tool to its job, use robots.txt block or noindex tag, which when.

How a blocked URL gets found

Source Example
Links on your own site A navigation link or footer link to a blocked section
Links from other sites Someone links to a blocked PDF or a staging URL
Your own sitemap The blocked URL is listed for submission

Google’s Page indexing help notes that robots.txt-blocked URLs in the report are “probably intentionally blocked”, so their presence in the report is not a fault by itself. It becomes one when a blocked URL is a page you wanted out of search.

How to keep a page out of results instead

  1. Remove the robots.txt rule that blocks the page.
  2. Add a noindex rule with a robots meta tag or an X-Robots-Tag HTTP header.
  3. Run the live test in URL Inspection to confirm Googlebot can fetch the page and sees the noindex, then wait for Google to recrawl it. Request indexing is not the way to hurry this: Google’s URL Inspection help says indexing cannot be requested for a page the live test finds non-indexable, and a page with noindex is one. Google’s blocking guidance says a revisit can take months for some pages, which is what step 5 is for.
  4. If the content must not be reachable at all, put it behind a login; a noindex rule hides a page from search results, not from people who have the link.
  5. For an urgent case, Search Console’s Removals tool can hide a URL temporarily. Google’s Removals help says a successful request “lasts only about six months”. Pair it with noindex or removal of the page.

Related posts:

  1. Does internal linking or backlinks matter more for getting a page indexed? For getting a page indexed, internal links do more of the work you control. Google has to find a URL before it can crawl and index it, and Search Console's......
  2. At what point is a “not indexed” page a content problem vs a domain-trust problem? Count how far the problem reaches before deciding. If one page or a small group is not indexed while comparable pages on the same site are, the cause sits with......
  3. Discovered vs Crawled “not indexed”, which is the worse signal and why? Neither is worse in every case; they fail at different stages and need different fixes. "Discovered - currently not indexed" means Google knows the URL but has not crawled it......
  4. Does linking to a page from every other page help it rank or look spammy? It depends on whether a visitor on any page might need that page. For main sections, key categories and contact pages, a sitewide link is ordinary navigation, and Google's ecommerce......
  5. Does a faster thin page beat a slower rich one for the same query? Not by being faster alone. Google's page experience documentation is explicit that it aims to show the most relevant content even when page experience is weaker, and that good Core......
  6. Does a wrong canonical tag actually deindex a page? It can remove the page from search results, though not as a punishment. A canonical that names another URL tells Google this page is a duplicate of that one. If......
  7. Does adding a city name to every page help local rankings or look spammy? It helps on a page that is about that city, and it does nothing useful on a page that is not. Naming the place you serve is how a reader,......
  8. Does adding an author byline change anything for a non-YMYL page? It helps readers, and it fits what Google's guidance asks, but it is not a ranking switch. Google's guidance on creating helpful, reliable, people-first content includes a "who" question in......
  9. Does adding an FAQ section help depth or just pad the page? It helps when its questions are ones the rest of the page does not answer, and pads the page when it restates the article as questions. An FAQ can be......
  10. Does adding unique copy to every product page actually pay off? On the products people search for, it can; across every product in a large catalog, the hours are unlikely to come back. Unique copy is an investment per page, and......
  11. Does an ecommerce category page need a wall of text to rank? No. A category page ranks by showing the products a shopper is looking for and helping them choose, and a long block of keyword-heavy text below the grid does neither.......
  12. Does an older page with authority beat a fresh one, when? It depends on whether the search needs current information. For a stable question, such as how something works or what a term means, an established page with links and a......
  13. Does internal linking reduce how many backlinks a page needs? It can, on a site that already has standing, but it does not replace links from other sites. Internal links show Google which pages your site treats as important and......
  14. Does page speed actually move rankings, or just help conversions? Both, but not equally. Google's ranking systems use Core Web Vitals, which measure loading speed, responsiveness, and visual stability, so speed is a real input. Google is just as clear......
  15. How do you find the one thing a ranking page does that you’re missing? Read the ranking page and yours side by side, as a searcher would, and ask which one you would trust to answer the question and why. The decisive difference is......

Comments on this answer

Add a comment

Your email address will not be published. Required fields are marked *