When should filtered/faceted URLs be indexed vs blocked?

Index a filtered view when people search for it as a thing in its own right and the page can serve them; keep the rest out of crawling, once you have checked that every product can still be reached without them. “Red rain jackets” or “packable rain jacket size medium tall” can be searches with their own demand, and a filtered category page answers them well. Other combinations are not: sort orders, price bands, and stacks of three or four filters create URLs no one searches for. Google’s faceted navigation guidance frames the first decision the same way: decide whether you need the filtered URLs to be indexable at all, and treat the two groups differently.

Two groups, two tools

Choosing the tool for the second group

The purpose decides the tool, and mixing them causes problems.

• To stop crawling, use robots.txt disallow rules on the filter parameters. This saves server load and crawl effort on large sites. Blocked URLs can still appear in search if other pages link to them, but they are not fetched.
• To keep a crawlable URL out of the index, use noindex, and leave the URL crawlable so Google can see the tag. This does nothing to reduce crawling, because Google has to fetch the page to read it.
• Do not combine them on the same URL. A robots.txt block hides the noindex from Google, so the page can stay listed with no content. The two tools do different jobs, laid side by side in when to block crawling and when to use a noindex tag.

Google’s faceted navigation guidance gives a robots.txt example of the first tool, blocking the filter parameters while leaving an unfiltered listing open:

user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$

The * stands for any run of characters and the $ marks the end of the URL. Replace the parameter names with your own, and leave out any filter you decided to index.

Google’s guidance also notes that a canonical from filtered views to the unfiltered category can gradually lower how much the filtered versions are crawled, which makes it a gentler option when you are not sure which filters matter.

Picking the filters to index

1. List the attributes shoppers use in searches: type, color, size, material, use.
2. Check search demand for type plus each attribute.
3. Index the single-attribute views with demand, and the occasional two-attribute view with clear demand, provided each shows enough products to be worth landing on and the count of indexable combinations stays manageable.
4. Before blocking the rest, check that every product page can still be reached through crawlable category and pagination links that do not pass through the filters you block, and that the filters still work for shoppers on the page. Google’s guidance describes the same arrangement: allow crawling of “just the individual items’ pages along with a dedicated listing page that shows all products without filters applied.” The crawl rule and the shopping experience are separate decisions.
5. Keep everything else out of crawling.

Index the filtered pages a searcher would type, and keep the combinations only a filter panel could produce out of Google’s way.