Faceted Navigation and the Crawl Space It Opens

Faceted navigation multiplies. Five filters with four options each, freely combinable, is 1,024 reachable combinations per category — and if the filters can be applied in different orders, or repeated with multiple values, the number grows past that. Every one of those combinations is a URL that your own navigation links to, which means every one of them is discoverable.

The problem is not that these URLs exist. It is that your site links to all of them, so a crawler has a legitimate path to every single one, and almost none of them are pages you would want in search results.

Why this is different from ordinary parameters

A tracking parameter is one extra address per page and the content is identical. A facet combination is a new set of items, so the content genuinely differs — which means the usual answer of “canonical everything to the clean URL” is weaker here. You are declaring a duplicate relationship between pages that are not duplicates, and a canonical between non-equivalent pages is frequently ignored, because the canonical is a hint that the engine weighs against the content itself.

The other difference is scale. Parameter variants of one page number in the tens. Facet combinations across a large catalogue number in the millions, and they are cheap for a crawler to enumerate and expensive for you to serve. The query parameter toolkit applies, but the sequencing has to change because of the volume.

Sort the facets into three tiers

The work is deciding, per facet and per combination, which of three treatments applies. Do this once, deliberately, and write it down — the failure mode is not any individual choice but the absence of a policy, which leaves the default (everything linked, everything crawlable) in force.

Tier one: facets that deserve a real page. A small number of high-demand single-facet views. /laptops/gaming/, /dresses/red/. These are searched for by name, they have enough inventory to be substantial, and they can carry their own title, description and copy.

Give them a path, not a parameter. That is what separates a landing page from a filtered view, and it is what lets you treat them as first-class pages with self-referencing canonicals and sitemap entries.

The list should be short and chosen from demand evidence rather than generated from your facet schema. A generated set of “SEO landing pages” for every facet value is how sites end up with thousands of thin near-duplicates.

Tier two: single-facet views with no independent demand. ?sort=price, ?in_stock=1, ?rating=4plus. Useful to a shopper, not something anyone searches. Keep them as parameters, canonical them to the unfiltered category, and — this is the part people skip — make sure the canonical target actually contains the items. A canonical from a filtered view to the parent category is defensible; a canonical to the site’s homepage is not, and gets ignored.

Tier three: multi-facet combinations. Two or more filters applied together. These should be reachable by a user clicking, and not reachable by a crawler following links. That distinction is the whole technique, and it is the next section.

Making a link clickable but not crawlable

The clean approach is to stop emitting the combination as a crawlable link once a threshold is crossed. Concretely: the first facet a user applies produces a normal <a href>; subsequent facet links on an already-filtered page are rendered as controls that a browser can act on and a crawler has no href to follow.

Options, roughly in order of robustness:

Render subsequent facets as form controls. A <select> or checkbox set inside a form that submits with GET. There is no anchor, so there is no link to follow. Works without JavaScript, degrades cleanly, and it is honest — a facet is a form control, semantically.

Apply additional facets client-side with a fragment. #colour=blue never reaches the server and produces no new URL from the crawler’s perspective. Requires JavaScript and changes the shareability of the filtered state, which may be unacceptable.

Use a POST for facet application. Effective and usually a worse user experience, because the filtered state is not linkable at all.

What does not work reliably: adding rel="nofollow" to facet links. As of this writing it is treated as a hint for crawling purposes rather than a directive, so the URL may still be discovered and crawled. It also does nothing about the URL being indexed once discovered from elsewhere. The distinction between the link attribute and the page-level directive is worth being precise about — see noindex and nofollow on the same page.

The robots.txt layer, and when to add it

Once the links are gone, discovery slows dramatically. But URLs already in the index, and URLs linked from elsewhere, will keep being requested. A Disallow pattern stops the requests:

User-agent: *
Disallow: /*?*colour=
Disallow: /*?*sort=
Disallow: /*&

That last line is worth explaining: /*& matches any URL containing an ampersand, which by construction means two or more parameters. It is a blunt but effective way to block multi-facet combinations while leaving single-parameter URLs crawlable. Verify it against your real URL patterns before deploying, because it will also catch legitimate two-parameter URLs — a paginated single facet, for instance — and you may need to be more specific.

Order of operations matters, and it is the reverse of what feels natural. Blocking first is the mistake:

1. Stop emitting crawlable links to tier-three combinations.
2. Add canonicals on tier-two views to the parent category.
3. Wait, and watch the indexed count for the parameterised patterns fall.
4. Then add Disallow rules for whatever is still being crawled in volume.

Blocking at step one means the canonicals in step two are never read, and any facet URL with an external link stays indexed as an undescribed address. That is the failure in robots.txt cannot deindex a page, applied to a very large number of URLs at once.

Empty facet combinations

Any combinatorial facet space generates combinations with zero results — red widgets in size 14 with next-day delivery. Those pages return 200 with an empty list, which is the textbook trigger for reclassification as a soft 404.

Two things to do:

• Do not link to empty combinations. Facet counts are usually already computed for the UI; if a value’s count is zero, render it disabled rather than as a link.
• Serve an appropriate response when one is requested anyway. Either a 404, or a 200 with genuine content — the filters applied, what was removed, and suggestions with results.

Watching the surface

Two measurements tell you whether the policy is holding.

Crawl volume on parameterised paths, from logs. Group verified crawler requests by whether the path contains a query string, and track the ratio over time. A site where the majority of crawl requests go to facet URLs is spending its crawl on pages it does not want indexed. Server logs are the only place that ratio is visible.

Indexed count for the parameter patterns. Search Console’s coverage report, filtered to the facet path patterns. You want this trending down after the changes, and flat afterwards.

Neither number has a target value that generalises. What you are looking for is direction and stability — a facet surface that stops growing is the outcome, and that is achievable in a way that “no facet URLs are ever crawled” is not.