Faceted Navigation SEO: Control Filter URLs
Filters for colour, size, price and brand are useful to shoppers but can create an almost unlimited number of URLs. Crawlers waste time on them and miss new pages. This guide gives Google’s two strategies, prevent crawling or optimise the URLs, with the exact options and their trade-offs.
- Why filters explode
- Prevent crawling options
- Allow and optimise
- 404 for empty results
/shoes/?color=red&size=9 /shoes/?size=9&color=red /shoes/?color=red&color=red
Filters implemented as URL parameters can create a practically unlimited number of URLs, and crawl capacity spent on them leaves less for new pages. If filtered URLs do not need to appear in Search, prevent crawling (robots.txt, URL fragments, canonical or nofollow). If they do, use & separators, keep filter order consistent and return a real 404 for empty or nonsensical combinations.
Why faceted navigation causes crawling problems
- Filters that change how items are displayed are common, but as URL parameters they can create a practically unlimited number of URLs.
- Crawlers cannot tell whether a new filtered URL is useful without fetching it, so they often request many useless URLs first.
- That crawl capacity leaves less for discovering new, useful pages.
- Every added filter multiplies the combinations, and rendering each page costs server resources.
Background: crawl budget.
Pick a strategy first
| If filtered URLs… | Then |
|---|---|
| do not need to appear in Search | Option 1: prevent crawling |
| should be crawled and indexed | Option 2: allow and optimise the URLs |
Option 1: prevent crawling
User-agent: * Disallow: /*?*color= Disallow: /*?*size=
The robots.txt example is my own illustration; Google’s page does not give wildcard syntax. See robots.txt syntax rules.
Which prevention method works best
Google says canonical and nofollow are generally less effective in the long term than robots.txt and fragments.
Option 2: allow crawling and optimise
- Use the standard & to separate parameters. Commas, semicolons and square brackets are hard for crawlers to recognise as separators.
- If filters live in the path, keep their order consistent and prevent duplicate filters.
- Return an HTTP 404 for combinations with no results, duplicated or nonsensical filters, or nonexistent pagination URLs.
/shoes/?color=red&size=9
/shoes/?filters=color:red,size:9
Handle faceted navigation step by step
List your filter URL patterns
Find which parameters create variations.
Which filtered pages deserve search traffic?
Most do not (my own view).
Prevent crawling of the rest
Robots.txt or fragments.
Keep key pages crawlable
Item pages and an unfiltered listing.
Return 404 for nonsense combos
At the URL itself.
The 404 rule
Return a 404 where the problem was encountered; do not redirect to a generic error page. Single-page apps may not be able to return proper 404s; Google points to its single-page app guidance (see JavaScript SEO basics).
Notes for WordPress and WooCommerce
- Layered-nav plugins create query parameters (my own note). Review which are crawlable.
- Plugin settings may add noindex or canonical to filters; verify the output.
- Be careful with noindex plus robots.txt on the same URLs. See noindex vs robots.txt. Google’s faceted page does not discuss noindex.
Troubleshooting
| Symptom | Likely cause (my own diagnosis) | Fix |
|---|---|---|
| Crawl stats full of parameter URLs | Open filters | Block patterns |
| New pages slow to be crawled | Capacity spent on filters | Reduce filter crawling |
| Duplicate filtered pages indexed | No canonical or block | Canonicalise or block |
| Empty filter pages return 200 | Soft 404 behaviour | Return 404 |
Checklist
Decide
- List filter parameters
- Choose strategy
- Keep valuable pages
- Document rules
Prevent
- Robots.txt rules
- Fragments where possible
- Canonical to unfiltered
- Nofollow on all links
Optimise
- Use & separators
- Consistent filter order
- No duplicate filters
- 404 for empty results
Monitor
- Crawl Stats
- Indexing report
- Server load
- Retest changes
What Google’s page does not say
It says nothing about noindex, the = separator, sitemaps, Search Console parameter tools, how long canonical or nofollow changes take, whether robots.txt blocking affects already-indexed URLs, or thresholds for what counts as overcrawling.
Key terms
- Facet
- A filter such as colour or size.
- Parameter
- A query string value that changes the page.
- Fragment
- URL text after #.
- Overcrawling
- Crawling many low-value URLs.
Common mistakes
Letting all filters crawl
Wastes capacity.
Partial nofollow
Must be on every link.
Odd separators
Hard to parse.
Inconsistent order
Duplicates.
Redirecting empties
Hides errors.
Blocking item pages
Loses products.
Related guides
Faceted navigation questions
Why are facets a problem?
Parameter filters can create practically unlimited URLs that waste crawl capacity.
How do I stop crawling?
Robots.txt disallow, URL fragments, canonical or nofollow.
Which prevention is strongest?
Google says robots.txt and fragments are generally more effective long term than canonical and nofollow.
Do fragments help?
Google generally does not support fragments for crawling or indexing, so they have no effect either way.
What separator should I use?
The standard &.
What for empty filter results?
Return HTTP 404.
Should I redirect empties to an error page?
No. Serve the 404 at that URL.
Does nofollow need to be everywhere?
Yes, every link to that URL must have it to be effective.
The problem description, both options, effectiveness comparison and 404 rule come from Google’s page. The robots.txt example, step routine, WordPress notes, troubleshooting and checklists are my own.