Link Plumbing

One robots.txt Per Host, Not Per Folder

A crawler fetches robots rules from exactly one place per host: /robots.txt at the root. There is no such thing as a robots.txt for a subdirectory. If your CMS serves a file at /blog/robots.txt, that file is a static asset with an interesting name — nothing fetches it as policy, and nothing in it applies to anything.

This matters because the setup is common. A marketing landing page sits at the root, a WordPress install or a docs generator sits at /blog/, and each was configured by a different person with a different robots file in mind. Only the root one exists as far as crawling is concerned, and its rules govern every path on the host, including the ones the other tool thinks it controls.

Host, not path

The scope of a robots file is the host — scheme, hostname and port. https://example.com/robots.txt governs every path under https://example.com/. It does not govern:

  • https://blog.example.com/ — different hostname, so a different file, fetched from https://blog.example.com/robots.txt.
  • http://example.com/ — different scheme, technically a different origin. Most sites redirect HTTP to HTTPS, in which case the crawler follows the redirect and uses the HTTPS file. If you serve HTTP directly, it has its own file.
  • https://example.com:8443/ — different port.

So the first diagnostic question when someone says “the blog has its own robots.txt” is whether the blog is a subdomain or a subdirectory. Subdomain: two hosts, two real files, and the root file genuinely does not reach the blog. Subdirectory: one host, one file, and whatever the blog plugin generates is decorative.

You can confirm which situation you’re in without touching the CMS. Fetch the root file and read it. Then fetch the nested one and notice that its contents have no bearing on anything:

curl -s https://example.com/robots.txt
curl -s https://example.com/blog/robots.txt   # served, but never consulted as policy

If the two disagree, the root wins by default, because the nested one was never in the running.

The failure this produces

The interesting part isn’t the inert file. It’s what the root file is doing to the blog’s URLs while everyone assumes the blog is governed by its own permissive rules.

A pattern worth recognising: a root file written for a static landing page blocks tracking-parameter URLs.

User-agent: *
Disallow: /*?
Disallow: /*utm_*

The intent is reasonable — keep parameterised duplicates out of the crawl. But Disallow: /*? matches every URL on the host containing a question mark, and that now includes /blog/some-article?utm_campaign=partner, the exact URL you handed a partner for a cross-promotion.

Here is the mechanism, in order:

  1. The partner publishes a link to /blog/some-article?utm_campaign=partner.
  2. A crawler discovers that URL from the partner’s page.
  3. It checks the root robots.txt, matches Disallow: /*?, and does not fetch the page.
  4. Because it never fetched the page, it never sees the rel="canonical" pointing at /blog/some-article.

The consequence is the opposite of what the rule was written to achieve. A blocked URL can still be indexed — URL-only, with no content, discovered purely from the inbound link — because robots.txt controls crawling, not indexing. That’s covered in more detail in Robots.txt Cannot Deindex a Page. And because the canonical was never read, no consolidation happens: the parameterised URL and the clean URL are two unrelated things as far as the index is concerned.

A canonical tag needs the page to be crawlable to do anything at all. Blocking a duplicate is not a substitute for canonicalising it; it actively prevents canonicalising it.

What to do with parameter URLs instead

If the URLs serve the same content as a clean equivalent, let them be crawled and let the canonical do its job:

  • Self-referencing canonical on the clean URL.
  • On the parameterised URL, a canonical pointing at the clean URL. Most platforms emit this automatically for tracking parameters, since the parameter doesn’t change what’s rendered.
  • No robots.txt rule at all for those parameters.

The crawler fetches the parameterised URL once, reads the canonical, consolidates onto the clean URL, and stops treating them as separate. That is the entire designed purpose of the tag.

Reach for Disallow when the URL space is genuinely unbounded and you need to stop the crawl from walking into it — faceted filter combinations, calendar pagination, internal search results. See Faceted Navigation and the Crawl Space It Opens and Query Parameters That Multiply Your URL Count for where that line sits. A handful of UTM variants on articles you promote deliberately is not an unbounded space; it’s a few extra URLs that consolidate cleanly if you let them.

Auditing a two-CMS host

A short pass that catches most of these:

1. Establish the host boundary. Is the second property a subdomain or a subdirectory? Everything else follows from the answer.

2. Read the root file as though it were written for the whole host, because it was. Take each Disallow line and ask what it matches under /blog/, /docs/, /help/ — not just under the paths the author had in mind. Wildcard rules written for one section routinely reach into others; the matching rules are in How Robots.txt Rules Are Matched.

3. Test real URLs, not patterns. Take an actual promoted URL with its actual query string and check it against the live file with a robots tester. Pattern-matching by eye is where this goes wrong, because * and $ behaviour is easy to misremember.

4. Check what’s already indexed URL-only. Pages a crawler was blocked from fetching but indexed anyway show up in coverage reporting as excluded-by-robots or indexed-though-blocked. Those are the receipts for a rule that fired when you didn’t mean it to.

5. Decide who owns the file. On a single host, one team’s plugin regenerating a robots file is a change to the whole site’s crawl policy — if it regenerates the root file. If it regenerates a nested one, nothing happens, which is its own kind of surprise when someone expects a rule to take effect and it never does.

The through-line: robots.txt is a host-level policy file that many stacks encourage you to think of as a per-app config. Treat it as infrastructure shared by every application on the hostname, and the surprises stop.

httpcanonicals