How Do You Verify a Scraper Is Actually Respecting Your Crawl-Delay?

Crawl-delay is a non-standard robots.txt directive: some crawlers read it and pace their requests accordingly, and at least one major search engine has documented that it ignores the field entirely in favor of its own adaptive rate logic. Publishing the directive tells you nothing on its own about whether any given client is following it — that’s a claim you verify from your own logs, the same way you’d verify any other claimed behavior you can’t directly observe.

Why the directive alone proves nothing

A Disallow rule and a Crawl-delay rule sit in the same file but work differently. Whether either one has any effect depends entirely on the requesting client choosing to read the file and comply — the point made in full in does robots.txt actually stop a paid scraping API. Crawl-delay inherits the same limitation and adds one more: even among clients that fetch and parse robots.txt, support for this specific field is inconsistent and undocumented for many of them. Publishing Crawl-delay: 10 is a request. Whether it changed anything is an empirical question about one specific client’s actual request pattern.

Measuring the interval, not trusting the number

The only record of what a client actually did is the access log, per reading log files for crawl evidence. Verifying a claimed crawl-delay is a matter of isolating one verified client’s requests and measuring the gap between them.

grep 'crawl-66-249-66-1\|scrapervendor' access.log | awk '{print $4}'

More usefully, extract just the timestamps for one verified client and compute the differences:

awk '$0 ~ /verified-client-string/ {print $4}' access.log \
  | tr -d '[]' \
  | while read -r ts; do date -d "${ts/:/ }" +%s; done \
  | awk 'NR>1{print $1-prev} {prev=$1}'

The output is a list of gaps, in seconds, between consecutive requests from that one client. If Crawl-delay: 10 was set and the client honors it, the gaps cluster at or above 10. If they don’t, either the client doesn’t support the directive, or — worth checking separately — the requests you isolated aren’t all from the same client, which is exactly the verification problem the log-reading post covers: confirm the IP against the hostname before trusting that a run of requests shares an origin.

What an inconsistent result usually means

A distribution that’s mostly at or above the requested interval with occasional shorter gaps is common and usually not evidence of non-compliance — concurrent connections, retried requests, or a client that batches lookahead fetches can produce brief clusters even from an otherwise-compliant crawler. What’s worth investigating is a sustained pattern well under the requested delay across the whole sample, not a handful of short gaps in an otherwise-compliant run.

A distribution with no relationship at all to the requested value — gaps that look identical to what the same client showed before the directive was added — is the actual finding: that specific client isn’t reading Crawl-delay, whatever it might read elsewhere in the file.

Concurrency is a separate variable from interval

A Crawl-delay value describes spacing between requests from a single sequential worker. It says nothing about a client that runs several connections in parallel — a crawler that opens four simultaneous connections and paces each one at the requested delay produces four times the aggregate request rate the directive was written to limit, while every individual connection, measured on its own, appears fully compliant.

This is worth checking for specifically before concluding a client honors the directive: group log entries not just by client identity but by source port or connection where that’s available, and check whether the aggregate rate across all of a client’s concurrent connections matches what a single-connection reading of the gaps would suggest. A compliant-looking per-connection interval can still add up to a request rate the directive was meant to prevent, if nothing constrains how many connections that client opens at once.

What to do if a client isn’t honoring it

Crawl-delay has no enforcement mechanism attached to it; a client that doesn’t support the field isn’t violating anything, because nothing obligates it to. If the request rate from a specific client is a genuine problem rather than a preference, the tools that actually constrain rate — a 429/503 response strategy, per what a 429 does to your crawl rate, applied against a verified client identity, per rate-limiting scraper traffic without blocking real crawlers — act on every request regardless of whether the client reads robots.txt at all.

What you can and can’t conclude

You can conclude, from measured intervals, whether one specific verified client’s behavior changed after you published a delay value. You can’t generalize that result to any other client, and you can’t conclude a directive “isn’t working” from a handful of requests — a short sample can look non-compliant purely from concurrency or retries. Measure over a window long enough to average that out before concluding anything.

A note on sample size

A handful of requests from a low-traffic client can produce gap measurements that look compliant or non-compliant almost by chance — one long gap because the client happened to be idle, one short gap because two of its lookahead fetches landed close together. The measurement is only meaningful across a window with enough requests that a couple of outliers don’t dominate the distribution; for an infrequent client, that can mean waiting days rather than hours before drawing a conclusion either way.

The durable point

Crawl-delay is a request written into a file, not a rate enforced by the protocol. Whether it did anything is answerable only by measuring the actual interval between a verified client’s requests in your own logs — the directive’s presence is not itself evidence, in either direction.