Reading Log Files for Crawl Evidence

Server access logs are the only source that records what a crawler requested and what your server returned. Every other source is a report about that — sampled, aggregated, delayed, or derived. When a crawl report and a log disagree about what was served, the log is the primary record.

That makes logs the right tool for a specific class of question: not “why is this page not ranking” but “was this URL fetched, when, and what did it get.”

What a log line contains

The combined format most servers default to:

66.249.66.1 - - [08/Jul/2026:14:22:07 +0000] "GET /blogs/some-post/ HTTP/1.1" 200 8412 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Reading left to right: source IP, timestamp, request method and path, status code, response size, referrer, user agent.

The two fields that do the work are the status code — what you actually served, as distinct from what your config says you serve — and the user agent plus IP, which together identify the client.

What logs do not contain: the response body, the response headers, or anything about rendering. A log tells you a 200 was served; it cannot tell you the 200 was a soft 404.

Verify the client before you count anything

The user-agent string is self-reported and trivially forged. A substantial share of requests claiming to be a major search crawler are not, and any analysis that filters on the string alone is measuring a mixture.

The verification method for the major crawlers is a reverse DNS lookup followed by a forward lookup:

# reverse: what hostname claims this IP?
dig +short -x 66.249.66.1
# → crawl-66-249-66-1.googlebot.com.

# forward: does that hostname resolve back to the same IP?
dig +short crawl-66-249-66-1.googlebot.com
# → 66.249.66.1

Both steps are required. A reverse lookup alone can be spoofed by anyone controlling the reverse DNS for their own IP range; the forward confirmation closes that, because the hostname is in a domain the crawler operator controls.

The operators also publish IP range lists as JSON, which is the practical approach for bulk log analysis — resolve once against the ranges rather than doing two DNS lookups per line. Refresh the lists periodically; they change.

Without this step, an apparent spike in crawl activity is as likely to be a scraper as a search engine, and the responses differ. This matters directly when you are considering rate limiting, as in what a 429 does to your crawl rate — throttling the wrong client is the whole risk.

The questions logs answer well

Was this URL ever fetched? The binary question that resolves most “is it indexed yet” confusion. If a URL has never appeared in the logs, no directive on it has taken effect and no content on it has been assessed.

grep ' /blogs/some-post/ ' access.log | awk '{print $4, $9}'

What status code is this URL actually returning to crawlers? Distinct from what it returns to you, because edge rules, geographic routing, and cache state can differ.

awk '$7 ~ /^\/blogs\// {print $9}' access.log | sort | uniq -c | sort -rn

Which URLs consume the most crawl requests? The single most valuable output, because it is almost always surprising. The list is routinely dominated by generated surfaces rather than content — internal search, facet combinations, calendar pages, parameter variants.

awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -30

If the top of that list is query-string URLs, the problem is the one in faceted navigation and the crawl space it opens, and the log is your evidence for prioritising it.

Are redirects being followed, and how many hops? Group 3xx responses by path and cross-reference each Location against the next request from the same client. Long chains show up as a sequence of 301s in quick succession from one IP, each one a hop nobody deployed deliberately.

Which URLs are being requested that you thought were gone? Filter for 404s with a non-empty referrer field. Those are live links pointing at dead URLs, and the referrer tells you where from. This is the highest-value 404 report there is, because it distinguishes dead URLs that matter from dead URLs that nobody visits — the input to the decision in retiring a page that still has inbound links.

awk '$9 == 404 && $11 != "\"-\"" {print $7, $11}' access.log | sort | uniq -c | sort -rn

Are there URLs being crawled that you did not know existed? Compare requested paths against your sitemap. Requests for paths absent from your own URL inventory point at something generating links — a plugin, a retired template, a partner integration.

The questions logs answer badly

Anything about ranking. Logs contain no ranking information.

Anything about rendering. If content only appears after JavaScript runs, the log shows the HTML fetch and, separately, fetches of the scripts. It cannot tell you what the rendered page looked like.

Anything about the reason for a crawl. You can see that a URL was fetched; the scheduling logic is not observable.

Crawl “budget” as a number. Request counts vary with site size, change rate, and response speed. There is no threshold in a log that means anything on its own — what is meaningful is the distribution across your URL types and how it changes after you act.

Practical constraints

Get the logs from the layer that answers your question. A CDN in front of your origin serves cached responses without the origin seeing them, so origin logs under-report crawl activity by however much is cached. For crawl analysis you want edge logs. For application-behaviour analysis you want origin logs. Knowing which of the two you have is a prerequisite for reading anything in them.

Retention is usually the binding constraint. Default rotation is often 7 or 14 days, which is too short to see a pattern across a crawl cycle. Extending retention, or shipping a filtered subset — verified crawler requests only — to longer storage, is the cheap fix. Filtered crawler traffic is a small fraction of total volume.

Anonymise before you analyse. Access logs contain IP addresses, which are personal data in several jurisdictions. For crawl analysis you need the verified-crawler flag, not the address, so resolve and discard.

Sample when the volume is large. A day of logs from a busy site is tens of gigabytes. For most questions a representative day is enough; for the 404-with-referrer report you want the full window, because the interesting entries are rare.

What to do with the output

The pattern that makes log analysis worth the effort is comparing two lists: URLs you want crawled (your sitemap, per what an XML sitemap is actually for) and URLs actually being crawled (the logs).

The difference in each direction is a finding. Sitemap URLs never requested are a discovery problem. Requested URLs absent from the sitemap are a surface you did not intend to expose. Both are actionable, and neither is visible from either source alone.