What an XML Sitemap Is Actually For

A sitemap is a list of URLs you would like a crawler to know about. That is the whole function. It assists discovery — it does not cause indexing, it does not confer importance, and it does not compensate for a page being unreachable by any other means.

The most useful way to think about it: a sitemap tells a crawler that a URL exists and roughly when it last changed. Everything else about how that URL is treated is decided elsewhere.

What it can fix

Newly published URLs being found faster. The primary legitimate use. A URL in a sitemap can be discovered before a crawler happens to follow a link to it, which matters when publication timing matters.

URLs with few internal links. A page three or four clicks deep with one inbound internal link is crawled infrequently. A sitemap entry gives it a second discovery path. Note that this treats a symptom — the real fix is the internal link graph, per internal linking as plumbing.

Large sites where crawling the whole link graph is slow. A million-URL catalogue is not fully traversed on every crawl. A sitemap lets the crawler prioritise by what changed.

Non-HTML resources. Images, video, and PDFs that are not linked prominently.

Signalling change. An accurate lastmod gives a crawler a reason to re-fetch a URL it has already seen, which is the one thing internal links cannot express.

What it cannot fix

A page will not be indexed because it is in a sitemap. Inclusion is not a request that is granted. If the page is thin, duplicated, or noindexed, listing it changes nothing.

A sitemap does not override a Disallow. If robots.txt blocks the URL, the crawler does not fetch it. Listing a blocked URL is a contradiction and Search Console will report it as one. The general shape of this problem is in robots.txt cannot deindex a page.

A sitemap does not override a noindex. The directive is in the response; the sitemap is not.

Orphan pages are still orphans. A URL reachable only from the sitemap can be crawled, and it has no internal links, which is itself a signal about how important you consider it. Sitemap inclusion is not a substitute for being linked.

It will not stop duplicate URLs being indexed. Listing only the canonical version is correct and helpful — it is one of the signals the engine weighs — but it does not by itself resolve duplication. See canonical tags are a hint, not a command.

The fields

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/blogs/some-post/</loc>
    <lastmod>2026-07-09</lastmod>
  </url>
</urlset>

<loc> — required. Must be the fully qualified canonical URL, on the canonical host and protocol, in the canonical trailing-slash and case form. Every mismatch here is you asking the crawler to fetch a redirect.

<lastmod> — optional, and the only optional field worth emitting. Documented behaviour is that it is used if it is consistently accurate. That condition is the whole thing: a lastmod that updates on every deploy for every URL, whether or not the content changed, is noise, and a crawler that learns your dates are meaningless stops using them. Set it from the content’s actual modification time, and if you cannot do that reliably, omit it.

<changefreq> — ignored. Google has stated it does not use it.

<priority> — ignored, and it never worked the way people hoped. It was never a ranking input; it was a hint about relative crawl priority within your own site, and it is not used.

Emitting the two ignored fields is harmless and clutters the file. Leaving them out is cleaner.

Limits and the index file

A single sitemap holds up to 50,000 URLs and 50 MB uncompressed. Past either limit, split into multiple files and reference them from a sitemap index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-1.xml</loc>
    <lastmod>2026-07-09</lastmod>
  </sitemap>
</sitemapindex>

Splitting by content type rather than arbitrarily is worth doing even below the limit, because it makes the coverage reporting per-sitemap useful — an index of products, posts and category pages as separate files lets you see which type has an indexing problem. That diagnostic value is the main reason to split early.

Gzipping is supported and the 50 MB limit applies to the uncompressed size.

Submitting it

Reference it from robots.txt. One line, no authentication, works for every crawler:

Sitemap: https://example.com/sitemap-index.xml

Submit it in Search Console (and the equivalent for other engines) for the per-sitemap coverage reporting. That reporting is the actual reason to submit — discovery works from the robots.txt reference alone.

The Sitemap: directive is independent of User-agent grouping in robots.txt; it applies globally regardless of where in the file it sits. Conventionally it goes at the end.

The failure modes

Redirecting URLs in the sitemap. The most common. Every entry that returns a 3xx is a URL you have declared as preferred while simultaneously declaring it is not. This accumulates silently after any URL change — the sitemap generator is often the last thing updated. Audit it as part of the post-migration checklist in preserving links through a site migration.

404s and noindexed URLs in the sitemap. Same class. Every entry should be a crawlable, indexable, 200-returning canonical URL. That is a testable assertion.

A stale sitemap. Generated once, never regenerated. Recognisable by a lastmod on the index file that is months old.

Parameterised or non-canonical variants included. Usually a generator that builds the list from a crawl of the site rather than from the content database, so it picks up whatever URLs it found.

A sitemap listing URLs on another host. Cross-host entries are only accepted under specific verification conditions and are otherwise ignored. If you have several hosts, give each one its own sitemap.

The test worth automating

# every URL in a sitemap, with its status code
curl -s https://example.com/sitemap-1.xml \
  | grep -oP '(?<=<loc>)[^<]+' \
  | while read -r u; do
      printf '%s %s\n' "$(curl -so /dev/null -w '%{http_code}' "$u")" "$u"
    done | grep -v '^200 '

The output should be empty. Anything it prints is a URL you are recommending that does not resolve cleanly, and this is cheap enough to run in CI against a generated sitemap before deploy — which catches the class of problem where a URL change lands and the sitemap does not.

For a large site, sample rather than checking all 50,000, and check the whole thing on a schedule instead.