Your XML sitemap is a claim about which pages matter. When crawl data and index data disagree with it, the mismatches tell you where discovery is leaking.

Sitemap hygiene: crawled but unlisted, listed but unindexed

A sitemap is a claim. Every URL you put in it says "this page matters to me"; every URL you leave out says "this one does not". Sitemap hygiene is the check that the claim matches reality — and two mismatches show up again and again: pages the crawler found that are not in the sitemap, and sitemap URLs the search engine has chosen not to index. Both are worth investigating, because each points at a different disagreement about what your site considers important.

The two mismatches

They sound symmetrical, but they mean opposite things.

The first mismatch means your sitemap is incomplete: a page exists and works, but you are not signalling it. The second means the sitemap is making a promise the search engine declines: you claim the page matters, but the engine disagrees. Fixing one without the other leaves the claim still wrong — just wrong in the other direction.

Why it matters for crawl budget

A sitemap does not only help discovery; it steers attention. Search engines use it to prioritise which URLs to crawl and re-crawl, especially on larger sites where crawl capacity is finite. Two consequences follow:

• Missing from the sitemap — an important page may be crawled less often. Updates to it are discovered later, through links alone.
• Listed but never indexed — every such URL spends attention on a page that will not appear in search. Fill the sitemap with enough of them and the signal "my sitemap describes my good pages" weakens.

This is why the fix is rarely "add everything to the sitemap" or "remove everything that is not indexed". Both mismatches ask you to make a decision per page first.

How to check this

In Be Seen Digitally, open Pages. Each URL carries its sitemap state and, when Google Search Console is connected, its index state, so the two mismatches are direct filters:

1. Crawled but not in sitemap. Filter the grid to pages that are live in the crawl but not present in the sitemap. Check the status and the internal link count before judging: a 200 page with healthy in-links that is absent from the sitemap is a gap; a redirect or a near-duplicate is often deliberately excluded and fine.
2. In sitemap but not indexed. Filter to sitemap URLs whose Search Console state is not indexed, and read the verdict Google reports — "Crawled – currently not indexed" points at quality or duplication, while a canonical conflict points at a different problem entirely (see the canonical conflict guide).

If Google Search Console is connected, the Dashboard tracks how many pages are in the sitemap and how many are indexed over time, so you can see whether the gap is shrinking after you fix it. For pages that turn out to be weakly linked, the Link graph shows where they sit in the structure.

What to do about it

Work the two lists in opposite directions — one decides what to add, the other what to fix or withdraw.

Crawled but not in sitemap:

1. Decide whether the URL should be indexed. Thank-you pages, filtered views and internal search results should stay out — ideally with a noindex. Only genuinely indexable pages belong in the sitemap.
2. Fix the generation. For important pages, check your CMS or sitemap plugin: a post type toggled off, or a static sitemap that has not been regenerated, is the usual cause. See the crawled but missing from the sitemap guide.
3. Regenerate and verify. After the change, confirm the URLs actually appear in the served XML file — not just in the plugin's settings screen.

In sitemap but not indexed:

1. Rule out conflicting signals. Check for a stray noindex meta tag or X-Robots header, and for a canonical pointing somewhere else. The sitemap then lists a page that the page itself asks search engines to ignore.
2. Judge the content. Thin or heavily duplicated pages are often consciously skipped. Add unique material, or merge the page into a stronger one and redirect — see the in the sitemap but not indexed guide.
3. Strengthen the internal linking. A page several clicks deep with few in-links competes poorly. Linking to it from a well-connected page is often enough to change the outcome.
4. Withdraw what you cannot justify. If a URL is in the sitemap, not indexed, and you cannot name a reason it should rank, removing it concentrates the sitemap's claim on pages you stand behind.

In both lists, turn each decision into a tracked item from the recommendation, so the fixes and their outcomes are visible on your Tasks board.

Closing

Sitemap hygiene is a comparison, not a document: the sitemap against the crawl, and the sitemap against the index. The crawl shows what exists; the index shows what the search engine accepted. Where those two disagree with the sitemap, you have a specific, finite list of URLs and a decision for each one. Re-run the comparison after the fixes and the remaining gap should be pages you deliberately excluded.