Sitemaps and indexing: the fundamentals
Updated June 25, 2026 · 5 min read
An XML sitemap is a file that lists the pages you want search and AI engines to discover and consider. Indexing is the separate step where an engine decides to store a crawled page so it can appear in results or answers. A sitemap helps discovery, but it does not force indexing - the page still has to be crawlable, canonical, and worth keeping.
Key takeaways
- A sitemap aids discovery; it doesn't guarantee a page gets indexed.
- Only include canonical, indexable, 200-status URLs you actually want found.
- Keep lastmod dates honest - misleading freshness signals erode trust.
- Indexing depends on crawlability, canonicals, and quality, not just sitemap inclusion.
- Monitor the gap between submitted and indexed pages to catch silent losses.
What a sitemap does (and doesn't) do
A sitemap is a discovery aid: it hands engines a clean list of the URLs you consider important, with optional hints like last-modified dates. It's especially useful for large sites, new sites with few inbound links, and pages that are hard to reach through normal navigation. What it does not do is compel indexing - submitting a URL is a suggestion, not a command. An engine still decides whether each page is worth storing.
What belongs in a sitemap
A sitemap should be a confident statement of your best, canonical pages - not a dump of every URL that exists.
- Only canonical URLs - never include pages that canonicalize elsewhere.
- Only indexable pages - exclude anything noindexed or blocked by robots.txt.
- Only live pages - no redirects, no 404s, no soft errors.
- Honest lastmod dates that reflect real content changes.
Why a page might not get indexed
Inclusion in a sitemap is no protection against the common reasons pages stay out of the index. Understanding them is most of the battle.
- It's blocked from crawling (robots.txt) or marked noindex.
- It canonicalizes to another URL, so the engine indexes that one instead.
- It's a near-duplicate of an existing page and gets folded into it.
- It's judged too thin or low-value to be worth indexing.
Indexing for AI engines
AI answer engines have their own crawling and retrieval. The same fundamentals apply - if a page can't be crawled and isn't clearly canonical, it won't be a reliable source to cite. Keeping your sitemap accurate and your indexing clean isn't just an SEO chore; it's what makes your best pages eligible to be retrieved and quoted in AI answers. Monitor the difference between what you submit and what actually gets indexed, so a silent deindexing doesn't go unnoticed.
Frequently asked questions
Does submitting a sitemap force Google to index my pages?
No. A sitemap helps engines discover pages, but indexing is a separate decision based on crawlability, canonicalization, and quality. A submitted URL can still go unindexed if it's blocked, duplicative, or judged low-value.
Should every page be in my sitemap?
No - only canonical, indexable, live pages you want found. Including redirects, noindexed pages, or duplicates sends mixed signals and wastes the engine's attention. Keep it a clean list of your best URLs.
How do I find pages that aren't getting indexed?
Compare the URLs you submit against what's actually indexed using a search console's coverage or pages report. A growing gap usually points to canonical conflicts, accidental noindex tags, or thin content.
Put this into practice — free.
Get your free AI-visibility audit and see where engines find you today.