Why is a page not indexed? Check sitemaps, robots rules and canonical signals

Choose the page and record what should happen

Start with one complete address copied from the public page. Record whether the content should be public and discoverable, the expected preferred URL, its publication date and any recent domain, permalink or SEO-setting change. Keep an old address separately if it redirects.

For a fictional workshop, the maintenance guide might be intended for Search while a private client document is not. An excluded private document is not a defect to fix by removing protection. Agree the publication purpose before changing a technical setting.

Record when each observation was made. A saved report from before yesterday’s correction and a live page response from today describe different states. This date distinction prevents a working correction from being repeatedly undone.

Check the sitemap that actually owns the URL list

Identify the generator: WordPress core, an SEO integration or the website platform. Use its documented sitemap address. A filename such as wp-sitemap.xml or sitemap_index.xml is a clue, not proof that a particular component owns the current response.

Open the sitemap index and the relevant child sitemap. Locate the complete page address, checking protocol, hostname, path and trailing form. An index lists other sitemap files; an article will normally be found inside one of those children rather than directly beside the index entries.

Google’s build-and-submit guidance calls for fully qualified URLs and accurate modification information. Use the preferred public addresses, not editor previews or temporary signed links. Do not refresh lastmod merely to make unchanged articles look new; it should reflect a significant update.

If an expected article is missing, check its publication state, the generator’s content-type inclusion and any intentional exclusion. Compare the saved setting with the current XML response. Keep one accountable source of the list instead of installing another generator over an unexplained first one.

Read the sitemap report as a retrieval report

In the authorised Search Console property, open the Sitemaps report for the exact submitted address. Success means Google fetched and read that sitemap without errors. Discovered pages counts parsed page URLs; it is not an indexed-page total.

Check Last read and the property scope. A report for a different protocol or hostname can conceal the sitemap you expected. The report covers sitemaps submitted through the report or API; absence from that list does not establish that Google never found another sitemap by another route.

For Couldn’t fetch or parsing errors, read the specific failure and test the actual sitemap response. A login screen, missing-file response or HTML error page is not valid evidence merely because the URL ends in .xml. Resolve that retrieval problem before interpreting a stale discovered count.

Check the page’s access and HTTP response separately

Open the article without an administrator session and confirm its final address, response and content. A sitemap can work while one child page redirects incorrectly, requires a login or serves an error. Record the redirect route separately from the final page.

Google’s HTTP-status documentation explains that a 2xx success does not guarantee indexing. An empty page or error message served with success can still be treated as a soft 404. Inspect what was returned, rather than judging only the status number.

Use the broken-link audit workflow to investigate missing and redirected destinations. For a crawler access refusal, correlate its URL and time with the responsible security or hosting evidence. Do not disable the whole firewall or expose private staging to clear a single diagnostic.

Distinguish robots.txt from noindex and private access

Google’s robots.txt guidance concerns crawling. A disallowed address can still appear in Search based on links from elsewhere. Robots.txt is not a reliable privacy boundary or a substitute for deciding which public content should be indexed.

Google’s noindex documentation covers a robots meta tag or X-Robots-Tag response header. Google must be able to retrieve the resource to see that instruction. Blocking crawling and expecting a newly added noindex to be read creates conflicting assumptions.

Inspect the article’s HTML head and response headers, including rules applied by its SEO integration or server. If exclusion is intentional, retain it and correct an inappropriate sitemap listing. If it is accidental, identify the setting owner and scope a correction to the intended public content.

Also identify which response carries a warning. Yoast’s HTTP-header specification documents noindex, follow on its XML sitemaps without preventing their consumption. Noindex on that XML resource is different from noindex on an article inside it. Do not remove all indexing headers based on a test of the wrong URL.

For genuinely private content, preserve access controls. The private staging guide explains why search instructions and authorised access are separate responsibilities.

Align canonical preference with the intended address

Google’s canonical guidance identifies sitemap inclusion as a weaker preference signal than redirects or rel=canonical. Keep the sitemap, canonical declaration and internal links consistent with the preferred page. Google can still choose another representative of similar content.

If an article declares the homepage as canonical, investigate the template or integration; do not assume all articles should consolidate there. If the intended page moved, review its relevant redirect and old internal links. The www and canonical-address guide covers hostname consistency without replacing this page-level check.

Compare current output with Google’s recorded version

The URL Inspection documentation distinguishes indexed data from the live test. Indexed data reports Google’s recorded state and selected canonical; compare its crawl date with the correction date. A live test examines current access but cannot predict canonical selection or verify every indexing condition.

For the fictional guide, suppose the current response has the intended canonical and no accidental exclusion, while the recorded crawl predates the repair. Preserve both observations. They support “the current output changed; Google’s stored result needs later rechecking,” not “Google has now indexed the corrected page.”

After a confirmed correction, use the authorised recrawl route when appropriate and monitor the exact URL later. An accepted request is not an indexing guarantee. If technical checks are coherent but the page remains excluded, review the recorded reason and the usefulness and distinct purpose of the content instead of repeatedly changing sitemap filenames.