How to Use Google Search Console to Diagnose Indexing Issues

When an important page does not appear in Google Search, Google Search Console is usually the best place to begin your investigation.

Search Console can show whether Google knows about a URL, whether it has crawled the page, whether indexing is allowed, which canonical URL Google selected, and why the page may have been excluded from the index.

However, Search Console does not automatically tell you which issues are important or exactly what change should be made. It reports what Google observed. You still need to interpret the information in the context of your website.

The most effective approach is to use two Search Console features together:

• The Page Indexing report for identifying site-wide patterns
• The URL Inspection tool for investigating individual pages

This guide explains how to use both tools to diagnose indexing issues without treating every excluded URL as an error.

What Is an Indexing Issue?

Google Search generally processes pages through three broad stages:

• Crawling: Google discovers and downloads a page.

Indexing: Google processes the page and may store information about it in its index.

Serving: Google may show the indexed page when it is relevant to a search.

An indexing issue occurs when a page that should appear in Google is not added to the index or when Google indexes a different URL from the one you intended.

A page can fail to reach the index because:

• Google has not discovered it
• Google cannot crawl it
• The page contains a noindex directive
• The URL redirects
• The page returns an error
• Google considers it a duplicate
• Google selected a different canonical URL
• The page was crawled but not selected for indexing
• The page has weak discovery or internal-linking signals
• Search Console helps you identify which of these situations may apply.

Start With the Page Indexing Report

Open your Search Console property and navigate to:

Indexing → Pages

The Page Indexing report provides an overall view of the URLs Google knows about on your website. It separates them into two broad groups:

• Indexed
• Not indexed
• The report also groups non-indexed pages by reason.

You may see categories such as:

• Page with redirect
• Not found
• Excluded by noindex
• Blocked by robots.txt
• Duplicate without user-selected canonical
• Duplicate, Google chose a different canonical
• Crawled – currently not indexed
• Discovered – currently not indexed
• Server error
• Soft 404

These categories are diagnostic starting points. They are not all warnings that require action.

For example, an old URL that correctly redirects to a new page should normally appear as Page with redirect. A deleted product may correctly return 404. A duplicate parameter URL may correctly remain outside the index.

Your task is to determine whether the affected URLs are:

• Intentionally excluded
• Unimportant duplicates
• Old or deleted pages
• Important pages that should be indexed

Do Not Judge the Report by the Total Number of Excluded Pages

Website owners sometimes assume that a healthy site should have no excluded pages.

That is not realistic.

Many websites contain URLs that should not be indexed separately, including:

• Redirected URLs
• Filter and sorting parameters
• Tracking URLs
• Account pages
• Shopping-cart pages
• Internal search pages
• Duplicate category paths
• Thank-you pages
• Expired content
• Deleted pages
• Non-canonical variants

A high number of non-indexed URLs is not automatically a technical SEO problem.

Instead of focusing only on the total, ask:

• Are important service or product pages excluded?
• Has the number changed unexpectedly?
• Did exclusions increase after a migration or redesign?
• Are entire page templates affected?
• Are URLs in the wrong exclusion category?
• Does the pattern match what the website is designed to do?

The report becomes useful when you interpret it by page type and business value.

Open an Indexing Reason and Review Example URLs

Click one of the reasons listed under Why pages aren’t indexed.

Search Console will show more information about the issue and a sample of affected URLs.

Review the examples carefully.

Look for patterns such as:

• All affected pages use the same template
• All URLs belong to one directory
• Only product variants are affected
• The problem began after a specific date
• Important URLs are mixed with low-value URLs
• The pages were recently migrated
• The affected URLs share the same canonical or robots directive
• A shared pattern usually points toward a shared cause.

For example, if dozens of service pages suddenly appear under Excluded by noindex, the problem may come from a template-level SEO setting rather than separate page edits.

If a group of URLs appears under Page with redirect, check whether those redirects are intentional and lead to the correct destinations.

The Page Indexing report identifies the category. It does not replace page-level investigation.

Use URL Inspection for an Individual Page

When you find an important affected URL, enter the exact address into the URL Inspection bar at the top of Search Console.

The tool provides information about Google’s indexed version of that page.

Depending on the URL, you may see messages such as:

• URL is on Google
• URL is not on Google
• URL is unknown to Google
• Page is not indexed
• Page is indexed, but has issues

The report may also show:

• The last crawl date
• Which crawler was used
• Whether crawling was allowed
• Whether indexing was allowed
• The page fetch result
• The referring page
• Sitemap discovery
• User-declared canonical
• Google-selected canonical

These fields help you determine where the indexing process may have failed.

Understand the Difference Between the Indexed Test and the Live Test

This is one of the most important distinctions in URL Inspection.

The initial inspection result describes what Google knows about the indexed or previously processed version of the page.

The Test Live URL option checks the page as it exists now.

These two results may differ.

For example:

• Google’s indexed version may still show an old noindex tag
• The live page may now be indexable
• Google may have previously encountered a server error
• The current server response may now be successful
• The old canonical may point elsewhere
• The live canonical may have been corrected

A successful live test does not mean the page has already been indexed. It means the current page may be technically eligible for indexing.

Google still needs to recrawl and process the updated version.

Check Whether Crawling Is Allowed

Inside URL Inspection, review the crawl information.

If Search Console says crawling is not allowed, investigate the website’s robots.txt file.

A rule like this could block an important directory:

User-agent: *
Disallow: /services/

Your robots.txt file normally appears at:

https://example.com/robots.txt

Remember that robots.txt controls crawler access. It is not the same as a noindex directive.

Blocking a page from crawling can prevent Google from seeing its content and current indexing directives. If you need Google to process a page and recognize that it should be indexed, Googlebot must usually be able to access it.

Crawl access may also be affected by:

• Firewall rules
• Security plugins
• Hosting problems
• DNS failures
• Authentication requirements
• Content delivery network settings
• Rate limiting
• Server errors

If the live test cannot fetch the page, the issue may be broader than a Search Console setting.

Review the Page Fetch and HTTP Response

A normal indexable page should generally return a successful 200 HTTP response.

Search Console may instead report:

• Redirect
• Not found
• Soft 404
• Server error
• Access denied
• Other fetch problems

A redirecting URL is not normally indexed as a separate page. Google evaluates the destination instead.

A 404 or 410 response indicates that the page is unavailable.

A server error may prevent Google from accessing the page at all.

A soft 404 can occur when the server returns 200, but the page appears empty, missing, or too similar to an error page.

Check both the response code and what the page actually displays.

Check Whether Indexing Is Allowed

URL Inspection can show whether indexing is allowed.

If indexing is blocked, inspect the page for a robots meta directive such as:

<meta name="robots" content="noindex">

Also check whether the directive is delivered through an X-Robots-Tag HTTP header.

Accidental noindex tags commonly appear after:

• A staging site goes live
• A redesign is launched
• An SEO plugin is reconfigured
• A template is copied
• A CMS visibility option is changed
• A developer temporarily blocks indexing
• If the page should appear in Google, remove the unintended directive.
• Then run the live test again to confirm that indexing is now allowed.

Compare the User-Declared and Google-Selected Canonicals

Canonical information is one of the most valuable parts of URL Inspection.

You may see:

• User-declared canonical
• Google-selected canonical
• The user-declared canonical is the URL specified by your page.

The Google-selected canonical is the URL Google decided to treat as the main version.

Ideally, these URLs should agree when you want the inspected page indexed independently.

If Google selects another canonical, review whether:

• The page duplicates another URL
• The canonical tag points elsewhere
• Internal links use a different version
• The sitemap lists a different URL
• HTTP and HTTPS versions conflict
• WWW and non-WWW versions conflict
• Parameter URLs duplicate the page
• Multiple pages target the same search intent
• The inspected page contains very little unique content

A canonical mismatch does not always indicate an error. Google may have correctly selected the stronger or cleaner duplicate.

The key question is whether it selected the version you actually want users to find.

Check How Google Discovered the Page

URL Inspection may display discovery information such as:

• Referring page
• Sitemap
• No referring sitemap detected
• These fields can help you assess how Google found the URL.

However, the discovery information is not always a complete list of every link or sitemap associated with the page.

Use it as one piece of evidence.

If an important page has weak discovery signals:

• Add relevant internal links
• Include the canonical URL in the XML sitemap
• Link from an indexed category or service hub
• Check that navigation links are crawlable
• Avoid relying only on JavaScript interactions or search forms

A page should normally be part of the website’s internal structure, not only present in the sitemap.

Review Rendered HTML and Resources

For JavaScript-heavy websites, the raw page response may not show everything Google processes after rendering.

URL Inspection can provide access to:

• Rendered HTML
• A screenshot
• Loaded resources
• JavaScript messages

Use this information to check whether Google can see:

• The main page content
• Important internal links
• Product or service information
• Headings
• Canonical tags
• Robots directives
• Structured data
• Content loaded through JavaScript

A browser may appear to display the page correctly while Google encounters blocked resources, rendering failures, or missing content.

Compare the rendered result with what a normal visitor sees.

If essential content or links are missing from the rendered HTML, the problem may require development work rather than a Search Console setting change.

Interpret Common Indexing Statuses Correctly

Discovered – currently not indexed

Google knows about the URL but has not crawled it yet.

Investigate:

• Internal linking
• Sitemap inclusion
• Server reliability
• Excessive low-value URL generation
• Website size
• Whether the page is new
• Crawled – currently not indexed
• Google crawled the page but did not add it to the index at that time.

Review:

• Content originality
• Page completeness
• Duplicate intent
• Canonical signals
• Internal links
• Whether the page deserves a separate search result
• This is not always solved through a technical change.
• Duplicate, Google chose a different canonical
• Google considers another URL the main version.

Review the canonical tag, internal links, sitemap entries, redirects, and content similarity.

Blocked by robots.txt

Google cannot crawl the page because of a robots rule.

Determine whether the block is intentional.

Excluded by noindex

Google detected a noindex directive.

Remove it only if the page should appear in search.

Page with redirect

The inspected URL redirects elsewhere.

Check whether the redirect is intentional and whether the destination is correct.

Not found

The URL returns a missing-page response.

Restore the page, redirect it, or leave it removed depending on the intended outcome.

Fix the Underlying Issue Before Requesting Indexing

The Request Indexing button is useful after you have:

• Published a new page
• Removed an unintended noindex
• Corrected a canonical
• Fixed a server problem
• Restored missing content
• Improved the page
• Updated internal links
• Corrected a redirect
• It should not be treated as a universal repair button.

Requesting indexing does not override:

• Robots blocks
• noindex
• Redirects
• Error responses
• Duplicate content
• Weak page value
• Canonical conflicts

Run a live test first. Confirm that Google can access the page and that the intended directives are present.

Then request indexing when appropriate.

Use Validate Fix for Group-Level Problems

For certain issue groups in the Page Indexing report, Search Console may provide a Validate Fix option.

Use this when you have corrected a shared problem affecting multiple URLs.

For example:

• A template-level noindex was removed
• A server error was fixed
• A redirect rule was corrected
• A blocked directory was made crawlable

Validation tells Google that the issue has been addressed and asks it to check affected URLs again.

Do not start validation before the fix is fully implemented. If several example URLs still fail, the validation may not complete successfully.

For a single updated page, URL Inspection and Request Indexing may be more appropriate.

Monitor the Result Over Time

Search Console does not always update immediately.

After making changes:

• Test the live URL.
• Confirm the technical fix.
• Request indexing or start validation when appropriate.
• Wait for Google to recrawl the page.
• Inspect the URL again later.
• Monitor the relevant Page Indexing category.
• Check whether impressions begin appearing in the Performance report.
• Remember that indexing and ranking are different.

A page can be indexed without receiving impressions or clicks. Search visibility also depends on relevance, content quality, competition, search demand, links, and other signals.

The first goal is to confirm that the page is technically accessible and indexable. Performance should be evaluated separately.

A Practical Search Console Indexing Checklist

When diagnosing an important page, check:

• Page Indexing report
• Which exclusion category contains the URL?
• Are similar pages affected?
• Is the exclusion intentional?
• Did the issue begin after a site change?
• URL Inspection
• Is the URL indexed?
• When was it last crawled?
• Is crawling allowed?
• Is indexing allowed?
• Did Google fetch the page successfully?
• What is the user-declared canonical?
• What is the Google-selected canonical?
• Was the page discovered through a sitemap or referring page?
• Live test
• Does the current page pass the live test?
• Does it return 200?
• Has noindex been removed?
• Is the correct canonical present?
• Can Google render the main content?
• Are important resources available?
• Website checks
• Is the page internally linked?
• Is the canonical URL in the sitemap?
• Is the content unique and complete?
• Does the page deserve to be indexed independently?
• Are redirects and duplicate URLs controlled?

Final Thoughts

Google Search Console is most useful when you treat it as a diagnostic system rather than an automated fix.

Start with the Page Indexing report to identify patterns across the website. Then use URL Inspection to investigate important pages individually.

Compare Google’s indexed information with the live version. Review crawl access, response codes, indexing directives, canonical signals, discovery sources, and rendered content.

Most importantly, decide whether the excluded URL should actually be indexed.

Not every non-indexed page is a problem. The objective is not to force every URL into Google. It is to ensure that your important, useful, canonical pages can be discovered, crawled, understood, and considered for indexing.