How to find pages Google can't index
Find pages Google can't index. Check crawl rules, page tags, and Search Console, then write the exact fix for each unintended indexing problem.
Free app. Runs on your Mac with the Claude or Codex plan you already have.
What a website indexing audit is
To find pages Google can't index, start with the pages you expect people to find through search and check the technical signals for each address. Look for blocked crawling, missing pages, redirects, instructions to keep an address out of the index and references to another version of the same content. Then compare the site's signals with Google Search Console, where you can see what Google reports for specific addresses. Keep the site's present settings and Google's reported state in separate columns.
An indexing audit ends with decisions. A thank-you page may be excluded on purpose, while a product page that uses the same setting may deserve a fix. Rank the indexing issues you find and give the site owner the affected pages, the evidence, the exact change and the outcome to look for on a recheck. This is also where you say which addresses you could not inspect and which decisions need the owner's input.
What it does for the business
- Focus developer time on pages that matterA blocked sales or service page can deserve attention before an untidy archive. Give the team a ranked set of fixes tied to the pages customers rely on.
- Find one edit that reaches a group of pagesWhen a template adds the same directive to many pages, the report gives the developer a place to look and a list of affected URLs. The team can review one change against several page examples.
- Keep utility pages out of the search work queueA clear record of intended exclusions helps the agency and the site owner agree on which pages should appear for customers. Include any cleanup work, such as removing an intentionally excluded address from the sitemap.
- Get better evidence for the next client reviewA live recheck and a later look at Search Console answer different questions about what changed. Together they help the client decide whether to continue investigating a particular page or move the team to another issue.
How to find pages Google can't index, step by step
Build a URL inventory and record the evidence you have
A useful audit includes the addresses you expect to find and the ones you actually discovered.
What to do
- Collect addresses from the links on your site and from each XML sitemap, a file that lists URLs for search engines. Merge the sources and keep a note of where each address was found.
- Export the page indexing data and the available address lists behind individual reasons in Search Console. Use URL Inspection on important pages when you need a closer view of Google's recorded state and selected version.
- Keep the date and source of each Search Console record. If the data only gives totals by reason, use those totals for context and mark the individual addresses without evidence as unknown.
- Add search clicks where available to help rank affected pages. Record any crawl limit, inspection limit or missing data that you expect will leave gaps in the report.
Why
Sitemaps can include pages that navigation misses, and Search Console can mention older addresses the site no longer links to. Reading the sources together covers more ways an important page can fall out of sight and shows which conclusions rest on incomplete evidence.
Read the technical signals for each address
Keep a record of the exact signal behind every finding.
What to do
- Check the server's response and follow redirects to their final destination. Read suspicious success responses for missing-page text and retry pages that fail before marking the evidence unknown.
- Evaluate robots.txt, the site's rules for what crawlers can request, for Googlebot. Record the applicable group and rule and keep blocked pages distinct from pages you could open and inspect.
- Read the server headers and original HTML for noindex, an instruction to leave a page out of search. Check the rendered page as well when scripts can change search directives, and record what the server originally sent. If the original response includes noindex, take it seriously even if a script removes it.
- Record the canonical URL, the address the site presents as the preferred version. Follow it to check that it loads and represents equivalent content, and record any conflict with sitemaps or internal links.
Why
Several signals can explain why an address is a poor candidate for indexing. A crawl block can prevent a crawler from reading page directives, while a canonical declaration can point to a different page that the site intends to show. The evidence should tell the developer where to look when those signals disagree.
Compare the crawl with what Google reports
Use the crawl to explain site settings and search data to report Google's state.
What to do
- Put the site's current technical eligibility beside the reported Google status for the same address. Record the date of the data because Google may have observed an earlier version of the page.
- For an important page labeled Crawled - currently not indexed, compare its content with similar pages and check how easily it can be found from related pages. Treat these observations as possible explanations.
- If Google reports a different preferred address, read both pages and check which version appears in the sitemap and internal links. Look for differences in purpose before calling either page a duplicate.
- Keep technical barriers, Google-reported exclusions and unknown observations visible in the report. When the sources disagree, state what each source shows and name the next check.
Why
A crawl can show a current technical barrier, but only the search data tells you how Google reported that particular address. This distinction helps the team decide whether to repair a site signal, investigate content or wait for Google to process a recent change.
Decide which exclusions are worth changing
A useful fix supports the intended role of the page.
What to do
- Read what the affected page is for. Keep purpose-built account pages, internal search results and thank-you pages out of the urgent queue when their search exclusion matches the owner's intent.
- Look for contradictory signals on important pages, such as a service landing page in the sitemap that contains a noindex instruction. Check several pages from the same template to see how far the issue reaches.
- Group repeated issues by cause and give the exact change, location and expected result. For a shared rule, check a nearby path that should keep its current treatment and choose the narrowest correction.
- Ask the owner to decide when the page's role is unclear. Write the two choices and the consequence for that page, then check the evidence for the most important fixes before handing them over.
Why
A single template setting can affect both pages that should be eligible and pages the site deliberately excludes. A precise change and a comparison page keep the team focused on the unintended exclusion while preserving the settings the owner chose.
Recheck the affected URLs after release
Verify the live signal first and track Google's response on its own timetable.
What to do
- Reread the live pages, applicable crawler rules and sitemaps after the owner deploys the change. Inspect the rendered page when the first finding involved a script.
- Check an affected page and representative neighboring pages for unintended changes. When a rule or template applies across the site, expand the recheck to cover the parts of the site that share it.
- Record which changes matched the plan, which were made differently and which could not be verified. Use the same evidence you recorded in the first audit so someone else can compare the results.
- Follow important addresses in Search Console after Google has had a chance to revisit the site. Escalate cases with a persistent difference between the expected live signal and Google's reported state.
Why
The live response can confirm a technical repair as soon as it is deployed. Google's index state updates on a separate timetable, so the team needs separate evidence before reporting that the page appears in the index.
Common mistakes
- Using a crawl as proof of index stateRecord the site's eligibility signals and use Search Console for claims about Google's index.
- Removing every noindex instructionConfirm what the owner wants eligible for search before changing a page or shared template.
- Treating all blocked requests the sameSeparate confirmed crawler rules from a refusal or timeout that leaves the page unknown.
- Checking only the browser viewRead the original response as well because Google may see a directive before a script changes it.
- Changing a shared rule without checking its neighborsInspect both affected and intentionally excluded pages before and after deployment.
What a good result looks like
- One record per known address with its source, evidence and any reason it could not be read.
- Separate technical and Google-reported status fields, with source dates where available.
- Evidence for intended exclusions, likely errors and any decision the owner still needs to make.
- A ranked list of exact edits, with locations, affected addresses and expected live signals.
- A record of the scope for a recheck and the pages that still need Search Console evidence.
And then what? Using the result
- Send shared template and crawler-rule fixes to the developer with sample affected URLs and expected results.
- Ask the owner to decide which pages with ambiguous signals should be eligible for search.
- Tidy up sitemaps and internal links that keep pointing to retired or intentionally excluded URLs.
- Recheck the amended pages after deployment and monitor Google's reported state for high-value URLs.
When to run it
- Before or after a website migrationCheck whether redirects, preferred addresses and sitemap entries agree with the new structure. Recheck templates that came from an environment where pages were intentionally kept out of search.
- When a valuable page drops out of Search Console resultsInspect the URL's live signals and compare them with Google's recorded status. Review nearby pages that share the template to see whether the problem extends further.
- When filters and alternate addresses multiplyCompare which variants deserve their own search entry and whether the site consistently points to the preferred versions.
- When taking over a client's siteCollect a baseline that shows which exclusions the client expects and which corrections deserve a place in the first development sprint.
Questions people ask
How do you find pages Google can't index?
Combine a crawl with sitemap and Search Console data, then check each page's status, crawler access, indexing directives and preferred address. Read the content and its role on the site before deciding whether any exclusion is a problem.
What does Crawled - currently not indexed mean?
It means Google reports that it crawled the URL and currently has it outside the index. Check when Google last crawled it, whether it duplicates another page and whether it serves a distinct purpose. A useful, technically eligible page can still be outside the index.
Does robots.txt keep a page out of Google?
A robots.txt block controls crawling. Google can sometimes show the address of a blocked URL and cannot read a noindex directive on a page the block prevents it from fetching. Check the exact rule and the reason the team wants the page excluded before changing how it is handled.
Why might Google select a different canonical URL?
Google may select another representative address when it sees similar or duplicate pages and other site signals favor a different version. Compare the content and check which address your sitemap and internal links point to. If the pages serve distinct needs, make those differences clear and check their other signals.
Can you check indexing without Search Console?
You can check the site's technical signals without Search Console and mark Google's reported index state as unknown. The crawl can find a blocked path, an unexpected noindex directive or a preferred URL pointing to a dead address. Use Search Console to verify what Google reports for particular URLs.
How do you check an indexing fix after deployment?
Reopen the affected page and verify that the exact signal changed while nearby pages still show the intended settings. Then use URL Inspection to follow what Google reports after it has had another chance to crawl and process the page.
More jobs in SEO agencies
Find pages Google can't index, on your own Mac.Free app. Runs on your Mac with the Claude or Codex plan you already have. Your files stay on your computer.
Download for Mac