How to list every page of a website
Learn how to list every page of a website, check sitemap and crawl coverage, and build a spreadsheet that shows which pages need attention.
Free app. Runs on your Mac with the Claude or Codex plan you already have.
What it takes to list every page of a website
To list every page of a website, start with the pages the site declares, then follow the paths visitors take through it. Record each address in one spreadsheet and check what it serves. The result is a website page inventory that shows where a page fits, where an old address redirects, and which pages need attention.
An inventory gives a content audit its coverage. The audit asks whether pages meet their goals, while the inventory establishes which pages exist and what you know about each one. A site crawl, which follows links and requests pages, can miss an unlinked campaign page. A sitemap can omit a useful guide or retain a retired URL. Compare both sources, record unresolved checks, and keep the evidence that explains each row.
What it does for the business
- Scope client work with a page listUse counts by section, language, and page type to estimate the work behind a redesign or consolidation. Your client can decide which sections to fund first and assign people who know the content.
- Reuse pages that already answer buyer questionsGroup pages by topic and reader to see where a useful guide already exists. The content team can update that guide and link to it from sales pages, saving the time it would spend commissioning a second article.
- Catch conflicting instructions for visitorsBring outdated policy pages, product guides, and current offer pages to the same review. The people responsible can agree on the correct instructions and point support staff to a page they can share.
- Plan moves with fewer missing pagesUse the inventory to connect old addresses with the pages customers will need after a migration. A launch owner can give priority to pages that prospects use to compare services or existing customers rely on for help.
How to list every page of a website, step by step
Set the scope and read every sitemap
A sitemap count gives you a set of addresses you can reconcile at the end of the audit.
What to do
- Define the public site you are inventorying. Identify which hostnames, language folders, and regional sections belong in your count. Keep links to partner sites out of your totals.
- Open robots.txt, the public file that provides crawl instructions, and find its sitemap references. Check the usual sitemap locations too. An XML sitemap is a machine-readable list of URLs. A sitemap index links to other sitemap files.
- Follow each index through its child files, including compressed lists. Record which files you read, how many entries they contain, and any that failed to open. Keep separate counts for entries as listed and for distinct addresses.
- Group addresses by language and section using their URL patterns, then verify these labels when you read the pages. Record the date of any earlier inventory you use for comparison.
Why
A sitemap index may point to separate files for articles, products, and translations. A count per file lets you trace a short total to a missing section or explain a surplus when two files list the same address.
Build the spreadsheet and verify what each URL serves
Keep the listed address and the address it serves so you can explain a redirect.
What to do
- Put every distinct sitemap URL into the spreadsheet before requesting pages. Capture its section, language, and any sitemap date. Preserve the original address when it differs from an address reached later.
- Fetch pages in batches at a pace the site can handle. Record response status and the final address after redirects. Read the HTML head, which contains the title and description, and find the H1, or main heading, on the page.
- Record the canonical URL, which is the address the page declares as its preferred version. Also capture language annotations and any noindex instruction, which asks search engines to exclude the page from their indexes. Keep those signals in separate fields.
- Follow links from menus, footers, categories, and older archive pages. Mark new addresses as absent from the sitemap. Check link variations such as filters or tracking parameters before counting them as distinct pages.
- Mark fetch failures separately from addresses you have yet to read. If you stop partway through the work, tally the rows still supported by sitemap evidence alone and say which sections they belong to.
Why
A guide linked from an old article can be absent from the sitemap, while the sitemap can still list a page that redirects. Both checks help you account for addresses that one source alone would miss or misclassify.
Label what each page is about and how fresh it is
A freshness label is useful when the reader can trace it to the date the page declares.
What to do
- Assign a page type based on the visitor's task. A pricing page helps them evaluate an offer. A category page helps them reach individual entries.
- Choose a short topic label you can reuse across related pages. Compare the address, title, and H1 before assigning a likely main phrase, and mark that phrase as inferred from the page.
- Record the public byline when available. Track maintenance assignments separately so a former contributor never appears to own today's work.
- Use the declared last modified date first, then the publication date, then the sitemap date. Label the source, keep conflicting dates visible, and mark the date unknown when the sources provide none.
- Read the oldest and most prominent pages for dated wording. Save the exact passage when you find an expired deadline described as upcoming or instructions that name a retired menu.
Why
Topic labels unite pages whose titles use different words. Date sources help you judge freshness. A recent sitemap date could reflect a template change on a page that still describes a discontinued service.
Reconcile duplicates, omissions, and changes
A flag should point to a page and give someone enough evidence to decide what to do.
What to do
- Find repeated titles and descriptions, missing fields, and pages that cover the same topic for the same reader. Open possible duplicates to see whether each has a purpose. A guide and a service page on the same topic can serve different needs.
- Check sitemap entries for redirects, errors, noindex instructions, and a canonical that points elsewhere. Compare those signals with the intended page role before choosing a fix. A saved old URL can serve a valid redirect and still belong outside the sitemap.
- Compare language versions and flag missing counterparts that affect a visitor's task, such as a translated product page whose help link leads to instructions in another language.
- Compare with the last inventory. Recheck any address missing from the new list before calling it gone. Record whether it still serves a page, moved, returned a response indicating removal, or could not be verified.
Why
A template error can give distinct pages the same title. Fix that by writing titles specific to each page. If the pages themselves do the same job, a merge into a named destination may make more sense.
Check the coverage and hand over a ranked list
The report should show which pages were verified and which addresses still need checking.
What to do
- Reconcile the number of rows with your sitemap counts. Explain the difference using repeated entries and URLs discovered through links. Check that redirects still have rows for the listed addresses.
- Check a random sample against the source pages. Make sure the spreadsheet represents redirects and missing fields correctly.
- Summarize the totals by language, page type, and age. Put the most useful fixes first, with the exact address, the proposed change, and a reason. Send each action to the person who can make that change.
- State where the evidence ends. Include the count of pages read, pages known from sitemaps alone, and failed requests. Date the inventory so the next person can compare the same scope after the site changes.
Why
The table lets a client check individual pages. The report tells them which fixes to make first and where a coverage gap could still affect a launch decision.
Common mistakes
- Treating the sitemap as the full siteCheck links from categories, archives, and individual pages to find addresses the sitemap omits.
- Dropping redirected URLs before countingKeep the original address and its destination so you can see which sitemap entries need attention.
- Using the day of the crawl as the update dateKeep the fetch date as a record of your work and store the page's declared date separately.
- Assigning the public author as the internal ownerRecord the byline as public evidence and verify the maintenance assignment with the team.
- Treating a failed request as a deletionVerify the response again before assigning a status that will drive migration or cleanup work.
What a good result looks like
- One row per discovered address, with the sitemap and link discoveries accounted for.
- The original URL, response status, final address, and declared canonical in separate fields.
- Language, section, type, topic, title, H1, and description where the evidence supports them.
- Public author and dates labeled with their source, including unknown values.
- An ordered action list with page-level evidence and a clear record of coverage gaps.
And then what? Using the result
- Give developers the outdated sitemap addresses and confirmed destinations so they can correct sitemap entries and internal links together.
- Have editors check the oldest pages that support active offers, then schedule the revisions customers are most likely to encounter.
- Use pages with overlapping topics to brief a consolidation and agree on the destination before changing any URLs.
- Use the language and page-type totals to budget translation or migration time and track progress by section.
- Save the dated inventory as a baseline and compare it with the next one after a release or site move.
When to run it
- At the start of a client engagementBuild the first inventory before estimating editorial or technical work. It tells you which sections and language versions belong in the estimate.
- Before a site migrationGive each public address a planned destination or a decision to retire it. Start with pages that support purchases, help requests, or ongoing campaigns.
- After merging two sitesFind where old URLs lead and where both sites still offer the same information. Assign an owner to decide which copy to keep for each reader.
- When a site has years of articlesRead beyond the first pages of the archive. Older guides often remain reachable through links long after they disappear from the main menu.
Questions people ask
How do you list every page of a website?
Read all sitemap files, crawl internal links, and reconcile the distinct addresses in one spreadsheet. Check each address for status and metadata and mark where you found it. For a public site, add any known addresses you can verify and record the limits of what your sources cover.
Can you use an XML sitemap to find all the pages on a site?
Use an XML sitemap as your starting list and compare it with links and any available site records to check coverage. A sitemap can omit a page that still works or list an address that has since moved.
How do you find orphan pages?
Compare addresses from sitemaps or authorized site records with those reached through internal links. Orphan pages have no internal links pointing to them, so a link crawl alone cannot discover them. Check whether part of the site was inaccessible before drawing that conclusion.
What should a website page inventory include?
Include the URL, language, section, type, topic, title, description, main heading, public author, and date with its source. Record response status, redirects, canonical, and sitemap membership so you can explain what you see and decide on a fix.
How do you know when a web page was last updated?
Look for a visible update date or a declared modified date in the page metadata. Where those are absent, record the publication date or sitemap date with its label. Leave the date unknown when you have no source.
How do you handle pages with more than one URL?
Record the distinct discovered addresses and group those that serve the same page using redirects, canonical declarations, and a check of the content. Keep the addresses visible so a developer can see which links or sitemap entries need to change.
More jobs in SEO agencies
List every page of a website, on your own Mac.Free app. Runs on your Mac with the Claude or Codex plan you already have. Your files stay on your computer.
Download for Mac