Ever feel like you’re flying blind with your website? You know you’ve published dozens of pages—service descriptions, FAQs, disclosures, maybe even a few blog posts—but can you confidently say you know all the URLs live on your domain right now?

For small businesses in finance, especially those juggling compliance, content updates, and client trust, overlooking orphaned pages or rogue URLs isn’t just sloppy SEO. It’s risky. Index bloat can tank crawl efficiency. Forgotten pages with outdated copy can erode authority. And let’s not even talk about thin pages you thought were noindexed… but somehow aren’t.

Let’s fix that.

This guide walks you through the most practical, free, and paid ways to uncover every page tied to your website. Whether you’re cleaning up for a redesign, auditing your SEO, or prepping for a Google Search Console migration, this checklist belongs in your toolkit.

1. Start with Google: The Quick-and-Dirty Inventory

Google sees your site in a way your CMS never will. Here’s how to tap into that perspective.

Type this into Google:

site:yourdomain.com -www

This search operator excludes subdomains and www, which just confuse things. You’ll get a rough snapshot of what Google has indexed. While it’s not exhaustive, it’s the fastest pulse check for index bloat or missing content. Pay close attention to:

  • Irrelevant parameters (e.g., ?replytocom= or print pages)

  • Duplicate-looking titles

  • Weird legacy URLs

Pro Tip: Scroll to the last page of results. If you see the dreaded “omitted results” message, Google is filtering out what it sees as similar or low-value content.

2. Google Search Console: Your Official Index Inventory

This one’s non-negotiable.

Log in to Google Search Console, and head to:

Index → Pages → All known pages

Here, you’ll see:

  • Indexed pages

  • Crawled but not indexed

  • Discovered but not crawled

  • Excluded (e.g., due to noindex)

What to do:

  • Export the list and keep it. It’s gold during audits.

  • Look for surprises: Are there PDFs indexed? Test pages? Unlinked privacy policies?

  • Compare this with your sitemap and CMS exports.

3. XML Sitemap(s): Your Official Page Manifest

If you’re serious about SEO, your site should have an up-to-date XML sitemap. It tells crawlers, “Here’s what matters.”

Method: Check /sitemap.xml

or https://yourdomain.com/sitemap.xml

If it doesn’t exist, you likely don’t have one. Some setups generate multiple nested sitemaps (e.g., one for blogs and one for products). Scrape them all. If your CMS or SEO plugin manages this (e.g., Yoast, Rank Math), you’re halfway there.

Use case for finance sites: Make sure legal pages like disclosures or terms are intentionally excluded or included. Don’t assume they’re “not important.” Google notices the gaps.

4. Run a Full Crawl with Screaming Frog

This is where the detective work starts.

Tool: Screaming Frog SEO Spider

  • Free for up to 500 URLs

  • Paid version ($259/year) unlocks full capability

Setup Tips:

  • Use your domain as the starting point

  • Enable JavaScript rendering if your site is JS-heavy

  • Don’t forget to include sitemap URLs in the config

The tool returns every internal page, response code, word count, canonical tag, and more. It’s the best way to spot:

  • Pages with no internal links (orphaned content)

  • Duplicates

  • Meta tag issues

  • Pages blocked by robots.txt

Finance-specific tip: Flag any landing pages used for campaigns or investor access. These are often forgotten and sometimes indexed when they shouldn’t be.

5. Check Hosting or CMS Exports

Sometimes, the most complete source of truth isn’t a tool—it’s your back-end.

If you’re using:

  • WordPress: Use a plugin like WP All Export or look inside the database.

  • Webflow, Wix, Squarespace: Export URLs from CMS Collections if possible.

  • Shopify: Use admin-level site export tools or apps like Matrixify.

Cross-reference this list against your crawl. Are all these pages linked from somewhere? Are they in your sitemap?

6. Audit with Ahrefs or Semrush: The External Eye

Third-party crawlers offer something unique: Google-like behavior and backlink data.

Tools:

These tools go beyond index status. They tell you:

  • Which pages have backlinks (that you didn’t know about)

  • Which pages are returning 404s but still being linked to

  • Crawl depth and internal linking problems

They also offer visualizations that simplify explaining issues to your team or client.

Pro tip: If you find a 404 page with backlinks, recover it with a redirect. Lost link equity is lost money.

7. Internet Archive & Historical Crawls

Ever deleted a page and thought, “No one will notice”? Google might not. But the Internet Archive will.

Tool: Wayback Machine

Search your domain and look at previous snapshots. You may discover:

  • Legacy content that’s still linked externally

  • Pages that were redirected away but not properly retired

  • Blog posts that mysteriously vanished

It’s not just digital archaeology; it’s a recovery mission.

8. Use Log File Analysis (Advanced)

If your site handles sensitive financial data or has compliance constraints, log files show you what bots (and users) actually access.

Tools like:

…can reveal crawl patterns, excessive hits to useless URLs, or stealth scrapers indexing content you didn’t mean to publish.

Finance-specific alert: If crawlers are hitting outdated pricing disclosures or archived PDFs, it’s time to prune or block them.

9. Build Your Master URL List

Once you’ve completed these steps, consolidate everything:

  • Google Search Console export

  • Screaming Frog crawl

  • Sitemap URLs

  • CMS export

  • Ahrefs/Semrush findings

  • Log files (optional)

De-duplicate, clean, and sort by:

  • Indexed vs non-indexed

  • Status code

  • Word count

  • Purpose (e.g., content, legal, product, utility)

This becomes your benchmark for future audits and a shield against index rot.

Final Words for Financial Sites

In finance, trust and transparency are everything. Having 17 different versions of your pricing policy or an indexed test page that says “Coming Soon” undercuts both.

You don’t need to be an SEO wizard to take control. You just need the right tools, a structured approach, and the discipline to do this regularly. If Google sees it, your clients might eventually do so, too.

Need help with your SEO strategy? Contact us today to learn how BizScribe has helped dozens of finance businesses improve their search engine ranking.