How to Extract Internal Links, Titles, and Meta Descriptions for Site Audits

Overview

A basic SEO audit scraper does not need to be complicated. In many cases, you only need to visit a starting URL, discover internal links, fetch each page, and extract a small set of fields:

• URL
• HTTP status code
• Canonical URL, if present
• Page title
• Meta description
• Indexability hints such as robots meta tags
• Outbound internal links found on the page

That dataset is enough to answer several useful audit questions:

• Which pages are missing titles or meta descriptions?
• Which titles are duplicated or overly long?
• Which internal links point to redirects, errors, or non-canonical URLs?
• Which sections of the site have weak internal linking?
• Which pages are orphaned from your chosen crawl entry point?

For most sites, the core workflow is straightforward:

1. Choose a crawl start point, usually the homepage or a section landing page.
2. Restrict the crawl to the target host or subdomain.
3. Normalise discovered URLs so duplicates do not multiply.
4. Extract title, meta description, and links from each HTML page.
5. Store results in CSV, JSON, SQLite, or another format suited to your review process.
6. Run a second pass to identify issues such as duplicates, missing fields, and broken internal targets.

If you are building this in Python, a simple stack of requests and Beautiful Soup is often enough for static pages. If the site relies heavily on client-side rendering, you may need browser automation. A Playwright web scraping setup is usually a cleaner choice than older browser-driving approaches when you need rendered DOM content. If you work in Node.js, a Puppeteer scraping or Playwright-based workflow can do the same job.

The key is not the library itself. The key is building a crawl process that produces consistent, audit-friendly output.

Before you start, define your scope clearly:

• Are you crawling one subdomain or the whole web property?
• Do you want only indexable pages, or every HTML page found?
• Will you follow query parameters?
• Do you treat trailing slash variants as separate URLs or normalise them?
• Do you need rendered content, or is server HTML enough?

Those decisions affect your totals, your duplicate counts, and your final conclusions. Many weak audits come from weak crawl rules, not weak analysis.

Checklist by scenario

Use this section as the part you return to before each audit. The right checklist depends on the kind of site and the kind of question you are trying to answer.

Scenario 1: Small brochure site or blog

For a smaller site with mostly static pages, keep the crawl simple.

• Start at the homepage.
• Limit the crawler to the primary domain.
• Fetch HTML with requests.
• Parse <title>, <meta name="description">, canonical tags, and internal links.
• Ignore non-HTML assets such as images, PDFs, CSS, and JavaScript files unless they matter to the audit.
• Export one row per page, plus a second file for link relationships if you want internal link graph analysis.

This is often enough for a content audit, title review, or migration QA pass.

Scenario 2: Large content site with many templates

On larger sites, your challenge is usually consistency rather than extraction.

• Sample by section first if a full crawl is expensive.
• Capture template signals such as article pages, category pages, author pages, and tag pages.
• Record heading data or breadcrumb data if you need to compare templates.
• Store crawl depth so you can see whether important pages sit too far from entry points.
• Deduplicate URLs aggressively after normalisation.
• Save the source of each discovered link so you can trace weak internal linking patterns.

On a site with many templates, duplicated metadata often follows template logic. If your data includes a page type field, your audit becomes much easier.

Scenario 3: JavaScript-heavy site

If critical links or metadata only appear after rendering, switch from a pure HTTP fetcher to browser automation.

• Test a small set of pages first by comparing raw HTML with rendered DOM.
• Use Playwright or Puppeteer only if needed.
• Wait for a stable selector rather than sleeping for an arbitrary number of seconds.
• Block unnecessary assets where appropriate to reduce crawl cost.
• Extract links after rendering, then continue the crawl with your chosen URL rules.

This is where a browser automation tutorial mindset helps: treat rendering as a targeted tool, not the default for every project. It is slower, more fragile, and often unnecessary unless you truly need it.

Scenario 4: Internal link audit

If your primary goal is to extract internal links rather than build a full metadata inventory, your export should focus on relationships.

• Store source_url and target_url for every internal link.
• Also store anchor text, rel attributes, and whether the link was found in navigation, footer, body, or another common area if you can classify it reliably.
• Resolve relative URLs to absolute URLs before storage.
• Record whether the target later returned 200, 301, 404, or another status.
• Create counts for inbound internal links per page.

This allows you to identify pages with very few internal links, links pointing to redirected URLs, and important pages that are difficult to discover through normal crawl paths.

Scenario 5: Metadata quality review

If you mainly want to scrape page titles and scrape meta descriptions for quality checks, build your export around review-ready columns.

• URL
• Title
• Title length in characters
• Meta description
• Description length in characters
• Duplicate title group
• Duplicate description group
• Missing title flag
• Missing description flag

This format helps you sort quickly for missing fields, repeated templates, or pages where metadata appears too generic to be useful.

Scenario 6: Pre-migration or post-launch verification

For migrations and launches, the crawler should help verify structure and continuity.

• Take a crawl snapshot before changes go live.
• Run the same crawl after launch.
• Compare titles, descriptions, canonicals, status codes, and internal links.
• Flag URLs that disappeared, changed unexpectedly, or lost metadata.
• Check whether internal links now point to redirected or broken destinations.

This is one of the most valuable uses of a simple website crawl metadata process because it turns vague launch feedback into a concrete checklist.

Scenario 7: Ongoing scheduled audits

If you want repeated checks, keep the crawler stable and the output comparable over time.

• Use a fixed start URL list.
• Keep normalisation rules consistent.
• Save crawl date and run identifier.
• Export in the same schema every time.
• Schedule regular runs with cron or CI workflows.

For teams doing repeated checks, it helps to pair the crawl with storage and scheduling guidance. See Schedule a Web Scraper With Cron, GitHub Actions, and Cloud Functions and Store Scraped Data in CSV, JSON, SQLite, or Postgres: What to Choose.

What to double-check

Once you have data, do not jump straight into reporting. Audit datasets often look complete while still containing subtle crawl errors. This is the stage where you validate that your scraper collected what you think it collected.

URL normalisation rules

Make sure your crawler treats equivalent URLs consistently. Common examples include:

• Trailing slash versus no trailing slash
• Uppercase versus lowercase paths where the site treats them the same
• Fragment identifiers such as #section, which usually should be removed
• Tracking parameters that create duplicate crawl entries

Without normalisation, duplicate counts become misleading and internal link analysis becomes noisy.

Internal versus external link classification

Check how you define an internal link. Some sites span multiple subdomains, while others use a separate host for assets or help content. Decide whether subdomains are included before the crawl starts. Then verify that your code reflects that rule.

Rendered versus raw metadata

On some sites, the title or meta description may be altered by JavaScript. Compare a few pages manually in the browser and in raw HTML. If they differ materially, your extraction method may need to change.

Canonical and redirect handling

A page can return 200 and still be the wrong URL to audit if it includes a canonical pointing elsewhere. Likewise, an internal link can look valid but resolve through one or more redirects. Capture both the originally requested URL and the final resolved URL when possible.

Status codes and soft errors

Do not assume every 200 response is healthy. A site may return a branded error page with a 200 status. If certain pages look suspicious, inspect the HTML for known error patterns or very short content. This is especially useful when checking internal links at scale.

Deduplication after extraction

Once the crawl is complete, clean your data before analysis. Remove duplicate rows, standardise null values, and validate unexpected blanks. For a deeper pass on this, see How to Clean Scraped Data: Deduplication, Normalisation, and Validation.

Rate limits and crawl reliability

If pages are intermittently timing out or returning blocked responses, your dataset may be incomplete. Build in retries, sensible timeouts, and logging. Related guidance: Web Scraping Error Handling Checklist: Retries, Timeouts, and Fallbacks and Rate Limiting for Web Scrapers: How to Crawl Responsibly Without Getting Blocked.

Common mistakes

The same problems appear in many first-pass audit crawlers. Avoiding them will usually improve your results more than switching libraries.

Crawling without a clear scope

If you do not define whether query strings, subdomains, or parameterised search pages are in scope, your crawl can expand quickly and distort findings. Always write down the rules first.

Using browser automation for everything

Headless browser scraping is useful, but it adds complexity and runtime cost. Start with raw requests unless the site genuinely requires rendering. If you need browser tools, a Playwright web scraping workflow is often a practical modern choice, but only for the pages that need it.

Ignoring pagination and hidden discovery paths

Some pages are only reachable through paginated archives, infinite scroll, or load-more interactions. If you skip those patterns, your crawl may miss large parts of the site. See How to Handle Pagination, Infinite Scroll, and Load More Buttons When Scraping.

Failing to separate page data from link data

If you store everything in one flat table, internal link analysis becomes harder. It is often better to keep one table for pages and another for edges between source and target URLs.

Trusting metadata without checking extraction logic

A missing title in your export may mean the page truly has no title, or it may mean your selector failed. Verify suspicious patterns with a small manual sample.

Skipping post-processing

Raw crawl output is rarely report-ready. You may need to normalise URLs, classify templates, compute lengths, deduplicate rows, and group repeated metadata. That processing is part of the audit, not an optional extra.

Overlooking operational constraints

Even a modest crawl can trigger defensive systems if you send too many requests too quickly. Respect rate limits, use delays where needed, and consider proxy strategy carefully for larger projects. If you work at higher scale, How to Use Proxies for Web Scraping: Rotation, Sessions, and Common Pitfalls is a useful companion.

When to revisit

This topic is worth revisiting because the inputs change. The crawler that worked well six months ago may no longer reflect the site, the audit goal, or the way metadata is rendered.

Review and update your crawl setup in these situations:

• Before seasonal planning cycles: run a fresh metadata and internal link audit before major campaigns, navigation changes, or content refreshes.
• When workflows or tools change: if you move from requests to Playwright, or from CSV to a database-backed workflow, review your schema and validation checks.
• After redesigns or migrations: compare pre- and post-change snapshots.
• When templates change: title or description logic may shift across whole sections.
• When indexing or performance issues appear: internal linking and metadata extraction can help surface structural problems quickly.

A practical maintenance routine looks like this:

1. Keep a saved crawl config with scope, user agent, delay, and URL normalisation rules.
2. Maintain a standard export schema for pages and links.
3. Store one baseline crawl for future comparison.
4. Schedule recurring runs for priority sections.
5. Review exceptions manually rather than changing rules blindly.

If you want to extend this workflow, useful next steps include storing historical crawl results, automating scheduled runs, and enriching your audit with additional fields such as headings, canonical tags, and structured data. For adjacent use cases, you may also find these guides helpful: How to Scrape Search Results for SEO Research and Rank Tracking and How to Scrape E-commerce Product Pages for Prices, Stock, and Variants.

Final action checklist:

• Define crawl scope before writing code.
• Decide whether raw HTML or rendered DOM is required.
• Extract page-level metadata and link-level relationships separately.
• Normalise URLs before analysis.
• Validate a manual sample before trusting the full export.
• Save results in a format that supports repeat audits.
• Re-run the crawl whenever site structure or reporting needs change.

That is what makes this kind of scraper useful over time: not just that it can scrape website data once, but that it can produce the same clear, audit-friendly view whenever you need to check a site again.