Build a Website SEO Crawler in Python: Titles, Canonicals, Links, and Status Codes

Overview

A useful SEO crawler does more than download pages. It creates a consistent dataset that helps you find broken URLs, redirect chains, missing metadata, conflicting canonicals, orphaned pages, and internal-linking problems. Python is well suited to this work because requests handles HTTP retrieval, Beautiful Soup extracts HTML fields, and standard libraries such as urllib.parse, urllib.robotparser, collections, and csv support the surrounding workflow.

For a first audit, use a normal HTTP crawler rather than a browser. It will usually be faster and easier to reproduce. Use Playwright or another browser automation tool only when important content or links are rendered client-side. A browser should solve a rendering requirement, not compensate for unclear crawl rules.

Before crawling, define the scope. Decide whether the audit includes only the preferred hostname, HTTPS URLs, subdomains, parameterised URLs, PDFs, images, or other file types. Also set a maximum page count, request timeout, delay between requests, and output location. Crawl only sites you are authorised to inspect, and treat access instructions and operational limits as part of the project requirements.

A practical crawl-data schema

Store one row per discovered URL. A useful export can include:

• requested_url: the URL placed in the queue.
• final_url: the URL after redirects.
• status_code: the final HTTP response status.
• redirect_chain: each hop between the requested and final URL.
• content_type: the response media type, where available.
• title and meta_description: trimmed text from the document head.
• canonical: the declared canonical URL, normalised where possible.
• robots_directives: the contents of relevant robots meta or header directives.
• h1_count and h1_text: useful for heading review.
• word_count: a consistent count of visible page text, clearly labelled as approximate.
• internal_links and external_links: counts found in the HTML.
• source_url: the page or sitemap where the URL was discovered.
• crawled_at and error: audit timing and failure details.

Keep the raw URL, normalised URL, and final URL separately. That distinction makes it easier to diagnose tracking parameters, redirects, and inconsistent internal links.

Checklist by scenario

1. Start with sitemap discovery

Check common sitemap locations such as /robots.txt, /sitemap.xml, and sitemap index files. A sitemap is a valuable seed list, but it is not a complete substitute for crawling internal links. Parse both individual URL sets and sitemap indexes, then record the sitemap source for every discovered URL.

• Fetch sitemap resources with a clear timeout.
• Handle XML namespaces rather than assuming a plain tag name.
• Ignore unsupported or out-of-scope URL schemes.
• Deduplicate URLs before adding them to the queue.
• Record malformed XML as an audit error instead of silently dropping it.

2. Crawl internal links

Use a queue and a set of visited URLs. For every successful HTML response, extract links from <a href> elements, resolve relative links against the page URL, remove fragments, and apply your scope rules. A basic breadth-first crawl gives you a useful view of pages reachable from the chosen starting points.

Normalisation needs to be deliberate. Lowercase the hostname, remove URL fragments, resolve relative paths, and decide how to treat a trailing slash. Preserve meaningful paths and query strings unless your project has a documented rule for excluding parameters. Do not blindly sort or remove query parameters: some sites use them to identify real content.

3. Extract page-level SEO fields

For HTML responses, parse the document head and body. Extract the first title element, the description meta tag, canonical link, robots meta tags, H1 elements, and links. Keep both the value and a diagnostic flag where useful, such as title_missing, canonical_multiple, or h1_empty.

For word counts, remove script and style elements before collecting visible text, collapse repeated whitespace, and document the method. A crawler’s word count is a comparison signal, not a definitive measure of content quality.

4. Capture statuses and redirects

Use a session so connection handling is consistent, and set a descriptive user agent that identifies the audit. Record the response history rather than only the final status. This exposes chains such as an old URL redirecting through several intermediate locations before reaching the preferred page.

Separate transport failures, timeouts, DNS errors, blocked responses, client errors, server errors, and non-HTML content in the export. Each category suggests a different follow-up action.

5. Export and analyse

Write a CSV for quick review and a JSON export when nested data such as redirect chains or individual link records matters. Sort the spreadsheet by status code, then review:

• 4xx and 5xx responses;
• redirects and multi-hop chains;
• duplicate or missing titles and descriptions;
• missing, relative, or conflicting canonicals;
• pages with no H1 or multiple H1 elements;
• URLs discovered internally but absent from the sitemap;
• sitemap URLs that cannot be reached through internal links;
• pages with unusually low or high link counts.

For larger crawls, store a crawl run identifier and compare exports over time. The comparison is often more valuable than a single list of issues because it shows whether fixes survived a release.

What to double-check

• Scope: Confirm the allowed hostname, protocols, paths, subdomains, and file types before interpreting results.
• Robots handling: Decide how the crawler should process robots instructions and document that decision. Do not present an incomplete crawl as a complete site inventory.
• Canonical interpretation: Treat a canonical as a declared signal to review, not proof that the URL is correct or honoured everywhere.
• Rendering: Compare a sample of raw HTML with the browser-rendered page. If important links or metadata exist only after JavaScript execution, run a targeted browser pass.
• HTTP methods and errors: A timeout or failed request should be retained with its error rather than converted into a false 404.
• International sites: Keep language and regional URL patterns in scope. If hreflang analysis is required, add separate fields instead of overloading the canonical column.
• Duplicate URLs: Test equivalent forms involving case, trailing slashes, encoded characters, fragments, and tracking parameters before deduplicating broadly.
• Test coverage: Use fixtures for redirects, malformed HTML, relative links, empty metadata, and non-HTML responses. Regression tests make crawler changes safer.

For related implementation guidance, see this Python web scraping tutorial, the guide to testing web scrapers with Pytest, and the workflow for cleaning and deduplicating scraped data.

Common mistakes

Using only the sitemap. This misses links, redirects, and pages that are reachable internally but not listed. Combine sitemap seeds with internal-link discovery.

Following every URL without limits. Calendars, search pages, filters, and session parameters can create effectively unlimited crawl spaces. Set a page limit, depth limit where appropriate, and parameter policy.

Overwriting redirect information. Saving only the final URL hides redirect chains and makes remediation harder. Preserve every hop.

Assuming every response is HTML. PDFs, feeds, images, JSON, and error pages require different handling. Check the content type before parsing.

Counting metadata without context. A long title or word count may be worth review, but automated thresholds are triage signals rather than final editorial decisions.

Starting with a headless browser unnecessarily. Browser automation adds runtime and operational complexity. Begin with HTTP requests, then use Playwright for pages that genuinely require rendering. The dynamic-site Playwright guide covers that next step.

When to revisit

Run the crawler before a major redesign, after a migration, when templates change, and before seasonal planning cycles. Re-run it after fixes have been deployed so the export can confirm whether errors disappeared rather than merely being marked complete.

Review the crawler itself at least whenever your workflow or dependencies change. Check URL normalisation, sitemap parsing, rendering assumptions, timeout values, output columns, and test fixtures. Metadata recommendations and search-engine guidance can evolve, so verify the interpretation of fields before reusing old thresholds.

For each audit, use this short operating checklist:

1. Define scope, limits, user agent, and request pacing.
2. Collect sitemap URLs and record their sources.
3. Crawl in-scope internal links with normalised URLs.
4. Save statuses, redirect chains, metadata, headings, links, and errors.
5. Export raw results before filtering or grouping.
6. Review high-impact findings manually and assign owners.
7. Compare the next crawl with the previous run.

Keep the script, configuration, crawl date, and output together. That simple record turns a one-off Python SEO crawler into a reusable audit process for technical SEO and data extraction work.