Extracting SEO metadata with Python turns a large technical audit into a repeatable comparison. This guide shows how to collect titles, descriptions, canonicals, hreflang annotations, robots directives, status codes, headings, and social tags, then normalize the results so missing values, duplicates, redirects, and international-page inconsistencies are easier to find.
Overview
A useful SEO metadata crawler does more than save the text inside a title tag. It records the response that delivered the page, the URL requested, the final URL after redirects, and the important signals found in the HTML. That distinction matters: a page can return a redirect, contain a canonical pointing elsewhere, or expose a different set of tags depending on whether the content is rendered server-side or by JavaScript.
For a straightforward crawl, Python's requests and Beautiful Soup are usually sufficient. A headless browser such as Playwright is more appropriate when the metadata is added after page load. Start with ordinary HTTP requests where possible because they are easier to schedule, test, and troubleshoot. Respect the site's access rules, use a clear user agent, limit request rates, and avoid crawling pages that are not needed for the audit.
| Field | What to capture | Useful checks |
|---|---|---|
| URL and status | Requested URL, final URL, status code, content type | Redirect chains, errors, non-HTML responses |
| Title | Text of the first or selected <title> | Missing, whitespace-only, duplicated, unusually long values |
| Meta description | meta[name="description"] content | Missing, duplicated, empty, or inconsistent template output |
| Canonical | Absolute value of link[rel="canonical"] | Invalid URL, conflicting canonicals, cross-domain targets |
| Hreflang | Every link[rel="alternate"][hreflang] and its URL | Missing return links, duplicate language codes, malformed URLs |
| Robots directives | Meta robots, other relevant meta names, and X-Robots-Tag | Conflicting values or unexpected noindex directives |
| Headings | Counts and text for H1, H2, and optionally H3 elements | Missing H1, repeated headings, template anomalies |
| Social metadata | Open Graph and selected Twitter card properties | Missing image, title, description, or URL values |
Keep the raw values as well as normalized values. Raw HTML-derived data helps explain an anomaly later, while normalized URLs make comparisons reliable. The crawler's output might include fields such as requested_url, final_url, status_code, title, meta_description, canonical_url, hreflang, robots_meta, x_robots_tag, h1_count, og_title, and issues.
Checklist by scenario
Standard site crawl
- Build a URL list from a sitemap, an approved seed list, or an existing crawl. Normalize obvious duplicates before making requests.
- Request each URL with a timeout and record the status code, content type, response URL, and redirect history.
- Parse only HTML responses. Save titles, descriptions, canonicals, robots metadata, headings, and social fields.
- Normalize whitespace and convert relative links to absolute URLs using the final response URL as the base.
- Flag missing fields, duplicate values, invalid URLs, and directives that differ from the page template or audit rules.
A compact extraction function can provide the core fields:
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
def extract_metadata(url):
response = requests.get(url, timeout=15, allow_redirects=True,
headers={"User-Agent": "SEO-audit-crawler/1.0"})
soup = BeautifulSoup(response.text, "html.parser")
def content(name):
tag = soup.find("meta", attrs={"name": name})
return tag.get("content", "").strip() if tag else ""
canonical = soup.find("link", rel=lambda value: value and "canonical" in value)
return {
"requested_url": url,
"final_url": response.url,
"status_code": response.status_code,
"title": soup.title.get_text(" ", strip=True) if soup.title else "",
"meta_description": content("description"),
"canonical_url": urljoin(response.url, canonical.get("href", "").strip()) if canonical else "",
"robots_meta": content("robots"),
"h1_count": len(soup.find_all("h1")),
"h1_text": [h.get_text(" ", strip=True) for h in soup.find_all("h1")]
}
This is a starting point rather than a complete crawler. Add exception handling, request pacing, content-type checks, logging, and a persistent output format before using it on a sizeable site. The related guide on building a website SEO crawler in Python can help extend the URL and status-code workflow.
International or multilingual site
- Store hreflang as a list of objects, for example
{"code": "en-gb", "url": "https://example.com/uk/"}, rather than as one concatenated string. - Lowercase language and region codes for comparison, while preserving the original value for review.
- Check that each alternate URL is absolute, returns an expected response, and has a corresponding reciprocal annotation when that is part of your audit rule.
- Compare canonical targets with language versions. A localized page that canonicals to an unrelated locale deserves review rather than automatic correction.
Migration, redesign, or template change
- Capture both the old and new crawl outputs using the same schema.
- Compare URL status, final destinations, title patterns, canonicals, robots directives, headings, and social fields.
- Prioritize changed directives, unexpected redirects, missing metadata, and changes affecting large URL groups.
- Keep a sample of raw responses so developers can distinguish a parser problem from a template problem.
What to double-check
URL normalization is often where comparison errors begin. Decide in advance how to handle trailing slashes, default ports, fragments, percent-encoding, repeated slashes, and query parameters. Do not remove query parameters blindly: some identify meaningful pages, while others are tracking noise. Use a documented allowlist or parameter policy.
Separate the requested URL, final URL, and canonical URL. They answer different questions. The requested URL shows what was crawled; the final URL shows where the server sent the crawler; and the canonical shows the page's declared preferred URL. A redirect is not automatically a canonical error, and a canonical pointing to another URL is not automatically a redirect.
Robots directives also need careful handling. A robots.txt file controls crawler access at the site or path level, while a meta robots tag and an X-Robots-Tag response header provide page or response directives. Record these separately. If both a meta tag and header exist, preserve both in the audit output so a reviewer can investigate their combined effect.
For JavaScript-heavy pages, compare the HTML returned by requests with the rendered DOM only when necessary. If the title or canonical appears after client-side rendering, use Playwright and record that the value came from a rendered page. This avoids treating a rendering difference as a broken parser.
Common mistakes
- Taking only the first matching tag: duplicate canonicals, descriptions, or social properties should be counted and reported.
- Comparing unnormalized strings: relative and absolute URLs, capitalization, and whitespace can create false differences.
- Ignoring headers: page-level directives may be sent through HTTP headers rather than HTML.
- Assuming one crawl is definitive: temporary errors, rate limits, consent pages, and server variation can affect a response.
- Using regex to parse HTML: regular expressions can assist with validation, but an HTML parser is safer for nested and malformed markup.
- Mixing extraction and decisions: first collect what the page says, then apply project-specific rules for severity and remediation.
For repeatable audits, add data-quality checks for missing fields, duplicates, unexpected schema changes, and invalid types. The guide to validating scraped data provides a useful testing mindset, while the selector cookbook covers practical field mapping when templates vary.
When to revisit
Run this checklist before a migration, after a major template release, when adding a language or market, and before important seasonal planning cycles. It is also worth scheduling smaller recurring crawls for high-value page groups rather than waiting for a full-site audit to reveal a missing directive.
Revisit the crawler whenever the site introduces a new rendering method, routing pattern, consent layer, CDN behavior, or metadata template. Review parser-library syntax and your interpretation rules periodically, especially when the audit begins collecting new fields. A practical maintenance routine is:
- Keep a small fixture set containing pages with redirects, missing tags, duplicate tags, hreflang links, and non-HTML responses.
- Run automated tests against those fixtures after code or dependency changes.
- Compare a new crawl with the previous baseline and investigate clusters of changes.
- Review the issue checklist at least every six months and update it when the site's templates or SEO requirements change.
- Save crawl date, crawler version, request settings, and normalization rules with every export.
The goal is not to produce a perfect spreadsheet on one occasion. It is to create a consistent evidence trail that makes technical SEO changes visible, explainable, and easier to fix.