Use a repeatable Python crawl to collect accessibility signals such as heading structure, image alternative text, form labels, language declarations, landmarks, link text, and HTTP status codes before a manual audit. This workflow helps SEO and development teams find patterns across many pages, while keeping human review at the centre of accessibility decisions.
Overview
A website accessibility crawler does not replace assistive-technology testing, keyboard checks, content review, or consultation with people who use different access needs. It does, however, provide a consistent first pass. A crawl can reveal missing attributes, repeated template problems, inconsistent headings, and links that return errors. It can also create a baseline that you can compare after a redesign or content migration.
For a small audit, Python’s requests and Beautiful Soup are sufficient for server-rendered HTML. They will not see content that appears only after JavaScript runs. If important controls or text are rendered dynamically, use a browser automation tool such as Playwright to capture the page after it loads, then pass the resulting HTML to the same parsing functions.
Before collecting data, define the crawl boundary. Start with an approved domain, a small page limit, a delay between requests, and a clear user agent. Respect the site’s access rules, avoid submitting forms, and do not crawl private or personal data without permission.
A practical audit-output schema
Store one record per URL, with repeated elements represented as arrays. This makes the results useful for filtering and comparison rather than leaving them as an unstructured report.
{
"url": "https://example.com/page",
"status_code": 200,
"language": "en",
"title": "Example page",
"headings": [
{"level": 1, "text": "Page heading"}
],
"images": [
{"src": "/images/item.jpg", "alt": "Short description", "alt_present": true}
],
"form_controls": [
{"type": "email", "id": "email", "label_text": "Email address", "label_found": true}
],
"landmarks": ["header", "main", "footer"],
"links": [
{"href": "/contact", "text": "Contact us", "status_code": 200}
]
}
Keep the raw URL and a crawl timestamp with every record. If the site changes, these fields help distinguish a genuine issue from a parser error or a temporary response.
Checklist by scenario
1. Extract heading structure
Collect every h1 through h6, their visible text, and their order in the document. A simple parser can identify pages with no heading, more than one apparent page heading, skipped levels, or headings that contain no useful text.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
headings = [
{
"level": int(tag.name[1]),
"text": " ".join(tag.get_text(" ", strip=True).split())
}
for tag in soup.find_all([f"h{i}" for i in range(1, 7)])
]
Use these findings as review signals, not automatic failures. Heading levels can be affected by component structures, and the importance of a sequence depends on the page’s content and layout.
2. Check image alternative text
For every img, record its source, whether the alt attribute exists, and its value. An absent attribute is different from an empty attribute. An empty alt="" can be appropriate for a decorative image, while an informative image usually needs a concise text alternative.
images = []
for image in soup.find_all("img"):
images.append({
"src": image.get("src", ""),
"alt_present": image.has_attr("alt"),
"alt": image.get("alt")
})
Flag missing attributes and suspiciously long or repeated values for review. Do not automatically label every empty value as an error: decorative images may intentionally have an empty alternative.
3. Find form controls without an associated label
Collect inputs, selects, textareas, and other interactive controls. For a first pass, check whether a control has an aria-label, an aria-labelledby, a matching label for, or is wrapped by a label element. This is a useful screening method, not a complete accessible-name calculation.
controls = []
for control in soup.find_all(["input", "select", "textarea"]):
control_id = control.get("id")
matching_label = soup.find("label", attrs={"for": control_id}) if control_id else None
wrapped_label = control.find_parent("label")
labelled = any([
control.has_attr("aria-label"),
control.has_attr("aria-labelledby"),
matching_label is not None,
wrapped_label is not None
])
controls.append({
"type": control.get("type", control.name),
"id": control_id,
"label_found": labelled
})
Exclude controls that are deliberately hidden from users, and inspect custom widgets separately. Placeholder text should not be treated as a reliable replacement for a persistent label.
4. Record language, landmarks, and link text
Read the document’s lang attribute and flag pages where it is absent or clearly inconsistent with the page content. Record structural landmarks such as header, nav, main, aside, and footer, along with explicit ARIA landmark roles.
For links, store the destination, visible text, and whether the text is empty or generic. A crawl can identify empty links and repeated phrases such as “read more”, but only a person with page context can decide whether the surrounding content makes the destination clear.
5. Collect HTTP status codes and broken-link candidates
First extract internal links, normalise relative URLs, remove fragments, and deduplicate them. Then request each destination carefully and record the status code, final URL, and any request error. Keep the link’s source page so editors can find the affected reference.
Use a timeout, retry policy, and modest rate limit. A temporary server error, redirect, login wall, or blocked request is not proof that a link is permanently broken. Mark uncertain responses for a later check rather than deleting them automatically.
What to double-check
- Rendered versus source HTML: Compare a requests-based crawl with a browser-rendered sample when content is loaded by JavaScript.
- Hidden content: A parser may collect text from menus, dialogs, templates, or elements hidden with CSS. Treat visibility as a review field, not an assumption.
- Accessible names: Matching a label element is helpful, but it does not fully evaluate naming, grouping, descriptions, or widget states.
- Decorative media: Empty alternative text can be intentional. Check the image’s purpose and its relationship to nearby content.
- Heading meaning: A numerical level sequence is a useful signal, but a heading’s clarity and hierarchy require editorial judgement.
- Template scope: Group identical findings by template, component, or content type. Fixing one shared component may resolve many URLs.
- URL variants: Canonicalise tracking parameters, fragments, trailing slashes, and redirects before comparing records.
- Manual interaction: Test keyboard navigation, focus visibility, zoom, error messages, dialogs, media controls, and dynamic updates with a person and appropriate tools.
For broader crawls, validate the extracted data as well as the pages. Check that URLs are unique, status codes are numeric, arrays are present when expected, and the crawl has not silently returned a block page or an error document. The guide on validating scraped data provides a useful quality-control pattern.
Common mistakes
- Calling a missing signal a confirmed violation. Label findings as “needs review”, “not detected”, or “likely issue” until a qualified review confirms them.
- Checking only the homepage. Include templates, product or service pages, articles, forms, search pages, error pages, and representative pages from each content type.
- Using regex to parse HTML. Regular expressions can help process extracted text, but an HTML parser is safer for nested elements, attributes, and malformed markup.
- Ignoring status context. A 3xx response, authentication requirement, rate-limit response, or intermittent timeout needs different handling from a stable client or server error.
- Saving only a pass/fail result. Preserve the selector, source URL, extracted value, timestamp, and parser version so someone can reproduce the finding.
- Overlooking privacy and crawl safety. Avoid storing form values or personal data, and do not submit forms while discovering links.
For selector design and field mapping, see the practical CSS selectors and XPath cookbook. If the crawler itself behaves unexpectedly, use the web scraper troubleshooting playbook before interpreting the results.
When to revisit
Run this accessibility-signal crawl before a redesign, during a content migration, after a component-library change, and before a seasonal publishing cycle. It is also useful as a recurring comparison against a known baseline, provided the crawl configuration remains documented.
Refresh the parser and review rules when your framework, rendering approach, URL structure, or form components change. Recheck HTML examples and accessibility guidance at least every six months, or sooner when your team adopts new standards, testing tools, or design patterns. Standards references and automated-check assumptions can change, so keep them separate from the extraction code.
Practical next step: begin with a representative sample, export the schema above, and manually verify a selection of flagged pages. Then classify findings into content, template, engineering, and false-positive queues. Fix the highest-reuse template issues first, rerun the crawl, and compare the new output with the saved baseline. Use automation to make review more consistent—not to decide accessibility on its own.