Collecting useful information from PDFs requires more than downloading every link that ends in .pdf. This practical Python checklist shows how to discover document URLs, download and validate files safely, extract text and metadata, parse tables, handle scanned pages with OCR, and preserve enough provenance to make the resulting dataset useful for SEO, research, and growth analysis.
Overview
PDF collections often contain reports, product catalogues, regulatory documents, price lists, research papers, and archived website content. They can be valuable sources for a data extraction workflow, but their structure varies considerably. One file may contain selectable text and clean tables; another may be an image-only scan; a third may use a layout that looks tabular to a person but has no machine-readable rows or columns.
A reliable PDF scraper therefore separates the work into stages:
- Discover: find candidate document URLs from HTML pages, sitemaps, feeds, navigation pages, or known URL patterns.
- Download: request files with sensible timeouts, rate limits, and error handling.
- Validate: confirm that the response is actually a PDF and record its URL, status, size, hash, and retrieval time.
- Extract: choose text, table, or OCR processing according to the document’s layout.
- Normalise: clean whitespace, numbers, dates, headers, and repeated page elements.
- Store: retain the original file or a stable reference alongside the extracted data.
- Check: measure missing fields, duplicate files, changed layouts, and extraction failures before using the data.
For broader field-mapping principles, see the guide to CSS selectors and XPath for web scraping. PDF links are usually discovered in HTML, but the extraction stage needs document-specific tools.
A small discovery and download example
The following pattern finds PDF links on a page and downloads them without assuming that a file extension alone proves the response is a PDF.
from pathlib import Path
from urllib.parse import urljoin
import hashlib
import requests
from bs4 import BeautifulSoup
page_url = "https://example.com/reports"
output_dir = Path("downloads")
output_dir.mkdir(exist_ok=True)
response = requests.get(page_url, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
for link in soup.select("a[href]"):
file_url = urljoin(page_url, link["href"])
if not file_url.lower().split("?")[0].endswith(".pdf"):
continue
pdf_response = requests.get(file_url, timeout=60)
pdf_response.raise_for_status()
content = pdf_response.content
if not content.startswith(b"%PDF"):
continue
digest = hashlib.sha256(content).hexdigest()[:16]
path = output_dir / f"{digest}.pdf"
path.write_bytes(content)
print(file_url, path)In production, add a deliberate delay, a clear user agent, retry rules for temporary failures, a maximum file size, and logging. Review the source website’s terms, access instructions, and applicable requirements before collecting documents at scale. Do not use a browser or proxy rotation simply to bypass access controls; first look for a permitted download route or an official data source.
Checklist by scenario
Scenario 1: PDFs are linked from ordinary HTML pages
- Crawl only the relevant listing pages or permitted site area.
- Resolve relative URLs with
urljoinand remove obvious tracking parameters where appropriate. - Deduplicate URLs before downloading.
- Capture the page URL where each document was found.
- Record anchor text, visible title, publication date, and any category shown beside the link.
If the site uses pagination, inspect next-page links and stable archive paths. If links are inserted after page load, inspect the page’s embedded data or network requests before reaching for a headless browser. The guide to extracting JSON from web pages can help when document metadata is stored in JSON-LD or embedded application state.
Scenario 2: You need plain text from digitally generated PDFs
Start with PyMuPDF, also used through the fitz import in many Python environments, when you need fast page text, page counts, links, or basic metadata. A simple extraction looks like this:
import fitz
doc = fitz.open("report.pdf")
text_by_page = [page.get_text("text") for page in doc]
full_text = "\n".join(text_by_page)Preserve page numbers rather than joining everything into one untraceable string. Page-level text makes it easier to locate a value, compare revised editions, and identify pages that returned no text.
Scenario 3: You need tables from a text-based PDF
Use pdfplumber when you need access to word positions, lines, rectangles, and table settings. It is useful for documents with consistent ruling lines or predictable whitespace. Camelot is often a practical choice for structured tables when you can tune extraction settings; its lattice and stream approaches suit different visual layouts. Tabula, commonly used through a desktop workflow or Python integrations, can be useful when you want to inspect and select table regions interactively.
Table extraction is layout-dependent. Test several representative pages, including a continuation page and a page with long values. Treat extracted cells as candidates until you check column alignment, merged cells, footnotes, and repeated headers. For HTML tables, the separate BeautifulSoup and pandas table guide is more appropriate.
| Tool or workflow | Best starting point | Watch for |
|---|---|---|
| PyMuPDF | Fast text, pages, links, and metadata | Complex table structure may need extra parsing |
| pdfplumber | Text positions and layout-aware extraction | Settings may need tuning for each document family |
| Camelot | Structured tables with detectable lines or spacing | Scans and irregular layouts are poor candidates without preprocessing |
| Tabula | Interactive table-region selection and repeatable exports | Results depend on accurately selecting the table area |
| OCR workflow | Image-only or scanned documents | Recognition errors, reading order, and table reconstruction |
Scenario 4: The PDF is scanned or image-only
Check whether a text layer exists before starting OCR. If page extraction returns empty or nearly empty text, render pages to images and pass them through an OCR engine. OCR output should be treated as an estimate: verify dates, decimal points, minus signs, units, and names against the page image. Tables usually need an additional reconstruction step because OCR may return words without reliable row and column boundaries.
Scenario 5: You are building a repeatable report or SEO dataset
Store one record per document and, where useful, one record per extracted item. A practical document schema includes:
{
"document_url": "https://example.com/report.pdf",
"source_page_url": "https://example.com/reports",
"retrieved_at": "2026-01-15T10:30:00Z",
"published_date": null,
"title": "Example report",
"file_sha256": "...",
"page_count": 12,
"extraction_method": "pymupdf",
"text_status": "complete",
"table_status": "not_attempted",
"revision_label": null
}For SEO and growth work, add fields such as topic, geography, product, metric, unit, period, and source page. Keep the raw value and the normalised value when conversion is involved.
What to double-check
- File identity: inspect the content signature and content type, not only the URL suffix. Error pages can be saved under a PDF-looking filename.
- Versioning: compare file hashes, publication dates, revision labels, and page counts. A changed document URL does not necessarily mean the content changed, and a stable URL can serve a new edition.
- Text completeness: measure blank pages, unusually short pages, and unexpected character ratios. These are useful signals for OCR or manual review.
- Numeric integrity: check decimal separators, thousands separators, currency symbols, percentages, negative values, and units before aggregation.
- Table shape: verify expected headers, column counts, row counts, and continuation markers. Flag rows that do not fit the schema instead of silently dropping them.
- Provenance: retain the source URL, page number, retrieval time, extraction method, and, when possible, a text snippet or bounding-box reference.
- Operational behaviour: use bounded concurrency, timeouts, backoff for temporary errors, and a cache so repeated runs do not redownload unchanged files.
Once extracted, apply the same quality discipline used for other scraped datasets. The checklist in How to validate scraped data covers missing values, duplicates, and schema changes.
Common mistakes
- Assuming every PDF is text-based. Test a sample page and route scanned files to OCR.
- Using one table setting for every file. Group documents by template and maintain settings per layout family.
- Discarding the original document. Keep a stable local copy or content hash so results can be audited and reprocessed.
- Flattening text without page boundaries. Page context is essential for citations, debugging, and locating extraction errors.
- Trusting successful code execution. A parser can complete without raising an error while returning shifted columns or missing rows.
- Ignoring revised editions. Deduplicating only by URL can overwrite historical values that should remain available for comparison.
- Overusing browser automation. Start with requests and HTML inspection; use Playwright or another browser only when the document links genuinely depend on client-side rendering.
- Skipping tests. Keep fixtures for representative PDFs and add regression checks for page count, required fields, and table dimensions. See the guide to testing web scrapers with pytest.
When to revisit
Revisit the workflow before each seasonal planning or reporting cycle, especially when a new batch of documents is expected. Run a small discovery check when the source website changes its archive navigation, download buttons, URL structure, or embedded data. Review package compatibility and OCR behaviour when you upgrade Python or extraction libraries.
Use a representative test set containing a clean text PDF, a multi-page table, a scanned document, a revised edition, and a deliberately invalid download. Re-run it whenever tools or workflows change. Record which parser handled each document family, which pages require OCR, and which fields need manual review.
A practical final checklist is: discover permitted URLs; deduplicate them; download with limits; validate file signatures; hash and archive inputs; classify document layouts; extract text or tables; OCR only when needed; preserve page-level provenance; validate numbers and schemas; compare revisions; and publish only records that pass quality checks. When the source changes or a batch fails, begin with the web scraper troubleshooting playbook rather than silently accepting incomplete output.