Scrape HTML Tables with Python: BeautifulSoup, Pandas, Spans, and Multiple Pages

Overview

A web table is usually represented by a <table> element containing rows, cells, and sometimes separate header cells. The simplest extraction workflow is:

1. Request the page and check that the response is usable.
2. Locate the correct table rather than assuming it is the first one.
3. Extract headers and cell values.
4. Normalise whitespace, links, numbers, dates, and missing values.
5. Validate row and column counts.
6. Export the result to CSV.

For well-formed tables, pandas.read_html() can perform much of this work in one call. BeautifulSoup is more useful when you need control over the HTML structure, such as extracting a link URL as well as its visible text, handling nested markup, or interpreting row and column spans.

Checklist by scenario

Scenario 1: Extract a simple table with pandas

Install the core packages if necessary:

pip install pandas lxml requests beautifulsoup4

Then try the shortest practical approach:

import pandas as pd

url = "https://example.com/report"
tables = pd.read_html(url)

if not tables:
    raise ValueError("No HTML tables found")

df = tables[0]
df.to_csv("report.csv", index=False)

read_html() returns a list because a page can contain several tables. Never assume the first result is the business table you need. Inspect the candidates before exporting:

for number, table in enumerate(tables):
    print(number, table.shape)
    print(table.head(2))

If the page contains a distinctive heading or table label, use that clue to select the correct result. You can also use match= when a table contains a predictable piece of text.

Scenario 2: Use requests and BeautifulSoup for precise control

BeautifulSoup is a better fit when you need to extract website data beyond visible cell text. The following template captures cell text and the first link in each cell:

import csv
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://example.com/report"
response = requests.get(
    url,
    timeout=30,
    headers={"User-Agent": "TableResearch/1.0"}
)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
table = soup.select_one("table.results")

if table is None:
    raise ValueError("Target table was not found")

rows = []
for tr in table.select("tr"):
    cells = tr.select("th, td")
    if not cells:
        continue

    row = []
    for cell in cells:
        text = " ".join(cell.stripped_strings)
        link = cell.select_one("a[href]")
        href = urljoin(url, link["href"]) if link else ""
        row.append({"text": text, "url": href})
    rows.append(row)

with open("report.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["name", "name_url", "value"])
    for row in rows[1:]:
        writer.writerow([
            row[0]["text"],
            row[0]["url"],
            row[1]["text"] if len(row) > 1 else ""
        ])

Adjust the selector and output columns to match the target table. Keeping link text and link URLs in separate fields is useful for SEO inventories, product catalogues, directory research, and internal-link analysis.

Scenario 3: Tables with missing headers or nested markup

Some tables use the first row as a header without a <thead>. Others use generic cells for everything. Inspect the HTML before deciding which row is the header. With pandas, you can prevent an incorrect first row from becoming a header:

df = pd.read_html(html, header=None)[0]
df.columns = ["name", "category", "value"]

For nested elements, prefer cell.stripped_strings over cell.get_text() when you want whitespace handled consistently. Preserve attributes such as href, data-id, or datetime before discarding the original HTML.

Scenario 4: Merged rows and columns

rowspan and colspan mean that the visual grid may contain fewer cell elements than apparent columns. Pandas can sometimes reconstruct a rectangular table, but the result should be checked rather than trusted automatically. BeautifulSoup gives you the raw spans, allowing you to expand them into explicit values.

A practical rule is to define what a merged value means in your dataset. If a category label applies to four following rows, repeat that category in each output row. If a merged cell represents a subtotal, keep it as a separate row or exclude it deliberately. Do not silently fill values simply because doing so makes the CSV rectangular.

Scenario 5: Multiple pages or pagination

For numbered pages, collect one DataFrame at a time, validate it, and concatenate only compatible results:

import time
import pandas as pd
import requests

frames = []

for page in range(1, 6):
    url = f"https://example.com/report?page={page}"
    response = requests.get(url, timeout=30)
    response.raise_for_status()

    tables = pd.read_html(response.text)
    if not tables:
        raise ValueError(f"No table found on page {page}")

    page_df = tables[0]
    page_df["source_page"] = page
    frames.append(page_df)
    time.sleep(1)

result = pd.concat(frames, ignore_index=True)
result.to_csv("all-pages.csv", index=False)

For a next-page link, follow the link until it is absent rather than guessing the page limit. Add a maximum-page guard, record the source URL, and stop if the same URL appears twice. Use measured request rates and follow the website's published access requirements.

What to double-check

• Table identity: Confirm that the selected table contains the intended records, not a navigation, comparison, or layout table.
• Headers: Check whether headers come from <th> elements, the first row, multiple header rows, or nowhere in the markup.
• Column alignment: Compare several rows, especially around totals, empty cells, and merged cells.
• Data types: Convert prices, percentages, counts, and dates explicitly. Remove display symbols only when their meaning is understood.
• Missing values: Distinguish an empty cell from zero, “not applicable”, and a value that failed to load.
• Duplicates: Pagination can repeat records, particularly when filters or offsets change. Check a stable identifier where available.
• Provenance: Store the source URL, retrieval time, page number, and parser version alongside the output when the data supports recurring analysis.
• Encoding: Open the exported CSV in a way that preserves accents, symbols, and non-Latin text. UTF-8 is a sensible default for Python output.

For SEO and growth work, validation matters as much as extraction. A clean-looking CSV can still omit rows, combine columns incorrectly, or capture a stale page state. Compare the extracted row count with the visible page, sample records manually, and retain a small raw HTML fixture for regression tests. See how to test web scrapers with pytest for a repeatable approach.

Common mistakes

Assuming every table is server-rendered

If the browser displays rows but the downloaded HTML does not contain them, the table may be populated by JavaScript. Inspect network requests for a JSON or CSV endpoint before introducing a browser. If browser rendering is necessary, use a controlled Playwright workflow and wait for the table to become meaningful, not merely for the page to finish loading.

Using regular expressions to parse HTML

Regex can help clean a known text value, but it is fragile for nested and irregular markup. Use an HTML parser for table structure, then apply targeted string or numeric transformations to individual fields.

Dropping useful context during cleaning

Removing links, footnotes, source labels, or page numbers too early can make later auditing difficult. Keep raw or lightly processed fields until the final transformation is verified.

Ignoring rate limits and failures

A scraper that works once may fail during a longer run. Use timeouts, retry only appropriate transient failures, limit concurrency, and log unsuccessful pages. Reliability guidance on rate limits, retries, proxy rotation, and monitoring can help when the workflow grows.

Exporting before normalising

CSV is an output format, not a data model. Before export, decide how to represent dates, decimal separators, currencies, duplicate records, and missing values. For a broader cleanup workflow, use this guide to clean and deduplicate scraped data with Python.

When to revisit

Revisit a table scraper before each seasonal planning or reporting cycle, especially when the output informs search visibility, competitor monitoring, product research, or lead-generation decisions. Also review it whenever the source website changes its table layout, pagination, filters, URL structure, or rendering method.

Keep a short maintenance checklist:

1. Open the source page and confirm the table still exists.
2. Run the scraper against a small sample.
3. Compare headers, row counts, and several representative values.
4. Check that links, spans, dates, and numeric formats still parse correctly.
5. Run automated tests using saved HTML fixtures.
6. Export a fresh CSV and record its source and retrieval date.

When the workflow changes tools, test the new parser against old fixtures before switching production output. If a table becomes a dynamic application, investigate its network data first; a structured response may be more stable than scraping rendered text. For related extraction patterns, see the guides to Python requests, BeautifulSoup, pagination, and clean data and extracting JSON from web pages.

The practical starting point is simple: try pandas.read_html(), inspect the result, and move to BeautifulSoup when the table's structure or metadata requires more control. Treat validation and provenance as part of the extraction, not as optional cleanup after the CSV has already been used.