Python Web Scraping Tutorial: Requests, Beautiful Soup, Pagination, and Clean Data

Overview

A small Python web scraper usually has five jobs: request a page, parse its HTML, select the fields you need, move through any additional pages, and save the result in a useful format. The code for each step is straightforward. The maintenance work comes from making sensible assumptions explicit and checking that those assumptions remain true.

Before collecting data, confirm that you are allowed to access and use it. Check the website's terms, robots.txt guidance, authentication requirements, and any applicable privacy or data protection obligations. Keep requests proportionate, identify your use case internally, and avoid attempting to bypass access controls.

For a simple static site, the Requests and Beautiful Soup combination is often a suitable starting point. Requests retrieves the HTML response, while Beautiful Soup provides a readable way to locate elements and extract text or attributes. If the required data is added only after JavaScript runs, first check whether an official API or embedded JSON is available. If not, a browser automation tool may be more appropriate. Our guide to APIs versus HTML parsing covers that decision in more detail.

Install the basic dependencies

Create a virtual environment for the project, then install the libraries:

python -m venv .venv
# macOS and Linux
source .venv/bin/activate
# Windows PowerShell
.venv\Scripts\Activate.ps1

pip install requests beautifulsoup4

A useful project layout separates fetching, parsing, and exporting. Even if the first version is a single file, keeping those responsibilities distinct makes later changes safer.

Checklist by scenario

Scenario 1: Fetch one page and extract fields

Start with one known page before adding pagination. This lets you inspect the HTML and verify selectors with a small, predictable input. Set a timeout, check the response, and pass a user agent that identifies the script rather than pretending to be a different browser.

from datetime import datetime, timezone

import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"
headers = {"User-Agent": "ExampleDataResearch/1.0"}

response = requests.get(url, headers=headers, timeout=20)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
product = soup.select_one("article.product")

if product is None:
    raise ValueError("Expected product element was not found")

record = {
    "name": product.select_one("h2").get_text(" ", strip=True),
    "url": product.select_one("a") ["href"],
    "collected_at": datetime.now(timezone.utc).isoformat(),
}

print(record)

The selector in this example is only a placeholder. Replace it after inspecting the target page. Prefer selectors based on stable attributes, semantic elements, or clearly named classes. Avoid relying on a long chain of positional selectors that can break when an unrelated element is added.

Scenario 2: Handle optional or missing fields

Real pages rarely have a complete record on every item. A product may have no sale price, an article may omit an author, or an image may have a lazy-loading attribute instead of a normal src. Use a helper that returns a default value instead of calling get_text() on a missing element.

def text_or_none(parent, selector):
    element = parent.select_one(selector)
    return element.get_text(" ", strip=True) if element else None


def attribute_or_none(parent, selector, attribute):
    element = parent.select_one(selector)
    return element.get(attribute) if element else None

record = {
    "name": text_or_none(product, "h2"),
    "price": text_or_none(product, ".price"),
    "image": attribute_or_none(product, "img", "src"),
}

Decide which fields are required and which are optional. A missing name may indicate a broken selector and should fail validation; a missing discount price may be normal and can be stored as null.

Scenario 3: Follow pagination

Pagination differs between sites. Some use a numbered query parameter, while others provide a “next” link. When a next link exists, follow that link rather than assuming the URL pattern. Resolve relative URLs with urljoin so links such as /page/2 become complete URLs.

from urllib.parse import urljoin


def fetch(url, session):
    response = session.get(url, headers=headers, timeout=20)
    response.raise_for_status()
    return BeautifulSoup(response.text, "html.parser")


records = []
next_url = "https://example.com/products"

with requests.Session() as session:
    session.headers.update(headers)

    while next_url:
        soup = fetch(next_url, session)

        for item in soup.select("article.product"):
            records.append({
                "name": text_or_none(item, "h2"),
                "url": urljoin(next_url, attribute_or_none(item, "a", "href")),
                "price": text_or_none(item, ".price"),
            })

        next_link = soup.select_one("a[rel='next'], a.next")
        next_url = urljoin(next_url, next_link["href"]) if next_link else None

Add a maximum page count or another stopping condition during development. It prevents an incorrect selector from creating an unintended loop. Also consider tracking visited URLs when the site can produce duplicate or cyclical pagination links.

Scenario 4: Export structured JSON or CSV

JSON is convenient when records contain nested values or may evolve over time. CSV is useful for spreadsheets and simple tabular workflows, but every record should use the same set of columns.

import csv
import json

with open("products.json", "w", encoding="utf-8") as file:
    json.dump(records, file, ensure_ascii=False, indent=2)

fields = ["name", "url", "price"]
with open("products.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=fields)
    writer.writeheader()
    writer.writerows(records)

For larger or recurring jobs, consider SQLite or Postgres instead of continually replacing files. The guide on choosing a storage format can help match the destination to the workflow.

What to double-check

• Response status and content: A successful HTTP response does not guarantee that the expected page was returned. Check the status code, content type, and a representative part of the response when debugging.
• Character encoding: Preserve Unicode text and test names, symbols, and accented characters before exporting.
• Relative URLs: Convert image, product, and next-page links with urljoin before storing them.
• Duplicate records: Pagination or repeated modules can produce duplicates. Define a stable key, such as a canonical URL or source identifier, and deduplicate deliberately.
• Whitespace and formatting: Use get_text(" ", strip=True), then normalise values only when the business meaning is clear. Do not remove characters that may be meaningful in prices, codes, or names.
• Validation: Check minimum and maximum record counts, required fields, URL formats, and unexpected empty output. A scraper that finishes without an exception can still have collected the wrong data.
• Request pacing: Add a measured delay where appropriate, avoid unnecessary requests, and use retries carefully. A separate guide covers retries, rate limits, and data validation.

Save a small sample of the raw HTML during development. When extraction fails, comparing the saved response with a current response helps distinguish a selector problem from a request, redirect, or content-delivery problem.

Common mistakes

Starting with every page at once. Test one page, then one complete pagination sequence, before scheduling a wider crawl. This limits the cost of a bad selector and makes debugging easier.

Assuming visible content is in the initial HTML. Requests and Beautiful Soup do not execute JavaScript. Inspect the response source, look for embedded JSON, or identify an appropriate API. If the page is genuinely dynamic, read our material on session-based websites and choose browser automation only when it is necessary.

Using regular expressions as the main HTML parser. Regex can help clean a known text value, but HTML structure is better handled with a parser and CSS selectors. This is especially important when tags are nested or optional.

Ignoring changes in page structure. A class name, pagination control, or embedded data format may change. Keep selectors in one place, log extraction counts, and alert when required fields disappear. See how to detect structure changes before a production job silently degrades.

Mixing extraction and cleaning too early. First capture the source value, then apply a separate normalisation step. This preserves traceability and makes it easier to correct a cleaning rule later. The data-cleaning checklist covers deduplication and validation in more detail.

When to revisit

Review the scraper before a seasonal planning cycle, a scheduled campaign, or any period when the source website is expected to change. Run a test extraction and compare the number of records, required-field completion, duplicate rate, and representative values with the previous run.

Revisit the implementation when the Python environment or dependencies change, when a site introduces a redesign, or when the output is moved from a local file to a database or scheduled job. Check that timeouts, logging, storage paths, and credentials still match the new environment. If you move from a script run manually to cron jobs, include failure notifications and retain enough run metadata to investigate problems.

Use this final checklist before putting a Python web scraper into regular use:

1. Confirm the collection is permitted and narrowly scoped.
2. Test the request and selectors against a known page.
3. Define required fields, optional fields, and normalised formats.
4. Set timeouts, sensible pacing, and bounded pagination.
5. Validate counts and key fields before exporting.
6. Save JSON or CSV with explicit encoding and predictable columns.
7. Record the source URL, collection time, and scraper version.
8. Schedule a periodic review of selectors, dependencies, and output quality.

A maintainable scraper is not just a parser that works today. It is a small data pipeline with clear assumptions, controlled requests, observable failures, and an update routine. That approach keeps a simple Requests and Beautiful Soup project useful as the surrounding workflow changes.