How to Scrape Product Prices with Python: Currency, Availability, and Price History
PythonEcommerce scrapingPrice extractionCurrency normalizationProduct dataPrice history

How to Scrape Product Prices with Python: Currency, Availability, and Price History

WWebScraper.uk Editorial Team
2026-11-10
7 min read

Build a reliable Python product-price scraper with currency normalization, availability checks, sale-price handling, validation, and price history.

Learn how to scrape product prices with Python in a way that remains useful after a retailer changes its layout. This guide covers product-page discovery, HTML and JSON-LD extraction, sale and subscription pricing, currency and decimal normalization, availability checks, validation, and timestamped price history.

Overview

A reliable ecommerce price scraper does more than find a number next to a currency symbol. It needs to answer several questions consistently: Which product is this? Is the displayed amount a standard price, a sale price, or a recurring subscription price? What currency and decimal convention are being used? Is the item available? When was the observation collected?

The most maintainable workflow separates extraction from normalization and validation. First, collect candidate values from the page. Next, convert them into a predictable internal format. Finally, check that the result makes sense before storing it in a price-history table.

For static pages, Python's requests and BeautifulSoup are often sufficient. Product information may also appear in JSON-LD structured data, which can be easier to parse than presentation markup. If the price is added after JavaScript runs, use a browser automation tool such as Playwright, or identify the permitted network response that supplies the product data. The choice of approach is discussed further in the Python web scraping stack guide.

Only collect data you are permitted to access, respect a site's terms and technical controls, and use conservative request rates. A price tracker should be dependable without placing unnecessary load on a retailer's site.

How to estimate

Before writing selectors, define the record that the scraper should produce. A practical minimum is:

  • product_id or a stable product URL
  • product_name
  • currency
  • price, stored as a decimal rather than a binary floating-point value
  • price_type, such as regular, sale, or subscription
  • availability
  • observed_at, preferably in UTC
  • source_url and an optional extraction method

Estimate the expected output before collecting at scale. If you monitor N product URLs at an interval of F runs per day, the approximate daily observation count is:

daily observations = N × F

This is useful for planning storage and checking whether a schedule is sensible. It does not predict whether every request will succeed. For that, track success rate, missing-price rate, response status, parsing errors, and the number of records whose values changed unexpectedly.

For price comparisons, decide whether the business question concerns the displayed consumer price, the lowest available offer, a unit price, or a recurring charge. These are different measurements. A monthly subscription price should not silently replace a one-time purchase price, and a crossed-out list price should not be treated as the current payable amount without an explicit rule.

Common product-price markup patterns

PatternWhat to look forRecommended handling
Visible price elementA product-page element with a price class or data attributeExtract text, then normalize symbols, separators, and whitespace.
Sale price and regular priceSeparate current and previous-price elementsStore both when available, but mark the current amount explicitly as the payable price.
JSON-LDProduct data containing an offers objectParse structured fields first or use them as a cross-check against visible markup.
Embedded application stateA script containing product or offer dataParse the documented, permitted data structure carefully; expect nesting to vary.
Dynamic priceNo price in the initial HTML, but a browser displays oneUse Playwright or inspect the relevant permitted network response.
Subscription or unit priceLabels such as “per month”, “per user”, or “per item”Store the amount and billing unit separately from the headline price.

Inputs and assumptions

Price extraction becomes more reliable when assumptions are explicit. Currency symbols are not always unique, and decimal punctuation varies by locale. For example, a value containing a comma may represent a decimal fraction in one locale and a thousands separator in another. Prefer a page-level currency code, structured-data field, URL locale, or known retailer locale over guessing from the symbol alone.

Use Decimal for monetary values. Keep the raw text as an audit field so a failed normalization can be investigated later. Do not use a regular expression as the only source of truth: regex can help identify a candidate, but it cannot reliably determine whether a number is a price, a product dimension, or a promotional percentage.

The following compact template illustrates the extraction boundary. Selectors are examples only; retailer-specific selectors must be inspected and tested.

from decimal import Decimal
from datetime import datetime, timezone
import re
from bs4 import BeautifulSoup

CURRENCY_MAP = {"£": "GBP", "$": "USD", "€": "EUR"}

def normalize_price(raw, decimal_mark="."):
    if not raw:
        return None
    text = raw.strip()
    symbol = next((s for s in CURRENCY_MAP if s in text), None)
    cleaned = re.sub(r"[^0-9,.-]", "", text)
    if decimal_mark == ",":
        cleaned = cleaned.replace(".", "").replace(",", ".")
    else:
        cleaned = cleaned.replace(",", "")
    try:
        return {
            "amount": Decimal(cleaned),
            "currency": CURRENCY_MAP.get(symbol)
        }
    except Exception:
        return None

def extract_product(html, url):
    soup = BeautifulSoup(html, "html.parser")
    name_node = soup.select_one("[data-product-name], h1")
    price_node = soup.select_one("[data-price], .price")
    availability_node = soup.select_one("[data-availability], .availability")

    parsed = normalize_price(price_node.get_text(" ", strip=True)
                             if price_node else None)
    return {
        "product_name": name_node.get_text(" ", strip=True)
            if name_node else None,
        "price": str(parsed["amount"]) if parsed else None,
        "currency": parsed["currency"] if parsed else None,
        "availability": availability_node.get_text(" ", strip=True)
            if availability_node else None,
        "source_url": url,
        "observed_at": datetime.now(timezone.utc).isoformat()
    }

In production, replace broad exception handling with targeted validation and return an extraction status. A record with no price should not be written as a zero price. Treat missing, malformed, and changed fields as distinct outcomes.

Structured data can provide fields such as price, priceCurrency, and availability. It is useful, but it should still be compared with the visible offer where possible. See the guide to extracting JSON from web pages for techniques covering JSON-LD and embedded state.

Worked examples

Example 1: a visible sale price

Suppose a page contains a current price of £49.99 and a previous price of £64.99. Store the current value as 49.99, the currency as GBP, and the previous value separately as 64.99. Add price_type = "sale" only when the page labels or markup supports that interpretation. This makes later comparisons clear: a change from 49.99 to 54.99 is different from a change caused by a promotion ending.

Example 2: locale-specific formatting

If a retailer presents 49,99 €, the scraper needs a rule that identifies the page locale or currency before converting the text to Decimal("49.99"). Store the raw value and the applied locale rule. If the locale cannot be established, flag the record for review rather than silently applying a guess.

Example 3: subscription pricing

A page might show 12.00 per month beside a one-time setup fee. These should become separate fields, for example amount = 12.00, billing_period = "month", and setup_fee = 0 only if the page clearly indicates no fee. Do not combine them into a single “price” field unless your comparison specifically defines how recurring and one-time charges should be calculated.

Validate before storing history

Useful checks include: a price must be present when the product is offered for sale; the currency must be known or explicitly marked unknown; an amount must not be negative unless the data model allows credits; and a large change should be flagged rather than automatically discarded. Compare multiple signals where available: visible markup, JSON-LD, and the selected availability label.

Store one observation per product and collection time. A simple table can include product_id, observed_at, amount, currency, price_type, availability, raw_price, and parser_version. The parser version helps explain why values changed after a selector update. For data cleaning and duplicate handling, see this Python data-cleaning guide.

When to recalculate

Revisit the scraper whenever a retailer changes its product template, locale, checkout model, structured-data implementation, or availability labels. A missing price, sudden drop to zero, repeated currency mismatch, or unusual increase in parsing failures is a reason to inspect the page rather than extend the historical trend.

Review collection frequency when your monitoring goal changes. A daily snapshot may suit broad comparison, while a shorter interval may be appropriate for a narrowly defined operational requirement. Recalculate expected request volume with product count × runs per day, then confirm that the schedule, retry policy, and storage design remain appropriate.

For a practical maintenance routine:

  1. Keep a small fixture of representative product pages and test it after selector changes.
  2. Record missing fields and parsing errors separately from genuine out-of-stock products.
  3. Run a validation report before publishing price comparisons.
  4. Review currency and subscription rules whenever a new retailer or region is added.
  5. Use bounded retries, delays, and monitoring rather than repeatedly requesting failed pages.

Automated tests can catch layout changes before they contaminate price history; the pytest web-scraper testing guide covers fixtures, mocks, and regression checks. If the page is rendered dynamically, consult the Playwright workflow for dynamic websites. Recheck this guide whenever the inputs change: a new retailer, a new currency, a new billing model, or a change in the page's product markup should lead to a fresh extraction test before the tracker resumes normal collection.

Related Topics

#Python#Ecommerce scraping#Price extraction#Currency normalization#Product data#Price history
W

WebScraper.uk Editorial Team

Technical SEO and Developer Tools Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.