How to Validate Scraped Data: Quality Checks for Missing Fields, Duplicates, and Unexpected Changes
Data qualityPythonValidationSchemasETLMonitoring

How to Validate Scraped Data: Quality Checks for Missing Fields, Duplicates, and Unexpected Changes

WWebscraper.uk Editorial Team
2026-11-13
8 min read

A practical workflow for validating scraped data with schemas, missing-field checks, duplicate detection, row counts, and change monitoring.

Reliable scraping starts after extraction. This guide shows how to validate scraped data before it reaches a report, database, model, or customer workflow, using schemas, required-field checks, type validation, duplicate detection, range tests, row-count monitoring, and HTML-change detection. Use the workflow for a new Python web scraper or as a recurring checkpoint for an established pipeline.

Overview

A scraper can complete successfully while returning incomplete or misleading data. A page redesign may cause selectors to return empty strings, a pagination loop may stop early, or a blocked request may produce an error page that looks like valid HTML. Without data-quality checks, these failures can be mistaken for real changes in the source.

Validation should therefore be treated as a separate stage of the extraction workflow:

  1. Extract: request pages, APIs, or browser-rendered content.
  2. Normalise: convert dates, prices, identifiers, and text into consistent formats.
  3. Validate: test structure, completeness, uniqueness, plausibility, and source changes.
  4. Store or publish: write only accepted records, while retaining rejected data and diagnostics.

Do not rely on a single check. A dataset can have the expected number of rows but still contain missing prices, duplicated products, or shifted columns. Strong validation combines record-level rules with dataset-level comparisons and source monitoring.

What to track

1. Define a schema and required fields

Start by describing what a valid record looks like. For a product scraper, the schema might include product_id, name, price, currency, availability, url, and scraped_at. Mark each field as required, optional, or conditionally required. For example, a sale-price field may be empty when an item is not discounted, while the regular price should always be present.

Schema rules should cover:

  • Allowed fields and their data types.
  • Required fields that cannot be null or blank.
  • Permitted values, such as a controlled availability status.
  • Format rules for URLs, dates, currencies, and identifiers.
  • Relationships between fields, such as a sale price being no greater than the regular price when both exist.

In Python, these rules can be implemented with ordinary functions, dataclasses, or a validation library. The choice matters less than making the rules explicit, version-controlled, and testable.

2. Check missing and blank values

Measure missingness by field rather than only counting invalid rows. A selector failure often affects one column across nearly every record. Treat null, empty strings, whitespace-only values, and placeholder text such as “N/A” according to the field’s meaning. Keep a missing-value report that includes both the count and percentage of affected records.

Set acceptance thresholds before a run. For example, an identifier or canonical URL might require zero missing values, while a description could permit a small proportion of blanks. Thresholds are business rules, not universal constants, so record why each threshold exists.

3. Validate types, formats, and normalisation

Extraction often produces strings even when the destination requires numbers or dates. Convert values deliberately and reject ambiguous conversions. A price such as 1,299.00 must be parsed according to the source’s number format, and a date should include timezone assumptions where they affect comparisons.

Useful checks include:

  • Prices are numeric, non-negative, and associated with a known currency.
  • Dates parse successfully and fall within a sensible range.
  • URLs use an expected scheme and point to the intended domain or path.
  • Identifiers match the expected pattern and preserve leading zeroes where relevant.
  • Text fields do not contain HTML fragments, error messages, or navigation labels unexpectedly.

4. Detect duplicates and key collisions

Choose a stable business key before deduplicating. A product ID is usually preferable to a name, while a URL may be suitable for an SEO crawler. If no single field is unique, use a composite key such as source, product ID, and variant.

Report duplicates before removing them. A sudden increase may indicate repeated pagination, a missing variant field, or a source that now returns the same item through multiple paths. Automatic deduplication should not hide the underlying extraction problem.

5. Test ranges, relationships, and outliers

Outlier checks help identify values that are technically valid but operationally implausible. Compare a new run with historical data where appropriate, but avoid treating every real change as an error. A price can change, a search result can move, and a page count can vary.

Use several levels of checks:

  • Hard bounds: reject impossible values, such as negative quantities.
  • Relative changes: flag unusually large changes from the previous accepted run for review.
  • Cross-field rules: compare related values, such as minimum and maximum prices.
  • Distribution checks: monitor counts by category, status, domain, or page.

6. Compare row counts and coverage

Record the number of requests, successful responses, extracted records, rejected records, unique records, and stored records. Also track pagination depth, category coverage, HTTP status groups, and the share of records with required fields.

A row-count alert is a useful signal, but it is not proof of quality. If a scraper normally returns 1,000 records and suddenly returns 120, investigate. If it returns 1,000 records with the same title repeated, row count alone will not reveal the failure.

7. Detect HTML and response changes

When a website changes, selectors may fail silently. Store lightweight diagnostics for each run, such as response status, content length, page title, selected container counts, and a hash of relevant HTML or embedded JSON. You do not need to archive every page indefinitely; retain enough evidence to compare a failed run with the last accepted run.

Check for signs of an unexpected response: login forms, access-denied text, challenge pages, consent screens, empty application shells, or a sudden change in the number of matching selectors. Respect the site’s terms, robots guidance where applicable, and reasonable request rates while diagnosing these changes.

Reusable validation template

def validate_records(records, previous_count=None):
    required = ["product_id", "name", "price", "url"]
    errors = []

    for index, record in enumerate(records):
        for field in required:
            value = record.get(field)
            if value is None or str(value).strip() == "":
                errors.append((index, field, "missing"))

        if record.get("price") is not None and record["price"] < 0:
            errors.append((index, "price", "negative"))

    ids = [r.get("product_id") for r in records]
    duplicate_ids = {item for item in ids if item and ids.count(item) > 1}

    summary = {
        "row_count": len(records),
        "error_count": len(errors),
        "duplicate_ids": sorted(duplicate_ids),
        "accepted": not errors and not duplicate_ids,
    }

    if previous_count and len(records) < previous_count * 0.5:
        summary["count_warning"] = "row count dropped sharply"

    return summary, errors

For production use, improve this example by counting missing values efficiently, separating warnings from blocking errors, validating types before comparisons, and returning structured error records. Store the validation result alongside the scraper run rather than printing it only to a console.

Common failures and diagnostic tests

Observed failureDiagnostic testLikely area to inspect
Many required fields are blankField-level missingness and selector match countsCSS/XPath selectors, page state, consent or challenge response
Row count falls sharplyCompare requests, status codes, pagination depth, and category coveragePagination logic, rate limiting, changed URLs, blocked responses
Duplicate records increaseUnique-key and composite-key countsPagination cursors, variant extraction, canonical identifiers
Prices or dates look incorrectType, currency, format, range, and cross-field checksLocale parsing, text cleanup, changed markup
All pages have similar contentContent hashes and repeated-title or repeated-ID countsMissing pagination parameter, cached response, failed navigation
Parser errors appear suddenlyHTML length, title, status, and sample-response comparisonTemplate redesign, access response, API or embedded-state change

Cadence and checkpoints

Run validation every time the scraper runs, but review the rules on a slower schedule. A recurring job should produce a compact quality record containing the run timestamp, code version, source scope, request totals, accepted and rejected rows, duplicate count, missingness by field, and warnings.

For daily or high-frequency extraction, use automated blocking checks for identifiers, response validity, and catastrophic row-count changes. Review trends weekly or monthly. For less frequent projects, perform the same checks on every run and compare with the last accepted dataset.

Keep a small set of representative fixtures for regression testing: a normal page, a page with optional fields missing, a paginated response, and a known error or challenge response. This complements live validation and makes parser changes safer. The related guide on testing web scrapers with Pytest provides a useful next step for turning these cases into repeatable tests.

How to interpret changes

A warning is not automatically a failure. First classify the change as expected, suspicious, or confirmed broken:

  • Expected: a seasonal category change, a known product removal, or a planned schema update.
  • Suspicious: an unusual count shift, new null pattern, or outlier that needs a sample review.
  • Confirmed broken: an error page, empty selector, invalid type conversion, or duplicate pattern caused by extraction logic.

Inspect a small sample of accepted and rejected records, compare raw responses with the previous run, and identify the earliest point where the numbers diverged. Do not silently widen thresholds to make a pipeline green. If the source has genuinely changed, update the schema, fixtures, parser, and acceptance rules together, documenting the reason.

For recurring datasets, retain the previous accepted version so downstream users have a stable reference. A failed run should normally be quarantined rather than overwriting a trusted dataset. This is especially important for price history, SERP tracking, lead lists, and SEO crawls, where a partial run can look like a meaningful business change.

When to revisit

Revisit your validation rules at least monthly or quarterly, depending on how often the source and scraper run. Also review them immediately after a failed validation, a source redesign, a parser update, a change in pagination or API behaviour, or a noticeable shift in the data’s normal range.

Use this practical checklist before accepting a new or recurring dataset:

  1. Confirm the schema and required fields still match the project’s purpose.
  2. Check missingness, type conversions, formats, and allowed values.
  3. Compare unique keys and investigate duplicates before deduplication.
  4. Review row counts, page coverage, status codes, and response samples.
  5. Run range, relationship, and outlier checks against documented expectations.
  6. Compare HTML or embedded-data fingerprints with the last accepted run.
  7. Quarantine failures, save diagnostics, and notify the owner with a specific reason.
  8. Update fixtures, schemas, and thresholds when a confirmed source change is intentional.

Validation is most useful when it creates a clear decision: accept, investigate, or reject. Build that decision into the scraper’s scheduled workflow, revisit the rules when the source or use case changes, and treat every unexpected result as evidence to improve the next run.

Related Topics

#Data quality#Python#Validation#Schemas#ETL#Monitoring
W

Webscraper.uk Editorial Team

Technical Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.