Web Scraping Reliability Checklist: Rate Limits, Retries, Proxy Rotation, and Monitoring
A scraper is reliable when it collects the right data consistently, responds safely to failures, and makes changes visible before they become expensive. This reusable checklist covers the operational details behind a dependable web scraper: request throttling, retries, proxy decisions, timeouts, validation, logging, alerting, and scheduled maintenance.
Overview
Many scraping projects work during development and become unreliable after they run unattended. A target site changes its HTML, a request takes longer than expected, a temporary error is treated as a permanent failure, or a scheduled job silently stops producing useful records. Reliability comes from treating scraping as an ongoing data pipeline rather than a single script.
Before collecting data, review the target site's robots.txt guidance and access requirements. Check the site's terms, authentication rules, and the volume your use case requires. Prefer an official API where it provides the data you need, and design requests to be proportionate. A proxy should not be used to bypass access controls or conceal abusive traffic; if proxies are appropriate for a legitimate workload, document why they are needed and manage them carefully.
A practical reliability plan answers five questions:
- How quickly may the scraper send requests?
- Which failures are safe to retry, and how many attempts are allowed?
- How will the scraper handle timeouts, blocked requests, and changing content?
- How will you know whether the output is complete and valid?
- Who or what receives an alert when the job falls outside its normal behaviour?
Keep these decisions in configuration rather than scattering them through the code. That makes a Python web scraper, Playwright workflow, or scheduled crawler easier to review and adjust.
What to track
Request rate and response behaviour
Rate limiting scraping is the first control to review. Set a deliberate delay between requests, then add jitter so a large batch does not produce a perfectly regular pattern. Use separate limits for different hosts when appropriate, because one site's acceptable pace may be unsuitable for another. Track request count, response status, latency, timeout count, and the number of pages skipped.
A simple configuration might look like this:
REQUEST_DELAY_SECONDS = 2.0
REQUEST_JITTER_SECONDS = 0.5
REQUEST_TIMEOUT_SECONDS = 20
MAX_RETRIES = 3
MAX_PAGES_PER_RUN = 1000These are starting points, not universal settings. Adjust them to the workload, the site's documented limits, and the consequences of missing data. A lower request rate is often preferable to repeatedly restarting a failed job.
Retries, backoff, and failure categories
Web scraping retries should be selective. A temporary network error or a service-unavailable response may be worth retrying. A malformed URL, a missing required field, or a consistent permission response usually needs investigation rather than repeated requests. Exponential backoff spaces out attempts after repeated failures:
delay = base_delay * (2 ** attempt) + random_jitterSet a maximum delay and an attempt limit. Record the final error after the limit is reached so the failed item can be reviewed or queued for a later run. Do not retry indefinitely; an unbounded retry loop can increase load while hiding a broken workflow.
Proxy strategy and session state
Proxy rotation for scraping is not a substitute for sensible request pacing. First determine whether you need a proxy at all. For a small, permitted workload, a stable connection may make debugging and attribution easier. If a distributed network is required for a legitimate operational reason, track the proxy used, its outcome, and its failure rate without retaining unnecessary personal data.
Rotate only when the workflow calls for it. Frequent, unexplained changes can complicate sessions, cookies, authentication, and troubleshooting. Keep credentials in a secret manager or environment variables, not in source code. For login-based collection, define how sessions expire and how the job should stop when authentication fails. See the guide to session-based websites for a fuller workflow.
Timeouts, validation, and completeness
Use separate timeouts for connection, response, and browser actions where your client supports them. A headless browser may need more time for page rendering than a direct HTTP request, but it should still have a clear upper bound. For dynamic targets, use stable readiness conditions instead of arbitrary long sleeps; the Playwright browser automation workflow provides useful context.
Validate data before writing it to storage. Check required fields, expected types, valid URLs, plausible ranges, duplicate keys, and the number of records returned. Compare the current count with recent runs, but treat volume changes as a signal for investigation rather than proof that a run succeeded. A scraper can return a successful HTTP response and still extract an empty page or an error document.
Logs, metrics, and alerts
At minimum, log the job identifier, start and end time, target, page or item identifier, status, duration, retry count, and final outcome. Avoid logging passwords, session cookies, tokens, or unnecessary personal information. Useful metrics include success rate, extraction yield, duplicate rate, validation failures, average latency, and the age of the newest successful dataset.
Alerts should be tied to action. Examples include a job that fails completely, a run with zero valid records, a sudden rise in timeouts, or data that has not refreshed within its expected window. A notification that fires for every minor variation will eventually be ignored, so use thresholds based on the normal behaviour of that particular job.
Cadence and checkpoints
Use a maintenance schedule that matches how often the data matters. A high-value daily pipeline may need a daily health check and a monthly review. A low-frequency research job may only need a check after each run and a quarterly audit.
| Cadence | Checkpoint | Action |
|---|---|---|
| Every run | Exit status, record count, validation errors, and newest timestamp | Stop bad output from replacing the last known-good dataset |
| Weekly | Latency, retry rate, skipped pages, and proxy or connection failures | Investigate gradual degradation before it becomes an outage |
| Monthly | Selectors, page structure, authentication, storage, and alert thresholds | Run a controlled test and update configuration where needed |
| Quarterly | Data necessity, access assumptions, request volume, and dependencies | Remove obsolete targets and confirm the workflow remains appropriate |
For scheduled execution, a cron job or task scheduler should expose failures rather than merely run commands. Capture standard output and errors, use a lock to prevent overlapping runs, and define what happens if a previous run is still active. Store the last successful run separately from the latest attempted run so downstream users can distinguish fresh data from stale data.
Keep a maintenance tracker with columns for target, owner, schedule, last successful run, last structure review, known rate limit, retry policy, proxy requirement, validation checks, and next review date. This turns reliability work into a repeatable process instead of relying on memory.
How to interpret changes
Not every change indicates a technical failure. A lower record count may reflect a genuine change in the source, a seasonal event, pagination differences, or a new filter. A higher count may indicate duplicates or an unexpectedly broad selector. Compare several signals before deciding what happened.
- High status success, low valid records: inspect the response body and selectors; the page may contain an error or a changed layout.
- Rising timeouts with stable counts: review page complexity, concurrency, connection health, and timeout settings.
- More retries and changing response codes: reduce request pressure, check access requirements, and inspect whether a dependency changed.
- Normal counts but stale timestamps: verify that the scraper is extracting current content rather than cached or default content.
- Duplicate growth: review pagination, item identifiers, resume logic, and deduplication before publishing the dataset.
When a structure change is suspected, save a small sample of the raw response and the parsed output, then compare it with a known-good run. A focused fixture test can detect a missing selector without repeatedly contacting the live site. The guide on detecting website structure changes can help you build this check into deployment and monitoring.
When to revisit
Revisit this checklist at least monthly for active scrapers and quarterly for lower-frequency projects. Review it sooner when the target changes its layout, introduces authentication changes, publishes new access guidance, or shows a sustained change in response behaviour. Also review it when your own requirements change: a larger crawl, a new data field, a tighter freshness requirement, or a move from direct requests to browser automation.
Use this practical closeout sequence:
- Confirm the permitted scope, target URLs, and data fields are still necessary.
- Record current request rates, timeout values, retry limits, and proxy settings.
- Run a small test against representative pages before a full scheduled job.
- Check both raw responses and validated output, including empty and duplicate results.
- Test one controlled failure so you know retries, logging, alerts, and recovery work.
- Update the maintenance tracker with the owner, evidence, decision, and next review date.
Reliability is maintained through these small recurring checks. If you need to revisit the extraction layer itself, compare the approaches in APIs versus HTML parsing, or review the reliable Python web scraper pattern before changing production code.
