A broken web scraper is usually diagnosable if you separate transport, access, rendering, extraction, and data-quality failures. This troubleshooting playbook gives you a repeatable workflow, an error-to-fix reference, quick test scripts, and a maintenance schedule for Requests and BeautifulSoup, Scrapy, Playwright, and Puppeteer projects.
Overview
When a scraper stops working, the visible symptom is often misleading. An empty result may mean that a CSS selector changed, JavaScript content has not rendered, the server returned a challenge page, or the request failed before any HTML was received. Treating every problem as a selector issue leads to wasted debugging time and unnecessary changes.
Use this order of investigation:
- Confirm the request: Check the URL, method, parameters, redirects, timeout, and final status code.
- Inspect the response: Save the status, headers, content length, and a small HTML snapshot.
- Confirm the content type: Establish whether the data is in the initial HTML, embedded JSON, an API response, or a browser-rendered view.
- Test extraction separately: Run selectors against a saved fixture rather than repeatedly requesting the live site.
- Check operational conditions: Review rate limits, concurrency, proxy health, cookies, authentication, and scheduled-job changes.
- Record the fix: Keep the failing response, the diagnosis, and the regression test in version control or an incident log.
Make only one material change at a time. A minimal reproduction is more valuable than a large rewrite: one URL, one request, one parser, and one expected field can usually identify the failing layer.
What to track
Request and response evidence
Log enough information to reproduce a failure without logging sensitive credentials. Useful fields include the timestamp, target host, URL pattern, HTTP method, status code, final URL after redirects, response content type, response size, elapsed time, retry count, and parser version. Store a redacted HTML or JSON snapshot for representative failures where permitted.
A small Requests diagnostic can expose the difference between a network problem and an extraction problem:
import requests
url = "https://example.com/page"
response = requests.get(
url,
headers={"User-Agent": "YourResearchBot/1.0"},
timeout=20,
allow_redirects=True,
)
print("status:", response.status_code)
print("final_url:", response.url)
print("type:", response.headers.get("content-type"))
print("bytes:", len(response.content))
print(response.text[:500])
Use an honest, identifiable user agent where appropriate, respect the site's terms and published access guidance, and keep request rates conservative. Do not use diagnostic logging as a reason to expose cookies, authorization headers, or personal data.
Extraction and data-quality signals
Track the number of records returned, required-field completeness, duplicate rate, parsing exceptions, and the proportion of pages that produce an empty result. A scraper can return HTTP 200 while silently extracting nothing, so a status-code check alone is not a health check.
For dynamic pages, also record browser launch failures, navigation timeouts, console errors, failed network requests, and whether the expected selector appeared. If the page contains structured data or an embedded application state object, compare that source with the visible DOM before adding brittle regular expressions. The guide to extracting JSON from web pages covers common places to look.
A compact error-to-fix reference
| Symptom | Likely cause | First fix to test |
|---|---|---|
| Connection timeout | Slow origin, network route, overloaded proxy, or an overly short timeout | Test the same URL from a stable environment, increase the timeout modestly, and inspect latency before adding retries. |
| Connection refused or DNS failure | Unavailable host, incorrect proxy configuration, or temporary network failure | Resolve the hostname and request the URL without the proxy; verify proxy credentials and endpoint settings. |
| HTTP 301 or 302 loop | Missing cookies, an incorrect canonical URL, or a redirect policy mismatch | Print every redirect, inspect the final URL, and confirm whether the client should retain a session. |
| HTTP 401 or 403 | Authentication, permissions, access controls, or a request that lacks required context | Confirm authorized access, required session state, and documented request requirements. Do not try to bypass controls. |
| HTTP 404 | Changed URL pattern, deleted content, or an incorrect identifier | Check URL construction and classify missing pages separately from parser failures. |
| HTTP 429 | Rate or concurrency limit | Reduce concurrency, apply exponential backoff with a limit, honor retry guidance, and cache successful responses. |
| HTTP 500, 502, 503, or 504 | Origin or upstream service failure | Retry only transient failures with bounded backoff; preserve the failed response and avoid creating a request storm. |
| HTTP 200 but empty fields | Selector change, consent page, challenge page, or client-rendered content | Save the HTML, compare it with a known-good fixture, and identify the actual data source. |
| Playwright or Puppeteer timeout | Wrong wait condition, slow resource, failed browser launch, or a page that never reaches the chosen state | Capture a screenshot and URL, wait for a specific stable element or response, and inspect browser logs. |
| JSON decode error | HTML was returned instead of JSON, malformed data, or an unexpected empty body | Check status and content type, print the first response characters, and parse only after validating the payload. |
Cadence and checkpoints
Use a layered maintenance schedule rather than waiting for a complete failure.
Every run
- Apply a timeout and bounded retries only for appropriate transient failures.
- Record status, latency, response size, and extraction counts.
- Fail loudly when required fields disappear or the result count falls below a sensible threshold.
- Keep requests within the rate and concurrency limits you have established for the target.
Weekly or after a notable incident
- Review failed URLs, empty responses, retry totals, proxy errors, and duplicate records.
- Compare a fresh HTML or JSON sample with the stored fixture.
- Run parser tests against saved pages so layout changes can be isolated from network failures.
- Verify that scheduled jobs, environment variables, browser binaries, and credentials have not changed.
Monthly or quarterly
- Review selectors, pagination rules, URL patterns, schemas, and required fields.
- Test representative pages across content types, regions, languages, and logged-in states where your project legitimately supports them.
- Audit proxy pools and remove unreliable endpoints rather than compensating with aggressive retries.
- Update dependencies deliberately and run the full test suite before deploying browser or parser changes.
For a broader operational checklist covering rate limits, retries, proxy rotation, and monitoring, see the web scraping reliability checklist. Keep a dated change log so a regression can be connected to a deployment, dependency update, or target-site change.
How to interpret changes
Look for patterns across metrics instead of reacting to a single failed request. A rise in latency followed by timeouts suggests a transport or capacity issue. Stable latency with a sudden drop in extracted fields points more strongly to a layout, selector, rendering, or access-page change. A fall in response size combined with a new title or body marker often indicates that the scraper received a login, consent, or challenge page rather than the intended document.
Compare failures by URL, status code, proxy, geographic route, browser mode, and time of day. If only one proxy fails, replace or quarantine that endpoint. If all proxies return the same changed markup, investigate the target response and your access assumptions instead of rotating faster. If Requests fails but a permitted browser workflow succeeds, determine whether the data is genuinely client-rendered or whether the browser is merely supplying cookies and headers that should be handled explicitly.
For parser changes, write a regression test around the business result, not just the selector. For example, assert that a product page produces a non-empty name, a valid currency value, and an availability state. A test suite using fixtures and mocks can protect against accidental breakage; see how to test web scrapers with Pytest.
When to revisit
Return to this playbook whenever a monitor reports an empty result, an unusual status-code distribution, a sustained latency increase, or a drop in required-field completeness. Revisit it after a target redesign, authentication change, browser upgrade, proxy change, dependency update, or migration between Requests, Scrapy, Playwright, and Puppeteer.
At the start of each monthly or quarterly review, choose a small set of representative URLs and run the same diagnostic checks. Save the date, response status, content type, record count, fixture comparison, and applied fix. This turns recurring debugging into a maintenance record instead of repeated guesswork.
- Reproduce the failure with one URL and one execution environment.
- Save a redacted response, screenshot, or network trace as evidence.
- Classify the failure as transport, access, rendering, extraction, or data quality.
- Apply the narrowest safe fix and add a regression check.
- Run a small canary batch before restoring normal volume.
- Document the date, cause, fix, and any follow-up review date.
For library-specific context, compare the trade-offs in the Python web scraping stack guide, and use the related Playwright dynamic-site guide when the diagnosis points to browser rendering. A scraper that is measured, tested, and reviewed on a schedule is easier to repair and less likely to fail silently.