Recurring scrapers do not always need to download a complete response. By storing a page’s ETag or Last-Modified value and sending it back in a conditional request, you can ask the server whether the resource changed first. This guide explains how to inspect cache headers, implement conditional GET requests in Python, handle 304 responses, set sensible expiry rules, and measure whether the approach is improving bandwidth without making your data stale.
Overview
HTTP caching for web scraping is most useful when a job revisits the same URLs on a schedule and many of those pages change infrequently. Instead of downloading the full response every time, the scraper records a validator from the previous response. On the next run, it includes that validator in a request:
If-None-Matchcarries a previously receivedETag.If-Modified-Sincecarries a previously receivedLast-Modifieddate.
If the server confirms that the representation has not changed, it can return 304 Not Modified without a response body. Your scraper then reuses the stored content and avoids downloading the page again. If the resource has changed, the server normally returns 200 OK with the new body and new metadata.
Conditional requests are not a substitute for respectful crawling. Check the target’s terms, access instructions, and applicable requirements before scraping, and continue to use a clear user agent, appropriate timeouts, concurrency limits, and rate limiting. Caching reduces repeated transfer; it does not grant permission to access a resource.
Cache headers and status codes at a glance
| Item | What it tells you | How a scraper can use it |
|---|---|---|
ETag | An identifier for a particular representation of the response. | Store it and send it as If-None-Match on a later request. |
Last-Modified | A server-provided modification date. | Store it and send it as If-Modified-Since. |
Cache-Control | Cache-related directives such as max-age or no-cache. | Use it as an input to freshness rules, not as proof that a page may be scraped. |
Expires | An older-style expiry date that may indicate when a response becomes stale. | Use cautiously, especially when it conflicts with other metadata. |
304 Not Modified | The selected representation has not changed since the validator was supplied. | Reuse the stored body and update access or validation timestamps. |
200 OK | The server returned a response body. | Validate and store the new body and its current validators. |
412 Precondition Failed | A supplied precondition was not met. | Log the event, discard or reassess the validator, and retry without it only when appropriate. |
Checklist by scenario
For a new crawl target
- Confirm that the URL is stable enough to revisit and that the intended collection is permitted.
- Make an initial request with a descriptive user agent, a timeout, and redirect handling suited to the target.
- Record the final URL, status code, response headers, retrieval time, content hash, and body or parsed result.
- Inspect whether the response includes
ETag,Last-Modified,Cache-Control, orExpires. - Choose a refresh interval based on how quickly the source is expected to change, rather than requesting as often as possible.
For a target with an ETag
Prefer the ETag as the primary validator because it identifies a representation directly and can distinguish changes that do not map neatly to a timestamp. Preserve the value exactly, including weak prefixes such as W/. Do not parse or normalise it casually. On the next request, send:
If-None-Match: "stored-etag-value"
A successful 304 means the stored body can be reused. A 200 means the body should be read, checked, and stored along with the new ETag.
For a target with Last-Modified only
Store the complete header value and send it unchanged as If-Modified-Since. Dates have limited precision, so a resource can change more than once within the same timestamp resolution. Treat this validator as a useful optimisation, not an absolute guarantee of content identity. If the page is important or changes frequently, compare a content hash after a normal response and schedule occasional unconditional checks.
For a target with neither validator
Use a local freshness window. For example, your scheduler might avoid requesting a URL again until its configured interval has elapsed. When it is due, make a normal request and store a hash of the relevant content. This does not produce a server-side 304, but it can prevent duplicate work within your own pipeline. The interval should reflect the data’s value, expected change rate, and the load your crawler creates.
Reusable Requests implementation
The following template stores response metadata in a simple record. In production, replace the in-memory dictionary with a durable database or key-value store, and store the body in a controlled cache or object store.
from datetime import datetime, timezone
import hashlib
import requests
cache = {}
session = requests.Session()
session.headers.update({
"User-Agent": "ExampleResearchBot/1.0 (+https://example.org/bot-info)"
})
def fetch_conditionally(url, max_age_seconds=3600):
now = datetime.now(timezone.utc)
record = cache.get(url)
if record:
age = (now - record["checked_at"]).total_seconds()
if age < max_age_seconds:
return {"status": "fresh-local-cache", "body": record["body"]}
headers = {}
if record and record.get("etag"):
headers["If-None-Match"] = record["etag"]
elif record and record.get("last_modified"):
headers["If-Modified-Since"] = record["last_modified"]
response = session.get(url, headers=headers, timeout=20)
if response.status_code == 304 and record:
record["checked_at"] = now
return {"status": "not-modified", "body": record["body"]}
response.raise_for_status()
body = response.content
cache[url] = {
"body": body,
"etag": response.headers.get("ETag"),
"last_modified": response.headers.get("Last-Modified"),
"checked_at": now,
"sha256": hashlib.sha256(body).hexdigest(),
}
return {"status": "downloaded", "body": body}
In a shared crawler, make cache writes atomic and key records by the canonical URL plus any request parameters that affect the representation. If cookies, authentication, language headers, or query parameters change the response, they must be part of the cache design. Never reuse private or user-specific content across users or accounts.
What to double-check
- Representation: Confirm that the validator relates to the exact variant you store. A response may vary by
Accept-Encoding,Accept-Language, cookies, or other request headers. - Freshness: A
304confirms the server’s conditional response, but it does not guarantee that your parser’s previous output was correct. Keep the original body or enough metadata to reproduce validation. - Compression: Hash consistently. Hashing compressed bytes and hashing decompressed content answer different questions.
- Redirects: Record the final URL and validators from the response that contains the actual representation. A redirect target can change independently.
- Failures: Do not replace a good cached body with an error page, challenge page, truncated download, or unexpected content type.
- Metrics: Track total requests, conditional requests,
304responses, full downloads, bytes received, timeouts, HTTP errors, parse failures, and stale-cache uses.
A practical success report should compare bandwidth and freshness, not just the number of requests. Useful measures include the percentage of validations ending in 304, bytes saved against a baseline, median age of the data when consumed, and the rate of unexpected content changes. A high 304 rate is not automatically good if the chosen refresh interval makes the data too old for its purpose.
Common mistakes
- Assuming every server supports validators: Some responses omit them, change them unpredictably, or ignore conditional headers. Fall back to a normal request and local freshness rules.
- Treating a missing body on 304 as an error: A 304 is expected to have no page body. Reuse the previously stored representation.
- Sending both validators without a plan: Servers can evaluate multiple conditions according to HTTP rules. Use ETag first where available, and document your fallback behaviour.
- Trusting timestamps blindly: Clock differences, coarse timestamp precision, and application-generated dates can make Last-Modified less reliable than expected.
- Discarding cache metadata after parsing: Store validators separately from extracted fields so a parser change does not force unnecessary downloads.
- Ignoring content changes hidden behind the same URL: Personalisation, rotating inventory, and query-driven responses may require shorter intervals or a different collection strategy.
When to revisit
Review this setup before seasonal planning cycles, when a source changes its site or API, or whenever a recurring job’s bandwidth and failure rates move unexpectedly. Recheck the target’s response headers, redirect behaviour, content type, and typical change frequency. A six-month editorial or engineering review is a useful baseline, but high-value or fast-changing targets deserve more frequent checks.
Use this action list when adding a new crawl target or changing an existing workflow:
- Run several ordinary requests and record the validators and cache directives returned.
- Run conditional requests and verify that a genuine
304reuses the correct stored body. - Force or observe a known content change and confirm that the scraper accepts the new
200response. - Test timeouts, redirects, malformed responses, and server errors without overwriting valid cached data.
- Compare bytes transferred, freshness, validation success, and parse quality with the previous implementation.
- Document the chosen local expiry, validator priority, fallback path, and alert thresholds for the next maintainer.
With those checks in place, conditional GET becomes a controlled reliability feature rather than an assumption about how every website behaves. It can reduce repeated downloads while keeping freshness, failure handling, and operational limits visible.