Is Web Scraping Legal in the UK? A Practical Compliance Checklist for Developers
Web scraping lawUK developersPrivacyCopyrightDatabase rightsCompliance

Is Web Scraping Legal in the UK? A Practical Compliance Checklist for Developers

WWebScraper.uk Editorial Team
2026-11-18
8 min read

A practical UK web scraping compliance checklist covering terms, robots.txt, copyright, privacy, access controls, rate limits, retention, and review triggers.

Is web scraping legal in the UK? There is no single yes-or-no answer. Legality depends on what you collect, how you access it, why you need it, and what you do with the results. This practical checklist helps developers assess a scraping project before it runs, document the decision, and revisit it when a website, use case, or data field changes. It is general information, not legal advice; higher-risk projects should be reviewed by a qualified UK lawyer.

Overview

Scraping a publicly visible page is not automatically authorised, and a website’s technical accessibility is not the same as permission. A responsible review considers several overlapping areas: contractual terms, copyright, database rights, privacy and data protection, computer misuse boundaries, authentication controls, and the operational impact of your requests.

Start by defining the project in plain language. “Collect product prices from public pages once per day for internal comparison” presents a different risk profile from “copy a competitor’s entire catalogue, including customer reviews and usernames, and republish it.” The purpose, fields, scale, and downstream use all matter.

For each target, keep an evidence record containing the URL, date reviewed, access method, relevant terms, robots.txt contents, collection fields, request schedule, retention period, and the person responsible for approval. This turns a one-off judgement into a repeatable compliance workflow and makes later changes easier to identify.

It is also useful to separate four questions:

  • Can the system technically access the page? This is an engineering question.
  • Does the website permit the activity? Review terms, notices, account rules, and explicit restrictions.
  • Are you allowed to copy or reuse the material? Consider copyright, database rights, licences, and substantial extraction.
  • Can you lawfully process the resulting data? This is especially important when personal data is present.

Do not treat a successful HTTP response, a missing block page, or an absent robots.txt file as proof that every proposed use is acceptable.

What to track

1. Terms and permission

Read the website’s terms of service, acceptable-use rules, API documentation, privacy notice, and any rules attached to an account. Look for restrictions on automated access, copying, commercial use, resale, bulk downloads, or creating derivative datasets. Record the exact page and date checked because online terms can change.

If a service offers an official API, feed, export, or licensing route, assess that option before building a scraper. An approved access method may provide clearer limits, more stable fields, and more defensible permission than automated page collection.

2. robots.txt and visible signals

Inspect /robots.txt for the relevant user-agent and paths. Robots.txt is an important signal of a publisher’s crawling preferences, but it is not a complete permission document and does not resolve copyright, privacy, or contractual questions. Respect disallowed paths in your crawler design, and document how you interpreted the file.

Also note login walls, “no automated access” notices, CAPTCHA challenges, technical access controls, and headers or documentation that define permitted use. Do not attempt to bypass authentication, paywalls, CAPTCHAs, IP blocks, or other controls simply because a technical workaround exists.

Separate facts from expressive material. A price, date, or product identifier may raise different questions from an editorial description, photograph, review, software code, or carefully structured collection. Copying a small selection for analysis is not the same as reproducing a substantial part of a database or publishing a competing copy.

Check the page’s licence, attribution requirements, syndication terms, and any database-specific notice. If your project stores raw HTML, images, long text passages, or a large structured collection, obtain a specific legal assessment rather than assuming that public visibility makes reuse free of restrictions.

4. Personal data and privacy

Identify whether the output contains information relating to identifiable people. Names, usernames, email addresses, profile URLs, photographs, location details, reviews, and combinations of seemingly harmless fields may be personal data. Public availability does not remove the need to assess a lawful basis, transparency, purpose limitation, minimisation, security, retention, and individuals’ rights under the applicable UK data protection framework.

Ask whether every field is necessary. A lead-generation project, for example, should not collect personal contact details merely because a page exposes them. Avoid sensitive or high-risk categories unless there is a clearly documented justification and specialist advice. Plan deletion, correction, opt-out, and access-request handling before collection begins.

5. Requests, rate limits, and operational impact

Set a conservative request rate, identify your crawler where appropriate, use caching, honour server errors, and stop when the site shows signs of strain. Do not create unnecessary parallel traffic or use proxy rotation to evade a stated limit. A technically reliable scraper can still be an unreasonable burden on the target.

For implementation guidance, pair this compliance review with practical engineering resources such as the web scraper troubleshooting playbook and the guide to validating scraped data. Reliability controls should support responsible access, not defeat restrictions.

6. Retention, security, and onward use

Define how long raw responses, parsed records, logs, and backups will be kept. Restrict access to collected data, encrypt sensitive stores where appropriate, and document who receives exports. If the dataset will be sold, published, used for profiling, or combined with another source, assess that downstream use separately. A low-risk collection can become higher risk after enrichment or publication.

Cadence and checkpoints

Use a checkpoint before launch, then review on a monthly or quarterly cadence according to the project’s scale and risk. A small internal collection of non-personal facts may need less frequent operational review than a daily crawl containing personal data or material used in a customer-facing product.

ScenarioInitial positionCheckpoint before proceeding
Public factual data, low volume, internal researchOften lower risk, but not automatically permittedCheck terms, robots.txt, reuse rights, fields, rate, and retention
Product prices and availabilityUsually requires careful copying and operational controlsConfirm permitted use, avoid unnecessary descriptions or images, record timestamps, and set request limits
Public profiles, reviews, or usernamesHigher privacy and reuse riskMinimise fields, document the lawful basis and notice approach, and obtain specialist advice where needed
Login-only, paywalled, or restricted contentHigh risk if access controls or contractual limits are involvedObtain explicit permission or use an authorised export or API
Republishing a substantial catalogue or article collectionHigher copyright and database-rights riskSeek a licence or legal review before collection and publication

At each checkpoint, compare the current project with its approved record. Confirm that the target domain, endpoints, fields, request volume, storage location, users, and purpose are unchanged. Test the crawler against a small sample before increasing coverage. A change from daily monitoring to continuous collection, or from internal analysis to public publication, should trigger a fresh review rather than waiting for the next scheduled date.

Keep a change log for terms, robots.txt, API rules, authentication requirements, privacy notices, and schema changes. Technical monitoring can help identify changes, but an alert is not a legal conclusion. Assign an owner who can pause collection while the change is assessed.

How to interpret changes

Not every change has the same significance. A renamed CSS class may be an engineering issue; a new restriction on automated access is a permission issue; a newly exposed email field is a privacy issue. Classify changes before deciding whether to modify code.

  • Green: the page layout changes but the approved fields, purpose, permissions, and traffic pattern remain the same. Update selectors, run data-quality checks, and retain the test evidence.
  • Amber: terms, robots.txt, rate guidance, fields, or data use have changed. Pause expansion, record the change, and ask the project owner or privacy lead to reassess.
  • Red: the project would require bypassing authentication or access controls, collecting sensitive personal data without a clear basis, ignoring an explicit prohibition, or reproducing substantial protected content. Stop and seek legal advice or a permitted alternative.

False positives and false negatives matter operationally. Missing records may indicate a selector break, a consent wall, a blocked request, or a deliberate site change. Compare status codes, response size, page titles, content hashes, and sample fields, but do not automatically respond to blocks with more aggressive automation. For SEO projects, related technical checks can be found in this metadata extraction guide; for field selection, see the CSS selector and XPath field-mapping guide.

Project documentation template

Save the following record with the code and review it at each checkpoint:

  • Purpose: what decision or activity will the data support?
  • Source: domain, URL patterns, country, and date first reviewed.
  • Fields: exact fields collected, with personal-data fields marked.
  • Access method: API, feed, ordinary HTTP requests, or browser automation, and why it is necessary.
  • Permission review: terms, robots.txt, licence, notices, authentication boundary, and reviewer.
  • Controls: request rate, concurrency, caching, error handling, and stop conditions.
  • Retention: raw-data period, parsed-data period, backup period, and deletion method.
  • Sharing: internal users, vendors, publication plans, and access controls.
  • Opt-out handling: contact address, suppression list, deletion workflow, and response owner.
  • Review date: next scheduled check and events that require an immediate reassessment.

When to revisit

Revisit this checklist before every new target website, geography, business purpose, or collection of fields. Repeat the review after changes to terms of service, robots.txt, privacy notices, APIs, authentication, rate limits, or the site’s technical behaviour. Also review it when a dataset moves from internal research into publication, profiling, lead generation, product development, or sale.

As a practical minimum, schedule a monthly or quarterly checkpoint and record the result even when nothing has changed. The record should say what was checked, when it was checked, who approved continued collection, and what evidence was retained. If the answers are unclear, reduce the scope, use an official access route, request permission, or pause the scraper while obtaining professional advice.

Before your next run, complete the documentation template, test a small sample, confirm the request limits, verify that unnecessary personal fields are excluded, and set a clear stop condition. That workflow will not eliminate legal uncertainty, but it makes your scraping infrastructure more controlled, reviewable, and easier to change responsibly.

Related Topics

#Web scraping law#UK developers#Privacy#Copyright#Database rights#Compliance
W

WebScraper.uk Editorial Team

Technical SEO and Automation Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.