Python Web Scraping Stack: Requests vs BeautifulSoup vs Scrapy vs Playwright

Overview

The four approaches fit different jobs:

• Requests: fetches HTTP responses with very little overhead. It is a strong starting point when the required data is present in the server-rendered HTML or an accessible endpoint.
• BeautifulSoup: parses HTML and XML into a convenient tree. It is usually paired with Requests rather than used as a complete downloader.
• Scrapy: provides a framework for larger crawlers, including request scheduling, item pipelines, concurrency controls, feed exports, and structured project conventions.
• Playwright: controls a real browser engine. It is useful when a page depends on JavaScript, browser events, interaction, or content that appears only after rendering.

A practical rule is to use the least complex tool that reliably produces the data. A Requests-and-BeautifulSoup scraper is often easier to run and debug than a browser-based scraper. Conversely, using a simple HTTP client for a genuinely interactive application can create brittle workarounds and more maintenance later.

The decision can be treated as a small estimation exercise. Record the page type, number of URLs, rendering requirements, extraction rules, acceptable delay, and expected change frequency. Then choose the smallest stack that satisfies those constraints with room for validation and retries.

For a broader implementation workflow, see the Python web scraping tutorial covering Requests, Beautiful Soup, pagination, and clean data.

How to estimate

Start by scoring the job against five questions. The result is not a benchmark; it is a planning tool that makes assumptions visible.

1. Is the data in the initial response?

Inspect the HTML returned by a normal HTTP request. If the target title, price, table, links, or product fields are already present, Requests plus BeautifulSoup may be sufficient. If the response contains only a shell and the values appear after JavaScript runs, investigate the underlying data request before moving directly to browser automation.

2. How many URLs must be processed?

A short list of pages can often be handled by a small script. A crawler that follows links, manages pagination, deduplicates URLs, retries failures, and exports structured records benefits from Scrapy’s project model. The threshold is not a fixed URL count: repeatability and workflow complexity matter as much as volume.

3. Does the workflow require a browser?

Choose Playwright when the task genuinely requires browser behaviour, such as clicking controls, waiting for a rendered component, selecting filters, handling a multi-step interaction, or reading content created by client-side code. Do not add a headless browser merely because the site looks dynamic in a normal browser; first check whether a stable, permitted data endpoint can be used.

4. How expensive is a failed record?

For exploratory work, a script that prints results may be enough. For price tracking, lead research, SEO data extraction, or scheduled reporting, missing or malformed records can be more costly than slower collection. Add time for retries, rate limiting, logging, schema checks, and sample-based validation.

5. How often will the scraper run?

A one-off extraction can tolerate more manual inspection. A scheduled scraper needs clear configuration, predictable output, failure alerts, and a recovery plan. The more often it runs, the more valuable a framework and explicit tests become.

A simple selection flowchart is:

1. If you need only a few HTTP responses and no parsing, start with Requests.
2. If you need fields from server-rendered HTML, use Requests plus BeautifulSoup.
3. If you need crawling, pipelines, pagination, retries, or many spider rules, evaluate Scrapy.
4. If the required result depends on browser execution or interaction, use Playwright, while keeping extraction and validation separate from browser control.

Inputs and assumptions

Before estimating effort, write down the inputs below. This prevents a fast prototype from being mistaken for a production-ready scraper.

Use three effort bands rather than false precision:

• Low complexity: one or a few page types, server-rendered HTML, stable selectors, and a small output file.
• Medium complexity: pagination, multiple templates, embedded JSON, retries, normalisation, and scheduled runs.
• High complexity: browser interactions, sessions, downloads, several workflows, frequent layout changes, or strict completeness requirements.

For each band, estimate four separate items: initial build time, per-run compute and network use, ongoing maintenance, and failure investigation. A browser scraper can have a larger per-run footprint than an HTTP scraper, while a Scrapy project may require more setup but reduce repeated engineering as the crawl grows.

Keep compliance and access assumptions explicit. Check the site’s published terms, access instructions, and applicable requirements before collecting data. Use conservative request rates, identify your own system in logs, avoid collecting unnecessary personal information, and do not treat technical accessibility as permission.

Reliability details such as backoff, proxy decisions, monitoring, and validation are covered in the web scraping reliability checklist.

Worked examples

Example 1: Extract article titles from server-rendered pages

Assume you have a known list of URLs and each response contains article headings in HTML. There is no interaction, and the output is a CSV file. Requests handles downloading and BeautifulSoup handles parsing:

from pathlib import Path
import csv
import requests
from bs4 import BeautifulSoup

URL = "https://example.com/articles"
response = requests.get(URL, timeout=20)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
rows = []
for heading in soup.select("article h2"):
    title = heading.get_text(" ", strip=True)
    link = heading.find_parent("article").find("a", href=True)
    rows.append({
        "title": title,
        "url": link["href"] if link else ""
    })

with Path("articles.csv").open("w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["title", "url"])
    writer.writeheader()
    writer.writerows(rows)

This is a low-complexity design if the selector is stable. Add a session, explicit timeouts, retries, URL normalisation, and checks for an unexpectedly empty result before scheduling it.

Example 2: Crawl paginated catalogue pages

Suppose a catalogue has many pages, a next-page link, and a consistent product card. Requests and BeautifulSoup can still work, but Scrapy becomes attractive when you need a queue, deduplication, controlled concurrency, item pipelines, and repeatable exports. A small spider might look like this:

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/catalogue"]

    def parse(self, response):
        for card in response.css("article.product"):
            yield {
                "name": card.css("h2::text").get(default="").strip(),
                "price": card.css(".price::text").get(default="").strip(),
                "url": response.urljoin(card.css("a::attr(href)").get())
            }

        next_url = response.css("a.next::attr(href)").get()
        if next_url:
            yield response.follow(next_url, callback=self.parse)

The important upgrade is not the selector syntax. It is the separation of crawling, extraction, and output, which makes later changes easier to test.

Example 3: Read a JavaScript-rendered result

Assume a search form must be submitted and the result cards appear only after the page runs JavaScript. Playwright is a better fit than trying to reconstruct every browser event:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/search", wait_until="domcontentloaded")
    page.fill("input[name=q]", "python")
    page.click("button[type=submit]")
    page.wait_for_selector("article.result")

    results = []
    for card in page.locator("article.result").all():
        results.append({
            "title": card.locator("h2").inner_text(),
            "url": card.locator("a").get_attribute("href")
        })

    browser.close()

Estimate this as higher complexity because browser startup, waits, session state, and rendering failures need attention. For a detailed browser workflow, see how to scrape dynamic websites with Playwright.

Migration paths

A sensible progression is:

1. Prototype extraction with Requests and BeautifulSoup.
2. Add sessions, timeouts, retries, validation, and structured logging.
3. Move repeated crawl logic into Scrapy when URL discovery and scheduling become central.
4. Introduce Playwright only for the page types that require browser execution, rather than converting every request to a browser.

This hybrid approach keeps simple pages inexpensive while reserving browser automation for the workflows that need it.

When to recalculate

Revisit the estimate whenever one of the main inputs changes. A redesign can turn a selector adjustment into a parser rewrite. A new pagination path can increase the number of requests. A move from weekly to hourly collection can make monitoring and rate limiting more important than initial code size. A new interaction requirement can justify Playwright even if the original Requests scraper remains healthy.

Recalculate after these events:

• The target adds client-side rendering, a login flow, filters, or infinite scrolling.
• The URL set expands beyond the original list or begins changing automatically.
• Run duration, error rate, empty outputs, or duplicate records trend upward.
• The output gains new fields, files, tables, or embedded JSON that require different validation.
• A scheduled job misses a delivery window or needs manual recovery.
• Access conditions, published instructions, or your permitted use changes.

Use a small review record: date, page type, URL count, selected tool, average run duration, failed requests, empty records, and maintenance notes. Compare the latest run with the original assumptions rather than relying on memory. If the scraper is becoming difficult to repair, inspect the page structure and add change detection before replacing the entire stack; the guide on detecting website structure changes provides a useful next step.

The practical decision is straightforward: begin with Requests and BeautifulSoup when the HTML already contains the data, choose Scrapy when crawling and repeatability dominate, and use Playwright when browser execution is genuinely part of the workflow. Estimate the operational cost—not only the first working script—and review that estimate whenever the pages, schedule, or quality requirements change.