CSS Selectors and XPath for Web Scraping: A Practical Field-Mapping Cookbook
CSS selectorsXPathBeautiful SoupScrapyPlaywrightHTML parsing

CSS Selectors and XPath for Web Scraping: A Practical Field-Mapping Cookbook

CCode Scrape Hub Editorial Team
2026-11-14
6 min read

A practical cookbook for choosing, testing, and maintaining CSS selectors and XPath mappings with Beautiful Soup, lxml, Scrapy, and Playwright.

Reliable selectors are the foundation of a maintainable web scraper. This practical cookbook shows how to choose CSS selectors and XPath expressions for common fields, apply them with Beautiful Soup, lxml, Scrapy, and Playwright, and test mappings against representative HTML before a site change turns into missing or duplicated data.

Overview

When you scrape website data, the difficult part is rarely writing one selector. The harder task is choosing a selector that still identifies the intended field when a page contains repeated cards, optional elements, nested markup, tracking links, or several similar headings. A selector should describe the relationship between an element and its meaning, not merely copy a long path from a browser inspector.

CSS selectors are often a good first choice because they are compact and widely supported. XPath is useful when you need relationships such as “the link inside the heading,” “the row whose first cell contains this label,” or “the image that has an ancestor with a product class.” Neither approach is automatically more reliable. Reliability comes from selecting stable attributes, limiting the search scope, handling missing values deliberately, and checking the result against realistic fixtures.

Before extracting any field, inspect the HTML delivered to your parser. A browser may show content created by JavaScript that is absent from the initial response. In that situation, requests and Beautiful Soup will not see the rendered nodes; you may need an API response, embedded JSON, or a browser automation tool such as Playwright. For JSON-LD and embedded state, see our guide to extracting JSON from web pages.

Template structure

Start every field mapping with the same worksheet. It makes assumptions visible and gives you a clear test plan when the publishing template changes.

FieldPreferred selectorFallbackValue ruleValidation
Titleh1.article-titlemeta[property="og:title"]Trim whitespaceExactly one non-empty value
Canonical URLlink[rel="canonical"]Page URLRead hrefAbsolute URL or explicit null
Card linkarticle.card a.card-linkFirst suitable link in cardResolve relative URLOne URL per card
Imageimg.product-imageimg[data-src]Use src or lazy-load attributeDo not return empty strings
Price.price .amountStructured dataKeep raw text before parsingRecord currency separately

For each row, record the sample HTML, the expected number of matches, whether the field is optional, and the normalization rule. Keep the raw value until validation is complete. For example, store £1,299.00 before converting it to a numeric value. This makes it easier to diagnose a selector problem separately from a parsing problem.

Common selector patterns

  • Class or ID: .article-title, #main-content. Use classes that describe a component or purpose rather than classes that look generated.
  • Attribute: a[href], img[data-src], [data-testid="product-card"]. Attribute selectors are useful when semantic hooks are available.
  • Descendant: article.card h2 a. This finds a link in a heading inside each card.
  • Direct child: nav > ul > li. Use this when an intermediate wrapper would indicate a different relationship.
  • XPath text relationship: //th[normalize-space()="Price"]/following-sibling::td[1]. This is useful for labelled data.
  • XPath ancestor relationship: //h2[normalize-space()="Example"]/ancestor::article[1]. This scopes extraction to the relevant repeated block.

How to customize

Choose a selector in three stages. First, identify the smallest stable container for the record, such as article.product-card. Second, search inside that container for each field. Third, define what should happen when the field is absent. Scoping selectors prevents a title in one card from being paired with a price in another.

Equivalent extraction examples

The following fixture represents a repeated product card:

<article class="product-card" data-testid="product-card">
  <h2><a class="product-link" href="/products/alpha">Alpha</a></h2>
  <img class="product-image" src="/images/alpha.jpg" alt="Alpha product">
  <span class="price" data-currency="GBP">£19.00</span>
</article>

With Beautiful Soup, select the card first and then query within it:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
records = []
for card in soup.select("article.product-card"):
    link = card.select_one("a.product-link")
    image = card.select_one("img.product-image")
    price = card.select_one(".price")
    records.append({
        "title": link.get_text(" ", strip=True) if link else None,
        "url": link.get("href") if link else None,
        "image": image.get("src") if image else None,
        "price_raw": price.get_text(" ", strip=True) if price else None,
        "currency": price.get("data-currency") if price else None,
    })

lxml supports CSS selectors through its CSS selector interface and provides XPath directly. An XPath equivalent for the same fields is:

cards = tree.xpath('//article[contains(concat(" ", normalize-space(@class), " "), " product-card ")]')
for card in cards:
    title = card.xpath('string(.//a[contains(@class, "product-link")])').strip()
    url = card.xpath('string(.//a[contains(@class, "product-link")]/@href)')
    image = card.xpath('string(.//img[contains(@class, "product-image")]/@src)')
    price = card.xpath('string(.//*[contains(concat(" ", normalize-space(@class), " "), " price ")])').strip()

In Scrapy, a Selector has both css and xpath methods. The same mapping can be written as:

for card in response.css("article.product-card"):
    yield {
        "title": card.css("a.product-link::text").get(default="").strip() or None,
        "url": card.css("a.product-link::attr(href)").get(),
        "image": card.css("img.product-image::attr(src)").get(),
        "price_raw": card.css(".price::text").get(default="").strip() or None,
        "currency": card.css(".price::attr(data-currency)").get(),
    }

Playwright uses locators that can target CSS or XPath. Prefer a locator scoped to the card rather than querying the whole page for every field:

cards = page.locator("article.product-card")
for i in range(await cards.count()):
    card = cards.nth(i)
    title = await card.locator("a.product-link").inner_text()
    url = await card.locator("a.product-link").get_attribute("href")
    image = await card.locator("img.product-image").get_attribute("src")
    price = await card.locator(".price").inner_text()

For optional fields in Playwright, check whether a locator exists before reading it. In Beautiful Soup, lxml, and Scrapy, use explicit null values rather than allowing a missing node to raise an exception. A missing image and a selector that matched the wrong image are different data-quality events and should be distinguishable in logs.

CSS or XPath decision table

NeedRecommended starting pointReason
Stable class, ID, or data attributeCSSShort and easy to review
Repeated recordsCSS container plus scoped CSSReduces field misalignment
Match a visible labelXPathCan express text and sibling relationships
Find an ancestor or preceding elementXPathSupports richer document relationships
Browser interactionPlaywright locatorCombines selection with waiting and actions

Examples

For a page title, try a semantic heading such as h1 when the page contract guarantees one. For a card title, use article.card h2 a and read both text and href. For metadata, target the attribute directly: meta[name="description"]::attr(content) in Scrapy or soup.select_one('meta[name="description"]')['content'] in Beautiful Soup after checking that the node exists.

Images and lazy-loaded values

Images may use src, srcset, or a data attribute until they are loaded. Map the source attributes intentionally instead of assuming src is always populated. A practical fallback is to inspect src first, then data-src, and record which attribute supplied the result. Do not treat a CSS background image as an ordinary image attribute; it may require parsing a style declaration or using a page-level data source.

Tables and labelled fields

For a table with stable headers, select rows and map each cell by position only when colspan and rowspan are controlled. Where a label identifies the value, XPath can be clearer: //tr[th[normalize-space()="Availability"]]/td[1]. For complex tables, preserve the original row and cell structure before normalizing it. Our guide to scraping HTML tables with Python covers spans and multi-page table workflows.

Testing against representative fixtures

Save small HTML fixtures for a normal page, a missing optional field, multiple repeated cards, and a known markup variation. Test expected counts as well as values: a title selector returning zero or five results where one is expected should fail loudly. Use regression tests with mocked responses before deploying a change; the web scraper testing guide provides a suitable structure.

When to update

Revisit a selector mapping when a scraper returns empty fields, duplicate records, shifted values, or a sudden change in field counts. Also review it after a site redesign, a framework migration, a change to lazy loading, or an update to your parser or browser automation library. A selector that still returns data can be more dangerous than one that fails if it silently captures a navigation label, promotional price, or unrelated image.

Make the review practical. Save a fresh representative response, compare it with the fixture, inspect the first incorrect match in browser developer tools, and update the field worksheet before changing production code. Test both the primary selector and its fallback, then add a regression case for the markup pattern that caused the failure. Finally, validate duplicates, missing values, data types, and unexpected schema changes using the checks in our scraped-data quality guide.

Keep selectors short, scoped, and tied to documented HTML relationships. When a mapping becomes difficult to express, look for structured data or an appropriate API response instead of adding more brittle path segments. This field-mapping cookbook is most useful as a working document: refresh it whenever the underlying page template or extraction workflow changes.

Related Topics

#CSS selectors#XPath#Beautiful Soup#Scrapy#Playwright#HTML parsing
C

Code Scrape Hub Editorial Team

Technical Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.