Extract JSON from Web Pages: JSON-LD, Embedded State, and API Responses

Overview

Web pages commonly expose structured data in several forms. A product page may include JSON-LD for search engines, an embedded state object for a JavaScript application, data attributes for individual components, or an API response used to render the page. These sources can contain useful SEO and growth data such as product names, prices, availability, article metadata, ratings, breadcrumbs, canonical URLs, and organisation details.

The first decision is whether to extract from the initial HTML or from the request that supplies the data. Static HTML parsing is usually simpler, faster, and easier to schedule. Direct API extraction can provide cleaner nested records when a page is populated dynamically. A browser automation tool such as Playwright or Puppeteer may be necessary when the relevant request only occurs after JavaScript runs, a user interaction, or a session is established.

Before collecting anything, check the website's terms, access controls, and applicable legal requirements. Limit request rates, identify the fields you need, and avoid collecting personal or sensitive information without a clear legitimate basis. For a broader comparison of extraction approaches, see the Python web scraping stack guide.

1. Extract JSON-LD from HTML

JSON-LD is commonly placed inside a <script type="application/ld+json"> element. It may describe a single object, an array, or a graph containing several related objects. The following Python example collects every JSON-LD block and keeps malformed blocks available for diagnostics rather than silently discarding them:

import json
import requests
from bs4 import BeautifulSoup

url = "https://example.com/page"
response = requests.get(url, timeout=20)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
records = []
errors = []

for index, node in enumerate(soup.select('script[type="application/ld+json"]')):
    raw = node.string or node.get_text()
    try:
        records.append(json.loads(raw))
    except json.JSONDecodeError as exc:
        errors.append({"index": index, "error": str(exc), "raw": raw[:500]})

print(records)
print(errors)

Do not assume that the first block is the only useful one. A page can contain separate objects for a product, breadcrumb list, organisation, and review. If the value is a graph, inspect the @graph array and select objects by their @type or another stable field.

2. Find embedded application state

JavaScript frameworks may place initial data in a script element with an identifier such as __NEXT_DATA__, or in a custom element created by the site. Names vary, so inspect the page source and search for likely markers such as props, initialState, dehydratedState, or __INITIAL_STATE__. Parse the complete object rather than using regular expressions to capture nested JSON. Regular expressions can help locate a marker, but they are brittle for balanced braces, escaped characters, and nested strings.

3. Read JSON from attributes

Some pages store small records in attributes such as data-product, data-config, or data-state. Decode the HTML attribute first, then parse its value as JSON:

import json
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
items = []

for element in soup.select("[data-product]"):
    raw = element.get("data-product")
    try:
        items.append(json.loads(raw))
    except (TypeError, json.JSONDecodeError):
        continue

Attribute data is often partial. Treat it as a source for identifiers or display values, then compare it with the page or API response before assuming it represents the complete record.

4. Capture API responses for dynamic pages

When the HTML contains only a shell, inspect the browser's network activity. In Playwright, response listeners can collect JSON returned by requests made after page load:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    api_records = []

    def handle_response(response):
        content_type = response.headers.get("content-type", "")
        if "application/json" in content_type:
            try:
                api_records.append({
                    "url": response.url,
                    "data": response.json()
                })
            except Exception:
                pass

    page.on("response", handle_response)
    page.goto("https://example.com/catalog", wait_until="networkidle")
    browser.close()

print(api_records)

In production, narrow the capture conditions by URL pattern, request method, or response shape. Saving every JSON response can create noisy, expensive datasets and may capture data that is unrelated to the extraction task. The article on Playwright waiting, pagination, and network data covers this workflow in more detail.

Maintenance cycle

JSON extraction should be treated as a small data pipeline, not a one-off parsing script. Start by defining a schema for the fields your SEO, content, or growth workflow actually uses. For example, a product record might require name, url, sku, price, currency, and availability. Store the source URL, retrieval timestamp, extraction method, and parser version alongside each record.

A practical maintenance cycle has four stages:

1. Inspect: save representative HTML and API responses from several page types, including pages with missing fields.
2. Parse: convert source data into a stable internal schema rather than passing site-specific structures through the rest of your system.
3. Validate: check required fields, data types, URL formats, currency values, and expected object counts.
4. Review: compare recent output with a small set of known examples and investigate unusual changes before publishing or exporting data.

Keep fixtures for tests. A saved JSON-LD block and a saved API response make it possible to test parser changes without repeatedly requesting a live website. Add a checksum or source snapshot identifier where appropriate so you can identify when the input changed. For operational controls such as retries, rate limiting, and monitoring, use the web scraping reliability checklist.

Schedule routine reviews according to how often the source changes and how costly an incorrect result would be. A monthly review may suit a stable editorial dataset; a price or availability feed may need automated checks on every run and a more formal review at a shorter interval.

Signals that require updates

Several signals indicate that an extractor needs investigation or a planned update:

• The number of JSON-LD blocks falls to zero or changes sharply.
• A required field becomes consistently empty, null, or a different data type.
• Validation finds a new structure, such as an object replacing an array.
• API response URLs, query parameters, pagination tokens, or response envelopes change.
• The same entity appears multiple times because HTML, JSON-LD, and API sources are being combined without a deduplication rule.
• Values no longer match visible page content on a sample of URLs.
• Response content is an access or error page instead of the expected HTML or JSON.
• Schema vocabulary or application state markers change during a site redesign.

Use thresholds rather than waiting for a complete failure. For example, alert when the proportion of records missing a required identifier exceeds an agreed limit, or when the median number of extracted items differs materially from recent runs. Keep alerts specific enough to guide diagnosis: “JSON-LD missing on product pages” is more useful than “scraper failed.”

Common issues

Malformed JSON

Trailing commas, HTML entities, unexpected byte-order marks, and JavaScript expressions can make a script look like JSON without being valid JSON. Record the original fragment, normalise only safe encoding issues, and avoid executing arbitrary script content as a repair strategy. If the source is JavaScript rather than JSON, identify whether an official endpoint or a safer browser-level observation is available.

Nested objects and schema variation

Use defensive access for optional fields and validate types explicitly. A product's price may be a number in one response and a nested object in another. Normalise it into your own representation while retaining the raw value for troubleshooting. JSON Schema or typed models can make these contracts visible and testable.

Duplicate records

Deduplicate using a stable key such as a canonical URL, product identifier, or a carefully constructed combination of fields. Do not use the array position as an identity. When multiple sources describe the same entity, define precedence—for example, use an API identifier when available, then fall back to a canonical URL.

HTML versus direct API extraction

Prefer HTML when the required fields are already present and the page is stable. Prefer a documented or clearly observable data response when it provides the complete record more directly. A browser is useful when JavaScript execution is essential, but it adds startup time, resource use, and more moving parts. The simplest source that produces accurate, validated data is usually the best maintenance choice.

Encoding and localisation

Prices, dates, decimal separators, and availability labels may vary by locale. Store raw source values and normalised values separately where interpretation matters. Confirm the page language, currency, and timezone before comparing records across URLs.

When to revisit

Revisit this extraction guide and your implementation on a scheduled review cycle, after a site redesign, or whenever search and growth requirements change. A new SEO reporting requirement may call for fields that were previously ignored, while a change from static pages to client-rendered pages may make HTML parsing incomplete. Likewise, an API response may become more useful than embedded JSON-LD as your dataset grows.

At each review, choose a small sample of representative URLs and verify four things: the source still exists, the parser still finds it, the output matches the intended schema, and duplicate handling still produces one record per entity. Review error logs and recent field-completeness metrics rather than testing only a successful page.

For a practical next step, create a fixture-based test for one JSON-LD page, one embedded-state page, and one dynamic API response. Add validation for required fields, log the source and parser version, and schedule a lightweight health check. If the source is a dynamic site, compare this workflow with the reliable Playwright browser automation workflow. If your extractor feeds SEO analysis, the Python SERP scraper guide provides a related example of turning collected web data into a repeatable reporting process.