Choosing between an API, an RSS or XML feed, embedded JSON, and HTML scraping affects reliability, cost, maintenance, and permission boundaries. This guide gives you a repeatable way to assess each source, estimate the real workload, plan pagination and authentication, and document a fallback before you build a data pipeline.
Overview
The best data source is not always the one that is easiest to request once. A useful choice must also provide the fields you need, remain available at the required frequency, fit your budget, and be appropriate for your intended use.
Start with this general preference order:
- Official API: usually the clearest option when it provides the required fields, access rights, and acceptable limits.
- Official RSS or XML feed: a practical lightweight source for published items, updates, or catalogue records with modest field requirements.
- Embedded JSON: useful when a page includes structured state for its own front end, but it still requires careful review of access conditions and stability.
- HTML scraping: a fallback when no suitable structured source exists or when the visible page contains information unavailable elsewhere.
This is a decision framework rather than a rule. An API may have incomplete coverage, expensive access, or a quota that does not suit a large historical export. Conversely, scraping a page may appear simple but become fragile after a redesign. Compare the whole workflow, not just the first successful request.
| Source | Typical strengths | Common constraints | Best first checks |
|---|---|---|---|
| API | Structured responses, documented fields, explicit pagination | Authentication, quotas, version changes, possible cost | Coverage, terms, limits, pagination, retention |
| RSS or XML | Simple polling, stable publication-focused records | Limited fields, incomplete history, update timing | Item fields, timestamps, archive depth, feed frequency |
| Embedded JSON | Often contains page data without parsing presentation markup | Undocumented structure, session requirements, front-end changes | Location, schema consistency, permissions, request volume |
| HTML | Visible content and broad coverage when other sources are absent | Selectors, rendering, layout changes, more validation work | Robots and terms, page variations, rate limits, field quality |
For a wider extraction project, pair this decision with a field-mapping plan. The CSS selectors and XPath field-mapping cookbook is useful when HTML is the chosen source.
How to estimate
Estimate a source using five scores: coverage, access, operational effort, risk, and cost. A simple worksheet is often more useful than a complicated numerical model because it makes assumptions visible.
1. Measure coverage
List every required field and mark whether the source provides it directly, provides an equivalent, or cannot provide it. Calculate:
Coverage ratio = usable required fields ÷ total required fields
Do this separately for current records and historical records. An endpoint may cover today’s items while offering no practical way to retrieve older data. Also record whether values are authoritative, derived, truncated, localised, or subject to delays.
2. Estimate request volume
For a recurring job, use:
Requests per run = discovery requests + page requests + detail requests + retries
Then estimate the monthly total:
Monthly requests = requests per run × runs per month
For pagination, divide the expected record count by the page size and round up. Add detail requests only if the list response does not contain the fields you need. Keep a separate retry allowance rather than hiding retries inside the page count.
3. Estimate engineering and maintenance effort
Record the time needed for discovery, authentication, pagination, parsing, validation, storage, monitoring, and recovery. HTML scraping usually needs more selector and layout testing; an API may need more work around tokens, versioning, quota handling, or nested JSON. A browser automation workflow can add browser runtime, rendering, and session-management concerns.
4. Estimate operational cost
Use a transparent model:
Total monthly cost = access charges + compute time + proxy or network costs + storage + monitoring and maintenance allowance
Leave unknown values as unknown. Do not assume that a free endpoint has no cost: engineering time, failed runs, data cleaning, and monitoring still affect the project. Conversely, do not assume an API is automatically cheaper than scraping; calculate the volume and required frequency.
5. Check freshness
Define the maximum acceptable age of a record. A daily feed may be adequate for an archive, but not for a workflow that reacts to frequent changes. Compare source update frequency with your polling schedule and include a method for detecting missed updates, such as timestamps, cursors, or conditional requests. See the guide to HTTP caching for web scrapers for a way to reduce unnecessary downloads where supported.
Inputs and assumptions
Before selecting a source, create a short project record. It should contain:
- Purpose: the decision the data will support and the users who will receive it.
- Required fields: names, types, examples, acceptable null values, and units.
- Time range: new records only, a fixed historical period, or the complete available archive.
- Freshness target: the maximum acceptable delay and polling schedule.
- Volume: expected records per run, page size, detail calls, and retry allowance.
- Authentication: public access, API key, OAuth flow, signed request, session, or another documented method.
- Pagination: page numbers, offsets, cursors, continuation links, or feed boundaries.
- Failure plan: retry rules, backoff, checkpointing, partial-run handling, and a fallback source.
- Permission review: applicable terms, licence conditions, privacy considerations, access controls, and the intended use of the collected data.
- Ownership: who reviews source changes, rotates credentials, investigates alerts, and approves schema changes.
Authentication and permission are different questions. A request can be technically possible while still requiring a review of the provider’s documentation, terms, licence, or access restrictions. For UK-focused projects, use the web scraping compliance checklist as a starting point, not as a substitute for advice tailored to your use case.
Save representative responses from each candidate source. Include an empty result, a large result, a missing field, a duplicate record, and an error response where possible. These fixtures make parser tests repeatable and help reveal whether an API response is genuinely more stable than page markup.
Worked examples
Example 1: A product price monitor
Suppose a project needs a product identifier, current price, availability, currency, and a product URL. An official API is the first candidate if it exposes all five fields and permits the intended polling frequency. An RSS feed may be unsuitable if it contains only new products or announcements. Embedded JSON could provide price and availability, but the team must test whether those values are consistently present across product variants. HTML is a fallback when the visible page is the only source, with validation for missing prices, currency changes, and out-of-stock labels.
If there are 2,400 products, a list endpoint returns 100 products per page, and each detail request is needed for the price, the estimate is 24 list requests plus 2,400 detail requests per run, before retries. A source that returns all required fields in its list response may be operationally preferable even if both sources are technically accessible.
Example 2: An SEO content feed
For newly published articles, an RSS or XML feed may provide enough information: title, URL, publication date, and summary. If the project also needs canonical tags, hreflang, robots directives, or page headings, it may need detail requests to the HTML pages. The SEO metadata extraction guide covers the additional fields to map and validate.
In this case, use the feed for discovery and HTML only for the fields it does not contain. That hybrid approach can reduce requests while preserving coverage. Record the feed’s archive depth and decide how to handle an item that appears late or changes after publication.
Example 3: A data archive in downloadable documents
A provider may publish reports as PDFs rather than through an API. If the required data exists in those files, downloading and parsing the documents can be more appropriate than scraping a summary page. Document the file URL, publication date, checksum or version marker, extraction method, and review process for changed layouts. The guide to scraping PDFs and downloadable documents discusses text, tables, and OCR considerations.
When to recalculate
Revisit the decision whenever an input changes, not only when a scraper fails. Recalculate after an API introduces authentication, changes pagination, reduces a quota, changes its pricing model, or removes a required field. Recheck RSS and XML feeds when their archive depth, item structure, or update schedule changes. Re-test embedded JSON and HTML extractors after a redesign, front-end framework change, rendering change, or unexpected increase in missing fields.
Set a review interval appropriate to the project’s risk and update frequency. At each review, compare actual requests with the estimate, measure missing and duplicate records, inspect response times, and verify that stored values still match the project’s purpose. The data validation checklist can help turn these checks into automated alerts.
Finish with an explicit decision: chosen source, rejected alternatives, assumptions, fallback, owner, and next review date. If the source becomes unreliable, do not immediately add more browser automation or proxy rotation. First confirm whether a documented endpoint or feed now exists, reduce unnecessary requests with caching, and revisit the permission boundary. A well-scoped data workflow is easier to maintain because it explains not only how to collect the data, but why that method remains appropriate.