How to Scrape Dynamic Websites with Playwright: Waiting, Pagination, and Network Data

Overview

A conventional HTTP request may return only a page shell when a site renders its content in the browser. Playwright is useful in this situation because it can load the page, execute JavaScript, interact with controls, and expose both the rendered document and the requests made by the page.

Before writing a browser automation script, identify the simplest reliable source of the data. If a documented or permitted data endpoint provides the required fields, using that endpoint may be more efficient than rendering every page. If the information is available only after interaction, Playwright provides a practical browser-based route. The related guide on APIs versus HTML parsing can help with that initial decision.

Install Playwright in a Node.js project with:

npm install playwright
npx playwright install chromium

A minimal scraper should create a browser, open a context, visit the target URL, wait for a meaningful condition, extract structured fields, and close resources:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const page = await context.newPage();

try {
  await page.goto('https://example.com/products', {
    waitUntil: 'domcontentloaded'
  });

  await page.locator('[data-testid="product-card"]').first().waitFor();

  const products = await page.locator('[data-testid="product-card"]').evaluateAll(cards =>
    cards.map(card => ({
      name: card.querySelector('[data-testid="name"]')?.textContent?.trim() ?? null,
      price: card.querySelector('[data-testid="price"]')?.textContent?.trim() ?? null
    }))
  );

  console.log(products);
} finally {
  await browser.close();
}

The selectors in this example are deliberately illustrative. Replace them with selectors that reflect the target site, and confirm that automated access and data use are appropriate for your project.

Checklist by scenario

When content is rendered after page load

• Open the page and inspect which element proves that the required content is ready.
• Prefer stable attributes such as data-testid, semantic roles, labels, or distinctive structure over long CSS paths.
• Wait for the specific result, not an arbitrary delay. For example, wait for a product card, table row, or result heading.
• Handle an empty state separately from a loading state so the scraper does not treat “no results” as a failed load.

For example:

await page.getByRole('heading', { name: 'Search results' }).waitFor();
const cards = page.locator('.result-card');
await cards.first().waitFor();

A fixed timeout can be useful while debugging, but it is usually a weak production synchronization strategy because network and rendering times vary.

When the site uses numbered pagination

• Extract the current page before clicking to the next one.
• Check whether the next control is disabled, hidden, or absent.
• Wait for the result set to change after navigation.
• Set a maximum page count or another stopping rule.
• Deduplicate records using a stable identifier or canonical URL.

const records = [];
const maxPages = 20;

for (let pageNumber = 1; pageNumber <= maxPages; pageNumber++) {
  await page.locator('.result-card').first().waitFor();

  const batch = await page.locator('.result-card').evaluateAll(cards =>
    cards.map(card => ({
      title: card.querySelector('.title')?.textContent?.trim() ?? null,
      url: card.querySelector('a')?.href ?? null
    }))
  );

  records.push(...batch);

  const next = page.getByRole('link', { name: /next/i });
  if (await next.count() === 0 || await next.isDisabled()) break;

  await Promise.all([
    page.waitForLoadState('domcontentloaded'),
    next.click()
  ]);
}

The exact navigation wait depends on the application. For client-side pagination, clicking may not trigger a full navigation. In that case, capture a value from the old result set and wait until it changes, or wait for a request and response associated with the control.

When the page uses infinite scroll

Infinite scroll needs both an interaction and a stopping condition. Scroll gradually, wait for additional items, and stop when the item count no longer increases, a loading indicator disappears permanently, or the page reports that there are no more results.

let previousCount = 0;
const maxRounds = 30;

for (let round = 0; round < maxRounds; round++) {
  const items = page.locator('.result-card');
  const currentCount = await items.count();

  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForTimeout(500);

  const newCount = await items.count();
  if (newCount === currentCount || newCount === previousCount) break;
  previousCount = newCount;
}

Replace the short delay with a condition tied to the site where possible, such as a loading indicator becoming hidden or a response completing. Always retain a maximum round count to prevent an unexpected page behaviour from creating an endless run.

When the useful data is in network responses

A rendered table is not always the best extraction target. If the page requests structured JSON, listening for the relevant response can reduce parsing work and preserve fields that are not visible in the interface.

const responsePromise = page.waitForResponse(response =>
  response.url().includes('/api/products') &&
  response.request().method() === 'GET' &&
  response.ok()
);

await page.getByRole('button', { name: 'Load more' }).click();
const response = await responsePromise;
const payload = await response.json();
console.log(payload);

Use a specific URL fragment, request method, and status check rather than accepting the first JSON response. Inspect the response shape, handle missing fields, and avoid assuming that an internal endpoint is stable. Network capture is most useful when it complements a clear, permitted workflow rather than bypassing access controls.

When the workflow includes downloads or separate browser sessions

Use a browser context to isolate cookies, local storage, permissions, and session state. This is useful when a run needs separate accounts or independent sessions. For downloads, start waiting for the download before clicking:

const downloadPromise = page.waitForEvent('download');
await page.getByRole('button', { name: 'Export' }).click();
const download = await downloadPromise;
await download.saveAs(`./data/${await download.suggestedFilename()}`);

Do not place session credentials directly in source code. Use environment variables or a secure secret store, and document the minimum permissions required.

What to double-check

• Wait logic: Confirm that the condition represents usable data, not merely a loaded shell or visible spinner.
• Selector resilience: Test selectors against realistic variations, including missing images, empty fields, and optional labels.
• Pagination boundaries: Verify the first page, final page, duplicate-page behaviour, and any maximum limit.
• Data quality: Normalise whitespace, preserve source URLs, parse prices and dates carefully, and validate required fields.
• Failure evidence: Save an error message, URL, screenshot, and relevant HTML or trace information when a run fails.
• Resource use: Close pages, contexts, and browsers in a finally block. Reuse a browser when appropriate, but isolate state with contexts.
• Request behaviour: Add deliberate rate limiting, bounded retries, and backoff. A browser can generate more requests than a simple HTTP client, especially on pages with many assets.
• Access and data handling: Check the site’s published instructions, your permissions, and the sensitivity of the data before scheduling collection.

For broader operational guidance, see the web scraping reliability checklist. If the task does not require a browser, the comparison of Requests, BeautifulSoup, Scrapy, and Playwright can help you choose a lighter tool.

Common mistakes

Using long arbitrary sleeps everywhere. Delays can hide synchronization problems and still fail when a page is slower than expected. Replace them with locator, response, or state-based waits wherever possible.

Depending on generated CSS classes. Classes produced by a build system may change without affecting the page’s user experience. Prefer accessible names, stable attributes, and structural relationships that reflect the interface.

Extracting before the list is complete. A first batch of cards may appear while more records are still loading. Define what “complete” means for the page and verify that condition before saving results.

Assuming every click causes navigation. Modern applications often update the DOM without changing the URL. Wait for the expected content or network response instead of always calling a navigation wait.

Ignoring duplicates and partial runs. Retries can repeat a page, while a timeout can leave a dataset incomplete. Store a source identifier, record the run status, and make downstream imports idempotent where possible.

Starting with a headless browser for every job. Browser automation has more overhead than direct requests and HTML parsing. Use it where JavaScript rendering or interaction is genuinely required, then keep extraction focused on the smallest useful surface.

When to revisit

Revisit this workflow before a seasonal planning cycle, a major change in the target site, or a change in the data fields your application needs. A scraper can continue running while quietly returning incomplete data, so a successful process exit is not enough evidence of correctness.

Use this maintenance checklist:

1. Run a small test against known pages and compare expected records with actual output.
2. Inspect selectors, pagination controls, empty states, and response patterns.
3. Review screenshots or traces from recent failures and check for layout changes.
4. Confirm that rate limits, retry rules, browser versions, and proxy or network settings still match the workload.
5. Check validation alerts for sudden drops in record counts, missing fields, duplicate URLs, or unusual response formats.
6. Update the runbook with changed selectors, stopping conditions, credentials, and recovery steps.

Keep the workflow small, observable, and easy to test. When the page changes, update the wait conditions and extraction contract together rather than patching one selector in isolation. That practice makes Playwright web scraping more dependable and gives you a repeatable way to scrape website data from dynamic interfaces without treating every failure as a mystery.