Web Scraping with Playwright: A Practical Guide to JavaScript-Rendered Websites

Overview

Traditional web scraping often starts with an HTTP request, followed by HTML parsing. That method works well when the data is present in the initial response. A JavaScript-heavy site may instead return a page shell and populate the visible content after scripts run. In that situation, a browser automation tool such as Playwright can load the page, interact with controls, and read the rendered result.

Playwright is useful when a workflow depends on actions such as clicking a tab, selecting a filter, submitting a form, or scrolling through a results list. It can also create isolated browser contexts, capture downloads, and work with multiple browser engines through a consistent API. The trade-off is greater resource usage and more moving parts than a direct request-and-parser workflow.

Before building a Playwright web scraper, identify the smallest reliable data path. If a documented or publicly observable endpoint returns the required data and you are permitted to use it, a direct request may be simpler. Use browser automation when rendering or interaction is genuinely part of the workflow. For a broader tool choice, compare browser automation with Scrapy and BeautifulSoup in this Playwright, Puppeteer, Scrapy, and BeautifulSoup comparison.

Minimal setup

Install Playwright in a JavaScript or TypeScript project, then install the browser binaries required by your environment:

npm install playwright
npx playwright install

A minimal scraper should create a browser, open a context and page, navigate to a target, extract only the fields it needs, and close resources in a finally block. Keeping these responsibilities separate makes later changes easier.

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
try {
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto('https://example.com/catalog', { waitUntil: 'domcontentloaded' });

  const records = await page.locator('[data-item]').evaluateAll(items =>
    items.map(item => ({
      name: item.querySelector('[data-name]')?.textContent?.trim() ?? null,
      url: item.querySelector('a')?.href ?? null
    }))
  );

  console.log(JSON.stringify(records, null, 2));
} finally {
  await browser.close();
}

Use this as a starting pattern, not a complete production pipeline. The selectors, navigation rules, error handling, and output validation must match the site and the purpose of the extraction.

Checklist by scenario

When content appears after page load

• Confirm that the target data is absent from the initial HTML but visible after the page runs.
• Prefer a semantic or stable selector, such as a meaningful role, label, or data attribute, over a deeply nested CSS path.
• Wait for the specific element or state you need instead of using a fixed delay.
• Check that the element contains usable text or attributes before extracting it.

For example, a targeted wait is more informative than waiting two seconds unconditionally:

await page.locator('[data-results]').waitFor({ state: 'visible' });
const count = await page.locator('[data-result]').count();

A fixed timeout can still be useful for diagnosing a race condition, but it should not be the main synchronization strategy. The guide to scraping dynamic websites with Playwright provides a complementary workflow for rendered content.

When the workflow uses pagination

• Record the current page or cursor so a failed run can be diagnosed.
• Extract records before moving to the next page.
• Detect the end using a disabled next button, a missing control, an unchanged URL, or an empty result set.
• Set a defensible maximum page limit to prevent an unexpected loop.
• Deduplicate records using a stable identifier or canonical URL.

for (let pageNumber = 1; pageNumber <= 100; pageNumber++) {
  await page.locator('[data-result]').first().waitFor({ state: 'visible' }).catch(() => {});
  await collectCurrentPage(page);

  const next = page.locator('[aria-label="Next page"]');
  if (!(await next.isVisible()) || await next.isDisabled()) break;
  await next.click();
}

Adjust the termination test to the actual interface. A button can remain visible while its function changes, so inspect the resulting page or URL when necessary.

When the page uses infinite scroll

• Measure whether the number of records increases after each scroll.
• Stop after several rounds with no growth, or when an explicit end marker appears.
• Allow time for content to render, but avoid an unbounded loop.
• Capture the final record count and any visible loading or error state.

let previousCount = 0;
let unchangedRounds = 0;

for (let round = 0; round < 50; round++) {
  const count = await page.locator('[data-result]').count();
  if (count === previousCount) unchangedRounds++;
  else unchangedRounds = 0;
  if (unchangedRounds >= 3) break;

  previousCount = count;
  await page.mouse.wheel(0, 1200);
  await page.waitForTimeout(500);
}

When downloads or authentication are involved

• Use a browser context to isolate cookies, local storage, and permissions for each run.
• For a download, start waiting for the event before clicking the control.
• Keep credentials outside source code and logs.
• Use only accounts and access you are authorized to use.
• Define the boundary of the workflow: public pages, an approved test account, or another explicitly permitted area.

const downloadPromise = page.waitForEvent('download');
await page.getByRole('button', { name: 'Export' }).click();
const download = await downloadPromise;
await download.saveAs('./output/export.csv');

Do not assume that a successful login means every subsequent page is available to the scraper. Model authentication as a controlled workflow with clear permissions, limited scope, and careful handling of session data.

What to double-check

Selectors and page states

Test selectors against more than one representative page. A selector that works for a full result may fail for an empty state, an item with missing fields, or a page containing an error message. Prefer extraction functions that return null or an explicit validation error instead of silently shifting fields into the wrong columns.

Use Playwright's locator model and accessible roles where they remain stable. If a visual redesign changes class names, data attributes or meaningful labels may provide a more durable boundary. Keep selectors in one place so a site change does not require editing the entire scraper.

Waits, retries, and throttling

Separate page readiness from network completion. A page may finish its initial navigation while a component is still requesting data. Conversely, waiting for every network request to stop can be unreliable on pages with analytics, polling, or long-lived connections. Wait for the exact result, table, card list, or state that proves the extraction can proceed.

Retry only operations that are plausibly transient, such as a navigation timeout or temporary server response. Keep retries bounded, add increasing delays, and log the attempt number. Add a modest pause between pages or actions where appropriate, and avoid parallelism until the sequential workflow is correct and its impact is understood. A reusable pipeline should also record failures rather than hiding them; see this guide to reliable Playwright scraping pipelines.

Output quality

• Define a schema before extraction: field names, types, required values, and normalization rules.
• Preserve the source URL and a collection timestamp when they are useful for auditing.
• Normalize whitespace, dates, prices, and links consistently.
• Validate record counts and required fields before publishing or loading data downstream.
• Save raw or diagnostic artifacts for failed runs when permitted and practical.

Structured JSON is convenient for nested records, while CSV can be useful for tabular review. Choose an export format based on how the next system consumes the data rather than on the scraping library alone.

Common mistakes

1. Using a browser for every task. If the required data is available in a permitted, stable response, a direct client and parser may be faster and easier to operate.
2. Relying on arbitrary sleeps. Time-based waits create slow runs and still fail when the page is slower than expected. Prefer state-based waits.
3. Choosing brittle selectors. Long chains of generated class names can break during minor interface changes. Record why a selector was chosen and keep a fallback only when it can be validated safely.
4. Ignoring empty and error states. A scraper can return technically valid but incorrect records if it treats an error message as content. Check page state before extraction.
5. Letting loops run indefinitely. Pagination and scrolling need explicit limits, progress checks, and termination conditions.
6. Skipping observability. Log the URL, stage, duration, record count, and error category. Without these details, maintenance becomes guesswork.
7. Mixing collection and transformation. Keep browser interaction, parsing, validation, and export as distinct steps. This makes it easier to test parsing without launching a browser.

When to revisit

Revisit a Playwright scraper before seasonal planning cycles, after a site redesign, when the workflow or browser tooling changes, and whenever output quality drops. A successful run is not proof that the data is still complete: a renamed control, an altered empty state, or a changed pagination rule can produce partial results without an obvious crash.

Use this maintenance checklist:

• Run a small test against known pages and compare expected fields.
• Verify selectors, waits, pagination limits, and infinite-scroll termination.
• Check authentication boundaries and rotate or remove test credentials as needed.
• Review retry counts, throttling, browser versions, and resource usage.
• Validate a sample of exported records, including links, dates, and missing values.
• Update the scraper's notes when the page structure or collection purpose changes.

Start with a narrow, observable run, confirm the output, and only then scale to more pages or parallel workers. If the task has become mostly static HTML parsing, reassess whether browser automation is still necessary. If it depends on interaction, keep the Playwright workflow focused, bounded, and easy to test. For use cases such as technical audits or structured table collection, pair this checklist with the technical SEO scraping workflow or the guide to extracting tables reliably.