How to use Playwright for scraping: setup, selectors, waits and scale

Playwright scrapes by driving a real browser: it launches Chromium, Firefox or WebKit, loads a page exactly as a user would — running the JavaScript that assembles modern sites — waits for the content you want, and extracts it with locators. That makes it the right tool for dynamic pages a plain HTTP request can't read, and the wrong tool the moment you need thousands of pages a minute.

Why Playwright and not requests

A plain HTTP fetch gets the raw HTML the server first sends. On a modern site that HTML is often an empty shell — the products, prices and reviews get built in the browser by JavaScript after load. Playwright runs that JavaScript, so it sees the finished page. It also handles the mechanics that trip up naive scrapers: automatic waiting for elements to appear, real clicks and typing, multiple browser engines, and a clean async API. The cost is weight: every page is a full browser, which is powerful and slow.

Setup and a first scrape

Install Playwright and its browser binaries, then launch, navigate and extract:

# install
pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")   # wait for JS content
    cards = page.query_selector_all(".product-card")
    for c in cards:
        title = c.query_selector(".title").inner_text()
        price = c.query_selector(".price").inner_text()
        print(title, price)
    browser.close()

That is a complete, working dynamic scraper. The wait_for_selector line is the one beginners skip and then wonder why they get empty results — it pauses until the JavaScript-rendered content actually exists.

The one skill that matters: waiting correctly

Ninety percent of Playwright scraping bugs are timing bugs. The page "loaded" but the data hasn't rendered yet, so you scrape nothing. Never solve this with fixed sleeps — they are slow when short and flaky when long. Wait for the thing you actually need:

Modern Playwright locators (page.locator(".price")) also auto-wait before acting, which removes most explicit waits — prefer them over query_selector for anything you interact with.

Extracting cleanly and handling pagination

Once the content is on the page, extraction is the easy part — but two habits separate a demo from a scraper you can leave running. First, extract defensively: a missing element should yield a null and a logged warning, not a crash that kills a thousand-page run on page seven. Wrap each field access so one malformed card does not sink the batch. Second, handle pagination as navigation, not guesswork. Sites paginate three ways, and each needs a different loop: classic numbered pages you visit by URL, "load more" buttons you click and then wait for the new elements, and infinite scroll where you scroll to the bottom and wait for the network to settle before scrolling again. Detect which pattern the target uses before writing the loop; a scroll loop on a numbered-pages site, or a URL loop on an infinite-scroll feed, is the second most common way Playwright scrapers silently miss most of the data.

For anything beyond a one-off script, put the browser lifecycle behind a small wrapper: launch once, reuse a context across pages rather than relaunching per URL, set sensible timeouts, and always close in a finally so a crash does not leak browser processes. Reusing the context also preserves cookies and session state, which is what lets a logged-in flow carry across the pages you scrape after signing in.

Proxies and staying unblocked

A headless browser on a datacenter IP gets flagged fast. Two additions keep a Playwright scraper alive at any real volume. First, route through proxies — pass them at launch:

browser = p.chromium.launch(
    proxy={"server": "http://pr.quanticdata.io:7777",
           "username": "USER", "password": "KEY"})

Use residential or rotating proxies for defended targets; a headless browser with a residential exit and human-like pacing is genuinely hard to distinguish from a real user. Second, reduce the automation fingerprint — headless browsers leak signals (navigator.webdriver, missing plugins) that detectors read. Stealth plugins patch the obvious ones, though the arms race never fully ends. The pairing that works: real browser + residential IP + human timing, the same recipe we describe for browser agents.

Where Playwright stops being the answer

Playwright is excellent for dynamic pages, logged-in flows and interaction-heavy targets. It is expensive for scale: each page is a full browser using hundreds of megabytes and seconds of CPU, so a thousand pages a minute means a browser farm to build and babysit, plus the proxy and anti-block layers on top. Two escape hatches. If the site's data is actually in the initial HTML or a JSON API call, skip the browser entirely — inspect the network tab first; you may not need Playwright at all. And if you do need rendering at scale but not the operational burden, a scraping API runs the browser fleet for you — JS rendering from $0.001 per page, proxies and anti-block included, clean Markdown or JSON out. The honest rule: Playwright when you need fine control over a browser; an API when you need pages at volume and would rather not operate the fleet. Many pipelines use both — Playwright for the tricky logged-in flows, the API for the bulk.