# How to Use Playwright for Scraping

> Use Playwright for web scraping: install and launch, wait for dynamic content, extract with locators, add proxies and stealth, and know when a scraping API is cheaper.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/How to Use Playwright for Scraping

# How to use Playwright for scraping: setup, selectors, waits and scale

Browser agentsJul 30, 2026·5 min read·QuanticData Team

On this page [Why Playwright and not requests](/blog/how-to-use-playwright-for-scraping/#why-playwright-and-not-requests) [Setup and a first scrape](/blog/how-to-use-playwright-for-scraping/#setup-and-a-first-scrape) [The one skill that matters: waiting correctly](/blog/how-to-use-playwright-for-scraping/#the-one-skill-that-matters-waiting-correctly) [Extracting cleanly and handling pagination](/blog/how-to-use-playwright-for-scraping/#extracting-cleanly-and-handling-pagination) [Proxies and staying unblocked](/blog/how-to-use-playwright-for-scraping/#proxies-and-staying-unblocked) [Where Playwright stops being the answer](/blog/how-to-use-playwright-for-scraping/#where-playwright-stops-being-the-answer)

Playwright scrapes by driving a real browser: it launches Chromium, Firefox or WebKit, loads a page exactly as a user would — running the JavaScript that assembles modern sites — waits for the content you want, and extracts it with locators. That makes it the right tool for dynamic pages a plain HTTP request can't read, and the wrong tool the moment you need thousands of pages a minute.

## Why Playwright and not requests

A plain HTTP fetch gets the raw HTML the server first sends. On a modern site that HTML is often an empty shell — the products, prices and reviews get built in the browser by JavaScript after load. Playwright runs that JavaScript, so it sees the finished page. It also handles the mechanics that trip up naive scrapers: automatic waiting for elements to appear, real clicks and typing, multiple browser engines, and a clean async API. The cost is weight: every page is a full browser, which is powerful and slow.

## Setup and a first scrape

Install Playwright and its browser binaries, then launch, navigate and extract:

```
# install
pip install playwright
playwright install chromium
```

```
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")   # wait for JS content
    cards = page.query_selector_all(".product-card")
    for c in cards:
        title = c.query_selector(".title").inner_text()
        price = c.query_selector(".price").inner_text()
        print(title, price)
    browser.close()
```

That is a complete, working dynamic scraper. The `wait_for_selector` line is the one beginners skip and then wonder why they get empty results — it pauses until the JavaScript-rendered content actually exists.

## The one skill that matters: waiting correctly

Ninety percent of Playwright scraping bugs are timing bugs. The page "loaded" but the data hasn't rendered yet, so you scrape nothing. Never solve this with fixed sleeps — they are slow when short and flaky when long. Wait for the thing you actually need:

| Wait for… | Method | Use when |
| --- | --- | --- |
| An element | `page.wait_for_selector(sel)` | Content appears after a render |
| Network idle | `page.wait_for_load_state("networkidle")` | Many async requests finish loading |
| A URL/navigation | `page.wait_for_url(pattern)` | After a click that navigates |
| A function | `page.wait_for_function(js)` | A custom "is it ready" condition |

Modern Playwright locators (`page.locator(".price")`) also auto-wait before acting, which removes most explicit waits — prefer them over `query_selector` for anything you interact with.

## Extracting cleanly and handling pagination

Once the content is on the page, extraction is the easy part — but two habits separate a demo from a scraper you can leave running. First, extract defensively: a missing element should yield a null and a logged warning, not a crash that kills a thousand-page run on page seven. Wrap each field access so one malformed card does not sink the batch. Second, handle pagination as navigation, not guesswork. Sites paginate three ways, and each needs a different loop: classic numbered pages you visit by URL, "load more" buttons you click and then wait for the new elements, and infinite scroll where you scroll to the bottom and wait for the network to settle before scrolling again. Detect which pattern the target uses before writing the loop; a scroll loop on a numbered-pages site, or a URL loop on an infinite-scroll feed, is the second most common way Playwright scrapers silently miss most of the data.

For anything beyond a one-off script, put the browser lifecycle behind a small wrapper: launch once, reuse a context across pages rather than relaunching per URL, set sensible timeouts, and always close in a `finally` so a crash does not leak browser processes. Reusing the context also preserves cookies and session state, which is what lets a logged-in flow carry across the pages you scrape after signing in.

## Proxies and staying unblocked

A headless browser on a datacenter IP gets flagged fast. Two additions keep a Playwright scraper alive at any real volume. First, route through proxies — pass them at launch:

```
browser = p.chromium.launch(
    proxy={"server": "http://pr.quanticdata.io:7777",
           "username": "USER", "password": "KEY"})
```

Use [residential](https://quanticdata.io/residential-proxies/) or [rotating proxies](https://quanticdata.io/rotating-proxies/) for defended targets; a headless browser with a residential exit and human-like pacing is genuinely hard to distinguish from a real user. Second, reduce the automation fingerprint — headless browsers leak signals (navigator.webdriver, missing plugins) that detectors read. Stealth plugins patch the obvious ones, though the arms race never fully ends. The pairing that works: real browser + residential IP + human timing, the same recipe we describe for [browser agents](https://quanticdata.io/blog/how-to-build-an-ai-browser-agent/).

## Where Playwright stops being the answer

Playwright is excellent for dynamic pages, logged-in flows and interaction-heavy targets. It is expensive for scale: each page is a full browser using hundreds of megabytes and seconds of CPU, so a thousand pages a minute means a browser farm to build and babysit, plus the proxy and anti-block layers on top. Two escape hatches. If the site's data is actually in the initial HTML or a JSON API call, skip the browser entirely — inspect the network tab first; you may not need Playwright at all. And if you do need rendering at scale but not the operational burden, a [scraping API](https://quanticdata.io/web-scraping-api/) runs the browser fleet for you — JS rendering from $0.001 per page, proxies and anti-block included, clean Markdown or JSON out. The honest rule: Playwright when you need fine control over a browser; an API when you need pages at volume and would rather not operate the fleet. Many pipelines use both — Playwright for the tricky logged-in flows, the API for the bulk.

### Sources & further reading

- [Oxylabs — Playwright web scraping tutorial](https://oxylabs.io/blog/playwright-web-scraping)

- [ScraperAPI — Playwright web scraping: complete guide](https://www.scraperapi.com/blog/playwright-web-scraping/)

## FAQ

Quick answers on how to use playwright for scraping.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### Is Playwright good for web scraping?

Yes, for dynamic and interaction-heavy sites: it runs JavaScript, auto-waits for elements, drives real clicks and typing, and supports Chromium, Firefox and WebKit. It is overkill and expensive for simple static pages or large-scale extraction, where a plain HTTP request or a scraping API is far cheaper per page.

### Playwright vs Selenium for scraping — which is better?

Playwright is generally the better modern choice: faster, a cleaner async API, built-in auto-waiting that eliminates most flaky-sleep bugs, and multi-engine support out of the box. Selenium has a larger legacy ecosystem. For a new scraping project, Playwright's auto-waiting alone saves the most common class of bugs.

### How do I add a proxy to Playwright?

Pass a proxy object at launch: proxy={"server": "http://host:port", "username": ..., "password": ...}. Use residential or rotating proxies for defended targets, since a headless browser on a datacenter IP is flagged quickly. You can also set the proxy per browser context for finer control.

### How do I scrape a page where content loads after the page loads?

Wait for the specific content, not a fixed sleep: page.wait_for_selector for an element, wait_for_load_state('networkidle') for many async requests, or wait_for_function for a custom condition. Modern locators also auto-wait before acting. Empty results almost always mean you scraped before the JavaScript rendered.

### Can websites detect Playwright?

Yes — headless browsers leak automation signals like navigator.webdriver and missing browser features that detectors read. Stealth plugins patch the obvious ones, but detection is an arms race. The durable approach pairs a real (optionally headed) browser with residential proxies and human-like timing so the session resembles a genuine user.

## Need rendering at scale without the browser farm?

The scraping API runs the browser fleet, proxies and anti-block for you — JavaScript rendering from $0.001 per page, clean Markdown or JSON out, pay per success. $2 free every month.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Cloud Browser for AI Agents](https://quanticdata.io/browser-ai/)

## Related reading

[Browser agents What Is Browser Automation? Browser automation drives a real browser with code or an AI agent. Here is how it works, when it beats a plain HTTP request, and what it actually costs. Read →](https://quanticdata.io/blog/what-is-browser-automation/) [Browser agents What Is AI Automation? AI automation puts a model in the middle of a workflow so it can read messy input, decide, and act. Definition, examples, tooling criteria and real cost math. Read →](https://quanticdata.io/blog/what-is-ai-automation/) [Browser agents How to Build an AI Browser Agent The observe-decide-act loop, the browser-control stack, prompt-injection defenses and honest cost math — everything a working browser agent actually needs. Read →](https://quanticdata.io/blog/how-to-build-an-ai-browser-agent/)

---

Source: https://quanticdata.io/blog/how-to-use-playwright-for-scraping/ · Site index for AI: https://quanticdata.io/llms.txt