# How to Use scrapy-playwright

> Use scrapy-playwright to render JavaScript in Scrapy: install and configure the download handler, request pages with Playwright, add proxies, and know the cost.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/How to Use scrapy-playwright

# How to use scrapy-playwright: render JavaScript inside your Scrapy spider

GuidesJul 30, 2026·5 min read·QuanticData Team

On this page [Why you'd need it](/blog/how-to-use-scrapy-playwright/#why-you-d-need-it) [Install and configure](/blog/how-to-use-scrapy-playwright/#install-and-configure) [Requesting a page with Playwright](/blog/how-to-use-scrapy-playwright/#requesting-a-page-with-playwright) [Handling interactions and pagination](/blog/how-to-use-scrapy-playwright/#handling-interactions-and-pagination) [Adding a proxy](/blog/how-to-use-scrapy-playwright/#adding-a-proxy) [The throughput cost — and the alternative](/blog/how-to-use-scrapy-playwright/#the-throughput-cost-and-the-alternative)

scrapy-playwright is a download handler that lets a Scrapy spider render JavaScript by routing chosen requests through a Playwright browser instead of a plain HTTP fetch. You keep Scrapy's spider, pipeline and scheduling, and flag only the requests that need a real browser — so dynamic pages become scrapeable without abandoning the framework. The tradeoff is throughput: a browser per page is far heavier than an HTTP request.

## Why you'd need it

Scrapy is fast and excellent, but its default downloader only sees the server-rendered HTML — content assembled in the browser by JavaScript is invisible to it. Rather than rewrite your crawler in a browser tool, scrapy-playwright bolts rendering onto Scrapy for the pages that need it, leaving the rest on the fast HTTP path. That selective approach is the whole point: you don't pay the browser cost for pages that don't need it.

## Install and configure

Install the package and its browser, then register the download handlers and the asyncio reactor in your settings:

```
pip install scrapy-playwright
playwright install chromium
```

```
# settings.py
DOWNLOAD_HANDLERS = {
    "http":  "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
```

The `TWISTED_REACTOR` line is mandatory and the number-one setup mistake — scrapy-playwright needs the asyncio reactor, and omitting it produces a cryptic error at startup. Set it and the handlers are ready.

## Requesting a page with Playwright

Rendering is opt-in per request via `meta`. Only requests you flag go through the browser:

```
import scrapy

class MySpider(scrapy.Spider):
    name = "js"
    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/products",
            meta={"playwright": True,
                  "playwright_page_methods": [
                      {"method": "wait_for_selector",
                       "args": [".product-card"]},
                  ]},
        )
    def parse(self, response):
        # response.text is now the RENDERED HTML
        for card in response.css(".product-card"):
            yield {"title": card.css(".title::text").get()}
```

The `playwright_page_methods` list runs browser actions before the page is handed back — waiting for a selector, scrolling, clicking. As with any browser scraping, waiting for the actual content (not a fixed sleep) is what makes it reliable; without the wait you parse an empty page.

## Handling interactions and pagination

Beyond a single wait, real spiders need to interact — click a load-more button, fill a search box, scroll an infinite feed — and scrapy-playwright exposes the page object so you can. Passing `playwright_include_page: True` in the request meta hands your callback the live Playwright page, which you drive directly and then close (always close it, or you leak browser pages and eventually stall the crawl). This is how you paginate a JavaScript feed inside Scrapy: get the page, scroll or click in a loop until no new content loads, then yield the accumulated data. It is more code than a static crawl, but it keeps a genuinely interactive target inside your existing Scrapy pipeline rather than forcing a separate tool. The discipline is the same as any browser automation — wait for what you need, act, and clean up the page handle when done.

## Adding a proxy

Proxies are passed to the browser context. scrapy-playwright reads them from a launch option in settings, applying to every Playwright request:

```
# settings.py
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {"server": "http://pr.quanticdata.io:7777",
              "username": "USER", "password": "PASS"},
}
```

For scraping at scale you want rotation and, on defended targets, residential IPs — a browser on a datacenter range gets flagged fast. Point the proxy at a [rotating endpoint](https://quanticdata.io/rotating-proxies/) for a fresh IP per request, and use [residential proxies](https://quanticdata.io/residential-proxies/) where the target blocks hard. The same pairing of browser plus trusted IP plus human pacing that keeps any browser scraper alive applies here; we cover it in [Playwright stealth](https://quanticdata.io/blog/playwright-stealth-in-python/).

## The throughput cost — and the alternative

Here is the reality to plan around: a Playwright request uses hundreds of megabytes and seconds of CPU where a plain Scrapy request uses almost nothing. Flag every request as `playwright: True` and your fast Scrapy crawler becomes a slow browser farm, with concurrency limited by RAM. The discipline is to render *only* the pages that truly need it — inspect the target first: if the data is in the initial HTML or a JSON API call the page makes, skip Playwright entirely and let Scrapy's fast path handle it. When you do need rendering at scale but not the operational weight of a browser fleet plus proxies inside your crawler, an alternative is to keep plain Scrapy and point its requests at a [scraping API](https://quanticdata.io/web-scraping-api/) that renders and returns clean HTML or Markdown server-side — your spider stays lightweight and the browser cost moves off your machine. Many teams run scrapy-playwright for the handful of genuinely interactive pages and the API for the high-volume rendered bulk. Either way, the principle holds: rendering is expensive, so spend it deliberately.

### Sources & further reading

- [scrapy-playwright on GitHub](https://github.com/scrapy-plugins/scrapy-playwright)

- [Scrapy documentation](https://docs.scrapy.org/)

## FAQ

Quick answers on how to use scrapy-playwright.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### What does scrapy-playwright do?

It's a Scrapy download handler that routes chosen requests through a Playwright browser so JavaScript renders before parsing. You keep Scrapy's spider, pipelines and scheduling, and flag only the requests that need a real browser with meta={'playwright': True}, leaving the rest on Scrapy's fast HTTP path.

### Why does scrapy-playwright fail on startup?

Almost always because the asyncio reactor isn't set. scrapy-playwright requires TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor' in settings.py alongside the DOWNLOAD_HANDLERS entries. Omitting the reactor line produces a cryptic error at startup; adding it resolves the most common setup failure.

### How do I add a proxy to scrapy-playwright?

Pass it in PLAYWRIGHT_LAUNCH_OPTIONS in settings.py as a proxy object with server, username and password. It applies to every Playwright request. For scraping at scale, point it at a rotating endpoint for per-request IPs and use residential proxies on targets that block datacenter ranges.

### Is scrapy-playwright slow?

Compared to plain Scrapy, yes — each Playwright request uses hundreds of MB and seconds of CPU where an HTTP request uses almost nothing, so concurrency is RAM-limited. The fix is selective use: flag only pages that genuinely need JavaScript rendering, and let Scrapy's fast path handle everything whose data is already in the HTML.

### When should I use a scraping API instead of scrapy-playwright?

When you need rendering at volume but don't want to run a browser fleet and proxies inside your crawler. Keep plain Scrapy and point its requests at a scraping API that renders and returns clean HTML or Markdown server-side — the spider stays lightweight and the browser cost moves off your machine. Many pipelines use both.

## Render at scale without the browser fleet in your crawler

Keep plain Scrapy and point its requests at the scraping API — JavaScript rendered server-side, clean HTML or Markdown back, residential proxies and anti-block included. Pay per success, $2 free monthly.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Rotating Proxies from $0.50/GB](https://quanticdata.io/rotating-proxies/)

## Related reading

[Guides How to Use a Proxy in Puppeteer Set the proxy in launch args, authenticate with page.authenticate, rotate per page via browser contexts, and dodge the mistakes that leak your real IP. Read →](https://quanticdata.io/blog/how-to-use-a-proxy-in-puppeteer/) [Guides How to Use undetected-chromedriver Install and launch the patched driver that evades Selenium detection, add a proxy, and understand the ceiling — IP reputation and behavior it can't fix. Read →](https://quanticdata.io/blog/how-to-use-undetected-chromedriver/) [Guides How to Use a Proxy in Node.js Native fetch needs an undici ProxyAgent; axios takes a proxy config or agent. The setup for both, authentication, HTTPS tunneling, and rotation for scraping. Read →](https://quanticdata.io/blog/how-to-use-a-proxy-in-nodejs/)

---

Source: https://quanticdata.io/blog/how-to-use-scrapy-playwright/ · Site index for AI: https://quanticdata.io/llms.txt