How to use scrapy-playwright: render JavaScript inside your Scrapy spider

scrapy-playwright is a download handler that lets a Scrapy spider render JavaScript by routing chosen requests through a Playwright browser instead of a plain HTTP fetch. You keep Scrapy's spider, pipeline and scheduling, and flag only the requests that need a real browser — so dynamic pages become scrapeable without abandoning the framework. The tradeoff is throughput: a browser per page is far heavier than an HTTP request.

Why you'd need it

Scrapy is fast and excellent, but its default downloader only sees the server-rendered HTML — content assembled in the browser by JavaScript is invisible to it. Rather than rewrite your crawler in a browser tool, scrapy-playwright bolts rendering onto Scrapy for the pages that need it, leaving the rest on the fast HTTP path. That selective approach is the whole point: you don't pay the browser cost for pages that don't need it.

Install and configure

Install the package and its browser, then register the download handlers and the asyncio reactor in your settings:

pip install scrapy-playwright
playwright install chromium

# settings.py
DOWNLOAD_HANDLERS = {
    "http":  "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

The TWISTED_REACTOR line is mandatory and the number-one setup mistake — scrapy-playwright needs the asyncio reactor, and omitting it produces a cryptic error at startup. Set it and the handlers are ready.

Requesting a page with Playwright

Rendering is opt-in per request via meta. Only requests you flag go through the browser:

import scrapy

class MySpider(scrapy.Spider):
    name = "js"
    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/products",
            meta={"playwright": True,
                  "playwright_page_methods": [
                      {"method": "wait_for_selector",
                       "args": [".product-card"]},
                  ]},
        )
    def parse(self, response):
        # response.text is now the RENDERED HTML
        for card in response.css(".product-card"):
            yield {"title": card.css(".title::text").get()}

The playwright_page_methods list runs browser actions before the page is handed back — waiting for a selector, scrolling, clicking. As with any browser scraping, waiting for the actual content (not a fixed sleep) is what makes it reliable; without the wait you parse an empty page.

Handling interactions and pagination

Beyond a single wait, real spiders need to interact — click a load-more button, fill a search box, scroll an infinite feed — and scrapy-playwright exposes the page object so you can. Passing playwright_include_page: True in the request meta hands your callback the live Playwright page, which you drive directly and then close (always close it, or you leak browser pages and eventually stall the crawl). This is how you paginate a JavaScript feed inside Scrapy: get the page, scroll or click in a loop until no new content loads, then yield the accumulated data. It is more code than a static crawl, but it keeps a genuinely interactive target inside your existing Scrapy pipeline rather than forcing a separate tool. The discipline is the same as any browser automation — wait for what you need, act, and clean up the page handle when done.

Adding a proxy

Proxies are passed to the browser context. scrapy-playwright reads them from a launch option in settings, applying to every Playwright request:

# settings.py
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {"server": "http://pr.quanticdata.io:7777",
              "username": "USER", "password": "PASS"},
}

For scraping at scale you want rotation and, on defended targets, residential IPs — a browser on a datacenter range gets flagged fast. Point the proxy at a rotating endpoint for a fresh IP per request, and use residential proxies where the target blocks hard. The same pairing of browser plus trusted IP plus human pacing that keeps any browser scraper alive applies here; we cover it in Playwright stealth.

The throughput cost — and the alternative

Here is the reality to plan around: a Playwright request uses hundreds of megabytes and seconds of CPU where a plain Scrapy request uses almost nothing. Flag every request as playwright: True and your fast Scrapy crawler becomes a slow browser farm, with concurrency limited by RAM. The discipline is to render only the pages that truly need it — inspect the target first: if the data is in the initial HTML or a JSON API call the page makes, skip Playwright entirely and let Scrapy's fast path handle it. When you do need rendering at scale but not the operational weight of a browser fleet plus proxies inside your crawler, an alternative is to keep plain Scrapy and point its requests at a scraping API that renders and returns clean HTML or Markdown server-side — your spider stays lightweight and the browser cost moves off your machine. Many teams run scrapy-playwright for the handful of genuinely interactive pages and the API for the high-volume rendered bulk. Either way, the principle holds: rendering is expensive, so spend it deliberately.