# Scraping Dynamic Web Pages in Python

> Scrape dynamic, JavaScript-built pages in Python three ways, tested on one page: read the hidden JSON, render with Playwright, or drive Selenium.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Scraping Dynamic Web Pages in Python

# Scraping dynamic web pages in Python: hidden JSON, Playwright or Selenium

Guides · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![Scraping a JavaScript page in Python: requests and Beautiful Soup received a 5,808-byte shell with 0 quotes, while the same quotes were in a JSON block inside a script tag and Playwright rendered all 10 in 3.34 seconds through a mobile proxy; the card ranks the three routes, hidden JSON first, then the API call, then a browser.](https://quanticdata.io/blog/dynamic-web-scraping-python/hero.webp?v=20261120h)

On this page

- [Why requests and Beautiful Soup see nothing](https://quanticdata.io/blog/dynamic-web-scraping-python/#why-requests-and-beautiful-soup-see-nothing)
- [Route 1: read the JSON inside the page](https://quanticdata.io/blog/dynamic-web-scraping-python/#route-1-read-the-json-inside-the-page)
- [Route 2: call the API the page uses](https://quanticdata.io/blog/dynamic-web-scraping-python/#route-2-call-the-api-the-page-uses)
- [Route 3: render with Playwright](https://quanticdata.io/blog/dynamic-web-scraping-python/#route-3-render-with-playwright)
- [Route 3b: render with Selenium](https://quanticdata.io/blog/dynamic-web-scraping-python/#route-3b-render-with-selenium)
- [Which route to choose](https://quanticdata.io/blog/dynamic-web-scraping-python/#which-route-to-choose)
- [Proxies for dynamic pages](https://quanticdata.io/blog/dynamic-web-scraping-python/#proxies-for-dynamic-pages)

To scrape a dynamic web page in Python, first check whether the data is already in the HTML (a JSON block inside a `<script>`) or comes from an API call you can request directly. If not, render the page in a real browser with Playwright or Selenium, wait for the element you need, then read it.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=dynamic-web-scraping-python) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

We tested all of it on 20 November 2026 with Python 3.12.10 on quotes.toscrape.com/js/, a public practice page whose quotes are written by JavaScript, through QuanticData mobile and residential proxies.

## Why requests and Beautiful Soup see nothing

```python
import requests
from bs4 import BeautifulSoup

PROXY = "http://USER-country-us-type-mobile-sid-d01-ttl-30m:PASS@mobile.quanticdata.io:8080"
r = requests.get("https://quotes.toscrape.com/js/", proxies={"http": PROXY, "https": PROXY}, timeout=30)
soup = BeautifulSoup(r.text, "html.parser")
print(r.status_code, len(r.content), len(soup.select("div.quote")))
# 200 5808 0
```

Status 200, 5,808 bytes, zero quotes. Nothing is wrong with the request: the server sent the page skeleton and a script, and the quotes appear only after a browser runs that script. Beautiful Soup does not run JavaScript.

## Route 1: read the JSON inside the page

Before reaching for a browser, look at what you downloaded. Search `r.text` for a value you can see on screen. On this page the two script tags hold the data as a JavaScript array:

```python
import re, json
m = re.search(r"var data = (\[.*?\]);", r.text, re.S)
quotes = json.loads(m.group(1))
print(len(quotes), sorted(quotes[0]), quotes[0]["author"]["name"])
# 10 ['author', 'tags', 'text'] Albert Einstein
```

Ten quotes, no browser, one request. Real sites do the same at scale: frameworks put their state in tags such as `__NEXT_DATA__` or `application/ld+json`. When the block is valid JSON, `json.loads` reads it; working with the result is covered in [parsing JSON in Python](https://quanticdata.io/blog/parse-json-python/).

## Route 2: call the API the page uses

Other pages fetch their data after loading. Open the browser's developer tools, Network tab, filter Fetch/XHR, and reload or scroll: the JSON request shows up, often with a page parameter. Playwright can find it for you. On the infinite-scroll version of the same site, we waited for the request the scroll triggered:

```
with page.expect_response(lambda r: "/api/quotes" in r.url and r.status == 200) as info:
    page.mouse.wheel(0, 20000)
data = info.value.json()
print(info.value.url, data["page"], len(data["quotes"]))
# https://quotes.toscrape.com/api/quotes?page=1  1  10
```

From then on you can call `/api/quotes?page=N` with requests and skip the browser. This is usually the fastest and most stable route.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=dynamic-web-scraping-python) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Route 3: render with Playwright

When the data exists only after rendering, let a browser build the page:

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    b = p.chromium.launch(proxy={"server": "http://mobile.quanticdata.io:8080",
                                 "username": "USER-country-us-type-mobile-sid-d01-ttl-30m",
                                 "password": "PASS"})
    page = b.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
    page.wait_for_selector("div.quote")
    for q in page.locator("div.quote").all():
        print(q.locator("small.author").inner_text())
    html = page.content()      # hand this to Beautiful Soup if you prefer
    b.close()
```

Through a US mobile line (Verizon Business, Conway, Arkansas), Chromium 156 rendered the page and found all 10 quotes in 3.34 s. Two habits matter more than anything else here:

- **Wait for the element, not for the page.** On the delayed version of the page, `goto(..., wait_until="networkidle")` returned after 1.67 s with 0 quotes; `wait_for_selector("div.quote")` found all 10 after 11.24 s. The full comparison of wait strategies is in [waiting for a page to load in Playwright](https://quanticdata.io/blog/playwright-wait-for-page-load/).
- **Proxy in launch().** Playwright takes the username and password in the proxy dictionary; more in [Playwright proxy setup](https://quanticdata.io/blog/playwright-proxy/).

## Route 3b: render with Selenium

Selenium drives the browsers installed on your machine. We ran Selenium 4.53.0 with Microsoft Edge 157 in headless mode on the delayed page:

```python
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver.get("https://quotes.toscrape.com/js-delayed/")
print(len(driver.find_elements(By.CLASS_NAME, "quote")))   # 0, too early
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CLASS_NAME, "quote")))
print(len(driver.find_elements(By.CLASS_NAME, "quote")))   # 10, after 10.8 s
```

Edge does not take a username and password inside `--proxy-server`, so for Selenium we ran a small local forwarder that adds them; the method and the selectors are in [Selenium find_element](https://quanticdata.io/blog/selenium-find-element/).

## Which route to choose

| Route | Requests per page | Our result | Use when |
| --- | --- | --- | --- |
| JSON in the HTML | 1 | 10 quotes, no browser | A script tag holds the data |
| The page's API | 1 per page of data | 10 quotes per call | Data loads by XHR or on scroll |
| Playwright | The page plus its scripts and assets | 10 quotes in 3.34 s | Data exists only after rendering or clicks |
| Selenium | The same | 10 quotes on the delayed page | You already have Selenium code or need a specific installed browser |

A browser downloads everything a page needs, so it uses more time and more proxy traffic per page than one JSON request. Render only the pages that need it. Playwright in depth: [Playwright for scraping](https://quanticdata.io/blog/how-to-use-playwright-for-scraping/).

## Proxies for dynamic pages

Many dynamic sites decide what to send based on the visitor: country, device, network. Our Playwright run used a QuanticData mobile line, which presents as a phone carrier's address; residential lines present as home connections in the country you choose. Both plug into Playwright's launch() and into requests for routes 1 and 2. Plans: [mobile proxies](https://quanticdata.io/mobile-proxies/) and [residential proxies](https://quanticdata.io/residential-proxies/). If you would rather not run browsers at all, our [web scraping API](https://quanticdata.io/web-scraping-api/) renders pages for you and returns the HTML or the data.

### Sources & further reading

- [Playwright for Python: Page.wait_for_selector](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)
- [Playwright for Python: Network](https://playwright.dev/python/docs/network)
- [Selenium documentation: waiting strategies](https://www.selenium.dev/documentation/webdriver/waits/)
- [Python docs: json](https://docs.python.org/3/library/json.html)

## FAQ

Quick answers on scraping dynamic web pages python.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I scrape a dynamic website with Python?

Check the HTML for a JSON block in a script tag, then the Network tab for an API call. If neither has the data, render the page with Playwright or Selenium, wait for the element with `wait_for_selector` or WebDriverWait, and read it.

### Can Beautiful Soup scrape dynamic content?

Not by itself: it does not run JavaScript. On our test page it found 0 quotes in the 5,808-byte HTML. Pass it the HTML of a rendered page (`page.content()` from Playwright), or parse the JSON the page carries.

### Is Playwright or Selenium better for dynamic pages?

Both render JavaScript. Playwright waits on elements automatically, takes a proxy with username and password in launch(), and found our 10 quotes in 3.34 s. Selenium fits existing Selenium code and installed browsers such as Edge or Chrome.

### Why does my scraper return empty results on a JavaScript page?

The data is added after the HTML loads. Either you used a plain HTTP client, or the browser read the page too early. In our test, networkidle returned with 0 quotes; waiting for the quote element returned all 10.

### How do I find the API behind a dynamic page?

Open developer tools, Network tab, filter Fetch/XHR, then reload, scroll or click. In Playwright, `page.expect_response()` catches it from code; on our scroll page it caught `/api/quotes?page=1` with 10 quotes.

## Render dynamic pages from a phone or home IP

Mobile and residential proxies with username and password that plug straight into Playwright, Selenium and requests.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=dynamic-web-scraping-python) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Collectors with Python examples

- [Google Maps scraper API in Python](https://quanticdata.io/collectors/google-maps-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Google Maps scraper collector, with Python examples.
- [Product price scraper API in Python](https://quanticdata.io/collectors/price-comparison-api/python/): Collector · Install, first call, async runs, output fields and errors of the Product price scraper collector, with Python examples.
- [Twitter (X) scraper API in Python](https://quanticdata.io/collectors/twitter-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Twitter (X) scraper collector, with Python examples.
- [Zalando Scraper API in Python](https://quanticdata.io/collectors/zalando-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Zalando Scraper collector, with Python examples.

## Related reading

- [Playwright Screenshot: Full Page, Element](https://quanticdata.io/blog/playwright-screenshot/): Guides · page.screenshot({ path }) captures the viewport, fullPage: true the whole page, and locator.screenshot() one element. Below: every option run with Playwright 1.61.1, with the real file sizes (a full page dropped from 625254 bytes as PNG to 170638 as JPEG at quality 50), masking, hiding elements with style, retina and phone captures, and screenshots through residential and mobile proxies.
- [Playwright: Wait for Page to Load in Python](https://quanticdata.io/blog/playwright-wait-for-page-load/): Guides · goto() waits for the load event by default, but the data you want often arrives later. On a page that adds its content after a delay, every wait_until option returned with 0 items; waiting for the element itself returned all 10. The timings, the code and when to use each wait.
- [Playwright Test Timeout: Defaults and Config](https://quanticdata.io/blog/playwright-test-timeout/): Guides · Playwright Test stops a test after 30 seconds and an assertion after 5 seconds by default; actions and navigations have their own optional limits. We reproduced each timeout error on 20 November 2026 against a deliberately slow local server, then fixed each one with the setting made for it.

---

Source: https://quanticdata.io/blog/dynamic-web-scraping-python/ · Site index for AI: https://quanticdata.io/llms.txt
