Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Scraping dynamic web pages in Python: hidden JSON, Playwright or Selenium

Scraping a JavaScript page in Python: requests and Beautiful Soup received a 5,808-byte shell with 0 quotes, while the same quotes were in a JSON block inside a script tag and Playwright rendered all 10 in 3.34 seconds through a mobile proxy; the card ranks the three routes, hidden JSON first, then the API call, then a browser.
Scraping a JavaScript page in Python: requests and Beautiful Soup received a 5,808-byte shell with 0 quotes, while the same quotes were in a JSON block inside a script tag and Playwright rendered all 10 in 3.34 seconds through a mobile proxy; the card ranks the three routes, hidden JSON first, then the API call, then a browser.

To scrape a dynamic web page in Python, first check whether the data is already in the HTML (a JSON block inside a <script>) or comes from an API call you can request directly. If not, render the page in a real browser with Playwright or Selenium, wait for the element you need, then read it.

We tested all of it on 20 November 2026 with Python 3.12.10 on quotes.toscrape.com/js/, a public practice page whose quotes are written by JavaScript, through QuanticData mobile and residential proxies.

Why requests and Beautiful Soup see nothing

import requests
from bs4 import BeautifulSoup

PROXY = "http://USER-country-us-type-mobile-sid-d01-ttl-30m:PASS@mobile.quanticdata.io:8080"
r = requests.get("https://quotes.toscrape.com/js/", proxies={"http": PROXY, "https": PROXY}, timeout=30)
soup = BeautifulSoup(r.text, "html.parser")
print(r.status_code, len(r.content), len(soup.select("div.quote")))
# 200 5808 0

Status 200, 5,808 bytes, zero quotes. Nothing is wrong with the request: the server sent the page skeleton and a script, and the quotes appear only after a browser runs that script. Beautiful Soup does not run JavaScript.

Route 1: read the JSON inside the page

Before reaching for a browser, look at what you downloaded. Search r.text for a value you can see on screen. On this page the two script tags hold the data as a JavaScript array:

import re, json
m = re.search(r"var data = (\[.*?\]);", r.text, re.S)
quotes = json.loads(m.group(1))
print(len(quotes), sorted(quotes[0]), quotes[0]["author"]["name"])
# 10 ['author', 'tags', 'text'] Albert Einstein

Ten quotes, no browser, one request. Real sites do the same at scale: frameworks put their state in tags such as __NEXT_DATA__ or application/ld+json. When the block is valid JSON, json.loads reads it; working with the result is covered in parsing JSON in Python.

Route 2: call the API the page uses

Other pages fetch their data after loading. Open the browser's developer tools, Network tab, filter Fetch/XHR, and reload or scroll: the JSON request shows up, often with a page parameter. Playwright can find it for you. On the infinite-scroll version of the same site, we waited for the request the scroll triggered:

with page.expect_response(lambda r: "/api/quotes" in r.url and r.status == 200) as info:
    page.mouse.wheel(0, 20000)
data = info.value.json()
print(info.value.url, data["page"], len(data["quotes"]))
# https://quotes.toscrape.com/api/quotes?page=1  1  10

From then on you can call /api/quotes?page=N with requests and skip the browser. This is usually the fastest and most stable route.

Route 3: render with Playwright

When the data exists only after rendering, let a browser build the page:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    b = p.chromium.launch(proxy={"server": "http://mobile.quanticdata.io:8080",
                                 "username": "USER-country-us-type-mobile-sid-d01-ttl-30m",
                                 "password": "PASS"})
    page = b.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
    page.wait_for_selector("div.quote")
    for q in page.locator("div.quote").all():
        print(q.locator("small.author").inner_text())
    html = page.content()      # hand this to Beautiful Soup if you prefer
    b.close()

Through a US mobile line (Verizon Business, Conway, Arkansas), Chromium 156 rendered the page and found all 10 quotes in 3.34 s. Two habits matter more than anything else here:

  • Wait for the element, not for the page. On the delayed version of the page, goto(..., wait_until="networkidle") returned after 1.67 s with 0 quotes; wait_for_selector("div.quote") found all 10 after 11.24 s. The full comparison of wait strategies is in waiting for a page to load in Playwright.
  • Proxy in launch(). Playwright takes the username and password in the proxy dictionary; more in Playwright proxy setup.

Route 3b: render with Selenium

Selenium drives the browsers installed on your machine. We ran Selenium 4.53.0 with Microsoft Edge 157 in headless mode on the delayed page:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver.get("https://quotes.toscrape.com/js-delayed/")
print(len(driver.find_elements(By.CLASS_NAME, "quote")))   # 0, too early
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CLASS_NAME, "quote")))
print(len(driver.find_elements(By.CLASS_NAME, "quote")))   # 10, after 10.8 s

Edge does not take a username and password inside --proxy-server, so for Selenium we ran a small local forwarder that adds them; the method and the selectors are in Selenium find_element.

Which route to choose

RouteRequests per pageOur resultUse when
JSON in the HTML110 quotes, no browserA script tag holds the data
The page's API1 per page of data10 quotes per callData loads by XHR or on scroll
PlaywrightThe page plus its scripts and assets10 quotes in 3.34 sData exists only after rendering or clicks
SeleniumThe same10 quotes on the delayed pageYou already have Selenium code or need a specific installed browser

A browser downloads everything a page needs, so it uses more time and more proxy traffic per page than one JSON request. Render only the pages that need it. Playwright in depth: Playwright for scraping.

Proxies for dynamic pages

Many dynamic sites decide what to send based on the visitor: country, device, network. Our Playwright run used a QuanticData mobile line, which presents as a phone carrier's address; residential lines present as home connections in the country you choose. Both plug into Playwright's launch() and into requests for routes 1 and 2. Plans: mobile proxies and residential proxies. If you would rather not run browsers at all, our web scraping API renders pages for you and returns the HTML or the data.

Sources & further reading

FAQ

Quick answers on scraping dynamic web pages python.

Something else? Ask us

How do I scrape a dynamic website with Python?

Check the HTML for a JSON block in a script tag, then the Network tab for an API call. If neither has the data, render the page with Playwright or Selenium, wait for the element with wait_for_selector or WebDriverWait, and read it.

Can Beautiful Soup scrape dynamic content?

Not by itself: it does not run JavaScript. On our test page it found 0 quotes in the 5,808-byte HTML. Pass it the HTML of a rendered page (page.content() from Playwright), or parse the JSON the page carries.

Is Playwright or Selenium better for dynamic pages?

Both render JavaScript. Playwright waits on elements automatically, takes a proxy with username and password in launch(), and found our 10 quotes in 3.34 s. Selenium fits existing Selenium code and installed browsers such as Edge or Chrome.

Why does my scraper return empty results on a JavaScript page?

The data is added after the HTML loads. Either you used a plain HTTP client, or the browser read the page too early. In our test, networkidle returned with 0 quotes; waiting for the quote element returned all 10.

How do I find the API behind a dynamic page?

Open developer tools, Network tab, filter Fetch/XHR, then reload, scroll or click. In Playwright, page.expect_response() catches it from code; on our scroll page it caught /api/quotes?page=1 with 10 quotes.

Render dynamic pages from a phone or home IP

Mobile and residential proxies with username and password that plug straight into Playwright, Selenium and requests.

Collectors with Python examples

Related reading