How to web scrape using Python: from requests to production

Web scraping in Python means three steps: fetch a page over HTTP with requests, parse the returned HTML with BeautifulSoup or lxml, and write the extracted fields to CSV or JSON. Static pages need nothing more. JavaScript-heavy pages, rate limits and blocks are where the real engineering starts.

The minimum viable Python scraper

Install two packages and you have a working scraper. requests handles the HTTP conversation; beautifulsoup4 turns the response body into a searchable tree. The standard library alternative, urllib.request, works too and needs no install, but you will spend the difference on manual header and encoding handling.

pip install requests beautifulsoup4 lxml

Here is the whole loop against a site built for practice scraping:

import csv, time
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "research-bot/1.0 (+mailto:[email protected])"}
BASE = "https://webscraper.io/test-sites/e-commerce/static/computers/laptops"

rows = []
with requests.Session() as s:
    s.headers.update(HEADERS)
    for page in range(1, 4):
        r = s.get(BASE, params={"page": page}, timeout=20)
        r.raise_for_status()
        soup = BeautifulSoup(r.text, "lxml")

        cards = soup.select("div.card, div.thumbnail")
        if not cards:
            break  # layout changed or pagination ended

        for card in cards:
            title = card.select_one("a.title")
            price = card.select_one(".price, h4.price")
            rows.append({
                "title": title.get("title") if title else None,
                "price": price.get_text(strip=True) if price else None,
                "url": requests.compat.urljoin(BASE, title["href"]) if title else None,
            })
        time.sleep(1)  # be polite

with open("laptops.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["title", "price", "url"])
    w.writeheader()
    w.writerows(rows)

print(len(rows), "rows")

Four details in that snippet matter more than the parsing itself. A Session reuses the TCP connection and keeps cookies, which is faster and looks less robotic. timeout stops a hung socket freezing the job forever. raise_for_status() makes a 403 fail loudly instead of silently writing zero rows. And the if not cards: break guard is your canary: when a selector stops matching, you want to know on run one, not after a week of empty CSVs.

Choosing selectors that survive a redesign

Beginners usually reach for find_all("div", class_="sc-1x9bd8a-3") — a hashed CSS-module class that will change the next time the site ships. Prefer anchors that carry meaning:

1. Structured data first. Look for <script type="application/ld+json"> or a __NEXT_DATA__ blob. Product, price and availability are often already there as JSON, and JSON-LD changes far less often than markup.
2. Then the hidden API. Open DevTools, Network, filter XHR/Fetch. Many "JavaScript sites" are a thin shell over a clean JSON endpoint you can call with requests directly. This is the single biggest speed win in scraping.
3. Then semantic HTML. article h2 a, [itemprop="price"], table tbody tr td:nth-of-type(3).
4. Last, hashed classes — and if you must, assert on shape (row count, price regex) so drift trips an alarm.

Also decide early where the parsing happens. BeautifulSoup with the lxml parser is forgiving and readable. Raw lxml.html plus XPath is faster and better at positional queries like //tr[td[2][contains(., "In stock")]]. Regular expressions on raw HTML work for one-off extraction of an ID or an inline JSON string, and fall apart on nested structure — Real Python's tutorial demonstrates this precisely: a single extra space inside <title > breaks a string-slicing approach that looked fine a moment earlier (Real Python).

Which Python web scraping library for which job

Layer	Typical choice	Good for	Cost you pay
Fetching	requests / httpx	Static HTML, hidden JSON APIs, anything without JS	Nothing — fastest path
Parsing	BeautifulSoup, lxml + XPath	Turning markup into fields	Selector maintenance
Crawling at scale	Scrapy	Thousands of URLs, concurrency, retries, pipelines built in	Framework learning curve
Rendered pages	Playwright, Selenium	Client-side rendering, logins, infinite scroll	~10-50x the CPU and RAM per page
Managed fetch	Scraping API	Blocks, proxy rotation, JS rendering, Markdown output	Per-page fee instead of engineer-hours

University library guides converge on roughly the same four names — requests, BeautifulSoup, Selenium, Scrapy — plus pandas for the analysis that follows (UT Austin Libraries). That shortlist is still correct. What changed is that step one, fetching, is now the hard part on most commercial sites, not step two.

When the HTML is empty: JavaScript-rendered pages

If print(r.text) shows a shell with an empty <div id="root">, the data arrives after render. Three options, in order of cost:

1. Call the underlying JSON endpoint

Copy the XHR request from DevTools as cURL, strip it back to the minimum headers, and replay it in Python. You get typed fields, pagination cursors and no parsing at all. Check the endpoint is not behind a signed token that rotates per session before committing to it.

2. Render with a headless browser

Playwright is the modern default; Selenium remains fine and is what most tutorials show, driving Chrome or Firefox through a WebDriver. Budget realistically: a browser context is hundreds of megabytes of RAM and a second or two per page, so 100,000 pages a day becomes an infrastructure project, not a script. If you are weighing this trade-off, our primer on browser automation covers where it earns its keep.

3. Offload rendering to an API

A web scraping API renders the page server-side and hands back clean Markdown, HTML or structured JSON. Your Python stays a single requests.post — no driver versions, no Docker image with Chromium in it.

import requests, os

resp = requests.post(
    "https://api.quanticdata.io/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['QD_API_KEY']}"},
    json={"url": "https://example.com/pricing", "render_js": True},
    timeout=60,
)
payload = resp.json()
print(payload["success"], payload["usage"]["cost_usd"])
print(payload["data"]["markdown"][:400])

Every QuanticData endpoint returns the same envelope — success, data, usage — so your Python error handling is one branch: if success is false, you get a machine-readable error code and the call is billed at $0.00. Rendering is $0.001 per page versus $0.0002 for plain fetch, which is a clean lever: render only the URLs that need it.

Being blocked is a fetching problem, not a parsing problem

Nothing in BeautifulSoup fixes a 403. Once a target starts caring, you are dealing with rate limits, TLS and header fingerprinting, and IP reputation. The practical checklist:

• Slow down and jitter. Concurrency of 2-5 with randomised delays beats 50 parallel requests that get your IP nulled in ten seconds.
• Send a coherent header set. A real browser User-Agent with no Accept-Language and no Accept-Encoding is an obvious tell.
• Retry with backoff on 429/503, respect Retry-After, and never retry a 404.
• Rotate egress IPs. Datacenter ranges are fine for tolerant targets; residential proxies from $0.80/GB are what you use when the target scores IP reputation. Our guide to rotating proxies in Python has the requests and httpx wiring.
• Cache aggressively. Write raw HTML to disk on first fetch. Re-parsing a local file while you debug selectors costs nothing and hits nobody's server.

proxies = {
    "http": "http://USER:[email protected]:8000",
    "https": "http://USER:[email protected]:8000",
}
r = s.get(url, proxies=proxies, timeout=30)

Honest cost math: script versus API

Take a realistic job: 50,000 product pages a month, half of them JavaScript-rendered, on a target that blocks naive traffic.

Line item	Self-hosted Python stack	Managed fetch layer
Fetch/render cost	Proxy bandwidth + a headless-browser worker box	25,000 x $0.0002 + 25,000 x $0.001 = $30
Failed requests	Billed as bandwidth regardless	$0.00 — pay per success
Engineering	Selector fixes, driver upgrades, block triage — call it 4 hours/month	Selector work only if you use CSS extraction
Ceiling	Your worker count	Batch up to 1,000 URLs per job

The point is not that one always wins. Under a few thousand static pages a month, plain requests plus BeautifulSoup is free and takes an afternoon — build it. Once four hours a month of maintenance shows up on the invoice, $30 of per-page fees is cheaper than the engineer, and the failed-request line is where the difference gets uncomfortable: bandwidth-billed proxies charge you for blocks, pay-per-success does not. For whole sites rather than URL lists, the crawl and map API handles BFS traversal at $0.0003 per page and refunds unfetched pages.

Structure a project you will still understand in six months

1. Separate fetch, parse and store. Three functions, three responsibilities. You should be able to re-run parsing over cached HTML without a single network call.
2. Version your schema. Every row gets source_url and fetched_at. Without provenance, a dataset is unauditable.
3. Validate before writing. A pydantic model or a handful of asserts catches "price is the string 'Add to cart'" on day one.
4. Make runs idempotent. Key on the canonical URL or product ID so a re-run updates rather than duplicates.
5. Log per-URL outcomes. Status code, attempt count, bytes. Debugging blocks without this is guesswork.
6. Pin dependencies. A silent lxml or Selenium major bump is the classic "it worked yesterday".

Good starter projects that exercise all of this: a price tracker over 20 SKUs writing to SQLite, a job-board aggregator, or a link-integrity crawler for your own site. All three exist in dozens of GitHub repositories, but write yours from the spec above rather than forking — the value is in the failure handling, which is exactly what copied code omits.

Scraping from an AI agent instead of a cron job

If the consumer of your data is an LLM, the shape changes. Agents want clean Markdown, not HTML soup, and they want to decide at runtime which URLs to fetch. That maps to tools rather than scripts: a web scraping MCP server exposes search, scrape, map, crawl, batch and SEO audit as callable tools inside Claude, Cursor or VS Code, and the web data API for AI returns the same JSON envelope for RAG pipelines. Practically, that means the Python you write shrinks to orchestration — schema definition, deduplication, storage — while fetching, rendering and proxying sit behind one call. For the discovery half of the loop, the SERP API gives you search results as JSON from $0.0005 so the agent finds URLs before it scrapes them.

Legality and etiquette, briefly

Scraping publicly available data is not inherently unlawful in most jurisdictions, but terms of use, copyright, database rights and personal-data rules all apply on top, and the analysis differs by country and by what you collect. Read the target's acceptable use policy and robots.txt, avoid data behind a login unless you have permission, and never collect personal data without a lawful basis. Rate-limit so you are not degrading someone's service. We cover the layers in is web scraping legal in the US. None of this is legal advice — get counsel for anything commercially material.