# How to Web Scrape Using Python

> Learn how to web scraping using Python with requests, BeautifulSoup, lxml and Selenium — plus honest cost math and when to swap your parser for an API.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/How to Web Scrape Using Python

# How to web scrape using Python: from requests to production

Web scrapingJul 29, 2026·9 min read·QuanticData Team

On this page [The minimum viable Python scraper](/blog/how-to-web-scraping-using-python/#the-minimum-viable-python-scraper) [Choosing selectors that survive a redesign](/blog/how-to-web-scraping-using-python/#choosing-selectors-that-survive-a-redesign) [Which Python web scraping library for which job](/blog/how-to-web-scraping-using-python/#which-python-web-scraping-library-for-which-job) [When the HTML is empty: JavaScript-rendered pages](/blog/how-to-web-scraping-using-python/#when-the-html-is-empty-javascript-rendered-pages) [Being blocked is a fetching problem, not a parsing problem](/blog/how-to-web-scraping-using-python/#being-blocked-is-a-fetching-problem-not-a-parsing-problem) [Honest cost math: script versus API](/blog/how-to-web-scraping-using-python/#honest-cost-math-script-versus-api) [Structure a project you will still understand in six months](/blog/how-to-web-scraping-using-python/#structure-a-project-you-will-still-understand-in-six-months) [Scraping from an AI agent instead of a cron job](/blog/how-to-web-scraping-using-python/#scraping-from-an-ai-agent-instead-of-a-cron-job) [Legality and etiquette, briefly](/blog/how-to-web-scraping-using-python/#legality-and-etiquette-briefly)

Web scraping in Python means three steps: fetch a page over HTTP with `requests`, parse the returned HTML with BeautifulSoup or lxml, and write the extracted fields to CSV or JSON. Static pages need nothing more. JavaScript-heavy pages, rate limits and blocks are where the real engineering starts.

## The minimum viable Python scraper

Install two packages and you have a working scraper. `requests` handles the HTTP conversation; `beautifulsoup4` turns the response body into a searchable tree. The standard library alternative, `urllib.request`, works too and needs no install, but you will spend the difference on manual header and encoding handling.

```
pip install requests beautifulsoup4 lxml
```

Here is the whole loop against a site built for practice scraping:

```
import csv, time
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "research-bot/1.0 (+mailto:you@example.com)"}
BASE = "https://webscraper.io/test-sites/e-commerce/static/computers/laptops"

rows = []
with requests.Session() as s:
    s.headers.update(HEADERS)
    for page in range(1, 4):
        r = s.get(BASE, params={"page": page}, timeout=20)
        r.raise_for_status()
        soup = BeautifulSoup(r.text, "lxml")

        cards = soup.select("div.card, div.thumbnail")
        if not cards:
            break  # layout changed or pagination ended

        for card in cards:
            title = card.select_one("a.title")
            price = card.select_one(".price, h4.price")
            rows.append({
                "title": title.get("title") if title else None,
                "price": price.get_text(strip=True) if price else None,
                "url": requests.compat.urljoin(BASE, title["href"]) if title else None,
            })
        time.sleep(1)  # be polite

with open("laptops.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["title", "price", "url"])
    w.writeheader()
    w.writerows(rows)

print(len(rows), "rows")
```

Four details in that snippet matter more than the parsing itself. A `Session` reuses the TCP connection and keeps cookies, which is faster and looks less robotic. `timeout` stops a hung socket freezing the job forever. `raise_for_status()` makes a 403 fail loudly instead of silently writing zero rows. And the `if not cards: break` guard is your canary: when a selector stops matching, you want to know on run one, not after a week of empty CSVs.

## Choosing selectors that survive a redesign

Beginners usually reach for `find_all("div", class_="sc-1x9bd8a-3")` — a hashed CSS-module class that will change the next time the site ships. Prefer anchors that carry meaning:

1. **Structured data first.** Look for `<script type="application/ld+json">` or a `__NEXT_DATA__` blob. Product, price and availability are often already there as JSON, and JSON-LD changes far less often than markup.

2. **Then the hidden API.** Open DevTools, Network, filter XHR/Fetch. Many "JavaScript sites" are a thin shell over a clean JSON endpoint you can call with `requests` directly. This is the single biggest speed win in scraping.

3. **Then semantic HTML.** `article h2 a`, `[itemprop="price"]`, `table tbody tr td:nth-of-type(3)`.

4. **Last, hashed classes** — and if you must, assert on shape (row count, price regex) so drift trips an alarm.

Also decide early where the parsing happens. BeautifulSoup with the `lxml` parser is forgiving and readable. Raw `lxml.html` plus XPath is faster and better at positional queries like `//tr[td[2][contains(., "In stock")]]`. Regular expressions on raw HTML work for one-off extraction of an ID or an inline JSON string, and fall apart on nested structure — Real Python's tutorial demonstrates this precisely: a single extra space inside `<title >` breaks a string-slicing approach that looked fine a moment earlier ([Real Python](https://realpython.com/python-web-scraping-practical-introduction/)).

## Which Python web scraping library for which job

| Layer | Typical choice | Good for | Cost you pay |
| --- | --- | --- | --- |
| Fetching | `requests` / `httpx` | Static HTML, hidden JSON APIs, anything without JS | Nothing — fastest path |
| Parsing | BeautifulSoup, `lxml` + XPath | Turning markup into fields | Selector maintenance |
| Crawling at scale | Scrapy | Thousands of URLs, concurrency, retries, pipelines built in | Framework learning curve |
| Rendered pages | Playwright, Selenium | Client-side rendering, logins, infinite scroll | ~10-50x the CPU and RAM per page |
| Managed fetch | Scraping API | Blocks, proxy rotation, JS rendering, Markdown output | Per-page fee instead of engineer-hours |

University library guides converge on roughly the same four names — requests, BeautifulSoup, Selenium, Scrapy — plus pandas for the analysis that follows ([UT Austin Libraries](https://guides.lib.utexas.edu/web-scrapping/scraping-with-python)). That shortlist is still correct. What changed is that step one, fetching, is now the hard part on most commercial sites, not step two.

## When the HTML is empty: JavaScript-rendered pages

If `print(r.text)` shows a shell with an empty `<div id="root">`, the data arrives after render. Three options, in order of cost:

### 1. Call the underlying JSON endpoint

Copy the XHR request from DevTools as cURL, strip it back to the minimum headers, and replay it in Python. You get typed fields, pagination cursors and no parsing at all. Check the endpoint is not behind a signed token that rotates per session before committing to it.

### 2. Render with a headless browser

Playwright is the modern default; Selenium remains fine and is what most tutorials show, driving Chrome or Firefox through a WebDriver. Budget realistically: a browser context is hundreds of megabytes of RAM and a second or two per page, so 100,000 pages a day becomes an infrastructure project, not a script. If you are weighing this trade-off, our primer on [browser automation](https://quanticdata.io/blog/what-is-browser-automation/) covers where it earns its keep.

### 3. Offload rendering to an API

A [web scraping API](https://quanticdata.io/web-scraping-api/) renders the page server-side and hands back clean Markdown, HTML or structured JSON. Your Python stays a single `requests.post` — no driver versions, no Docker image with Chromium in it.

```
import requests, os

resp = requests.post(
    "https://api.quanticdata.io/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['QD_API_KEY']}"},
    json={"url": "https://example.com/pricing", "render_js": True},
    timeout=60,
)
payload = resp.json()
print(payload["success"], payload["usage"]["cost_usd"])
print(payload["data"]["markdown"][:400])
```

Every QuanticData endpoint returns the same envelope — `success`, `data`, `usage` — so your Python error handling is one branch: if `success` is false, you get a machine-readable error code and the call is billed at $0.00. Rendering is $0.001 per page versus $0.0002 for plain fetch, which is a clean lever: render only the URLs that need it.

## Being blocked is a fetching problem, not a parsing problem

Nothing in BeautifulSoup fixes a 403. Once a target starts caring, you are dealing with rate limits, TLS and header fingerprinting, and IP reputation. The practical checklist:

- **Slow down and jitter.** Concurrency of 2-5 with randomised delays beats 50 parallel requests that get your IP nulled in ten seconds.

- **Send a coherent header set.** A real browser User-Agent with no `Accept-Language` and no `Accept-Encoding` is an obvious tell.

- **Retry with backoff on 429/503**, respect `Retry-After`, and never retry a 404.

- **Rotate egress IPs.** Datacenter ranges are fine for tolerant targets; [residential proxies](https://quanticdata.io/residential-proxies/) from $0.80/GB are what you use when the target scores IP reputation. Our guide to [rotating proxies in Python](https://quanticdata.io/blog/how-to-rotate-proxies-python/) has the `requests` and `httpx` wiring.

- **Cache aggressively.** Write raw HTML to disk on first fetch. Re-parsing a local file while you debug selectors costs nothing and hits nobody's server.

```
proxies = {
    "http": "http://USER:PASS@gate.example:8000",
    "https": "http://USER:PASS@gate.example:8000",
}
r = s.get(url, proxies=proxies, timeout=30)
```

## Honest cost math: script versus API

Take a realistic job: 50,000 product pages a month, half of them JavaScript-rendered, on a target that blocks naive traffic.

| Line item | Self-hosted Python stack | Managed fetch layer |
| --- | --- | --- |
| Fetch/render cost | Proxy bandwidth + a headless-browser worker box | 25,000 x $0.0002 + 25,000 x $0.001 = $30 |
| Failed requests | Billed as bandwidth regardless | $0.00 — pay per success |
| Engineering | Selector fixes, driver upgrades, block triage — call it 4 hours/month | Selector work only if you use CSS extraction |
| Ceiling | Your worker count | Batch up to 1,000 URLs per job |

The point is not that one always wins. Under a few thousand static pages a month, plain `requests` plus BeautifulSoup is free and takes an afternoon — build it. Once four hours a month of maintenance shows up on the invoice, $30 of per-page fees is cheaper than the engineer, and the failed-request line is where the difference gets uncomfortable: bandwidth-billed proxies charge you for blocks, pay-per-success does not. For whole sites rather than URL lists, the [crawl and map API](https://quanticdata.io/crawl-map/) handles BFS traversal at $0.0003 per page and refunds unfetched pages.

## Structure a project you will still understand in six months

1. **Separate fetch, parse and store.** Three functions, three responsibilities. You should be able to re-run parsing over cached HTML without a single network call.

2. **Version your schema.** Every row gets `source_url` and `fetched_at`. Without provenance, a dataset is unauditable.

3. **Validate before writing.** A `pydantic` model or a handful of asserts catches "price is the string 'Add to cart'" on day one.

4. **Make runs idempotent.** Key on the canonical URL or product ID so a re-run updates rather than duplicates.

5. **Log per-URL outcomes.** Status code, attempt count, bytes. Debugging blocks without this is guesswork.

6. **Pin dependencies.** A silent `lxml` or Selenium major bump is the classic "it worked yesterday".

Good starter projects that exercise all of this: a price tracker over 20 SKUs writing to SQLite, a job-board aggregator, or a link-integrity crawler for your own site. All three exist in dozens of GitHub repositories, but write yours from the spec above rather than forking — the value is in the failure handling, which is exactly what copied code omits.

## Scraping from an AI agent instead of a cron job

If the consumer of your data is an LLM, the shape changes. Agents want clean Markdown, not HTML soup, and they want to decide at runtime which URLs to fetch. That maps to tools rather than scripts: a [web scraping MCP server](https://quanticdata.io/mcp-server/) exposes search, scrape, map, crawl, batch and SEO audit as callable tools inside Claude, Cursor or VS Code, and the [web data API for AI](https://quanticdata.io/web-data-api-for-ai/) returns the same JSON envelope for RAG pipelines. Practically, that means the Python you write shrinks to orchestration — schema definition, deduplication, storage — while fetching, rendering and proxying sit behind one call. For the discovery half of the loop, the [SERP API](https://quanticdata.io/serp-api/) gives you search results as JSON from $0.0005 so the agent finds URLs before it scrapes them.

## Legality and etiquette, briefly

Scraping publicly available data is not inherently unlawful in most jurisdictions, but terms of use, copyright, database rights and personal-data rules all apply on top, and the analysis differs by country and by what you collect. Read the target's acceptable use policy and `robots.txt`, avoid data behind a login unless you have permission, and never collect personal data without a lawful basis. Rate-limit so you are not degrading someone's service. We cover the layers in [is web scraping legal in the US](https://quanticdata.io/blog/is-web-scraping-legal-in-us/). None of this is legal advice — get counsel for anything commercially material.

### Sources & further reading

- [A Practical Introduction to Web Scraping in Python — Real Python](https://realpython.com/python-web-scraping-practical-introduction/)

- [Web Scraping with Python — UT Austin Libraries LibGuides](https://guides.lib.utexas.edu/web-scrapping/scraping-with-python)

- [Beautiful Soup 4 documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)

- [Scrapy — open source web crawling framework](https://scrapy.org/)

## FAQ

Quick answers on how to web scraping using python.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### How do I start web scraping with Python as a beginner?

Install `requests` and `beautifulsoup4`, pick a site built for practice such as webscraper.io's test store, and write ten lines: GET the page, parse it with BeautifulSoup, select one element, print it. Then add a second field, then pagination, then CSV output. Do not start with Selenium or Scrapy.

### Is BeautifulSoup or Scrapy better for web scraping in Python?

BeautifulSoup is a parser, Scrapy is a full crawling framework — they solve different problems. Use requests plus BeautifulSoup for a handful of pages or a single site. Move to Scrapy when you need concurrency, automatic retries, deduplication and item pipelines across thousands of URLs, and are willing to learn its architecture.

### Can Python scrape JavaScript-rendered websites?

Yes, three ways. Find the JSON endpoint the page calls and hit it with `requests` — fastest and cheapest. Or drive a real browser with Playwright or Selenium, paying roughly a second and hundreds of megabytes of RAM per page. Or offload rendering to a scraping API and keep your Python to a single POST request.

### Why does my Python scraper get blocked or return 403?

Almost always fetching, not parsing: too many requests per second, an incoherent header set, or an IP with poor reputation. Add jittered delays, send a full browser-like header set, retry 429 and 503 with backoff, and rotate egress IPs — residential exits when the target scores IP reputation. Cache raw HTML so debugging costs no extra requests.

### How much does Python web scraping cost at scale?

The script is free; fetching is not. Self-hosting means proxy bandwidth, headless-browser compute and ongoing maintenance for selector drift and blocks. A managed fetch layer prices it per page — from $0.0002, or $0.001 with JS rendering — with failed calls charged at $0.00. Below a few thousand static pages a month, DIY usually wins.

### Do I need proxies for web scraping with Python?

Not for small volumes on tolerant sites. You need them when a target rate-limits by IP, geo-restricts content, or blocks datacenter ranges. Start with datacenter proxies for tolerant targets and move to residential IPs when reputation scoring kicks in. Rotate per request for breadth, use sticky sessions when a flow needs the same IP.

## Keep the Python, drop the fetching problem

Point your scraper at one endpoint and get clean Markdown, HTML or structured JSON from $0.0002 per page — $0.001 with JS rendering — with failed calls billed at $0.00. Start with $2 of free usage every month, no card required.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

[Web scraping What Is a Web Scraper API? One endpoint, one URL in, structured data out. How web scraper APIs work under the hood, what they cost per page, and how to judge one. Read →](https://quanticdata.io/blog/what-is-a-web-scraper-api/) [Web scraping Does Web Scraping Use an API? Web scraping and APIs are not opposites. Most modern scrapers hit a JSON endpoint, a public API, or a third-party scraping API — here is how to pick. Read →](https://quanticdata.io/blog/does-web-scraping-use-api/) [SEO data How to Perform an SEO Audit A practical six-step SEO audit process with a checklist, the crawler-vs-user diff most audits skip, and how to run the whole thing programmatically. Read →](https://quanticdata.io/blog/how-to-perform-an-seo-audit/)

---

Source: https://quanticdata.io/blog/how-to-web-scraping-using-python/ · Site index for AI: https://quanticdata.io/llms.txt