# How to Web Crawl in Python

> How to web crawl with Python: a working requests + BeautifulSoup crawler, a Scrapy spider, library trade-offs, block handling and honest cost math per 10k pages.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/How to Web Crawl in Python

# How to web crawl with Python: from a 40-line loop to production

CrawlingJul 29, 2026·10 min read·QuanticData Team

On this page [Crawling and scraping are different jobs](/blog/how-to-web-crawl-python/#crawling-and-scraping-are-different-jobs) [A minimal Python web crawler example](/blog/how-to-web-crawl-python/#a-minimal-python-web-crawler-example) [Choosing a Python web crawler library](/blog/how-to-web-crawl-python/#choosing-a-python-web-crawler-library) [The same crawl as a Scrapy spider](/blog/how-to-web-crawl-python/#the-same-crawl-as-a-scrapy-spider) [Where Python crawlers actually break](/blog/how-to-web-crawl-python/#where-python-crawlers-actually-break) [Skipping the loop: crawling through an API](/blog/how-to-web-crawl-python/#skipping-the-loop-crawling-through-an-api) [Honest cost math for 10,000 pages](/blog/how-to-web-crawl-python/#honest-cost-math-for-10-000-pages) [Crawling from an agent](/blog/how-to-web-crawl-python/#crawling-from-an-agent) [Crawl politely, and check the rules](/blog/how-to-web-crawl-python/#crawl-politely-and-check-the-rules)

To web crawl in Python you need four things: a queue of URLs to visit, a set of URLs already seen, a fetcher, and a link extractor. Forty lines of `requests` plus `BeautifulSoup` gets you a working crawler; Scrapy or a crawl API takes it to thousands of pages without babysitting.

## Crawling and scraping are different jobs

Crawling is *discovery*: given one or more seed URLs, find every page worth looking at by following links. Scraping is *extraction*: turn one page's HTML into structured fields. Almost every real project does both, but they fail differently. A broken scraper returns empty fields; a broken crawler either misses 80% of the site or hammers the same three pages until you get blocked.

Everything below assumes the crawl side is what you are trying to solve. Keep the two layers separate in your code — one function returns HTML, another returns links, a third returns data — and you can swap the fetcher for a proxy, a headless browser or an API later without rewriting the loop.

## A minimal Python web crawler example

This is a breadth-first crawler with the five invariants that matter: a frontier queue, a de-duplication set, a depth cap, a same-host filter and a delay between requests. It respects `robots.txt` using the standard library's `urllib.robotparser`, so you do not need an extra dependency for that.

```
import time
from collections import deque
from urllib.parse import urljoin, urlparse, urldefrag
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

UA = "my-crawler/0.1 (+https://example.com/bot)"
START = "https://quotes.toscrape.com/"
MAX_PAGES, MAX_DEPTH, DELAY = 200, 3, 1.0

root = urlparse(START)
robots = RobotFileParser()
robots.set_url(f"{root.scheme}://{root.netloc}/robots.txt")
robots.read()

session = requests.Session()
session.headers["User-Agent"] = UA

frontier = deque([(START, 0)])
seen = {START}
rows = []

while frontier and len(rows) < MAX_PAGES:
    url, depth = frontier.popleft()
    if not robots.can_fetch(UA, url):
        continue
    try:
        resp = session.get(url, timeout=15)
        resp.raise_for_status()
    except requests.RequestException as exc:
        print("skip", url, exc)
        continue
    if "text/html" not in resp.headers.get("content-type", ""):
        continue

    soup = BeautifulSoup(resp.text, "html.parser")
    rows.append({
        "url": url,
        "title": soup.title.get_text(strip=True) if soup.title else "",
    })

    if depth < MAX_DEPTH:
        for a in soup.select("a[href]"):
            link, _ = urldefrag(urljoin(url, a["href"]))
            if urlparse(link).netloc == root.netloc and link not in seen:
                seen.add(link)
                frontier.append((link, depth + 1))

    time.sleep(DELAY)

print(len(rows), "pages crawled,", len(seen), "urls discovered")
```

### Why each line is there

1. **deque, not recursion.** Recursive crawling blows the stack and gives you depth-first order, which tends to tunnel into pagination instead of covering the site.

2. **Add to `seen` at enqueue time**, not after fetching. Otherwise the same URL enters the queue a dozen times before the first fetch completes.

3. **`urldefrag` and `urljoin`.** `/page/2`, `page/2` and `#top` variants are the top source of duplicate work. Normalise before comparing.

4. **Content-type check.** Without it you will feed 40 MB PDFs into an HTML parser.

5. **Depth and page caps.** Every crawl needs a stop condition that does not depend on the site being sane. Calendar widgets and faceted search generate infinite URL spaces.

Add persistence when the crawl outgrows one process: write `seen` and `frontier` to SQLite or Redis so a crash does not restart the whole job.

## Choosing a Python web crawler library

Pick on the shape of the job, not on popularity. These are the criteria that actually change your architecture.

| Criterion | requests + BeautifulSoup | Scrapy | Crawlee / browser automation |
| --- | --- | --- | --- |
| Concurrency model | You write it (threads, asyncio) | Built-in async scheduler, per-domain limits | Async pool, browser contexts |
| Retries, throttling, dedupe | Manual | Included (AutoThrottle, dupe filter) | Included, plus session rotation |
| JavaScript-rendered pages | No | Only with a rendering add-on | Native (Playwright/Chromium) |
| Cost per page at scale | Low CPU, low RAM | Low CPU, low RAM | High: ~0.3-1 CPU core and hundreds of MB per browser |
| Best fit | Under ~1,000 static pages, one-off scripts | Repeatable pipelines, 10k+ pages, static or API-backed HTML | Login walls, infinite scroll, client-rendered SPAs |

Scrapy is the default answer for structured, repeatable crawling in Python — it is an open-source framework built around exactly the loop above, with the queue, dupe filter, retry middleware and export pipeline already written ([Scrapy docs](https://docs.scrapy.org/en/latest/)). Crawlee for Python covers the browser-heavy end and is actively maintained on [GitHub](https://github.com/apify/crawlee-python), where it lists 9.4k stars at the time of writing — useful if you want a Python web crawler on GitHub to read rather than a blog snippet. University reading lists still recommend the same four building blocks: requests, BeautifulSoup, Selenium and Scrapy ([UT Austin Libraries](https://guides.lib.utexas.edu/web-scrapping/scraping-with-python)).

## The same crawl as a Scrapy spider

Scrapy replaces about 60 lines of plumbing with settings. This spider crawls a host to depth 3, obeys robots.txt, throttles itself and writes JSON Lines:

```
import scrapy

class SiteSpider(scrapy.Spider):
    name = "site"
    start_urls = ["https://quotes.toscrape.com/"]
    custom_settings = {
        "DEPTH_LIMIT": 3,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 4,
        "DOWNLOAD_DELAY": 0.5,
        "AUTOTHROTTLE_ENABLED": True,
        "ROBOTSTXT_OBEY": True,
        "HTTPCACHE_ENABLED": True,
    }

    def parse(self, response):
        yield {"url": response.url, "title": response.css("title::text").get()}
        yield from response.follow_all(css="a[href]", callback=self.parse)

# scrapy crawl site -O pages.jsonl
```

`HTTPCACHE_ENABLED` is the underrated one: while you are iterating on selectors, cached responses mean you develop against disk instead of re-hitting the target site. `follow_all` handles relative URLs and the dupe filter handles `seen`.

## Where Python crawlers actually break

The loop is easy. Three things break it in production.

### 1. Blocks and rate limits

A single datacenter IP crawling a few hundred pages per minute gets 403s, CAPTCHAs or silent HTML swaps. The fix is not a longer sleep: it is IP diversity plus honest concurrency limits. Route the fetcher through [rotating proxies](https://quanticdata.io/rotating-proxies/) so each request leaves from a different exit, or use [residential proxies](https://quanticdata.io/residential-proxies/) when the target treats datacenter ranges as hostile. Keep the retry policy explicit: exponential backoff, a cap on attempts, and a log line per attempt so "why was this slow" has an answer.

### 2. JavaScript-rendered content

If `requests` returns a shell with an empty `<div id="root">`, the data arrives via XHR. Check the network tab first — a JSON endpoint you can call directly is faster and cheaper than any browser. Only reach for rendering when the payload is genuinely client-assembled. If you need to know which parts of a page exist only after JS runs, the [SEO audit API](https://quanticdata.io/seo-audit/) fetches a URL twice — pure HTTP and fully rendered — and diffs the two views.

### 3. URL explosion

Faceted navigation multiplies: 5 filters with 10 values each is 100,000 URLs that all render the same 200 products. Before crawling, map the site. Sitemaps plus homepage links usually reveal the real page count and the sections that matter, which lets you crawl 800 useful URLs instead of 80,000 near-duplicates.

## Skipping the loop: crawling through an API

If the crawl is a means to an end — a RAG corpus, a price sweep, a competitor's docs — running your own fetch layer means owning proxies, retries, rendering and parsing forever. A crawl endpoint collapses that into one call. QuanticData's [Crawl & Map API](https://quanticdata.io/crawl-map/) runs the same breadth-first walk server-side and returns Markdown per page, with residential exits and retries underneath:

```
curl -X POST https://api.quanticdata.io/v1/crawl \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com", "limit": 200, "max_depth": 3 }'
# { "success": true, "data": { "jobId": "..." }, "usage": { "cost_usd": 0.06 } }
```

Crawl is async, so you poll for the result — the same envelope comes back on every endpoint:

```
import os, time, requests

AUTH = {"Authorization": f"Bearer {os.environ['QD_API_KEY']}"}
BASE = "https://api.quanticdata.io/v1"

job = requests.post(f"{BASE}/crawl", headers=AUTH, json={
    "url": "https://example.com", "limit": 200, "max_depth": 3,
}).json()

job_id = job["data"]["jobId"]
while True:
    res = requests.get(f"{BASE}/crawl/{job_id}", headers=AUTH).json()
    if res["data"]["status"] in ("completed", "failed"):
        break
    time.sleep(5)

for page in res["data"].get("pages", []):
    print(page["url"], len(page["markdown"]))
```

Two patterns are worth knowing. `POST /v1/map` lists every URL of a site from sitemaps plus homepage links with per-section totals for $0.0005 — run it first to size the job. Then feed the URLs you actually want to `POST /v1/batch` (up to 1,000 known URLs at $0.0002 each) instead of crawling blindly. Full parameter lists live in the [API docs](https://quanticdata.io/docs/), and single-page extraction with CSS or AI selectors sits in the [Web Scraping API](https://quanticdata.io/web-scraping-api/). For background on what a hosted fetch layer replaces, see [what a web scraper API is](https://quanticdata.io/blog/what-is-a-web-scraper-api/).

## Honest cost math for 10,000 pages

Numbers below use QuanticData list prices and an assumption of ~0.5 MB of HTML per page — check your own average, image-heavy pages are far larger.

| Approach | Direct cost for 10,000 pages | What you still own |
| --- | --- | --- |
| Scrapy + residential proxies ($0.80/GB) | ~5 GB = $4.00, plus ~20% retry overhead ≈ $4.80 | Scheduler, retries, parsing, monitoring, selector rot |
| Crawl API ($0.0003/page) | $3.00, unfetched pages refunded | Deciding the seed and the budget |
| Map + batch ($0.0005 + $0.0002/URL) | $2.0005 for 10,000 targeted URLs | Choosing which URLs matter |
| Rendered pages ($0.001/page) | $10.00, only for pages that need JS | Knowing which pages need rendering |

Raw bandwidth on a self-hosted crawler is genuinely cheap. What is not cheap is the week you spend on the block-handling middleware and the afternoon every month when a layout change breaks it. The failure math also differs: pay-per-success means a call that returns `success: false` costs $0.00, whereas a proxy-metered crawl pays for every blocked response it downloaded. Cross the two and the sensible split is usually: self-host the crawls you run continuously and understand well, and pay per page for the long tail of sites you touch once.

## Crawling from an agent

If the consumer of the data is an LLM, the crawl loop belongs behind a tool call rather than in a script the model writes from scratch. The [MCP server](https://quanticdata.io/mcp-server/) exposes search, scrape, map, crawl, crawl_status, batch, batch_status and seo_audit as tools to Claude Code, Cursor and other MCP clients, so an agent can map a site, pick 30 URLs and batch-fetch them without you shipping any crawler code. The same endpoints are available as a plain [web data API for AI](https://quanticdata.io/web-data-api-for-ai/) when you want the pipeline inside your own Python service.

## Crawl politely, and check the rules

Three habits keep you out of trouble and out of block lists. Send a real `User-Agent` with a contact URL. Honour `robots.txt` — it is a standardised protocol ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)), and ignoring it is the fastest way to turn a technical question into a legal one. Cap concurrency per host rather than globally; four in-flight requests per domain is a reasonable default for most sites, lower for small ones.

Legality depends on what you collect, where the data subjects are and what the site's terms say — public data, personal data and logged-in content are not the same risk profile. Our overview of [web scraping legality in the US](https://quanticdata.io/blog/is-web-scraping-legal-in-us/) covers the main cases. This article is engineering guidance, not legal advice; get counsel for anything commercial or personal-data adjacent.

### Sources & further reading

- [Scrapy documentation](https://docs.scrapy.org/en/latest/)

- [Crawlee for Python (apify/crawlee-python) on GitHub](https://github.com/apify/crawlee-python)

- [Web Scraping with Python — University of Texas at Austin Libraries](https://guides.lib.utexas.edu/web-scrapping/scraping-with-python)

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)

- [Building a Web Crawler in Python — Zyte](https://www.zyte.com/learn/building-a-web-crawler-in-python/)

- [Python web crawler: from setup to web crawling — ScrapingBee](https://www.scrapingbee.com/blog/crawling-python/)

## FAQ

Quick answers on how to web crawl python.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### How do you write a web crawler in Python?

Keep a queue of URLs to visit and a set of URLs already seen. Pop a URL, fetch it with `requests`, parse it with BeautifulSoup, extract absolute links with `urljoin`, push unseen same-host links back onto the queue with depth+1, and stop at a page or depth cap. That is about 40 lines.

### Should I use Scrapy or BeautifulSoup for crawling?

BeautifulSoup is a parser, not a crawler — pair it with requests for one-off jobs under roughly a thousand static pages. Scrapy is a full framework: it ships the scheduler, duplicate filter, retries, AutoThrottle and export pipelines, so it wins for repeatable crawls at 10k+ pages. Neither renders JavaScript on its own.

### How do I crawl JavaScript-rendered pages in Python?

First check whether the page loads its data from a JSON endpoint you can call directly — that is faster and cheaper than any browser. If the HTML is genuinely assembled client-side, use a headless browser library or an API with JS rendering, and only render the pages that need it, since rendering costs an order of magnitude more per page.

### Where can I find a Python web crawler example on GitHub?

Scrapy's own repository and its example spiders are the canonical reference, and Crawlee for Python (apify/crawlee-python, 9.4k stars at the time of writing) is a readable async crawler with browser support. Reading a maintained framework's scheduler teaches more than copying a single-file gist.

### How fast can I crawl without getting blocked?

There is no universal number. Cap concurrency per host — four in-flight requests with a short delay is a safe default — obey any `Crawl-delay` in robots.txt, back off exponentially on 429 and 403, and spread requests across rotating IPs if you need more throughput. Sudden bursts from one IP trigger blocks far faster than sustained polite rates.

### How much does it cost to crawl 10,000 pages?

On self-hosted Scrapy with residential proxies at $0.80/GB and roughly 0.5 MB per page, bandwidth is about $4 plus retry overhead — engineering time dominates. Through a crawl API at $0.0003 per page it is $3.00 with unfetched pages refunded, or about $2 if you map the site first and batch only the URLs you want.

## Crawl a whole site without writing the fetch layer

Point the Crawl & Map API at a domain and get every page back as clean Markdown from $0.0003 per page, with unfetched pages refunded and failed calls charged at $0.00. $2 of free usage every month, no card required.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Website Crawler API](https://quanticdata.io/crawl-map/)

## Related reading

[Crawling Is Web Crawling Legal? Rules by Layer Crawling public pages is generally lawful. Legality turns on four separate layers — access, contract, content rights and privacy — plus how much load you create. Read →](https://quanticdata.io/blog/is-web-crawling-legal/) [Use cases How to Price Watch on Amazon Three ways to price watch on Amazon — native price history and alerts, third-party trackers, or your own API watcher — with honest cost math for each. Read →](https://quanticdata.io/blog/how-to-price-watch-on-amazon/) [SEO data How to Perform an SEO Audit A practical six-step SEO audit process with a checklist, the crawler-vs-user diff most audits skip, and how to run the whole thing programmatically. Read →](https://quanticdata.io/blog/how-to-perform-an-seo-audit/)

---

Source: https://quanticdata.io/blog/how-to-web-crawl-python/ · Site index for AI: https://quanticdata.io/llms.txt