How to web crawl with Python: from a 40-line loop to production

To web crawl in Python you need four things: a queue of URLs to visit, a set of URLs already seen, a fetcher, and a link extractor. Forty lines of requests plus BeautifulSoup gets you a working crawler; Scrapy or a crawl API takes it to thousands of pages without babysitting.

Crawling and scraping are different jobs

Crawling is discovery: given one or more seed URLs, find every page worth looking at by following links. Scraping is extraction: turn one page's HTML into structured fields. Almost every real project does both, but they fail differently. A broken scraper returns empty fields; a broken crawler either misses 80% of the site or hammers the same three pages until you get blocked.

Everything below assumes the crawl side is what you are trying to solve. Keep the two layers separate in your code — one function returns HTML, another returns links, a third returns data — and you can swap the fetcher for a proxy, a headless browser or an API later without rewriting the loop.

A minimal Python web crawler example

This is a breadth-first crawler with the five invariants that matter: a frontier queue, a de-duplication set, a depth cap, a same-host filter and a delay between requests. It respects robots.txt using the standard library's urllib.robotparser, so you do not need an extra dependency for that.

import time
from collections import deque
from urllib.parse import urljoin, urlparse, urldefrag
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

UA = "my-crawler/0.1 (+https://example.com/bot)"
START = "https://quotes.toscrape.com/"
MAX_PAGES, MAX_DEPTH, DELAY = 200, 3, 1.0

root = urlparse(START)
robots = RobotFileParser()
robots.set_url(f"{root.scheme}://{root.netloc}/robots.txt")
robots.read()

session = requests.Session()
session.headers["User-Agent"] = UA

frontier = deque([(START, 0)])
seen = {START}
rows = []

while frontier and len(rows) < MAX_PAGES:
    url, depth = frontier.popleft()
    if not robots.can_fetch(UA, url):
        continue
    try:
        resp = session.get(url, timeout=15)
        resp.raise_for_status()
    except requests.RequestException as exc:
        print("skip", url, exc)
        continue
    if "text/html" not in resp.headers.get("content-type", ""):
        continue

    soup = BeautifulSoup(resp.text, "html.parser")
    rows.append({
        "url": url,
        "title": soup.title.get_text(strip=True) if soup.title else "",
    })

    if depth < MAX_DEPTH:
        for a in soup.select("a[href]"):
            link, _ = urldefrag(urljoin(url, a["href"]))
            if urlparse(link).netloc == root.netloc and link not in seen:
                seen.add(link)
                frontier.append((link, depth + 1))

    time.sleep(DELAY)

print(len(rows), "pages crawled,", len(seen), "urls discovered")

Why each line is there

1. deque, not recursion. Recursive crawling blows the stack and gives you depth-first order, which tends to tunnel into pagination instead of covering the site.
2. Add to seen at enqueue time, not after fetching. Otherwise the same URL enters the queue a dozen times before the first fetch completes.
3. urldefrag and urljoin. /page/2, page/2 and #top variants are the top source of duplicate work. Normalise before comparing.
4. Content-type check. Without it you will feed 40 MB PDFs into an HTML parser.
5. Depth and page caps. Every crawl needs a stop condition that does not depend on the site being sane. Calendar widgets and faceted search generate infinite URL spaces.

Add persistence when the crawl outgrows one process: write seen and frontier to SQLite or Redis so a crash does not restart the whole job.

Choosing a Python web crawler library

Pick on the shape of the job, not on popularity. These are the criteria that actually change your architecture.

Scrapy is the default answer for structured, repeatable crawling in Python — it is an open-source framework built around exactly the loop above, with the queue, dupe filter, retry middleware and export pipeline already written (Scrapy docs). Crawlee for Python covers the browser-heavy end and is actively maintained on GitHub, where it lists 9.4k stars at the time of writing — useful if you want a Python web crawler on GitHub to read rather than a blog snippet. University reading lists still recommend the same four building blocks: requests, BeautifulSoup, Selenium and Scrapy (UT Austin Libraries).

The same crawl as a Scrapy spider

Scrapy replaces about 60 lines of plumbing with settings. This spider crawls a host to depth 3, obeys robots.txt, throttles itself and writes JSON Lines:

import scrapy

class SiteSpider(scrapy.Spider):
    name = "site"
    start_urls = ["https://quotes.toscrape.com/"]
    custom_settings = {
        "DEPTH_LIMIT": 3,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 4,
        "DOWNLOAD_DELAY": 0.5,
        "AUTOTHROTTLE_ENABLED": True,
        "ROBOTSTXT_OBEY": True,
        "HTTPCACHE_ENABLED": True,
    }

    def parse(self, response):
        yield {"url": response.url, "title": response.css("title::text").get()}
        yield from response.follow_all(css="a[href]", callback=self.parse)

# scrapy crawl site -O pages.jsonl

HTTPCACHE_ENABLED is the underrated one: while you are iterating on selectors, cached responses mean you develop against disk instead of re-hitting the target site. follow_all handles relative URLs and the dupe filter handles seen.

Where Python crawlers actually break

The loop is easy. Three things break it in production.

1. Blocks and rate limits

A single datacenter IP crawling a few hundred pages per minute gets 403s, CAPTCHAs or silent HTML swaps. The fix is not a longer sleep: it is IP diversity plus honest concurrency limits. Route the fetcher through rotating proxies so each request leaves from a different exit, or use residential proxies when the target treats datacenter ranges as hostile. Keep the retry policy explicit: exponential backoff, a cap on attempts, and a log line per attempt so "why was this slow" has an answer.

2. JavaScript-rendered content

If requests returns a shell with an empty <div id="root">, the data arrives via XHR. Check the network tab first — a JSON endpoint you can call directly is faster and cheaper than any browser. Only reach for rendering when the payload is genuinely client-assembled. If you need to know which parts of a page exist only after JS runs, the SEO audit API fetches a URL twice — pure HTTP and fully rendered — and diffs the two views.

3. URL explosion

Faceted navigation multiplies: 5 filters with 10 values each is 100,000 URLs that all render the same 200 products. Before crawling, map the site. Sitemaps plus homepage links usually reveal the real page count and the sections that matter, which lets you crawl 800 useful URLs instead of 80,000 near-duplicates.

Skipping the loop: crawling through an API

If the crawl is a means to an end — a RAG corpus, a price sweep, a competitor's docs — running your own fetch layer means owning proxies, retries, rendering and parsing forever. A crawl endpoint collapses that into one call. QuanticData's Crawl & Map API runs the same breadth-first walk server-side and returns Markdown per page, with residential exits and retries underneath:

curl -X POST https://api.quanticdata.io/v1/crawl \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com", "limit": 200, "max_depth": 3 }'
# { "success": true, "data": { "jobId": "..." }, "usage": { "cost_usd": 0.06 } }

Crawl is async, so you poll for the result — the same envelope comes back on every endpoint:

import os, time, requests

AUTH = {"Authorization": f"Bearer {os.environ['QD_API_KEY']}"}
BASE = "https://api.quanticdata.io/v1"

job = requests.post(f"{BASE}/crawl", headers=AUTH, json={
    "url": "https://example.com", "limit": 200, "max_depth": 3,
}).json()

job_id = job["data"]["jobId"]
while True:
    res = requests.get(f"{BASE}/crawl/{job_id}", headers=AUTH).json()
    if res["data"]["status"] in ("completed", "failed"):
        break
    time.sleep(5)

for page in res["data"].get("pages", []):
    print(page["url"], len(page["markdown"]))

Two patterns are worth knowing. POST /v1/map lists every URL of a site from sitemaps plus homepage links with per-section totals for $0.0005 — run it first to size the job. Then feed the URLs you actually want to POST /v1/batch (up to 1,000 known URLs at $0.0002 each) instead of crawling blindly. Full parameter lists live in the API docs, and single-page extraction with CSS or AI selectors sits in the Web Scraping API. For background on what a hosted fetch layer replaces, see what a web scraper API is.

Honest cost math for 10,000 pages

Numbers below use QuanticData list prices and an assumption of ~0.5 MB of HTML per page — check your own average, image-heavy pages are far larger.

Raw bandwidth on a self-hosted crawler is genuinely cheap. What is not cheap is the week you spend on the block-handling middleware and the afternoon every month when a layout change breaks it. The failure math also differs: pay-per-success means a call that returns success: false costs $0.00, whereas a proxy-metered crawl pays for every blocked response it downloaded. Cross the two and the sensible split is usually: self-host the crawls you run continuously and understand well, and pay per page for the long tail of sites you touch once.

Crawling from an agent

If the consumer of the data is an LLM, the crawl loop belongs behind a tool call rather than in a script the model writes from scratch. The MCP server exposes search, scrape, map, crawl, crawl_status, batch, batch_status and seo_audit as tools to Claude Code, Cursor and other MCP clients, so an agent can map a site, pick 30 URLs and batch-fetch them without you shipping any crawler code. The same endpoints are available as a plain web data API for AI when you want the pipeline inside your own Python service.

Crawl politely, and check the rules

Three habits keep you out of trouble and out of block lists. Send a real User-Agent with a contact URL. Honour robots.txt — it is a standardised protocol (RFC 9309), and ignoring it is the fastest way to turn a technical question into a legal one. Cap concurrency per host rather than globally; four in-flight requests per domain is a reasonable default for most sites, lower for small ones.

Legality depends on what you collect, where the data subjects are and what the site's terms say — public data, personal data and logged-in content are not the same risk profile. Our overview of web scraping legality in the US covers the main cases. This article is engineering guidance, not legal advice; get counsel for anything commercial or personal-data adjacent.