What is web crawling in Python? The algorithm, the code, the limits

Web crawling in Python means writing a program that starts from a seed URL, downloads the page, extracts its links, adds the new ones to a queue and repeats — walking a site breadth-first until a depth or page limit stops it. Scraping extracts data from pages; crawling is the discovery loop that finds those pages in the first place.

Crawling vs scraping: the distinction that unlocks everything

The two words get used interchangeably, but they name different jobs. A crawler answers "which pages exist?" — it navigates link structure and produces URLs. A scraper answers "what does this page say?" — it parses one document into fields. Every real pipeline composes them: crawl to discover, scrape to extract. Keeping the roles separate is what lets you swap either half — a sitemap can replace the crawler, an extraction API can replace the scraper — without rewriting the other.

The algorithm: a frontier, a seen-set, and politeness

Under every crawler from a 20-line script to Googlebot sits the same structure:

1. A frontier — the queue of URLs waiting to be fetched, usually FIFO for breadth-first order so shallow pages come before deep ones.
2. A seen-set — normalized URLs already visited, so cycles and duplicate links do not trap the loop.
3. A fetch-parse-enqueue cycle — download, extract <a href> targets, normalize them (absolute URLs, stripped fragments), keep the in-scope ones, enqueue the unseen.
4. Politeness — respect robots.txt, rate-limit per host, and cap depth and page count so the crawl terminates on purpose rather than by accident.

A minimal working crawler in Python

import urllib.parse, collections
import requests
from bs4 import BeautifulSoup

def crawl(seed, max_pages=100, max_depth=3):
    frontier = collections.deque([(seed, 0)])
    seen, out = {seed}, []
    domain = urllib.parse.urlparse(seed).netloc
    while frontier and len(out) < max_pages:
        url, depth = frontier.popleft()
        try:
            resp = requests.get(url, timeout=10,
                headers={"User-Agent": "my-crawler/0.1"})
        except requests.RequestException:
            continue
        out.append((url, resp.status_code))
        if depth >= max_depth:
            continue
        soup = BeautifulSoup(resp.text, "html.parser")
        for a in soup.select("a[href]"):
            nxt = urllib.parse.urljoin(url, a["href"]).split("#")[0]
            if urllib.parse.urlparse(nxt).netloc == domain and nxt not in seen:
                seen.add(nxt)
                frontier.append((nxt, depth + 1))
    return out

Forty lines, and it genuinely crawls: breadth-first, same-domain, depth-capped, cycle-safe. Everything a production crawler adds — concurrency, retries, robots.txt parsing, JavaScript rendering, storage — bolts onto this skeleton. For the extraction half and a deeper build-out, see our step-by-step Python crawling guide.

Scrapy, Crawlee and friends: when to reach for a framework

Frameworks earn their learning curve the moment you need concurrency and resilience. Scrapy is the Python veteran: asynchronous fetching, middleware for retries and throttling, item pipelines, feed exports. Crawlee brings the same idea with first-class browser automation for JavaScript-heavy sites. The trade is control for convention — you write spiders and let the framework own the loop. A framework does not, however, solve the two problems that actually kill crawls in production: getting blocked and rendering JavaScript at scale. Those need an IP strategy — typically rotating proxies — and a browser fleet, which is infrastructure, not code.

Where DIY crawling hits the wall

A crawler that works on 100 pages of one site meets four compounding problems at scale: anti-bot systems that throttle datacenter IPs within seconds, client-side rendering that hides links from plain HTTP fetches, queue-and-state management once a crawl outgrows one process, and the eternal maintenance tax of layouts and defenses that shift under you. This is the point where offloading the loop beats improving it. A crawl API runs the whole discovery-and-fetch cycle server-side — async BFS up to 500 pages and depth 10, every page returned as clean Markdown at $0.0003 per page, with unfetched pages auto-refunded. And when you only need the URL list, a map call reads sitemaps plus homepage links and returns the site's structure for $0.0005 — often the smarter first move: map first, then scrape only the sections you care about.

One more scaling detail that surprises people: most production crawling is re-crawling. The first pass discovers the site; every pass after that exists to catch what changed. Naive re-crawls refetch everything and burn budget on unchanged pages, so mature pipelines track content hashes or last-modified signals per URL and prioritize sections that historically change often — product listings daily, documentation weekly, legal pages almost never. Designing the schedule is usually worth more than optimizing the fetcher.

The Python that remains

Offloading the crawl does not remove Python from the picture — it moves your code up a level, from managing frontiers to consuming results:

import requests

job = requests.post(
    "https://api.quanticdata.io/v1/crawl",
    headers={"Authorization": f"Bearer {KEY}"},
    json={"url": "https://example.com", "limit": 200, "depth": 5},
).json()["data"]

# poll until done, then iterate pages as Markdown
status = requests.get(
    f"https://api.quanticdata.io/v1/crawl/{job['id']}",
    headers={"Authorization": f"Bearer {KEY}"},
).json()["data"]

The judgment call is simple: crawling as a learning exercise or a small internal tool — write the loop, it is a genuinely instructive algorithm. Crawling as production data infrastructure — buy the loop, keep your Python for the part that is actually yours: deciding what the data means. The API quickstart covers job submission, polling and the response envelope.