# What Is Web Crawling in Python?

> Web crawling in Python explained: the BFS frontier algorithm, crawler vs scraper, a minimal working example, Scrapy and Crawlee, and when to use a crawl API.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/What Is Web Crawling in Python?

# What is web crawling in Python? The algorithm, the code, the limits

CrawlingJul 30, 2026·5 min read·QuanticData Team

On this page [Crawling vs scraping: the distinction that unlocks everything](/blog/what-is-web-crawling-in-python/#crawling-vs-scraping-the-distinction-that-unlocks-everything) [The algorithm: a frontier, a seen-set, and politeness](/blog/what-is-web-crawling-in-python/#the-algorithm-a-frontier-a-seen-set-and-politeness) [A minimal working crawler in Python](/blog/what-is-web-crawling-in-python/#a-minimal-working-crawler-in-python) [Scrapy, Crawlee and friends: when to reach for a framework](/blog/what-is-web-crawling-in-python/#scrapy-crawlee-and-friends-when-to-reach-for-a-framework) [Where DIY crawling hits the wall](/blog/what-is-web-crawling-in-python/#where-diy-crawling-hits-the-wall) [The Python that remains](/blog/what-is-web-crawling-in-python/#the-python-that-remains)

Web crawling in Python means writing a program that starts from a seed URL, downloads the page, extracts its links, adds the new ones to a queue and repeats — walking a site breadth-first until a depth or page limit stops it. Scraping extracts data from pages; crawling is the discovery loop that finds those pages in the first place.

## Crawling vs scraping: the distinction that unlocks everything

The two words get used interchangeably, but they name different jobs. A **crawler** answers "which pages exist?" — it navigates link structure and produces URLs. A **scraper** answers "what does this page say?" — it parses one document into fields. Every real pipeline composes them: crawl to discover, scrape to extract. Keeping the roles separate is what lets you swap either half — a sitemap can replace the crawler, an [extraction API](https://quanticdata.io/web-scraping-api/) can replace the scraper — without rewriting the other.

## The algorithm: a frontier, a seen-set, and politeness

Under every crawler from a 20-line script to Googlebot sits the same structure:

1. A **frontier** — the queue of URLs waiting to be fetched, usually FIFO for breadth-first order so shallow pages come before deep ones.

2. A **seen-set** — normalized URLs already visited, so cycles and duplicate links do not trap the loop.

3. A **fetch-parse-enqueue cycle** — download, extract `<a href>` targets, normalize them (absolute URLs, stripped fragments), keep the in-scope ones, enqueue the unseen.

4. **Politeness** — respect robots.txt, rate-limit per host, and cap depth and page count so the crawl terminates on purpose rather than by accident.

## A minimal working crawler in Python

```
import urllib.parse, collections
import requests
from bs4 import BeautifulSoup

def crawl(seed, max_pages=100, max_depth=3):
    frontier = collections.deque([(seed, 0)])
    seen, out = {seed}, []
    domain = urllib.parse.urlparse(seed).netloc
    while frontier and len(out) < max_pages:
        url, depth = frontier.popleft()
        try:
            resp = requests.get(url, timeout=10,
                headers={"User-Agent": "my-crawler/0.1"})
        except requests.RequestException:
            continue
        out.append((url, resp.status_code))
        if depth >= max_depth:
            continue
        soup = BeautifulSoup(resp.text, "html.parser")
        for a in soup.select("a[href]"):
            nxt = urllib.parse.urljoin(url, a["href"]).split("#")[0]
            if urllib.parse.urlparse(nxt).netloc == domain and nxt not in seen:
                seen.add(nxt)
                frontier.append((nxt, depth + 1))
    return out
```

Forty lines, and it genuinely crawls: breadth-first, same-domain, depth-capped, cycle-safe. Everything a production crawler adds — concurrency, retries, robots.txt parsing, JavaScript rendering, storage — bolts onto this skeleton. For the extraction half and a deeper build-out, see [our step-by-step Python crawling guide](https://quanticdata.io/blog/how-to-web-crawl-python/).

## Scrapy, Crawlee and friends: when to reach for a framework

Frameworks earn their learning curve the moment you need concurrency and resilience. [Scrapy](https://scrapy.org/) is the Python veteran: asynchronous fetching, middleware for retries and throttling, item pipelines, feed exports. Crawlee brings the same idea with first-class browser automation for JavaScript-heavy sites. The trade is control for convention — you write spiders and let the framework own the loop. A framework does not, however, solve the two problems that actually kill crawls in production: getting blocked and rendering JavaScript at scale. Those need an IP strategy — typically [rotating proxies](https://quanticdata.io/rotating-proxies/) — and a browser fleet, which is infrastructure, not code.

## Where DIY crawling hits the wall

A crawler that works on 100 pages of one site meets four compounding problems at scale: anti-bot systems that throttle datacenter IPs within seconds, client-side rendering that hides links from plain HTTP fetches, queue-and-state management once a crawl outgrows one process, and the eternal maintenance tax of layouts and defenses that shift under you. This is the point where offloading the loop beats improving it. A [crawl API](https://quanticdata.io/crawl-map/) runs the whole discovery-and-fetch cycle server-side — async BFS up to 500 pages and depth 10, every page returned as clean Markdown at $0.0003 per page, with unfetched pages auto-refunded. And when you only need the URL list, a *map* call reads sitemaps plus homepage links and returns the site's structure for $0.0005 — often the smarter first move: map first, then scrape only the sections you care about.

One more scaling detail that surprises people: most production crawling is *re*-crawling. The first pass discovers the site; every pass after that exists to catch what changed. Naive re-crawls refetch everything and burn budget on unchanged pages, so mature pipelines track content hashes or last-modified signals per URL and prioritize sections that historically change often — product listings daily, documentation weekly, legal pages almost never. Designing the schedule is usually worth more than optimizing the fetcher.

## The Python that remains

Offloading the crawl does not remove Python from the picture — it moves your code up a level, from managing frontiers to consuming results:

```
import requests

job = requests.post(
    "https://api.quanticdata.io/v1/crawl",
    headers={"Authorization": f"Bearer {KEY}"},
    json={"url": "https://example.com", "limit": 200, "depth": 5},
).json()["data"]

# poll until done, then iterate pages as Markdown
status = requests.get(
    f"https://api.quanticdata.io/v1/crawl/{job['id']}",
    headers={"Authorization": f"Bearer {KEY}"},
).json()["data"]
```

The judgment call is simple: crawling as a learning exercise or a small internal tool — write the loop, it is a genuinely instructive algorithm. Crawling as production data infrastructure — buy the loop, keep your Python for the part that is actually yours: deciding what the data means. The [API quickstart](https://quanticdata.io/docs/) covers job submission, polling and the response envelope.

### Sources & further reading

- [Scrapy — an open-source web crawling framework for Python](https://scrapy.org/)

- [ScrapingBee — Python web crawler: from setup to web crawling](https://www.scrapingbee.com/blog/crawling-python/)

## FAQ

Quick answers on what is web crawling in python.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### What is the difference between web crawling and web scraping?

Crawling is discovery: following links to enumerate pages, producing URLs. Scraping is extraction: parsing a known page into structured fields. Pipelines compose them — crawl to find pages, scrape each one. Search engines crawl and index; a price monitor scrapes specific product pages it already knows.

### Is Scrapy still worth learning?

Yes, if you operate crawlers yourself: its async engine, middleware and pipelines encode a decade of hard lessons. If your goal is the data rather than the crawler, framework skills matter less than pipeline design — an API call replaces the spider and your code starts at parsed Markdown.

### What is a web crawling algorithm?

The standard one is breadth-first search over the link graph: a FIFO frontier queue, a seen-set for deduplication, and a fetch-parse-enqueue loop bounded by depth and page limits, with per-host rate limiting and robots.txt compliance. Priority variants re-order the frontier by URL importance instead of arrival order.

### Is web crawling legal?

Crawling public pages politely — honoring robots.txt, reasonable rates, no login circumvention — is generally lawful in most jurisdictions, and it is how every search engine works. Legal risk concentrates in what you do with content and personal data afterwards; we cover specifics in our web crawling legality guide.

### How do I crawl a JavaScript-heavy site in Python?

Plain requests only sees server-rendered HTML, so links assembled client-side are invisible. Options: drive a real browser via Playwright and crawl the rendered DOM, check for the site's sitemap or JSON endpoints first, or use a crawl API whose fetch layer escalates to rendering automatically when a page needs it.

## Keep the Python, skip the frontier management

Crawl whole sites to clean Markdown — async BFS to 500 pages and depth 10 at $0.0003 per page, unfetched pages refunded — or map every URL of a site for $0.0005. $2 free every month.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Website Crawler API](https://quanticdata.io/crawl-map/)

## Related reading

[Crawling How to Web Crawl in Python A working Python crawl loop in 40 lines, the same job in Scrapy, and when to swap the loop for a crawl API — with real cost math per 10,000 pages. Read →](https://quanticdata.io/blog/how-to-web-crawl-python/) [Crawling Is Web Crawling Legal? Rules by Layer Crawling public pages is generally lawful. Legality turns on four separate layers — access, contract, content rights and privacy — plus how much load you create. Read →](https://quanticdata.io/blog/is-web-crawling-legal/) [SERP & search How a SERP API Works, End to End The full path from query to JSON: localization parameters, the identity layer, HTTP vs rendered fetches, parsing rich blocks, verticals and pricing. Read →](https://quanticdata.io/blog/how-a-serp-api-works/)

---

Source: https://quanticdata.io/blog/what-is-web-crawling-in-python/ · Site index for AI: https://quanticdata.io/llms.txt