Is web crawling legal? The four layers that decide it

Web crawling is legal in most jurisdictions when you fetch publicly reachable pages at a sane rate and handle what you collect lawfully. There is no statute banning crawlers. Risk comes from four separate layers — how you access the site, what terms bind you, what content you copy, and whose personal data ends up in your store.

This is an engineering guide to those layers, not legal advice. Nothing here substitutes for a lawyer who knows your jurisdiction and your data.

Crawling and scraping are not the same legal question

Most "is web scraping legal" articles collapse two activities. Crawling is discovery and traversal: fetch a page, parse links, fetch more, repeat until a budget is hit. Scraping is extraction and reuse: pull specific fields out of a document and store them. They fail differently.

A crawler that stores nothing at all can still create liability, because its exposure is volume — bandwidth, CPU and cache pressure on someone else's servers. The Wikimedia Foundation reported in April 2025 that bandwidth for multimedia downloads had grown 50% since January 2024, driven largely by automated bots harvesting content for AI training rather than by human readers (Wikimedia, 2025). That is an infrastructure argument, and it lands even when every page crawled is public and unencumbered.

A scraper, by contrast, can hit one URL and still breach privacy or copyright law, because its exposure is the payload. So the useful question is not "is web crawling legal" in the abstract but "which of the five layers below does this job touch, and how hard".

The four layers that decide whether a crawl is legal (plus one always on)

Access: public means less exposure, not immunity

The reference point in the US remains hiQ Labs v. LinkedIn, where the Ninth Circuit held that scraping data available to the general public without authentication likely does not amount to accessing a computer "without authorization" under the CFAA (9th Cir., 2022). That ruling is narrow: it addresses one anti-hacking theory. It says nothing about copyright, privacy or server burden. And Facebook v. Power Ventures shows the other edge — continuing to access a service after being told to stop and after technical blocks were imposed can be unauthorised access.

Content rights: facts travel further than text

Copyright protects original expression, not raw facts, which is why price and availability monitoring is a comparatively settled use case and wholesale copying of editorial prose is not. In the EU, the Database Directive adds a separate right against systematic extraction of a substantial part of a protected database, even when each record is individually public. Downstream use is where this compounds: in Thomson Reuters v. Ross Intelligence (February 2025) the court rejected a fair-use defence for headnotes used to build a competing legal research tool.

Personal data: the layer that ignores "but it was public"

Under GDPR, personal data is protected regardless of where it was found. A publicly visible LinkedIn profile of an EU resident is still personal data, and you need a lawful basis — usually legitimate interest, which requires a balancing test that regulators have rejected for large-scale silent collection. In the US, most state privacy laws carve out data an individual made public, which is more permissive, but biometrics are the exception: the Clearview AI settlement, valued by the court at roughly $51.75 million and approved in May 2025, came from scraping faces without consent.

Is web crawling legal in the US, Europe and Asia?

United States

No single statute governs it. Exposure is assembled from the CFAA, copyright and the DMCA's anti-circumvention rules, state privacy laws, and common-law contract and tort claims. Public, factual, logged-out crawling for internal analysis is the lowest-risk profile available. We go deeper on the case law in is web scraping legal in the US.

Europe

Is web scraping legal in Europe? For non-personal public data, generally yes. The moment personal data enters the pipeline, GDPR applies and you need a documented lawful basis, retention limit and deletion path. Separately, the Digital Single Market Directive lets rights holders reserve text-and-data-mining rights by machine-readable means (Directive (EU) 2019/790, Art. 4) — in practice, a robots.txt or metadata opt-out aimed at AI crawlers is a legal signal in the EU, not just a courtesy. The UK mirrors most of this through UK GDPR, the Data Protection Act 2018 and the Computer Misuse Act.

Asia-Pacific

The picture diverges. China's Network Data Security Management Regulations took effect on 1 January 2025 with localisation and audit duties for major platform operators; limiting collection to public non-personal data simplifies compliance considerably. India's Digital Personal Data Protection Act carves out information a data subject voluntarily made public, which is friendlier to public-data crawlers than the EU model. Japan has moved in the most permissive direction on data use for AI training.

Commercial use, AI training and the 2025–26 shift

Is web scraping for commercial use legal? Commerciality is not the trigger by itself — price monitoring, market sizing and lead-quality research are ordinary commercial activities. What changed is the AI training question. Reddit sued Anthropic in June 2025 and Perplexity plus three data companies in October 2025; the New York Times sued OpenAI; Getty sued Stability. The US Copyright Office's Part 3 pre-publication report on generative AI training (May 2025) signals that copying expressive works to build a directly competing model is unlikely to sit comfortably inside fair use.

The practical consequence for engineers: separate your corpora. A crawl that feeds internal dashboards, retrieval over your own documents, or factual monitoring is a very different risk object from a crawl that feeds model weights. If you are building the latter, licensing is now the industry norm rather than an edge case — see is AI web scraping legal for the detail.

What good-faith crawling looks like in the config

Ethics and defensibility converge on the same settings. "Is web scraping ethical" has a boring engineering answer: be cheap to serve, honest about who you are, and minimal in what you keep.

1. Read robots.txt before the first fetch, honour the disallow rules for your user-agent, and re-read it on a schedule — sites change it without notice. Snapshot what it said on the day you crawled.
2. Set a request budget: concurrency, requests per second per host, and a hard page cap. Exponential backoff on 429 and 5xx, and stop entirely after repeated blocks.
3. Use conditional requests and caching. If nothing changed, do not refetch the body.
4. Identify yourself honestly. Never impersonate Googlebot. A contact address in the user-agent turns an abuse report into an email.
5. Stay logged out. Credentialed crawling converts a technical question into a contract question.
6. Extract only the fields you need. Drop emails, phone numbers and names with a filter at the ingestion layer if they are not part of your lawful purpose.
7. Keep an audit trail: target list, robots snapshot, terms review date, rate settings, retention and deletion schedule.
8. Have a cease-and-desist runbook — pause the crawler, preserve logs, map the complaint against the layers above, escalate to counsel.

Two calls implement most of this. Map a site first to see its real URL inventory, then crawl only the sections that matter with an explicit depth and page cap:

curl https://api.quanticdata.io/v1/map \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com" }'

# then crawl a bounded slice, not the whole domain
curl https://api.quanticdata.io/v1/crawl \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/pricing",
        "limit": 200,
        "maxDepth": 3 }'

{ "success": true,
  "data":    { "jobId": "crawl_7f2a" },
  "usage":   { "cost_usd": 0.06 } }

Poll GET /v1/crawl/:jobId for the Markdown. The same job also gives you the audit artefact: one envelope per call with cost and a per-attempt retry log, so "what did we request, when, and how often did it retry" is answerable months later. Full parameters live in the Crawl & Map API reference and the quickstart.

Honest cost math: the polite crawl is also the cheap one

Compliance advice often reads as a tax. In crawling it is the opposite, because the behaviours that get you sued are the same ones that inflate your bill.

Take a 500-page competitor site you monitor weekly. A brute-force full re-crawl is 500 pages at $0.0003 = $0.15 per run, about $7.80 a year per site — trivial in isolation, but it is also 26,000 requests a year against a server that gains nothing from them, and it is exactly the traffic pattern that shows up in an abuse report. The alternative: one map call at $0.0005 to see the current URL set, then crawl only the 40 pages whose content you actually track — $0.0005 + 40 × $0.0003 = $0.0125 per run. Twelve times cheaper, ~95% fewer requests, and a defensible story about data minimisation.

Two further mechanics matter for cost discipline. Pay-per-success means a blocked or failed fetch is billed at $0.00, so there is no financial incentive to retry aggressively into a wall. And async crawl jobs are charged on requested volume then auto-refund the unfetched share, so setting a generous cap does not punish you when robots rules or depth limits cut the job short. If you only need single pages rather than traversal, the Web Scraping API at $0.0002 per page is the smaller footprint; JS rendering at $0.001 costs five times more and should be reserved for pages that genuinely need it.

Agents make policy an infrastructure problem

When a human writes the crawler, the rate limit lives in the code. When an LLM agent decides what to fetch next, it does not — and an agent looping on a 403 will happily generate the traffic pattern your lawyer least wants to explain. Put the policy where the agent cannot route around it: allowlists, per-host caps, PII filters and page budgets in the tool layer, not the prompt.

That is why the same crawl, map, search and scrape endpoints are exposed as agent tools through the MCP server and the web data API for AI: every agent action becomes a logged, priced, individually auditable API call rather than an opaque browser session. If a rights holder asks what you fetched on 14 March, you can answer with the envelopes.

None of this is legal advice, and the AI-training questions above are actively being litigated. Treat the five layers as a triage tool, keep your logs, and get counsel before any crawl that touches personal data, credentials or model training.