# Is Web Crawling Legal? Rules by Layer

> Is web crawling legal? Usually yes for public pages — but access method, terms, copyright, privacy and server load each carry separate risk. Rules, cases, config.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/Is Web Crawling Legal? Rules by Layer

# Is web crawling legal? The four layers that decide it

CrawlingJul 29, 2026·10 min read·QuanticData Team

On this page [Crawling and scraping are not the same legal question](/blog/is-web-crawling-legal/#crawling-and-scraping-are-not-the-same-legal-question) [The four layers that decide whether a crawl is legal (plus one always on)](/blog/is-web-crawling-legal/#the-four-layers-that-decide-whether-a-crawl-is-legal-plus-on) [Is web crawling legal in the US, Europe and Asia?](/blog/is-web-crawling-legal/#is-web-crawling-legal-in-the-us-europe-and-asia) [Commercial use, AI training and the 2025–26 shift](/blog/is-web-crawling-legal/#commercial-use-ai-training-and-the-2025-26-shift) [What good-faith crawling looks like in the config](/blog/is-web-crawling-legal/#what-good-faith-crawling-looks-like-in-the-config) [Honest cost math: the polite crawl is also the cheap one](/blog/is-web-crawling-legal/#honest-cost-math-the-polite-crawl-is-also-the-cheap-one) [Agents make policy an infrastructure problem](/blog/is-web-crawling-legal/#agents-make-policy-an-infrastructure-problem)

Web crawling is legal in most jurisdictions when you fetch publicly reachable pages at a sane rate and handle what you collect lawfully. There is no statute banning crawlers. Risk comes from four separate layers — how you access the site, what terms bind you, what content you copy, and whose personal data ends up in your store.

This is an engineering guide to those layers, not legal advice. Nothing here substitutes for a lawyer who knows your jurisdiction and your data.

## Crawling and scraping are not the same legal question

Most "is web scraping legal" articles collapse two activities. **Crawling** is discovery and traversal: fetch a page, parse links, fetch more, repeat until a budget is hit. **Scraping** is extraction and reuse: pull specific fields out of a document and store them. They fail differently.

A crawler that stores nothing at all can still create liability, because its exposure is volume — bandwidth, CPU and cache pressure on someone else's servers. The Wikimedia Foundation reported in April 2025 that bandwidth for multimedia downloads had grown 50% since January 2024, driven largely by automated bots harvesting content for AI training rather than by human readers ([Wikimedia, 2025](https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/)). That is an infrastructure argument, and it lands even when every page crawled is public and unencumbered.

A scraper, by contrast, can hit one URL and still breach privacy or copyright law, because its exposure is the payload. So the useful question is not "is web crawling legal" in the abstract but "which of the five layers below does this job touch, and how hard".

## The four layers that decide whether a crawl is legal (plus one always on)

| Layer | What it turns on | Typical hook | Lower risk | Higher risk |
| --- | --- | --- | --- | --- |
| Access | Were you authorised to be there? | CFAA (US), Computer Misuse Act (UK) | Logged-out pages, no blocks bypassed | Credentials, paywalls, CAPTCHA circumvention, crawling after an IP block |
| Contract | Did you agree to terms? | Breach of contract | No account, no click-through, terms unreviewed but unnotified | Signed-up account, explicit notice, cease-and-desist ignored |
| Content rights | Is the payload expression or fact? | Copyright, DMCA §1201, EU Database Directive | Prices, SKUs, dates, counts | Articles, photos, reviews, systematic extraction of a whole database |
| Personal data | Does a row identify a human? | GDPR, UK GDPR, CCPA/CPRA, BIPA | No PII collected, or PII dropped at ingestion | Names, emails, photos, biometrics — public visibility is not a defence in the EU |
| Load | Did you degrade the service? | Trespass to chattels, tortious interference | Throttled, cached, conditional requests | Unbounded concurrency, no backoff, repeated full-site refetches |

### Access: public means less exposure, not immunity

The reference point in the US remains *hiQ Labs v. LinkedIn*, where the Ninth Circuit held that scraping data available to the general public without authentication likely does not amount to accessing a computer "without authorization" under the CFAA ([9th Cir., 2022](https://law.justia.com/cases/federal/appellate-courts/ca9/17-16783/17-16783-2022-04-18.html)). That ruling is narrow: it addresses one anti-hacking theory. It says nothing about copyright, privacy or server burden. And *Facebook v. Power Ventures* shows the other edge — continuing to access a service after being told to stop and after technical blocks were imposed can be unauthorised access.

### Content rights: facts travel further than text

Copyright protects original expression, not raw facts, which is why price and availability monitoring is a comparatively settled use case and wholesale copying of editorial prose is not. In the EU, the Database Directive adds a separate right against systematic extraction of a substantial part of a protected database, even when each record is individually public. Downstream use is where this compounds: in *Thomson Reuters v. Ross Intelligence* (February 2025) the court rejected a fair-use defence for headnotes used to build a competing legal research tool.

### Personal data: the layer that ignores "but it was public"

Under GDPR, personal data is protected regardless of where it was found. A publicly visible LinkedIn profile of an EU resident is still personal data, and you need a lawful basis — usually legitimate interest, which requires a balancing test that regulators have rejected for large-scale silent collection. In the US, most state privacy laws carve out data an individual made public, which is more permissive, but biometrics are the exception: the Clearview AI settlement, valued by the court at roughly $51.75 million and approved in May 2025, came from scraping faces without consent.

## Is web crawling legal in the US, Europe and Asia?

### United States

No single statute governs it. Exposure is assembled from the CFAA, copyright and the DMCA's anti-circumvention rules, state privacy laws, and common-law contract and tort claims. Public, factual, logged-out crawling for internal analysis is the lowest-risk profile available. We go deeper on the case law in [is web scraping legal in the US](https://quanticdata.io/blog/is-web-scraping-legal-in-us/).

### Europe

Is web scraping legal in Europe? For non-personal public data, generally yes. The moment personal data enters the pipeline, GDPR applies and you need a documented lawful basis, retention limit and deletion path. Separately, the Digital Single Market Directive lets rights holders reserve text-and-data-mining rights by machine-readable means ([Directive (EU) 2019/790, Art. 4](https://eur-lex.europa.eu/eli/dir/2019/790/oj/eng)) — in practice, a `robots.txt` or metadata opt-out aimed at AI crawlers is a legal signal in the EU, not just a courtesy. The UK mirrors most of this through UK GDPR, the Data Protection Act 2018 and the Computer Misuse Act.

### Asia-Pacific

The picture diverges. China's Network Data Security Management Regulations took effect on 1 January 2025 with localisation and audit duties for major platform operators; limiting collection to public non-personal data simplifies compliance considerably. India's Digital Personal Data Protection Act carves out information a data subject voluntarily made public, which is friendlier to public-data crawlers than the EU model. Japan has moved in the most permissive direction on data use for AI training.

## Commercial use, AI training and the 2025–26 shift

Is web scraping for commercial use legal? Commerciality is not the trigger by itself — price monitoring, market sizing and lead-quality research are ordinary commercial activities. What changed is the AI training question. Reddit sued Anthropic in June 2025 and Perplexity plus three data companies in October 2025; the New York Times sued OpenAI; Getty sued Stability. The US Copyright Office's Part 3 pre-publication report on generative AI training ([May 2025](https://www.copyright.gov/ai/)) signals that copying expressive works to build a directly competing model is unlikely to sit comfortably inside fair use.

The practical consequence for engineers: separate your corpora. A crawl that feeds internal dashboards, retrieval over your own documents, or factual monitoring is a very different risk object from a crawl that feeds model weights. If you are building the latter, licensing is now the industry norm rather than an edge case — see [is AI web scraping legal](https://quanticdata.io/blog/is-ai-web-scraping-legal/) for the detail.

## What good-faith crawling looks like in the config

Ethics and defensibility converge on the same settings. "Is web scraping ethical" has a boring engineering answer: be cheap to serve, honest about who you are, and minimal in what you keep.

1. Read `robots.txt` before the first fetch, honour the disallow rules for your user-agent, and re-read it on a schedule — sites change it without notice. Snapshot what it said on the day you crawled.

2. Set a request budget: concurrency, requests per second per host, and a hard page cap. Exponential backoff on 429 and 5xx, and stop entirely after repeated blocks.

3. Use conditional requests and caching. If nothing changed, do not refetch the body.

4. Identify yourself honestly. Never impersonate Googlebot. A contact address in the user-agent turns an abuse report into an email.

5. Stay logged out. Credentialed crawling converts a technical question into a contract question.

6. Extract only the fields you need. Drop emails, phone numbers and names with a filter at the ingestion layer if they are not part of your lawful purpose.

7. Keep an audit trail: target list, robots snapshot, terms review date, rate settings, retention and deletion schedule.

8. Have a cease-and-desist runbook — pause the crawler, preserve logs, map the complaint against the layers above, escalate to counsel.

Two calls implement most of this. Map a site first to see its real URL inventory, then crawl only the sections that matter with an explicit depth and page cap:

```
curl https://api.quanticdata.io/v1/map \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com" }'

# then crawl a bounded slice, not the whole domain
curl https://api.quanticdata.io/v1/crawl \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/pricing",
        "limit": 200,
        "maxDepth": 3 }'

{ "success": true,
  "data":    { "jobId": "crawl_7f2a" },
  "usage":   { "cost_usd": 0.06 } }
```

Poll `GET /v1/crawl/:jobId` for the Markdown. The same job also gives you the audit artefact: one envelope per call with cost and a per-attempt retry log, so "what did we request, when, and how often did it retry" is answerable months later. Full parameters live in the [Crawl & Map API](https://quanticdata.io/crawl-map/) reference and the [quickstart](https://quanticdata.io/docs/).

## Honest cost math: the polite crawl is also the cheap one

Compliance advice often reads as a tax. In crawling it is the opposite, because the behaviours that get you sued are the same ones that inflate your bill.

Take a 500-page competitor site you monitor weekly. A brute-force full re-crawl is 500 pages at $0.0003 = $0.15 per run, about $7.80 a year per site — trivial in isolation, but it is also 26,000 requests a year against a server that gains nothing from them, and it is exactly the traffic pattern that shows up in an abuse report. The alternative: one map call at $0.0005 to see the current URL set, then crawl only the 40 pages whose content you actually track — $0.0005 + 40 × $0.0003 = $0.0125 per run. Twelve times cheaper, ~95% fewer requests, and a defensible story about data minimisation.

Two further mechanics matter for cost discipline. Pay-per-success means a blocked or failed fetch is billed at $0.00, so there is no financial incentive to retry aggressively into a wall. And async crawl jobs are charged on requested volume then auto-refund the unfetched share, so setting a generous cap does not punish you when robots rules or depth limits cut the job short. If you only need single pages rather than traversal, the [Web Scraping API](https://quanticdata.io/web-scraping-api/) at $0.0002 per page is the smaller footprint; JS rendering at $0.001 costs five times more and should be reserved for pages that genuinely need it.

## Agents make policy an infrastructure problem

When a human writes the crawler, the rate limit lives in the code. When an LLM agent decides what to fetch next, it does not — and an agent looping on a 403 will happily generate the traffic pattern your lawyer least wants to explain. Put the policy where the agent cannot route around it: allowlists, per-host caps, PII filters and page budgets in the tool layer, not the prompt.

That is why the same crawl, map, search and scrape endpoints are exposed as agent tools through the [MCP server](https://quanticdata.io/mcp-server/) and the [web data API for AI](https://quanticdata.io/web-data-api-for-ai/): every agent action becomes a logged, priced, individually auditable API call rather than an opaque browser session. If a rights holder asks what you fetched on 14 March, you can answer with the envelopes.

None of this is legal advice, and the AI-training questions above are actively being litigated. Treat the five layers as a triage tool, keep your logs, and get counsel before any crawl that touches personal data, credentials or model training.

### Sources & further reading

- [How crawlers impact the operations of the Wikimedia projects](https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/)

- [hiQ Labs, Inc. v. LinkedIn Corp., 9th Cir. (2022)](https://law.justia.com/cases/federal/appellate-courts/ca9/17-16783/17-16783-2022-04-18.html)

- [U.S. Copyright Office — Copyright and Artificial Intelligence (Part 3, Generative AI Training)](https://www.copyright.gov/ai/)

- [Directive (EU) 2019/790 on copyright in the Digital Single Market](https://eur-lex.europa.eu/eli/dir/2019/790/oj/eng)

- [Regulation (EU) 2016/679 (GDPR)](https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=celex%3A32016R0679)

- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)

## FAQ

Quick answers on is web crawling legal.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### Is web scraping legal in Europe?

Crawling public, non-personal data in the EU is generally lawful. Personal data is the dividing line: GDPR applies even to publicly visible information, so you need a documented lawful basis, retention limit and deletion path. The Database Directive also restricts systematic extraction of protected databases, and DSM Article 4 lets rights holders reserve text-and-data-mining rights in machine-readable form.

### Is web scraping for commercial use legal?

Commercial purpose alone is not illegal — price monitoring, market research and competitor tracking are ordinary business activities built on public facts. Risk rises with what you do next: reselling raw datasets, republishing expressive content, or training a model that competes directly with the source. Internal analytics sits at the low end; competing products sit at the high end.

### Is web scraping Amazon or eBay legal?

Both prohibit automated collection in their terms, so scraping them is primarily a contract and enforcement question rather than a criminal one. Logged-out collection of public factual data such as price and availability is the lowest-risk pattern; using an account, bypassing bot defences, or copying reviews and images escalates it. Expect blocking regardless.

### Is web scraping ethical?

It can be. The working test: do not overburden the target, stay out of authenticated areas, prefer factual data over expression, respect robots.txt and machine-readable opt-outs, and build something transformative rather than a substitute for the source. Ethical crawling and cheap crawling point at the same settings — fewer requests, smaller payloads.

### What is the difference between web crawling and web scraping?

Crawling is discovery: fetch a page, parse its links, fetch more, until a budget is reached. Scraping is extraction: pull defined fields out of a document and store them. Crawling risk is mostly volume and server load; scraping risk is mostly the payload — copyright and personal data. Most real jobs do both.

### Does violating a website's terms of service make crawling illegal?

Not automatically. Terms are a contract, not a statute, and courts generally require clear notice before enforcing them. But a terms breach supports civil claims and is strong evidence of bad faith, especially combined with bypassed technical controls or continued crawling after a cease-and-desist. Ignoring a legal notice is the expensive mistake.

## Crawl less, log everything

Map a site for $0.0005, then crawl only the pages you need from $0.0003 each — with unfetched pages refunded and failed calls billed at $0.00. Every request returns a priced envelope with a retry log, so your crawl has an audit trail by default. $2 free every month, no card.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Website Crawler API](https://quanticdata.io/crawl-map/)

## Related reading

[Crawling How to Web Crawl in Python A working Python crawl loop in 40 lines, the same job in Scrapy, and when to swap the loop for a crawl API — with real cost math per 10,000 pages. Read →](https://quanticdata.io/blog/how-to-web-crawl-python/) [Use cases How to Price Watch on Amazon Three ways to price watch on Amazon — native price history and alerts, third-party trackers, or your own API watcher — with honest cost math for each. Read →](https://quanticdata.io/blog/how-to-price-watch-on-amazon/) [SEO data How to Perform an SEO Audit A practical six-step SEO audit process with a checklist, the crawler-vs-user diff most audits skip, and how to run the whole thing programmatically. Read →](https://quanticdata.io/blog/how-to-perform-an-seo-audit/)

---

Source: https://quanticdata.io/blog/is-web-crawling-legal/ · Site index for AI: https://quanticdata.io/llms.txt