Is web scraping legal in the US? The rules, the cases, the risk lines

Yes — collecting publicly available data from US websites is generally legal, and no federal statute bans web scraping. Liability comes from how you get the data and what you do with it: bypassing logins, ignoring a revocation notice, harvesting personal data, or republishing copyrighted text. This is information, not legal advice.

The short answer, and the four questions behind it

There is no "web scraping law" in the United States. Courts decide scraping disputes by mapping the conduct onto general-purpose legal theories, so the useful question is never "is web scraping illegal?" but "which of these four doors is my pipeline walking through?"

1. Access: did you get the data the same way any browser would, or did you defeat an authentication or access control?
2. Contract: did you ever accept the site's terms — by creating an account, clicking through, or logging in?
3. Content: is what you keep factual (price, stock, address, job title) or expressive (article text, photos, reviews as prose)?
4. Subject: is any of it personal data about identifiable people?

Answer "public, no contract, factual, non-personal" and you are in the zone US courts have repeatedly treated as lawful. Change any one answer and the analysis changes with it.

What web scraping actually is in legal terms

Web scraping is automated retrieval of pages a server already serves publicly, plus parsing of those pages into structured records. Technically it is the same HTTP request a browser sends; the difference is volume and persistence. That is why the "scraping equals hacking" framing fails — a scraper does not exploit a vulnerability, it asks the same question faster.

But automation changes the legal texture in three ways: it can impose load on a server (trespass-to-chattels territory), it can aggregate small facts into a competing product (unfair competition and copyright territory), and it can accumulate personal data at a scale no human browsing session would (privacy territory). Each of those is a separate exposure with a separate control.

The main legal theories, and what actually triggers them

Theory	What triggers it	Practical control
Computer Fraud and Abuse Act (CFAA) and state analogues such as California Penal Code 502	Access after authorisation is revoked; using credentials you were not given; data behind a gate	Stop on a cease-and-desist; never rotate around a targeted block; no logged-in collection without a licence
Breach of contract (Terms of Service)	Clickwrap acceptance, account creation, or logged-in access followed by automated collection	Collect logged-off; do not hold accounts on targets you scrape; record whether terms were ever accepted
Copyright (and DMCA anti-circumvention)	Storing or republishing expressive content verbatim; defeating technical protection measures	Keep facts, not prose; short excerpts with attribution; no circumvention of DRM-style protections
Trespass to chattels / server harm	Request volume that measurably degrades the target	Concurrency caps, exponential backoff, off-peak scheduling, cache aggressively
Privacy: CCPA/CPRA, Virginia VCDPA, Colorado CPA	Collecting names, emails, IPs, profiles tied to identifiable residents	Field-level PII filters at extraction time, retention limits, deletion workflow
Unfair competition / misappropriation	A substitutive product that free-rides on the target's investment	Transformative output: analysis, comparison, aggregation — not a mirror

The US cases that set the risk lines

Craigslist v. 3taps (2013). 3taps scraped a public site, received a cease-and-desist, was IP-blocked, and then continued using fresh IPs and rotating proxies. The court held that Craigslist had validly revoked authorisation, so continued access fell under the CFAA. The lesson is not "proxies are illegal" — it is that circumventing a block aimed specifically at you, after written revocation, is the single fastest route to CFAA exposure.

Facebook v. Power Ventures (2016). Same shape: user permission gave Power an initial basis for access, a cease-and-desist removed it, and continued access after the notice was held to violate the CFAA. Facebook's claim was supported in part by more than $5,000 in employee time spent responding, which cleared the statute's loss threshold, as SerpApi's case walkthrough documents.

Van Buren v. United States (2021). The Supreme Court narrowed "exceeds authorized access" to a gates-up-or-down inquiry: it covers reaching areas of a system you are not entitled to reach, not misusing data you were entitled to see. The opinion removed a lot of the "violating the terms of use is a federal crime" theory from circulation.

hiQ Labs v. LinkedIn (9th Cir., 2022). The court upheld hiQ's injunction: scraping profiles that are publicly visible, with no login required, does not amount to access "without authorization" under the CFAA, because there is no gate to lower. Read the 2022 opinion rather than the headlines — the ruling was about the CFAA, and LinkedIn separately pursued contract claims.

Meta v. Bright Data (2024). The court found that logged-off scraping of public Facebook and Instagram pages did not breach Meta's terms, because a party that is not a logged-in user is not bound by the user agreement. This is why serious operators keep collection strictly logged-off and hold no accounts on target platforms.

Where the spectrum sits

Is web scraping legal in Europe and elsewhere?

The EU starts from a different premise. Under the GDPR, personal data is protected regardless of whether it was publicly visible, so you need a lawful basis (usually legitimate interest, documented in a balancing test), transparency, and a route for data subjects to object. The Database Directive adds a sui generis right over substantial investment in database contents, which has no direct US equivalent, and the DSM Directive's text-and-data-mining exceptions favour research bodies over commercial actors. The UK mirrors much of this through the Data Protection Act, the Copyright, Designs and Patents Act and the Computer Misuse Act. Brazil's LGPD and Canada's PIPEDA follow GDPR-style logic.

The practical consequence for a US team: the same crawler can be low-risk on US retail catalogues and high-risk on EU-facing pages containing user profiles. Scope your collectors by jurisdiction and data category, not by target site alone.

Is scraping Amazon or eBay legal? Marketplace-specific reality

Marketplaces are the most common source of "is web scraping amazon legal" and "is web scraping ebay legal" questions, and the answer is structural rather than special. Product titles, prices, availability and seller names on public listing pages are factual data on pages served without a login — the hiQ and Bright Data reasoning applies. What creates exposure is everything else: their terms prohibit automated collection (a contract issue if you hold an account), review text and product photography are copyrighted, buyer and seller contact details are personal data, and aggressive crawling triggers both technical blocks and harm arguments.

So the workable pattern is: logged-off collection only, keep the numeric and categorical fields, discard review prose and images, rate-limit per host, and use an official API where one exists for the field you need. Commercial use does not change the analysis — commercial scraping of public data is generally lawful in the US; it just attracts more scrutiny and better-funded plaintiffs.

A compliance checklist you can implement in code

1. Fetch and honour robots.txt per host, and re-check it on a schedule. It is not binding law, but ignoring it destroys any good-faith argument.
2. Never authenticate against a target unless you hold a written licence. No accounts, no cookies from human sessions.
3. Cap concurrency and add jittered backoff per domain. Log request counts per host so you can prove you were not the cause of an outage.
4. Filter PII at extraction time, not in a later cleanup job. Drop emails, phone numbers and free-text profiles unless a documented lawful basis exists.
5. Store facts, link to prose. Keep the price and the URL; keep at most a short excerpt of the copy.
6. Keep an audit trail: URL, timestamp, HTTP status, and which extraction rule produced each field. This is the artefact that answers a legal letter.
7. Route cease-and-desist letters to a human on day one, and add the sender to a hard blocklist in your crawler config. This is the single control that would have changed the outcome in 3taps and Power Ventures.

Most of this is easier when collection runs through an API that returns provenance with the payload. Our Web Scraping API returns Markdown or structured JSON in one envelope with the cost of the call, and failed calls cost nothing — which means your audit log and your invoice describe the same set of successful fetches:

curl https://api.quanticdata.io/v1/scrape \
  -H "Authorization: Bearer $QD_API_KEY" \
  -d '{
    "url": "https://example.com/product/123",
    "formats": ["markdown"],
    "onlyMainContent": true
  }'

{ "success": true,
  "data": { "markdown": "# Example Product\n…" },
  "usage": { "cost_usd": 0.0002 },
  "retries": [ { "attempt": 1, "outcome": "ok" } ] }

For discovery, keep it logged-off too: SERP results to find sellers or sources, then map a site's URLs from its sitemaps before you crawl anything, so you can review the target list before a single page is fetched.

Honest cost math for compliant collection

Compliance costs less than teams assume; what costs money is unbounded retry loops against hostile targets. At list pay-as-you-go prices:

Job	Volume	Endpoint	Unit	Total
Catalogue pages, no JS	100,000	/v1/scrape	$0.0002	$20
Same pages, JS rendered	100,000	/v1/scrape	$0.001	$100
Whole-site crawl to Markdown	5,000 pages	/v1/crawl	$0.0003	$1.50
Seller and source discovery	2,000 searches	/v1/serp	$0.0005	$1.00
Bot-view vs rendered diff	500 URLs	/v1/seo-audit	$0.0012	$0.60

Two details matter for budgeting under a pay-per-success model: a call that fails is charged $0.00, and async crawl and batch jobs are charged on requested volume with the unfetched share auto-refunded when the job settles. That removes the perverse incentive to hammer a blocked target — retrying harder does not buy you a cheaper unit cost, so the polite configuration is also the cheap one. If you need household exit IPs for geo-specific pricing pages, residential proxies start at $0.80/GB and run for the whole plan term — up to 360 days — so low-and-slow schedules do not waste prepaid capacity.

AI agents, MCP and the same old rules

Agent workflows have not created a new legal category; they have made the old one harder to supervise. When an LLM decides at runtime which URLs to fetch, your compliance boundary has to live in the tool layer rather than in a reviewed script. Practically that means: the tool refuses authenticated fetches, enforces per-host rate limits regardless of what the model asks for, applies PII filters before text reaches the model context, and logs every URL the agent touched with a timestamp.

That is how our MCP server exposes search, scrape, map, crawl, batch and SEO audit to Claude, Cursor and other clients: fixed tool surfaces with a uniform response envelope, so the agent's actions are auditable after the fact. The same pipeline sits under our AI web scraping service, where a plain-language dataset request is decomposed into search, map and scrape steps — each of which is a logged, priced, public-page fetch you can review, rather than an opaque crawl.

Is web scraping ethical? Use the test that reads well in a courtroom and in a blog post: you did not overburden the site, you took only what was publicly available, you took facts rather than someone's expression, and you built something new instead of a substitute for the source. If a step in your pipeline fails that test, no amount of infrastructure fixes it.

This article is technical information, not legal advice. Scraping disputes turn on specific facts and jurisdictions — consult qualified counsel before running a collection programme at scale.