You can make web scraping harder, slower and more expensive — you cannot make public pages unscrapeable. The realistic goal is to filter out casual and abusive bots, protect what actually matters (logins, personal data, expensive queries), and stop wasting effort trying to hide prices that anyone with a browser can read. Here is what each defense actually buys you.
Set the goal before you set the defenses
"Stop all scraping" is not achievable and usually not what you want — you need Googlebot, preview crawlers and legitimate partners. The workable objectives are narrower: stop abusive traffic that degrades your servers, stop credential and personal-data harvesting, and raise the cost of large-scale copying enough that it is not worth it. Anything you publish on a public URL for humans to read is, by definition, readable by software too. Defend the gates, not the open field.
Defenses that genuinely help
Rate limiting
The highest-value, lowest-cost defense. Cap requests per IP and per session over rolling windows, and the crude bots — the ones hammering thousands of requests from one address — get throttled or blocked automatically. It also protects your infrastructure, which is often the real problem behind "we're being scraped". Rate limiting will not stop a distributed collector, but it removes the majority of low-effort abuse and the accidental self-DoS from a misconfigured crawler.
Bot detection and challenges
Services like Cloudflare, DataDome and F5 fingerprint requests — TLS signature, header order, JavaScript execution, behavioral signals — and challenge the suspicious ones. This is the strongest single layer: it reliably stops scripts that do not run a real browser, and forces everyone else to pay the cost of one. As Cloudflare's own explainer concedes, though, it manages and prices scraping rather than eliminating it — a determined collector using a real browser and residential IPs can pass the same challenge a human passes.
Authentication for anything valuable
The one genuinely effective move: put your most valuable data behind a login. Authentication converts "scraping public data" (broadly legal, hard to stop) into "circumventing access controls" (both a technical barrier and, in the US, a much stronger legal position under the CFAA). If a dataset is your crown jewel, do not leave it on an open URL.
Defenses that mostly add friction
| Tactic | Stops | Reality |
|---|---|---|
| Honeypot links | Naive link-following crawlers | Real-browser agents ignore hidden links; cheap to add, catches amateurs |
| Obfuscated/rotating class names | Selector-based scrapers | Breaks brittle scrapers for one release; AI extractors read by meaning and shrug |
| Rendering content in JavaScript | Plain HTTP scrapers | Stops curl; any headless browser renders it, and it hurts your own SEO |
| Blocking datacenter IP ranges | Cheap datacenter scrapers | Determined collectors switch to residential IPs and blend in |
| User-Agent blocklists | Honest bots that identify themselves | Trivially spoofed; mostly blocks the polite scrapers, not the rude ones |
None of these are useless — layered, they raise cost. But treated as a wall, each has a one-line bypass. The uncomfortable truth every scraping vendor knows: infrastructure built to collect public data — real browser fingerprints, residential exits, human-like pacing — is designed precisely to look like the legitimate traffic your defenses must let through. We describe that collection side plainly in our legality guide; understanding it is what makes a defender realistic.
The legal layer
Terms of service, cease-and-desist letters and copyright notices are part of the toolkit, and they matter more than technical measures for the cases that actually harm you. US case law is clear that revoking authorization in writing, then blocking, strengthens a claim against someone who circumvents the block — but the same case law (hiQ v. LinkedIn) confirms that scraping genuinely public pages is not, by itself, unlawful access. Legal defenses work against identifiable, high-value adversaries; they do nothing against anonymous distributed collection. Pair them with technical measures, do not rely on either alone, and get counsel for anything you intend to enforce.
A defender's priority list
- Rate-limit everything — protects infrastructure, stops crude abuse, near-zero downside.
- Put a managed bot-detection layer in front — the best single filter for scripted traffic.
- Authenticate your valuable data — turns a losing battle into a defensible one.
- Protect personal data hardest — PII harvesting is where real legal and reputational risk lives, for you and the scraper.
- Accept that public prices are public — spend your effort where it changes an outcome, not on hiding numbers competitors can read in a browser.
The mature posture is not "stop all scraping" but "make abuse expensive, keep valuable data behind auth, and stop fighting the parts you can't win". If your actual goal is the opposite — collecting public data responsibly — that is a different guide: our AI scraping service and scraping API exist for the collection side, with the same respect-the-gates principles in reverse.