# Web Scraping CAPTCHA: How Not to Get One

> Web scraping CAPTCHA guide: which captcha is which (reCAPTCHA, hCaptcha, Turnstile and more), why a scraper is served one, and how to get the page without it.

[Home](https://quanticdata.io/) / [Web Unlocker](https://quanticdata.io/web-unlocker/) / CAPTCHA when scraping

Web Unlocker · CAPTCHAs

# CAPTCHA when web scraping: why you get one, and how not to

A CAPTCHA is what a site serves when it does not trust a request. A scraper gets one for its address, client, pace or missing session, so the way through is not to be served one. The Web Unlocker sends each request from a residential or mobile IP with a browser-grade client, and retries on a fresh exit.

[Start free](https://app.quanticdata.io/register?next=%2Funlock&utm_source=web-unlocker&utm_medium=website&utm_campaign=unlocker-guides&utm_content=captcha-when-scraping-start-1) · [The block page dictionary](https://quanticdata.io/web-unlocker/block-pages/)

- **88.9%**: calls answered with the real page across 27 protected targets, canary of 11 November 2026
- **81**: calls, 3 runs, default settings
- **4.1 s**: median response across the run

- [Which captcha is which](https://quanticdata.io/web-unlocker/captcha-when-scraping/#seen)
- [If you are a visitor](https://quanticdata.io/web-unlocker/captcha-when-scraping/#visitor)
- [If your script gets it](https://quanticdata.io/web-unlocker/captcha-when-scraping/#script)
- [Get the page](https://quanticdata.io/web-unlocker/captcha-when-scraping/#call)
- [Measured](https://quanticdata.io/web-unlocker/captcha-when-scraping/#measured)
- [Limits](https://quanticdata.io/web-unlocker/captcha-when-scraping/#limits)
- [FAQ](https://quanticdata.io/web-unlocker/captcha-when-scraping/#faq)

## Which captcha is which

These are the checks a scraper meets most often. Some are widgets a site adds to a form; others belong to a protection system in front of the whole site. They are triggered by different things.

| CAPTCHA | What it looks like | Who shows it | What triggers it | Explained on |
| --- | --- | --- | --- | --- |
| reCAPTCHA v2 | A box labelled "I'm not a robot". Its invisible form has no box and runs when a button on the page is clicked. | Google, on a form or a page where the site put it. | Google's judgement of the visitor. A tick passes at once or is followed by a challenge; in the invisible form, by default only the most suspicious traffic is challenged. | This page |
| reCAPTCHA v3 | Nothing. It never interrupts the visitor. | Google, on sites that score every interaction. | It returns a score for each request, and the site decides what to do with a low one: ask for more proof, moderate, or throttle. | This page |
| hCaptcha | A box to tick inside a form, then a task such as picking the images that match. An invisible mode shows no box. | hCaptcha, where the site put the widget. | The confidence that the visitor is human and the difficulty the site chose. Failing one brings another. | This page |
| Cloudflare Turnstile | "Verify you are human" with a box, a spinner, or nothing at all. No images or text to decipher. | Cloudflare, inside its challenge pages and as a widget on any site's forms. | What it reads in the browser: a tick is asked only when a further check is needed. | [Cloudflare Turnstile](https://quanticdata.io/web-unlocker/cloudflare-turnstile/) |
| DataDome slider | A slider puzzle to drag, behind the sentence "Please enable JS and disable any ad blocker". | DataDome, in front of the whole site. | A request it scored as automated, when its device check alone was not enough to decide. | ["Please enable JS and disable any ad blocker"](https://quanticdata.io/web-unlocker/please-enable-js-and-disable-any-ad-blocker/) |
| AWS WAF puzzle | "Let's confirm you are human", a puzzle with an audio option, served with status `405`. | AWS WAF, on a rule the owner of the site set to CAPTCHA. | A request that matched that rule without a valid `aws-waf-token`. | [AWS WAF guide](https://quanticdata.io/web-unlocker/bypass-aws-waf/) |
| Press & Hold | One button: "Press & Hold to confirm you are a human". | HUMAN Security, formerly PerimeterX, in front of the whole site. | A session with a low score. The hold is a behaviour test, not a puzzle. | [Press and hold CAPTCHA](https://quanticdata.io/web-unlocker/press-and-hold-captcha/) |

![Why a scraper is served a CAPTCHA: first the four things a site judges in a request, the address, the client, the pace and the session, then a page that holds either a captcha box or the content, last a card that says what comes back: with little trust a captcha or a refusal, with enough trust the page with status 200, and the blocked and blockClass fields of the answer.](https://quanticdata.io/web-unlocker/captcha-when-scraping/diagram.webp?v=20261111b1)

*What a site judges in a request, and what it serves: a captcha, or the page.*

## If you are a visitor

In your own browser a captcha is the same judgement, made about you: something in the visit looked unusual. Solve it once and it normally goes away. When it returns on every page, the cause is nearly always one of a few.

- Turn off the VPN or proxy. A shared address is the most common reason for repeated captchas.
- Allow cookies for the site. The pass is stored in one, and without it every page asks again.
- Pause extensions that block scripts, and use an up-to-date browser: a check that cannot run cannot pass.
- Slow down. Many tabs or fast reloads look like a machine.
- If a puzzle is hard to see or to use, look for its audio or accessibility option, or ask the site for another way in.

## If your script gets it

A site does not show a captcha at random. It is the answer to doubt, and four things raise that doubt for a script. Lower them and the captcha is not served in the first place.

- The address: requests from hosting ranges, or from an address that already sent a lot, are doubted first. A residential or mobile address starts from the trust of an ordinary visitor.
- The client: an HTTP library does not connect like the browser it names, and an automated browser leaves marks. A client that is coherent with a real browser removes the most common trigger.
- The pace: many requests in a short time from one identity look like a machine, whatever sent them. Fewer requests per address, and pauses between them, keep the count under the line.
- The session: a visitor who passed a check carries a cookie that says so. A script that throws its cookies away is a stranger on every request, and is tested every time.
- What does not help: a new `User-Agent` alone, or a token or a cookie taken from another machine.

Each system has its guide, and the [block page dictionary](https://quanticdata.io/web-unlocker/block-pages/) tells them apart: [Cloudflare](https://quanticdata.io/web-unlocker/bypass-cloudflare/), [DataDome](https://quanticdata.io/web-unlocker/bypass-datadome/), [Akamai Bot Manager](https://quanticdata.io/web-unlocker/bypass-akamai/), [PerimeterX (HUMAN)](https://quanticdata.io/web-unlocker/bypass-perimeterx/), [Kasada](https://quanticdata.io/web-unlocker/bypass-kasada/), [Imperva (Incapsula)](https://quanticdata.io/web-unlocker/bypass-imperva/), [F5 BIG-IP and Shape](https://quanticdata.io/web-unlocker/bypass-f5/), [AWS WAF](https://quanticdata.io/web-unlocker/bypass-aws-waf/).

## Get the page with one call

Send the URL to the REST endpoint. The request leaves from a residential or mobile exit with a browser-grade client. When the site answers with a check that needs no person, a real browser opens the page and lets it clear. A blocked attempt is retried on a fresh exit, and after a pass the session is reused for your next calls to that site.

cURL:

```bash
curl -X POST https://app.quanticdata.io/api/v1/scraper/unlock \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/", "country": "us"}'
```

Python:

```python
import requests

r = requests.post(
    "https://app.quanticdata.io/api/v1/scraper/unlock",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url": "https://example.com/", "country": "us"},
    timeout=130,
)
page = r.json()["payload"]
print(page["status"], page["blocked"], page["latency_ms"], page["cost"])
print(page["body"][:500])
```

Node.js:

```javascript
const res = await fetch("https://app.quanticdata.io/api/v1/scraper/unlock", {
  method: "POST",
  headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://example.com/", country: "us" }),
});
const { payload: page } = await res.json();
console.log(page.status, page.blocked, page.latency_ms, page.cost);
console.log(page.body.slice(0, 500));
```

When a captcha is still served, the answer says so: `blocked: true`, `blockClass: "captcha"`, and in `vendor` the system or the widget, such as `recaptcha` or `hcaptcha`. Your code can skip the page or send it again later.

### What the answer tells you

| Field | What it says |
| --- | --- |
| `blocked` | `false` when the real page came back. `true` only when every attempt met the block; the block page is then in `body`. |
| `vendor` | The system or the widget the engine recognised on a blocked attempt, such as `recaptcha`, `hcaptcha`, `cloudflare`, `datadome`, `awswaf` or `perimeterx`. |
| `blockClass` | `captcha` when the page asks a person: a tick, a puzzle, a slider, press and hold. `js_challenge` for a check that a browser clears by itself. |
| `tier_served` | `tls` for a browser-grade request, `browser` when a real browser had to open the page. |
| `attempts`, `latency_ms` | How many tries the call took and how long it lasted. |
| `cost` | The bytes this call moved and what it was charged. |

## Measured, with a date

The canary of 11 November 2026 runs a fixed list of 27 protected targets, sites of the eight systems of the guides among them: 72 of 81 calls came back with the real page, with a median response of 4.1 s.

A call counts only when the answer is 2xx and carries the real page, not a challenge. The list, the method and every run are on the [Web Unlocker status page](https://quanticdata.io/web-unlocker/status/).

## What it is for, and what it does not do

- It is for pages anyone can open in a browser without signing in: prices, listings, reviews, search results, public profiles.
- Challenges that need no human input are handled by the browser tier. An interactive captcha that asks a person to click, pick images or type is not solved: the answer comes back with `blocked: true` and `blockClass: "captcha"`, so your code can skip the page or try later.
- It does not log in for you and it does not create accounts. Pages behind a login stay behind it.
- Whether you may collect a page depends on the terms of the site and on the law that covers the data. A protection system does not change that, in either direction.

## CAPTCHA when scraping: questions people ask

### Why do I get a CAPTCHA when web scraping?

Because the site, or the protection in front of it, did not trust the request. The usual reasons are an address from a hosting range, a client that does not look like a real browser, too many requests in a short time, or no session from an earlier visit. The captcha is the site asking for proof.

### How do I avoid CAPTCHAs when scraping?

Remove the reasons for doubt before the request is sent: a residential or mobile address, a client that is coherent with a real browser, a pace a person could keep, and the cookies of a session that already passed. A request that looks ordinary is answered with the page, and no captcha is served.

### Can I scrape a website that uses reCAPTCHA or hCaptcha?

Often yes, because the widget usually guards a form, not the pages. Public pages load without it. Where a site shows a captcha before the content, it does so when it doubts the request, so a trusted request gets the page. Read blocked in the Web Unlocker answer to know which case you met.

### How does reCAPTCHA v3 affect a scraper?

reCAPTCHA v3 shows nothing. Per Google's documentation it returns a score for each request, and the site chooses what to do with a low one: ask for more proof, moderate, or throttle bots that may be scraping content. A scraper never sees a puzzle; it sees slower or refused answers instead.

### What is the difference between a CAPTCHA and a JavaScript challenge?

A JavaScript challenge is a check the browser passes by itself: a script runs, and the page follows with no click. A CAPTCHA asks a person to do something: tick, drag, hold, pick images. The Web Unlocker reports the first as blockClass js_challenge and the second as blockClass captcha.

### What does the Web Unlocker return when a CAPTCHA is served?

The same JSON as for a page, with blocked: true, blockClass captcha and the system or the widget in vendor, such as recaptcha or hcaptcha. The body carries the page that was served. The call was already retried on a fresh exit, so your code can skip the page or send it again later.

### Is it legal to scrape a site that shows a CAPTCHA?

The captcha does not decide that. What matters is the page and the data: whether the page is public, what the terms of the site say, and which law covers the data, personal data above all. The Web Unlocker fetches pages anyone can open without signing in; it does not log in and it does not create accounts.

## Send the URL that shows you a captcha

Create the free account, open Web Unlocker in the dashboard and paste the URL into the test box. The answer shows the status, the tier that served the page, the attempts and the cost of the call.

[Start free](https://app.quanticdata.io/register?next=%2Funlock&utm_source=web-unlocker&utm_medium=website&utm_campaign=unlocker-guides&utm_content=captcha-when-scraping-start-2) · [How the Web Unlocker works](https://quanticdata.io/web-unlocker/)

Block pages:

- [Block page dictionary](https://quanticdata.io/web-unlocker/block-pages/)
- ["Please enable JS and disable any ad blocker"](https://quanticdata.io/web-unlocker/please-enable-js-and-disable-any-ad-blocker/)
- ["Access Denied ... on this server"](https://quanticdata.io/web-unlocker/access-denied-on-this-server/)
- [Error 1015](https://quanticdata.io/web-unlocker/cloudflare-error-1015/)
- [Error 1020](https://quanticdata.io/web-unlocker/cloudflare-error-1020/)
- ["Just a moment..."](https://quanticdata.io/web-unlocker/cloudflare-just-a-moment/)
- ["Pardon Our Interruption"](https://quanticdata.io/web-unlocker/pardon-our-interruption/)
- [Press and hold CAPTCHA](https://quanticdata.io/web-unlocker/press-and-hold-captcha/)
- ["The requested URL was rejected"](https://quanticdata.io/web-unlocker/request-rejected-support-id/)
- [Cloudflare Turnstile](https://quanticdata.io/web-unlocker/cloudflare-turnstile/)

Each system:

- [Cloudflare](https://quanticdata.io/web-unlocker/bypass-cloudflare/)
- [DataDome](https://quanticdata.io/web-unlocker/bypass-datadome/)
- [Akamai Bot Manager](https://quanticdata.io/web-unlocker/bypass-akamai/)
- [PerimeterX (HUMAN)](https://quanticdata.io/web-unlocker/bypass-perimeterx/)
- [Kasada](https://quanticdata.io/web-unlocker/bypass-kasada/)
- [Imperva (Incapsula)](https://quanticdata.io/web-unlocker/bypass-imperva/)
- [F5 BIG-IP and Shape](https://quanticdata.io/web-unlocker/bypass-f5/)
- [AWS WAF](https://quanticdata.io/web-unlocker/bypass-aws-waf/)

Also:

- [Web Unlocker](https://quanticdata.io/web-unlocker/)
- [Status and canary](https://quanticdata.io/web-unlocker/status/)
- [Residential proxies](https://quanticdata.io/residential-proxies/)
- [Mobile proxies](https://quanticdata.io/mobile-proxies/)

## References

By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData · Published 11 November 2026 · Updated 11 November 2026 · [Read as Markdown](https://quanticdata.io/web-unlocker/captcha-when-scraping/index.md)

- [Google reCAPTCHA docs: choosing the type of reCAPTCHA (v3, v2 checkbox, v2 invisible)](https://developers.google.com/recaptcha/docs/versions)
- [Google reCAPTCHA docs: reCAPTCHA v3 (the score and what a site does with it)](https://developers.google.com/recaptcha/docs/v3)
- [hCaptcha docs: how the widget and its token work](https://docs.hcaptcha.com/)
- [hCaptcha docs: invisible and passive modes](https://docs.hcaptcha.com/invisible)
- [Cloudflare docs: Turnstile widgets (managed, non-interactive and invisible modes, error states)](https://developers.cloudflare.com/turnstile/concepts/widget/)
- [DataDome docs: response pages (device check, captcha, block)](https://docs.datadome.co/docs/response-pages)
- [DataDome docs: Device Check](https://docs.datadome.co/docs/device-check)
- [AWS WAF developer guide: CAPTCHA and Challenge action behavior (status 405 and 202, the x-amzn-waf-action header)](https://docs.aws.amazon.com/waf/latest/developerguide/waf-captcha-and-challenge-actions.html)
- [HUMAN docs: the HUMAN Challenge (press and hold)](https://docs.humansecurity.com/applications/human-challenge)
- [Web Unlocker in the QuanticData API docs: request, answer, errors and how to read a block (vendor and blockClass)](https://quanticdata.io/docs/#tag/Web-Unlocker)
- [Web Unlocker status: the canary list, the method and every run](https://quanticdata.io/web-unlocker/status/)

---

Source: https://quanticdata.io/web-unlocker/captcha-when-scraping/ · Site index for AI: https://quanticdata.io/llms.txt