CAPTCHA when web scraping: why you get one, and how not to

These are the checks a scraper meets most often. Some are widgets a site adds to a form; others belong to a protection system in front of the whole site. They are triggered by different things.

In your own browser a captcha is the same judgement, made about you: something in the visit looked unusual. Solve it once and it normally goes away. When it returns on every page, the cause is nearly always one of a few.

• Turn off the VPN or proxy. A shared address is the most common reason for repeated captchas.
• Allow cookies for the site. The pass is stored in one, and without it every page asks again.
• Pause extensions that block scripts, and use an up-to-date browser: a check that cannot run cannot pass.
• Slow down. Many tabs or fast reloads look like a machine.
• If a puzzle is hard to see or to use, look for its audio or accessibility option, or ask the site for another way in.

A site does not show a captcha at random. It is the answer to doubt, and four things raise that doubt for a script. Lower them and the captcha is not served in the first place.

• The address: requests from hosting ranges, or from an address that already sent a lot, are doubted first. A residential or mobile address starts from the trust of an ordinary visitor.
• The client: an HTTP library does not connect like the browser it names, and an automated browser leaves marks. A client that is coherent with a real browser removes the most common trigger.
• The pace: many requests in a short time from one identity look like a machine, whatever sent them. Fewer requests per address, and pauses between them, keep the count under the line.
• The session: a visitor who passed a check carries a cookie that says so. A script that throws its cookies away is a stranger on every request, and is tested every time.
• What does not help: a new User-Agent alone, or a token or a cookie taken from another machine.

Each system has its guide, and the block page dictionary tells them apart: Cloudflare, DataDome, Akamai Bot Manager, PerimeterX (HUMAN), Kasada, Imperva (Incapsula), F5 BIG-IP and Shape, AWS WAF.

Send the URL to the REST endpoint. The request leaves from a residential or mobile exit with a browser-grade client. When the site answers with a check that needs no person, a real browser opens the page and lets it clear. A blocked attempt is retried on a fresh exit, and after a pass the session is reused for your next calls to that site.

curl -X POST https://app.quanticdata.io/api/v1/scraper/unlock \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/", "country": "us"}'

import requests

r = requests.post(
    "https://app.quanticdata.io/api/v1/scraper/unlock",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url": "https://example.com/", "country": "us"},
    timeout=130,
)
page = r.json()["payload"]
print(page["status"], page["blocked"], page["latency_ms"], page["cost"])
print(page["body"][:500])

const res = await fetch("https://app.quanticdata.io/api/v1/scraper/unlock", {
  method: "POST",
  headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://example.com/", country: "us" }),
});
const { payload: page } = await res.json();
console.log(page.status, page.blocked, page.latency_ms, page.cost);
console.log(page.body.slice(0, 500));

When a captcha is still served, the answer says so: blocked: true, blockClass: "captcha", and in vendor the system or the widget, such as recaptcha or hcaptcha. Your code can skip the page or send it again later.

What the answer tells you

The canary of 11 November 2026 runs a fixed list of 27 protected targets, sites of the eight systems of the guides among them: 72 of 81 calls came back with the real page, with a median response of 4.1 s.

A call counts only when the answer is 2xx and carries the real page, not a challenge. The list, the method and every run are on the Web Unlocker status page.

• It is for pages anyone can open in a browser without signing in: prices, listings, reviews, search results, public profiles.
• Challenges that need no human input are handled by the browser tier. An interactive captcha that asks a person to click, pick images or type is not solved: the answer comes back with blocked: true and blockClass: "captcha", so your code can skip the page or try later.
• It does not log in for you and it does not create accounts. Pages behind a login stay behind it.
• Whether you may collect a page depends on the terms of the site and on the law that covers the data. A protection system does not change that, in either direction.