# Web Scraping 403 Forbidden: Causes and Fixes

> A scraping 403 means the site saw you and refused. Read the block from its headers, the five causes, a 5-minute decision tree, and what 16 major sites returned.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/Web Scraping 403 Forbidden: Causes and Fixes

# Web Scraping 403 Forbidden: Causes and Fixes

TroubleshootingSep 3, 2026·7 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

On this page [Read the 403 before you touch your code](/blog/web-scraping-403-forbidden/#read-the-403-before-you-touch-your-code) [The five causes, in order](/blog/web-scraping-403-forbidden/#the-five-causes-in-order) [A decision tree you can run in five minutes](/blog/web-scraping-403-forbidden/#a-decision-tree-you-can-run-in-five-minutes) [What we saw fetching 16 major sites](/blog/web-scraping-403-forbidden/#what-we-saw-fetching-16-major-sites) [Fixes in Python, and when to stop fighting](/blog/web-scraping-403-forbidden/#fixes-in-python-and-when-to-stop-fighting) [403 versus its neighbours](/blog/web-scraping-403-forbidden/#403-versus-its-neighbours)

A 403 Forbidden while scraping means the server received your request, understood it, and refused to serve it. If the same URL loads in a browser, the refusal is about *you*, not the page: something in the request identified it as automated. The response headers usually name the system that made that call, and knowing which one it was decides whether the fix is a user agent, a header set, a different kind of IP, or a TLS fingerprint.

## Read the 403 before you touch your code

The wrong instinct is to rotate everything at once. The right one is to look at the response, because bot-management products sign their work. A short body and one of these headers tells you what refused you and how hard it will be to get past:

| Signal in the response | Who refused you | What it usually means |
| --- | --- | --- |
| `server: cloudflare` + `cf-mitigated: challenge` or a body mentioning error 1020 | Cloudflare WAF / Bot Management | Firewall rule or bot score; often TLS fingerprint plus IP |
| `server: AkamaiGHost`, tiny “Error Page” body | Akamai Bot Manager | Header consistency and TLS fingerprint |
| `x-datadome: protected`, body loading captcha-delivery.com | DataDome | Fingerprint plus IP reputation; a challenge is a soft 403 |
| `x-amzn-waf-action: challenge`, status 202 | AWS WAF | JavaScript challenge; the 202 is a refusal in disguise |
| `x-iinfo`, `incap_ses` cookie | Imperva / Incapsula | Fingerprint and behaviour scoring |
| Plain 403, normal server header, HTML error page | The site itself (mod_security, a rule on UA or IP) | Usually the easiest: user agent or IP range |

The free [WAF detector](https://quanticdata.io/tools/waf-detector/) runs this classification for any URL and reports which product fronts it. Two shapes deserve a special note because they are not literally 403s: a **202 with a challenge body** and a **200 whose body is a challenge page**. Both count as blocks, and a scraper that only checks `status_code == 200` will happily store the challenge HTML and call it data.

## The five causes, in order

1. **The default user agent.** `python-requests/2.32`, `axios/1.x`, `Go-http-client/1.1` and `curl/8` are on every blocklist. A 403 on the very first request, on a site that is not behind a bot-management product, is nearly always this.

2. **An inconsistent header set.** A Chrome user agent with no `Accept-Language`, no `Sec-Fetch-*` headers and `Accept: */*` is a contradiction, and the contradiction is what gets scored. Send what the browser sends, in the order it sends it.

3. **The IP.** Cloud and datacenter ranges are labelled as hosting in every IP intelligence database, and whole subnets carry the reputation of their worst tenant. A 403 that appears only from your server and not from your laptop is this one. [Residential exits](https://quanticdata.io/residential-proxies/) fix the label; they do not fix the two causes above.

4. **The TLS and HTTP/2 fingerprint.** The ClientHello of Python's `ssl` module does not look like Chrome's, and neither does its HTTP/2 SETTINGS frame. Cloudflare, Akamai and DataDome all compare them to the user agent you claim. Perfect headers on a Python TLS stack still fail here.

5. **Rate and geography.** A run that succeeds for a while and then turns into 403s is rate-based, even if the code is not 429. A 403 only from certain countries is a geo rule, and the fix is a proxy exit in the right country rather than any header.

## A decision tree you can run in five minutes

Each step isolates one cause. Stop at the first step that changes the outcome.

```
# 1. Is it really about you? Same 403 in a browser (private window) = permission-based, not anti-bot.
# 2. User agent only:
curl -sS -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "$URL"
# 3. Full header set (Accept, Accept-Language, Accept-Encoding, Sec-Fetch-Dest/Mode/Site, Upgrade-Insecure-Requests):
curl -sS -o /dev/null -w "%{http_code}\n" -H @browser-headers.txt "$URL"
# 4. Same headers from a residential exit instead of your server:
curl -sS -o /dev/null -w "%{http_code}\n" -H @browser-headers.txt -x pr.quanticdata.io:7777 -U "USER-country-us:PASS" "$URL"
# 5. Browser TLS fingerprint (curl_cffi impersonates Chrome's ClientHello and HTTP/2 settings):
python -c "from curl_cffi import requests; print(requests.get('$URL', impersonate='chrome').status_code)"
```

If step 2 fixes it, you were on cause 1. If step 3 fixes it, cause 2. Step 4, cause 3. Step 5, cause 4. If nothing fixes it and the block appears only after N requests, it is cause 5 and the answer is pacing and rotation, covered in the [429 guide](https://quanticdata.io/blog/429-too-many-requests-web-scraping/). The [curl converter](https://quanticdata.io/tools/curl-converter/) turns the working curl line into requests, httpx or Node code so the fix survives the translation.

## What we saw fetching 16 major sites

On 3 September 2026 we fetched one representative page from each of 16 well-known sites with a pure HTTP client, no JavaScript execution, using a real Chrome TLS profile and US residential exits. This is the cheapest tier of the [web scraping API](https://quanticdata.io/web-scraping-api/), with escalation to a browser switched off so the refusals would be visible.

| Outcome | Sites | Shape of the refusal |
| --- | --- | --- |
| Full HTML on first attempt | Amazon search, Walmart search, Zillow, Indeed, Wikipedia, BBC News, GitHub, Tripadvisor, LinkedIn company, Reddit, NYTimes | — |
| 403 | eBay (Akamai), Yelp (DataDome) | Short error page; DataDome served a captcha loader |
| 202 challenge | IMDb (AWS WAF), Booking.com | “Verify that you're not a robot” body, no data |
| 200 with challenge body | Google Search | A 200 that contains no results, only a retry script |

Three things follow. With a browser-grade TLS profile and residential IPs, two thirds of hard commercial targets did not need a browser at all, which is where the cost difference lives: a plain fetch is $0.0002 per page against $0.001 rendered. Only two of the five refusals were literal 403s, so a scraper must classify bodies, not just status codes. And every one of the five refused on fingerprint or challenge grounds, which no user agent change would have touched.

## Fixes in Python, and when to stop fighting

For causes 1 and 2, the fix is a real header set, kept consistent with the user agent it claims:

```
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}
s = requests.Session(); s.headers.update(HEADERS)
```

For cause 3, route the session through a residential exit, exactly as in the [requests proxy guide](https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/). For cause 4, `requests` cannot help because the fingerprint is decided below it; switch the transport to `curl_cffi` with `impersonate="chrome"`, which keeps the same API surface. For a browser-rendered fallback there is Playwright, and the [stealth guide](https://quanticdata.io/blog/playwright-stealth-in-python/) covers what it does and does not hide.

At some point the maintenance cost exceeds the value of doing it yourself: fingerprints drift with every Chrome release, WAF vendors ship weekly, and a pipeline that was green on Monday is red on Thursday. That is the point of a pay-per-success API. With `engine: auto` the scraping API tries the TLS tier first and escalates to a real browser only when the response is classified as a block, so you pay the rendered price only for the pages that needed it, and nothing for the ones that failed at every tier.

## 403 versus its neighbours

| Code | Who sent it | Meaning | Where to read more |
| --- | --- | --- | --- |
| 401 | Site | Log in first; the resource needs credentials | — |
| 403 | Site or its WAF | Identified and refused; anti-bot in nearly every scraping case | This guide |
| 407 | Your proxy | Proxy credentials or allowlist; the site never saw the request | 407 guide |
| 429 | Site | Too fast; a rate limit keyed by IP, session or account | 429 guide |

A 403 is also the code a site returns when its terms say no. Whether you may proceed past it is a separate question from whether you can, and it is covered in [is web scraping legal in the US](https://quanticdata.io/blog/is-web-scraping-legal-in-us/).

### Sources & further reading

- [MDN — 403 Forbidden](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/403)

- [RFC 9110 — HTTP Semantics, §15.5.4 403 Forbidden](https://www.rfc-editor.org/rfc/rfc9110.html#name-403-forbidden)

- [Cloudflare — Error 1020: Access denied](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1020/)

- [AWS WAF — Challenge action](https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-action.html)

- [curl_cffi — Python binding for curl-impersonate](https://github.com/lexiforest/curl_cffi)

- [MDN — Sec-Fetch-* request headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Sec-Fetch-Dest)

## FAQ

Quick answers on web scraping 403 forbidden.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### What causes a 403 Forbidden error when web scraping?

The site or its bot-management layer identified the request as automated and refused it. In order of likelihood: a library default user agent, an incomplete or inconsistent set of browser headers, an IP from a hosting range with poor reputation, a TLS or HTTP/2 fingerprint that does not match the claimed browser, and finally rate or geographic rules. The response headers usually name the product that refused you.

### How do I fix a 403 error in Python requests?

Work through the causes in order and stop when the status changes: send a current Chrome user agent, then the full browser header set (Accept, Accept-Language, Sec-Fetch-*), then route through a residential proxy, then switch the transport to curl_cffi with impersonate="chrome" so the TLS fingerprint matches. If the 403 appears only after many requests, it is a rate limit and the fix is pacing and rotation.

### Why does my script get 403 when the browser works?

Because the site compares dozens of signals the browser gets right by default: header set and order, TLS ClientHello, HTTP/2 settings, IP type, cookies and JavaScript challenges. A Python or Node client differs on several of them at once. The browser working proves the URL is public; the 403 tells you which signal to fix.

### Does a 403 always mean I am blocked?

Almost always in scraping. The exception is a resource that genuinely requires permission, which returns 403 in a browser too. Also note that blocks do not always arrive as 403: AWS WAF returns a 202 challenge, and some sites return a 200 whose body is a challenge page, so check the body as well as the status.

### How do I avoid 403 errors at scale?

Keep the header set consistent with the user agent, match the TLS fingerprint, rotate residential exits, keep per-IP request rates under the target threshold, back off on 403 and 429, and classify response bodies so a challenge page is never stored as data. Or hand the escalation to a pay-per-success API that tries a plain fetch first and renders only when blocked.

## Let the escalation happen for you

The web scraping API fetches with a browser-grade TLS profile behind residential IPs at $0.0002 per page and escalates to a real browser only when the response is classified as a block, at $0.001. Pages that fail at every tier are never billed, and every account gets $2 of free usage a month.

[Start free — $2/month included](https://quanticdata.io/signup/)[Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

[Troubleshooting How to Fix 407 Proxy Authentication Required A 407 is the proxy refusing to forward your request until it sees credentials it accepts, so rotating headers or slowing down cannot fix it. On HTTPS it does not even arrive as a status code. How to read the challenge, tell the five causes apart, and fix each one in curl, Python requests, httpx, Node and Scrapy. Read →](https://quanticdata.io/blog/how-to-fix-407-proxy-authentication-required/) [Troubleshooting How to Fix 429 Too Many Requests When Scraping A 429 is the one block that tells the truth: you crossed a rate limit. What it does not tell you is what the limit is keyed on, and that decides whether proxies help at all. A two-request test to find the key, backoff code that honours Retry-After, the concurrency formula that turns pages per hour into IPs in flight, and a per-host token bucket. Read →](https://quanticdata.io/blog/429-too-many-requests-web-scraping/) [Troubleshooting Proxy Not Working? A 10-Step Checklist “Proxy not working” is four different problems wearing one name: you cannot reach the proxy, the proxy refuses you, the proxy reaches the site but the site refuses it, or it works and is slow. Each layer has its own error strings and its own fix. A table that maps the message to the layer, curl timings that separate slow from broken, and a 10-step checklist in the order that saves the most time. Read →](https://quanticdata.io/blog/proxy-not-working-checklist/)

## Also on this site

Quantic**Data**

Residential proxies & web data APIs for AI.

#### Proxies

- [Residential Basic](https://quanticdata.io/residential-proxies/#basic)

- [Residential Premium](https://quanticdata.io/residential-proxies/#plans)

- [Cheap Residential](https://quanticdata.io/cheap-residential-proxies/)

- [Mobile Proxies](https://quanticdata.io/mobile-proxies/)

- [Datacenter Proxies](https://quanticdata.io/datacenter-proxies/)

- [ISP Proxies](https://quanticdata.io/isp-proxies/)

- [Rotating Proxies](https://quanticdata.io/rotating-proxies/)

- [Sneaker Proxies](https://quanticdata.io/sneaker-proxies/)

- [SOCKS5 Proxies](https://quanticdata.io/socks5-proxies/)

- [IPv6 Proxies](https://quanticdata.io/ipv6-proxies/)

- [Proxy locations](https://quanticdata.io/proxies/)

#### Data APIs

- [MCP Server](https://quanticdata.io/mcp-server/)

- [Web Scraper API](https://quanticdata.io/web-scraping-api/)

- [SERP API](https://quanticdata.io/serp-api/)

- [Collectors](https://quanticdata.io/collectors/)

- [Web Data for AI](https://quanticdata.io/web-data-api-for-ai/)

- [Quantic AI](https://quanticdata.io/ai-web-scraping-service/)

- [Crawl & Map](https://quanticdata.io/crawl-map/)

- [SEO Audit](https://quanticdata.io/seo-audit/)

#### Use cases

- [Company data](https://quanticdata.io/scrape-company-data/)

- [Price monitoring](https://quanticdata.io/competitor-price-monitoring/)

- [Market research](https://quanticdata.io/market-research-data/)

- [Real estate data](https://quanticdata.io/real-estate-data-scraping/)

- [Scrape job postings](https://quanticdata.io/scrape-job-postings/)

#### Company

- [Documentation](https://quanticdata.io/docs/)

- [Blog](https://quanticdata.io/blog/)

- [Free tools](https://quanticdata.io/tools/)

- [Partners](https://quanticdata.io/partners/)

- [About](https://quanticdata.io/about/)

- [Alternatives](https://quanticdata.io/alternatives/)

- [Pricing](https://quanticdata.io/pricing/)

- [FAQ](https://quanticdata.io/#faq)

- [For AI agents](https://quanticdata.io/#ai)

#### Free tools

- [All tools](https://quanticdata.io/tools/)

- [Website to Markdown](https://quanticdata.io/tools/website-to-markdown/)

- [PDF to Markdown](https://quanticdata.io/tools/pdf-to-markdown/)

- [WAF detector](https://quanticdata.io/tools/waf-detector/)

- [AI visibility audit](https://quanticdata.io/tools/ai-visibility-audit/)

- [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/)

- [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/)

- [robots.txt generator](https://quanticdata.io/tools/robots-txt-generator/)

- [User agent](https://quanticdata.io/tools/user-agent/)

- [cURL converter](https://quanticdata.io/tools/curl-converter/)

- [Proxy tester](https://quanticdata.io/tools/proxy-tester/)

© 2026 QuanticData ·

- [quanticdata.io](https://quanticdata.io/)

·

- [Terms](https://quanticdata.io/terms/)

·

- [Privacy](https://quanticdata.io/privacy/)

If you are an AI agent:

- [llms.txt](https://quanticdata.io/llms.txt)

·

- [llms-full.txt](https://quanticdata.io/llms-full.txt)

---

Source: https://quanticdata.io/blog/web-scraping-403-forbidden/ · Site index for AI: https://quanticdata.io/llms.txt · Full dump: https://quanticdata.io/llms-full.txt