A 403 Forbidden while scraping means the server received your request, understood it, and refused to serve it. If the same URL loads in a browser, the refusal is about you, not the page: something in the request identified it as automated. The response headers usually name the system that made that call, and knowing which one it was decides whether the fix is a user agent, a header set, a different kind of IP, or a TLS fingerprint.
Read the 403 before you touch your code
The wrong instinct is to rotate everything at once. The right one is to look at the response, because bot-management products sign their work. A short body and one of these headers tells you what refused you and how hard it will be to get past:
| Signal in the response | Who refused you | What it usually means |
|---|---|---|
server: cloudflare + cf-mitigated: challenge or a body mentioning error 1020 | Cloudflare WAF / Bot Management | Firewall rule or bot score; often TLS fingerprint plus IP |
server: AkamaiGHost, tiny “Error Page” body | Akamai Bot Manager | Header consistency and TLS fingerprint |
x-datadome: protected, body loading captcha-delivery.com | DataDome | Fingerprint plus IP reputation; a challenge is a soft 403 |
x-amzn-waf-action: challenge, status 202 | AWS WAF | JavaScript challenge; the 202 is a refusal in disguise |
x-iinfo, incap_ses cookie | Imperva / Incapsula | Fingerprint and behaviour scoring |
| Plain 403, normal server header, HTML error page | The site itself (mod_security, a rule on UA or IP) | Usually the easiest: user agent or IP range |
The free WAF detector runs this classification for any URL and reports which product fronts it. Two shapes deserve a special note because they are not literally 403s: a 202 with a challenge body and a 200 whose body is a challenge page. Both count as blocks, and a scraper that only checks status_code == 200 will happily store the challenge HTML and call it data.
The five causes, in order
- The default user agent.
python-requests/2.32,axios/1.x,Go-http-client/1.1andcurl/8are on every blocklist. A 403 on the very first request, on a site that is not behind a bot-management product, is nearly always this. - An inconsistent header set. A Chrome user agent with no
Accept-Language, noSec-Fetch-*headers andAccept: */*is a contradiction, and the contradiction is what gets scored. Send what the browser sends, in the order it sends it. - The IP. Cloud and datacenter ranges are labelled as hosting in every IP intelligence database, and whole subnets carry the reputation of their worst tenant. A 403 that appears only from your server and not from your laptop is this one. Residential exits fix the label; they do not fix the two causes above.
- The TLS and HTTP/2 fingerprint. The ClientHello of Python's
sslmodule does not look like Chrome's, and neither does its HTTP/2 SETTINGS frame. Cloudflare, Akamai and DataDome all compare them to the user agent you claim. Perfect headers on a Python TLS stack still fail here. - Rate and geography. A run that succeeds for a while and then turns into 403s is rate-based, even if the code is not 429. A 403 only from certain countries is a geo rule, and the fix is a proxy exit in the right country rather than any header.
A decision tree you can run in five minutes
Each step isolates one cause. Stop at the first step that changes the outcome.
# 1. Is it really about you? Same 403 in a browser (private window) = permission-based, not anti-bot.
# 2. User agent only:
curl -sS -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "$URL"
# 3. Full header set (Accept, Accept-Language, Accept-Encoding, Sec-Fetch-Dest/Mode/Site, Upgrade-Insecure-Requests):
curl -sS -o /dev/null -w "%{http_code}\n" -H @browser-headers.txt "$URL"
# 4. Same headers from a residential exit instead of your server:
curl -sS -o /dev/null -w "%{http_code}\n" -H @browser-headers.txt -x pr.quanticdata.io:7777 -U "USER-country-us:PASS" "$URL"
# 5. Browser TLS fingerprint (curl_cffi impersonates Chrome's ClientHello and HTTP/2 settings):
python -c "from curl_cffi import requests; print(requests.get('$URL', impersonate='chrome').status_code)"If step 2 fixes it, you were on cause 1. If step 3 fixes it, cause 2. Step 4, cause 3. Step 5, cause 4. If nothing fixes it and the block appears only after N requests, it is cause 5 and the answer is pacing and rotation, covered in the 429 guide. The curl converter turns the working curl line into requests, httpx or Node code so the fix survives the translation.
What we saw fetching 16 major sites
On 3 September 2026 we fetched one representative page from each of 16 well-known sites with a pure HTTP client, no JavaScript execution, using a real Chrome TLS profile and US residential exits. This is the cheapest tier of the web scraping API, with escalation to a browser switched off so the refusals would be visible.
| Outcome | Sites | Shape of the refusal |
|---|---|---|
| Full HTML on first attempt | Amazon search, Walmart search, Zillow, Indeed, Wikipedia, BBC News, GitHub, Tripadvisor, LinkedIn company, Reddit, NYTimes | — |
| 403 | eBay (Akamai), Yelp (DataDome) | Short error page; DataDome served a captcha loader |
| 202 challenge | IMDb (AWS WAF), Booking.com | “Verify that you're not a robot” body, no data |
| 200 with challenge body | Google Search | A 200 that contains no results, only a retry script |
Three things follow. With a browser-grade TLS profile and residential IPs, two thirds of hard commercial targets did not need a browser at all, which is where the cost difference lives: a plain fetch is $0.0002 per page against $0.001 rendered. Only two of the five refusals were literal 403s, so a scraper must classify bodies, not just status codes. And every one of the five refused on fingerprint or challenge grounds, which no user agent change would have touched.
Fixes in Python, and when to stop fighting
For causes 1 and 2, the fix is a real header set, kept consistent with the user agent it claims:
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
}
s = requests.Session(); s.headers.update(HEADERS)For cause 3, route the session through a residential exit, exactly as in the requests proxy guide. For cause 4, requests cannot help because the fingerprint is decided below it; switch the transport to curl_cffi with impersonate="chrome", which keeps the same API surface. For a browser-rendered fallback there is Playwright, and the stealth guide covers what it does and does not hide.
At some point the maintenance cost exceeds the value of doing it yourself: fingerprints drift with every Chrome release, WAF vendors ship weekly, and a pipeline that was green on Monday is red on Thursday. That is the point of a pay-per-success API. With engine: auto the scraping API tries the TLS tier first and escalates to a real browser only when the response is classified as a block, so you pay the rendered price only for the pages that needed it, and nothing for the ones that failed at every tier.
403 versus its neighbours
| Code | Who sent it | Meaning | Where to read more |
|---|---|---|---|
| 401 | Site | Log in first; the resource needs credentials | — |
| 403 | Site or its WAF | Identified and refused; anti-bot in nearly every scraping case | This guide |
| 407 | Your proxy | Proxy credentials or allowlist; the site never saw the request | 407 guide |
| 429 | Site | Too fast; a rate limit keyed by IP, session or account | 429 guide |
A 403 is also the code a site returns when its terms say no. Whether you may proceed past it is a separate question from whether you can, and it is covered in is web scraping legal in the US.