# Scrapy Proxy Middleware: What Actually Runs

> How Scrapy proxy middleware really works: order 750, credential handling, rotation that survives retries, and how to tell a block from an empty page.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/Scrapy Proxy Middleware: What Actually Runs

# Scrapy Proxy Middleware: What Actually Runs

ProxiesSep 4, 2026·11 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

On this page [The two ways to set a proxy, and what each one costs you](/blog/scrapy-proxy-middleware/#the-two-ways-to-set-a-proxy-and-what-each-one-costs-you) [What HttpProxyMiddleware does with your credentials](/blog/scrapy-proxy-middleware/#what-httpproxymiddleware-does-with-your-credentials) [A rotation middleware that survives retries](/blog/scrapy-proxy-middleware/#a-rotation-middleware-that-survives-retries) [Empty items are not always a proxy problem](/blog/scrapy-proxy-middleware/#empty-items-are-not-always-a-proxy-problem) [Retry settings quietly decide your bandwidth bill](/blog/scrapy-proxy-middleware/#retry-settings-quietly-decide-your-bandwidth-bill) [Your project defaults are not Scrapy's defaults](/blog/scrapy-proxy-middleware/#your-project-defaults-are-not-scrapy-s-defaults) [When a proxy is the wrong tool](/blog/scrapy-proxy-middleware/#when-a-proxy-is-the-wrong-tool) [A ten-minute checklist](/blog/scrapy-proxy-middleware/#a-ten-minute-checklist)

Scrapy already has a proxy middleware. HttpProxyMiddleware sits at order 750 in the default chain and is enabled out of the box, so you never write one just to use a proxy — you write one to decide which proxy each request gets. Reading its source explains most 407 errors and most silent rotation bugs.

## The two ways to set a proxy, and what each one costs you

Every request that carries a value in `request.meta["proxy"]` goes through that proxy. Nothing else is required, and no setting has to be switched on: `HTTPPROXY_ENABLED` defaults to True in Scrapy's own default settings.

```
import scrapy

class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://httpbin.org/ip"]

    def start_requests(self):
        proxy = "http://USER-country-us:PASS@rotate.quanticdata.io:8000"
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": proxy}, callback=self.parse)

    def parse(self, response):
        self.logger.info("exit ip: %s", response.text.strip())
```

The second route is the environment. When no request carries a proxy in its meta, the middleware falls back to what Python's `urllib.request.getproxies()` reports, which means the standard `http_proxy` and `https_proxy` variables work with no code at all, and `no_proxy` is honoured for http and https URLs. That is the fastest way to test whether a proxy is the problem: run the same spider once with the variables set and once without.

A custom middleware is the third route, and the only one worth the file. Its job is not to enable proxying; it is to answer the question "which exit should this specific request use" on every attempt, including retries.

## What HttpProxyMiddleware does with your credentials

This is the part every tutorial skips, and it is the part that produces the support tickets. In `process_request` the middleware takes the proxy URL, splits the credentials out of it with `_parse_proxy`, base64-encodes them, and sets a `Proxy-Authorization` header. The URL that stays in `request.meta["proxy"]` is the stripped one, without user and password. It also records the proxy those credentials belong to in `request.meta["_auth_proxy"]`.

Three consequences follow directly from that code, and all three bite in production.

- **Order matters, and 750 is the deadline.** Middlewares run in increasing order, so a component that sets `meta["proxy"]` at order 800 sets it after HttpProxyMiddleware has already run. The credentials never become a header, and the target answers 407.

- **Changing the proxy drops the header.** If a later hook rewrites `meta["proxy"]` to a different host than the one in `_auth_proxy`, the middleware deletes `Proxy-Authorization` rather than sending the old credentials to a new server. Rotation code that assigns a bare host and expects the previous credentials to follow silently produces unauthenticated requests.

- **Credentials are encoded as latin-1.** `HTTPPROXY_AUTH_ENCODING` defaults to "latin-1", so a generated password containing characters outside that range raises on encode. If a password works in cURL and fails in Scrapy, this is usually why.

The practical rule is short: always assign the full URL with credentials in it, always before order 750. Our walkthrough of [407 Proxy Authentication Required](https://quanticdata.io/blog/how-to-fix-407-proxy-authentication-required/) covers the other half of these failures, the ones that come from the provider side.

## A rotation middleware that survives retries

The skeleton copied around the web looks like this: check whether the request already has a proxy, and if not, set one from settings. It works exactly once. When RetryMiddleware re-schedules a failed request, the meta dictionary is carried over, so the guard sees an existing proxy and keeps the exit that just failed. You retry a dead IP twice and log three failures.

```
import random

class RotatingProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        proxies = crawler.settings.getlist("PROXY_POOL")
        if not proxies:
            from scrapy.exceptions import NotConfigured
            raise NotConfigured("PROXY_POOL is empty")
        return cls(proxies)

    def process_request(self, request, spider):
        # assign on every attempt, retries included, credentials included
        request.meta["proxy"] = random.choice(self.proxies)

    def process_response(self, request, response, spider):
        if response.status in (403, 407, 429):
            spider.logger.warning(
                "status %s via %s", response.status, request.meta.get("proxy")
            )
        return response
```

```
# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 740,
}
PROXY_POOL = [
    "http://USER-country-us:PASS@rotate.quanticdata.io:8000",
    "http://USER-country-de:PASS@rotate.quanticdata.io:8000",
]
```

Two details make this different from the usual version. It assigns unconditionally, so a retry gets a fresh choice. And it logs the exit alongside the status, which is the only way to find out later whether one subnet is carrying all your blocks.

If the pool is a list of single IPs you also need health tracking, a cooldown, and a refresh job. A [rotating endpoint](https://quanticdata.io/rotating-proxies/) removes that work: one host, a new IP per request, and a sticky session when you need one by appending a session identifier to the username. In that case `PROXY_POOL` holds one entry per country, not per IP, and the pool never goes stale. The same pattern in plain Python is in our guide to [rotating proxies in Python](https://quanticdata.io/blog/how-to-rotate-proxies-python/).

## Empty items are not always a proxy problem

Before you buy more IPs, find out which failure you have. On 2026-09-04 we fetched twelve pages a Scrapy tutorial would plausibly target, twice each, from a United States residential exit: once as a plain HTTP client with no JavaScript execution — which is exactly what Scrapy is — and once fully rendered in a browser. The word counts are what a text extractor found in each view.

| Target page | No-JS view (what Scrapy sees) | Rendered view | Diagnosis |
| --- | --- | --- | --- |
| books.toscrape.com | 200, 308 words | 200, 308 words | Complete in the HTML |
| news.ycombinator.com | 200, 606 words | 200, 606 words | Complete in the HTML |
| amazon.com search | 200, 4,103 words | 200, 4,389 words | Complete in the HTML |
| indeed.com jobs | 200, 1,832 words | 403 interstitial, 70 words | Plain HTTP won, the browser was challenged |
| walmart.com search | 200, 890 words | 200, 4,148 words | Partial: about a fifth of the content |
| zillow.com listings | 200, 138 words | 403 captcha page | Metadata only, no listings |
| target.com search | 200, 27 words | 200, 29 words | Shell page, data behind a private API |
| quotes.toscrape.com/js | 200, 14 words | 200, 227 words | Content injected by script |
| ebay.com search | 403 | 200, 4,918 words | Blocked without a browser |
| etsy.com search | 403 | 403 | Blocked both ways |
| yelp.com search | 403 | 403 | Blocked both ways |
| g2.com category | 403 | 403 | Blocked both ways |

Four of the twelve gave a plain HTTP client everything. Four returned a hard 403. The interesting four are the ones in between: they answered **HTTP 200 and no data**. A spider that only counts non-200 responses sees four failures and quietly mis-parses four more, and no quantity of proxies changes that column. Two rows also cut the other way — on Indeed and Zillow the headless browser was challenged while the plain fetch went through, so "render it" is not a general fix either.

This is one sample per site, on one day, through one exit; treat it as a triage method rather than a scoreboard. The method is what transfers: fetch twice, compare, then decide. The measurement above came from our own [audit endpoint](https://quanticdata.io/seo-audit/), which does exactly that double fetch and returns the diff.

The equivalent inside a spider is three lines and a saved body:

```
def parse(self, response):
    rows = response.css("div.product")
    if not rows:
        self.logger.error(
            "no rows: status=%s bytes=%s proxy=%s",
            response.status, len(response.body), response.meta.get("proxy"),
        )
        with open("debug-%s.html" % response.status, "wb") as fh:
            fh.write(response.body)
        return
    for row in rows:
        yield {"title": row.css("h3 a::attr(title)").get()}
```

Open that file. If it holds a consent wall or a challenge page, it is a block — see [403 Forbidden while scraping](https://quanticdata.io/blog/web-scraping-403-forbidden/). If it holds a valid page with an empty container, the data arrives by script and you need a rendering step such as [scrapy-playwright](https://quanticdata.io/blog/how-to-use-scrapy-playwright/) or the site's own JSON endpoint. If it holds a rate-limit notice, read [429 Too Many Requests](https://quanticdata.io/blog/429-too-many-requests-web-scraping/) instead of adding IPs.

## Retry settings quietly decide your bandwidth bill

Scrapy's defaults retry on `[500, 502, 503, 504, 522, 524, 408, 429]` with `RETRY_TIMES = 2`, which is three attempts in total. Two things about that list matter when you are paying per gigabyte.

First, **403 is not in it**. A blocked crawl looks fast and cheap in the stats and produces nothing. Adding 403 to `RETRY_HTTP_CODES` only helps if the retry uses a different exit — which is precisely what the unconditional assignment in the middleware above guarantees. Add it without rotating, and you buy the same ban three times.

Second, tunnel failures are already covered: `scrapy.core.downloader.handlers.http11.TunnelError` is in the default `RETRY_EXCEPTIONS`, so a proxy that refuses the CONNECT is retried like any transport error. What is not covered is a proxy that answers with a captive portal at HTTP 200.

```
# settings.py — retry a ban only when the retry will change exit
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429, 403]
RETRY_TIMES = 2
DOWNLOAD_TIMEOUT = 30
```

The arithmetic is worth doing before the invoice arrives. Suppose your responses average 60 KB and you crawl 100,000 pages: that is roughly 6 GB. If a fifth of them fail twice before succeeding, you are at about 8.4 GB for the same 100,000 items, a 40 percent surcharge that appears nowhere in your item count. At the entry price for our rotating pool, $0.50/GB, that is the difference between $3.00 and $4.20 for the run — small here, linear as you scale, and the reason the retry policy belongs in the cost model. Our note on [how much proxy data you need](https://quanticdata.io/blog/how-much-proxy-data-do-i-need/) works through sizing from the other direction. Lower `DOWNLOAD_TIMEOUT` from its 180-second default too: a hung connection through a bad exit holds a concurrency slot for three minutes.

## Your project defaults are not Scrapy's defaults

The settings file that `scrapy startproject` writes is deliberately more conservative than the library, and people tune the wrong file for days without noticing.

| Setting | Library default | What startproject writes |
| --- | --- | --- |
| ROBOTSTXT_OBEY | False | True |
| CONCURRENT_REQUESTS_PER_DOMAIN | 8 | 1 |
| DOWNLOAD_DELAY | 0 | 1 |
| USER_AGENT | Scrapy/x.y (+https://scrapy.org) | commented out, so the library value is sent |

Two of those explain complaints that get blamed on proxies. A spider crawling at one request per second per domain is not slow because of the network. And a request that announces itself as Scrapy in the User-Agent header is filtered by ordinary bot rules before your exit IP is even considered — the cheapest fix in this entire article is setting a real browser User-Agent. Leaving `ROBOTSTXT_OBEY` on is a policy decision rather than a bug, but know that it is on: it is why some start URLs are dropped before a single proxy byte is spent.

## When a proxy is the wrong tool

Proxies solve one class of problem: the target refuses your network identity. They do not solve rendering, they do not solve challenge pages, and they do not solve a site that ships an empty shell and fetches its data from a private API.

- **Content arrives by script.** Render it or call the underlying endpoint. Scrapy stays the scheduler either way.

- **Blocked with and without a browser.** Four of our twelve sample pages were in this state. That is a full anti-bot stack, and beating it is a maintenance commitment, not a settings change.

- **You need one field from many pages.** Paying per successful page can be cheaper than paying per gigabyte plus the engineering to keep a pool healthy.

Our [scraping API](https://quanticdata.io/web-scraping-api/) starts at $0.0002 per page and $0.001 with JavaScript rendering, and it returns Markdown or structured JSON, so the parsing step disappears too. Called from inside a spider it is an ordinary request, which means your pipelines, feeds and scheduling stay exactly where they are:

```
yield scrapy.Request(
    "https://api.quanticdata.io/v1/scrape",
    method="POST",
    headers={"Authorization": "Bearer YOUR_KEY", "Content-Type": "application/json"},
    body=json.dumps({"url": target, "format": "markdown", "render": True}),
    callback=self.parse_markdown,
)
```

The honest split: use [residential proxies](https://quanticdata.io/residential-proxies/) when the site serves complete HTML to a plain client and only your network identity is in the way, and use a success-priced API when the page needs a browser or an anti-bot stack stands in front of it.

## A ten-minute checklist

1. Set `meta["proxy"]` with the credentials in the URL, from a middleware ordered under 750.

2. Assign on every attempt, not only when the key is missing, so retries change exit.

3. Log status, body length and exit together, on every response.

4. Save the body of the first empty parse and read it before touching settings.

5. Fetch one target twice, with and without JavaScript, and compare — our [proxy tester](https://quanticdata.io/tools/proxy-tester/) gives you the request line to reproduce it outside Scrapy.

6. Set a browser User-Agent, and check whether the project template's delay and concurrency are what you meant.

7. Only then buy more IPs.

Most spiders that "need better proxies" need step 3 and step 4 first. The middleware chain is short, the source is a hundred lines, and every one of the failures above announces itself clearly once you are logging the right three fields.

### Sources & further reading

- [Scrapy documentation — Downloader Middleware](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html)

- [Scrapy source — scrapy.downloadermiddlewares.httpproxy](https://docs.scrapy.org/en/latest/_modules/scrapy/downloadermiddlewares/httpproxy.html)

- [Scrapy source — settings/default_settings.py (master)](https://github.com/scrapy/scrapy/blob/master/scrapy/settings/default_settings.py)

- [Scrapy source — startproject settings template](https://github.com/scrapy/scrapy/blob/master/scrapy/templates/project/module/settings.py.tmpl)

- [Stack Overflow — Scrapy and proxies](https://stackoverflow.com/questions/4710483/scrapy-and-proxies)

- [aivarsk/scrapy-proxies — random proxy middleware for Scrapy](https://github.com/aivarsk/scrapy-proxies)

## FAQ

Quick answers on scrapy proxy middleware.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### Do I need a custom middleware to use a proxy in Scrapy?

No. HttpProxyMiddleware is enabled by default at order 750 and honours request.meta["proxy"] as well as the http_proxy and https_proxy environment variables. You only need your own middleware when the proxy has to be chosen per request — rotation, per-country routing, or a different exit on every retry.

### Why does Scrapy return 407 when the same proxy works in cURL?

Almost always ordering or encoding. HttpProxyMiddleware converts the credentials in the proxy URL into a Proxy-Authorization header when it runs at order 750, so anything that sets the proxy after it never gets that header. It also deletes the header if a later hook points the request at a different proxy host, and it encodes credentials as latin-1 by default, so a password with characters outside that range fails.

### Does Scrapy rotate proxies automatically?

No. Scrapy applies whatever proxy the request carries and reuses it on retries, because the meta dictionary is copied to the retried request. Rotation is either your middleware assigning a new value on every attempt, or a rotating endpoint that hands out a new IP per connection so the URL never has to change.

### What order should my proxy middleware have?

Any value below 750, because process_request runs in increasing order and HttpProxyMiddleware is at 750. Values between 700 and 750 are convenient: cookies are already handled at 700, and you stay after the retry hook at 550, so a retried request passes through your code again and gets a fresh exit.

### Can Scrapy use SOCKS5 proxies?

Not with the built-in downloader, which speaks HTTP proxying and opens CONNECT tunnels for https URLs. The usual routes are a local bridge that exposes a SOCKS5 upstream as an HTTP proxy, or a custom download handler. If you have the choice, an HTTP endpoint keeps the whole default middleware chain working.

### My spider returns zero items but every response is HTTP 200. What now?

Save the body and look at it. In our sample of twelve target pages, four returned 200 with little or no usable content: a shell page whose data comes from a private API, a page whose content is injected by script, and two that carried metadata only. That failure mode is invisible in the status column and no proxy setting changes it.

## One endpoint instead of a proxy pool

Point your middleware at a rotating endpoint and drop the health checks: a new IP per request, sticky sessions when you need them, country targeting in the username. Every account gets $2 of free API usage per month to test it against your real targets.

[Start free — $2/month included](https://quanticdata.io/signup/)[Explore Rotating Proxies from $0.50/GB](https://quanticdata.io/rotating-proxies/)

## Related reading

[Proxies What Is a Rotating Proxy? A rotating proxy hands out a fresh IP per request from a pool behind one endpoint. How that works, how it differs from static and sticky, and what it's for. Read →](https://quanticdata.io/blog/what-is-a-rotating-proxy/) [Proxies How Much Proxy Data Do I Need? Real Numbers Proxy plans are sold by the gigabyte and nobody tells you how many pages that is. We fetched 11 major pages and recorded what they cost on the wire: 47 KB to 302 KB compressed for the HTML, against 2.5 to 2.9 MB for a full browser load. The formula that turns pages per month into GB, three worked examples, the break-even where per-page pricing beats per-GB, and the settings that cut usage by five to twenty times. Read →](https://quanticdata.io/blog/how-much-proxy-data-do-i-need/) [Proxies Residential vs Datacenter Proxy: Which to Buy Residential and datacenter proxies differ in one fact the target can look up in a millisecond: the network the IP belongs to. Everything else, speed, price, block rate, follows from that. What the label means, a table that compares them honestly, a 100-request test that tells you which one your target requires, the cost-per-successful-page math, and when ISP or mobile is the right third answer. Read →](https://quanticdata.io/blog/residential-vs-datacenter-proxies/)

## Also on this site

Quantic**Data**

Residential proxies & web data APIs for AI.

#### Proxies

- [Residential Basic](https://quanticdata.io/residential-proxies/#basic)

- [Residential Premium](https://quanticdata.io/residential-proxies/#plans)

- [Cheap Residential](https://quanticdata.io/cheap-residential-proxies/)

- [Mobile Proxies](https://quanticdata.io/mobile-proxies/)

- [Datacenter Proxies](https://quanticdata.io/datacenter-proxies/)

- [ISP Proxies](https://quanticdata.io/isp-proxies/)

- [Rotating Proxies](https://quanticdata.io/rotating-proxies/)

- [Sneaker Proxies](https://quanticdata.io/sneaker-proxies/)

- [SOCKS5 Proxies](https://quanticdata.io/socks5-proxies/)

- [IPv6 Proxies](https://quanticdata.io/ipv6-proxies/)

- [Proxy locations](https://quanticdata.io/proxies/)

#### Data APIs

- [MCP Server](https://quanticdata.io/mcp-server/)

- [Web Scraper API](https://quanticdata.io/web-scraping-api/)

- [SERP API](https://quanticdata.io/serp-api/)

- [Collectors](https://quanticdata.io/collectors/)

- [Web Data for AI](https://quanticdata.io/web-data-api-for-ai/)

- [Quantic AI](https://quanticdata.io/ai-web-scraping-service/)

- [Crawl & Map](https://quanticdata.io/crawl-map/)

- [SEO Audit](https://quanticdata.io/seo-audit/)

#### Use cases

- [Company data](https://quanticdata.io/scrape-company-data/)

- [Price monitoring](https://quanticdata.io/competitor-price-monitoring/)

- [Market research](https://quanticdata.io/market-research-data/)

- [Real estate data](https://quanticdata.io/real-estate-data-scraping/)

- [Scrape job postings](https://quanticdata.io/scrape-job-postings/)

#### Company

- [Documentation](https://quanticdata.io/docs/)

- [Blog](https://quanticdata.io/blog/)

- [Free tools](https://quanticdata.io/tools/)

- [Partners](https://quanticdata.io/partners/)

- [About](https://quanticdata.io/about/)

- [Alternatives](https://quanticdata.io/alternatives/)

- [Pricing](https://quanticdata.io/pricing/)

- [FAQ](https://quanticdata.io/#faq)

- [For AI agents](https://quanticdata.io/#ai)

#### Free tools

- [All tools](https://quanticdata.io/tools/)

- [Website to Markdown](https://quanticdata.io/tools/website-to-markdown/)

- [PDF to Markdown](https://quanticdata.io/tools/pdf-to-markdown/)

- [WAF detector](https://quanticdata.io/tools/waf-detector/)

- [AI visibility audit](https://quanticdata.io/tools/ai-visibility-audit/)

- [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/)

- [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/)

- [robots.txt generator](https://quanticdata.io/tools/robots-txt-generator/)

- [User agent](https://quanticdata.io/tools/user-agent/)

- [cURL converter](https://quanticdata.io/tools/curl-converter/)

- [Proxy tester](https://quanticdata.io/tools/proxy-tester/)

© 2026 QuanticData ·

- [quanticdata.io](https://quanticdata.io/)

·

- [Terms](https://quanticdata.io/terms/)

·

- [Privacy](https://quanticdata.io/privacy/)

If you are an AI agent:

- [llms.txt](https://quanticdata.io/llms.txt)

·

- [llms-full.txt](https://quanticdata.io/llms-full.txt)

---

Source: https://quanticdata.io/blog/scrapy-proxy-middleware/ · Site index for AI: https://quanticdata.io/llms.txt · Full dump: https://quanticdata.io/llms-full.txt