Scrapy Proxy Middleware: What Actually Runs

Scrapy already has a proxy middleware. HttpProxyMiddleware sits at order 750 in the default chain and is enabled out of the box, so you never write one just to use a proxy — you write one to decide which proxy each request gets. Reading its source explains most 407 errors and most silent rotation bugs.

The two ways to set a proxy, and what each one costs you

Every request that carries a value in request.meta["proxy"] goes through that proxy. Nothing else is required, and no setting has to be switched on: HTTPPROXY_ENABLED defaults to True in Scrapy's own default settings.

import scrapy

class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://httpbin.org/ip"]

    def start_requests(self):
        proxy = "http://USER-country-us:PASS@rotate.quanticdata.io:8000"
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": proxy}, callback=self.parse)

    def parse(self, response):
        self.logger.info("exit ip: %s", response.text.strip())

The second route is the environment. When no request carries a proxy in its meta, the middleware falls back to what Python's urllib.request.getproxies() reports, which means the standard http_proxy and https_proxy variables work with no code at all, and no_proxy is honoured for http and https URLs. That is the fastest way to test whether a proxy is the problem: run the same spider once with the variables set and once without.

A custom middleware is the third route, and the only one worth the file. Its job is not to enable proxying; it is to answer the question "which exit should this specific request use" on every attempt, including retries.

What HttpProxyMiddleware does with your credentials

This is the part every tutorial skips, and it is the part that produces the support tickets. In process_request the middleware takes the proxy URL, splits the credentials out of it with _parse_proxy, base64-encodes them, and sets a Proxy-Authorization header. The URL that stays in request.meta["proxy"] is the stripped one, without user and password. It also records the proxy those credentials belong to in request.meta["_auth_proxy"].

Three consequences follow directly from that code, and all three bite in production.

• Order matters, and 750 is the deadline. Middlewares run in increasing order, so a component that sets meta["proxy"] at order 800 sets it after HttpProxyMiddleware has already run. The credentials never become a header, and the target answers 407.
• Changing the proxy drops the header. If a later hook rewrites meta["proxy"] to a different host than the one in _auth_proxy, the middleware deletes Proxy-Authorization rather than sending the old credentials to a new server. Rotation code that assigns a bare host and expects the previous credentials to follow silently produces unauthenticated requests.
• Credentials are encoded as latin-1. HTTPPROXY_AUTH_ENCODING defaults to "latin-1", so a generated password containing characters outside that range raises on encode. If a password works in cURL and fails in Scrapy, this is usually why.

The practical rule is short: always assign the full URL with credentials in it, always before order 750. Our walkthrough of 407 Proxy Authentication Required covers the other half of these failures, the ones that come from the provider side.

A rotation middleware that survives retries

The skeleton copied around the web looks like this: check whether the request already has a proxy, and if not, set one from settings. It works exactly once. When RetryMiddleware re-schedules a failed request, the meta dictionary is carried over, so the guard sees an existing proxy and keeps the exit that just failed. You retry a dead IP twice and log three failures.

import random

class RotatingProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        proxies = crawler.settings.getlist("PROXY_POOL")
        if not proxies:
            from scrapy.exceptions import NotConfigured
            raise NotConfigured("PROXY_POOL is empty")
        return cls(proxies)

    def process_request(self, request, spider):
        # assign on every attempt, retries included, credentials included
        request.meta["proxy"] = random.choice(self.proxies)

    def process_response(self, request, response, spider):
        if response.status in (403, 407, 429):
            spider.logger.warning(
                "status %s via %s", response.status, request.meta.get("proxy")
            )
        return response

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 740,
}
PROXY_POOL = [
    "http://USER-country-us:PASS@rotate.quanticdata.io:8000",
    "http://USER-country-de:PASS@rotate.quanticdata.io:8000",
]

Two details make this different from the usual version. It assigns unconditionally, so a retry gets a fresh choice. And it logs the exit alongside the status, which is the only way to find out later whether one subnet is carrying all your blocks.

If the pool is a list of single IPs you also need health tracking, a cooldown, and a refresh job. A rotating endpoint removes that work: one host, a new IP per request, and a sticky session when you need one by appending a session identifier to the username. In that case PROXY_POOL holds one entry per country, not per IP, and the pool never goes stale. The same pattern in plain Python is in our guide to rotating proxies in Python.

Empty items are not always a proxy problem

Before you buy more IPs, find out which failure you have. On 2026-09-04 we fetched twelve pages a Scrapy tutorial would plausibly target, twice each, from a United States residential exit: once as a plain HTTP client with no JavaScript execution — which is exactly what Scrapy is — and once fully rendered in a browser. The word counts are what a text extractor found in each view.

Target page	No-JS view (what Scrapy sees)	Rendered view	Diagnosis
books.toscrape.com	200, 308 words	200, 308 words	Complete in the HTML
news.ycombinator.com	200, 606 words	200, 606 words	Complete in the HTML
amazon.com search	200, 4,103 words	200, 4,389 words	Complete in the HTML
indeed.com jobs	200, 1,832 words	403 interstitial, 70 words	Plain HTTP won, the browser was challenged
walmart.com search	200, 890 words	200, 4,148 words	Partial: about a fifth of the content
zillow.com listings	200, 138 words	403 captcha page	Metadata only, no listings
target.com search	200, 27 words	200, 29 words	Shell page, data behind a private API
quotes.toscrape.com/js	200, 14 words	200, 227 words	Content injected by script
ebay.com search	403	200, 4,918 words	Blocked without a browser
etsy.com search	403	403	Blocked both ways
yelp.com search	403	403	Blocked both ways
g2.com category	403	403	Blocked both ways

Four of the twelve gave a plain HTTP client everything. Four returned a hard 403. The interesting four are the ones in between: they answered HTTP 200 and no data. A spider that only counts non-200 responses sees four failures and quietly mis-parses four more, and no quantity of proxies changes that column. Two rows also cut the other way — on Indeed and Zillow the headless browser was challenged while the plain fetch went through, so "render it" is not a general fix either.

This is one sample per site, on one day, through one exit; treat it as a triage method rather than a scoreboard. The method is what transfers: fetch twice, compare, then decide. The measurement above came from our own audit endpoint, which does exactly that double fetch and returns the diff.

The equivalent inside a spider is three lines and a saved body:

def parse(self, response):
    rows = response.css("div.product")
    if not rows:
        self.logger.error(
            "no rows: status=%s bytes=%s proxy=%s",
            response.status, len(response.body), response.meta.get("proxy"),
        )
        with open("debug-%s.html" % response.status, "wb") as fh:
            fh.write(response.body)
        return
    for row in rows:
        yield {"title": row.css("h3 a::attr(title)").get()}

Open that file. If it holds a consent wall or a challenge page, it is a block — see 403 Forbidden while scraping. If it holds a valid page with an empty container, the data arrives by script and you need a rendering step such as scrapy-playwright or the site's own JSON endpoint. If it holds a rate-limit notice, read 429 Too Many Requests instead of adding IPs.

Retry settings quietly decide your bandwidth bill

Scrapy's defaults retry on [500, 502, 503, 504, 522, 524, 408, 429] with RETRY_TIMES = 2, which is three attempts in total. Two things about that list matter when you are paying per gigabyte.

First, 403 is not in it. A blocked crawl looks fast and cheap in the stats and produces nothing. Adding 403 to RETRY_HTTP_CODES only helps if the retry uses a different exit — which is precisely what the unconditional assignment in the middleware above guarantees. Add it without rotating, and you buy the same ban three times.

Second, tunnel failures are already covered: scrapy.core.downloader.handlers.http11.TunnelError is in the default RETRY_EXCEPTIONS, so a proxy that refuses the CONNECT is retried like any transport error. What is not covered is a proxy that answers with a captive portal at HTTP 200.

# settings.py — retry a ban only when the retry will change exit
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429, 403]
RETRY_TIMES = 2
DOWNLOAD_TIMEOUT = 30

The arithmetic is worth doing before the invoice arrives. Suppose your responses average 60 KB and you crawl 100,000 pages: that is roughly 6 GB. If a fifth of them fail twice before succeeding, you are at about 8.4 GB for the same 100,000 items, a 40 percent surcharge that appears nowhere in your item count. At the entry price for our rotating pool, $0.50/GB, that is the difference between $3.00 and $4.20 for the run — small here, linear as you scale, and the reason the retry policy belongs in the cost model. Our note on how much proxy data you need works through sizing from the other direction. Lower DOWNLOAD_TIMEOUT from its 180-second default too: a hung connection through a bad exit holds a concurrency slot for three minutes.

Your project defaults are not Scrapy's defaults

The settings file that scrapy startproject writes is deliberately more conservative than the library, and people tune the wrong file for days without noticing.

Setting	Library default	What startproject writes
ROBOTSTXT_OBEY	False	True
CONCURRENT_REQUESTS_PER_DOMAIN	8	1
DOWNLOAD_DELAY	0	1
USER_AGENT	Scrapy/x.y (+https://scrapy.org)	commented out, so the library value is sent

Two of those explain complaints that get blamed on proxies. A spider crawling at one request per second per domain is not slow because of the network. And a request that announces itself as Scrapy in the User-Agent header is filtered by ordinary bot rules before your exit IP is even considered — the cheapest fix in this entire article is setting a real browser User-Agent. Leaving ROBOTSTXT_OBEY on is a policy decision rather than a bug, but know that it is on: it is why some start URLs are dropped before a single proxy byte is spent.

When a proxy is the wrong tool

Proxies solve one class of problem: the target refuses your network identity. They do not solve rendering, they do not solve challenge pages, and they do not solve a site that ships an empty shell and fetches its data from a private API.

• Content arrives by script. Render it or call the underlying endpoint. Scrapy stays the scheduler either way.
• Blocked with and without a browser. Four of our twelve sample pages were in this state. That is a full anti-bot stack, and beating it is a maintenance commitment, not a settings change.
• You need one field from many pages. Paying per successful page can be cheaper than paying per gigabyte plus the engineering to keep a pool healthy.

Our scraping API starts at $0.0002 per page and $0.001 with JavaScript rendering, and it returns Markdown or structured JSON, so the parsing step disappears too. Called from inside a spider it is an ordinary request, which means your pipelines, feeds and scheduling stay exactly where they are:

yield scrapy.Request(
    "https://api.quanticdata.io/v1/scrape",
    method="POST",
    headers={"Authorization": "Bearer YOUR_KEY", "Content-Type": "application/json"},
    body=json.dumps({"url": target, "format": "markdown", "render": True}),
    callback=self.parse_markdown,
)

The honest split: use residential proxies when the site serves complete HTML to a plain client and only your network identity is in the way, and use a success-priced API when the page needs a browser or an anti-bot stack stands in front of it.

A ten-minute checklist

1. Set meta["proxy"] with the credentials in the URL, from a middleware ordered under 750.
2. Assign on every attempt, not only when the key is missing, so retries change exit.
3. Log status, body length and exit together, on every response.
4. Save the body of the first empty parse and read it before touching settings.
5. Fetch one target twice, with and without JavaScript, and compare — our proxy tester gives you the request line to reproduce it outside Scrapy.
6. Set a browser User-Agent, and check whether the project template's delay and concurrency are what you meant.
7. Only then buy more IPs.

Most spiders that "need better proxies" need step 3 and step 4 first. The middleware chain is short, the source is a hundred lines, and every one of the failures above announces itself clearly once you are logging the right three fields.