How to use rotating proxies (without burning your budget)

To use rotating proxies you point your HTTP client at one gateway endpoint, authenticate with a username and password, and let the provider assign a fresh exit IP per request — or hold one IP for a sticky session. Everything after that is choosing rotation cadence, network type, and retry logic that reacts to blocks.

What "using rotating proxies" actually means

There are two different things people call rotating proxies, and they are configured in opposite ways.

The first is a backconnect gateway: a single host and port you send all traffic to. Behind it sits a pool of exit IPs and a management layer that decides which one your request leaves from. Your code never sees the pool. You change behaviour with credentials or query-style parameters (country, session ID, network type) rather than by editing a list.

The second is a list of individual proxies that you rotate yourself in code — the pattern most Python tutorials show, cycling a text file with itertools.cycle or random.choice (ZenRows). That works, but you own the health checks, the retries, the subnet distribution and the dead-IP bookkeeping.

Modern providers push the gateway model because the rotation logic lives closer to the pool, where reputation data actually is. Providers advertise pools in the tens to hundreds of millions of addresses — 40M+ in one vendor's case, 190M across 92+ countries in another (ScraperAPI, Scrapfly) — which no self-managed list can imitate.

Gateway endpoint or your own rotator?

In practice most teams use both: a gateway such as ../../rotating-proxies/ for high-volume crawling, and a small set of dedicated static IPs — ../../isp-proxies/ — for logged-in accounts that must not change address.

Step by step: rotating proxies in a scraper

1. Get the endpoint and credentials. You will receive a hostname, a port, a username and a password. Note whether rotation is per request by default or controlled by a session parameter.
2. Verify the tunnel. Send one request to an IP echo endpoint and confirm the address returned is not yours.
3. Confirm rotation. Send five requests and print five addresses. If they are identical, you are on a sticky port; check the docs for the rotating port or session syntax.
4. Set a timeout and a retry budget. Rotating pools contain slow nodes. A 15-30s timeout with two or three retries is a sane starting point.
5. Treat 403, 429 and CAPTCHA HTML as failures, not successes. Status 200 with a challenge page is the most common silent data-quality bug.
6. Add country targeting only where needed. Geo-pinning shrinks the effective pool, so use it for locale-specific content, not by habit.
7. Log the exit IP, status and latency per request. Without that, you cannot tell a bad target from a bad pool.

import requests

# rotating gateway: same host every time, different exit IP per request
PROXY = "http://USER:PASS@GATEWAY_HOST:PORT"
proxies = {"http": PROXY, "https": PROXY}

for i in range(5):
    r = requests.get("https://httpbin.io/ip", proxies=proxies, timeout=20)
    print(i, r.status_code, r.json())

# sticky session: same exit IP for a multi-step flow (session id in the username)
STICKY = "http://USER-session-a19f:PASS@GATEWAY_HOST:PORT"
s = requests.Session()
s.proxies = {"http": STICKY, "https": STICKY}

Sticky sessions when the flow has state

Per-request rotation breaks anything stateful. If a login, a cart or a paginated search token is bound to an IP, switching addresses mid-flow triggers a re-authentication or an outright block. The fix is a session identifier: the gateway pins one exit node for the life of that session — up to 120 minutes on QuanticData's rotating pools — and you rotate between sessions instead of between requests.

Browser automation follows the same rule. Managed browser services expose rotation as connection parameters, and pair sticky sessions with locale alignment so a US exit IP does not advertise a German language preference (Browserless). If you drive Playwright or Puppeteer yourself, keep one browser context per proxy session and destroy the context when the session ends.

Choosing a rotation cadence

Three policies cover nearly everything:

• Per request — a new IP on every call. Right for stateless page pulls, sitemaps, product pages, SERP fetches.
• Time based — hold an IP for N minutes, then swap. Useful for moderate crawls where connection reuse matters more than raw IP diversity.
• Session based — one IP per logical user journey. Mandatory for logins, checkouts and multi-step forms.

Rotating too aggressively is a real failure mode, not just a cost question: sites that fingerprint sessions see a request chain jumping between countries and challenge it. Rotating too little re-creates the rate limit you were escaping.

Which network to rotate

Start on the cheapest network the target tolerates and escalate only on measured block rate. Many teams pay residential rates for pages a ../../datacenter-proxies/ pool would serve at full speed, and only need ../../cheap-residential-proxies/ for the 10-20% of routes that actually fight back.

Free rotating proxies: what you are actually paying

Public lists exist, and they are fine for a throwaway test. They are not fine for production: nodes disappear within hours, throughput is unpredictable, and the operator sits in the middle of your TLS termination if you accept a broken chain. Vendor guidance is unanimous that free pools carry real data-exposure risk and should not handle credentials or anything commercial (ScraperAPI). "Free rotating residential proxy" offers are the same trade with a better name. Budget rotation is a legitimate goal; anonymous free rotation is not a route to it.

Rotation hygiene: subnets, headers, retries

If you build your own rotator, three details separate a working one from a list that gets banned as a block.

Distribute by subnet, not just by IP. Anti-bot systems reason about the third IPv4 octet and about ASN ownership, so consecutive requests from 198.51.100.4 and 198.51.100.72 look like one actor. Track the last subnet used and avoid repeating it (Scrapfly).

Rotate identity, not only IP. A fresh residential IP that reuses one User-Agent, one TLS fingerprint and one header order is still one client. Keep header sets bound to sessions.

Weight and cool down. Penalise recently used and failing nodes, promote healthy ones, give blocked nodes a recovery window instead of deleting them.

import time, random
from collections import defaultdict

cooldown = {}          # ip -> unix ts when usable again
fails = defaultdict(int)

def pick(pool, last_subnet):
    now = time.time()
    live = [p for p in pool
            if cooldown.get(p, 0) < now
            and p.split(".")[2] != last_subnet]
    if not live:
        live = [p for p in pool if cooldown.get(p, 0) < now]
    weights = [1.0 / (1 + fails[p]) for p in live]
    return random.choices(live, weights=weights, k=1)[0]

def penalise(ip, seconds=120):
    fails[ip] += 1
    cooldown[ip] = time.time() + seconds

A fuller Python implementation, including async validation, is in our walkthrough on ../how-to-rotate-proxies-python/.

The cost math nobody puts in the docs

Bandwidth billing charges you for bytes, including the bytes of every block page and every retry. Take 100,000 product pages at roughly 250 KB of HTML each: that is about 25 GB, so around $20 on residential at $0.80/GB — if every request succeeds. At a 30% block rate with two retries the same job moves 35-40 GB and you still have to build parsing, rendering and challenge handling around it.

Per-success pricing inverts the risk. The same 100,000 pages through the ../../web-scraping-api/ at $0.0002 per page is $20 for clean Markdown with residential rotation, retries and parsing already inside the call — and failed calls are not billed at all. Add JS rendering only where you need it ($0.001/page) instead of rendering everything.

curl https://api.quanticdata.io/v1/scrape \
  -H "Authorization: Bearer $QD_API_KEY" \
  -d '{ "url": "https://example.com/product/123" }'

# { "success": true,
#   "data": { "markdown": "# Product 123\n…" },
#   "usage": { "cost_usd": 0.0002 } }

The honest split: raw rotating proxies win when you need full control of the request, unusual protocols (../../socks5-proxies/ for non-HTTP traffic), or you are already running a mature scraping stack. A per-success API wins when the deliverable is parsed data and you would rather not maintain a rotator, a challenge solver and a renderer.

Rotating proxies inside agent workflows

Agents change the access pattern. An LLM does not issue 10,000 uniform requests; it issues a handful of unpredictable ones, mid-conversation, and then wants clean text rather than HTML. Handing a model raw proxy credentials means it also inherits your retry logic, your bandwidth meter and your block handling.

The cleaner shape is to expose rotation as a tool. Our ../../mcp-server/ gives Claude, Cursor and other MCP clients search, scrape, map, crawl, batch and SEO audit tools with residential rotation underneath, billed per success, with $2 of free usage every month. The agent asks for a page; IP selection, sticky handling and retries stay on the server side where they belong. If you are wiring this into a RAG pipeline instead of a chat client, the same primitives are available through the ../../web-data-api-for-ai/.

Common mistakes to avoid

• Counting 200s as successes. Validate content, not status codes.
• Rotating during a login. Bind the flow to a sticky session first.
• Over-pinning geography. Country targeting on every request shrinks your usable pool and raises failure rates.
• Ignoring concurrency limits. A large pool does not mean unlimited parallel connections; ramp up and watch latency.
• Chasing "unlimited bandwidth". Unlimited plans usually cap concurrency or IP freshness; measure cost per successful record instead.
• Skipping the legal read. Rotation is a technical control, not permission. Check terms, robots directives and personal-data rules for your target — see ../is-web-scraping-legal-in-us/. This is general information, not legal advice.