How to use a proxy with Python requests: syntax, auth, rotation and pitfalls

To use a proxy with Python requests, pass a proxies dictionary mapping the URL scheme to your proxy URL. That is the whole basic idea — one dict, applied per request or set once on a session. The details that trip people up are authentication, HTTPS coverage, SOCKS5 support and rotation, and they are the difference between a scraper that works and one that silently leaks your real IP.

The basic syntax

The proxies dict keys are schemes (http and https), and the values are the proxy URL to use for each:

import requests

proxies = {
    "http":  "http://pr.quanticdata.io:7777",
    "https": "http://pr.quanticdata.io:7777",
}
r = requests.get("https://ipinfo.io/ip", proxies=proxies)
print(r.text)   # the proxy's exit IP, not yours

Note the values are the same proxy for both schemes here — the key is the scheme of the target URL, not the proxy. This is the number-one confusion: people set only http, then hit an https:// site and wonder why the proxy is ignored. Almost every real target is HTTPS, so you almost always need the https key.

Authenticated proxies

Paid proxies require credentials, which go inline in the proxy URL as user:pass@host:port:

proxy = "http://USER:[email protected]:7777"
proxies = {"http": proxy, "https": proxy}
r = requests.get("https://example.com", proxies=proxies)

If your password contains special characters (@, :, /), URL-encode them with urllib.parse.quote or the credentials will be parsed wrong and you'll get a 407. This is the second most common failure after the HTTPS-key mistake.

SOCKS5 proxies

For SOCKS5 (useful for non-HTTP traffic or stricter anonymity), install the extra and use the socks5 scheme in the proxy URL:

pip install "requests[socks]"

proxy = "socks5://USER:PASS@host:1080"
proxies = {"http": proxy, "https": proxy}

Use socks5h:// (with the h) to have the proxy resolve DNS, which prevents your machine's DNS lookups from leaking the domains you're visiting — worth it for anything privacy-sensitive.

Reuse and rotation for scraping

For more than one request, use a Session so the proxy (and connection pooling) applies to every call without repeating yourself:

s = requests.Session()
s.proxies = proxies
for url in urls:
    r = s.get(url)   # all go through the proxy

Rotation is where scraping gets serious. If you point the session at a rotating proxy endpoint, the provider assigns a fresh IP per request automatically — no rotation logic in your code. That is far cleaner than maintaining and health-checking your own IP list, which is the DIY approach that eats afternoons. For defended targets use residential IPs; for high-volume calls to tolerant targets, datacenter is cheaper. We cover the setup for rotating endpoints in how to set up rotating proxies.

The pitfalls that silently break it

Setting verify=False to make SSL errors go away is a common mistake — it disables certificate checking entirely and opens you to interception. Fix the actual cause (usually the proxy's CA) instead.

Setting a timeout and handling proxy failures

One habit that separates a toy script from a working scraper: always set a timeout, and handle the proxy-specific errors. A dead or slow proxy will otherwise hang your request indefinitely, and at scale one bad exit IP shouldn't kill the whole run. Wrap each call so a proxy failure is caught, logged and retried rather than crashing the loop:

from requests.exceptions import ProxyError, ConnectTimeout

try:
    r = s.get(url, timeout=15)
    r.raise_for_status()
except (ProxyError, ConnectTimeout):
    # bad exit IP or slow proxy — retry (a rotating
    # endpoint hands you a different IP on the next call)
    r = s.get(url, timeout=15)

With a rotating endpoint this retry pattern is especially effective, because the retry automatically comes from a fresh IP — a transient block or a dead exit self-heals on the next attempt. Without a timeout, a single unresponsive proxy can stall a scrape for minutes; fifteen seconds is a sane default for most targets.

When requests isn't enough

A proxy solves the IP problem, but requests only fetches raw HTML — it doesn't run JavaScript, so client-rendered content is invisible, and you still write your own parsing. When a target renders in the browser or blocks harder than proxies alone can beat, point requests at a scraping API instead of the site directly: one POST with the URL, and you get back clean Markdown or structured JSON with the proxy, rendering and anti-block handled server-side. Your Python stays exactly this simple — requests.post(api, json={"url": target}) — but the fragile parts move off your machine. Keep raw requests+proxy for simple static targets; reach for the API when the page fights back.