Golang HTTP Client Proxy: Auth, SOCKS5

Go has no client.SetProxy. The proxy is a property of the Transport, not of the request, and that single fact explains the rest: why the environment variable sometimes does nothing, why credentials go in the URL, why a 407 on an HTTPS target arrives as a bare error string, and why rotating exits per request is four lines rather than a library.

Three ways to set a proxy, and when each is right

The standard library gives you exactly one hook, Transport.Proxy, a function called once per connection with the request. Everything else is a wrapper around it.

The first way is the environment. Go's DefaultTransport uses ProxyFromEnvironment, so an exported variable is picked up with no code change at all:

export HTTPS_PROXY="http://USER:PASSWORD@proxy.example.net:8080"
export HTTP_PROXY="http://USER:PASSWORD@proxy.example.net:8080"
export NO_PROXY="127.0.0.1,localhost,.internal.example.com"
go run ./cmd/crawler

The second way is an explicit client, which is what you want in anything long-lived: it does not depend on how the process was started, and it lets you size the connection pool for scraping instead of for a CLI tool.

proxyURL, err := url.Parse("http://USER:PASSWORD@proxy.example.net:8080")
if err != nil {
    log.Fatal(err)
}

tr := &http.Transport{
    Proxy:               http.ProxyURL(proxyURL),
    MaxIdleConns:        100,
    MaxIdleConnsPerHost: 10,
    TLSHandshakeTimeout: 10 * time.Second,
}
client := &http.Client{Transport: tr, Timeout: 30 * time.Second}

resp, err := client.Get("https://example.com/")

The third way is to reassign http.DefaultTransport. It works, it is one line, and it changes the behaviour of every package in your binary that quietly uses http.Get or http.DefaultClient — including your telemetry and your cloud SDKs. Use it in a throwaway script, never in a service.

Method	Scope	Good for	Trap
HTTP_PROXY / HTTPS_PROXY	Whole process	CLI tools, corporate egress, containers	Read once per process; ignored for localhost
Transport.Proxy with ProxyURL	One client	Services, scrapers, anything with a pool	A second client with no Proxy silently bypasses it
Reassigning DefaultTransport	Whole binary	Scripts	Every dependency inherits it, including SDKs
Your own func(*http.Request)	Per request	Rotation, per-target routing	Each distinct URL opens its own connection pool

Proxy authentication: Basic, and the 407 you never get to read

If the proxy URL carries a userinfo component, the Transport turns it into a Proxy-Authorization: Basic header for you. It does this in both directions: on plain http:// targets the header is added to every request sent down the proxied connection, and on https:// targets it is added to the CONNECT request that opens the tunnel. You do not write that header yourself, and in fact you cannot: a Proxy-Authorization header you set on req.Header travels inside the tunnel to the origin server, where it is meaningless.

Credentials with symbols in them must be percent-encoded, which is the single most common cause of a mystifying 407. Build the URL with url.UserPassword and Go encodes it correctly:

proxy := &url.URL{
    Scheme: "http",
    Host:   "proxy.example.net:8080",
    User:   url.UserPassword(os.Getenv("PROXY_USER"), os.Getenv("PROXY_PASS")),
}

tr := &http.Transport{
    Proxy: http.ProxyURL(proxy),
    OnProxyConnectResponse: func(ctx context.Context, p *url.URL, req *http.Request, res *http.Response) error {
        if res.StatusCode != 200 {
            log.Printf("CONNECT %s -> %s", req.URL.Host, res.Status)
        }
        return nil
    },
}

That hook matters more than it looks. When a CONNECT gets a non-200 answer, the Transport closes the connection and returns an error built from the status text alone — you get Proxy Authentication Required as an error string, with no status code, no headers and no body, because there is no *http.Response to hand back. OnProxyConnectResponse, added in Go 1.20, is the only place where you can see what the proxy actually said. On a plain http:// target there is no tunnel, so the 407 comes back as an ordinary response you can read normally. Our field guide to that status code is in ../how-to-fix-407-proxy-authentication-required/, and the CONNECT-level failures have their own walkthrough in ../err-tunnel-connection-failed/.

SOCKS5 with the standard library, no extra package

You do not need golang.org/x/net/proxy to send HTTP traffic over SOCKS5. Since Go 1.9 the Transport accepts a socks5:// proxy URL directly, and since Go 1.23 it also accepts socks5h:// — with the two treated identically, meaning the hostname is resolved by the proxy and not by your machine. Username and password are read from the same userinfo component as HTTP proxies and negotiated with the username/password auth method.

proxy := &url.URL{
    Scheme: "socks5h",
    Host:   "proxy.example.net:1080",
    User:   url.UserPassword(os.Getenv("PROXY_USER"), os.Getenv("PROXY_PASS")),
}
client := &http.Client{
    Transport: &http.Transport{Proxy: http.ProxyURL(proxy)},
    Timeout:   30 * time.Second,
}

The limit is that net/http only ever asks the SOCKS proxy for a TCP connection. If you need UDP, or you are proxying something that is not HTTP, dial through x/net/proxy yourself. Our SOCKS5 endpoints answer on the same credentials as the HTTP ones, so switching scheme is a one-character change in the code above.

Rotating the proxy per request without leaking connections

Because Proxy is a function of the request, rotation needs no third-party middleware. Return a different URL per call and keep one client for the whole program:

var gateways = []string{
    "gw-1.example.net:8080",
    "gw-2.example.net:8080",
    "gw-3.example.net:8080",
}
var counter uint64

func rotate(req *http.Request) (*url.URL, error) {
    i := atomic.AddUint64(&counter, 1)
    return &url.URL{
        Scheme: "http",
        Host:   gateways[i%uint64(len(gateways))],
        User:   url.UserPassword(os.Getenv("PROXY_USER"), os.Getenv("PROXY_PASS")),
    }, nil
}

client := &http.Client{
    Transport: &http.Transport{
        Proxy:               rotate,
        MaxIdleConns:        200,
        MaxIdleConnsPerHost: 4,
    },
    Timeout: 30 * time.Second,
}

Two things follow from how the Transport caches connections, and both bite people who build this in a hurry. The idle-connection pool is keyed by the triple of proxy URL, target scheme and target address, so every distinct proxy string you return gets its own pool — and MaxIdleConnsPerHost defaults to 2. With sticky-session usernames, where each session is a different username on the same gateway, you are minting a new pool entry per session and then abandoning it; call client.CloseIdleConnections() when you retire a batch of sessions, or keep the number of live sessions bounded on purpose.

The second is the mistake underneath most "my Go scraper eats file descriptors" reports: building a fresh http.Transport inside the request loop. A Transport is a connection pool, it is designed to be created once and shared across goroutines, and one per request means zero reuse plus a slow leak. One client, one Transport, a rotating Proxy func. If you want the rotation handled on the far side instead, our rotating endpoint gives a new exit per request behind a single host and a single credential, so the Go side stays a two-line ProxyURL.

HTTP_PROXY, NO_PROXY and the environment that is read exactly once

Four behaviours of ProxyFromEnvironment account for nearly every "the environment variable does nothing" thread, and all four are in the standard library source rather than in folklore.

• The environment is read once per process. The parsed configuration is cached behind a sync.Once on first use, so calling os.Setenv("HTTP_PROXY", ...) after your program's first HTTP request has no effect whatsoever. Set it before you start, or set Transport.Proxy explicitly.
• Lowercase wins. When both http_proxy and HTTP_PROXY are present, the lowercase one takes precedence — the reverse of what most people assume.
• localhost and loopback are never proxied. If the request host is localhost or a loopback address, with or without a port, Go returns no proxy as a documented special case. Testing your proxy setup against a local server will always look like the proxy is being ignored, because it is.
• CGI processes ignore HTTP_PROXY. If REQUEST_METHOD is set in the environment, Go treats the process as a CGI handler and refuses to apply HTTP_PROXY, because a remote client could otherwise set it through the Proxy header. That is the httpoxy vulnerability, closed at the library level.

NO_PROXY is a comma-separated list, and its grammar is more capable than the one-host-per-entry version most people use. Each entry can be a domain, an IP prefix, a CIDR block, or a single asterisk.

NO_PROXY entry	Matches	Does not match
foo.com	foo.com and bar.foo.com	notfoo.com
.y.com	x.y.com	y.com
10.0.0.0/8	Any address in that block	Hostnames that resolve into it
1.2.3.4:80	That address on that port only	1.2.3.4:443
*	Everything: proxying is disabled	—

The errors Go returns, and what each one means

Go's proxy errors are terse, and two of them are terse in a way that hides the status code. This table is the translation.

What you see	What happened	Where to look
proxyconnect tcp: dial tcp …: i/o timeout	The TCP connection to the proxy itself never opened	Port, firewall, egress rules. The target is not involved yet
proxyconnect tcp: lookup …: no such host	The proxy hostname does not resolve	Copy the host from your dashboard; check DNS before blaming auth
Proxy Authentication Required	A 407 on the CONNECT, returned as a bare status text	Percent-encoding of the password, IP allowlist, exhausted plan
Forbidden / Bad Gateway (bare, no code)	Non-200 CONNECT: the proxy refused or could not reach the target	OnProxyConnectResponse to see the real status and headers
EOF immediately after CONNECT	The proxy accepted the TCP connection then closed it	Wrong port for the protocol, or a plan with no bandwidth left
tls: failed to verify certificate	The TLS handshake is with something that is not your target	An intercepting proxy, or an https:// scheme on an HTTP proxy port
context deadline exceeded (Client.Timeout exceeded while awaiting headers)	The client deadline covered CONNECT, TLS and the wait for headers	Raise Client.Timeout, or use per-attempt contexts

One nuance worth internalising: http.Client.Timeout is a deadline on the whole exchange, including the proxy handshake and the reading of the body. A 30-second timeout on a residential exit is not generous; it is the budget for a CONNECT, a TLS handshake, a request and a full HTML download over a home line.

What a plain Go client actually gets: 12 pages measured

The interesting question is not how to attach a proxy. It is how far a proxied HTTP client gets before you need a browser. On 14 September 2026, between 05:33 and 05:40 UTC, we fetched twelve public listing pages twice each through US residential exits — once as a pure HTTP client with no JavaScript, which is what net/http is, and once fully rendered — and compared the word counts. Each pair was one call to our own SEO audit endpoint, which exists precisely to answer this question.

Page	Words, no JS	Words, rendered	What a Go client gets
en.wikipedia.org (article)	6,306	6,383	Everything
etsy.com (category)	5,046	5,188	Everything
booking.com (city page)	4,437	4,409	Everything
github.com/trending	2,248	2,248	Everything, byte for byte
news.ycombinator.com	603	603	Everything
producthunt.com	562	676	Most of it; 83% of the words
walmart.com (category)	823	2,838	Copy and FAQ, not the product grid
zillow.com (city listings)	137	403 on render	Thin page, but with an ItemList in the structured data
target.com (category)	27	64	An empty shell: no title, no canonical
ebay.com (category)	403	403	Nothing
indeed.com (job search)	403	403	Nothing
yelp.com (search)	403	403	Nothing

Six of twelve returned the complete page to a client that cannot run a line of JavaScript. Three returned something partial — Walmart's category page gives you 29% of its rendered words, and the missing 71% is the product grid, which is the only part anyone wants. Three refused outright with a 403 before any HTML existed to parse.

Two honest caveats. First, those fetches used a browser-grade TLS fingerprint; Go's own ClientHello from crypto/tls is distinctive enough to be classified on sight, so a stock net/http client will do the same or worse on the hostile half, not better. Second, the Zillow row is a useful inversion: the plain HTTP fetch succeeded with a usable ItemList in the structured data while the rendered fetch drew a captcha. More machinery is not automatically more access, and the structured data block is often the shortest path to the fields you were going to parse out of the DOM anyway.

Bandwidth, and when to stop writing client code

Residential traffic is metered by the gigabyte, and HTML is heavier than it looks once images, fonts and analytics beacons are counted — which is why a Go scraper that only ever reads resp.Body is cheaper per page than a headless browser doing the same job. We put real numbers on that in ../how-much-proxy-data-do-i-need/; the short version is that page weight, not request count, is what your bill tracks. Our residential pool starts at $0.80/GB on the Basic tier with no subscription, and rotating exits start at $0.50/GB.

Write the client yourself for the six-out-of-twelve half: those pages are a Transport, a rotating Proxy func and golang.org/x/net/html, and nothing more. For the other half — the 403s, the shells, the grids that only exist after JavaScript — the work is no longer Go code, it is fingerprinting and rendering maintenance. That is what our scraping API is for: one HTTP call from the same http.Client you already built, clean Markdown or structured JSON back, from $0.0002 per page and $0.001 with rendering, billed only on success. Every account gets $2 of free API usage a month, which is several thousand pages before you decide anything.