Best Buy Proxies: 285 Words, Never Render

Best Buy proxies are the cleanest case in this series for spending less. On 26 September 2026 bestbuy.com gave a plain HTTP client behind a United States residential exit 285 words; a headless browser got 284. On 28 September we weighed the two fetches: 506,484 bytes in 5.1 seconds without JavaScript, 4,968,555 bytes in 45.9 seconds with it. The browser costs 9.8 times the bytes and 9 times the wait to lose a word. Target, which sells the same categories, multiplies by 6.8 when rendered; Best Buy divides by nothing. The setting is residential, plain HTTP, a US exit, and never the browser, and the one thing the exit country changes is whether you get the store or a list of countries.

Rendering Best Buy buys minus one word

We audited the home with the tool in our MCP server, which fetches once as a pure HTTP bot and once fully rendered and counts the words in each view, then weighed both fetches two days later.

Three fetches from the same exit on two days, 285, 284 and 288 words: the page is stable, the browser adds nothing, and the plain response is the real home, with the title "Best Buy | Official Online Store | Shop Now & Save", the meta description and a canonical of https://www.bestbuy.com/. What it does not have is also worth knowing: no h1, no JSON-LD and no robots meta tag, so a classifier that treats missing structured data as a sign of a block page will call this real page blocked. Classify on the canonical and the title.

The reason the browser adds nothing is in the response headers of the plain fetch. Best Buy serves its home HTML with cache-control: public, max-age=286, s-maxage=300 and x-cached-response: true: the page is a pre-built document that Akamai hands out from cache for five minutes at a time, brotli-compressed, in one attempt. The product tiles a shopper sees are loaded afterwards by scripts that call Best Buy's own APIs, and those are the paths its robots.txt closes. So the rendered view has the same words as the plain one because the words are the shell, and the data was never going to be in the DOM of the home page either way.

Ten times the bytes for the same page

Row four is the budget line. The browser downloaded 4,968,555 bytes and held a residential session for 45.9 seconds; the plain client downloaded 506,484 bytes in 5.1 seconds. On a per-gigabyte network the render is 9.8 times the bill for the same 285 words, and on a pool it is nine times the occupancy. Of the 52 targets we measured in September, Best Buy is one of 26 where rendering is useless or worse, and it is the one with the widest gap between what the browser costs and what it returns: 4.5 MB of scripts, fonts and images for the page you already had.

The cache header gives you a second saving. With max-age=286, polling the same URL faster than every five minutes returns the same cached document from the edge; a monitor that fetches a Best Buy page every 60 seconds pays five times for one change. Set the interval to the cache and spend the difference on more pages. For what a 429 looks like when you do exceed a limit, and how to back off without stalling a pool, 429 Too Many Requests while scraping covers it; we did not meet one in this measurement.

From Canada the same URL is a country chooser

Row five is the geo point. A Canadian residential exit, sending the same request to https://www.bestbuy.com/, was answered with a different page: title "Best Buy International: Select your Country - Best Buy", h1 "Choose a country.", 103 words, no canonical. Best Buy operates separate stores per country, and rather than redirecting a foreign visitor it serves a chooser at the US address. A pool that includes Canadian exits will therefore return, for a fraction of its requests, a 200 OK page with no products and no canonical, and a parser that keys on status will store it as a valid empty catalogue.

So pin country: "us" and check for the canonical. The chooser is not an error; it is the site telling you that the exit was outside the United States. If your job is the Canadian store, that is bestbuy.ca, a different host with its own pages and its own terms, and a Canadian exit is the right one for it.

Best Buy's robots.txt is written in buckets, and one bucket blocks nothing

bestbuy.com/robots.txt is 15,016 bytes, last modified on 17 September 2026, and it announces itself in the first line as the official enterprise robots.txt, "optimized for search engines, AI discovery, ads, and crawl control". It is organised in five commented buckets. Bucket 1 lists 12 AI search and discovery agents, OAI-SearchBot, ChatGPT-User, Perplexity-User, GPTBot, Claude-User, Claudebot, PerplexityBot, Applebot, Google-Extended, Gemini-Deep-Research, AmazonBot and one misspelled Gemini entry, and allows them /site/ and /discover-learn/ while closing the API, cart, pricing, availability and fulfilment paths. Bucket 3 gives the same core exclusions to Googlebot, Bingbot, DuckDuckBot, Slurp and the wildcard. Eleven sitemaps on sitemaps.bestbuy.com close the file.

Bucket 2 is labelled an AI training and data ingestion firewall and names GPTBot-Web, CCBot, Anthropic-AI, Claude-Web and Bytespider, and its only rule is Disallow: /[1.1]. That is a literal path, not the root, and a standards-compliant parser reads it as "do not fetch a page called /[1.1]", which does not exist; everything else stays open to those five agents. Bucket 5, the block for aggressive aggregators, carries the same annotation and the same effect. The file's intent is clear and its syntax says the opposite, which is worth a line for anyone who reads robots files as policy: read the directive, not the comment. For the general shape of these lists across retailers, the AI crawler user-agent list has the census.

The Terms and Conditions are the contract, and their Security section says it in one clause: you may not use any engine, software, tool, agent or other device, including browsers, spiders, robots, avatars or intelligent agents, to navigate or search any Best Buy Property other than the search engine and search agents Best Buy provides and generally publicly available browsers. That is the line, and we leave it there. What Best Buy licenses is its developer programme: the portal at developer.bestbuy.com advertises 100+ brands and 725,000+ products behind an API key, with Products, Stores and Categories endpoints, and public documentation whose last modification date is 26 April 2021. For catalogue data with a licence attached, that is the route, and it is the one the vendor pages ranking for this keyword never mention.

What a gigabyte buys on Best Buy

Prices from our pricing page: residential Basic $0.80/GB, and the web scraping API from $0.0002 per page, or $0.001 with rendering. A gigabyte here is 10^9 bytes.

Forty-one cents against three dollars and ninety-seven for the same thousand pages, and the expensive column has one word fewer in it. There is no version of this table in which the browser wins; the only question is how much of the $3.56 difference a team has already spent before measuring. If you are sizing a pool for a catalogue crawl, how much proxy data you need does the arithmetic in the other direction, and Target is the page to read next, because it is the same category with the opposite answer.

There is no Best Buy collector in our catalogue. For a cross-retailer price on the same product, product_offers returns every seller offer at $0.002 per delivered offer, and google_shopping returns priced listings per query. For the Best Buy page itself, the web scraping API with engine: tls and country: us is the job.

The setting that works on Best Buy

• Network: residential, Basic line at $0.80/GB. Every fetch in this post went through it, one attempt each, answered from Akamai's cache in 5.1 seconds.
• Fetch mode: engine: tls, never rendered. 285 words without JavaScript against 284 rendered, at 506,484 bytes against 4,968,555 and 5.1 seconds against 45.9.
• Country: country: "us". A Canadian exit is handed a 103-word country chooser with no canonical; a US exit is handed the store. Classify on the canonical, not the status.
• When the proxy is not enough: the web scraping API with engine: tls at $0.0002 per page, never with rendering. For offers across retailers, product_offers at $0.002 per delivered offer.
• Free tier: every account gets $2 of free API usage per month, which is 10,000 Best Buy pages over plain HTTP at the API price.