Autotrader is the United Kingdom's largest car marketplace, and on 28 September 2026 its home page gave a plain HTTP client through a British residential exit 200 OK, 366,925 bytes and 1,115 words: the title, the h1, the canonical and JSON-LD for the Corporation and the WebSite. Rendered, the same page cost 835,543 bytes and returned 1,133 words. From a US exit it returned the same 1,115. What sets Autotrader apart is not the page but the fence: its robots.txt, updated 24 September 2026, disallows /car-search, /car-details and the GraphQL gateway to every user agent. The setting that works on Autotrader is residential, plain HTTP, country=gb, for the pages that are open; the autotrader_search collector for the US site, which returned 10 Honda Civic listings with VIN and mileage in 15.3 seconds; and Auto Trader Connect, the official API, for UK inventory on Auto Trader's terms.
The home is complete over plain HTTP, and the browser adds 18 words
Autotrader's home is served by an Express application behind Cloudflare (x-at-service: sauron-home-pages-app, cache-control max-age=60), and it is server-rendered. The seo_audit tool in our MCP server fetched it as a pure HTTP bot and fully rendered; both passes returned the same title, meta description, canonical, the h1 "Discover more from Autotrader", an og:title and two JSON-LD blocks, Corporation and WebSite. The rendered page added 18 words.
| Fetch (28 September 2026) | Status | Bytes returned | Time | Words | Notes |
|---|---|---|---|---|---|
| autotrader.co.uk, plain HTTP, UK exit | 200 | 366,925 | 8.3 s | 1,115 | Title, description, canonical, h1, JSON-LD Corporation + WebSite |
| autotrader.co.uk, rendered, UK exit | 200 | 835,543 | 38.7 s | 1,133 | Same head; 18 more words |
| autotrader.co.uk, plain HTTP, US exit | 200 | n/a | 13.7 s (audit) | 1,115 | Identical page; no redirect, no country notice |
| autotrader.co.uk/robots.txt | 200 | 2,489 | 0.6 s | n/a | Last modified 24 September 2026; cached four hours at Cloudflare |
| developers.autotrader.co.uk | 200 | 954,514 | 16.9 s | n/a | Redirects to /api: the Auto Trader Connect documentation, one page of 8,800 lines |
On 26 September the same audit counted 1,113 and 1,129; the numbers are stable to within a few words across days and across fetch modes. There is nothing on the home that a browser earns, and it costs 2.3 times the bytes and 4.6 times the wait to confirm it. Read Autotrader over plain HTTP.
The country does not change the page; the path does
Autotrader serves the UK and does not check where a reader is: the home came back identical, 1,115 words, from a British and from an American residential exit, with no redirect and no notice. The number that changes when you pick the wrong country is zero. Pin country=gb anyway, because a session that reads UK car prices from a UK address is the pattern of the site's real visitors, and Cloudflare in front of Autotrader scores sessions, not just requests.
What changes is the path. Autotrader's robots.txt is 2,489 bytes and was modified four days before our fetch. For every user agent it disallows /car-search, /car-details*, /van-search, /van-details*, /bike-search, /bike-details*, /caravan-search, /motorhome-search, /truck-search, /plant-search, /farm-search, /new-cars/search, /new-cars/generation, /new-cars/derivative, the leasing searches, every URL with ?postcode=, ?radius=, ?sort=, ?body-type=, ?county= or ?region=, the JSON autocomplete and finance endpoints, and four gateway paths: /at-graphql, /at-secure-graphql, /at-gateway and /at-secure-gateway. Baiduspider and a few ad-tech crawlers are excluded from the whole site; Meta's crawlers are kept only out of /contact-dealer/. The sitemap index is public.
That list is the answer to "can I scrape Autotrader". The home, the editorial content, the reviews and whatever the sitemap lists are open to a crawler. The inventory, search results and vehicle detail pages, is closed to every automated agent by name of path, and so is the GraphQL layer the site itself uses to load it. The GitHub scraper that ranks for "autotrader uk scraper" notes that a search exposes at most its top 1,000 results; robots.txt says a crawler should not be reading them at all. We do not route around that, and a proxy is not a licence.
The sanctioned route is Auto Trader Connect; the collector covers the US site
Autotrader publishes an API. The developer portal at developers.autotrader.co.uk redirects to a single 954,514-byte documentation page for Auto Trader Connect, which describes itself as Autotrader's public API platform, a suite of APIs connecting automotive systems with the marketplace and with Autotrader's data and insight. It is aimed at integrators and retailers, and it is the route for anyone whose product needs UK stock, valuations or advert data on Autotrader's terms. The autocomplete for "autotrader uk api" is small but real: "autotrader co uk api", "autotrader apis", "autotrader api cost".
Our own Autotrader collector covers Autotrader.com, the US marketplace, which is a separate company with a separate site and its own rules. We ran it once on 28 September 2026 around ZIP 90210 for make Honda and model Civic with max_results 10. It delivered 10 rows in 15.34 seconds.
| Field in the run (Autotrader.com, US) | What came back for 10 rows |
|---|---|
| Price | 10 of 10, from $18,072 for a 2017 Civic EX with 73,086 miles to $33,267 for a new 2026 Civic Si |
| MSRP | 3 of 10, the three new cars, $32,690 to $33,145, each priced slightly above it |
| VIN | 10 of 10 |
| Condition | 6 used, 3 new, 1 certified |
| Days on site | 10 of 10, from 63 to 149 |
| Specifications | MPG, engine, transmission, drive type, exterior and interior colour on every row |
Days on site is the field a dealer analyst wants and a card parser rarely gets: it turns a list of prices into a measure of how long inventory sits, and in this run the range was 63 to 149 days. At $0.001 per delivered listing the run cost one cent. If your question is about UK stock, this collector is the wrong tool and Auto Trader Connect is the right one; if your question is about US pricing by ZIP, it answers in one call.
Cost math with real numbers
Prices from our pricing page: residential Basic $0.80/GB, Premium $2.20/GB, the web scraping API from $0.0002 per page ($0.001 rendered) and the Autotrader collector at $0.001 per delivered listing. One gigabyte is counted as 10^9 bytes.
| Approach | Bytes per page | Pages per GB | Cost per 1,000 |
|---|---|---|---|
| Plain HTTP over residential Basic ($0.80/GB) | 366,925 | about 2,725 | about $0.37 per 1,000 pages |
| Plain HTTP over residential Premium ($2.20/GB) | 366,925 | about 2,725 | about $1.01 per 1,000 pages |
| Rendered over residential Basic | 835,543 | about 1,197 | about $0.84 per 1,000 pages |
| Web scraping API, plain HTTP | n/a | n/a | $0.20 per 1,000 pages |
| autotrader_search collector (US site) | n/a | n/a | $1.00 per 1,000 listings, VIN, MSRP, mileage and days on site parsed |
Byte figures are decoded HTML; wire transfer after compression is lower and headers add a little back. Rendering doubles the bill for 18 words; skip it. The comparison that matters on this target is not bytes but scope: the open pages are cheap and few, the inventory is closed to crawlers by Autotrader's own file, and the official API is the route for it. For a classifieds site where the fence is at page six rather than at the search path, see Kleinanzeigen through residential proxies; for one where the browser is the wrong client altogether, Idealista.
The setting that works on Autotrader
- Network: residential, Basic at $0.80/GB. Nothing in our fetches challenged a residential exit from either country, and at 367 KB a page the tier is a small part of the bill.
- Fetch mode: engine: tls, plain HTTP. 1,115 words with the full head and both JSON-LD blocks in 8.3 seconds; the rendered page returns 1,133 words for 835,543 bytes and 38.7 seconds.
- Country: country=gb. The page is the same from the US (1,115 words, no redirect), so the exit is pinned for a British session pattern, not for access.
- When the proxy is not enough: for UK inventory, Auto Trader Connect, the official API; for US listings, the autotrader_search collector, 10 cars in 15.3 seconds with VIN, price, MSRP, mileage, days on site and full specifications, $0.001 per delivered listing.
- Every account gets $2 of free API usage per month, and failed requests are never billed.