Documentation Python quickstart Blog Free tools hello@quanticdata.ioLog in

Twitter Proxies: What X Returns to a Bot

Two requests from the same residential proxy exit to x.com: the profile path returns five posts in 254 KB while the search path returns an empty challenge shell of 299 KB, both answering HTTP 200
Two requests from the same residential proxy exit to x.com: the profile path returns five posts in 254 KB while the search path returns an empty challenge shell of 299 KB, both answering HTTP 200

A Twitter proxy is an exit IP you route requests through when you read x.com from outside a browser. On 23 September 2026 we fetched seven x.com surfaces through residential exits with no login and no JavaScript. Public profiles and individual posts render in full over plain HTTP, search returns an empty shell, and every failure arrives as HTTP 200.

What x.com actually serves a proxied client

Most pages that rank for this query compare providers. None of them measure the thing that decides your bill: what x.com puts in the response body when an anonymous HTTP client asks for it. So we asked. Every row below is one fetch through a residential exit on 23 September 2026, no cookies, no account, no browser unless the row says so.

SurfaceStatusBytesTimeWhat came back
Profile, plain HTTP (/NASA)200253,9581.6 sBio, 92.3M followers, 74.3K posts, and 5 posts with full text, timestamps, media alt text and engagement counts
Profile, headless browser2004,623,11822.3 sThe same 5 post IDs
Single post (/NASA/status/...)200195,5301.9 sPost text, exact timestamp, view count, and 3 public replies from other accounts
Search (/search?q=nasa)200299,3363.7 sNothing. Challenge shell, no title, 227 characters of text
Unknown handle200298,8785.2 sNothing. "JavaScript is not available"
Public oEmbed endpoint2008292.6 sPost text, author name, author URL
/robots.txt2002,6780.8 sThe crawl rules, quoted later in this post

Two numbers in that table are worth reading twice. The headless browser spent 18 times the bandwidth and 14 times the wall clock to return the same five post IDs as the plain HTTP fetch. And the two failures cost more bytes than the successes: an empty challenge page is roughly 299 KB of markup that contains no data at all.

Two backends hide behind one hostname

The response headers explain the split. The profile and the status page came back with x-server: x-web and a server-rendered timeline: the post cards are in the HTML, marked with data-timeline-entry attributes, before a single line of script runs. The search page and the unknown handle came back from a different stack — x-powered-by: Express, x-frame-options: DENY — carrying the old "JavaScript is not available. We've detected that JavaScript is disabled in this browser" interstitial and the "Something went wrong, but don't fret" retry box.

So x.com is not one wall. It is a server-rendered public surface, which an HTTP client reads for free, sitting next to a challenge-gated surface that no proxy type will open for you. Choosing the surface matters more than choosing the IP.

The trap: failures are HTTP 200

Every one of the seven fetches returned status 200, including the two that returned no data. A scraper that branches on response.status will record a nonexistent handle and a blocked search as successes and write empty rows into your dataset. Detect failure on content, not on status:

def is_real_x_page(html: str) -> bool:
    if "JavaScript is not available" in html:
        return False            # challenge shell, served with status 200
    if "Something went wrong, but don" in html:
        return False            # legacy error page, also status 200
    return 'data-timeline-entry' in html or 'property="og:description"' in html

The same rule applies to a nonexistent or suspended handle: x.com does not answer 404 on the path we tested, it answers 200 with the challenge shell. If you need to tell "this handle is gone" apart from "we got challenged", retry through a second exit before you conclude anything — an account that is really gone fails the same way every time, a challenge usually does not.

Which proxy type X actually needs

The honest answer is smaller than the listicles suggest. For reading public profiles and public posts, the request that worked is an ordinary HTTP GET with a real browser TLS fingerprint. We ran ours over rotating residential IPs because that is what our own platform uses, and every profile fetch succeeded first try. We did not test datacenter exits on X for this post, so we will not tell you they work — but note that the document is server-rendered and publicly indexable, which is not the shape of a surface defended by IP reputation.

  • Residential — the safe default for public profile and post reads. Country selection per request, which matters for the locale effect described below.
  • Mobile — 4G and 5G device IPs are the most trusted addresses on any social platform, and the most expensive per GB. On a 250 KB document that difference is real money at volume. Reach for them only when residential actually starts failing, not preemptively.
  • ISP static — a fixed address per IP with unlimited bandwidth. Attractive for a long-running monitor with a steady, modest request rate, because the bill stops scaling with page weight.
  • Datacenter — the cheapest bandwidth we sell, and the first thing to test if your target is only the public surface. Measure it yourself before you commit.

What no proxy type changes: the challenge-gated surfaces. We got the same empty shell on search through a US exit and the same shape on the unknown handle. If search results are what you need, the supported route is the official API, not a better IP.

Geo targeting: what the exit country really changes

We fetched the identical profile through a German exit. The five post IDs came back identical, and the post text was unchanged — posts are served in the author's language, not the reader's. What changed was the chrome and, more usefully, the number formatting:

FieldUS exitDE exit
html langende
Interface stringsLog in, Following, Joined December 2007Anmelden, Folge ich, Beigetreten Dezember 2007
Likes on the equinox post15.3K15.340
Followers92.3M92,3 Mio.
Document size253,958 bytes251,535 bytes

That third row is the practical find. The English locale abbreviates engagement counts to three significant figures; the German locale prints them in full, because German thousands separators do not collapse. If you are tracking engagement on public posts and you want exact numbers without the API, request through an exit whose locale does not abbreviate. It is the same document, the same cost, and the rounding disappears.

Geo targeting also matters for a second reason we could not measure in one session: X withholds specific content in specific countries on legal request. A creative or a post that renders from a German exit may be absent from a Turkish or Indian one. If brand or ad verification is your use case, that variation is the whole job, and it is the reason to buy country targeting at all.

Rate limits and what the errors look like

We did not trigger a 429 in this run, and we are not going to invent one. What we did hit is more useful to know about, because it is what you will hit first:

  • The challenge shell — around 299 KB, no title, the "JavaScript is not available" text. Our fetcher retried three times before giving up, which means one blocked search cost roughly 900 KB of billed bandwidth for zero rows. Cap retries on this shape at one, or you pay triple for nothing.
  • The legacy error box — "Something went wrong, but don't fret." Transient on profile paths, permanent on the surfaces X does not serve anonymously.
  • 429 — the documented shape when you do earn a rate limit. Back off exponentially and rotate the exit; a retry from the same address at the same rate is how a soft limit becomes a hard one. We wrote up the general pattern in the 429 post.
  • 403 — worth distinguishing from a challenge, because it usually means the fingerprint, not the IP. Our 403 walkthrough covers the order to test in.

X's own robots.txt sets Crawl-delay: 1 for the crawlers it names. One request per second per target is a defensible starting rate for anything you run, and it is slow enough that you will find out whether your pipeline is correct before you find out whether it is blocked.

What a thousand profiles cost

Bandwidth is the whole bill on this platform, because the documents are heavy. Using our published per-GB rates and the byte counts we measured:

Job, 1,000 itemsBytes movedResidential Basic, $0.80/GBMobile, $2.30/GB
Profiles over plain HTTP0.254 GB$0.20$0.58
Profiles through a headless browser4.62 GB$3.70$10.63
Single posts by URL0.196 GB$0.16$0.45
Post text via the oEmbed endpoint0.0008 GBunder $0.01under $0.01
Searches that return nothing0.299 GB$0.24$0.69

Two caveats so the numbers stay honest. These are the byte counts our fetcher reported for the documents; X serves them compressed, so a per-GB meter on the wire bills less than this — treat the table as a ceiling that compares surfaces fairly rather than as an invoice. And the profile page carries five posts, so the marginal cost of a post collected this way is about 50 KB, not 250 KB.

The row that should change your architecture is the second one. Rendering costs eighteen times the bandwidth and returns the same five posts. On a per-page API the multiplier is similar: our scraping API bills $0.0002 for a page fetched over HTTP and $0.001 with a browser, and on this surface the browser buys nothing. Check before you pay for rendering — that is exactly what our audit endpoint does, fetching any URL twice and returning the diff. On the NASA profile the diff was twelve words.

The cheaper route when you already have the URL

If your job is "show the text of these posts" rather than "discover posts", the documented oEmbed endpoint answers in 829 bytes what the status page answers in 195,530 — a factor of 236. It returns the post text, the author name and the author URL as JSON, it needs no key, and it is a published integration surface rather than a page you are scraping around.

curl -s "https://publish.twitter.com/oembed?url=https://x.com/NASA/status/2102425682286412261&omit_script=1"
# {"url":"...","author_name":"NASA","author_url":"https://x.com/NASA","html":"...","provider_name":"X"}

It will not enumerate anything — you must already know the post URL — and it carries no engagement counts. As a pairing, discovery through the profile page and text through oEmbed is the cheapest correct combination we measured.

Read X's own rules before you scale

This is the part the provider pages leave out, so here it is plainly. X's robots.txt on the day we fetched it grants a short allow-list to Googlebot, Bingbot and facebookexternalhit — profiles and statuses yes, /*/followers, /*/following, /*/likes, /*/retweets, /*/media, /search/realtime and /search/users no — explicitly disallows Google-Extended, FacebookBot, Discordbot and the Meta crawlers, and closes with:

User-agent: * / Disallow: /

Every client that is not on the named list, yours included, is disallowed by that file. A proxy changes which IP the request comes from; it does not change that line, and it does not grant permission. X's Terms of Service and Developer Agreement govern what you may do with the data, and both are linked in the sources below. Read them against your actual use case before you scale, and take legal advice for anything commercial.

What this post is for: brand and ad verification, checking how public content renders in a given country, social listening across posts you already have URLs for, and creator or campaign research on public accounts. What it is not for, and what we will not write: creating or farming accounts, automating engagement, evading a suspension, or any form of credential stuffing. A proxy does not restore a banned account, and any vendor who implies otherwise is selling you a story. The same boundary applies on every network in this series — we drew it the same way for Instagram and for Mastodon, where the federated answer turns out to be very different.

One more thing about "Twitter proxy sites"

The autocomplete data splits this query in two. Half the people typing it want an exit IP for their tooling, which is everything above. The other half want a mirror front-end to read X without an account — the Nitter instances and redirectors that still rank for it. Those front-ends depend on unauthenticated access that X has repeatedly closed, and we did not test any of them for this post. The measurement above suggests they are also largely unnecessary now: a public profile renders its last five posts to any client that asks politely.

How to reproduce this

Nothing here needs our platform. Fetch https://x.com/NASA with any HTTP client that presents a real browser TLS fingerprint, no cookies, and count the occurrences of data-timeline-entry in the response. Then fetch the same URL through a headless browser and compare the transferred bytes. Then fetch /search?q=nasa and look at the status code before you look at the body. The three results are the whole argument of this post, and they take about a minute.

If you want the same shape of answer for a platform we have already built a collector for, the per-result pages are cheaper to reason about than bytes: the Reddit posts collector and the YouTube search collector bill per delivered row instead of per gigabyte. We do not publish an X collector, for the reasons in the section above.

Sources & further reading

FAQ

Quick answers on twitter proxies.

Something else? Ask us →

Do I need a proxy to read a public X profile?

Not to read one. On 23 September 2026 a single anonymous HTTP request to a public profile returned the bio, the follower counts and the last five posts with full text and engagement numbers, status 200. You need proxies when you read many profiles, because that is when rate limiting and country targeting start to matter.

Which proxy type works best for X?

Residential is the safe default for public profiles and posts, and it is what we measured. Mobile IPs carry the highest trust and the highest per-GB price, so treat them as an escalation rather than a starting point. Datacenter is worth testing first on a surface this public. No proxy type opens the search endpoint, which answers with a challenge shell regardless of the exit.

Why does my X scraper return empty pages with status 200?

Because X serves its challenge and error pages with HTTP 200. The blocked search page we fetched was 299,336 bytes with no title and 227 characters of text, and an unknown handle returned the same shape. Branch on the response body — look for the string about JavaScript not being available, or for the presence of timeline entries — never on the status code alone.

Is it worth rendering X pages in a headless browser?

On the profile surface, no. The rendered fetch moved 4,623,118 bytes in 22.3 seconds and returned the same five post IDs as a 253,958-byte HTTP fetch that took 1.6 seconds. That is eighteen times the bandwidth for twelve extra words of text. Verify the diff on your own target URLs before paying for rendering.

Does the exit country change what X returns?

The posts do not change: a German exit returned the same five post IDs with the same English text. The interface language, the date formatting and the number formatting do change, and the last one is useful — the English locale rounds likes to 15.3K where the German locale prints 15.340. Country targeting also matters because X withholds specific content in specific jurisdictions.

Is scraping X allowed?

X’s robots.txt ends with a blanket disallow for every user agent it does not name, so an ordinary client is outside what that file permits, proxy or no proxy. The Terms of Service and the Developer Agreement govern what you may collect and what you may do with it. Read both against your use case, take legal advice for commercial work, and do not use any of this to create accounts, automate engagement or evade a suspension.

Measure your own target before you buy bandwidth

Every number in this post came from the same platform: fetch any URL over HTTP or through a browser, compare the two views, and pay only for what succeeds. Every account gets $2 of free API usage each month, and failed requests are never billed.

Related reading