# What Is a Web Scraper API?

> A web scraper API turns one HTTP request into clean HTML, Markdown or JSON — proxies, rendering and retries handled. How they work, code and real cost math.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/What Is a Web Scraper API?

# What is a web scraper API? A practical explanation

Web scrapingJul 29, 2026·10 min read·QuanticData Team

On this page [What a web scraper API actually is](/blog/what-is-a-web-scraper-api/#what-a-web-scraper-api-actually-is) [How the request path works](/blog/what-is-a-web-scraper-api/#how-the-request-path-works) [Web scraper API vs proxies vs an official API](/blog/what-is-a-web-scraper-api/#web-scraper-api-vs-proxies-vs-an-official-api) [What a web scraper API looks like in code](/blog/what-is-a-web-scraper-api/#what-a-web-scraper-api-looks-like-in-code) [Free tiers, open source, and where each breaks](/blog/what-is-a-web-scraper-api/#free-tiers-open-source-and-where-each-breaks) [Scraper API pricing, with the arithmetic shown](/blog/what-is-a-web-scraper-api/#scraper-api-pricing-with-the-arithmetic-shown) [AI scraper APIs and agent workflows](/blog/what-is-a-web-scraper-api/#ai-scraper-apis-and-agent-workflows) [How to evaluate a web scraper API in an afternoon](/blog/what-is-a-web-scraper-api/#how-to-evaluate-a-web-scraper-api-in-an-afternoon)

A web scraper API is an HTTP endpoint you send a target URL to and get that page's content back — raw HTML, clean Markdown or structured JSON. Proxy rotation, browser rendering, block and CAPTCHA handling, retries and parsing all happen server-side, so one request replaces an entire stack you would otherwise run and babysit yourself.

## What a web scraper API actually is

Every scraper you have ever written does four things: pick an exit IP, fetch a URL, render whatever JavaScript the page needs, then turn the DOM into fields. A web scraper API collapses those four things into a single POST request. You send `{ "url": "..." }` plus a few options; the service returns content and a status. That is the whole product.

The distinction worth internalising is between a *proxy API* and a *scraper API*. A proxy or unblocker routes your request through a different IP and hands back whatever the origin server said — you still own the browser, the parser and the failure handling. A scraper API layers browser control, session management and extraction on top, so what comes back is data rather than a page you still have to process. [Zyte draws the same line](https://www.zyte.com/learn/guide-to-web-scraping-ap-is-key-features-and-benefits/) in its guide, and it matters commercially: the two are billed differently and fail differently.

What a scraper API is not: a permission slip. It does not change what a site's terms allow, what personal data you may store, or which jurisdiction you are in. The technical question (can I fetch this page reliably?) and the legal question (should I?) are separate — see our overview of [whether web scraping is legal in the US](https://quanticdata.io/blog/is-web-scraping-legal-in-us/). Nothing here is legal advice.

## How the request path works

Behind a one-line call, a typical request goes through roughly this sequence:

1. **Authentication and routing.** A bearer key identifies the account; the target domain determines which strategy profile is used.

2. **Exit selection.** The request is assigned an IP — datacenter for soft targets, residential or mobile for hard ones — optionally pinned to a country.

3. **Fingerprint assembly.** TLS signature, HTTP/2 frame order and headers are made to match a plausible real browser rather than a default HTTP client.

4. **Fetch or render.** Static HTML goes over plain HTTP. Pages that build their content client-side get a real browser, a wait condition and sometimes scrolling or clicking.

5. **Validation.** A 200 status is not success. Challenge interstitials, empty shells and consent walls all return 200. Good services assert that expected markup exists before calling it done.

6. **Retry with variation.** On failure, the exit IP and fingerprint change and the fetch repeats, up to a budget.

7. **Transformation.** HTML is converted to Markdown, trimmed to main content, or reduced to fields by CSS selectors or an LLM against a schema.

Step 5 is where most cheap comparisons fall apart. A benchmark of 16 scraping APIs run against seven hard targets — Amazon, Indeed, GitHub, Zillow, Capterra, Google and X — [validated response bodies rather than status codes](https://scrape.do/blog/best-web-scraping-api/), and average success rates spread from roughly 60% to over 98% across providers. Same request, same URL, order-of-magnitude difference in whether you actually got the page.

## Web scraper API vs proxies vs an official API

The most common question on this topic is what separates "using an API" from "web scraping". If a site publishes an official API, that is nearly always the better path: stable contracts, documented rate limits, no anti-bot arms race. Scraper APIs exist for the other 95% of the web, where no official endpoint exists, the endpoint omits the fields you need, or access is gated behind a partnership.

| Dimension | Proxy / unblocker | Web scraper API | Official site API |
| --- | --- | --- | --- |
| You send | Any HTTP request via a gateway | A URL plus options | A documented query |
| You get | Raw origin response | HTML, Markdown or JSON fields | Contract-stable JSON |
| Blocks and CAPTCHAs | Your problem | Vendor's problem | Not applicable |
| JS rendering | You run the browser | A flag on the request | Not applicable |
| Parsing | You write and fix selectors | Selectors or schema in the call | Already structured |
| Typical billing unit | Bandwidth (per GB) or per IP | Per successful request or credit | Per call or per seat |
| Breaks when | Fingerprint or IP quality slips | Layout changes and no schema is set | Vendor deprecates or gates it |

Plenty of teams run both: a scraper API for awkward targets and raw [residential proxies](https://quanticdata.io/residential-proxies/) for high-volume, easy pages where paying per request would be wasteful. That split is usually cheaper than picking one religion.

## What a web scraper API looks like in code

The interface should be boring. Here is the same call three ways against the QuanticData [web scraping API](https://quanticdata.io/web-scraping-api/) — curl, Python and JavaScript, which covers most of what people mean when they search for a web scraper API example.

```
curl https://api.quanticdata.io/v1/scrape \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com" }'

{ "success": true,
  "data": { "markdown": "# Example Domain\n…" },
  "usage": { "cost_usd": 0.0002 } }
```

Python, with nothing but `requests`:

```
import os, requests

r = requests.post(
    "https://api.quanticdata.io/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['QD_API_KEY']}"},
    json={"url": "https://example.com"},
    timeout=120,
)
body = r.json()
if body["success"]:
    print(body["data"]["markdown"][:500])
    print("cost:", body["usage"]["cost_usd"])
else:
    print("failed, charged nothing")
```

Node, with the built-in fetch:

```
const res = await fetch("https://api.quanticdata.io/v1/scrape", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.QD_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com" }),
});
const { success, data, usage } = await res.json();
```

Notice there is no client library requirement and no per-language SDK to learn. Every endpoint — single page, [search results](https://quanticdata.io/serp-api/), [crawl and map](https://quanticdata.io/crawl-map/), batch, SEO audit — answers with the same `success` / `data` / `usage` envelope, so error handling is written once.

## Free tiers, open source, and where each breaks

A free web scraper API generally means one of three things: a trial measured in successful results, a permanent small monthly allowance, or a free tier that quietly omits JS rendering and premium exits — the two features you need most on the targets that were hard enough to send you looking. Free allowances are for validating the interface and measuring success rate on *your* URLs, not for running production.

Open source is a real option, not a lesser one. Scrapy gives you a mature crawling framework, Playwright and Puppeteer give you browser control, Crawlee handles queues and retries, and several self-hostable extraction servers will convert HTML to Markdown or JSON. The code is free; the network is not. You still need IP diversity, fingerprint maintenance and someone on call when a target changes its challenge flow, and the sites you most want are exactly the ones that make that a recurring cost. The honest rule of thumb: open source wins when your targets are permissive and your volume is high; a hosted scraper API wins when your targets are hostile and your engineers are expensive. One vendor reports serving [over 11 billion requests in a 30-day window](https://www.scraperapi.com/) — that is the scale at which block-handling becomes a full product rather than a script.

## Scraper API pricing, with the arithmetic shown

Three billing models dominate, and they are not comparable at face value:

- **Per successful request.** Simplest to forecast. Watch for whether a JS-rendered page costs the same as a static one.

- **Credits with multipliers.** A base call is one credit, rendering might be five, premium IPs ten, both together twenty-five. Your effective price depends entirely on your feature mix, so a headline plan size tells you almost nothing.

- **Bandwidth (per GB).** Normal for proxies. Cheap for lean HTML, punishing for image-heavy pages you did not want to download.

Concrete example. Say you need 250,000 product pages a month and 40% of them require rendering. On QuanticData list rates — $0.0002 per page, $0.001 rendered — that is 150,000 x $0.0002 = $30, plus 100,000 x $0.001 = $100, so $130 for the month. Adding 20,000 searches at $0.0005 puts you at $140. The same workload on a credit model with a 5x render multiplier and a 10x premium-IP multiplier can land anywhere between a third and five times that number depending on how often the vendor decides you need premium exits — which is a decision you do not control.

Two mechanics change the maths more than the unit price. First, **pay per success**: if a failed call costs nothing, a 70%-success provider is not 30% more expensive, it is just slower — but if you pay per attempt, that same failure rate silently inflates your bill. Second, **refunds on async jobs**: crawl and batch jobs are charged on requested volume up front, so ask whether the unfetched share comes back when a 500-page crawl finds 180 pages.

## AI scraper APIs and agent workflows

The reason "AI scraper API" is now a search term is that two different AI angles collapsed into one phrase.

### AI inside the extraction step

Instead of writing CSS selectors that break on the next redesign, you pass a schema — field names and types — and an LLM fills it from the page. This is slower and costlier per page than selectors, so the pragmatic pattern is selectors for the ten sites you scrape daily, AI extraction for the long tail and for one-off research runs.

### The API as a tool for an agent

An agent cannot pip-install a scraping framework mid-conversation. It needs callable tools with typed inputs, and that is what the Model Context Protocol standardises — a client-server interface for exposing tools to LLM applications ([spec](https://modelcontextprotocol.io/)). Our [web scraping MCP server](https://quanticdata.io/mcp-server/) exposes eight tools — search, scrape, map, crawl, crawl_status, batch, batch_status and seo_audit — to Claude, Claude Code, Cursor, Windsurf, VS Code and Cline, so "find the top ten suppliers and pull their pricing pages" becomes a sequence of tool calls rather than a code-generation exercise.

One detail that matters for token budgets: return Markdown, not HTML. A product page can be 400 KB of markup and 6 KB of actual text. Feeding the raw DOM to a model wastes context and money; a smart-content Markdown mode usually cuts input tokens by an order of magnitude and improves extraction accuracy at the same time.

## How to evaluate a web scraper API in an afternoon

1. **Test your own URLs, not the demo.** Take 200 real targets, run them through each candidate's free allowance, and score on validated content — a required selector present — not HTTP 200.

2. **Separate render cost from base cost.** Measure what share of your pages genuinely need a browser, then price that mix. Many pages that look dynamic ship their data in a JSON blob in the initial HTML.

3. **Check the failure contract.** Are failures free? Is there a machine-readable error code? Does the response include a per-attempt retry log so you can explain a slow call?

4. **Check output shapes.** Markdown, HTML, text, CSS-extracted fields, schema-driven extraction. If you can only get HTML, you are buying half a product.

5. **Check async coverage.** Batch endpoints for known URL lists, crawl for site-wide sweeps, and geo-targeting per request rather than per account.

6. **Check the exit ceiling.** When a target hardens, can you escalate to residential or mobile IPs without migrating vendors?

7. **Read the compliance posture.** How are IPs sourced, what does the provider prohibit, and does that match your own policy on personal data.

A web scraper API is infrastructure, and infrastructure should be judged on failure behaviour and unit economics, not on feature grids. If the price per successful page is predictable, the envelope is uniform across endpoints, and failures cost you nothing, the rest is just your code.

### Sources & further reading

- [Guide to Web Scraping APIs: Key Features and Benefits — Zyte](https://www.zyte.com/learn/guide-to-web-scraping-ap-is-key-features-and-benefits/)

- [Best Web Scraping APIs: benchmark methodology and results — Scrape.do](https://scrape.do/blog/best-web-scraping-api/)

- [ScraperAPI — request volume and platform scale](https://www.scraperapi.com/)

- [Model Context Protocol — specification](https://modelcontextprotocol.io/)

## FAQ

Quick answers on what is web scraper api.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### What is a web scraper API in Python?

It is a normal HTTP call — no scraping library needed. You POST a JSON body containing the target URL to an endpoint with a bearer token, using `requests` or `httpx`, and read the returned Markdown, HTML or extracted fields. The proxy rotation, headless browser and retries run on the provider's side, so your Python code stays about ten lines long.

### What is the difference between using an API and web scraping?

An official API is a contract the site publishes: documented fields, stable shapes, declared rate limits. Scraping reads the public page a browser would render and derives fields from markup. Use the official API when one exists and exposes what you need; use a scraper API for the vast majority of sites that publish no API, or omit the fields you actually want.

### Is there a free web scraper API?

Most providers offer either a trial measured in successful results or a small permanent monthly allowance — QuanticData includes $2 of free usage every month with no card. Free tiers are ideal for measuring success rate on your own URLs, but they usually restrict rendering or premium exits, so treat them as validation rather than production capacity.

### Are there open source web scraper APIs?

Yes. Scrapy for crawling, Playwright or Puppeteer for browser control, Crawlee for queue and retry management, and several self-hostable HTML-to-Markdown extraction servers. The software is free; IP diversity, fingerprint maintenance and on-call time for broken targets are not. Open source wins on permissive, high-volume targets and loses on heavily defended ones.

### How much does a scraper API cost?

Anywhere from fractions of a cent to several cents per page depending on billing model. Per-request pricing is easiest to forecast — QuanticData lists $0.0002 per page and $0.001 with JS rendering — while credit systems apply multipliers for rendering and premium IPs that can move your effective rate several times over. Always price your real feature mix, and check whether failed calls are billed.

### Can an AI agent use a web scraper API directly?

Yes, if the API is exposed as tools. Through the Model Context Protocol, an agent in Claude, Cursor or VS Code can call search, scrape, map, crawl, batch and audit tools with typed arguments and receive Markdown it can reason over. Returning Markdown instead of raw HTML keeps input token counts, and cost, an order of magnitude lower.

## Try one request before you plan a pipeline

Point the QuanticData scrape endpoint at your hardest URL and read the envelope: Markdown or structured JSON from $0.0002 per page, $0.001 with JS rendering, and failed calls cost nothing. $2 of free usage every month, no card required.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

[Web scraping Does Web Scraping Use an API? Web scraping and APIs are not opposites. Most modern scrapers hit a JSON endpoint, a public API, or a third-party scraping API — here is how to pick. Read →](https://quanticdata.io/blog/does-web-scraping-use-api/) [Web scraping How to Web Scrape Using Python A practical Python scraping walkthrough: fetch, parse, paginate, handle JavaScript pages — and the cost math for when to stop maintaining your own stack. Read →](https://quanticdata.io/blog/how-to-web-scraping-using-python/) [SEO data How to Perform an SEO Audit A practical six-step SEO audit process with a checklist, the crawler-vs-user diff most audits skip, and how to run the whole thing programmatically. Read →](https://quanticdata.io/blog/how-to-perform-an-seo-audit/)

---

Source: https://quanticdata.io/blog/what-is-a-web-scraper-api/ · Site index for AI: https://quanticdata.io/llms.txt