Use the official API when one exists, it returns the fields you need, its quota fits your volume, and its terms allow your use. Scrape when any of those four fails, which is more often than the API's documentation suggests. And know that “scrape” has itself split in two: writing and maintaining a scraper, or calling a scraping API that returns structured data for any page and bills per success. Most production pipelines end up using the official API for what it covers and one of the other two for the gaps.
Three options, not two
| Official API | Scrape it yourself | Scraping API / collector | |
|---|---|---|---|
| What you get | The fields the owner chose to expose, as JSON | Anything visible on the page, after you parse it | Anything on the page, returned structured |
| Stability | Versioned; breaks on deprecations, months of notice | Breaks on any layout change, no notice | Provider maintains parsers and anti-bot handling |
| Rate limits | Explicit, per key, often per minute and per day | Whatever the site tolerates before it blocks you | Provider's concurrency, billed per success |
| Freshness | What the API serves; sometimes cached or delayed | What the page shows right now | What the page shows right now |
| Cost model | Free tier, then per call or per month; can change | Engineering time, proxies, rendering, retries | Per page or per result, e.g. from $0.0002/page |
| Legal position | Contract: you agreed to the terms | Depends on the data, the access and the jurisdiction | Same as scraping, with the provider's own policy on top |
| Skill needed | Read docs, handle auth and pagination | HTTP, parsing, proxies, fingerprints, rendering, monitoring | One HTTP call and a schema |
Seven questions that decide it
- Does an official API exist for this data? Not for the site; for the data. Many platforms have an API for posting and account management and none for the public listings you want.
- Does it return the fields you need? Check the response schema against your target columns. Price without shipping, a listing without its seller, a job without its salary range: partial coverage is the most common reason an API is not enough.
- Does the quota fit? Multiply your daily volume by the calls each record costs, including pagination. A limit of 5,000 calls a day is 200 records an hour for a paginated list of 25.
- Do the terms allow your use? Redistribution, caching, commercial use, training models: read the clause, not the summary.
- Is the API's data what the page shows? Some APIs serve a delayed or aggregated view. If freshness is the point, compare a sample against the live page.
- How much will the scraper cost to keep alive? One target with a stable layout is a weekend. Twenty targets behind bot management is a job.
- Is this data already a product? If a ready-made collector returns the exact record for a semantic input, such as a keyword and a location, the third column wins on both cost and time.
Yes to the first five and no to the sixth points at the API. Any no in the first five points at scraping, and question seven decides which kind.
Where official APIs stop being the answer
An API is a promise the owner can revise. The last few years produced three well-known revisions: Twitter's API moved to paid tiers in 2023 with free access reduced to almost nothing, Reddit priced its Data API the same year and third-party clients shut down, and Google has retired more public APIs than most companies have launched. Beyond pricing, the practical failures are narrower and more common. The API exposes 20 fields where the page shows 60. The API returns the seller's list price and the page shows the promotion. The API paginates at 25 with a 60-call-a-minute limit and you need a million rows by Monday. The API requires an approved app and the approval queue is four months long. None of these is a reason to avoid APIs; each is a reason to have a second option ready.
What scraping costs you
Scraping trades the owner's constraints for your own. The page changes and your selectors go stale, silently, so you need monitoring that compares extracted rows against expectations. The site adds bot management and your fetches turn into 403s, so you need proxies whose type matches the target and a TLS fingerprint the WAF accepts. The site rate-limits and you get 429s, so you need pacing and rotation. The data moves behind JavaScript and you need a rendering tier that costs ten times the bandwidth. Each of these is solvable, and each is an ongoing cost that an API bill makes visible and a scraper hides in engineering time.
Then there is the legal side, which is a separate question from the technical one. Public data, accessed without circumventing authentication, has fared well in US courts; personal data has GDPR on top in Europe; and a site's terms can still make a breach-of-contract claim even where no statute applies. The US guide and the European guide walk the layers.
A worked cost example
Suppose you need 10,000 product pages a day from a large marketplace, 30 fields each, fresh, for price intelligence.
- Official API: if one exists for the listings, check field coverage first. Marketplace product APIs commonly expose catalogue data but not the live buy-box price or competing offers, which is usually the point of the exercise. If coverage fails, the price of the API is irrelevant.
- Scrape it yourself: 10,000 pages at roughly 370 KB each on the wire is about 4 GB a day through residential exits, which is around $3 a day in bandwidth at $0.80/GB. The real cost is the parser for 30 fields, the fingerprint work, the monitoring, and the day it breaks. Budget an engineer-day a month at minimum and more when the site redesigns.
- Scraping API: 10,000 pages at $0.0002 is $2 a day for clean Markdown or HTML, with CSS or AI extraction to your 30-field schema in the same call, and failed pages unbilled. Or, for marketplaces where a collector exists, a semantic call that takes a product id and returns the structured record; the collector catalogue lists which sources are covered and at what price per result.
The scraping API is cheaper than the bandwidth alone of doing it yourself in this example, before any engineering time, because per-page pricing does not scale with page weight. The break-even where raw proxies become cheaper is at about 250 KB per page on the wire, worked out in how much proxy data do I need.
The hybrid that usually wins
Use the official API for the entities it covers well, typically identifiers, canonical names and catalogue attributes, because it is stable and cheap and you agreed to its terms. Use scraping for the fields the API does not expose, joined on the identifier the API gave you. Prefer the site's own internal JSON endpoints where the page has them, because they are lighter than HTML and rarely change; the scraping API's xhr and app_state options exist to find and return them. Put a schema check between every source and your database, so a silent layout change fails loudly. And if an agent is doing the work rather than a pipeline, the same three options are exposed as tools through the MCP server, where “search, then scrape the top result” is one call each.
The question was never scraping or API. It is which fields come from where, and what you pay when each source changes its mind.