Web data is any information that exists on the public web or is produced by web activity: page text, prices, listings, reviews, media, plus the clickstream your own site records. It only becomes usable once it is collected, structured and timestamped — by an analytics tag, a crawler, or a web data API.
What is web data, precisely
Most definitions of web data are too loose to be useful. A tighter one has two halves. The first is web content: everything a browser can render from a public URL — product pages, job posts, company profiles, news articles, search engine result pages, images and video metadata. The second is web activity: the events generated when people interact with a site, which is what people mean by web analytics.
The enterprise framing predates modern scraping tools. Wikipedia's entry on web data services describes service-oriented architecture applied to information sourced from the web, covering structured tables, semi-structured XML and unstructured feeds — the same three shapes you still deal with today, just now behind REST endpoints instead of SOAP.
What counts and what does not
- Counts: anything served to an anonymous visitor at a public URL, and any event your own property records about its own visitors.
- Does not count as public web data: content behind a login, paywall or registration wall. Coresignal's guide makes the same distinction — reaching that data means accepting a platform's terms as a member, which is a different legal and ethical situation entirely.
- Not web data at all: your CRM records, invoices or internal spreadsheets. They live in a browser UI, but the web is not their source.
Types of web data
By structure
Structured web data arrives in consistent fields: a price, a currency, a stock flag, a posting date. It is cheap to store and query, and it is what most pipelines are trying to produce. Semi-structured data has a schema but a loose one — JSON-LD product markup, sitemaps, RSS feeds, embedded API responses. Unstructured data is free text and media: review bodies, forum threads, PDFs, transcripts. It carries the richest signal and needs the most work, which is exactly why LLM extraction has taken over that end of the pipeline.
By origin
First-party web data is measured on properties you control. Third-party web data is collected from the open web about markets, competitors, employers or listings. Most real projects join the two: your conversion data is meaningless without the competitor price that existed on the same day.
First-party website data vs third-party web data
The phrase "website data" gets used for two unrelated things, which is why search results for it are a mess.
When people ask about website data on iPhone, they mean Safari's per-site storage list — cookies, caches and local storage held on the device. Clearing it deletes local state on that phone; it does nothing to server-side records. When people ask about website data in Google, they usually mean first-party measurement in Google Analytics or Search Console: sessions, pages, queries, impressions.
Neither is the same as third-party web data acquisition, where you fetch pages you do not own and turn them into rows. Same words, different engineering problem: one is a tag and a retention policy, the other is discovery, fetching, blocking and parsing.
What is web data example: five concrete ones
- Retail price and availability. URL, SKU, title, price, currency, in-stock flag, seller, captured-at. Useful only if refreshed on a known cadence — a price without a timestamp is a rumour.
- Job postings. Company, title, location, remote flag, posted date, salary range if stated. Aggregated, it becomes a hiring signal that leads revenue by a quarter or two.
- Company profiles. Domain, description, industry, HQ, contact page, technology hints — the raw material for lead lists and market maps.
- Reviews and forum threads. Unstructured text, ideal for sentiment and feature-request mining, terrible for anything requiring precise counts.
- Search engine result pages. The rank, title, snippet and URL for a query — both a competitive dataset in itself and the cheapest discovery mechanism for everything above.
Web data in data analytics
On the first-party side, Digital.gov's introduction to analytics gives the cleanest chain of reasoning: state a goal, define objectives, name the calls to action, pick KPIs that measure them, then set targets. Their warning is the one most teams ignore — do not focus on traffic alone; more pageviews and lower bounce rates are not user success.
The same discipline applies to third-party web data, and it is the main reason acquisition projects fail. Teams collect a million rows before deciding what decision the rows are supposed to change. Write the decision down first, then derive the schema, then derive the URL list. A useful third-party dataset is usually narrow: 40 fields across 20,000 entities, refreshed weekly, beats 4 million unvalidated pages of HTML.
How web data gets collected
Whatever the tool, the pipeline is the same six steps:
- Define the output schema. Field names, types, units, and what "missing" means.
- Discover URLs. Search queries via a SERP API for breadth; sitemap and homepage-link mapping for depth on known domains — see the crawl and map API for that side.
- Fetch. Plain HTTP where possible; a rendered browser only where content is JavaScript-injected. The difference is roughly 5x in cost, so test before you assume.
- Parse. CSS or XPath selectors when the layout is stable; LLM extraction against a schema when it is not, or when the source is free text.
- Validate. Range checks, null rates, duplicate keys, and a sample reviewed by a human who knows the domain.
- Refresh. Decide the cadence per field. Prices move daily, company HQs move once a decade.
Steps 2 to 4 are where blocking lives. A public page served to a browser is frequently not served to a datacenter IP, which is why collection sits on top of a proxy layer — residential proxies from $0.80/GB if you run your own fetcher, or an API that owns that problem for you. Our explainer on what a web scraper API is covers the trade-off in detail.
curl https://api.quanticdata.io/v1/scrape \
-H "Authorization: Bearer $QD_API_KEY" \
-d '{ "url": "https://example.com/product/123" }'
{ "success": true,
"data": { "markdown": "# Widget Pro\n$249.00 · In stock" },
"usage": { "cost_usd": 0.0002 } }
Dataset, API or your own scraper
There are three honest routes to third-party web data, and the right one depends on how specific and how fresh your question is.
| Route | Best when | Time to first row | Freshness control | Where the cost hides |
|---|---|---|---|---|
| Prebuilt dataset from a data marketplace | You need broad coverage of a well-known source and can live with someone else's schema | Days (contract, sample, delivery) | Vendor's refresh cycle | Annual minimums; paying for records you never query |
| On-demand web data API | The question is specific, the URL set changes, or you need a live refresh | Minutes | Yours, per call | Retries and rendered pages — unless failures are free |
| Self-hosted scraper plus proxies | Very high volume on a handful of stable targets | Weeks | Yours | Engineer time on blocks, layout changes and queue infrastructure |
Note the evaluation criteria rather than brand names: schema fit, refresh cadence, per-row provenance (source URL and capture timestamp), documented block-handling, and whether failed attempts are billed. That last one quietly decides your real unit cost on hard targets.
The cost math, honestly
Take a concrete brief: track 12,000 competitor product URLs across 40 domains, daily, and add 500 discovery searches a month.
- Discovery: 500 searches at $0.0005 = $0.25; 40 site maps at $0.0005 = $0.02.
- Extraction, plain HTML: 12,000 pages at $0.0002 = $2.40 per run, so about $72/month daily.
- If a third of those pages need JS rendering at $0.001: 4,000 x $0.001 + 8,000 x $0.0002 = $5.60 per run, about $168/month.
- Whole-site crawls, where you do not have the URL list, run $0.0003 per page with unfetched pages refunded when the job settles.
Two things dominate that arithmetic. First, rendering discipline: measure which domains genuinely need a browser instead of rendering everything by default. Second, failure billing. Under pay-per-success pricing a blocked or errored call costs $0.00, so a 15% block rate does not silently inflate your bill by 15% — it just means fewer rows that hour. Under per-request billing it does.
Web data for AI: RAG, agents and MCP
Web data is now mostly consumed by models rather than dashboards, and that changes the requirements. A RAG index wants clean Markdown with the boilerplate stripped, not raw HTML. An agent wants a single response envelope it can branch on, and a schema it can validate against. Both want provenance on every chunk, because a citation is the only way to audit a generated answer.
That is the shape of a web data API for AI: search, scrape, map, crawl, batch and audit exposed as tools, one success / data / usage envelope, and per-success billing so an agent that retries ten times does not produce a surprise invoice. Wired through an MCP server, the model calls those tools directly from Claude, Cursor or your own runtime — no glue code per source. If the protocol layer is new to you, is an MCP server like an API? explains where it differs.
curl https://api.quanticdata.io/v1/serp \
-H "Authorization: Bearer $QD_API_KEY" \
-d '{ "query": "industrial pump supplier germany" }'
# success:false -> machine-readable error code, $0.00 charged
Quality and compliance checks before you trust a row
Run these on every new source, and again whenever the source redesigns:
- Provenance. Every row carries its source URL and capture timestamp, or it is not evidence.
- Completeness. Null rate per field, plus a check for collection bias — a review dataset with only five-star entries has a sampling bug, not a happy customer base.
- Duplicates and drift. Stable entity keys; alert when a selector's yield drops sharply, which usually means a layout change rather than a market change.
- Access basis. Public, anonymous access only. Logins, paywalls and personal data raise separate obligations under GDPR, CCPA and site terms.
On the legal side, treat public availability as a starting point, not a permission slip: robots directives, terms of service, copyright and data-protection law all apply independently. Our post on whether web scraping is legal in the US walks through the leading cases. None of this is legal advice — if a project touches personal data or contested sources, get counsel before you scale it.
Web data, in the end, is not a mysterious asset class. It is public pages plus your own event stream, reduced to fields you can join, with a date on every row. The engineering work is in the reduction, and the economics are in not paying for the attempts that failed.