Using data for AI means three separate jobs: training or fine-tuning a model on a corpus, grounding a running model in facts it never memorised, and analysing your own data with AI as an assistant. Each needs different freshness, volume and cost discipline. This guide covers all three, with code and numbers.
The three jobs people mean by "using data for AI"
Most confusion about data and AI comes from collapsing three very different workloads into one word. They have different failure modes, so they need different pipelines.
| Job | What the data does | Volume | Freshness need | Main failure mode |
|---|---|---|---|---|
| Pre-training / fine-tuning | Shapes the weights: style, domain vocabulary, task format | Very high (millions of documents) or very curated (hundreds of examples) | Low — a snapshot is fine | Contamination, duplication, licensing gaps |
| Grounding (RAG and agent tools) | Supplies facts at answer time; the model reasons over text you hand it | Moderate, query-driven | High — hours or minutes | Stale or missing retrieval, no provenance |
| Analysis | The model is the analyst; data stays in your warehouse or spreadsheet | Whatever the question needs | Matches your reporting cycle | Confident wrong answers, silent schema drift |
A fine-tune will not make a model know yesterday's prices. A retrieval pipeline will not teach it a new output format reliably. Pick the job first, then pick the data.
Where does AI get its data from?
Foundation models are trained mostly on large public text collections — open web crawls, encyclopaedic sources, code repositories, licensed archives — then cleaned, filtered and de-duplicated before a single training step runs. VAST Data's overview of the AI data pipeline lists public datasets such as Common Crawl and Wikipedia alongside customer data and industry archives as the standard ingestion mix, followed by cleaning and transformation stages that strip markup, menus and boilerplate.
That answers the training half. The other half — where a model gets its answers — is more mundane: from the prompt, from whatever documents your retrieval layer attached to the prompt, and from tools it can call. Anything outside that window is either memorised (and possibly out of date) or invented.
Your own data is usually the bottleneck
Internal data is the differentiator, and it is also where projects stall. IBM reports that 82% of enterprises experience data silos that stymie key workflows, and as much as 68% of organisational data never gets analysed. Before you buy anything, run a discovery pass: what exists, who owns it, what classification it carries, and whether it is queryable at all.
The live web fills the gaps
Almost every useful AI application needs facts that are not in your warehouse and were not in the training set: current prices, current job postings, current company details, current documentation. That is public web data, fetched on demand. Our primer on what web data is breaks down the categories; the practical point is that this is the only tier of your stack that has to be fresh at request time.
A six-step workflow to get data ready for AI
- Write the question down. "Which of our 40 competitors changed price this week" is a data spec. "Use AI on our data" is not. The question determines schema, refresh interval and acceptable error rate.
- Inventory and classify what you already hold. Tag every source public, internal, restricted or regulated. This single step decides which tools you may legally point at it.
- Fill the gaps from the open web in three moves: search to discover candidate URLs, map a site to enumerate its URLs, then scrape only the pages that matter. Discovery before extraction keeps volume — and cost — down.
- Normalise to one text-first shape. Markdown for prose, JSON for fields, and a fixed metadata block on every record: source URL, fetch timestamp, content hash, extraction method. Without provenance you cannot debug a bad answer or defend a good one.
- De-duplicate and chunk. Near-duplicate pages inflate embedding bills and skew retrieval. Chunk on headings, not fixed character counts, so a chunk is still a coherent thought.
- Evaluate with a frozen question set. Twenty to fifty questions with known answers, run on every pipeline change. Track retrieval hit rate separately from answer quality — otherwise you will tune the model when the fetcher is the problem.
Steps 3 and 4 are where teams burn months building infrastructure. A hosted web scraping API collapses fetch, retry, proxy rotation and Markdown conversion into one call, and a crawl and map API handles the enumeration half. If you would rather own the fetch layer, you still need an exit network: residential proxies from $0.80/GB, billed per GB over the plan term, are the usual base for self-hosted crawlers.
How to use AI for data analysis (and where the free tier stops)
For analysis, AI is a fast junior analyst, not an oracle. Intuit's guide to AI-assisted analysis cites a Nucleus Research finding that AI-powered analytics improved productivity by 27% to 43% — gains that come from compressing prep and exploration, not from replacing judgement.
The reliable pattern, in order of trustworthiness:
- Code generation over direct answers. Ask the model for the SQL, pandas or Excel formula, then run it yourself. The output is reviewable and reproducible; a number typed into chat is neither.
- Schema and profile first. Paste column names, types and a few rows — not the whole dataset. The model needs structure, not volume.
- Spreadsheet copilots for shape work. AI data analysis in Excel or Sheets is strongest at formula authoring, pivot suggestions, text cleanup and flagging outliers. It is weakest at anything requiring context outside the file.
- Narration last. Let the model turn verified numbers into a summary for stakeholders. That is genuinely good at it.
On free options: the built-in assistants in mainstream spreadsheets and notebooks, plus free tiers of chat models with file upload, cover most learning and one-off analysis. What free tiers do not give you is stable throughput, provenance, or permission to send restricted data — see the governance section. Judge tools by criteria, not brand: does it show the code it ran, does it state its assumptions, can you pin a version, and can you audit what data left your network.
Serving data to agents: tools beat batch exports
Once an LLM can call tools, the shape of the data problem changes. You stop pre-loading a corpus and start answering questions on demand. An agent that can search, map, scrape and crawl needs no nightly export to answer "what does this vendor charge today".
That is the pattern behind our web data API for AI: the same endpoints exposed as agent tools through an MCP server, returning one JSON envelope so the model always parses the same shape. If you are weighing protocols, how MCP compares to a plain API is worth ten minutes before you wire anything.
The single-page call, which is also the unit an agent invokes:
curl https://api.quanticdata.io/v1/scrape \
-H "Authorization: Bearer $QD_API_KEY" \
-d '{ "url": "https://example.com/pricing", "formats": ["markdown"] }'
{ "success": true,
"data": { "markdown": "# Pricing\n…" },
"usage": { "cost_usd": 0.0002 } }For corpus building, batch the URLs you already discovered and poll the job:
import os, time, requests
h = {"Authorization": f"Bearer {os.environ['QD_API_KEY']}"}
job = requests.post("https://api.quanticdata.io/v1/batch", headers=h,
json={"urls": urls, "formats": ["markdown"]}).json()
job_id = job["data"]["jobId"]
while True:
r = requests.get(f"https://api.quanticdata.io/v1/batch/{job_id}", headers=h).json()
if r["data"]["status"] in ("completed", "failed"):
break
time.sleep(5)
for page in r["data"]["pages"]:
index(page["markdown"], source=page["url"], fetched_at=page["fetched_at"])Note the last line. Indexing without source and fetched_at is the most common self-inflicted wound in RAG builds: when an answer is wrong, you have no way to tell whether the retrieval missed, the page changed, or the extraction broke.
Honest cost math
Data for AI is priced per unit of work, and the units are small enough that people guess badly in both directions. List prices, with the arithmetic done:
| Work | Endpoint | List price | 10,000 units |
|---|---|---|---|
| Page to Markdown, no JS | /v1/scrape | $0.0002/page | $2.00 |
| Page with JS rendering | /v1/scrape | $0.001/page | $10.00 |
| Search results as JSON | /v1/serp | from $0.0005/search | $5.00 |
| Enumerate a site's URLs | /v1/map | $0.0005/call | $5.00 |
| Whole-site crawl | /v1/crawl | $0.0003/page | $3.00 |
| No-JS vs rendered audit | /v1/seo-audit | $0.0012/URL | $12.00 |
A 50,000-page domain corpus, crawled and converted, is $15 of fetch. The embedding, storage and inference around it will cost more than the acquisition — which is exactly why per-page fetch cost is the wrong thing to over-engineer, and per-page reliability is the right thing to care about.
Two mechanics matter more than the headline rate. First, pay per success: a failed call returns success: false and costs nothing, so blocked pages do not quietly bill you. Second, async jobs are charged on requested volume and auto-refund the unfetched share when they settle, so an over-specified crawl does not become an over-specified invoice. Compare that with per-seat dashboards, where the bill is identical whether you pull 100 rows or 100,000. For discovery-heavy work, the same logic applies to a SERP API billed per search rather than per month.
Governance: what you may point AI at
Two questions decide whether a data-for-AI project survives review: where does the prompt go, and where did the corpus come from.
On prompts, university IT guidance is refreshingly blunt. The University of Iowa's ITS advises that public AI tools store prompts and responses on the vendor's systems and that sensitive or restricted data should never be pasted into them without a security review — and that even with an opt-out, you should assume anything sent to a consumer tool could be reviewed inside that service. Classify data first, then choose the lowest-risk tool that can do the job.
On corpora, the relevant layers are site terms, robots directives, copyright and database rights, and personal-data law such as the GDPR. Public accessibility is not the same as permission to redistribute, and "we only used it for training" is not a legal category in most jurisdictions. Our write-ups on AI web scraping and the law and web scraping legality in the US map the layers in detail. None of this is legal advice; if your dataset touches personal data, paywalled content or a regulated industry, get counsel before you index it.
How to evaluate a data-for-AI stack
Ignore feature lists. Score candidates — build or buy — on seven things:
- Freshness guarantee. Can you get a page fetched in this request, or only from a cache of unknown age?
- Provenance per record. Source URL, timestamp and extraction method, or nothing.
- Cost per successful record, not per attempt, per month or per seat.
- Failure semantics. Machine-readable error codes and a retry log beat a 200 response containing a block page.
- Rendering fallback. Cheap HTTP fetch by default, JS rendering when the DOM demands it — chosen per URL, not per plan.
- Geo and locale control. Prices, availability and search results differ by country; a stack that cannot target one is measuring the wrong market.
- Schema stability. One envelope across endpoints means agents and parsers do not break when you add a source.
Get those seven right and the AI part becomes the easy part. Models improve every few months without you doing anything; a pipeline that cannot say where a number came from stays broken until you fix it.