# How to Use Data for AI

> How to use data for AI in practice: where AI gets its data, a six-step readiness workflow, MCP tool patterns for agents, and per-page cost math you can check.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/How to Use Data for AI

# How to use data for AI: sources, pipelines and honest cost math

Data for AIJul 29, 2026·10 min read·QuanticData Team

On this page [The three jobs people mean by "using data for AI"](/blog/how-to-use-data-for-ai/#the-three-jobs-people-mean-by-using-data-for-ai) [Where does AI get its data from?](/blog/how-to-use-data-for-ai/#where-does-ai-get-its-data-from) [A six-step workflow to get data ready for AI](/blog/how-to-use-data-for-ai/#a-six-step-workflow-to-get-data-ready-for-ai) [How to use AI for data analysis (and where the free tier stops)](/blog/how-to-use-data-for-ai/#how-to-use-ai-for-data-analysis-and-where-the-free-tier-stop) [Serving data to agents: tools beat batch exports](/blog/how-to-use-data-for-ai/#serving-data-to-agents-tools-beat-batch-exports) [Honest cost math](/blog/how-to-use-data-for-ai/#honest-cost-math) [Governance: what you may point AI at](/blog/how-to-use-data-for-ai/#governance-what-you-may-point-ai-at) [How to evaluate a data-for-AI stack](/blog/how-to-use-data-for-ai/#how-to-evaluate-a-data-for-ai-stack)

Using data for AI means three separate jobs: training or fine-tuning a model on a corpus, grounding a running model in facts it never memorised, and analysing your own data with AI as an assistant. Each needs different freshness, volume and cost discipline. This guide covers all three, with code and numbers.

## The three jobs people mean by "using data for AI"

Most confusion about data and AI comes from collapsing three very different workloads into one word. They have different failure modes, so they need different pipelines.

| Job | What the data does | Volume | Freshness need | Main failure mode |
| --- | --- | --- | --- | --- |
| Pre-training / fine-tuning | Shapes the weights: style, domain vocabulary, task format | Very high (millions of documents) or very curated (hundreds of examples) | Low — a snapshot is fine | Contamination, duplication, licensing gaps |
| Grounding (RAG and agent tools) | Supplies facts at answer time; the model reasons over text you hand it | Moderate, query-driven | High — hours or minutes | Stale or missing retrieval, no provenance |
| Analysis | The model is the analyst; data stays in your warehouse or spreadsheet | Whatever the question needs | Matches your reporting cycle | Confident wrong answers, silent schema drift |

A fine-tune will not make a model know yesterday's prices. A retrieval pipeline will not teach it a new output format reliably. Pick the job first, then pick the data.

## Where does AI get its data from?

Foundation models are trained mostly on large public text collections — open web crawls, encyclopaedic sources, code repositories, licensed archives — then cleaned, filtered and de-duplicated before a single training step runs. VAST Data's overview of the AI data pipeline lists [public datasets such as Common Crawl and Wikipedia alongside customer data and industry archives](https://www.vastdata.com/blog/how-does-ai-get-its-data) as the standard ingestion mix, followed by cleaning and transformation stages that strip markup, menus and boilerplate.

That answers the training half. The other half — where a model gets its *answers* — is more mundane: from the prompt, from whatever documents your retrieval layer attached to the prompt, and from tools it can call. Anything outside that window is either memorised (and possibly out of date) or invented.

### Your own data is usually the bottleneck

Internal data is the differentiator, and it is also where projects stall. IBM reports that [82% of enterprises experience data silos that stymie key workflows, and as much as 68% of organisational data never gets analysed](https://www.ibm.com/think/topics/ai-data-management). Before you buy anything, run a discovery pass: what exists, who owns it, what classification it carries, and whether it is queryable at all.

### The live web fills the gaps

Almost every useful AI application needs facts that are not in your warehouse and were not in the training set: current prices, current job postings, current company details, current documentation. That is public web data, fetched on demand. Our primer on [what web data is](https://quanticdata.io/blog/what-is-web-data/) breaks down the categories; the practical point is that this is the only tier of your stack that has to be fresh at request time.

## A six-step workflow to get data ready for AI

1. **Write the question down.** "Which of our 40 competitors changed price this week" is a data spec. "Use AI on our data" is not. The question determines schema, refresh interval and acceptable error rate.

2. **Inventory and classify what you already hold.** Tag every source public, internal, restricted or regulated. This single step decides which tools you may legally point at it.

3. **Fill the gaps from the open web in three moves:** search to discover candidate URLs, map a site to enumerate its URLs, then scrape only the pages that matter. Discovery before extraction keeps volume — and cost — down.

4. **Normalise to one text-first shape.** Markdown for prose, JSON for fields, and a fixed metadata block on every record: source URL, fetch timestamp, content hash, extraction method. Without provenance you cannot debug a bad answer or defend a good one.

5. **De-duplicate and chunk.** Near-duplicate pages inflate embedding bills and skew retrieval. Chunk on headings, not fixed character counts, so a chunk is still a coherent thought.

6. **Evaluate with a frozen question set.** Twenty to fifty questions with known answers, run on every pipeline change. Track retrieval hit rate separately from answer quality — otherwise you will tune the model when the fetcher is the problem.

Steps 3 and 4 are where teams burn months building infrastructure. A hosted [web scraping API](https://quanticdata.io/web-scraping-api/) collapses fetch, retry, proxy rotation and Markdown conversion into one call, and a [crawl and map API](https://quanticdata.io/crawl-map/) handles the enumeration half. If you would rather own the fetch layer, you still need an exit network: [residential proxies](https://quanticdata.io/residential-proxies/) from $0.80/GB, billed per GB over the plan term, are the usual base for self-hosted crawlers.

## How to use AI for data analysis (and where the free tier stops)

For analysis, AI is a fast junior analyst, not an oracle. Intuit's guide to AI-assisted analysis cites a Nucleus Research finding that [AI-powered analytics improved productivity by 27% to 43%](https://www.intuit.com/blog/innovative-thinking/using-ai-for-data-science-analysis/) — gains that come from compressing prep and exploration, not from replacing judgement.

The reliable pattern, in order of trustworthiness:

- **Code generation over direct answers.** Ask the model for the SQL, pandas or Excel formula, then run it yourself. The output is reviewable and reproducible; a number typed into chat is neither.

- **Schema and profile first.** Paste column names, types and a few rows — not the whole dataset. The model needs structure, not volume.

- **Spreadsheet copilots for shape work.** AI data analysis in Excel or Sheets is strongest at formula authoring, pivot suggestions, text cleanup and flagging outliers. It is weakest at anything requiring context outside the file.

- **Narration last.** Let the model turn verified numbers into a summary for stakeholders. That is genuinely good at it.

On free options: the built-in assistants in mainstream spreadsheets and notebooks, plus free tiers of chat models with file upload, cover most learning and one-off analysis. What free tiers do not give you is stable throughput, provenance, or permission to send restricted data — see the governance section. Judge tools by criteria, not brand: does it show the code it ran, does it state its assumptions, can you pin a version, and can you audit what data left your network.

## Serving data to agents: tools beat batch exports

Once an LLM can call tools, the shape of the data problem changes. You stop pre-loading a corpus and start answering questions on demand. An agent that can search, map, scrape and crawl needs no nightly export to answer "what does this vendor charge today".

That is the pattern behind our [web data API for AI](https://quanticdata.io/web-data-api-for-ai/): the same endpoints exposed as agent tools through [an MCP server](https://quanticdata.io/mcp-server/), returning one JSON envelope so the model always parses the same shape. If you are weighing protocols, [how MCP compares to a plain API](https://quanticdata.io/blog/is-mcp-server-like-an-api/) is worth ten minutes before you wire anything.

The single-page call, which is also the unit an agent invokes:

```
curl https://api.quanticdata.io/v1/scrape \
  -H "Authorization: Bearer $QD_API_KEY" \
  -d '{ "url": "https://example.com/pricing", "formats": ["markdown"] }'

{ "success": true,
  "data":    { "markdown": "# Pricing\n…" },
  "usage":   { "cost_usd": 0.0002 } }
```

For corpus building, batch the URLs you already discovered and poll the job:

```
import os, time, requests

h = {"Authorization": f"Bearer {os.environ['QD_API_KEY']}"}

job = requests.post("https://api.quanticdata.io/v1/batch", headers=h,
                    json={"urls": urls, "formats": ["markdown"]}).json()

job_id = job["data"]["jobId"]
while True:
    r = requests.get(f"https://api.quanticdata.io/v1/batch/{job_id}", headers=h).json()
    if r["data"]["status"] in ("completed", "failed"):
        break
    time.sleep(5)

for page in r["data"]["pages"]:
    index(page["markdown"], source=page["url"], fetched_at=page["fetched_at"])
```

Note the last line. Indexing without `source` and `fetched_at` is the most common self-inflicted wound in RAG builds: when an answer is wrong, you have no way to tell whether the retrieval missed, the page changed, or the extraction broke.

## Honest cost math

Data for AI is priced per unit of work, and the units are small enough that people guess badly in both directions. List prices, with the arithmetic done:

| Work | Endpoint | List price | 10,000 units |
| --- | --- | --- | --- |
| Page to Markdown, no JS | `/v1/scrape` | $0.0002/page | $2.00 |
| Page with JS rendering | `/v1/scrape` | $0.001/page | $10.00 |
| Search results as JSON | `/v1/serp` | from $0.0005/search | $5.00 |
| Enumerate a site's URLs | `/v1/map` | $0.0005/call | $5.00 |
| Whole-site crawl | `/v1/crawl` | $0.0003/page | $3.00 |
| No-JS vs rendered audit | `/v1/seo-audit` | $0.0012/URL | $12.00 |

A 50,000-page domain corpus, crawled and converted, is $15 of fetch. The embedding, storage and inference around it will cost more than the acquisition — which is exactly why per-page fetch cost is the wrong thing to over-engineer, and per-page *reliability* is the right thing to care about.

Two mechanics matter more than the headline rate. First, pay per success: a failed call returns `success: false` and costs nothing, so blocked pages do not quietly bill you. Second, async jobs are charged on requested volume and auto-refund the unfetched share when they settle, so an over-specified crawl does not become an over-specified invoice. Compare that with per-seat dashboards, where the bill is identical whether you pull 100 rows or 100,000. For discovery-heavy work, the same logic applies to a [SERP API](https://quanticdata.io/serp-api/) billed per search rather than per month.

## Governance: what you may point AI at

Two questions decide whether a data-for-AI project survives review: where does the prompt go, and where did the corpus come from.

On prompts, university IT guidance is refreshingly blunt. The University of Iowa's ITS advises that [public AI tools store prompts and responses on the vendor's systems and that sensitive or restricted data should never be pasted into them](https://its.uiowa.edu/news/2025/12/how-ai-actually-works-your-data) without a security review — and that even with an opt-out, you should assume anything sent to a consumer tool could be reviewed inside that service. Classify data first, then choose the lowest-risk tool that can do the job.

On corpora, the relevant layers are site terms, robots directives, copyright and database rights, and personal-data law such as the GDPR. Public accessibility is not the same as permission to redistribute, and "we only used it for training" is not a legal category in most jurisdictions. Our write-ups on [AI web scraping and the law](https://quanticdata.io/blog/is-ai-web-scraping-legal/) and [web scraping legality in the US](https://quanticdata.io/blog/is-web-scraping-legal-in-us/) map the layers in detail. None of this is legal advice; if your dataset touches personal data, paywalled content or a regulated industry, get counsel before you index it.

## How to evaluate a data-for-AI stack

Ignore feature lists. Score candidates — build or buy — on seven things:

- **Freshness guarantee.** Can you get a page fetched in this request, or only from a cache of unknown age?

- **Provenance per record.** Source URL, timestamp and extraction method, or nothing.

- **Cost per successful record**, not per attempt, per month or per seat.

- **Failure semantics.** Machine-readable error codes and a retry log beat a 200 response containing a block page.

- **Rendering fallback.** Cheap HTTP fetch by default, JS rendering when the DOM demands it — chosen per URL, not per plan.

- **Geo and locale control.** Prices, availability and search results differ by country; a stack that cannot target one is measuring the wrong market.

- **Schema stability.** One envelope across endpoints means agents and parsers do not break when you add a source.

Get those seven right and the AI part becomes the easy part. Models improve every few months without you doing anything; a pipeline that cannot say where a number came from stays broken until you fix it.

### Sources & further reading

- [What is AI data management? — IBM Think](https://www.ibm.com/think/topics/ai-data-management)

- [How Does AI Get Its Data? Understanding the Fuel Behind the Intelligence — VAST Data](https://www.vastdata.com/blog/how-does-ai-get-its-data)

- [How to Use AI for Data Analysis: Workflows, Use Cases, and Best Practices — Intuit Blog](https://www.intuit.com/blog/innovative-thinking/using-ai-for-data-science-analysis/)

- [How AI actually works with your data — University of Iowa ITS](https://its.uiowa.edu/news/2025/12/how-ai-actually-works-your-data)

## FAQ

Quick answers on how to use data for ai.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### Where does AI get its data from?

Training data comes mostly from large public collections — open web crawls, encyclopaedic sources, code repositories and licensed archives — plus proprietary and customer data, all cleaned and de-duplicated first. At answer time, a model uses only its prompt, retrieved documents and tool results. Anything outside that window is memorised and possibly stale.

### Where does AI get its answers from?

Three places: patterns learned during training, text you attached to the prompt, and live tool calls such as search or a scrape. If none of those contain the fact, the model may still produce a fluent guess. Grounding with retrieval plus source URLs is the only reliable fix.

### How do I use AI for data analysis?

Give the model schema and a sample rather than the whole dataset, ask it for code (SQL, pandas, a formula) instead of a final number, run that code yourself, then let the model write the plain-language summary. Keep a frozen set of questions with known answers to check its work.

### Is there a good free AI for data analysis?

Free tiers of mainstream chat models and the built-in assistants in spreadsheets and notebooks handle learning and one-off analysis well. Judge them on whether they show the code they ran, state assumptions, and let you control where data goes — not on brand. Free tiers rarely offer throughput guarantees or approval for restricted data.

### Can AI analyse data in Excel?

Yes, and it is strongest at formula authoring, pivot suggestions, text cleanup and outlier flagging inside the sheet. It is weakest at anything needing context outside the file. Ask it to insert formulas you can inspect rather than pasted static values, so the workbook stays auditable and recalculates correctly.

### What does an AI data analyst earn?

Ranges vary widely by country, seniority and industry, and public aggregator figures are noisy self-reported samples. Treat them as a rough band, not a benchmark. What consistently raises value is the pipeline side of the role: provenance, evaluation harnesses and cost control, rather than prompt-writing alone.

## Feed your AI fresh data, pay only for successes

Scrape, search, map and crawl the open web through one API with a single JSON envelope — from $0.0002 per page, with failed calls charged at $0.00. Start on $2 of free usage every month, no card, and wire the same tools into Claude or Cursor via MCP.

[Start free — $2/month included](https://app.quanticdata.io/register)[Explore Web Data API for AI Agents](https://quanticdata.io/web-data-api-for-ai/)

## Related reading

[Data for AI What is web data? Types, examples and uses A precise definition of web data, its types and examples, how it is collected, and what it actually costs to acquire at scale for analytics and AI. Read →](https://quanticdata.io/blog/what-is-web-data/) [Data for AI How to Get Data for AI Five real sources of AI training data, how to judge them, and the cost math nobody publishes — plus working API calls for live web data. Read →](https://quanticdata.io/blog/how-to-get-data-for-ai/) [SEO data How to Perform an SEO Audit A practical six-step SEO audit process with a checklist, the crawler-vs-user diff most audits skip, and how to run the whole thing programmatically. Read →](https://quanticdata.io/blog/how-to-perform-an-seo-audit/)

---

Source: https://quanticdata.io/blog/how-to-use-data-for-ai/ · Site index for AI: https://quanticdata.io/llms.txt