AI readiness API

An AI readiness API that audits how a website presents itself to AI systems and returns one row per domain: which of 13 AI crawlers are allowed, partially fenced or blocked, whether llms.txt exists and how many links it lists, how much text the page serves without JavaScript, and a published 0–100 score. Pass a list of domains — yours, your clients' or your competitors' — and get the dataset behind answer-engine optimization.

By Aldo Morese, founder of QuanticData · Published Aug 21, 2026 · Updated Sep 3, 2026

Get my free API key See the request

$0.002 per delivered site · $2 free every month · Failed runs never billed

POST /v1/scraper/collectors/ai_readiness/run

$ curl $QD/ai_readiness/run \
    -H "Authorization: Bearer $QD_API_KEY" \
    -d '{"domains": ["quanticdata.io", "stripe.com"]}'
{ "status": "done", "count": 10,
  "results": [
    {
      "domain": "…",
      "url": "…",
      "robots_txt": false,
      "ai_bots": "…" } ],
  "cost": 0.02 }
# 10 sites × $0.002 · nothing delivered, nothing charged

AI readiness API: domains and country in, a residential exit in the middle, delivered rows with domain, robots_txt and llms_txt on the right, $0.002 per site. — You send domains and country; the run goes out through a residential exit and comes back as sites with domain, robots_txt and llms_txt — 21 fields on every row — and you are billed $0.002 for each site actually delivered, nothing for a run that delivers none.

$0.002 / site1,000 sites on the free $2 every month

Semantic inputdomains, country, max_results — no URL lists

Up to 50sites per run, pagination handled for you

No browserread over HTTP/TLS — cheaper and faster than rendering

Try it

AI readiness audit, running now

Change the input and run it against the live collector — nothing to install, no sign-up.

Run it from your own code, on your own inputs

Same collector, same rows — $2 of free API credit every month, no card.

Get my free API key

What an AI readiness API does

Every robots.txt answer here is computed with RFC 9309 precedence, not with a substring search: the group that applies is the longest matching user-agent, with * counting as length zero, and inside it the longest matching rule wins with Allow taking the tie. That is why a verdict is three-valued instead of two: blocked is a disallow at the root, partial means the crawler is welcome except on fenced paths, and allowed means no restriction. A site with no robots.txt at all is reported as allowed — that is a verdict about the site, not an error about the fetch.

The rest of the row is the other half of the question, the one a robots checker never asks: whether there is anything for an AI crawler to read once it is let in. ssr_text_chars counts the visible text the page serves without JavaScript — what an LLM crawler that does not render actually sees — and js_dependent flags the pages that serve almost nothing. Add llms.txt and llms-full.txt (validated as real Markdown, never a courtesy HTML page returned by a catch-all route), the sitemaps robots declares, the schema.org types on the homepage, and noai meta and header signals, and the ai_readiness_score composes them with its formula printed in the field description: crawler access 55, llms.txt 15, no-JS text 15, sitemap 10, structured data 5.

Limits, in plain numbers

Everything that bounds one run of this collector. No hidden throttles.

Max per run

50 sites

Price

$0.002 / site

Per 1,000

$2.00

Failed runs

Free zero rows, zero charge

Free every month

$2 no card

Rate limit

60 req/min on the free tier

What one site looks like

Every delivered site carries these fields. Nullable means the source did not publish it — the field stays empty instead of being guessed.

Field	Type	What it holds
`domain`	string	Registrable host (no www).
`url`	string	Homepage URL fetched.
`robots_txt`	boolean	Whether a robots.txt exists.
`ai_bots`	object	Per-crawler verdict: { gptbot: "allowed\|partial\|blocked", claudebot: …, … }.
`allowed_bots`	string[]	AI crawlers with no restrictions.
`partial_bots`	string[]	AI crawlers allowed at the root but fenced off some paths.
`blocked_bots`	string[]	AI crawlers disallowed from the whole site.
`llms_txt`	boolean	Whether /llms.txt exists and is valid Markdown.
`llms_txt_url`	string · nullable	URL of the llms.txt when present.
`llms_txt_links`	integer	Links listed in the llms.txt.
`llms_full_txt`	boolean	Whether /llms-full.txt exists.
`sitemap`	boolean	Whether robots.txt declares at least one sitemap.
`sitemaps`	string[]	Declared sitemap URLs (max 10).
`noai_meta`	boolean	Homepage carries a noai/noimageai robots meta.
`x_robots_noai`	boolean	X-Robots-Tag response header carries noai.
`jsonld_types`	string[]	schema.org @type values found on the homepage.
`og_meta`	boolean	OpenGraph tags present.
`h1`	boolean	Homepage has an h1.
`ssr_text_chars`	integer	Visible text characters served without JavaScript.
`js_dependent`	boolean	True when the no-JS page has almost no text (< 400 chars).
`ai_readiness_score`	integer	0–100: crawler access (55) + llms.txt (15) + no-JS text (15) + sitemap (10) + structured data (5).

Inputs

The whole request. Anything you leave out falls back to the default shown in the catalog.

Input	Type	Required	What it does
`domains`	array	yes	Domains or site URLs, one per line (max 50).
`country`	string	no	ISO 3166-1 alpha-2 code — proxy exit geo and Google locale (gl). Omit for the default pool.
`max_results`	integer	no	How many sites to deliver at most (1–50). You pay only for delivered sites.

Pricing

AI readiness API pricing

$0.002 per delivered site. A run that delivers nothing costs nothing: blocked pages, challenges and retries are on us, and the $2 monthly allowance covers about 1,000 sites before you spend anything.

$0.002per delivered site$2 per 1,000 delivered sites

1,000 siteson the free allowance$2 every month, no card

Zero rowszero chargeblocks, captchas and retries are on us

−30%on volume tiersthe catalog returns your key's price

Pay as you go

$0/mo

$2 free credit / month
60 requests / min
List unit prices

Starter

$19/mo

$15 free credit / month
300 requests / min
10% off unit prices

Growth

$79/mo

$50 free credit / month
600 requests / min
20% off unit prices

Scale

$299/mo

$250 free credit / month
1,200 requests / min
30% off unit prices

Same wallet, same key and same $2 monthly allowance as every other Data API. Prices are launch pricing read live from the billing config — GET /v1/scraper/collectors returns the price your key actually pays.

Integration

One POST, typed rows

Base URL https://api.quanticdata.io/v1, Bearer auth, the same key as every other Data API. Endpoint: POST /v1/scraper/collectors/ai_readiness/run.

curl -X POST https://api.quanticdata.io/v1/scraper/collectors/ai_readiness/run \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domains":["quanticdata.io","stripe.com"]}'

import requests

r = requests.post(
    "https://api.quanticdata.io/v1/scraper/collectors/ai_readiness/run",
    headers={"Authorization": f"Bearer {QD_API_KEY}"},
    json={
        "domains": [
            "quanticdata.io",
            "stripe.com"
        ]
    },
    timeout=120,
)
for row in r.json()["payload"]["results"]:
    print(row)

const res = await fetch(
  "https://api.quanticdata.io/v1/scraper/collectors/ai_readiness/run",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.QD_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({"domains":["quanticdata.io","stripe.com"]}),
  },
);
const { payload } = await res.json();
console.table(payload.results);

claude mcp add quanticdata \
  -e QUANTICDATA_API_KEY=qd_live_your_key_here \
  -- npx -y quanticdata-mcp

# then, in the chat:
> run the ai_readiness collector with domains=["quanticdata.io","stripe.com"]

What people build with the AI readiness API

Three shapes of work this endpoint was designed around.

Answer-engine optimization

Run your own domain and the competitors ranking in AI answers side by side. The score says where you stand; the columns say why — a blocked ClaudeBot, a JS-only homepage and a missing llms.txt are three different fixes with three different owners.

Agency portfolio audits

One call over every client domain produces a comparable table instead of a folder of screenshots — and it re-runs monthly to show what the last quarter of work actually moved.

AI-policy monitoring

Sites change their mind about GPTBot and CCBot quietly. Schedule the same domain list and diff blocked_bots run over run to see who closed the door, and when.

AI readiness API versus rolling your own

The differences that actually cost time when you build this in-house.

	DIY scraper	This collector
robots.txt reading	A substring match that misreads groups and precedence	RFC 9309: longest agent match, longest rule, Allow wins ties
What it measures	Crawler access only	Access + llms.txt + no-JS text + sitemap + structured data
Scale	One domain at a time, in a browser tab	Up to 50 domains per run, one comparable row each

What people search for

Live autocomplete demand around AI readiness audit, pulled with our own Keyword ideas collector.

AI readiness API: the POST to /v1/scraper/collectors/ai_readiness/run with domains and country on the left, and the JSON envelope on the right with 20 sites and usage.cost_usd $0.04. — The same call you would paste into a terminal: a Bearer key, domains and country in the body, and back the envelope every QuanticData endpoint returns — type, message, payload — where count is how many sites arrived and usage.cost_usd is $0.04, which is 20 × $0.002. A run that delivers nothing costs nothing. The key is good for 60 req/min on the free tier.

Sources and standards

The platform documentation and standards this collector is built against — check any claim on this page against the primary source:

FAQ

Questions we get about the AI readiness API.

Something else? Ask us →

Which AI crawlers does it check?

Thirteen: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot and DuckAssistBot. Each gets its own verdict in ai_bots, plus the three roll-up arrays.

What does the score actually measure?

The formula ships in the field itself, so nothing is hidden: crawler access is worth 55 points, a valid llms.txt 15, text served without JavaScript 15, a sitemap declared in robots.txt 10, and structured data 5. It is a description of a site's posture towards AI, not a ranking prediction — no one, including us, can promise the second.

How is this different from your free AI crawler checker?

The free tool answers one question about one domain in a browser, with no account. This collector answers the whole question for up to 50 domains per run and returns it as rows you can store, diff and join — same verdicts underneath, different shape and scale.

Does it need to render the page?

No, and that is the point: three or four small fetches per domain — robots.txt, llms.txt, the homepage — with no browser at all. Reading the page exactly as a non-rendering crawler reads it is what makes ssr_text_chars meaningful in the first place.

Is there a free AI readiness API?

Every account gets $2 of credit every month with no card, which is about 1,000 delivered sites on this endpoint at $0.002 each. It renews monthly, and a run that delivers nothing is never billed — so a failed or blocked attempt does not eat the allowance.

How much does one run cost?

Multiply the rows you actually receive by $0.002. A run capped at 50 sites — the maximum for this collector — costs $0.1 if every row comes back, and less when the source has fewer. Volume tiers take up to 30% off, and GET /v1/scraper/collectors returns the price your key actually pays.

Run the AI readiness API now

$2 of free credit every month, no card. Your key returns its own prices from GET /v1/scraper/collectors.

Get my free API key