Collectors · SEO & keywords

AI readiness API in Python

Which AI crawlers a site lets in, llms.txt adoption, no-JS readability — one audit row per domain. One POST from Python with domains, rows back as JSON with 28 fields, $0.003 per delivered site. No proxies or parsers to maintain.

Get an API key See the collector

A Python requests snippet that posts { — A Python requests call to the ai_readiness collector returns one dict per site, at $0.003 each.

First call

Install, call, read the rows.

pip install requests

The only dependency.

import os, requests

r = requests.post(
    "https://api.quanticdata.io/v1/scraper/collectors/ai_readiness/run",
    headers={"Authorization": f"Bearer {os.environ['QD_API_KEY']}"},
    json={
        "domains": ["quanticdata.io","stripe.com"],
    },
    timeout=120,
)
r.raise_for_status()
run = r.json()["payload"]
print(run["count"], "rows,", run["cost"]["usd"], "USD")
for row in run["results"]:
    print(row.get("domain"), row.get("url"), row.get("robots_txt"), row.get("ai_bots"))

Python: one AI readiness audit run (Audit two sites), printing domain, url, robots_txt, ai_bots

The key comes from an environment variable, never from the code. The run answers in the same request when it finishes in time; payload.results holds the rows and payload.cost what they cost.

Request parameters

What the collector takes.

Input of the AI readiness audit collector
Parameter	Type	Required	Meaning
`domains`	array	yes	Domains or site URLs, one per line (max 50).
`country`	string	no	ISO 3166-1 alpha-2 code — proxy exit geo and Google locale (gl). Omit for the default pool.
`max_results`	integer	no (default 20)	How many sites to deliver at most (1–50). You pay only for delivered sites.

Response fields

28 fields per site.

Fields of each row (site)
Field	Type	What it holds
`domain`	string	Registrable host (no www).
`url`	string	Homepage URL fetched.
`robots_txt`	boolean	Whether a robots.txt exists.
`ai_bots`	object	Per-crawler verdict: { gptbot: "allowed\|partial\|blocked", claudebot: …, … }.
`allowed_bots`	string[]	AI crawlers with no restrictions.
`partial_bots`	string[]	AI crawlers allowed at the root but fenced off some paths.
`blocked_bots`	string[]	AI crawlers disallowed from the whole site.
`llms_txt`	boolean	Whether /llms.txt exists and is valid Markdown.
`llms_txt_url`	string, nullable	URL of the llms.txt when present.
`llms_txt_links`	integer	Links listed in the llms.txt.
`llms_full_txt`	boolean	Whether /llms-full.txt exists.
`sitemap`	boolean	Whether robots.txt declares at least one sitemap.
`sitemaps`	string[]	Declared sitemap URLs (max 10).
`noai_meta`	boolean	Homepage carries a noai/noimageai robots meta.
`x_robots_noai`	boolean	X-Robots-Tag response header carries noai.
`jsonld_types`	string[]	schema.org @type values found on the homepage.
`og_meta`	boolean	OpenGraph tags present.
`h1`	boolean	Homepage has an h1.
`ssr_text_chars`	integer	Visible text characters served without JavaScript.
`js_dependent`	boolean	True when the no-JS page has almost no text (< 400 chars).
`ai_readiness_score`	integer	0–100: crawler access (55) + llms.txt (15) + no-JS text (15) + sitemap (10) + structured data (5).
`status`	string	ok for delivered rows.
`extra_ai_bots`	object	Verdicts for further AI agents (Perplexity-User, MistralAI-User, Meta-ExternalFetcher, Google-CloudVertexBot, cohere-ai, YouBot, Diffbot, AI2Bot, Timpibot). Not part of the score.
`content_signals`	string[]	Content-Signal directives in robots.txt (search=yes, ai-input=yes, ai-train=no…).
`homepage_read`	boolean	Whether the homepage could be read without a browser.
`title`	string, nullable	Homepage <title>.
`meta_description`	string, nullable	Homepage meta description.
`lang`	string, nullable	Homepage language attribute.

Add ?format=csv to the runs endpoint to get the same rows as CSV.

Long runs

Run in the background, then poll.

A run that needs more time can go to the background: send "async": true, get 202 with the run id, and poll the runs endpoint until the status is done. Reading a run is free.

import os, time, requests

H = {"Authorization": f"Bearer {os.environ['QD_API_KEY']}"}
r = requests.post("https://api.quanticdata.io/v1/scraper/collectors/ai_readiness/run", headers=H,
                  json={**{"domains":["quanticdata.io","stripe.com"]}, "async": True}, timeout=30)
run_id = r.json()["payload"]["run_id"]
while True:
    run = requests.get(f"https://api.quanticdata.io/v1/scraper/collectors/runs/{run_id}", headers=H, timeout=30).json()["payload"]
    if run["status"] in ("done", "failed"):
        break
    time.sleep(5)
print(run["status"], run["count"])

Python: background run and polling

Errors

What can come back, and what is billed.

Collector run errors
Status	Meaning	Billed
`400`	The input does not match the collector (`payload.errors`)	No
`401`	Missing or wrong API key	No
`402`	Balance and free credit do not cover the run: top up	No
`422`	The input can never resolve; retrying the same input cannot help	No
`424`	The run failed at the source (blocked, timed out, layout changed); safe to retry	No
`429`	Rate limit or a per-account brake; wait for `Retry-After`	No

Only delivered rows are billed: a run that delivers nothing costs nothing.

Price

$0.003 per delivered site.

What AI readiness audit rows cost at list price
Rows	Cost
1	$0.003
1,000	$3.00
10,000	$30.00

Up to 50 sites per run; split larger jobs into several runs. The $2 free credit every month covers about 666 sites. No subscription; paid tiers lower the unit price. All collector prices: collector pricing.

Before you run it

Questions developers ask

Is there a free AI readiness audit API?

Every account gets $2 of free usage each month, about 666 sites of this collector at $0.003 each, with no card.

Do I have to handle proxies, retries or blocks?

No. The collector runs on our network and handles them; a run blocked at the source returns 424 and is not billed.

How many rows can one run return?

Up to 50 sites per run (the max_results input, where the collector has one).

Can an AI agent run it?

Yes: the MCP server exposes it as the run_collector tool, with the same input and price. See the MCP server.

Run it on the free credit.

Create an API key, export it as QD_API_KEY and run the example. $0.003 per delivered site, no subscription.

Get an API key AI readiness API