Website crawler API from $0.0003 per page

A website crawler API is a service that starts from a seed URL, follows links across a site and returns every page it fetches as structured content — here, clean Markdown. It replaces running your own crawler: proxies, retries, anti-bot escalation, politeness and parsing are handled server-side, and you collect the results from a job endpoint.

Scraping fetches pages you already know, one URL per call. Crawling discovers the pages for you: it starts from a seed, follows links breadth-first to the depth you set, and fetches everything it finds. If you already hold a URL list, the batch endpoint is cheaper at $0.0002 per URL; if you need the site explored, that is the crawler's job.

Use map. It reads the site's robots.txt sitemaps, /sitemap.xml and homepage links in a single $0.0005 call and returns up to 5,000 URLs plus a site-wide total and a per-section summary — so you see the site's shape before spending anything on fetching. Filter with search or ask for the path tree with group_by.

Markdown keeps the document structure — headings, lists, tables, links — while dropping the HTML noise that wastes context tokens. Each crawled page arrives as Markdown ready to chunk and embed, with navigation, footers and cookie chrome already stripped in the default smart mode. An article mode narrows to the main content only, and full keeps the entire body.

$0.0003 per crawled page — $0.30 per thousand — with no subscription or monthly minimum. Mapping a site costs $0.0005 per call, and batch-scraping a known URL list costs $0.0002 per URL. Every account gets $1 of free usage per month, enough for about 3,300 crawled pages.

No. A crawl job is charged up front on the requested volume, and when the job settles the failed and unfetched share is automatically refunded. If you request 500 pages and the site only yields 180, you pay for 180. Pay-per-success is the billing rule across all QuanticData Data APIs.

Up to 500 pages per job (default 50) and up to 10 link levels deep (default 3). The crawl is breadth-first, so shallow pages — usually the important ones — are fetched before deep archive pages. For larger sites, run map first and split the work into multiple crawls or a batch job over the URL list.

Yes. Pass include and exclude URL patterns — substrings or globs — to keep the crawler inside /docs and out of /tag pages, for example. Combined with the depth and page limits, filters are how you cap both the scope and the maximum spend of a job before it starts.

Each page goes through the same pipeline as the scraping API: a real-browser TLS fingerprint on the request tier, escalating to a headless browser when a page is bot-challenged, with residential exits underneath. You can also pin the proxy exit to a country so geo-sensitive sites are seen from the right market.

Starting a crawl returns a job id immediately. Poll the status endpoint to watch progress and collect pages as they arrive; the job reports per-page outcomes so you can see what succeeded, what failed and why. Nothing blocks your side while the crawl runs.

Crawling publicly accessible pages is generally lawful in most jurisdictions, but sites' terms of service restrict automated access and the case law differs by country. Respect the data you collect, avoid personal data you have no basis to process, and get proper legal advice for your specific use case; this answer is not it.

Yes. The QuanticData MCP server exposes crawl, crawl_status and map as native tools, so Claude, Cursor or any MCP-capable agent can map a site, decide what matters and crawl it autonomously. Results come back as Markdown the model can read directly.

There are free options and they are real: open-source crawlers like Crawl4AI and Crawlee run on your own machine for nothing, and several hosted APIs have a free tier with a monthly cap. What free rarely covers is the part that costs money — residential exits for sites that block datacenter IPs, browser rendering when the markup needs JavaScript, and retries. This API gives every account $1 of free usage each month with no card, which is roughly 3,300 crawled pages, and after that you pay per page actually fetched. If a local open-source crawler gets your pages, it is genuinely the cheaper answer.

Start the job with a POST and poll it — no SDK required, because it is plain HTTP. Send the start URL with your limit and depth to the crawl endpoint using requests, keep the returned job id, then poll crawl_status until it reports done and read the pages array, where each entry carries its source URL and Markdown. The page's code tab has the full working snippet. The same two calls work from Node, Go or curl; nothing about the API is language-specific.

A crawler is the thing that walks a site; an API is how you ask for it. The confusion usually means one of two other questions. If a site publishes its own API, use that — it is faster, cleaner and sanctioned. If it does not, a crawler API is how you get the same data from the pages themselves: you call one endpoint, it discovers the URLs, fetches them through the proxy network and returns Markdown, and you never run or maintain the crawler yourself.