Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Crawl a whole website to Markdown: video guide

0:53··English captions

A 0:53 screen recording with voice and captions of Map and Crawl in the QuanticData Playground: list the URLs of a public sample bookshop with a count for each section, start a crawl of ten pages at depth two as a background job, check the job, read each page as Markdown and copy the same call in Python.

The crawl and map API in writing, with the options and the price: Website crawler API: crawl, map and the price per page.

Chapters

  1. 0:00The result: every page as Markdown
  2. 0:06Map the site
  3. 0:19Set up the crawl
  4. 0:29Start the job and check it
  5. 0:36Read the pages
  6. 0:42The same call in Python

Transcript

The captions of the video, line by line. Click a time to jump there.

  1. 0:00A whole website as clean Markdown, page by page. Here is how.
  2. 0:06In the Playground, start with Map.
  3. 0:09Type the site, here a public sample bookshop.
  4. 0:12Click Send.
  5. 0:14The answer lists the URLs of the site, with a count for each section.
  6. 0:19Now switch to Crawl, and give the same start URL.
  7. 0:23Set the maximum pages, here ten.
  8. 0:26And the depth, here two.
  9. 0:29Click Send: the crawl starts as a background job.
  10. 0:32Click Check job: completed, ten pages crawled.
  11. 0:36Each page comes back with its URL, its title and its Markdown.
  12. 0:42The same call is ready in Python, with your own key.
  13. 0:46Map and crawl work from the API too. Read the docs at quanticdata.io/docs.

Where to go next

One key for every web data API

Scrape, crawl, map, search and audit pages with one API key: try each call in the Playground, then copy it as cURL, Python, Node.js, PHP or Go.

More video guides