Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Web scraping from a Supabase Edge Function, Deno and Bun

One fetch call to a scraping API from three JavaScript runtimes: the same script ran on Deno 2.9.8, Bun 1.4.5 and Node.js 22 and each printed status 200 with 8402 characters of Markdown; the card lists the three runtimes and the size of the answer each one printed.
One fetch call to a scraping API from three JavaScript runtimes: the same script ran on Deno 2.9.8, Bun 1.4.5 and Node.js 22 and each printed status 200 with 8402 characters of Markdown; the card lists the three runtimes and the size of the answer each one printed.

To scrape a web page from a Supabase Edge Function, call a scraping API with fetch and return its JSON. Edge Functions run on Deno, where fetch is built in, so there is nothing to install: the function sends the URL, the API fetches the page through its own proxies and answers with the title and the content as Markdown.

We ran the code in this post on 20 November 2026 with Deno 2.9.8, Bun 1.4.5 and Node.js 22.23.2 on Windows 11. The Edge Function was served locally with Deno; the deployment commands below are the ones in the Supabase documentation.

Why call an API instead of scraping inside the function

An Edge Function is a short-lived worker close to your users. That shapes what it is good at:

Inside an Edge FunctionWhat it means for scraping
No headless browserPages that need JavaScript rendering cannot be opened there
Requests leave from the platform's addressesThe page you get can differ from the one a visitor in the target country sees
A request must answer within 150 seconds (Supabase limits page)Long crawls do not fit in one invocation
fetch, JSON and secrets are built inCalling an HTTP API is the natural job

So the function does the part it is good at, receiving a request and returning JSON, and hands the fetching to a web scraping API that runs the request through residential proxies, renders when the page needs it and returns clean text.

One file for Deno, Bun and Node.js

Start outside Supabase, with the smallest script that proves your key works. It uses only what the three runtimes share: top-level await, fetch and AbortSignal.timeout.

// scrape.mjs
const key = globalThis.Deno ? Deno.env.get("QD_API_KEY") : process.env.QD_API_KEY;

const res = await fetch("https://api.quanticdata.io/v1/scraper/extract", {
  method: "POST",
  headers: { Authorization: "Bearer " + key, "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://books.toscrape.com/", format: "markdown" }),
  signal: AbortSignal.timeout(90_000),
});

const body = await res.json();
if (body.type === "error") throw new Error(res.status + " " + body.message);

console.log(res.status, body.payload.title, body.payload.content.length + " characters of Markdown");

The same file, three commands, three identical answers:

$ deno run --allow-net=api.quanticdata.io --allow-env=QD_API_KEY scrape.mjs
200 All products | Books to Scrape - Sandbox 8402 characters of Markdown

$ bun run scrape.mjs
200 All products | Books to Scrape - Sandbox 8402 characters of Markdown

$ node scrape.mjs
200 All products | Books to Scrape - Sandbox 8402 characters of Markdown

Two differences are worth knowing. Deno reads environment variables with Deno.env.get and asks for permission to use the network and the environment, which is what the two --allow flags grant; naming the host and the variable keeps the permission narrow. Bun and Node read process.env and need no flags.

Every answer from the API has the same envelope: type is "response" or "error", message says what happened, and payload holds the data. For a scrape, payload.title, payload.content and payload.metadata are the fields most functions return.

The Edge Function

A Supabase Edge Function is a file that calls Deno.serve with a handler. This one accepts {"url": "..."} and answers with the title and the Markdown:

// supabase/functions/scrape/index.ts
const API = "https://api.quanticdata.io/v1/scraper/extract";

Deno.serve(async (req) => {
  if (req.method !== "POST") return new Response("POST a JSON body with a url", { status: 405 });

  const { url } = await req.json();
  const upstream = await fetch(API, {
    method: "POST",
    headers: {
      Authorization: "Bearer " + Deno.env.get("QD_API_KEY"),
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, format: "markdown" }),
    signal: AbortSignal.timeout(90_000),
  });

  const body = await upstream.json();
  if (body.type === "error") {
    // Pass the status and Retry-After through, so the caller can wait and try again on a 429.
    const headers = new Headers({ "Content-Type": "application/json" });
    const retryAfter = upstream.headers.get("Retry-After");
    if (retryAfter) headers.set("Retry-After", retryAfter);
    return new Response(JSON.stringify({ error: body.message }), { status: upstream.status, headers });
  }

  const { title, content, finalUrl } = body.payload;
  return Response.json({ title, url: finalUrl, markdown: content });
});

Served with plain Deno on our machine and called with curl:

$ deno run --allow-net --allow-env=QD_API_KEY index.ts
Listening on http://localhost:8000/

$ curl -s -X POST http://localhost:8000/ -H "Content-Type: application/json" -d '{"url":"https://books.toscrape.com/"}'
{"title":"All products | Books to Scrape - Sandbox","url":"https://books.toscrape.com/",
 "markdown":"[Books to Scrape](https://books.toscrape.com/index.html) We love being scraped!\n\n-   [Home](...) ..."}

$ curl -s http://localhost:8000/
POST a JSON body with a url

The POST came back with status 200, and the GET with 405; in our run the answer came back in about 3 seconds.

Run it on Supabase: secrets, serve, deploy

The Supabase documentation gives the steps; they are short.

  • Create the function: supabase functions new scrape creates supabase/functions/scrape/index.ts. Replace its starter code with the function above.
  • Local secret: put QD_API_KEY=qd_live_... in supabase/functions/.env, which the local stack loads, and add that file to .gitignore.
  • Serve locally: supabase functions serve scrape.
  • Production secret: supabase secrets set --env-file .env, or set the single value. The function reads it with the same Deno.env.get("QD_API_KEY").
  • Deploy: supabase functions deploy scrape.

The key belongs in the function's secrets and nowhere else. A browser or mobile app calls your function; it never sees the key, because the function adds the Authorization header on the server side.

Timeouts, rate limits and bigger jobs

  • Timeout. The function above gives the API 90 seconds, inside the 150-second window Supabase documents for a request.
  • Rate limits. If the API answers 429, the function returns the same status with the Retry-After header, so your caller knows how long to wait. How to read that header is in Retry-After header.
  • Many URLs. Do not loop over a long list inside one invocation. Submit the list as one batch job (POST /scraper/batch, up to 5,000 URLs), return the job id, and let a second function or a webhook collect the result.
  • Structured fields. To get fields instead of Markdown, send an extract object that maps field names to CSS selectors, or parse the HTML yourself with Cheerio on Bun or Node.

The same pattern on Bun as a server

Bun has its own built-in server. The handler is the same code; only the entry point and the way to read the key change:

// server.ts, run with: bun run server.ts
const API = "https://api.quanticdata.io/v1/scraper/extract";

Bun.serve({
  port: 8001,
  async fetch(req) {
    if (req.method !== "POST") return new Response("POST a JSON body with a url", { status: 405 });

    const { url } = await req.json();
    const upstream = await fetch(API, {
      method: "POST",
      headers: { Authorization: "Bearer " + process.env.QD_API_KEY, "Content-Type": "application/json" },
      body: JSON.stringify({ url, format: "markdown" }),
      signal: AbortSignal.timeout(90_000),
    });

    const body = await upstream.json();
    if (body.type === "error") return Response.json({ error: body.message }, { status: upstream.status });
    return Response.json({ title: body.payload.title, url: body.payload.finalUrl, markdown: body.payload.content });
  },
});
console.log("Listening on http://localhost:8001/");

On Bun 1.4.5 the same curl POST returned the same JSON with status 200, and a GET returned 405. Request options beyond url and format, such as country or rendering, are in the API reference, and more JavaScript samples are in the Node.js examples.

Sources & further reading

FAQ

Quick answers on Supabase Edge Function web scraping.

Something else? Ask us

Can a Supabase Edge Function scrape a website?

Yes, by calling a scraping API with fetch. The function sends the URL and returns the JSON; the API does the fetching. Our Deno.serve function returned the title and Markdown of a page when served locally with Deno.

Do I need to install a package?

No. fetch, Response and AbortSignal.timeout are built into Deno, Bun and Node.js 18 or later. The script in this post has no imports.

Where do I store the API key in Supabase?

In the function's secrets: supabase/functions/.env for local runs and supabase secrets set for production, read with Deno.env.get("QD_API_KEY"). Never put it in client-side code.

Does the same code run on Deno, Bun and Node.js?

Yes. Our scrape.mjs ran unchanged on Deno 2.9.8, Bun 1.4.5 and Node.js 22.23.2 and printed the same line each time: status 200 and 8402 characters of Markdown.

Which permissions does Deno need for this?

Network access to the API host and read access to one environment variable: --allow-net=api.quanticdata.io --allow-env=QD_API_KEY. On Supabase the platform grants these to the function.

How do I scrape many pages from an Edge Function?

Submit them as one batch job and return the job id, then read the result from a second call or a webhook. One invocation should not wait for hundreds of pages.

One fetch call from your function

The scraping API returns a page as Markdown or structured fields; your Edge Function only needs fetch and a key.

Related reading