How to use a SERP API: keys, parameters, parsing and real cost math

To use a SERP API you do three things: create an account and copy the API key, send one HTTP request containing the query plus locale parameters (q, gl, hl, num), and read structured JSON back instead of HTML. Everything else — proxies, CAPTCHAs, parsing — happens server side.

What a SERP API actually does for you

A search results page is one of the most defended pages on the web. Requesting it at volume from a single IP gets you soft-blocked, redirected to a consent interstitial, or served a shell page whose markup changes weekly. A SERP API removes three layers of work: it routes the request through an exit IP that looks like a real user in the right country, it fetches (and if needed renders) the page, and it converts the DOM into a stable JSON schema so your code never touches a CSS selector.

That last point is the one people underestimate. Selectors are the maintenance tax of DIY search scraping. When Google reshuffles a snippet block, a good provider's parser absorbs the change; your rank tracker keeps running. Providers advertise structured output for organic results plus the surrounding modules — ads, knowledge panels, local packs, People Also Ask — and support several engines beyond Google, including Bing, DuckDuckGo, Yandex and Baidu (Bright Data SERP API docs).

Also worth knowing before you start: Google has no general-purpose public search API. The official Custom Search JSON API is scoped to programmable search engines, gives 100 free queries a day, then charges $5 per 1,000 queries with a 10,000/day ceiling (Google docs). That is why third-party SERP APIs exist at all.

Step by step: your first SERP API request

1. Create the account and copy the key. Every provider gives you a dashboard — the "SERP API login" people search for — where the key lives. Keys are usually a single bearer token that works across all endpoints. Store it in an environment variable, never in the repo.
2. Send one request with a single keyword. Don't start with a loop. One query, printed to stdout, so you can see the exact shape you get back.
3. Pin the locale. Results differ by country, language and device. Set them explicitly on request one, or you will compare apples to oranges later.
4. Inspect the JSON keys you actually need. Usually that is the organic array (title, link, position, snippet) plus one or two modules.
5. Persist raw responses. Keep the JSON blob alongside your normalised rows. When a number looks wrong three weeks later, the raw payload is your only evidence.
6. Then add concurrency. Start at 3-5 parallel requests, watch for 429s, and move to async or queued jobs for big batches.

Here is a first call against the QuanticData SERP API, which returns SerpApi-compatible JSON so existing parsers keep working:

export QD_API_KEY=qd_live_your_key_here

curl https://api.quanticdata.io/v1/serp \
  -H "Authorization: Bearer $QD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "q": "best crm for agencies",
    "engine": "google",
    "gl": "us",
    "hl": "en",
    "num": 10
  }'

The response uses the same envelope as every other endpoint on the platform — success, data, usage:

{ "success": true,
  "data": { "organic_results": [ { "position": 1, "title": "…", "link": "https://…" } ] },
  "usage": { "cost_usd": 0.0005 } }

And the Python version you would actually put in a job, looping keywords with a shared session:

import os, requests

API = "https://api.quanticdata.io/v1/serp"
HEAD = {"Authorization": f"Bearer {os.environ['QD_API_KEY']}"}

def ranks(keyword, domain, gl="us", hl="en"):
    r = requests.post(API, headers=HEAD, json={
        "q": keyword, "engine": "google",
        "gl": gl, "hl": hl, "num": 20,
    }, timeout=60)
    r.raise_for_status()
    body = r.json()
    if not body.get("success"):
        return None                      # failed call, nothing charged
    for row in body["data"].get("organic_results", []):
        if domain in row.get("link", ""):
            return row["position"]
    return None

for kw in ["serp api", "google serp api", "cheap serp api"]:
    print(kw, ranks(kw, "quanticdata.io"))

The parameters that change your results

Most SERP API documentation lists dozens of parameters. In practice five decide whether your dataset is usable.

Two further switches are worth understanding. First, verticals: a Google search API is rarely just web results. Maps, shopping, news, jobs, scholar, hotels and flights each have their own schema — QuanticData exposes 17 of them, and vertical choice is usually a parameter rather than a different endpoint. Second, rendering: unrendered fetches are cheaper and faster; JS rendering ($0.002 per search here versus $0.0005) is only needed for result types that hydrate client side. Test both on ten queries before you decide.

Reading the response without building a fragile pipeline

A SERP payload is a page, not a list. Alongside the organic array you typically get People Also Ask questions, related searches, ads, a knowledge graph block and pagination links. Treat every module as optional — presence varies by query — and write your extraction defensively:

• Key on position, not order of appearance. Ads and modules interleave with organic entries in the visual page but are separate arrays in JSON.
• Store the full URL and the display URL. Domain matching on the display string will fail on subdomains and redirects.
• Keep PAA and related searches. They are free keyword research: PAA text is literal user phrasing, which is what you want for FAQ sections and content briefs.
• Timestamp every row. A rank without a fetch time and a locale is not data.

If you then need the content of the ranking pages — for content gap analysis or RAG grounding — chain the SERP call into a fetch step rather than writing a second scraper. Our Web Scraping API returns each result URL as clean Markdown from $0.0002 a page, and crawl and map handle the case where you want the whole competitor site. If the concept of an API-as-scraper is new, this primer covers the model.

SERP API pricing: do the arithmetic before you subscribe

The honest way to compare providers is cost per successful search at your volume — not the headline plan price. Three things distort the comparison:

• Credit multipliers. Some providers deduct several credits per Google search, or extra credits for location targeting and rendering. Divide plan credits by the multiplier before you compute unit cost.
• Failures. Ask whether blocked or empty responses are billed. Pay-per-success billing — where success: false costs $0.00 — makes your invoice track delivered rows instead of attempts.
• Unused allowance. Monthly subscription credits that expire punish spiky workloads. Rank tracking is spiky by nature: a quarterly audit can be 20x a normal week.

Worked example. Tracking 500 keywords daily across two countries is 500 × 2 × 30 = 30,000 searches a month. At $0.0005 per unrendered search that is $15.00; with JS rendering on every call, $60.00. Add a one-off competitive audit of 5,000 queries and you have added $2.50. Compare that with a fixed plan: if a plan bundles 30,000 searches for a monthly fee, your effective rate is the fee divided by the searches you actually make — which, for most teams, is well under the allowance.

"Free SERP API" tiers exist and are genuinely useful for evaluation: they are typically a few hundred searches a month, enough to validate schema and locale accuracy, not enough to run production. QuanticData's pay-as-you-go tier includes $2 of free usage every month with no card, which at list price is 4,000 unrendered searches — plenty to build and test a tracker before you spend anything.

Using a SERP API from an AI agent

The fastest-growing use of search APIs is not rank tracking; it is grounding. An agent asked "who are the three biggest suppliers of X in Germany" needs a live search, then a fetch of the top results, then extraction. Wiring that as three separate HTTP integrations is where agent projects stall.

The cleaner pattern is to expose search as a tool. Our MCP server publishes eight tools — including search, scrape, map, crawl and batch — to Claude, Claude Code, Cursor, Windsurf, VS Code and Cline, so the model calls search directly and receives the same JSON envelope your Python job sees. The Web Data API for AI is the same platform over plain REST if you are building the loop yourself.

claude mcp add quantumproxies \
  -e QUANTUMPROXIES_API_KEY=qp_live_your_key_here \
  -- npx -y quantumproxies-mcp

Two practical notes for agent use. Keep result depth small — 10 results is usually enough context and keeps token cost down. And cache aggressively: agents re-ask the same question constantly, and a one-hour cache on identical q+gl+hl tuples cuts spend more than any pricing negotiation.

Debugging: the five errors you will actually hit

1. 401 / 403. Wrong key, or a key scoped to a different product or zone. Echo the header you are sending; a trailing newline from export is a classic.
2. 429. You exceeded concurrency or hourly throughput. Reduce parallelism, add exponential backoff, or move large jobs to async endpoints — the standard advice across providers (Bright Data docs).
3. Empty or partial results. Almost always a parameter problem: missing gl/hl, an unsupported location string, or a vertical that has no results for that query. Reproduce the same query in a browser with the same locale before blaming the API.
4. Garbled results for non-Latin queries. Encode the query properly; unencoded Cyrillic, Greek or CJK strings return the wrong page rather than an error.
5. Silent schema drift. A module you depend on disappears for a subset of queries. Assert on the shape you need and alert when the assertion rate drops, instead of writing zeros to your database.

Evaluating providers: criteria, not leaderboards

One compliance note: collecting public search results is common practice, but the legal picture depends on jurisdiction, the data involved (personal data especially) and how you use it. Read our overview of US web scraping law and take your own counsel — nothing here is legal advice. Full parameter reference, endpoints and the shared envelope live in the documentation.