How to scrape Google search results with Python: 10 queries, measured

On 8 October 2026 we ran 10 Google searches from a Python script, one POST request each, and got 100 of 100 organic results back as JSON, every row with rank, title, link and snippet. The median call took 3.5 seconds, the JSON averaged 7.8 KB against a 130 KB Google results page, and the 10 searches cost $0.005. Below: the code, why the classic BeautifulSoup tutorial now prints nothing, and how to go past 10 results.

Why the BeautifulSoup tutorial returns nothing

The most copied way to scrape Google search results with Python is requests.get("https://www.google.com/search?q=...") followed by soup.find_all("h3"), one title per result. We ran exactly that on 8 October 2026 from a laptop in Italy, for the query "coffee grinder":

Request	What came back	Bytes	h3 titles found
requests default headers	Google's consent page, "Before you continue to Google Search"	34,463	0
requests with a desktop Chrome user agent	A Google Search page without the results in the HTML	93,752	0

Two changes on Google's side explain it. Visitors from the EU first get a consent interstitial, and since January 2025 Google Search requires JavaScript to show results, which ended the plain-HTML approach that tutorials written before then rely on. Swapping user agents or adding a sleep between requests does not bring the results back. What works today is a real browser that runs JavaScript, or an API that hands you the parsed page.

Three ways to scrape Google search results in Python

Route	Results	Status in October 2026	You maintain
Custom Search JSON API (Google)	10 per call, from a Programmable Search Engine index	Closed to new customers, discontinued on 1 January 2027	API key and search engine id
Headless browser (Playwright, Selenium)	What google.com renders	Works, heavy	Chrome, consent clicks, selectors, IP rotation, rate
requests + BeautifulSoup on google.com	None without JavaScript	Broken for results since January 2025	Everything
SERP API (one POST, JSON back)	Organic, AI Overview, related searches, rich blocks	Works, measured below	One HTTP call

Google's own documentation for the Custom Search JSON API now states that it is not available for new customers and that existing customers keep 100 free queries a day until the service ends on 1 January 2027. If you are starting a project today, that route is closed. A headless browser works, but you then own a fleet of Chrome instances, consent handling and residential IPs; our guide to using a proxy with Python requests covers the proxy half. The rest of this post uses the SERP API, which does the fetching and parsing and returns rows.

The Python script

This is the script we ran, unchanged. It needs only the requests library, sends one query, writes the organic results to a CSV file and prints the AI Overview and related searches when Google shows them.

import csv, os, requests

API = "https://api.quanticdata.io/v1/serp"
HEADERS = {"Authorization": "Bearer " + os.environ["QD_API_KEY"]}

def google_search(query, country="us", lang="en", num=10):
    r = requests.post(API, headers=HEADERS, timeout=120, json={
        "query": query, "engine": "google",
        "country": country, "lang": lang, "num": num})
    r.raise_for_status()
    return r.json()["payload"]

serp = google_search("what is a vector database")
with open("serp.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["rank", "title", "link", "snippet"])
    for row in serp["organic"]:
        w.writerow([row["rank"], row["title"], row["link"], row.get("description", "")])

print(len(serp["organic"]), "organic results")
print("AI Overview:", (serp.get("ai_overview") or "")[:80])
print("Related:", [r["query"] for r in serp.get("related_searches") or []][:3])

Output on 8 October 2026: 10 organic results, the first sentence of Google's AI Overview ("A vector database is a specialized type of data storage system designed to store..."), and three related searches. Set country and lang to the market you track: the same query returns a different page in Germany or Brazil, and rank tracking only makes sense with both fixed.

What we measured on 8 October 2026

We picked 10 queries that look like real workloads, from product research and local services to finance and developer questions, and ran each once with country: us and lang: en. Time is wall-clock from our client to the JSON in hand; page size is the Google results page that was fetched and parsed on our side.

Query	Organic	AI Overview	Seconds	Google page
best running shoes	10	Yes	3.4	121 KB
python requests timeout	10	No	4.4	56 KB
coffee grinder	10	Yes	2.2	206 KB
how to make sourdough bread	10	Yes	1.3	122 KB
cheap flights to tokyo	10	No (flights block)	1.9	103 KB
best crm for small business	10	Yes	3.8	117 KB
iphone 17 pro review	10	No	3.6	126 KB
what is a vector database	10	Yes	4.1	181 KB
plumber near me	10	No	1.9	152 KB
nvidia stock	10	Yes	4.9	117 KB

All 10 calls answered 200, all 100 organic rows had a title, a link and a snippet, and every query came with 8 related searches, 80 in total. Six of the 10 pages carried an AI Overview, and the flights query also returned Google's flights block as structured data. Calls took 1.3 to 4.9 seconds, with a median of 3.5 seconds.

The size column is the part people underestimate. The Google pages weighed 56 to 206 KB, 130 KB on average, while the JSON our script received averaged 7.8 KB: about 17 times less to download, store and parse. For a job of 100,000 queries that is the difference between roughly 12 GB of HTML and under 1 GB of rows.

What the JSON contains

Each organic row carries rank, title, link, display_link, source, description (the snippet), snippet_highlighted_words, date when Google shows one, and sitelinks when the result has them. Next to organic, the payload has fixed keys for the other blocks, filled when Google shows them:

• ai_overview holds the opening answer of the AI Overview as text, which is what you need to track whether a brand is named in it.
• related_searches lists the queries at the foot of the page. They are a free source of secondary keywords; we use them to choose the subheadings of posts like this one.
• people_also_ask, knowledge_graph, ads, places, shopping, flights and the other rich blocks share one envelope, so the parser on your side never changes when Google adds a module.

To scrape only the URLs from Google search results, read row["link"] from organic: the links are the final destination URLs, not Google redirect links, so you can feed them straight into a page scraper.

Getting more than 10 results

Google serves about 10 organic results per page. Pass num up to 100 and the API fetches consecutive pages and merges them; search_metadata.paging tells you how many pages were fetched. On 8 October 2026, "best crm for small business" with num: 30 fetched 3 pages and returned 29 unique organic results in 16.7 seconds. If you need one specific page, pass page instead.

For many queries, send them concurrently rather than in a loop. A thread pool of 5 to 10 workers keeps the code simple:

from concurrent.futures import ThreadPoolExecutor

queries = ["best running shoes", "coffee grinder", "plumber near me"]
with ThreadPoolExecutor(max_workers=5) as pool:
    pages = list(pool.map(google_search, queries))

for q, serp in zip(queries, pages):
    print(q, "->", serp["organic"][0]["link"])

If you prefer rows to whole pages, the Google search results collector returns one row per organic result for a query and a country, and the keyword ideas collector expands a seed into the suggestions people type, the same autocomplete data we used to pick this topic.

What it costs to scrape Google search results

The SERP API bills $0.0005 per search over HTTP, and only when results come back. Our 10 searches cost $0.005.

Job	Searches	Cost
This test: 10 queries	10	$0.005
1,000 queries	1,000	$0.50
Free monthly allowance ($2)	4,000	$0
Rank tracking: 200 keywords daily for 30 days	6,000	$3.00

Every account gets $2 of free API usage per month, which covers 4,000 Google searches: enough to track 130 keywords every day of the month before paying anything. The free route, a browser script on your own machine, has no invoice, but you pay in hours on consent pages, Chrome updates and IP rotation each time Google changes the page.

Scrape Google search results from an AI agent

The same search is a tool in our MCP server, so Claude, Cursor or any MCP client can run it without a script: "search Google US for best crm for small business, list the 10 domains and tell me which ones appear in the AI Overview". The agent calls search, reads the JSON, and can chain scrape on the top links. For research we ran this post's keyword study the same way, with autocomplete, one full results page and two scraped competitors.

Is scraping Google search results legal?

Search results pages are public, and collecting titles, links and snippets for SEO monitoring, research or price comparison is a common, long-standing practice. Google's terms of service restrict automated access, so read them for your use case, keep request rates reasonable, and do not republish snippets wholesale. We cover the US case law on public data in is web scraping legal in the US.

The settings to use

Endpoint POST /v1/serp with engine: google, country and lang fixed to your market, num up to 100 for deeper pages, and a thread pool for volume. Read organic for rankings, ai_overview for AI visibility, related_searches for keyword ideas. Price: $0.0005 per search, billed only on results, with $2 of free API usage per month. For a walkthrough of every parameter, see how to use a SERP API.