# Scrape Google Search Results in Python, Tested > We scraped Google for 10 queries in Python on 8 October 2026: 100 of 100 organic results, a median 3.5 seconds per call, 7.8 KB of JSON. Code, fields and cost. [Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/Scrape Google Search Results in Python, Tested # How to scrape Google search results with Python: 10 queries, measured SERP & searchOct 8, 2026·8 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData A Python request for a Google search goes in on the left; two bars compare the 130 KB average Google results page with the 7.8 KB of JSON the script receives, and a card sums up the 8 October 2026 test: 100 of 100 organic results, a median 3.5 seconds per call, $0.005 for 10 searches. On this page [Why the BeautifulSoup tutorial returns nothing](/blog/scrape-google-search-results-python/#why-the-beautifulsoup-tutorial-returns-nothing) [Three ways to scrape Google search results in Python](/blog/scrape-google-search-results-python/#three-ways-to-scrape-google-search-results-in-python) [The Python script](/blog/scrape-google-search-results-python/#the-python-script) [What we measured on 8 October 2026](/blog/scrape-google-search-results-python/#what-we-measured-on-8-october-2026) [What the JSON contains](/blog/scrape-google-search-results-python/#what-the-json-contains) [Getting more than 10 results](/blog/scrape-google-search-results-python/#getting-more-than-10-results) [What it costs to scrape Google search results](/blog/scrape-google-search-results-python/#what-it-costs-to-scrape-google-search-results) [Scrape Google search results from an AI agent](/blog/scrape-google-search-results-python/#scrape-google-search-results-from-an-ai-agent) [Is scraping Google search results legal?](/blog/scrape-google-search-results-python/#is-scraping-google-search-results-legal) [The settings to use](/blog/scrape-google-search-results-python/#the-settings-to-use) On 8 October 2026 we ran 10 Google searches from a Python script, one POST request each, and got 100 of 100 organic results back as JSON, every row with rank, title, link and snippet. The median call took 3.5 seconds, the JSON averaged 7.8 KB against a 130 KB Google results page, and the 10 searches cost $0.005. Below: the code, why the classic BeautifulSoup tutorial now prints nothing, and how to go past 10 results. ## Why the BeautifulSoup tutorial returns nothing The most copied way to scrape Google search results with Python is `requests.get("https://www.google.com/search?q=...")` followed by `soup.find_all("h3")`, one title per result. We ran exactly that on 8 October 2026 from a laptop in Italy, for the query "coffee grinder": | Request | What came back | Bytes | h3 titles found | | --- | --- | --- | --- | | requests default headers | Google's consent page, "Before you continue to Google Search" | 34,463 | 0 | | requests with a desktop Chrome user agent | A Google Search page without the results in the HTML | 93,752 | 0 | Two changes on Google's side explain it. Visitors from the EU first get a consent interstitial, and since January 2025 Google Search requires JavaScript to show results, which ended the plain-HTML approach that tutorials written before then rely on. Swapping user agents or adding a sleep between requests does not bring the results back. What works today is a real browser that runs JavaScript, or an API that hands you the parsed page. **Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month. [Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=scrape-google-search-results-python)[Explore SERP API](https://quanticdata.io/serp-api/) ## Three ways to scrape Google search results in Python | Route | Results | Status in October 2026 | You maintain | | --- | --- | --- | --- | | Custom Search JSON API (Google) | 10 per call, from a Programmable Search Engine index | Closed to new customers, discontinued on 1 January 2027 | API key and search engine id | | Headless browser (Playwright, Selenium) | What google.com renders | Works, heavy | Chrome, consent clicks, selectors, IP rotation, rate | | requests + BeautifulSoup on google.com | None without JavaScript | Broken for results since January 2025 | Everything | | SERP API (one POST, JSON back) | Organic, AI Overview, related searches, rich blocks | Works, measured below | One HTTP call | Google's own documentation for the Custom Search JSON API now states that it is not available for new customers and that existing customers keep 100 free queries a day until the service ends on 1 January 2027. If you are starting a project today, that route is closed. A headless browser works, but you then own a fleet of Chrome instances, consent handling and residential IPs; our guide to [using a proxy with Python requests](https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/) covers the proxy half. The rest of this post uses the [SERP API](https://quanticdata.io/serp-api/), which does the fetching and parsing and returns rows. ## The Python script This is the script we ran, unchanged. It needs only the `requests` library, sends one query, writes the organic results to a CSV file and prints the AI Overview and related searches when Google shows them. ``` import csv, os, requests API = "https://api.quanticdata.io/v1/serp" HEADERS = {"Authorization": "Bearer " + os.environ["QD_API_KEY"]} def google_search(query, country="us", lang="en", num=10): r = requests.post(API, headers=HEADERS, timeout=120, json={ "query": query, "engine": "google", "country": country, "lang": lang, "num": num}) r.raise_for_status() return r.json()["payload"] serp = google_search("what is a vector database") with open("serp.csv", "w", newline="") as f: w = csv.writer(f) w.writerow(["rank", "title", "link", "snippet"]) for row in serp["organic"]: w.writerow([row["rank"], row["title"], row["link"], row.get("description", "")]) print(len(serp["organic"]), "organic results") print("AI Overview:", (serp.get("ai_overview") or "")[:80]) print("Related:", [r["query"] for r in serp.get("related_searches") or []][:3]) ``` Output on 8 October 2026: `10 organic results`, the first sentence of Google's AI Overview ("A vector database is a specialized type of data storage system designed to store..."), and three related searches. Set `country` and `lang` to the market you track: the same query returns a different page in Germany or Brazil, and rank tracking only makes sense with both fixed. ## What we measured on 8 October 2026 We picked 10 queries that look like real workloads, from product research and local services to finance and developer questions, and ran each once with `country: us` and `lang: en`. Time is wall-clock from our client to the JSON in hand; page size is the Google results page that was fetched and parsed on our side. | Query | Organic | AI Overview | Seconds | Google page | | --- | --- | --- | --- | --- | | best running shoes | 10 | Yes | 3.4 | 121 KB | | python requests timeout | 10 | No | 4.4 | 56 KB | | coffee grinder | 10 | Yes | 2.2 | 206 KB | | how to make sourdough bread | 10 | Yes | 1.3 | 122 KB | | cheap flights to tokyo | 10 | No (flights block) | 1.9 | 103 KB | | best crm for small business | 10 | Yes | 3.8 | 117 KB | | iphone 17 pro review | 10 | No | 3.6 | 126 KB | | what is a vector database | 10 | Yes | 4.1 | 181 KB | | plumber near me | 10 | No | 1.9 | 152 KB | | nvidia stock | 10 | Yes | 4.9 | 117 KB | All 10 calls answered 200, all 100 organic rows had a title, a link and a snippet, and every query came with 8 related searches, 80 in total. Six of the 10 pages carried an AI Overview, and the flights query also returned Google's flights block as structured data. Calls took 1.3 to 4.9 seconds, with a median of 3.5 seconds. The size column is the part people underestimate. The Google pages weighed 56 to 206 KB, 130 KB on average, while the JSON our script received averaged 7.8 KB: about 17 times less to download, store and parse. For a job of 100,000 queries that is the difference between roughly 12 GB of HTML and under 1 GB of rows. ## What the JSON contains Each organic row carries `rank`, `title`, `link`, `display_link`, `source`, `description` (the snippet), `snippet_highlighted_words`, `date` when Google shows one, and `sitelinks` when the result has them. Next to `organic`, the payload has fixed keys for the other blocks, filled when Google shows them: - **`ai_overview`** holds the opening answer of the AI Overview as text, which is what you need to track whether a brand is named in it. - **`related_searches`** lists the queries at the foot of the page. They are a free source of secondary keywords; we use them to choose the subheadings of posts like this one. - **`people_also_ask`, `knowledge_graph`, `ads`, `places`, `shopping`, `flights`** and the other rich blocks share one envelope, so the parser on your side never changes when Google adds a module. To scrape only the URLs from Google search results, read `row["link"]` from `organic`: the links are the final destination URLs, not Google redirect links, so you can feed them straight into a page scraper. ## Getting more than 10 results Google serves about 10 organic results per page. Pass `num` up to 100 and the API fetches consecutive pages and merges them; `search_metadata.paging` tells you how many pages were fetched. On 8 October 2026, "best crm for small business" with `num: 30` fetched 3 pages and returned 29 unique organic results in 16.7 seconds. If you need one specific page, pass `page` instead. For many queries, send them concurrently rather than in a loop. A thread pool of 5 to 10 workers keeps the code simple: ``` from concurrent.futures import ThreadPoolExecutor queries = ["best running shoes", "coffee grinder", "plumber near me"] with ThreadPoolExecutor(max_workers=5) as pool: pages = list(pool.map(google_search, queries)) for q, serp in zip(queries, pages): print(q, "->", serp["organic"][0]["link"]) ``` If you prefer rows to whole pages, the [Google search results collector](https://quanticdata.io/collectors/google-search-results-api/) returns one row per organic result for a query and a country, and the [keyword ideas collector](https://quanticdata.io/collectors/keyword-research-api/) expands a seed into the suggestions people type, the same autocomplete data we used to pick this topic. ## What it costs to scrape Google search results The SERP API bills $0.0005 per search over HTTP, and only when results come back. Our 10 searches cost $0.005. | Job | Searches | Cost | | --- | --- | --- | | This test: 10 queries | 10 | $0.005 | | 1,000 queries | 1,000 | $0.50 | | Free monthly allowance ($2) | 4,000 | $0 | | Rank tracking: 200 keywords daily for 30 days | 6,000 | $3.00 | Every account gets $2 of free API usage per month, which covers 4,000 Google searches: enough to track 130 keywords every day of the month before paying anything. The free route, a browser script on your own machine, has no invoice, but you pay in hours on consent pages, Chrome updates and IP rotation each time Google changes the page. ## Scrape Google search results from an AI agent The same search is a tool in our [MCP server](https://quanticdata.io/mcp-server/), so Claude, Cursor or any MCP client can run it without a script: "search Google US for best crm for small business, list the 10 domains and tell me which ones appear in the AI Overview". The agent calls `search`, reads the JSON, and can chain `scrape` on the top links. For research we ran this post's keyword study the same way, with autocomplete, one full results page and two scraped competitors. ## Is scraping Google search results legal? Search results pages are public, and collecting titles, links and snippets for SEO monitoring, research or price comparison is a common, long-standing practice. Google's terms of service restrict automated access, so read them for your use case, keep request rates reasonable, and do not republish snippets wholesale. We cover the US case law on public data in [is web scraping legal in the US](https://quanticdata.io/blog/is-web-scraping-legal-in-us/). ## The settings to use Endpoint `POST /v1/serp` with `engine: google`, `country` and `lang` fixed to your market, `num` up to 100 for deeper pages, and a thread pool for volume. Read `organic` for rankings, `ai_overview` for AI visibility, `related_searches` for keyword ideas. Price: $0.0005 per search, billed only on results, with $2 of free API usage per month. For a walkthrough of every parameter, see [how to use a SERP API](https://quanticdata.io/blog/how-to-use-a-serp-api/). ### Sources & further reading - [Custom Search JSON API overview and pricing, Google for Developers](https://developers.google.com/custom-search/v1/overview) - [Google begins requiring JavaScript for Google Search, Hacker News discussion (January 2025)](https://news.ycombinator.com/item?id=42747092) - [Scrape Google Search Results using Python BeautifulSoup, GeeksforGeeks](https://www.geeksforgeeks.org/python/scrape-google-search-results-using-python-beautifulsoup/) - [How do I scrape Google Search results (on a big scale kinda)?, Stack Overflow](https://stackoverflow.com/questions/67179806/how-do-i-scrape-google-search-results-on-a-big-scale-kinda) ## FAQ Quick answers on scrape google search results python. [Something else? Ask us](mailto:hello@quanticdata.io) ### How do I scrape Google search results with Python? Send a POST request with the query, engine, country and language to a SERP API and read the organic list from the JSON. On 8 October 2026 this returned 100 of 100 organic results for 10 queries, in a median 3.5 seconds per call, with a short script that needs only the requests library. ### Why does BeautifulSoup return nothing when I scrape Google? Because Google no longer puts the results in the plain HTML. From the EU a script first gets a consent page, and since January 2025 Google Search requires JavaScript. In our test a plain requests call found 0 result titles, with default headers and with a Chrome user agent alike. ### Can I still use the Google Custom Search JSON API? Only if you are an existing customer. Google states the API is not available for new customers and will be discontinued on 1 January 2027; until then existing users keep 100 free queries per day. ### How do I scrape more than 10 Google results? Pass num up to 100. The API fetches consecutive result pages and merges them: num 30 fetched 3 pages and returned 29 unique organic results in our test. To fetch one specific page, pass page instead. ### How much does it cost to scrape Google search results? $0.0005 per search, billed only when results come back: $0.50 per 1,000 queries. Every account gets $2 of free API usage per month, which covers 4,000 searches. ### Is it legal to scrape Google search results? Search results are public and widely collected for SEO and research, but Google's terms restrict automated access. Read the terms for your use case, keep rates reasonable and do not republish snippets wholesale. ## Run your first 4,000 Google searches free 100 of 100 organic results on 8 October 2026, a median 3.5 seconds per call, JSON ready for Python. You pay $0.0005 per search, only on results, and every account gets $2 of free API usage per month. [Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=scrape-google-search-results-python)[Explore SERP API](https://quanticdata.io/serp-api/) ## Related reading [SERP & search How to Get a SERP API in Minutes A practical guide to getting a SERP API — sign-up, key generation, first request, free tier limits and the cost math that decides which provider fits. Read more](https://quanticdata.io/blog/how-to-get-serp-api/) [SERP & search How a SERP API Works, End to End The full path from query to JSON: localization parameters, the identity layer, HTTP vs rendered fetches, parsing rich blocks, verticals and pricing. Read more](https://quanticdata.io/blog/how-a-serp-api-works/) [SERP & search How to Scrape Google Reviews: 800 Measured On 7 October 2026 we pulled the 100 newest Google reviews of 8 US places through one API call each: 800 of 800 delivered, every row with rating, ISO date and review id. Here is the Python code, what the rows contain, how fast reviews pile up, and what it costs. Read more](https://quanticdata.io/blog/scrape-google-reviews/) --- Source: https://quanticdata.io/blog/scrape-google-search-results-python/ · Site index for AI: https://quanticdata.io/llms.txt