https://quanticdata.io/blog/ Practical guides on web scraping, SERP data, proxies, crawling and web data pipelines for AI. en Thu, 03 Sep 2026 22:19:26 GMT https://quanticdata.io/quanticdata-favicon-512.pnghttps://quanticdata.io/blog/ https://quanticdata.io/blog/how-to-find-b2b-clients-from-public-data/ https://quanticdata.io/blog/how-to-find-b2b-clients-from-public-data/ Thu, 03 Sep 2026 22:19:26 GMT Businesses publish their category, location, phone and website on maps, and their email addresses on their own sites. Collecting that into a targeted prospect list is the cheap part: our measurement puts a usable business record at $0.0143. The workflow, the yield you should expect at each step from 458 measured records, and the part most guides skip: which lawful basis lets you actually email the list in Italy, Germany, Spain, France and the UK, with the regulator page for each. lead-gen https://quanticdata.io/blog/how-much-does-a-lead-cost/ https://quanticdata.io/blog/how-much-does-a-lead-cost/ Thu, 03 Sep 2026 22:19:26 GMT The word "lead" covers a scraped business record and a booked sales meeting, and they differ in price by four orders of magnitude. Published per-record prices from named list vendors, agency retainer and pay-per-lead rates, paid-search cost per lead, blended cost-per-lead benchmarks by industry, and our own first-hand measurement of $0.0101 to $0.0143 per usable business record from public data. What each number does and does not include. lead-gen https://quanticdata.io/blog/what-a-google-maps-lead-contains/ https://quanticdata.io/blog/what-a-google-maps-lead-contains/ Thu, 03 Sep 2026 22:19:26 GMT Every Google Maps scraper page lists the fields you get. None of them publish how often those fields are actually filled. We collected 458 dentist listings across Milan, Berlin, Madrid, Paris and London on one afternoon and counted: phone 99.3%, website 96.5%, rating 99.6%, review count 12.7%, an email on the site pass 60.3%. Cost per usable lead, the Paris booking-platform problem, the method, the limitations and the published dataset. lead-gen https://quanticdata.io/blog/is-buying-leads-worth-it/ https://quanticdata.io/blog/is-buying-leads-worth-it/ Thu, 03 Sep 2026 08:15:14 GMT “Buying leads” covers five different products, from real-time shared leads to a spreadsheet of contacts, and they are worth it at very different prices. The break-even formula that turns your margin and close rate into a maximum price per lead, benchmark costs by industry, why bought leads underperform and how to test a vendor with a small sample, where the law actually bites, and the public-data route that produces a contactable business record for cents. lead-gen https://quanticdata.io/blog/web-scraping-vs-api/ https://quanticdata.io/blog/web-scraping-vs-api/ Thu, 03 Sep 2026 08:15:14 GMT Web scraping versus API is a false binary: there are three options, and most real pipelines use two of them. When an official API is the right answer and the four ways it stops being one; what scraping costs in maintenance and risk; where scraping APIs and ready-made collectors fit; a seven-question decision tree, a worked cost example, and the hybrid pattern that holds up. scraping-api https://quanticdata.io/blog/residential-vs-datacenter-proxies/ https://quanticdata.io/blog/residential-vs-datacenter-proxies/ Thu, 03 Sep 2026 08:15:14 GMT Residential and datacenter proxies differ in one fact the target can look up in a millisecond: the network the IP belongs to. Everything else, speed, price, block rate, follows from that. What the label means, a table that compares them honestly, a 100-request test that tells you which one your target requires, the cost-per-successful-page math, and when ISP or mobile is the right third answer. proxy-types https://quanticdata.io/blog/how-much-proxy-data-do-i-need/ https://quanticdata.io/blog/how-much-proxy-data-do-i-need/ Thu, 03 Sep 2026 08:15:14 GMT Proxy plans are sold by the gigabyte and nobody tells you how many pages that is. We fetched 11 major pages and recorded what they cost on the wire: 47 KB to 302 KB compressed for the HTML, against 2.5 to 2.9 MB for a full browser load. The formula that turns pages per month into GB, three worked examples, the break-even where per-page pricing beats per-GB, and the settings that cut usage by five to twenty times. proxy-types https://quanticdata.io/blog/proxy-not-working-checklist/ https://quanticdata.io/blog/proxy-not-working-checklist/ Thu, 03 Sep 2026 08:15:14 GMT “Proxy not working” is four different problems wearing one name: you cannot reach the proxy, the proxy refuses you, the proxy reaches the site but the site refuses it, or it works and is slow. Each layer has its own error strings and its own fix. A table that maps the message to the layer, curl timings that separate slow from broken, and a 10-step checklist in the order that saves the most time. troubleshooting https://quanticdata.io/blog/429-too-many-requests-web-scraping/ https://quanticdata.io/blog/429-too-many-requests-web-scraping/ Thu, 03 Sep 2026 08:15:14 GMT A 429 is the one block that tells the truth: you crossed a rate limit. What it does not tell you is what the limit is keyed on, and that decides whether proxies help at all. A two-request test to find the key, backoff code that honours Retry-After, the concurrency formula that turns pages per hour into IPs in flight, and a per-host token bucket. troubleshooting https://quanticdata.io/blog/web-scraping-403-forbidden/ https://quanticdata.io/blog/web-scraping-403-forbidden/ Thu, 03 Sep 2026 08:15:14 GMT The browser loads the page and your script gets 403. The response headers usually name the system that refused you, and that decides the fix: a user agent, the full header set, the IP type, or the TLS fingerprint. A decision tree you can run in five minutes, Python fixes for each branch, and what we saw fetching 16 major sites with a pure HTTP client behind US residential IPs. troubleshooting https://quanticdata.io/blog/how-to-fix-407-proxy-authentication-required/ https://quanticdata.io/blog/how-to-fix-407-proxy-authentication-required/ Thu, 03 Sep 2026 08:15:14 GMT A 407 is the proxy refusing to forward your request until it sees credentials it accepts, so rotating headers or slowing down cannot fix it. On HTTPS it does not even arrive as a status code. How to read the challenge, tell the five causes apart, and fix each one in curl, Python requests, httpx, Node and Scrapy. troubleshooting https://quanticdata.io/blog/llms-txt-vs-robots-txt/ https://quanticdata.io/blog/llms-txt-vs-robots-txt/ Wed, 02 Sep 2026 18:52:59 GMT robots.txt answers whether an agent may fetch a page and is honoured by every major AI operator. llms.txt answers what is worth reading and is documented as read by none of them. The formats, the resolution rules, what neither file can do, and measured adoption for both across the same 355 domains. seo-data https://quanticdata.io/blog/how-many-sites-use-llms-txt/ https://quanticdata.io/blog/how-many-sites-use-llms-txt/ Wed, 02 Sep 2026 18:52:59 GMT 64 of 355 top domains publish a real llms.txt. Count by status code instead and you get 39.7%, because 77 sites answer 200 with their homepage. Includes what the real files contain, and the cross-tab nobody has run: publishers of llms.txt block AI crawlers four to seven times less, and none of them blocks a retrieval crawler. seo-data https://quanticdata.io/blog/ai-crawler-user-agent-list/ https://quanticdata.io/blog/ai-crawler-user-agent-list/ Wed, 02 Sep 2026 17:52:16 GMT A reference table of every AI user agent: which operator runs it, whether its job is training, retrieval or a user-triggered fetch, what blocking it actually costs you, and the share of 326 top domains that block it today. Plus the robots.txt mechanics that make these rules mean something other than intended. seo-data https://quanticdata.io/blog/should-i-block-ai-crawlers/ https://quanticdata.io/blog/should-i-block-ai-crawlers/ Wed, 02 Sep 2026 17:52:16 GMT We parsed the robots.txt of 326 of the web’s top domains. AI crawlers are blocked seven times more often than Googlebot, training bots twice as often as the search bots from the same company — and a large share of the blocking lands on crawlers that would have cited the site and never trained on it. seo-data https://quanticdata.io/blog/how-to-find-business-email-addresses/ https://quanticdata.io/blog/how-to-find-business-email-addresses/ Wed, 02 Sep 2026 15:49:18 GMT An address is published, inferred or guessed, and the three behave completely differently once you press send. Where business addresses actually live, why role mailboxes are a category rather than a fallback, what verification can and cannot prove, and the obligations that attach the moment you find one. lead-gen https://quanticdata.io/blog/how-to-build-a-b2b-lead-list/ https://quanticdata.io/blog/how-to-build-a-b2b-lead-list/ Wed, 02 Sep 2026 15:49:18 GMT Four passes: express the ICP as predicates a machine can execute, source companies from public surfaces that answer different questions, enrich each row into a contactable record with provenance, and verify before anything is sent. Includes decay maths derived from official turnover statistics rather than a vendor deck. lead-gen https://quanticdata.io/blog/is-scraping-google-maps-legal/ https://quanticdata.io/blog/is-scraping-google-maps-legal/ Wed, 02 Sep 2026 15:49:18 GMT The question hides four: the CFAA, contract, data protection and copyright. hiQ won the CFAA argument and lost on breach of contract; Google’s terms define prohibited automated access by pointing at robots.txt, which explicitly allows /maps/search/ and /maps/place/; and GDPR applies regardless of whether the data was public. lead-gen https://quanticdata.io/blog/how-to-scrape-leads-from-google-maps/ https://quanticdata.io/blog/how-to-scrape-leads-from-google-maps/ Wed, 02 Sep 2026 15:49:18 GMT A Maps listing gives you name, category, address, phone, rating, coordinates and a domain — but no email address. Building a usable lead list therefore means tiling the city, de-duplicating on place id, crawling each domain for a published contact, and knowing which of those businesses you are actually allowed to email. lead-gen https://quanticdata.io/blog/do-ai-crawlers-render-javascript/ https://quanticdata.io/blog/do-ai-crawlers-render-javascript/ Tue, 25 Aug 2026 17:57:22 GMT We audited 800 top domains with JS on and off: 1 in 15 is invisible to AI crawlers, 1 in 6 loses half its words — and the biggest sites fare worst. seo-data https://quanticdata.io/blog/how-to-use-scrapy-playwright/ https://quanticdata.io/blog/how-to-use-scrapy-playwright/ Thu, 30 Jul 2026 17:14:34 GMT Install the download handler, flag requests to render with Playwright, wait for content, add proxies — and understand the throughput cost of a browser per page. tool-scrapy https://quanticdata.io/blog/how-to-use-a-proxy-in-nodejs/ https://quanticdata.io/blog/how-to-use-a-proxy-in-nodejs/ Thu, 30 Jul 2026 17:12:09 GMT Native fetch needs an undici ProxyAgent; axios takes a proxy config or agent. The setup for both, authentication, HTTPS tunneling, and rotation for scraping. tool-node https://quanticdata.io/blog/how-to-use-undetected-chromedriver/ https://quanticdata.io/blog/how-to-use-undetected-chromedriver/ Thu, 30 Jul 2026 16:28:52 GMT Install and launch the patched driver that evades Selenium detection, add a proxy, and understand the ceiling — IP reputation and behavior it can't fix. tool-selenium https://quanticdata.io/blog/how-to-use-a-proxy-in-puppeteer/ https://quanticdata.io/blog/how-to-use-a-proxy-in-puppeteer/ Thu, 30 Jul 2026 16:27:14 GMT Set the proxy in launch args, authenticate with page.authenticate, rotate per page via browser contexts, and dodge the mistakes that leak your real IP. tool-puppeteer https://quanticdata.io/blog/how-to-use-mcp-in-cursor/ https://quanticdata.io/blog/how-to-use-mcp-in-cursor/ Thu, 30 Jul 2026 16:24:58 GMT What MCP gives Cursor's agent, how to add a server in mcp.json, project vs global scope, approving tool calls, and connecting a web-data server for live scraping. ai-dev-stack https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/ https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/ Thu, 30 Jul 2026 16:23:30 GMT The proxies dict, authenticated and SOCKS5 proxies, session reuse, rotating per request, and the mistakes — HTTPS key, verify, env vars — that silently break it. tool-python https://quanticdata.io/blog/playwright-stealth-in-python/ https://quanticdata.io/blog/playwright-stealth-in-python/ Thu, 30 Jul 2026 16:21:46 GMT Install playwright-stealth, understand which automation tells it patches, why sophisticated detectors still win, and the IP-plus-fingerprint combination that lasts. tool-playwright https://quanticdata.io/blog/how-to-use-a-proxy-in-n8n/ https://quanticdata.io/blog/how-to-use-a-proxy-in-n8n/ Thu, 30 Jul 2026 16:20:06 GMT Two ways to route n8n through a proxy — per-node in the HTTP Request node or globally with env vars — the gotchas that trip people up, and rotation for scraping. automation https://quanticdata.io/blog/how-to-feed-data-to-an-llm/ https://quanticdata.io/blog/how-to-feed-data-to-an-llm/ Thu, 30 Jul 2026 15:38:09 GMT Context window, RAG, tool calls or fine-tuning — the four ways to give an LLM your data, how each works, when to pick it, and how to keep the source fresh. data-for-ai https://quanticdata.io/blog/how-to-create-an-mcp-server/ https://quanticdata.io/blog/how-to-create-an-mcp-server/ Thu, 30 Jul 2026 15:36:37 GMT Pick the SDK, define tools with clear schemas, choose a transport, test in a client — plus the tool-description rules that decide whether a model actually calls it. mcp https://quanticdata.io/blog/is-web-scraping-legal-in-europe/ https://quanticdata.io/blog/is-web-scraping-legal-in-europe/ Thu, 30 Jul 2026 15:27:52 GMT The EU has no anti-scraping law, but four layers draw the lines: the GDPR, the database right, the DSM text-and-data-mining exception, and unfair-competition rules. scraping-api https://quanticdata.io/blog/what-is-a-rotating-proxy/ https://quanticdata.io/blog/what-is-a-rotating-proxy/ Thu, 30 Jul 2026 15:27:41 GMT A rotating proxy hands out a fresh IP per request from a pool behind one endpoint. How that works, how it differs from static and sticky, and what it's for. proxy-types https://quanticdata.io/blog/how-do-data-pipelines-work/ https://quanticdata.io/blog/how-do-data-pipelines-work/ Thu, 30 Jul 2026 15:23:19 GMT The four stages every pipeline shares, ETL vs ELT, batch vs streaming, how orchestration ties it together — and where web data feeds in at the ingest step. data-for-ai https://quanticdata.io/blog/how-to-create-an-llm-dataset/ https://quanticdata.io/blog/how-to-create-an-llm-dataset/ Thu, 30 Jul 2026 15:04:22 GMT Choose the format for your goal, source raw text from the web, clean and deduplicate, structure the examples, and quality-check — the pipeline that decides model quality. data-for-ai https://quanticdata.io/blog/how-to-use-playwright-for-scraping/ https://quanticdata.io/blog/how-to-use-playwright-for-scraping/ Thu, 30 Jul 2026 15:02:12 GMT Install and launch, wait for content that loads late, extract with locators, add proxies and stealth — and the point where a browser per page stops being worth it. browser-agents https://quanticdata.io/blog/how-can-residential-proxies-be-legal/ https://quanticdata.io/blog/how-can-residential-proxies-be-legal/ Thu, 30 Jul 2026 15:00:33 GMT The legality of a residential proxy comes down to one thing — did the person whose IP you're using agree? The ethical/illegal line, and how to check which pool you're buying. residential https://quanticdata.io/blog/how-to-stop-web-scraping/ https://quanticdata.io/blog/how-to-stop-web-scraping/ Thu, 30 Jul 2026 14:58:26 GMT A defender's honest guide: the tactics that stop casual scrapers, the ones that only add friction, and why protecting logins and PII beats blocking public prices. ai-scraping https://quanticdata.io/blog/how-to-build-an-ai-web-scraper/ https://quanticdata.io/blog/how-to-build-an-ai-web-scraper/ Thu, 30 Jul 2026 14:31:04 GMT Where the LLM actually belongs in a scraping pipeline, how prompt-based extraction survives redesigns, and how to keep token costs from eating the project. ai-scraping https://quanticdata.io/blog/how-to-set-up-rotating-proxies/ https://quanticdata.io/blog/how-to-set-up-rotating-proxies/ Thu, 30 Jul 2026 14:26:37 GMT How the single rotating endpoint works, config in curl/Python/Node, sticky vs per-request sessions, and the settings that decide whether you get banned. proxy-types https://quanticdata.io/blog/how-mcp-servers-work/ https://quanticdata.io/blog/how-mcp-servers-work/ Thu, 30 Jul 2026 14:24:04 GMT Host, client, server; JSON-RPC and transports; tools, resources and prompts — and a single tool call traced from user question to structured result. mcp https://quanticdata.io/blog/what-is-web-crawling-in-python/ https://quanticdata.io/blog/what-is-web-crawling-in-python/ Thu, 30 Jul 2026 14:22:28 GMT The frontier algorithm behind every crawler, a minimal Python example, how Scrapy and Crawlee fit in, and the scaling wall where DIY stops paying. crawl https://quanticdata.io/blog/how-a-serp-api-works/ https://quanticdata.io/blog/how-a-serp-api-works/ Thu, 30 Jul 2026 14:19:53 GMT The full path from query to JSON: localization parameters, the identity layer, HTTP vs rendered fetches, parsing rich blocks, verticals and pricing. serp https://quanticdata.io/blog/is-web-scraping-legal-in-germany/ https://quanticdata.io/blog/is-web-scraping-legal-in-germany/ Thu, 30 Jul 2026 14:17:55 GMT Germany has no anti-scraping law — but GDPR, database rights, the TDM exception and unfair-competition rules draw real lines. Here is where they sit. scraping-api https://quanticdata.io/blog/can-ai-work-without-data/ https://quanticdata.io/blog/can-ai-work-without-data/ Thu, 30 Jul 2026 14:16:13 GMT Two different questions hide in one query: AI runs offline just fine, but AI without data is a contradiction — and stale data is a slower version of none. ai-scraping https://quanticdata.io/blog/how-to-build-an-ai-browser-agent/ https://quanticdata.io/blog/how-to-build-an-ai-browser-agent/ Thu, 30 Jul 2026 14:13:57 GMT The observe-decide-act loop, the browser-control stack, prompt-injection defenses and honest cost math — everything a working browser agent actually needs. browser-agents https://quanticdata.io/blog/how-to-perform-an-seo-audit/ https://quanticdata.io/blog/how-to-perform-an-seo-audit/ Thu, 30 Jul 2026 14:12:07 GMT A practical six-step SEO audit process with a checklist, the crawler-vs-user diff most audits skip, and how to run the whole thing programmatically. seo-data https://quanticdata.io/blog/how-to-price-watch-on-amazon/ https://quanticdata.io/blog/how-to-price-watch-on-amazon/ Wed, 29 Jul 2026 15:56:29 GMT Three ways to price watch on Amazon — native price history and alerts, third-party trackers, or your own API watcher — with honest cost math for each. use-cases https://quanticdata.io/blog/how-to-browser-fingerprint/ https://quanticdata.io/blog/how-to-browser-fingerprint/ Wed, 29 Jul 2026 15:50:44 GMT The signals, hashing and detection logic behind browser fingerprinting — plus how automation stacks get caught, and what it costs to avoid the problem entirely. anti-bot https://quanticdata.io/blog/how-to-rotate-proxy-in-selenium-python/ https://quanticdata.io/blog/how-to-rotate-proxy-in-selenium-python/ Wed, 29 Jul 2026 15:48:30 GMT Chrome fixes its proxy at launch. Here are the three real ways to rotate IPs in Selenium Python, with code, auth handling and honest bandwidth math. proxy-types https://quanticdata.io/blog/how-to-get-data-for-ai/ https://quanticdata.io/blog/how-to-get-data-for-ai/ Wed, 29 Jul 2026 15:41:42 GMT Five real sources of AI training data, how to judge them, and the cost math nobody publishes — plus working API calls for live web data. data-for-ai