# Honeypot Traps in Web Scraping, Tested

> Honeypot traps in web scraping are links and form fields people cannot see. We built eight traps, caught all of them, and ran the same check on four real sites.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Honeypot Traps in Web Scraping, Tested

# Honeypot traps in web scraping: what they look like, and a check that finds them

Anti-bot · Nov 20, 2026 · 6 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![Honeypot traps on a web page: two product links a visitor can see, and eight hidden trap links made invisible with display none, visibility hidden, zero opacity, off-screen position, a zero-size box, white text on white and an empty link; a visibility check in Playwright keeps the two visible links and flags all eight traps.](https://quanticdata.io/blog/honeypot-traps-web-scraping/hero.webp?v=20261120h)

On this page

- [What a honeypot trap is](https://quanticdata.io/blog/honeypot-traps-web-scraping/#what-a-honeypot-trap-is)
- [Our test page: eight traps, two real links](https://quanticdata.io/blog/honeypot-traps-web-scraping/#our-test-page-eight-traps-two-real-links)
- [The same check on four real pages](https://quanticdata.io/blog/honeypot-traps-web-scraping/#the-same-check-on-four-real-pages)
- [How a well-behaved crawler stays out of them](https://quanticdata.io/blog/honeypot-traps-web-scraping/#how-a-well-behaved-crawler-stays-out-of-them)
- [When you need a rendered page to check visibility](https://quanticdata.io/blog/honeypot-traps-web-scraping/#when-you-need-a-rendered-page-to-check-visibility)

Honeypot traps in web scraping are links, pages or form fields that a person never sees but a crawler finds in the HTML: a link styled with display:none, a URL disallowed in robots.txt, a hidden "website" field on a form. A visitor never touches them, so anything that does is a bot. A well-behaved crawler stays out of them by following only what a visitor could see and use.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=honeypot-traps-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

We tested that on 20 November 2026 with Chromium 149 and Playwright: first on a page we built with eight common trap patterns, then on four real public pages loaded through a QuanticData residential IP. The results show both halves of the problem: the check catches the traps, and "hidden" alone is not proof of a trap.

## What a honeypot trap is

The word comes from security, where a honeypot is a decoy system that attracts attackers so they can be watched. On websites it means the same thing in miniature:

- **Hidden links.** An anchor that no person can see or click, placed where a link extractor will collect it. Any request for that URL comes from software.
- **Trap URLs in robots.txt.** A path that robots.txt disallows and nothing on the site needs. A crawler that ignores robots.txt walks straight in.
- **Hidden form fields.** A field hidden by CSS, often with a tempting name like "website" or "email2". People leave it empty; form-filling bots fill everything.
- **Decoy data.** Fake records shown only to clients a site already suspects, so a scraped dataset gives itself away later.

Hitting a trap rarely brings an error on the spot. The usual consequence is quieter: the client's IP or session gets a worse score, later requests meet challenges or 403s, or the data starts to differ from what visitors see.

## Our test page: eight traps, two real links

We built a page with two ordinary product links and eight trap links, each hidden a different way, plus a form with a visible email field and a hidden "website" field. A naive link extractor, the kind that collects every `a[href]`, returned all 10 links. Then we asked the rendered page which links a person could actually see:

```javascript
const visibleLinks = () => [...document.querySelectorAll('a[href]')].map(a => {
  const r = a.getBoundingClientRect(); const s = getComputedStyle(a);
  let why = '';
  if (!a.checkVisibility({ checkOpacity: true, checkVisibilityCSS: true })) why = 'not rendered';
  else if (r.width < 1 || r.height < 1) why = 'zero size';
  else if (r.right < 0 || r.bottom < 0 || r.left > document.documentElement.scrollWidth) why = 'off-screen';
  else {
    let el = a.parentElement, clipped = false;
    while (el && el !== document.body) { const es = getComputedStyle(el), er = el.getBoundingClientRect();
      if (es.overflow !== 'visible' && (er.width < 1 || er.height < 1)) { clipped = true; break; } el = el.parentElement; }
    if (clipped) why = 'clipped by a zero-size parent';
    else if (s.color === getComputedStyle(document.body).backgroundColor && !a.querySelector('img')) why = 'text colour = background';
    else if (!a.textContent.trim() && !a.querySelector('img,svg')) why = 'no text or image';
  }
  return { href: a.getAttribute('href'), visible: !why, why };
});
const links = await page.evaluate(visibleLinks);   // Playwright
```

| Link on the test page | How it was hidden | Flagged by the check |
| --- | --- | --- |
| /products/1, /products/2 | Not hidden | Kept as visible |
| /trap/display-none | `display: none` | Yes: not rendered |
| /trap/hidden-attr | Inside a `<div hidden>` | Yes: not rendered |
| /trap/visibility-hidden | `visibility: hidden` | Yes: not rendered |
| /trap/opacity-0 | `opacity: 0` | Yes: not rendered |
| /trap/off-screen | `position: absolute; left: -9999px` | Yes: off-screen |
| /trap/zero-size | Inside a 0 x 0 box with overflow hidden | Yes: clipped by a zero-size parent |
| /trap/white-on-white | White text on a white page | Yes: text colour = background |
| /trap/nofollow | An empty anchor with rel="nofollow" | Yes: zero size |

All 8 traps flagged, both real links kept. The hidden "website" form field also reported as not visible, which is the rule for forms: never fill a field a person cannot see.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=honeypot-traps-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## The same check on four real pages

We then loaded four public pages through Residential Premium and ran the identical check:

| Page | Links | Flagged as not visible | What the sampled links were |
| --- | --- | --- | --- |
| books.toscrape.com | 94 | 0 | Nothing hidden |
| quotes.toscrape.com | 55 | 40 | Tag labels: the colour rule compared the link with the page background, not with the coloured label behind it |
| en.wikipedia.org/wiki/Web_crawler | 770 | 181 | Collapsed menus and panels (82 visibility:hidden, 73 display:none, 26 inside a hidden parent) |
| www.python.org | 215 | 71 | Drop-down menus of the main navigation (65 inside a hidden parent) |

292 links were flagged across the three sites; the ones we sampled, five per page, were menus and labels, not traps. Menus that open on hover or click are hidden until a person opens them, and every site with a mega-menu has dozens of them. The quotes.toscrape.com result is a lesson about checks themselves: a colour rule must compare the link with the background it actually sits on, or it flags perfectly visible labels.

## How a well-behaved crawler stays out of them

1. **Read robots.txt and obey it.** Trap URLs are often disallowed paths, so a crawler that respects the file never requests them. Our [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/) checks a URL against a site's rules.
2. **Prefer the sitemap to link discovery.** A sitemap is the list of pages the owner publishes.
3. **Skip invisible links, do not panic about them.** Use a check like the one above to decide what to follow, and expect menus among the hidden links. If a hidden link also appears visibly elsewhere or in the sitemap, it is a normal page.
4. **Never fill hidden form fields**, and do not submit forms at all unless that is the point of an agreed integration.
5. **Stay at a visitor's pace.** A trap only matters when it is combined with other signals; [crawl a website without getting blocked](https://quanticdata.io/blog/crawl-a-website-without-getting-blocked/) shows a run of 30 pages at one request per second, all answered 200.

## When you need a rendered page to check visibility

Visibility can only be judged on a rendered page: the CSS that hides a link is often in a stylesheet or applied by a script, invisible in the raw HTML. That means a browser, like the Playwright run above (see [Playwright for scraping](https://quanticdata.io/blog/how-to-use-playwright-for-scraping/)), or a service that renders for you: for public pages that need a real browser, see the [Web Unlocker](https://quanticdata.io/web-unlocker/). The [website crawler API](https://quanticdata.io/crawl-map/) lists and fetches the pages of a whole site, and [residential proxies](https://quanticdata.io/residential-proxies/) carry your own browser's traffic. For the other signals a site combines with a trap, read [how websites detect bots](https://quanticdata.io/blog/how-do-websites-detect-bots/); if you run a site and want to set traps of your own, [how to stop web scraping](https://quanticdata.io/blog/how-to-stop-web-scraping/) covers that side.

### Sources & further reading

- [MDN: Element.checkVisibility()](https://developer.mozilla.org/en-US/docs/Web/API/Element/checkVisibility)
- [MDN: the hidden global attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/hidden)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
- [Wikipedia: Honeypot (computing)](https://en.wikipedia.org/wiki/Honeypot_%28computing%29)

## FAQ

Quick answers on honeypot traps in web scraping.

[Something else? Ask us](mailto:hello@quanticdata.io)

### What is a honeypot trap in web scraping?

A link, page or form field that a person cannot see but a crawler finds in the HTML, such as a link with display:none or a hidden form field. Anything that requests or fills it is software, so the site can flag it.

### How do I detect honeypot links?

Render the page and check each link's visibility: whether it is rendered, its size, its position and its colour against the background. On our test page that check flagged all 8 trap links and kept the 2 real ones.

### Is every hidden link a honeypot?

No. On Wikipedia and python.org our check found 181 and 71 hidden links, and the ones we sampled were collapsed menus. Skip hidden links when crawling, but expect most of them to be ordinary navigation.

### How does a well-behaved crawler stay out of honeypot traps?

Obey robots.txt, prefer the sitemap to link discovery, follow only links a visitor can see, never fill hidden form fields, and keep a visitor's pace. Traps matter most when they add to other signals.

### What happens if a scraper hits a honeypot?

Usually nothing visible at once. The IP or session gets a worse score, and later requests meet challenges, 403 responses or, on some sites, data that differs from what visitors see.

## Render the page, then decide what to follow

The website crawler API maps a whole site from its sitemap and links and returns its pages as clean Markdown.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=honeypot-traps-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

- [How Do Websites Detect Bots? Signals Tested](https://quanticdata.io/blog/how-do-websites-detect-bots/): Anti-bot · Websites detect bots by stacking signals: the IP and its network, the headers, the TLS handshake, what JavaScript finds in the browser, and how the visitor behaves. We read those signals from six real clients, five of them also through residential and mobile proxies, to show what a proxy changes: the IP, not the client.
- [TLS Fingerprinting: JA3 and JA4 Tested](https://quanticdata.io/blog/tls-fingerprinting-ja3-ja4/): Anti-bot · A TLS fingerprint is a hash of the hello your client sends before any HTTP header. We measured JA3 and JA4 for curl, Python Requests, httpx and Chromium, directly and through three proxy networks, and Node directly. For the four clients we ran through the proxies, the IP changed every time and the JA4 never did.
- [Canvas Fingerprinting: What a Proxy Changes](https://quanticdata.io/blog/canvas-fingerprinting/): Anti-bot · Canvas fingerprinting draws hidden text and shapes, reads the pixels back and hashes them: the result depends on the graphics stack, fonts and browser, not on the network. We measured it in Chromium, headless and with a window, directly and through three proxy networks. Eight runs, two hashes.

---

Source: https://quanticdata.io/blog/honeypot-traps-web-scraping/ · Site index for AI: https://quanticdata.io/llms.txt
