Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Honeypot traps in web scraping: what they look like, and a check that finds them

Honeypot traps on a web page: two product links a visitor can see, and eight hidden trap links made invisible with display none, visibility hidden, zero opacity, off-screen position, a zero-size box, white text on white and an empty link; a visibility check in Playwright keeps the two visible links and flags all eight traps.
Honeypot traps on a web page: two product links a visitor can see, and eight hidden trap links made invisible with display none, visibility hidden, zero opacity, off-screen position, a zero-size box, white text on white and an empty link; a visibility check in Playwright keeps the two visible links and flags all eight traps.

Honeypot traps in web scraping are links, pages or form fields that a person never sees but a crawler finds in the HTML: a link styled with display:none, a URL disallowed in robots.txt, a hidden "website" field on a form. A visitor never touches them, so anything that does is a bot. A well-behaved crawler stays out of them by following only what a visitor could see and use.

We tested that on 20 November 2026 with Chromium 149 and Playwright: first on a page we built with eight common trap patterns, then on four real public pages loaded through a QuanticData residential IP. The results show both halves of the problem: the check catches the traps, and "hidden" alone is not proof of a trap.

What a honeypot trap is

The word comes from security, where a honeypot is a decoy system that attracts attackers so they can be watched. On websites it means the same thing in miniature:

  • Hidden links. An anchor that no person can see or click, placed where a link extractor will collect it. Any request for that URL comes from software.
  • Trap URLs in robots.txt. A path that robots.txt disallows and nothing on the site needs. A crawler that ignores robots.txt walks straight in.
  • Hidden form fields. A field hidden by CSS, often with a tempting name like "website" or "email2". People leave it empty; form-filling bots fill everything.
  • Decoy data. Fake records shown only to clients a site already suspects, so a scraped dataset gives itself away later.

Hitting a trap rarely brings an error on the spot. The usual consequence is quieter: the client's IP or session gets a worse score, later requests meet challenges or 403s, or the data starts to differ from what visitors see.

We built a page with two ordinary product links and eight trap links, each hidden a different way, plus a form with a visible email field and a hidden "website" field. A naive link extractor, the kind that collects every a[href], returned all 10 links. Then we asked the rendered page which links a person could actually see:

const visibleLinks = () => [...document.querySelectorAll('a[href]')].map(a => {
  const r = a.getBoundingClientRect(); const s = getComputedStyle(a);
  let why = '';
  if (!a.checkVisibility({ checkOpacity: true, checkVisibilityCSS: true })) why = 'not rendered';
  else if (r.width < 1 || r.height < 1) why = 'zero size';
  else if (r.right < 0 || r.bottom < 0 || r.left > document.documentElement.scrollWidth) why = 'off-screen';
  else {
    let el = a.parentElement, clipped = false;
    while (el && el !== document.body) { const es = getComputedStyle(el), er = el.getBoundingClientRect();
      if (es.overflow !== 'visible' && (er.width < 1 || er.height < 1)) { clipped = true; break; } el = el.parentElement; }
    if (clipped) why = 'clipped by a zero-size parent';
    else if (s.color === getComputedStyle(document.body).backgroundColor && !a.querySelector('img')) why = 'text colour = background';
    else if (!a.textContent.trim() && !a.querySelector('img,svg')) why = 'no text or image';
  }
  return { href: a.getAttribute('href'), visible: !why, why };
});
const links = await page.evaluate(visibleLinks);   // Playwright
Link on the test pageHow it was hiddenFlagged by the check
/products/1, /products/2Not hiddenKept as visible
/trap/display-nonedisplay: noneYes: not rendered
/trap/hidden-attrInside a <div hidden>Yes: not rendered
/trap/visibility-hiddenvisibility: hiddenYes: not rendered
/trap/opacity-0opacity: 0Yes: not rendered
/trap/off-screenposition: absolute; left: -9999pxYes: off-screen
/trap/zero-sizeInside a 0 x 0 box with overflow hiddenYes: clipped by a zero-size parent
/trap/white-on-whiteWhite text on a white pageYes: text colour = background
/trap/nofollowAn empty anchor with rel="nofollow"Yes: zero size

All 8 traps flagged, both real links kept. The hidden "website" form field also reported as not visible, which is the rule for forms: never fill a field a person cannot see.

The same check on four real pages

We then loaded four public pages through Residential Premium and ran the identical check:

PageLinksFlagged as not visibleWhat the sampled links were
books.toscrape.com940Nothing hidden
quotes.toscrape.com5540Tag labels: the colour rule compared the link with the page background, not with the coloured label behind it
en.wikipedia.org/wiki/Web_crawler770181Collapsed menus and panels (82 visibility:hidden, 73 display:none, 26 inside a hidden parent)
www.python.org21571Drop-down menus of the main navigation (65 inside a hidden parent)

292 links were flagged across the three sites; the ones we sampled, five per page, were menus and labels, not traps. Menus that open on hover or click are hidden until a person opens them, and every site with a mega-menu has dozens of them. The quotes.toscrape.com result is a lesson about checks themselves: a colour rule must compare the link with the background it actually sits on, or it flags perfectly visible labels.

How a well-behaved crawler stays out of them

  1. Read robots.txt and obey it. Trap URLs are often disallowed paths, so a crawler that respects the file never requests them. Our robots.txt tester checks a URL against a site's rules.
  2. Prefer the sitemap to link discovery. A sitemap is the list of pages the owner publishes.
  3. Skip invisible links, do not panic about them. Use a check like the one above to decide what to follow, and expect menus among the hidden links. If a hidden link also appears visibly elsewhere or in the sitemap, it is a normal page.
  4. Never fill hidden form fields, and do not submit forms at all unless that is the point of an agreed integration.
  5. Stay at a visitor's pace. A trap only matters when it is combined with other signals; crawl a website without getting blocked shows a run of 30 pages at one request per second, all answered 200.

When you need a rendered page to check visibility

Visibility can only be judged on a rendered page: the CSS that hides a link is often in a stylesheet or applied by a script, invisible in the raw HTML. That means a browser, like the Playwright run above (see Playwright for scraping), or a service that renders for you: for public pages that need a real browser, see the Web Unlocker. The website crawler API lists and fetches the pages of a whole site, and residential proxies carry your own browser's traffic. For the other signals a site combines with a trap, read how websites detect bots; if you run a site and want to set traps of your own, how to stop web scraping covers that side.

Sources & further reading

FAQ

Quick answers on honeypot traps in web scraping.

Something else? Ask us

What is a honeypot trap in web scraping?

A link, page or form field that a person cannot see but a crawler finds in the HTML, such as a link with display:none or a hidden form field. Anything that requests or fills it is software, so the site can flag it.

How do I detect honeypot links?

Render the page and check each link's visibility: whether it is rendered, its size, its position and its colour against the background. On our test page that check flagged all 8 trap links and kept the 2 real ones.

Is every hidden link a honeypot?

No. On Wikipedia and python.org our check found 181 and 71 hidden links, and the ones we sampled were collapsed menus. Skip hidden links when crawling, but expect most of them to be ordinary navigation.

How does a well-behaved crawler stay out of honeypot traps?

Obey robots.txt, prefer the sitemap to link discovery, follow only links a visitor can see, never fill hidden form fields, and keep a visitor's pace. Traps matter most when they add to other signals.

What happens if a scraper hits a honeypot?

Usually nothing visible at once. The IP or session gets a worse score, and later requests meet challenges, 403 responses or, on some sites, data that differs from what visitors see.

Render the page, then decide what to follow

The website crawler API maps a whole site from its sitemap and links and returns its pages as clean Markdown.

Related reading