Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

What is an HTML parser? Tokens, the tree and broken markup, on a real page

An HTML parser at work on 20 November 2026: one real page of 244,347 characters became a tree of 1,973 elements; the code window shows a table written without closing tags and the path each Python parser built for its row, table then tr for html.parser and lxml, table then tbody then tr for html5lib; the card lists the three jobs of a parser.
An HTML parser at work on 20 November 2026: one real page of 244,347 characters became a tree of 1,973 elements; the code window shows a table written without closing tags and the path each Python parser built for its row, table then tr for html.parser and lxml, table then tbody then tr for html5lib; the card lists the three jobs of a parser.

An HTML parser is a program that reads the text of a web page and turns it into a tree of elements that code can search and change. It splits the text into tokens (tags, attributes, text, comments), nests them into a tree, and repairs markup that is incomplete or wrong. Browsers, search engine crawlers and scrapers all start with one.

The definition is short; the behaviour is where parsers differ. On 20 November 2026 we fetched one real page and ran five parsers over it, three in Python and two in Node.js, then gave each of them the same pieces of broken HTML. The output below is from that run.

What an HTML parser does, in three steps

StepInputOutputOn our test page
1. TokeniseCharactersStart tags, end tags, text, comments1,973 start tags, 1,930 end tags, 8 comments
2. Build the treeTokensNested elements and text nodes (the DOM)1,973 elements, up to 28 levels deep
3. RepairMissing or misplaced tagsA complete, well-nested tree43 tags never closed in the source, all closed in the tree

After that, your code asks the tree questions: "every link", "the first h1", "the price inside this card". That selecting step is not the parser's job, but it only works because the parser built the tree first. Turning the selected pieces into typed records is the wider subject of data parsing.

Step 1: the tokens

Python ships a tokeniser in its standard library, html.parser.HTMLParser. It calls a method for every token it meets, so a few lines are enough to see them:

from html.parser import HTMLParser

class Tok(HTMLParser):
    def handle_starttag(self, tag, attrs): print("  start  ", tag, dict(attrs) if attrs else "")
    def handle_endtag(self, tag): print("  end    ", tag)
    def handle_data(self, data):
        if data.strip(): print("  text   ", repr(data))
    def handle_comment(self, data): print("  comment", repr(data))

Tok().feed('<p class="lead">Hello <a href="/docs">docs</a><!-- note --></p>')

  start   p {'class': 'lead'}
  text    'Hello '
  start   a {'href': '/docs'}
  text    'docs'
  end     a
  comment ' note '
  end     p

Seven tokens for one short paragraph. The tokeniser has already done useful work: it separated the attribute name from its value and removed the quotes. It has not built anything yet: nothing here says that the link is inside the paragraph.

Then the real page. We fetched the English Wikipedia article on web scraping through a QuanticData residential proxy line in the United States, with curl and our own User-Agent, and got status 200 and 244,849 bytes of HTML (244,347 characters). Its tokens:

--- tokens of the real page: {'start': 1973, 'end': 1930, 'text': 2132, 'comment': 8}

There are 43 more start tags than end tags. The page has 113 tags for void elements such as <meta>, <link> and <img>, which never have an end tag; 70 of them are written in the self-closing form and the other 43 are not. This is normal HTML, and it is the first reason a parser has to know the rules of the language and not only match brackets.

Step 2: the tree

The tree builder takes the tokens and nests them. We built the tree of the same page with the three parsers that BeautifulSoup 4.14.3 can use, parsed 20 times each, and counted what came out:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")      # or "html.parser", "html5lib"

parser       elements text nodes comments  links  h2 depth      ms
html.parser     1,973      2,133        8    498   7    28    27.4
lxml            1,973      2,132        8    498   7    27    21.8
html5lib        1,973      2,130        8    498   7    27    66.0
title: Web scraping - Wikipedia | h1: Web scraping | first h2: Contents

On this page, which is valid HTML, the three parsers agreed on everything that matters to a scraper: 1,973 elements, 498 links, 7 second-level headings, the same title. The parse time is the median of 20 parses on one Windows 11 PC. The same page in Node.js v22.23.2 with cheerio 1.2.0, which can use two parsers:

cheerio (parse5)       elements 1972 links 498 h2 7 ms 9.1 | title: Web scraping - Wikipedia
cheerio (htmlparser2)  elements 1973 links 498 h2 7 ms 2.4 | title: Web scraping - Wikipedia
ParserLanguageElementsLinksMedian parseFollows the HTML standard's repair rules
html.parser (standard library)Python1,97349827.4 msNo
lxml 6.1.1Python, C library underneath1,97349821.8 msPartly
html5lib 1.1Python1,97349866.0 msYes
parse5, in cheerio 1.2.0JavaScript1,9724989.1 msYes
htmlparser2, in cheerio 1.2.0JavaScript1,9734982.4 msNo

The last column is filled in from the next test, not from documentation. The timings are one run on one page: they show the order of magnitude, tens of milliseconds for a quarter of a megabyte, not a ranking to rely on for other pages. A different page, timed in its own run, is in web scraping with BeautifulSoup.

Step 3: broken markup, where parsers disagree

Much of the HTML on the web is not valid. The HTML standard therefore defines what a browser must build for every possible mistake, and parsers differ in how closely they follow it. We gave each parser the same four fragments and printed the tree back as HTML.

Two paragraphs, neither closed:

input        <p>one<p>two
  html.parser <p>one<p>two</p></p>
  lxml        <html><body><p>one</p><p>two</p></body></html>
  html5lib    <html><head></head><body><p>one</p><p>two</p></body></html>
  parse5      <html><head></head><body><p>one</p><p>two</p></body></html>
  htmlparser2 <p>one</p><p>two</p>

Tags closed in the wrong order:

input        <b>bold <i>both</b> italic</i>
  html.parser <b>bold <i>both</i></b> italic
  lxml        <html><body><b>bold <i>both</i></b> italic</body></html>
  html5lib    <html><head></head><body><b>bold <i>both</i></b><i> italic</i></body></html>
  parse5      <html><head></head><body><b>bold <i>both</i></b><i> italic</i></body></html>
  htmlparser2 <b>bold <i>both</i></b> italic

A table with a row, a cell and no closing tags for either:

input        <table><tr><td>cell</table>
  html.parser <table><tr><td>cell</td></tr></table>
  lxml        <html><body><table><tr><td>cell</td></tr></table></body></html>
  html5lib    <html><head></head><body><table><tbody><tr><td>cell</td></tr></tbody></table></body></html>
  parse5      <html><head></head><body><table><tbody><tr><td>cell</td></tr></tbody></table></body></html>
  htmlparser2 <table><tr><td>cell</td></tr></table>

Text with no tags around it:

input        just text &amp; <br> a break
  html.parser just text &amp; <br/> a break
  lxml        <html><body><p>just text &amp; <br/> a break</p></body></html>
  html5lib    <html><head></head><body>just text &amp; <br/> a break</body></html>
  parse5      <html><head></head><body>just text &amp; <br> a break</body></html>
  htmlparser2 just text &amp; <br> a break

What the four cases show:

  • html5lib and parse5 gave the same tree every time. Both implement the parsing algorithm of the HTML standard, the one browsers use: they add <html>, <head> and <body>, close the first paragraph when the second opens, reopen the italic after the bold ends, and insert the <tbody> a browser would insert.
  • html.parser and htmlparser2 add nothing. They return a fragment as a fragment. On the two paragraphs they differed from each other: html.parser nested the second inside the first, htmlparser2 made them siblings.
  • lxml sits in between. It wrapped everything in <html><body> and closed the paragraphs, did not add a <tbody>, and put the bare text inside a <p> of its own.

Why this matters when you scrape

Because a selector is a path through the tree, and a different tree means a different path. The same table fragment, the same two selectors, three Python parsers:

html.parser  select('table > tr'): 1   select('table > tbody > tr'): 0   select('tr'): 1
lxml         select('table > tr'): 1   select('table > tbody > tr'): 0   select('tr'): 1
html5lib     select('table > tr'): 0   select('table > tbody > tr'): 1   select('tr'): 1

This is the classic surprise: you copy table > tbody > tr from the browser's developer tools, where the tbody exists because the browser inserted it, and your script finds nothing because its parser did not. Two habits avoid it. Prefer selectors that do not depend on the exact chain of parents (tr matched once with all three parsers). And when a selector that works in the browser returns nothing in your code, check which parser you are using before blaming the site. Selector syntax itself is compared in CSS selectors vs XPath.

Why not a regular expression?

Because a regular expression matches characters, not structure. Two checks from the same run:

regex  <td>(.*?)</td>  -> []
parser select("td")    -> ['cell']
regex  href="(.*?)"    -> ['/a']
parser a[href]         -> ['/a', '/b', '/c']

The first pattern found nothing in <table><tr><td>cell</table> because the closing </td> it waits for is not in the text; the parser returned the cell. The second met three links written three legal ways (double quotes, no quotes, upper case with single quotes) and matched one; the parser read all three. A regular expression is the right tool after the parser, on a short string such as a price or a date.

Which HTML parser to use

NeedPythonNode.js
Nothing to installhtml.parserNone built in; cheerio is the usual choice
The same tree a browser buildshtml5libparse5 (the default in cheerio)
Short parse times on large pages, plus XPathlxml (21.8 ms on our page)htmlparser2 (2.4 ms on our page)

The tutorials for each are web scraping with BeautifulSoup, lxml and XPath in Python and web scraping with cheerio.

One limit applies to all five: a parser works on the HTML it is given. If a page builds its content with JavaScript after loading, that content is not in the downloaded text and no parser will find it; the page has to be rendered first, as in scraping dynamic pages with Python. Our Web Scraping API does both steps in one request: it returns the page as HTML or Markdown, rendered when the page needs it, or the fields you name with CSS selectors.

Sources & further reading

FAQ

Quick answers on html parser.

Something else? Ask us

What is an HTML parser?

A program that reads the text of an HTML page and builds a tree of elements from it. It splits the text into tokens, nests them, and repairs missing or misplaced tags, so that code can search the page by element, attribute or position.

What does an HTML parser do?

Three things: it tokenises the characters into tags, text and comments; it builds the tree; and it repairs invalid markup. On our test page of 244,347 characters, that meant 1,973 start tags turned into a tree of 1,973 elements with 498 links.

What is html.parser in BeautifulSoup?

The parser from Python's standard library, used when you write BeautifulSoup(html, "html.parser"). It needs no installation. In our test it built the same 1,973 elements as lxml and html5lib on a valid page, and it does not add missing html, body or tbody elements.

Which HTML parser is best for Python?

It depends on the need. html.parser is built in; lxml had the shortest median parse in our run (21.8 ms) and adds XPath; html5lib builds the same tree a browser does, which matters for broken markup. All three work through BeautifulSoup.

Why does my selector work in the browser but not in my script?

Often because the trees differ. Browsers insert elements such as tbody; in our test table > tbody > tr matched with html5lib and matched nothing with html.parser and lxml. The other common cause is content added by JavaScript, which is not in the downloaded HTML.

Can I parse HTML with a regular expression?

Not reliably. In our test a pattern for table cells found nothing in a table with omitted closing tags, and a pattern for href="..." matched one of three validly written links. A parser returned all of them.

Fetch the page, then parse it

Residential proxies to fetch public pages from the country you choose, and a Web Scraping API that returns HTML, Markdown or the fields you select.

Related reading