# HTML Parser: What It Is and How It Works

> An HTML parser turns the text of a page into a tree of elements. We ran five parsers on one real page: 1,973 elements, timings, and how each fixes broken HTML.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / HTML Parser: What It Is and How It Works

# What is an HTML parser? Tokens, the tree and broken markup, on a real page

Web scraping · Nov 20, 2026 · 9 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![An HTML parser at work on 20 November 2026: one real page of 244,347 characters became a tree of 1,973 elements; the code window shows a table written without closing tags and the path each Python parser built for its row, table then tr for html.parser and lxml, table then tbody then tr for html5lib; the card lists the three jobs of a parser.](https://quanticdata.io/blog/html-parser/hero.webp?v=20261120h)

On this page

- [What an HTML parser does, in three steps](https://quanticdata.io/blog/html-parser/#what-an-html-parser-does-in-three-steps)
- [Step 1: the tokens](https://quanticdata.io/blog/html-parser/#step-1-the-tokens)
- [Step 2: the tree](https://quanticdata.io/blog/html-parser/#step-2-the-tree)
- [Step 3: broken markup, where parsers disagree](https://quanticdata.io/blog/html-parser/#step-3-broken-markup-where-parsers-disagree)
- [Why this matters when you scrape](https://quanticdata.io/blog/html-parser/#why-this-matters-when-you-scrape)
- [Why not a regular expression?](https://quanticdata.io/blog/html-parser/#why-not-a-regular-expression)
- [Which HTML parser to use](https://quanticdata.io/blog/html-parser/#which-html-parser-to-use)

An HTML parser is a program that reads the text of a web page and turns it into a tree of elements that code can search and change. It splits the text into tokens (tags, attributes, text, comments), nests them into a tree, and repairs markup that is incomplete or wrong. Browsers, search engine crawlers and scrapers all start with one.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=html-parser) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

The definition is short; the behaviour is where parsers differ. On 20 November 2026 we fetched one real page and ran five parsers over it, three in Python and two in Node.js, then gave each of them the same pieces of broken HTML. The output below is from that run.

## What an HTML parser does, in three steps

| Step | Input | Output | On our test page |
| --- | --- | --- | --- |
| 1. Tokenise | Characters | Start tags, end tags, text, comments | 1,973 start tags, 1,930 end tags, 8 comments |
| 2. Build the tree | Tokens | Nested elements and text nodes (the DOM) | 1,973 elements, up to 28 levels deep |
| 3. Repair | Missing or misplaced tags | A complete, well-nested tree | 43 tags never closed in the source, all closed in the tree |

After that, your code asks the tree questions: "every link", "the first h1", "the price inside this card". That selecting step is not the parser's job, but it only works because the parser built the tree first. Turning the selected pieces into typed records is the wider subject of [data parsing](https://quanticdata.io/blog/data-parsing/).

## Step 1: the tokens

Python ships a tokeniser in its standard library, `html.parser.HTMLParser`. It calls a method for every token it meets, so a few lines are enough to see them:

```python
from html.parser import HTMLParser

class Tok(HTMLParser):
    def handle_starttag(self, tag, attrs): print("  start  ", tag, dict(attrs) if attrs else "")
    def handle_endtag(self, tag): print("  end    ", tag)
    def handle_data(self, data):
        if data.strip(): print("  text   ", repr(data))
    def handle_comment(self, data): print("  comment", repr(data))

Tok().feed('<p class="lead">Hello <a href="/docs">docs</a><!-- note --></p>')

  start   p {'class': 'lead'}
  text    'Hello '
  start   a {'href': '/docs'}
  text    'docs'
  end     a
  comment ' note '
  end     p
```

Seven tokens for one short paragraph. The tokeniser has already done useful work: it separated the attribute name from its value and removed the quotes. It has not built anything yet: nothing here says that the link is inside the paragraph.

Then the real page. We fetched the English Wikipedia article on web scraping through a QuanticData [residential proxy](https://quanticdata.io/residential-proxies/) line in the United States, with curl and our own User-Agent, and got status 200 and 244,849 bytes of HTML (244,347 characters). Its tokens:

```
--- tokens of the real page: {'start': 1973, 'end': 1930, 'text': 2132, 'comment': 8}
```

There are 43 more start tags than end tags. The page has 113 tags for void elements such as `<meta>`, `<link>` and `<img>`, which never have an end tag; 70 of them are written in the self-closing form and the other 43 are not. This is normal HTML, and it is the first reason a parser has to know the rules of the language and not only match brackets.

## Step 2: the tree

The tree builder takes the tokens and nests them. We built the tree of the same page with the three parsers that BeautifulSoup 4.14.3 can use, parsed 20 times each, and counted what came out:

```python
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")      # or "html.parser", "html5lib"

parser       elements text nodes comments  links  h2 depth      ms
html.parser     1,973      2,133        8    498   7    28    27.4
lxml            1,973      2,132        8    498   7    27    21.8
html5lib        1,973      2,130        8    498   7    27    66.0
title: Web scraping - Wikipedia | h1: Web scraping | first h2: Contents
```

On this page, which is valid HTML, the three parsers agreed on everything that matters to a scraper: 1,973 elements, 498 links, 7 second-level headings, the same title. The parse time is the median of 20 parses on one Windows 11 PC. The same page in Node.js v22.23.2 with cheerio 1.2.0, which can use two parsers:

```
cheerio (parse5)       elements 1972 links 498 h2 7 ms 9.1 | title: Web scraping - Wikipedia
cheerio (htmlparser2)  elements 1973 links 498 h2 7 ms 2.4 | title: Web scraping - Wikipedia
```

| Parser | Language | Elements | Links | Median parse | Follows the HTML standard's repair rules |
| --- | --- | --- | --- | --- | --- |
| html.parser (standard library) | Python | 1,973 | 498 | 27.4 ms | No |
| lxml 6.1.1 | Python, C library underneath | 1,973 | 498 | 21.8 ms | Partly |
| html5lib 1.1 | Python | 1,973 | 498 | 66.0 ms | Yes |
| parse5, in cheerio 1.2.0 | JavaScript | 1,972 | 498 | 9.1 ms | Yes |
| htmlparser2, in cheerio 1.2.0 | JavaScript | 1,973 | 498 | 2.4 ms | No |

The last column is filled in from the next test, not from documentation. The timings are one run on one page: they show the order of magnitude, tens of milliseconds for a quarter of a megabyte, not a ranking to rely on for other pages. A different page, timed in its own run, is in [web scraping with BeautifulSoup](https://quanticdata.io/blog/beautifulsoup-web-scraping/).

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=html-parser) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Step 3: broken markup, where parsers disagree

Much of the HTML on the web is not valid. The HTML standard therefore defines what a browser must build for every possible mistake, and parsers differ in how closely they follow it. We gave each parser the same four fragments and printed the tree back as HTML.

Two paragraphs, neither closed:

```
input        <p>one<p>two
  html.parser <p>one<p>two</p></p>
  lxml        <html><body><p>one</p><p>two</p></body></html>
  html5lib    <html><head></head><body><p>one</p><p>two</p></body></html>
  parse5      <html><head></head><body><p>one</p><p>two</p></body></html>
  htmlparser2 <p>one</p><p>two</p>
```

Tags closed in the wrong order:

```
input        <b>bold <i>both</b> italic</i>
  html.parser <b>bold <i>both</i></b> italic
  lxml        <html><body><b>bold <i>both</i></b> italic</body></html>
  html5lib    <html><head></head><body><b>bold <i>both</i></b><i> italic</i></body></html>
  parse5      <html><head></head><body><b>bold <i>both</i></b><i> italic</i></body></html>
  htmlparser2 <b>bold <i>both</i></b> italic
```

A table with a row, a cell and no closing tags for either:

```
input        <table><tr><td>cell</table>
  html.parser <table><tr><td>cell</td></tr></table>
  lxml        <html><body><table><tr><td>cell</td></tr></table></body></html>
  html5lib    <html><head></head><body><table><tbody><tr><td>cell</td></tr></tbody></table></body></html>
  parse5      <html><head></head><body><table><tbody><tr><td>cell</td></tr></tbody></table></body></html>
  htmlparser2 <table><tr><td>cell</td></tr></table>
```

Text with no tags around it:

```
input        just text &amp; <br> a break
  html.parser just text &amp; <br/> a break
  lxml        <html><body><p>just text &amp; <br/> a break</p></body></html>
  html5lib    <html><head></head><body>just text &amp; <br/> a break</body></html>
  parse5      <html><head></head><body>just text &amp; <br> a break</body></html>
  htmlparser2 just text &amp; <br> a break
```

What the four cases show:

- **html5lib and parse5 gave the same tree every time.** Both implement the parsing algorithm of the HTML standard, the one browsers use: they add `<html>`, `<head>` and `<body>`, close the first paragraph when the second opens, reopen the italic after the bold ends, and insert the `<tbody>` a browser would insert.
- **html.parser and htmlparser2 add nothing.** They return a fragment as a fragment. On the two paragraphs they differed from each other: html.parser nested the second inside the first, htmlparser2 made them siblings.
- **lxml sits in between.** It wrapped everything in `<html><body>` and closed the paragraphs, did not add a `<tbody>`, and put the bare text inside a `<p>` of its own.

## Why this matters when you scrape

Because a selector is a path through the tree, and a different tree means a different path. The same table fragment, the same two selectors, three Python parsers:

```
html.parser  select('table > tr'): 1   select('table > tbody > tr'): 0   select('tr'): 1
lxml         select('table > tr'): 1   select('table > tbody > tr'): 0   select('tr'): 1
html5lib     select('table > tr'): 0   select('table > tbody > tr'): 1   select('tr'): 1
```

This is the classic surprise: you copy `table > tbody > tr` from the browser's developer tools, where the `tbody` exists because the browser inserted it, and your script finds nothing because its parser did not. Two habits avoid it. Prefer selectors that do not depend on the exact chain of parents (`tr` matched once with all three parsers). And when a selector that works in the browser returns nothing in your code, check which parser you are using before blaming the site. Selector syntax itself is compared in [CSS selectors vs XPath](https://quanticdata.io/blog/css-selector-vs-xpath/).

## Why not a regular expression?

Because a regular expression matches characters, not structure. Two checks from the same run:

```
regex  <td>(.*?)</td>  -> []
parser select("td")    -> ['cell']
regex  href="(.*?)"    -> ['/a']
parser a[href]         -> ['/a', '/b', '/c']
```

The first pattern found nothing in `<table><tr><td>cell</table>` because the closing `</td>` it waits for is not in the text; the parser returned the cell. The second met three links written three legal ways (double quotes, no quotes, upper case with single quotes) and matched one; the parser read all three. A regular expression is the right tool after the parser, on a short string such as a price or a date.

## Which HTML parser to use

| Need | Python | Node.js |
| --- | --- | --- |
| Nothing to install | html.parser | None built in; cheerio is the usual choice |
| The same tree a browser builds | html5lib | parse5 (the default in cheerio) |
| Short parse times on large pages, plus XPath | lxml (21.8 ms on our page) | htmlparser2 (2.4 ms on our page) |

The tutorials for each are [web scraping with BeautifulSoup](https://quanticdata.io/blog/beautifulsoup-web-scraping/), [lxml and XPath in Python](https://quanticdata.io/blog/lxml-xpath-python/) and [web scraping with cheerio](https://quanticdata.io/blog/cheerio-web-scraping/).

One limit applies to all five: a parser works on the HTML it is given. If a page builds its content with JavaScript after loading, that content is not in the downloaded text and no parser will find it; the page has to be rendered first, as in [scraping dynamic pages with Python](https://quanticdata.io/blog/dynamic-web-scraping-python/). Our [Web Scraping API](https://quanticdata.io/web-scraping-api/) does both steps in one request: it returns the page as HTML or Markdown, rendered when the page needs it, or the fields you name with CSS selectors.

### Sources & further reading

- [HTML Living Standard: parsing HTML documents](https://html.spec.whatwg.org/multipage/parsing.html)
- [Python documentation: html.parser](https://docs.python.org/3/library/html.parser.html)
- [Beautiful Soup documentation: differences between parsers](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#differences-between-parsers)
- [MDN: Document Object Model (DOM)](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model)
- [cheerio documentation: loading documents](https://cheerio.js.org/docs/basics/loading)

## FAQ

Quick answers on html parser.

[Something else? Ask us](mailto:hello@quanticdata.io)

### What is an HTML parser?

A program that reads the text of an HTML page and builds a tree of elements from it. It splits the text into tokens, nests them, and repairs missing or misplaced tags, so that code can search the page by element, attribute or position.

### What does an HTML parser do?

Three things: it tokenises the characters into tags, text and comments; it builds the tree; and it repairs invalid markup. On our test page of 244,347 characters, that meant 1,973 start tags turned into a tree of 1,973 elements with 498 links.

### What is html.parser in BeautifulSoup?

The parser from Python's standard library, used when you write `BeautifulSoup(html, "html.parser")`. It needs no installation. In our test it built the same 1,973 elements as lxml and html5lib on a valid page, and it does not add missing `html`, `body` or `tbody` elements.

### Which HTML parser is best for Python?

It depends on the need. html.parser is built in; lxml had the shortest median parse in our run (21.8 ms) and adds XPath; html5lib builds the same tree a browser does, which matters for broken markup. All three work through BeautifulSoup.

### Why does my selector work in the browser but not in my script?

Often because the trees differ. Browsers insert elements such as `tbody`; in our test `table > tbody > tr` matched with html5lib and matched nothing with html.parser and lxml. The other common cause is content added by JavaScript, which is not in the downloaded HTML.

### Can I parse HTML with a regular expression?

Not reliably. In our test a pattern for table cells found nothing in a table with omitted closing tags, and a pattern for `href="..."` matched one of three validly written links. A parser returned all of them.

## Fetch the page, then parse it

Residential proxies to fetch public pages from the country you choose, and a Web Scraping API that returns HTML, Markdown or the fields you select.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=html-parser) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

- [Web Scraping Project Ideas, Started for Real](https://quanticdata.io/blog/web-scraping-project-ideas/): Web scraping · Five web scraping project ideas, each one started for real on 20 November 2026: a price tracker, a library release dashboard, a question analysis, an open-source landscape and a topic timeline, with the code, the first run and what to build next.
- [Retry-After Header: Read It, Wait, Retry](https://quanticdata.io/blog/retry-after-header/): Web scraping · Retry-After tells a client how long to wait before the next request, as a number of seconds or as an HTTP date. Below: both formats, a Python and a Node helper we ran against a real 429 on 20 November 2026 (each waited 10 seconds and then got 200), the rate-limit headers of the QuanticData API, and how one batch job replaced 20 calls.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/html-parser/ · Site index for AI: https://quanticdata.io/llms.txt
