An HTML parser is a program that reads the text of a web page and turns it into a tree of elements that code can search and change. It splits the text into tokens (tags, attributes, text, comments), nests them into a tree, and repairs markup that is incomplete or wrong. Browsers, search engine crawlers and scrapers all start with one.
The definition is short; the behaviour is where parsers differ. On 20 November 2026 we fetched one real page and ran five parsers over it, three in Python and two in Node.js, then gave each of them the same pieces of broken HTML. The output below is from that run.
What an HTML parser does, in three steps
| Step | Input | Output | On our test page |
|---|---|---|---|
| 1. Tokenise | Characters | Start tags, end tags, text, comments | 1,973 start tags, 1,930 end tags, 8 comments |
| 2. Build the tree | Tokens | Nested elements and text nodes (the DOM) | 1,973 elements, up to 28 levels deep |
| 3. Repair | Missing or misplaced tags | A complete, well-nested tree | 43 tags never closed in the source, all closed in the tree |
After that, your code asks the tree questions: "every link", "the first h1", "the price inside this card". That selecting step is not the parser's job, but it only works because the parser built the tree first. Turning the selected pieces into typed records is the wider subject of data parsing.
Step 1: the tokens
Python ships a tokeniser in its standard library, html.parser.HTMLParser. It calls a method for every token it meets, so a few lines are enough to see them:
from html.parser import HTMLParser
class Tok(HTMLParser):
def handle_starttag(self, tag, attrs): print(" start ", tag, dict(attrs) if attrs else "")
def handle_endtag(self, tag): print(" end ", tag)
def handle_data(self, data):
if data.strip(): print(" text ", repr(data))
def handle_comment(self, data): print(" comment", repr(data))
Tok().feed('<p class="lead">Hello <a href="/docs">docs</a><!-- note --></p>')
start p {'class': 'lead'}
text 'Hello '
start a {'href': '/docs'}
text 'docs'
end a
comment ' note '
end p
Seven tokens for one short paragraph. The tokeniser has already done useful work: it separated the attribute name from its value and removed the quotes. It has not built anything yet: nothing here says that the link is inside the paragraph.
Then the real page. We fetched the English Wikipedia article on web scraping through a QuanticData residential proxy line in the United States, with curl and our own User-Agent, and got status 200 and 244,849 bytes of HTML (244,347 characters). Its tokens:
--- tokens of the real page: {'start': 1973, 'end': 1930, 'text': 2132, 'comment': 8}
There are 43 more start tags than end tags. The page has 113 tags for void elements such as <meta>, <link> and <img>, which never have an end tag; 70 of them are written in the self-closing form and the other 43 are not. This is normal HTML, and it is the first reason a parser has to know the rules of the language and not only match brackets.
Step 2: the tree
The tree builder takes the tokens and nests them. We built the tree of the same page with the three parsers that BeautifulSoup 4.14.3 can use, parsed 20 times each, and counted what came out:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml") # or "html.parser", "html5lib"
parser elements text nodes comments links h2 depth ms
html.parser 1,973 2,133 8 498 7 28 27.4
lxml 1,973 2,132 8 498 7 27 21.8
html5lib 1,973 2,130 8 498 7 27 66.0
title: Web scraping - Wikipedia | h1: Web scraping | first h2: Contents
On this page, which is valid HTML, the three parsers agreed on everything that matters to a scraper: 1,973 elements, 498 links, 7 second-level headings, the same title. The parse time is the median of 20 parses on one Windows 11 PC. The same page in Node.js v22.23.2 with cheerio 1.2.0, which can use two parsers:
cheerio (parse5) elements 1972 links 498 h2 7 ms 9.1 | title: Web scraping - Wikipedia
cheerio (htmlparser2) elements 1973 links 498 h2 7 ms 2.4 | title: Web scraping - Wikipedia
| Parser | Language | Elements | Links | Median parse | Follows the HTML standard's repair rules |
|---|---|---|---|---|---|
| html.parser (standard library) | Python | 1,973 | 498 | 27.4 ms | No |
| lxml 6.1.1 | Python, C library underneath | 1,973 | 498 | 21.8 ms | Partly |
| html5lib 1.1 | Python | 1,973 | 498 | 66.0 ms | Yes |
| parse5, in cheerio 1.2.0 | JavaScript | 1,972 | 498 | 9.1 ms | Yes |
| htmlparser2, in cheerio 1.2.0 | JavaScript | 1,973 | 498 | 2.4 ms | No |
The last column is filled in from the next test, not from documentation. The timings are one run on one page: they show the order of magnitude, tens of milliseconds for a quarter of a megabyte, not a ranking to rely on for other pages. A different page, timed in its own run, is in web scraping with BeautifulSoup.
Step 3: broken markup, where parsers disagree
Much of the HTML on the web is not valid. The HTML standard therefore defines what a browser must build for every possible mistake, and parsers differ in how closely they follow it. We gave each parser the same four fragments and printed the tree back as HTML.
Two paragraphs, neither closed:
input <p>one<p>two
html.parser <p>one<p>two</p></p>
lxml <html><body><p>one</p><p>two</p></body></html>
html5lib <html><head></head><body><p>one</p><p>two</p></body></html>
parse5 <html><head></head><body><p>one</p><p>two</p></body></html>
htmlparser2 <p>one</p><p>two</p>
Tags closed in the wrong order:
input <b>bold <i>both</b> italic</i>
html.parser <b>bold <i>both</i></b> italic
lxml <html><body><b>bold <i>both</i></b> italic</body></html>
html5lib <html><head></head><body><b>bold <i>both</i></b><i> italic</i></body></html>
parse5 <html><head></head><body><b>bold <i>both</i></b><i> italic</i></body></html>
htmlparser2 <b>bold <i>both</i></b> italic
A table with a row, a cell and no closing tags for either:
input <table><tr><td>cell</table>
html.parser <table><tr><td>cell</td></tr></table>
lxml <html><body><table><tr><td>cell</td></tr></table></body></html>
html5lib <html><head></head><body><table><tbody><tr><td>cell</td></tr></tbody></table></body></html>
parse5 <html><head></head><body><table><tbody><tr><td>cell</td></tr></tbody></table></body></html>
htmlparser2 <table><tr><td>cell</td></tr></table>
Text with no tags around it:
input just text & <br> a break
html.parser just text & <br/> a break
lxml <html><body><p>just text & <br/> a break</p></body></html>
html5lib <html><head></head><body>just text & <br/> a break</body></html>
parse5 <html><head></head><body>just text & <br> a break</body></html>
htmlparser2 just text & <br> a break
What the four cases show:
- html5lib and parse5 gave the same tree every time. Both implement the parsing algorithm of the HTML standard, the one browsers use: they add
<html>,<head>and<body>, close the first paragraph when the second opens, reopen the italic after the bold ends, and insert the<tbody>a browser would insert. - html.parser and htmlparser2 add nothing. They return a fragment as a fragment. On the two paragraphs they differed from each other: html.parser nested the second inside the first, htmlparser2 made them siblings.
- lxml sits in between. It wrapped everything in
<html><body>and closed the paragraphs, did not add a<tbody>, and put the bare text inside a<p>of its own.
Why this matters when you scrape
Because a selector is a path through the tree, and a different tree means a different path. The same table fragment, the same two selectors, three Python parsers:
html.parser select('table > tr'): 1 select('table > tbody > tr'): 0 select('tr'): 1
lxml select('table > tr'): 1 select('table > tbody > tr'): 0 select('tr'): 1
html5lib select('table > tr'): 0 select('table > tbody > tr'): 1 select('tr'): 1
This is the classic surprise: you copy table > tbody > tr from the browser's developer tools, where the tbody exists because the browser inserted it, and your script finds nothing because its parser did not. Two habits avoid it. Prefer selectors that do not depend on the exact chain of parents (tr matched once with all three parsers). And when a selector that works in the browser returns nothing in your code, check which parser you are using before blaming the site. Selector syntax itself is compared in CSS selectors vs XPath.
Why not a regular expression?
Because a regular expression matches characters, not structure. Two checks from the same run:
regex <td>(.*?)</td> -> []
parser select("td") -> ['cell']
regex href="(.*?)" -> ['/a']
parser a[href] -> ['/a', '/b', '/c']
The first pattern found nothing in <table><tr><td>cell</table> because the closing </td> it waits for is not in the text; the parser returned the cell. The second met three links written three legal ways (double quotes, no quotes, upper case with single quotes) and matched one; the parser read all three. A regular expression is the right tool after the parser, on a short string such as a price or a date.
Which HTML parser to use
| Need | Python | Node.js |
|---|---|---|
| Nothing to install | html.parser | None built in; cheerio is the usual choice |
| The same tree a browser builds | html5lib | parse5 (the default in cheerio) |
| Short parse times on large pages, plus XPath | lxml (21.8 ms on our page) | htmlparser2 (2.4 ms on our page) |
The tutorials for each are web scraping with BeautifulSoup, lxml and XPath in Python and web scraping with cheerio.
One limit applies to all five: a parser works on the HTML it is given. If a page builds its content with JavaScript after loading, that content is not in the downloaded text and no parser will find it; the page has to be rendered first, as in scraping dynamic pages with Python. Our Web Scraping API does both steps in one request: it returns the page as HTML or Markdown, rendered when the page needs it, or the fields you name with CSS selectors.


