Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

lxml XPath in Python: the expressions that scrape real pages, tested

lxml XPath on a product page in Python: doc.xpath with //article[@class="product_pod"]/h3/a/@title returns 20 book titles, contains(@class, "star-rating") finds 20 ratings where the exact match finds 0, and ancestor::article climbs from a price to its book; the card shows timings of 25 microseconds for a compiled XPath and 61 for cssselect.
lxml XPath on a product page in Python: doc.xpath with //article[@class="product_pod"]/h3/a/@title returns 20 book titles, contains(@class, "star-rating") finds 20 ratings where the exact match finds 0, and ancestor::article climbs from a price to its book; the card shows timings of 25 microseconds for a compiled XPath and 61 for cssselect.

To use XPath with lxml in Python, parse the page with lxml.html.fromstring(r.content), then call doc.xpath("expression"). It returns a list: of elements for //article, of strings for //a/@href or //p/text(). Install it with pip install lxml.

Each expression in this tutorial ran on 20 November 2026 with Python 3.12.10 and lxml 6.1.1, on a catalogue page of books.toscrape.com (a public shop made for scraping practice) downloaded through a QuanticData mobile proxy in the US. The comments show what it returned.

Parse the page

import requests, lxml.html

PROXY = "http://USER-country-us-type-mobile-sid-x01-ttl-30m:PASS@mobile.quanticdata.io:8080"
r = requests.get("https://books.toscrape.com/catalogue/page-1.html",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)
doc = lxml.html.fromstring(r.content)

Pass r.content (bytes), not r.text: lxml then reads the page's own charset, and the £ in prices stays a £. For XML documents such as sitemaps and feeds use lxml.etree.fromstring instead; that case, with namespaces, is in parsing XML in Python.

The XPath expressions you will use most

GoalExpressionOur result
An attribute of every match//article[@class="product_pod"]/h3/a/@title20 titles, first "A Light in the Attic"
Text of elements//p[@class="price_color"]/text()20 prices, first "£51.77"
A class among several//p[contains(@class, "star-rating")]20 (the exact @class="star-rating" found 0)
Text with spaces trimmednormalize-space(//p[contains(@class, "instock")])"In stock"
Link by its text//a[contains(text(), "Travel")]/@hrefcategory/books/travel_2/index.html
Climb to a parent block//p[@class="price_color"][text()="£51.77"]/ancestor::article//h3/a/@titleA Light in the Attic
The last match(//article[@class="product_pod"])[last()]//h3/a/@titleIt's Only the Himalayas
Countcount(//article)20.0 (a float)
Next page link//li[@class="next"]/a/@href['page-2.html']

contains(): the fix for classes and messy text

The rating paragraph on this page has two classes, star-rating Three. @class="star-rating" compares the whole attribute string, so it matched nothing; contains(@class, "star-rating") matched all 20. The same goes for text with spaces and line breaks around it: contains(text(), "Travel") found the link where an exact text match would miss. One caution: contains(@class, "star") would also match a class called starred. For an exact class among several, use contains(concat(" ", normalize-space(@class), " "), " star-rating "), or a CSS selector, which handles classes natively.

Loop over blocks with relative paths

To keep each title with its own price, select the blocks first and run XPath from each one. Start the inner expressions with . so they search inside the block, not the whole page:

rows = []
for pod in doc.xpath('//article[@class="product_pod"]'):
    rows.append({
        "title": pod.xpath("./h3/a/@title")[0],
        "price": pod.xpath('.//p[@class="price_color"]/text()')[0],
        "stars": pod.xpath('.//p[contains(@class, "star-rating")]/@class')[0].split()[1],
    })
print(len(rows), rows[0])
# 20 {'title': 'A Light in the Attic', 'price': '£51.77', 'stars': 'Three'}

Without the dot, pod.xpath("//h3/a/@title") returns all 20 titles from every block, the most common lxml surprise. Since xpath always returns a list, [0] raises IndexError when nothing matches; use next(iter(result), None) for optional fields.

Axes: siblings, parents and the brackets trap

XPath can move in any direction from an element, which CSS selectors cannot. From the first price on the page:

ExpressionOur result
(//p[@class="price_color"])[1]/following-sibling::p[1]/@class['instock availability']
(//p[@class="price_color"])[1]/../@class['product_price'], the parent
(//div[@class="product_price"])[1]/preceding-sibling::h3/a/@title['A Light in the Attic']
//a[starts-with(@href, "category/")]51 links
//p[contains(@class, "star-rating") and (contains(@class, "Five") or contains(@class, "One"))]10 books rated five or one

Mind the brackets. We asked for the first two books with //article[@class="product_pod"][position() <= 2] and got all 20 titles back: on this page every article sits alone inside its own list item, so each one is "position 1" among its siblings. Wrapping the path in parentheses, (//article[@class="product_pod"])[position() <= 2], counts across the whole document and returns two. The same rule explains (//p[@class="price_color"])[1] above.

Faster: compile the expression

We timed each way of running the same query (20 links) 500 times on the parsed page:

MethodMedian per query
etree.XPath(expr), compiled once25 microseconds
doc.xpath(expr)31 microseconds
CSSSelector(css), compiled once33 microseconds
doc.cssselect(css)61 microseconds
from lxml import etree
titles = etree.XPath('//article[@class="product_pod"]/h3/a')
for page_doc in docs:
    links = titles(page_doc)

All of these are microseconds; downloading the page takes far longer. Compile when the same expression runs over thousands of pages.

CSS selectors in lxml

With pip install cssselect, lxml also takes CSS: doc.cssselect("article.product_pod h3 a") found the same 20 links. It works by translating CSS into XPath; CSSSelector("article.product_pod h3 a").path shows the result, which starts with descendant-or-self::article[@class and contains(@class, .... That is why compiled CSS ran close to XPath and uncompiled CSS slower. When to pick which is covered in CSS selector vs XPath.

lxml vs Beautiful Soup

On the same page, lxml with XPath extracted the 20 books in 1.81 ms; Beautiful Soup with the lxml parser took 12.91 ms. Beautiful Soup has the friendlier API and repairs broken HTML more forgivingly; lxml is faster and gives you XPath's axes (parent, ancestor, following-sibling) in one expression. Both read only the HTML the server sends. The Beautiful Soup side: Beautiful Soup web scraping; all the options compared: Python web scraping libraries.

Where the page comes from

lxml parses; the download decides what there is to parse. If a site sends a block page to scripts, your XPath returns empty lists and the bug looks like a selector problem. Print len(r.content) and the title (doc.findtext(".//title")) before debugging expressions. We fetched through QuanticData mobile proxies; residential proxies work the same way, with the country and session in the username. There is a Residential Basic free trial to try it.

Sources & further reading

FAQ

Quick answers on lxml xpath.

Something else? Ask us

How do I use XPath with lxml in Python?

Parse with doc = lxml.html.fromstring(r.content), then call doc.xpath("//tag[@attr='value']"). Add /@attr or /text() to get strings instead of elements. The result is always a list.

How do I select an element by class with XPath?

Use contains(@class, "name"). An exact @class="name" fails when the element has more than one class: on our page it found 0 ratings where contains() found 20.

Why does my XPath inside a loop return results from the whole page?

An expression that starts with // searches the whole document even when called on a child element. Start it with .// or ./ to search inside that element only.

Is lxml faster than Beautiful Soup?

Yes. On our test page, lxml with XPath extracted 20 books in 1.81 ms against 12.91 ms for Beautiful Soup with the lxml parser. A compiled XPath query took 25 microseconds.

Can lxml use CSS selectors?

Yes, with the cssselect package: doc.cssselect("article.product_pod h3 a"). It translates the CSS into XPath; compiled with CSSSelector it ran in 33 microseconds on our page.

Give your XPath a real page to read

Mobile and residential proxies for requests and httpx, with country and session settings in the username.

Collectors with Python examples

Related reading