To use XPath with lxml in Python, parse the page with lxml.html.fromstring(r.content), then call doc.xpath("expression"). It returns a list: of elements for //article, of strings for //a/@href or //p/text(). Install it with pip install lxml.
Each expression in this tutorial ran on 20 November 2026 with Python 3.12.10 and lxml 6.1.1, on a catalogue page of books.toscrape.com (a public shop made for scraping practice) downloaded through a QuanticData mobile proxy in the US. The comments show what it returned.
Parse the page
import requests, lxml.html
PROXY = "http://USER-country-us-type-mobile-sid-x01-ttl-30m:PASS@mobile.quanticdata.io:8080"
r = requests.get("https://books.toscrape.com/catalogue/page-1.html",
proxies={"http": PROXY, "https": PROXY}, timeout=30)
doc = lxml.html.fromstring(r.content)
Pass r.content (bytes), not r.text: lxml then reads the page's own charset, and the £ in prices stays a £. For XML documents such as sitemaps and feeds use lxml.etree.fromstring instead; that case, with namespaces, is in parsing XML in Python.
The XPath expressions you will use most
| Goal | Expression | Our result |
|---|---|---|
| An attribute of every match | //article[@class="product_pod"]/h3/a/@title | 20 titles, first "A Light in the Attic" |
| Text of elements | //p[@class="price_color"]/text() | 20 prices, first "£51.77" |
| A class among several | //p[contains(@class, "star-rating")] | 20 (the exact @class="star-rating" found 0) |
| Text with spaces trimmed | normalize-space(//p[contains(@class, "instock")]) | "In stock" |
| Link by its text | //a[contains(text(), "Travel")]/@href | category/books/travel_2/index.html |
| Climb to a parent block | //p[@class="price_color"][text()="£51.77"]/ancestor::article//h3/a/@title | A Light in the Attic |
| The last match | (//article[@class="product_pod"])[last()]//h3/a/@title | It's Only the Himalayas |
| Count | count(//article) | 20.0 (a float) |
| Next page link | //li[@class="next"]/a/@href | ['page-2.html'] |
contains(): the fix for classes and messy text
The rating paragraph on this page has two classes, star-rating Three. @class="star-rating" compares the whole attribute string, so it matched nothing; contains(@class, "star-rating") matched all 20. The same goes for text with spaces and line breaks around it: contains(text(), "Travel") found the link where an exact text match would miss. One caution: contains(@class, "star") would also match a class called starred. For an exact class among several, use contains(concat(" ", normalize-space(@class), " "), " star-rating "), or a CSS selector, which handles classes natively.
Loop over blocks with relative paths
To keep each title with its own price, select the blocks first and run XPath from each one. Start the inner expressions with . so they search inside the block, not the whole page:
rows = []
for pod in doc.xpath('//article[@class="product_pod"]'):
rows.append({
"title": pod.xpath("./h3/a/@title")[0],
"price": pod.xpath('.//p[@class="price_color"]/text()')[0],
"stars": pod.xpath('.//p[contains(@class, "star-rating")]/@class')[0].split()[1],
})
print(len(rows), rows[0])
# 20 {'title': 'A Light in the Attic', 'price': '£51.77', 'stars': 'Three'}
Without the dot, pod.xpath("//h3/a/@title") returns all 20 titles from every block, the most common lxml surprise. Since xpath always returns a list, [0] raises IndexError when nothing matches; use next(iter(result), None) for optional fields.
Axes: siblings, parents and the brackets trap
XPath can move in any direction from an element, which CSS selectors cannot. From the first price on the page:
| Expression | Our result |
|---|---|
(//p[@class="price_color"])[1]/following-sibling::p[1]/@class | ['instock availability'] |
(//p[@class="price_color"])[1]/../@class | ['product_price'], the parent |
(//div[@class="product_price"])[1]/preceding-sibling::h3/a/@title | ['A Light in the Attic'] |
//a[starts-with(@href, "category/")] | 51 links |
//p[contains(@class, "star-rating") and (contains(@class, "Five") or contains(@class, "One"))] | 10 books rated five or one |
Mind the brackets. We asked for the first two books with //article[@class="product_pod"][position() <= 2] and got all 20 titles back: on this page every article sits alone inside its own list item, so each one is "position 1" among its siblings. Wrapping the path in parentheses, (//article[@class="product_pod"])[position() <= 2], counts across the whole document and returns two. The same rule explains (//p[@class="price_color"])[1] above.
Faster: compile the expression
We timed each way of running the same query (20 links) 500 times on the parsed page:
| Method | Median per query |
|---|---|
etree.XPath(expr), compiled once | 25 microseconds |
doc.xpath(expr) | 31 microseconds |
CSSSelector(css), compiled once | 33 microseconds |
doc.cssselect(css) | 61 microseconds |
from lxml import etree
titles = etree.XPath('//article[@class="product_pod"]/h3/a')
for page_doc in docs:
links = titles(page_doc)
All of these are microseconds; downloading the page takes far longer. Compile when the same expression runs over thousands of pages.
CSS selectors in lxml
With pip install cssselect, lxml also takes CSS: doc.cssselect("article.product_pod h3 a") found the same 20 links. It works by translating CSS into XPath; CSSSelector("article.product_pod h3 a").path shows the result, which starts with descendant-or-self::article[@class and contains(@class, .... That is why compiled CSS ran close to XPath and uncompiled CSS slower. When to pick which is covered in CSS selector vs XPath.
lxml vs Beautiful Soup
On the same page, lxml with XPath extracted the 20 books in 1.81 ms; Beautiful Soup with the lxml parser took 12.91 ms. Beautiful Soup has the friendlier API and repairs broken HTML more forgivingly; lxml is faster and gives you XPath's axes (parent, ancestor, following-sibling) in one expression. Both read only the HTML the server sends. The Beautiful Soup side: Beautiful Soup web scraping; all the options compared: Python web scraping libraries.
Where the page comes from
lxml parses; the download decides what there is to parse. If a site sends a block page to scripts, your XPath returns empty lists and the bug looks like a selector problem. Print len(r.content) and the title (doc.findtext(".//title")) before debugging expressions. We fetched through QuanticData mobile proxies; residential proxies work the same way, with the country and session in the username. There is a Residential Basic free trial to try it.


