# lxml XPath in Python: Scraping Tutorial

> lxml XPath tutorial for Python scraping: parse with lxml.html, select with xpath(), contains(), text(), ancestors and last(), and compile for speed.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / lxml XPath in Python: Scraping Tutorial

# lxml XPath in Python: the expressions that scrape real pages, tested

Guides · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![lxml XPath on a product page in Python: doc.xpath with //article\[@class="product_pod"\]/h3/a/@title returns 20 book titles, contains(@class, "star-rating") finds 20 ratings where the exact match finds 0, and ancestor::article climbs from a price to its book; the card shows timings of 25 microseconds for a compiled XPath and 61 for cssselect.](https://quanticdata.io/blog/lxml-xpath-python/hero.webp?v=20261120h)

On this page

- [Parse the page](https://quanticdata.io/blog/lxml-xpath-python/#parse-the-page)
- [The XPath expressions you will use most](https://quanticdata.io/blog/lxml-xpath-python/#the-xpath-expressions-you-will-use-most)
- [contains(): the fix for classes and messy text](https://quanticdata.io/blog/lxml-xpath-python/#contains-the-fix-for-classes-and-messy-text)
- [Loop over blocks with relative paths](https://quanticdata.io/blog/lxml-xpath-python/#loop-over-blocks-with-relative-paths)
- [Axes: siblings, parents and the brackets trap](https://quanticdata.io/blog/lxml-xpath-python/#axes-siblings-parents-and-the-brackets-trap)
- [Faster: compile the expression](https://quanticdata.io/blog/lxml-xpath-python/#faster-compile-the-expression)
- [CSS selectors in lxml](https://quanticdata.io/blog/lxml-xpath-python/#css-selectors-in-lxml)
- [lxml vs Beautiful Soup](https://quanticdata.io/blog/lxml-xpath-python/#lxml-vs-beautiful-soup)
- [Where the page comes from](https://quanticdata.io/blog/lxml-xpath-python/#where-the-page-comes-from)

To use XPath with lxml in Python, parse the page with `lxml.html.fromstring(r.content)`, then call `doc.xpath("expression")`. It returns a list: of elements for `//article`, of strings for `//a/@href` or `//p/text()`. Install it with `pip install lxml`.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=lxml-xpath-python) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

Each expression in this tutorial ran on 20 November 2026 with Python 3.12.10 and lxml 6.1.1, on a catalogue page of books.toscrape.com (a public shop made for scraping practice) downloaded through a QuanticData mobile proxy in the US. The comments show what it returned.

## Parse the page

```python
import requests, lxml.html

PROXY = "http://USER-country-us-type-mobile-sid-x01-ttl-30m:PASS@mobile.quanticdata.io:8080"
r = requests.get("https://books.toscrape.com/catalogue/page-1.html",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)
doc = lxml.html.fromstring(r.content)
```

Pass `r.content` (bytes), not `r.text`: lxml then reads the page's own charset, and the £ in prices stays a £. For XML documents such as sitemaps and feeds use `lxml.etree.fromstring` instead; that case, with namespaces, is in [parsing XML in Python](https://quanticdata.io/blog/parse-xml-python/).

## The XPath expressions you will use most

| Goal | Expression | Our result |
| --- | --- | --- |
| An attribute of every match | `//article[@class="product_pod"]/h3/a/@title` | 20 titles, first "A Light in the Attic" |
| Text of elements | `//p[@class="price_color"]/text()` | 20 prices, first "£51.77" |
| A class among several | `//p[contains(@class, "star-rating")]` | 20 (the exact `@class="star-rating"` found 0) |
| Text with spaces trimmed | `normalize-space(//p[contains(@class, "instock")])` | "In stock" |
| Link by its text | `//a[contains(text(), "Travel")]/@href` | category/books/travel_2/index.html |
| Climb to a parent block | `//p[@class="price_color"][text()="£51.77"]/ancestor::article//h3/a/@title` | A Light in the Attic |
| The last match | `(//article[@class="product_pod"])[last()]//h3/a/@title` | It's Only the Himalayas |
| Count | `count(//article)` | 20.0 (a float) |
| Next page link | `//li[@class="next"]/a/@href` | ['page-2.html'] |

## contains(): the fix for classes and messy text

The rating paragraph on this page has two classes, `star-rating Three`. `@class="star-rating"` compares the whole attribute string, so it matched nothing; `contains(@class, "star-rating")` matched all 20. The same goes for text with spaces and line breaks around it: `contains(text(), "Travel")` found the link where an exact text match would miss. One caution: `contains(@class, "star")` would also match a class called `starred`. For an exact class among several, use `contains(concat(" ", normalize-space(@class), " "), " star-rating ")`, or a CSS selector, which handles classes natively.

## Loop over blocks with relative paths

To keep each title with its own price, select the blocks first and run XPath from each one. Start the inner expressions with `.` so they search inside the block, not the whole page:

```
rows = []
for pod in doc.xpath('//article[@class="product_pod"]'):
    rows.append({
        "title": pod.xpath("./h3/a/@title")[0],
        "price": pod.xpath('.//p[@class="price_color"]/text()')[0],
        "stars": pod.xpath('.//p[contains(@class, "star-rating")]/@class')[0].split()[1],
    })
print(len(rows), rows[0])
# 20 {'title': 'A Light in the Attic', 'price': '£51.77', 'stars': 'Three'}
```

Without the dot, `pod.xpath("//h3/a/@title")` returns all 20 titles from every block, the most common lxml surprise. Since xpath always returns a list, `[0]` raises IndexError when nothing matches; use `next(iter(result), None)` for optional fields.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=lxml-xpath-python) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## Axes: siblings, parents and the brackets trap

XPath can move in any direction from an element, which CSS selectors cannot. From the first price on the page:

| Expression | Our result |
| --- | --- |
| `(//p[@class="price_color"])[1]/following-sibling::p[1]/@class` | ['instock availability'] |
| `(//p[@class="price_color"])[1]/../@class` | ['product_price'], the parent |
| `(//div[@class="product_price"])[1]/preceding-sibling::h3/a/@title` | ['A Light in the Attic'] |
| `//a[starts-with(@href, "category/")]` | 51 links |
| `//p[contains(@class, "star-rating") and (contains(@class, "Five") or contains(@class, "One"))]` | 10 books rated five or one |

Mind the brackets. We asked for the first two books with `//article[@class="product_pod"][position() <= 2]` and got all 20 titles back: on this page every article sits alone inside its own list item, so each one is "position 1" among its siblings. Wrapping the path in parentheses, `(//article[@class="product_pod"])[position() <= 2]`, counts across the whole document and returns two. The same rule explains `(//p[@class="price_color"])[1]` above.

## Faster: compile the expression

We timed each way of running the same query (20 links) 500 times on the parsed page:

| Method | Median per query |
| --- | --- |
| `etree.XPath(expr)`, compiled once | 25 microseconds |
| `doc.xpath(expr)` | 31 microseconds |
| `CSSSelector(css)`, compiled once | 33 microseconds |
| `doc.cssselect(css)` | 61 microseconds |

```python
from lxml import etree
titles = etree.XPath('//article[@class="product_pod"]/h3/a')
for page_doc in docs:
    links = titles(page_doc)
```

All of these are microseconds; downloading the page takes far longer. Compile when the same expression runs over thousands of pages.

## CSS selectors in lxml

With `pip install cssselect`, lxml also takes CSS: `doc.cssselect("article.product_pod h3 a")` found the same 20 links. It works by translating CSS into XPath; `CSSSelector("article.product_pod h3 a").path` shows the result, which starts with `descendant-or-self::article[@class and contains(@class, ...`. That is why compiled CSS ran close to XPath and uncompiled CSS slower. When to pick which is covered in [CSS selector vs XPath](https://quanticdata.io/blog/css-selector-vs-xpath/).

## lxml vs Beautiful Soup

On the same page, lxml with XPath extracted the 20 books in 1.81 ms; Beautiful Soup with the lxml parser took 12.91 ms. Beautiful Soup has the friendlier API and repairs broken HTML more forgivingly; lxml is faster and gives you XPath's axes (parent, ancestor, following-sibling) in one expression. Both read only the HTML the server sends. The Beautiful Soup side: [Beautiful Soup web scraping](https://quanticdata.io/blog/beautifulsoup-web-scraping/); all the options compared: [Python web scraping libraries](https://quanticdata.io/blog/python-web-scraping-libraries/).

## Where the page comes from

lxml parses; the download decides what there is to parse. If a site sends a block page to scripts, your XPath returns empty lists and the bug looks like a selector problem. Print `len(r.content)` and the title (`doc.findtext(".//title")`) before debugging expressions. We fetched through QuanticData [mobile proxies](https://quanticdata.io/mobile-proxies/); [residential proxies](https://quanticdata.io/residential-proxies/) work the same way, with the country and session in the username. There is a [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/) to try it.

### Sources & further reading

- [lxml: XPath and XSLT with lxml](https://lxml.de/xpathxslt.html)
- [lxml: parsing HTML (lxml.html)](https://lxml.de/lxmlhtml.html)
- [lxml: cssselect](https://lxml.de/cssselect.html)
- [W3C: XML Path Language (XPath) 1.0](https://www.w3.org/TR/1999/REC-xpath-19991116/)

## FAQ

Quick answers on lxml xpath.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I use XPath with lxml in Python?

Parse with `doc = lxml.html.fromstring(r.content)`, then call `doc.xpath("//tag[@attr='value']")`. Add `/@attr` or `/text()` to get strings instead of elements. The result is always a list.

### How do I select an element by class with XPath?

Use `contains(@class, "name")`. An exact `@class="name"` fails when the element has more than one class: on our page it found 0 ratings where contains() found 20.

### Why does my XPath inside a loop return results from the whole page?

An expression that starts with `//` searches the whole document even when called on a child element. Start it with `.//` or `./` to search inside that element only.

### Is lxml faster than Beautiful Soup?

Yes. On our test page, lxml with XPath extracted 20 books in 1.81 ms against 12.91 ms for Beautiful Soup with the lxml parser. A compiled XPath query took 25 microseconds.

### Can lxml use CSS selectors?

Yes, with the cssselect package: `doc.cssselect("article.product_pod h3 a")`. It translates the CSS into XPath; compiled with `CSSSelector` it ran in 33 microseconds on our page.

## Give your XPath a real page to read

Mobile and residential proxies for requests and httpx, with country and session settings in the username.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=lxml-xpath-python) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## Collectors with Python examples

- [eBay scraper API in Python](https://quanticdata.io/collectors/ebay-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the eBay scraper collector, with Python examples.
- [Google Images scraper API in Python](https://quanticdata.io/collectors/image-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Google Images scraper collector, with Python examples.
- [Video search API in Python](https://quanticdata.io/collectors/search-videos-api/python/): Collector · Install, first call, async runs, output fields and errors of the Video search collector, with Python examples.
- [Scrape YouTube videos in Python](https://quanticdata.io/collectors/youtube-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Scrape YouTube videos collector, with Python examples.

## Related reading

- [urllib3 vs requests: Speed and Bandwidth](https://quanticdata.io/blog/urllib3-vs-requests/): Guides · requests is built on urllib3 and adds sessions, JSON, exceptions on errors and compression by default. urllib3 alone was faster on a local server, the same through a proxy, and downloaded 9.5 times more bytes until we asked for compression. Measured, with the code.
- [Python requests POST: JSON, Form Data, Files](https://quanticdata.io/blog/python-requests-post/): Guides · requests.post(url, json=payload) sends a JSON body with the right Content-Type; data= sends form fields; files= uploads. We posted each one through a residential proxy to a test server and print what the server received, including the redirect that silently turns a POST into a GET.
- [Concurrency vs Parallelism, Measured in Python](https://quanticdata.io/blog/concurrency-vs-parallelism/): Guides · Concurrency is making progress on several tasks by switching between them; parallelism is running several at the same instant on different CPU cores. We timed both on a real scraping job in Python on 20 November 2026: threads cut 30 downloads from 10.3 s to 3.5 s, and only processes cut the parsing, from 3.5 s to 0.8 s.

---

Source: https://quanticdata.io/blog/lxml-xpath-python/ · Site index for AI: https://quanticdata.io/llms.txt
