Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Beautiful Soup web scraping: find, find_all, select and the traps, tested

Beautiful Soup scraping a product list: requests downloads the page through a proxy, BeautifulSoup parses r.content with lxml, and the title, price and star rating of each book are read from three marked parts of the page; the card shows the first row, A Light in the Attic at 51.77 with three stars, and a note warns to pass r.content, not r.text, so the pound sign does not turn into £.
Beautiful Soup scraping a product list: requests downloads the page through a proxy, BeautifulSoup parses r.content with lxml, and the title, price and star rating of each book are read from three marked parts of the page; the card shows the first row, A Light in the Attic at 51.77 with three stars, and a note warns to pass r.content, not r.text, so the pound sign does not turn into £.

To scrape a page with Beautiful Soup, download it with requests, parse r.content with BeautifulSoup(r.content, "lxml"), then pick elements with find(), find_all() or the CSS method select() and read their text or attributes. Install it with pip install beautifulsoup4 lxml requests.

Every line of output below is real: we ran the code on 20 November 2026 with Python 3.12.10 and beautifulsoup4 4.14.3, against books.toscrape.com (a public shop built for scraping practice), through a QuanticData Residential Premium session in the UK.

Step by step: your first Beautiful Soup scraper

  1. Install the three packages: pip install beautifulsoup4 lxml requests. The import name is bs4.
  2. Fetch the page with requests, through a proxy if you scrape more than a few pages.
  3. Parse the bytes into a soup.
  4. Find the repeating block (here one article.product_pod per book) and read each field inside it.
import requests
from bs4 import BeautifulSoup

PROXY = "http://USER-country-gb-sid-bs01-ttl-30m:PASS@residentialpremium.quanticdata.io:8080"
r = requests.get("https://books.toscrape.com/catalogue/page-1.html",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)
soup = BeautifulSoup(r.content, "lxml")

rows = []
for pod in soup.find_all("article", class_="product_pod"):
    rows.append({
        "title": pod.h3.a["title"],
        "price": float(pod.select_one(".price_color").text.lstrip("£")),
        "stars": pod.select_one(".star-rating")["class"][1],
    })
print(len(rows), rows[:2])
20 [{'title': 'A Light in the Attic', 'price': 51.77, 'stars': 'Three'},
    {'title': 'Tipping the Velvet', 'price': 53.74, 'stars': 'One'}]

find, find_all and select

CallReturnsOur output
soup.find_all("article", class_="product_pod")A list of every match20 elements
pod.find("p", class_="price_color").get_text(strip=True)The first match, or None£51.77
pod.h3.a["title"]An attribute (KeyError if missing)A Light in the Attic
pod.h3.a.get("href")An attribute, or Nonea-light-in-the-attic_1000/index.html
soup.select("article.product_pod h3 a")A list, CSS selector20 elements
soup.select_one("li.next a")["href"]The first CSS matchpage-2.html
pod.select_one(".star-rating")["class"]class is a list['star-rating', 'Three']
  • Find by class with class_= (the underscore, because class is a Python keyword) or with select(".name"). class_="star-rating" matches an element that has several classes.
  • find() returns None when nothing matches, so soup.find("div", class_="nope").text raises AttributeError. Check for None on optional fields.
  • Find by text needs the exact string: soup.find("a", string="Travel") returned None on this page because the link text has spaces and line breaks around it. string=re.compile(r"^\s*Travel\s*$") found it.

Move around the tree: parents and siblings

Sometimes the field you know is not the one you want. Start from the price and climb to the book it belongs to:

price = soup.find("p", string="£51.77")
book = price.find_parent("article")
print(book.h3.a["title"])                          # A Light in the Attic
print(price.parent.name, price.parent["class"])    # div ['product_price']
print(price.find_next_sibling("p")["class"])       # ['instock', 'availability']
print(book.find("div", class_="product_price").get_text(" | ", strip=True))
# £51.77 | In stock | Add to basket
print(book.h3.a.attrs)
# {'href': 'a-light-in-the-attic_1000/index.html', 'title': 'A Light in the Attic'}

find_parent(), find_next_sibling() and .parent walk the tree; get_text(" | ", strip=True) joins the text of every child with a separator; .attrs gives all attributes as a dictionary.

Which parser: html.parser, lxml or html5lib

Beautiful Soup does not parse by itself; it drives a parser. All three found the same 20 books on our page. They differ in speed and in how they repair broken HTML:

ParserInstallMedian, parse only, our page<ul><li>one<li>two</ul> becomes
html.parserBuilt into Python7.8 ms<li>one<li>two</li></li>, the second li nested in the first
lxmlpip install lxml6.3 msTwo sibling li elements
html5libpip install html5lib16.7 msTwo sibling li elements, like a browser

Use lxml by default. Switch to html5lib when a badly broken page parses differently from what you see in the browser. Always name the parser: without it, Beautiful Soup picks the best one installed, and the same script can behave differently on another machine. For bigger jobs, the comparison with lxml and parsel is in Python web scraping libraries.

The encoding trap: r.text versus r.content

The practice shop sends Content-Type: text/html without a charset. requests then assumes ISO-8859-1 for r.text, while the page is UTF-8. Same response, two results:

print(r.headers["content-type"], r.encoding)   # text/html ISO-8859-1
BeautifulSoup(r.text, "lxml").select_one(".price_color").text     # '£51.77'
BeautifulSoup(r.content, "lxml").select_one(".price_color").text  # '£51.77'

Pass r.content (bytes) and Beautiful Soup reads the charset from the page's own meta tag. Or set r.encoding = "utf-8" before using r.text.

Save the rows

import csv
with open("books.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["title", "price", "stars"])
    w.writeheader()
    w.writerows(rows)

For more than one page, wrap the scraper in a loop that follows the Next link: the full loop is in web scraping pagination. Beautiful Soup reads only the HTML the server sends. If the data appears only after JavaScript runs, the soup will be empty: render the page with Playwright first, or find the JSON the page loads.

Beautiful Soup with a proxy

Beautiful Soup never touches the network; requests (or httpx, or aiohttp) does. So the proxy goes in the request, as in the code above, and works the same with every parser. The proxy decides which address the site sees:

  • Residential Premium, which we used here: a home IP in the country you set in the username, and a session ID in the username when you want one.
  • Residential Basic: a rotating gateway and a session gateway, chosen by port, with the country in the username.
  • Mobile uses carrier IPs, for sites that treat phone traffic differently.

Setup for requests is in a proxy in Python requests. Plans: residential proxies and mobile proxies; to run this tutorial on a real exit first, there is a Residential Basic free trial.

Sources & further reading

FAQ

Quick answers on beautifulsoup web scraping.

Something else? Ask us

How do I use Beautiful Soup for web scraping?

Download the page with requests, parse it with BeautifulSoup(r.content, "lxml"), then use find_all() or select() to get the repeating elements and read .get_text() or ["attribute"] from each. Our run returned 20 books with title, price and rating.

What is the difference between find and find_all in Beautiful Soup?

find() returns the first match or None; find_all() returns a list of every match, empty if none. select_one() and select() do the same with CSS selectors.

How do I find elements by class in Beautiful Soup?

Use soup.find_all("p", class_="price_color") with the underscore, or soup.select("p.price_color"). An element with several classes, such as star-rating Three, matches either class.

Which parser should I use with Beautiful Soup?

lxml for most work: parsing alone took 6.3 ms on our page against 7.8 ms for html.parser and 16.7 ms for html5lib. Use html5lib when a broken page must be repaired the way a browser does it.

Why does Beautiful Soup show strange characters like £?

The server did not declare a charset and requests decoded r.text as ISO-8859-1. Pass r.content to BeautifulSoup instead, or set r.encoding = "utf-8" first.

Can Beautiful Soup scrape JavaScript pages?

No. It parses the HTML you give it and does not run scripts. Render the page with Playwright or Selenium and pass page.content() to BeautifulSoup, or fetch the JSON the page loads.

Feed Beautiful Soup pages from a home or mobile IP

Residential and mobile proxies for requests, httpx and aiohttp, with country and session settings in the username.

Related reading