To scrape a page with Beautiful Soup, download it with requests, parse r.content with BeautifulSoup(r.content, "lxml"), then pick elements with find(), find_all() or the CSS method select() and read their text or attributes. Install it with pip install beautifulsoup4 lxml requests.
Every line of output below is real: we ran the code on 20 November 2026 with Python 3.12.10 and beautifulsoup4 4.14.3, against books.toscrape.com (a public shop built for scraping practice), through a QuanticData Residential Premium session in the UK.
Step by step: your first Beautiful Soup scraper
- Install the three packages:
pip install beautifulsoup4 lxml requests. The import name isbs4. - Fetch the page with requests, through a proxy if you scrape more than a few pages.
- Parse the bytes into a soup.
- Find the repeating block (here one
article.product_podper book) and read each field inside it.
import requests
from bs4 import BeautifulSoup
PROXY = "http://USER-country-gb-sid-bs01-ttl-30m:PASS@residentialpremium.quanticdata.io:8080"
r = requests.get("https://books.toscrape.com/catalogue/page-1.html",
proxies={"http": PROXY, "https": PROXY}, timeout=30)
soup = BeautifulSoup(r.content, "lxml")
rows = []
for pod in soup.find_all("article", class_="product_pod"):
rows.append({
"title": pod.h3.a["title"],
"price": float(pod.select_one(".price_color").text.lstrip("£")),
"stars": pod.select_one(".star-rating")["class"][1],
})
print(len(rows), rows[:2])
20 [{'title': 'A Light in the Attic', 'price': 51.77, 'stars': 'Three'},
{'title': 'Tipping the Velvet', 'price': 53.74, 'stars': 'One'}]
find, find_all and select
| Call | Returns | Our output |
|---|---|---|
soup.find_all("article", class_="product_pod") | A list of every match | 20 elements |
pod.find("p", class_="price_color").get_text(strip=True) | The first match, or None | £51.77 |
pod.h3.a["title"] | An attribute (KeyError if missing) | A Light in the Attic |
pod.h3.a.get("href") | An attribute, or None | a-light-in-the-attic_1000/index.html |
soup.select("article.product_pod h3 a") | A list, CSS selector | 20 elements |
soup.select_one("li.next a")["href"] | The first CSS match | page-2.html |
pod.select_one(".star-rating")["class"] | class is a list | ['star-rating', 'Three'] |
- Find by class with
class_=(the underscore, becauseclassis a Python keyword) or withselect(".name").class_="star-rating"matches an element that has several classes. - find() returns None when nothing matches, so
soup.find("div", class_="nope").textraises AttributeError. Check for None on optional fields. - Find by text needs the exact string:
soup.find("a", string="Travel")returned None on this page because the link text has spaces and line breaks around it.string=re.compile(r"^\s*Travel\s*$")found it.
Move around the tree: parents and siblings
Sometimes the field you know is not the one you want. Start from the price and climb to the book it belongs to:
price = soup.find("p", string="£51.77")
book = price.find_parent("article")
print(book.h3.a["title"]) # A Light in the Attic
print(price.parent.name, price.parent["class"]) # div ['product_price']
print(price.find_next_sibling("p")["class"]) # ['instock', 'availability']
print(book.find("div", class_="product_price").get_text(" | ", strip=True))
# £51.77 | In stock | Add to basket
print(book.h3.a.attrs)
# {'href': 'a-light-in-the-attic_1000/index.html', 'title': 'A Light in the Attic'}
find_parent(), find_next_sibling() and .parent walk the tree; get_text(" | ", strip=True) joins the text of every child with a separator; .attrs gives all attributes as a dictionary.
Which parser: html.parser, lxml or html5lib
Beautiful Soup does not parse by itself; it drives a parser. All three found the same 20 books on our page. They differ in speed and in how they repair broken HTML:
| Parser | Install | Median, parse only, our page | <ul><li>one<li>two</ul> becomes |
|---|---|---|---|
| html.parser | Built into Python | 7.8 ms | <li>one<li>two</li></li>, the second li nested in the first |
| lxml | pip install lxml | 6.3 ms | Two sibling li elements |
| html5lib | pip install html5lib | 16.7 ms | Two sibling li elements, like a browser |
Use lxml by default. Switch to html5lib when a badly broken page parses differently from what you see in the browser. Always name the parser: without it, Beautiful Soup picks the best one installed, and the same script can behave differently on another machine. For bigger jobs, the comparison with lxml and parsel is in Python web scraping libraries.
The encoding trap: r.text versus r.content
The practice shop sends Content-Type: text/html without a charset. requests then assumes ISO-8859-1 for r.text, while the page is UTF-8. Same response, two results:
print(r.headers["content-type"], r.encoding) # text/html ISO-8859-1
BeautifulSoup(r.text, "lxml").select_one(".price_color").text # '£51.77'
BeautifulSoup(r.content, "lxml").select_one(".price_color").text # '£51.77'
Pass r.content (bytes) and Beautiful Soup reads the charset from the page's own meta tag. Or set r.encoding = "utf-8" before using r.text.
Save the rows
import csv
with open("books.csv", "w", newline="", encoding="utf-8") as f:
w = csv.DictWriter(f, fieldnames=["title", "price", "stars"])
w.writeheader()
w.writerows(rows)
For more than one page, wrap the scraper in a loop that follows the Next link: the full loop is in web scraping pagination. Beautiful Soup reads only the HTML the server sends. If the data appears only after JavaScript runs, the soup will be empty: render the page with Playwright first, or find the JSON the page loads.
Beautiful Soup with a proxy
Beautiful Soup never touches the network; requests (or httpx, or aiohttp) does. So the proxy goes in the request, as in the code above, and works the same with every parser. The proxy decides which address the site sees:
- Residential Premium, which we used here: a home IP in the country you set in the username, and a session ID in the username when you want one.
- Residential Basic: a rotating gateway and a session gateway, chosen by port, with the country in the username.
- Mobile uses carrier IPs, for sites that treat phone traffic differently.
Setup for requests is in a proxy in Python requests. Plans: residential proxies and mobile proxies; to run this tutorial on a real exit first, there is a Residential Basic free trial.


