# BeautifulSoup Web Scraping in Python, Tested

> Beautiful Soup web scraping in Python: fetch with requests, parse with find, find_all and select, pick a parser, and avoid the encoding trap. Real output.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / BeautifulSoup Web Scraping in Python, Tested

# Beautiful Soup web scraping: find, find_all, select and the traps, tested

Guides · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![Beautiful Soup scraping a product list: requests downloads the page through a proxy, BeautifulSoup parses r.content with lxml, and the title, price and star rating of each book are read from three marked parts of the page; the card shows the first row, A Light in the Attic at 51.77 with three stars, and a note warns to pass r.content, not r.text, so the pound sign does not turn into Â£.](https://quanticdata.io/blog/beautifulsoup-web-scraping/hero.webp?v=20261120h)

On this page

- [Step by step: your first Beautiful Soup scraper](https://quanticdata.io/blog/beautifulsoup-web-scraping/#step-by-step-your-first-beautiful-soup-scraper)
- [find, find_all and select](https://quanticdata.io/blog/beautifulsoup-web-scraping/#find-find-all-and-select)
- [Move around the tree: parents and siblings](https://quanticdata.io/blog/beautifulsoup-web-scraping/#move-around-the-tree-parents-and-siblings)
- [Which parser: html.parser, lxml or html5lib](https://quanticdata.io/blog/beautifulsoup-web-scraping/#which-parser-html-parser-lxml-or-html5lib)
- [The encoding trap: r.text versus r.content](https://quanticdata.io/blog/beautifulsoup-web-scraping/#the-encoding-trap-r-text-versus-r-content)
- [Save the rows](https://quanticdata.io/blog/beautifulsoup-web-scraping/#save-the-rows)
- [Beautiful Soup with a proxy](https://quanticdata.io/blog/beautifulsoup-web-scraping/#beautiful-soup-with-a-proxy)

To scrape a page with Beautiful Soup, download it with requests, parse `r.content` with `BeautifulSoup(r.content, "lxml")`, then pick elements with `find()`, `find_all()` or the CSS method `select()` and read their text or attributes. Install it with `pip install beautifulsoup4 lxml requests`.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=beautifulsoup-web-scraping) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

Every line of output below is real: we ran the code on 20 November 2026 with Python 3.12.10 and beautifulsoup4 4.14.3, against books.toscrape.com (a public shop built for scraping practice), through a QuanticData Residential Premium session in the UK.

## Step by step: your first Beautiful Soup scraper

1. **Install** the three packages: `pip install beautifulsoup4 lxml requests`. The import name is `bs4`.
2. **Fetch** the page with requests, through a proxy if you scrape more than a few pages.
3. **Parse** the bytes into a soup.
4. **Find** the repeating block (here one `article.product_pod` per book) and read each field inside it.

```python
import requests
from bs4 import BeautifulSoup

PROXY = "http://USER-country-gb-sid-bs01-ttl-30m:PASS@residentialpremium.quanticdata.io:8080"
r = requests.get("https://books.toscrape.com/catalogue/page-1.html",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)
soup = BeautifulSoup(r.content, "lxml")

rows = []
for pod in soup.find_all("article", class_="product_pod"):
    rows.append({
        "title": pod.h3.a["title"],
        "price": float(pod.select_one(".price_color").text.lstrip("£")),
        "stars": pod.select_one(".star-rating")["class"][1],
    })
print(len(rows), rows[:2])
```

```
20 [{'title': 'A Light in the Attic', 'price': 51.77, 'stars': 'Three'},
    {'title': 'Tipping the Velvet', 'price': 53.74, 'stars': 'One'}]
```

## find, find_all and select

| Call | Returns | Our output |
| --- | --- | --- |
| `soup.find_all("article", class_="product_pod")` | A list of every match | 20 elements |
| `pod.find("p", class_="price_color").get_text(strip=True)` | The first match, or None | £51.77 |
| `pod.h3.a["title"]` | An attribute (KeyError if missing) | A Light in the Attic |
| `pod.h3.a.get("href")` | An attribute, or None | a-light-in-the-attic_1000/index.html |
| `soup.select("article.product_pod h3 a")` | A list, CSS selector | 20 elements |
| `soup.select_one("li.next a")["href"]` | The first CSS match | page-2.html |
| `pod.select_one(".star-rating")["class"]` | class is a list | ['star-rating', 'Three'] |

- **Find by class** with `class_=` (the underscore, because `class` is a Python keyword) or with `select(".name")`. `class_="star-rating"` matches an element that has several classes.
- **find() returns None** when nothing matches, so `soup.find("div", class_="nope").text` raises AttributeError. Check for None on optional fields.
- **Find by text** needs the exact string: `soup.find("a", string="Travel")` returned None on this page because the link text has spaces and line breaks around it. `string=re.compile(r"^\s*Travel\s*$")` found it.

## Move around the tree: parents and siblings

Sometimes the field you know is not the one you want. Start from the price and climb to the book it belongs to:

```
price = soup.find("p", string="£51.77")
book = price.find_parent("article")
print(book.h3.a["title"])                          # A Light in the Attic
print(price.parent.name, price.parent["class"])    # div ['product_price']
print(price.find_next_sibling("p")["class"])       # ['instock', 'availability']
print(book.find("div", class_="product_price").get_text(" | ", strip=True))
# £51.77 | In stock | Add to basket
print(book.h3.a.attrs)
# {'href': 'a-light-in-the-attic_1000/index.html', 'title': 'A Light in the Attic'}
```

`find_parent()`, `find_next_sibling()` and `.parent` walk the tree; `get_text(" | ", strip=True)` joins the text of every child with a separator; `.attrs` gives all attributes as a dictionary.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=beautifulsoup-web-scraping) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## Which parser: html.parser, lxml or html5lib

Beautiful Soup does not parse by itself; it drives a parser. All three found the same 20 books on our page. They differ in speed and in how they repair broken HTML:

| Parser | Install | Median, parse only, our page | `<ul><li>one<li>two</ul>` becomes |
| --- | --- | --- | --- |
| html.parser | Built into Python | 7.8 ms | `<li>one<li>two</li></li>`, the second li nested in the first |
| lxml | `pip install lxml` | 6.3 ms | Two sibling li elements |
| html5lib | `pip install html5lib` | 16.7 ms | Two sibling li elements, like a browser |

Use lxml by default. Switch to html5lib when a badly broken page parses differently from what you see in the browser. Always name the parser: without it, Beautiful Soup picks the best one installed, and the same script can behave differently on another machine. For bigger jobs, the comparison with lxml and parsel is in [Python web scraping libraries](https://quanticdata.io/blog/python-web-scraping-libraries/).

## The encoding trap: r.text versus r.content

The practice shop sends `Content-Type: text/html` without a charset. requests then assumes ISO-8859-1 for `r.text`, while the page is UTF-8. Same response, two results:

```
print(r.headers["content-type"], r.encoding)   # text/html ISO-8859-1
BeautifulSoup(r.text, "lxml").select_one(".price_color").text     # 'Â£51.77'
BeautifulSoup(r.content, "lxml").select_one(".price_color").text  # '£51.77'
```

Pass `r.content` (bytes) and Beautiful Soup reads the charset from the page's own meta tag. Or set `r.encoding = "utf-8"` before using `r.text`.

## Save the rows

```python
import csv
with open("books.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["title", "price", "stars"])
    w.writeheader()
    w.writerows(rows)
```

For more than one page, wrap the scraper in a loop that follows the Next link: the full loop is in [web scraping pagination](https://quanticdata.io/blog/web-scraping-pagination/). Beautiful Soup reads only the HTML the server sends. If the data appears only after JavaScript runs, the soup will be empty: render the page with Playwright first, or find the JSON the page loads.

## Beautiful Soup with a proxy

Beautiful Soup never touches the network; requests (or httpx, or aiohttp) does. So the proxy goes in the request, as in the code above, and works the same with every parser. The proxy decides which address the site sees:

- **Residential Premium**, which we used here: a home IP in the country you set in the username, and a session ID in the username when you want one.
- **Residential Basic**: a rotating gateway and a session gateway, chosen by port, with the country in the username.
- **Mobile** uses carrier IPs, for sites that treat phone traffic differently.

Setup for requests is in [a proxy in Python requests](https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/). Plans: [residential proxies](https://quanticdata.io/residential-proxies/) and [mobile proxies](https://quanticdata.io/mobile-proxies/); to run this tutorial on a real exit first, there is a [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/).

### Sources & further reading

- [Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
- [Beautiful Soup: differences between parsers](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#differences-between-parsers)
- [Requests: response content and encoding](https://requests.readthedocs.io/en/latest/user/quickstart/#response-content)
- [beautifulsoup4 on PyPI](https://pypi.org/project/beautifulsoup4/)

## FAQ

Quick answers on beautifulsoup web scraping.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I use Beautiful Soup for web scraping?

Download the page with requests, parse it with `BeautifulSoup(r.content, "lxml")`, then use `find_all()` or `select()` to get the repeating elements and read `.get_text()` or `["attribute"]` from each. Our run returned 20 books with title, price and rating.

### What is the difference between find and find_all in Beautiful Soup?

`find()` returns the first match or None; `find_all()` returns a list of every match, empty if none. `select_one()` and `select()` do the same with CSS selectors.

### How do I find elements by class in Beautiful Soup?

Use `soup.find_all("p", class_="price_color")` with the underscore, or `soup.select("p.price_color")`. An element with several classes, such as `star-rating Three`, matches either class.

### Which parser should I use with Beautiful Soup?

lxml for most work: parsing alone took 6.3 ms on our page against 7.8 ms for html.parser and 16.7 ms for html5lib. Use html5lib when a broken page must be repaired the way a browser does it.

### Why does Beautiful Soup show strange characters like Â£?

The server did not declare a charset and requests decoded `r.text` as ISO-8859-1. Pass `r.content` to BeautifulSoup instead, or set `r.encoding = "utf-8"` first.

### Can Beautiful Soup scrape JavaScript pages?

No. It parses the HTML you give it and does not run scripts. Render the page with Playwright or Selenium and pass `page.content()` to BeautifulSoup, or fetch the JSON the page loads.

## Feed Beautiful Soup pages from a home or mobile IP

Residential and mobile proxies for requests, httpx and aiohttp, with country and session settings in the username.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=beautifulsoup-web-scraping) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## Related reading

- [urllib3 vs requests: Speed and Bandwidth](https://quanticdata.io/blog/urllib3-vs-requests/): Guides · requests is built on urllib3 and adds sessions, JSON, exceptions on errors and compression by default. urllib3 alone was faster on a local server, the same through a proxy, and downloaded 9.5 times more bytes until we asked for compression. Measured, with the code.
- [Python requests POST: JSON, Form Data, Files](https://quanticdata.io/blog/python-requests-post/): Guides · requests.post(url, json=payload) sends a JSON body with the right Content-Type; data= sends form fields; files= uploads. We posted each one through a residential proxy to a test server and print what the server received, including the redirect that silently turns a POST into a GET.
- [Concurrency vs Parallelism, Measured in Python](https://quanticdata.io/blog/concurrency-vs-parallelism/): Guides · Concurrency is making progress on several tasks by switching between them; parallelism is running several at the same instant on different CPU cores. We timed both on a real scraping job in Python on 20 November 2026: threads cut 30 downloads from 10.3 s to 3.5 s, and only processes cut the parsing, from 3.5 s to 0.8 s.

---

Source: https://quanticdata.io/blog/beautifulsoup-web-scraping/ · Site index for AI: https://quanticdata.io/llms.txt
