# Scrapy vs BeautifulSoup: Same 1,000 Items

> Scrapy vs BeautifulSoup on the same 50 pages through the same proxy: 4.6 s against 14.9 s, the code for both, and when each one is the right choice.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Scrapy vs BeautifulSoup: Same 1,000 Items

# Scrapy vs Beautiful Soup: the same 1,000 items scraped both ways

Guides · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![Scrapy and Beautiful Soup compared on the same 50 pages and 1,000 books through one QuanticData proxy: Scrapy with 16 requests at a time finished in 4.6 seconds, requests with Beautiful Soup one page after another in 14.9 seconds; the card lists what each one does, parsing only for Beautiful Soup, downloading, scheduling and export for Scrapy.](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/hero.webp?v=20261120h)

On this page

- [The result: 4.6 s against 14.9 s](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#the-result-4-6-s-against-14-9-s)
- [The requests + Beautiful Soup version](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#the-requests-beautiful-soup-version)
- [The Scrapy version](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#the-scrapy-version)
- [Side by side](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#side-by-side)
- [How to choose in three questions](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#how-to-choose-in-three-questions)
- [Using them together](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#using-them-together)
- [What about Selenium?](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#what-about-selenium)
- [Proxies for each approach](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/#proxies-for-each-approach)

Scrapy vs Beautiful Soup is not a like-for-like choice. Beautiful Soup is a parser: it reads HTML you already downloaded. Scrapy is a crawling framework: it downloads, schedules, retries, follows links, parses and exports. For a few pages, use requests with Beautiful Soup. For a whole site or thousands of pages, use Scrapy.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=scrapy-vs-beautifulsoup) · [Explore Rotating proxies for scraping from $0.50/GB](https://quanticdata.io/rotating-proxies/)

To put numbers on it, on 20 November 2026 we scraped the same 50 catalogue pages of books.toscrape.com, a public shop made for scraping practice, both ways, through the same QuanticData proxy, with Python 3.12.10, Scrapy 2.20.0 and beautifulsoup4 4.14.3. This was its own run: the requests figure here (14.9 s) and the one in our async scraping comparison (15.6 s) come from separate runs of the same kind of test, each with its own date and time.

## The result: 4.6 s against 14.9 s

| Setup | Items | Time |
| --- | --- | --- |
| Scrapy, 50 start URLs, 16 requests at a time | 1,000 | 4.6 s |
| requests + Beautiful Soup (lxml), one page after another | 1,000 | 14.9 s |

Same pages, same items, same proxy. The gap is not parsing speed but waiting: requests waits for each page before asking for the next, while Scrapy keeps 16 downloads in flight. Parsing is the smaller part either way: to parse and extract one page, Beautiful Soup with lxml took 12.91 ms and parsel, the selector library inside Scrapy, 1.58 ms.

## The requests + Beautiful Soup version

```python
import requests
from bs4 import BeautifulSoup

PROXY = "http://USER:PASS@HOST:PORT"   # your QuanticData proxy line
URLS = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 51)]

s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}
items = []
for u in URLS:
    soup = BeautifulSoup(s.get(u, timeout=30).content, "lxml")
    for a in soup.select("article.product_pod"):
        items.append({"title": a.h3.a["title"],
                      "price": a.select_one(".price_color").text})
print(len(items))   # 1000, in 14.9 s
```

Twelve lines, no project, easy to read and to debug in a notebook. Everything else (retries, saving, resuming after a crash, concurrency) is yours to write. The details of the parsing side are in [Beautiful Soup web scraping](https://quanticdata.io/blog/beautifulsoup-web-scraping/).

## The Scrapy version

```python
import scrapy
from scrapy.crawler import CrawlerProcess

class Books(scrapy.Spider):
    name = "books"
    custom_settings = {"CONCURRENT_REQUESTS": 16,
                       "CONCURRENT_REQUESTS_PER_DOMAIN": 16,
                       "FEEDS": {"books.json": {"format": "json"}}}

    async def start(self):
        for u in URLS:
            yield scrapy.Request(u, meta={"proxy": PROXY})

    def parse(self, r):
        for b in r.css("article.product_pod"):
            yield {"title": b.css("h3 a::attr(title)").get(),
                   "price": b.css("p.price_color::text").get()}

process = CrawlerProcess()
process.crawl(Books)
process.start()     # 1,000 items in 4.6 s
```

More setup, but the output file, retries of failed requests, concurrency, throttling and the request queue come with it. The proxy rides on each request in `meta`; to set it once for every request, see [Scrapy proxy middleware](https://quanticdata.io/blog/scrapy-proxy-middleware/). We used `async def start()`, the entry point of recent Scrapy versions; `start_requests()`, which older guides use, also ran in our test on 2.20.0.

Scrapy can also follow links instead of a list of URLs. Yield the items, then `response.follow(next_href, self.parse)`: started on page 1 with only the Next link to go on, our spider read the same 50 pages and 1,000 items. Each page reveals the next one, so a crawl like that runs one page at a time; when you can build the list of URLs up front, as above, Scrapy can fetch them in parallel.

## Side by side

|  | Beautiful Soup (+ requests) | Scrapy |
| --- | --- | --- |
| What it is | An HTML/XML parser | A crawling framework |
| Downloads pages | No, requests or httpx does | Yes |
| Concurrency | Yours to add (threads or asyncio) | Built in (16 in our run) |
| Selectors | find, find_all, CSS | CSS and XPath (parsel) |
| Retries, throttling, link following | Yours to write | Built in, set in settings |
| Export to JSON or CSV | Yours to write | FEEDS setting |
| JavaScript pages | No | No, unless you add [scrapy-playwright](https://quanticdata.io/blog/how-to-use-scrapy-playwright/) |
| Learning curve | Minutes | An afternoon |
| Best for | Scripts, notebooks, a few hundred pages | Whole sites, scheduled crawls, large volumes |

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=scrapy-vs-beautifulsoup) · [Explore Rotating proxies for scraping from $0.50/GB](https://quanticdata.io/rotating-proxies/)

## How to choose in three questions

1. **How many pages?** Up to a few hundred, run once: requests and Beautiful Soup. Thousands, or the same crawl every day: Scrapy, whose concurrency and retries you would otherwise rebuild by hand.
2. **Do you know the URLs?** A fixed list runs fine in a loop. If the scraper has to discover pages by following links, Scrapy's scheduler and link following save the most code.
3. **Where does the data go?** Into a notebook or a pandas DataFrame: Beautiful Soup keeps everything in one script. Into files or a database on a schedule: Scrapy's feed exports and item pipelines are built for it.

## Using them together

Nothing stops you from parsing inside a Scrapy callback with Beautiful Soup: `BeautifulSoup(response.body, "lxml")`. We ran it that way on 5 pages through the same proxy: 100 items, written to books.json by the FEEDS setting. You keep Scrapy's downloading and scheduling and lose only the faster selectors. And when you outgrow the requests loop but not the code, httpx or aiohttp can run the same Beautiful Soup parsing concurrently; the comparison of all the options is in [Python web scraping libraries](https://quanticdata.io/blog/python-web-scraping-libraries/).

## What about Selenium?

Selenium (and Playwright) drive a real browser, which neither Scrapy nor Beautiful Soup does. Use one when the data appears only after JavaScript runs. It is far heavier per page, so the usual pattern is to render only the pages that need it and parse the HTML it returns with Beautiful Soup or parsel.

## Proxies for each approach

Scrapy's 16 parallel requests are 16 connections at once from your machine. QuanticData [mobile proxies](https://quanticdata.io/mobile-proxies/) and [residential proxies](https://quanticdata.io/residential-proxies/) take their settings from the username: the rotating gateway, or a session ID when a crawl needs one. The same proxy URL works in Scrapy's `meta` and in the `proxies` dictionary of requests. More on rotation settings: [rotating proxies](https://quanticdata.io/rotating-proxies/).

### Sources & further reading

- [Scrapy documentation](https://docs.scrapy.org/en/latest/)
- [Scrapy: settings, CONCURRENT_REQUESTS](https://docs.scrapy.org/en/latest/topics/settings.html#concurrent-requests)
- [Scrapy: feed exports](https://docs.scrapy.org/en/latest/topics/feed-exports.html)
- [Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)

## FAQ

Quick answers on scrapy vs beautifulsoup.

[Something else? Ask us](mailto:hello@quanticdata.io)

### Is Scrapy faster than Beautiful Soup?

For many pages, yes, because Scrapy downloads in parallel. In our test, Scrapy with 16 concurrent requests scraped 1,000 items from 50 pages in 4.6 s; requests with Beautiful Soup, one page at a time, took 14.9 s through the same proxy.

### Should I learn Beautiful Soup or Scrapy first?

Beautiful Soup with requests: it takes minutes and teaches selectors. Move to Scrapy when you need whole sites, retries, scheduling or export without writing them yourself.

### Can I use Beautiful Soup inside Scrapy?

Yes. In a callback, parse `response.body` with `BeautifulSoup(response.body, "lxml")`. Scrapy still downloads and schedules; you only replace its selectors.

### Can Scrapy or Beautiful Soup scrape JavaScript pages?

Not on their own: both read the HTML the server sends. Add scrapy-playwright to Scrapy, or render with Playwright or Selenium and pass the HTML to Beautiful Soup.

### How do I use a proxy with Scrapy and Beautiful Soup?

In Scrapy, set `meta={"proxy": "http://USER:PASS@HOST:PORT"}` on each request or use a downloader middleware. With Beautiful Soup, the proxy goes in the download call, for example the `proxies` dictionary of requests.

## Proxies for Scrapy and requests

Residential and mobile proxies for Scrapy and requests, with rotation and session settings in the username.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=scrapy-vs-beautifulsoup) · [Explore Rotating proxies for scraping from $0.50/GB](https://quanticdata.io/rotating-proxies/)

## Related reading

- [How to Use scrapy-playwright](https://quanticdata.io/blog/how-to-use-scrapy-playwright/): Guides · Install the download handler, flag requests to render with Playwright, wait for content, add proxies — and understand the throughput cost of a browser per page.
- [Direct Image URL: What It Is, How to Check](https://quanticdata.io/blog/direct-image-url/): SERP & search · A direct image URL answers with the image file: its Content-Type starts with image/. A page that displays the picture answers with text/html. Below: how to tell them apart in one curl command, how to get the direct link from a page or a photo host, and what our reverse image search API did with each kind of URL in tests on 20 November 2026.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/scrapy-vs-beautifulsoup/ · Site index for AI: https://quanticdata.io/llms.txt
