Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Concurrency vs parallelism: the difference, measured on a real scraping job

One Python scraping job timed four ways on 20 November 2026: downloading 30 pages took 10.3 s one after another and 3.5 s with 10 threads; parsing 240 pages took 3.3 s with 8 threads and 0.8 s with 8 processes; the card says threads help the waiting part and processes help the computing part.
One Python scraping job timed four ways on 20 November 2026: downloading 30 pages took 10.3 s one after another and 3.5 s with 10 threads; parsing 240 pages took 3.3 s with 8 threads and 0.8 s with 8 processes; the card says threads help the waiting part and processes help the computing part.

Concurrency means a program makes progress on several tasks in overlapping periods, usually by switching between them while one is waiting. Parallelism means several tasks run at the same instant on different CPU cores. Concurrency is how a program is organised; parallelism is how it is executed. Waiting work gains from concurrency, computing work only from parallelism.

The usual explanation stops at an analogy about cooks and kitchens. We timed the difference on a job every scraper has, download pages and parse them, on 20 November 2026 in Python 3.12 through one QuanticData residential proxy line. The script and its output are below.

Concurrency vs parallelism in one table

ConcurrencyParallelism
MeaningSeveral tasks in progress over the same periodSeveral tasks executing at the same instant
Needs more than one CPU coreNoYes
Helps when the taskWaits: network, disk, databaseComputes: parsing, hashing, compression, maths
Python toolsthreading, ThreadPoolExecutor, asynciomultiprocessing, ProcessPoolExecutor
In our run30 downloads: 10.3 s in a row, 3.5 s with 10 threads240 parses: 3.5 s on one core, 0.8 s with 8 processes

The two are not opposites and a program can be both: eight processes, each downloading with ten threads, is concurrent and parallel at once.

I/O-bound and CPU-bound: the question to ask first

Before choosing a tool, ask what the program is doing while time passes.

  • I/O-bound: it is waiting for something outside the CPU. A page download is almost all waiting: the request travels, the server works, the answer travels back. During that wait the CPU is free.
  • CPU-bound: it is computing. Turning 50,000 characters of HTML into a tree and running selectors over it keeps one core busy from start to finish.

A scraper has both phases, which makes it a good test.

The test script

The target is books.toscrape.com, a shop built for scraping practice: 30 catalogue pages of 20 books each. The script downloads them twice (one after another, then with 10 threads), then parses 240 pages three ways (one after another, 8 threads, 8 processes). Python 3.12.10, Requests 2.34.2, BeautifulSoup 4.14.3 with the pure-Python html.parser, on a Windows 11 PC with 32 logical CPUs.

import os, sys, threading, time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import requests
from bs4 import BeautifulSoup

PROXY = os.environ["QD_PROXY"]   # http://USER:PASS@HOST:PORT of your proxy line
PROXIES = {"http": PROXY, "https": PROXY}
URLS = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 31)]

local = threading.local()

def fetch(url):
    if not hasattr(local, "session"):          # one keep-alive session per thread
        local.session = requests.Session()
        local.session.proxies = PROXIES
    r = local.session.get(url, timeout=30)
    r.raise_for_status()
    return r.text

def parse(html):
    soup = BeautifulSoup(html, "html.parser")
    return [(a["title"], p.get_text()) for a, p in
            zip(soup.select("article.product_pod h3 a"), soup.select("p.price_color"))]

def timed(label, fn):
    t = time.perf_counter(); out = fn(); dt = time.perf_counter() - t
    print(f"{label:<34}{dt:6.1f} s   {out}")
    return dt

if __name__ == "__main__":
    print("Python", sys.version.split()[0], "| logical CPUs:", os.cpu_count())
    pages = []
    def io_seq():
        pages[:] = [fetch(u) for u in URLS]; return f"{len(pages)} pages"
    def io_threads():
        with ThreadPoolExecutor(max_workers=10) as ex:
            got = list(ex.map(fetch, URLS))
        return f"{len(got)} pages"
    print("-- I/O-bound: download 30 pages through one proxy")
    timed("one after another", io_seq)
    timed("10 threads", io_threads)

    work = pages * 8          # 240 parse jobs, pure Python (html.parser)
    def cpu_seq():
        return f"{sum(len(parse(h)) for h in work)} books"
    def cpu_threads():
        with ThreadPoolExecutor(max_workers=8) as ex:
            return f"{sum(len(x) for x in ex.map(parse, work))} books"
    def cpu_procs():
        with ProcessPoolExecutor(max_workers=8) as ex:
            return f"{sum(len(x) for x in ex.map(parse, work, chunksize=10))} books"
    print("-- CPU-bound: parse 240 pages with html.parser")
    timed("one after another", cpu_seq)
    timed("8 threads", cpu_threads)
    timed("8 processes", cpu_procs)
Python 3.12.10 | logical CPUs: 32
-- I/O-bound: download 30 pages through one proxy
one after another                   10.3 s   30 pages
10 threads                           3.5 s   30 pages
-- CPU-bound: parse 240 pages with html.parser
one after another                    3.5 s   4800 books
8 threads                            3.3 s   4800 books
8 processes                          0.8 s   4800 books

Concurrency at work: the downloads

One after another, the 30 pages took 10.3 seconds. With 10 threads the same 30 pages took 3.5 seconds, about a third of the time, and the script returned 30 of 30 pages both ways.

Nothing ran faster. Each request took as long as before; the threads simply overlapped the waiting. While one thread waits for its answer, Python hands control to another, which sends its own request. This is concurrency, and it would work the same on a single core, because a waiting thread needs no CPU.

asyncio gets the same effect with one thread and an event loop instead of ten threads; the mechanics and a measured example are in async web scraping in Python. For a few dozen simultaneous requests, a thread pool around Requests is the smallest change to an existing script; the proxy settings it needs are in a proxy in Python Requests.

Parallelism at work: the parsing

The parsing phase tells the opposite story. One after another, 240 parses took 3.5 seconds. With 8 threads they took 3.3 seconds: no real gain. With 8 processes they took 0.8 seconds, more than four times faster, and all three returned the same 4,800 books.

The reason threads did not help is the global interpreter lock (GIL). In the standard build of Python 3.12 only one thread executes Python bytecode at a time. Threads can overlap waiting, because a thread that waits on the network releases the lock, but they cannot overlap computing: eight parsing threads take turns on one core.

Processes do not share that lock. ProcessPoolExecutor starts eight Python interpreters, each with its own, and the operating system places them on eight cores. That is parallelism. It has costs that threads do not: each process takes time to start, and every page and every result is copied between processes, which is why the script sends the work in chunks of 10 and why the gain is four times and not eight.

PhaseOne after anotherThreadsProcessesWhat helped
Download 30 pages (I/O-bound)10.3 s3.5 s with 10Not neededConcurrency
Parse 240 pages (CPU-bound)3.5 s3.3 s with 80.8 s with 8Parallelism

These are the timings of one run on one PC and one network on 20 November 2026; the shape of the result, not the exact seconds, is what carries over to other machines.

Concurrency vs parallelism vs multithreading vs async

The four words are often mixed up because two of them are ideas and two are tools.

TermWhat it isIn PythonConcurrentParallel
ConcurrencyAn idea: overlapping tasksThreads or asyncioYesNot necessarily
ParallelismAn idea: simultaneous executionProcessesYesYes
MultithreadingA tool: several threads in one processthreading, ThreadPoolExecutorYesNot for Python code under the GIL: 3.3 s against 3.5 s in our run
AsyncA tool: one thread, tasks that yield while waitingasyncio, awaitYesNo
MultiprocessingA tool: several processesmultiprocessing, ProcessPoolExecutorYesYes: 0.8 s against 3.5 s in our run

In languages without a global lock, such as Go or Java, threads can run in parallel, so multithreading there covers both columns. The distinction between the two ideas stays the same in every language.

Which to use in a scraper

  • Downloading: use concurrency. A thread pool or asyncio, with a fixed number of workers.
  • Parsing a lot of HTML in Python: use processes, or a parser written in C such as lxml, which does the heavy part outside the interpreter. The selectors themselves are in our BeautifulSoup guide.
  • Both: download with threads into a queue, parse in a process pool. Keep the two phases separate so each gets the tool that fits.
  • A handful of pages: a plain loop. In our run the sequential version already parsed 4,800 books in 3.5 seconds; concurrency adds moving parts that a small job does not need.

Concurrency is also a matter of manners

Ten threads mean ten requests in flight at the same site. The site we used exists for practice; a real one has to serve its visitors too. Choose the number of workers on purpose and keep it small, add a pause between requests when you crawl many pages of one site (pagination shows the loop), set timeouts and a bounded retry for transient errors (retries and timeouts in Python Requests), and if the site answers 429, lower the concurrency: that status is the site telling you its limit, as our post on rate limiting explains. The wider list of habits is in crawling a website politely.

Where the proxy comes in

A proxy does not change which tool fits: downloads stay I/O-bound. What matters is that every worker uses the proxy the same way. In the script each thread keeps one requests.Session with the proxy set, so its connection to the gateway is reused from page to page. The run above used one residential proxy line with the country set to the United States in the username. Plans and countries for residential proxies and mobile proxies are on their pages.

Sources & further reading

FAQ

Quick answers on concurrency vs parallelism.

Something else? Ask us

What is the difference between concurrency and parallelism?

Concurrency is several tasks making progress over the same period, usually by switching between them while one waits. Parallelism is several tasks executing at the same instant on different CPU cores. In our run concurrency cut 30 downloads from 10.3 s to 3.5 s; parallelism cut 240 parses from 3.5 s to 0.8 s.

Is multithreading concurrency or parallelism?

It is a tool for concurrency, and in some languages for parallelism too. In standard Python the global interpreter lock lets one thread run Python code at a time: 8 threads parsed our pages in 3.3 s against 3.5 s for a plain loop, while they cut the downloads to a third.

Is asyncio concurrent or parallel?

Concurrent, not parallel. An asyncio program runs in one thread and switches between tasks when one awaits the network. It suits I/O-bound work such as many HTTP requests, and does nothing for CPU-bound work.

What is the Python GIL?

The global interpreter lock: in the standard build of CPython only one thread executes Python bytecode at a time. A thread waiting on the network releases it, which is why threads speed up downloads and not parsing.

Should a web scraper use threads or processes?

Threads or asyncio for downloading, which is waiting; processes for heavy parsing, which is computing. In our test 10 threads downloaded 30 pages in 3.5 s, and 8 processes parsed 240 pages in 0.8 s.

Can a program be concurrent without being parallel?

Yes. A single-core machine, or a single asyncio thread, can keep many downloads in progress by switching between them. It is concurrent, and nothing in it runs at the same instant.

Residential and mobile proxies for your scraper

One gateway, the country in the username, and as many workers as your job politely needs.

Related reading