# Web Scraping Pagination in Python, 4 Ways

> Web scraping pagination in Python, four ways run for real: follow the Next link, loop page numbers, page through a JSON API, scroll with Playwright.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Web Scraping Pagination in Python, 4 Ways

# Web scraping pagination in Python: next links, page numbers, APIs and infinite scroll

Web scraping · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![Four kinds of pagination scraped with Python: follow the Next link until it disappears, loop numbered page URLs until a page comes back empty, read a JSON API until has_next is false, and scroll with Playwright until no new items load; the card shows our results, 1,000 books from 50 pages and 100 quotes from each of the other three.](https://quanticdata.io/blog/web-scraping-pagination/hero.webp?v=20261120h)

On this page

- [The four kinds of pagination](https://quanticdata.io/blog/web-scraping-pagination/#the-four-kinds-of-pagination)
- [1. Follow the Next link](https://quanticdata.io/blog/web-scraping-pagination/#1-follow-the-next-link)
- [2. Loop the page numbers, and stop on an empty page](https://quanticdata.io/blog/web-scraping-pagination/#2-loop-the-page-numbers-and-stop-on-an-empty-page)
- [3. Page through the JSON API](https://quanticdata.io/blog/web-scraping-pagination/#3-page-through-the-json-api)
- [4. Infinite scroll with Playwright](https://quanticdata.io/blog/web-scraping-pagination/#4-infinite-scroll-with-playwright)
- [Pagination with Scrapy](https://quanticdata.io/blog/web-scraping-pagination/#pagination-with-scrapy)
- [When pagination is not working](https://quanticdata.io/blog/web-scraping-pagination/#when-pagination-is-not-working)
- [Which proxy for long paginated runs](https://quanticdata.io/blog/web-scraping-pagination/#which-proxy-for-long-paginated-runs)

To scrape paginated pages in Python, find which kind of pagination the site uses, then loop until its own stop signal: follow the Next link until there is none, request page numbers until a page has no items, call the JSON API until `has_next` is false, or scroll in a browser until no new items appear.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=web-scraping-pagination) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

We ran all four on 20 November 2026 against books.toscrape.com and quotes.toscrape.com, two public sites made for scraping practice, with Python 3.12.10, requests, Beautiful Soup and Playwright, through QuanticData residential and mobile proxies. Each section shows the code we ran and what came back.

## The four kinds of pagination

| Kind | How to spot it | Loop | Stop when | Our run |
| --- | --- | --- | --- | --- |
| Next link | A "next" button with an href | Follow the href | No next link | 50 pages, 1,000 books |
| Page numbers | `?page=2` or `/page/2/` in the URL | Count up | A page with no items | 10 pages, 100 quotes |
| JSON API | An XHR call in the browser's Network tab | Count up the page parameter | `has_next` false (or an empty list) | 10 calls, 100 quotes |
| Infinite scroll | More items load as you scroll, no buttons | Scroll in a browser | The item count stops growing | 100 quotes |

## 1. Follow the Next link

The safest loop: the site tells you the next URL, so you never guess. Resolve the href against the current URL with `urljoin`, since it is usually relative.

```python
import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

PROXY = "http://USER-type-residential-country-us-session-p01-lifetime-30:PASS@residentialboson.quanticdata.io:10000"
s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}

url, books = "https://books.toscrape.com/catalogue/page-1.html", []
while url:
    r = s.get(url, timeout=30)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "lxml")
    books += [a["title"] for a in soup.select("article.product_pod h3 a")]
    nxt = soup.select_one("li.next > a")
    url = urljoin(url, nxt["href"]) if nxt else None

print(len(books))   # 1000, from 50 pages
```

Through one Residential Basic session, set in the username, it read 50 pages and 1,000 books. The relative href on page 1 is `page-2.html`; without urljoin, the second request goes to the wrong place.

## 2. Loop the page numbers, and stop on an empty page

When the URL carries the page number, you can count up. The trap is the stop condition. quotes.toscrape.com has 10 pages; we asked for page 11 and it did **not** return 404:

```
page = 1
while True:
    r = s.get(f"https://quotes.toscrape.com/page/{page}/", timeout=30)
    items = BeautifulSoup(r.text, "lxml").select("div.quote")
    if r.status_code != 200 or not items:
        break                  # page 11: status 200, 0 quotes
    # ... save items ...
    page += 1
```

A loop that waits for a 404 would never end on this site. Check for items, keep a hard upper limit, and stop if a page repeats the previous one: some sites send the last page again for any number past the end.

## 3. Page through the JSON API

Many sites load their lists from an API, which is cleaner to scrape than HTML. Open the browser's developer tools, Network tab, filter by Fetch/XHR, and change page: the call shows up. On quotes.toscrape.com it is `/api/quotes?page=N`, and it says when to stop:

```
page, quotes = 1, []
while True:
    d = s.get("https://quotes.toscrape.com/api/quotes",
              params={"page": page}, timeout=30).json()
    quotes += d["quotes"]
    if not d["has_next"]:
        break
    page += 1
print(len(quotes))   # 100, from 10 calls
```

The response had five keys: `has_next`, `page`, `quotes`, `tag` and `top_ten_tags`. Other APIs use a cursor instead of a page number (a `next_cursor` or `after` value you send back); the loop is the same, and it ends when the cursor comes back empty.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=web-scraping-pagination) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## 4. Infinite scroll with Playwright

quotes.toscrape.com/scroll loads ten more quotes each time you reach the bottom. A browser does the scrolling; the loop stops when the count stops growing:

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    b = p.chromium.launch(proxy={"server": "http://mobile.quanticdata.io:8080",
                                 "username": "USER-country-us-type-mobile-sid-p04-ttl-30m",
                                 "password": "PASS"})
    page = b.new_page()
    page.goto("https://quotes.toscrape.com/scroll")
    page.wait_for_selector("div.quote")
    prev = -1
    while True:
        n = page.locator("div.quote").count()
        if n == prev:
            break
        prev = n
        page.mouse.wheel(0, 20000)
        try:
            page.wait_for_function(f"document.querySelectorAll('div.quote').length > {n}", timeout=5000)
        except Exception:
            pass
    print(prev)   # 100
    b.close()
```

Through a mobile line it collected all 100 quotes, and the next scroll added nothing. Simpler still: infinite scroll is usually an API underneath. `page.expect_response()` caught the call the scroll made, `/api/quotes?page=1` with 10 quotes, and from there you can drop the browser and use the loop of section 3. More waiting patterns are in [Playwright for scraping](https://quanticdata.io/blog/how-to-use-playwright-for-scraping/).

## Pagination with Scrapy

Scrapy follows the Next link for you: yield the items, then `response.follow(next_href, self.parse)`. Our spider crawled the same 50 catalogue pages and 1,000 items with no loop of ours. Crawling beyond one list is covered in [how to web crawl with Python](https://quanticdata.io/blog/how-to-web-crawl-python/).

## When pagination is not working

- **The loop never ends.** The site returns 200 past the last page (as page 11 did). Stop on no items, a repeated page, or a maximum page count.
- **Page 2 equals page 1.** The page parameter lives in a cookie, a form field or the API, not in the URL you are changing. Check the Network tab.
- **Next is a button with no href.** It calls JavaScript; use the API it calls, or Playwright.
- **Later pages fail.** You are going too fast: slow down and honour Retry-After (see [429 Too Many Requests](https://quanticdata.io/blog/429-too-many-requests-web-scraping/)).

## Which proxy for long paginated runs

A list of 50 pages is 50 requests, so keep a modest pace and a pause between pages. With QuanticData [residential proxies](https://quanticdata.io/residential-proxies/), the country, the rotating gateway or a session ID are settings in the username; we ran the Next-link loop on one session, which suits lists that depend on cookies. [mobile proxies](https://quanticdata.io/mobile-proxies/) put the scroll test on a carrier IP. To skip the loop entirely, our [web scraping API](https://quanticdata.io/web-scraping-api/) returns the page for you; to try the code here first, take the [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/). A first scraper end to end: [web scraping with Python](https://quanticdata.io/blog/how-to-web-scraping-using-python/).

### Sources & further reading

- [Python docs: urllib.parse.urljoin](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin)
- [Beautiful Soup documentation: CSS selectors](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors)
- [Scrapy tutorial: following links](https://docs.scrapy.org/en/latest/intro/tutorial.html#following-links)
- [Playwright for Python: Mouse](https://playwright.dev/python/docs/api/class-mouse)
- [MDN: Link header and rel=next](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link)

## FAQ

Quick answers on web scraping pagination.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do you handle pagination in web scraping?

Find the kind first: a Next link, numbered URLs, a JSON API or infinite scroll. Then loop with the stop signal that kind gives you: no Next link, a page with no items, `has_next` false, or a scroll that adds nothing.

### How do I scrape multiple pages with Beautiful Soup?

Fetch each page with requests, parse it, collect items, then find the next link with `soup.select_one("li.next > a")` and resolve it with `urljoin`. Stop when there is none. Our loop read 50 pages and 1,000 books.

### Why does my pagination loop never stop?

Many sites answer 200 for pages past the end. quotes.toscrape.com returned 200 with zero quotes for page 11. Stop when a page has no items or repeats the last one, and set a maximum.

### How do I scrape infinite scroll pages in Python?

Scroll in Playwright with `page.mouse.wheel()` and stop when the number of items stops growing; we got all 100 quotes. Or find the API the scroll calls and page through it without a browser.

### Should I use the same session for every page?

Use one session when the list depends on cookies or a login; set it with a session ID in the proxy username.

## Proxies for long paginated runs

Residential and mobile proxies for requests, Scrapy and Playwright, with country, rotation and session settings in the username.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=web-scraping-pagination) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

- [Web Scraping Project Ideas, Started for Real](https://quanticdata.io/blog/web-scraping-project-ideas/): Web scraping · Five web scraping project ideas, each one started for real on 20 November 2026: a price tracker, a library release dashboard, a question analysis, an open-source landscape and a topic timeline, with the code, the first run and what to build next.
- [Retry-After Header: Read It, Wait, Retry](https://quanticdata.io/blog/retry-after-header/): Web scraping · Retry-After tells a client how long to wait before the next request, as a number of seconds or as an HTTP date. Below: both formats, a Python and a Node helper we ran against a real 429 on 20 November 2026 (each waited 10 seconds and then got 200), the rate-limit headers of the QuanticData API, and how one batch job replaced 20 calls.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/web-scraping-pagination/ · Site index for AI: https://quanticdata.io/llms.txt
