Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Web scraping pagination in Python: next links, page numbers, APIs and infinite scroll

Four kinds of pagination scraped with Python: follow the Next link until it disappears, loop numbered page URLs until a page comes back empty, read a JSON API until has_next is false, and scroll with Playwright until no new items load; the card shows our results, 1,000 books from 50 pages and 100 quotes from each of the other three.
Four kinds of pagination scraped with Python: follow the Next link until it disappears, loop numbered page URLs until a page comes back empty, read a JSON API until has_next is false, and scroll with Playwright until no new items load; the card shows our results, 1,000 books from 50 pages and 100 quotes from each of the other three.

To scrape paginated pages in Python, find which kind of pagination the site uses, then loop until its own stop signal: follow the Next link until there is none, request page numbers until a page has no items, call the JSON API until has_next is false, or scroll in a browser until no new items appear.

We ran all four on 20 November 2026 against books.toscrape.com and quotes.toscrape.com, two public sites made for scraping practice, with Python 3.12.10, requests, Beautiful Soup and Playwright, through QuanticData residential and mobile proxies. Each section shows the code we ran and what came back.

The four kinds of pagination

KindHow to spot itLoopStop whenOur run
Next linkA "next" button with an hrefFollow the hrefNo next link50 pages, 1,000 books
Page numbers?page=2 or /page/2/ in the URLCount upA page with no items10 pages, 100 quotes
JSON APIAn XHR call in the browser's Network tabCount up the page parameterhas_next false (or an empty list)10 calls, 100 quotes
Infinite scrollMore items load as you scroll, no buttonsScroll in a browserThe item count stops growing100 quotes

The safest loop: the site tells you the next URL, so you never guess. Resolve the href against the current URL with urljoin, since it is usually relative.

import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

PROXY = "http://USER-type-residential-country-us-session-p01-lifetime-30:PASS@residentialboson.quanticdata.io:10000"
s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}

url, books = "https://books.toscrape.com/catalogue/page-1.html", []
while url:
    r = s.get(url, timeout=30)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "lxml")
    books += [a["title"] for a in soup.select("article.product_pod h3 a")]
    nxt = soup.select_one("li.next > a")
    url = urljoin(url, nxt["href"]) if nxt else None

print(len(books))   # 1000, from 50 pages

Through one Residential Basic session, set in the username, it read 50 pages and 1,000 books. The relative href on page 1 is page-2.html; without urljoin, the second request goes to the wrong place.

2. Loop the page numbers, and stop on an empty page

When the URL carries the page number, you can count up. The trap is the stop condition. quotes.toscrape.com has 10 pages; we asked for page 11 and it did not return 404:

page = 1
while True:
    r = s.get(f"https://quotes.toscrape.com/page/{page}/", timeout=30)
    items = BeautifulSoup(r.text, "lxml").select("div.quote")
    if r.status_code != 200 or not items:
        break                  # page 11: status 200, 0 quotes
    # ... save items ...
    page += 1

A loop that waits for a 404 would never end on this site. Check for items, keep a hard upper limit, and stop if a page repeats the previous one: some sites send the last page again for any number past the end.

3. Page through the JSON API

Many sites load their lists from an API, which is cleaner to scrape than HTML. Open the browser's developer tools, Network tab, filter by Fetch/XHR, and change page: the call shows up. On quotes.toscrape.com it is /api/quotes?page=N, and it says when to stop:

page, quotes = 1, []
while True:
    d = s.get("https://quotes.toscrape.com/api/quotes",
              params={"page": page}, timeout=30).json()
    quotes += d["quotes"]
    if not d["has_next"]:
        break
    page += 1
print(len(quotes))   # 100, from 10 calls

The response had five keys: has_next, page, quotes, tag and top_ten_tags. Other APIs use a cursor instead of a page number (a next_cursor or after value you send back); the loop is the same, and it ends when the cursor comes back empty.

4. Infinite scroll with Playwright

quotes.toscrape.com/scroll loads ten more quotes each time you reach the bottom. A browser does the scrolling; the loop stops when the count stops growing:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    b = p.chromium.launch(proxy={"server": "http://mobile.quanticdata.io:8080",
                                 "username": "USER-country-us-type-mobile-sid-p04-ttl-30m",
                                 "password": "PASS"})
    page = b.new_page()
    page.goto("https://quotes.toscrape.com/scroll")
    page.wait_for_selector("div.quote")
    prev = -1
    while True:
        n = page.locator("div.quote").count()
        if n == prev:
            break
        prev = n
        page.mouse.wheel(0, 20000)
        try:
            page.wait_for_function(f"document.querySelectorAll('div.quote').length > {n}", timeout=5000)
        except Exception:
            pass
    print(prev)   # 100
    b.close()

Through a mobile line it collected all 100 quotes, and the next scroll added nothing. Simpler still: infinite scroll is usually an API underneath. page.expect_response() caught the call the scroll made, /api/quotes?page=1 with 10 quotes, and from there you can drop the browser and use the loop of section 3. More waiting patterns are in Playwright for scraping.

Pagination with Scrapy

Scrapy follows the Next link for you: yield the items, then response.follow(next_href, self.parse). Our spider crawled the same 50 catalogue pages and 1,000 items with no loop of ours. Crawling beyond one list is covered in how to web crawl with Python.

When pagination is not working

  • The loop never ends. The site returns 200 past the last page (as page 11 did). Stop on no items, a repeated page, or a maximum page count.
  • Page 2 equals page 1. The page parameter lives in a cookie, a form field or the API, not in the URL you are changing. Check the Network tab.
  • Next is a button with no href. It calls JavaScript; use the API it calls, or Playwright.
  • Later pages fail. You are going too fast: slow down and honour Retry-After (see 429 Too Many Requests).

Which proxy for long paginated runs

A list of 50 pages is 50 requests, so keep a modest pace and a pause between pages. With QuanticData residential proxies, the country, the rotating gateway or a session ID are settings in the username; we ran the Next-link loop on one session, which suits lists that depend on cookies. mobile proxies put the scroll test on a carrier IP. To skip the loop entirely, our web scraping API returns the page for you; to try the code here first, take the Residential Basic free trial. A first scraper end to end: web scraping with Python.

Sources & further reading

FAQ

Quick answers on web scraping pagination.

Something else? Ask us

How do you handle pagination in web scraping?

Find the kind first: a Next link, numbered URLs, a JSON API or infinite scroll. Then loop with the stop signal that kind gives you: no Next link, a page with no items, has_next false, or a scroll that adds nothing.

How do I scrape multiple pages with Beautiful Soup?

Fetch each page with requests, parse it, collect items, then find the next link with soup.select_one("li.next > a") and resolve it with urljoin. Stop when there is none. Our loop read 50 pages and 1,000 books.

Why does my pagination loop never stop?

Many sites answer 200 for pages past the end. quotes.toscrape.com returned 200 with zero quotes for page 11. Stop when a page has no items or repeats the last one, and set a maximum.

How do I scrape infinite scroll pages in Python?

Scroll in Playwright with page.mouse.wheel() and stop when the number of items stops growing; we got all 100 quotes. Or find the API the scroll calls and page through it without a browser.

Should I use the same session for every page?

Use one session when the list depends on cookies or a login; set it with a session ID in the proxy username.

Proxies for long paginated runs

Residential and mobile proxies for requests, Scrapy and Playwright, with country, rotation and session settings in the username.

Related reading