Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Web scraping with Java: HttpClient, jsoup and a proxy, run on Java 25

A Java web scraper on Java 25: the HttpClient built into the JDK downloads each page, optionally through a proxy, jsoup 1.24.1 selects every article.product_pod card and reads title, price and rating, the li.next link leads to the next page and Files.write saves books.csv; the card shows the real run: 60 books over 3 pages, the same directly and through residential and mobile proxies.
A Java web scraper on Java 25: the HttpClient built into the JDK downloads each page, optionally through a proxy, jsoup 1.24.1 selects every article.product_pod card and reads title, price and rating, the li.next link leads to the next page and Files.write saves books.csv; the card shows the real run: 60 books over 3 pages, the same directly and through residential and mobile proxies.

To scrape a website with Java, download the page with java.net.http.HttpClient, which ships with the JDK, and parse the HTML with jsoup and its CSS selectors. Our 86-line scraper ran on Java 25 with jsoup 1.24.1, collected 60 books from 3 pages into a CSV file, and worked the same through a proxy.

Everything below ran on 20 November 2026 (UTC) on Windows 11 against books.toscrape.com, a sandbox made for scraping practice, and against one page of our own site. The output is copied from those runs.

Which tools, and the versions we ran

JobToolVersion in this run
JDKEclipse Temurin JDK25.0.5.1
Download pagesjava.net.http.HttpClientpart of the JDK since Java 11
Parse HTMLjsoup1.24.1
Build and runApache Maven with exec-maven-plugin3.9.16 and 3.7.0

jsoup can also download pages by itself, and a section below shows that. We keep the JDK client for the main scraper because it gives you timeouts, redirects, proxies and parallel requests with no extra dependency.

Set up the project

Create a folder with this pom.xml and put the Java files in src/main/java:

<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>java-scraper</artifactId>
  <version>1.0</version>

  <properties>
    <maven.compiler.release>25</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.24.1</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.7.0</version>
      </plugin>
    </plugins>
  </build>
</project>

The scraper: HttpClient downloads, jsoup parses

import java.net.Authenticator;
import java.net.InetSocketAddress;
import java.net.PasswordAuthentication;
import java.net.ProxySelector;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

public class Scraper {
    record Book(String title, String price, String rating, String url) {}

    static HttpClient newClient() {
        HttpClient.Builder builder = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(20))
            .followRedirects(HttpClient.Redirect.NORMAL);
        String proxy = System.getenv("PROXY");   // http://USER:PASS@HOST:PORT, or unset for a direct connection
        if (proxy != null && !proxy.isBlank()) {
            URI p = URI.create(proxy);
            String[] cred = p.getUserInfo().split(":", 2);
            builder.proxy(ProxySelector.of(new InetSocketAddress(p.getHost(), p.getPort())));
            builder.authenticator(new Authenticator() {
                @Override protected PasswordAuthentication getPasswordAuthentication() {
                    if (getRequestorType() != RequestorType.PROXY) return null;
                    return new PasswordAuthentication(cred[0], cred[1].toCharArray());
                }
            });
        }
        return builder.build();
    }

    static Document get(HttpClient client, String url) throws Exception {
        HttpRequest request = HttpRequest.newBuilder(URI.create(url))
            .header("User-Agent", "my-scraper/1.0 (+https://example.com/bot)")
            .timeout(Duration.ofSeconds(30))
            .GET().build();
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
        if (response.statusCode() != 200) throw new IllegalStateException(response.statusCode() + " for " + url);
        return Jsoup.parse(response.body(), url);
    }

    static String csv(String s) {
        return "\"" + s.replace("\"", "\"\"") + "\"";
    }

    public static void main(String[] args) throws Exception {
        // Let the JDK answer a proxy's Basic challenge on HTTPS tunnels (off by default since Java 8u111).
        System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
        HttpClient client = newClient();

        List<Book> books = new ArrayList<>();
        String url = "https://books.toscrape.com/catalogue/page-1.html";
        for (int page = 1; page <= 3 && url != null; page++) {
            Document doc = get(client, url);
            for (Element card : doc.select("article.product_pod")) {
                Element link = card.selectFirst("h3 a");
                books.add(new Book(
                    link.attr("title"),
                    card.selectFirst(".price_color").text().replaceAll("[^\\d.]", ""),
                    card.selectFirst("p.star-rating").className().replace("star-rating", "").trim(),
                    link.absUrl("href")));
            }
            System.out.printf("page %d: %d books so far%n", page, books.size());
            Element next = doc.selectFirst("li.next a");
            url = next == null ? null : next.absUrl("href");
            Thread.sleep(1000);   // one page a second is plenty
        }

        List<String> lines = new ArrayList<>();
        lines.add("title,price,rating,url");
        for (Book b : books) lines.add(String.join(",", csv(b.title()), b.price(), b.rating(), b.url()));
        Files.write(Path.of("books.csv"), lines, StandardCharsets.UTF_8);

        for (Book b : books.subList(0, 3)) System.out.println(b.title() + " | " + b.price() + " | " + b.rating());
        System.out.println(books.size() + " books written to books.csv");
    }
}
$ mvn -q compile exec:java -Dexec.mainClass=Scraper
page 1: 20 books so far
page 2: 40 books so far
page 3: 60 books so far
A Light in the Attic | 51.77 | Three
Tipping the Velvet | 53.74 | One
Soumission | 50.10 | One
60 books written to books.csv

The first lines of books.csv:

title,price,rating,url
"A Light in the Attic",51.77,Three,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
"Tipping the Velvet",53.74,One,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
"Soumission",50.10,One,https://books.toscrape.com/catalogue/soumission_998/index.html

How it works

  • The User-Agent names the scraper and gives a URL where a site owner can read about it. httpbin.org, which echoes request headers, returned exactly that string.
  • HttpClient.send returns the status and the body. It does not throw on a 404 or a 500, so the code checks the status itself.
  • Jsoup.parse(body, url) builds the document. The second argument is the base address, which is what lets absUrl("href") turn a relative link into a full one.
  • doc.select("article.product_pod") returns every book card. Inside a card, selectFirst finds one element: the title is in the link's title attribute, where it is not shortened, and the rating is the second class name of p.star-rating. More on choosing selectors in CSS selectors and XPath.
  • Pagination follows the link in li.next a until there is none or the page limit is reached, one page a second. Other layouts are covered in pagination patterns.
  • The record Book gives each row named, typed fields; the small csv method quotes the title so that commas inside it do not break the file.

Running it through a proxy

The JDK client takes the proxy in two parts: proxy(ProxySelector.of(...)) for the host and port, and an Authenticator for the username and password. The scraper reads one environment variable, so the same class runs both ways:

PROXY="http://USER-type-residential-country-us-session-jv01-lifetime-30:PASS@residentialboson.quanticdata.io:10000" mvn -q compile exec:java -Dexec.mainClass=Scraper
PROXY="http://USER-country-us-sid-jv02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080" mvn -q compile exec:java -Dexec.mainClass=Scraper
PROXY="http://USER-country-us-type-mobile-sid-jv03-ttl-30m:PASS@mobile.quanticdata.io:8080" mvn -q compile exec:java -Dexec.mainClass=Scraper
NetworkExit location (ipinfo.io)Result
Residential BasicNew York City, New York, US60 books over 3 pages
Residential PremiumBridgeport, Connecticut, US60 books over 3 pages
MobileAustin, Texas, US60 books over 3 pages

One line in main matters whenever an HTTPS site is reached through a proxy that asks for a username and password: System.setProperty("jdk.http.auth.tunneling.disabledSchemes", ""). The JDK ships with that property set to Basic in conf/net.properties, which means it does not send a username and password when it opens an HTTPS tunnel through a proxy. With the property left alone, the same client returned status 407, the answer explained in 407 Proxy Authentication Required; with it set to an empty value at the start of main, before the client is created, every run above returned 200. You can pass it on the command line instead, as -Djdk.http.auth.tunneling.disabledSchemes=.

The session name and its length are set in the username: use one name per run. To see where the requests leave from, ask an IP service through the same client:

for (int i = 1; i <= 3; i++) {
    HttpResponse<String> r = client.send(req("https://ipinfo.io/json"), HttpResponse.BodyHandlers.ofString());
    System.out.printf("request %d: %d %s | %s, %s, %s%n", i, r.statusCode(), field(r.body(), "ip"),
        field(r.body(), "city"), field(r.body(), "region"), field(r.body(), "country"));
}
HttpResponse<String> r = client.send(req("https://httpbin.org/headers"), HttpResponse.BodyHandlers.ofString());
System.out.println("User-Agent seen by the site: " + field(r.body(), "User-Agent"));
request 1: 200 x.x.x.x | New York City, New York, US
request 2: 200 x.x.x.x | New York City, New York, US
request 3: 200 x.x.x.x | New York City, New York, US
User-Agent seen by the site: my-scraper/1.0 (+https://example.com/bot)

In this run the three requests of each session reported the same exit location (addresses masked here). field is a three-line helper that reads one value out of the JSON with a regular expression, and req builds the same request as in the scraper.

jsoup on its own, with a proxy

For a quick job jsoup can fetch the page too. A session holds the proxy, the credentials and the user agent, and each newRequest reuses them:

import java.net.URI;
import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class JsoupOnly {
    public static void main(String[] args) throws Exception {
        System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
        URI p = URI.create(System.getenv("PROXY"));          // http://USER:PASS@HOST:PORT
        String[] cred = p.getUserInfo().split(":", 2);

        Connection session = Jsoup.newSession()
            .proxy(p.getHost(), p.getPort())
            .auth(auth -> auth.isProxy() ? auth.credentials(cred[0], cred[1]) : null)
            .userAgent("my-scraper/1.0 (+https://example.com/bot)")
            .timeout(30_000);

        for (String url : args) {
            Document doc = session.newRequest(url).get();
            System.out.println(url);
            System.out.println("  title: " + doc.title());
            System.out.println("  h1:    " + doc.selectFirst("h1").text());
            System.out.println("  links: " + doc.select("a[href]").size());
        }
    }
}

Run through Residential Basic against the sandbox and against our own residential proxies page:

$ mvn -q compile exec:java -Dexec.mainClass=JsoupOnly -Dexec.args="https://books.toscrape.com/ https://quanticdata.io/residential-proxies/"
https://books.toscrape.com/
  title: All products | Books to Scrape - Sandbox
  h1:    All products
  links: 94
https://quanticdata.io/residential-proxies/
  title: Residential Proxies from $0.80/GB | QuanticData
  h1:    Residential Proxies: Real Home IPs in 228 Countries
  links: 482

The same system property applies here, because jsoup sends its requests with the JDK's networking classes.

Several pages at once with sendAsync

When you already know the addresses, sendAsync starts the requests without waiting for each answer and returns a CompletableFuture per page:

long start = System.nanoTime();
List<CompletableFuture<String>> jobs = new ArrayList<>();
for (int page = 1; page <= 5; page++) {
    String url = "https://books.toscrape.com/catalogue/page-" + page + ".html";
    jobs.add(client.sendAsync(req(url), HttpResponse.BodyHandlers.ofString())
        .thenApply(r -> r.statusCode() + " " + Jsoup.parse(r.body()).select("article.product_pod").size() + " books  " + url));
}
for (CompletableFuture<String> job : jobs) System.out.println(job.get());
System.out.printf("5 pages in %.1f s%n", (System.nanoTime() - start) / 1e9);
200 20 books  https://books.toscrape.com/catalogue/page-1.html
200 20 books  https://books.toscrape.com/catalogue/page-2.html
200 20 books  https://books.toscrape.com/catalogue/page-3.html
200 20 books  https://books.toscrape.com/catalogue/page-4.html
200 20 books  https://books.toscrape.com/catalogue/page-5.html
5 pages in 2.2 s

That is one run through one Residential Basic session. Keep the number of requests in flight small and fixed: five pages of a practice site is fine, hundreds at once against a real site is not. If a site answers 429 or 403, slow down or stop.

JavaScript pages

An HTTP client receives the HTML the server sends, and jsoup does not run scripts. quotes.toscrape.com has a JavaScript version of its home page; counting div.quote in each:

0 quotes in https://quotes.toscrape.com/js/
10 quotes in https://quotes.toscrape.com/

When a page shows data in the browser and jsoup finds none, the data arrives by JavaScript. Look in the browser's network tab for the JSON request the page makes and fetch that with HttpClient, or render the page with a browser tool such as Selenium or Playwright for Java and pass the resulting HTML to Jsoup.parse. The idea is the same in every language, and scraping dynamic pages walks through it.

Java next to other languages

LanguageDownloadParseOur guide
JavaHttpClientjsoupthis page
PythonrequestsBeautiful Soupweb scraping with Python
Node.jsfetchCheerioCheerio in Node.js
PerlHTTP::TinyHTML::TokeParserweb scraping with Perl
PowerShellInvoke-WebRequestRegular expressionsweb scraping with PowerShell

The same scraper as a compiled program with no JVM is in web scraping with Rust.

Keep to each site's terms and robots.txt, and pause between requests. Our residential proxies and mobile proxies plug into HttpClient and jsoup as shown, with a Residential Basic free trial; our web scraping API returns rendered pages from one call when a site needs a browser.

Sources & further reading

FAQ

Quick answers on web scraping with java.

Something else? Ask us

How do I scrape a website with Java?

Download the page with java.net.http.HttpClient, which is part of the JDK since Java 11, parse the HTML with jsoup and its CSS selectors, and write the fields to a file. Our 86-line scraper did that on Java 25 with jsoup 1.24.1 and collected 60 books from 3 pages.

Which Java library is used for web scraping?

jsoup is the usual choice for pages whose data is in the HTML: it parses real-world markup and selects elements with CSS selectors. It can also download pages itself. For requests, the JDK's own HttpClient needs no dependency. Pages that fill in their data with JavaScript need a browser tool or the JSON the page loads.

How do I use a proxy with Java HttpClient?

Pass ProxySelector.of(new InetSocketAddress(host, port)) to HttpClient.newBuilder().proxy(...) and supply the username and password with an Authenticator. For HTTPS sites also set jdk.http.auth.tunneling.disabledSchemes to an empty value, because the JDK does not answer a Basic challenge on a tunnel by default.

Why does Java return 407 through an authenticated proxy on HTTPS?

The JDK ships with jdk.http.auth.tunneling.disabledSchemes=Basic in conf/net.properties, so it does not send Basic credentials when it opens an HTTPS tunnel. In our test the same client returned status 407 without the property and 200 with it set to an empty value.

Can jsoup scrape JavaScript pages?

jsoup parses the HTML the server sends and does not run scripts. On the JavaScript version of quotes.toscrape.com it found 0 quotes, against 10 on the static version. Fetch the JSON the page loads, or render the page with a browser tool first, then hand the HTML to jsoup.

Is Java good for web scraping?

It fits well where the rest of the system is on the JVM: the HTTP client is in the JDK, jsoup covers parsing, records give typed rows, and sendAsync runs several requests at once. In our run five pages came back in 2.2 seconds through one residential session.

A home or carrier IP for your Java scraper

Residential and mobile proxies with country targeting and named sessions, ready for HttpClient and jsoup.

Related reading