Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Perl web scraping with HTTP::Tiny and HTML::TokeParser, no CPAN installs

A Perl web scraper built only from modules that were already installed: HTTP::Tiny downloads each page over HTTPS, optionally through a proxy, HTML::TokeParser walks the article, a and p tags to read title, rating and price, and JSON::PP writes books.json; the card shows the real run: 60 books over 3 pages, the same through residential and mobile proxies.
A Perl web scraper built only from modules that were already installed: HTTP::Tiny downloads each page over HTTPS, optionally through a proxy, HTML::TokeParser walks the article, a and p tags to read title, rating and price, and JSON::PP writes books.json; the card shows the real run: 60 books over 3 pages, the same through residential and mobile proxies.

To scrape a website with Perl, download the page with HTTP::Tiny (part of core Perl), parse the HTML with HTML::TokeParser or another parser, and save the fields with JSON::PP. Our 48-line scraper used only modules that were already installed, collected 60 books from 3 pages, and ran the same through a proxy by adding one option.

Everything below ran on 20 November 2026 on Windows 11 with Perl 5.42.3, the perl that comes with Git for Windows, against books.toscrape.com, a sandbox made for scraping practice. Output is copied from those runs.

Which modules, and which ones you already have

JobModuleOn our Perl
Download pagesHTTP::Tiny0.096, installed
HTTPS for HTTP::TinyIO::Socket::SSL2.099, installed; HTTP::Tiny->can_ssl printed yes
Parse HTMLHTML::TokeParser (with HTML::Parser)3.85, installed
Write JSONJSON::PP4.16, installed
Download pages, the older wayLWP::UserAgent6.83 installed, but https:// failed: 501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)

Check your own Perl with perl -MHTTP::Tiny -e 'print $HTTP::Tiny::VERSION' for each module. If you can install from CPAN, Mojo::UserAgent and HTML::TreeBuilder are popular choices with CSS or tree selectors; the approach below works where you cannot install anything.

The scraper

#!/usr/bin/perl
use strict;
use warnings;
use HTTP::Tiny;
use HTML::TokeParser;
use JSON::PP;
use Encode qw(decode);

my $proxy = $ENV{PROXY};    # http://USER:PASS@HOST:PORT, or unset for a direct connection
my $http  = HTTP::Tiny->new(
    agent   => 'my-scraper/1.0 (+https://example.com/bot)',
    timeout => 30,
    ($proxy ? (proxy => $proxy, https_proxy => $proxy) : ()),
);

sub get_html {
    my ($url) = @_;
    my $res = $http->get($url);
    die "$res->{status} $res->{reason} for $url\n" unless $res->{success};
    return decode('UTF-8', $res->{content});
}

my @books;
my $url = 'https://books.toscrape.com/catalogue/page-1.html';
for my $page (1 .. 3) {
    my $p = HTML::TokeParser->new(\ get_html($url));
    my ($in_book, %book, $next);
    while (my $t = $p->get_tag('article', 'a', 'p', 'li')) {
        my ($tag, $attr) = @$t;
        if ($tag eq 'article' && ($attr->{class} // '') eq 'product_pod') { $in_book = 1; %book = (); }
        elsif ($in_book && $tag eq 'a' && $attr->{title}) { $book{title} = $attr->{title}; }
        elsif ($in_book && $tag eq 'p' && ($attr->{class} // '') =~ /^star-rating (\w+)/) { $book{rating} = $1; }
        elsif ($in_book && $tag eq 'p' && ($attr->{class} // '') eq 'price_color') {
            ($book{price} = $p->get_trimmed_text('/p')) =~ s/[^\d.]//g;
            push @books, {%book}; $in_book = 0;
        }
        elsif ($tag eq 'li' && ($attr->{class} // '') eq 'next') { my $a = $p->get_tag('a'); $next = $a->[1]{href}; }
    }
    printf "page %d: %d books so far\n", $page, scalar @books;
    last unless $next;
    $url = "https://books.toscrape.com/catalogue/$next";
}

open my $fh, '>:encoding(UTF-8)', 'books.json' or die $!;
print $fh JSON::PP->new->pretty->canonical->encode(\@books);
close $fh;
printf "%s | %s | %s\n", @{$books[$_]}{qw(title price rating)} for 0 .. 2;
print scalar(@books), " books written to books.json\n";
$ perl scrape.pl
page 1: 20 books so far
page 2: 40 books so far
page 3: 60 books so far
A Light in the Attic | 51.77 | Three
Tipping the Velvet | 53.74 | One
Soumission | 50.10 | One
60 books written to books.json

How it works

  • agent names the scraper and gives a URL where a site owner can read about it.
  • HTTP::Tiny returns a hash: success, status, reason, headers and content. It does not die on a 404, so the script checks success itself.
  • decode('UTF-8', ...) turns the bytes into Perl characters, so prices with the pound sign and accented titles stay intact. The :encoding(UTF-8) layer writes them back out correctly.
  • HTML::TokeParser reads the page as a stream of tags. get_tag('article', 'a', 'p', 'li') skips everything else; the attributes come as a hash, so $attr->{title} gives the full title even where the link text is shortened.
  • get_trimmed_text('/p') returns the text up to the closing tag; the regex keeps digits and the dot, so £51.77 becomes 51.77.
  • Pagination follows the link inside <li class="next"> until there is none or the page limit is reached.

A token parser is more code than CSS selectors, but it needs nothing extra and does not depend on whitespace or on the order of attributes, which is what most often breaks regular expressions on HTML.

Running it through a proxy

HTTP::Tiny takes the proxy in its constructor: proxy for http:// URLs and https_proxy for https:// URLs, with the username and password in the proxy URL. The script reads it from the environment, so the same file runs both ways:

PROXY="http://USER-type-residential-country-us-session-pl01-lifetime-30:PASS@residentialboson.quanticdata.io:10000" perl scrape.pl
PROXY="http://USER-country-us-sid-pl02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080" perl scrape.pl
PROXY="http://USER-country-us-type-mobile-sid-pl03-ttl-30m:PASS@mobile.quanticdata.io:8080" perl scrape.pl
NetworkExit location (ipinfo.io)Result
Residential BasicNew York City, New York, US60 books over 3 pages
Residential PremiumPalisades Park, New Jersey, US60 books over 3 pages
MobileFort George G Mead Junction, Maryland, US60 books over 3 pages

The username carries a session ID: use one per run. With a wrong proxy password the script stopped at the first page with 407 Proxy Authentication Required, the gateway's answer explained in 407 Proxy Authentication Required. Read the exit location yourself with a one-liner:

PROXY="http://USER:PASS@HOST:PORT" perl -MHTTP::Tiny -MJSON::PP -e 'my $h=HTTP::Tiny->new(proxy=>$ENV{PROXY}, https_proxy=>$ENV{PROXY}); my $r=$h->get("https://ipinfo.io/json"); my $j=decode_json($r->{content}); print "$r->{status} $j->{city}, $j->{region}, $j->{country}\n"'

JavaScript pages

Perl, like any HTTP client, gets the HTML the server sends. quotes.toscrape.com has a JavaScript version of its home page; counting quote blocks in each:

quote divs in /js/: 0
quote divs in /: 10

When a page shows data in the browser but your script finds nothing, the data is loaded by JavaScript. Look for the JSON request the page makes and fetch that with HTTP::Tiny and decode_json, or render the page with a browser tool. The same check works in web scraping with PowerShell and with Cheerio in Node.js.

Perl next to other languages

LanguageDownloadParseOur guide
PerlHTTP::TinyHTML::TokeParserthis page
PowerShellInvoke-WebRequestRegular expressionsweb scraping with PowerShell
Node.jsfetchCheerioCheerio in Node.js
PythonrequestsBeautiful Soupweb scraping with Python

Keep to each site's terms and pause between requests. Our residential proxies and mobile proxies plug into HTTP::Tiny as shown, with a Residential Basic free trial; our web scraping API returns rendered pages from one call when a site needs a browser.

Sources & further reading

FAQ

Quick answers on perl web scraping.

Something else? Ask us

How do I scrape a website with Perl?

Download the page with HTTP::Tiny, parse it with HTML::TokeParser (or HTML::TreeBuilder or Mojo::DOM if you can install from CPAN), and write the results with JSON::PP. Our 48-line script collected 60 books from 3 pages using only modules that were already installed.

Does HTTP::Tiny support HTTPS?

Yes, when IO::Socket::SSL is installed. HTTP::Tiny->can_ssl tells you whether it can; on our Perl it returned true, and every page in our test was https.

Why does LWP::UserAgent fail on https URLs?

LWP needs the separate LWP::Protocol::https module for https. Without it, our request returned 501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed). Install that module, or use HTTP::Tiny.

How do I use a proxy in Perl?

With HTTP::Tiny, pass proxy => "http://USER:PASS@HOST:PORT" and https_proxy => "http://USER:PASS@HOST:PORT" to new. We ran the same scraper through residential and mobile proxies this way and got 60 books each time.

Can Perl scrape JavaScript websites?

Not by itself: HTTP::Tiny gets the HTML the server sends, and on a JavaScript page that HTML has no data (0 quotes against 10 on the static version in our test). Fetch the JSON the page loads, or render the page with a browser first.

A home or carrier IP for your Perl scripts

Residential and mobile proxies with country targeting and sticky sessions, ready for HTTP::Tiny proxy and https_proxy.

Related reading