To scrape a website with Perl, download the page with HTTP::Tiny (part of core Perl), parse the HTML with HTML::TokeParser or another parser, and save the fields with JSON::PP. Our 48-line scraper used only modules that were already installed, collected 60 books from 3 pages, and ran the same through a proxy by adding one option.
Everything below ran on 20 November 2026 on Windows 11 with Perl 5.42.3, the perl that comes with Git for Windows, against books.toscrape.com, a sandbox made for scraping practice. Output is copied from those runs.
Which modules, and which ones you already have
| Job | Module | On our Perl |
|---|---|---|
| Download pages | HTTP::Tiny | 0.096, installed |
| HTTPS for HTTP::Tiny | IO::Socket::SSL | 2.099, installed; HTTP::Tiny->can_ssl printed yes |
| Parse HTML | HTML::TokeParser (with HTML::Parser) | 3.85, installed |
| Write JSON | JSON::PP | 4.16, installed |
| Download pages, the older way | LWP::UserAgent | 6.83 installed, but https:// failed: 501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed) |
Check your own Perl with perl -MHTTP::Tiny -e 'print $HTTP::Tiny::VERSION' for each module. If you can install from CPAN, Mojo::UserAgent and HTML::TreeBuilder are popular choices with CSS or tree selectors; the approach below works where you cannot install anything.
The scraper
#!/usr/bin/perl
use strict;
use warnings;
use HTTP::Tiny;
use HTML::TokeParser;
use JSON::PP;
use Encode qw(decode);
my $proxy = $ENV{PROXY}; # http://USER:PASS@HOST:PORT, or unset for a direct connection
my $http = HTTP::Tiny->new(
agent => 'my-scraper/1.0 (+https://example.com/bot)',
timeout => 30,
($proxy ? (proxy => $proxy, https_proxy => $proxy) : ()),
);
sub get_html {
my ($url) = @_;
my $res = $http->get($url);
die "$res->{status} $res->{reason} for $url\n" unless $res->{success};
return decode('UTF-8', $res->{content});
}
my @books;
my $url = 'https://books.toscrape.com/catalogue/page-1.html';
for my $page (1 .. 3) {
my $p = HTML::TokeParser->new(\ get_html($url));
my ($in_book, %book, $next);
while (my $t = $p->get_tag('article', 'a', 'p', 'li')) {
my ($tag, $attr) = @$t;
if ($tag eq 'article' && ($attr->{class} // '') eq 'product_pod') { $in_book = 1; %book = (); }
elsif ($in_book && $tag eq 'a' && $attr->{title}) { $book{title} = $attr->{title}; }
elsif ($in_book && $tag eq 'p' && ($attr->{class} // '') =~ /^star-rating (\w+)/) { $book{rating} = $1; }
elsif ($in_book && $tag eq 'p' && ($attr->{class} // '') eq 'price_color') {
($book{price} = $p->get_trimmed_text('/p')) =~ s/[^\d.]//g;
push @books, {%book}; $in_book = 0;
}
elsif ($tag eq 'li' && ($attr->{class} // '') eq 'next') { my $a = $p->get_tag('a'); $next = $a->[1]{href}; }
}
printf "page %d: %d books so far\n", $page, scalar @books;
last unless $next;
$url = "https://books.toscrape.com/catalogue/$next";
}
open my $fh, '>:encoding(UTF-8)', 'books.json' or die $!;
print $fh JSON::PP->new->pretty->canonical->encode(\@books);
close $fh;
printf "%s | %s | %s\n", @{$books[$_]}{qw(title price rating)} for 0 .. 2;
print scalar(@books), " books written to books.json\n";
$ perl scrape.pl
page 1: 20 books so far
page 2: 40 books so far
page 3: 60 books so far
A Light in the Attic | 51.77 | Three
Tipping the Velvet | 53.74 | One
Soumission | 50.10 | One
60 books written to books.json
How it works
agentnames the scraper and gives a URL where a site owner can read about it.- HTTP::Tiny returns a hash:
success,status,reason,headersandcontent. It does not die on a 404, so the script checkssuccessitself. decode('UTF-8', ...)turns the bytes into Perl characters, so prices with the pound sign and accented titles stay intact. The:encoding(UTF-8)layer writes them back out correctly.- HTML::TokeParser reads the page as a stream of tags.
get_tag('article', 'a', 'p', 'li')skips everything else; the attributes come as a hash, so$attr->{title}gives the full title even where the link text is shortened. get_trimmed_text('/p')returns the text up to the closing tag; the regex keeps digits and the dot, so£51.77becomes51.77.- Pagination follows the link inside
<li class="next">until there is none or the page limit is reached.
A token parser is more code than CSS selectors, but it needs nothing extra and does not depend on whitespace or on the order of attributes, which is what most often breaks regular expressions on HTML.
Running it through a proxy
HTTP::Tiny takes the proxy in its constructor: proxy for http:// URLs and https_proxy for https:// URLs, with the username and password in the proxy URL. The script reads it from the environment, so the same file runs both ways:
PROXY="http://USER-type-residential-country-us-session-pl01-lifetime-30:PASS@residentialboson.quanticdata.io:10000" perl scrape.pl
PROXY="http://USER-country-us-sid-pl02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080" perl scrape.pl
PROXY="http://USER-country-us-type-mobile-sid-pl03-ttl-30m:PASS@mobile.quanticdata.io:8080" perl scrape.pl
| Network | Exit location (ipinfo.io) | Result |
|---|---|---|
| Residential Basic | New York City, New York, US | 60 books over 3 pages |
| Residential Premium | Palisades Park, New Jersey, US | 60 books over 3 pages |
| Mobile | Fort George G Mead Junction, Maryland, US | 60 books over 3 pages |
The username carries a session ID: use one per run. With a wrong proxy password the script stopped at the first page with 407 Proxy Authentication Required, the gateway's answer explained in 407 Proxy Authentication Required. Read the exit location yourself with a one-liner:
PROXY="http://USER:PASS@HOST:PORT" perl -MHTTP::Tiny -MJSON::PP -e 'my $h=HTTP::Tiny->new(proxy=>$ENV{PROXY}, https_proxy=>$ENV{PROXY}); my $r=$h->get("https://ipinfo.io/json"); my $j=decode_json($r->{content}); print "$r->{status} $j->{city}, $j->{region}, $j->{country}\n"'
JavaScript pages
Perl, like any HTTP client, gets the HTML the server sends. quotes.toscrape.com has a JavaScript version of its home page; counting quote blocks in each:
quote divs in /js/: 0
quote divs in /: 10
When a page shows data in the browser but your script finds nothing, the data is loaded by JavaScript. Look for the JSON request the page makes and fetch that with HTTP::Tiny and decode_json, or render the page with a browser tool. The same check works in web scraping with PowerShell and with Cheerio in Node.js.
Perl next to other languages
| Language | Download | Parse | Our guide |
|---|---|---|---|
| Perl | HTTP::Tiny | HTML::TokeParser | this page |
| PowerShell | Invoke-WebRequest | Regular expressions | web scraping with PowerShell |
| Node.js | fetch | Cheerio | Cheerio in Node.js |
| Python | requests | Beautiful Soup | web scraping with Python |
Keep to each site's terms and pause between requests. Our residential proxies and mobile proxies plug into HTTP::Tiny as shown, with a Residential Basic free trial; our web scraping API returns rendered pages from one call when a site needs a browser.


