# Perl Web Scraping with HTTP::Tiny, No CPAN

> Scrape websites in Perl with modules you already have: HTTP::Tiny over HTTPS, HTML::TokeParser for the HTML, JSON::PP for output, pagination and proxies.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Perl Web Scraping with HTTP::Tiny, No CPAN

# Perl web scraping with HTTP::Tiny and HTML::TokeParser, no CPAN installs

Web scraping · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![A Perl web scraper built only from modules that were already installed: HTTP::Tiny downloads each page over HTTPS, optionally through a proxy, HTML::TokeParser walks the article, a and p tags to read title, rating and price, and JSON::PP writes books.json; the card shows the real run: 60 books over 3 pages, the same through residential and mobile proxies.](https://quanticdata.io/blog/perl-web-scraping/hero.webp?v=20261120h)

On this page

- [Which modules, and which ones you already have](https://quanticdata.io/blog/perl-web-scraping/#which-modules-and-which-ones-you-already-have)
- [The scraper](https://quanticdata.io/blog/perl-web-scraping/#the-scraper)
- [How it works](https://quanticdata.io/blog/perl-web-scraping/#how-it-works)
- [Running it through a proxy](https://quanticdata.io/blog/perl-web-scraping/#running-it-through-a-proxy)
- [JavaScript pages](https://quanticdata.io/blog/perl-web-scraping/#javascript-pages)
- [Perl next to other languages](https://quanticdata.io/blog/perl-web-scraping/#perl-next-to-other-languages)

To scrape a website with Perl, download the page with `HTTP::Tiny` (part of core Perl), parse the HTML with `HTML::TokeParser` or another parser, and save the fields with `JSON::PP`. Our 48-line scraper used only modules that were already installed, collected 60 books from 3 pages, and ran the same through a proxy by adding one option.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=perl-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

Everything below ran on 20 November 2026 on Windows 11 with Perl 5.42.3, the perl that comes with Git for Windows, against books.toscrape.com, a sandbox made for scraping practice. Output is copied from those runs.

## Which modules, and which ones you already have

| Job | Module | On our Perl |
| --- | --- | --- |
| Download pages | HTTP::Tiny | 0.096, installed |
| HTTPS for HTTP::Tiny | IO::Socket::SSL | 2.099, installed; `HTTP::Tiny->can_ssl` printed yes |
| Parse HTML | HTML::TokeParser (with HTML::Parser) | 3.85, installed |
| Write JSON | JSON::PP | 4.16, installed |
| Download pages, the older way | LWP::UserAgent | 6.83 installed, but https:// failed: `501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)` |

Check your own Perl with `perl -MHTTP::Tiny -e 'print $HTTP::Tiny::VERSION'` for each module. If you can install from CPAN, Mojo::UserAgent and HTML::TreeBuilder are popular choices with CSS or tree selectors; the approach below works where you cannot install anything.

## The scraper

```
#!/usr/bin/perl
use strict;
use warnings;
use HTTP::Tiny;
use HTML::TokeParser;
use JSON::PP;
use Encode qw(decode);

my $proxy = $ENV{PROXY};    # http://USER:PASS@HOST:PORT, or unset for a direct connection
my $http  = HTTP::Tiny->new(
    agent   => 'my-scraper/1.0 (+https://example.com/bot)',
    timeout => 30,
    ($proxy ? (proxy => $proxy, https_proxy => $proxy) : ()),
);

sub get_html {
    my ($url) = @_;
    my $res = $http->get($url);
    die "$res->{status} $res->{reason} for $url\n" unless $res->{success};
    return decode('UTF-8', $res->{content});
}

my @books;
my $url = 'https://books.toscrape.com/catalogue/page-1.html';
for my $page (1 .. 3) {
    my $p = HTML::TokeParser->new(\ get_html($url));
    my ($in_book, %book, $next);
    while (my $t = $p->get_tag('article', 'a', 'p', 'li')) {
        my ($tag, $attr) = @$t;
        if ($tag eq 'article' && ($attr->{class} // '') eq 'product_pod') { $in_book = 1; %book = (); }
        elsif ($in_book && $tag eq 'a' && $attr->{title}) { $book{title} = $attr->{title}; }
        elsif ($in_book && $tag eq 'p' && ($attr->{class} // '') =~ /^star-rating (\w+)/) { $book{rating} = $1; }
        elsif ($in_book && $tag eq 'p' && ($attr->{class} // '') eq 'price_color') {
            ($book{price} = $p->get_trimmed_text('/p')) =~ s/[^\d.]//g;
            push @books, {%book}; $in_book = 0;
        }
        elsif ($tag eq 'li' && ($attr->{class} // '') eq 'next') { my $a = $p->get_tag('a'); $next = $a->[1]{href}; }
    }
    printf "page %d: %d books so far\n", $page, scalar @books;
    last unless $next;
    $url = "https://books.toscrape.com/catalogue/$next";
}

open my $fh, '>:encoding(UTF-8)', 'books.json' or die $!;
print $fh JSON::PP->new->pretty->canonical->encode(\@books);
close $fh;
printf "%s | %s | %s\n", @{$books[$_]}{qw(title price rating)} for 0 .. 2;
print scalar(@books), " books written to books.json\n";
```

```bash
$ perl scrape.pl
page 1: 20 books so far
page 2: 40 books so far
page 3: 60 books so far
A Light in the Attic | 51.77 | Three
Tipping the Velvet | 53.74 | One
Soumission | 50.10 | One
60 books written to books.json
```

## How it works

- **`agent`** names the scraper and gives a URL where a site owner can read about it.
- **HTTP::Tiny** returns a hash: `success`, `status`, `reason`, `headers` and `content`. It does not die on a 404, so the script checks `success` itself.
- **`decode('UTF-8', ...)`** turns the bytes into Perl characters, so prices with the pound sign and accented titles stay intact. The `:encoding(UTF-8)` layer writes them back out correctly.
- **HTML::TokeParser** reads the page as a stream of tags. `get_tag('article', 'a', 'p', 'li')` skips everything else; the attributes come as a hash, so `$attr->{title}` gives the full title even where the link text is shortened.
- **`get_trimmed_text('/p')`** returns the text up to the closing tag; the regex keeps digits and the dot, so `£51.77` becomes `51.77`.
- **Pagination** follows the link inside `<li class="next">` until there is none or the page limit is reached.

A token parser is more code than CSS selectors, but it needs nothing extra and does not depend on whitespace or on the order of attributes, which is what most often breaks regular expressions on HTML.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=perl-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Running it through a proxy

HTTP::Tiny takes the proxy in its constructor: `proxy` for http:// URLs and `https_proxy` for https:// URLs, with the username and password in the proxy URL. The script reads it from the environment, so the same file runs both ways:

```
PROXY="http://USER-type-residential-country-us-session-pl01-lifetime-30:PASS@residentialboson.quanticdata.io:10000" perl scrape.pl
PROXY="http://USER-country-us-sid-pl02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080" perl scrape.pl
PROXY="http://USER-country-us-type-mobile-sid-pl03-ttl-30m:PASS@mobile.quanticdata.io:8080" perl scrape.pl
```

| Network | Exit location (ipinfo.io) | Result |
| --- | --- | --- |
| Residential Basic | New York City, New York, US | 60 books over 3 pages |
| Residential Premium | Palisades Park, New Jersey, US | 60 books over 3 pages |
| Mobile | Fort George G Mead Junction, Maryland, US | 60 books over 3 pages |

The username carries a session ID: use one per run. With a wrong proxy password the script stopped at the first page with `407 Proxy Authentication Required`, the gateway's answer explained in [407 Proxy Authentication Required](https://quanticdata.io/blog/how-to-fix-407-proxy-authentication-required/). Read the exit location yourself with a one-liner:

```
PROXY="http://USER:PASS@HOST:PORT" perl -MHTTP::Tiny -MJSON::PP -e 'my $h=HTTP::Tiny->new(proxy=>$ENV{PROXY}, https_proxy=>$ENV{PROXY}); my $r=$h->get("https://ipinfo.io/json"); my $j=decode_json($r->{content}); print "$r->{status} $j->{city}, $j->{region}, $j->{country}\n"'
```

## JavaScript pages

Perl, like any HTTP client, gets the HTML the server sends. quotes.toscrape.com has a JavaScript version of its home page; counting quote blocks in each:

```
quote divs in /js/: 0
quote divs in /: 10
```

When a page shows data in the browser but your script finds nothing, the data is loaded by JavaScript. Look for the JSON request the page makes and fetch that with HTTP::Tiny and `decode_json`, or render the page with a browser tool. The same check works in [web scraping with PowerShell](https://quanticdata.io/blog/powershell-web-scraping/) and with [Cheerio in Node.js](https://quanticdata.io/blog/cheerio-web-scraping/).

## Perl next to other languages

| Language | Download | Parse | Our guide |
| --- | --- | --- | --- |
| Perl | HTTP::Tiny | HTML::TokeParser | this page |
| PowerShell | Invoke-WebRequest | Regular expressions | [web scraping with PowerShell](https://quanticdata.io/blog/powershell-web-scraping/) |
| Node.js | fetch | Cheerio | [Cheerio in Node.js](https://quanticdata.io/blog/cheerio-web-scraping/) |
| Python | requests | Beautiful Soup | [web scraping with Python](https://quanticdata.io/blog/how-to-web-scraping-using-python/) |

Keep to each site's terms and pause between requests. Our [residential proxies](https://quanticdata.io/residential-proxies/) and [mobile proxies](https://quanticdata.io/mobile-proxies/) plug into HTTP::Tiny as shown, with a [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/); our [web scraping API](https://quanticdata.io/web-scraping-api/) returns rendered pages from one call when a site needs a browser.

### Sources & further reading

- [HTTP::Tiny on MetaCPAN](https://metacpan.org/pod/HTTP::Tiny)
- [HTML::TokeParser on MetaCPAN](https://metacpan.org/pod/HTML::TokeParser)
- [JSON::PP on MetaCPAN](https://metacpan.org/pod/JSON::PP)
- [Books to Scrape sandbox](https://books.toscrape.com/)

## FAQ

Quick answers on perl web scraping.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I scrape a website with Perl?

Download the page with HTTP::Tiny, parse it with HTML::TokeParser (or HTML::TreeBuilder or Mojo::DOM if you can install from CPAN), and write the results with JSON::PP. Our 48-line script collected 60 books from 3 pages using only modules that were already installed.

### Does HTTP::Tiny support HTTPS?

Yes, when IO::Socket::SSL is installed. `HTTP::Tiny->can_ssl` tells you whether it can; on our Perl it returned true, and every page in our test was https.

### Why does LWP::UserAgent fail on https URLs?

LWP needs the separate LWP::Protocol::https module for https. Without it, our request returned `501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)`. Install that module, or use HTTP::Tiny.

### How do I use a proxy in Perl?

With HTTP::Tiny, pass `proxy => "http://USER:PASS@HOST:PORT"` and `https_proxy => "http://USER:PASS@HOST:PORT"` to `new`. We ran the same scraper through residential and mobile proxies this way and got 60 books each time.

### Can Perl scrape JavaScript websites?

Not by itself: HTTP::Tiny gets the HTML the server sends, and on a JavaScript page that HTML has no data (0 quotes against 10 on the static version in our test). Fetch the JSON the page loads, or render the page with a browser first.

## A home or carrier IP for your Perl scripts

Residential and mobile proxies with country targeting and sticky sessions, ready for HTTP::Tiny proxy and https_proxy.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=perl-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

- [Web Scraping Project Ideas, Started for Real](https://quanticdata.io/blog/web-scraping-project-ideas/): Web scraping · Five web scraping project ideas, each one started for real on 20 November 2026: a price tracker, a library release dashboard, a question analysis, an open-source landscape and a topic timeline, with the code, the first run and what to build next.
- [Retry-After Header: Read It, Wait, Retry](https://quanticdata.io/blog/retry-after-header/): Web scraping · Retry-After tells a client how long to wait before the next request, as a number of seconds or as an HTTP date. Below: both formats, a Python and a Node helper we ran against a real 429 on 20 November 2026 (each waited 10 seconds and then got 200), the rate-limit headers of the QuanticData API, and how one batch job replaced 20 calls.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/perl-web-scraping/ · Site index for AI: https://quanticdata.io/llms.txt
