# Web Scraping with Ruby: Nokogiri and Net::HTTP

> Web scraping with Ruby, tested on Ruby 4.0: Net::HTTP downloads the pages, Nokogiri parses them, with pagination, CSV output, proxies and threads.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Web Scraping with Ruby: Nokogiri and Net::HTTP

# Web scraping with Ruby: Nokogiri, Net::HTTP and a proxy, run on Ruby 4.0

Web scraping · Nov 21, 2026 · 9 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![A Ruby web scraper on Ruby 4.0: Net::HTTP downloads each page, optionally through a proxy, Nokogiri 1.19.7 selects every article.product_pod card and reads title, price and rating, the li.next link leads to the next page and the csv library writes books.csv; the card shows the real run: 60 books over 3 pages, the same directly and through residential and mobile proxies.](https://quanticdata.io/blog/web-scraping-ruby/hero.webp?v=20261121h)

On this page

- [Which libraries, and the versions we ran](https://quanticdata.io/blog/web-scraping-ruby/#which-libraries-and-the-versions-we-ran)
- [Set up](https://quanticdata.io/blog/web-scraping-ruby/#set-up)
- [The scraper: Net::HTTP downloads, Nokogiri parses](https://quanticdata.io/blog/web-scraping-ruby/#the-scraper-net-http-downloads-nokogiri-parses)
- [How it works](https://quanticdata.io/blog/web-scraping-ruby/#how-it-works)
- [Running it through a proxy](https://quanticdata.io/blog/web-scraping-ruby/#running-it-through-a-proxy)
- [The same request with HTTParty](https://quanticdata.io/blog/web-scraping-ruby/#the-same-request-with-httparty)
- [Several pages at once with threads](https://quanticdata.io/blog/web-scraping-ruby/#several-pages-at-once-with-threads)
- [When a page is missing](https://quanticdata.io/blog/web-scraping-ruby/#when-a-page-is-missing)
- [JavaScript pages](https://quanticdata.io/blog/web-scraping-ruby/#javascript-pages)
- [Ruby next to other languages](https://quanticdata.io/blog/web-scraping-ruby/#ruby-next-to-other-languages)

To scrape a website with Ruby, download the page with `Net::HTTP`, which is part of Ruby, and parse the HTML with the Nokogiri gem and its CSS selectors. A 38-line script with a 24-line helper ran on Ruby 4.0.10 with Nokogiri 1.19.7, collected 60 books from 3 pages into a CSV file, and worked the same through a proxy.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=web-scraping-ruby) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

Everything below ran on 21 November 2026 (UTC) on Windows 11 against books.toscrape.com, a sandbox made for scraping practice, and against one page of our own site. The output is copied from those runs.

## Which libraries, and the versions we ran

| Job | Library | Version in this run | Where it came from |
| --- | --- | --- | --- |
| Download pages | Net::HTTP (net-http) | 0.9.1 | with Ruby 4.0.10 |
| Parse HTML with CSS selectors | Nokogiri | 1.19.7 | `gem install` |
| Write the CSV file | csv | 3.3.5 | with Ruby 4.0.10 |
| Read JSON answers | json | 2.18.3 | with Ruby 4.0.10 |
| A shorter HTTP client (optional) | HTTParty | 0.25.0 | `gem install` |

## Set up

```bash
$ ruby -v
ruby 4.0.10 (2026-11-19 revision a704b7cd95) +PRISM [x64-mingw-ucrt]
```

```bash
$ gem install nokogiri httparty
Successfully installed nokogiri-1.19.7-x64-mingw-ucrt
Successfully installed multi_xml-0.9.1
Successfully installed mini_mime-1.1.5
When you HTTParty, you must party hard!
Successfully installed httparty-0.25.0
4 gems installed
```

Nokogiri arrived as a precompiled build for our platform (`x64-mingw-ucrt`), so nothing had to be compiled. On Windows we unpacked the RubyInstaller 7-Zip archive into a folder and ran `ruby.exe` from there, with no setup program and without the Devkit. Setting the `GEM_HOME` environment variable to a folder of the project keeps the gems next to the code, which is what we did.

## The scraper: Net::HTTP downloads, Nokogiri parses

A small helper file, `common.rb`, opens the connection, directly or through a proxy:

```
require "net/http"
require "nokogiri"

UA = "my-scraper/1.0 (+https://example.com/bot)"

# An HTTP connection to the site in +uri+, through the proxy in PROXY when it is set.
def connect(uri)
  proxy = ENV["PROXY"] && URI(ENV["PROXY"]) # http://USER:PASS@HOST:PORT
  http = if proxy
    Net::HTTP.new(uri.host, uri.port, proxy.host, proxy.port, proxy.user, proxy.password)
  else
    Net::HTTP.new(uri.host, uri.port, nil) # nil: no proxy, not even from the environment
  end
  http.use_ssl = uri.scheme == "https"
  http.open_timeout = 30
  http.read_timeout = 30
  http
end

def fetch(uri)
  res = connect(uri).get(uri.request_uri, "User-Agent" => UA)
  res.value # raises unless the answer is 2xx
  res
end
```

`scraper.rb`:

```
require "csv"
require_relative "common"

url = URI("https://books.toscrape.com/catalogue/page-1.html")
http = connect(url).start # one connection for all the pages
books = []
page = 0

while url
  page += 1
  res = http.get(url.request_uri, "User-Agent" => UA)
  res.value
  doc = Nokogiri::HTML(res.body)

  doc.css("article.product_pod").each do |card|
    link = card.at_css("h3 a")
    books << {
      title: link["title"],
      price: card.at_css(".price_color").text[/[\d.]+/],
      rating: (card.at_css("p.star-rating")["class"].split - ["star-rating"]).first,
      url: URI.join(url, link["href"]).to_s
    }
  end
  puts "page #{page}: #{books.size} books so far"

  nxt = doc.at_css("li.next a")
  url = nxt && page < 3 ? URI.join(url, nxt["href"]) : nil
  sleep 1 # one page a second is plenty
end
http.finish

CSV.open("books.csv", "w") do |csv|
  csv << books.first.keys
  books.each { |book| csv << book.values }
end

books.first(3).each { |b| puts "#{b[:title]} | #{b[:price]} | #{b[:rating]}" }
puts "#{books.size} books written to books.csv"
```

```bash
$ ruby scraper.rb
page 1: 20 books so far
page 2: 40 books so far
page 3: 60 books so far
A Light in the Attic | 51.77 | Three
Tipping the Velvet | 53.74 | One
Soumission | 50.10 | One
60 books written to books.csv
```

The first lines of `books.csv`:

```
title,price,rating,url
A Light in the Attic,51.77,Three,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,One,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
Soumission,50.10,One,https://books.toscrape.com/catalogue/soumission_998/index.html
```

## How it works

- **The User-Agent** names the scraper and gives a URL where a site owner can read about it. httpbin.org, which echoes request headers, returned exactly that string.
- **`connect(url).start`** opens one connection and the loop reuses it for the three pages; `http.finish` closes it.
- **`res.value`** raises an error unless the answer is 2xx, so the script stops instead of parsing an error page. Net::HTTP does not follow redirects by itself, so a 301 or 302 raises too.
- **`Nokogiri::HTML(res.body)`** parses the page. `doc.css(...)` returns every book card, and `card.at_css(...)` returns the first match inside one card. More on choosing selectors in [CSS selectors and XPath](https://quanticdata.io/blog/css-selector-vs-xpath/).
- **Attributes and text**: the title is in the link's `title` attribute, where it is not shortened; the price is the text of `.price_color` cut down to digits and the dot by `[/[\d.]+/]`; the rating is the class next to `star-rating`.
- **`URI.join`** turns a relative link into a full address, for the book links and for the next page.
- **Pagination** follows `li.next a` until there is none or the page limit is reached, one page a second. Other layouts are covered in [pagination patterns](https://quanticdata.io/blog/web-scraping-pagination/).
- **`CSV.open`** writes the header from the keys of the first row and one line per book.

## Running it through a proxy

`Net::HTTP.new` takes the proxy as four more arguments after the site's host and port: proxy host, proxy port, username and password. `common.rb` reads them from one URL in an environment variable, so the same script runs both ways:

```
PROXY="http://USER-type-residential-country-us-session-rb01-lifetime-30:PASS@residentialboson.quanticdata.io:10000" ruby scraper.rb
PROXY="http://USER-country-us-sid-rb02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080" ruby scraper.rb
PROXY="http://USER-country-us-type-mobile-sid-rb03-ttl-30m:PASS@mobile.quanticdata.io:8080" ruby scraper.rb
```

| Network | Exit location (ipinfo.io) | Result |
| --- | --- | --- |
| Residential Basic | Highland Heights, Ohio, US | 60 books over 3 pages |
| Residential Premium | Homestead, Florida, US | 60 books over 3 pages |
| Mobile | Columbus, Ohio, US | 60 books over 3 pages |

The four runs, one direct and three through a proxy, wrote the same `books.csv`. The session name and its length are set in the username: use one name per run. To see where the requests leave from, a second script, `check.rb`, uses the same helper and asks an IP service (the part of `check.rb` that runs for `ip`; the file starts with `require "json"` and `require_relative "common"`):

```
ips = []
3.times do |i|
  res = fetch(URI("https://ipinfo.io/json"))
  info = JSON.parse(res.body)
  ips << info["ip"]
  puts "request #{i + 1}: #{res.code} #{res.message} #{info['ip']} | #{info['city']}, #{info['region']}, #{info['country']}"
end
puts "different exit addresses in 3 requests: #{ips.uniq.size}"
echo = JSON.parse(fetch(URI("https://httpbin.org/headers")).body)
puts "User-Agent seen by the site: #{echo['headers']['User-Agent']}"
```

```bash
$ ruby check.rb ip
request 1: 200 OK x.x.x.x | Highland Heights, Ohio, US
request 2: 200 OK x.x.x.x | Highland Heights, Ohio, US
request 3: 200 OK x.x.x.x | Highland Heights, Ohio, US
different exit addresses in 3 requests: 1
User-Agent seen by the site: my-scraper/1.0 (+https://example.com/bot)
```

In this run the three requests of this check left from one address in each session (masked here), on all three networks.

### The http_proxy environment variable

When no proxy is passed, Net::HTTP looks for one in the environment. A connection made with only `Net::HTTP.new(uri.host, uri.port)` and `http_proxy` set to the same proxy URL went out through the proxy, username and password included:

```bash
$ ruby check.rb env
proxy in use: true
x.x.x.x | Highland Heights, Ohio, US
```

Net::HTTP looks up `http_proxy` for https:// addresses too; it does not read `https_proxy`. Passing `nil` as the third argument, as `common.rb` does when `PROXY` is not set, turns the lookup off.

### Another page, same helper

Through Residential Basic, the sandbox and our own residential proxies page:

```
ARGV.each do |url|
  doc = Nokogiri::HTML(fetch(URI(url)).body)
  puts url
  puts "  title: #{doc.at_css('title').text.strip}"
  puts "  h1:    #{doc.at_css('h1').text.strip}"
  puts "  links: #{doc.css('a[href]').size}"
end
```

```bash
$ ruby check.rb page https://books.toscrape.com/ https://quanticdata.io/residential-proxies/
https://books.toscrape.com/
  title: All products | Books to Scrape - Sandbox
  h1:    All products
  links: 94
https://quanticdata.io/residential-proxies/
  title: Residential Proxies from $0.80/GB | QuanticData
  h1:    Residential Proxies: Real Home IPs in 228 Countries
  links: 482
```

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=web-scraping-ruby) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## The same request with HTTParty

HTTParty wraps Net::HTTP in one call. The proxy goes in four options:

```
require "httparty"
require "nokogiri"

options = { headers: { "User-Agent" => "my-scraper/1.0 (+https://example.com/bot)" }, timeout: 30 }
if ENV["PROXY"]
  proxy = URI(ENV["PROXY"])
  options.merge!(http_proxyaddr: proxy.host, http_proxyport: proxy.port,
                 http_proxyuser: proxy.user, http_proxypass: proxy.password)
end

res = HTTParty.get("https://books.toscrape.com/", options)
doc = Nokogiri::HTML(res.body)
puts "#{res.code} #{doc.css('article.product_pod').size} books, first: #{doc.at_css('h3 a')['title']}"
```

```bash
$ ruby party.rb
200 20 books, first: A Light in the Attic
```

That line is from the Residential Basic session; the Mobile session printed the same.

## Several pages at once with threads

When you already know the addresses, a few threads can share the work. Each thread takes the next address from a queue, so the number of threads is the number of requests in flight. `fast.rb`:

```
require_relative "common"

urls = (1..5).map { |n| URI("https://books.toscrape.com/catalogue/page-#{n}.html") }
queue = Queue.new
urls.each_with_index { |uri, i| queue << [i, uri] }
queue.close
results = []
start = Process.clock_gettime(Process::CLOCK_MONOTONIC)

size = (ARGV[0] || 3).to_i # requests in flight at once
workers = size.times.map do
  Thread.new do
    while (job = queue.pop)
      i, uri = job
      res = fetch(uri)
      books = Nokogiri::HTML(res.body).css("article.product_pod").size
      results[i] = "#{res.code} #{res.message} #{books} books  #{uri}"
    end
  end
end
workers.each(&:join)

puts results
puts format("5 pages, #{size} at a time: %.1f s", Process.clock_gettime(Process::CLOCK_MONOTONIC) - start)
```

```bash
$ ruby fast.rb 3
200 OK 20 books  https://books.toscrape.com/catalogue/page-1.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-2.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-3.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-4.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-5.html
5 pages, 3 at a time: 3.5 s
```

That is one run of 21 November 2026 through one Residential Basic session. Keep the number small: a script can send requests faster than a site should receive them. If a site answers 429 or 403, slow down or stop.

## When a page is missing

`res.value` turns an error status into an exception you can rescue. Asking the sandbox for a page that does not exist:

```
begin
  fetch(URI("https://books.toscrape.com/catalogue/page-999.html"))
rescue Net::HTTPClientException => e
  puts "#{e.class}: #{e.message}"
end
```

```
Net::HTTPClientException: 404 "Not Found"
```

## JavaScript pages

An HTTP client receives the HTML the server sends, and Nokogiri does not run scripts. quotes.toscrape.com has a JavaScript version of its home page; counting `div.quote` in each:

```
0 quotes in https://quotes.toscrape.com/js/
10 quotes in https://quotes.toscrape.com/
```

When a page shows data in the browser and your selectors find none, the data arrives by JavaScript. Look in the browser's network tab for the JSON request the page makes and read it with `JSON.parse`, as the IP check above does, or render the page with a browser tool and parse the result. The idea is the same in every language, and [scraping dynamic pages](https://quanticdata.io/blog/dynamic-web-scraping-python/) walks through it.

## Ruby next to other languages

| Language | Download | Parse | Our guide |
| --- | --- | --- | --- |
| Ruby | Net::HTTP | Nokogiri | this page |
| Python | requests | Beautiful Soup | [web scraping with Python](https://quanticdata.io/blog/how-to-web-scraping-using-python/) |
| Node.js | fetch | Cheerio | [Cheerio in Node.js](https://quanticdata.io/blog/cheerio-web-scraping/) |
| Java | HttpClient | jsoup | [web scraping with Java](https://quanticdata.io/blog/web-scraping-java/) |
| Rust | reqwest | scraper | [web scraping with Rust](https://quanticdata.io/blog/web-scraping-rust/) |
| Perl | HTTP::Tiny | HTML::TokeParser | [web scraping with Perl](https://quanticdata.io/blog/perl-web-scraping/) |

Keep to each site's terms and robots.txt, and pause between requests. Our [residential proxies](https://quanticdata.io/residential-proxies/) and [mobile proxies](https://quanticdata.io/mobile-proxies/) plug into `Net::HTTP.new` and HTTParty as shown, with a [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/); our [web scraping API](https://quanticdata.io/web-scraping-api/) returns rendered pages from one call when a site needs a browser.

### Sources & further reading

- [Net::HTTP (Ruby 4.0 documentation)](https://docs.ruby-lang.org/en/4.0/Net/HTTP.html)
- [Nokogiri](https://nokogiri.org/)
- [CSV (Ruby documentation)](https://ruby.github.io/csv/)
- [HTTParty](https://github.com/jnunemaker/httparty)
- [RubyInstaller for Windows: downloads](https://rubyinstaller.org/downloads/)
- [Books to Scrape sandbox](https://books.toscrape.com/)

## FAQ

Quick answers on web scraping with ruby.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I scrape a website with Ruby?

Download the page with Net::HTTP, parse it with Nokogiri and its CSS selectors, and write the rows with the csv library. A 38-line script with a 24-line helper did that on Ruby 4.0.10 with Nokogiri 1.19.7 and collected 60 books from 3 pages.

### Which Ruby gems are used for web scraping?

Nokogiri for HTML with CSS selectors or XPath is the one gem this tutorial has to install. Net::HTTP, json and csv came with Ruby 4.0 in our install. HTTParty is a common shorter client; we ran the same request with HTTParty 0.25.0.

### How do I use a proxy with Net::HTTP in Ruby?

Pass the proxy as four more arguments: `Net::HTTP.new(host, port, proxy_host, proxy_port, proxy_user, proxy_pass)`. Net::HTTP also reads the `http_proxy` environment variable, username and password included; in our test a connection with no proxy in the code went out through the proxy named there.

### How do I use a proxy with HTTParty?

Pass `http_proxyaddr`, `http_proxyport`, `http_proxyuser` and `http_proxypass` in the options of `HTTParty.get`. In our run that request returned 200 and the 20 books of the page through a residential and a mobile proxy.

### Can Ruby scrape JavaScript pages?

Net::HTTP receives the HTML the server sends and Nokogiri does not run scripts. On the JavaScript version of quotes.toscrape.com we counted 0 quotes, against 10 on the static version. Fetch the JSON the page loads, or render the page with a browser tool first.

### Do I need a compiler to install Nokogiri on Windows?

Not in our install. `gem install nokogiri` fetched a precompiled build for Windows (nokogiri 1.19.7, platform x64-mingw-ucrt) on a Ruby 4.0.10 unpacked from the RubyInstaller archive without the Devkit.

## A home or carrier IP for your Ruby scraper

Residential and mobile proxies with country targeting and named sessions, ready for Net::HTTP and HTTParty.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=web-scraping-ruby) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.
- [Web Scraping with Java: jsoup and HttpClient](https://quanticdata.io/blog/web-scraping-java/): Web scraping · In Java, the HttpClient that ships with the JDK downloads the page and jsoup reads the HTML with CSS selectors. Below: an 86-line scraper that collected 60 books over 3 pages, run directly and through residential and mobile proxies, the one system property that proxy authentication needs on HTTPS, and five pages fetched at once.
- [Web Scraping with Rust: reqwest and scraper](https://quanticdata.io/blog/web-scraping-rust/): Web scraping · In Rust, reqwest downloads the page and the scraper crate reads the HTML with CSS selectors. Below: a 72-line scraper that collected 60 books over 3 pages, run directly and through residential and mobile proxies, an async version that fetched five pages in 2.1 seconds, and a Windows build that needs no Visual Studio.

---

Source: https://quanticdata.io/blog/web-scraping-ruby/ · Site index for AI: https://quanticdata.io/llms.txt
