Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Web scraping with Ruby: Nokogiri, Net::HTTP and a proxy, run on Ruby 4.0

A Ruby web scraper on Ruby 4.0: Net::HTTP downloads each page, optionally through a proxy, Nokogiri 1.19.7 selects every article.product_pod card and reads title, price and rating, the li.next link leads to the next page and the csv library writes books.csv; the card shows the real run: 60 books over 3 pages, the same directly and through residential and mobile proxies.
A Ruby web scraper on Ruby 4.0: Net::HTTP downloads each page, optionally through a proxy, Nokogiri 1.19.7 selects every article.product_pod card and reads title, price and rating, the li.next link leads to the next page and the csv library writes books.csv; the card shows the real run: 60 books over 3 pages, the same directly and through residential and mobile proxies.

To scrape a website with Ruby, download the page with Net::HTTP, which is part of Ruby, and parse the HTML with the Nokogiri gem and its CSS selectors. A 38-line script with a 24-line helper ran on Ruby 4.0.10 with Nokogiri 1.19.7, collected 60 books from 3 pages into a CSV file, and worked the same through a proxy.

Everything below ran on 21 November 2026 (UTC) on Windows 11 against books.toscrape.com, a sandbox made for scraping practice, and against one page of our own site. The output is copied from those runs.

Which libraries, and the versions we ran

JobLibraryVersion in this runWhere it came from
Download pagesNet::HTTP (net-http)0.9.1with Ruby 4.0.10
Parse HTML with CSS selectorsNokogiri1.19.7gem install
Write the CSV filecsv3.3.5with Ruby 4.0.10
Read JSON answersjson2.18.3with Ruby 4.0.10
A shorter HTTP client (optional)HTTParty0.25.0gem install

Set up

$ ruby -v
ruby 4.0.10 (2026-11-19 revision a704b7cd95) +PRISM [x64-mingw-ucrt]
$ gem install nokogiri httparty
Successfully installed nokogiri-1.19.7-x64-mingw-ucrt
Successfully installed multi_xml-0.9.1
Successfully installed mini_mime-1.1.5
When you HTTParty, you must party hard!
Successfully installed httparty-0.25.0
4 gems installed

Nokogiri arrived as a precompiled build for our platform (x64-mingw-ucrt), so nothing had to be compiled. On Windows we unpacked the RubyInstaller 7-Zip archive into a folder and ran ruby.exe from there, with no setup program and without the Devkit. Setting the GEM_HOME environment variable to a folder of the project keeps the gems next to the code, which is what we did.

The scraper: Net::HTTP downloads, Nokogiri parses

A small helper file, common.rb, opens the connection, directly or through a proxy:

require "net/http"
require "nokogiri"

UA = "my-scraper/1.0 (+https://example.com/bot)"

# An HTTP connection to the site in +uri+, through the proxy in PROXY when it is set.
def connect(uri)
  proxy = ENV["PROXY"] && URI(ENV["PROXY"]) # http://USER:PASS@HOST:PORT
  http = if proxy
    Net::HTTP.new(uri.host, uri.port, proxy.host, proxy.port, proxy.user, proxy.password)
  else
    Net::HTTP.new(uri.host, uri.port, nil) # nil: no proxy, not even from the environment
  end
  http.use_ssl = uri.scheme == "https"
  http.open_timeout = 30
  http.read_timeout = 30
  http
end

def fetch(uri)
  res = connect(uri).get(uri.request_uri, "User-Agent" => UA)
  res.value # raises unless the answer is 2xx
  res
end

scraper.rb:

require "csv"
require_relative "common"

url = URI("https://books.toscrape.com/catalogue/page-1.html")
http = connect(url).start # one connection for all the pages
books = []
page = 0

while url
  page += 1
  res = http.get(url.request_uri, "User-Agent" => UA)
  res.value
  doc = Nokogiri::HTML(res.body)

  doc.css("article.product_pod").each do |card|
    link = card.at_css("h3 a")
    books << {
      title: link["title"],
      price: card.at_css(".price_color").text[/[\d.]+/],
      rating: (card.at_css("p.star-rating")["class"].split - ["star-rating"]).first,
      url: URI.join(url, link["href"]).to_s
    }
  end
  puts "page #{page}: #{books.size} books so far"

  nxt = doc.at_css("li.next a")
  url = nxt && page < 3 ? URI.join(url, nxt["href"]) : nil
  sleep 1 # one page a second is plenty
end
http.finish

CSV.open("books.csv", "w") do |csv|
  csv << books.first.keys
  books.each { |book| csv << book.values }
end

books.first(3).each { |b| puts "#{b[:title]} | #{b[:price]} | #{b[:rating]}" }
puts "#{books.size} books written to books.csv"
$ ruby scraper.rb
page 1: 20 books so far
page 2: 40 books so far
page 3: 60 books so far
A Light in the Attic | 51.77 | Three
Tipping the Velvet | 53.74 | One
Soumission | 50.10 | One
60 books written to books.csv

The first lines of books.csv:

title,price,rating,url
A Light in the Attic,51.77,Three,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,One,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
Soumission,50.10,One,https://books.toscrape.com/catalogue/soumission_998/index.html

How it works

  • The User-Agent names the scraper and gives a URL where a site owner can read about it. httpbin.org, which echoes request headers, returned exactly that string.
  • connect(url).start opens one connection and the loop reuses it for the three pages; http.finish closes it.
  • res.value raises an error unless the answer is 2xx, so the script stops instead of parsing an error page. Net::HTTP does not follow redirects by itself, so a 301 or 302 raises too.
  • Nokogiri::HTML(res.body) parses the page. doc.css(...) returns every book card, and card.at_css(...) returns the first match inside one card. More on choosing selectors in CSS selectors and XPath.
  • Attributes and text: the title is in the link's title attribute, where it is not shortened; the price is the text of .price_color cut down to digits and the dot by [/[\d.]+/]; the rating is the class next to star-rating.
  • URI.join turns a relative link into a full address, for the book links and for the next page.
  • Pagination follows li.next a until there is none or the page limit is reached, one page a second. Other layouts are covered in pagination patterns.
  • CSV.open writes the header from the keys of the first row and one line per book.

Running it through a proxy

Net::HTTP.new takes the proxy as four more arguments after the site's host and port: proxy host, proxy port, username and password. common.rb reads them from one URL in an environment variable, so the same script runs both ways:

PROXY="http://USER-type-residential-country-us-session-rb01-lifetime-30:PASS@residentialboson.quanticdata.io:10000" ruby scraper.rb
PROXY="http://USER-country-us-sid-rb02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080" ruby scraper.rb
PROXY="http://USER-country-us-type-mobile-sid-rb03-ttl-30m:PASS@mobile.quanticdata.io:8080" ruby scraper.rb
NetworkExit location (ipinfo.io)Result
Residential BasicHighland Heights, Ohio, US60 books over 3 pages
Residential PremiumHomestead, Florida, US60 books over 3 pages
MobileColumbus, Ohio, US60 books over 3 pages

The four runs, one direct and three through a proxy, wrote the same books.csv. The session name and its length are set in the username: use one name per run. To see where the requests leave from, a second script, check.rb, uses the same helper and asks an IP service (the part of check.rb that runs for ip; the file starts with require "json" and require_relative "common"):

ips = []
3.times do |i|
  res = fetch(URI("https://ipinfo.io/json"))
  info = JSON.parse(res.body)
  ips << info["ip"]
  puts "request #{i + 1}: #{res.code} #{res.message} #{info['ip']} | #{info['city']}, #{info['region']}, #{info['country']}"
end
puts "different exit addresses in 3 requests: #{ips.uniq.size}"
echo = JSON.parse(fetch(URI("https://httpbin.org/headers")).body)
puts "User-Agent seen by the site: #{echo['headers']['User-Agent']}"
$ ruby check.rb ip
request 1: 200 OK x.x.x.x | Highland Heights, Ohio, US
request 2: 200 OK x.x.x.x | Highland Heights, Ohio, US
request 3: 200 OK x.x.x.x | Highland Heights, Ohio, US
different exit addresses in 3 requests: 1
User-Agent seen by the site: my-scraper/1.0 (+https://example.com/bot)

In this run the three requests of this check left from one address in each session (masked here), on all three networks.

The http_proxy environment variable

When no proxy is passed, Net::HTTP looks for one in the environment. A connection made with only Net::HTTP.new(uri.host, uri.port) and http_proxy set to the same proxy URL went out through the proxy, username and password included:

$ ruby check.rb env
proxy in use: true
x.x.x.x | Highland Heights, Ohio, US

Net::HTTP looks up http_proxy for https:// addresses too; it does not read https_proxy. Passing nil as the third argument, as common.rb does when PROXY is not set, turns the lookup off.

Another page, same helper

Through Residential Basic, the sandbox and our own residential proxies page:

ARGV.each do |url|
  doc = Nokogiri::HTML(fetch(URI(url)).body)
  puts url
  puts "  title: #{doc.at_css('title').text.strip}"
  puts "  h1:    #{doc.at_css('h1').text.strip}"
  puts "  links: #{doc.css('a[href]').size}"
end
$ ruby check.rb page https://books.toscrape.com/ https://quanticdata.io/residential-proxies/
https://books.toscrape.com/
  title: All products | Books to Scrape - Sandbox
  h1:    All products
  links: 94
https://quanticdata.io/residential-proxies/
  title: Residential Proxies from $0.80/GB | QuanticData
  h1:    Residential Proxies: Real Home IPs in 228 Countries
  links: 482

The same request with HTTParty

HTTParty wraps Net::HTTP in one call. The proxy goes in four options:

require "httparty"
require "nokogiri"

options = { headers: { "User-Agent" => "my-scraper/1.0 (+https://example.com/bot)" }, timeout: 30 }
if ENV["PROXY"]
  proxy = URI(ENV["PROXY"])
  options.merge!(http_proxyaddr: proxy.host, http_proxyport: proxy.port,
                 http_proxyuser: proxy.user, http_proxypass: proxy.password)
end

res = HTTParty.get("https://books.toscrape.com/", options)
doc = Nokogiri::HTML(res.body)
puts "#{res.code} #{doc.css('article.product_pod').size} books, first: #{doc.at_css('h3 a')['title']}"
$ ruby party.rb
200 20 books, first: A Light in the Attic

That line is from the Residential Basic session; the Mobile session printed the same.

Several pages at once with threads

When you already know the addresses, a few threads can share the work. Each thread takes the next address from a queue, so the number of threads is the number of requests in flight. fast.rb:

require_relative "common"

urls = (1..5).map { |n| URI("https://books.toscrape.com/catalogue/page-#{n}.html") }
queue = Queue.new
urls.each_with_index { |uri, i| queue << [i, uri] }
queue.close
results = []
start = Process.clock_gettime(Process::CLOCK_MONOTONIC)

size = (ARGV[0] || 3).to_i # requests in flight at once
workers = size.times.map do
  Thread.new do
    while (job = queue.pop)
      i, uri = job
      res = fetch(uri)
      books = Nokogiri::HTML(res.body).css("article.product_pod").size
      results[i] = "#{res.code} #{res.message} #{books} books  #{uri}"
    end
  end
end
workers.each(&:join)

puts results
puts format("5 pages, #{size} at a time: %.1f s", Process.clock_gettime(Process::CLOCK_MONOTONIC) - start)
$ ruby fast.rb 3
200 OK 20 books  https://books.toscrape.com/catalogue/page-1.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-2.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-3.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-4.html
200 OK 20 books  https://books.toscrape.com/catalogue/page-5.html
5 pages, 3 at a time: 3.5 s

That is one run of 21 November 2026 through one Residential Basic session. Keep the number small: a script can send requests faster than a site should receive them. If a site answers 429 or 403, slow down or stop.

When a page is missing

res.value turns an error status into an exception you can rescue. Asking the sandbox for a page that does not exist:

begin
  fetch(URI("https://books.toscrape.com/catalogue/page-999.html"))
rescue Net::HTTPClientException => e
  puts "#{e.class}: #{e.message}"
end
Net::HTTPClientException: 404 "Not Found"

JavaScript pages

An HTTP client receives the HTML the server sends, and Nokogiri does not run scripts. quotes.toscrape.com has a JavaScript version of its home page; counting div.quote in each:

0 quotes in https://quotes.toscrape.com/js/
10 quotes in https://quotes.toscrape.com/

When a page shows data in the browser and your selectors find none, the data arrives by JavaScript. Look in the browser's network tab for the JSON request the page makes and read it with JSON.parse, as the IP check above does, or render the page with a browser tool and parse the result. The idea is the same in every language, and scraping dynamic pages walks through it.

Ruby next to other languages

LanguageDownloadParseOur guide
RubyNet::HTTPNokogirithis page
PythonrequestsBeautiful Soupweb scraping with Python
Node.jsfetchCheerioCheerio in Node.js
JavaHttpClientjsoupweb scraping with Java
Rustreqwestscraperweb scraping with Rust
PerlHTTP::TinyHTML::TokeParserweb scraping with Perl

Keep to each site's terms and robots.txt, and pause between requests. Our residential proxies and mobile proxies plug into Net::HTTP.new and HTTParty as shown, with a Residential Basic free trial; our web scraping API returns rendered pages from one call when a site needs a browser.

Sources & further reading

FAQ

Quick answers on web scraping with ruby.

Something else? Ask us

How do I scrape a website with Ruby?

Download the page with Net::HTTP, parse it with Nokogiri and its CSS selectors, and write the rows with the csv library. A 38-line script with a 24-line helper did that on Ruby 4.0.10 with Nokogiri 1.19.7 and collected 60 books from 3 pages.

Which Ruby gems are used for web scraping?

Nokogiri for HTML with CSS selectors or XPath is the one gem this tutorial has to install. Net::HTTP, json and csv came with Ruby 4.0 in our install. HTTParty is a common shorter client; we ran the same request with HTTParty 0.25.0.

How do I use a proxy with Net::HTTP in Ruby?

Pass the proxy as four more arguments: Net::HTTP.new(host, port, proxy_host, proxy_port, proxy_user, proxy_pass). Net::HTTP also reads the http_proxy environment variable, username and password included; in our test a connection with no proxy in the code went out through the proxy named there.

How do I use a proxy with HTTParty?

Pass http_proxyaddr, http_proxyport, http_proxyuser and http_proxypass in the options of HTTParty.get. In our run that request returned 200 and the 20 books of the page through a residential and a mobile proxy.

Can Ruby scrape JavaScript pages?

Net::HTTP receives the HTML the server sends and Nokogiri does not run scripts. On the JavaScript version of quotes.toscrape.com we counted 0 quotes, against 10 on the static version. Fetch the JSON the page loads, or render the page with a browser tool first.

Do I need a compiler to install Nokogiri on Windows?

Not in our install. gem install nokogiri fetched a precompiled build for Windows (nokogiri 1.19.7, platform x64-mingw-ucrt) on a Ruby 4.0.10 unpacked from the RubyInstaller archive without the Devkit.

A home or carrier IP for your Ruby scraper

Residential and mobile proxies with country targeting and named sessions, ready for Net::HTTP and HTTParty.

Related reading