To scrape a website with Ruby, download the page with Net::HTTP, which is part of Ruby, and parse the HTML with the Nokogiri gem and its CSS selectors. A 38-line script with a 24-line helper ran on Ruby 4.0.10 with Nokogiri 1.19.7, collected 60 books from 3 pages into a CSV file, and worked the same through a proxy.
Everything below ran on 21 November 2026 (UTC) on Windows 11 against books.toscrape.com, a sandbox made for scraping practice, and against one page of our own site. The output is copied from those runs.
Which libraries, and the versions we ran
| Job | Library | Version in this run | Where it came from |
|---|---|---|---|
| Download pages | Net::HTTP (net-http) | 0.9.1 | with Ruby 4.0.10 |
| Parse HTML with CSS selectors | Nokogiri | 1.19.7 | gem install |
| Write the CSV file | csv | 3.3.5 | with Ruby 4.0.10 |
| Read JSON answers | json | 2.18.3 | with Ruby 4.0.10 |
| A shorter HTTP client (optional) | HTTParty | 0.25.0 | gem install |
Set up
$ ruby -v
ruby 4.0.10 (2026-11-19 revision a704b7cd95) +PRISM [x64-mingw-ucrt]
$ gem install nokogiri httparty
Successfully installed nokogiri-1.19.7-x64-mingw-ucrt
Successfully installed multi_xml-0.9.1
Successfully installed mini_mime-1.1.5
When you HTTParty, you must party hard!
Successfully installed httparty-0.25.0
4 gems installed
Nokogiri arrived as a precompiled build for our platform (x64-mingw-ucrt), so nothing had to be compiled. On Windows we unpacked the RubyInstaller 7-Zip archive into a folder and ran ruby.exe from there, with no setup program and without the Devkit. Setting the GEM_HOME environment variable to a folder of the project keeps the gems next to the code, which is what we did.
The scraper: Net::HTTP downloads, Nokogiri parses
A small helper file, common.rb, opens the connection, directly or through a proxy:
require "net/http"
require "nokogiri"
UA = "my-scraper/1.0 (+https://example.com/bot)"
# An HTTP connection to the site in +uri+, through the proxy in PROXY when it is set.
def connect(uri)
proxy = ENV["PROXY"] && URI(ENV["PROXY"]) # http://USER:PASS@HOST:PORT
http = if proxy
Net::HTTP.new(uri.host, uri.port, proxy.host, proxy.port, proxy.user, proxy.password)
else
Net::HTTP.new(uri.host, uri.port, nil) # nil: no proxy, not even from the environment
end
http.use_ssl = uri.scheme == "https"
http.open_timeout = 30
http.read_timeout = 30
http
end
def fetch(uri)
res = connect(uri).get(uri.request_uri, "User-Agent" => UA)
res.value # raises unless the answer is 2xx
res
end
scraper.rb:
require "csv"
require_relative "common"
url = URI("https://books.toscrape.com/catalogue/page-1.html")
http = connect(url).start # one connection for all the pages
books = []
page = 0
while url
page += 1
res = http.get(url.request_uri, "User-Agent" => UA)
res.value
doc = Nokogiri::HTML(res.body)
doc.css("article.product_pod").each do |card|
link = card.at_css("h3 a")
books << {
title: link["title"],
price: card.at_css(".price_color").text[/[\d.]+/],
rating: (card.at_css("p.star-rating")["class"].split - ["star-rating"]).first,
url: URI.join(url, link["href"]).to_s
}
end
puts "page #{page}: #{books.size} books so far"
nxt = doc.at_css("li.next a")
url = nxt && page < 3 ? URI.join(url, nxt["href"]) : nil
sleep 1 # one page a second is plenty
end
http.finish
CSV.open("books.csv", "w") do |csv|
csv << books.first.keys
books.each { |book| csv << book.values }
end
books.first(3).each { |b| puts "#{b[:title]} | #{b[:price]} | #{b[:rating]}" }
puts "#{books.size} books written to books.csv"
$ ruby scraper.rb
page 1: 20 books so far
page 2: 40 books so far
page 3: 60 books so far
A Light in the Attic | 51.77 | Three
Tipping the Velvet | 53.74 | One
Soumission | 50.10 | One
60 books written to books.csv
The first lines of books.csv:
title,price,rating,url
A Light in the Attic,51.77,Three,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,One,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
Soumission,50.10,One,https://books.toscrape.com/catalogue/soumission_998/index.html
How it works
- The User-Agent names the scraper and gives a URL where a site owner can read about it. httpbin.org, which echoes request headers, returned exactly that string.
connect(url).startopens one connection and the loop reuses it for the three pages;http.finishcloses it.res.valueraises an error unless the answer is 2xx, so the script stops instead of parsing an error page. Net::HTTP does not follow redirects by itself, so a 301 or 302 raises too.Nokogiri::HTML(res.body)parses the page.doc.css(...)returns every book card, andcard.at_css(...)returns the first match inside one card. More on choosing selectors in CSS selectors and XPath.- Attributes and text: the title is in the link's
titleattribute, where it is not shortened; the price is the text of.price_colorcut down to digits and the dot by[/[\d.]+/]; the rating is the class next tostar-rating. URI.jointurns a relative link into a full address, for the book links and for the next page.- Pagination follows
li.next auntil there is none or the page limit is reached, one page a second. Other layouts are covered in pagination patterns. CSV.openwrites the header from the keys of the first row and one line per book.
Running it through a proxy
Net::HTTP.new takes the proxy as four more arguments after the site's host and port: proxy host, proxy port, username and password. common.rb reads them from one URL in an environment variable, so the same script runs both ways:
PROXY="http://USER-type-residential-country-us-session-rb01-lifetime-30:PASS@residentialboson.quanticdata.io:10000" ruby scraper.rb
PROXY="http://USER-country-us-sid-rb02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080" ruby scraper.rb
PROXY="http://USER-country-us-type-mobile-sid-rb03-ttl-30m:PASS@mobile.quanticdata.io:8080" ruby scraper.rb
| Network | Exit location (ipinfo.io) | Result |
|---|---|---|
| Residential Basic | Highland Heights, Ohio, US | 60 books over 3 pages |
| Residential Premium | Homestead, Florida, US | 60 books over 3 pages |
| Mobile | Columbus, Ohio, US | 60 books over 3 pages |
The four runs, one direct and three through a proxy, wrote the same books.csv. The session name and its length are set in the username: use one name per run. To see where the requests leave from, a second script, check.rb, uses the same helper and asks an IP service (the part of check.rb that runs for ip; the file starts with require "json" and require_relative "common"):
ips = []
3.times do |i|
res = fetch(URI("https://ipinfo.io/json"))
info = JSON.parse(res.body)
ips << info["ip"]
puts "request #{i + 1}: #{res.code} #{res.message} #{info['ip']} | #{info['city']}, #{info['region']}, #{info['country']}"
end
puts "different exit addresses in 3 requests: #{ips.uniq.size}"
echo = JSON.parse(fetch(URI("https://httpbin.org/headers")).body)
puts "User-Agent seen by the site: #{echo['headers']['User-Agent']}"
$ ruby check.rb ip
request 1: 200 OK x.x.x.x | Highland Heights, Ohio, US
request 2: 200 OK x.x.x.x | Highland Heights, Ohio, US
request 3: 200 OK x.x.x.x | Highland Heights, Ohio, US
different exit addresses in 3 requests: 1
User-Agent seen by the site: my-scraper/1.0 (+https://example.com/bot)
In this run the three requests of this check left from one address in each session (masked here), on all three networks.
The http_proxy environment variable
When no proxy is passed, Net::HTTP looks for one in the environment. A connection made with only Net::HTTP.new(uri.host, uri.port) and http_proxy set to the same proxy URL went out through the proxy, username and password included:
$ ruby check.rb env
proxy in use: true
x.x.x.x | Highland Heights, Ohio, US
Net::HTTP looks up http_proxy for https:// addresses too; it does not read https_proxy. Passing nil as the third argument, as common.rb does when PROXY is not set, turns the lookup off.
Another page, same helper
Through Residential Basic, the sandbox and our own residential proxies page:
ARGV.each do |url|
doc = Nokogiri::HTML(fetch(URI(url)).body)
puts url
puts " title: #{doc.at_css('title').text.strip}"
puts " h1: #{doc.at_css('h1').text.strip}"
puts " links: #{doc.css('a[href]').size}"
end
$ ruby check.rb page https://books.toscrape.com/ https://quanticdata.io/residential-proxies/
https://books.toscrape.com/
title: All products | Books to Scrape - Sandbox
h1: All products
links: 94
https://quanticdata.io/residential-proxies/
title: Residential Proxies from $0.80/GB | QuanticData
h1: Residential Proxies: Real Home IPs in 228 Countries
links: 482
The same request with HTTParty
HTTParty wraps Net::HTTP in one call. The proxy goes in four options:
require "httparty"
require "nokogiri"
options = { headers: { "User-Agent" => "my-scraper/1.0 (+https://example.com/bot)" }, timeout: 30 }
if ENV["PROXY"]
proxy = URI(ENV["PROXY"])
options.merge!(http_proxyaddr: proxy.host, http_proxyport: proxy.port,
http_proxyuser: proxy.user, http_proxypass: proxy.password)
end
res = HTTParty.get("https://books.toscrape.com/", options)
doc = Nokogiri::HTML(res.body)
puts "#{res.code} #{doc.css('article.product_pod').size} books, first: #{doc.at_css('h3 a')['title']}"
$ ruby party.rb
200 20 books, first: A Light in the Attic
That line is from the Residential Basic session; the Mobile session printed the same.
Several pages at once with threads
When you already know the addresses, a few threads can share the work. Each thread takes the next address from a queue, so the number of threads is the number of requests in flight. fast.rb:
require_relative "common"
urls = (1..5).map { |n| URI("https://books.toscrape.com/catalogue/page-#{n}.html") }
queue = Queue.new
urls.each_with_index { |uri, i| queue << [i, uri] }
queue.close
results = []
start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
size = (ARGV[0] || 3).to_i # requests in flight at once
workers = size.times.map do
Thread.new do
while (job = queue.pop)
i, uri = job
res = fetch(uri)
books = Nokogiri::HTML(res.body).css("article.product_pod").size
results[i] = "#{res.code} #{res.message} #{books} books #{uri}"
end
end
end
workers.each(&:join)
puts results
puts format("5 pages, #{size} at a time: %.1f s", Process.clock_gettime(Process::CLOCK_MONOTONIC) - start)
$ ruby fast.rb 3
200 OK 20 books https://books.toscrape.com/catalogue/page-1.html
200 OK 20 books https://books.toscrape.com/catalogue/page-2.html
200 OK 20 books https://books.toscrape.com/catalogue/page-3.html
200 OK 20 books https://books.toscrape.com/catalogue/page-4.html
200 OK 20 books https://books.toscrape.com/catalogue/page-5.html
5 pages, 3 at a time: 3.5 s
That is one run of 21 November 2026 through one Residential Basic session. Keep the number small: a script can send requests faster than a site should receive them. If a site answers 429 or 403, slow down or stop.
When a page is missing
res.value turns an error status into an exception you can rescue. Asking the sandbox for a page that does not exist:
begin
fetch(URI("https://books.toscrape.com/catalogue/page-999.html"))
rescue Net::HTTPClientException => e
puts "#{e.class}: #{e.message}"
end
Net::HTTPClientException: 404 "Not Found"
JavaScript pages
An HTTP client receives the HTML the server sends, and Nokogiri does not run scripts. quotes.toscrape.com has a JavaScript version of its home page; counting div.quote in each:
0 quotes in https://quotes.toscrape.com/js/
10 quotes in https://quotes.toscrape.com/
When a page shows data in the browser and your selectors find none, the data arrives by JavaScript. Look in the browser's network tab for the JSON request the page makes and read it with JSON.parse, as the IP check above does, or render the page with a browser tool and parse the result. The idea is the same in every language, and scraping dynamic pages walks through it.
Ruby next to other languages
| Language | Download | Parse | Our guide |
|---|---|---|---|
| Ruby | Net::HTTP | Nokogiri | this page |
| Python | requests | Beautiful Soup | web scraping with Python |
| Node.js | fetch | Cheerio | Cheerio in Node.js |
| Java | HttpClient | jsoup | web scraping with Java |
| Rust | reqwest | scraper | web scraping with Rust |
| Perl | HTTP::Tiny | HTML::TokeParser | web scraping with Perl |
Keep to each site's terms and robots.txt, and pause between requests. Our residential proxies and mobile proxies plug into Net::HTTP.new and HTTParty as shown, with a Residential Basic free trial; our web scraping API returns rendered pages from one call when a site needs a browser.


