Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

PowerShell web scraping: Invoke-WebRequest, regex, CSV and proxies

A PowerShell web scraper in four steps: Invoke-WebRequest downloads the page, the bytes are decoded as UTF-8, a regular expression turns each book into an object with title and price, and Export-Csv writes the rows; the card shows the real run: 60 rows from 3 pages, cheapest 12.84 and dearest 57.31.
A PowerShell web scraper in four steps: Invoke-WebRequest downloads the page, the bytes are decoded as UTF-8, a regular expression turns each book into an object with title and price, and Export-Csv writes the rows; the card shows the real run: 60 rows from 3 pages, cheapest 12.84 and dearest 57.31.

Yes, PowerShell can scrape websites. Invoke-WebRequest downloads the page, [regex]::Matches() pulls the fields you want out of the HTML into objects, and Export-Csv saves them. For JSON APIs, Invoke-RestMethod returns objects directly. Pages built by JavaScript need a browser instead.

Everything below ran in Windows PowerShell 5.1 (5.1.26100.9818) on Windows 11 on 20 November 2026, against books.toscrape.com and quotes.toscrape.com, two sandboxes made for scraping practice. The output is copied from those runs.

Step 1: download the page

$r = Invoke-WebRequest -Uri 'https://books.toscrape.com/' -UseBasicParsing
"$($r.StatusCode) $($r.StatusDescription) len=$($r.Content.Length) links=$($r.Links.Count)"

200 OK len=51294 links=94

-UseBasicParsing returns the HTML, the status code, the headers and a list of links without any HTML engine. We used it everywhere. Without it, Windows PowerShell tries its older Internet Explorer based parser (ParsedHtml), which did not run in our non-interactive session; scripts that run unattended should not depend on it.

The Links property is already useful: it lists every <a> with its href and title.

$r.Links | Where-Object { $_.href -like 'catalogue/*' -and $_.title } | Select-Object -First 3 title, href

title                href
-----                ----
A Light in the Attic catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet   catalogue/tipping-the-velvet_999/index.html
Soumission           catalogue/soumission_998/index.html

Step 2: decode the text correctly

Our first run printed prices as £51.77. The site sends Content-Type: text/html without a character set, so Windows PowerShell decoded the UTF-8 bytes with the wrong code page. Decode the raw bytes yourself:

$html = [System.Text.Encoding]::UTF8.GetString($r.RawContentStream.ToArray())

After that line the first price read £51.77. Check this on any site where accents or currency signs come out garbled.

Step 3: turn HTML into objects with a regular expression

Windows PowerShell has no CSS selector engine built in, so a regular expression with named groups is the tool that is always there. Each match becomes a [pscustomobject]:

$pattern = '(?s)<article class="product_pod">.*?<a href="(?<url>[^"]+)" title="(?<title>[^"]+)">.*?<p class="price_color">(?<price>[^<]+)</p>'
$books = [regex]::Matches($html, $pattern) | ForEach-Object {
  [pscustomobject]@{
    Title = [System.Net.WebUtility]::HtmlDecode($_.Groups['title'].Value)
    Price = $_.Groups['price'].Value
    Url   = 'https://books.toscrape.com/' + $_.Groups['url'].Value
  }
}
$books.Count
20
  • (?s) lets . match line breaks, because one book spans many lines of HTML.
  • .*? is lazy: it stops at the first price after each title instead of running to the last one on the page.
  • HtmlDecode turns &amp; and &#39; in titles back into characters.

The same approach reads a two-column table, such as the product information on a book page:

$rows = [regex]::Matches($h, '<tr>\s*<th>(?<k>[^<]+)</th>\s*<td>(?<v>[^<]*)</td>') |
  ForEach-Object { [pscustomobject]@{ Field = $_.Groups['k'].Value; Value = $_.Groups['v'].Value } }

Field             Value
-----             -----
UPC               a897fe39b1053632
Product Type      Books
Price (excl. tax) £51.77
Price (incl. tax) £51.77
Tax               £0.00
Availability      In stock (22 available)
Number of reviews 0

Step 4: several pages to a CSV file

$all = foreach ($n in 1..3) {
  $page = Invoke-WebRequest -Uri "https://books.toscrape.com/catalogue/page-$n.html" -UseBasicParsing
  $html = [System.Text.Encoding]::UTF8.GetString($page.RawContentStream.ToArray())
  [regex]::Matches($html, $pattern) | ForEach-Object {
    [pscustomobject]@{ Page = $n; Title = [System.Net.WebUtility]::HtmlDecode($_.Groups['title'].Value); Price = $_.Groups['price'].Value }
  }
  Start-Sleep -Milliseconds 500
}
$all | Export-Csv -Path books.csv -NoTypeInformation -Encoding UTF8

"Page","Title","Price"
"1","A Light in the Attic","£51.77"
"1","Tipping the Velvet","£53.74"

Three pages gave 60 rows. Once the price is a number, PowerShell can summarise it: [decimal]($_.Price -replace '[^\d.]', '') piped to Measure-Object -Average -Minimum -Maximum gave 60 prices from 12.84 to 57.31, average 35.00. The half-second pause keeps the request rate polite.

JSON APIs: Invoke-RestMethod does the parsing

Many sites load their data from a JSON endpoint. When you find one, skip the HTML: Invoke-RestMethod parses JSON into objects.

$q = Invoke-RestMethod -Uri 'https://quotes.toscrape.com/api/quotes?page=1'
"page=$($q.page) has_next=$($q.has_next) quotes=$($q.quotes.Count)"
page=1 has_next=True quotes=10

$q.quotes | Select-Object -First 2 @{n='Author';e={$_.author.name}}, @{n='Tags';e={$_.tags -join ','}}

Author          Tags
------          ----
Albert Einstein change,deep-thoughts,thinking,world
J.K. Rowling    abilities,choices

Headers, user agent and cookies

NeedParameterWhat our test showed
Your own user agent-UserAgent 'my-script/1.0 (+https://example.com/bot)'Sent as given; the default is Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.26100.9818, which names PowerShell
Other headers-Headers @{ 'Accept-Language' = 'en-US' }Sent as given
Cookies across requests-SessionVariable s on the first call, -WebSession $s afterA cookie set by the first call came back on the second
HTTP errorstry { ... } catch { $_.Exception.Response.StatusCode }A missing page threw "(404) Not Found"

Scraping through a proxy

Invoke-WebRequest takes -Proxy and -ProxyCredential. We ran the three pages through a QuanticData Residential Basic line with one session for the whole run, our own user agent, a one-second pause between pages, and -DisableKeepAlive so PowerShell opens a fresh connection for each request:

$proxy = 'http://residentialboson.quanticdata.io:10000'
$pass  = ConvertTo-SecureString 'PASS' -AsPlainText -Force
$cred  = New-Object System.Management.Automation.PSCredential('USER-type-residential-country-us-session-run1-lifetime-30', $pass)
$rows = foreach ($n in 1..3) {
  $p = Invoke-WebRequest -Uri "https://books.toscrape.com/catalogue/page-$n.html" -Proxy $proxy -ProxyCredential $cred -UseBasicParsing -DisableKeepAlive -UserAgent 'my-script/1.0 (+https://example.com/bot)'
  $html = [System.Text.Encoding]::UTF8.GetString($p.RawContentStream.ToArray())
  [regex]::Matches($html, $pattern) | ForEach-Object { [pscustomobject]@{ Page = $n; Title = $_.Groups['title'].Value; Price = $_.Groups['price'].Value } }
  Start-Sleep -Seconds 1
}
$rows.Count
60

The username carries the country and a session ID; keep one session ID for one job. Setting the credential once for a whole window, and SOCKS5, are in our PowerShell proxy setup guide.

The same from curl.exe

curl.exe, which ships with Windows 11, works from PowerShell too, and its output goes into the same regex. We fetched the first catalogue page this way with a QuanticData line and got the same 20 books:

$html = curl.exe -s -A "my-script/1.0 (+https://example.com/bot)" -x "http://HOST:PORT" -U "USER:PASS" https://books.toscrape.com/catalogue/page-1.html | Out-String
[regex]::Matches($html, $pattern).Count
20

When PowerShell is not enough: JavaScript pages

quotes.toscrape.com/js/ shows the same quotes as the home page, but builds them in the browser. Invoke-WebRequest only gets the HTML the server sends: our regex found 0 quote blocks there, and the page source contained the data inside a var data = script instead. Two ways forward: find the JSON the page loads and call it with Invoke-RestMethod, or use a browser, as in Playwright for scraping. For larger jobs, web scraping with Python has more parsing libraries to choose from.

Before you schedule it

Keep to each site's terms and robots rules, pause between requests, and keep passwords out of scripts you share. When the job grows, our residential proxies and mobile proxies give your scripts a home or carrier IP in the country you choose (there is a Residential Basic free trial), and our web scraping API returns rendered pages as clean data from one call.

Sources & further reading

FAQ

Quick answers on powershell web scraping.

Something else? Ask us

Can PowerShell do web scraping?

Yes. Invoke-WebRequest downloads pages, regular expressions extract the fields into objects, and Export-Csv saves them. We collected 60 books from 3 pages this way in Windows PowerShell 5.1. Pages that are built by JavaScript need a browser or the JSON endpoint behind them.

How do I scrape a web page with Invoke-WebRequest?

Run $r = Invoke-WebRequest -Uri URL -UseBasicParsing, decode $r.RawContentStream as UTF-8 if characters look wrong, then use [regex]::Matches() with named groups and build a [pscustomobject] per match. $r.Links already lists every link with its href and title.

Why do prices or accents look wrong when I scrape with PowerShell?

The page is UTF-8 but the server did not say so, and Windows PowerShell decoded it with another code page: we got £51.77 instead of £51.77. Use [System.Text.Encoding]::UTF8.GetString($r.RawContentStream.ToArray()).

How do I scrape a table with PowerShell?

Match each row with a regular expression such as <tr>\s*<th>(?<k>[^<]+)</th>\s*<td>(?<v>[^<]*)</td> and turn each match into an object with the two groups. On a product page it gave seven fields, from UPC to the number of reviews.

How do I use a proxy for web scraping in PowerShell?

Pass -Proxy "http://HOST:PORT" and -ProxyCredential (a PSCredential with the proxy username and password) to Invoke-WebRequest, and add -DisableKeepAlive so each request opens a fresh connection. curl.exe with -x and -U works from PowerShell too.

A home or carrier IP for your PowerShell scripts

Residential and mobile proxies with country targeting, for your PowerShell scripts.

Related reading