Yes, PowerShell can scrape websites. Invoke-WebRequest downloads the page, [regex]::Matches() pulls the fields you want out of the HTML into objects, and Export-Csv saves them. For JSON APIs, Invoke-RestMethod returns objects directly. Pages built by JavaScript need a browser instead.
Everything below ran in Windows PowerShell 5.1 (5.1.26100.9818) on Windows 11 on 20 November 2026, against books.toscrape.com and quotes.toscrape.com, two sandboxes made for scraping practice. The output is copied from those runs.
Step 1: download the page
$r = Invoke-WebRequest -Uri 'https://books.toscrape.com/' -UseBasicParsing
"$($r.StatusCode) $($r.StatusDescription) len=$($r.Content.Length) links=$($r.Links.Count)"
200 OK len=51294 links=94
-UseBasicParsing returns the HTML, the status code, the headers and a list of links without any HTML engine. We used it everywhere. Without it, Windows PowerShell tries its older Internet Explorer based parser (ParsedHtml), which did not run in our non-interactive session; scripts that run unattended should not depend on it.
The Links property is already useful: it lists every <a> with its href and title.
$r.Links | Where-Object { $_.href -like 'catalogue/*' -and $_.title } | Select-Object -First 3 title, href
title href
----- ----
A Light in the Attic catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet catalogue/tipping-the-velvet_999/index.html
Soumission catalogue/soumission_998/index.html
Step 2: decode the text correctly
Our first run printed prices as £51.77. The site sends Content-Type: text/html without a character set, so Windows PowerShell decoded the UTF-8 bytes with the wrong code page. Decode the raw bytes yourself:
$html = [System.Text.Encoding]::UTF8.GetString($r.RawContentStream.ToArray())
After that line the first price read £51.77. Check this on any site where accents or currency signs come out garbled.
Step 3: turn HTML into objects with a regular expression
Windows PowerShell has no CSS selector engine built in, so a regular expression with named groups is the tool that is always there. Each match becomes a [pscustomobject]:
$pattern = '(?s)<article class="product_pod">.*?<a href="(?<url>[^"]+)" title="(?<title>[^"]+)">.*?<p class="price_color">(?<price>[^<]+)</p>'
$books = [regex]::Matches($html, $pattern) | ForEach-Object {
[pscustomobject]@{
Title = [System.Net.WebUtility]::HtmlDecode($_.Groups['title'].Value)
Price = $_.Groups['price'].Value
Url = 'https://books.toscrape.com/' + $_.Groups['url'].Value
}
}
$books.Count
20
(?s)lets.match line breaks, because one book spans many lines of HTML..*?is lazy: it stops at the first price after each title instead of running to the last one on the page.HtmlDecodeturns&and'in titles back into characters.
The same approach reads a two-column table, such as the product information on a book page:
$rows = [regex]::Matches($h, '<tr>\s*<th>(?<k>[^<]+)</th>\s*<td>(?<v>[^<]*)</td>') |
ForEach-Object { [pscustomobject]@{ Field = $_.Groups['k'].Value; Value = $_.Groups['v'].Value } }
Field Value
----- -----
UPC a897fe39b1053632
Product Type Books
Price (excl. tax) £51.77
Price (incl. tax) £51.77
Tax £0.00
Availability In stock (22 available)
Number of reviews 0
Step 4: several pages to a CSV file
$all = foreach ($n in 1..3) {
$page = Invoke-WebRequest -Uri "https://books.toscrape.com/catalogue/page-$n.html" -UseBasicParsing
$html = [System.Text.Encoding]::UTF8.GetString($page.RawContentStream.ToArray())
[regex]::Matches($html, $pattern) | ForEach-Object {
[pscustomobject]@{ Page = $n; Title = [System.Net.WebUtility]::HtmlDecode($_.Groups['title'].Value); Price = $_.Groups['price'].Value }
}
Start-Sleep -Milliseconds 500
}
$all | Export-Csv -Path books.csv -NoTypeInformation -Encoding UTF8
"Page","Title","Price"
"1","A Light in the Attic","£51.77"
"1","Tipping the Velvet","£53.74"
Three pages gave 60 rows. Once the price is a number, PowerShell can summarise it: [decimal]($_.Price -replace '[^\d.]', '') piped to Measure-Object -Average -Minimum -Maximum gave 60 prices from 12.84 to 57.31, average 35.00. The half-second pause keeps the request rate polite.
JSON APIs: Invoke-RestMethod does the parsing
Many sites load their data from a JSON endpoint. When you find one, skip the HTML: Invoke-RestMethod parses JSON into objects.
$q = Invoke-RestMethod -Uri 'https://quotes.toscrape.com/api/quotes?page=1'
"page=$($q.page) has_next=$($q.has_next) quotes=$($q.quotes.Count)"
page=1 has_next=True quotes=10
$q.quotes | Select-Object -First 2 @{n='Author';e={$_.author.name}}, @{n='Tags';e={$_.tags -join ','}}
Author Tags
------ ----
Albert Einstein change,deep-thoughts,thinking,world
J.K. Rowling abilities,choices
Headers, user agent and cookies
| Need | Parameter | What our test showed |
|---|---|---|
| Your own user agent | -UserAgent 'my-script/1.0 (+https://example.com/bot)' | Sent as given; the default is Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.26100.9818, which names PowerShell |
| Other headers | -Headers @{ 'Accept-Language' = 'en-US' } | Sent as given |
| Cookies across requests | -SessionVariable s on the first call, -WebSession $s after | A cookie set by the first call came back on the second |
| HTTP errors | try { ... } catch { $_.Exception.Response.StatusCode } | A missing page threw "(404) Not Found" |
Scraping through a proxy
Invoke-WebRequest takes -Proxy and -ProxyCredential. We ran the three pages through a QuanticData Residential Basic line with one session for the whole run, our own user agent, a one-second pause between pages, and -DisableKeepAlive so PowerShell opens a fresh connection for each request:
$proxy = 'http://residentialboson.quanticdata.io:10000'
$pass = ConvertTo-SecureString 'PASS' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential('USER-type-residential-country-us-session-run1-lifetime-30', $pass)
$rows = foreach ($n in 1..3) {
$p = Invoke-WebRequest -Uri "https://books.toscrape.com/catalogue/page-$n.html" -Proxy $proxy -ProxyCredential $cred -UseBasicParsing -DisableKeepAlive -UserAgent 'my-script/1.0 (+https://example.com/bot)'
$html = [System.Text.Encoding]::UTF8.GetString($p.RawContentStream.ToArray())
[regex]::Matches($html, $pattern) | ForEach-Object { [pscustomobject]@{ Page = $n; Title = $_.Groups['title'].Value; Price = $_.Groups['price'].Value } }
Start-Sleep -Seconds 1
}
$rows.Count
60
The username carries the country and a session ID; keep one session ID for one job. Setting the credential once for a whole window, and SOCKS5, are in our PowerShell proxy setup guide.
The same from curl.exe
curl.exe, which ships with Windows 11, works from PowerShell too, and its output goes into the same regex. We fetched the first catalogue page this way with a QuanticData line and got the same 20 books:
$html = curl.exe -s -A "my-script/1.0 (+https://example.com/bot)" -x "http://HOST:PORT" -U "USER:PASS" https://books.toscrape.com/catalogue/page-1.html | Out-String
[regex]::Matches($html, $pattern).Count
20
When PowerShell is not enough: JavaScript pages
quotes.toscrape.com/js/ shows the same quotes as the home page, but builds them in the browser. Invoke-WebRequest only gets the HTML the server sends: our regex found 0 quote blocks there, and the page source contained the data inside a var data = script instead. Two ways forward: find the JSON the page loads and call it with Invoke-RestMethod, or use a browser, as in Playwright for scraping. For larger jobs, web scraping with Python has more parsing libraries to choose from.
Before you schedule it
Keep to each site's terms and robots rules, pause between requests, and keep passwords out of scripts you share. When the job grows, our residential proxies and mobile proxies give your scripts a home or carrier IP in the country you choose (there is a Residential Basic free trial), and our web scraping API returns rendered pages as clean data from one call.


