# PowerShell Web Scraping with Invoke-WebRequest

> Scrape a website with PowerShell: Invoke-WebRequest, regex into objects, Export-Csv over several pages, JSON APIs with Invoke-RestMethod and proxies, tested.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / PowerShell Web Scraping with Invoke-WebRequest

# PowerShell web scraping: Invoke-WebRequest, regex, CSV and proxies

Web scraping · Nov 20, 2026 · 6 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![A PowerShell web scraper in four steps: Invoke-WebRequest downloads the page, the bytes are decoded as UTF-8, a regular expression turns each book into an object with title and price, and Export-Csv writes the rows; the card shows the real run: 60 rows from 3 pages, cheapest 12.84 and dearest 57.31.](https://quanticdata.io/blog/powershell-web-scraping/hero.webp?v=20261120h)

On this page

- [Step 1: download the page](https://quanticdata.io/blog/powershell-web-scraping/#step-1-download-the-page)
- [Step 2: decode the text correctly](https://quanticdata.io/blog/powershell-web-scraping/#step-2-decode-the-text-correctly)
- [Step 3: turn HTML into objects with a regular expression](https://quanticdata.io/blog/powershell-web-scraping/#step-3-turn-html-into-objects-with-a-regular-expression)
- [Step 4: several pages to a CSV file](https://quanticdata.io/blog/powershell-web-scraping/#step-4-several-pages-to-a-csv-file)
- [JSON APIs: Invoke-RestMethod does the parsing](https://quanticdata.io/blog/powershell-web-scraping/#json-apis-invoke-restmethod-does-the-parsing)
- [Headers, user agent and cookies](https://quanticdata.io/blog/powershell-web-scraping/#headers-user-agent-and-cookies)
- [Scraping through a proxy](https://quanticdata.io/blog/powershell-web-scraping/#scraping-through-a-proxy)
- [When PowerShell is not enough: JavaScript pages](https://quanticdata.io/blog/powershell-web-scraping/#when-powershell-is-not-enough-javascript-pages)
- [Before you schedule it](https://quanticdata.io/blog/powershell-web-scraping/#before-you-schedule-it)

Yes, PowerShell can scrape websites. `Invoke-WebRequest` downloads the page, `[regex]::Matches()` pulls the fields you want out of the HTML into objects, and `Export-Csv` saves them. For JSON APIs, `Invoke-RestMethod` returns objects directly. Pages built by JavaScript need a browser instead.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=powershell-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

Everything below ran in Windows PowerShell 5.1 (5.1.26100.9818) on Windows 11 on 20 November 2026, against books.toscrape.com and quotes.toscrape.com, two sandboxes made for scraping practice. The output is copied from those runs.

## Step 1: download the page

```
$r = Invoke-WebRequest -Uri 'https://books.toscrape.com/' -UseBasicParsing
"$($r.StatusCode) $($r.StatusDescription) len=$($r.Content.Length) links=$($r.Links.Count)"

200 OK len=51294 links=94
```

`-UseBasicParsing` returns the HTML, the status code, the headers and a list of links without any HTML engine. We used it everywhere. Without it, Windows PowerShell tries its older Internet Explorer based parser (`ParsedHtml`), which did not run in our non-interactive session; scripts that run unattended should not depend on it.

The `Links` property is already useful: it lists every `<a>` with its `href` and `title`.

```
$r.Links | Where-Object { $_.href -like 'catalogue/*' -and $_.title } | Select-Object -First 3 title, href

title                href
-----                ----
A Light in the Attic catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet   catalogue/tipping-the-velvet_999/index.html
Soumission           catalogue/soumission_998/index.html
```

## Step 2: decode the text correctly

Our first run printed prices as `Â£51.77`. The site sends `Content-Type: text/html` without a character set, so Windows PowerShell decoded the UTF-8 bytes with the wrong code page. Decode the raw bytes yourself:

```
$html = [System.Text.Encoding]::UTF8.GetString($r.RawContentStream.ToArray())
```

After that line the first price read `£51.77`. Check this on any site where accents or currency signs come out garbled.

## Step 3: turn HTML into objects with a regular expression

Windows PowerShell has no CSS selector engine built in, so a regular expression with named groups is the tool that is always there. Each match becomes a `[pscustomobject]`:

```
$pattern = '(?s)<article class="product_pod">.*?<a href="(?<url>[^"]+)" title="(?<title>[^"]+)">.*?<p class="price_color">(?<price>[^<]+)</p>'
$books = [regex]::Matches($html, $pattern) | ForEach-Object {
  [pscustomobject]@{
    Title = [System.Net.WebUtility]::HtmlDecode($_.Groups['title'].Value)
    Price = $_.Groups['price'].Value
    Url   = 'https://books.toscrape.com/' + $_.Groups['url'].Value
  }
}
$books.Count
20
```

- **`(?s)`** lets `.` match line breaks, because one book spans many lines of HTML.
- **`.*?`** is lazy: it stops at the first price after each title instead of running to the last one on the page.
- **`HtmlDecode`** turns `&amp;` and `&#39;` in titles back into characters.

The same approach reads a two-column table, such as the product information on a book page:

```
$rows = [regex]::Matches($h, '<tr>\s*<th>(?<k>[^<]+)</th>\s*<td>(?<v>[^<]*)</td>') |
  ForEach-Object { [pscustomobject]@{ Field = $_.Groups['k'].Value; Value = $_.Groups['v'].Value } }

Field             Value
-----             -----
UPC               a897fe39b1053632
Product Type      Books
Price (excl. tax) £51.77
Price (incl. tax) £51.77
Tax               £0.00
Availability      In stock (22 available)
Number of reviews 0
```

## Step 4: several pages to a CSV file

```
$all = foreach ($n in 1..3) {
  $page = Invoke-WebRequest -Uri "https://books.toscrape.com/catalogue/page-$n.html" -UseBasicParsing
  $html = [System.Text.Encoding]::UTF8.GetString($page.RawContentStream.ToArray())
  [regex]::Matches($html, $pattern) | ForEach-Object {
    [pscustomobject]@{ Page = $n; Title = [System.Net.WebUtility]::HtmlDecode($_.Groups['title'].Value); Price = $_.Groups['price'].Value }
  }
  Start-Sleep -Milliseconds 500
}
$all | Export-Csv -Path books.csv -NoTypeInformation -Encoding UTF8

"Page","Title","Price"
"1","A Light in the Attic","£51.77"
"1","Tipping the Velvet","£53.74"
```

Three pages gave 60 rows. Once the price is a number, PowerShell can summarise it: `[decimal]($_.Price -replace '[^\d.]', '')` piped to `Measure-Object -Average -Minimum -Maximum` gave 60 prices from 12.84 to 57.31, average 35.00. The half-second pause keeps the request rate polite.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=powershell-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## JSON APIs: Invoke-RestMethod does the parsing

Many sites load their data from a JSON endpoint. When you find one, skip the HTML: `Invoke-RestMethod` parses JSON into objects.

```
$q = Invoke-RestMethod -Uri 'https://quotes.toscrape.com/api/quotes?page=1'
"page=$($q.page) has_next=$($q.has_next) quotes=$($q.quotes.Count)"
page=1 has_next=True quotes=10

$q.quotes | Select-Object -First 2 @{n='Author';e={$_.author.name}}, @{n='Tags';e={$_.tags -join ','}}

Author          Tags
------          ----
Albert Einstein change,deep-thoughts,thinking,world
J.K. Rowling    abilities,choices
```

## Headers, user agent and cookies

| Need | Parameter | What our test showed |
| --- | --- | --- |
| Your own user agent | `-UserAgent 'my-script/1.0 (+https://example.com/bot)'` | Sent as given; the default is `Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.26100.9818`, which names PowerShell |
| Other headers | `-Headers @{ 'Accept-Language' = 'en-US' }` | Sent as given |
| Cookies across requests | `-SessionVariable s` on the first call, `-WebSession $s` after | A cookie set by the first call came back on the second |
| HTTP errors | `try { ... } catch { $_.Exception.Response.StatusCode }` | A missing page threw "(404) Not Found" |

## Scraping through a proxy

Invoke-WebRequest takes `-Proxy` and `-ProxyCredential`. We ran the three pages through a QuanticData Residential Basic line with one session for the whole run, our own user agent, a one-second pause between pages, and `-DisableKeepAlive` so PowerShell opens a fresh connection for each request:

```
$proxy = 'http://residentialboson.quanticdata.io:10000'
$pass  = ConvertTo-SecureString 'PASS' -AsPlainText -Force
$cred  = New-Object System.Management.Automation.PSCredential('USER-type-residential-country-us-session-run1-lifetime-30', $pass)
$rows = foreach ($n in 1..3) {
  $p = Invoke-WebRequest -Uri "https://books.toscrape.com/catalogue/page-$n.html" -Proxy $proxy -ProxyCredential $cred -UseBasicParsing -DisableKeepAlive -UserAgent 'my-script/1.0 (+https://example.com/bot)'
  $html = [System.Text.Encoding]::UTF8.GetString($p.RawContentStream.ToArray())
  [regex]::Matches($html, $pattern) | ForEach-Object { [pscustomobject]@{ Page = $n; Title = $_.Groups['title'].Value; Price = $_.Groups['price'].Value } }
  Start-Sleep -Seconds 1
}
$rows.Count
60
```

The username carries the country and a session ID; keep one session ID for one job. Setting the credential once for a whole window, and SOCKS5, are in our [PowerShell proxy setup](https://quanticdata.io/blog/powershell-proxy-setup/) guide.

### The same from curl.exe

curl.exe, which ships with Windows 11, works from PowerShell too, and its output goes into the same regex. We fetched the first catalogue page this way with a QuanticData line and got the same 20 books:

```
$html = curl.exe -s -A "my-script/1.0 (+https://example.com/bot)" -x "http://HOST:PORT" -U "USER:PASS" https://books.toscrape.com/catalogue/page-1.html | Out-String
[regex]::Matches($html, $pattern).Count
20
```

## When PowerShell is not enough: JavaScript pages

quotes.toscrape.com/js/ shows the same quotes as the home page, but builds them in the browser. Invoke-WebRequest only gets the HTML the server sends: our regex found 0 quote blocks there, and the page source contained the data inside a `var data =` script instead. Two ways forward: find the JSON the page loads and call it with Invoke-RestMethod, or use a browser, as in [Playwright for scraping](https://quanticdata.io/blog/how-to-use-playwright-for-scraping/). For larger jobs, [web scraping with Python](https://quanticdata.io/blog/how-to-web-scraping-using-python/) has more parsing libraries to choose from.

## Before you schedule it

Keep to each site's terms and robots rules, pause between requests, and keep passwords out of scripts you share. When the job grows, our [residential proxies](https://quanticdata.io/residential-proxies/) and [mobile proxies](https://quanticdata.io/mobile-proxies/) give your scripts a home or carrier IP in the country you choose (there is a [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/)), and our [web scraping API](https://quanticdata.io/web-scraping-api/) returns rendered pages as clean data from one call.

### Sources & further reading

- [Microsoft Learn: Invoke-WebRequest (5.1)](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-5.1)
- [Microsoft Learn: Invoke-RestMethod (5.1)](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-5.1)
- [Microsoft Learn: Export-Csv](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/export-csv?view=powershell-5.1)
- [Books to Scrape sandbox](https://books.toscrape.com/)
- [Quotes to Scrape sandbox](https://quotes.toscrape.com/)

## FAQ

Quick answers on powershell web scraping.

[Something else? Ask us](mailto:hello@quanticdata.io)

### Can PowerShell do web scraping?

Yes. Invoke-WebRequest downloads pages, regular expressions extract the fields into objects, and Export-Csv saves them. We collected 60 books from 3 pages this way in Windows PowerShell 5.1. Pages that are built by JavaScript need a browser or the JSON endpoint behind them.

### How do I scrape a web page with Invoke-WebRequest?

Run `$r = Invoke-WebRequest -Uri URL -UseBasicParsing`, decode `$r.RawContentStream` as UTF-8 if characters look wrong, then use `[regex]::Matches()` with named groups and build a `[pscustomobject]` per match. `$r.Links` already lists every link with its href and title.

### Why do prices or accents look wrong when I scrape with PowerShell?

The page is UTF-8 but the server did not say so, and Windows PowerShell decoded it with another code page: we got `Â£51.77` instead of `£51.77`. Use `[System.Text.Encoding]::UTF8.GetString($r.RawContentStream.ToArray())`.

### How do I scrape a table with PowerShell?

Match each row with a regular expression such as `<tr>\s*<th>(?<k>[^<]+)</th>\s*<td>(?<v>[^<]*)</td>` and turn each match into an object with the two groups. On a product page it gave seven fields, from UPC to the number of reviews.

### How do I use a proxy for web scraping in PowerShell?

Pass `-Proxy "http://HOST:PORT"` and `-ProxyCredential` (a PSCredential with the proxy username and password) to Invoke-WebRequest, and add `-DisableKeepAlive` so each request opens a fresh connection. curl.exe with `-x` and `-U` works from PowerShell too.

## A home or carrier IP for your PowerShell scripts

Residential and mobile proxies with country targeting, for your PowerShell scripts.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=powershell-web-scraping) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

- [Web Scraping Project Ideas, Started for Real](https://quanticdata.io/blog/web-scraping-project-ideas/): Web scraping · Five web scraping project ideas, each one started for real on 20 November 2026: a price tracker, a library release dashboard, a question analysis, an open-source landscape and a topic timeline, with the code, the first run and what to build next.
- [Retry-After Header: Read It, Wait, Retry](https://quanticdata.io/blog/retry-after-header/): Web scraping · Retry-After tells a client how long to wait before the next request, as a number of seconds or as an HTTP date. Below: both formats, a Python and a Node helper we ran against a real 429 on 20 November 2026 (each waited 10 seconds and then got 200), the rate-limit headers of the QuanticData API, and how one batch job replaced 20 calls.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/powershell-web-scraping/ · Site index for AI: https://quanticdata.io/llms.txt
