---
title: "Web scraping in C#"
url: https://recursiveai.org/onehtml/articles/web-scraping-csharp/
description: "Web scraping in C# with HttpClient and CSS selectors: fetch a page, extract data, follow pagination politely, and handle pages that JavaScript builds."
date: 2026-10-27
author: onehtml team
---

# Web scraping in C#

Web scraping in C# with HttpClient and CSS selectors: fetch a page, extract data, follow pagination politely, and handle pages that JavaScript builds.

October 27, 2026 · onehtml team

Web scraping in C# takes two parts: `HttpClient` to download the page, and an HTML parser to turn it into a tree you can query with CSS selectors. .NET ships the first and not the second. This tutorial uses onehtml as the parser: it parses pages the way a browser does, detects the character encoding from the bytes and the `Content-Type` header, and supports Selectors Level 4, so `:has()` and `:not()` do filtering that would otherwise take a loop. It covers one page, then pagination with polite request pacing, then pages that build their content with JavaScript. The samples scrape [books.toscrape.com](https://books.toscrape.com/) and [quotes.toscrape.com](https://quotes.toscrape.com/), two sandboxes built for scraping practice, and were run on .NET 10 with onehtml 1.0.0 and Microsoft.Playwright 1.64.1.

## Set up the project

*Terminal · NuGet*

```
dotnet new console -o BookScraper
cd BookScraper
dotnet add package RecursiveAI.OneHtml --version 1.0.0
```

onehtml targets .NET 10 and has no dependencies outside .NET. It never makes network requests itself, so all HTTP stays in your `HttpClient` code, where you control headers, timeouts, proxies and retries.

## Scrape a page

Open the page in your browser’s DevTools first and find the elements that hold the data. On books.toscrape.com each book is an `article.product_pod`, with the title in the `title` attribute of the link inside its `h3`, and the price in `.price_color`.

*Program.cs · onehtml 1.0.0 · .NET 10*

```
using RecursiveAI.OneHtml;
 
using var http = new HttpClient();
http.DefaultRequestHeaders.UserAgent.ParseAdd("BookPriceBot/1.0 (+https://example.com/bot)");
 
var pageUrl = new Uri("https://books.toscrape.com/");
using HttpResponseMessage response = await http.GetAsync(pageUrl);
response.EnsureSuccessStatusCode();
 
// Pass the bytes and the Content-Type: the parser picks the encoding as a browser would
byte[] body = await response.Content.ReadAsByteArrayAsync();
Document doc = Document.Parse(body, new ParseOptions { ContentType = response.Content.Headers.ContentType?.ToString() });
 
List<Node> books = doc.Root.QuerySelectorAll("article.product_pod");
Console.WriteLine($"{books.Count} books on {pageUrl}");
 
foreach (Node book in books.Take(5))
{
    Node link = book.QuerySelector("h3 > a");
    string title = link.GetAttribute("title")!;                          // the link text is truncated, the title isn't
    string price = book.QuerySelector(".price_color").TextContent!;
    string rating = book.QuerySelector("p.star-rating").GetAttribute("class")!.Split(' ')[1];
    string stock = book.QuerySelector(".availability").TextContent!.Trim();
    var url = new Uri(pageUrl, link.GetAttribute("href"));                // resolve the relative link
    Console.WriteLine($"{price,7}  {rating,-5}  {stock}  {title}");
    Console.WriteLine($"         {url}");
}
 
// :has() filters on what an element contains, so no loop is needed
int fiveStars = doc.Root.QuerySelectorAll("article.product_pod:has(p.star-rating.Five)").Count;
Console.WriteLine($"Five-star books on this page: {fiveStars}");
```

*Output · Console*

```
20 books on https://books.toscrape.com/
 £51.77  Three  In stock  A Light in the Attic
         https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
 £53.74  One    In stock  Tipping the Velvet
         https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
 £50.10  One    In stock  Soumission
         https://books.toscrape.com/catalogue/soumission_998/index.html
 £47.82  Four   In stock  Sharp Objects
         https://books.toscrape.com/catalogue/sharp-objects_997/index.html
 £54.23  Five   In stock  Sapiens: A Brief History of Humankind
         https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html
Five-star books on this page: 4
```

- **Pass bytes, not a string.** `ReadAsByteArrayAsync` plus the `Content-Type` header lets the parser choose the encoding as a browser does: a byte order mark, then the header’s charset, then a `<meta>` declaration near the top of the page. This site sends `text/html` with no charset and declares UTF-8 in a `<meta http-equiv>` tag, which is how the “£” comes out right. The [encoding recipe](https://recursiveai.org/onehtml/cookbook/detect-encoding/) shows the rules.
- **Read attributes when the text is cut.** The link text on this site is truncated (“A Light in the …”), and the `title` attribute holds the full title.
- **Resolve links against the page URL.** `new Uri(pageUrl, href)` applies the same rules a browser does to relative links. If the page has a `<base href>`, resolve against that instead.
- **Selectors return a null node, not an exception,** when nothing matches. Check `IsNull` on anything optional before you read it, because reading attributes from the null node throws.
- **Identify your scraper.** A `User-Agent` with a contact URL lets a site owner reach you instead of blocking you.

The parser follows the WHATWG HTML standard, so a page with missing end tags or misnested elements becomes the same tree the browser showed you in DevTools, and the selectors you tested there select the same elements. The [CSS selectors article](https://recursiveai.org/onehtml/articles/css-selectors-csharp/) covers the selector syntax in more depth.

## Follow pagination

Most listings span several pages. Follow the site’s own “next” link until there isn’t one, wait between requests, and cap the number of pages so a mistake can’t turn into an endless crawl:

*Program.cs · onehtml 1.0.0 · .NET 10*

```
using System.Globalization;
using RecursiveAI.OneHtml;
 
using var http = new HttpClient();
http.DefaultRequestHeaders.UserAgent.ParseAdd("BookPriceBot/1.0 (+https://example.com/bot)");
 
var rows = new List<string> { "title,price_gbp,url" };
Uri? next = new Uri("https://books.toscrape.com/catalogue/category/books/mystery_3/index.html");
int pages = 0;
 
while (next is not null && pages < 10)
{
    Document doc = await FetchAsync(next);
    pages++;
 
    foreach (Node book in doc.Root.QuerySelectorAll("article.product_pod"))
    {
        Node link = book.QuerySelector("h3 > a");
        decimal price = decimal.Parse(book.QuerySelector(".price_color").TextContent!.TrimStart('£'), CultureInfo.InvariantCulture);
        var url = new Uri(next, link.GetAttribute("href"));
        rows.Add($"\"{link.GetAttribute("title")!.Replace("\"", "\"\"")}\",{price.ToString(CultureInfo.InvariantCulture)},{url}");
    }
 
    // Follow the "next" link until there isn't one
    Node nextLink = doc.Root.QuerySelector("li.next > a");
    Console.WriteLine($"Page {pages}: {next.Segments[^1]}, {rows.Count - 1} books so far");
    next = nextLink.IsNull ? null : new Uri(next, nextLink.GetAttribute("href"));
 
    await Task.Delay(TimeSpan.FromSeconds(1));   // one request a second keeps the load on the site low
}
 
File.WriteAllLines("mystery.csv", rows);
Console.WriteLine($"Wrote {rows.Count - 1} books to mystery.csv");
 
async Task<Document> FetchAsync(Uri url)
{
    using HttpResponseMessage response = await http.GetAsync(url);
    response.EnsureSuccessStatusCode();
    byte[] body = await response.Content.ReadAsByteArrayAsync();
    return Document.Parse(body, new ParseOptions { ContentType = response.Content.Headers.ContentType?.ToString() });
}
```

*Output · Console*

```
Page 1: index.html, 20 books so far
Page 2: page-2.html, 32 books so far
Wrote 32 books to mystery.csv
```

*mystery.csv · CSV*

```
title,price_gbp,url
"Sharp Objects",47.82,https://books.toscrape.com/catalogue/sharp-objects_997/index.html
"In a Dark, Dark Wood",19.63,https://books.toscrape.com/catalogue/in-a-dark-dark-wood_963/index.html
"The Past Never Ends",56.50,https://books.toscrape.com/catalogue/the-past-never-ends_942/index.html
```

- **Resolve the next link against the current page.** Page 2’s address is relative (`page-2.html`), and `new Uri(next, href)` resolves it against the page it came from.
- **Parse numbers with `CultureInfo.InvariantCulture`,** or a machine set to a locale that uses a decimal comma will read “47.82” wrong.
- **Reuse one `HttpClient`** for the whole run rather than creating one per request.
- **Keep the delay.** One request a second is slow for a program and fast for a person, and it keeps you from loading the server. If a site answers with HTTP 429 or 503, slow down further, and honor a `Retry-After` header if there is one.

## robots.txt and terms of use

Before you scrape a site that isn’t yours, read two things:

- **robots.txt.** Fetch `/robots.txt` from the site’s root. The format is standardized in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309): groups of `User-agent` lines followed by `Allow` and `Disallow` path rules. The RFC states that these rules “are not a form of access authorization”: they are requests that well-behaved crawlers honor. When robots.txt returns a status in the 400–499 range, as both sandboxes here do, the RFC says crawlers “MAY access any resources on the server”.
- **The site’s terms of use,** and any API it offers. An official API is usually more stable than HTML that changes with each redesign.

Whether scraping a particular site is allowed depends on the site’s terms, the data and where you are. This article isn’t legal advice.

## Scrape pages that JavaScript builds

onehtml doesn’t run scripts. When the data you see in the browser isn’t in the HTML the server sends, a script built it after the page loaded, and you need a browser to run that script. Playwright for .NET drives Chromium. Let it render the page, then pass the finished DOM from `ContentAsync` to the parser:

*Terminal · NuGet*

```
dotnet add package Microsoft.Playwright --version 1.64.1
dotnet build
pwsh bin/Debug/net10.0/playwright.ps1 install chromium
```

*Program.cs · onehtml 1.0.0 · Playwright 1.64.1*

```
using Microsoft.Playwright;
using RecursiveAI.OneHtml;
 
const string url = "https://quotes.toscrape.com/js/";
 
// The HTML the server sends has no quotes in it: a script builds them in the browser
using var http = new HttpClient();
Document raw = Document.Parse(await http.GetByteArrayAsync(url));
Console.WriteLine($"Quotes in the server's HTML: {raw.Root.QuerySelectorAll(".quote").Count}");
 
// Let a real browser run the script, then parse the DOM it built
using IPlaywright playwright = await Playwright.CreateAsync();
await using IBrowser browser = await playwright.Chromium.LaunchAsync();
IPage page = await browser.NewPageAsync();
await page.GotoAsync(url);
await page.WaitForSelectorAsync(".quote");
Document rendered = Document.Parse(await page.ContentAsync());
 
List<Node> quotes = rendered.Root.QuerySelectorAll(".quote");
Console.WriteLine($"Quotes in the rendered DOM:  {quotes.Count}");
foreach (Node quote in quotes.Take(3))
{
    string author = quote.QuerySelector(".author").TextContent!;
    string tags = string.Join(", ", quote.QuerySelectorAll(".tag").Select(t => t.TextContent));
    Console.WriteLine($"{author}: {tags}");
}
```

*Output · Console*

```
Quotes in the server's HTML: 0
Quotes in the rendered DOM:  10
Albert Einstein: change, deep-thoughts, thinking, world
J.K. Rowling: abilities, choices
Albert Einstein: inspirational, life, live, miracle, miracles
```

`WaitForSelectorAsync` waits until the script has added the first quote, so the DOM is complete before you read it. Check for a JSON API first, though. Pages that render on the client often fetch their data from one, and the browser’s Network tab shows it. Calling that API with `HttpClient` is faster than running a browser. On this sandbox the data is inline in a `<script>` element instead, as a JavaScript array.

## Choosing a parser for scraping

HtmlAgilityPack and AngleSharp are the other common .NET choices. Here is how they differ on the points that matter for scraping, as each publisher states them, for the versions on NuGet in October 2026:

|  | onehtml 1.0.0 | HtmlAgilityPack 1.13.1 | AngleSharp 1.8.6 |
|---|---|---|---|
| Queries | CSS Selectors Level 4; no XPath | XPath; CSS through the third-party Fizzler package ([HAP site](https://html-agility-pack.net/third-party-library)) | CSS selectors; XPath through [AngleSharp.XPath](https://www.nuget.org/packages/AngleSharp.XPath/2.0.10) |
| Downloads pages itself | No; you pass the bytes | `HtmlWeb` ([docs](https://html-agility-pack.net/from-web)) | `BrowsingContext` with a loader ([README](https://github.com/AngleSharp/AngleSharp)) |
| Builds in the package | .NET 10 | .NET Framework 3.5 to 4.5, .NET 8, .NET Standard 2.0 and others ([NuGet](https://www.nuget.org/packages/HtmlAgilityPack/1.13.1)) | .NET 10 and 8, .NET Standard 2.0, .NET Framework 4.6.2 and 4.7.2 ([NuGet](https://www.nuget.org/packages/AngleSharp/1.8.6)) |
| License | MIT OR Apache-2.0 | MIT | MIT |

On parsing speed, the project’s [published benchmark](https://github.com/recursive-ai-oss/onehtml-csharp/blob/v1.0.0/README.md#performance) parses a 100-page sample of crawled pages from bytes in 125.0 ms per sweep with onehtml, 341.7 ms with AngleSharp 1.8.6 and 292.8 ms with HtmlAgilityPack 1.13.1, on an AMD Ryzen 9 7950X3D with .NET 10. For a polite scraper the network sets the pace, so parse speed matters most when you reprocess pages you’ve already saved. The [HtmlAgilityPack vs AngleSharp comparison](https://recursiveai.org/onehtml/articles/htmlagilitypack-vs-anglesharp/) covers the rest.

## Where next

- [Parse HTML in C#](https://recursiveai.org/onehtml/articles/parse-html-csharp/): files, strings, encodings, malformed markup and fragments.
- [CSS selectors in C#](https://recursiveai.org/onehtml/articles/css-selectors-csharp/): Level 4 selectors, selecting by text, and CSS versus XPath.
- [Extract visible text from HTML in C#](https://recursiveai.org/onehtml/articles/extract-visible-text-from-html-csharp/), without script and style.
- [Extract all links and images from a page](https://recursiveai.org/onehtml/cookbook/extract-links-and-images/), with resolved URLs.