---
title: "Web scraping in C#"
url: https://recursiveai.org/onehtml/cookbook/web-scraping/
description: "Web scraping in C#: download a page with HttpClient, parse it, pull data out with CSS selectors, follow pagination politely, and scrape pages that JavaScript builds. Runnable code."
version: "1.5.1"
---

# Web scraping in C#

Download the page with `HttpClient`, parse the bytes with onehtml, and pull out the data with CSS selectors. The program below scrapes the first page of [books.toscrape.com](https://books.toscrape.com/), a sandbox built for scraping practice.

```shell
dotnet new console -o BookScraper && cd BookScraper
dotnet add package RecursiveAI.OneHtml --version 1.5.1
```

Put this in `Program.cs` and run `dotnet run`:

```csharp
using RecursiveAI.OneHtml;

using var http = new HttpClient();
http.DefaultRequestHeaders.UserAgent
    .ParseAdd("BookPriceBot/1.0 (+https://example.com/bot)");

var pageUrl = new Uri("https://books.toscrape.com/");
using HttpResponseMessage response = await http.GetAsync(pageUrl);
response.EnsureSuccessStatusCode();

// Pass the bytes and the Content-Type: the parser picks the encoding as a browser does
byte[] body = await response.Content.ReadAsByteArrayAsync();
string? type = response.Content.Headers.ContentType?.ToString();
var options = new ParseOptions { ContentType = type };
Document doc = Document.Parse(body, options);

List<Node> books = doc.Root.QuerySelectorAll("article.product_pod");
Console.WriteLine($"{books.Count} books on {pageUrl}");

foreach (Node book in books.Take(4))
{
    Node link = book.QuerySelector("h3 > a");
    string title = link.GetAttribute("title")!;   // the link text is cut short
    string price = book.QuerySelector(".price_color").TextContent!;
    string stock = book.QuerySelector(".availability").TextContent!.Trim();
    var url = new Uri(pageUrl, link.GetAttribute("href"));   // a relative link
    Console.WriteLine($"{price,7}  {stock}  {title}");
    Console.WriteLine($"         {url}");
}

// :has() filters on what an element contains, so no loop is needed
int fiveStars = doc.Root
    .QuerySelectorAll("article.product_pod:has(.star-rating.Five)").Count;
Console.WriteLine($"Five-star books on this page: {fiveStars}");
```

```text
20 books on https://books.toscrape.com/
 £51.77  In stock  A Light in the Attic
         https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
 £53.74  In stock  Tipping the Velvet
         https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
 £50.10  In stock  Soumission
         https://books.toscrape.com/catalogue/soumission_998/index.html
 £47.82  In stock  Sharp Objects
         https://books.toscrape.com/catalogue/sharp-objects_997/index.html
Five-star books on this page: 4
```

- **Find the selectors in DevTools.** Inspect the element in your browser and write the selector there first. onehtml builds the same tree the browser does, so the selector you tested selects the same elements here.
- **Pass bytes, not a string.** With the bytes and the `Content-Type`, the parser chooses the encoding as a browser does. This site declares UTF-8 only in a `<meta>` tag, which is why the “£” comes out right.
- **Read attributes when the text is cut.** The link text here is truncated (“A Light in the …”); the `title` attribute has the whole title.
- **Optional elements give the null node,** not an exception. Check `IsNull` before reading from anything that may be missing.
- **Say who you are.** A `User-Agent` with a contact URL lets a site owner reach you instead of blocking you.

## Follow pagination

Follow the site’s own “next” link until there isn’t one, wait between requests, and cap the page count so a mistake can’t become an endless crawl. This writes a CSV:

```csharp
using System.Globalization;
using RecursiveAI.OneHtml;

using var http = new HttpClient();
http.DefaultRequestHeaders.UserAgent
    .ParseAdd("BookPriceBot/1.0 (+https://example.com/bot)");

var rows = new List<string> { "title,price_gbp,url" };
Uri? next = new Uri(
    "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html");
int pages = 0;

while (next is not null && pages < 10)
{
    Document doc = await FetchAsync(next);
    pages++;

    foreach (Node book in doc.Root.QuerySelectorAll("article.product_pod"))
    {
        Node link = book.QuerySelector("h3 > a");
        string price = book.QuerySelector(".price_color").TextContent!.TrimStart('£');
        decimal gbp = decimal.Parse(price, CultureInfo.InvariantCulture);
        string title = link.GetAttribute("title")!.Replace("\"", "\"\"");
        var url = new Uri(next, link.GetAttribute("href"));
        rows.Add($"\"{title}\",{gbp.ToString(CultureInfo.InvariantCulture)},{url}");
    }

    Console.WriteLine($"Page {pages}: {next.Segments[^1]}, {rows.Count - 1} books");
    Node nextLink = doc.Root.QuerySelector("li.next > a");
    next = nextLink.IsNull ? null : new Uri(next, nextLink.GetAttribute("href"));

    await Task.Delay(TimeSpan.FromSeconds(1));   // one request a second
}

File.WriteAllLines("mystery.csv", rows);
Console.WriteLine($"Wrote {rows.Count - 1} books to mystery.csv");
Console.WriteLine(rows[1]);

async Task<Document> FetchAsync(Uri url)
{
    using HttpResponseMessage response = await http.GetAsync(url);
    response.EnsureSuccessStatusCode();
    byte[] body = await response.Content.ReadAsByteArrayAsync();
    string? type = response.Content.Headers.ContentType?.ToString();
    return Document.Parse(body, new ParseOptions { ContentType = type });
}
```

```text
Page 1: index.html, 20 books
Page 2: page-2.html, 32 books
Wrote 32 books to mystery.csv
"Sharp Objects",47.82,https://books.toscrape.com/catalogue/sharp-objects_997/index.html
```

Page 2’s address is relative (`page-2.html`), so resolve the next link against the page it came from. Parse numbers with `CultureInfo.InvariantCulture`, or a machine set to a decimal-comma locale reads “47.82” wrong. Reuse one `HttpClient` for the whole run. If a site answers 429 or 503, slow down and honor `Retry-After`.

## Pages that JavaScript builds

When the data you see in the browser isn’t in the HTML the server sends, a script built it, and you need a browser to run that script. Let Playwright render the page, then parse the finished DOM:

```shell
dotnet add package Microsoft.Playwright --version 1.64.1
```

```csharp
using Microsoft.Playwright;
using RecursiveAI.OneHtml;

const string url = "https://quotes.toscrape.com/js/";

// The HTML the server sends: a script builds the quotes later, in the browser
using var http = new HttpClient();
Document raw = Document.Parse(await http.GetByteArrayAsync(url));
int inHtml = raw.Root.QuerySelectorAll(".quote").Count;
Console.WriteLine($"Quotes in the server's HTML: {inHtml}");

// Downloads Chromium the first time; does nothing once it is there
Microsoft.Playwright.Program.Main(["install", "chromium"]);
using IPlaywright playwright = await Playwright.CreateAsync();
await using IBrowser browser = await playwright.Chromium.LaunchAsync();
IPage page = await browser.NewPageAsync();
await page.GotoAsync(url);
await page.WaitForSelectorAsync(".quote");

Document rendered = Document.Parse(await page.ContentAsync());
List<Node> quotes = rendered.Root.QuerySelectorAll(".quote");
Console.WriteLine($"Quotes in the rendered DOM:  {quotes.Count}");
foreach (Node quote in quotes.Take(3))
{
    string author = quote.QuerySelector(".author").TextContent!;
    var tags = quote.QuerySelectorAll(".tag").Select(t => t.TextContent);
    Console.WriteLine($"{author}: {string.Join(", ", tags)}");
}
```

```text
Quotes in the server's HTML: 0
Quotes in the rendered DOM:  10
Albert Einstein: change, deep-thoughts, thinking, world
J.K. Rowling: abilities, choices
Albert Einstein: inspirational, life, live, miracle, miracles
```

The first run also prints Chromium’s download progress before this output. Check the browser’s Network tab for a JSON API first: pages that render on the client often fetch their data from one, and calling it with `HttpClient` is faster than running a browser.

## Before you scrape a site that isn’t yours

- **robots.txt.** Fetch `/robots.txt` from the site’s root. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) defines its `User-agent`, `Allow` and `Disallow` rules, and says they “are not a form of access authorization”: they are requests that well-behaved crawlers honor.
- **Terms of use and APIs.** An official API is usually more stable than HTML that changes with each redesign. Whether scraping a site is allowed depends on its terms, the data and where you are; this isn’t legal advice.

## Limits

- **onehtml doesn’t fetch or run scripts.** HTTP stays in your code, where you control headers, timeouts, proxies and retries; JavaScript needs a browser, as above.
- **No XPath.** Selectors plus LINQ cover the same ground; see [Use CSS selectors in C#](https://recursiveai.org/onehtml/cookbook/css-selectors/).
- **.NET 10 only.**

For the text of a page, see [Extract text from HTML](https://recursiveai.org/onehtml/cookbook/extract-visible-text/); for its links and images, [Extract links and images](https://recursiveai.org/onehtml/cookbook/extract-links-and-images/). How onehtml compares with HtmlAgilityPack and AngleSharp: [HtmlAgilityPack vs AngleSharp](https://recursiveai.org/onehtml/articles/htmlagilitypack-vs-anglesharp/).