Decode HTML bytes the way a browser does, using the byte order mark, the HTTP Content-Type header, the meta charset prescan and the windows-1252 fallback, and override the fallback when you know better.
Document.Parse(bytes) implements the WHATWG Encoding Standard and the HTML encoding-sniffing algorithm. The input here is a French menu saved as windows-1252, which declares itself with <meta charset="windows-1252">:
<!DOCTYPE html> <html lang="fr"> <head> <meta charset="windows-1252"> <title>Menu</title> </head> <body> <h1>Café crème – 3,50 €</h1> </body> </html>
using System.Text; using RecursiveAI.OneHtml; using RecursiveAI.OneHtml.Encodings; // A windows-1252 page that declares its charset in a <meta> byte[] declared = File.ReadAllBytes("menu-1252.html"); Document doc = Document.Parse(declared); Console.WriteLine($"{doc.Source.Encoding.Name} ({doc.Source.Confidence}): {doc.Root.QuerySelector("h1").TextContent}"); // Sniff without parsing: BOM, then transport Content-Type, then the <meta> prescan EncodingSniffResult sniffed = EncodingSniffer.Sniff(declared); Console.WriteLine($"Sniffed: {sniffed.Encoding.Name}, {sniffed.Confidence}"); // The HTTP header wins over the <meta> when you pass it Document fromHeader = Document.Parse(declared, new ParseOptions { ContentType = "text/html; charset=utf-8" }); Console.WriteLine($"With a UTF-8 header: {fromHeader.Source.Encoding.Name}, {fromHeader.Source.DecodeErrorCount} decode errors"); // A byte order mark beats everything byte[] bom = [0xEF, 0xBB, 0xBF, .. Encoding.UTF8.GetBytes("<meta charset=windows-1252><p>café")]; Console.WriteLine($"With a BOM: {Document.Parse(bom).Source.Encoding.Name}"); // No BOM, no header, no <meta>: the last-resort default is windows-1252 byte[] bare = Encoding.UTF8.GetBytes("<p>café"); Document guessed = Document.Parse(bare); Console.WriteLine($"Undeclared: {guessed.Source.Encoding.Name} ({guessed.Source.Confidence}): {guessed.Root.QuerySelector("p").TextContent}"); // Choose the fallback yourself when you know your sources Document utf8 = Document.Parse(bare, new ParseOptions { DefaultEncoding = WhatwgEncoding.Utf8 }); Console.WriteLine($"Default set to UTF-8: {utf8.Root.QuerySelector("p").TextContent}"); // Labels resolve the way browsers resolve them Console.WriteLine($"latin1 -> {WhatwgEncoding.ForLabel("latin1")}, gb2312 -> {WhatwgEncoding.ForLabel("gb2312")}");
windows-1252 (Certain): Café crème – 3,50 € Sniffed: windows-1252, Tentative With a UTF-8 header: UTF-8, 4 decode errors With a BOM: UTF-8 Undeclared: windows-1252 (Tentative): café Default set to UTF-8: café latin1 -> windows-1252, gb2312 -> GBK
The order of precedence:
- A byte order mark (UTF-8, UTF-16LE, UTF-16BE) decides, whatever the page declares.
- The transport
Content-Type, which you pass asParseOptions.ContentTypebecause the library doesn’t make HTTP requests. Pass the header your HTTP client received. Here a wrong UTF-8 header produced 4 decode errors on the windows-1252 bytes, as it would in a browser. - The prescan of the first 1,024 bytes for a
<meta charset>, a<meta http-equiv="Content-Type">, or an XML declaration. Change the length withParseOptions.PrescanLimit. - The fallback,
ParseOptions.DefaultEncoding, which is windows-1252 unless you set it. The HTML standard recommends windows-1252 for most locales. That is why undeclared UTF-8 bytes show up ascafé. If you know your sources are UTF-8, setDefaultEncoding = WhatwgEncoding.Utf8.
ParseOptions.OverrideEncoding forces an encoding, though a byte order mark still wins. doc.Source tells you what was used: Encoding, Confidence (Tentative for a guess or a prescan result, Certain once confirmed), BomLength and DecodeErrorCount. EncodingSniffer.Sniff runs the same steps without parsing.
Labels follow the Encoding Standard’s table, so latin1 and iso-8859-1 mean windows-1252, and gb2312 means GBK. When you edit a page, the output is written back in the encoding it was decoded with (see Edit HTML without reformatting it).
More on loading real pages: Parse HTML in C#.
Encoding in a full scraper: Web scraping in C#.