Recursive AI

Detect the encoding of an HTML page

All onehtml recipes

Decode HTML bytes the way a browser does, using the byte order mark, the HTTP Content-Type header, the meta charset prescan and the windows-1252 fallback, and override the fallback when you know better.

Document.Parse(bytes) implements the WHATWG Encoding Standard and the HTML encoding-sniffing algorithm. The input here is a French menu saved as windows-1252, which declares itself with <meta charset="windows-1252">:

HTML
<!DOCTYPE html>
<html lang="fr">
<head>
<meta charset="windows-1252">
<title>Menu</title>
</head>
<body>
<h1>Café crème – 3,50 €</h1>
</body>
</html>
C#
using System.Text;
using RecursiveAI.OneHtml;
using RecursiveAI.OneHtml.Encodings;
 
// A windows-1252 page that declares its charset in a <meta>
byte[] declared = File.ReadAllBytes("menu-1252.html");
Document doc = Document.Parse(declared);
Console.WriteLine($"{doc.Source.Encoding.Name} ({doc.Source.Confidence}): {doc.Root.QuerySelector("h1").TextContent}");
 
// Sniff without parsing: BOM, then transport Content-Type, then the <meta> prescan
EncodingSniffResult sniffed = EncodingSniffer.Sniff(declared);
Console.WriteLine($"Sniffed: {sniffed.Encoding.Name}, {sniffed.Confidence}");
 
// The HTTP header wins over the <meta> when you pass it
Document fromHeader = Document.Parse(declared, new ParseOptions { ContentType = "text/html; charset=utf-8" });
Console.WriteLine($"With a UTF-8 header: {fromHeader.Source.Encoding.Name}, {fromHeader.Source.DecodeErrorCount} decode errors");
 
// A byte order mark beats everything
byte[] bom = [0xEF, 0xBB, 0xBF, .. Encoding.UTF8.GetBytes("<meta charset=windows-1252><p>café")];
Console.WriteLine($"With a BOM: {Document.Parse(bom).Source.Encoding.Name}");
 
// No BOM, no header, no <meta>: the last-resort default is windows-1252
byte[] bare = Encoding.UTF8.GetBytes("<p>café");
Document guessed = Document.Parse(bare);
Console.WriteLine($"Undeclared: {guessed.Source.Encoding.Name} ({guessed.Source.Confidence}): {guessed.Root.QuerySelector("p").TextContent}");
 
// Choose the fallback yourself when you know your sources
Document utf8 = Document.Parse(bare, new ParseOptions { DefaultEncoding = WhatwgEncoding.Utf8 });
Console.WriteLine($"Default set to UTF-8: {utf8.Root.QuerySelector("p").TextContent}");
 
// Labels resolve the way browsers resolve them
Console.WriteLine($"latin1 -> {WhatwgEncoding.ForLabel("latin1")}, gb2312 -> {WhatwgEncoding.ForLabel("gb2312")}");
Text
windows-1252 (Certain): Café crème – 3,50 €
Sniffed: windows-1252, Tentative
With a UTF-8 header: UTF-8, 4 decode errors
With a BOM: UTF-8
Undeclared: windows-1252 (Tentative): café
Default set to UTF-8: café
latin1 -> windows-1252, gb2312 -> GBK

The order of precedence:

  1. A byte order mark (UTF-8, UTF-16LE, UTF-16BE) decides, whatever the page declares.
  2. The transport Content-Type, which you pass as ParseOptions.ContentType because the library doesn’t make HTTP requests. Pass the header your HTTP client received. Here a wrong UTF-8 header produced 4 decode errors on the windows-1252 bytes, as it would in a browser.
  3. The prescan of the first 1,024 bytes for a <meta charset>, a <meta http-equiv="Content-Type">, or an XML declaration. Change the length with ParseOptions.PrescanLimit.
  4. The fallback, ParseOptions.DefaultEncoding, which is windows-1252 unless you set it. The HTML standard recommends windows-1252 for most locales. That is why undeclared UTF-8 bytes show up as café. If you know your sources are UTF-8, set DefaultEncoding = WhatwgEncoding.Utf8.

ParseOptions.OverrideEncoding forces an encoding, though a byte order mark still wins. doc.Source tells you what was used: Encoding, Confidence (Tentative for a guess or a prescan result, Certain once confirmed), BomLength and DecodeErrorCount. EncodingSniffer.Sniff runs the same steps without parsing.

Labels follow the Encoding Standard’s table, so latin1 and iso-8859-1 mean windows-1252, and gb2312 means GBK. When you edit a page, the output is written back in the encoding it was decoded with (see Edit HTML without reformatting it).

More on loading real pages: Parse HTML in C#.

Encoding in a full scraper: Web scraping in C#.