# onearchive docs, all pages (OneArchive 0.7.0-preview.1)

Every page of https://recursiveai.org/onearchive/docs/ and https://recursiveai.org/onearchive/cookbook/ as markdown, getting started first. Index: https://recursiveai.org/onearchive/llms.txt


<!-- https://recursiveai.org/onearchive/docs/getting-started/ -->

# Getting started with OneArchive

Install the OneArchive and OneCompress previews for .NET 10, compress a buffer with Zstandard, then create, list, read and safely extract a `.tar.zst` archive, all in managed code with no native libraries.

## Install

Both packages target `net10.0`, so you need the .NET 10 SDK. They are previews: this guide uses 0.7.0-preview.1 of both, and the API may change between previews.

```shell
dotnet new console -o OneArchiveStart
cd OneArchiveStart
dotnet add package RecursiveAI.OneArchive --version 0.7.0-preview.1
```

`RecursiveAI.OneArchive` brings in `RecursiveAI.OneCompress`, the codecs. Use OneCompress on its own if you only need codecs and checksums.

## Opt in to the preview APIs

Every codec and the archive layer carry `[Experimental]` until they meet their speed targets, so using one is a compiler error until you opt in. Each has its own diagnostic ID: Zstandard is `ONECOMPRESS001`, Deflate, zlib and gzip `ONECOMPRESS003`, the archive layer `ONEARCHIVE001` (the full table is in [the overview](https://recursiveai.org/onearchive/docs/overview/#preview-codecs)). Add the ones you use to the project file:

```xml
<PropertyGroup>
  <NoWarn>$(NoWarn);ONECOMPRESS001;ONEARCHIVE001</NoWarn>
</PropertyGroup>
```

## A first program

The program compresses a buffer, then archives an `assets/` folder (three JSON level files) and `notes.txt`:

```csharp
using System.Text;
using RecursiveAI.OneArchive;
using RecursiveAI.OneCompress.Zstd;

// One codec call: compress and decompress a buffer.
byte[] data = Encoding.UTF8.GetBytes(string.Concat(Enumerable.Repeat("level=3 tiles=64x64 seed=42\n", 2000)));
byte[] packed = Zstd.Compress(data, new ZstdCompressionOptions { Level = 19 });
byte[] again = Zstd.Decompress(packed);
Console.WriteLine($"zstd: {data.Length:N0} -> {packed.Length:N0} bytes, round trip {again.AsSpan().SequenceEqual(data)}");

// Create an archive: format and compression come from the extension.
Archive.Create("backup.tar.zst").Add("assets/").Add("notes.txt", "docs/notes.txt").Save();
Console.WriteLine($"backup.tar.zst: {new FileInfo("backup.tar.zst").Length:N0} bytes");

// List it, and read one entry by name.
using (var archive = Archive.Open("backup.tar.zst"))
{
    foreach (var e in archive.Entries) Console.WriteLine($"  {e.Kind,-9} {e.Size,7:N0}  {e.Name}");
    using var s = archive.OpenEntry(archive.Find("docs/notes.txt")!);
    Console.WriteLine($"docs/notes.txt says: {new StreamReader(s).ReadToEnd().Trim()}");
}

// Extract it: safe by default, nothing is written outside "out".
Archive.Extract("backup.tar.zst", "out");
Console.WriteLine($"extracted {Directory.GetFiles("out", "*", SearchOption.AllDirectories).Length} files");
```

```text
zstd: 56,000 -> 48 bytes, round trip True
backup.tar.zst: 4,846 bytes
  Directory       0  assets
  Directory       0  assets/levels
  File       20,632  assets/levels/level1.json
  File       20,632  assets/levels/level2.json
  File       20,632  assets/levels/level3.json
  File           15  docs/notes.txt
docs/notes.txt says: Read me first.
extracted 4 files
```

## What each part does

- **`Zstd.Compress` and `Zstd.Decompress`** are one call each; `Level` runs from negative fast levels to 22. Every codec has the same shape: `Lz4`, `Deflate`, `GZip`, `ZLib`, `BZip2`, `Xz`, `Brotli` and the rest, plus stream types for data that doesn’t fit in memory.
- **`Archive.Create(path)`** picks the format and compression from the extension: `.tar`, `.tar.zst`, `.tar.gz`, `.tar.xz`, `.tar.lz4`, `.tar.lz`, `.zip`, `.cfb`, or `.gz`, `.zst`, `.lz4`, `.xz` and `.lz` for one compressed file. `Add(folder)` adds a directory tree; `Add(file, name)` stores a file under another name.
- **`Archive.Open`** finds the format from the bytes, not the name. `Entries` lists them, and `OpenEntry` reads one by random access when the source can seek.
- **`Archive.Extract`** is safe by default: nothing is written outside the target, links can’t escape it, and limits on size, ratio, entry count and path length are on. See [Extract untrusted archives](https://recursiveai.org/onearchive/cookbook/safe-extraction/).

## What it reads and writes

Read and written: tar, zip (with WinZip AES), 7z (with AES-256), cpio, ar, ISO 9660 with Joliet and Rock Ridge, and Microsoft compound files. Read only: cab, and RAR 5 (listing, metadata and stored entries). The compressed bytes are identical on every platform, with and without hardware intrinsics, at any thread count and in WebAssembly.

## Next

- [Extract untrusted archives](https://recursiveai.org/onearchive/cookbook/safe-extraction/)
- [Zip files with AES encryption](https://recursiveai.org/onearchive/cookbook/zip-encryption/)
- [Zstandard dictionaries for small messages](https://recursiveai.org/onearchive/cookbook/zstd-dictionaries/)
- [Archives and safety](https://recursiveai.org/onearchive/docs/archive/): the model, the safety rules and the limits.


<!-- https://recursiveai.org/onearchive/docs/overview/ -->

# onearchive-csharp

Compression codecs and archive formats for .NET, written from first principles in managed code.
No native dependencies; the compressed bytes are identical on every platform.

Two packages:

- `RecursiveAI.OneCompress` - codecs and checksums.
- `RecursiveAI.OneArchive` - archive containers and a simple `Archive` API, built on OneCompress.

**Status: preview.** This page describes 0.7.0-preview.1. No codec changed since 0.5.1-preview.1, so the measurements below, from 0.5.0-preview.1 and 0.5.1-preview.1, stand. OneCompress has Zstandard (frames, skippable frames, dictionaries and
dictionary training, long-distance matching, multithreaded compression in fixed-size jobs, the seekable
format), LZ4 (block and frame), Deflate, zlib, gzip and Deflate64, LZMA, LZMA2, xz (with the branch and delta
filters) and lzip, bzip2, Brotli (with its built-in dictionary), the LZX decoder, and CRC32, CRC32C, CRC64,
Adler-32 and xxHash64, plus PPMd (variants H and I). OneArchive reads and writes tar, zip, 7z, cpio, ar and
ISO 9660 (with Joliet and Rock Ridge), with zip and 7z encryption, reads cab and RAR 5 (listing, metadata and stored
entries), and handles single compressed streams. Everything is preview: see below.

```csharp
byte[] packed = Zstd.Compress(data, new ZstdCompressionOptions { Level = 19 });
byte[] again = Zstd.Decompress(packed);
```

## Build

Requires the .NET 10 SDK (see `global.json`).

```sh
build/ci.sh              # build, test (Checked configuration), NativeAOT/trim smoke
build/ci.sh wasm         # browser-wasm build and a headless run under Node (needs the wasm-tools workload)
```

Build output goes to `artifacts/` (ignored by git). Set `ONEARCHIVE_ARTIFACTS` to put it elsewhere,
for example one directory per git worktree.

## Preview codecs

Zstandard, LZ4, Deflate, zlib, gzip and the LZMA family (LZMA, LZMA2, xz, lzip) are preview while their speed is
below the reference tools' in some rows. Their compression ratio and byte-identical output are already at the
target; speed is what is still being tuned. Brotli (compress is below the tool's speed at qualities 0 and 5, and quality 11 is
2% larger) and PPMd (decompression at the target on both machines; compression 0.85-0.91 of 7-Zip's speed, same sizes) are preview too.
The bzip2 codec, the lzip decoder, Deflate64 and LZX meet their targets on both machines and stay preview with the rest. The archive layer
is preview while the formats' interop and fuzz coverage grows. BCJ2 is not supported: 7z entries that use it are listed,
and opening them is a typed "unsupported method" error. RAR 5 is read only: compressed entries raise a typed error.
Their public types carry `[Experimental]`, so using them is a compiler error until you opt in per codec:

| Codec | Diagnostic ID |
|---|---|
| Zstandard (including dictionary training and the seekable format) | `ONECOMPRESS001` |
| LZ4 | `ONECOMPRESS002` |
| Deflate, zlib, gzip and Deflate64 | `ONECOMPRESS003` |
| bzip2 | `ONECOMPRESS004` |
| LZMA and LZMA2 | `ONECOMPRESS005` |
| xz and the branch and delta filters | `ONECOMPRESS006` |
| lzip | `ONECOMPRESS007` |
| Brotli | `ONECOMPRESS008` |
| LZX (decoder) | `ONECOMPRESS009` |
| PPMd (variants H and I) | `ONECOMPRESS010` |
| OneArchive: `Archive`, the entry model, readers and writers, every container, encryption | `ONEARCHIVE001` |

```xml
<PropertyGroup>
  <NoWarn>$(NoWarn);ONECOMPRESS001;ONEARCHIVE001</NoWarn>
</PropertyGroup>
```

or `#pragma warning disable ONECOMPRESS001` around the code that uses it. The marker comes off once the codec
meets its speed target.

### Where they stand

The target for every codec: compressed size within 1% of the reference tool, and compress and decompress
speed at least 0.87 of the reference tool's, at every level. Measured memory to memory against
`zstd` 1.5.7, `lz4` 1.10.0, libdeflate 1.26 (one call), zlib-ng 2.3.3 (streaming), `bzip2` 1.0.8, `xz` 5.8.4, `lzip` 1.26 and
7-Zip `7zz` 26.05, single-threaded unless noted, on Silesia and enwik8 and on small game
snapshot payloads with trained dictionaries. Speed is ours divided by the reference (1.00 = same speed).
x64 is an AMD Ryzen 9 7950X3D on Linux; arm64 is an Apple M2 on macOS. .NET 10.0.402 with the JIT.

The table is the 0.5.0-preview.1 measurement, with the rows for the codecs that changed in 0.5.1-preview.1 remeasured on Silesia:
Zstandard levels 1, 3, 5, 9, 12 and 15, LZ4, Deflate in one call, and dictionary training. Those rows say "0.5.1".

| Rows | x64 | arm64 |
|---|---|---|
| Zstandard compress, levels 1-4 | 0.5.1, levels 1 and 3: 0.76 and 0.81 (0.5: 0.71-0.80); size within +0.94% | 0.5.1: 0.82 and 0.81 (0.5: 0.76-0.86) |
| Zstandard compress, levels -5 to 15 (other than 1-4) | 0.5.1, levels 5, 9, 12, 15: 0.68, 0.63, 0.81, 0.72; the rest 0.56-0.84 (0.5); size within +0.93% | 0.5.1: 0.74, 0.71, 0.95 (at the target), 0.65; the rest 0.63-0.87 (level -5 at the target) |
| Zstandard compress, levels 16-19 | 0.76-0.85 (enwik8 16-18 at the target) | 0.78-0.87 |
| Zstandard compress, levels 20 and 22 | 0.86-1.07; enwik8 level 22 size +1.17% | 0.90-1.02; enwik8 level 22 size +1.17% |
| Zstandard decompress | 0.5.1, Silesia levels 1-15: 0.89-0.98, at the target; 0.5: 12 of 16 rows at the target, the rest 0.86-0.87 | 0.5.1: 0.88-0.93, at the target |
| Zstandard long-distance matching | 0.60-0.75; Silesia level 3 size +1.04% | 0.66-0.78 |
| Zstandard multithreaded jobs (levels 1 and 3, 4 to 16 workers) | 0.62-0.86; size within +0.82% | 0.68-0.85 |
| Game payloads with dictionaries, compress | per-tick frames 1.0-1.4x faster; snapshots and saves 0.26-0.58 at level 3, 0.06-0.39 at level 19 (size -4.8% to +0.44%) | 10 of 48 rows at the target |
| Game payloads with dictionaries, decompress | 29 of 48 rows at the target; frames 1.02-1.44x faster; the rest 0.83-0.86 | not yet measured |
| LZ4 compress, 0.5.1, levels 1, 2, 3, 5, 9, 12 | 0.82, 1.04, 0.97, 0.87, 1.11, 1.76 (all but level 1 at the target); size within +0.88% | 0.87, 1.10, 0.94, 0.81, 0.92, 1.70 (all but level 5 at the target); same sizes |
| LZ4 decompress | 0.5.1: 0.85-0.92 | 0.5.1: 0.83-0.93 |
| Small payloads (256 B to 16 KiB) with a dictionary, reused encoder and decoder, Zstandard levels 1 and 3 | compress 0.26-0.74, decompress 0.68-1.06; size 0.73-1.01; no allocation per payload | not yet measured |
| Small payloads with a dictionary, LZ4 | compress 0.27-0.86, decompress 0.45-1.31; size within +2.9% (level 3), +2.5% (fast levels) | not yet measured |
| Deflate compress, one call, levels 1-9 (against libdeflate) | 0.5.1, levels 1, 3, 6, 9: 0.71, 0.68, 0.75, 1.28; size +0.63% to +0.81% | 0.5.1: 0.80, 0.69, 0.76, 1.34; same sizes |
| Deflate compress, one call, levels 10-12 | Silesia 0.89-0.96, enwik8 0.84-0.86; size within +0.90% | at the target |
| Deflate compress, streaming in 64 KiB pieces, levels 1-9 (against zlib-ng; 0.5 measurement, levels 1-9 changed in 0.5.1, where enwik8 level 7 is +0.14%) | levels 1-4 0.56-0.79, Silesia 5-8 and enwik8 6, 7, 9 at the target; output 0.5-26% smaller | levels 7 and 8 at the target, the rest 0.64-0.79 |
| Deflate decompress | 0.60-0.71 of libdeflate, 0.88-0.97 of zlib-ng | 0.72-0.78 of libdeflate, 1.14-1.29 of zlib-ng |
| bzip2 decompress (against `bzip2 -d`, start-up subtracted) | 1.28-1.47, at the target | 1.00-1.10, at the target |
| xz compress, presets 0-3 | 0.71-1.29; size -0.29% to +0.79%, except preset 1 (+2.29% Silesia, +1.44% enwik8) and enwik8 preset 3 (+1.28%) | 0.69-1.14; same sizes |
| xz compress, presets 4-9 | 0.87-0.96 (Silesia preset 4 0.87); size +0.23% to +0.72% | enwik8 0.88-0.93, Silesia 0.82-0.84; same sizes |
| xz decompress | 0.71-0.73 | 0.99-1.02, at the target |
| lzip compress, levels 0-3 | 0.37-0.85; size -4.3% to +0.02% | 0.44-0.84 |
| lzip compress, levels 4-9 | enwik8 0.91-0.97, Silesia 0.85-0.92 (levels 4, 8, 9 below 0.87); size -0.02% to +0.48% | enwik8 0.89-0.93, Silesia 0.81-0.89; same sizes |
| lzip decompress | 0.99-1.14, at the target | 1.28-1.44, at the target |
| 7z archive size (LZMA2, against `7zz -mx=1/5/9 -mmt=1`) | -0.89% to +0.91%, at the target | |
| bzip2 compress, levels 1 and 9 (against `bzip2`, start-up subtracted) | 1.11-1.14, at the target; size -0.02% to +0.01% | 1.09-1.10, at the target; same sizes |
| Brotli compress, quality 0 and 5 (window 22) | 0.57 and 0.62; size -7.17% and -0.11% | 0.47 and 0.61; same sizes |
| Brotli compress, quality 11 | 1.10; size +2.04% | 1.17; size +2.04% |
| Brotli decompress, quality 5 and 11 (against `brotli` 1.2.1) | 0.83-0.93 (0.83 on one stream written by the tool, quality 5); other streams at the target | 0.92-1.01, at the target |
| Deflate64 compress, levels 1, 5, 9 (against `7zz` zip Deflate64) | 1.03, 0.92 and 3.03, at the target; size -1.04% to +0.38% | 1.00, 0.92 and 3.22, at the target; same sizes |
| Deflate64 decompress | 2.26-2.60, at the target | 2.05-2.45, at the target |
| LZX decompress (against `cabextract` 1.11, on a cabinet from our test encoder) | 1.06, at the target | 1.01, at the target |
| PPMd compress, 7z PPMD, mx 1, 5 and 9 (against `7zz`) | 0.89, 0.87 and 0.86 (mx 1 and 5 at the target); size equal | 0.85, 0.87 and 0.91 (mx 5 and 9 at the target); size equal |
| PPMd decompress, mx 1, 5 and 9 | 0.91-0.95, at the target | 0.89-0.93, at the target |
| Dictionary training, 0.5.1 (three game sample sets, 110 KiB dictionaries, times include .NET start-up) | 3-17x the tool's time on one thread, 2.2-9x with the default thread count; 2.4-3.8x its memory | 2-17x on one thread, 1.6-9.5x with the default thread count; 3.9-5.9x its memory; output with our dictionaries within about 1% of the tool's (worst +1.2%, at a 4 KiB dictionary) |

Peak memory of the LZMA family's in-memory calls is 0.5-2 GB on these corpora, because they hold the whole input and
output; the tools stream (5-675 MB). Use the stream types to bound it.

Output is byte-identical across x64 and arm64, the JIT and NativeAOT, with and without hardware
intrinsics, at any thread count, and in the browser (WebAssembly).

## Archives

```csharp
Archive.Extract("data.tar.zst", "out");                         // safe by default: no writes outside "out", limits on
Archive.Create("backup.tar.zst").Add("assets/").Save();          // format and compression from the extension
using var archive = Archive.Open("data.tar");                  // entries, random access when the source can seek
```

Read and written: tar (ustar, pax, GNU, sparse files); zip (store, Deflate, Deflate64, bzip2, PPMd and Zstandard; LZMA and xz
entries read; zip64, data descriptors; WinZip AES, and traditional encryption for reading); 7z (Copy, Deflate,
Deflate64, BZip2, LZMA, LZMA2, PPMd, Delta, the branch filters, Zstandard; solid folders, encoded headers, AES-256); cpio
(newc, crc, odc), ar (common, GNU, BSD) and ISO 9660 with Joliet and Rock Ridge. Microsoft compound files (.doc, .xls, .ppt, .msg; versions 3 and 4, storages as directories; `docs/cfb.md`) are read and written. Read only: cab (stored, MSZIP, LZX) and RAR 5 (list and metadata, stored entries, multi-volume sets; compressed entries raise a typed error). Single streams (`.gz`, `.zst`, `.lz4`,
`.bz2`, `.xz`, `.lz`, and `.br`, which is recognised by its name) and tar inside them (`.tar.gz`, `.tar.zst`, ...) are
one archive. AES needs `System.Security.Cryptography.Aes`, which the browser does not provide. An app that uses few codecs can trim the rest: set the `RecursiveAI.OneArchive.AllCodecs` feature switch to false and register what it needs with `ArchiveCodecs.AddX()` (`docs/archive-codecs.md`). The other containers plug into the same reader and writer interfaces. The model, the safety defaults and limits, and the tar coverage are in `docs/archive.md`.

## Layout

- `src/OneCompress`, `src/OneArchive` - the two packages.
- `tests/` - xUnit tests.
- `fuzz/` - in-process mutational fuzz driver and targets.
- `bench/` - the bar runner, micro-benchmarks and `bench/Managed`, which compares against the fastest managed .NET libraries; the only projects that may reference comparison libraries.
- `tools/` - corpus and reference-tool scripts, table generators, the hash-matrix runner and the per-codec bench drivers.
- `docs/` - normative notes on each codec and container (formats, tuning choices, benchmark method).
- `smoke/aot-trim`, `smoke/wasm` - NativeAOT/trim and browser-wasm smoke apps.
- `build/ci.sh` - the script CI and developers run.

## Licence

Licensed under either of [MIT](https://github.com/recursive-ai-oss/onearchive-csharp/tree/3fa11f22dbe4e13c90ae9c63f393e945a0c09bba/LICENSE-MIT) or [Apache-2.0](https://github.com/recursive-ai-oss/onearchive-csharp/tree/3fa11f22dbe4e13c90ae9c63f393e945a0c09bba/LICENSE-APACHE), at your option.
Copyright (c) Recursive AI, LLC.


<!-- https://recursiveai.org/onearchive/docs/archive/ -->

# Archives: the shared layer, tar, zip and single compressed files

How OneArchive models an archive, what extraction refuses by default, and what the tar reader and writer do. This
is normative: a port must reproduce the writer's bytes and the defaults below from this text.

## The model

A container is read through an `ArchiveReader` and written through an `ArchiveWriter`. Both are small abstract classes;
each format supplies one pair (`TarReader`/`TarWriter`), and everything above them (`Archive`, extraction, limits,
the builder) is written once against the base classes. A new container (zip, 7z, cpio, ar, ISO, cab, RAR5) adds its
pair and one case in `Archive`'s format switch; it does not change these interfaces.

Why this shape:

- **One entry type for every format.** `ArchiveEntry` carries what the formats have in common: name, kind, size,
  times, mode, owner ids and names, link target, device numbers. Format-specific data lives in a derived class
  (`TarEntry` has every header field, the pax records and the sparse map) and in `Extras`, a string map for attributes
  with no property (tar pax records, later zip extra fields). Extraction and the builder only know `ArchiveEntry`.
- **Sequential first.** `GetNextEntry()` plus `OpenEntryStream()` works on any stream, including a pipe. Random access
  (`SupportsRandomAccess`, `OpenEntryStream(entry)`) is optional and only offered when the source can seek and the format
  can address entries; zip and 7z will offer it from their central directories, tar from offsets it recorded while
  scanning.
- **Limits live in the base reader.** Entry count, name length, name depth and total declared size are checked in
  `ArchiveReader` as each entry is admitted, so no container has to remember them, and they apply before any content is
  read. Format-specific limits (tar's pax header size, sparse region count) live in the format's own options.
- **Sync and async share one body.** Every operation that does I/O is written once as an `async` method with a flag;
  with the flag off every `await` is on an already-completed operation, so the synchronous methods do not block a
  thread pool thread and do not duplicate code.
- **Safety is in the extractor, not in the readers.** A reader reports the archive as stored (a name `../x` is returned
  as `../x`); the extractor (`ArchiveExtractor`, shared by all formats) decides what may be written.
- **Errors are one hierarchy.** Everything derives from `CompressionException`: `CompressionFormatException` (format
  name, byte offset, what was expected, `IsTruncated`), `CompressionLimitException` (format, option name, limit, requested),
  and for archives `ArchiveSafetyException` (format, `Rule`, entry name) and `ArchiveUnsupportedException` (format, feature).

## Happy path

```csharp
using var archive = Archive.Open("data.tar.zst");           // path or Stream; format found from the bytes
foreach (var e in archive.Entries) { ... }                    // seekable sources only
using var s = archive.OpenEntry(archive.Find("a/b.txt")!);    // random access (re-decompresses when compressed)
using var reader = archive.OpenReader();                      // sequential, works on any source, once for non-seekable

Archive.Extract("data.tar.zst", "out");                       // safe defaults
Archive.Create("backup.tar.zst").Add("assets/").Add("notes.txt", "docs/notes.txt").Save();
```

`Archive.Create(path)` chooses the format and compression from the extension (`.tar`, `.tar.zst`/`.tzst`, `.tar.lz4`/`.tlz4`,
`.tar.gz`/`.tgz`, `.tar.xz`/`.txz`, `.tar.lz`/`.tlz`, `.zip`, `.cfb`, and `.gz`/`.zst`/`.lz4`/`.xz`/`.lz` for one compressed file) unless `ArchiveCreateOptions.Format`/`Compression` say otherwise. An extension
that names no writable format and no `Format` option is an `ArgumentException`. `Archive.Create(Stream)` defaults to
uncompressed tar. `.tar.bz2` and `.bz2` can be read, not written.

### Detection

1. Compression from the first bytes: Zstandard `28 B5 2F FD`, LZ4 frame `04 22 4D 18`, gzip `1F 8B`, bzip2 `BZh`, xz `FD 37 7A 58 5A 00`, lzip `LZIP`. Magic bytes win over the file name.
2. If compressed, the first 512 bytes of the decompressed stream are examined; otherwise the first 512 bytes.
3. Container from those bytes: `PK\3\4` (also `PK\5\6`, the whole of an empty zip, and the span markers `PK\7\8` and `PK00`) is
   zip; `MSCF` is a cabinet (see `cab-reader.md`; `.cab` by extension); `D0 CF 11 E0 A1 B1 1A E1` is a compound file (see `cfb.md`; `.cfb`, `.doc`, `.xls`, `.ppt`, `.msg` and `.msi` by extension); `ustar` at offset 257 is tar; a block with no magic but a valid header checksum, numeric mode and size fields and a
   non-empty name is a version 7 tar; 512 zero bytes inside a compressed stream are an empty tar, unless a name is known and
   does not say tar (a disk image starts with zeros too).
4. If nothing matched, the file name's extension decides (this is how an empty uncompressed `.tar` is recognised).
5. A compressed stream that is still unrecognised is a single compressed file (below).
6. Otherwise `CompressionFormatException` (`archive`, offset 0).

gzip (RFC 1952, all members in a row), bzip2, xz and lzip are read and written; zstd and LZ4 are written too (gzip at
`CompressionLevel`, default 6, through the Deflate encoder; MTIME is the entry's modification time or 0, OS byte 255; bzip2 at
`CompressionLevel` 1-9, default 9, through `BZip2Stream`; xz at preset `CompressionLevel`, default 6, with a CRC64 check; lzip at level
`CompressionLevel`, default 6, one member). The xz and lzip dictionary sizes are checked against `MaxWindowSize` before any allocation.
The place for a new layer is `Internal/CompressionFilters.cs` (`OpenDecompress`, `OpenCompress`, `CanCompress`).

## Formats

| Format | `ArchiveFormat` | Reader / writer | Notes |
|---|---|---|---|
| tar | `Tar` | `TarReader` / `TarWriter` | ustar, pax, GNU, sparse files |
| zip | `Zip` | `ZipReader` / `ZipWriter` | zip64, WinZip AES; ZipCrypto is read only |
| 7z | `SevenZip` | `SevenZipReader` / `SevenZipWriter` | solid folders, encoded headers, AES-256 |
| cpio, ar | `Cpio`, `Ar` | `CpioReader`, `ArReader` / `CpioWriter`, `ArWriter` | newc, crc, odc; common, GNU, BSD |
| ISO 9660 | `Iso` | `IsoReader` / `IsoWriter` | Joliet, Rock Ridge |
| Compound file (CFB) | `CompoundFile` | `CompoundFileReader` / `CompoundFileWriter` | storages are directories, streams are files; versions 3 and 4 (`cfb.md`) |
| cab | `Cab` | `CabReader` | read only; stored, MSZIP, LZX |
| RAR 5 | `Rar` | `RarReader` | read only; stored entries |
| single stream | `SingleStream` | `SingleStreamReader` / `SingleStreamWriter` | one compressed file |

## Entry model

`ArchiveEntry`: `Name` (`/`-separated, no trailing slash), `Kind` (File, Directory, Symlink, HardLink, CharacterDevice,
BlockDevice, Fifo), `Size`, `ModifiedTime`/`AccessTime`/`ChangeTime` (`DateTimeOffset?`, 100 ns), `Mode` (`UnixFileMode?`, 12 bits),
`Uid`, `Gid`, `UserName`, `GroupName`, `LinkTarget`, `DeviceMajor`, `DeviceMinor`, `Extras`.

## Safety defaults

Everything below is on by default; each loosening is an option on `ArchiveExtractOptions`.

| Rule | Default | Loosened by |
|---|---|---|
| Name leaves the target (`..`, including after `\` is read as `/`) | `ArchiveSafetyException`, rule `path-traversal` | `UnsafePaths = Sanitize` removes `.`/`..` components |
| Absolute name: leading `/` or `\` (UNC included) or a drive letter | rule `absolute-path` | `UnsafePaths = Sanitize` drops the root |
| NUL in a name | rule `invalid-name` (always) | |
| Symbolic link whose target is absolute, or has a `..` after a name, or climbs above the target directory | rule `link-escape` | `Symlinks = Unsafe` (create anyway) or `Skip` |
| Entry written under a path component that is a symbolic link (from the archive or already on disk) | rule `link-write-through` | `AllowWriteThroughSymlinks = true` |
| Hard link whose target is not a safe relative name of a file extracted earlier from this archive | rule `hardlink-target` | `HardLinks = Skip` |
| Device, FIFO | rule `special-file` | `SpecialFiles = Skip` (they are never created) |
| Destination exists (including a second entry with the same name) | rule `existing-file` | `Existing = Overwrite` or `Skip` |
| setuid, setgid, sticky bits | dropped | `KeepSpecialModeBits` |

Other behaviour, fixed:

- `\` and `/` both separate components on every platform; `.` and empty components are dropped; an entry named `.` or
  `./` that is a directory is the target itself, a file with that name is `invalid-name`. A first component of the form
  `X:` (a letter and a colon) is a drive.
- Symbolic-link targets are checked lexically (relative, no `..` after a name, leading `..` count at most the depth of the
  link's own directory). They are created with `File.CreateSymbolicLink`.
- A link replaced under `Existing = Overwrite` is deleted, never written through; files are created with
  `CreateNew`, which does not follow a link that appeared after the check.
- Hard links are extracted as **copies** of the file already written (managed code cannot create hard links). The
  target must be a regular file this run extracted.
- Parent directories that the archive does not list are created. Directory modes and times are applied after all
  entries (so a read-only directory does not block its contents).
- Ownership is not restored (no `chown` from managed code). Permissions are applied where the platform has them
  (`RestorePermissions`, default true); modification times for files and directories (`RestoreTimes`, default true).
- A file that fails midway (limit hit, truncation, cancellation) is deleted; earlier entries stay.
- Races with other processes changing the target while it is extracted are not defended against.

### Limits (`ArchiveLimits`)

A limit hit raises `CompressionLimitException` whose `LimitName` is the property name; never a truncated result.

| Property | Default | Checked |
|---|---|---|
| `MaxTotalSize` | 1 GiB | sum of declared entry sizes (sparse files at full size), as each header is read, before data |
| `MaxEntryCount` | 100,000 | per entry |
| `MaxPathLength` | 4096 chars | entry name and link target |
| `MaxPathDepth` | 64 components | entry name |
| `MaxExpansionRatio` | 1000 | bytes produced (tar stream bytes plus sparse holes) per byte read from the file, only once more than 32 MiB were produced; `Archive` only; 0 disables |
| `MaxWindowSize` | 8 MiB | Zstandard window, or xz/lzip dictionary, a compressed archive may declare (xz presets 0 to 6 and lzip levels 0 to 6 fit; raise it for the larger presets) |

Tar adds (`TarReaderOptions`): `MaxExtendedHeaderSize` 1 MiB (one pax or long-name record block, checked before it is
buffered) and `MaxSparseRegions` 100,000.

For a compressed archive the whole decompressed stream is metered, and after the last entry the rest of the stream is read
so that the codec's own checks (Zstandard content checksum, truncation) run.

## Tar

### Reading

Dialects: V7, ustar, pax and GNU; `TarEntry.Format` says which a header was. Offsets in errors and in `TarEntry` count
bytes of the tar stream from where the reader started.

- **Header**: 512 bytes; the checksum (octal, unsigned or signed byte sum with the field read as spaces) must match, else
  `tar: invalid data at byte offset N: expected a valid header checksum`. Numeric fields are octal with space/NUL padding or
  GNU base-256 (bit 7 of the first byte; `0xFF` is negative). Names are NUL-terminated, decoded as UTF-8 when valid and
  as Latin-1 otherwise. The ustar `prefix` joins as `prefix/name` for ustar headers only.
- **Types**: `0`/NUL file, `1` hard link, `2` symlink, `3`/`4` devices, `5` directory, `6` FIFO, `7` file, `D` directory
  with a listing (skipped), `S` old GNU sparse, `x`/`X` pax extended header, `g` global pax header, `L`/`K` GNU long name and
  link name, `V` volume label (skipped), `M` multi-volume (`ArchiveUnsupportedException`). Any other type is a regular file
  (POSIX). A regular file whose name ends in `/` is a directory. Types 1-6 have no content whatever their size field says.
- **pax**: records `length key=value\n` (the length counts itself). Global records apply to all later entries until
  replaced; an empty value removes a key. Used: `path`, `linkpath`, `size`, `uid`, `gid`, `uname`, `gname`, `mtime`,
  `atime`, `ctime` (decimal seconds with optional fraction, digits beyond 100 ns dropped). Other keys become `Extras`.
- **GNU**: `L`/`K` records, atime/ctime in the old header, and sparse files in every form: old GNU (`S`, map in the header
  and extension blocks), pax 0.0 (repeated `GNU.sparse.offset`/`numbytes`), 0.1 (`GNU.sparse.map`) and 1.0 (map in the
  first blocks of the data, `GNU.sparse.major=1`, `minor=0`). A sparse entry's `Size` is its real size,
  `StoredSize` the bytes in the archive; its stream reads zeros for holes and, over a seekable source, seeks. Regions must be
  in order, inside the real size, and add up to the stored size. Extraction leaves holes as holes.
- **End**: a zero block ends the archive (one is enough; the rest is ignored). End of input exactly between entries with
  no zero block is `IsTruncated` unless `RequireEndOfArchive = false`; `IgnoreZeroBlocks` reads concatenated archives.
- Every length is validated before it is used: sizes against the stream length (seekable) or while skipping,
  pax and long-name sizes against `MaxExtendedHeaderSize`, region counts against `MaxSparseRegions`, times against years 1-9999,
  all arithmetic overflow-checked.

### Writing (byte-exact)

`TarWriter` writes `Pax` (default) or `Ustar`. For every entry, in this order: an optional pax extended header, the header, the
content padded with zeros to a multiple of 512. At the end: two zero blocks, then zeros to a multiple of `RecordSize`
(default 10240; 512 for none).

Header fields, all fixed width:

| Field | Written |
|---|---|
| name | UTF-8; directories get a trailing `/`. At most 100 bytes of ASCII; otherwise see below |
| mode | 7 octal digits + NUL of `Mode & 0xFFF`; default 0644 (directory 0755, symlink 0777) |
| uid, gid | 7 octal digits + NUL; above 2,097,151: field 0 and a pax record (ustar: error) |
| size | 11 octal digits + NUL, files only (0 for everything else); above 8,589,934,591: field 0 and pax `size` |
| mtime | 11 octal digits + NUL of whole seconds (floor); missing: 0. A fraction, a negative time or one beyond the field adds pax `mtime` (ustar: fraction dropped, out of range is an error) |
| chksum | 6 octal digits, NUL, space, computed with the field as 8 spaces |
| typeflag | `0` `5` `2` `1` `3` `4` `6` |
| linkname | link target, at most 100 ASCII bytes, else pax `linkpath` (ustar: error) |
| magic, version | `ustar\0`, `00` |
| uname, gname | at most 31 ASCII bytes, else pax `uname`/`gname` (ustar: error); empty when unset |
| devmajor, devminor | 7 octal digits + NUL; 0 for non-devices |
| prefix | ustar only: a name over 100 bytes is split at the leftmost `/` that leaves at most 100 bytes in `name`; the prefix must be at most 155 bytes, else error |

With `Pax`, a name over 100 bytes or with non-ASCII characters goes to pax `path` and the name field holds its first
100 bytes (cut at a character boundary). `atime`/`ctime` are written as pax records when the entry has them.
`Extras` are written as pax records (keys that the writer models itself, `charset` and `hdrcharset` are skipped; empty values are skipped).
Records are sorted by key in ordinal order and each is `length key=value\n`. The extended header is an entry named
`././@PaxHeader`, type `x`, mode 0000644, uid, gid and mtime 0, empty owner names, magic `ustar\0` `00`, zero device numbers.

Nothing host-dependent is written: no process id, clock, user database or locale. Output depends only on the entries and
the options; the hash matrix entries `tar-write/*`, `tar-zst/*`, `tar-lz4/*` and `tar-read/*` check it across platforms.

### The builder

`Archive.Create(...).Add(path)` records entries in call order; a directory is its own entry followed by its children
sorted by name (ordinal, UTF-16 code units), depth first. Times come from the file system (100 ns), modes from the file
(`0644`/`0755` where the platform has none), uid and gid are 0 with no names (not readable from managed code), symbolic links are
stored as links (`FollowSymlinks` stores their content), special files are not detected. `Normalize = true` sorts all entries by
name (stable, ordinal), sets the time to `NormalizedTime` (default the epoch), owners to 0 with no names, drops access and
change times, and sets modes to 0755 (directories, and files with any execute bit) or 0644 (symbolic links 0777), so equal
trees give equal bytes. Compression: Zstandard level 3 with a content checksum, LZ4 level 1; `CompressionLevel` overrides.
Single-threaded, so the bytes do not depend on the machine.

## Single compressed files

A `.gz`, `.bz2`, `.zst` or `.lz4` file (any compressed stream the detector finds no container in) is an archive of one regular file,
`ArchiveFormat.SingleStream`, with `Archive.Compression` saying which codec. The entry's name is the gzip header's FNAME when it
has one (ISO 8859-1, as RFC 1952 2.3.1 says), otherwise the archive's file name without its `.gz`/`.bz2`/`.zst`/`.zstd`/`.lz4`/`.xz`/`.lz`
extension, otherwise `data`. The gzip MTIME, when not zero, is the entry's modification time. `Size` is 0: the content's length
is not known before it has been read (it is limited and metered like every compressed archive). Writing takes exactly one file
entry (`ArchiveUnsupportedException` for a second entry or for anything but a regular file); gzip stores the entry's base name as FNAME (ISO 8859-1; left out when it has characters above U+00FF) and its modification time
as MTIME, the other formats store no name. `Archive.Create("a.txt.zst")` and `ArchiveCreateOptions { Format = SingleStream, Compression = ... }`
produce them. `.tar.gz`/`.tgz` is a tar inside gzip, like `.tar.zst`.

## Zip

`ZipReader`/`ZipWriter` (namespace `RecursiveAI.OneArchive.Zip`), with `ZipEntry` carrying every header field. Section numbers
are those of PKWARE's APPNOTE 6.3.10; its text is not reproduced here.

### Reading

- **Two modes.** A stream that can seek is read through the central directory (4.3.12), which is authoritative; any other stream
  (or `ZipReaderOptions.ForceStreaming`) is read forward through the local headers (4.3.7), and the central directory is never
  read. Entries read forward have no comment, no Unix mode and no link kind (those live in the central directory), so a symbolic
  link arrives as a regular file holding its target and permissions are the extractor's defaults. `ArchiveComment` is only
  known in directory mode.
- **Finding the directory.** The end record (4.3.16) is searched in the last 22 + 65535 bytes, from the end; a candidate whose
  comment length ends exactly at the end of the data wins, else the nearest candidate whose comment fits. The zip64 locator
  (4.3.15) right before it and the zip64 end record (4.3.14) it points to (or the one just before the locator) replace the
  end record's values; a `0xFFFF`/`0xFFFFFFFF` marker without them is an error. Disk numbers other than 0 are
  `ArchiveUnsupportedException` (spanned archives). Data before the archive (a self-extractor stub) is allowed: when the
  directory is not at the stored offset but is at the offset shifted by the difference to the end record, every offset is shifted.
- **Checks before any entry is handed out** (all `CompressionFormatException` unless noted): the entry count is at most
  `ArchiveLimits.MaxEntryCount` (`CompressionLimitException`) and at most directory size / 46; the directory is at most
  `ZipReaderOptions.MaxCentralDirectorySize` (default 128 MiB, `CompressionLimitException`); every entry's local header lies
  before the directory; no two entries share a local header offset; the entry's compressed size fits between its local header
  and the next entry (or the directory). Together these make overlapping entries - one stored blob claimed by many names,
  data running into the next entry - impossible, so the total compressed bytes read never exceed the file's size.
- **Opening an entry** reads its local header and requires it to agree with the directory: same method, same name bytes, same
  encryption flag, and (when bit 3 is clear) the same sizes and CRC-32 where the local header states them. The data must end
  before the entry's boundary. A stored entry's compressed size must equal its size.
- **Content** is delivered through a stream that decodes on the fly and checks, at the last byte: the decoder consumed exactly the
  compressed size (deflate: the stream ends there; zstd: one or more frames fill it), the output is exactly the declared size
  (more output is detected one byte past it, never buffered), and the CRC-32 matches (`ZipReaderOptions.VerifyChecksum`, default on).
  Every byte is metered against `MaxExpansionRatio` (compressed bytes read versus bytes produced, whole archive, checked after
  32 MiB) and `MaxTotalSize`. Entry streams are forward-only.
- **Forward-only and data descriptors.** With bit 3 set, deflate and zstd entries are self-terminating, so the decoder finds the
  end and the descriptor (4.3.9) follows; the reader tries the four layouts (with or without the signature, 4- or 8-byte sizes, the
  one implied by a zip64 field in the local header first) and accepts the one whose CRC-32 and sizes equal what it decoded.
  `ZipEntry.IsSizeKnown` is false for such entries and `Size` is 0; the declared total limit cannot see them, so produced bytes are
  counted against `MaxTotalSize` as they are read. A stored entry with bit 3 is read only if its local header carries the size
  (our writer does this); otherwise it is `ArchiveUnsupportedException` (open a seekable stream instead). An entry the reader
  cannot decode but whose compressed size is known is skipped.
- **Names.** Bit 11 set: UTF-8. Otherwise the Info-ZIP Unicode Path field (0x7075, version 1) when its CRC-32 equals the CRC-32 of the
  stored name, otherwise IBM code page 437 (4.4.17), or `ZipReaderOptions.LegacyNameEncoding`. Names are returned as stored apart
  from a trailing `/` or `\`, which makes the entry a directory and is removed; `\` separates components for the extractor's safety
  rules (zip written on Windows). The archive comment and entry comments follow the same rules (the archive comment takes UTF-8 when valid).
- **Kind and mode.** Host system 3 (Unix) or 19: the high 16 bits of the external attributes are `st_mode`; the type bits give symlink,
  directory, device or FIFO (a socket is reported as FIFO), the low 12 bits the mode. Other hosts: DOS directory bit 0x10, no mode.
  A symbolic link's target is its data (UTF-8 when valid, else Latin-1), read when the directory is loaded, at most `4 * MaxPathLength` bytes.
- **Times.** NTFS field 0x000a (100 ns) wins over the extended timestamp 0x5455 (Unix seconds; modification, access, creation), which wins
  over the DOS date and time, read as UTC (the fields carry no zone); an impossible DOS value is no time.
- **Methods.** 0 store, 8 deflate, 9 deflate64 (raw stream, `Deflate64Decoder`, see `deflate64-decoder.md`), 12 bzip2 (read and written), 93 zstd (4.4.5). Anything else, and PKWARE strong encryption (bit 6), are listed and raise
  `ArchiveUnsupportedException` naming the method when opened. Encrypted entries (bit 0 or method 99) are described in `zip-encryption.md`. Zip64 extra fields (0x0001) hold exactly the values whose header field is
  saturated, in the order size, compressed size, offset, disk; a short field is an error.

### Writing (byte-exact)

Everything not listed is as APPNOTE says. The same entries and options give the same bytes on every platform; the clock is never read.

- **Layout**: for each entry a local header, the data, and for descriptor entries the descriptor; then the central directory, the zip64
  end record and locator if needed, and the end record. "Version made by" is `0x033F` (Unix, 6.3); "version needed" is 10 (store), 20
  (deflate, directories and descriptor entries), 21 (deflate64; at least 20 for directories and descriptor entries, as for deflate), 46 (bzip2), 63 (zstd), at least 45 with zip64. Compression method per entry: `ZipWriterOptions.Method`
  (default deflate); directories, symbolic links and empty files are stored. Hard links, devices and FIFOs are
  `ArchiveUnsupportedException`. Deflate entries are raw Deflate at `Level` (default 6, 0-12, through `DeflateEncoderStream`); deflate64 entries are a raw Deflate64 stream at `Level` (default 6, 0-9; higher values mean 9; `Deflate64Stream`); bzip2 entries are one bzip2 stream at `Level` (default 9, values outside 1-9 are clamped; `BZip2Stream`); zstd entries are one frame at `Level` (default 3) with the content size and no frame checksum (the zip CRC-32
  covers the data).
- **Names**: UTF-8, bit 11 set when any byte is not ASCII (name or comment); directories get a trailing `/`. An empty name or NUL is an
  `ArgumentException`; names and comments are at most 65535 bytes.
- **Attributes**: external attributes = `(file type | mode) << 16 | (0x10 for directories)`; type is `0x8000` file, `0x4000` directory,
  `0xA000` symbolic link (whose data is its target, UTF-8); mode is the entry's, else 0644 (0755 directories, 0777 links).
- **Times**: DOS date and time are the entry's UTC time, `1980-01-01 00:00` when there is none or it is earlier, clamped to 2107-12-31 23:59:58.
  The extended timestamp field (0x5455) is written when the modification time fits 32 bits: flags then modification, access, change times
  (those that exist and fit) in the local header, modification time alone in the central directory. The NTFS field (0x000a, tag 1: modified,
  accessed, changed, missing ones taking the modification time) is written in both places when any time has a fractional second or does not fit 32 bits.
- **Extra field order**: zip64 (local header only when needed, see below), 0x5455, 0x000a, then the `ZipEntry.ExtraFields` the caller passed
  (except 0x0001, 0x5455, 0x000a and 0x7075, which the writer owns). The central directory repeats them, with its own zip64 field first.
- **Local headers and descriptors**: `DataDescriptors.Auto` patches the local header with CRC-32 and sizes when the output can seek and uses
  descriptors (bit 3, signature, 4-byte sizes, or 8-byte with a zip64 field) when it cannot; `Always` and `Never` force it. Entries whose data is
  known up front (directories, links, empty files) never have descriptors. A stored entry with a descriptor still states its sizes in the local
  header so that forward-only readers can find its end; other methods write zero there. The output of `Always` is the same on every stream.
- **Zip64**: an entry with a declared size of 3.75 GiB (`0xF0000000`) or more, or any entry with `Zip64 = Always`, gets a 16-byte zip64 field in
  its local header (`0xFFFFFFFF` in the 32-bit size fields; the values in the field). The central directory gets a zip64 field only for the values that
  reach `0xFFFFFFFF` (size, compressed size, offset, in that order). The zip64 end record and locator are written when the count reaches 65535, the
  directory's size or offset reaches `0xFFFFFFFF`, or `Always`; the end record then holds `0xFFFF`/`0xFFFFFFFF` for values that do not fit and the real
  values otherwise. `Never` turns each of these into an `ArgumentException`. An entry that compresses to 4 GiB or more without a zip64 local field fails the same way.
- **Checks**: content must be exactly `Size` bytes (`ArgumentException`); a failed write marks the writer failed and `Dispose` does not finish it.

## Platforms

Everything except the file-system parts (`Archive.Extract`, `Archive.Create(path)`, `Add(path)`, opening a path) works with streams only,
including under browser-WASM. Symbolic links and permissions need a platform that has them; on others the extractor raises
`ArchiveUnsupportedException` for links and skips permissions.


<!-- https://recursiveai.org/onearchive/docs/archive-codecs/ -->

# Choosing codecs for a small binary

`Archive` and the per-format readers and writers reach codecs only through a registry. By default nothing needs
configuring: the first time an archive needs a codec, every codec is registered, so every format and method works, and
the app carries every codec.

An app that needs a small NativeAOT or trimmed binary turns the default off and registers what it uses:

```xml
<ItemGroup>
  <RuntimeHostConfigurationOption Include="RecursiveAI.OneArchive.AllCodecs" Value="false" Trim="true" />
</ItemGroup>
```

```csharp
ArchiveCodecs.AddZstd();   // .zst and .tar.zst; AddLz4, AddDeflate, AddDeflate64, AddBZip2, AddLzma, AddXz,
                           // AddLzip, AddBrotli, AddLzx and AddFilters register the others
```

Without the trimmer the switch can be set before the first archive is touched with
`AppContext.SetSwitch("RecursiveAI.OneArchive.AllCodecs", false)`.

With the default off, an archive that needs a codec the app did not register fails with an `ArchiveUnsupportedException`
naming the format, the method and the `ArchiveCodecs.Add...` call that fixes it. It is never skipped or truncated.

Which call serves what:

| Need | Register |
|---|---|
| `.zst`, `.tar.zst`, zip method 93, 7z Zstandard | `AddZstd` |
| `.lz4`, `.tar.lz4` | `AddLz4` |
| `.gz`, `.tar.gz`, zip method 8 (the zip writer's default), 7z Deflate, cabinet MSZIP | `AddDeflate` |
| zip method 9, 7z Deflate64 | `AddDeflate64` |
| `.bz2`, `.tar.bz2`, zip method 12, 7z BZip2 | `AddBZip2` |
| zip method 14, 7z LZMA and LZMA2, 7z compressed headers | `AddLzma` |
| `.xz`, `.tar.xz`, zip method 95 | `AddXz` |
| `.lz`, `.tar.lz` | `AddLzip` |
| `.br`, `.tar.br` (and brotli's dictionary) | `AddBrotli` |
| cabinet LZX folders | `AddLzx` |
| 7z branch converters and delta | `AddFilters` |

The smoke app `smoke/aot-trim-slim` reads and writes only `.tar.zst` this way; `build/ci.sh aot` publishes it with
NativeAOT and checks that no other codec's types and not brotli's dictionary are in the binary.


<!-- https://recursiveai.org/onearchive/docs/conventions/ -->

# Conventions for codecs

How every OneCompress codec is shaped, so the codecs look alike and a port can follow them. The Zstandard
decoder (`src/OneCompress/Zstd/`) is the reference implementation of each point.

## Namespaces and tiers

- One namespace per codec: `RecursiveAI.OneCompress.<Codec>`. Shared types (the exceptions below) live in
  `RecursiveAI.OneCompress`.
- Three tiers, each built only on the one below:
  1. **One-liners**: a static class named for the codec (`Zstd.Decompress(bytes, options)`).
  2. **Decoder / encoder / stream**: `XxxDecoder`, `XxxEncoder` (sealed classes that own their buffers and tables and
     reuse them) and `XxxStream`.
  3. **Raw parts**, public and documented: bit readers, entropy tables, sequence decoders, dictionaries.
- Inside a namespace named like the codec, the static class has the same name (`RecursiveAI.OneCompress.Zstd.Zstd`).
  Code that lives under `RecursiveAI.OneCompress` therefore writes `Zstd.Zstd.Decompress`; code elsewhere writes
  `Zstd.Decompress` after `using RecursiveAI.OneCompress.Zstd;`.

## Errors

All errors a codec raises on its input derive from `CompressionException`:

| Type | Raised when | Properties |
|---|---|---|
| `CompressionFormatException` | the data is not valid for the format: truncated, corrupt, a forbidden value, a checksum mismatch | `Format`, `Offset` (byte offset in the compressed input), `Expected` (what the format requires there), `IsTruncated` |
| `CompressionLimitException` | a configured limit is exceeded (window size, output size, entry size...). Raised **before** the memory is allocated | `Format`, `LimitName` (the option's name), `Limit`, `Requested` |
| `CompressionDictionaryException` | a frame names a dictionary that was not supplied | `Format`, `DictionaryId` |

Rules:

- Messages read `<format>: invalid data at byte offset <n>: expected <what>`. `Expected` is a phrase that completes
  "expected ...", naming the value the format allows, not the value found.
- `Offset` counts bytes of the compressed input from the start of what the caller gave the decoder (across calls for a
  streaming decoder). For a problem inside a block, it is the offset of the block or of the field being read.
- A decoder never lets `IndexOutOfRangeException`, `NullReferenceException`, `OverflowException` or
  `OutOfMemoryException` (from sizes the input chose) escape for malformed input. Argument errors
  (`ArgumentException` and subclasses), `OperationCanceledException` and I/O errors from the caller's own streams are
  the only other exceptions.
- Truncated input is reported with `IsTruncated = true` and the offset where the input ended.
- Never return a truncated result as a success: a decoder that cannot finish throws.

## Options

- Every tunable is a property on a sealed options class: `ZstdDecompressionOptions`, `ZstdCompressionOptions`.
  No tunable is an overload or an extra parameter.
- Properties are `init`-only, with the default in the property's XML doc and as a `Default...` constant when it is a
  number. The class has a shared `Default` instance. Options are read once, when a decoder or stream is constructed.
- Limit properties are named `Max<Thing>` (`MaxWindowSize`, `MaxOutputSize`). They are checked against the sizes the
  input *declares* before allocating, and against the real sizes as output is produced. A hit throws
  `CompressionLimitException` with `LimitName` set to the property name.
- Safe defaults: memory a decoder may reserve is bounded by the declared window and a configured maximum
  (8 MiB for Zstandard); total output is bounded (1 GiB). Callers raise them knowingly.

## Tier 2: the decoder shape

```csharp
public sealed class XxxDecoder
{
    public XxxDecoder(XxxDecompressionOptions? options = null);
    public OperationStatus Decompress(ReadOnlySpan<byte> source, Span<byte> destination, out int consumed, out int written);
    public bool IsAtFrameBoundary { get; }
    public void ThrowIfIncomplete();
    public void Reset();
}
```

- `Decompress` returns `System.Buffers.OperationStatus`: `Done` (a unit of the format, such as one frame, is complete and
  fully written), `DestinationTooSmall` (output is waiting), `NeedMoreData` (all input used, unit incomplete).
  It never returns `InvalidData`: malformed input throws, so a corrupt stream is never mistaken for a short one.
- Callers pass the bytes not consumed by earlier calls, plus any new ones. `consumed` and `written` are always set,
  also when the call throws. Any split of input and output must give the same result as no split.
- `ThrowIfIncomplete` is how a caller says "no more input will come": it throws a truncated `CompressionFormatException`
  when the decoder stopped inside a unit.
- After an exception the decoder refuses further calls (`InvalidOperationException`) until `Reset()`.
- A decoder instance is not thread-safe; immutable parts such as dictionaries are, and can be shared.
- Steady state allocates nothing: buffers are created on first need, grow only, and are reused across units.
  A test with `GC.GetAllocatedBytesForCurrentThread` guards this.
- Where the caller's buffer can hold the whole output of a unit that states its size, the decoder writes into it
  directly instead of through its own window.

## Streams

- `XxxStream : Stream`, sync and async paths, `CancellationToken` on `ReadAsync`/`WriteAsync`.
- Input is read into a pooled buffer. `leaveOpen` as in the BCL. End of data is reported only at a unit boundary;
  an inner stream that ends early throws the truncated format error.

## Hot paths and `Unsafe`

- Write loops on spans first. `Unsafe`/`MemoryMarshal` appear only behind small internal accessors whose bounds hold
  by construction, each with a comment stating the bound; in the `Checked` build configuration they are bounds-checked.
- No `Vector<T>`; fixed-width `Vector128/256/512` only, with a scalar path giving the same result.

## Tests and data

- Unit tests live in `tests/OneCompress.Tests/<Codec>/`. Test vectors written by reference tools are generated by a
  script in `testdata/<codec>/` (the tool and its version are recorded in the manifest) and committed when small.
- Tests that need the reference tool or large corpora are tagged `[Trait("Category", "Interop")]` and run only with
  `ONEARCHIVE_INTEROP=1`; with it set, a missing tool or corpus fails the run.
- Every decoder has hostile-input tests: every prefix of a frame, bit flips, and each limit.
- Hash-matrix entries for a codec live in `tools/hashmatrix/Host/Entries/<Codec>*.cs`.


<!-- https://recursiveai.org/onearchive/docs/zip-encryption/ -->

# Zip encryption

WinZip AES (AE-1 and AE-2) is read and written; traditional PKWARE encryption is read only. PKWARE's strong encryption (bit 6)
and central-directory encryption are not supported: they sit under PKWARE's licence terms. Sources: the WinZip document "AES
Encryption Information: Encryption Specification AE-1 and AE-2" (version 1.04; cited here by its section names) and PKWARE
APPNOTE 6.3.10 section 6.1 (traditional encryption), 4.4.4 (flag bits) and 4.4.5 (method 99).

## Reading

- `ZipEntry.Encryption` (`None`, `ZipCrypto`, `Aes128`, `Aes192`, `Aes256`, `Other`), `AesVersion` (1, 2 or 0), and
  `CompressionMethod` (the method inside the encryption; `Method` stays 99 for AES entries, as the headers say). Listing needs
  no password.
- `ZipReaderOptions.Password` (an `ArchivePassword`: a string, char array or `ReadOnlyMemory<char>` converts to it). Encoded as
  UTF-8, which is what 7-Zip decodes our archives with (it refuses to *create* a zip with a non-ASCII password, so there is no
  reference to compare against for creation).
- Opening an encrypted entry without a password raises `ArchivePasswordException` (`Required`, naming the entry). A wrong
  password raises it with `Incorrect`: AES entries compare the 2-byte verifier (1 in 65536 wrong passwords gets past it and
  then fails the authentication code, which is a `CompressionFormatException`); ZipCrypto entries compare the check byte (1 in
  256 gets past it and then fails the CRC-32 or the decoder). Messages never contain the password.
- A malformed AES field (missing, shorter than 7 bytes, vendor version other than 1 or 2, vendor id other than `AE`, strength
  other than 1 to 3, the real method 99, bit 0 clear) is a `CompressionFormatException` naming what was expected. The local header's
  method, name and bit 0 must agree with the central directory as for any entry.
- **AES data** (WinZip document, "Encrypted file storage format"): `salt | verifier (2) | data | authentication code (10)`; the
  compressed size in the headers covers all four. Salt is 8, 12 or 16 bytes for 128, 192 and 256-bit keys.
  **Key schedule** ("Key generation"): PBKDF2 with HMAC-SHA1, 1000 iterations, over the password and salt, producing the
  encryption key (key size), the authentication key (key size) and the verifier (2 bytes), in that order. **Cipher**: AES in counter
  mode with a 128-bit little-endian counter that starts at 1 and counts whole 16-byte blocks (the last may be partial). **Authentication
  code**: HMAC-SHA1 over the ciphertext with the authentication key, first 10 bytes. The code is compared in constant time.
- **AE-1 and AE-2** differ only in the CRC-32: AE-1 stores and checks it (`VerifyChecksum`), AE-2 stores 0 and the reader does not
  check it. The authentication code is checked in both and always, whatever `VerifyChecksum` says.
- **Streaming.** Decrypted bytes are handed to the caller before the authentication code at the end has been read, as for any
  streamed decryption; a mismatch raises `CompressionFormatException` ("an authentication code that matches the encrypted data") from
  the read that reaches the end of the entry. A modified ciphertext often fails in the decoder first, which is also a
  `CompressionFormatException`.
- **Traditional encryption** (APPNOTE 6.1): three 32-bit keys start at `0x12345678`, `0x23456789`, `0x34567890`, are updated with the
  password's bytes, and then with each decrypted byte (CRC-32 step, multiply by 134775813 plus 1, CRC-32 step); the stream byte is
  `(t * (t ^ 1)) >> 8` with `t = key2 | 2`. The 12-byte header's last decrypted byte must be the high byte of the CRC-32 or, when
  bit 3 is set, the high byte of the DOS time (6.1.6); with bit 3 set either is accepted, since writers differ.
- **Forward-only reads** decrypt entries whose sizes are in the local header. An encrypted entry with its sizes only in a data
  descriptor needs the central directory (the end of its data is not known), and raises `ArchiveUnsupportedException` saying so.
  Info-ZIP's streamed archives and our own descriptor-mode output are of this kind; a seekable stream reads them.
- Encrypted symbolic links are listed with an empty target when the password is missing or wrong.

## Writing

`ZipWriterOptions.Password` non-empty: every file and symbolic link entry, empty files included, is encrypted; directories are
not (the WinZip document recommends both). Defaults: `AesStrength = Aes256`, `AesVersion = Ae2`. ZipCrypto is never written.

- Headers: method 99, bit 0 set, the AES extra field `0x9901` (7 data bytes: version, `AE`, strength, real method) in the local
  header and the central directory, before the caller's own fields. Version-needed and version-made-by are what they would be
  without encryption (the document says so). The compressed size is data plus salt, verifier and 10-byte code. AE-2 writes CRC 0
  in the local header, the descriptor and the central directory.
- Order of work: compress, then encrypt. Data descriptors, patching and zip64 behave as for plain entries; the overhead counts
  in the compressed size.
- **Salt**: `RandomNumberGenerator`, one per entry, or `ZipWriterOptions.RandomBytes(Span<byte>)`, which fills the salt (the span
  is 8, 12 or 16 bytes). The hook exists so tests and the hash matrix can reproduce output; real archives need unpredictable salts.
- Passwords never reach messages, logs or `ToString` (`ArchivePassword.ToString` returns a fixed text). The password is
  copied into a byte array for the key derivation and cleared right after; derived keys are held by the cipher object and
  cleared when it is disposed.

## Platforms

On browser-wasm `System.Security.Cryptography.Aes` throws `PlatformNotSupportedException` (the BCL annotates it
`UnsupportedOSPlatform("browser")`), so AES zips and 7z archives cannot be read or written there. Traditional zip encryption is
plain managed code and works. SHA-256, HMAC-SHA1, PBKDF2 and `RandomNumberGenerator` do work in the browser (hash matrix
entries `encrypt-primitive/*`).


<!-- https://recursiveai.org/onearchive/docs/7z-reader/ -->

# 7z reader

How `SevenZipReader` (`src/OneArchive/SevenZip/`) reads 7z archives and the choices the format document leaves open.
Sources: `7zFormat.txt` (header structures) and `Methods.txt` (method IDs) from the 7-Zip documentation; every point the
documents do not settle was fixed by a black-box run of 7-Zip 26.00 (`testdata/7z/generate.py` writes the vectors).

## What is read

- **Signature header** (32 bytes): signature `37 7A BC AF 27 1C`, version (major must be 0), the start-header CRC over
  bytes 12–31, then the next header's offset, size and CRC. A next-header size of 0 is an empty archive (7-Zip writes
  one for an empty input).
- **Header**: plain (`0x01`) or encoded (`0x17`: a streams info whose single folder decodes to the real header; up to four
  levels). Archive properties are skipped. Additional-streams info is parsed and ignored; names, times or attributes that
  point into it (`External != 0`) raise `ArchiveUnsupportedException`.
- **Streams info**: pack info, folders (coders, bond pairs, packed stream indices, unpack sizes, folder CRC), substreams
  (counts, sizes, CRCs).
- **Files info**: empty-stream, empty-file and anti bit vectors, names (UTF-16LE), creation, access and modification
  times (100 ns ticks since 1601-01-01 UTC), Windows attributes. Dummy padding, start positions, comments and unknown
  properties are skipped by their size.

## Choices where the document is silent (all checked against 7-Zip output)

| Point | What the reader does |
|---|---|
| Substream sizes | A folder with n files stores n−1 sizes; the last is the folder's unpack size minus the others. With no `kSize`, only folders with one file are allowed. |
| Substream CRCs | A folder with exactly one file and a folder CRC takes that CRC; digests are stored only for the other files, in folder order. |
| Directory | An empty-stream entry that is not marked empty-file, or any entry whose attributes carry `0x10`. Its size is 0 and any data stream is ignored. |
| Unix mode | Attribute bit `0x8000` says the high 16 bits are `st_mode`. The permission bits (`& 0xFFF`) become `Mode`; the type bits choose symlink (`0xA000`), directory, device or FIFO. Without the bit, `Mode` is null. |
| Symbolic link | A file of type symlink whose content is the target; `LinkTarget` is read when the header is read (at most 4 × `MaxPathLength` bytes). |
| Anti-items | Listed in `Items` with `IsAnti`; `GetNextEntry` skips them, so extraction never sees a deletion marker. |
| Names | Kept exactly as stored, without a trailing `/`. Extraction applies the shared path rules. |
| Coder order | Coders are kept in header order. BCJ2 (four inputs) is parsed and its graph checked, but not decoded. |
| Stream directions | `NumInStreams` are the coder's packed-side inputs when decoding, `NumOutStreams` its outputs; a bond feeds a coder input from another coder's output; unbound inputs are the folder's packed streams, in the listed order; the one unbound output is the result. |

## Coders

Decoded: Copy (`00`), Deflate (`04 01 08`), Deflate64 (`04 01 09`, raw blocks, no properties, same as Deflate), BZip2 (`04 02 02`), LZMA (`03 01 01`, 5 property bytes: lc/lp/pb and the
dictionary size; the unpack size comes from the header, an end marker is optional), LZMA2 (`21`, one property byte, the
dictionary size code, at most 40), Delta (`03`, one property byte: distance minus one), the branch filters x86, PowerPC,
IA-64, ARM, ARM Thumb, SPARC, ARM64 and RISC-V (short IDs `04`–`0B`; the older long IDs `03 03 xx xx` for the first six;
an optional 4-byte start offset as property), and Zstandard under the ID `F7 11 01` that `Methods.txt` lists for external
codecs (the frame is self-describing, so the coder's properties are ignored). Any other coder, or one with several
streams (BCJ2), raises `ArchiveUnsupportedException` naming the method ID, when the content is opened;
listing works. Each coder's output is cut to the unpack size the header gives; a coder that ends early is a truncated-data
error naming the folder's first packed stream offset.

## Limits and safety

- `SevenZipReaderOptions.MaxHeaderSize` (default 64 MiB) bounds the next header and any encoded header, checked on the
  declared size before allocating.
- `MaxDictionarySize` (default 64 MiB) bounds an LZMA or LZMA2 dictionary, checked on the header's property bytes before
  the decoder allocates (a typed limit error otherwise). It is separate from `ArchiveLimits.MaxWindowSize` because 7-Zip's
  own default dictionary is 16 MiB. The zip LZMA and xz entries use `MaxWindowSize`.
- `MaxCodersPerFolder` (default 32) and 64 streams per folder; every count read from the header is checked against the
  remaining header bytes or the limit before an array is allocated.
- File count against `ArchiveLimits.MaxEntryCount`; each folder's unpack size against `MaxTotalSize` before any decoding;
  the declared entry sizes through the shared `Admit` check; the expansion ratio over the bytes read from packed streams
  and bytes produced.
- Coder graphs: bonds and packed indices must use each stream once, and every coder must be reachable from the result
  (a loop is refused).
- Packed stream positions are checked against the file length when the header is read.
- A file's CRC-32 is checked when it has been read to its end (`VerifyChecksum`, default on). The header CRCs always are.
- The 7z header is at the end of the file, so the source must seek. `Archive.Open` copies a non-seekable source into
  memory first, up to `MaxTotalSize`.

## Random access and streaming

`OpenEntryStream(entry)` leases a decoder for the entry's folder and gives it back when the entry stream is disposed; the
reader keeps the two most recently used idle decoders, so opening a solid folder's entries in archive order decodes the
folder once, while going back to an earlier entry decodes from the folder's start. A decoder that threw is discarded and
the next use starts the folder over. In sequential reading the reader keeps the folder's decoder open, so an
archive read in order decodes each folder once. An entry stream can seek; seeking backwards re-decodes, and a seek turns
off the CRC check for that stream.

## Not done yet

The BCJ2 coder. PPMd (variant H) is read (`ppmd.md`). AES-256 is read (`7z-aes-derivation.md`). Writing is in `7z-writer.md`.

## Zip entries (methods 14 and 95)

Method 14 is read as APPNOTE 5.8 describes: a 4-byte header (LZMA SDK major and minor version, then the properties size,
which must be 5), the 5 property bytes, then raw LZMA data. The uncompressed size comes from the zip headers; a streamed
entry (size unknown) must end with an end marker. Method 95 is one xz stream. Both use `ArchiveLimits.MaxWindowSize` for
the dictionary. Both are read only.


<!-- https://recursiveai.org/onearchive/docs/7z-writer/ -->

# 7z writer

How `SevenZipWriter` (`src/OneArchive/SevenZip/`) writes 7z archives and the choices the format document leaves open.
Sources: `7zFormat.txt` (structures) and `Methods.txt` (method IDs); every point they do not settle was fixed by listing
archives written by 7-Zip 26.00 (see the reader notes in `7z-reader.md`). `Archive.Create("a.7z")` uses the writer;
`ArchiveCreateOptions.SevenZip` carries its options and `CompressionLevel` overrides `Level`.

## Determinism

The bytes depend only on the entries (in the order the caller gives them) and `SevenZipWriterOptions`. No clock is read, no
thread is used, and nothing depends on the platform, the CPU's vector support or how the content streams chunk their reads.

## Layout

32-byte start header (signature, version 0.4, CRC over bytes 12–31, next-header offset, size and CRC), the packed streams of
every folder in order, then the header. The start header is written as zeros first and patched last, so the output must seek.
**A non-seekable output is supported by building the whole archive in a `MemoryStream` and copying it out at `Finish`**
(limited to 2 GiB); a seekable output is written in place, starting at its current position.

With `EncodeHeader` (default) the header is compressed with LZMA (preset 5, dictionary up to 16 MiB but no larger than the
header, size known so no end marker) and stored after the packed streams; the next header is then an `EncodedHeader` (0x17)
record: one pack stream, one LZMA folder with its unpack size and CRC. Without it the plain `Header` (0x01) is the next header.
An archive with no entries is just the 32-byte start header with next-header size 0, as 7-Zip writes it.

## Header content

- `MainStreamsInfo` only when some file has content: pack sizes, folders, and for each folder its file count, the sizes of
  all but the last file and a CRC-32 for every file. Folders carry no folder CRC; the per-file CRCs are in the substreams.
- `FilesInfo` in caller order: `EmptyStream` (directories, empty files and nothing else), `EmptyFile` (the empty-stream
  entries that are files), names (UTF-16LE, NUL-terminated, a trailing `/` removed, other characters kept as given),
  creation, access and modification times (100 ns ticks since 1601-01-01, only the properties some entry has, an
  all-defined byte or a bit vector), and attributes. No padding (`kDummy`) entries are written.
- Attributes: `0x20` (archive) for files and symbolic links, `0x10` for directories; when the entry has a `Mode`, or is a
  symbolic link, `0x8000` is added and the high 16 bits hold `st_mode` (type `0x8000`, `0x4000` or `0xA000` plus the low 12
  permission bits). A symbolic link without a mode gets `0777`.
- Symbolic links are files whose content is the UTF-8 link target. Hard links, devices and FIFOs raise
  `ArchiveUnsupportedException`. Anti-items are not written. Times before 1601 raise `ArgumentException`.

## Folders and coders

A file with content goes into the current folder, or starts a new one. A folder is closed before a file when `Solid` is off,
the filter chosen for the file differs from the folder's, the folder already holds `SolidFileCount` files (when non-zero), or
adding the file would pass `SolidBlockSize` bytes of content (unless the folder is empty, so a huge file still gets a folder).
Empty files and directories never touch folders.

Coder order in the header follows 7-Zip: the compression method first, then the filter, with one bond pair (input 1 ← output
0); the folder's single packed stream is the method's input. Both coders list the folder's unpack size.

| Method | ID | Properties |
|---|---|---|
| Copy | `00` | none |
| LZMA2 | `21` | 1 byte, the dictionary size code |
| LZMA | `03 01 01` | 5 bytes: `(pb*5+lp)*9+lc`, dictionary size LE32 |
| Deflate | `04 01 08` | none; raw Deflate at the given level (0–9) |
| Deflate64 | `04 01 09` | none; raw Deflate64 at the given level (0–9) |
| Zstandard | `F7 11 01` (the external-codec ID in `Methods.txt`) | none; the frame is self-describing. Standard 7-Zip builds cannot read it. |

Filters: Delta `03` (property: distance − 1); x86 `03 03 01 03`, PowerPC `03 03 02 05`, IA-64 `03 03 04 01`, ARM `03 03 05 01`,
ARM Thumb `03 03 07 01`, SPARC `03 03 08 05` (the long IDs 7-Zip itself writes for these, checked); ARM64 `0A` and RISC-V `0B`
(short IDs). Start offset 0, no properties.

`Filter = Auto` (default) looks at the first up to 4096 bytes of the file that would open a folder: ELF (class any, machine
3/62 x86, 40 ARM, 183 ARM64, 243 RISC-V, 50 IA-64 little-endian; 20/21 PowerPC, 2/43 SPARC big-endian) and PE (`MZ`, the
`PE\0\0` signature at the offset in the header, machine `014C`/`8664` x86, `01C0` ARM, `01C2`/`01C4` Thumb, `AA64` ARM64)
get their instruction set's filter; everything else gets none. The choice is made for every file, and a file whose choice
differs from the open folder's starts a new folder, so interleaved executables and data make many folders; sort entries by
kind or set the filter explicitly to avoid that. With `Copy`, Auto means no filter.

## Levels

`Level` 0 is Copy whatever `Method` says. 1–9 mean the following, chosen by measurement against `7z a -mx=N` (Silesia text and
executables) and by listing 7-Zip's dictionary choices (LZMA2 at 256 KiB, 4 MiB and 32 MiB for -mx=1, 3 and 5):

| Level | LZMA dictionary | Encoder preset |
|---|---|---|
| 1 | 256 KiB | 3 (hash chain, fast mode) |
| 2 | 1 MiB | 3 |
| 3 | 4 MiB | 3 |
| 4 | 16 MiB | 4 |
| 5, 6 | 32 MiB | 5, 6 |
| 7–9 | 64 MiB | 7–9 |

`DictionarySize` overrides the dictionary and `Lzma` replaces the whole LZMA configuration. Deflate uses the level as is;
Zstandard maps 1–9 to levels 1, 2, 3, 5, 7, 9, 12, 16, 19.

**Dictionary and memory.** The LZMA family sizes its tables from the folder's size when it is known. A non-solid folder knows
it (the file's size). A solid folder's first 16 MiB are held back: if the folder ends within them the compressor is created
knowing the size, otherwise without; the output is a function of the data either way.

## Streaming

File content is read sequentially from the caller's stream (any `Stream`, seekable or not) in 64 KiB pieces and must be
exactly `Size` bytes; a shorter or longer stream raises `ArgumentException` and the writer refuses further writes. Each piece
goes through the CRC, the filter and the compressor straight to the output, so memory is bounded by the compressor's tables
(plus the first 16 MiB of a solid folder and the entry names and CRCs until `Finish`).

## Not written

BCJ2 and BZip2 coders (PPMd is written, see `ppmd.md`), archive properties, additional streams, start-position and
comment properties.

## Encryption

With `SevenZipWriterOptions.Password` set, every folder gets a 7zAES coder (method ID `06 F1 07 01`) as its first coder, ahead of
the compression method and filter, which are bound to it one after another (`7z-aes-derivation.md` has the layout and key derivation).
The AES stream sits between the compressor and the file: the coder's unpack size is the compressed size, the packed size is that
rounded up to 16, and the pad bytes are zero. With `EncryptHeader` (default on) the header is always stored as an encoded
header whose folder is `[AES, LZMA]`, so names and sizes are hidden too; without it the header stays readable and only file
content is encrypted. The IV (16 bytes) comes from `RandomNumberGenerator`, or from `SevenZipWriterOptions.RandomBytes`, which
exists for tests and the hash matrix and makes the output reproducible. The key is derived once per writer with
`KeyDerivationPower` (default 19, the cost 7-Zip uses; 0 to 24 accepted).


<!-- https://recursiveai.org/onearchive/docs/7z-aes-derivation/ -->

# 7z AES-256 (method 06 F1 07 01)

`7zFormat.txt` and `Methods.txt` give the method ID and say nothing about its properties, key derivation or padding. Those were
pinned by experiment against 7-Zip 26.00 (`7z`, used as a black box), written down here before the code that uses them.
A value enters the code only through one of the experiments below. Scripts: `testdata/encryption/generate.py` makes the
reference archives; the experiments themselves used a small Python script with `hashlib` and the OpenSSL-backed
`cryptography` package as an independent implementation of the hypothesis.

## What the code does

- **Coder properties** (experiments 1 to 3): byte 0 holds the key-derivation power in bits 0-5, and bits 6 and 7 are the extra
  one of the IV size and salt size. Byte 1 holds the salt size's remaining four bits in its high nibble and the IV size's in its
  low nibble. So `saltSize = bit7 + (b1 >> 4)` and `ivSize = bit6 + (b1 & 15)`. The salt bytes follow, then the IV bytes.
  The reader requires `ivSize = 16` and rejects other sizes with a typed error (7-Zip itself refuses them). A writer uses no
  salt, a 16-byte IV, and so writes `0x40 | power`, `0x0F`, IV.
- **Key** (experiment 1): SHA-256 over 2^power repetitions of `salt || password as UTF-16LE || round counter`, where the
  counter is the round number as an 8-byte little-endian integer starting at 0, all fed to one running hash. The 32-byte digest
  is the AES-256 key. 7-Zip writes power 19. The reader refuses a power above `SevenZipReaderOptions.MaxKeyDerivationPower`
  (default 24) with `CompressionLimitException` before any hashing, since 2^power rounds is the work a hostile archive asks for.
- **Cipher** (experiment 1): AES-256 in CBC mode without padding, with the 16-byte IV, over the whole packed stream. The packed
  size is a multiple of 16; the coder's unpack size (in the folder's unpack-size list) is the true length and the rest is padding.
- **Padding** (experiment 4): the encoder pads the last block with zero bytes.
- **Wrong password**: there is no verifier. A wrong key decrypts to noise, which fails the CRC or the next coder. The reader
  reports that as `ArchivePasswordException` (`Problem = Incorrect`) for folders that have an AES coder, and says in the message
  that 7z cannot tell it from damage.
- **Encrypted header** (experiment 5): the signature header's next-header record is an EncodedHeader (0x17) whose single folder
  is `[AES, LZMA]`: the AES coder first, its output bound to the LZMA coder's input; unpack sizes `[LZMA packed size, header size]`.

## Experiments

All with password `test123` (and, for 6, a Unicode one), source file `a.txt` of 1200 bytes, `7z a -t7z -mx0 -mhe=off`.

1. **Layout and key derivation.** The coder in the archive read `24 06 F1 07 01 12 | 53 0F | 16 bytes`: flag byte, ID, property
   size 18, then `53 0F` and 16 random-looking bytes. Hypothesis: power 0x13 (19) from `0x53 & 0x3F`; 16-byte IV (bit 6 plus nibble 15);
   no salt; key as above. Decrypting the 1200 packed bytes with that key and IV reproduced `a.txt` and its CRC-32. Result: confirmed.
2. **A salt goes before the IV.** An archive was rebuilt in Python with a 4-byte salt `01 02 03 04` (properties: byte 0 `0x40|19`, byte 1
   `(4<<4)|15`, the salt, the IV), the data re-encrypted with the salted key, and both header CRCs recomputed. `7z t -ptest123`:
   `Everything is Ok`; `-pwrong`: `CRC Failed in encrypted file. Wrong password?`. Confirms the nibble layout and the position of the salt.
3. **Extra bits and sizes.** Same construction with a 5-byte salt (flag bit 7 set, nibble 4) and a 16-byte salt (bit 7, nibble 15):
   both `Everything is Ok`. An 8-byte IV (no bit 6, nibble 8, zero-padded to 16 for the cipher): `Unsupported Method`. Powers 10 and 0:
   `Everything is Ok` (the power field means 2^power rounds).
4. **Padding.** A 1001-byte file: packed size 1008, unpack size 1001. Decrypting and looking at bytes 1001-1007: all zero.
5. **Encrypted header.** `-mhe=on` archives open only with the password and decode with the construction above. The reader's
   test vectors in `testdata/encryption/vectors/` come from these runs.
6. **Password encoding.** A password with `ä ö €` and a non-BMP character produced an archive whose key matched UTF-16LE of the
   string (surrogate pairs included) in the key derivation above.

The round trip the other way is part of the test suite: archives from `SevenZipWriter` pass `7z t -p...` and extract byte-exact,
with and without header encryption, and `7z t` with a wrong password fails.

## Limits and behaviour

- A missing password for a folder names the entry (`ArchivePasswordException.EntryName`, `Problem = Required`); for an
  encrypted header the name is empty and the exception comes from the first call that needs the header.
- Listing an archive whose header is not encrypted works without a password. Links in encrypted folders are listed with an
  empty target in that case.
- Memory: one 64 KiB buffer per open decryptor; key material is cleared on dispose where the BCL allows (the key array held by
  the reader's cache is zeroed when the reader is disposed; `Aes` copies the key it is given).


<!-- https://recursiveai.org/onearchive/docs/cpio-ar/ -->

# cpio and ar

Normative notes for the `Cpio` and `Ar` containers in `RecursiveAI.OneArchive`: what is read, what is written, and every
choice the formats leave open. A port should reproduce the same bytes from this text.

Sources: the GNU cpio manual ("new ASCII" and "crc" formats), POSIX.1 pax's cpio interchange format ("odc"), the `ar(5)` and
`deb(5)` man pages, and public descriptions of the GNU and BSD long-name conventions. Gaps were closed by running GNU cpio 2.15,
GNU ar (binutils 2.46) and bsdtar 3.8 as black boxes on small trees; the findings are marked **[tool]**. The vectors in
`testdata/cpio/` and `testdata/ar/` come from `python3 testdata/cpio/generate.py`.

## cpio

### Header layouts

| Variant | Magic | Header | Fields | Padding |
|---|---|---|---|---|
| newc | `070701` | 110 bytes | 13 fields of 8 upper-case hex digits after the magic: inode, mode, uid, gid, nlink, mtime, filesize, dev major, dev minor, rdev major, rdev minor, namesize, check | header + name padded to 4; data padded to 4 |
| crc | `070702` | as newc | as newc; `check` is the 32-bit sum of the data bytes (regular files only) | as newc |
| odc | `070707` | 76 bytes | octal: dev (6), inode (6), mode (6), uid (6), gid (6), nlink (6), rdev (6), mtime (11), namesize (6), filesize (11) | none |

- `namesize` counts the NUL that ends the name. The name is UTF-8.
- The archive ends with an entry named `TRAILER!!!` (nlink 1, everything else zero), then zero bytes up to the block size
  (default 512, as GNU cpio writes). Readers ignore what follows the trailer.
- The mode is the full `st_mode`: type in `0170000` (`0100000` file, `040000` directory, `0120000` symlink, `020000` character
  device, `060000` block device, `010000` FIFO, `0140000` socket), permissions in the low 12 bits.
- A symbolic link's data is its target; `filesize` is the target's length.
- Device numbers: newc and crc store rdev major and minor separately. **[tool]** odc stores one 18-bit number, `major * 256 + minor`
  (GNU cpio and bsdtar agree: major 259, minor 3 is `0o201403`). The writer refuses an odc major above 1023 or minor above 255.
- The old binary format (`0xC771`) is recognised and raises `ArchiveUnsupportedException`.

### Hard links

newc and crc store a file with a link count above one once: the headers of all names but the last have `filesize` 0, the last one
carries the data. **[tool]** GNU cpio -o does this for the names it meets in the input; odc stores the data under every name.

Reader: a newc or crc regular file with `nlink > 1` and `filesize 0` is a *stub* (`IsLinkStub`). On a seekable stream the reader
makes one pass over the headers (at most `MaxEntryCount` of them; a header that fails to parse ends the pass, and the main read
reports the same error in order), finds the member with the same (dev major, dev minor, inode) that holds data, and gives the stub
that content (`ContentFromLaterLink`, `Size` is the data size, `StoredSize` stays 0). Every stub counts toward
`MaxTotalSize`. On a forward-only stream, or with `ResolveLinkStubs = false`, a stub is empty.

Writer: a generic `ArchiveEntryKind.HardLink` raises `ArchiveUnsupportedException`, because the data belongs to the last name
and a streaming writer cannot know which that is. A caller who wants links writes `CpioEntry` values with the same `Inode` and
`NLink`, empty content for all names but the last, and the data with the last. **[tool]** GNU cpio -i and bsdtar both restore those
as one file.

### Writer defaults

- Inode: `CpioEntry.Inode` if above zero, else a counter that starts at 1 and counts every entry written. Device fields are zero.
- Link count: `CpioEntry.NLink` if above zero, else 2 for a directory and 1 otherwise.
- Mode: the entry's permission bits, or `0644` (files), `0755` (directories), `0777` (symlinks). Uid and gid: the entry's, else 0.
- Time: the entry's modification time in seconds (before 1970 is refused), else 0. Nothing reads the clock.
- A directory's trailing `/` is dropped. Sockets are not written; a FIFO is.
- crc: **[tool]** GNU cpio sums regular files only; a symlink's `check` is 0, and so is ours. The sum comes before the data, so
  seekable content is read twice; content that cannot seek is buffered up to `MaxBufferedSize` (64 MiB), else
  `ArchiveUnsupportedException`.
- Field overflow (odc: 18 bits for inode, mode, uid, gid, nlink and dev, 33 bits for time and size; newc: 32 bits) raises
  `ArchiveUnsupportedException` naming the field and entry.

### Reader rules and limits

- All entries must carry the magic of the first.
- Every number is validated as digits of the right base. `namesize` must be at least 1, the name must end in NUL and hold no other,
  and the name length is checked against `MaxPathLength` *before* the buffer is allocated. A symlink's data is checked against
  `MaxPathLength` before it is read.
- A stored size is checked against `MaxTotalSize` through the shared entry admission, before any content is read.
- The crc sum is checked when an entry's data is read from its first to its last byte in order (`VerifyChecksum`, default on).
  Skipped entries and entries read by seeking are not checked. **[tool]** A symlink's check is not verified (GNU writes 0).
- `RequireTrailer` (default on): an archive that ends at an entry boundary without `TRAILER!!!` is a truncated-archive error.
- A socket reads as `ArchiveEntryKind.Fifo`; `RawMode` tells them apart. A mode with no file type is an error.
- Device and FIFO entries are listed with their numbers; safe extraction never creates them (`SpecialFiles` policy).

## ar

### Layout

`!<arch>\n`, then members. A member is a 60-byte header, its data, and one `\n` if the data length is odd:

| Bytes | Field | Encoding |
|---|---|---|
| 0–15 | name | ASCII/UTF-8 padded with spaces |
| 16–27 | mtime | decimal, space padded (blank reads as 0) |
| 28–33 | uid | decimal |
| 34–39 | gid | decimal |
| 40–47 | mode | octal (blank reads as 0) |
| 48–57 | size | decimal, required |
| 58–59 | end marker | `` `\n `` |

The archive has no trailer. A final odd-sized member whose pad byte is missing is accepted.

### Names

- **Inline.** The name followed by `/` (System V and GNU), or the bare name (BSD). The reader removes trailing spaces, then one
  trailing `/`.
- **GNU long names.** A member named `//` holds the long names, each ending in `/\n`; a member whose name field is `/` followed by
  digits takes the name at that offset in the table, up to the newline, minus one trailing `/`. The table comes before the
  members that use it. The reader reads it into memory (at most `MaxNameTableSize`, default 8 MiB).
- **BSD long names.** The field is `#1/` and the name's byte length; the name is the first bytes of the data (trailing NULs removed)
  and the member's size counts it. **[tool]** bsdtar writes the name unpadded and uses `#1/` for names longer than 16 bytes or with a
  space.
- **Symbol tables** are `/` and `/SYM64/` (GNU) and `__.SYMDEF`, `__.SYMDEF SORTED`, `__.SYMDEF_64`, `__.SYMDEF_64 SORTED` (BSD).
  They are not interpreted. By default they and the `//` table are left out of the listing; `IncludeSpecialMembers` returns them
  as entries with `MemberKind` set and their data at `DataOffset`. They are not safe extraction names, which is why they are off by
  default. A `.deb` package is an ar archive of `debian-binary`, `control.tar.*` and `data.tar.*`, and reads like any other.
- **[tool]** GNU ar writes mtime 0, uid 0, gid 0 and mode `644` in its default deterministic mode, and leaves the time, uid, gid and
  mode fields of `/` and `//` blank. bsdtar writes real times and the mode with type bits (`100664`). The reader takes all of these.
- A thin archive (`!<thin>\n`) raises `ArchiveUnsupportedException`.

### Writer

- Only regular files can be stored; other kinds raise `ArchiveUnsupportedException`. `ArchiveBuilder` leaves directories out of an
  ar and keeps the files' paths in their names.
- Header values: mtime 0 unless the entry has one; uid and gid from the entry (0 default, at most 999999); mode `0100000 | permission
  bits` (default `0644`) in octal; size at most 9,999,999,999.
- GNU variant (default): a name of up to 15 bytes that does not start with `/` is written as `name/`; others go to the `//` table
  as `name/\n` and the field is `/offset`. The table (and an optional caller-built `/` symbol table, `SetSymbolTable`) is written
  before the members, so the members are held until `Finish`: in memory up to `MaxMemoryBuffer` (16 MiB), then in a temporary
  file. Where temporary files are not available, a larger archive raises `ArchiveUnsupportedException`.
- BSD variant: names of up to 16 bytes with no space, not ending in `/` and not starting `#1/` are written bare; all others as
  `#1/length` with the name unpadded at the start of the data. It streams.
- Common variant: `name/` only, names above 15 bytes are refused. It streams.
- A name may not be empty or contain NUL (and for ar a line feed).

### Reader limits

Header fields are validated as digits; the end marker must be present. A BSD name length is checked against `MaxPathLength` and
against the member's size before the name is read; the `//` table size against `MaxNameTableSize` before it is read; a `/N` reference
against the table's length. Left-out special members count toward `MaxEntryCount`.

## Archive happy path

- Detection: the magic (`!<arch>\n`, `!<thin>\n`, `070701`, `070702`, `070707`), then the extension: `.cpio` and `.cpio.gz|zst|zstd|lz4|bz2|xz`;
  `.a`, `.ar`, `.deb`, `.udeb`.
- `Archive.Create` picks the format from the extension, or `ArchiveCreateOptions.Format`; `.cpio.gz` and `.cpio.zst` compress the whole
  archive, as for tar. cpio reads and writes `ArchiveCreateOptions.Cpio` / `ArchiveOpenOptions.Cpio`, ar the same with `Ar`.
- Extraction applies the shared path rules to every name; a stored `../x`, an absolute path or a symlink escape is refused as for tar.


<!-- https://recursiveai.org/onearchive/docs/iso/ -->

# ISO 9660 reader and writer

Normative notes for `RecursiveAI.OneArchive.Iso`. Clause numbers refer to ECMA-119; SUSP is IEEE P1281 and RRIP is IEEE P1282.

## Reading

- Descriptors are scanned from sector 16 (clause 8): primary (type 1), supplementary/Joliet (type 2, escape sequences `%/@`, `%/C`, `%/E`), boot record (listed only), terminator (type 255, version 1, identifier `CD001`). At most 256 sectors are examined; a missing terminator is a format error.
- The logical block size must be 2048; any other value raises `ArchiveUnsupportedException`. Both-endian fields whose halves disagree are a format error.
- Rock Ridge is active when the root directory's "." record carries `SP` and either a `PX` entry or an `ER` entry naming RRIP/IEEE. It applies to the primary hierarchy only. Entries read: `PX` (36- and 44-byte forms), `PN`, `SL`, `NM`, `TF` (short and long time forms), `CL`/`PL`/`RE` (relocated directories; the `RR_MOVED` placeholder at depth 1 is hidden), `CE`, `ZF` (listed; extracting raises `ArchiveUnsupportedException`), `SP`, `ER`, `ST`.
- `Tree = Auto` lists the primary hierarchy when Rock Ridge is active, else Joliet when present, else the primary hierarchy. `IsoEntry.Name` is the name from the chosen source; `IsoEntry.IsoPath` is always the identifier path of the hierarchy read.
- Multi-extent files (several directory records with the multi-extent flag, clause 9.1.6) are one entry whose `Extents` list the pieces.
- Safety: every extent must lie inside the image; a directory seen twice (loop or duplicate) is a format error; continuation areas (`CE`) must lie inside one sector, inside the image, must not overlap one another or the area being read, and are limited by `MaxSystemUseSize` per record and `MaxContinuationBytes` per image (default 64 KiB and 128 MiB); directories are limited by `MaxDirectorySize` (16 MiB); the shared `ArchiveLimits` bound entry count, path length and depth. Traversal uses an explicit stack, not recursion.

## Writing

- Defaults: interchange level 3, Joliet on, Rock Ridge on. Level 3 is the default because real names travel in Joliet and Rock Ridge, so the level only decides how much the plain names are cut, and it is the only level that allows files of 4 GiB or more (written as several extents of at most 4 GiB - 2048 bytes).
- Layout: 16 zero sectors, the primary descriptor, the Joliet descriptor, the terminator, the path tables (little-endian then big-endian, primary then Joliet), the primary directories, the Rock Ridge continuation areas, the Joliet directories, then file data in the order the entries were written, each file starting on a sector. The continuation areas follow the primary directories directly because some readers (bsdtar 3.8) only find an `NM` or `SL` continuation that lies next to the directory being parsed.
- Directory records are sorted as clause 9.3 asks, path tables as clause 6.9.1 asks. Names cut to the level that collide get a three-digit counter in the order the entries were written. Joliet names are cut to 64 UTF-16 units keeping a short extension, forbidden characters become `_`.
- Rock Ridge per record: `PX` (44 bytes), `TF` (modify, plus access and attributes when the entry has them), `NM`, `SL`, `PN` for devices. The root "." record carries `SP` and `ER` (`RRIP_1991A`). A name or link target that does not fit the record is split: the first part fills the record, the rest goes into a continuation area.
- Determinism: no clock is read. Volume dates are "not specified" unless `VolumeTime` is set; entries and implied directories without a time use `DefaultTime` (1970-01-01). Hard links share one extent. Zero-length files get extent 0. Deep directories (more than eight levels) are written as they are, not relocated.
- Symbolic links, devices and FIFOs need Rock Ridge; without it the writer raises `ArchiveUnsupportedException`.


<!-- https://recursiveai.org/onearchive/docs/cfb/ -->

# Compound file reader and writer

Normative notes for `CompoundFileReader` and `CompoundFileWriter` in `RecursiveAI.OneArchive.Cfb`. Source: [MS-CFB] Compound File Binary File Format,
revision 13.0, sections 2.1 to 2.9 (cited as "MS-CFB 2.x"). The format is the container of legacy Office documents (.doc, .xls, .ppt),
Outlook messages (.msg) and installer packages (.msi). Choices the specification leaves open are marked **[choice]**.

## Structures read

All integers are little-endian.

- **Header** (MS-CFB 2.2, 512 bytes at offset 0; in a version 4 file the rest of the first 4096-byte sector is skipped): signature
  `D0 CF 11 E0 A1 B1 1A E1`, major version 3 or 4, byte order mark `0xFFFE`, sector shift 9 (version 3) or 12 (version 4), mini sector shift 6,
  mini stream cutoff 4096, and the counts and first sectors of the directory, FAT, mini FAT and DIFAT. The header CLSID, minor version,
  reserved fields, directory sector count and transaction signature are exposed (`CompoundFileHeader`) and not checked.
- **Sectors** (MS-CFB 2.1): sector `n` starts at byte `(n + 1) * sector size`. A sector must lie wholly inside the file; the file's
  sector count is `floor(length / sector size) - 1`. Numbers above `0xFFFFFFFA` are never sectors.
- **DIFAT** (2.5): the first 109 FAT sector numbers are in the header, the rest in a chain of DIFAT sectors (each holds `sector size / 4 - 1`
  numbers and a link to the next). The chain has exactly the declared number of sectors and its last link is `ENDOFCHAIN`.
- **FAT** (2.3): the `FAT sector count` sectors named by the DIFAT, concatenated. A chain follows `FAT[s]` until `ENDOFCHAIN`.
- **Mini FAT and mini stream** (2.4): the mini FAT is a chain of exactly the declared number of sectors; the mini stream is the root
  entry's stream. A stream smaller than 4096 bytes lives in the mini stream in 64-byte mini sectors (a size of exactly 4096 is a normal stream).
- **Directory** (2.6): a chain of 128-byte entries. Entry 0 is the root storage and must have object type 5; no other entry may.
  Fields read: name (UTF-16, at most 31 characters and a terminating null), name length, object type (0 unallocated, 1 storage, 2 stream,
  5 root), color flag, left and right sibling IDs, child ID, CLSID, state bits, creation and modified FILETIME, starting sector and
  size. In version 3 files only the low 32 bits of the size count (2.6.1: older writers left the high bits uninitialized); in version 4 the
  size is 64 bits and must be below 2^63. Times that are zero, or beyond the range of `DateTimeOffset`, give no `ModifiedTime`/`CreationTime`;
  the raw FILETIME values are always exposed.

## Validation

The whole structure is checked the first time the reader is used, before any stream is read. Every failure is a
`CompressionFormatException` (format `cfb`) giving the byte offset of the offending field and what was expected, or
`CompressionLimitException` for a limit.

- Header fields as listed above; a cutoff other than 4096, a sector shift that does not match the version, a mini shift other than 6.
- The FAT sector count and DIFAT sector count may not exceed the number of sectors in the file; every sector number must be inside the file.
- **Each sector belongs to at most one chain** (MS-CFB 2.1): DIFAT sectors, FAT sectors, the directory, the mini FAT, the mini stream and every
  stream's chain claim their sectors in a bitmap, and claiming one twice is an error. This also catches loops in any chain. Mini sectors are
  claimed the same way, so streams cannot share mini sectors either. The total size declared by streams therefore never exceeds the file.
- A chain may not enter a free sector (`0xFFFFFFFF`), a FAT or DIFAT sector marker, or a sector outside the FAT's range.
- A stream's chain must be at least `ceil(size / sector size)` sectors long (mini: 64-byte units, and each mini sector must lie inside the
  mini stream). A longer chain is accepted **[choice]**; the extra sectors are claimed and not read. Zero-size streams need no chain.
- Directory: a type other than 0, 1, 2, 5; a color other than 0 or 1; an odd name length, one below 2 or above 64; a name whose terminating
  null is missing or that contains a null before it; an empty name on a non-root entry; a size of 2^63 or more.
- Tree: the tree under each storage is walked with explicit stacks (no recursion follows the input). Every sibling or child ID must be below the
  directory length or `NOSTREAM`, must name a storage or stream, and may be reached once (a loop or a shared subtree is an error). A stream's
  child ID must be `NOSTREAM`. Each storage's children, taken in tree order, must be strictly increasing under the comparison of MS-CFB 2.6.4:
  a shorter name is less; names of equal length are compared one UTF-16 code unit at a time after `char.ToUpperInvariant` (surrogates are
  not changed). Equal names are "unique name" errors, other disorder "order" errors. Colors are exposed, not checked **[choice]**
  (the red-black balance does not matter to a reader).
- Entries that are allocated but not reachable from the root are ignored.

## Limits

`CompoundFileReaderOptions.Limits` (`ArchiveLimits`) and `MaxDirectoryEntries`:

- `MaxDirectoryEntries` (default 262,144, **[choice]**): directory entries of all kinds, checked against the directory chain's length before
  anything is read.
- `MaxEntryCount` and `MaxPathDepth` are checked during the tree walk, so a hostile file fails before names are built; `MaxPathLength` and
  `MaxTotalSize` are checked per entry as it is returned. Depth counts storages from the root (a stream directly under the root has depth 1).
- There is no expansion ratio: nothing is decompressed, and the exclusive-sector rule bounds the output by the file's size.
- Memory is proportional to the file: the FAT and mini FAT are held as arrays (4 bytes per sector), the directory as a struct array.

## Listing and names

- `GetNextEntry` returns storages as `ArchiveEntryKind.Directory` and streams as `ArchiveEntryKind.File`; the root storage is not listed (it is
  `Root`). Order: depth first, each storage before its children, siblings in the tree's name order **[choice]**.
- `ArchiveEntry.Name` is the path from the root, components joined with `/`, each passed through `SanitizeName` **[choice]**. `RawName` is the
  entry's own stored name, unchanged.
- **Name mapping.** CFB names may hold any UTF-16 code unit, including control characters (`\u0005SummaryInformation` is common) and the
  separators `/` and `\`. A name component is rewritten one UTF-16 code unit at a time: each of U+0000-U+001F, U+007F-U+009F, `/ \ : * ? " < > | %`
  and each unpaired surrogate becomes `%` followed by its code unit in four uppercase hex digits (`%0005`, `%002F`). A component that is exactly
  `.` or `..` has its first character escaped, and one that ends in `.` or a space has its last character escaped (so Windows keeps the name).
  Valid surrogate pairs and all other characters are kept. Because `%` is always escaped, the mapping is one-to-one: different stored names never
  give the same component. (Names are unique only under the case-insensitive comparison, so on a case-insensitive file system two entries
  that differ only in case would still collide; extraction then follows the usual `ExistingFilePolicy`.)
- Lookups (`Find`, `OpenStream`) take the stored names separated by `/` and compare them as MS-CFB 2.6.4 does, by binary search over the
  sorted children.

## Access

- The source is a seekable `Stream` or `ReadOnlyMemory<byte>`. `Archive` copies a forward-only source into memory first, subject to `MaxTotalSize`.
- `OpenEntryStream(entry)` gives a seekable read-only stream for any entry in any order. Runs of consecutive sectors are read with one call.
  A storage opens as an empty stream.
- A reader is not thread-safe.

## Not done

Transacted or in-place editing; property-set encoding; the range lock sector (never read or written); transactions; the old beta-2 signature (`0E 11 FC 0D D0 CF 11 E0`), which is
rejected as a bad signature; reading the contents of streams (property sets, Word and Excel records, MAPI properties).

## Tests and tools

`tools/cfb/gen_cfb.py` writes the vectors in `testdata/cfb/` (version 3 and 4, DIFAT beyond 109 entries, the 4096-byte boundary, empty and
deeply nested storages, unusual names, a message-like tree, and about fifty damaged files that each break one rule) and
`manifest.json`; `olefile_check.py` confirms with the olefile package that the good ones read as described. `make-office-vectors.sh` makes
.doc/.xls/.ppt files with LibreOffice and records olefile's view of them in `testdata/cfb/office/`. The `cfb-read`, `cfb-rewrite` and `cfb-write` fuzz targets and their
seeds are in `fuzz/`. `check-written.sh` rewrites every good vector and the Office files in both versions and has olefile list the result.

# Writer

`CompoundFileWriter` (an `ArchiveWriter`; `Archive.Create("x.cfb")`) writes version 3 (512-byte sectors, the default) or version 4
(4096-byte sectors) from storages and streams. Options (`CompoundFileWriterOptions`): `MajorVersion` (3 or 4), `StrictNames` (default true) and
`MaxBufferedBytes` (default 1 GiB). The output depends only on the entries and the options: the clock is never read, and the bytes are the same on
every platform.

## Entries

- A directory entry is a storage, a file entry a stream; other kinds raise `ArchiveUnsupportedException`. The entry name is a path of
  components separated by `/` (empty components are skipped; `.` and `..` components are rejected). Missing parent storages are created with
  zero CLSID, state bits and times. An explicit directory entry for a storage that was only implied sets its properties; a second explicit
  entry, or a stream and a storage with the same name, is a duplicate.
- An entry with an empty name and kind directory describes the root storage: its CLSID, state bits and times. The root's stored name is
  `Root Entry`.
- Properties: a `CfbEntry` supplies `ClassId`, `StateBits` and the raw FILETIMEs (`CreationFileTime`, `ModifiedFileTime`; a zero raw value
  falls back to the time below). Any entry supplies `ModifiedTime` (modified) and `ChangeTime` (created) as FILETIMEs, exact to 100 ns; no
  time gives 0, "not recorded". A time before 1601-01-01 is an `ArgumentException`.
- **Names [choice].** Each path component is the reader's `SanitizeName` form of the stored name, and the writer inverts it
  (`UnsanitizeName`): `%` followed by exactly four uppercase hex digits is that UTF-16 code unit; any other `%` is itself. So the entry
  names the reader returns can be fed straight back to the writer, and a rewrite keeps every stored name. A literal `%` followed by four
  uppercase hex digits therefore has to be written as `%0025` plus the digits. The stored name must have 1 to 31 UTF-16 code units and no
  NUL (MS-CFB 2.6.1), and be unique within its storage under the 2.6.4 comparison (case-insensitive), else `ArgumentException`. With
  `StrictNames` (default) a name containing any of `/ \ : !` is also rejected, as MS-CFB 2.6.1 forbids them; turn it off to store such names (the reader of this
  library accepts them, which is how every vector is rewritten in the tests). A rejected entry changes nothing: the writer stays usable.
- Version 3 streams larger than 2^31 - 1 bytes raise `ArchiveUnsupportedException`. The directory is limited to 262,144 entries, so every
  output reads with the default `MaxDirectoryEntries`; more raises `CompressionLimitException`, as does exceeding `MaxBufferedBytes`.

## Layout [choice]

The layout needs every stream's size, so content is buffered until `Finish` and the file is written in one forward pass, sectors in this order:
header (padded to a whole sector in version 4), directory, mini FAT, mini stream, the data of the streams of 4096 bytes or more in directory
order, FAT, DIFAT. Each sector chain is contiguous.

- **Directory** (2.6): entry 0 is the root; the others follow in preorder, the children of each storage sorted by the 2.6.4 comparison
  (shorter name first, then `char.ToUpperInvariant` per UTF-16 code unit, as the reader does). The unused tail of the last directory sector
  holds free entries (type 0, all-zero except left, right and child = `NOSTREAM`). Version 4 files store the directory sector count in the
  header; version 3 files store 0. Storages have start sector 0 and size 0; empty streams have start sector `ENDOFCHAIN` and size 0. Times, CLSID
  and state bits are written as given. The root's start sector and size describe the mini stream (`ENDOFCHAIN` and 0 when there is none).
- **Siblings (2.6.4).** The children of a storage are linked as a binary search tree by repeatedly taking the middle of the sorted run (the
  upper middle for an even count) as the node; `Child` of the storage is the top node. Colors keep the tree a valid red-black tree: all nodes
  are black when the count is 2^k - 1, otherwise the nodes on the deepest level, which is not full, are red. Every path then has the same number of
  black nodes and no red node has a red child.
- **Mini stream (2.4, 2.5).** A stream of 1 to 4095 bytes is stored in 64-byte mini sectors, each stream's chain contiguous, in directory
  order, and padded with zeros to a mini sector; the mini stream is padded with zeros to a whole sector. The mini FAT has one entry per
  mini sector, padded with `FREESECT`. A stream of exactly 4096 bytes or more is a normal stream.
- **FAT and DIFAT (2.3, 2.5).** The FAT holds one chain per structure, `FATSECT` for each FAT sector and `DIFSECT` for each DIFAT sector;
  unused entries are `FREESECT`. The FAT sector count is the smallest `n` such that all sectors, FAT and DIFAT included, fit in `n` sectors of
  entries. The first 109 FAT sector numbers go in the header (the rest `FREESECT`); the others go in DIFAT sectors of `sector size / 4 - 1`
  numbers each, unused ones `FREESECT`, with the last entry linking to the next DIFAT sector or `ENDOFCHAIN`. The header holds the first
  DIFAT sector (`ENDOFCHAIN` when none) and the count.
- **Header (2.2).** Signature, zero CLSID, minor version `0x3E`, major version 3 or 4, byte order `0xFFFE`, sector shift 9 or 12, mini shift 6,
  zero reserved bytes, mini cutoff 4096, transaction signature 0, first mini FAT sector (`ENDOFCHAIN` when none) and count.
- The last sector is padded with zeros. Sector numbers stay below `0xFFFFFFFA`, else `CompressionLimitException`.


<!-- https://recursiveai.org/onearchive/docs/cab-reader/ -->

# Cabinet reader

Normative notes for `CabReader` in `RecursiveAI.OneArchive.Cab`: read only. Source: Microsoft's [MS-CAB] (Cabinet File Format) 2.1 to
2.4 and 3.1; LZX is in `lzx-decoder.md`. Gaps were closed with cabextract 1.11 and gcab 1.6 run as black boxes; findings are marked
**[tool]**.

## Layout read

All integers are little-endian.

- **CFHEADER** (36 bytes): `MSCF`, reserved, `cbCabinet`, reserved, `coffFiles`, reserved, version minor and major, `cFolders`, `cFiles`,
  `flags`, `setID`, `iCabinet`. Then, if the reserve flag (4) is set, `cbCFHeader` (u16, at most 60000), `cbCFFolder` (u8), `cbCFData` (u8) and
  `cbCFHeader` reserved bytes; then the NUL-terminated `szCabinetPrev`, `szDiskPrev` (flag 1) and `szCabinetNext`, `szDiskNext` (flag 2), each
  at most 255 bytes. Reserved fields are not checked; major version must be 1 (minor is exposed, not checked); unknown flag bits are ignored.
  `coffFiles` must lie after the folder table and inside the file.
- **CFFOLDER** (8 bytes + `cbCFFolder`): `coffCabStart`, `cCFData`, `typeCompress`. Method = `typeCompress & 15` (0 none, 1 MSZIP, 2
  Quantum, 3 LZX); LZX window = `typeCompress >> 8 & 31`, which must be 15 to 21 (other values raise `ArchiveUnsupportedException`).
- **CFFILE** (16 bytes + name): `cbFile`, `uoffFolderStart`, `iFolder`, DOS date, DOS time, `attribs`, NUL-terminated name of at most 256
  bytes. `iFolder` 0xFFFD, 0xFFFE and 0xFFFF mark files continued from the previous cabinet, into the next one, or both; they are listed
  (`CabEntry.Continuation`) but cannot be opened (`ArchiveUnsupportedException` naming the neighbouring cabinet). A file that is not continued
  must satisfy `uoffFolderStart + cbFile <= min(0x7FFF8000, cCFData * 32768)`.
- **CFDATA**: `csum` (u32), `cbData` (u16), `cbUncomp` (u16), `cbCFData` reserved bytes, `cbData` bytes. Blocks of a folder are contiguous
  from `coffCabStart`. `cbUncomp` 0 means the block continues in the next cabinet (`ArchiveUnsupportedException`); more than 32768 is an error.

## Names, times, attributes

- A name with `_A_NAME_IS_UTF` (0x80) is UTF-8 (invalid sequences become U+FFFD); otherwise it is ISO 8859-1 unless
  `CabReaderOptions.LegacyNameEncoding` says otherwise. `\` becomes `/` in `ArchiveEntry.Name`; the stored bytes are in `RawName`.
- DOS date and time are read as UTC, like zip's. A date or time that is not valid gives no `ModifiedTime`.
- `Mode` is 0444, plus owner write unless `_A_RDONLY`, plus execute for everyone with `_A_EXEC`. Hidden and system are only in `Attributes`.

## Data blocks

- **Checksum** ([MS-CAB] 3.1): XOR of the little-endian 32-bit words of the data, then the same function continued over `cbData` and
  `cbUncomp` (four bytes), where a trailing partial word is built from the last byte down. **[tool]** The data reserve is not included,
  although the specification calls the second part the "header area": with reserve bytes present, only this reading extracts under
  cabextract. A zero `csum` is not checked. A mismatch raises `CompressionFormatException` at the block's offset unless
  `VerifyChecksum` is false.
- **Stored**: `cbData` must equal `cbUncomp`.
- **MSZIP**: the block is `CK` then a raw deflate stream that ends with a final block and produces exactly `cbUncomp` bytes. Matches may
  reach into the previous 32 KiB of the folder's output (which is the history of the folder, not of the block); the reader keeps that
  history across blocks and uses the deflate decoder's preset dictionary for it. **[tool]** Reference tools assume every block but the last
  has 32768 bytes; this reader accepts any size.
- **LZX**: one frame per block with `cbUncomp` 32768 (the last block may be shorter; any other short block is an error); a cabinet is
  decoded with one `LzxDecoder` per folder.

## Access and limits

- The source must be seekable (`Archive` copies a forward-only source into memory first, subject to `MaxTotalSize`).
- A folder is decoded from its first block. Reading files in table order costs one pass; going back decodes from the start again. At
  most two idle decoders are kept. Entries are listed in table order.
- Limits: `MaxEntryCount`, `MaxPathLength`, `MaxPathDepth`, `MaxTotalSize` (declared sizes, checked while listing),
  `MaxExpansionRatio` (compressed bytes counted per block including headers, output per block), `MaxWindowSize` (the LZX window,
  checked when an entry is opened). Errors name the format `cab` and the offset of the field or block at fault; errors from the
  deflate and LZX decoders are re-raised with the cabinet offset of the block and their own description.
- Not supported: Quantum folders (`ArchiveUnsupportedException`), files and blocks that span cabinets, LZX windows beyond 2^21, writing.


<!-- https://recursiveai.org/onearchive/docs/rar-reader/ -->

# RAR 5 reader

Normative notes for `RarReader` in `RecursiveAI.OneArchive.Rar`: read only. Source: RARLAB's "RAR 5.0 archive format" technote (section names are
cited in the code). The technote sends readers to the vendor's source for algorithms; that is never followed. Gaps were closed by running
`rar` 7.23 as a black box (`tools/rar-testdata/generate.sh` makes the committed archives); findings are marked **[tool]**.

## What is read

Signature (`52 61 72 21 1A 07 01 00`, searched for within `MaxSfxSize`, default 1 MiB, to skip a self-extracting module), then headers in file
order: optional archive encryption header, main archive header, an optional `CMT` comment service header, then file headers, their service headers,
and an end-of-archive header. The reader never uses the locator, the quick-open data or the recovery record: it walks the headers in order, so the
names listed are the names extracted (the technote's security note). The source need not seek.

- **Header block**: CRC-32 (u32), header size (vint, at most 3 bytes, so at most 2 MiB; at least 2), then that many bytes: type, flags, optional
  extra-area size, optional data size, type fields, extra area. The CRC covers the size field and the body and is always checked. An extra area
  larger than the header, a record that overruns the area, a record field that overruns its record, a vint over 64 bits or over 10 bytes, a data size
  of 2^63 or more: `CompressionFormatException` naming the offset (counted from the first byte of the file, SFX module included).
  Unknown header types, and service headers other than the first `CMT`, are skipped with their data.
- **vint**: seven bits per byte, low group first. Padding bytes of `0x80` are accepted. **[tool]** `rar` pre-allocates several fields and pads them
  (a compression-information field of `80 00`, a quick-open offset of several bytes).
- **Main header**: archive flags (`0x1` volume, `0x2` volume number present, `0x4` solid, `0x8` recovery record, `0x10` locked), volume number, and the
  locator (quick-open and recovery offsets, 0 meaning "not filled in", read as absent) and metadata (original name cut at the first zero byte; a zero first
  byte means no name; creation time in FILETIME, Unix seconds or Unix nanoseconds) records. **[tool]** `-m0 -s` does not set the solid flag: `rar` ignores
  `-s` for stored data. With compression, solid archives list entries in a different order than given, and may swap which of two hard-linked names holds the
  data.
- **File header**: flags (`0x1` directory, `0x2` Unix mtime, `0x4` CRC-32, `0x8` unpacked size unknown), unpacked size, attributes, optional mtime (u32), optional
  CRC-32, compression information, host OS, name (UTF-8, no zero). Compression information: bits 0-5 algorithm version, bit 6 solid, bits 7-9 method (0 stored,
  1-5), bits 10-14 `N`, bits 15-19 and bit 20 only for version 1. Dictionary size is `128 KiB << N`; for version 1 plus `(that / 32) * bits 15-19`.
- **Extra records** (file and service headers): encryption (presence only), hash (type 0 = BLAKE2sp, 32 bytes), file time, file version, redirection, Unix owner,
  service data (ignored). **[tool]** The file time record holds a u32 for each time present when the Unix flag is set (not u64), then one u32 of nanoseconds per time
  when the nanosecond flag is also set; without the Unix flag each time is a u64 FILETIME. Times are kept to 100 ns; a value outside the representable range reads as absent.
- **End header**: flag `0x1` means the set continues in another volume.

## Entry model

| RAR | `RarEntry` |
|---|---|
| directory flag | `Kind = Directory`, size 0 |
| redirection 1 (Unix symlink), 2 (Windows symlink), 3 (junction) | `Symlink`, `LinkTarget`; for 2 and 3 `\` in the target becomes `/` |
| redirection 4 (hard link), 5 (file copy) | `HardLink`, `LinkTarget`. A file copy is stored without data and names the file it copies; extracting it as a hard link gives the same content. Size is 0 for every link: **[tool]** the header carries the target's size, which is not content of this entry. |
| host OS 1 (Unix) | `Mode` = low 12 bits of the attribute word (`st_mode`), `Attributes` = the whole word |
| host OS 0 (Windows) | `Mode` null, `Attributes` = DOS bits |
| header mtime, then the file time record | `ModifiedTime` (the record wins); atime in `AccessTime`; creation time in `CreationTime`; `ChangeTime` is not set |
| Unix owner record | `UserName`, `GroupName`, `Uid`, `Gid` |
| unknown-size flag | `SizeIsUnknown`; `Size` is the first part's data size |

Names are exactly as stored: `\` is data in a Unix-host name and invalid in a Windows-host name, but the extraction rules treat both as separators, so neither can
escape. Names with bytes that are not valid UTF-8 read with replacement characters.

## Stored entries and checks

Method 0 data is the data area. A stored entry that does not continue must have a data area equal to its unpacked size (unless the size is flagged unknown or the data
is encrypted, which pads to 16 bytes). CRC-32 and BLAKE2sp are checked when the stream is read to its end (`VerifyChecksum`, default true); a mismatch is
`CompressionFormatException` naming the stored and computed values. BLAKE2sp is in `Blake2sp.cs`: BLAKE2s from RFC 7693 and the eight-leaf tree of the BLAKE2 paper (fanout 8,
depth 2, inner length 32; leaf `i` hashes the 64-byte blocks `i`, `i + 8`, ...; the last leaf and the root carry the last-node flag). It is internal because RAR is its only
user; it is tested against RFC 7693's BLAKE2s vectors, digests computed with Python's `hashlib` in tree mode, and the hashes `rar` writes.

Compressed entries (methods 1-5) are listed in full and raise `RarUnsupportedException` (a `ArchiveUnsupportedException`) from `OpenEntryStream`:
"RAR compressed entries are not supported yet", naming the entry, method and dictionary size. Nothing is written for them. Encrypted data raises the same type with kind `EncryptedEntry`.
A RAR 4 signature (`52 61 72 21 1A 07 00`) is detected as RAR and raises kind `Rar4Archive` from `Archive.Open` and the reader. An archive encryption header (`rar -hp`) raises kind
`EncryptedHeaders`; reading it would need AES-256 with PBKDF2 and a password check the technote leaves to the vendor's source, so it is not implemented.

## Volumes

The reader is given the first volume and `RarReaderOptions.Volumes`, a function from volume number to a stream (or null). At an end-of-archive header with flag `0x1`, or when a file's data
continues (header flag `0x10`), it opens the next volume, checks its signature and main header (volume flag set, volume number equal to the expected one) and takes over the continuation file
header (flag `0x8`, same name). A file split over volumes reads as one stream. **[tool]** `rar -v40k` names volumes `name.part1.rar`, `name.part2.rar`, ...; `-vn` did not change this.

The CRC-32 or BLAKE2sp in a file header covers that volume's data for every part but the last, and the whole file in the last part's header; the reader checks each part and the whole.
A missing volume is `CompressionFormatException` with `IsTruncated`, raised where the volume is needed (when its data is read or skipped, or when listing reaches it).
`RarVolumes.ForFile` finds volumes on disk: `name.partN.rar` (digit width kept) or `name.rar`, `name.r00` ... `name.r99`, `name.s00` ...; `Archive.Open(path)` uses it. Opening a
volume other than the first is `ArchiveUnsupportedException`.

## Limits

Shared `ArchiveLimits`: entry count, name and link-target length, path depth, total declared size (checked from the header, before data is read; an unpacked size of 2^62 on a
compressed entry is a limit error, 2^63 or more a format error). Header size is bounded by the format (2 MiB); no allocation depends on a data size.


<!-- https://recursiveai.org/onearchive/docs/zstd-encoder/ -->

# Zstandard encoder

What the encoder does where the format leaves a choice. Compressed bytes depend only on the input and the options, so a later
port that follows this text and the format (RFC 8878) writes the same bytes. Source: `src/OneCompress/Zstd/`.

## Layers

| tier | types |
|---|---|
| 1, one call | `Zstd.Compress`, `Zstd.TryCompress`, `Zstd.GetMaxCompressedLength` |
| 2, encoder and stream | `ZstdEncoder` (`CompressFrame`, `Compress` with `isFinalBlock`, `Flush`, `Reset`), `ZstdStream` in write mode, `ZstdCompressionOptions`, `ZstdCompressionDictionary` |
| 3, raw parts | `FseEncodingTable` (normalize, build, describe, price), `HuffmanEncodingTable` (length-limited tree, description, 1 and 4 streams), `ZstdBlockWriter` (literals and sequences section writer, table-mode choice, repeat offsets), `ZstdFrameWriter` (header, checksum, skippable frames), `ZstdMatchFinder` (the seam every strategy plugs into), `ZstdSequenceStore`, `ZstdSequence`, `ZstdRepeatOffsets`, `ZstdCompressionParameters` |

A match finder turns a window into sequences with **real offsets**; the block writer decides which of them are written as
repeat codes (RFC 8878 section 3.1.1.5), updates the repeat offsets, and chooses the entropy modes. A finder is given the offsets
in force at the start of each block so it can look for cheap matches. A block stored raw or as RLE leaves the offsets and the
carried tables untouched, as the decoder sees it. A frame writer can take blocks from any source between a header and a checksum.

## Levels

Level numbers and meanings follow the `zstd` command line tool. Level 0 means 3. Levels 1 to 22 and every negative level down to
-131072 are implemented; the negative level is the acceleration (the fast strategy's target length). Higher levels throw
`ArgumentOutOfRangeException`. Levels 20 to 22 are the tool's ultra levels: their windows reach 128 MiB (window log 27), so a decoder
has to allow that window (`ZstdDecompressionOptions.MaxWindowSize` defaults to 8 MiB; `zstd -d` allows 128 MiB without a flag).

Parameters are looked up by the source size class, then shrunk to what the source can use. The size used is the content size plus
the dictionary's content size; it is unknown for a streaming encoder that was not told the content size, and then the largest class
applies.

| class (size) | level | window log | chain log | hash log | search log | min match | target length | strategy |
|---|---|---|---|---|---|---|---|---|
| > 256 KiB or unknown | negative | 19 | 12 | 13 | 1 | 6 | acceleration | fast |
|  | 1 | 19 | 13 | 14 | 1 | 7 | 0 | fast |
|  | 2 | 20 | 15 | 16 | 1 | 6 | 0 | fast |
|  | 3 | 21 | 16 | 17 | 1 | 5 | 0 | double fast |
|  | 4 | 21 | 18 | 18 | 1 | 5 | 0 | double fast |
|  | 5 | 21 | 18 | 19 | 3 | 5 | 2 | greedy |
|  | 6 | 21 | 18 | 19 | 3 | 5 | 4 | lazy |
|  | 7 | 21 | 19 | 20 | 4 | 5 | 8 | lazy |
|  | 8 | 21 | 19 | 20 | 4 | 5 | 16 | lazy2 |
|  | 9 | 22 | 20 | 21 | 4 | 5 | 16 | lazy2 |
|  | 10 | 22 | 21 | 22 | 4 | 5 | 16 | lazy2 |
|  | 11 | 22 | 21 | 22 | 5 | 5 | 16 | lazy2 |
|  | 12 | 22 | 22 | 23 | 5 | 5 | 32 | lazy2 |
|  | 13 | 22 | 22 | 22 | 4 | 5 | 32 | btlazy2 |
|  | 14 | 22 | 22 | 23 | 5 | 5 | 32 | btlazy2 |
|  | 15 | 22 | 23 | 23 | 6 | 5 | 32 | btlazy2 |
|  | 16 | 22 | 22 | 22 | 5 | 5 | 48 | btopt |
|  | 17 | 23 | 23 | 22 | 5 | 4 | 64 | btopt |
|  | 18 | 23 | 23 | 22 | 6 | 3 | 64 | btultra |
|  | 19 | 23 | 24 | 22 | 7 | 3 | 256 | btultra2 |
|  | 20 | 25 | 25 | 23 | 7 | 3 | 256 | btultra2 |
|  | 21 | 26 | 26 | 24 | 7 | 3 | 512 | btultra2 |
|  | 22 | 27 | 27 | 25 | 9 | 3 | 999 | btultra2 |
| <= 256 KiB | negative | 18 | 12 | 13 | 1 | 5 | acceleration | fast |
|  | 1 | 18 | 13 | 14 | 1 | 6 | 0 | fast |
|  | 2 | 18 | 14 | 14 | 1 | 5 | 0 | double fast |
|  | 3 | 18 | 16 | 16 | 1 | 4 | 0 | double fast |
|  | 4 | 18 | 16 | 17 | 3 | 5 | 2 | greedy |
|  | 5 | 18 | 17 | 18 | 5 | 5 | 2 | greedy |
|  | 6 | 18 | 18 | 19 | 3 | 5 | 4 | lazy |
|  | 7 | 18 | 18 | 19 | 4 | 4 | 4 | lazy |
|  | 8 | 18 | 18 | 19 | 4 | 4 | 8 | lazy2 |
|  | 9 | 18 | 18 | 19 | 5 | 4 | 8 | lazy2 |
|  | 10 | 18 | 18 | 19 | 6 | 4 | 8 | lazy2 |
|  | 11 | 18 | 18 | 19 | 5 | 4 | 12 | btlazy2 |
|  | 12 | 18 | 19 | 19 | 7 | 4 | 12 | btlazy2 |
|  | 13 | 18 | 18 | 19 | 4 | 4 | 16 | btopt |
|  | 14 | 18 | 18 | 19 | 4 | 3 | 32 | btopt |
|  | 15 | 18 | 18 | 19 | 6 | 3 | 128 | btopt |
|  | 16 | 18 | 19 | 19 | 6 | 3 | 128 | btultra |
|  | 17 | 18 | 19 | 19 | 8 | 3 | 256 | btultra |
|  | 18 | 18 | 19 | 19 | 6 | 3 | 128 | btultra2 |
|  | 19 | 18 | 19 | 19 | 8 | 3 | 256 | btultra2 |
|  | 20 | 18 | 19 | 19 | 10 | 3 | 512 | btultra2 |
|  | 21 | 18 | 19 | 19 | 12 | 3 | 512 | btultra2 |
|  | 22 | 18 | 19 | 19 | 13 | 3 | 999 | btultra2 |
| <= 128 KiB | negative | 17 | 12 | 12 | 1 | 5 | acceleration | fast |
|  | 1 | 17 | 12 | 13 | 1 | 6 | 0 | fast |
|  | 2 | 17 | 13 | 15 | 1 | 5 | 0 | fast |
|  | 3 | 17 | 15 | 16 | 2 | 5 | 0 | double fast |
|  | 4 | 17 | 17 | 17 | 2 | 4 | 0 | double fast |
|  | 5 | 17 | 16 | 17 | 3 | 4 | 2 | greedy |
|  | 6 | 17 | 16 | 17 | 3 | 4 | 4 | lazy |
|  | 7 | 17 | 16 | 17 | 3 | 4 | 8 | lazy2 |
|  | 8 | 17 | 16 | 17 | 4 | 4 | 8 | lazy2 |
|  | 9 | 17 | 16 | 17 | 5 | 4 | 8 | lazy2 |
|  | 10 | 17 | 16 | 17 | 6 | 4 | 8 | lazy2 |
|  | 11 | 17 | 17 | 17 | 5 | 4 | 8 | btlazy2 |
|  | 12 | 17 | 18 | 17 | 7 | 4 | 12 | btlazy2 |
|  | 13 | 17 | 18 | 17 | 3 | 4 | 12 | btopt |
|  | 14 | 17 | 18 | 17 | 4 | 3 | 32 | btopt |
|  | 15 | 17 | 18 | 17 | 6 | 3 | 256 | btopt |
|  | 16 | 17 | 18 | 17 | 6 | 3 | 128 | btultra |
|  | 17 | 17 | 18 | 17 | 8 | 3 | 256 | btultra |
|  | 18 | 17 | 18 | 17 | 10 | 3 | 512 | btultra |
|  | 19 | 17 | 18 | 17 | 5 | 3 | 256 | btultra2 |
|  | 20 | 17 | 18 | 17 | 7 | 3 | 512 | btultra2 |
|  | 21 | 17 | 18 | 17 | 9 | 3 | 512 | btultra2 |
|  | 22 | 17 | 18 | 17 | 11 | 3 | 999 | btultra2 |
| <= 16 KiB | negative | 14 | 12 | 13 | 1 | 5 | acceleration | fast |
|  | 1 | 14 | 14 | 15 | 1 | 5 | 0 | fast |
|  | 2 | 14 | 14 | 15 | 1 | 4 | 0 | fast |
|  | 3 | 14 | 14 | 15 | 2 | 4 | 0 | double fast |
|  | 4 | 14 | 14 | 14 | 4 | 4 | 2 | greedy |
|  | 5 | 14 | 14 | 14 | 3 | 4 | 4 | lazy |
|  | 6 | 14 | 14 | 14 | 4 | 4 | 8 | lazy2 |
|  | 7 | 14 | 14 | 14 | 6 | 4 | 8 | lazy2 |
|  | 8 | 14 | 14 | 14 | 8 | 4 | 8 | lazy2 |
|  | 9 | 14 | 15 | 14 | 5 | 4 | 8 | btlazy2 |
|  | 10 | 14 | 15 | 14 | 9 | 4 | 8 | btlazy2 |
|  | 11 | 14 | 15 | 14 | 3 | 4 | 12 | btopt |
|  | 12 | 14 | 15 | 14 | 4 | 3 | 24 | btopt |
|  | 13 | 14 | 15 | 14 | 5 | 3 | 32 | btultra |
|  | 14 | 14 | 15 | 15 | 6 | 3 | 64 | btultra |
|  | 15 | 14 | 15 | 15 | 7 | 3 | 256 | btultra |
|  | 16 | 14 | 15 | 15 | 5 | 3 | 48 | btultra2 |
|  | 17 | 14 | 15 | 15 | 6 | 3 | 128 | btultra2 |
|  | 18 | 14 | 15 | 15 | 7 | 3 | 256 | btultra2 |
|  | 19 | 14 | 15 | 15 | 8 | 3 | 256 | btultra2 |
|  | 20 | 14 | 15 | 15 | 8 | 3 | 512 | btultra2 |
|  | 21 | 14 | 15 | 15 | 9 | 3 | 512 | btultra2 |
|  | 22 | 14 | 15 | 15 | 10 | 3 | 999 | btultra2 |

**Levels 10 to 12 above 256 KiB search one step less deep than the tool.** The search log is 4, 5 and 5 where the tool's is 5, 6 and 6
(every other value is the tool's). The row search spends most of its time on candidates that fail the longer-match check, so its time
is close to proportional to the candidates visited. On silesia.tar the sizes are +0.55%, +0.32% and +0.32% against the tool at levels
10, 11 and 12, and on enwik8 -0.2% to +0.2%. Level 10 now has 16-entry rows, which take the insert stride (see Match finders). Sizes
fall and times rise from level 10 to 12 (11.66, 11.58 and 11.57 MB on the first 32 MiB of silesia.tar).

The strategies above btlazy2 (btopt, btultra, btultra2) take levels 13 to 15 of the 256 KiB and 128 KiB classes, 11 to 15 of the 16 KiB class
and levels 16 to 22 of every class; they are the optimal parsers described below.

The values were read off the reference tool's own report (`zstd --show-default-cparams`) for sources of the class boundary sizes;
they are measured behaviour, not copied code. Shrinking: `windowLog = min(table, max(10, ceil(log2(size))))` when the size is
known, `hashLog = min(hashLog, windowLog + 1)`, `chainLog = min(chainLog, windowLog)` (`windowLog + 1` for the binary-tree strategies, whose table holds two links per position). For negative levels the target length
(acceleration) is `-level`. `ZstdCompressionOptions.WindowLog` replaces the window log (and tightens the other two logs).

**Literals of negative levels are stored raw** (the Huffman pass costs more time than it is worth at those speeds). This is part
of the level definition, `ZstdCompressionParameters.CompressLiterals`.

## Match finders (greedy, lazy, lazy2, btlazy2)

One finder, `ZstdLazyMatchFinder`, implements the four strategies. They differ in the search structure (rows of tagged entries or a binary tree)
and in how far the parse looks ahead.

- **Index:** every position with eight bytes after it inside the block is entered, in order, by the search that reaches it (positions
  skipped over by a match or a step are entered first). The hash is the fast finder's: the low `minMatch` bytes of an 8-byte read,
  multiplied by `0x9E3779B185EBCA87`, top `hashLog` bits. Tables hold window positions as `uint` with 0 for empty; the first byte of a
  window is never a candidate. A stream that slides its window rebases positions and, for the binary tree (indexed by position
  modulo its size), turns it around by the slide distance. Blocks continue each other: the last seven positions of a block are entered
  by the first search of the next one.
- **Rows** (greedy, lazy, lazy2; `chainLog` is not used by the frame's own table): the table holds `2^hashLog` entries in rows of `2^rowLog` entries, `rowLog = clamp(searchLog, 4, 6)`
  (16, 32 or 64), `2^(hashLog - rowLog)` rows (at least 2). The 64-bit hash picks the row with its top bits and a tag byte with the
  next 8. Each row is a ring: a new entry (tag and position) goes just below the row's head, which then moves to it, so walking
  from the head visits entries newest first. A search takes the entries whose tag equals its own, newest first, and stops at the first
  one that is empty or farther back than the window size; it visits at most `min(2^searchLog, 2^rowLog)` of them. Only the tags are
  compared before the match is looked at. A visited candidate is skipped without measuring when its four bytes ending at the current
  best length differ from the same bytes at the current position (a longer match must agree on them); otherwise the match length
  is measured and the candidate becomes the best if its gain is higher. With the current best shorter than `minMatch - 1` the
  skip test looks at the single byte at index `minMatch - 1` instead (`minMatch` 3 only).
- **Insert stride** (rows with `rowLog` 4, no dictionary, `minMatch` at least 4, `hashLog` at least 19): positions skipped over by a match are
  entered into their rows lazily by the next search (see Index). When more than 16 positions lie between the last one entered and the position
  being searched, they are entered every 4th position (the first of them, then every fourth), and the last 16 are all entered. A long match
  usually repeats text whose positions are already in the table, and each entered position is a write to a row far from the cache. Entering
  every fourth loses almost no match, because the parse extends a match backwards over equal bytes. This is a rule of the output bytes: it
  applies at levels 5 to 9 for sources over 256 KiB (`hashLog` 19 to 21) and at levels 6 to 8 for sources of 128 KiB to 256 KiB when their
  `hashLog` reaches 19. On silesia.tar it costs 0.04% to 0.17% in size.
- **Binary tree** (btlazy2): `2^(chainLog-1)` nodes, two links each (smaller and larger), one tree per hash value rooted at
  `head[hash]`. Entering a position compares it against nodes from the root down, splitting them into those below and those above
  it (the standard suffix-tree-like insertion of binary-tree match finders); the comparisons give every match met, in increasing
  length, with at most `2^searchLog` nodes visited. Comparison length is capped at 256 (the nice length): a node that matches that
  far is replaced by the new one, which takes its children. A node entered with fewer than 256 bytes left in its block was ordered
  by fewer bytes, so the shortcut that starts a comparison at the shorter of the two bounding prefix lengths is not used on it, and
  the bounds are forgotten after it. The search then extends the best match past the cap.
- **Gain:** a match of length `L` at offset `d` has gain `4 L - floor(log2 d)`; a repeat-offset match has `4 L - 1`. The search keeps
  the candidate of highest gain (earliest on ties). Repeat offsets are tried before the table: with a literal in front, the three
  repeat offsets; with none, the second, the third and the first minus one (the cases RFC 8878 section 3.1.1.5 gives codes 1 to 3 to).
  A repeat match needs 4 equal bytes.
- **Parse:** greedy takes the best match at the current position, after also trying the first repeat offset one byte further on, which
  wins if its gain is higher. Lazy (lazy2) takes the best match at `ip`, then looks at `ip + 1` (`ip + 2`): a later match replaces
  it when its gain is higher by more than 4 (7 for two positions on), and the check restarts from the new position. A match of 128
  bytes or more is not looked past. The match is extended backwards over equal bytes down to the last match end, emitted, and the
  search resumes at its end. With nothing found the position advances by `1 + (ip - anchor) / 256`.
- **With a dictionary:** the frame's own candidates are visited first; a search that has attempts left continues along the
  dictionary's hash chains (built once per dictionary and parameter set, for every position with 8 bytes after it, sized by
  `DictionaryHashLog` and `DictionaryChainLog` when set). The dictionary's candidates count only while the bytes produced do not
  exceed the window size, and the binary-tree strategy searches the dictionary through chains, not a tree. Repeat offsets that
  reach before the frame's first byte are tried against the dictionary content.

## Optimal parsing (btopt, btultra, btultra2)

`ZstdOptimalMatchFinder` implements the three strategies. They differ only in the parameters of their rows (search depth, target length,
minimum match) and in two details: btopt skips searching a position whose successor is already reachable at nearly the same price, and
btultra2 parses the first block of a frame twice.

- **Tree:** the binary tree of btlazy2 (one tree per hash of `minMatch` bytes, `2^(chainLog-1)` nodes, every position entered once, in order),
  compared up to 1024 bytes (the tree's "long enough"), at most `2^searchLog` nodes per search. The matches met on the way down, each longer
  than the one before, are the candidates; with a dictionary, the dictionary's hash chains add candidates longer than the tree's best.
  The chain walk has its own budget of 16 times the level's attempts (the frame's tree does not use it up): the dictionary has no tree,
  and its best match is often far down a chain. The three repeat offsets (with a literal in front; with none, the second, the third and the first minus
  one, RFC 8878 section 3.1.1.5) are candidates too, from four bytes (three when `minMatch` is 3). A candidate that reaches the
  target length ends the stretch at once: the longest candidate is measured to its end and taken.
- **Prices:** in 1/256 bit. A symbol costs `log2(total) - log2(count)` (counts below 1 are 1; `log2` is the integer table of `ZstdCost`), plus
  its extra bits. Four count sets: literals (256), literal length codes (36), match length codes (53) and offset codes (32; the offset
  value's base-2 logarithm, so the repeat codes 1 to 3 are codes 0 and 1). Counts of the first block start from the block's literal
  histogram and the predefined distributions (a distribution entry of -1 counts 1); with a dictionary in the Zstandard format they start from
  the dictionary's tables instead, since the first block can be coded with them: a literal counts `2^(maxBits - codeLength)` (1 if it has no code)
  and a sequence symbol its normalized count (1 for -1 and 0). Prices are rebuilt from the statistics alone at the start of each
  block, and again every 4096 bytes parsed (half the statistics at the block start plus the counts of this block so far). When a block
  finishes, its counts are added to half the old statistics. A match of length 3 costs two bits more than its symbols.
- **Path search:** from the start of a stretch (the first position with a candidate, with the literals since the last match pending) a
  cell per following position records the cheapest known price of reaching it, whether a literal or a match got there, the match's length,
  offset, starting cell and literals before it, and the repeat offsets after that path. The price of a cell includes the code of the literal
  run pending at it, so a literal extends a cell by its own price plus the difference between the run's code prices, and a match adds its
  offset, length and (for the next run) zero-length literal code prices. Cell by cell: the literal transition is applied (ties go to the
  literal), the cell's repeat offsets are derived from its path, candidates at the position are found (using that path's repeat offsets)
  and every length from the candidate's first new length to its length relaxes the cell it reaches. Offsets that equal a repeat offset
  of the path are priced as repeat codes (the same resolution the block writer uses). The stretch ends when the cell reached is the
  farthest any match got to; the cheapest path there is written out and the literals at its end stay pending, so a stretch is never forced
  to end with a match that costs more than literals. A stretch is at most 4096 positions; a candidate that would pass that, or that reaches
  the target length, is taken as the stretch's last match.
- **btultra2:** the first block of a frame is parsed once with the starting prices, the counts of that parse become the statistics, the
  tree is emptied, and the block is parsed again. (Blocks after the first use the statistics of the blocks before them.)
- **Window sliding:** as for btlazy2: stored positions are rebased, the tree is turned around by the distance moved.

## Block splitting

Blocks of the optimal strategies are cut again after parsing. `ZstdBlockSplitter` groups the block's sequences into chunks of about
4096 input bytes and estimates the cost of coding a run of chunks as one block: the entropy of its literals, literal length codes, match
length codes and offset codes (integer, from the counts), plus the extra bits, plus an allowance for the headers and table descriptions of
`12 + distinct literals / 2 + 3/4 * (distinct codes of the three sequence alphabets)` bytes (a literal alphabet is not counted when raw
literals are cheaper). A shortest-path search over the chunk boundaries (at most 32 blocks per 128 KiB) picks the cuts of least total
cost. Each part is written as a block of its own with its own tables; repeat offsets run on from part to part exactly as the decoder
sees them. Because every part may cost three header bytes more than the single block would have, the bound of `GetMaxCompressedLength`
allows 32 parts per 128 KiB.

## Frames

- Content size is written when it is known (one-shot calls, or `ContentSize` for a stream) unless `WriteContentSize` is false.
- A frame with a known content size no larger than the window is **single-segment**: no window descriptor, the content size is the
  window. Content size field: 1 byte below 256 (single-segment only), 2 bytes (value minus 256) below 65,792, 4 bytes up to 2^32-1,
  else 8.
- The dictionary ID is written when the dictionary has one and `WriteDictionaryId` is true, in the fewest of 1, 2, 4 bytes.
- The checksum (`Checksum`, default false) is the low 32 bits of XXH64 of the content.
- Empty input: header plus one empty raw last block.
- Blocks are cut every 128 KiB of input. A one-shot call and a stream with the same content size make the same bytes however the
  input is split across calls; `Flush` ends a block early (and so changes the bytes). A stream without a content size uses the
  largest class and writes no content size, whether or not its first call happened to carry the whole input.

## Blocks

- A block whose bytes are all equal (two or more) is an RLE block. Otherwise the block is compressed, and stored raw when the
  compressed body would not be smaller than the block.
- **Literals section:** fewer than 8 literals, or raw literals mode, are stored raw; one distinct byte value is RLE. Otherwise the
  smallest of raw, a new Huffman tree, and the previous tree (treeless) is written, by exact size. One stream below 256
  literals, four streams from 256. A tree description uses 4-bit weights or FSE-compressed weights, whichever is smaller; the
  compressed form is read back with the decoder's own routine and dropped for the direct form if it does not decode to the same
  weights. Codes are limited to 11 bits: the Huffman tree is built by a two-queue merge, and too-long codes are shortened by
  lengthening the least frequent of the longest codes until the Kraft sum fits, then spare code space is given back to the most
  frequent symbols.
- **Skipping work that cannot pay (small blocks):** a new Huffman tree is built only when the literals number at least 64 and an estimate of
  its size (the entropy of the literals, rounded down by the integer logarithm's error, plus `8 + 3/8` bytes per distinct symbol for the
  description) is below both the raw size and the cost of the previous tree. A new FSE table is tried only when the block has at least 64
  sequences and the entropy of its codes plus 16 bits is below the cost of the best of repeat and predefined; with neither available a
  table is always built. The estimates are integers; they decide bytes, so they are part of the format of our output.
  At the fast and double-fast strategies, a block whose previous tree can code its literals (the dictionary's, or the last block's)
  tries a new tree only from 128 literals; below that the saving never repaid the description in the measurements behind this rule.
- **Sequences section:** for each of literal lengths, offsets and match lengths the cheapest of RLE (one symbol), repeat (the
  previous table from the frame or the dictionary), predefined and a new FSE table is chosen. Cost is in 1/256 bit, integer:
  blocks with at most 1,500 sequences price each choice by running the encoder's state machine over the symbols (the table
  description's bytes included); larger blocks, and every block at the fast and double-fast strategies (levels up to 4 and the
  negative levels), use the histogram estimate `count * (accuracyLog - log2(probability))`. `log2` is
  a table built with integer arithmetic. After an RLE or predefined block the repeat mode is not offered (the next block offers
  a new table or the predefined one at the same cost).
- **Normalization:** symbols that occur get at least one state; symbols with at most `total >> accuracyLog` occurrences get the
  "less than one" probability; the rest are scaled with rounding, then states are removed from or given to the symbols that lose
  or gain the fewest bits (ties: lower symbol first).
- **Accuracy log** of a new FSE table, from the number of symbols coded (`sampleLog = floor(log2(count - 1))`): start from
  `min(maxLog, sampleLog - 2)`, raise it to at least `min(sampleLog + 1, floor(log2(maxSymbol)) + 2)`, and clamp to 5 up to
  `maxLog`. `maxLog` is 9 for literal and match lengths, 8 for offsets, 6 for Huffman weights. These values were fitted to the
  reference tool's table sizes by black-box comparison, not derived from a specification.

## Match finders (fast, double fast)

- Hash: the low `minMatch` bytes of an 8-byte little-endian read, shifted up, multiplied by `0x9E3779B185EBCA87`, top `log` bits.
  The double-fast long table hashes 8 bytes. Tables hold window positions as `uint`; an empty slot reads as position 0, which is checked like any
  other candidate. Candidates must be within the window size; the candidate is accepted when its first 4 bytes (double fast long: 8)
  equal the current bytes.
- **Fast:** each round hashes two neighbouring positions, `ip` and `ip + 1`, writes both into the table, and tests (in this order)
  the first repeat offset at `ip + step`, the candidate of `ip`, the candidate of `ip + 1`. With no match, `ip += step`. The
  step starts at `targetLength + 1` (2 when the target length is 0) and grows by one each 128 bytes since the last match. A match
  is extended backward over equal bytes (not before the last match end) and forward. After a match two positions are indexed (the
  match start + 2 and the position 2 before its end) and the second repeat offset is tried repeatedly at the end of each match.
- **Double fast:** one position per round, `ip += 1 + (ip - anchor) / 256`. Tests the first repeat offset at `ip + 1`, the long
  candidate, the short candidate (then also the long candidate at `ip + 1`, which wins if it matches 8 bytes). Both tables are
  filled at the same positions after a match.
- **With a dictionary:** the dictionary content sits logically right before the first byte. A match into it has offset
  `bytes produced + dictionary length - position in dictionary`, and is allowed while the bytes produced so far do not exceed
  the window size (RFC 8878 section 3.1.1.1.2). Repeat offsets that reach before the frame start are tried against the
  dictionary. A match in the dictionary may run on into the start of the frame. The dictionary's search tables are built once per
  (minimum match, hash log, chain log, strategy) with an entry for every position (a later position replaces an earlier one with
  the same hash). With a dictionary and the fast strategy, one position is tried per round (repeat at `ip + 1`, own table, dictionary table).
- **Tables of a small source with a dictionary:** the window log, minimum match and strategy come from the size class of source plus
  dictionary, but the hash and chain tables a frame fills are sized for the source alone (`min(table, sourceWindowLog + 1)`, and
  `sourceWindowLog` for the chain), so a 100-byte payload does not clear 100 KiB of table. The dictionary's own search index keeps the
  sizes its length calls for (`DictionaryHashLog`, `DictionaryChainLog` in the parameters).
- Window sliding (streams): the window buffer keeps the last `windowSize + 1` bytes; stored positions are rebased by the
  amount moved and those that fall below become 0. Matches never reach past the window size, so a stream and a one-shot call
  find the same matches.

## Dictionaries

`ZstdCompressionDictionary.Parse` accepts the Zstandard dictionary format (entropy tables read back into encoder tables, repeat
offsets, content) and, for anything else, raw content (8 bytes or more; ID 0 unless given). The Huffman tree, the three FSE tables
and the repeat offsets start the frame's carried state, so the first block can use repeat modes and treeless literals.

A dictionary builds search tables over its content for each set of match-finder parameters it is used with (level, size class and
window change them). It keeps at most 128 MiB of such tables; the least recently used sets are dropped and rebuilt on demand, which
costs time but never changes the output. A long-lived dictionary used at many settings therefore stays bounded in memory.

The fast and double-fast strategies search a dictionary through its index (every content position, the latest of each hash). For
content under 16 MiB each index entry holds the position in its low 24 bits and, in its top 8 bits, a tag: the top byte of
`bytes * 0x9E3779B1` (32-bit, wrapping) for the four bytes at that position. A search compares the tag of the four bytes at the
current position before it reads the dictionary content, so a slot that holds other bytes costs no second memory access. A tag
is a function of the four bytes a match must share, so it never rejects a match the untagged search would take: the output is the
same with or without it. Content of 16 MiB and more has untagged entries.

## Windows

`WindowLog` (10 to 31) replaces the level's window log. The header states it; matches never reach back further than 2^30 bytes, which is
the most a window buffer can hold, so 31 is a declaration only (`zstd -d --long=31` or `--memory` is needed to decode it). A log below
17 asked for explicitly is written as 17 in the header, because a block (128 KiB) may not exceed the window (RFC 8878 section 3.1.1.2.3);
matching keeps to the smaller window. A frame whose content size is known and fits the window is single-segment and needs no window
descriptor.

What a decoder accepts: this library's `ZstdDecompressionOptions.MaxWindowSize` defaults to 8 MiB and is raised by the caller (it
plays the part of `zstd -d --long=N` and `--memory`); it holds up to two windows, so it refuses windows of 2^30 and above. The `zstd`
command line decodes up to 2^27 by default.

## Memory

What an encoder holds depends on the parameters and on whether it is told the size.

- **Known size** (one call, or `ContentSize`): the window and tables are sized to the source as the parameter table above says, and
  allocated whole when the frame starts.
- **Unknown size** (a stream without `ContentSize`): the parameters are those of the largest size class, as `zstd` itself uses for a
  pipe, so the output is the same as before. What the encoder allocates is not: the window buffer and the binary-tree link table of
  btopt, btultra and btultra2 start small (1 MiB; 65,536 positions) and are enlarged as data arrives, to four times their size, or to
  the whole once a quarter of it is needed. A node of the link table lives at the slot of its position modulo the full table size;
  below the current size that slot is the position itself, so enlarging moves nothing and cannot change a match. Where the window
  slides depends on the full capacity, never on how much is allocated. The hash tables are indexed by hash and are allocated whole
  (up to 128 MiB at level 22), as are the chain tables of the other strategies (up to 16 MiB at level 12).
- **`SizeHint`**: for a stream whose length is only roughly known. The window and tables are chosen as for a source of that size and
  allocated whole; the header states no content size, and its window log is at least 17 (a block may not exceed the window,
  RFC 8878 section 3.1.1.2.3) while matching keeps to the smaller window. Input beyond the hint is still compressed correctly, with
  that smaller window. `ContentSize`, when given, wins.
- **Dictionary search index**: the index a dictionary builds for a parameter set holds one entry per content position, so its tables
  are capped by the content, not by the source: the chain index at `ceil(log2(content length))` (at least 10), which keeps every
  position and so cannot change a match, and the hash tables at four bits more (sixteen slots per position). A 4 KiB dictionary at
  level 22 used to build 256 MiB of index for a large source and now builds under 1 MiB.
- Sliding the tree-link table is done in place (reverse the two parts, then the whole) rather than through a copy of the table.

Peak memory (measured when these allocation changes were made), one process per row, managed bytes allocated and peak working set in MiB, before and after, for generated text
(`GC.GetTotalAllocatedBytes` and `Process.PeakWorkingSet64`; levels 1 and 3 did not change). Each cell is the no-dictionary row / the
row with a 4 KiB dictionary.

| level, input | one call | stream, no size | stream, size known |
|---|---|---|---|
| 9, 16 MiB | 30/73 > 30/73 / 42/77 > 30/74 | 24/80 > 25/82 / 36/85 > 26/83 | 24/81 > 24/81 / 36/85 > 25/82 |
| 19, 1 KiB | 0/39 > 0/39 / 1/40 > 1/40 | 80/119 > 17/56 / 160/122 > 17/57 | 0/39 > 0/39 / 1/40 > 1/40 |
| 19, 64 KiB | 2/41 > 2/41 / 3/43 > 2/43 | 81/120 > 17/57 / 161/124 > 18/58 | 2/41 > 2/41 / 3/43 > 2/42 |
| 19, 16 MiB | 99/142 > 99/142 / 180/146 > 100/144 | 161/217 > 145/194 / 242/221 > 145/195 | 161/217 > 97/153 / 242/221 > 98/155 |
| 22, 1 KiB | 0/39 > 0/39 / 1/40 > 1/40 | 384/423 > 129/168 / 640/427 > 129/169 | 0/39 > 0/39 / 1/40 > 1/40 |
| 22, 64 KiB | 2/41 > 2/41 / 3/43 > 2/43 | 385/425 > 129/169 / 641/428 > 130/170 | 2/41 > 2/41 / 3/43 > 2/42 |
| 22, 16 MiB | 275/318 > 275/318 / 532/322 > 276/320 | 521/465 > 393/401 / 778/469 > 393/402 | 281/337 > 281/337 / 554/341 > 298/339 |

The compressed bytes of all 90 rows are unchanged.

## Long-distance matching

`LongDistanceMatching` is `zstd --long`: with it on and no `WindowLog`, the window is 2^27 (smaller when the content size is known to be
smaller: the log of the size, at least 10). A second matcher finds long repeats anywhere in the window; the level's own match finder
parses the rest, whatever its strategy. The matcher is `ZstdLongDistanceMatcher` and the glue is `ZstdLongSequencer`, both public.

- **Anchoring.** A gear hash runs over every byte: `h = (h << 1) + G[byte]`, with 256 constants from the SplitMix64 sequence seeded with
  `0x2545F4914F6CDD1D` (each step adds `0x9E3779B97F4A7C15`, then `z = (z ^ z >> 30) * 0xBF58476D1CE4E5B9`, `z = (z ^ z >> 27) *
  0x94D049BB133111EB`, `G = z ^ z >> 31`). The hash after byte `j` covers bytes `j-63` to `j`; position `i = j - 63` is a checkpoint when the
  top `HashRateLog` bits of the hash are zero (content-defined sampling, so both copies of a repeat are anchored at the same place).
- **Table.** `2^HashLog` entries of 8 bytes in buckets of `2^BucketSizeLog`: `check << 32 | (position + 1)`, newest first, the oldest falls
  out. With `mixed = (h ^ h >> 29) * 0xBF58476D1CE4E5B9`, the bucket is the top `HashLog - BucketSizeLog` bits of `mixed` and the check
  is its low 32 bits. Defaults: `HashLog = clamp(min(windowLog, sizeLog) - 7, 6, 24)`, `BucketSizeLog = 2`,
  `HashRateLog = clamp(min(windowLog, sizeLog) - HashLog, 1, 24)`, `MinMatch = 64` (16 to 4096), where `sizeLog` is the log of the content
  size when it is known.
- **Search.** At a checkpoint at or after the end of the previous long match and the block start, each bucket entry with the same check
  and a distance of at most the window is verified byte by byte, extended forward to the block end and backward to the previous match end
  or block start; the longest wins, the newest on a tie, and it is taken when at least `MinMatch` long. Then the checkpoint is stored.
  Positions scanned inside a match are stored but not searched from. The dictionary content is not indexed.
- **Blocks.** Long matches are found per 128 KiB block (matches stay inside a block). In a block, each long match becomes a sequence; the
  match finder parses the stretches between them, told the repeat offsets that follow from what came before, and the last 256 bytes of a
  long match are added to its tables so that what follows can refer back into it. Hash tables hold position and check only, never
  pointers, so the matches do not depend on the platform.
- Sliding the stream's window buffer drops entries older than the window and changes nothing else: a stream and a one-shot call make the
  same matches.

## Jobs and threads

`JobSize` (bytes, 0 = off) cuts the input into jobs; `Workers` (0 = one per processor, default 1) is how many are compressed at once.
The bytes depend on the job size, the overlap and the other options, never on `Workers`, the processor count, the platform or the order
in which jobs finish; a browser runs the jobs one after another. With `JobSize` 0 there is one job and `Workers` is ignored.

- **Job size.** `AutoJobSize` (-1) is `max(1 MiB, 4 * min(windowSize, 4 MiB))`. Any size is rounded up to a multiple of 128 KiB (so the
  jobs add no blocks of their own and `GetMaxCompressedLength` holds) and capped at 1 GiB. Job `k` covers input bytes `k * size` to
  `(k + 1) * size`. An input no longer than one job is compressed exactly as without jobs.
- **Overlap.** The bytes before a job are history for matching. `OverlapSize` (default `min(windowSize / 8, 1 MiB)`, at most the window)
  of them are added to the match finder's tables when the job starts (`ZstdMatchFinder.Prime`: every position of the overlap, hashed as
  during parsing); with long-distance matching the whole window before the job can be matched into, found by the one long-distance
  matcher, which sees the whole input in order whichever thread runs which job (jobs take their turn at it in job order, then
  compress in parallel).
- **Job frames.** The header and checksum are written once. The jobs' blocks follow one another; every job writes its own blocks from
  fresh tables: its first block uses no table of the previous job and no repeat offset code until it has coded three offsets of its own
  (the decoder's offsets come from the end of the previous job; `ZstdBlockWriter.ForgetRepeatOffsets` makes the writer assume three
  offsets above any real one). Only the job that starts the frame uses the dictionary. The last block of the last job is the frame's
  last block.
- **Streaming.** `ZstdEncoder.Compress` buffers the window (or overlap) behind the next batch of `Workers` jobs and compresses a batch when
  more input than a batch has arrived, or at the end; the bytes do not depend on how the input is split across calls. Memory is
  the window plus `Workers * JobSize` plus the batch's output. `Flush` compresses what is buffered as a short job and ends it (and so
  changes the bytes). Each new job loads its overlap into the match finder again, so flushing after every few bytes costs time that
  grows with the square of the stream length (about 3.7 s for 80 KB written one byte per flush). Writers that flush often should leave
  jobs off.

## Measuring

- Ratio: compressed bytes of the whole file in one frame, no checksum, against `zstd -N --no-check` (negative levels
  `--fast=N`), on the same machine.
- Speed: `zstd -b<level>` in-memory modes against `tools/zstd-encode-bench` (`big`): both warm, memory to memory, single thread,
  best of several seconds. Decompression compares our decoder on our frame with the tool decoding its own frame.
- Peak memory: `/usr/bin/time -v` on both sides.
- Game snapshot data: `tools/zstd-encode-bench/game.sh` trains with `zstd --train` on seed 1 and compresses seed 2 one frame per
  payload.

## Results (zstd 1.5.7, one thread, memory to memory)

The tables in this section and the next are dated measurements; each says which build or levels it covers. They are not kept current level by level. The size bar (+1%) is met except where a table says otherwise; the known misses are enwik8 level 22 (+1.16%) and the long-distance and dictionary rows noted below.

Method as above (`tools/zstd-encode-bench/run.sh`, `game.sh --timing`); `time ratio` is our time divided by the tool's, so the bar is 1.15.
Sizes are identical on both machines (the output does not depend on the processor). x64: Ryzen 9 7950X3D, 32 cores, .NET 10.0.402, load 10 to 12
from other work on the machine; arm64: Apple M2, 8 cores, load 2 to 4.

| file, level | size vs tool | compress time ratio x64 | arm64 |
|---|---|---|---|
| silesia.tar -5 | -3.86% | 1.38 | 1.32 |
| silesia.tar -1 | +0.20% | 1.43 | 1.33 |
| silesia.tar 1 | +0.45% | 1.52 | 1.38 |
| silesia.tar 2 | +0.36% | 1.46 | 1.30 |
| silesia.tar 3 | +0.93% | 1.33 | 1.26 |
| silesia.tar 4 | +0.93% | 1.49 | 1.32 |
| enwik8 -5 | -0.23% | 1.35 | 1.33 |
| enwik8 -1 | +0.36% | 1.34 | 1.36 |
| enwik8 1 | +0.63% | 1.51 | 1.44 |
| enwik8 2 | +0.41% | 1.74 | 1.33 |
| enwik8 3 | +0.67% | 1.34 | 1.28 |
| enwik8 4 | +0.69% | 1.37 | 1.33 |

These are the first full measurement of levels -5 to 4. Sizes were within the 1% bar at those levels; compression speed was not (1.26 to 1.74 times the tool's time). Levels 1 to 9 have since been retuned (faster search loops, a sparser insert at levels 5 to 9), so the time ratios and the split below are superseded; the current rows are in the README, and sizes are unchanged at levels -5 to 4. Where the time went at level 1 on
silesia.tar (per 212 MB pass, x64): match finding about 290 ms, literals about 90 ms, sequence coding about 115 ms; the sequence
coding took twice that before its loop was split out and its state kept in registers. Dynamic profile guidance in
the JIT matters: without it the search loop is 30% slower.

Game payloads with trained dictionaries (seed 2, one frame per payload, our size and payloads per second against the tool's; geometric
means over three games, levels 1 to 3): frames (tens of bytes) are 1.0 to 1.4 times the tool's rate on x64 and 1.0 to 1.2 on arm64 with
sizes 0.2 to 1.1% smaller; snapshots and saves (hundreds of bytes to 6 KB) run at 0.32 to 0.42 times the tool's rate with sizes
2 to 10% smaller.

The unchecked accessors (`Raw`) are worth 5 to 7% of the encoder's speed: the same code built in the `Checked` configuration (every accessor
bounds-checked) runs at 564/598 (level -5), 400/430 (level 1) and 259/273 MB/s (level 3), match finding 10% slower and block writing level.

### Block writer speed work (arm64, zstd 1.5.7; measured before the levels 1 to 9 retuning)

The block writer prices the three sequence tables side by side (their state machines are independent, so the walks overlap), keeps
the table it priced instead of building it again, and keeps the Huffman bit container in registers; the fast and double-fast
strategies choose table modes from the histogram estimate at every block size (above). Output is unchanged except for one small
block in the matrix (same size). Apple M2, `time ratio` as above (reference speed divided by ours, bar 1.15), load 3 to 6 from other
work on the machine; the whole-file rows move by less than their run-to-run spread, because the match finders dominate there:

| file | level | before | after |
|---|---|---|---|
| silesia.tar | 1 / 3 / 6 / 9 | 1.37 / 1.25 / 1.46 / 1.47 | 1.41 / 1.30 / 1.46 / 1.42 |
| enwik8 | 1 / 3 / 6 / 9 | 1.49 / 1.29 / 1.57 / 1.52 | 1.49 / 1.28 / 1.44 / 1.50 |
| silesia.tar | 12 / 16 / 19 | | 1.35 / 1.18 / 1.20 |
| enwik8 | 12 / 16 / 19 | | 1.49 / 1.05 / 0.92 |

Small payloads with a 16 or 112 KiB trained dictionary (payloads per second divided by the tool's, geometric mean of three games and
two dictionary sizes; above 1 is faster than the tool; bar 0.87):

| payloads | level | before | after |
|---|---|---|---|
| frames | 1 / 3 | 1.20 / 1.06 | 1.26 / 1.07 |
| snapshots | 3 / 9 / 19 | 0.39 / 0.97 / 0.45 | 0.48 / 1.11 / 0.50 |
| saves | 3 / 9 / 19 | 0.39 / 0.81 / 0.42 | 0.54 / 0.89 / 0.45 |

At a few hundred bytes to a few kilobytes a block costs about a hundred matches and three table decisions, and the time is spread
over the match loop (about 45 ns a match), the literal section and the sequence section; no single step is more than a third.

## Results: long-distance matching and jobs (zstd 1.5.7; measured before the levels 1 to 4 retuning)

Method as above, `run.sh --ref-opts ... --ours-opts ...` (`zstd -b<L> -e<L> -T<n> [--long=27]` against `ZstdEncodeBench big ... --workers n --job auto [--long 27]`, best of 3 s).
`time ratio` is our time over the tool's. x64: Ryzen 9 7950X3D, 32 cores, load 2 to 7 from other work; arm64: Apple M2, 8 cores, load 3 to 6.
Sizes are the same on both machines. Jobs cost 0.03% in size (silesia.tar level 1: 73,520,243 bytes as one job, 73,540,841 with jobs).

| file, level | size vs tool | x64 time ratio: 1 thread, long | 4 jobs at once, long | 8 | 16 | arm64: 1 thread, long | 4 | 8 |
|---|---|---|---|---|---|---|---|---|
| silesia.tar 1 | +0.57% | 1.44 | 1.13 | 0.67 | 0.62 | 1.38 | 1.06 | 0.85 |
| silesia.tar 3 | +1.04% | 1.28 | 1.22 | 0.88 | 0.71 | 1.31 | 1.26 | 1.16 |
| enwik8 1 | +0.68% | 1.57 | 1.83 | 1.04 | 0.93 | 1.53 | 1.65 | 1.22 |
| enwik8 3 | +0.69% | 1.44 | 1.76 | 1.19 | 1.12 | 1.39 | 1.85 | 1.46 |

The long-distance matcher alone scans 2.1 GB/s of silesia.tar (x64). On an input with far repeats (1.2 GB, three 32 MB pieces recurring
at about 96 MB with one byte in 64 KiB changed) the output is 47% smaller than the tool's at levels 1 and 3, at 1.18 and 0.90 times its time.
With jobs and no long matching the ratio of times is 1.25 to 1.5 (the single-thread gap) at 4 to 16 jobs, and size is +0.34% (level 1) and
+0.82% (level 3) on silesia.tar.

### Levels 16 to 22 (optimal parsing)

Sizes against `zstd --ultra -N` (1.5.7) on the full files, our size divided by the tool's minus one; the tool decodes every frame we write.
Level 21 is not measured on the big files. Times are from the Apple M2 (8 cores, load 2.4 to 4.2 from other work), `run.sh --seconds 1`; the
x64 timing rows wait for a quiet window.

| file, level | size vs tool | compress time ratio | decompress time ratio (our decoder, our output, against the tool's own) | peak memory ours / tool (MB) |
|---|---|---|---|---|
| silesia.tar 16 | -0.17% | 1.18 | 1.23 | 767 / 642 |
| silesia.tar 17 | -0.29% | | | |
| silesia.tar 18 | +0.41% | | | |
| silesia.tar 19 | +0.30% | 1.37 | 1.18 | 813 / 690 |
| silesia.tar 20 | +0.38% | | | |
| silesia.tar 22 | +0.29% | 1.36 | 1.32 | 1117 / 1315 |
| enwik8 16 | +0.02% | 1.22 | 1.28 | 303 / 321 |
| enwik8 17 | -0.30% | | | |
| enwik8 18 | +0.59% | | | |
| enwik8 19 | -0.01% | 1.26 | 1.14 | 337 / 369 |
| enwik8 20 | +0.50% | | | |
| enwik8 22 | +1.16% | 0.82 | 1.69 | 876 / 994 |

Compression time is over the 1.15 bar everywhere but enwik8 22 (where the size is also over the 1% bar), so levels 16 to 22 stay preview.
Outliers by file (first 4 MB of Silesia files, levels 16 and 19): mr +1.1% and +1.3%, x-ray 0.0% and +1.7%, nci +0.9% and +0.4%; the others
are within 0.4% (osdb is 1.3% smaller). x-ray at levels 18 and up is 1.7% larger than at level 17 for us but 1.2% smaller for the tool: skipping the search at a
position where a literal is almost as cheap (what level 17 does) brings it to +0.4% and costs 0.3% on text, so the other levels do not do it.
Inputs of 16 KB or less at levels 13 to 15 can be up to 1.8% larger than the tool's.

Small game payloads with a trained dictionary at levels 16 and 19 (seed 2, one frame per payload, dictionaries of 16 and 112 KiB trained on seed 1;
size against the tool's at the same level; worst cell, then the range over all 24 cells):
before the dictionary-aware starting prices and the dictionary's own chain-walk budget, L16 up to +8.7% and L19 up to +5.8%; after, L16 -1.5% to +1.2%
(worst: lanterns saves, 16 KiB dictionary) and L19 -4.8% to +0.4%. Levels 13 and 15 on the same payloads are -2.2% to +0.4%.


<!-- https://recursiveai.org/onearchive/docs/zstd-decoder/ -->

# Zstandard decoder: behavior

Normative notes for `RecursiveAI.OneCompress.Zstd`'s decoder. The format itself is RFC 8878; this page records what the
format leaves to the decoder and the choices made, so another implementation can match them.

## What is decoded

- Zstandard frames (RFC 8878 §3.1.1) and skippable frames (§3.1.2), one after another. Skippable frames are handed to
  `ZstdDecompressionOptions.SkippableFrameHandler` (variant = low four bits of the magic number, payload as a span).
  Without a handler they are skipped without being read into memory.
- Raw, RLE and compressed blocks; raw, RLE, compressed (one and four streams) and treeless literals; predefined, RLE,
  FSE-compressed and repeat sequence tables; Huffman weights stored directly or FSE-compressed; repeat offsets with the
  literals-length-zero rule.
- Window descriptors and single-segment frames; the content size field in all four widths; the dictionary ID field; the
  content checksum (low 32 bits of XXH64, seed 0), verified unless `VerifyChecksum` is false.
- Dictionaries in the Zstandard dictionary format (entropy tables and repeat offsets, content) and raw-content dictionaries.

## Choices the format leaves open

| Matter | Choice |
|---|---|
| Maximum window | `MaxWindowSize`, default 8 MiB. A larger declared window raises `CompressionLimitException` before any buffer is reserved. For single-segment frames the window is the content size. |
| Maximum output | `MaxOutputSize`, default 1 GiB, counted across frames since the last `Reset`. A frame whose declared content size exceeds what remains is refused up front; for other frames the count is checked as blocks are produced. |
| Memory of a streaming decoder | The decoder keeps a window of up to `2 x window size` (the history plus one block), grown in steps as output appears, never reserved up front from the header. Frames above 1 GiB of window therefore need `Array.MaxLength / 2`; they raise a limit exception rather than a framework `OutOfMemoryException`. A caller that gives a buffer large enough for a whole frame whose size is declared bypasses the window entirely: the output buffer is the history. |
| Declared size on one-shot calls | `Zstd.Decompress` allocates the declared size only when it is at most `1 MiB + 255 x input length`; otherwise it starts small and grows, so a tiny input cannot make it reserve the whole limit. |
| Content size mismatch | A frame that decodes to a different size than it declares is corrupt. |
| Block sizes | A block (content and decoded size) larger than `min(window size, 128 KiB)` is corrupt. |
| Offsets | A match offset must be at most the window size and at most the bytes produced so far in the frame plus the dictionary content; the dictionary is reachable only while the frame's output is within its window (RFC 8878 §5). Offset codes above 31 are rejected. |
| FSE table descriptions | Accuracy log at most 9 (literals and match lengths), 8 (offsets), 6 (Huffman weights). A description that does not complete its probabilities, or names symbols beyond the table's alphabet, is corrupt. A table of a single symbol is accepted in FSE-compressed mode. |
| Huffman | Weights up to 11; the implied last weight must complete a power of two with at least one weight-1 symbol; the code depth is at most 11. A stream that does not end exactly at its first bit is corrupt. A four-stream block with fewer than four literals whose last stream would be negative is corrupt. |
| Sequence section | A sequence count of zero must end the block; reserved mode bits must be zero; the sequence bitstream must be consumed exactly. |
| Dictionary selection | A frame naming an ID is matched against `Dictionaries`, then `Dictionary` (a raw-content dictionary with ID 0 matches any ID). A frame naming no ID uses `Dictionary`. A frame naming an unknown ID raises `CompressionDictionaryException`. |
| Dictionary validation | Formatted dictionaries: magic, ID, tables, three repeat offsets (each at least 1 and at most the content size), content. Raw content: at least 8 bytes. |
| Empty input | Decoding zero bytes is an error (a stream needs at least one frame); it raises a truncated `CompressionFormatException`. |
| Trailing bytes | Anything after a frame must be another frame or a skippable frame; otherwise the magic number is reported as wrong. |
| Skippable payloads | Buffered only when a handler is set, up to `MaxSkippableFrameSize` (default 1 MiB). |

## Streaming

`ZstdDecoder.Decompress(source, destination, out consumed, out written)` accepts any split of input and output. It returns
`Done` at the end of each frame (and each skippable frame), `DestinationTooSmall` when output is pending, and `NeedMoreData`
when it used all input mid-frame. Compressed blocks (at most 128 KiB) are decoded only once complete, from the caller's
span when it already holds the whole block and from a reused internal buffer otherwise. Raw and RLE blocks stream through.

## Hot-path safety

The decoder is written on spans. Nothing in `src/` uses `Unsafe` or `MemoryMarshal` unless this page lists it.

`src/OneCompress/Zstd/Raw.cs` is the only file that skips bounds checks: `Get`, `Ref`, `ReadUInt64`, `Copy16`, `Copy8` and
`Set`. Each states the bound its caller guarantees (a table index below the table's size, a copy inside a buffer that was
checked or kept 32 bytes short of its end). With the `Checked` configuration every one of them is an ordinary checked access,
and the tests and the fuzzer run that way.

## Sequence loop

Blocks with fewer than 4 sequences run the plain loop over the tier-3 sequence decoder, because setting up the fast loops costs
more than they save. Blocks with 4 to 63 sequences (the size of game payloads) run `RunSmall`, one function that also handles
the last sequence and the stream's start. Larger blocks run `RunFast`, with a second instance that stops its refill short of
the stream's start so that only the last sequence needs the careful path.

- The three FSE tables' rows are expanded when the block's tables are known (`FseDecodingTable.FillSequenceEntries`) and
  stored in one array, so a state indexes it directly. A row is one 64-bit value: next-state baseline (16 bits), total bits
  (8), state bits (8), extra bits (8) and the baseline value (24).
- Each sequence reads the bitstream once into a 64-bit register and cuts the six fields from it with the bit counts held
  in the rows; sequences that need more than 56 bits take a slower read.
- Literals are copied with 16-byte moves. The literals buffer keeps 32 bytes of slack for that, and raw literals up to
  4096 bytes are copied into it.
- Matches with an offset of 8 or more copy 8 or 16 bytes at a time; offsets below 8 use a pattern fill; matches that reach
  into a dictionary are copied inline when 32 bytes of dictionary follow the match, otherwise by the careful path.
- Corrupt data, a copy that would pass the safe end of the output buffer and anything else unusual goes to `RunGeneral`, which
  checks every bound and is the only code that raises errors.

## Literals

The four-stream Huffman loop keeps one register per stream with a marker bit above the data, so the bits used come from a
trailing-zero count, and it refills in batches sized for the worst case. A single-stream section of 32 or more literals that
uses a dictionary's table decodes two symbols per lookup from a pair table built on first use (published with an
`Interlocked` exchange; the dictionary is shared between threads and the table is never changed after it is published).

## Speed

Bar rows are in `docs/zstd-decoder-bench.md` (method: memory to memory, warm, best of the pass, against `zstd -b<L> -e<L>`
on the same frames; `tools/zstd-decode-bench/run.sh` and `game.sh`). With the `Checked` configuration the decoder is about
27% slower than with the unchecked accessors (silesia.tar, level 3 frames, x64: 1787 against 1315 MB/s).


<!-- https://recursiveai.org/onearchive/docs/zstd-dictionary-training/ -->

# Zstandard dictionary training

`ZstdDictionary.Train(samples, options)` turns a set of sample payloads into a dictionary in the Zstandard dictionary format
(RFC 8878 §5). The pieces are public so that they can be composed: `ZstdCoverTrainer` (content selection, parameter search, and the
whole pipeline), `ZstdDictionaryBuilder` (content and samples to a complete dictionary) and `ZstdDictionaryTrainingOptions`.

The output depends only on the samples and the options: not on the platform, vector width, runtime, or the number of threads the
parameter search uses. All scoring is integer arithmetic. This page is normative: a port must produce the same bytes from it.

## Samples

- Empty samples are dropped. The samples are laid end to end in the order given.
- If their total exceeds `MaxTrainingBytes` (default 64 MiB), every `n`-th sample is kept (sample indexes `0, n, 2n, ...`) where
  `n = ceil(total / MaxTrainingBytes)`, and what is still over the limit is cut off at the limit (the sample that crosses it is
  truncated, later ones are dropped).
- Fewer than 8 bytes in total is a `CompressionDictionaryException`.

## Content selection (COVER)

Source: Liao, Petri, Moffat and Wirth, "Effective Construction of Relative Lempel-Ziv Dictionaries", WWW 2016, §3 (local
segments chosen per epoch by the frequencies of their k-mers, covered k-mers set to zero, ℓ0.5 scoring). Parameters: `k`
(segment size), `d` (d-mer size, the paper's k-mer), `D` (the content budget).

1. **d-mers.** A d-mer is the `d` bytes starting at a position, wholly inside one sample. Its key is the little-endian value of its
   bytes (`d <= 8`), or for `d` of 9 to 16 the mix `(a * 0xC2B2AE3D27D4EB4F) xor rotl(b * 0x165667B19E3779F9, 31)` of the first and
   last eight bytes read as little-endian 64-bit values (`a`, `b`), computed modulo 2^64.
   - *Cover* gives each distinct key its own id (exact).
   - *FastCover* has `2^b` ids with `b = clamp(floor(log2(N)) + 3, 12, HashBits)` for `N` total sample bytes (a table much larger
     than the data only costs time; `b` comes from the whole sample set, whichever part of it is trained on),
     `id = (key * 0x9E3779B97F4A7C15 mod 2^64) >> (64 - b)` (collisions share an id), and only positions whose offset within their
     sample is a multiple of `Acceleration` start a d-mer.
2. **Frequency.** The frequency of an id is the number of samples that contain a d-mer with that id (a sample counts once). It is
   counted per *group* of samples (see "Held-out parts") and saturates at 65535 in each group; the frequency over a part of the
   samples is the sum over its groups, capped at 65535 (so a d-mer found in 65535 or more samples counts as exactly that many).
3. **Weight.** A frequency below 2 has weight 0 (a d-mer in one sample only cannot help any other). Otherwise `Frequency` score:
   the frequency. `SquareRoot` score: `isqrt(frequency << 16)`, the floor of the integer square root (the paper's ℓ0.5 norm up to
   the final monotone square, with 8 fractional bits). Weights are 16 bits.
4. **Epochs.** `E = clamp(ceil(D / k), 1, max(1, N / k))` where `N` is the number of bytes trained on (all the samples, or all
   but a held-out part); epoch `e` covers positions `[e * (N / E), (e + 1) * (N / E))` of those bytes laid end to end, the last
   one up to `N`.
5. **Segment.** In epoch `e` with room `R` left of the budget, the window is `w = min(k, R, epoch length)`; epochs with `w < d` are
   skipped. The score of a window placement is the sum of the weights of the *distinct* ids of the d-mers starting inside it
   (positions `s .. s + w - d`). The best-scoring placement wins; among equal scores the earliest. A best score of 0 adds nothing.
   The window is `w` bytes from the winning start. The weights of all ids of its d-mers are then set to 0 for later epochs.
6. **Order.** The chosen segments are written by increasing score, ties by increasing epoch, so the most useful content ends up
   last, nearest the data it serves and with the smallest offsets. If nothing scored, the content is the first
   `min(D, N)` bytes of the samples.

## Parameter search

With `Optimize` (the default), `ZstdCoverTrainer.Optimize` looks for the combination of `DmerSizeCandidates`, `ScoreCandidates` and
`SegmentSizeCandidates` (distinct values, `k >= d`) with the smallest total. Candidates are numbered in the order d ascending, then
score (`SquareRoot` before `Frequency`, when both are given), then k ascending; a *variant* is one pair of d and score and its
candidates differ in k. A candidate is scored by the total size of the held-out samples compressed at `ScoringLevel` by this
library's encoder with the dictionary the candidate's content makes (a raw-content dictionary: the content only, no entropy tables,
so that only the content is judged); a held-out sample is a frame of its own. **The smallest total wins, and the first of equal
totals** (lowest candidate number). Scoring runs on `MaxDegreeOfParallelism` threads (0: the processor count, at most 8); each total
is independent of the others, so the choice does not depend on scheduling.

- **`SearchMode.Exhaustive`** scores every candidate.
- **`SearchMode.Guided`** (the default) scores a subset, chosen from totals only (the same totals whatever the thread count):
  1. If there are more than 6 candidates and the samples exceed 4 MiB, only the first variant is searched (below).
  2. The *first variant* is the one with the largest d up to 8 (else the first variant). Its segment sizes `k_0 < ... < k_(m-1)`
     are scored at positions `(m-1)/6`, `3(m-1)/6` and `5(m-1)/6` (integer division) of the list.
  3. From the best candidate so far, its two neighbours in the list (previous and next k) are scored, unless already scored. If a
     neighbour beat the best by more than a quarter of a percent (`(best - new) * 400 >= best`), this repeats from it; otherwise
     it stops. The smallest total scored so far stays the best.
  4. Each other variant is scored at the segment size nearest the best one (the smaller on a tie).
  5. With 6 or fewer candidates all are scored.
- **Held-out parts.** `TrainingPercent` (default 75) gives `F = ceil(100 / (100 - TrainingPercent))` folds (4 for 75; 1 and
  train = test when 100). The samples are cut into blocks of `max(1, count / 16)` consecutive samples; block `j` belongs to *group*
  `j mod F`, and fold `f` holds out group `f` and trains on the rest. Blocks, not single samples, are held out because neighbouring
  samples (ticks of one game, pieces of one file) are alike. If a group would be empty or the other groups hold fewer than 8 bytes
  (very few samples), there is one fold that trains and is scored on everything. The training samples of a fold are a few stretches
  of the original data and are never copied; the d-mers are counted once for all folds.
- **Folds scored.** Sample sets up to 1 MiB score the first `min(F, 4)` folds and add their totals; up to 4 MiB the first
  `min(F, 2)`; larger ones the first fold only. A fold is scored on at most `MaxScoringSamples` (default 2000) of its held-out samples
  and 1 MiB of them: every n-th, where `n = max(ceil(samples / MaxScoringSamples), ceil(bytes / 1 MiB))`.

Defaults: `d` in {6, 8}; `k` in {32, 48, 64, 96, 128, 192, 256, 384, 512, 768, 1024, 1536, 2048}; the `SquareRoot` score only (26
combinations; a guided search scores about 6 to 9 of them). Without `Optimize`, `SegmentSize` (256), `DmerSize` (8) and `Score`
(`SquareRoot`) are used.

## The dictionary file

`ZstdDictionaryBuilder.Build(content, samples, id)` writes: magic `0xEC30A437`, the ID, the Huffman table for literals, the FSE
tables for offsets, match lengths and literal lengths, three repeat offsets (1, 4, 8) and the content (RFC 8878 §5). Total size
is at most `DictionarySize`: content for the held-out parts is chosen with 350 bytes reserved for the tables, the tables are
measured, and the content from all the samples is then chosen with the real size of the tables (a whole-set choice of the same
length as the held-out content is kept as it is); if the final tables are bigger, the oldest (first) content bytes are dropped.

- **Statistics.** The tables are measured on held-out frames. With one fold (see "Held-out parts"), the content chosen from all
  the samples is used and every sample is parsed. Otherwise, for each fold `f` of the first `min(F, 4)` (all of them when the
  samples total at most 24 MiB, else the first only), content of the reserved size is chosen from the samples outside the fold with
  the final `k`, `d` and score, and the fold's samples (every n-th when more than 20000 or 8 MiB, as for scoring) are parsed
  against it. The histograms are added. Measuring against the final content itself would parse samples that the content was cut
  from, so almost every sample would be one long match and the tables would describe nothing like new data; the effect grows as
  the dictionary approaches the size of the samples. Each sample is parsed by the level-3 fast match finder with the content as
  history (one sample is one frame; blocks of 128 KiB). Every sequence is counted in the literal length code, match length code and
  offset code histograms (offset *value*: 1 to 3 for repeat offsets as the decoder resolves them, otherwise offset + 3; code =
  floor(log2 value)), starting each sample from the repeat offsets 1, 4, 8. All literals, including those after the last match, go
  into the byte histogram. The header is as large as the first fold's content and these histograms need.
- **Huffman.** Every byte gets a code: weight counts are `max(count, 1)`, built with the library's length-limited builder (11
  bits). If that cannot be described (no samples; a perfectly even spread) a text-like prior replaces the counts: space 64,
  `a`-`z` 32, `A`-`Z` and digits 16, other printable 8, control bytes 2, others 1.
- **FSE.** Counts are `max(count, 1)` for every symbol of the range: literal lengths 0..35, match lengths 0..52, offsets 0..max(17,
  floor(log2(content + 3))) (at most 31). The accuracy log is `OptimalAccuracyLog` (max 9; offsets 8), counts are normalized
  (counts at or below `total >> accuracyLog` become probability -1). With no sequences at all the format's default
  distributions are written.
- **ID.** The given `DictionaryId`, or if 0 the xxHash64 (seed 0) of the whole dictionary written with ID 0, folded as
  `32768 + hash mod (2^31 - 32768)`. IDs 1 to 32767 and 2^31 and up are refused (RFC 8878 §5).
- Passing a whole dictionary (magic first) as the content keeps its content and ID and replaces its tables: this is how
  dictionaries made by another tool get tables measured on your samples.

## Limits and errors

`DictionarySize` 256 to 2^30; `d` 2 to 16 and `k >= d` (at most 16 MiB); `HashBits` 8 to 24; `Acceleration` 1 to 10;
`TrainingPercent` 1 to 100; `ScoringLevel` 1 to 22 (a level above the encoder's highest is scored at its highest). Bad options throw `ArgumentException`; samples that cannot give a dictionary throw
`CompressionDictionaryException`; cancellation throws `OperationCanceledException`.

## Time and memory

The d-mers of the samples are counted once per d-mer size, in one pass that keeps the counts of each group apart; a fold's
weights are the sum of the other groups' counts, so holding out a part costs no second pass. Working arrays come from a pool and are
reused by the next candidate; none of this affects the output, which does not depend on what the arrays held before.
`tools/zstd-train-bench/timing.sh` measures time and peak memory against `zstd --train`, and `bench/Managed` against the fastest
managed implementation; method and figures are in the changelog and `docs/bench-method.md`.

## Measured against `zstd --train`

`tools/zstd-train-bench/quality.sh` compresses held-back payloads with zstd's own compressor and a dictionary from each trainer, so
only the dictionary differs. On game snapshot data (trained on one run, judged on another) the total over all kinds and dictionary
sizes is within 0.3% of `zstd --train` (smaller), with most rows within 1%.
Known gaps, left for later tuning: sample sets of a few hundred small payloads against a dictionary larger than the data (the
reference writes a dictionary smaller than the data; ours keeps nearly all of it), and heterogeneous bulk data cut into 4 KiB
chunks with dictionaries of 64 KiB and up (about 2% larger totals; the entropy tables are not the cause, the chosen content is).


<!-- https://recursiveai.org/onearchive/docs/zstd-seekable/ -->

# Zstandard seekable format

Reading and writing seekable Zstandard files: ordinary frames, each compressed on its own, followed by one
skippable frame (the seek table) that maps content offsets to frames. A seekable file is a valid Zstandard file;
any decoder reads it start to end and ignores the table. The format is the Zstandard seekable format document,
version 0.1.0. Code: `src/OneCompress/Zstd/Seekable/`.

## Layout

```
frame 0 | frame 1 | ... | frame n-1 | seek table frame
```

Seek table frame (all fields little-endian):

| field | size | value |
|---|---|---|
| Skippable_Magic_Number | 4 | `0x184D2A5E` |
| Frame_Size | 4 | `entries * entry size + 9` |
| entry × n: Compressed_Size | 4 | the frame's size in the file |
| entry × n: Decompressed_Size | 4 | the frame's content size; 0 for a skippable or empty frame |
| entry × n: Checksum | 4 | only with the checksum flag: low 32 bits of XXH64 (seed 0) of the frame's content |
| Number_Of_Frames | 4 | n |
| Seek_Table_Descriptor | 1 | bit 7 = checksum flag; bits 6-2 reserved; bits 1-0 unused |
| Seekable_Magic_Number | 4 | `0x8F92EAB1` |

## What the writer does

- The content is cut every `FrameSize` bytes (default 1 MiB, 1 to 2^30). The last frame holds the rest. An input of
  length 0 gives no frames and a table with zero entries. An input that is an exact multiple of the frame size
  gets no extra empty frame.
- Each frame is compressed by one `ZstdEncoder` with the options in `ZstdSeekableOptions.Compression`, as a
  complete frame: nothing carries over between frames, so no frame depends on another. The header declares the
  frame's content size when `WriteContentSize` is set (the default). `ContentSize` must be null.
- The frame's own optional 32-bit checksum (`ZstdCompressionOptions.Checksum`, default off) is independent of the
  seek table checksums (`ZstdSeekableOptions.Checksums`, default on).
- The bytes depend only on the content, the options, and `EndFrame` calls (which cut a frame early at the caller's
  choice). They do not depend on how the content arrives across `Write` calls, on threads or on vector paths.
- The descriptor's unused bits are written as zero.
- Frames are compressed one after another; the writer does not use worker threads across frames. (Options that
  split a single frame into jobs apply inside each frame as usual.)

## What the reader checks

On open (`ZstdSeekTable.Read`):

1. The file is at least 17 bytes; its last four bytes are `0x8F92EAB1`.
2. The descriptor's reserved bits 6-2 are zero.
3. `Number_Of_Frames` is at most `MaxFrames` (default 4,194,304; `CompressionLimitException`, before anything is
   allocated), and a table of that many entries fits in the file.
4. The table frame starts with `0x184D2A5E` and its `Frame_Size` equals `entries * entry size + 9`.
5. Every `Compressed_Size` is at least 8 (the smallest frame, an empty skippable frame).
6. The sizes add up: the frames fill the file exactly up to the table frame.
7. No frame declares more than `MaxFrameSize` bytes of content (default 256 MiB), so the buffer for one decoded frame
   is bounded before it exists.

Every failure is a `CompressionFormatException` (format `zstd-seekable`) with the byte offset and what was
expected. On a read, the frame is decoded with `ZstdDecoder` and must produce exactly `Decompressed_Size` bytes,
and match the table checksum when the table has them (`VerifyChecksums`, default on); otherwise
`CompressionFormatException`. Decoder errors inside a frame are re-raised with the offset relative to the file.
Frames whose `Decompressed_Size` is 0 are never read by a range read. `Verify()` decodes every frame and also
checks that a zero-content frame is a skippable frame of the listed size or an empty Zstandard frame.

The table does not describe frames inside the file beyond their sizes, so a file whose table is consistent but
whose frames are not frames fails when those frames are read (or by `Verify()`), not when it opens.

## Random access

`FindFrame(offset)` binary-searches the cumulative content offsets for the frame whose end lies beyond the
offset. A read decodes each frame it touches once: a read of `L` bytes touches `ceil(L / F)` frames, or one more
when it straddles a frame boundary, for frame size `F`. The reader caches the last
decoded frame, so sequential small reads decode each frame once, and a read that covers a whole frame decodes
straight into the caller's buffer. `ZstdSeekableReader.FramesDecoded` and `CompressedBytesDecoded` count the work.

## Non-seekable input

A seekable reader needs to read the end of the file and then jump around, so it takes a readable stream that
can seek (or a memory block). A stream that cannot seek is rejected with `ArgumentException`. To read a seekable
file from such a stream, use `ZstdStream`, which decodes the frames in order and skips the seek table.

## Tiers

| Tier | Types |
|---|---|
| 1 | `ZstdSeekable.Compress`, `Decompress`, `DecompressRange`, `ReadTable` |
| 2 | `ZstdSeekableWriter` (to a `Stream` or `IBufferWriter<byte>`, reusable through `Reset`), `ZstdSeekableReader` (a stream or memory, positional `Read`/`ReadAsync`), `ZstdSeekableStream` (read-only, seekable `Stream`) |
| 3 | `ZstdSeekTable` (parse, index, `FindFrame`), `ZstdSeekTableFooter`, `ZstdSeekFrame`, `ZstdSeekTableBuilder` (encode) |

## Testing

- Unit and property tests: `tests/OneCompress.Tests/Zstd/ZstdSeekableTests.cs`.
- Interop (`ONEARCHIVE_INTEROP=1`): our files decode with the `zstd` tool and read with the seekable-format
  example reader; files from the example producer open in our reader. The example programs are built by
  `tools/build-zstd-seekable-examples.sh` from the pinned release tarball as black boxes.
- Fuzz targets `zstd-seekable-read` (memory and stream readers must agree) and `zstd-seekable-roundtrip`;
  seeds from `fuzz/seeds-seekable.sh`.
- Hash matrix entries `zstd-seekable/*`.

## Size overhead

silesia.tar (211,957,760 bytes), compressed size against one frame at the same level; the table is 8 + 9 bytes plus
8 or 12 bytes per frame. The cost is the frames, not the table.

| level | single frame | frame 64 KiB | frame 1 MiB | frame 8 MiB |
|---|---|---|---|---|
| 3 | 66,748,199 | +10.43% (3,235 frames) | +1.70% (203 frames) | +0.15% (26 frames) |
| 15 | 56,938,984 | +18.74% | +7.28% | +1.34% |

With table checksums on, the table is 38,837 / 2,453 / 329 bytes at the three frame sizes; off, 25,897 / 1,641 / 225.

Frames touched by a read at level 3 (1,000 random reads each, last-frame cache on): reads of 1 byte and 4 KiB touch
1.0 frame at every frame size; 64 KiB reads touch 2.0 frames at F = 64 KiB and 1.05 at F = 1 MiB; 1 MiB reads touch
17.0 frames at F = 64 KiB and 2.0 at F = 1 MiB.


<!-- https://recursiveai.org/onearchive/docs/message-mode/ -->

# Streams of tiny messages

For a connection that carries many messages of tens to a few hundred bytes, there are two ways to compress them. Both keep the
encoder and its buffers for the life of the connection, so a steady state allocates nothing.

## Frame per message

`ZstdEncoder.TryCompressFrame(message, buffer, out n)` (or `Lz4Encoder.TryCompressFrame`, or the headerless
`Lz4BlockEncoder.TryCompressPrefixed`) turns each message into a self-contained unit that can be decoded alone, lost or reordered.
A zstd frame costs a 4-byte magic number, a descriptor, optionally the content size and dictionary ID, and a 3-byte block
header; an LZ4 frame costs 11 bytes. With a trained dictionary this is the best choice when messages arrive independently.

- Set `WriteContentSize = false` and `WriteDictionaryId = false` when both ends agree on the dictionary out of band (saves up to 8 bytes).
- Set `Checksum = false` (the zstd default) unless the transport does not already protect the bytes.
- LZ4 frames: `ContentChecksum = false` saves 4 bytes; `Lz4BlockEncoder.TryCompressPrefixed` removes the frame entirely (a varint
  length and the block, not an LZ4 frame, read back with `Lz4PrefixedBlock`).

## One stream, flushed per message

A single `ZstdEncoder` (or `ZstdStream`) calls `Compress(message, ..., isFinalBlock: false)` and then `Flush(...)` after every message.
The receiver's `ZstdDecoder` sees one frame and can decode each message as soon as its bytes arrive. Later messages can match
earlier ones, so the output is much smaller than frame-per-message when messages resemble each other: measured on 232 game
snapshot payloads of 256 bytes, with a 16 KiB dictionary at level 1, 111 bytes per message in a stream against 137 as frames, and
125 against 222 without a dictionary.

- **Cost of a flush.** Each `Flush` ends a block, so it writes a 3-byte block header and the block's literal and sequence
  headers, and the next message starts a new block (entropy tables may be repeated from the previous block). At 256-byte messages the
  work per message is the same few microseconds as a frame; it does not grow with the length of the stream. Do not set `JobSize`
  in this mode: with jobs every flush reloads the overlap, and a flush after every few bytes costs time quadratic in the stream.
- **Memory per stream.** The encoder holds a window buffer of `2^WindowLog + 2 * 128 KiB` bytes, plus the match finder's tables
  (`2^HashLog` four-byte entries, and a chain table at levels 5 and up). Left at 0, `WindowLog` follows the level's table, which is
  megabytes of window for an unsized stream; set it. The decoder needs the same window.
- **Recommended options.** `Level` 1 to 3, `WindowLog` 14 to 17 (16 to 128 KiB of history; messages older than the window are not matched anyway), `Checksum = false`, a trained `Dictionary`, and `ContentSize`
  unset. Set the decoder's `MaxWindowSize` to match.
- **LZ4** streams: `Lz4Encoder` with `BlocksIndependent = false` keeps 64 KiB of history across messages; its `Flush` ends a block, so the
  per-message cost is a 4-byte block size. Memory is 64 KiB of history plus one block (`BlockSize`, default 4 MiB), so
  set `BlockSize` to `Size64KiB` unless messages are larger.

A frame boundary or a lost message breaks the shared history of a stream, so use frame-per-message over unreliable or
unordered transports.


<!-- https://recursiveai.org/onearchive/docs/lz4-encoder/ -->

# LZ4 encoder

How the LZ4 compressors in `RecursiveAI.OneCompress.Lz4` choose and write bytes. A later port reproduces the same
output from this page and the format documents (LZ4 Block Format and LZ4 Frame Format). The decoder is described in
`lz4-decoder.md`.

## Layers

| Tier | Types | Use |
|---|---|---|
| 1 | `Lz4.Compress`, `Lz4.TryCompress`, `Lz4Block.Compress`, `Lz4Block.TryCompress` | one call, whole input |
| 2 | `Lz4Encoder`, `Lz4Stream` (write mode) | input in pieces, frames one after another, reuse, flush |
| 3 | `Lz4BlockCompressor` (`Lz4FastBlockCompressor`, `Lz4HighBlockCompressor`), `Lz4BlockWriter`, `Lz4Dictionary` | match finders over a window the caller manages; the sequence writer |

Tier 1 calls tier 2, tier 2 calls tier 3. Every tunable is on `Lz4CompressionOptions`.

## Output depends on the input and the options only

- Integer arithmetic and fixed multiplicative hashes only. The cost model of the optimal parser is integer. No floating
  point, no `HashCode`, no timing, no thread count, no machine feature in any decision. Vector instructions are used only
  to copy bytes and to compare eight bytes at a time (the same count as a byte loop).
- A compressor instance gives the same bytes as a new one: starting a block without history moves the running position
  past everything seen (so older table entries are more than 64 KiB away and never used) instead of clearing tables.
  A stale entry and an empty one behave identically.
- The frame bytes do not depend on how the input is split between calls or on the sizes of output buffers. They do depend
  on where `Flush` is called (it ends the block).

## Levels

`Level` follows the `lz4` tool: `-N` is `--fast=N`, 1 is the default, 2 to 12 are high compression. Level 0 is level 1.
The range is -65536 to 12 (a level outside it is an `ArgumentOutOfRangeException`).

| Level | Finder | Parse | Parameters |
|---|---|---|---|
| 1, 0 | hash table | greedy | acceleration 1 |
| -N | hash table | greedy | acceleration N + 1 (so `--fast=1` skips like acceleration 2) |
| 2 | hash table (2^18 entries, every second interior position indexed) | lazy, one-byte lookahead | acceleration 1 |
| 3 to 9 | hash chain | lazy, two-byte lookahead | chain candidates at a position with no match yet 10, 24, 48, 88, 136, 192, 256; at the next position 4, 9, 19, 35, 54, 76, 102; at the one after 1, 2, 4, 8, 13, 19, 25; stop at match length 256; no look-ahead from a match of 64 or more |
| 10 to 12 | hash chain | optimal | chain candidates 56, 100, 128; stop at match length 256 |

The candidate counts were chosen so that the output stays within 1% of the `lz4` tool's size at the same level, at the
least search work that achieves it (the measured sizes are under "Measured sizes" below). A longer match is rarer than a
first match and is only an improvement, so the look-ahead positions get a shorter walk than the first one: about 40% of
the candidates one byte on and about 10% two bytes on. A deeper first walk with shallow look-ahead walks gives a smaller
block for the same work than the same count at every position.

### Fast levels (hash table, greedy)

- One table of 4096 `uint` entries (16 KiB), indexed by `((eight bytes << 24) * 889523592379) >> 52` (a hash of five bytes,
  little-endian). An entry is the position of the last occurrence.
- At each probe the entry for the current position is read and replaced. A candidate is a match when it lies 1 to 65535
  bytes back and its first four bytes agree. The first byte of a block is only indexed.
- While nothing matches, the step between probes starts at 1 and grows by one every 64 failed probes, scaled by the
  acceleration: the step after `n` failures is `(n + acceleration * 64) >> 6` (the first probe step is 1).
- Level 2 only: lazy matching. After a match is found (and extended backwards), the position one byte later is looked up in
  the table (and indexed); a candidate there that agrees on four bytes and gives a match more than one byte longer than the
  one found replaces it, and the byte before it becomes a literal. Level 2 indexes every second position inside each match
  (`interiorStride` 2).
- A found match is extended backwards over pending literals (not past the start of the window or the dictionary), then
  forwards up to five bytes before the end of the block. After a match, the position two bytes before its end is indexed
  and the position at its end is tried at once (no step).
- Block end rules (Block Format, "End of block conditions"): the last match starts at least 12 bytes before the end and
  ends at least 5 bytes before it; a block shorter than 13 bytes is literals only.

### High levels

- Hash chain: `head[hash]` holds the last position with that hash of four bytes (`(bytes * 2654435761) >> 16`, 65536
  slots); `chain[position & 0xFFFF]` holds the distance to the previous position with the same hash (0 ends the chain, and
  a distance of 65536 or more is stored as 0). Every position is inserted, in order, before it is searched, including the
  positions inside matches. A position is searched right after it is inserted, from the head entry it replaced.
- Levels 6 and up store a 32-bit chain entry: the link above in the low 16 bits and, in the high 16 bits, the distance to
  the position that link points to (0 when either link is missing or the sum exceeds 65535). The search reads two
  candidates from one entry, so the dependent load for the third candidate starts sooner. The candidates, their order and
  the output are exactly those of the 16-bit chain; only the speed differs.
- A search at a position examines at most that search's number of chain candidates (the level's table row above), nearest first. A candidate counts when it
  lies 1 to 65535 bytes back, agrees on the four bytes ending at the current best length, agrees on the first four bytes,
  and extends past the best length so far (ties keep the nearest). A candidate that fails the four-byte test cannot be
  longer than the best, so the test changes no result. The search stops early when the best length reaches the level's stop
  length or the end of the block. Matches end at least 5 bytes before the end of the block and start at least 12 bytes before it.
- **Lazy parse (levels 3 to 9).** At a position with a match, the next position is searched (with the level's second
  candidate count) for a match longer than the current one; one found replaces it (the bytes before it become literals) and
  the check repeats from there. When the next position has none, the position after it is searched (with the third count)
  for a match more than one byte longer than the current one, and one found replaces the current match the same way. A match
  of 64 bytes or more is taken without either search. The chosen match is then extended backwards over pending literals.
- **Optimal parse (levels 10 to 12).** Starting at a position with a match, the encoder plans the cheapest sequence of
  literals and matches for the stretch ahead, by dynamic programming over byte positions. Prices are bytes of output:
  a literal costs 1, plus 1 more when the run it extends reaches 15 literals and every 255 after; a match costs 3 (token
  and offset) plus length bytes when the length minus 4 reaches 15 and every 255 after. Every length from 4 up to the longest
  match found at a position is available at that position's offset. A position is not searched when the next position is no
  dearer to reach and a four-byte match here would not improve the price four bytes on (a match starting here could be
  replaced by one starting a byte later). The plan ends where no match reaches further, or at the first match of at least the
  stop length, or when a match would reach 4096 bytes ahead; matches are then written front to back.

## Dictionaries

A dictionary (`Lz4Dictionary`, at most the last 64 KiB) is history in front of a block that has no history of its own.

- Fast levels: the dictionary has its own table, built once per dictionary object (every position that has eight bytes
  after it, later positions replacing earlier ones). When the current table has no usable candidate, the dictionary table is
  tried. A match may start in the dictionary and run on into the block, because the history is the dictionary followed by the
  block. No dictionary bytes are copied per block.
- High levels: the dictionary and the block are placed in one buffer, and the chains are copied from a snapshot built once
  per dictionary (positions with four bytes after them). A block of at most 64 KiB without history leaves the dictionary's
  index in place between calls instead: it records the head and chain values each insertion replaces and puts them back
  afterwards, so many small blocks with one dictionary cost the block's own work. The bytes produced are the same either way.
- Frames: independent blocks all start from the dictionary; with linked blocks only the first block does. The frame header
  names the dictionary by `Lz4Dictionary.Id` when it is not zero.

## Frames

- Header: version 1, block independence from `BlocksIndependent`, block checksum, content size if known, content checksum,
  dictionary ID if the dictionary has one; block maximum size from `BlockSize` (default 4 MiB). The one-call
  `Lz4.Compress` and `Lz4Encoder.TryCompressFrame` write the content size unless `IncludeContentSize` is false.
- Blocks hold `BlockSize` bytes except the last (and blocks ended by `Flush`). A block that does not compress below its input
  size is stored (high bit of the size field). Block checksums are xxHash32 of the stored bytes; the content checksum is
  xxHash32 of all the input.
- Linked blocks keep the last 64 KiB of the data as history; the encoder moves it to the front of its window between blocks.
- An empty input is a frame with no blocks. After a frame is complete, the next call to `Lz4Encoder.Compress` starts a new one.

## Limits

- A single block holds at most 2,113,929,216 bytes (`Lz4BlockCompressor.MaxInputLength`). A destination for a block must
  hold `n + n / 255 + 16` bytes (`Lz4BlockWriter.GetMaxCompressedLength`); the one-call `Lz4Block.TryCompress` accepts less and
  reports whether the result fit.
- Position arithmetic is in 32 bits. After about two gigabytes through one instance the tables are cleared (a ratio cost
  only, and still deterministic).

## Memory

A fast compressor holds 16 KiB (level 2: 1 MiB); a high-level compressor holds 384 KiB (a 256 KiB head table and 128 KiB of
chain links; 512 KiB from level 6 up, where the links are 256 KiB), plus 64 KiB of plan state for levels 10 to 12. With a dictionary, a high-level compressor also keeps a copy
buffer (the dictionary and the block) and, for blocks up to 64 KiB, undo records of 6 bytes per input byte. A frame encoder
adds its staging buffer (one block) and, for independent blocks that arrive in pieces or linked blocks, a window (block
size, plus 64 KiB when linked).

## Measured sizes

Block sizes (4 MiB independent blocks, as `lz4 -b<level> -B7` reports them) against the `lz4` tool 1.10.0, measured on the
library at commit 5d6f9e5. A negative figure means ours is smaller. Levels 10 to 12 were not measured again; their bytes are
the same as before that commit.

| Level | silesia.tar | enwik8 |
|---|---|---|
| 1 | -0.001% | +0.000% |
| 2 | +0.11% | -0.95% |
| 3 | +0.80% | +0.42% |
| 4 | +0.72% | +0.49% |
| 5 | +0.80% | +0.61% |
| 6 | +0.84% | +0.63% |
| 7 | +0.90% | +0.64% |
| 8 | +0.90% | +0.62% |
| 9 | +0.88% | +0.61% |

The level 2 figures are for the lazy fast parse described above.

## Measuring

`tools/lz4-bench` holds the encoder's measurements. All timing is memory to memory and warm, and runs one process at a time
under the shared timing lock, recording the machine's load before, during and after.

- `bar.sh` times block compression the way `lz4 -b<level>` does (independent 4 MiB blocks, one reused compressor, the best
  of several passes inside each process, `lz4 -b<level> -B7 -i<seconds>` for the reference), repeating each side `--reps`
  times in fresh processes. A row gives both compressed sizes, the min/median/max compression and decompression speeds of
  the repeats, the speed ratios from the medians, and both sides' peak resident memory (`/usr/bin/time`).
  The sizes are those of the blocks (the frame adds a few bytes).
- `game-prep.sh` samples the game snapshot data into per-game payload directories and builds raw 8 KiB and 64 KiB
  dictionaries from the training set. `game-bar.sh` compresses every payload as its own frame with a reused encoder and a
  dictionary prepared once, against the lz4 library's frame compressor with a prepared dictionary (`refsmall.c`, built
  against the pinned lz4 install's shared library, never the system one, and called as a black box), and prints sizes and payloads per second.
- `Lz4Bench size` prints frame sizes only, for tuning.

A row passes when our size is at most 1.01 times the reference's and our compression and decompression times are each at most
1.15 times the reference's (speed ratio at least 0.87).


<!-- https://recursiveai.org/onearchive/docs/lz4-decoder/ -->

# LZ4 decoder

What the LZ4 decoder accepts, rejects and bounds. The format sources are the LZ4 Block Format and Frame Format documents
(`lz4_Block_format.md`, `lz4_Frame_format.md`); section names below refer to them. A later port should reproduce these
choices from this page alone.

## Layers

| Tier | Types | Use |
|---|---|---|
| 1 | `Lz4.Decompress`, `Lz4.TryDecompress`, `Lz4Block.Decompress`, `Lz4Block.TryDecompress` | one call, whole input |
| 2 | `Lz4Decoder`, `Lz4Stream` | any split of input and output; reuse across frames |
| 3 | `Lz4BlockDecoder`, `Lz4FrameHeader`, `Lz4Dictionary` | the block decoder over spans, the frame descriptor parser |

Tier 1 calls tier 2, tier 2 calls tier 3. `Lz4Block` decodes raw blocks whose compressed size and a bound on the decoded size
come from outside the data; `Lz4` decodes frames.

## Blocks (Block Format)

- A block is a run of sequences: token, literals, 2-byte little-endian offset, match. The last sequence has literals only and
  no offset. A block ends exactly where its input span ends, so the caller supplies the block's compressed length.
- Length fields: the high nibble of the token is the literal count, the low nibble the match length minus 4; 15 means more
  bytes follow, each added, and a byte of 255 means another follows. There is no limit on the number of bytes; sums are
  taken in 64 bits, and a literal count larger than the remaining input is a truncation error.
- Matches copy from `offset` bytes back, byte by byte in effect, so a match may be longer than its offset and repeats the
  last `offset` bytes.
- **Offset 0 is an error**, and so is an offset reaching before the start of the history (the first
  bytes of the destination, or the dictionary). The decoder never reads or writes outside the destination.
- **End of block conditions are enforced.** Unless the block is a single sequence, the last sequence must hold at least 5
  literals and the last match must start at least 12 bytes before the end of the block. The format allows a decoder to reject
  other blocks; this one does. An empty block (no bytes) is an error; the empty output is the single byte `0x00`.
- Errors carry the offset in the block's input: the offset field for offset errors, the end of the input for truncation and
  for the end rules, and the position after the length bytes for output overflow.
- `Lz4BlockDecoder.TryDecode(source, destination, historyLength, out written)`: the first `historyLength` bytes of
  `destination` are history (dictionary or earlier blocks); output follows. It returns false (no exception) when the output
  does not fit. `GetDecompressedSize` validates the block's structure and returns the exact output size without writing.
- `Lz4Block.Decompress(source, options)` measures first, checks the size against `Lz4BlockOptions.MaxOutputSize` (default
  64 MiB) before allocating, then decodes. `Lz4Block.Decompress(source, destination)` throws `CompressionLimitException`
  (`LimitName` = `destination.Length`) if the output does not fit. A dictionary is copied in front of the output for each call.

## Frames (Frame Format)

- Magic `0x184D2204`, then the descriptor: FLG (version 01 in bits 7-6, block independence, block checksum, content size,
  content checksum, reserved bit 1, dictionary ID), BD (block maximum size code 4 to 7 = 64 KiB, 256 KiB, 1 MiB, 4 MiB; bits
  7 and 3-0 reserved), optional 8-byte content size, optional 4-byte dictionary ID, and the header checksum (second byte of
  xxHash32 of the descriptor, seed 0). Wrong version, set reserved bit, block size code below 4, a content size of 2^63 or
  more, or a wrong header checksum is an error at that field's offset. The header checksum is checked even when
  `VerifyChecksum` is false.
- Data blocks: 4-byte size; bit 31 set means stored (uncompressed). A size of 0 is the EndMark. A size above the block
  maximum is an error at the size field. A stored block of size 0 is valid. With block checksums each block, empty ones
  included, is followed by xxHash32 (seed 0) of its raw bytes. A compressed block decodes to at most the block maximum bytes
  and, when the content size is declared, at most what remains of it.
- After the EndMark: the content checksum (xxHash32 of the decoded content) when flagged. When the content size is declared,
  the decoded size must equal it, checked at the EndMark.
- **Linked blocks** (independence flag clear) may refer to the previous 64 KiB of the same frame's output; a new frame never
  sees an earlier frame's output. Independent blocks refer to nothing before their own start, except the dictionary.
- **Dictionaries.** With independent blocks every block starts with the dictionary as history; with linked blocks the
  dictionary is history once, at the start of the frame. Only the last 64 KiB of a dictionary count. The frame's dictionary ID
  selects a dictionary: a nonzero ID is looked up in `Dictionaries` by `Lz4Dictionary.Id`, then `Dictionary` is used if its
  ID is 0 or equal; no match raises `CompressionDictionaryException`. A frame with no ID (or ID 0) uses `Dictionary` if set.
  The lz4 tool writes no dictionary ID when given `-D`, so its frames need `Dictionary`.
- **Concatenated frames** decode in sequence; the decoder returns `Done` after each. **Skippable frames** (magic
  `0x184D2A50` to `0x184D2A5F`) are skipped, or passed to `SkippableFrameHandler(variant, data)` when set; a payload larger
  than `MaxSkippableFrameSize` (1 MiB) raises `CompressionLimitException` before it is buffered.
- **Legacy frames** (magic `0x184C2102`): blocks of at most 8 MiB decoded, each preceded by its 4-byte compressed size
  (at most 8 MiB + 8 MiB / 255 + 16). The frame ends at the next frame's magic number (standard, skippable or legacy), at a
  zero size (the Linux kernel's end marker, accepted), or where the input ends. A size above the bound is an error.
  Dictionaries and checksums do not apply to legacy frames.
- **Empty input is an error** (a truncated frame at offset 0), unlike the lz4 tool, which accepts it: a network payload that
  is empty is not an LZ4 frame.
- Trailing bytes after the last frame that are not a frame are an error at their offset.

## Limits (`Lz4DecompressionOptions`)

| Option | Default | Meaning |
|---|---|---|
| `MaxBlockSize` | 8 MiB | largest block maximum a frame may declare (legacy frames count as 8 MiB); checked at the header, before buffers are allocated |
| `MaxOutputSize` | 1 GiB | total output between resets; a declared content size that would exceed it is refused at the header; output that exceeds it mid-frame stops the decoder with the limit error |
| `VerifyChecksum` | true | block and content checksums |
| `Dictionary`, `Dictionaries` | none | see above |
| `SkippableFrameHandler`, `MaxSkippableFrameSize` | none, 1 MiB | see above |

Memory: a decoder that decodes straight into the caller's buffer allocates nothing beyond a small header buffer. Otherwise it
holds a window of `dictionary + block size` bytes (independent blocks) or `64 KiB + 2 x block size` (linked blocks), where
the block size is the smaller of the frame's block maximum and its declared content size, plus a buffer for a compressed
block that arrives in pieces. Buffers grow only and are reused; a dictionary stays in the window between frames that use
the same one.

## Streaming (`Lz4Decoder`)

`Decompress(source, destination, out consumed, out written)` follows the shared decoder shape (`docs/conventions.md`).
- A compressed block is decoded only after the whole block and its checksum have arrived and been verified. A stored block is
  handed out as it arrives and its checksum is checked at its end.
- `Done` comes at the end of each standard or skippable frame, with its output written. A legacy frame has no end mark, so its
  `Done` comes with the next frame's magic number; at the end of the input it simply ends (`IsAtFrameBoundary`).
- `TryDecompress(source, destination, out written)` (whole input, whole output) lets blocks decode directly into
  `destination` when no dictionary is involved: linked blocks then reach their history in place. `Decompress` with a
  destination at least as large as a block does the same for independent blocks without a dictionary.

## Unchecked accessors

`Lz4Raw` holds the only unchecked code: 16-byte and 8-byte wide copies. A caller uses them only after comparing the room in the
destination and the remaining input with the copy width plus its overshoot (at most 15 bytes, overwritten by the following
sequence). With the `Checked` configuration they are ordinary bounds-checked copies, and the tests and fuzzers run that way.


<!-- https://recursiveai.org/onearchive/docs/deflate-encoder/ -->

# Deflate, zlib and gzip encoder

Normative and language-neutral: a port that follows this text produces the same bytes. RFC 1951 leaves the parse, the block
boundaries and the Huffman code construction to the encoder; everything it leaves open is fixed here. Output depends only on the
input bytes, the options, and where flushes fall. It never depends on the CPU, the runtime or how the caller splits the input
into calls.

## API shape

- Tier 1: `Deflate`, `ZLib`, `GZip` each have `Compress`, `TryCompress`, `GetMaxCompressedLength` taking a `DeflateOptions`.
- Tier 2: `DeflateEncoder` (`OperationStatus Compress(source, destination, out consumed, out written, DeflateFlush)`, a
  `bool isFinalBlock` overload, `Flush`, `Reset`) and `DeflateEncoderStream` (write-only stream, `Flush()` is a sync flush).
- Tier 3: `DeflateHashChainMatchFinder`, `DeflateBinaryTreeMatchFinder`, `DeflateSequenceBuffer`, `DeflateBlockSplitter`,
  `DeflateHuffmanBuilder`, `DeflateBlockWriter`. The encoder is built from exactly these.

## Options

| Option | Default | Meaning |
|---|---|---|
| `Level` | 6 | 0 to 12, table below. |
| `Strategy` | `Default` | `HuffmanOnly` (no matches), `RunLength` (matches of distance 1 only), `FixedCodes` (fixed Huffman codes or stored, never dynamic), `Default`. |
| `Dictionary` | empty | Preset dictionary. zlib: sets FDICT and writes the Adler-32 of the dictionary; raw: used silently; gzip: rejected. Only the last 32768 bytes are used. |
| `GZipFileName`, `GZipComment`, `GZipExtra`, `GZipHeaderCrc`, `GZipIsText`, `GZipModificationTime`, `GZipOperatingSystem` | none, none, none, false, false, 0, 255 | RFC 1952 section 2.3 header fields. Names are written as Latin-1 with a terminating 0. |

## Levels

Level 0 stores. Levels 1 to 9 follow the zlib scale (1 fastest, 9 smallest); 10 to 12 are optimal parses and correspond to
libdeflate 10 to 12. Levels 1 to 9 are compared with zlib-ng 1 to 9 (streaming), and 1 to 12 with libdeflate 1 to 12 (one-shot);
the mapping is by number.

| Level | Parser | Hash | Chain | Nice | MaxLazy | Tail | Good | Passes |
|---|---|---|---|---|---|---|---|---|
| 0 | stored blocks of at most 65535 bytes | | | | | | | |
| 1 | greedy, one probe, 2-way bucket | 4 bytes, 17 bits | 1 | 32 | 16 | 4 | | |
| 2 | greedy | 4 bytes, 15 bits | 10 | 40 | 32 | 4 | | |
| 3 | greedy | 4 bytes, 15 bits | 18 | 48 | 48 | 4 | | |
| 4 | lazy | 4 bytes, 15 bits | 8 | 32 | 16 | | 8 | |
| 5 | lazy | 3 bytes, 15 bits | 10 | 48 | 24 | | 12 | |
| 6 | lazy | 3 bytes, 15 bits | 20 | 64 | 64 | | 8 | |
| 7 | lazy | 3 bytes, 15 bits | 32 | 100 | 100 | | 16 | |
| 8 | lazy, look-ahead 2 | 3 bytes, 15 bits | 48 | 160 | 200 | | 16 | |
| 9 | lazy, look-ahead 2 | 3 bytes, 15 bits | 64 | 258 | 258 | | 16 | |
| 10 | optimal, binary tree | 4 bytes 16 bits + 3 bytes 16 bits | 24 | 64 | | | | 1 |
| 11 | optimal | as 10 | 64 | 128 | | | | 2 |
| 12 | optimal | as 10 | 256 | 258 | | | | 3 |

(The table above is the single source; the code's `LevelParameters.For` must match it.)

### Hashing and match finding

- Every hash is `(bytes_little_endian * 2654435761) mod 2^32`, shifted right to the table size. The 4-byte hash reads 4
  bytes, the 3-byte hash the low 3 of them.
- Window: 32768 bytes of history. Candidates farther than 32767 back are never taken. A match of length 3 farther than 4096 is
  not taken (it costs more bits than three literals).
- Hash chain finder: `head[hash]` holds the latest position, `previous[position & 32767]` the one before. Candidates are visited
  newest first, at most `Chain` of them; a candidate must beat the best length so far; the search ends at length `Nice`. Once a
  previous match is at least `Good` long, the next search uses a quarter of the chain. A match is the longest found; at equal
  length the newest (smallest distance) wins.
- Level 1 probes one bucket of two entries (newest and older) of the 4-byte hash (the bucket index is the hash with its low bit
  cleared) and takes the longest, ties to the newer.
- Levels 2 and 3 are greedy: take the match found at a position, then insert the positions inside it into the finder: all of
  them when the match is at most `MaxLazy` long, otherwise only its last `Tail` positions. Level 1 does the same.
  Levels 5 to 9 also keep a table of the latest position per 3-byte hash (15 bits), consulted when no match was found yet; levels 2
  to 4 hash 4 bytes only, find no 3-byte matches, and so skip that table (a measured trade: the table costs a store and a load
  per position, and at these levels the 3-byte matches it finds are few and mostly too far to pay).
- Dry spells (levels 1 to 7): the parser counts the positions it searched in a row without finding a match (a match resets the
  count; bytes it steps over are not counted). Once the count exceeds 16, each further failed search is followed by
  `(count - 16) / 8` bytes emitted as literals without searching or entering them in the finder (never past the end of the run being
  parsed, and never past the point where the sequence buffer must be checked), so data that does not compress costs little time.
  The count restarts for every call of the parser, as the other parser state does.
- Lazy (4 to 9): after finding a match shorter than `MaxLazy`, look at the next position. Matches are compared by a score,
  `4 * length - bsr(distance)` (`bsr` = index of the highest set bit; a nearer match costs fewer bits). The next position's match
  replaces the current one when its length is at least the current length and its score is higher by more than 1: one literal is
  emitted and the longer or nearer match is carried forward. A match of length 3 farther than 8192 is never taken (levels 2 and 3
  keep 4096). Repeat. Positions inside a taken match are inserted. Levels 8 and 9 also look two positions ahead when the next
  position did not replace the match: a match there that is longer and scores higher by more than 4 replaces it, with two literals
  emitted. The search at the next position asks for a match as long as the current one (`length - 1` is the length to beat) so
  that an equal-length nearer match is found.
- Binary tree finder (10 to 12): a binary tree per 4-byte hash plus a 16-bit 3-byte table; the search follows the tree for at
  most `Chain` nodes, returning every candidate that is longer than the previous one (so lengths ascend and distances ascend).
- A position with fewer than `Nice` bytes of input after it is searched but not inserted into the binary tree: its comparisons are cut
  short by the end of the data, bytes that agree up to the cut would be treated as equal for ever, and a flush followed by more
  input could then make the tree report matches that are not there. (The hash chain finder has no ordering to break and inserts them.)
- The optimal parser closes a block when the sequence buffer is within one run (16384 entries) of its nominal capacity of 65536
  entries, whatever size the buffer really has, so a one-call encoder (whose buffers are sized to the input) and a streaming one
  agree.
- Positions near the end of the buffered input, where fewer than 4 bytes remain, are not hashed and find no match until more
  input arrives; at the end of the stream the remaining bytes are literals.

### Optimal parse (10 to 12)

The input is parsed in runs of up to 16384 positions. For each run: collect matches at every position, then price a shortest path
by dynamic programming over costs in 1/16 bit, then emit the path. Costs come from the symbol statistics so far: literal and
a symbol with count `f` costs `(log2(2*total + symbols) - log2(2*f + 1))` in 1/16 bit, at least 1, where `log2` is computed by
integer squaring with 8 fractional bits (no floating point); match lengths add their extra bits. Before any statistics exist the
costs are those of the fixed codes (literal 8.5 bits, length 7 bits plus extra bits, distance slot 5 bits). The statistics are
updated with each emitted run and halved after every run. With `Passes` > 1 the run is parsed again with prices from the previous pass
and the cheapest result is kept. Matches longer than `Nice` end the search at that position and are taken whole.

### Sequence storage and writing

The parsers of levels 1 to 9 (and the two strategies that bypass the finder) do not store literals. A block holds the matches in
order, each with the input position it starts at; the literals between two matches are the input bytes between them, and the writer
reads them back from the input. The entry count that decides when a block is checked still counts literals and matches together,
so block boundaries and the bytes written are the same as if every literal had been stored. Levels 10 to 12 store every entry.

## Blocks

- Parsed entries accumulate in a sequence buffer of at most 65536 entries (literal or match). A block ends when the buffer is
  full, when the splitter asks, at a flush, or when the 256 KiB input window is processed.
- Splitter: every 512 entries (from 4096 entries into the block) the new entries' mix over 11 classes (literals in eight groups of
  32 values, then matches of length 3 to 5, 6 to 9, and longer) is compared with the block so far. When the sum of absolute
  differences of the class shares exceeds 450/1000, the block ends. All integer arithmetic.
- Writing: the writer prices stored, fixed and dynamic exactly (header bits included) and writes the cheapest; a tie prefers
  fixed, then dynamic, then stored. `FixedCodes` chooses between fixed and stored only. Dynamic codes are built by the length-limited Huffman
  builder (maximum 15 bits, 7 for the code-length alphabet): standard Huffman lengths from frequencies sorted by (frequency,
  symbol), then, if any length exceeds the limit, overlong codes are shortened and the freed space is given to the most frequent
  symbols. At least two literal/length and two distance symbols are always coded so every code is complete. Code lengths are
  sent with run-length symbols 16, 17 and 18 chosen greedily. Codes are assigned canonically (RFC 1951 section 3.2.2).
- The final block carries BFINAL. An empty stream is one empty fixed block (zlib and gzip: header, `03 00`, trailer).

## Window and slide

The encoder buffers up to 262144 bytes. When the buffer is full it parses everything except the last 266 bytes, closes the block,
and slides the buffer by a multiple of 32768 so that at least 32768 bytes of history remain. Output does not depend on how the
caller splits its input among calls, because parsing waits for a full window or a flush.

## Flush

- `None`: nothing forced.
- `Sync`: parse everything, close the block, then write an empty stored block (`00 00 FF FF` after byte alignment). Matches may
  still reach earlier data.
- `Full`: as `Sync`, and later matches may not point before the flush point (the stream can be decoded from there on).
- `Finish`: parse, close the final block, align, write the trailer. A second flush with nothing new written in between writes
  nothing more.

## Wrappers

- zlib header: CMF `78`; FLEVEL 0 when the level is below 2 or the strategy is HuffmanOnly or RunLength, 1 for levels 2 to 5,
  2 for level 6, 3 above; FDICT as above; FCHECK makes the pair a multiple of 31. Trailer: Adler-32 big endian.
- gzip header: `1F 8B 08`, FLG from the options, MTIME, XFL 2 for level 9 and above, 4 for level 1, else 0, OS (default 255);
  trailer CRC-32 and ISIZE (mod 2^32), little endian.
- `GetMaxCompressedLength(n)` = n + n/512 + 64 plus the wrapper size. A destination of that size never reports
  `DestinationTooSmall` for a one-call compression.

## Memory

A reused encoder owns about 2 MiB (window, sequence buffer, writer buffer, finders; the optimal parser a few MiB more) and
allocates nothing per call once warm.


<!-- https://recursiveai.org/onearchive/docs/deflate-decoder/ -->

# Deflate, zlib and gzip decoder

What the decoder accepts, rejects and bounds. The format sources are RFC 1951 (Deflate), RFC 1950 (zlib) and RFC 1952
(gzip); section names below refer to them. A later port should reproduce these choices from this page alone.

## Layers

| Tier | Types | Use |
|---|---|---|
| 1 | `Deflate.Decompress`, `ZLib.Decompress`, `GZip.Decompress` and their `TryDecompress` | one call, whole input |
| 2 | `DeflateDecoder` (`DeflateFormat` Raw, ZLib or GZip), `DeflateCodecStream` (read mode) | any split of input and output; reuse across units |
| 3 | `DeflateHuffmanTable`, `DeflateBitReader`, `DeflateTables` | table building, bit input, the constant tables of RFC 1951 |

Tier 1 calls tier 2, tier 2 calls tier 3. The stream is `DeflateCodecStream` because `DeflateStream` is the BCL's name.

## Units and `Done`

A unit is one raw Deflate stream, one zlib stream or one gzip member. `Decompress` returns `Done` when a unit is complete and
fully written, and then stops; the next call starts a new unit (a gzip file with several members is decoded by calling again with
the bytes not consumed; tier 1 and the stream do this). Each unit starts with an empty window. Tier 1 and the stream refuse bytes
after the end of a raw or zlib stream unless `AllowTrailingData` is set; for gzip, bytes after a member are the next member
(trailing bytes that do not start with 1F 8B are ignored under `AllowTrailingData` and an error otherwise).

A call that returns `NeedMoreData` has consumed all of its input: bits and partial header fields are kept inside the decoder,
so the caller never holds input back. `consumed` and `written` are set even when the call throws.

## Blocks (RFC 1951)

- BTYPE 3 is an error. Stored blocks require NLEN to be the one's complement of LEN. The bits up to the byte boundary after a
  block header, and after the final block, are padding and are ignored.
- Dynamic blocks (3.2.7): **HLIT above 286 and HDIST above 30 are errors** (the fields can encode 288 and 32; zlib refuses
  them and no encoder writes them). The code-length code must be a complete code (or have no symbols, in which case the first
  code-length symbol fails). Repeat code 16 with no previous length, and any repeat running past HLIT + HDIST, are errors. The
  literal/length code must give the end-of-block symbol (256) a length.
- Huffman codes (`DeflateHuffmanTable.TryBuild`): over-subscribed codes are errors; incomplete codes are errors **except** a
  single code of length 1 and a code with no symbols (a block with no distance codes is legal while it has no matches).
  Code lengths above 15 cannot occur (3.2.7) and are refused by the public builder. Literal/length symbols 286 and 287 and
  distance symbols 30 and 31 may take part in a code but are errors when decoded.
- A match distance reaching before the start of the unit's output (or its preset dictionary) is an error.

## zlib (RFC 1950)

- CM must be 8, CINFO at most 7 (window up to 32 KiB), and CMF * 256 + FLG a multiple of 31.
- FDICT: the stream names its dictionary by the Adler-32 of the dictionary bytes. The options' `Dictionary` is used when its
  Adler-32 matches; no dictionary, or a different one, raises `CompressionDictionaryException` with the ID. The last 32 KiB of
  the dictionary are the history.
- The trailing Adler-32 (big-endian) is checked against the output unless `VerifyChecksum` is false.

## gzip (RFC 1952)

- ID1 ID2 = 1F 8B, CM = 8, and **reserved flag bits (0xE0) must be zero**. FEXTRA, FNAME and FCOMMENT are read in bulk. FNAME and
  FCOMMENT are kept as bytes (ISO 8859-1 by the RFC) without their zero terminator; each field is limited to `MaxHeaderFieldSize`
  bytes (default 65535; `CompressionLimitException`). FHCRC is the low 16 bits of the CRC-32 of all header bytes before it and is
  checked unless `VerifyChecksum` is false.
- The trailer's CRC-32 and ISIZE (length modulo 2^32) are checked unless `VerifyChecksum` is false.
- The header fields are available through `DeflateDecoder.CurrentGZipHeader` (and the stream's) once the header is consumed.

## Limits

- `MaxOutputSize` (default 1 GiB) counts output since construction or `Reset`. The decoder is given a destination shortened to
  what the limit still allows and throws `CompressionLimitException` the moment it needs to write one more byte; nothing is
  written past the limit. Tier 1 grows its output buffer by doubling from a size taken from the input (the gzip ISIZE only when it is at most
  1100 times the input) and never allocates more than the limit lets through.
- Memory: a decoder owns at most a 64 KiB window, a few hundred bytes of tables and the gzip header fields; all of it is allocated
  on first need and reused. No allocation per call in steady state.
- No recursion; work is linear in input and output.

## Errors

`CompressionFormatException` with `Format` "deflate", "zlib" or "gzip"; `Offset` is the byte offset in the compressed input
of the position when the problem was found (for truncation, the end of the input); `IsTruncated` is set when the input ended inside
a unit. After an exception the decoder refuses calls until `Reset`.

## Window and speed notes

The decoder writes into the caller's buffer and keeps the last 32 KiB of each call's output in a window (a copy of at most 32 KiB
per call) so matches can reach into earlier calls. The hot loop (`DeflateFastLoop`) runs while 16 input and 280 output bytes remain:
one 8-byte load tops the bit buffer up to at least 56 bits per symbol, enough for a literal/length code (15 bits), its extra
bits (5), a distance code (15) and its extra bits (13) together. Entries of the lookup tables carry the length or distance base
and extra-bit count, so a symbol is one or two table reads. The low byte of an entry is the code length plus its extra bits, so one
shift drops both and the extra bits are read from the bits before the shift (the code length sits in bits 8 to 11). Tables have a
10-bit (literal/length), 8-bit (distance) or 7-bit (code-length) first level and second-level tables for longer codes.

Choices that make the loop fast without changing what it decodes:

- **The next symbol's refill and table read are issued before a match is copied**, since they do not depend on the copy; the loop
  therefore needs 16 input bytes in hand (two refills) rather than 8.
- **The literal/length table keeps a second copy for the loop** (`FastEntries`). Its kind bits are one flag per kind (literal 8,
  length 4, link 2, end of block 1 in bits 12 to 15), so each test is one bit test; the plain table and `Lookup` keep
  `DeflateSymbolKind`. In the copy, a first-level literal entry whose code is followed, within the same ten index bits, by a complete
  literal code stands for both literals (one two-byte store). After the first literal of a refill the loop decodes up to two more
  first-level literals before refilling. Making these follow-on literals branch-free (masking a non-literal entry to nothing) was
  measured slower on binary data (silesia) and faster only by 1% on text, so they are ordinary branches.
- **Matches copy 16 bytes at a time** when the distance is at least 16 (two unconditional 16-byte copies, a loop beyond 32 bytes), eight
  at a time for distances 8 to 15, and by pattern for shorter distances. The output margin is the longest match plus the width of
  the last copy that may run past it.
- **Table building** visits only the first-level prefixes that have codes longer than the first level (a list is kept while sizing),
  and fills unused entries only for an incomplete code, since a complete code leaves none.
- A first-level size of 9 or 11 bits for the literal/length table was measured slower on binary data or on all data respectively; 10 stays.

A match that reaches back before the call's output ends the loop, and the caller copies it from the window and re-enters, so the loop makes no
calls. None of this changes what a caller can observe, since the slow path and `Lookup` read the plain table.

## Tests and data

`testdata/deflate/generate.py` writes the vectors (zlib through Python, gzip and pigz) with a manifest of output hashes;
the bit-flip, prefix and limit tests, the BCL cross-checks and the optional interop tests (`ONEARCHIVE_INTEROP=1`) live in
`tests/OneCompress.Tests/Deflate/`.


<!-- https://recursiveai.org/onearchive/docs/deflate64-encoder/ -->

# Deflate64 encoder

Normative notes for `Deflate64Encoder`: what a later port must do to reproduce the same bytes. The symbols and limits are in
`deflate64-derivation.md`.

## Structure

- A buffer of 64 KiB of history plus a 128 KiB chunk. Input is parsed only when the buffer is full, on `Flush` and on the final
  call, so the output depends on the input bytes, the level and where flushes fall, never on how the caller splits its calls.
  When the buffer is full, the last 64 KiB stay as history and the rest is dropped.
- Matches: lengths 3 to 65538 (lengths of 258 and above always use symbol 285 with 16 extra bits), distances 1 to 65535.
  The encoder never emits distance 65536 or symbol 284 with extra 31 (both are accepted by the decoder, fact F3/F4).
- Match finders, by level (below):
  - Levels 1 to 4: a hash chain keyed by the first three bytes (multiplicative hash `(b0 | b1<<8 | b2<<16) * 2654435761 >> 16`,
    a 65536-entry head table, one `ushort` distance link per window position). A chain walk ends at distance 65535, at a
    zero link, or after the level's chain count. After a match, only the last 64 of its positions are entered into the finder.
  - Levels 5 to 9: the binary-tree finder of the Deflate encoder (`DeflateBinaryTreeMatchFinder`) over a 64 KiB window: trees
    keyed by a 16-bit hash of 4 bytes, plus a table of the latest position for each 16-bit hash of 3 bytes. A search reports,
    for every length from 3 up, the nearest distance that reaches it. Positions inside a taken long match are all entered.

## Levels

Level 0 stores (RFC 1951 stored blocks of at most 65535 bytes). Levels 1 to 9 are in four classes, which follow the classes of
the reference tool's `-mx` levels: 1 to 4, 5 and 6, 7 and 8, and 9 (7-Zip gives identical output within a class, and so do we).

| Levels | Parse | Depth | Nice length | Max lazy | Good length | Passes |
|---|---|---|---|---|---|---|
| 1 to 4 | lazy, hash chain | 16 | 32 | 16 | 8 | |
| 5, 6 (default 6) | optimal, binary tree | 16 | 64 | | | 1 |
| 7, 8 | optimal, binary tree | 32 | 128 | | | 1 |
| 9 | optimal, binary tree | 256 | 257 | | | 3 |

**Lazy parse (1 to 4).** Try the next position while the current match is shorter than Max lazy, searching with a quarter of
the chain when the current match is at least Good length; a longer match there wins and the current byte becomes a literal. A
search stops early at a match of at least Nice length. Matches of three bytes are taken at any distance.

**Optimal parse (5 to 9).** Work proceeds in runs of 4096 positions. Every match the finder reports at each position of the run
is collected (a search stops descending a tree at Nice length, and at Depth nodes). A match that reaches Nice length ends the
run: it is extended to its full length (up to 65538) and taken whole, and every position inside it is entered into the finder.
Otherwise the cheapest path through the run's positions (a shortest path over literals and matches of 3 to 257 bytes) is found
by dynamic programming over a price list in sixteenths of a bit. The first run is priced as the fixed Huffman code; later
runs are priced by `-log2` of the symbol's share of the statistics so far (a symbol with count c of total T costs
`(log2(2T + n) - log2(2c + 1))` bits, at least 1/16 bit, with `log2` by integer squaring to eight fractional bits and n the
alphabet size; the statistics are halved and the last run's counts added after each run). With more than one pass, the run is
parsed again with prices from the previous pass's choices (the run's counts plus a quarter of the earlier statistics). A match
length 258 or more is priced as symbol 285 with 16 extra bits only when taken whole as above.

The numbers are this library's choice, not 7-Zip's.

## Blocks

A block is closed at 32768 tokens, at the end of a parse round, or when the mix of symbols drifts: every 512 tokens (at the
end of each run for levels 5 to 9) the block's counts are given to `DeflateBlockSplitter` (at least 2048 tokens in a block, a
difference threshold of 300 per mille), which asks for a split when a new Huffman code would pay for its header. Each block is written as the cheapest of stored, fixed and
dynamic (a tie prefers fixed, then dynamic, then stored). Dynamic codes are length-limited Huffman codes from
`DeflateHuffmanBuilder` (limit 15 for literals/lengths and distances, 7 for code lengths), with HLIT at most 286 and HDIST at
most 32, at least two symbols in each alphabet, and the code lengths run-length coded with symbols 16, 17 and 18 greedily.
`Flush` ends the pending block, then writes an empty stored block (a sync flush, byte aligned). The final call writes the
final block and pads to a byte boundary; an empty input gives the two bytes `03 00` at levels 1 to 9 and one empty final stored block at level 0.

`GetMaxCompressedLength(n)` is `n + n/4096 + 64`.


<!-- https://recursiveai.org/onearchive/docs/deflate64-decoder/ -->

# Deflate64 decoder

Normative notes for `Deflate64Decoder`. The format facts (symbols, extra bits, window, header limits) are in
`deflate64-derivation.md`, with the experiment behind each. This file records what the decoder does with them.

- **Input:** one raw stream (zip method 9 stores nothing around it). No wrapper, no checksum; the zip layer checks the CRC-32.
- **Window:** 65536 bytes of history, kept in a 128 KiB ring. A match whose distance exceeds the output produced so far is a
  format error naming the offset. Distance 65536 is accepted (fact F1).
- **Symbols:** RFC 1951 for everything not listed in the derivation. Literal/length 285 = 3 plus 16 extra bits; distance
  symbols 30 and 31 = bases 32769 and 49153 with 14 extra bits. Literal/length 286 and 287 fail only when decoded.
- **Dynamic headers:** HLIT up to 288 and HDIST up to 32 are accepted (fact F9); the code lengths must still form a complete
  or single-symbol code per RFC 1951, otherwise a format error.
- **Block type 3** is a format error. A stored block checks LEN against its complement.
- **Output limit:** `MaxOutputSize` (default 1 GiB) is checked before bytes are written; reaching it throws
  `CompressionLimitException` and writes nothing past the limit. The decoder allocates only its fixed window and tables.
- **End:** `Done` is returned after the final block; bytes after it are left unconsumed. The one-call and stream readers refuse
  them unless `AllowTrailingData` is set. A truncated stream is a `CompressionFormatException` with `IsTruncated` set,
  raised by `ThrowIfIncomplete` or by the readers.
- **Speed:** a fast loop (`Deflate64FastLoop`, built like Deflate's) runs while 24 input and 280 output bytes remain. A second bit refill follows a length whose code and extra bits leave fewer than 29 bits (symbol 285: 15 + 16 bits, then a distance code and 14 extra bits do not fit one 56-bit refill). Matches longer than 258 bytes, or reaching into the window, leave the loop and are copied by the per-symbol path, which also handles the ends of the buffers.


<!-- https://recursiveai.org/onearchive/docs/deflate64-derivation/ -->

# Deflate64: derivation and pinned facts

Deflate64 (zip method 9) has no specification we may implement from. This note records, for every difference from
Deflate (RFC 1951), the hypothesis, where it came from, the experiment that tested it, and the value the codec uses.
A value enters the code only through a public description cited here or an experiment committed beside it.

Sources:

- **A.** PKWARE APPNOTE 6.3.10, sections 4.4.3.2 (version needed 2.1 for Deflate64), 4.4.5 (method 9, "Enhanced Deflating
  using Deflate64") and 5.6.1 ("similar to Deflate but uses a sliding dictionary of up to 64K"). It states nothing else.
- **B.** Public description of Deflate64 in the encyclopedia article "Deflate" (section Deflate64): the dictionary grows from
  32 KB to 64 KB, "an extension of the distance codes to 16 bits so that they may address a range of 64 KB", and the length
  code "extended to 16-bit, so that it may define lengths of three to 65,538 bytes". It gives no symbol numbers or extra-bit
  counts; those are hypotheses H1 and H2 below, derived from the stated ranges and then tested.
- **C.** RFC 1951 for everything shared (block types, Huffman code construction, code-length alphabet, tables of length and
  distance codes 0 to 29 and 257 to 284).
- **D.** Black-box runs of the local 7-Zip 26.00 (`7z`): streams it writes, and hand-built streams it accepts or rejects.

The experiments are `testdata/deflate64/experiments.py` (using the laboratory `d64lab.py`: a bit writer, a zip wrapper and a
reference decoder with the Deflate64 parameters as switches); `experiments-output.txt` is its output, tagged E1 to E7 below.
Hand-built streams are wrapped in a zip entry with method 9 and run through `7z e -so`; "accepted, output identical" means
exit code 0 and the extracted bytes equal what the hand-built stream means under the hypothesis.

## Pinned facts

| # | Fact | Hypothesis from | Experiment | Value in the codec |
|---|---|---|---|---|
| F1 | History window | A (64K), B | E4: a match at distance 65536 after exactly 65536 bytes is accepted; E4c: after 65535 bytes it is rejected | Window 65536. A distance larger than the output so far (or the preset history) is an error |
| F2 | Length symbol 285 | B (lengths 3 to 65538 with 16 bits) | E3: lengths 258, 259, 1000, 40000, 65538 as symbol 285 plus 16 extra bits (length − 3) are accepted and decode identically; E3c: symbol 285 with no extra bits is rejected | Symbol 285 = base 3, 16 extra bits (3 to 65538) |
| F3 | Length symbols 257 to 284 | C (unchanged) | E1: 7-Zip's own streams parse with RFC 1951 tables for them; E3d: symbol 284 with extra 31 (length 258) is accepted | Unchanged: RFC 1951 bases and extra bits. 284 + 31 = 258 is accepted |
| F4 | Distance symbols 30 and 31 | B (distances reach 64 KB; the ranges that follow symbol 29) | E2: distances 32769, 40000, 49152 as symbol 30 and 49153, 60000, 65535, 65536 as symbol 31, each with 14 extra bits (distance − base), are accepted and decode identically | Symbol 30 = base 32769, symbol 31 = base 49153, 14 extra bits each (maximum 65536) |
| F5 | Wrong extra-bit count for symbols 30/31 | negative control | E2c: 13 and 15 extra bits for a distance that needs 14 are rejected or give a CRC failure | 14, as F4 |
| F6 | Distance symbols 0 to 29 | C (unchanged) | E1 (7-Zip streams), E2d (32768 as symbol 29) | Unchanged |
| F7 | Symbols 286 and 287 (literal/length) | C | E6: used in a fixed block, rejected | Still invalid when decoded |
| F8 | Block structure: stored, fixed, dynamic, BTYPE 3 | C | E6: BTYPE 3 rejected; E6: a lone empty fixed block (`03 00`) accepted; E1: 7-Zip's streams use the RFC 1951 headers and code-length alphabet | Unchanged |
| F9 | Dynamic header limits | none in A or B: tested directly | E5: HDIST 2, 30, 31 and 32 accepted; HLIT 258, 286, 287 and 288 accepted; E5b: lengths given to symbols 286 and 287, unused, accepted. E1: 7-Zip itself writes HDIST = 32 (and 23 to 31 at levels 7 and 9) | The decoder accepts HLIT up to 288 and HDIST up to 32, so symbols 30 and 31 can be coded; a code that gives 286/287 a length is accepted and the symbols fail only if decoded. The encoder writes HLIT at most 286 and HDIST at most 32 |
| F10 | Distance symbols 30/31 in a dynamic block | B | E5: symbols 30 and 31 used with HDIST 31 and 32 after 40000 and 60000 bytes of history, accepted | As F4 |
| F11 | Fixed-code block | C (fixed distance codes are 5 bits) | E2 to E4 use fixed blocks with distance symbols 30 and 31 as 5-bit codes 30 and 31 | Distance symbols 30 and 31 are valid in fixed blocks |
| F12 | Zip headers for method 9 | A (version needed 2.1) | E7: 7-Zip writes version needed 21, general purpose flags 0 at `-mx=1` and `-mx=9`, no extra field; made-by 0x033F; an empty file is stored (method 0), not method 9 | The writer sets version needed 21 and flags 0 for method 9 |
| F13 | Longest match an encoder may use | F2 | E3 | 65538. (7-Zip's own encoder was seen to emit lengths of at most 257, E1; that is its choice, not a limit) |

## What 7-Zip's encoder does (observed, not required)

E1 parses `7z a -tzip -mm=Deflate64 -mx=1,3,5,7,9` output for four inputs with the pinned parameters and with plain
Deflate parameters. On text and on data with far matches the Deflate64 parameters reproduce the input exactly and plain Deflate
fails on symbol 30 or 31; on long runs (all distances at most 2) both parameter sets succeed. Levels 1 and 3 give identical
output on the text input. Random data is stored (method 0). The longest length seen is 257 and the farthest distance 65534;
symbol 285 was not emitted by 7-Zip in these inputs, so F2 rests on E3 (7-Zip's decoder) and B, not on 7-Zip's output.

## Gaps

None open. F2's 16 extra bits and F4's 14 are fixed by E2 and E3 together with the ranges in B: 3 + 65535 = 65538, and
49153 + 16383 = 65536. Details not tested (and so not relied on): behaviour of 7-Zip when a stream continues past its final
block, and a Deflate64 stream in a gzip or zlib wrapper (not a format we offer).

## Choices beyond the experiments

- The decoder follows 7-Zip's acceptance in F9 (HLIT up to 288, HDIST up to 32) because the interoperability target is 7-Zip's
  decoder; the encoder stays inside the narrower range.
- The Deflate64 decoder shares the Deflate decoder's structure (bit reader, Huffman table builder, window) with the Deflate64
  tables in place of the Deflate ones; no behaviour of the plain Deflate decoder changes.


<!-- https://recursiveai.org/onearchive/docs/lzma-encoder/ -->

# LZMA, LZMA2, xz and lzip encoders

Normative and language-neutral: a port that follows this text produces the same bytes. The LZMA specification, the xz file format
specification and the lzip manual fix the streams a decoder accepts; everything they leave to the encoder (the match finders, the
parse, the price model, the chunking, the preset parameters) is fixed here. Output depends only on the input bytes, the options and,
for LZMA2 and xz, where flushes fall. It never depends on the CPU, the runtime, thread counts or how the caller splits the input into
calls. Byte-identical output with the `xz` and `lzip` tools is not a goal: the streams are valid and the ratio is at parity.

## API shape

- Tier 1: `Lzma.Compress/TryCompress` (the `.lzma` format), `Lzma2.Compress` (raw LZMA2 plus its dictionary property byte),
  `Xz.Compress/TryCompress`, `Lzip.Compress/TryCompress`.
- Tier 2: `LzmaEncoder`, `Lzma2Encoder`, `XzEncoder`, `LzipEncoder` with
  `OperationStatus Compress(source, destination, out consumed, out written, bool isFinalBlock)` (`Lzma2Encoder` also has `Flush`), and
  the write-only streams `LzmaEncoderStream`, `XzEncoderStream`, `LzipEncoderStream` (`Flush()` flushes the inner stream only; it does
  not end a chunk, because that costs ratio).
- Tier 3: `LzmaRangeEncoder`, `LzmaPrices`, `LzmaMatchFinder` with `LzmaHashChainMatchFinder` and `LzmaBinaryTreeMatchFinder`,
  `LzmaEncoderCore` (window, probabilities, the fast and the normal parser, symbol coding) and `Lzma2Chunker`. The encoders are built
  from exactly these. `XzFilter` describes the filters in front of LZMA2.

## Options

`LzmaOptions` (used by every encoder; zero or null means "the preset's"):

| Option | Default | Meaning |
|---|---|---|
| `Preset` | 6 | 0 to 9. |
| `Extreme` | false | The slower variant of the preset (below). |
| `Lc`, `Lp`, `Pb` | 3, 0, 2 | Literal context bits (0 to 8, LZMA2: 0 to 4), literal position bits (0 to 4), position bits (0 to 4). LZMA2 requires `Lc + Lp <= 4`. |
| `DictionarySize` | preset's | 4096 to 768 MiB (a binary-tree finder keeps two 32-bit links per dictionary byte, which must fit one array). |
| `NiceLength` | preset's | 2 to 273, raised to the match finder's minimum (2, 3 or 4). |
| `Depth` | 0 = automatic | Candidates examined per position. |
| `Mode` | preset's | `Fast` or `Normal`. |
| `MatchFinder` | preset's | `HashChain3`, `HashChain4`, `BinaryTree2`, `BinaryTree3`, `BinaryTree4`. |
| `WriteEndMarker` | false | `.lzma` only; a stream of unknown length always gets the marker. |

`XzCompressionOptions`: `Lzma`, `Check` (default CRC-64), `BlockSize` (default 0 = one block), `Filters` (at most three before LZMA2).
`LzipOptions`: `Level` (6), `DictionarySize`, `MatchLength`, `MemberSize` (0 = one member).

## Presets

The dictionary sizes, match finders and modes of presets 0 to 9 follow the `xz` manual (dictionary 256 KiB, 1, 2, 4, 4, 8, 8, 16, 32
and 64 MiB; `hc3` for 0, `hc4` for 1 to 3, `bt4` for 4 to 9; fast mode for 0 to 3, normal for 4 to 9). The manual leaves the nice
lengths and depths to the implementation; they were set by measuring size against `xz` on the corpus subset below, so that every
preset is within 1% of the tool's size.

| Preset | Dictionary | Match finder | Mode | Nice length | Depth |
|---|---|---|---|---|---|
| 0 | 256 KiB | HashChain3 | Fast | 6 | 6 |
| 1 | 1 MiB | HashChain4 | Fast | 8 | 8 |
| 2 | 2 MiB | HashChain4 | Fast | 24 | 16 |
| 3 | 4 MiB | HashChain4 | Fast | 48 | 24 |
| 4 | 4 MiB | BinaryTree4 | Normal | 16 | 24 |
| 5 | 8 MiB | BinaryTree4 | Normal | 32 | 32 |
| 6 | 8 MiB | BinaryTree4 | Normal | 64 | 48 |
| 7 | 16 MiB | BinaryTree4 | Normal | 64 | 48 |
| 8 | 32 MiB | BinaryTree4 | Normal | 64 | 48 |
| 9 | 64 MiB | BinaryTree4 | Normal | 64 | 48 |

Depth when not given: `16 + nice / 2` for binary trees, `4 + nice / 4` for hash chains (presets 0 to 3 use the table's depths).
`Extreme` switches presets 0 to 3 to `BinaryTree4` and `Normal` with nice length 64 and the binary-tree depth, and presets 4 to 9 to nice
length 273; the depth is at least 512 (binary tree) or 100 (hash chain).

Lzip levels (`lzip` manual, "Invoking lzip"): the dictionary size and match length of level 0 to 9 are 64 KiB/16, 1 MiB/5, 1.5 MiB/6,
2 MiB/8, 3 MiB/12, 4 MiB/20, 8 MiB/36, 16 MiB/68, 24 MiB/132 and 32 MiB/273. Level 0 uses `HashChain4` and the fast parser, the
others `BinaryTree4` and the normal parser. lc, lp and pb are 3, 0, 2.

## Window and determinism

The encoder copies input into a window with index 0 reserved (a zero byte before the stream, so the first literal's previous byte is 0).
Stream position of window index `i` is `i - 1 + slid`, where `slid` counts the bytes dropped when the window moved down.

- Capacity is `1 + history + slack + 4385`, with `history = max(dictionary, 65536) + 16`, `slack = clamp(dictionary, 64 KiB, 8 MiB)`.
  When the window is full it moves down so that `history` bytes before the next position stay; every stored position moves with it.
  Positions at or below the shift become "empty" (0).
- Parsing starts only when 4385 bytes (`4096` parse window, `273` longest match, `16` margin) follow the position, unless the caller
  drains (final block, `Flush`). So a decision never depends on how the input was cut into calls. A drain shortens the parse window near
  the end of the data, which is why flush positions are part of an LZMA2 stream's definition.
- `inputLength` and block/member sizes only change memory: a known input shortens the dictionary used for distances and for the cyclic
  arrays to the input length (no distance can exceed it), and an xz block or lzip member size caps both the dictionary and the hash table
  size to that size. Hash table sizes otherwise depend on the options' dictionary size, never on the input length.

## Match finders

All positions are window indexes; a stored 0 means empty. Distances are `position - stored`; a candidate farther than the dictionary
size is rejected. `available` is the number of bytes from the position to the end of the data; positions with fewer than the hash
width available are not inserted (the cyclic slot is cleared).

- **Hashes.** `hash2` is the first two bytes as a 16-bit number indexing a table of 65536 entries. `hash3` (finders with a 4-byte main
  hash) is `((v & 0xFFFFFF) * 2654435761) >> 14` into 2^18 entries, where `v` is the next four bytes little endian (computed in 32-bit
  unsigned arithmetic). The main hash is `(v * 2654435761) >> (32 - bits)` for 4 bytes, `((v & 0xFFFFFF) * 2654435761) >> (32 - bits)` for 3,
  and the 16-bit direct value for `BinaryTree2`; `bits = clamp(32 - lzcnt(dictionary - 1), 16, 24)`.
- **Short matches.** From `hash2`: if the stored position is within the dictionary and its first two bytes equal, a match of length 2
  is reported. From `hash3`: the common length with that position is measured (it can be 0 to 273) and reported if longer than what is
  known so far.
- **Hash chain.** A cyclic array of `dictionary + 1` links; the new position links to the old head of its main hash. A search walks
  `depth` links, skipping candidates whose bytes at the current best length differ, and reports each candidate that is longer than the best
  so far. It stops at the nice length (or `available`).
- **Binary tree.** Two links per position (smaller and greater subtree) in a cyclic array of `2 * (dictionary + 1)`. Inserting a
  position walks down from the main hash's old head for at most `depth` nodes, comparing from the shorter of the two matched prefixes
  `len0` and `len1`, reporting every node that beats the best length, and re-hanging the visited nodes on the new node's two sides. When
  the match reaches the nice length, the new node takes over the found node's two subtrees and the walk stops. `Skip` does the same
  walk without reporting.
- The lengths reported are strictly increasing; each is the closest distance found for it.

## Symbol coding

The probability layout is `LzmaModel`'s; everything is as in the LZMA specification's decoder, run backwards. The range encoder keeps a
33-bit `low`, a 32-bit `range`, a cache byte and a pending count: the first byte written is always 0, and `Flush` shifts out five
bytes. An end marker is a match of length 2 with distance field `0xFFFFFFFF`. Matched literals are used after a match or repeated match
(state at least 7) with the byte at `rep0`. Repeated-distance indexes rotate as in the specification.

## Prices

Prices are in 1/16 bit. The table has 128 entries, entry `i` for a probability `p` in `16 i` to `16 i + 15`: `round(16 * log2(2048 / (16 i + 8)))`,
computed by integer repeated squaring (the binary logarithm in fixed point with 8 fraction bits, rounded half up when scaled to 1/16);
the price of a 1 bit is the table entry of `2048 - p`. No floating point is used. The generator is `LzmaPrices.BuildTable`.

Derived tables are rebuilt from the current probabilities when `_age` (the number of symbols coded since the last rebuild) reaches 2048
in fast mode or 256 in normal mode, and after every state reset: 4 x 64 slot prices; 4 x 128 prices of distances below 128 (slot plus
low bits); 16 align prices; for the match and the repeated-match length coders and each position state, 272 length prices. A distance
of 128 or more costs its slot price plus 16 per direct bit plus its align price.

## Parsers

Both parsers run on a *horizon*: at most 4096 bytes, at most the bytes left, at most the room left in the chunk (below). The choice of
symbols for those bytes is made, coded, and the next decision starts after them.

**Fast.** At position `p`: the repeated lengths of the four distances and the finder's matches are known. If neither exists, a literal
is coded (or a one-byte repeated match when the byte equals the one at `rep0`). Otherwise each candidate (every repeated distance with
length 2 or more, every finder match with length clamped to the horizon) is priced as a fresh symbol, and the one with the largest
saving `length * literal - price` wins (a later match wins a tie, a repeated distance only wins by being strictly better), where
`literal` is the price of a literal at `p` including the is-match flag. A winner of at least the nice length is coded at once. Otherwise
the same choice is made at `p + 1` (after a literal; matches at `p + 1` come from the finder, repeated lengths use the current distances);
if coding a literal and then that candidate is cheaper than the first candidate over the same span (the shorter plan is padded with
literals at `literal` each), a literal is coded now and the decision is retaken at `p + 1`.

**Normal.** A shortest-path search over the horizon. Node `i` holds the cheapest price to reach `p0 + i`, its predecessor, the symbol
that was used, and the state and four repeated distances after it. Edges from node `i`: a literal (matched-literal pricing when the
state is 7 or more); a one-byte repeated match if the byte equals the one at `rep0`; for each repeated distance with length `l >= 2`,
every length 2 to `l`; for each finder match, every length from the previous match's length plus one up to its own. The search visits
nodes in order, asking the finder for matches at each, until the last reachable node is passed. A repeated or finder match that reaches
the nice length ends the search: that edge is taken unconditionally and its target is the end. Nodes beyond the horizon are never created. The path to the end node is coded
symbol by symbol and the finder is advanced (skipping) to the end. Pricing the same symbol twice never uses information that depends on
how many bytes were appended: all lengths are clamped to the horizon.

## Fast and normal modes with budgets

The caller (the LZMA2 chunker) limits how much output a call may produce. Before each decision the horizon is reduced to
`(limit - pending - 24) / 10` input bytes, where `pending` is the range encoder's delayed-plus-flush size. 10 is more than the worst
output per input byte (a 2-byte match with the least probable bits throughout is about 20 bytes), so a chunk can never pass its
limit. Coding stops when that horizon is below 1 (the budget is exhausted).

## LZMA2 chunks

A chunk is closed when it holds 2 MiB of input, when the budget above is exhausted (64 KiB of output), or when the caller drains. A
chunk left open when input runs out stays open: the next call continues it, so chunk boundaries do not depend on how input arrives.
When a chunk is closed, the range encoder is flushed. If `coded + header >= input + 3 * ceil(input / 65536)` the input is written as
stored chunks of at most 64 KiB (the first of the stream with control 1, the rest 2), and the next LZMA chunk resets the state. The window keeps at least 64 KiB of
history so the stored bytes are always available. Otherwise it is an LZMA chunk with control `0x80 | reset << 5 | (size - 1) >> 16` where
`reset` is 3 for the first chunk of a stream (dictionary, state and properties), 2 for the first LZMA chunk after a stored first
chunk (properties), 1 after a stored chunk (state), else 0; then the low 16 bits of the size minus one, the compressed size minus one, and
the properties byte when `reset >= 2`. A reset of the state restores the initial probabilities, state 0 and repeated distances 0. The
stream ends with control byte 0. The dictionary property byte is the smallest of 0 to 40 whose size is at least the dictionary size.

## `.lzma`

13-byte header: properties byte `(pb * 5 + lp) * 9 + lc`, dictionary size (the options' size, as a 32-bit little-endian number),
uncompressed size (64-bit; all ones when unknown). With a known size the end marker is written only if `WriteEndMarker`; the encoder
throws `InvalidOperationException` if the input differs from the declared length.

## xz

Stream header, blocks, index, stream footer as in the xz file format specification. A block header never carries the optional
sizes. Filter flags are written in the order of `Filters` (applied first to last when compressing), then LZMA2 (ID 0x21, one property
byte). Branch filters write their start offset (four bytes) only if it is not 0; delta writes `distance - 1`. Each block restarts the
dictionary, the filters (positions count from the filter's start offset) and the check. Bytes a branch filter holds back at the end of a
block are passed on unconverted. The check covers the block's original data. Block padding is zeros to a multiple of 4, and the index
lists `(header + compressed + check size, uncompressed size)` per block. Empty input gives a stream with no blocks. `BlockSize` fixes
the block boundaries (they are at exact multiples of it); with 0 the whole input is one block.

## lzip

Header `LZIP`, version 1, dictionary byte; the LZMA stream with lc/lp/pb 3/0/2 ends with the end marker; trailer CRC-32, data size and
member size (little endian). The dictionary byte is `log2 | wedges << 5` for the smallest `2^log2 - wedges * 2^(log2 - 4)` (log2
12 to 29, wedges 0 to 7) that is at least the dictionary size. `MemberSize` cuts the input into members of that many uncompressed bytes;
empty input is one empty member.

## Results against the reference tools

Measured with `xz 5.8.3` and `lzip 1.21` as black boxes (one-process CPU time for speed, so direction only), compressed size of this
library divided by the tool's, on dickens, mr, xml, ooffice, osdb, reymont, x-ray (Silesia) and the first 10 MB of enwik8.

`.lzma` against `xz --format=lzma -N` (preset N, `e` is extreme):

| Level | Ours (bytes) | Tool (bytes) | Size ratio | CPU time ratio |
|---|---|---|---|---|
| 0 | 25,143,280 | 25,284,236 | 0.9944 | 1.53 |
| 1 | 23,732,924 | 23,550,183 | 1.0078 | 1.40 |
| 2 | 22,968,553 | 22,881,358 | 1.0038 | 1.00 |
| 3 | 22,683,224 | 22,525,190 | 1.0070 | 0.79 |
| 4 | 20,475,572 | 20,489,845 | 0.9993 | 1.16 |
| 5 | 19,920,755 | 19,928,660 | 0.9996 | 1.13 |
| 6 | 19,837,931 | 19,838,748 | 1.0000 | 1.16 |
| 7 | 19,835,425 | 19,835,719 | 1.0000 | 1.12 |
| 8 | 19,835,425 | 19,835,719 | 1.0000 | 1.21 |
| 9 | 19,835,425 | 19,835,719 | 1.0000 | 1.08 |
| 0e | 21,033,266 | 21,057,324 | 0.9989 | 1.13 |
| 3e | 19,911,504 | 19,887,305 | 1.0012 | 1.10 |
| 6e | 19,811,459 | 19,816,691 | 0.9997 | 1.26 |
| 9e | 19,808,375 | 19,813,460 | 0.9997 | 1.17 |

`.lzip` against `lzip -N`:

| Level | Ours (bytes) | Tool (bytes) | Size ratio | CPU time ratio |
|---|---|---|---|---|
| 0 | 24,244,297 | 24,947,330 | 0.9718 | 2.69 |
| 1 | 24,076,232 | 24,078,459 | 0.9999 | 1.60 |
| 2 | 22,941,796 | 22,942,576 | 1.0000 | 1.36 |
| 3 | 21,907,588 | 21,888,925 | 1.0009 | 1.24 |
| 4 | 20,906,976 | 20,919,118 | 0.9994 | 1.12 |
| 5 | 20,256,806 | 20,257,304 | 1.0000 | 1.15 |
| 6 | 19,899,103 | 19,888,561 | 1.0005 | 1.09 |
| 7 | 19,831,810 | 19,811,712 | 1.0010 | 1.13 |
| 8 | 19,815,436 | 19,788,519 | 1.0014 | 1.18 |
| 9 | 19,809,609 | 19,780,671 | 1.0015 | 1.14 |

Presets 7 to 9 give one size here because no file in the subset is larger than 8 MiB. The CPU time ratio is one process each, cold start included, on a shared machine: it shows the direction only (below 1 is faster than the tool). The bar timings are taken separately on the benchmark machine. Levels 0 and 1 are the slowest relative to the tool (1.4 to 2.7 times).

## Experiments

- Preset parameters: the `xz` manual names the match finder, mode and dictionary of each preset; nice length and depth were tuned by
  running both encoders over the corpus subset and keeping the cheapest setting within 1% of the tool's size (fast presets: nice
  4 to 48 and depth 4 to 32 were compared; 0.9 to 1.04 of the tool's size across settings).
- Fast parser: a first version that took the longest match with a preference for repeated distances was 2.5% to 5.6% larger than `xz`
  on presets 1 to 3; pricing the candidates (above) brought it to within 1%.

## Speed notes (no effect on output)

The match finders issue prefetch hints for the hash buckets of positions 16 ahead and for the node a bucket 6 positions ahead holds,
and the tree walk prefetches both children of the node it is at. Hints change no table and no result. Reads in the walk, the
hash updates and the chain steps skip bounds checks behind `LzmaMatchRaw` (bounds-checked in the `Checked` build); every entry
point first checks that `position + min(available, 273)` lies inside the buffer, and the walk accepts a stored position only when
it lies behind the current one and within the dictionary.


<!-- https://recursiveai.org/onearchive/docs/lzma-decoder/ -->

# LZMA, LZMA2, xz and lzip decoders

`RecursiveAI.OneCompress.Lzma`, `.Xz`, `.Lzip` and `.Filters`. The formats are specified by the LZMA specification
document (`lzma-specification.txt`), the xz file format specification (`xz-file-format.txt`) and the lzip manual's format
chapter. This document records only what those texts leave open, and how the decoders behave. A later port must reproduce
the decoder's accept/reject behaviour from this text and the specifications.

## Layers

| Tier | Types |
|---|---|
| 3 (raw) | `LzmaRangeDecoder`, `LzmaModel` (probability layout, state machine), `LzmaDecoderCore` (symbol decoder and window), `BranchFilter`, `DeltaFilter` |
| 2 | `LzmaDecoder` (`.lzma` alone format, and `CreateRaw` for a bare stream with known properties), `Lzma2Decoder`, `XzDecoder`, `LzipDecoder`: `OperationStatus Decompress(source, destination, out consumed, out written)`; `LzmaStream`, `XzStream`, `LzipStream` |
| 1 | `Lzma.Decompress`, `Lzma2.Decompress`, `Xz.Decompress`, `Lzip.Decompress` and the `TryDecompress` forms |

The filters are standalone span transforms that work in both directions, so an encoder shares them.

## Defaults and limits

| Option | Default | Meaning |
|---|---|---|
| `MaxDictionarySize` | 64 MiB | A header declaring more raises `CompressionLimitException` before memory is reserved. 64 MiB is the dictionary of the reference tool's `-9`. Larger streams (`xz --lzma2=dict=1536MiB`) need it raised. |
| `MaxOutputSize` | 1 GiB | Total output of a decoder instance (all streams and members). A declared size above it is rejected before decoding; output reaching it raises `CompressionLimitException`. |
| `AllowTrailingData` | false | Bytes after the last stream, member or `.lzma` end: error when false. The span decoders never look at them and report `Done` with the bytes unconsumed. |
| `VerifyChecks` (xz) / `VerifyChecksum` (lzip) | true | Verify CRC32, CRC64, SHA-256 / the lzip CRC-32. When false the check is still skipped over and the sizes in the trailer and index are still verified. |
| `AllowUnsupportedChecks` (xz) | false | Check IDs other than 0, 1, 4 and 10 are reserved. They are rejected unless this is set; then the block checks, whose size follows from the ID (4 bytes for 1-3, 8 for 4-6, 16 for 7-9, 32 for 10-12, 64 for 13-15), are skipped unverified. |

Memory is the declared dictionary at most (the window grows with the output, starting at 64 KiB and doubling), plus 32 spare bytes behind the ring (a match copy rounds up to whole 16-byte steps; the ring is the dictionary size plus 32, so the overshoot lands on bytes too old for any match to reach), plus the
probability tables (`0x300 << (lc + lp)` 16-bit entries plus a fixed part; at most about 6 MiB, since the `.lzma` format allows
lc up to 8 and lp up to 4), plus 64 KiB of
chain buffer when an xz block has filters. Steady-state decoding allocates nothing.

## LZMA

- The `.lzma` header is 13 bytes: properties byte `(pb * 5 + lp) * 9 + lc`, a 32-bit little-endian dictionary size, and a 64-bit
  size (all ones = unknown). A dictionary size below 4096 is raised to 4096. A properties byte above 224 is an error.
- With an unknown size the stream must end with the end marker (distance 2^32 - 1). With a known size an end marker before
  that size is an error. At the declared size the stream ends if the range decoder is finished (code value zero); otherwise an
  end marker must follow (the specification allows both), and anything else is an error. A match that would run past the
  declared size is an error.
- A match whose distance exceeds the bytes produced so far, or the dictionary size, is `CompressionFormatException` at the
  input offset of the symbol.
- The range decoder's first byte must be 0, as the specification requires.
- **Split input.** A symbol reads at most about 20 bytes. With 32 bytes of input available the symbol decodes directly; with
  less, the decoder journals the probability updates, and if the input ends inside a symbol it rolls the symbol back and
  carries the unread bytes inside the decoder, so `NeedMoreData` means all input was consumed. Any split of the input gives the
  same output and the same errors. A match that does not fit in the destination is finished by the next call.

- **Fast loop.** While at least 32 bytes of input remain at the start of a symbol, `LzmaDecoderCore` decodes with the range
  coder state in locals and no undo log (`LzmaDecoderCore.Fast.cs`); literal and tree bits are decoded without a branch on the
  bit. The last few symbols of an input piece use the journalling loop. Both loops make the same decisions, so the output and
  the errors do not depend on where the input is cut. Probability and input reads in this loop and the wide match copy skip bounds checks behind the accessors in `LzmaRaw`
  (bounds-checked in the `Checked` build): a symbol starts only with 32 input bytes left and consumes at most 25, and every probability
  index is a fixed offset of the model layout plus a state or tree node inside its table. Literals are decoded in their own methods and
  the bit trees are unrolled so that the range coder state stays in registers; none of this changes a decision. Details that only
  affect speed: range and code are held as 64-bit values (both stay below 2^32), tree nodes are read through bounds-checked views of
  the probability array, a matched literal reads the probabilities of both possible next nodes before the current bit is known,
  the state transitions are arithmetic or a table lookup rather than branches, and the symbol methods are marked for full
  optimization so the runtime's profile-guided tiering does not replace them.

## LZMA2

- Chunks as in the xz specification's LZMA2 description: control 0 ends; 1 and 2 are uncompressed chunks (1 resets the
  dictionary); 0x80 and above are LZMA chunks, bits 5-6 choosing the reset level (none, state, state + new properties,
  all + dictionary). Sizes are 21 bits of unpacked and 16 bits of packed size, each plus one.
- Rules the decoder enforces (violations are `CompressionFormatException` naming the offset): the first chunk resets the
  dictionary; after a dictionary reset the first LZMA chunk carries properties (control >= 0xC0); an LZMA chunk that follows
  an uncompressed chunk needs at least a state reset; control values 3-0x7F are invalid; `lc + lp <= 4`; the LZMA data of a chunk
  uses exactly its packed size and produces exactly its unpacked size, and the chunk's range decoder must end clean.
- The dictionary property byte is 0-40: size `(2 | (b & 1)) << (b / 2 + 11)`, and 40 means 4 GiB - 1. Above 40 is an error.

## xz

- Stream header and footer: magic, flags (check ID in the low nibble of the second flag byte; all other bits zero), CRC-32 of
  the flags; the footer repeats the flags and holds the backward size. Both are compared.
- Block header: size byte `(n / 4) - 1` (a zero size byte is the index), flags (filter count 1-4, bits 6 and 7 mark the
  optional compressed and uncompressed sizes), variable-length integers (at most 9 bytes, no redundant zero continuation), filter
  IDs and property sizes, zero padding to a multiple of 4, CRC-32. A reserved flag bit, a non-zero padding byte or a bad CRC is an error.
- Filter chain rules: the last filter is LZMA2 (0x21) and no other filter may be last; LZMA2 may appear only last; at most
  three non-last filters; the non-last filters must be branch (0x04-0x0B) or delta (0x03) filters. Anything else is
  `CompressionFormatException("unsupported filter")`.
- Filters: x86 0x04, PowerPC 0x05, IA-64 0x06, ARM 0x07, ARM-Thumb 0x08, SPARC 0x09, ARM64 0x0A, RISC-V 0x0B. A branch filter's
  properties are empty (start offset 0) or a 4-byte little-endian start offset aligned to the architecture. Delta's property
  is one byte, distance - 1. Filters run in reverse order of the chain, in place over a 64 KiB buffer.
- A block's compressed size field (when present) must match what LZMA2 consumed, and the uncompressed size field must match the
  output. After the LZMA2 end marker come 0-3 zero padding bytes to a multiple of 4, then the check.
- The index lists the (unpadded size, uncompressed size) record of each block. The decoder keeps the same pairs as it decodes and
  compares them with the index, and the backward size with the index's real size. The index's own CRC-32 is verified. (The
  pairs are hashed with CRC-64 for the comparison; this is internal and affects nothing a caller can see.)
- Multiple streams may follow each other, each optionally followed by zero padding in multiples of 4 bytes. Anything else
  after a stream is trailing data (error unless `AllowTrailingData`).
- Checks: none (0), CRC-32 (1), CRC-64 (4, ECMA-182 polynomial reflected), SHA-256 (10, from the BCL). CRC-32 and CRC-64 come from the
  library's checksum primitives.

## Branch and delta filters

Both are exact inverses of the reference tool's encode direction and work on any split of the data: `Convert` returns the
number of leading bytes that are final and the caller presents the rest again with more data behind it. At the end of the
data the last bytes (under 4, 8, 16 depending on the architecture) stay unconverted, which is what the reference encoder does.
The x86 filter carries its previous-mask state across calls; the others are stateless apart from the position.

- x86: E8/E9 with a 32-bit operand whose top byte is 00 or FF, and the prefix mask of the previous 5 bytes, as in the public
  descriptions of the BCJ x86 filter.
- PowerPC, ARM, ARM-Thumb, SPARC, ARM64, IA-64: as in the public descriptions; ARM64 converts BL (26-bit word offset) and ADRP
  (21-bit page offset, only when it is small enough to look like code).
- **RISC-V** has no public description other than the xz format list. The rules were derived by running `xz --riscv` as a black box over
  hand-made instruction sequences (the derivation is described in the comment above `BranchFilter.RiscV`): JAL with a link
  register x1 or x5 becomes an absolute target; AUIPC followed by an instruction using the same register is converted as a
  pair, stored as a marker AUIPC (register x2) holding the low bits of the second instruction followed by a big-endian
  absolute target; an original AUIPC that looks like a marker is stored escaped. Decoding reverses both.
- Delta: `out[i] = in[i] + out[i - distance]` modulo 256, distance 1-256, history kept across calls.

## lzip

- Header: `LZIP`, version 1 (version 0 is rejected as unsupported), dictionary-size byte: `2^(b & 31)` minus
  `(2^(b & 31) / 16) * (b >> 5)`; the size must be 4 KiB to 512 MiB.
- Body: LZMA with lc 3, lp 0, pb 2, ending with the end marker. Trailer: CRC-32 of the data, data size, member size (each little
  endian, 4 + 8 + 8 bytes); all three are checked, the member size even with checks off.
- Members follow each other. Trailing bytes that are not a member header are an error unless `AllowTrailingData`.

## Errors

`CompressionFormatException` (format name, input offset, what was expected), `CompressionLimitException` (the limit's name,
its value, the requested value), both under `CompressionException`. Offsets are absolute input offsets of the field
that failed, counted from the start of the data given to the decoder. Truncated input ends in a format error naming the offset
where more data was expected, through `ThrowIfIncomplete` or the one-call methods and the streams.

## Tests

`testdata/lzma/generate.py` writes the vectors with the xz, lzip and 7-Zip tools (versions recorded in `manifest.tsv`). The
hash matrix (`lzma-decode`, `lzma-filter` entries) decodes the smaller vectors one-shot, through a stream, in pieces and with bit
flips, and runs both filter directions; the outputs and error texts must be equal on every platform. Fuzz targets: `lzma-decode`,
`xz-decode`, `lzip-decode`.


<!-- https://recursiveai.org/onearchive/docs/bzip2-encoder/ -->

# bzip2 encoder

`RecursiveAI.OneCompress.BZip2`: compression to bzip2 streams. The format is described in `bzip2-decoder.md`; every choice the format
leaves to the writer, with the experiments that pinned or bounded it, is in `bzip2-encoder-derivation.md` (fact numbers F1…F9 below). A later
port must reproduce the same bytes from these two documents.

## Options

- `Level` 1–9, default 9: the block size level in the header; blocks hold at most `Level × 100 000 − 19` first-stage bytes (F1).
- `MaxCodeLength` 9–20, default 17: the longest Huffman code (F5). The format allows 20; the tool writes at most 17.

The output depends only on the input bytes and these two options, never on how the input is cut into calls, the output buffer size,
the platform, vector support or the number of threads (there is one).

## The stream written

`BZh` + the level digit; one block per full block of input and one for the remainder; the end marker with the combined CRC; zero
bits to a byte boundary. Empty input gives the 14-byte stream with no block (F7). Several streams are made by concatenating outputs.

## Per block

1. **First stage** (`BZip2RunLengthEncoder`). Runs of equal bytes: a run of 1–3 bytes is written as is; a run of 4–255 is the byte four
times and a count byte of the length minus 4 (F2). Longer runs are cut at 255 and the rest starts a new run, even in the same byte. The block
capacity is `Level × 100 000 − 19` bytes of output; a unit (a literal, or a run with its count) that does not fit starts the next block, so a
run in progress at the end of a block is written out with its count and the next block begins afresh (F3). The block CRC is the CRC of the input
bytes of the block (F7); the combined CRC is `rotl(combined, 1) xor blockCrc`.
2. **Sort** (`BZip2BlockSorter`). The last column of the sorted cyclic rotations of the block, and the row of rotation 0 (F6). Ties between
equal rotations (a periodic block) go by rotation index, so the lowest index comes first and the origin is its row. The sorted order is
fully defined by that rule, so any correct sort gives the same bytes; the method below is a speed choice, not a format fact.
Method: the rotations are bucketed by their first two bytes (the block is read as cyclic), and each bucket is sorted on the five bytes after
them, with the rotation index in the low 24 bits of the same 64-bit word as the key (large buckets are first split on one key byte at a time,
smaller ones go to a quicksort with a branch-free partition). Rotations tied on those seven bytes go to prefix doubling over group ranks
(Manber and Myers; Larsson and Sadakane): a group tied on `h` bytes is sorted by (rank of the rotation `h` bytes on, rotation index), which
orders it on `2h` bytes, and only groups that are still tied are revisited as `h` doubles.
3. **Move-to-front and zero runs** (`BZip2BlockWriter`). The list starts as the bytes in use, ascending. A move-to-front position 0 repeated
*n* times is *n* in bijective base 2, least significant digit first, `RUNA` (0) for digit 1 and `RUNB` (1) for digit 2; any other position *p*
is symbol *p* + 1; the block ends with symbol `used + 1` (F8). A run still open at the end is written before it. The first sixteen list positions are kept in a vector register while a block is coded; this computes what a position-by-position search and shift would.
4. **Tables** (`BZip2TableChooser`, `BZip2HuffmanBuilder`). Symbols are split in groups of 50 (the last may be shorter), one selector per
group, ceil(symbols / 50) in all (F4). The number of tables is 2 for fewer than 200 symbols, 3 below 600, 4 below 1 200, 5 below 2 400, else 6.
The groups start dealt to the tables in runs of consecutive groups (group *g* of *G* to table *g × k / G*, integer division), each table's
lengths are built from its groups' symbol counts, and six passes follow: each group goes to the table with the least total code length for
its symbols (ties to the lower number), then each table is rebuilt from the groups it received. A pass in which no group changes table ends the passes early (the next ones would repeat it); a table's lengths are rebuilt only when a group moved into or out of it. Tables that no group uses are removed and the
remaining tables keep their order; if only one is left it is written twice. Lengths: Huffman's construction with every count raised to at
least 1, leaves ordered by (count, symbol), a merge taking the leaf on equal weights; if a length passes `MaxCodeLength`, all counts
become `(count + 1) / 2` and the tree is rebuilt. Codes are canonical (`BZip2HuffmanTable`).
5. **Header and data.** Block magic, CRC, a zero randomised bit, the origin row (24 bits), the symbol map, the table count (3 bits), the
selector count (15 bits), the selectors as unary move-to-front indices over the table numbers (F9), each table as a 5-bit start length (the
first symbol's) and for every symbol `10` per +1 and `11` per −1 step then `0` (F5), then the symbols.

## Tiers

- **One call:** `BZip2.Compress`, `BZip2.TryCompress`. There is no tight bound on the output size ahead of time; incompressible input grows by about an eighth.
- **Spans:** `BZip2Encoder.Compress(source, destination, out consumed, out written, isFinalBlock)` and `Flush`. A call may consume
much and write nothing; a full block is coded in one go and its output (up to about 1 MB) is handed out over as many calls as the
destination needs. `Flush` hands out what is already coded and does not end the block in progress: ending a block early would change the output.
- **Stream:** `BZip2Stream(stream, BZip2Options, leaveOpen)` for writing; disposing ends the stream.
- **Parts:** `BZip2RunLengthEncoder`, `BZip2BlockSorter`, `BZip2BlockWriter` (move-to-front, runs, header, symbols), `BZip2TableChooser`,
`BZip2HuffmanBuilder`, `BZip2BitWriter`, plus the decoder's `BZip2Crc`, `BZip2MoveToFront` and `BZip2HuffmanTable`.

## Memory

At level 9: the block (900 000 bytes), the last column (900 000), the sorter (two `int` and two `ulong` per block byte, 21.6 MB, plus a copy
of the block with 15 bytes of its own beginning appended), the symbols (1.8 MB) and the output of one block (about 1 MB) — about 27 MB,
allocated on first use and kept for later blocks.

## Checked against

`bzip2` 1.0.8 decodes our output at levels 1–9 over generated inputs (empty, one byte, runs of 3, 4, 5, 255, 256, 259, 260, 1 000 bytes,
text, random, periodic, 2.5 MB of zeros, inputs of capacity − 1 … 2 × capacity + 7 and a run across a block end), over Silesia files, and
concatenated with other streams; 7-Zip, Info-ZIP `unzip`, GNU tar and bsdtar read our zip entries (method 12) and `.tar.bz2` files. A table with
lengths up to 20 is accepted by the tool. Sizes against the tool: see the unit's report; within about 0.3% on dickens, xml, mr, reymont and
ooffice at levels 1, 5 and 9.

## Speed

Measured on one core of a desktop x64 machine under load (a quick comparison, not a bar run), in memory against `bzip2` 1.0.8 on silesia.tar:
about 18 MB/s at level 9 and 21 to 23 MB/s at level 1, where the tool reaches 17 to 20 and about 21; the sizes did not change (the bytes are
the same as before the sort was rewritten). The sort takes about four fifths of the time. Highly repetitive input is the slow case: a period
of five bytes over 3 MB needs many doubling rounds.


<!-- https://recursiveai.org/onearchive/docs/bzip2-decoder/ -->

# bzip2 decoder

`RecursiveAI.OneCompress.BZip2`: decoding of bzip2 data. bzip2 has no official specification. This document records the
format as the decoder reads it, how each point was established, and every choice the format leaves open. A later port
must reproduce the decoder's behaviour from this text.

## Sources and how gaps were closed

Public descriptions of the format (the Wikipedia article's file-format section, the bzip2 manual page), the Burrows–Wheeler
paper (DEC SRC report 124, 1994) and black-box runs of `bzip2` 1.0.8 (and 7-Zip's and pbzip2's encoders for streams
written by other implementations). Two things were not in any description and were measured:

- **The randomisation schedule** (see below): `tools/bzip2/derive-randomization-table.py` recovers it from the tool.
- **Which malformed streams the tool rejects** (see "Strictness"): hand-built streams, written by the test builder in
  `tests/OneCompress.Tests/BZip2/BZip2TestBuilder.cs`, were fed to the tool and the outcomes recorded.

## Stream layout

All fields are most significant bit first. Bytes are not aligned except at the start and end of a stream.

```
stream   = "BZh" level  block*  end-of-stream
level    = ASCII '1'..'9'         block size limit = level * 100000 bytes (after the first run-length stage)
block    = 0x314159265359 (48 bits)  block-crc (32)  randomised (1)  original-row (24)
           symbol-map  table-count (3)  selector-count (15)  selectors  tables  symbols
end      = 0x177245385090 (48 bits)  combined-crc (32)  zero padding to a byte boundary
```

- **Symbol map:** 16 bits, one per group of 16 byte values, then 16 bits for each group present. The set bits, in order, are
  the bytes used in the block (`used`, 1 to 256).
- **Table count** 2 to 6. **Selector count** 1 to 32767; a block uses at most 18002, and bzip2 1.0.8 reads the rest and ignores them (checked at 18003 and 32767), so the decoder does too. Each selector is a unary number (that many 1 bits, then a 0 bit)
  below the table count, undone with move-to-front over the table numbers `0..tables-1`. Selector `i` names the table for
  symbols `50 i` to `50 i + 49`.
- **Tables:** per table, a 5-bit start length, then for each of the `used + 2` symbols: while the next bit is 1, one more bit
  (0: length + 1, 1: length - 1); a 0 bit ends the symbol. A length must be 1 to 20 whenever it is examined.
- **Symbols:** RUNA = 0, RUNB = 1, 2 to `used` = move-to-front position 1 to `used - 1`, `used + 1` = end of block. A run of RUNA
  and RUNB is a bijective base-2 number, least significant digit first (RUNA = 1, RUNB = 2, so the digits `A B` mean 1 + 2×2 = 5),
  and repeats the front of the move-to-front list that many times. The list starts as the used bytes in increasing order.
- **Codes:** canonical. Codes of the same length are consecutive in symbol order; a length's first code is twice the
  previous length's first code plus its count.
- **After the symbols:** the sequence is the last column of the sorted rotations; the original-row number picks the row that
  is the original block. Inverse transform: `T[count[b]++] = i` for each last-column byte `b` at position `i` (counts are
  prefix sums of byte frequencies); start at `row = T[original-row]` and repeat `n` times: output `L[row]`, `row = T[row]`.
- **First stage run-length coding:** after four equal bytes in a row comes a count byte, 0 to 255, of further copies; the
  byte after the count starts a new run. Runs do not cross blocks.
- **Block CRC:** CRC-32, polynomial 0x04C11DB7, MSB first, initial 0xFFFFFFFF, final complement, over the block's output
  bytes. The check value of `123456789` is 0xFC891918. **Combined CRC:** `c = rotl(c, 1) xor block_crc` over the blocks, from 0.
- **Randomised blocks** (written only by versions before 0.9.5): after the inverse transform, before the first-stage run-length
  decoding, the lowest bit of the bytes at scheduled positions is flipped. The first flip is at position `gap[0] - 2` and each
  next flip comes `gap[(k + 1) mod 512]` bytes after flip `k`. The 512 gaps are in `BZip2RandomizationTable.g.cs`, measured by
  `tools/bzip2/derive-randomization-table.py`: set the flag of a one-block stream of 899 000 bytes in which no byte repeats and
  no two neighbours differ only in the lowest bit, let the tool decode it (it delivers the bytes before it reports the CRC
  error), and read the positions where the output differs. The gaps repeat with period 512 (three full cycles are
  observed). The tool accepts a hand-made randomised stream with correct CRCs (`testdata/bzip2/vectors/randomized20k.bz2`).

## Decoder behaviour (choices)

**Strictness.** The decoder rejects, with `CompressionFormatException` naming the byte offset and what was expected:

| Input | Why |
|---|---|
| a level digit outside '1'..'9'; a wrong signature | format |
| an empty symbol map; fewer than 2 or more than 6 tables; 0 selectors; a selector at or above the table count | format; the tool rejects the same (0 selectors, 1 table) |
| a code length outside 1..20, or lengths that oversubscribe the code space | format |
| a symbol with a length walk of more than 40 steps | choice: an encoder needs at most 19; the tool has no limit. 40 bounds the header size at about 40 KB |
| a code not assigned by the table (lengths whose Kraft sum is below 1 are accepted) | the tool accepts incomplete tables used legitimately; so do we |
| more than `level × 100000` symbols in a block, or a run that would pass it (exactly that many is accepted) | the tool rejects 100 001 symbols at level 1 and accepts 100 000 |
| a block with no symbols; an original row at or above the block length | the tool rejects both |
| a block ending right after four equal bytes (no count byte) | the tool rejects it |
| 0 selectors | the tool rejects it |
| a block or combined CRC mismatch | format (off with `VerifyChecksum = false`, and then the stored values are still folded into the combined CRC) |

A block with more selectors than it needs is fine. A block that needs more selectors than it has is an error.

**Limits** (`BZip2DecompressionOptions`): `MaxOutputSize` (default 1 GiB) is checked before bytes are written, against the real
output, so a hostile stream (a 900 000-byte block can expand to about 46 MB, and a stream may hold many blocks) stops at the
limit with `CompressionLimitException`; no output is written past it. `VerifyChecksum` default true. `AllowTrailingData`
default false: input after the last stream is an error (the tool ignores it with a warning); when allowed, input that does not
start with `BZh` ends the data.

**Memory.** One 32-bit word per byte of the stream's block size limit (up to 3.6 MB), a 64 KiB staging buffer, a 4 KiB output
chunk and six tables of about 3 KB. All are allocated when first needed and kept across blocks, streams and `Reset`.

**Streaming.** `BZip2Decoder.Decompress` ends each stream with `Done` and consumes exactly its bytes. It reads the symbols
straight from the caller's buffer through a 64-bit accumulator; block headers are parsed from a staging buffer so that a header
split across calls is re-parsed from its start once more input arrives (a header is at most about 40 KB, so feeding one byte at
a time costs time quadratic in the header size, not in the stream size). A symbol is decoded only with 20 bits at hand, which
never stalls a valid stream because at least 80 bits (the next block header or the end marker) follow every block.

**Concatenated streams.** The one-shot and stream APIs decode streams one after the other and join their outputs, as
`cat a.bz2 b.bz2 | bzip2 -d` does; an empty stream (header and end marker only) contributes nothing.

## Tier 3 parts

`BZip2Crc` (block and combined CRC), `BZip2Bwt` (inverse transform: `BuildVector`, `Next`, `Inverse`), `BZip2MoveToFront`
(symbols and selectors, both directions), `BZip2HuffmanTable` (canonical tables with the format's limits, both directions),
`BZip2RunLengthDecoder` (first stage, resumable), `BZip2Randomization` (the schedule). An encoder reuses the CRC, the move-to-front
list, the table codes and the run-length rule.

## Checked against

`bzip2` 1.0.8 at -1 to -9 over the Canterbury files, Silesia and enwik8 (see the test class `BZip2InteropTests` for the
environment variables), 7-Zip's bzip2 encoder at several levels, passes and thread counts, pbzip2 1.1.13 output (several
streams), concatenated streams, runs that exercise the run-length stages (4, 5, 255, 259, 260 bytes; 1 MB and 50 MB of one byte), a randomised
block over a full cycle of the schedule with and without runs, and a bit flip in every byte of a small stream (the tool and the decoder agree
on which are rejected).


<!-- https://recursiveai.org/onearchive/docs/bzip2-encoder-derivation/ -->

# bzip2 encoder: derivation of the writer's choices

bzip2 has no official specification, and the decoder note (`bzip2-decoder.md`) already records what a reader must accept.
This note covers the other half: every choice the format leaves to the *writer* that changes the compression ratio or whether
a reader accepts the stream. For each one: where the hypothesis came from, the experiment, and the value pinned. A value goes
into the encoder only through a public description or an experiment recorded here.

**Sources.** The Burrows–Wheeler report (DEC SRC 124, 1994: block sorting, move-to-front, the idea of Huffman-coding the result);
Manber and Myers (suffix arrays, prefix doubling, 1990); Larsson and Sadakane ("Faster suffix sorting", 1999) and Sadakane (a
BWT-oriented suffix sorter, 1998) for the sorting design; Bentley and Sedgewick (multikey quicksort, 1997); the public
descriptions of the bzip2 stream layout (the encyclopedia article's file-format section, the bzip2 manual page); the decoder note. Experiments run
`bzip2` 1.0.8 (13-Jul-2019) as a black box. The reference compressor's source and its heuristics are not consulted: where the tool's *observed*
output is listed below, it is evidence of what a reader accepts and of what a ratio comparison is up against, not a design to copy.

**Experiments.** `tools/bzip2/derive-encoder-facts.py` generates its own inputs from fixed seeds, compresses them with the tool, parses the
streams with `tools/bzip2/bzip2_inspect.py` (a parser written from the format description) and prints one `F` line per fact.
`testdata/bzip2/encoder-facts.txt` is its recorded output (E-numbers below name the experiment groups in the script).

## Pinned facts

| # | Question | Hypothesis and source | Experiment | Pinned value |
|---|---|---|---|---|
| F1 | How many first-stage bytes fit in a block at level L? | The level digit is the block size in units of 100 000 bytes (format description; decoder note: the tool accepts exactly `L × 100 000` symbols). | E1: random bytes without runs, 2.2 blocks long, levels 1, 2, 5, 9 | Blocks of `L × 100 000 − 19` bytes (99 981, 199 981, 499 981, 899 981), the rest in the next block. The reader accepts up to `L × 100 000`, so any limit up to that is valid. |
| F2 | How are runs coded in the first stage? | After four equal bytes comes a count of further copies (format description). The largest count is the writer's choice. | E2: `x`, *n* × `a`, `y` for *n* = 1…520 | A run of *n* ≥ 4 is `aaaa` plus a count byte *n*−4, and a run is cut at **255 bytes** (count 251, `fb`): 259 `a` become `aaaa fb aaaa 00`. A run of exactly 4 still gets its count byte (`00`). The next run starts fresh after a count byte even when the next byte is the same. The reader accepts counts up to 255 (decoder note); we write at most 251, as the tool does, so that a reader that checks the count against the tool's range accepts our streams. |
| F3 | May a run span blocks? What happens at the boundary? | Decoder note: runs do not cross blocks; a block that ends right after four equal bytes is rejected. | E2/F3: level 1, 99 960…99 981 varied bytes followed by 1 000 `z` | The tool closes the block between two pieces of a long run (`… zzzz fb` ends block 1; block 2 starts a new run of `z`). A block that is full is closed even though the same byte continues. Each run piece ends with its count byte. The tool's blocks can exceed 99 981 by up to 4 (the piece that crosses the mark is finished); ours never exceed the cap (a unit that does not fit starts the next block). |
| F3b | Exact capacity with no runs | as F1 | E1/F3b: 99 982 … 100 002 bytes | the first block holds exactly 99 981 bytes; 99 981 bytes fill a block completely. |
| F4 | How many tables, how many selectors? | Decoder note: 2…6 tables are accepted, 1…32 767 selectors, one selector per group of 50 symbols. | E3: text inputs of 1…200 000 bytes | Selectors = ⌈symbols / 50⌉ where symbols counts the end-of-block symbol. The tool writes 2 tables for fewer than 200 symbols, 3 up to about 600, 4 up to about 1 200, 5 up to about 2 400 and 6 above that, and always at least 2 (a 1-byte input has 2). The format accepts 2…6 for any block size, so the count is ours to choose (see "Our choices"). |
| F5 | Code lengths | Decoder note: lengths 1…20, delta-coded from a 5-bit start, canonical codes. | E4: skewed, uniform and two-symbol inputs, and a Fibonacci-weighted input | Every symbol of the alphabet gets a length, used or not (the two-symbol input has 16 coded symbols all with a length; each table's Kraft sum is exactly 1). The longest length seen is **17**, on a skewed input that needs more (the Fibonacci-weighted input needs only 12); the tool limits to 17. The reader accepts up to 20 (decoder note; our interop test of 18…20 is in `BZip2EncoderTests`). |
| F6 | The sort and the origin pointer | The report: sort all cyclic rotations, send the last column and the index of the original. | E5: *banana*, *abracadabra*, *mississippi*, 8 more | The last column is that of the sorted **cyclic** rotations (no end marker), and the origin is the row holding rotation 0, in all 11 cases without ties. With periodic input (`abab…`) several rows hold equal rotations and the tool names a different one from the first; any of them decodes to the same bytes, so tie order is ours (we use the lowest rotation index first). |
| F7 | CRCs, framing, padding | Decoder note (CRC over the block's input bytes before the first stage; combined = rotate-left-1 xor). | E6: empty, 1 byte, text, runs, 3 blocks | The block CRC is the CRC of the block's *original* bytes (before the first stage); the combined CRC is the rotate-xor of the block CRCs, stored after the end marker (0 for no blocks). An empty input is 14 bytes: `BZh`+level, the end marker, a zero CRC, no padding bit. Padding to a byte boundary is zero bits. The level digit is the requested level even for 3 bytes of input. |
| F8 | Symbols | Decoder note (RUNA/RUNB, end of block = used + 1, list starts as used bytes ascending). | E7 | confirmed: `abcabczzz` → `[4,4,3,0,4,0,4,0,4,5]` with 4 used bytes; a block using all 256 values has alphabet 258. A zero run still open at the end is written before the end symbol. |
| F9 | Selector coding | Decoder note (unary move-to-front index over table numbers). | E7/F9 | confirmed: a repeated table is index 0, written as a single 0 bit. |

## Our choices (the reader accepts the whole range each time; the reason is ratio unless stated)

- **Block size.** `L × 100 000 − 19` first-stage bytes, never split a unit (a literal, or a run piece with its count byte). Same cut points as the tool's for inputs without runs (F1), so ratio comparisons are like for like.
- **First stage.** F2: pieces of at most 255 bytes. Runs are cut at block ends (F3).
- **Sort.** Cyclic rotation sort by prefix doubling over group ranks (Manber–Myers, as organised by Larsson–Sadakane: bucket by the first bytes, then refine only unsorted groups), ties broken by rotation index. The result is the same for any sorting algorithm that sorts rotations and breaks ties by index, so the sorter can change without changing bytes. Origin = the row of rotation 0 (lowest index among equal rotations).
- **Tables.** 2…6 chosen from the number of coded symbols by the thresholds F4 shows (fewer than 200, 600, 1 200, 2 400 symbols: 2, 3, 4, 5; 6 above). The tool's own thresholds are an observation; ours were checked against always writing 6 (tables no group picks are dropped anyway): the sizes are the same on seven Canterbury files and 2.2% larger on the 3.7 KB grammar.lsp, so the small-block thresholds earn their place. The groups of 50 symbols start dealt to the tables in runs of consecutive groups, and six refinement passes follow: each group goes to the table that codes it in the fewest bits (ties to the lower table), each table's lengths are rebuilt from the groups it received. Tables no group chose are dropped, and the table count is kept at 2 or more. Integer arithmetic only. Measured alternatives (dickens, xml and mr at level 9, size relative to the tool's): start from a split of the alphabet into ranges of equal frequency, 2 passes 1.0086, 8 passes 1.0018; start from runs of groups, 4 passes 0.9995, 6 passes 0.9996 on those three files (0.9996 over five files at levels 1, 5, 9). Not copied from the tool: the choice came from trying the two natural starts.
- **Selectors.** Exactly ⌈symbols / 50⌉, move-to-front coded (F4, F9).
- **Header.** Start length = the first symbol's length, then ±1 steps; the largest step between neighbours is 16, far below the reader's 40.
- **Concatenation.** Multiple streams are not produced by one encoder; callers concatenate encoder outputs (the decoder reads concatenated streams).


<!-- https://recursiveai.org/onearchive/docs/brotli-encoder/ -->

# Brotli encoder

Normative and language-neutral: a port that follows this text produces the same bytes. RFC 7932 fixes what a brotli stream
means and leaves the parse, the block boundaries, the context modelling and the prefix-code construction to the encoder;
everything it leaves open is fixed here. Output depends only on the input bytes, the options, and where flushes fall. It never
depends on the CPU, the runtime, the thread count or how the caller splits the input into calls. Nothing that decides a byte
uses floating point: costs are integers or fixed-point.

The design is this library's own, built from the RFC and from the published description of brotli (Alakuijala et al., "Brotli:
A General-Purpose Data Compressor", ACM TOIS 2019) and the literature on cost-based (shortest-path) parsing. The numbers
below are measured, not copied from another encoder. Its streams decode with `brotli` 1.2.1 and with this library's decoder.

## API shape

- Tier 1: `Brotli.Compress(data, BrotliOptions?)`, `Brotli.TryCompress`, `Brotli.GetMaxCompressedLength`.
- Tier 2: `BrotliEncoder` (`OperationStatus Compress(source, destination, out consumed, out written, bool isFinalBlock)`,
  `Flush`, `Reset`) and `BrotliStream(Stream, BrotliOptions?, leaveOpen)` in write mode (`Flush` is the flush below).
- Tier 3, public: `BrotliBitWriter` (LSB-first bit writer, RFC 7932 section 1.5.1), `BrotliHuffmanCode` (length-limited
  prefix-code lengths and canonical codes from a histogram) and `BrotliPrefixCodeWriter` (the simple and complex prefix-code
  descriptions of section 3.4 and 3.5). Together they are enough to write a meta-block by hand.
- Internal on purpose: the match finders, the block splitter, the clusterer, the dictionary index and the meta-block writer. They
  share one buffer and one set of tables with the engine, and their interfaces still move with the tuning; making them public
  would freeze that. They are described below so that a port can reproduce them.

## Options

| Option | Default | Meaning |
|---|---|---|
| `Quality` | 11 | 0 to 11, table below. Same numbering and meaning as the `brotli` tool's `-q`. |
| `WindowBits` | 22 | 10 to 24. A copy reaches back at most `2^WindowBits - 16` bytes (RFC 7932 section 9.1). |

Large-window brotli (window above 24) is not part of RFC 7932 and is not produced.

## Meta-blocks and flushing

- Input is gathered in a buffer holding the window followed by the block being filled. A block is coded when it holds
  `BlockSize` bytes (table below), on a flush, and at the end. Block boundaries therefore depend on the input length and the
  flush positions only, never on how the input was cut into `Compress` calls.
- Each block is one meta-block. The stream header is the WBITS field of section 9.1 (one bit for 16, otherwise the shortest of
  the three forms that can name the window).
- **Flush** codes the pending bytes as a (non-last) meta-block, then writes an empty metadata meta-block (ISLAST 0, MNIBBLES
  code 3, reserved bit 0, MSKIPBYTES 0, padding to a byte), which ends the output on a byte boundary. A flush with nothing new
  since the last one writes nothing.
- **End** codes the pending bytes as the last meta-block; with no pending bytes it writes the empty last meta-block
  (ISLAST, ISLASTEMPTY).
- **Uncompressed fallback:** if a meta-block's compressed form takes more bits than the uncompressed one (header, padding and the
  bytes), it is written uncompressed. A last meta-block cannot be uncompressed, so it is written as a non-last uncompressed one followed by
  an empty last meta-block.
- State carried across blocks: the window, the match finder tables and the four last distances (so a block may begin with a
  copy that repeats the previous block's last distance).

## Levels

| Q | Block | Table | Hash bytes | Depth | Long table | Parse | Contexts, splitting | Dictionary |
|---|---|---|---|---|---|---|---|---|
| 0 | 128 KiB | 2^15 heads | 5 | 1 | | greedy, one probe | none | no |
| 1 | 128 KiB | 2^17 heads | 6 | 1 | | greedy, one probe | none | no |
| 2 | 128 KiB | 2^16 x 2 | 5 | 2 | | greedy | none | no |
| 3 | 256 KiB | 2^15 x 8 | 6 | 4 | | lazy 1 | literal contexts | no |
| 4 | 1 MiB | 2^15 x 8 | 6 | 8 | | lazy 1 | + block splitting | no |
| 5 | 4 MiB | 2^16 x 16 | 6 | 8 | | lazy 1 | + context mode search | no |
| 6 | 4 MiB | 2^18 x 16 | 6 | 16 | 2^16 x 8, 8 deep | lazy 1 | + distance parameter search | identity |
| 7 | 4 MiB | 2^18 x 32 | 6 | 24 | 2^16 x 16, 16 deep | lazy 2 | same, finer literal splitting | identity |
| 8 | 4 MiB | 2^18 x 32 | 6 | 32 | 2^17 x 16, 16 deep | lazy 2 | same | identity |
| 9 | 4 MiB | 2^17 x 64 | 6 | 48 | 2^17 x 32, 32 deep | lazy 2 | same | identity |
| 10 | 1 MiB | 2^20 binary tree | 4 | 64 | | shortest path, 2 passes | same, finest literal splitting | all transforms |
| 11 | 1 MiB | 2^20 binary tree | 4 | 128 | | shortest path, 3 passes | same | all transforms |

"Table" is the number of hash values times the slots each owns ("x 16"); "Depth" is the number of entries of a bucket (or tree nodes at q10
and q11) visited per position. Qualities 0 to 5 skip ahead on incompressible data: after a miss the position advances by
`1 + (bytes since the last copy >> S)`, with S = 6 at qualities 0 to 2 and 5 at qualities 3 to 5 (the positions skipped over are still
entered in the table at q4 and q5; at q2 and q3 see "Sparse entry" below). Qualities 6 to 11 do not skip. The nice length (a match this long ends the search) is 64, and 128 at qualities 10 and 11. The
table is the single source; `BrotliQualityPlan.For` must match it.

## Match finding

- **Hash of a position**: for 4 bytes, `(little-endian 4 bytes * 0x1E35A7BD mod 2^32) >> (32 - HashBits)`; for 5 to 8 bytes, the little-endian
  64-bit word with all but its first `n` bytes cleared (shifted up so the kept bytes are the high ones), times `0x9E3779B185EBCA87 mod 2^64`,
  shifted down to `HashBits`. (The tables of q0 to q2 do not shift the kept bytes up: their hash is `((word & (2^(8n) - 1)) * 0x9E3779B185EBCA87 mod 2^64) >> (64 - HashBits)`.) A position is entered only if every byte its hash reads is inside the block; the others are entered with
  the next block, because bytes past the end of the block are not input yet.
- **Hash head** (q0, q1): one candidate per bucket, the latest position, stored with the four bytes found there (a slot is two
  ints) so that a candidate that does not agree on them is refused without reading the input at its position. Hashes are of 5 bytes at q0 and 6 at q1,
  taken from the 64-bit word at the position, so a table candidate always agrees on that many bytes before its length is measured and
  copies found through the table are at least that long (`FastHashLength` in the plan). Only positions with eight bytes inside the
  block are probed or entered; the last seven bytes of a block are literals.
- **Two-way table** (q2): each hash value (5 bytes, taken the same way) owns two slots, the two latest positions, each with its four
  bytes; entering a position moves the newer slot to the older and writes the new one. Both are candidates, newest first. This is the
  bucketed table of q2 to q9 with a bucket size of 2 and a depth of 2, minus the entry counts.
- **Bucketed table** (q2 to q9): each hash value owns `2^BucketBits` slots holding the latest positions with that hash, the oldest slot
  overwritten first (a count of entries made per bucket says where the next goes). Candidates are visited newest first, at most Depth
  of them; a candidate must agree with the position on the hashed bytes and on the byte just past the best length so far; the search ends
  at the nice length. Every position that is not skipped is entered before the next one is searched, so a lazy probe sees the
  positions the main probe entered.
- **Long table** (q6 to q9): a second bucketed table keyed by a hash of the first twelve bytes
  (`(word0 * 0x9E3779B185EBCA87) xor (word1 * 0xC2B2AE3D27D4EB4F) >> (64 - bits)`, where word0 is the first eight bytes and word1 the next four).
  It is searched first, newest first and at most its own depth, so that long matches in repetitive input are found even when many newer
  positions share the shorter hash. Positions enter it together with the main table, only when twelve bytes are inside the block.
- **Binary tree** (q10, q11): each bucket is the root of a tree of positions keyed by the string starting there, re-rooted at
  each new position (each position stores a smaller and a larger link). The nodes met on the way down are the candidates, with
  ever longer common prefixes. Comparisons stop at the nice length, and a node that agrees that far is replaced by the new
  position, which inherits its subtrees. A position is entered only if `position + niceLength` is inside the block; positions
  nearer the end are searched but entered with the next block, because the order of a string against the others must not be
  decided by bytes that are not there yet. A candidate's length is checked by comparing bytes before it is used.
- The four last distances are always tried first (any length from 3 in the greedy and lazy parses, from 2 in the shortest-path
  parse), then the finder's candidates. Distances beyond the window or the data so far are never used.
- **Static dictionary** (q6 and up, only when the best match found is shorter than 12): words are indexed by a hash (18 bits) of
  their first four bytes, the lowest word ids first. A hit is accepted only after the whole transformed word has been compared
  with the input. q6 to q9 try the identity transform, q10 and q11 all 121. A dictionary copy uses the distance
  `maxDistance + 1 + word id` where `maxDistance = min(window, bytes so far)` (RFC 7932 section 8) and does not enter the
  last-distance ring. A word is looked for only if it could beat the match already found: its copy costs at least
  `224 + 16 * (extra bits of maxDistance + 1)` in the score below, so words covering fewer bytes than `(best score + that cost) / 90`
  (rounded up) are skipped; this changes no choice, it only saves work. The all-transforms search of q8 and q9 in earlier
  versions gained under 0.3% in size on text and XML at 6 to 10 times the time, so those qualities use the identity transform only. At q5 the identity search gained under 0.1% on text, XML and mixed data and cost
  a fifth of the time, so q5 does not search the dictionary.

## Greedy and lazy parse (q0 to q9)

Costs are in 1/16 bit. A match's score is `covered * 90 - cost`, where cost is 144 for a copy at the last distance, 200 for one of
the other three recent distances, and `224 + 16 * (extra bits of the distance)` for a new distance or a dictionary word. A match
is taken only if its score is positive; ties go to the one covering more bytes. With Lazy of 1 or 2, a match is postponed while
a match one (or two) positions later scores more by over 90 (180) than this one, for matches covering under 64 bytes; the probe
for the later positions uses half the depth (and half the long table's depth).

**q0 and q1** use one loop. At each position with eight bytes inside the block it reads the table slot and replaces it with the
position. If the slot's four bytes agree with the position's and the distance is within the window, the table has a candidate. Only
then is anything else done, and the candidate is the only copy considered: a position with no table candidate is a miss, whether or
not some other distance would have matched. The copy's length is the number of equal bytes from the position and the candidate,
limited by the end of the block; it is taken only if it is at least `FastHashLength` and its score is positive. A copy at the last
distance used (a distance changes only through a copy with an explicit distance) is coded with short code 0 and scores with a cost of
144; any other distance is coded in full (RFC 7932 section 4, distance code 16 and up) and scores with `224 + 16 * (extra bits)`. The
other short codes (the three older recent distances and the nearest-neighbour codes) are not used. They save little on these inputs
and testing for them costs more than the bits they save. After a copy, its second byte and its last two enter the table (only when the
copy ends at least eight bytes inside the block). After a miss the position advances by `1 + (bytes since the last copy >> 6)`.

**q2** looks at the two table slots at a position with eight bytes inside the block. A slot is a candidate when it agrees on the
first four bytes and lies within the window. Each candidate's copy is measured as at q0 and scored with the q0 costs; copies shorter
than `FastHashLength` are not candidates. The better score wins; on equal scores the newer slot wins. Only the last distance has a short
code, as at q0. A position with no candidate whose score is positive is a miss. Entry follows "Sparse entry".

**q3 to q9** try the four last distances first (any length from 3), then the bucket candidates, then the dictionary. After a copy
every position it covers is entered, except at q3 (below).

**Sparse entry (q2 and q3).** After a copy of 32 bytes or more only its last two positions are entered, and positions stepped over by
the skip-ahead are not entered; the position just before the next probe is. The positions are a small share of what a later search
finds, and entering them costs a hash and two stores each. The measured price is under 0.2% of size on text.

**Command records (q0 to q2).** The parse computes each command's insert-and-copy symbol, extra bits, distance symbol and distance
context at the moment it finds the command, and counts the symbols into the histograms the writer builds its prefix codes from. The
literal histogram is counted by the writer over the input between copies. In a block whose commands carry four literals or more on average,
the writer counts and writes the first eight literals of every run through lookups that read eight bytes from the start of the run
whatever its length and give zero bits for those past its end; other blocks use plain loops. Neither changes a bit written. The buffer
keeps sixteen bytes beyond the data for those reads.

## Shortest-path parse (q10, q11)

Per block (1 MiB):

1. **Candidates.** For every position the binary tree (and, where the tree's best candidate is under 12 long, the dictionary)
   lists candidates `(length, distance)`.
2. **Costs.** Costs are in 1/256 bit and path costs are 64-bit. The first pass uses the block's byte histogram for literals
   (`log2(total / count)`, unseen bytes `+ 3` bits, at least 1/8 bit), 9 bits per command symbol, 4 bits for distance symbols
   below 16 and 6 for the others. Later passes use the symbol statistics of the previous pass's commands (unseen symbols
   priced 3 bits, literals 4 bits above the seen ones).
3. **Path.** A forward dynamic programme over positions, one node each, holding the cheapest cost, its predecessor, the number
   of literals pending and the four last distances. Edges from a node: one literal; each of the four recent distances at lengths
   `max(2, len - 6)` to `len`; for each tree candidate its lengths from just above the next shorter candidate (at least 4,
   and at most 10 below its own length) using the nearest distance among candidates at least that long; each dictionary
   candidate at its covered length. A command's price is its command-symbol cost, its insert and copy extra bits, and its distance
   symbol cost (none if the distance is the last one and the command can leave it implicit) plus distance extra bits. A candidate
   of at least the nice length is taken whole and its interior is skipped.
4. **Distances in the model** are priced as NPOSTFIX 0, NDIRECT 0; the writer may then choose better parameters.

## Entropy coding of a block

Done by the meta-block writer, in this order (each choice by estimated size, integer arithmetic, ties to the lower index):

1. **Distance parameters.** q6 and up try `(NPOSTFIX, NDIRECT)` = (0,0), (0,8), (1,0), (1,8), (2,0), (2,12) and keep the one with
   the smallest distance entropy; below q6 it is (0,0). Distances are mapped onto the short codes (RFC 7932 section 4) first.
2. **Block splitting** (q4 and up) separately for literals, commands and distances (section 6). The sequence is cut into equal
   segments: for commands and distances at least 256 symbols and at most 128 segments (at q4: 32, merged pair by pair up to 16 of them;
   the pairwise merging of long histograms is the largest cost of the splitter, and the estimated gain of the extra segments is
   small); for literals at least 1024 symbols and at most
   128 segments at q4 to q6, at least 512 and at most 256 at q7 to q9, at least 256 and at most 2048 at q10 and q11.
   Segment histograms are clustered into at most 256 block types, then refined up to four times (stopping early when nothing
   moves) by a Viterbi search over the segments where a type change costs 14 bits. The split is kept only if its estimated cost, headers and switches included,
   beats a single type.
3. **Clustering** (`BrotliClusterer`): the cost of a histogram is its entropy; merging two costs the entropy of the sum minus the
   parts'. A merge is taken when the increase is less than the estimated cost of one more prefix code (4 bits per used symbol
   plus 20). Up to 160 histograms (64 for the literal block types at q4 to q6, 96 at q7 to q9) the cheapest pair is merged repeatedly;
   beyond that histograms are taken largest first, each joining the nearest group or opening one, then two rounds move each histogram to the group that codes it best. Entropy is
   computed with an integer table of `x log2 x` in 1/65536 bit.
4. **Literal contexts** (q3 and up): each literal block type gets a context mode (RFC 7932 section 7.1): LSB6, MSB6, UTF8 or
   Signed. q5 and up try all four per type, by entropy plus an estimate of the header; below that UTF8 is used. The 64 contexts of
   each type are clustered onto trees (at most 256 trees in total). Distance contexts (4 per type, section 7.2) are clustered
   the same way. q0 to q2 use one tree and one context.
5. **Prefix codes.** Lengths from `BrotliHuffmanCode` (plain Huffman: leaves ordered by (count, symbol), two merged queues,
   maximum length 15, a code longer than the limit is rebuilt with every count raised to a floor of 1, 2, 4, ... until it fits). Codes with one used symbol are written as
   simple codes with zero bits per symbol; codes of up to four symbols use the simple form, the rest the complex form with
   run-length codes 16 and 17 for repeats, and a code-length code of at least two symbols.
6. **Commands** are written with insert-and-copy symbols (an implicit last distance when allowed), then the extra bits, the
   literals, and the distance.

## Determinism notes

- Hashes are the fixed multiplicative hash above. No `HashCode`, no `Dictionary` order, no timing.
- Every cost is an integer; logarithms come from a table built by integer digit-by-digit squaring (1/65536 bit).
- SIMD is not used in the encoder; the only vectorised helper is the common-prefix length, which computes the same function. Cache
  prefetch hints in the match finders (the tables of q1 and q2, and the bucketed tables) load table lines early and change no result. The meta-block writer packs a command's
  codes into a bit accumulator held in locals; that only changes how the bits are put out, never which bits.
- Block size depends on the quality alone, so one thread or many (the encoder is single-threaded) gives the same bytes.

## Measured

Sizes against `brotli` 1.2.1 at `-w 22` are in the unit report and the bar results, not here: a number that moves with tuning
does not belong in a normative document.


<!-- https://recursiveai.org/onearchive/docs/brotli-decoder/ -->

# Brotli decoder

`RecursiveAI.OneCompress.Brotli`: decoding of brotli data as RFC 7932 defines it. This document records the choices the RFC
leaves to the implementation. The format itself is the RFC; section numbers below are its.

## Names

`Brotli` (one-shot), `BrotliDecoder` (span decoder with `OperationStatus`) and `BrotliStream` (read-only stream) keep this
library's naming convention. They share simple names with the types in `System.IO.Compression`, but live in a different
namespace; a file that uses both aliases one of them. The public tier-3 parts are `BrotliPrefixCode` (prefix-code tables),
`BrotliWords` (the static dictionary) and `BrotliTransforms` (the 121 word transformations).

## Static data

The dictionary (section 8 and Appendix A), the transformation list (Appendix B) and the context lookup tables (section 7.1)
are generated by `tools/brotli/generate-tables.py` from the RFC text, identified by its SHA-256. The script verifies every
CRC-32 the RFC states for them. The generated files carry the IETF Trust's licence notice (see THIRD-PARTY-NOTICES.md). They
are reachable only from the brotli types, so an application that does not use brotli does not contain them after trimming.

## Limits (BrotliDecompressionOptions)

- `MaxWindowSize`, default 2^24 - 16 (the largest window the format allows). A stream whose window exceeds it throws
  `CompressionLimitException` after the first header byte(s). Memory for the output ring is bounded by the window and by the
  declared output of the meta-blocks seen so far; the ring grows by doubling until it first wraps.
- `MaxOutputSize`, default 1 GiB, checked against each meta-block's declared length before any of it is produced.
- `AllowTrailingData`, default false: bytes after the last meta-block are an error naming their offset.

## Choices

- **Large-window brotli** (window-bit pattern `0010001`, written by the tool's `--large_window`) is not part of RFC 7932 and is
  rejected as an invalid window size.
- **Context maps:** the RFC says every tree value must be used by the map; the decoder does not check this, since an unused
  tree changes nothing a decoder can observe.
- **Errors:** malformed data throws `CompressionFormatException` (format `brotli`, byte offset, what was expected;
  `IsTruncated` when more input would have been needed). The span decoder never returns `InvalidData`. After a throw the decoder
  must be `Reset`.
- **Streaming:** header fields are decoded atomically; when input ends inside one, the decoder keeps the unfinished bytes
  and resumes when more arrive, so input and output may be cut anywhere. Output is never withheld: a call writes as much as
  fits in the destination and returns `DestinationTooSmall` when more is waiting.
- **Metadata meta-blocks** are skipped. The bytes after the last meta-block's final bit up to the byte boundary must be zero.


<!-- https://recursiveai.org/onearchive/docs/ppmd/ -->

# PPMd (variants H and I)

Two codecs from Dmitry Shkarin's PPMd family, byte-identical to what 7-Zip writes and reads:

| | Variant H | Variant I (revision 1) |
|---|---|---|
| Used by | 7z coder `03 04 01` | zip method 98 |
| Types | `PpmdH`, `PpmdHEncoder`, `PpmdHDecoder`, `PpmdHModel` | `PpmdI`, `PpmdIEncoder`, `PpmdIDecoder`, `PpmdIModel` |
| Order | 2 to 64 | 2 to 16 |
| Model memory | 2 KiB to 0x7FFFFF00 bytes | 64 KiB to 0x7FFFFF00 bytes |
| Range coder | 7z flavour (carry propagation, 5-byte start) | Subbotin carry-less coder |
| Restore method | restart only | restart (0) or cut-off (1) |

The model parameters are not in the compressed data. The decoder must be given the encoder's order, memory size and (for I) restore
method; the container carries them: the 7z coder properties (5 bytes: order, then memory as a little-endian 32-bit number), and the
two-byte header at the start of a zip method 98 entry (APPNOTE 5.10.4: bits 0–3 order − 1, bits 4–11 memory in MiB − 1,
bits 12–15 restore method). The zip header can express only whole MiB from 1 to 256.

## Origin

The model and coders are ported from the public-domain sources named in the header of each file in `src/OneCompress/Ppmd`
(see `THIRD-PARTY-NOTICES.md`). The model keeps the original's memory layout (12-byte units, 32-bit offsets) because the sequence of
allocations decides when memory runs out and the model restarts, so any other layout would write different bytes. The model is
a byte array with bounds-checked access; no unchecked accessors are used.

## Behaviour the formats leave open

- **End marker.** 7z data has none; the size is known from the folder. 7-Zip's zip writer appends an end marker (a symbol coded as
  escape at the root with no successor), and `PpmdIOptions.WriteEndMarker` defaults to true; `PpmdHOptions.WriteEndMarker` to false.
  A reader with a known size stops at the size and does not require the marker.
- **Levels** (the numbering of 7-Zip's `-mx`), pinned by `testdata/ppmd/levels` (a script and a note: 7-Zip 26.05 run on a 212 MB input, where no
  size reduction applies):

  | Level | 7z: order, memory | zip: order, memory (MiB), restore |
  |---|---|---|
  | 1 | 4, 1 MiB | 4, 1, restart |
  | 2 | 4, 2 MiB | 5, 2, restart |
  | 3 | 5, 4 MiB | 6, 4, restart |
  | 4 | 5, 8 MiB | 7, 8, restart |
  | 5 | 6, 16 MiB | 8, 16, restart |
  | 6 | 8, 32 MiB | 9, 32, restart |
  | 7 | 16, 64 MiB | 10, 64, cut-off |
  | 8 | 24, 128 MiB | 11, 128, cut-off |
  | 9 | 32, 256 MiB | 12, 256, cut-off |

  Levels above 9 behave as 9. The default level is 5.
- **Small inputs.** When the size is known, the model memory is reduced to the smallest power of two 2^k (k from 16 to 31) that is at
  least 16 times the size, if that is smaller than the requested memory; the zip writer then rounds up to whole MiB, minimum 1.
  The reduced value is what the header or properties record, and the decoder uses the recorded value.
- **Variant I minimum memory** is 64 KiB: with less, the cut-off restore cannot make room when the root context takes more than
  three quarters of memory.
- **Freeze restore method** (variant I's third method) is not supported; a header that asks for it is refused.
- The 7z encoder of the reference tool refuses orders above 32; the decoder here accepts up to 64 as the format allows.

## Limits and hostile input

- `PpmdHDecompressionOptions.MaxMemorySize` / `PpmdIDecompressionOptions.MaxMemorySize` (default 256 MiB) is checked before the model is
  allocated; a larger stated size raises `CompressionLimitException`. Archives use `ArchiveLimits.MaxPpmdMemorySize` (default 256 MiB).
- `MaxOutputSize` (default 1 GiB) bounds the output of the one-shot calls. The archive layer applies its own total-size and
  expansion-ratio limits.
- An order or restore method outside the ranges above, or a model memory below the minimum, is a `CompressionFormatException`
  naming the field and its offset in the properties or header.
- Truncated input raises `CompressionFormatException` with `IsTruncated`; a decoder given a damaged stream may produce wrong bytes before
  it notices (the format has no checksum), so archives rely on their own CRCs.

## Streaming

The span decoder (`OperationStatus Decompress(...)`) may be given input in pieces of any size. A symbol that needs bytes
beyond the end of the input is rolled back through a small journal (a few scalars, one binary-context probability and the
SEE sums) and the call returns `NeedMoreData` with that symbol's bytes unconsumed, so the next call re-reads them. Output
can be drained in pieces as well. `PpmdStream` wraps both directions over a `Stream`.

## Determinism

No SIMD and no floating point are used. The hash matrix has `ppmd-h` and `ppmd-i` entries (orders 2 to 64, memory from the
minimum to 16 MiB, both restore methods, end marker on and off) over the standard inputs; the manifests are equal on x64 JIT,
NativeAOT, scalar, reduced-AVX and browser-wasm.


<!-- https://recursiveai.org/onearchive/docs/lzx-decoder/ -->

# LZX decoder

Normative notes for `RecursiveAI.OneCompress.Lzx`: LZX as cabinet folders store it, decode only. A port should decode the same
streams from this text. No encoder exists.

Sources: Microsoft's [MS-PATCH] (LZX DELTA Compression and Decompression) 2.1 to 2.7, which describes the bit stream, the block
types, the trees and the call translation, and [MS-CAB] 2.4 for the data blocks that carry the frames. Where the two differ, or where
the wording is open, the reference extractor cabextract 1.11 was run as a black box on cabinets whose LZX streams came from a small
test compressor (`tests/OneCompress.Tests/Lzx/LzxTestEncoder.cs`, written from the same documents). Each choice below marked
**[tool]** was settled by building the stream both ways and seeing which one cabextract extracts byte for byte; the checks are
`CabInteropTests.FormatChoicesPinnedByTheReferenceTool` and the cabinets in `CabInteropTests`.

## Framing

- The stream is a sequence of **frames**. Every frame decodes to 32768 bytes except possibly the last. A cabinet keeps one frame in each
  data block (`cbUncomp` 32768; `cbData` the compressed size). There is no chunk-size prefix (that belongs to LZX DELTA).
- The bit stream is 16-bit little-endian words consumed from the most significant bit. After the last token of a frame the stream
  is padded with zero bits to a 16-bit boundary, and the next frame starts at the next word.
- Blocks, trees, the three repeated offsets and the window carry over from frame to frame; a block may span frames. A match never
  crosses a frame boundary or the end of its block (the decoder rejects one that does).
- The window is 2^15 to 2^21 bytes (the cabinet folder's `typeCompress >> 8 & 31`). It starts empty: an offset larger than the bytes
  decoded so far is an error. The decoder allocates the window up front.
- First bit of the first frame: 1 = x86 call translation on, followed by the 32-bit translation size (high 16 bits first); 0 = off.

## Blocks

Each block starts with a 3-bit type (1 verbatim, 2 aligned offset, 3 uncompressed; anything else is an error) and a 24-bit size, the
number of uncompressed bytes the block represents. A zero-size block is accepted and ignored.

- **Verbatim and aligned offset** blocks carry (aligned only) eight 3-bit code lengths for the aligned-offset tree, then the main
  tree as two parts (256 literals; then 8 x slots match symbols) and the length tree (249 symbols), each preceded by a 20-entry pretree
  of 4-bit lengths. Lengths are sent as differences `(previous - symbol + 17) mod 17` from the previous block's lengths (zero for the
  first block); the persisted lengths survive uncompressed blocks. Pretree symbols 17 and 18 are runs of zero lengths (4 + 4 bits,
  20 + 5 bits), 19 is a run of 4 or 5 (1 bit) equal lengths followed by one more pretree symbol giving their common value.
  **[tool]** The value of a run (19) is one absolute length for the whole run, computed from the previous length of the run's first
  element; it is not a run of equal differences.
- Codes are canonical: shorter codes first, equal lengths in symbol order. A set of lengths that oversubscribes the code space is an
  error. An incomplete set is accepted and fails only when an unassigned bit pattern is read (the reference tool also refuses an
  *empty* aligned-offset tree, so the test compressor never writes one; the decoder is more lenient).
- **Uncompressed** blocks: after the 27 bits of header the stream is padded with zero bits to a word boundary, **[tool]** and a
  header that already ends on a word boundary is followed by a whole zero word (1 to 16 bits of padding, as [MS-PATCH] 2.3.2.1 says).
  Then R0, R1 and R2 as 32-bit little-endian values (they replace the repeated offsets), then the raw bytes; an odd byte count is
  followed by one zero byte. An uncompressed block may span frames: each frame's compressed bytes hold that frame's part, with no
  padding word between.

## Matches

- A main symbol below 256 is a literal. Otherwise `m = symbol - 256`: length header `m & 7`, position slot `m >> 3`. Header 7 means
  the length tree gives `symbol + 9`; otherwise the length is `header + 2`.
- Slots 0, 1, 2 reuse R0, R1, R2 (slot 1 and 2 swap with R0). Other slots give formatted offset
  `base[slot] + footer`, the real offset is `formatted - 2`, and `R2, R1, R0 = R1, R0, offset`.
- Footer bits per slot: 0 for slots 0 to 3, then `min(17, slot / 2 - 1)`; bases are the running sums (`LzxPositionSlots`). A window of
  2^15, 2^16, ... 2^21 uses 30, 32, 34, 36, 38, 42, 50 slots.
- In an aligned-offset block a footer of 3 or more bits is split: the low 3 bits come from the aligned-offset tree, the rest are read
  verbatim, most significant first.
- **[tool]** Cabinet matches are at most 257 bytes. The extra-length field of [MS-PATCH] 2.6.6 does not exist in cabinet LZX: a
  stream written with it is not extracted correctly by the reference tool, and the decoder does not read it.

## Call translation

When enabled, each frame is processed after decoding with `frameOffset` the number of stream bytes before it ([MS-PATCH] 2.2.2):
nothing is done to frames starting at or beyond 2^30 or to frames of 10 bytes or fewer; otherwise for each `E8` byte at `i < length -
10`, with `value` the next four bytes as a signed little-endian integer and `current = frameOffset + i`, if `-current <= value <
size` the operand becomes `value - current` (when `value >= 0`) or `value + size`; then `i` advances by 5 in every case.
**[tool]** The four operand bytes are skipped after every `E8`, translated or not. The window keeps the untranslated bytes.

## Errors and limits

`CompressionFormatException` names the format `lzx` and the byte offset (the compressed bytes given so far plus the position in the
current frame): bad block type, oversubscribed tree, a code no symbol owns, a pretree run past the end of a tree, a match that ends
beyond its block or frame, an offset below 1 or beyond the decoded bytes or the window, and input that ends inside a frame. Output
beyond `LzxDecompressionOptions.MaxOutputSize` (default 1 GiB) raises `CompressionLimitException` before the frame is decoded.
After an error the decoder refuses frames until `Reset`.

## Speed

Direction only, one loaded x64 host, memory to memory, best of 7, streams from the test compressor at window 2^21, against
`cabextract -p` to `/dev/null` with its process startup (3.5 ms) subtracted: dickens (Silesia, 10.2 MB) 357 MB/s against about 384 MB/s;
the first 20 MB of mozilla (with call translation) 408 MB/s against about 304 MB/s. Not a bar run.


<!-- https://recursiveai.org/onearchive/docs/checksums/ -->

# Checksums

Normative description of the checksums in `RecursiveAI.OneCompress.Checksums`. A port must reproduce every value from this text alone.

| Type | Width | Definition | Check value of `123456789` | Empty input |
|---|---|---|---|---|
| `Crc32` | 32 | reflected polynomial `0xEDB88320`, init and final xor `0xFFFFFFFF` | `0xCBF43926` | `0` |
| `Crc32C` | 32 | reflected polynomial `0x82F63B78`, init and final xor `0xFFFFFFFF` | `0xE3069283` | `0` |
| `Crc64` | 64 | reflected polynomial `0xC96C5795D7870F42` (ECMA-182 as the xz format uses it), init and final xor all ones | `0x995DC9BBDF1939FA` | `0` |
| `Adler32` | 32 | RFC 1950 section 8.2: sums modulo 65521, initial value 1, second sum in the high half | `0x091E01DE` | `1` |
| `XxHash32` | 32 | xxHash specification, XXH32, 32-bit seed (default 0) | — | `0x02CC5D05` (seed 0) |
| `XxHash64` | 64 | xxHash specification, XXH64, 64-bit seed (default 0) | — | `0xEF46DB3751D8E999` (seed 0) |

## API shape (identical for all six)

- `Compute(data)` returns the value for a buffer. The xxHash types also take `Compute(data, seed)`.
- The CRCs and Adler-32 have `Update(value, data)`, which continues a finished value: `Update(Compute(a), b) == Compute(a ++ b)`. A value is the externally visible one, with the final xor applied, so values written into container formats can be fed straight back in.
- The struct (`Crc32`, …) is the incremental form: `Append(data)`, `GetCurrentValue()` (does not disturb the state), `Reset()`. Any split of the input, including empty pieces, gives the same value. `default(T)` is the empty checksum. `XxHash32(seed)` / `XxHash64(seed)` set the seed; `Reset()` keeps it.
- `Crc32.Combine(crcA, crcB, lengthB)` (also `Crc32C`, `Crc64`) returns the checksum of A followed by B. `lengthB` outside `[0, 2^60]` throws `ArgumentOutOfRangeException`.

## CRC combine

With finished values (init and final xor equal), `crc(A ++ B) = crc(A) * x^(8·len(B)) xor crc(B)` in GF(2)[x] modulo the CRC polynomial. `x^n` is computed by square-and-multiply with the bit-reflected representation (bit `w-1-i` holds the coefficient of `x^i`). Cost is O(log n) multiplications of w iterations each.

## Folding constants

The carry-less-multiply kernels fold 128-bit blocks. For a width-`w` CRC, folding a register forward by `d` bits uses the pair `(x^(d+w-1) mod P, x^(d+w-65) mod P)` (low, high), each in the reflected `w`-bit form. The `-1` compensates for the one-bit offset of a reflected 64×64 carry-less product. Distances used: 1024, 512, 256, 128. When the data runs out, the 16-byte remainder is fed through the table-driven update from state zero, which completes the reduction. Every path (tables, instructions, folding with 128-bit or 256-bit multiplies, 4 or 8 accumulators) computes the same function; only speed differs.

## Adler-32 deferral

Sums are reduced modulo 65521 at least every 5552 bytes (the largest run in which the second sum stays below 2^32 for byte input).


<!-- https://recursiveai.org/onearchive/docs/bench-method/ -->

# Benchmark method

Normative description of how the bar is measured. It is language-neutral: another implementation of the
runner must produce comparable numbers by following this text.

## What a row is

One row compares one implementation of a codec with one pinned reference tool at one level on one file:

| field | meaning |
|---|---|
| size ratio | our compressed size / reference compressed size |
| compress, decompress time ratio | our best time / reference best time (equivalently reference MB/s / ours) |
| MB/s | input bytes / 10^6 / seconds, for both directions, on the uncompressed size |
| peak memory | maximum resident set size of each side, in MB (see below) |
| verdict | `pass` when size ratio <= 1.01 and both time ratios <= 1.15, else `FAIL`; `-` for comparison rows |

A level "N" always means the reference tool's level N. A codec passes the bar only when every row for it passes
on both the x64 and the arm64 bench host.

Implementations are either **ours** (gated), including stand-ins used to prove the plumbing before a codec
exists, or **comparison** libraries (ZstdSharp, K4os.Compression.LZ4, the BCL codecs, and so on), which are
reported and never gated.

## Conditions

- **Memory to memory, warm.** The whole input is read into memory before timing; our side compresses into a
  pre-allocated buffer and decompresses into a pre-allocated buffer of the original size. Disk I/O and
  allocation are outside the timed region. Two untimed warm-up runs precede the timed runs so that the JIT has
  reached its final tier.
- **Best of N.** N is 5 unless stated. The row records the fastest run, which is the least disturbed by
  other activity.
- **One thread** unless the level is multithreaded. For a multithreaded level the reference is run with the
  same thread count as ours.
- **Fresh process per measurement** on our side: compression, decompression and the round-trip check each
  run in their own process, so the peak memory of each can be read from the operating system. The round-trip
  check compares the decompressed bytes with the input; a mismatch fails the run.
- **Reference files** live in RAM where the host allows (`/dev/shm` on Linux). On macOS, which has no
  user-mountable RAM file system without privileges, they are ordinary files that are warm in the page cache;
  the row records which was used.
- **Quiet machine.** The benchmark takes a host-wide lock (an atomically created file naming the project and the
  expected end time) and refuses to start while the other work on the host uses more than 2 cores, measured over
  3 seconds from the kernel's CPU counters. A run that overlapped another project's heavy job is discarded and
  repeated.
- **Load.** The 1-minute load average is recorded before, during (every 5 seconds; maximum and mean) and after.
  The run's own single busy thread contributes about 1.

## References

### Tools with an in-memory benchmark mode

`zstd -b<L> -e<L> -i<S> -T<threads> file` and `lz4 -b<L> -e<L> -i<S> -T<threads> file` (S seconds, default 3).
The tool's reported compressed size and best compression and decompression speeds are used as they are. The
tool holds the file in memory and times the codec only.

### Tools without one

The command is run through `sh` on files in the work directory with standard input and output redirected
(`tool -L < in > out`), so the output carries no file name:

1. The same compression and decompression commands run on an empty input, N times; the best time is the
   process **start-up cost** of that command.
2. The command runs on the real input once as a warm-up and then N times; the best time minus the start-up cost
   is the reference time. Decompression is timed the same way on the reference's own output. The decompressed
   length must equal the input length.

The reference's own timed region therefore includes reading the input and writing the output, which is
slightly in its disfavor against our in-memory timing; this is deliberate and stated in each row's method
column. Small inputs (under about 1 MB) make the start-up subtraction noisy; use larger files for gating.

### Container framing

Where the reference writes a framed format but the codec under test is a bare stream, the reference's size is
corrected: the gzip framing (10-byte header and 8-byte trailer, measured as the size of the reference's
output for an empty input minus the 2 bytes of an empty deflate stream) is removed for the `deflate` codec, and
the same with 6 bytes of zlib framing added back for `zlib`. For `gzip` no correction is made.

### Pinned tools

The references are the tools pinned in `tools/pins.tsv` (version, URL, SHA-256), installed user-locally by
`tools/install-tools.sh`:

| codec | reference |
|---|---|
| zstd | `zstd` |
| lz4 | `lz4` |
| deflate, zlib, gzip | zlib-ng's `minigzip` (streaming) and `libdeflate-gzip` (one-shot); both rows are reported |
| brotli | `brotli` |
| xz, lzma | `xz` (single-threaded) |
| bzip2 | `bzip2` |
| lzip | `lzip` |

7-Zip (`7zz`), pigz, zlib's `minigzip`, bsdtar, GNU tar, Info-ZIP, cpio, ar, xorriso, cabextract and the RARLAB
`rar`/`unrar` binaries are installed for interoperability tests and as references for the container and decoder rows.
Every row records the reference's version string.

## Peak memory

Peak memory is the maximum resident set size reported by `/usr/bin/time -v` (Linux, kilobytes) or
`/usr/bin/time -l` (macOS, bytes) for a run of the whole command. Our figure is the larger of the compression
and decompression processes and includes the .NET runtime itself, so the row also records a **baseline**: the same
process compressing an empty input. Net memory of the codec is peak minus baseline. For references with a
benchmark mode (zstd, lz4) the figure is one process that compresses and decompresses.

## Corpora

Fetched, never committed, by `tools/fetch-corpora.sh` into `$ONEARCHIVE_HOME/corpora`, each file pinned by
SHA-256 in `tools/corpora.sha256`:

- **Silesia** (12 files; the published per-file MD5s are checked too), plus `silesia.tar`: a ustar archive of
  the files sorted by name, with owner, group and time zeroed, so that every host builds identical bytes.
- **Canterbury**: the `cantrbry`, `large` and `artificl` sets.
- **enwik8**: the first 10^8 bytes of an English Wikipedia dump (MD5 `a1fa5ffddb56f4953e226637dabbb36a`).
- **Game snapshot data** (`game/seed1`, `game/seed2`): generated, not downloaded, by the `OneGame.Corpus`
  tool of the public onegame repository (MIT OR Apache-2.0) at the commit pinned in `tools/game-corpus.tsv`,
  with `tools/fetch-game-corpus.sh`. Each seed is about 66 MB in about 800,000 mostly tiny files:
  `<game>/<kind>/NNNNNN.bin` plus `manifest.tsv` (game, kind, file, size, sha256, schema, tick); games are
  lanterns, bricks and solitaire, kinds are frames, snapshots, saves, recordings and transfer. The generator is
  deterministic, and the manifest SHA-256 it prints is pinned per seed, so every host builds identical data.
  Dictionaries are trained on seed 1 and benchmarked on seed 2, per (game, kind). Rows for these payload sets
  (ratio against dictionary size, speed on small frames) are measured by `tools/zstd-encode-bench`, `tools/zstd-decode-bench`, `tools/lz4-bench` and `tools/small-bench`.

## Deflate, zlib and gzip

The references for this family have no in-process benchmark mode, so `tools/deflate-bench/refbench.c` (built by
`tools/deflate-bench/build.sh` against the pinned libdeflate and zlib-ng installs) drives their public APIs as a
black box on a file held in memory and times only the library call:

- `libdeflate enc <level>`: one-shot gzip compression into a buffer sized by the library's bound. Our side compresses the
  same file in one call (`DeflateBench enc <file> <level>`), levels 1 to 12.
- `zlibng enc <level>`: streaming deflate fed in 64 KiB pieces. Our side feeds the same pieces through the encoder
  (`--stream`), levels 1 to 9.
- `dec`: decompression of the other tool's own gzip output and of ours (written with `DeflateBench enc --out`), by the
  reference library and by us, at levels 1, 6, 9 and 12 (zlib-ng up to 9).
- `bzip2`: our bzip2 decoder against `bzip2 -dc` on the tool's own `-1` and `-9` output. The CLI has no in-memory mode, so
  its wall time (output to the null device, best of N) has the process start-up subtracted, measured the same way on a
  one-byte stream.

Both sides use the best of N passes after warm-up (N is `--reps`, default 5); ours also reports the median and slowest pass. The gzip wrapper is included on both
sides, so sizes compare directly. `tools/deflate-bench/bar.sh --mode oneshot|stream|decode` prints one row per file and
level with sizes, MB/s of uncompressed data, the time ratio (the reference's MB/s divided by ours),
peak resident memory of each process, and pass or FAIL against the usual limits (size at most 1.01, time at most 1.15).
The row header records CPU, core count, SDK and load before and after. Corpora are silesia.tar and enwik8 for the
bar; the Canterbury and single Silesia files are used for direction.

## LZMA family (xz, lzip, 7z)

References: `xz` (pinned 5.8.4, `-T1`), `lzip` (pinned 1.26) and `7zz` (pinned), all run as black-box command-line programs.
Ours: `Xz.TryCompress` at preset N, `Lzip.TryCompress` at level N, `Xz.TryDecompress`/`Lzip.TryDecompress`, and `SevenZipWriter`
at level N, all one call in memory over a buffer provided by the caller (`tools/lzma-bench/LzmaBench`).

- **Compress and decode rows** (`tools/lzma-bench/bar.sh --mode enc|dec`). The tools have no in-memory benchmark mode, so a
  tool's time is the wall time of one run with the file on standard input and the output to a file or `/dev/null`, minus the
  wall time of the same tool on a one-byte input (best of 20, about a millisecond), best of the repeats. Our time is the best
  of the repeats of the in-process call. A first pass of ours longer than 8 s counts as a repeat; a shorter one is a warm-up and
  is dropped. Rates are MB/s of uncompressed data. Peak memory comes from `/usr/bin/time -v` (`-l` on macOS) around the tool
  and around our process; ours holds the whole input and output, as the in-memory API does, the tool streams.
- **Levels.** xz presets 0-9 and lzip levels 0-9, the same number on both sides; the extreme variants (`-9e`) are not run. Both
  sides use one thread (`xz -T1`; lzip has no threads).
- **Decode rows** take the tool's own output at the level and ours at the level, and decode each with the tool (`-d -c`) and
  with ours; the default levels are 0, 6 and 9. The compressed files are made once per job and shared between steps.
- **Round trip.** Every compressed output of ours that is written is decoded by our decoder and compared with the input, outside
  the timed region. The reference tool's decode of ours is covered by the interop tests.
- **7z rows** compare sizes only: a one-file archive written by `SevenZipWriter` at level N against `7zz a -mx=N -mmt=1` (LZMA2,
  solid, compressed header on both sides), N = 1, 5, 9.
- **Pass rule** as for the other codecs: size at most 1.01 times the tool's, and the tool's rate at most 1.15 times ours.
  Chunk `0.3` of `tools/bar-x64/run.sh` runs the steps in order of cost (presets 0-3, decode, lzip 0-3, 7z sizes, then presets
  4-6, then 7-9 with a single repeat) so that a deadline drops the slowest rows first.

## brotli, bzip2, Deflate64 and LZX (chunk 0.4)

References: `brotli` 1.2.1, `bzip2` 1.0.8, `7zz` 26.05 and `cabextract` 1.11, all pinned and run as command-line programs. The tools' time
is the wall time of one run with output to `/dev/null` (or a RAM file for 7-Zip archives) minus the wall time of the same command on a
one-byte input, best of the repeats; ours is the best of the repeats of the in-process call (`tools/bar04-bench`, and
`tools/brotli-bench` for the brotli encoder). Rates are MB/s of uncompressed data; peak memory comes from `/usr/bin/time` around both
sides (for the brotli encoder, around the tool inside `BrotliBench` and from the process's own peak for ours).

- **brotli encode** (`tools/brotli-bench/bar.sh`): qualities 0 and 5 (3 repeats) and 11 (1 repeat) at window 22, `brotli -q N -w 22`.
  Chunk 0.4 measures each codec at its fastest, default and maximum level on silesia.tar, to keep a run under about 45 minutes.
- **brotli decode** (`bar04-bench/bar.sh --mode brotli-dec`): the tool's stream and ours at qualities 5 and 11, decoded by `brotli -d -c`
  and by `Brotli.TryDecompress` into a buffer that is already allocated. Our output is compared with the original after the timing.
- **bzip2 encode** (`--mode bz2-enc`): levels 1 and 9 against `bzip2 -N -c` on stdin. Our stream is decoded
  by `bzip2 -dc` and compared with the input (column `interop`; a mismatch is a MISS).
- **Deflate64** (`--mode d64-enc|d64-dec`): levels 1, 5, 9 against `7zz a -tzip -mm=Deflate64 -mx=N -mmt=1`. 7-Zip has no raw-stream
  mode, so its compress time includes reading the file, CRC-32 and writing the zip (visible at `-mx=1`, which favours the reference
  there), and its size is the entry's compressed size from the central directory. Decode compares our raw-stream decode with
  `7zz e -so` on a zip (which also verifies the CRC-32); the zip around our stream is written by `d64-zip`, which `7zz` must read
  back to the original bytes (column `interop`).
- **LZX decode** (`--mode lzx-dec`): there are no Microsoft-made cabinets of corpus size, so a one-file cabinet (window 2^21, 32 KiB
  frames, verbatim and aligned blocks) is made by the test compressor in `tests/OneCompress.Tests/Lzx`, compiled into the bench. `cabextract -q -p`
  must extract it byte-exact before it is timed. Both sides read the same bytes; ours goes through `CabReader` with checksums verified.
  The rate depends on how the test compressor parses (greedy, no optimal parsing), so it says how fast the decoder is on such streams only.
- **Quick mode.** `BAR_QUICK=1` (with `tools/bar-x64/run.sh 0.4` or either `bar.sh`) cuts each input to its first 1 MiB and uses one
  repeat, to prove the scripts run; its numbers mean nothing. Use a private `BAR_LOCK_FILE` and `BAR_OUT`.
- **Pass rule** as above. `run.sh 0.4 --list` shows the steps and their expected minutes.

## Managed .NET comparison

`bench/Managed/run.sh` compares our codecs with the fastest implementations written entirely in managed code (ZstdSharp, K4os, SharpCompress, SharpZipLib, DotNetZip, BrotliSharpLib and others, as packages used only from `bench/`). It is single-threaded (except one two-worker Zstandard row), memory to memory, best of three interleaved runs, and decodes each candidate's compressed output with our decoder to prove it is a real stream. Rows give speed and size of each candidate; the native tool is shown for information. It is not part of the pass rule above.

## Running

```
bench/run-bar.sh <codec[,codec]> <levels|all> <file[,file]> [--runs N] [--ref-seconds S] [--threads N]
```

The script builds the runner, takes the lock, checks the machine is quiet, runs, and writes
`bar-<timestamp>.csv` and `.md` into `$ONEARCHIVE_HOME/results`. The CSV is self-contained: each row repeats the
machine, CPU, core count, architecture, SDK, runtime, load figures, reference version, the reference method and
the file-system kind of the work directory.

Machines: the x64 bench host (Linux) and the arm64 bench host (macOS). Timings from different machines are
never compared with each other, only each machine's ours-versus-reference ratio.

## Adding a codec

Implement `IBarCodec` (compress and decompress memory to memory, the level list, whether a level is
multithreaded) and add it to `Registry`. If its reference is not yet in `References`, add the reference with
its command line. Nothing else changes.


<!-- https://recursiveai.org/onearchive/docs/small-payload-bench/ -->

# Small-payload rows

How the per-message path is measured: payloads of a few hundred bytes to a few KiB, each compressed or decompressed on
its own with a trained dictionary, through one reused encoder or decoder. Two numbers matter: payloads per second, and
bytes allocated per payload (which must be zero).

## Inputs

Synthetic payloads are cut from the game snapshot data (see the corpus notes): for a game and a kind (`frames` or
`snapshots`) the bench set's payloads are concatenated in manifest order and cut into pieces of exactly 256, 1,024, 4,096 or
16,384 bytes (2,000 pieces unless the data runs out). Dictionaries of 16 and 112 KiB come from the training set:

- zstd: `zstd --train --maxdict=N` on the first 5,000 payloads of the training set (the reference's trainer). One extra row
  per case set uses a dictionary of 16 KiB trained by this library's trainer on the first 2,000 payloads.
- LZ4: a raw dictionary of evenly spaced training payloads concatenated (LZ4 uses the last 64 KiB of a dictionary).

## Reference

The reference is the pinned zstd and lz4 *libraries* (the install prefixes under `tools/opt/` named in `tools/pins.tsv`),
driven by `tools/small-bench/refsmall.c` through their documented APIs: `ZSTD_compress_usingCDict` and
`ZSTD_decompress_usingDDict` with one context each and the dictionary prepared once; `LZ4F_compressFrame_usingCDict` and
`LZ4F_decompress_usingDict` with one context each. The program is compiled against the pinned prefix, never the system
library. It writes the frames it produced; our side decodes exactly those frames, so both decoders do the same work, and our
output is only checked to round-trip through our own decoder. No checksums and no content size in the LZ4 frames; zstd
frames carry the content size, as the library writes them.

## Method

Each side makes repeated passes over the whole payload set after a warm-up pass; a pass is timed as a whole and a payload
rate is payloads divided by pass time. Rows report the best, the median and the worst pass. A row passes when the output is
at most 1.01 times the reference's size, the median compress and decompress rates are each at least 0.87 of the
reference's, and the reused paths allocate nothing (under half a byte per payload; the counter is the thread's allocated
bytes over all timed passes). Runs go under the bench lock on the bench hosts, with load recorded before and after; the row
set and the tool versions are written at the top of the output.

LZ4 levels: `-8` and `-1` are the fast levels with acceleration 9 and 2 (`--fast=8`, `--fast=1`), `1` and `3` the default
fast level and the first high-compression level.

## Running

```
BENCH_DLL=<SmallBench.dll> tools/small-bench/run.sh [--games ...] [--kinds ...] [--sizes ...] [--dicts ...] [--ours-dict]
```

The bar driver runs it as the `small` step of chunk 3 and summarizes it as PASS or MISS per row.

## Browser-wasm

`smoke/wasm` run with `small` measures the same path under Node, ours only, on generated record-shaped payloads (the host
has no files and Node has no reference tool): `node smoke/wasm/run.mjs <wwwroot> small [seconds]`. A reported number, not a
gate. Build it with `dotnet publish smoke/wasm -c Release -p:RunAOTCompilation=true` for ahead-of-time compiled code; a plain
`dotnet build` runs the interpreter, about eight times slower.


<!-- https://recursiveai.org/onearchive/docs/zstd-decoder-bench/ -->

# Zstandard decoder speed

Method: memory to memory, warm, best pass over 4 s, one thread. The reference is `zstd -b<L> -e<L> -i4` (decompression MB/s)
on the same file. Frames come from the tool (`--no-check`, as `zstd -b` makes them) and from this library's encoder. Our side is
`tools/zstd-decode-bench` (`run.sh` for whole files, `game.sh` for small payloads with trained dictionaries; the dictionary is
trained by the tool on the first 5000 payloads of one game-corpus seed and the payloads decoded come from another). Each row
repeats three times. The pass line is ours ≥ 0.87 × the reference's rate (time ≤ 1.15×).

## Apple M2 (8 cores), zstd 1.5.7, .NET 10.0.402, load 4–11 during the whole-file rows

Ours ÷ reference decode rate, whole files, tool-made frames (three repetitions, range):

| file | L1 | L3 | L9 | L19 |
|---|---|---|---|---|
| silesia.tar | 0.87 | 0.90 | 0.91–0.92 | 0.84–0.90 |
| enwik8 | 0.87–0.89 | 0.93–0.94 | 0.86–0.97 | 0.90–0.92 |

Frames made by this library's encoder (L1–L9) decode at 0.82–0.95 of the tool's rate on the same file; the spread between
repetitions is as large as the difference, so those rows are not a separate result. Peak memory: ours 331 MB on silesia.tar
and 190 MB on enwik8, against 640–980 MB and 309–577 MB for the tool's benchmark mode. NativeAOT decode on silesia.tar
(oneshot, MB/s): L1 1470–1545, L3 1406–1480, L9 1680–1686, L19 1413–1440, close to the JIT numbers.

Game payloads, dictionary 16 KiB and 112 KiB, load 3–4, ours ÷ reference payload rate (two passes):

| game | frames L3 | snapshots L9 | saves L19 |
|---|---|---|---|
| lanterns | 1.74–1.79 | 1.12–1.14 | 0.93–0.97 |
| bricks | 1.89–2.01 | 1.01–1.05 | 1.00–1.02 |
| solitaire | 1.08–1.10 | 0.88–0.89 | 0.90–0.91 |

All game rows are at or above 0.87; the closest are solitaire snapshots (0.88-0.89) and solitaire saves (0.90). Whole-file rows at or near the limit: silesia.tar L1 (0.87) and one L19 repetition (0.84), enwik8 L9 (one 0.86); these were taken under load and need the quiet-window rerun.

## Pending

x64 rows on the benchmark worker and a second Apple M2 pass in a quiet window (the whole-file rows above ran at load 4–11).


<!-- https://recursiveai.org/onearchive/docs/hash-matrix/ -->

# Hash matrix

The compressed bytes of every codec depend only on the input and the options. The hash matrix checks
that: it runs the same list of entries in every column below, writes a manifest of SHA-256 digests per
column, and a compare step names the first entry that differs.

## Entries

An entry is `(codec, option set, input) -> bytes`, named `codec/options/input`. Its digest is the
SHA-256 of the output bytes; the manifest also records the output length. An entry that throws gets the
digest `ERROR:<exception type>`, so a failure in one column shows as a mismatch against the others.
Entries are listed in ordinal (byte-wise) name order.

Entries receive a context with a `Threads` count. An entry for a multithreaded codec passes it on as the
codec's thread option; its output must be the same for every value.

## Inputs

Generated by committed code from SplitMix64 streams with fixed seeds (no `System.Random`, no files):

| name | size | content |
|---|---|---|
| `empty` | 0 | |
| `one-byte` | 1 | `0x5A` |
| `text-small` | 300 | word-frequency text from a generated 512-word vocabulary |
| `text-64k` | 64 KiB | same generator |
| `text-3m` | 3 MiB | same generator |
| `binary-1m` | 1 MiB | 32-byte records: counter, record type, stride-addressed pointer, payload from a 16-entry pool |
| `runs-1m` | 1 MiB | runs of one byte, lengths 1 to about 300, skewed short |
| `zeros-4m` | 4 MiB | all zero |
| `random-1m` | 1 MiB | incompressible |

An entry registers for all of them unless it names a subset.

## Columns

| column | what differs |
|---|---|
| `jit` | x64, JIT, 1 thread (the reference) |
| `aot` | x64, NativeAOT, trimmed |
| `scalar` | `DOTNET_EnableHWIntrinsic=0` |
| `noavx` | `DOTNET_EnableAVX512F=0` and `DOTNET_EnableAVX2=0` |
| `threadsN` | JIT with N threads, N = the machine's core count |
| arm64 columns | `jit`, `aot`, `scalar`, `threadsN` on an arm64 host (osx-arm64) |
| `wasm` | browser-wasm under Node, headless |

Every manifest must equal the reference, `x64-jit`. Manifests are named `<prefix>-<column>.tsv`.

## Manifest format

Tab-separated, UTF-8, `\n` line ends. Lines starting with `#` are the header (column name, framework, OS,
architecture, thread count, vector widths, entry count); the compare step ignores them. Each other line is
`entry <TAB> sha256-hex-lowercase <TAB> output-length`.

## Commands

All scripts live in `tools/hashmatrix/` and take paths and hosts as arguments.

- `columns.sh --out DIR --prefix NAME [--columns jit,aot,scalar,noavx,threadsN,wasm]` runs columns on this
  machine. Add `--include-opt-in` to include opt-in entries.
- `remote.sh --target user@host --prefix NAME --out DIR [--columns ...]` copies the working tree to another
  machine over ssh, runs `columns.sh` there, and fetches the manifests. Extra ssh options go in
  `HASHMATRIX_SSH_OPTS`. For the arm64 host: `--prefix arm64 --columns jit,aot,scalar,threadsN`.
- `compare.sh DIR` compares every `*.tsv` in DIR against `x64-jit.tsv`, prints `equal` or the first
  differing entry per manifest (with both digests), and exits 1 on any mismatch.
- `install-dotnet.sh [dir]` and `tools/node/install.sh [dir]` install the pinned .NET SDK and Node.js
  user-locally; both skip an install that already matches the pin. Node is needed only for `wasm`.

The matrix is not a timing run. The `threadsN` column briefly uses every core; on a shared machine, run it
when no benchmark holds the machine.

## Registering entries

The host is `tools/hashmatrix/Host` (JIT and NativeAOT) with `tools/hashmatrix/Wasm` compiling the same
sources for browser-wasm. A codec adds one file under `tools/hashmatrix/Host/Entries/` with a module
initializer; the runner is never edited:

```csharp
namespace RecursiveAI.HashMatrix;

internal static class FooEntries
{
    [ModuleInitializer]
    internal static void Register()
    {
        for (int level = 1; level <= 9; level++)
        {
            int l = level;
            Matrix.Add("foo", $"level{l}", (input, ctx) => Foo.Compress(input, new FooOptions { Level = l, Threads = ctx.Threads }));
        }
    }
}
```

`Matrix.Add(codec, options, run, inputs...)` registers one entry per input (all standard inputs when none
are named). `Matrix.AddOptIn` registers entries that run only with `--include-opt-in`; the `divergent`
entry (it reads the process architecture and vector width) is one, and exists to show that the compare step
fails when output depends on the machine. Registering the same name twice is an error.

Option sets that the codec's options object can express belong in the matrix: every level, and every
option that changes the stream (dictionary, window size, checksum, block size, filters, job size).

## Entries a platform cannot run

An entry that throws `PlatformNotSupportedException` gets the digest `ERROR:PlatformNotSupportedException`. The run does not count
it as a failure, and the compare step reports it as "not run" instead of a mismatch when the reference column produced a digest. Today
this is the encrypted zip and 7z entries (`zip-encrypt`, `sevenzip-encrypt`) in the `wasm` column, because the browser has no AES;
the `encrypt-primitive` entries show which primitive each platform lacks. Encrypted entries pass `RandomBytes` to fix the salt or IV.


<!-- https://recursiveai.org/onearchive/docs/changelog/ -->

# Changelog

All notable changes are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and versions follow [Semantic Versioning](https://semver.org/).

## [Unreleased]

## [0.7.0-preview.1] - 2026-10-29

Everything is still preview and carries `[Experimental]`. No codec changed since 0.5.1-preview.1, so the README measurements stand. Version 0.6 is reserved for RAR compressed entries.

### Added

- Microsoft compound file writer (`CompoundFileWriter`, `CompoundFileWriterOptions`): versions 3 and 4, FAT, DIFAT, mini stream, red-black directory trees, CLSIDs, state bits and times, reachable as `Archive.Create("x.cfb")` (`docs/cfb.md`).

## [0.5.1-preview.1] - 2026-10-29

Everything is still preview and carries `[Experimental]`. The README has the new measurements from both machines.

### Changed

- Deflate, zlib and gzip: faster one-shot and streaming compression at levels 1-9. The greedy and lazy parsers no longer store literals and the block writer emits literal runs straight from the input (identical bytes by itself); levels 1-3 and 5-9 use shallower hash-chain searches (level 1 enters fewer positions of long matches, levels 6-9 also give up on the lazy look-ahead sooner), so their compressed bytes differ from 0.5 while staying within 1% of libdeflate's size. Levels 0, 4 and 10-12 and Deflate64 are unchanged.
- Zstandard: levels 5-9 (and 6-8 for sources of 128-256 KiB) enter only every fourth position that a long match skips over into the match table, which saves writes to a table far larger than the cache. Compressed bytes change, within 0.2% in size, and stay identical on every platform.
- LZ4: faster high-compression levels (3-12) through a tighter search, and a lazy level 2 that is within 0.2% of the `lz4` tool's size (it was +2.2%). Levels 3-9 search deeper where no match is known yet and only briefly for a longer match one or two bytes later, and take a match of 64 bytes or more without looking further; this makes them 8-19% faster at about the same size. Levels 2-9 are retuned, so their compressed bytes differ from 0.5 while staying within 1% of the tool's size; levels 1 and 10-12 are unchanged.
- Zstandard: levels 1-2 compress 3-11% faster (most on text), with identical output.
- Zstandard dictionary training: COVER and fast-COVER training is 3-9 times faster and uses 4-12 times less memory (a 1024 x 8 KiB sample set trains in under a second in under 100 MiB, from 6.6 s and 1.2 GiB). D-mers are counted once per size and the parameter search is guided; sets up to 256 KiB are searched exhaustively. The new `SearchMode` option selects the search. Trained dictionaries differ from 0.5 (still identical on every platform and thread count) and compress held-out game data within about 1% of the previous ones and of `zstd --train` (the worst case is +1.2% against the tool, at a 4 KiB dictionary).
- Zstandard: levels 10-15 compress 25-45% faster. Levels 10-12 search one step less deep than the `zstd` tool, so their bytes change (within 0.6% in size); levels 13-15 are unchanged.

### Added

- Microsoft compound file reader (`CompoundFileReader`, `ArchiveFormat.CompoundFile`): .doc, .xls, .ppt and .msg containers, versions 3 and 4, with strict validation and `Archive` support (`docs/cfb.md`).
- `bench/Managed`: compares each codec with the fastest managed .NET implementations (ZstdSharp, K4os, SharpCompress and others), reporting speed and size per row.
- `THIRD-PARTY-NOTICES.md`: an entry for the RFC 8878 code tables used by the Zstandard codec.

## [0.5.0-preview.1] - 2026-10-29

Everything is still preview and carries `[Experimental]`. The README has the new measurements from both machines.

### Changed

- Deflate64: a fast decode loop; decompression now runs at 2.0-2.6 times the speed of 7-Zip's on both machines.
- PPMd: encoding and decoding are 1.4-1.6 times faster, with identical output. Decoding now runs at 0.89-0.95 of 7-Zip's speed and encoding at 0.85-0.91.
- LZMA, LZMA2, xz and lzip: faster decoding (fewer unpredictable branches per symbol), with identical output.
- Brotli: faster encoding at qualities 0-5 (quality 0 is about 1.5 times as fast as in 0.4). Qualities 0-2 now find matches and emit symbols in one pass, so their compressed bytes differ from 0.4 while staying within 1% of the `brotli` tool's size; qualities 3-11 are unchanged.
- Zstandard: levels 1-4 compress 2-10% faster, with identical output.

### Fixed

- Zstandard: negative compression levels could write invalid frames on large inputs. A sequence whose state and extra bits filled the encoder's 64-bit bit container exactly was written twice, so the `zstd` tool and this library's decoder rejected the frame or decoded garbage. The same defect could in principle affect any level; it showed first at negative levels on the Silesia corpus. Frames that were valid are unchanged byte for byte.

## [0.4.0-preview.1] - 2026-10-29

Everything new here is preview and carries `[Experimental]`. The README has the measurements against the reference tools on both machines, misses included.

### Added

- PPMd (`ONECOMPRESS010`): variants H and I as `PpmdH`/`PpmdI`, encoders, decoders and `PpmdStream`, byte-identical to 7-Zip's output; zip method 98 and the 7z PPMD coder, read and write. The codec is ported from the public-domain reference sources (see `THIRD-PARTY-NOTICES.md`). Levels 8 and 9 use 128 and 256 MiB of model memory, as 7-Zip does on large inputs. New limit `ArchiveLimits.MaxPpmdMemorySize` (256 MiB). See `docs/ppmd.md`.
- ISO 9660 read and write, with Joliet and Rock Ridge (`docs/iso.md`).
- RAR 5 (read only): list and metadata, stored entries, multi-volume sets; compressed entries raise a typed error. RAR 4 archives and header-encrypted archives are also refused with a typed error.
- Codec registration for the archive layer: by default every codec is available, and an app that wants a smaller trimmed or NativeAOT binary sets the `RecursiveAI.OneArchive.AllCodecs` feature switch to false and registers only the codecs it uses with `ArchiveCodecs`. A missing codec raises `ArchiveUnsupportedException` naming the format, the method and the call to add. A `.tar.zst`-only NativeAOT app shrank from 3.82 to 3.07 MB, without the brotli dictionary (`docs/archive-codecs.md`).
- Benchmarks against the reference tools for brotli, bzip2 compress, Deflate64, LZX and PPMd.

### Changed

- Brotli: faster decoder (a fast command loop) and a much faster encoder at every quality: a bucketed match finder, a fused meta-block writer for qualities 0-5, smaller tables. Qualities 0-9 are still below the tool's speed (see the README). Compressed bytes at qualities 0-9 differ from 0.3, still identical on every platform.
- bzip2 encoder: about twice as fast (two-phase block sort, vector move-to-front, incremental table choice); on x64 it now runs at 1.11-1.14 of `bzip2`. Compressed bytes are unchanged.
- Deflate64 encoder: levels 5-9 use an optimal parse with block splitting, and levels 1-4 share one fast class; output is smaller at levels 4-9.
- Deflate64, bzip2, PPMd, AES and ZipCrypto entries in zip, and the 7z Deflate64, PPMd and AES-256 coders (including encrypted headers), are covered by interop tests with 7-Zip and Info-ZIP. AES needs `System.Security.Cryptography.Aes`, which the browser does not provide; there only ZipCrypto reading works.

### Not supported

- BCJ2 in 7z is listed but refused with a typed "unsupported method" error. Quantum cabinets and cabinets that span files are refused with a typed error.

## [0.3.0-preview.1] - 2026-10-28

Everything new here is preview and carries `[Experimental]`. The LZMA family meets the size target and is below the reference speed in some rows; the README has the measurements from both machines.

### Added

- Brotli (RFC 7932, `ONECOMPRESS008`): decoder with the built-in dictionary and transforms, and encoder at qualities 0-11 (windows 10-24), in three layers. Single streams `.br` and `.tar.br` in the archive layer, detected by file name (brotli streams have no magic number).
- bzip2 encoder at levels 1-9 (`ONECOMPRESS004`), zip method 12 writing, and `.bz2`/`.tar.bz2` writing.
- Deflate64 decoder and encoder (`ONECOMPRESS003`), `Deflate64Stream`, zip method 9 read and write, and the 7z Deflate64 coder.
- LZX decoder (`ONECOMPRESS009`) with verbatim, aligned and uncompressed blocks and call translation.
- 7z writing: solid and non-solid folders, LZMA2, LZMA, Copy, Deflate and Zstandard coders, the branch and delta filters, and encoded headers.
- cpio (newc, crc and odc) and ar (common, GNU and BSD) read and write.
- cab read: stored, MSZIP and LZX folders, every header field, checksums, and a typed error for spanned cabinets and Quantum.
- xz and lzip single streams: `.xz`, `.lz`, `.tar.xz` and `.tar.lz`, read and write.
- Encryption (`ONEARCHIVE001`): zip WinZip AES-128/192/256 (AE-1 and AE-2) read and write, traditional zip encryption read; 7z AES-256 read and write, with encrypted headers. `ZipReaderOptions.Password`, `ZipWriterOptions.Password`, `SevenZipReaderOptions.Password`, `SevenZipWriterOptions.Password`, and `Password` on `ArchiveOpenOptions` and `ArchiveCreateOptions`. A missing or wrong password is an `ArchivePasswordException`. AES needs `System.Security.Cryptography.Aes`, which the browser does not provide.

### Changed

- LZMA-family speed: decoder fast paths and match-finder prefetching. Deflate speed: a faster decoder loop and table build, and faster encoder levels 1-7. Deflate levels 2-4 now hash four bytes, so their output differs from 0.2 (still identical on every platform); other compressed bytes are unchanged.
- 7z reading: empty-stream vectors in the order the format defines, and coders without streams.

### Not supported

- PPMd (zip method 98, 7z PPMD) and BCJ2 are listed but refused with a typed "unsupported method" error.

## [0.2.0-preview.1] - 2026-10-28

Everything new here is preview and carries `[Experimental]`. Deflate, zlib and gzip meet the size target and are below the reference speed; the bzip2 decoder meets both. The README has the measurements.

### Added

- Deflate (RFC 1951), zlib (RFC 1950) and gzip (RFC 1952): decoder and encoder at levels 0-12 with strategies, preset dictionaries and flushes, in three layers (`ONECOMPRESS003`).
- bzip2 decoder: multi-stream and randomized blocks, with limits for hostile input (`ONECOMPRESS004`).
- LZMA and LZMA2 (`ONECOMPRESS005`), xz with CRC32, CRC64 and SHA-256 checks, the branch filters (x86, ARM, ARM-Thumb, ARM64, PowerPC, SPARC, IA-64, RISC-V) and Delta (`ONECOMPRESS006`), and lzip (`ONECOMPRESS007`): decoders and encoders at every preset.
- OneArchive (`ONEARCHIVE001`): `Archive.Open`, `Archive.Extract` and `Archive.Create`, with detection by content, extraction that never writes outside the target, and limits on size, ratio, entry count, path length and nesting.
  - tar: v7, ustar, pax and GNU (long names, sparse files), read and write.
  - zip: read and write with store, Deflate and Zstandard, zip64 and data descriptors; LZMA, xz and bzip2 entries read.
  - 7z: read, with Copy, Deflate, BZip2, LZMA, LZMA2, Delta, the branch filters and Zstandard, solid folders and encoded headers.
  - Single streams: gz, zst and lz4 (read and write) and bz2 (read), and tar inside them.
- `Lz4BlockEncoder`, a reusable block compressor, and a length-prefixed block form read back by `Lz4PrefixedBlock` (not an LZ4 frame). `Lz4Dictionary.Train`.
- `ZstdCompressionDictionary.ForDecompression` and `ToArray`, so one trained dictionary serves both directions.
- `ZstdCompressionOptions.SizeHint`: an estimate of a stream's length that sizes the window and tables without being written to the frame.

### Changed

- Zstandard encoder memory: a stream of unknown length no longer reserves its whole window and binary-tree tables up front (level 22 took 384 MiB for a 1 KiB frame, 640 MiB with a small dictionary; now 129 MiB), and a dictionary's search index is sized by the dictionary, not the source (4 KiB at level 22: 256 MiB, now under 1 MiB). Compressed bytes are unchanged.
- Zstandard and Deflate speed work; compressed bytes of Zstandard are unchanged.

### Not supported

- PPMd (zip method 98, 7z PPMD), BCJ2, Deflate64 and encryption are listed but refused with a typed "unsupported method" error.

## [0.1.0-preview.1] - 2026-10-28

First preview. Both codecs are preview and their public types carry `[Experimental]` (`ONECOMPRESS001` for Zstandard, `ONECOMPRESS002` for LZ4): compressed sizes are at or near the reference tools', and speed is still below them. The README has the measurements.

### Added

- Zstandard (RFC 8878): every level from -7 to 22, frames and skippable frames, dictionaries (raw and formatted), dictionary training, long-distance matching, multithreaded compression in fixed-size jobs (the same bytes at any thread count), and the seekable format. One-liners on `Zstd`; `ZstdEncoder`/`ZstdDecoder` over spans with `OperationStatus`, and `ZstdStream`; the parts beneath (FSE and Huffman tables, match finders, block writer) are public.
- LZ4 block and frame formats, fast and high-compression levels, with dictionaries and linked blocks, in the same three layers.
- Checksums: CRC32, CRC32C, CRC64, Adler-32 and xxHash64, with vector paths.
- Input is treated as untrusted: the Zstandard decoder caps the window it will allocate (8 MiB by default), malformed input is a typed error, and every decoder has a fuzz target.
- NativeAOT, trimming and browser WebAssembly are supported; output is byte-identical on every platform.

[Unreleased]: https://github.com/recursive-ai-oss/onearchive-csharp/compare/v0.7.0-preview.1...HEAD
[0.7.0-preview.1]: https://github.com/recursive-ai-oss/onearchive-csharp/releases/tag/v0.7.0-preview.1
[0.5.1-preview.1]: https://github.com/recursive-ai-oss/onearchive-csharp/releases/tag/v0.5.1-preview.1
[0.5.0-preview.1]: https://github.com/recursive-ai-oss/onearchive-csharp/releases/tag/v0.5.0-preview.1
[0.4.0-preview.1]: https://github.com/recursive-ai-oss/onearchive-csharp/releases/tag/v0.4.0-preview.1
[0.3.0-preview.1]: https://github.com/recursive-ai-oss/onearchive-csharp/releases/tag/v0.3.0-preview.1
[0.2.0-preview.1]: https://github.com/recursive-ai-oss/onearchive-csharp/releases/tag/v0.2.0-preview.1
[0.1.0-preview.1]: https://github.com/recursive-ai-oss/onearchive-csharp/releases/tag/v0.1.0-preview.1


<!-- https://recursiveai.org/onearchive/cookbook/safe-extraction/ -->

# Extract untrusted archives

Extract an archive a user uploaded with OneArchive, and see what its safe defaults refuse: a path that climbs out of the target, and an archive larger than you allow.

`upload.tar` holds one ordinary file and one entry named `../../home/user/.bashrc`, the classic “zip slip”. It was made with Python’s `tarfile`, as an attacker would:

```python
import tarfile, io
with tarfile.open('upload.tar', 'w') as t:
    for name, data in [('readme.txt', b'hello\n'), ('../../home/user/.bashrc', b'echo pwned\n')]:
        i = tarfile.TarInfo(name); i.size = len(data); t.addfile(i, io.BytesIO(data))
```

```csharp
using RecursiveAI.OneArchive;
using RecursiveAI.OneCompress;

// upload.tar came from a user: one good entry, and one whose name climbs out of the target.
using (var archive = Archive.Open("upload.tar"))
    foreach (var e in archive.Entries) Console.WriteLine($"entry: {e.Name}");

try
{
    Archive.Extract("upload.tar", "out");
}
catch (ArchiveSafetyException e)
{
    Console.WriteLine($"refused: rule {e.Rule}, entry {e.EntryName}");
}
Console.WriteLine($"files in out/: {string.Join(", ", Directory.GetFiles("out", "*", SearchOption.AllDirectories))}");

// Opting in to Sanitize drops the ".." components and keeps the file inside the target.
Archive.Extract("upload.tar", "clean", new ArchiveExtractOptions { UnsafePaths = UnsafePathHandling.Sanitize });
Console.WriteLine($"files in clean/: {string.Join(", ", Directory.GetFiles("clean", "*", SearchOption.AllDirectories).Order())}");

// Limits: a 1 KiB budget for the whole archive.
Archive.Create("big.tar").Add(new ArchiveEntry { Name = "big.bin", Kind = ArchiveEntryKind.File, Size = 65536 }, new byte[65536]).Save();
try
{
    using var big = Archive.Open("big.tar", new ArchiveOpenOptions { Limits = new ArchiveLimits { MaxTotalSize = 1024 } });
    big.Extract("limited");
}
catch (CompressionLimitException e)
{
    Console.WriteLine($"limit: {e.LimitName}: {e.Message}");
}
```

```text
entry: readme.txt
entry: ../../home/user/.bashrc
refused: rule path-traversal, entry ../../home/user/.bashrc
files in out/: out/readme.txt
files in clean/: clean/home/user/.bashrc, clean/readme.txt
limit: MaxTotalSize: tar: MaxTotalSize limit exceeded: the input needs 65536 but the limit is 1024
```

## What the defaults refuse

| Rule | Refused by default |
|---|---|
| `path-traversal` | a name that leaves the target, `..` included after `\` is read as `/` |
| `absolute-path` | a leading `/` or `\`, a UNC path, a drive letter |
| `link-escape` | a symbolic link whose target is absolute or climbs above the target |
| `link-write-through` | writing under a path component that is a symbolic link |
| `hardlink-target` | a hard link to anything but a file extracted earlier from the same archive |
| `special-file` | devices and FIFOs |
| `existing-file` | a file that already exists |

Each is an `ArchiveSafetyException` with the rule and the entry name, and each can be loosened one at a time on `ArchiveExtractOptions` (`UnsafePaths`, `Symlinks`, `HardLinks`, `SpecialFiles`, `Existing`).

## Good to know

- Entries before the bad one are already written when it is refused (`out/readme.txt` here). Extract into a fresh directory and delete it on failure.
- Limits are on `ArchiveOpenOptions.Limits`: `MaxTotalSize` (1 GiB by default), `MaxEntryCount` (100,000), `MaxPathLength`, `MaxPathDepth`, `MaxExpansionRatio` (1000) and `MaxWindowSize`. They are checked as each header is read, before any data, and a hit is a `CompressionLimitException` naming the limit, never a truncated file.
- setuid, setgid and sticky bits are dropped, and ownership isn’t restored.

All the rules are in [archives and safety](https://recursiveai.org/onearchive/docs/archive/#safety-defaults).


<!-- https://recursiveai.org/onearchive/cookbook/zip-encryption/ -->

# Zip files with AES encryption

Write zip files with OneArchive that every zip tool opens, encrypt one with WinZip AES-256, and handle a missing or wrong password.

```csharp
using System.IO.Compression;
using RecursiveAI.OneArchive;

Directory.CreateDirectory("report");
File.WriteAllText("report/summary.txt", "Q3: 444 orders shipped.\n");
File.WriteAllText("report/data.csv", string.Concat(Enumerable.Range(1, 500).Select(i => $"{i},order-{i},shipped\n")));

// A plain zip opens with System.IO.Compression and every other zip tool.
Archive.Create("report.zip").Add("report/").Save();
using (var zip = ZipFile.OpenRead("report.zip"))
    foreach (var e in zip.Entries) Console.WriteLine($"System.IO.Compression reads {e.FullName} ({e.Length:N0} bytes)");

// WinZip AES-256: every file entry is encrypted with the password.
Archive.Create("secret.zip", new ArchiveCreateOptions { Password = "correct horse" }).Add("report/").Save();

try { Archive.Extract("secret.zip", "nopass"); }
catch (ArchivePasswordException e) { Console.WriteLine($"no password: {e.Problem}, {e.EntryName}"); }

try
{
    using var wrong = Archive.Open("secret.zip", new ArchiveOpenOptions { Password = "battery staple" });
    wrong.Extract("wrong");
}
catch (ArchivePasswordException e) { Console.WriteLine($"wrong password: {e.Problem}, {e.EntryName}"); }

using (var right = Archive.Open("secret.zip", new ArchiveOpenOptions { Password = "correct horse" }))
    right.Extract("opened");
Console.WriteLine($"right password: {File.ReadAllText("opened/report/summary.txt").Trim()}");
```

```text
System.IO.Compression reads report/ (0 bytes)
System.IO.Compression reads report/data.csv (10,784 bytes)
System.IO.Compression reads report/summary.txt (24 bytes)
no password: Required, report/data.csv
wrong password: Incorrect, report/data.csv
right password: Q3: 444 orders shipped.
```

## Good to know

- `ArchiveCreateOptions.Password` encrypts every file entry, empty files included, with WinZip AES-256; listing needs no password. 7-Zip opens these archives.
- A missing password is an `ArchivePasswordException` with `Problem` `Required`; a wrong one is `Incorrect`, found from the entry’s 2-byte password verifier. The 1 in 65,536 wrong passwords that pass the verifier then fail the authentication code, a `CompressionFormatException`.
- Traditional zip encryption (ZipCrypto) is read, never written: it is broken.
- Passwords never appear in exception messages, logs or `ToString`.
- Zip entries are stored or compressed with Deflate, Deflate64, bzip2, PPMd or Zstandard; zip64 and data descriptors are handled. 7z archives can be encrypted too, with AES-256 and encrypted headers.
- AES needs `System.Security.Cryptography.Aes`, which the browser doesn’t provide, so encryption is not available in WebAssembly.

The encryption formats are in [zip encryption](https://recursiveai.org/onearchive/docs/zip-encryption/).


<!-- https://recursiveai.org/onearchive/cookbook/zstd-dictionaries/ -->

# Zstandard dictionaries for small messages

Train a Zstandard dictionary with OneCompress on samples of small, similar messages, and compress new messages to a fraction of what plain Zstandard manages.

Small payloads, such as game state updates, API messages or log lines, compress badly on their own: each one is too short for the compressor to learn its structure. A dictionary carries that structure in advance.

```csharp
using System.Text;
using RecursiveAI.OneCompress.Zstd;

// Many small, similar payloads: player state messages of 100-200 bytes.
var rng = new Random(1);
string Message(int i) =>
    $$"""{"type":"player_state","tick":{{1000 + i}},"player":"p{{rng.Next(1, 64)}}","pos":[{{rng.Next(0, 4096)}},{{rng.Next(0, 4096)}}],"hp":{{rng.Next(0, 101)}},"weapon":"{{(i % 3 == 0 ? "bow" : "sword")}}","status":["{{(i % 2 == 0 ? "running" : "idle")}}"]}""";
var samples = Enumerable.Range(0, 2000).Select(i => (ReadOnlyMemory<byte>)Encoding.UTF8.GetBytes(Message(i))).ToList();

// Train on part of the traffic; compress messages it has not seen.
var dictionary = ZstdCompressionDictionary.Train(samples[..1500], new ZstdDictionaryTrainingOptions { DictionarySize = 4096 });
Console.WriteLine($"dictionary id {dictionary.Id}, {dictionary.ToArray().Length:N0} bytes");

long raw = 0, plain = 0, withDict = 0;
foreach (var m in samples[1500..])
{
    raw += m.Length;
    plain += Zstd.Compress(m.Span, new ZstdCompressionOptions { Level = 3 }).Length;
    byte[] packed = Zstd.Compress(m.Span, new ZstdCompressionOptions { Level = 3, Dictionary = dictionary });
    withDict += packed.Length;
    byte[] back = Zstd.Decompress(packed, new ZstdDecompressionOptions { Dictionary = dictionary.ForDecompression });
    if (!back.AsSpan().SequenceEqual(m.Span)) throw new Exception("round trip failed");
}
Console.WriteLine($"500 messages: {raw:N0} bytes raw, {plain:N0} with zstd, {withDict:N0} with zstd and the dictionary");
```

```text
dictionary id 958003025, 4,096 bytes
500 messages: 55,508 bytes raw, 52,488 with zstd, 19,536 with zstd and the dictionary
```

## Good to know

- Training is deterministic: the same samples and options give the same dictionary bytes on every platform and with any thread count, so peers can train their own copy. `ToArray()` gives the bytes to store or ship; `ZstdCompressionDictionary.Parse` loads them, and dictionaries in the standard Zstandard format.
- `ForDecompression` is the same dictionary prepared for decoding, made once and kept.
- The frame records the dictionary ID, and a decoder can hold several dictionaries (`ZstdDecompressionOptions.Dictionaries`) and picks the one the frame names.
- For a stream of tiny messages, reuse one encoder and decoder to avoid per-message allocations: see [streams of tiny messages](https://recursiveai.org/onearchive/docs/message-mode/).

Training options, the parameter search and how it compares with `zstd --train` are in [dictionary training](https://recursiveai.org/onearchive/docs/zstd-dictionary-training/).


<!-- https://recursiveai.org/onearchive/cookbook/gzip-interop/ -->

# gzip that other tools read

Compress with OneCompress’s gzip and read it with .NET’s `GZipStream`, and the other way round: the formats are the standard ones, byte for byte.

This recipe uses the Deflate family, so it opts in to `ONECOMPRESS003` as well:

```xml
<NoWarn>$(NoWarn);ONECOMPRESS001;ONECOMPRESS003;ONEARCHIVE001</NoWarn>
```

```csharp
using System.IO.Compression;
using System.Text;
using RecursiveAI.OneCompress.Checksums;
using RecursiveAI.OneCompress.Deflate;

byte[] log = Encoding.UTF8.GetBytes(string.Concat(Enumerable.Range(0, 5000).Select(i => $"2026-10-30T12:{i % 60:D2}:00Z GET /api/items/{i % 97} 200\n")));

// OneCompress writes gzip; System.IO.Compression reads it.
byte[] gz = GZip.Compress(log, new DeflateOptions { Level = 9 });
using (var input = new GZipStream(new MemoryStream(gz), CompressionMode.Decompress))
using (var output = new MemoryStream())
{
    input.CopyTo(output);
    Console.WriteLine($"gzip level 9: {log.Length:N0} -> {gz.Length:N0} bytes; GZipStream reads it back: {output.ToArray().AsSpan().SequenceEqual(log)}");
}

// And the other way round.
var net = new MemoryStream();
using (var w = new GZipStream(net, CompressionLevel.SmallestSize, leaveOpen: true)) w.Write(log);
Console.WriteLine($"GZipStream output read by OneCompress: {GZip.Decompress(net.ToArray()).AsSpan().SequenceEqual(log)}");

// The same bytes on every machine: hash the compressed output to check.
Console.WriteLine($"CRC32 of the gzip bytes: {Crc32.Compute(gz):X8}");
```

```text
gzip level 9: 214,480 -> 16,205 bytes; GZipStream reads it back: True
GZipStream output read by OneCompress: True
CRC32 of the gzip bytes: BF755A76
```

## Good to know

- `GZip`, `ZLib` and `Deflate` take `DeflateOptions`: levels 0 to 12, strategies, preset dictionaries and flushes.
- The compressed bytes are the same on every platform, so a CRC32 or SHA-256 of the output is a valid cache key or test fixture. `RecursiveAI.OneCompress.Checksums` also has CRC32C, CRC64, Adler-32 and xxHash64.

Encoder choices and speeds are in [the Deflate encoder](https://recursiveai.org/onearchive/docs/deflate-encoder/) and [checksums](https://recursiveai.org/onearchive/docs/checksums/).


<!-- https://recursiveai.org/onearchive/cookbook/compound-files/ -->

# Read and write compound files

Write a Microsoft compound file with OneArchive and read it back the same way as a `.doc`, `.xls`, `.ppt` or `.msg` file: storages are directories and streams are files.

The writer is new in 0.7.0-preview.1:

```csharp
using RecursiveAI.OneArchive;
using System.Text;

// Write a Microsoft compound file: storages are directories, streams are files.
byte[] summary = Encoding.UTF8.GetBytes("Quarterly notes");
byte[] chart = new byte[10_000];
Archive.Create("notes.cfb")
    .Add(new ArchiveEntry { Name = "Summary", Kind = ArchiveEntryKind.File, Size = summary.Length }, summary)
    .Add(new ArchiveEntry { Name = "Attachments", Kind = ArchiveEntryKind.Directory }, [])
    .Add(new ArchiveEntry { Name = "Attachments/chart", Kind = ArchiveEntryKind.File, Size = chart.Length }, chart)
    .Save();

byte[] head = File.ReadAllBytes("notes.cfb")[..8];
Console.WriteLine($"notes.cfb: {new FileInfo("notes.cfb").Length:N0} bytes, signature {Convert.ToHexString(head)}");

// Read it back, the same way as a .doc, .xls, .ppt or .msg file.
using var cfb = Archive.Open("notes.cfb");
foreach (var e in cfb.Entries) Console.WriteLine($"  {e.Kind,-9} {e.Size,6:N0}  {e.Name}");
using var s = cfb.OpenEntry(cfb.Find("Summary")!);
Console.WriteLine($"Summary: {new StreamReader(s).ReadToEnd()}");
```

```text
notes.cfb: 12,800 bytes, signature D0CF11E0A1B11AE1
  File          15  Summary
  Directory      0  Attachments
  File      10,000  Attachments/chart
Summary: Quarterly notes
```

## Good to know

- `Archive.Open` detects a compound file by its signature, `D0 CF 11 E0 A1 B1 1A E1`, so legacy Office files and Outlook messages open without naming the format. `Entries` then lists their storages and streams, such as `WordDocument` or `Workbook`.
- `ArchiveBuilder.Add(entry, bytes)` needs the entry’s `Size` to match the content; a mismatch is an `ArgumentException`.
- The writer writes version 3 (512-byte sectors, the default) or version 4 (4096-byte sectors), set with `CompoundFileWriterOptions`.
- The reader validates strictly and has its own limits, including `MaxDirectoryEntries`.

The format details are in [compound files](https://recursiveai.org/onearchive/docs/cfb/).