onearchiveDocs
onearchive docs
Everything about OneArchive and OneCompress: getting started, the archive layer and its safety defaults, each container, each codec, checksums, and how they are measured. Both packages are previews: 0.7.0-preview.1.
Start
Getting startedInstall the OneArchive and OneCompress previews for .NET 10, compress a buffer with Zstandard, then create, list, read and safely extract a .tar.zst archive, all in managed code with no native libraries.OverviewCompression codecs and archive formats for .NET, written from first principles in managed code. No native dependencies; the compressed bytes are identical on every platform.Archives and safetyHow OneArchive models an archive, what extraction refuses by default, and what the tar reader and writer do. This is normative: a port must reproduce the writer's bytes and the defaults below from this text.Codecs for a small binaryArchive and the per-format readers and writers reach codecs only through a registry. By default nothing needs configuring: the first time an archive needs a codec, every codec is registered, so every format and method works, and the app carries every codec.Codec conventionsHow every OneCompress codec is shaped, so the codecs look alike and a port can follow them. The Zstandard decoder (src/OneCompress/Zstd/) is the reference implementation of each point.
Containers
Zip encryptionWinZip AES (AE-1 and AE-2) is read and written; traditional PKWARE encryption is read only. PKWARE's strong encryption (bit 6) and central-directory encryption are not supported: they sit under PKWARE's licence terms. Sources: the WinZip document "AES Encryption Information…7z readerHow SevenZipReader (src/OneArchive/SevenZip/) reads 7z archives and the choices the format document leaves open. Sources: 7zFormat.txt (header structures) and Methods.txt (method IDs) from the 7-Zip documentation; every point the documents do not settle was fixed by a…7z writerHow SevenZipWriter (src/OneArchive/SevenZip/) writes 7z archives and the choices the format document leaves open. Sources: 7zFormat.txt (structures) and Methods.txt (method IDs); every point they do not settle was fixed by listing archives written by 7-Zip 26.00 (see the…7z AES-2567zFormat.txt and Methods.txt give the method ID and say nothing about its properties, key derivation or padding. Those were pinned by experiment against 7-Zip 26.00 (7z, used as a black box), written down here before the code that uses them. A value enters the code only…cpio and arNormative notes for the Cpio and Ar containers in RecursiveAI.OneArchive: what is read, what is written, and every choice the formats leave open. A port should reproduce the same bytes from this text.ISO 9660Normative notes for RecursiveAI.OneArchive.Iso. Clause numbers refer to ECMA-119; SUSP is IEEE P1281 and RRIP is IEEE P1282.Compound filesNormative notes for CompoundFileReader and CompoundFileWriter in RecursiveAI.OneArchive.Cfb. Source: [MS-CFB] Compound File Binary File Format, revision 13.0, sections 2.1 to 2.9 (cited as "MS-CFB 2.x"). The format is the container of legacy Office documents (.doc, .xls…Cabinet readerNormative notes for CabReader in RecursiveAI.OneArchive.Cab: read only. Source: Microsoft's [MS-CAB] (Cabinet File Format) 2.1 to 2.4 and 3.1; LZX is in lzx-decoder.md. Gaps were closed with cabextract 1.11 and gcab 1.6 run as black boxes; findings are marked [tool].RAR 5 readerNormative notes for RarReader in RecursiveAI.OneArchive.Rar: read only. Source: RARLAB's "RAR 5.0 archive format" technote (section names are cited in the code). The technote sends readers to the vendor's source for algorithms; that is never followed. Gaps were closed by…
Zstandard
Zstandard encoderWhat the encoder does where the format leaves a choice. Compressed bytes depend only on the input and the options, so a later port that follows this text and the format (RFC 8878) writes the same bytes. Source: src/OneCompress/Zstd/.Zstandard decoderNormative notes for RecursiveAI.OneCompress.Zstd's decoder. The format itself is RFC 8878; this page records what the format leaves to the decoder and the choices made, so another implementation can match them.Dictionary trainingZstdDictionary.Train(samples, options) turns a set of sample payloads into a dictionary in the Zstandard dictionary format (RFC 8878 §5). The pieces are public so that they can be composed: ZstdCoverTrainer (content selection, parameter search, and the whole pipeline)…Seekable formatReading and writing seekable Zstandard files: ordinary frames, each compressed on its own, followed by one skippable frame (the seek table) that maps content offsets to frames. A seekable file is a valid Zstandard file; any decoder reads it start to end and ignores the table.…Streams of tiny messagesFor a connection that carries many messages of tens to a few hundred bytes, there are two ways to compress them. Both keep the encoder and its buffers for the life of the connection, so a steady state allocates nothing.
Other codecs
LZ4 encoderHow the LZ4 compressors in RecursiveAI.OneCompress.Lz4 choose and write bytes. A later port reproduces the same output from this page and the format documents (LZ4 Block Format and LZ4 Frame Format). The decoder is described in lz4-decoder.md.LZ4 decoderWhat the LZ4 decoder accepts, rejects and bounds. The format sources are the LZ4 Block Format and Frame Format documents (lz4_Block_format.md, lz4_Frame_format.md); section names below refer to them. A later port should reproduce these choices from this page alone.Deflate encoderNormative and language-neutral: a port that follows this text produces the same bytes. RFC 1951 leaves the parse, the block boundaries and the Huffman code construction to the encoder; everything it leaves open is fixed here. Output depends only on the input bytes, the…Deflate decoderWhat the decoder accepts, rejects and bounds. The format sources are RFC 1951 (Deflate), RFC 1950 (zlib) and RFC 1952 (gzip); section names below refer to them. A later port should reproduce these choices from this page alone.Deflate64 encoderNormative notes for Deflate64Encoder: what a later port must do to reproduce the same bytes. The symbols and limits are in deflate64-derivation.md.Deflate64 decoderNormative notes for Deflate64Decoder. The format facts (symbols, extra bits, window, header limits) are in deflate64-derivation.md, with the experiment behind each. This file records what the decoder does with them.Deflate64 derivationDeflate64 (zip method 9) has no specification we may implement from. This note records, for every difference from Deflate (RFC 1951), the hypothesis, where it came from, the experiment that tested it, and the value the codec uses. A value enters the code only through a public…LZMA and xz encodersNormative and language-neutral: a port that follows this text produces the same bytes. The LZMA specification, the xz file format specification and the lzip manual fix the streams a decoder accepts; everything they leave to the encoder (the match finders, the parse, the price…LZMA and xz decodersRecursiveAI.OneCompress.Lzma, .Xz, .Lzip and .Filters. The formats are specified by the LZMA specification document (lzma-specification.txt), the xz file format specification (xz-file-format.txt) and the lzip manual's format chapter. This document records only what those…bzip2 encoderRecursiveAI.OneCompress.BZip2: compression to bzip2 streams. The format is described in bzip2-decoder.md; every choice the format leaves to the writer, with the experiments that pinned or bounded it, is in bzip2-encoder-derivation.md (fact numbers F1…F9 below). A later port…bzip2 decoderRecursiveAI.OneCompress.BZip2: decoding of bzip2 data. bzip2 has no official specification. This document records the format as the decoder reads it, how each point was established, and every choice the format leaves open. A later port must reproduce the decoder's behaviour…bzip2 encoder derivationbzip2 has no official specification, and the decoder note (bzip2-decoder.md) already records what a reader must accept. This note covers the other half: every choice the format leaves to the writer that changes the compression ratio or whether a reader accepts the stream. For…Brotli encoderNormative and language-neutral: a port that follows this text produces the same bytes. RFC 7932 fixes what a brotli stream means and leaves the parse, the block boundaries, the context modelling and the prefix-code construction to the encoder; everything it leaves open is…Brotli decoderRecursiveAI.OneCompress.Brotli: decoding of brotli data as RFC 7932 defines it. This document records the choices the RFC leaves to the implementation. The format itself is the RFC; section numbers below are its.PPMdTwo codecs from Dmitry Shkarin's PPMd family, byte-identical to what 7-Zip writes and reads:LZX decoderNormative notes for RecursiveAI.OneCompress.Lzx: LZX as cabinet folders store it, decode only. A port should decode the same streams from this text. No encoder exists.ChecksumsNormative description of the checksums in RecursiveAI.OneCompress.Checksums. A port must reproduce every value from this text alone.
How it is measured
Benchmark methodNormative description of how the bar is measured. It is language-neutral: another implementation of the runner must produce comparable numbers by following this text.Small payloadsHow the per-message path is measured: payloads of a few hundred bytes to a few KiB, each compressed or decompressed on its own with a trained dictionary, through one reused encoder or decoder. Two numbers matter: payloads per second, and bytes allocated per payload (which…Zstandard decoder speedMethod: memory to memory, warm, best pass over 4 s, one thread. The reference is zstd -b -e -i4 (decompression MB/s) on the same file. Frames come from the tool (--no-check, as zstd -b makes them) and from this library's encoder. Our side is tools/zstd-decode-bench (run.sh…Hash matrixThe compressed bytes of every codec depend only on the input and the options. The hash matrix checks that: it runs the same list of entries in every column below, writes a manifest of SHA-256 digests per column, and a compare step names the first entry that differs.