Zstandard dictionaries for small messages · onearchive cookbook

Train a Zstandard dictionary with OneCompress on samples of small, similar messages, and compress new messages to a fraction of what plain Zstandard manages.

Small payloads, such as game state updates, API messages or log lines, compress badly on their own: each one is too short for the compressor to learn its structure. A dictionary carries that structure in advance.

using System.Text;
using RecursiveAI.OneCompress.Zstd;
 
// Many small, similar payloads: player state messages of 100-200 bytes.
var rng = new Random(1);
string Message(int i) =>
    $$"""{"type":"player_state","tick":{{1000 + i}},"player":"p{{rng.Next(1, 64)}}","pos":[{{rng.Next(0, 4096)}},{{rng.Next(0, 4096)}}],"hp":{{rng.Next(0, 101)}},"weapon":"{{(i % 3 == 0 ? "bow" : "sword")}}","status":["{{(i % 2 == 0 ? "running" : "idle")}}"]}""";
var samples = Enumerable.Range(0, 2000).Select(i => (ReadOnlyMemory<byte>)Encoding.UTF8.GetBytes(Message(i))).ToList();
 
// Train on part of the traffic; compress messages it has not seen.
var dictionary = ZstdCompressionDictionary.Train(samples[..1500], new ZstdDictionaryTrainingOptions { DictionarySize = 4096 });
Console.WriteLine($"dictionary id {dictionary.Id}, {dictionary.ToArray().Length:N0} bytes");
 
long raw = 0, plain = 0, withDict = 0;
foreach (var m in samples[1500..])
{
    raw += m.Length;
    plain += Zstd.Compress(m.Span, new ZstdCompressionOptions { Level = 3 }).Length;
    byte[] packed = Zstd.Compress(m.Span, new ZstdCompressionOptions { Level = 3, Dictionary = dictionary });
    withDict += packed.Length;
    byte[] back = Zstd.Decompress(packed, new ZstdDecompressionOptions { Dictionary = dictionary.ForDecompression });
    if (!back.AsSpan().SequenceEqual(m.Span)) throw new Exception("round trip failed");
}
Console.WriteLine($"500 messages: {raw:N0} bytes raw, {plain:N0} with zstd, {withDict:N0} with zstd and the dictionary");

Text

dictionary id 958003025, 4,096 bytes
500 messages: 55,508 bytes raw, 52,488 with zstd, 19,536 with zstd and the dictionary

Good to know

Training is deterministic: the same samples and options give the same dictionary bytes on every platform and with any thread count, so peers can train their own copy. ToArray() gives the bytes to store or ship; ZstdCompressionDictionary.Parse loads them, and dictionaries in the standard Zstandard format.
ForDecompression is the same dictionary prepared for decoding, made once and kept.
The frame records the dictionary ID, and a decoder can hold several dictionaries (ZstdDecompressionOptions.Dictionaries) and picks the one the frame names.
For a stream of tiny messages, reuse one encoder and decoder to avoid per-message allocations: see streams of tiny messages.

Training options, the parameter search and how it compares with zstd --train are in dictionary training.