Compress a buffer in parallel into one Zstandard frame.
dotnet add package RecursiveAI.OneCompressRequires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneCompress.Zstd; byte[] data = new byte[300000]; byte[] packed = Zstd.Compress(data, new() { JobSize = 128 * 1024, Workers = 2 }); Console.WriteLine(packed.Length);
21Unzip the example, then dotnet run.
The full program
The snippet at the top creates a 300,000-byte zero-filled buffer in memory and prints the compressed length. It needs no input file.
The full program creates a 500,000-byte UTF-8 buffer of lines from item=00000; status=ready through item=19999; status=ready, each ending with a newline. It compresses the same buffer with one, two and four workers, keeping job size, overlap, level and checksum settings fixed.
Invalid compression options throw ArgumentOutOfRangeException or ArgumentException; decompression rejects malformed frames or configured resource-limit violations.
using System.Text; using RecursiveAI.OneCompress.Zstd; byte[] input = Encoding.UTF8.GetBytes(string.Concat( Enumerable.Range(0, 20000).Select(i => $"item={i:D5}; status=ready\n"))); byte[]? baseline = null; foreach (int workers in new[] { 1, 2, 4 }) { byte[] compressed = Zstd.Compress(input, new ZstdCompressionOptions { Level = 3, JobSize = 128 * 1024, OverlapSize = 32 * 1024, Workers = workers, Checksum = true }); baseline ??= compressed; bool same = baseline.AsSpan().SequenceEqual(compressed); bool roundTrip = input.AsSpan().SequenceEqual(Zstd.Decompress(compressed)); Console.WriteLine($"Workers={workers}: {input.Length} -> {compressed.Length} bytes; same bytes={same}; round trip={roundTrip}"); }
dotnet run prints:
Workers=1: 500000 -> 12013 bytes; same bytes=True; round trip=True Workers=2: 500000 -> 12013 bytes; same bytes=True; round trip=True Workers=4: 500000 -> 12013 bytes; same bytes=True; round trip=True
Variations
Use one worker per processor. Workers = 0 selects the processor count; the job size must still be nonzero:
using RecursiveAI.OneCompress.Zstd; byte[] data = new byte[300000]; byte[] packed = Zstd.Compress(data, new() { JobSize = 128 * 1024, Workers = 0 }); Console.WriteLine(data.AsSpan().SequenceEqual(Zstd.Decompress(packed)));
TrueRelated recipes
- Train and use Zstandard dictionaries
- Seek in a Zstandard compressed stream
- Compress and decompress LZ4
The short answer
Set both JobSize and Workers in ZstdCompressionOptions, then pass the options to Zstd.Compress. JobSize splits the input into pieces; Workers sets how many pieces can be compressed at once. Setting workers alone does not enable parallel compression because the default job size is zero.
OneCompress writes the jobs into a single Zstandard frame. With the same input, job size, overlap and other compression options, changing the worker count does not change the compressed bytes. The full program demonstrates that rule with one, two and four workers, then decompresses every result.
How do I use Zstandard parallel compression in C#?
The jobs and threads documentation defines the two controls separately. Workers defaults to 1; 0 means one per processor. JobSize = 0 disables splitting and makes the worker setting irrelevant. An input no longer than one job is compressed exactly as without jobs, so use input spanning several jobs when you want parallel work.
The full program fixes the job size at 128 KiB and overlap at 32 KiB. The buffer contains 500,000 bytes, so it spans four jobs. Workers affect scheduling, not the job boundaries. The returned array contains the entire frame; you do not concatenate separately framed files yourself.
Job size is rounded up to a multiple of 128 KiB and capped at 1 GiB. ZstdCompressionOptions.AutoJobSize selects an automatic size, documented as the larger of 1 MiB and four times the smaller of the window size and 4 MiB. Select an explicit size when you need the job boundaries to stay fixed across changes in other settings.
Overlap gives a job bytes from before its starting position to use as matching history. It can affect compressed output, unlike worker count. Keep it fixed when comparing reproducibility. The frame header and checksum are written once; each job contributes blocks in input order regardless of completion order.
For streaming compression, the documented memory model is the window plus Workers * JobSize plus the batch’s output. More workers can therefore require more memory. Streaming call boundaries do not change the bytes, but flushing ends a short job and does change them. Writers that flush frequently should leave jobs off.
What is the default compression level for zstd?
OneCompress uses level 3 by default, including in the snippet at the top. The full program states Level = 3 explicitly so its output comparison has a visible fixed setting. Level 0 also selects the default. The compression options describe positive levels 1 through 22 and negative levels.
Level is separate from parallelism. Keep the level constant when testing whether different worker counts produce identical output. Changing job size, overlap, dictionary or other compression settings is not the same test and may change the bytes.
Is zstd compression lossless?
Yes. Decompression recovers the original byte sequence. The full program checks this with SequenceEqual for every worker count. It also enables Checksum, which adds a content checksum to the frame; checksum writing is off by default in OneCompress.
Lossless does not mean every input becomes smaller. The compressed length depends on the input and options, so the example’s byte count is only its own result. It is not a promised compression ratio for your files. Measure size on your own payloads, and compare decoded bytes when checking that an integration preserves binary content.