---
title: "Compress Zstandard with parallel jobs in C#"
url: https://recursiveai.org/onearchive/cookbook/compress-zstd-with-parallel-jobs/
description: "Split a buffer into compression jobs and run several at once while writing one Zstandard frame. Runnable C# example using OneCompress."
version: "1.2.0"
---

# Compress Zstandard with parallel jobs in C#

**OneArchive** is a free, open-source archive and compression library for .NET from Recursive AI: archives (zip, tar, 7z, RAR 5, cpio, ar, ISO) in RecursiveAI.OneArchive, codecs (zlib, gzip, xz, Zstandard, Brotli) in RecursiveAI.OneCompress. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneCompress) · [GitHub](https://github.com/recursive-ai-oss/onearchive-csharp) · [Docs](https://recursiveai.org/onearchive/docs/) · [Download example (.zip)](https://recursiveai.org/onearchive/cookbook/compress-zstd-with-parallel-jobs/compress-zstd-with-parallel-jobs.zip)

Compress a buffer in parallel into one Zstandard frame.

```shell
dotnet add package RecursiveAI.OneCompress
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneCompress.Zstd;
byte[] data = new byte[300000];
byte[] packed = Zstd.Compress(data, new()
{
    JobSize = 128 * 1024, Workers = 2
});
Console.WriteLine(packed.Length);
```

```text
21
```

Unzip [the example](https://recursiveai.org/onearchive/cookbook/compress-zstd-with-parallel-jobs/compress-zstd-with-parallel-jobs.zip), then `dotnet run`.

## The full program

The snippet at the top creates a 300,000-byte zero-filled buffer in memory and prints the compressed length. It needs no input file.

The full program creates a 500,000-byte UTF-8 buffer of lines from `item=00000; status=ready` through `item=19999; status=ready`, each ending with a newline. It compresses the same buffer with one, two and four workers, keeping job size, overlap, level and checksum settings fixed.

Invalid compression options throw `ArgumentOutOfRangeException` or `ArgumentException`; decompression rejects malformed frames or configured resource-limit violations.

```csharp
using System.Text;
using RecursiveAI.OneCompress.Zstd;

byte[] input = Encoding.UTF8.GetBytes(string.Concat(
    Enumerable.Range(0, 20000).Select(i => $"item={i:D5}; status=ready\n")));
byte[]? baseline = null;
foreach (int workers in new[] { 1, 2, 4 })
{
    byte[] compressed = Zstd.Compress(input, new ZstdCompressionOptions
    {
        Level = 3,
        JobSize = 128 * 1024,
        OverlapSize = 32 * 1024,
        Workers = workers,
        Checksum = true
    });
    baseline ??= compressed;
    bool same = baseline.AsSpan().SequenceEqual(compressed);
    bool roundTrip = input.AsSpan().SequenceEqual(Zstd.Decompress(compressed));
    Console.WriteLine($"Workers={workers}: {input.Length} -> {compressed.Length} bytes; same bytes={same}; round trip={roundTrip}");
}
```

`dotnet run` prints:

```text
Workers=1: 500000 -> 12013 bytes; same bytes=True; round trip=True
Workers=2: 500000 -> 12013 bytes; same bytes=True; round trip=True
Workers=4: 500000 -> 12013 bytes; same bytes=True; round trip=True
```

## Variations

**Use one worker per processor.** `Workers = 0` selects the processor count; the job size must still be nonzero:

```csharp
using RecursiveAI.OneCompress.Zstd;

byte[] data = new byte[300000];
byte[] packed = Zstd.Compress(data, new()
{
    JobSize = 128 * 1024, Workers = 0
});
Console.WriteLine(data.AsSpan().SequenceEqual(Zstd.Decompress(packed)));
```

```text
True
```

## Related recipes

- [Train and use Zstandard dictionaries](https://recursiveai.org/onearchive/cookbook/zstd-dictionaries/)
- [Seek in a Zstandard compressed stream](https://recursiveai.org/onearchive/cookbook/seek-in-zstd-compressed-stream/)
- [Compress and decompress LZ4](https://recursiveai.org/onearchive/cookbook/lz4-compress-decompress/)

<!-- article -->

## The short answer

Set both `JobSize` and `Workers` in `ZstdCompressionOptions`, then pass the options to `Zstd.Compress`. `JobSize` splits the input into pieces; `Workers` sets how many pieces can be compressed at once. Setting workers alone does not enable parallel compression because the default job size is zero.

OneCompress writes the jobs into a single Zstandard frame. With the same input, job size, overlap and other compression options, changing the worker count does not change the compressed bytes. The full program demonstrates that rule with one, two and four workers, then decompresses every result.

## How do I use Zstandard parallel compression in C#?

The [jobs and threads documentation](https://recursiveai.org/onearchive/docs/zstd-encoder/#jobs-and-threads) defines the two controls separately. `Workers` defaults to 1; 0 means one per processor. `JobSize = 0` disables splitting and makes the worker setting irrelevant. An input no longer than one job is compressed exactly as without jobs, so use input spanning several jobs when you want parallel work.

The full program fixes the job size at 128 KiB and overlap at 32 KiB. The buffer contains 500,000 bytes, so it spans four jobs. Workers affect scheduling, not the job boundaries. The returned array contains the entire frame; you do not concatenate separately framed files yourself.

Job size is rounded up to a multiple of 128 KiB and capped at 1 GiB. `ZstdCompressionOptions.AutoJobSize` selects an automatic size, documented as the larger of 1 MiB and four times the smaller of the window size and 4 MiB. Select an explicit size when you need the job boundaries to stay fixed across changes in other settings.

Overlap gives a job bytes from before its starting position to use as matching history. It can affect compressed output, unlike worker count. Keep it fixed when comparing reproducibility. The frame header and checksum are written once; each job contributes blocks in input order regardless of completion order.

For streaming compression, the documented memory model is the window plus `Workers * JobSize` plus the batch’s output. More workers can therefore require more memory. Streaming call boundaries do not change the bytes, but flushing ends a short job and does change them. Writers that flush frequently should leave jobs off.

## What is the default compression level for zstd?

OneCompress uses level 3 by default, including in the snippet at the top. The full program states `Level = 3` explicitly so its output comparison has a visible fixed setting. Level 0 also selects the default. The [compression options](https://recursiveai.org/onearchive/docs/zstd-encoder/) describe positive levels 1 through 22 and negative levels.

Level is separate from parallelism. Keep the level constant when testing whether different worker counts produce identical output. Changing job size, overlap, dictionary or other compression settings is not the same test and may change the bytes.

## Is zstd compression lossless?

Yes. Decompression recovers the original byte sequence. The full program checks this with `SequenceEqual` for every worker count. It also enables `Checksum`, which adds a content checksum to the frame; checksum writing is off by default in OneCompress.

Lossless does not mean every input becomes smaller. The compressed length depends on the input and options, so the example’s byte count is only its own result. It is not a promised compression ratio for your files. Measure size on your own payloads, and compare decoded bytes when checking that an integration preserves binary content.
