Recursive AI

Seek in a Zstd-compressed stream in C#

OneArchive is a free, open-source archive library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Read ranges without decoding the whole file. Requires seekable-format Zstd input.

Terminal
dotnet add package RecursiveAI.OneCompress

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneCompress.Zstd;
 
byte[] data = "0123456789abcdef"u8.ToArray();
byte[] packed = ZstdSeekable.Compress(data,
    new() { FrameSize = 8 });
byte[] part = ZstdSeekable.DecompressRange(
    packed, offset: 6, length: 4);
Console.WriteLine(Convert.ToHexString(part));
Text
36373839

The range crosses the eight-byte frame boundary; the output is the hex for 6789.

Unzip the example, then dotnet run.

The full program

The program generates bytes 0 through 99, writes sample.zst in 16-byte frames, and reopens the file. It reads a range across three frames, compares it with the original bytes, and tests a damaged footer. No input download is needed.

C#
using RecursiveAI.OneCompress;
using RecursiveAI.OneCompress.Zstd;
 
byte[] original = Enumerable.Range(0, 100).Select(i => (byte)i).ToArray();
using (var file = File.Create("sample.zst"))
using (var writer = new ZstdSeekableWriter(file,
    new ZstdSeekableOptions { FrameSize = 16 }))
{
    writer.Write(original.AsSpan(0, 37));
    writer.Write(original.AsSpan(37));
    writer.Complete();
}
using (var reader = new ZstdSeekableReader(File.OpenRead("sample.zst")))
{
    byte[] range = new byte[20];
    int count = reader.Read(14, range);
    Console.WriteLine($"Range: {count} bytes; matches: {range.AsSpan().SequenceEqual(original.AsSpan(14, 20))}");
    Console.WriteLine($"Frames decoded: {reader.FramesDecoded}");
    byte[] tail = new byte[10];
    Console.WriteLine($"Tail: {reader.Read(95, tail)} bytes");
}
byte[] corrupt = File.ReadAllBytes("sample.zst");
corrupt[^1] ^= 1;
try
{
    using var reader = new ZstdSeekableReader(corrupt);
}
catch (CompressionFormatException)
{
    Console.WriteLine("Corrupt seek table refused");
}

dotnet run prints:

Text
Range: 20 bytes; matches: True
Frames decoded: 3
Tail: 5 bytes
Corrupt seek table refused

A malformed seek table raises CompressionFormatException; exceeding a configured reader budget raises CompressionLimitException.

Variations

Use the .NET Stream interface. Run the full program first to create sample.zst, then use this standalone snippet:

C#
using RecursiveAI.OneCompress.Zstd;
 
using var stream = new ZstdSeekableStream(
    File.OpenRead("sample.zst"));
stream.Seek(14, SeekOrigin.Begin);
byte[] part = new byte[4];
stream.ReadExactly(part);
Console.WriteLine(Convert.ToHexString(part));
Text
0E0F1011

The short answer

For C# seekable Zstd random access, use ZstdSeekableWriter to produce independent compressed frames and a seek table, then open ZstdSeekableReader. Pass an offset in the uncompressed content and a destination buffer to Read. The one-call DecompressRange API does the same task for a file already held in memory.

This is a format choice made when writing the data. An ordinary Zstandard frame does not gain a random-access index merely because its underlying file stream supports Seek. The seekable-format documentation describes the independent frames and the trailing table used by these APIs.

How do I use seekable Zstd in C#?

Choose the input form first. For an existing file on disk, open a readable, seekable stream and pass it to ZstdSeekableReader. For bytes already in memory, pass ReadOnlyMemory<byte>. A forward-only input is rejected with ArgumentException: the reader needs to inspect the end of the file before it can locate the frames covering your range.

Use the positional reader when your application already knows the logical offset. Use ZstdSeekableStream when a consumer expects the usual read-only .NET Stream interface, including Position and Seek. The variation above reads the same logical content through that interface. To decode from a non-seekable input in order, use ZstdStream instead; it skips the trailing table rather than offering indexed access.

Keep the reader open for related reads. It caches the last decoded frame. Creating a new reader for each tiny range discards that cache and reparses the index. Read returns the actual number of bytes supplied: near the end of the content, that can be less than your buffer length. The full program requests ten bytes at offset 95 and receives five.

How does Zstd seekable random access work?

The table maps compressed frame sizes to decompressed sizes. The reader finds the frame covering your logical offset, decodes it, and continues into additional frames if the requested range crosses a boundary. In the full program, offsets 14 through 33 cross frames starting at 0, 16 and 32, so the range requires three frames.

FrameSize controls where the writer normally cuts the content; EndFrame can cut a frame early. Chunk boundaries in calls to Write do not otherwise determine frame boundaries. The default frame size is 1 MiB. Select a size with your access pattern in mind: a small range still requires decoding its containing frame. These are independent frames; compression history does not carry between them.

Opening validates the seek table’s structure and size accounting, not all compressed payloads. A consistent table can still point at damaged frame data. Frame decoding checks the declared output size and, by default, the table checksum when present. The writer enables table checksums by default; they are independent of the optional checksum inside each Zstandard frame.

The reader also bounds table entry count and decoded frame size before allocating their buffers. Corrupting the final magic, as the example does, is detected on open. Damage inside a frame can instead appear when that frame is read. Handle both stages rather than treating successful construction as evidence that every range is readable.

All onearchive recipes