Open a large zip file in C#

Open the zip with ArchiveLimits.LargeArchive and ZipReaderOptions.LazyDirectory, then list it with archive.EnumerateEntries(): memory stays flat however many entries it has. The program below lists the Linux kernel source, a 295 MB zip of 92,438 entries, through Entries and through EnumerateEntries().

Three things matter with a large zip. The default limits, which suit archives of up to about 100,000 entries and 1 GiB of content, refuse it. The usual zip reader keeps every directory record in memory; LazyDirectory reads them one at a time instead. And Archive.Entries builds a list of every entry, while EnumerateEntries() hands them out one at a time and keeps none.

dotnet new console -o LargeZip && cd LargeZip
dotnet add package RecursiveAI.OneArchive --version 1.1.1
curl -L -o linux-6.12.zip https://github.com/torvalds/linux/archive/refs/tags/v6.12.zip

Put this in Program.cs and run dotnet run -c Release:

using System.Diagnostics;
using RecursiveAI.OneArchive;
using RecursiveAI.OneArchive.Zip;
using RecursiveAI.OneCompress;
 
const string path = "linux-6.12.zip";
Console.WriteLine($"{path}: {new FileInfo(path).Length:N0} bytes");
 
// The default limits refuse it: its files add up to more than 1 GiB.
try
{
    using var archive = Archive.Open(path);
    foreach (var entry in archive.EnumerateEntries()) { }
}
catch (CompressionLimitException e)
{
    Console.WriteLine($"default limits: {e.Message}");
}
 
// LargeArchive raises the entry count and total size limits; every
// other check stays on. LazyDirectory reads the zip's directory one
// record at a time instead of loading it.
var options = new ArchiveOpenOptions
{
    Limits = ArchiveLimits.LargeArchive,
    Zip = new ZipReaderOptions { LazyDirectory = true },
};
List("Entries", archive => archive.Entries);
List("EnumerateEntries()", archive => archive.EnumerateEntries());
 
void List(string how, Func<Archive, IEnumerable<ArchiveEntry>> entries)
{
    long before = GC.GetTotalMemory(true);
    var time = Stopwatch.StartNew();
    using var archive = Archive.Open(path, options);
    long count = 0, size = 0, held = 0;
    foreach (var entry in entries(archive))
    {
        count++;
        size += entry.Size;
        if (count == 50_000)   // halfway: how much memory is in use?
        {
            time.Stop();
            held = GC.GetTotalMemory(true) - before;
            time.Start();
        }
    }
    double mib = held / 1048576.0;
    Console.WriteLine($"{how}: {count:N0} entries, {size:N0} bytes");
    Console.WriteLine($"  {time.ElapsedMilliseconds} ms, {mib:F1} MiB held halfway");
}

linux-6.12.zip: 295,237,397 bytes
default limits: zip: MaxTotalSize limit exceeded: the total reached 1073928450 when it passed the limit of 1073741824; the archive may be larger
Entries: 92,438 entries, 1,476,498,481 bytes
  236 ms, 59.6 MiB held halfway
EnumerateEntries(): 92,438 entries, 1,476,498,481 bytes
  216 ms, 0.1 MiB held halfway

With LargeArchive the archive opens. Listed through EnumerateEntries() with LazyDirectory, it holds 0.1 MiB, against 59.6 MiB for Entries, and lists a little faster (one run on an AMD Ryzen 9 7950X3D; three runs agreed). The gap grows with the entry count: Entries costs about 700 bytes per zip entry, and the project measured 630 MiB held for a million-entry zip without LazyDirectory, against under 1 MiB with it.

The limit message gives the total when the limit was passed, not the archive’s size: these files add up to 1.48 GB.

Extract it

Archive.Extract takes the same two settings, and keeps no state that grows with the number of files:

var options = new ArchiveExtractOptions
{
    Limits = ArchiveLimits.LargeArchive,
    Zip = new ZipReaderOptions { LazyDirectory = true },
};
Archive.Extract("linux-6.12.zip", "linux", options);

On the same machine it wrote 86,618 files, 62 links and 5,758 folders (1.6 GB) in 5.6 s, with a peak working set of 100 MiB.

Limits

• Flat memory needs both. EnumerateEntries() without LazyDirectory still holds the zip’s directory, and Entries with it still builds its list. Use EnumerateEntries(), OpenReader() or Extract with LazyDirectory on.
• A seekable stream. Lazy mode reads the directory and then jumps to each entry, so it needs a file or another stream that can seek.
• Listing doesn’t read content. Entries from EnumerateEntries() aren’t tied to OpenEntry, and Find builds the full list. To read files as you go, use archive.OpenReader() and OpenEntryStream(), as in Extract a tar.gz file.
• Zip only. LazyDirectory is a zip option; the other formats keep the per-entry state their readers need.

Good to know

• LargeArchive changes two limits: MaxEntryCount to 4,000,000 and MaxTotalSize (the sum of declared sizes, which also bounds the disk an extraction can fill) to 256 GiB. The expansion ratio, path, window and link checks stay as they are, so a zip bomb or a “zip slip” entry is still refused. For other sizes, set the two values on your own ArchiveLimits.
• EnumerateEntries() reads the archive again from the start on each call, with the limits applied.
• The directory is still checked as a whole before the first entry, with the same overlap and shared-header checks and the same errors as the usual reader.

The details are in Archives and safety and the lazy central directory.