---
title: "Hash matrix"
url: https://recursiveai.org/onearchive/docs/hash-matrix/
description: "The compressed bytes of every codec depend only on the input and the options. The hash matrix checks that: it runs the same list of entries in every…"
version: "0.7.0-preview.1"
source: https://github.com/recursive-ai-oss/onearchive-csharp/blob/3fa11f22dbe4e13c90ae9c63f393e945a0c09bba/docs/hash-matrix.md
---

# Hash matrix

The compressed bytes of every codec depend only on the input and the options. The hash matrix checks
that: it runs the same list of entries in every column below, writes a manifest of SHA-256 digests per
column, and a compare step names the first entry that differs.

## Entries

An entry is `(codec, option set, input) -> bytes`, named `codec/options/input`. Its digest is the
SHA-256 of the output bytes; the manifest also records the output length. An entry that throws gets the
digest `ERROR:<exception type>`, so a failure in one column shows as a mismatch against the others.
Entries are listed in ordinal (byte-wise) name order.

Entries receive a context with a `Threads` count. An entry for a multithreaded codec passes it on as the
codec's thread option; its output must be the same for every value.

## Inputs

Generated by committed code from SplitMix64 streams with fixed seeds (no `System.Random`, no files):

| name | size | content |
|---|---|---|
| `empty` | 0 | |
| `one-byte` | 1 | `0x5A` |
| `text-small` | 300 | word-frequency text from a generated 512-word vocabulary |
| `text-64k` | 64 KiB | same generator |
| `text-3m` | 3 MiB | same generator |
| `binary-1m` | 1 MiB | 32-byte records: counter, record type, stride-addressed pointer, payload from a 16-entry pool |
| `runs-1m` | 1 MiB | runs of one byte, lengths 1 to about 300, skewed short |
| `zeros-4m` | 4 MiB | all zero |
| `random-1m` | 1 MiB | incompressible |

An entry registers for all of them unless it names a subset.

## Columns

| column | what differs |
|---|---|
| `jit` | x64, JIT, 1 thread (the reference) |
| `aot` | x64, NativeAOT, trimmed |
| `scalar` | `DOTNET_EnableHWIntrinsic=0` |
| `noavx` | `DOTNET_EnableAVX512F=0` and `DOTNET_EnableAVX2=0` |
| `threadsN` | JIT with N threads, N = the machine's core count |
| arm64 columns | `jit`, `aot`, `scalar`, `threadsN` on an arm64 host (osx-arm64) |
| `wasm` | browser-wasm under Node, headless |

Every manifest must equal the reference, `x64-jit`. Manifests are named `<prefix>-<column>.tsv`.

## Manifest format

Tab-separated, UTF-8, `\n` line ends. Lines starting with `#` are the header (column name, framework, OS,
architecture, thread count, vector widths, entry count); the compare step ignores them. Each other line is
`entry <TAB> sha256-hex-lowercase <TAB> output-length`.

## Commands

All scripts live in `tools/hashmatrix/` and take paths and hosts as arguments.

- `columns.sh --out DIR --prefix NAME [--columns jit,aot,scalar,noavx,threadsN,wasm]` runs columns on this
  machine. Add `--include-opt-in` to include opt-in entries.
- `remote.sh --target user@host --prefix NAME --out DIR [--columns ...]` copies the working tree to another
  machine over ssh, runs `columns.sh` there, and fetches the manifests. Extra ssh options go in
  `HASHMATRIX_SSH_OPTS`. For the arm64 host: `--prefix arm64 --columns jit,aot,scalar,threadsN`.
- `compare.sh DIR` compares every `*.tsv` in DIR against `x64-jit.tsv`, prints `equal` or the first
  differing entry per manifest (with both digests), and exits 1 on any mismatch.
- `install-dotnet.sh [dir]` and `tools/node/install.sh [dir]` install the pinned .NET SDK and Node.js
  user-locally; both skip an install that already matches the pin. Node is needed only for `wasm`.

The matrix is not a timing run. The `threadsN` column briefly uses every core; on a shared machine, run it
when no benchmark holds the machine.

## Registering entries

The host is `tools/hashmatrix/Host` (JIT and NativeAOT) with `tools/hashmatrix/Wasm` compiling the same
sources for browser-wasm. A codec adds one file under `tools/hashmatrix/Host/Entries/` with a module
initializer; the runner is never edited:

```csharp
namespace RecursiveAI.HashMatrix;

internal static class FooEntries
{
    [ModuleInitializer]
    internal static void Register()
    {
        for (int level = 1; level <= 9; level++)
        {
            int l = level;
            Matrix.Add("foo", $"level{l}", (input, ctx) => Foo.Compress(input, new FooOptions { Level = l, Threads = ctx.Threads }));
        }
    }
}
```

`Matrix.Add(codec, options, run, inputs...)` registers one entry per input (all standard inputs when none
are named). `Matrix.AddOptIn` registers entries that run only with `--include-opt-in`; the `divergent`
entry (it reads the process architecture and vector width) is one, and exists to show that the compare step
fails when output depends on the machine. Registering the same name twice is an error.

Option sets that the codec's options object can express belong in the matrix: every level, and every
option that changes the stream (dictionary, window size, checksum, block size, filters, job size).

## Entries a platform cannot run

An entry that throws `PlatformNotSupportedException` gets the digest `ERROR:PlatformNotSupportedException`. The run does not count
it as a failure, and the compare step reports it as "not run" instead of a mismatch when the reference column produced a digest. Today
this is the encrypted zip and 7z entries (`zip-encrypt`, `sevenzip-encrypt`) in the `wasm` column, because the browser has no AES;
the `encrypt-primitive` entries show which primitive each platform lacks. Encrypted entries pass `RandomBytes` to fix the salt or IV.