# onemodel docs, all pages (OneModel 0.3.0-preview.1)

Every page of https://recursiveai.org/onemodel/docs/ and https://recursiveai.org/onemodel/cookbook/ as markdown, getting started first. Index: https://recursiveai.org/onemodel/llms.txt


<!-- https://recursiveai.org/onemodel/docs/getting-started/ -->

# Getting started with OneModel

Install the OneModel preview for .NET 10, run an ONNX model with one call, read its inputs and outputs, and see which kernels it picked. No native binaries, no ONNX Runtime.

## Install

OneModel targets `net10.0`, so you need the .NET 10 SDK. It is a preview: this guide uses 0.3.0-preview.1, and the API may change before 1.0.

```shell
dotnet new console -o OneModelStart
cd OneModelStart
dotnet add package RecursiveAI.OneModel --version 0.3.0-preview.1
```

The package has no dependency beyond the .NET base class library and contains no native code. It works under NativeAOT and trimming; for a NativeAOT build, see [publishing with the right instruction set](https://recursiveai.org/onemodel/docs/overview/#nativeaot-users-set-ilcinstructionset).

## The sample models

The guide and the [cookbook](https://recursiveai.org/onemodel/cookbook/) use three small models made with the `onnx` Python package (`pip install onnx numpy`), so the outputs below can be reproduced exactly. Any float32 ONNX model with one input and one output works the same way. `models.py`:

```python
import numpy as np, onnx
from onnx import helper as h, TensorProto as T, numpy_helper as nh

rng = np.random.default_rng(7)

def w(name, *shape):
    return nh.from_array(rng.standard_normal(shape).astype(np.float32) * 0.5, name)

# classifier.onnx: 4 features -> 8 hidden (Relu) -> 3 classes (Softmax)
g = h.make_graph(
    [h.make_node("Gemm", ["x", "W1", "b1"], ["h"], transB=1),
     h.make_node("Relu", ["h"], ["hr"]),
     h.make_node("Gemm", ["hr", "W2", "b2"], ["logits"], transB=1),
     h.make_node("Softmax", ["logits"], ["probs"], axis=1)],
    "classifier",
    [h.make_tensor_value_info("x", T.FLOAT, ["batch", 4])],
    [h.make_tensor_value_info("probs", T.FLOAT, ["batch", 3])],
    [w("W1", 8, 4), w("b1", 8), w("W2", 3, 8), w("b2", 3)])
m = h.make_model(g, opset_imports=[h.make_opsetid("", 17)], ir_version=8)
onnx.checker.check_model(m)
onnx.save(m, "classifier.onnx")

# scorer.onnx: two named inputs, two outputs
g = h.make_graph(
    [h.make_node("MatMul", ["features", "W"], ["proj"]),
     h.make_node("Add", ["proj", "bias"], ["score"]),
     h.make_node("ReduceMax", ["score"], ["best"], axes=[1], keepdims=0)],
    "scorer",
    [h.make_tensor_value_info("features", T.FLOAT, ["batch", 3]),
     h.make_tensor_value_info("bias", T.FLOAT, [1, 2])],
    [h.make_tensor_value_info("score", T.FLOAT, ["batch", 2]),
     h.make_tensor_value_info("best", T.FLOAT, ["batch"])],
    [nh.from_array(np.array([[1, 0], [0, 1], [1, 1]], np.float32), "W")])
m = h.make_model(g, opset_imports=[h.make_opsetid("", 13)], ir_version=8)
onnx.checker.check_model(m)
onnx.save(m, "scorer.onnx")

# wide.onnx: a larger float32 model for half-precision weight storage (256 -> 512 -> 64)
g = h.make_graph(
    [h.make_node("MatMul", ["x", "W1"], ["h"]),
     h.make_node("Relu", ["h"], ["hr"]),
     h.make_node("MatMul", ["hr", "W2"], ["y"])],
    "wide",
    [h.make_tensor_value_info("x", T.FLOAT, [1, 256])],
    [h.make_tensor_value_info("y", T.FLOAT, [1, 64])],
    [nh.from_array((rng.standard_normal((256, 512)) * 0.05).astype(np.float32), "W1"),
     nh.from_array((rng.standard_normal((512, 64)) * 0.05).astype(np.float32), "W2")])
m = h.make_model(g, opset_imports=[h.make_opsetid("", 17)], ir_version=8)
onnx.checker.check_model(m)
onnx.save(m, "wide.onnx")
print("ok")
```

`classifier.onnx` takes four features and returns three class probabilities. Run `python3 models.py` and copy `classifier.onnx` next to `Program.cs`.

## Run a model

`Model.Load` reads the file and builds a session; `Run` takes the input values in row-major order and returns the output as a new array. Names and shapes come from the model. Here the model declares its input as `[batch, 4]`, so four values are one row:

```csharp
using RecursiveAI.OneModel;

using var model = Model.Load("classifier.onnx");

float[] features = [5.1f, 3.5f, 1.4f, 0.2f];
float[] probs = model.Run(features);

Console.WriteLine(string.Join(", ", probs.Select(p => p.ToString("F4"))));
Console.WriteLine($"Kernels: {model.Session.KernelPath}");
foreach (var input in model.Session.Inputs)
    Console.WriteLine($"Input {input.Name}: {input.DType}, rank {input.Rank}");
```

```text
0.2323, 0.2813, 0.4864
Kernels: Avx512
Input x: F32, rank 2
```

`KernelPath` says which kernels the session chose from the hardware at start-up: `Avx512`, `Avx2`, `Neon`, `Portable128` or `Scalar`. On another machine it may differ; in the default fast mode the results then agree to within rounding, and in [exact mode](https://recursiveai.org/onemodel/cookbook/exact-mode/) they are the same bits.

## What `Run` accepts

- `float[]`, `long[]` (token ids) and `int[]` inputs, with an optional shape: `model.Run(values, [2, 4])` for two rows.
- A span form, `Run<TIn, TOut>(input, shape, output)`, that writes into your buffer and allocates nothing once the model has run once with that shape.
- One input and one output. For several of either, named tensors, or options such as exact mode, use `model.Session` or `Session.Load`: see [Named inputs and outputs](https://recursiveai.org/onemodel/cookbook/named-inputs-and-outputs/).

`Run` may be called from any number of threads at once. Fast-mode results don’t depend on the thread count.

## When a model doesn’t load

A model that uses an operator OneModel doesn’t implement fails in `Model.Load` with a `SessionException` that lists every missing operator at once, not the first one it meets: see [When a model won’t load](https://recursiveai.org/onemodel/cookbook/missing-operators/). Which operators pass the ONNX backend tests is in [conformance](https://recursiveai.org/onemodel/docs/conformance/).

## Next

- [Exact mode](https://recursiveai.org/onemodel/cookbook/exact-mode/): the same bits on every platform and thread count.
- [The OneModel format](https://recursiveai.org/onemodel/cookbook/convert-to-onemodel-format/): convert an ONNX file once and load it as memory-mapped weights.
- [Half-precision weights](https://recursiveai.org/onemodel/cookbook/half-precision-weights/): halve the memory of large weights.
- [Sessions and plans](https://recursiveai.org/onemodel/docs/graph-runtime/): how a model is planned, cached and run.


<!-- https://recursiveai.org/onemodel/docs/overview/ -->

# OneModel

A neural-network runtime for .NET in pure managed C#, with no native binaries and no dependency beyond the shared framework. It loads ONNX models and our own format and runs them on SIMD CPU kernels, with training to follow on the same kernels. The name is provisional.

**Status:** preview, 0.3.0-preview.1. ONNX models and the OneModel format load and run through `Model.Load(...).Run(...)` or a `Session`, in fast mode or an exact mode that gives the same bits on every platform. Of the 1905 `onnx/backend/test` cases, 1580 pass in fast mode and 1382 in exact mode, none fail, and the rest are listed gaps ([docs/conformance.md](https://recursiveai.org/onemodel/docs/conformance/)). Training is not in the previews yet. See [CHANGELOG.md](https://recursiveai.org/onemodel/docs/changelog/).

## Layout

| Path | Holds |
|---|---|
| `src/OneModel` | the library, package `RecursiveAI.OneModel` |
| `tests/OneModel.Tests` | unit tests (xUnit v3) |
| `tests/AotSmoke`, `tests/WasmSmoke` | NativeAOT and browser-wasm smoke apps |
| `bench/Micro` | BenchmarkDotNet micro-benchmarks and kernel timing |
| `bench/Bar` | side-by-side timing against ONNX Runtime (the only place it is referenced) |
| `tools/` | scripts that fetch and export reference models, produce reference outputs and compare output hashes across machines |
| `fuzz/` | SharpFuzz targets |
| `docs/` | normative notes; see [docs/README.md](https://recursiveai.org/onemodel/docs/docs-map/) |

## Running a model

    using var m = Model.Load("model.onnx");
    float[] v = m.Run(tokens);          // float[], long[] or int[] in; names and shapes come from the model

For several inputs or outputs, tensors, exact mode (`new SessionOptions { Mode = ExecutionMode.Exact }`) or a thread limit, use `m.Session` or `Session.Load` directly. A model with an operator the library does not implement fails to load with an exception that lists every missing one; which operators pass the ONNX backend tests is in [docs/conformance.md](https://recursiveai.org/onemodel/docs/conformance/). How graphs are planned and run: [docs/graph-runtime.md](https://recursiveai.org/onemodel/docs/graph-runtime/).

### NativeAOT users: set `IlcInstructionSet`

A NativeAOT binary targets the x86-64 baseline unless told otherwise, and the library then silently runs its 4-lane kernels instead of the AVX2 or AVX-512 ones. Results are the same, but slower. Publish with the instruction set you target, for example:

    dotnet publish -c Release -p:IlcInstructionSet=x86-64-v3     # AVX2 and FMA machines (2013 and later)
    dotnet publish -c Release -p:IlcInstructionSet=native         # the machine you publish on

`Session.KernelPath` reports which kernels a session chose (`Avx2`, `Avx512`, `Neon`, `Portable128`, `Scalar`). `tests/AotSmoke` publishes with `x86-64-v3` and fails if an x64 machine does not pick AVX2 or AVX-512. The ordinary JIT build needs nothing: it checks the hardware at start-up.

## Building

The .NET SDK in `global.json` is required. Build output goes to `$ONEMODEL_ARTIFACTS` (default: a folder in the temp directory), never into the checkout.

    dotnet build -c Release
    dotnet test --solution OneModel.slnx -c Release

`-c Checked` builds optimized code with the `CHECKED` symbol, which enables bounds assertions at kernel entry points and poisons freed buffers. Tests and fuzzing run in `Checked`.

## Licence

`MIT OR Apache-2.0`, see [LICENSE-MIT](https://github.com/recursive-ai-oss/onemodel-csharp/tree/a603a81de3ac7eb2fe4ef4a5961c59822c93ae8c/LICENSE-MIT) and [LICENSE-APACHE](https://github.com/recursive-ai-oss/onemodel-csharp/tree/a603a81de3ac7eb2fe4ef4a5961c59822c93ae8c/LICENSE-APACHE). Third-party material is listed in [THIRD-PARTY-NOTICES.md](https://github.com/recursive-ai-oss/onemodel-csharp/blob/a603a81de3ac7eb2fe4ef4a5961c59822c93ae8c/THIRD-PARTY-NOTICES.md).


<!-- https://recursiveai.org/onemodel/docs/graph-runtime/ -->

# Graph runtime: operator contract, plans and sessions

This note says how an imported model becomes something that runs: how operators are described, how a plan for one set of input shapes is made, how memory is reused, and what a session promises. The C# types are named for orientation; the rules are the contract.

## 1. Layers

```
Model.Load(path|stream)            happy path: arrays/spans in, float[] out, no names or shapes needed
  └─ Session                       graph layer: named inputs/outputs as tensors or spans, thread-safe Run
       └─ CompiledGraph            operators resolved, weights loaded; independent of input shapes
            └─ ExecutionPlan       per set of input element types and shapes: shapes, constants, schedule, memory
                 └─ RunState       per run: value table, slab, arena
```

Each layer uses only the one below. The happy path never exposes ONNX names, layouts or kernel choices.

## 2. Operator contract

One operator type (or one family pattern) is one class under `Graph/Operators/` plus one `OpDef` line in `OpTable.cs`. The registry is a static table; nothing is found by reflection.

| Piece | When | What it does |
|---|---|---|
| `OpDef(opType, minOpset, maxOpset, factory, domain)` | table | Names the operator and the opset range one class serves. Ranges of one operator must not overlap; a lookup picks the definition whose range contains the node's opset. The default domain is `""` (`ai.onnx` is the same). |
| factory `(NodeInfo) → OpKernel` | session creation, once per node | Parses attributes (`GetInt`, `GetFloat`, `GetString`, `GetInts`, `GetFloats`, `Find`) and builds the kernel. Malformed attributes raise `SessionException(InvalidModel)` naming the node. |
| `Infer(InferContext)` | plan creation, once per node and shape set | Reads input element types and shapes (and values, if constant) and states every output with `SetOutput(i, type, shape)`. |
| `Prepare(PrepareContext)` | plan creation, after `Infer`, for nodes that will run | Optional. Turns constant inputs into a faster form (a transposed weight) with `CreateOwned`; the returned object comes back as `OpRun.State` on every run. |
| `Run(in OpRun)` | every run | Reads `Inputs`, fills `Outputs` (already allocated, contiguous, for the shapes `Infer` stated). Allocates no managed memory. Uses `Scratch(type, shape)` for temporaries. |

A kernel object is shared by every plan and every concurrent run. It holds what its attributes fixed and nothing else.

Flags on a kernel:

- **`SupportsExact`.** False means "not exact". A session in `ExecutionMode.Exact` over a model that uses such an operator fails at creation with `SessionException(ExactModeUnavailable)`, whose message and `Operators` list name every offending operator type. A kernel that supports exact receives `OpRun.Exact` and must then follow `exact-mode.md`: no fused multiply-add, one canonical accumulation order, NaN results rewritten to `0x7FC00000`.
- **`InPlaceInput`.** Output 0 may reuse that input's storage when the input is not needed afterwards and has the same element type and no fewer bytes. The kernel must be correct with output and input being the same memory.
- **`ViewOfInput`.** Output 0 is the input reinterpreted with the shape `Infer` stated (Reshape, Identity, Squeeze, ...). `Run` is never called; the session builds the view without a copy, and the source's storage stays alive as long as the view does.
- **`OutputsDataDependent`.** The output shapes depend on input values (NonZero, TopK with a computed k). `Infer` is not called; `Run` allocates its outputs with `AllocateOutput`, and everything downstream is shaped at run time (section 4).

Special outputs of `Infer`:

- `SetConstantOutput(i, tensor, planOwns)` states that the output is a constant for these shapes (Constant, Shape, Size). The node is not scheduled. All outputs of a node are constant or none are.
- `RequireRunTimeValues()` says the node needs an input *value* that is not known at plan time (Reshape with a computed shape). The node is then shaped again, with real values, in every run.

### Adding an operator

Write `Graph/Operators/<Name>.cs`: a class deriving `OpKernel` (or a family base such as `BinaryArithmeticKernel<TOp>`) with a `public static readonly OpDef Def`; add `<Name>Op.Def` to `OpTable.All`. Patterns to copy: `Binary.cs` (broadcasting element-wise), `Unary.cs`, `MatMul.cs` (matrix products with `Prepare`), `Views.cs` (views), `Constant.cs`.

### Subgraph operators (If, Loop, Scan)

The contract has what they need, and all three are implemented (section 8). A factory receives the `IrNode` (`NodeInfo.Ir`) and can read its graph attributes. It compiles each subgraph once with `CompiledGraph.Create` (outer-scope values it captures become extra inputs), and keeps the compiled graphs as kernel state. `Run` makes or reuses an `ExecutionPlan` for the subgraph's input shapes and runs it with a `RunState` taken from that plan. Because the subgraph's shapes may differ per iteration, the node is declared `OutputsDataDependent` unless its output shapes follow from its inputs. The plan cache and per-plan state pools are what make repeated iterations allocation-free.

## 3. Plans

A plan is made the first time a model runs with a given list of input element types and shapes, and cached (default 16 per session, oldest dropped first). Making one:

1. **Types and shapes.** Nodes are visited in graph order. Input types and shapes are the caller's. Each kernel's `Infer` states its outputs. Shapes are concrete: the plan is for exactly these input shapes. Symbolic dimensions in the model are not propagated through operators; they are metadata (`TensorInfo.Dims`), checked against the caller's shapes (rank and fixed extents) before a plan is made.
2. **Constant evaluation.** A node whose inputs are all constants is run once now and its outputs become constants, unless the result would be larger than both its inputs and 64 KiB. A view of a constant is a constant. This is what turns `Shape → Gather → Concat → Reshape` chains into a literal target shape. It is bit-preserving by construction: the same kernel produces the same bits, only earlier.
3. **Dead nodes** (no path to an output) are dropped when the session is created.
4. **Dynamic nodes.** A node with a run-time-shaped input, with `OutputsDataDependent`, or that required run-time values, is scheduled as dynamic: shaped again in each run from the actual tensors, with outputs taken from the run's arena.
5. **Memory.** Section 5.

A plan is immutable after creation and is shared by any number of concurrent runs. Plans are reference-counted: the cache holds one reference and each run another, so evicting a plan never frees memory a run is using.

## 4. Sessions

- **Creation** (`Session.Create(IrModel, options, ct)`, `Session.Load(path|stream, options, ct)`): resolves every operator and reports *all* missing ones together (`UnsupportedOperator`, with `Operators` listing `Type (opset N)`), checks the mode, copies weights out of the model (the `IrModel` can be disposed afterwards), picks the kernel path once from the hardware checks (`KernelPaths.Best`). The token is checked between operators and while importing.
- **Options.** `Mode` (`Fast` default, `Exact`), `IntraOpThreads` (0 = one per logical core, up to 256; changes speed only), `OptimizationLevel` (`None`, `Basic`, `All`; default `All`), `WeightStorage` (`Original`, `Binary16`, `BFloat16`; [fp16.md](https://recursiveai.org/onemodel/docs/fp16/)), `MaxCachedPlans`.
- **`Run`** has two forms. The tensor form takes names and tensors for every input and the outputs wanted; empty output tensors are allocated for the caller, others are checked and overwritten. The span form (single-input models) copies the input into the plan's staging area and the output out of the result and returns the output shape. Inputs are never modified or kept. Non-contiguous input tensors are copied first.
- **Errors** are `SessionException` with a `Kind`: `UnsupportedOperator`, `ExactModeUnavailable`, `InvalidModel`, `Unsupported`, `InvalidInput`, `Disposed`, and the node name where there is one. Input validation (unknown name, wrong element type, wrong rank or fixed extent, short output buffer) happens before anything runs.
- **Threads.** `Run` may be called from any number of threads. The plan cache is a copy-on-write array read without locks; building a plan takes a lock. Each run takes a `RunState` from its plan's idle list (at most 8 kept), so concurrent runs share weights and plan but nothing mutable. `Dispose` waits for nothing: runs in progress finish, and the weights are freed when the last one leaves.
- **Determinism.** Fast mode: results depend on inputs, options and kernel path only, never on thread count or on how runs interleave. Exact mode: bit-identical everywhere. Neither depends on whether a plan was just made or reused, or on which pooled state served the run.
- **Allocation.** After a plan exists and the pools are warm, a `Run` on one intra-op thread allocates no managed memory in either form. The parallel-for dispatch allocates nothing either: the job objects are kept per thread and the claim word of the pool carries the generation, the chunk count and the next chunk in one atomic value.

## 5. Memory

All intermediate values of a plan that have static shapes live in one **slab** per run state, 64-byte aligned, at offsets the plan chose. The slab also holds one staging area per input (where span inputs are copied). Dynamic values and kernel scratch come from the run state's **arena**, which is reset at the start of every run.

Offsets come from liveness:

1. `lastUse[v]` is the last step that reads `v` (or the step that makes it, if nobody reads it); graph outputs live to the end. A view extends its source's life to its own.
2. Steps are walked in order. A new value takes the smallest free block that fits (best fit), else the slab grows. Outputs are placed *before* the step's dying inputs are freed, so a kernel never writes over what it is still reading, except through an explicit in-place alias.
3. In-place: if the kernel allows it, the first input is static, dies at this step, is the only user of its block, and the block is large enough with the same element type, the output takes over the input's block.
4. A view shares its source's block (reference-counted). Free blocks are coalesced.

A plan whose slab would exceed 2 GiB is refused (`Unsupported`).

Checked builds poison each block as soon as its last value dies, so a stale read shows up as garbage in the tests; freed arena memory is poisoned too.

## 6. Graph passes

Between operator lookup and the first plan, `CompiledGraph` runs the **graph passes** the session's `OptimizationLevel` and mode select (details: [graph-passes.md](https://recursiveai.org/onemodel/docs/graph-passes/)). A pass is an internal `IGraphPass` (`Graph/Passes/GraphPass.cs`) with a name, a `MinLevel`, a `PreservesBits` flag and `Run(PassGraph, ct)`. The `PassGraph` it edits is mutable and shape-free: the node list (kept in dependency order), value ids with names and constants, and the graph's output ids. A pass may replace nodes, add values and constants (`AddValue`, `SetConstant`; the graph owns and disposes them) and add kernels (`Own`; disposed with the session). A fused node reuses the `NodeInfo` of the node it replaces, so errors still name an operator the model contains. Passes run in the order of the table `GraphPasses.s_all`; nodes nothing reads are removed after each pass. A pass that does not preserve bits is skipped in exact mode. Nothing a pass does may depend on input shapes: shape-dependent work stays in plans. To add a pass, write a class in `Graph/Passes/` and add it to the table.

## 7. Bits

- Plans, scheduling, buffer reuse, constant evaluation (in plans and in the `constant-fold` pass) and views are bit-preserving: they change where and when a result is computed, never how.
- Element-wise add and multiply are single correctly rounded operations, so fast and exact give the same bits for the same inputs except for NaN payloads, which exact mode rewrites to `0x7FC00000`. Relu is a compare and select; `-0` becomes `+0`; a NaN passes through.
- Gemm and MatMul follow `kernels-gemm.md`. Gemm's `alpha` and `beta` epilogue is separate multiplies and an add, never fused, in both modes.
- Exact-mode status of the operators so far: Add, Mul, Relu, MatMul, Gemm, Reshape, Identity, Constant are exact for float32; the integer and double paths of Add, Mul, Relu and MatMul are exact by construction (wrap-around integer arithmetic; doubles use the same single operations).

## 8. Control flow, sequences and optionals

**Bodies.** `If`, `Loop`, `Scan` and `SequenceMap` hold graphs. When the compiled graph creates such a kernel it compiles every body into a nested session of the same mode and kernel path (no graph passes). The outer values a body reads (ONNX lets a body read any value of the enclosing graphs) become extra inputs of the nested session, and of the node: they follow the node's declared inputs in the compiled node, in first-use order, so planning and freeing see the reads. A body that cannot be compiled (an operator is missing, or exact mode lacks a kernel) fails the whole model at creation, naming the operators. Body inputs may be untyped; whatever arrives is accepted.

**Running.** The kernels declare data-dependent outputs, so their results are allocated as they finish. `If` runs the chosen branch once. `Loop` runs its body while the trip count and the condition allow, carrying values from one iteration to the next, and stacks scan outputs on a new first axis (an empty `[0]` tensor of the declared type when there were no iterations). `Scan` slices its scan inputs along their axes (reversed when asked) and places scan outputs the same way; opset 8's batch axis and sequence lengths are supported. A nested session runs concurrently like any other; each run allocates its iteration tensors from the pool (steady state is not allocation-free for these operators).

**Sequences and optionals** are carried through plans as one rank-1 `uint8` tensor each (`PackedValue`): a four-byte tag, a kind byte and the elements back to back (type, rank, extents, row-major bytes), little-endian. A graph input or output declared as a sequence or an optional reports `TensorInfo.Kind` `Sequence` or `Optional` and a U8 element type with no shape. Kernels decide whether an input is packed by the model's declared type and, where the model gives none, by the tag. This keeps one value kind in sessions, plans and the memory planner; the cost is a copy per sequence operation. Sequences hold tensors only (ONNX's rule); an optional holds a tensor, a sequence, or nothing.


<!-- https://recursiveai.org/onemodel/docs/onnx-import/ -->

# ONNX import

How a model file becomes the in-memory graph. A port can follow this text without the C# code. The schema is `onnx.proto` and the
rules are ONNX's `IR.md`; the wire format is the protobuf encoding specification. Nothing here depends on a protobuf library.

## 1. Protobuf reading

The reader works on one contiguous byte range and never reads outside it.

* **Keys.** A key is a varint of at most 32 bits: `field << 3 | wire`. Field 0 is an error. Wire types 0 (varint), 1 (64-bit),
  2 (length-delimited) and 5 (32-bit) are read. Wire types 3 and 4 (groups) and 6 and 7 are errors.
* **Varints.** At most 10 bytes. The tenth byte may be 0 or 1; anything larger overflows 64 bits and is an error. A varint cut off by
  the end of the data is an error. Overlong encodings (`0x81 0x80 0x00`) are accepted, as the specification allows.
* **Lengths.** A length is read as 64 bits and compared with the bytes that remain *before* any slice is taken. A length larger than
  the remainder is an error. Nothing is allocated from a length.
* **Integers.** `int32` and enum fields keep the low 32 bits of the varint, as protobuf does. `int64` and `uint64` keep all 64.
* **Strings** (`string` fields) must be valid UTF-8; invalid bytes are an error. `bytes` fields are never decoded.
* **Repeated scalars** are accepted packed (one length-delimited field) or unpacked (one key per element), mixed in any order.
* **Unknown fields**, and known fields that arrive with the wrong wire type, are skipped.
* **Repeated occurrences** of a singular field: the last one wins. Singular message fields are not merged.
* **Nesting.** Each nested message is one level. Reading stops with an error beyond `MaxNesting` levels (default 128, range 8 to 250).
  A graph inside a node's attribute inside a graph costs three levels.
* **Size.** A model file can be at most 2^31 − 1 bytes (protobuf's own limit). Bigger models keep their weights in external data.

## 2. Messages and fields used

| Message | Fields read (number) |
|---|---|
| ModelProto | ir_version 1, producer_name 2, producer_version 3, domain 4, model_version 5, doc_string 6, graph 7, opset_import 8, metadata_props 14, functions 25. `training_info` and `configuration` are skipped |
| OperatorSetIdProto | domain 1, version 2 |
| GraphProto | node 1, name 2, initializer 5, doc_string 10, input 11, output 12, value_info 13, sparse_initializer 15. `quantization_annotation` and `metadata_props` are skipped |
| NodeProto | input 1, output 2, name 3, op_type 4, attribute 5, doc_string 6, domain 7, overload 8. `metadata_props` and `device_configurations` are skipped |
| AttributeProto | name 1, f 2, i 3, s 4, t 5, g 6, floats 7, ints 8, strings 9, tensors 10, graphs 11, doc_string 13, tp 14, type_protos 15, type 20, ref_attr_name 21, sparse_tensor 22, sparse_tensors 23 |
| TensorProto | dims 1, data_type 2, segment 3, float_data 4, int32_data 5, string_data 6, int64_data 7, name 8, raw_data 9, double_data 10, uint64_data 11, doc_string 12, external_data 13, data_location 14 |
| SparseTensorProto | values 1, indices 2, dims 3 |
| ValueInfoProto | name 1, type 2 |
| TypeProto | tensor_type 1, sequence_type 4, map_type 5, optional_type 9, sparse_tensor_type 8, opaque_type 7 |
| TensorShapeProto.Dimension | dim_value 1, dim_param 2 (last one set wins) |
| FunctionProto | name 1, input 4, output 5, attribute 6, node 7, doc_string 8, opset_import 9, domain 10, attribute_proto 11, overload 13. `value_info` and `metadata_props` are skipped |

**Model checks.** `ir_version` must be present and at least 3; a larger version than any known is accepted (new fields are skipped).
A model without a graph is an error.

**Attributes.** If `type` is absent the type is inferred from the field that is set (`f`, `i`, `s`, `t`, `g`, `sparse_tensor`, `tp`,
then the repeated fields in the order floats, ints, strings, tensors, graphs, sparse_tensors, type_protos). An attribute with
neither a type nor a value (and no `ref_attr_name`) is an error. A scalar attribute whose field is absent reads as 0 or empty. An
attribute declared as tensor, graph, sparse tensor or type whose field is absent is an error. An attribute with `ref_attr_name`
(function bodies) carries no value.

**Types.** A `TypeProto` with no member set is "no type". A tensor type that has a `shape` field has a shape (possibly rank 0);
one without it has unknown rank. A dimension with neither `dim_value` nor `dim_param` is unknown.

## 3. Tensors

A tensor has an element type code (1 to 28, `TensorProto.DataType`), dimensions, and its values as **little-endian bytes** in the
raw layout of `TensorProto.raw_data`.

* Dimensions must be non-negative. The element count is their product, which must not exceed 2^60.
* The expected byte length is `ceil(count × bits / 8)` with bits: 32 for float, int32, uint32; 8 for uint8, int8, bool, the 8-bit
  floats; 16 for uint16, int16, float16, bfloat16; 64 for int64, uint64, double, complex64; 128 for complex128; 4 for the 4-bit
  types; 2 for the 2-bit types; 6 for the 6-bit floats. Sub-byte elements are packed as `onnx.proto` describes. A tensor that would
  need more than 2^31 − 1 bytes is *unsupported*, and this is decided from the dimensions alone, before any data is read.
* **Where the values come from**, in this order: `data_location = EXTERNAL` (section 5); else `raw_data`; else the typed field
  for the element type. Mixing sources is an error. A tensor with elements and no source is an error (its byte length is zero).
* **Typed fields**, converted to the raw layout:
  * `float_data` (float, complex64): 4 bytes per entry. `double_data` (double, complex128): 8 bytes. `int64_data` (int64): 8 bytes.
  * `uint64_data` serves uint64 (8 bytes) and uint32 (the low 32 bits, 4 bytes).
  * `int32_data` serves every other numeric type, one element per entry: int32 as is; 16-bit types keep the low 16 bits; 8-bit types
    and bool keep the low 8 bits (bool: any non-zero entry becomes 1). For 4-bit and 2-bit types an entry's low byte is one *packed*
    byte. For the 6-bit types an entry's low 6 bits are one element and the elements are packed into the 4-in-3-bytes bit stream.
  * A typed field that does not belong to the element type is an error.
  * The resulting byte length must equal the expected length.
* **Strings** (type 8) live only in `string_data`, one entry per element; the count must equal the element count.
* **Segments** (`segment`) are not supported.
* `raw_data` must have exactly the expected length. Values are not byte-swapped; raw data is little-endian by definition.

A sparse tensor is imported as written (`values`, `indices`, dense `dims`) and is not converted to dense.

## 4. The graph

Reading produces the parsed graph; resolution then builds the graph proper.

* **Values.** Per graph, in this order: each input (name must be non-empty and unique); each initializer (an initializer whose name is
  an input that has no initializer yet becomes that input's default value, which is how IR versions before 4 list initializers; any other
  repeated name is an error); each sparse initializer; each non-empty node output. A name defined twice is an error. An empty name
  among inputs or outputs of a *node* means "omitted optional" and gives no value.
* **Scope.** A subgraph (an attribute of type graph or graphs) resolves after its enclosing graph has defined all its values, and
  sees them; names it defines itself shadow outer names. A subgraph's `Captures` list the outer values it (or its own subgraphs) reads.
* **References.** Every non-empty node input, and every graph output, must name a defined value. Otherwise: error, naming the node.
* **Opsets.** A node's opset version is the version imported for its domain; the empty domain and `ai.onnx` are the same domain.
  A node whose domain is not imported is an error. Function bodies are not resolved.
* **Types** come from input and output declarations first, and from `value_info` for values that still have none. A `value_info` for a
  name that is not defined is ignored.
* **Order.** IR.md requires nodes in topological order. Input in that order is kept as is. Otherwise nodes are sorted: repeatedly take
  the node with the smallest file index whose inputs (including values read by its subgraphs) are all produced; the flag
  `NodesWereReordered` records that this happened. A cycle is an error.
* **Consumers.** For each value, the nodes that read it in sorted order, each once; a node counts as a reader of what its subgraphs read.
* **Functions** are imported as written (name, domain, overload, formal inputs, outputs and attributes, attribute defaults, body nodes
  by name, opset imports). They are not expanded.

## 5. External data

`data_location = EXTERNAL` takes the values from a file. Entries of `external_data`: `location` (required), `offset` (default 0),
`length` (default: the tensor's expected length), `checksum` (ignored).

1. `location` must be a relative path: not empty, no NUL, not starting with `/` or `\`, not `X:`-prefixed, not rooted on the platform,
   and with no `..` segment when split at either `/` or `\`.
2. It is joined to the **external-data directory** (for a file path, the model's directory; for a stream or byte array, only if the
   caller names one) and every symbolic link on the way is resolved. The resolved file must lie inside the resolved directory and be a
   file. Otherwise: external-data error.
3. `offset` and `length` must be non-negative decimal integers. `length`, if given, must equal the expected length. `offset + length`
   must not exceed the file's size (checked in 64 bits).
4. The file is memory-mapped where the platform allows it (one mapping per file, shared by its tensors) and the tensor is a view into
   it. Where mapping is unavailable, only the requested range is read into memory.

A swap of a file or link after the check is not defended against.

## 6. Memory

Initializer bytes are views: into the mapped model file, into a byte array the caller supplied, or into a small array holding converted
typed-field data. Importing therefore allocates only the graph itself. Disposing the model releases the mappings; reading tensor data
afterwards throws. Copying a tensor out (`ToTensor`) is the only step that allocates its bytes. Models up to 1 MiB are read instead of
mapped. 

Indicative figures from one Apple M2 (arm64, .NET 10, warm file cache, three runs, the steady ones), importing the whole graph and
touching no tensor data. Peak resident size is the operating system's figure for the process, including the runtime (about 45 MiB):

| Model (file size) | Mapped: load, peak RSS | Read into memory: load, peak RSS |
|---|---|---|
| BERT-base (416 MiB) | 17 ms, 48 MiB | 68 ms, 463 MiB |
| MiniLM-L6 (86 MiB) | 17 ms, 46 MiB | 66 ms, 132 MiB |
| Whisper-small decoder (738 MiB) | 19 ms, 51 MiB | 110 ms, 787 MiB |
| Whisper-small encoder (336 MiB) | 18 ms, 48 MiB | 55 ms, 383 MiB |

On an x64 desktop (Ryzen 9 7950X3D, Linux) mapped loads took 21 to 25 ms and peaked at 181 to 281 MiB, because Linux counts the file pages
it reads ahead around each header; reading took 140 to 300 ms and 370 to 790 MiB. The managed heap after loading is 3 to 5 MiB when mapped.
These are single-machine, indicative figures, not a gated benchmark.

## 7. Errors

Everything wrong with the input is one exception type carrying a kind (`Malformed` for bytes that are not valid protobuf, `InvalidModel`
for valid protobuf that is not a valid model, `Unsupported`, `ExternalData`, `Io`), the input's name, the byte offset when one is known,
and the field path (`ModelProto.graph.node[3].attribute[1].t`). Offsets are from the start of the model file or byte array.

## 8. How the importer is checked

* `tests/OnnxSweep` imports every `model.onnx` of the ONNX backend test data and writes a structural dump of each (producer, opsets,
  every node with its attributes, value types, initializer element type, dimensions and SHA-256 of the data). `tools/onnx_dump.py` writes
  the same dump from the `onnx` Python package using only `onnx.load` and the message fields, and `tools/compare_onnx_dump.sh`
  diffs the two. Floats are compared as bit patterns.
* The hostile inputs in `testdata/onnx/hostile` are named `<kind>-<case>.onnx`; the test suite regenerates them from code and expects
  each to be refused with that kind.
* `fuzz/` holds libFuzzer targets for the model reader and the tensor reader; `tools/setup_fuzz.sh` and `tools/run_fuzz.sh` run them.


<!-- https://recursiveai.org/onemodel/docs/format/ -->

# The OneModel model format

A model file with the graph and its weights in one place, laid out so the weights can be used straight from a memory map. This
text is normative: a reader or writer in another language can be built from it alone. **Format version 1. Unstable until 1.0:**
version 1 files may stop loading, or change meaning, before the library reaches 1.0.

The format stores an ONNX graph (the same model that the ONNX importer produces, see [onnx-import.md](https://recursiveai.org/onemodel/docs/onnx-import/)) without
protobuf: a fixed header, a JSON manifest and the raw tensor bytes. It carries no optimisation results. What is saved is the
imported graph before any pass, so a saved file optimises and runs exactly as the ONNX file it came from, and bit-identically
(`graph-passes.md` says what passes do at load).

## 1. File layout

All integers are little endian. Offsets are from the start of the file.

```
offset 0                 header                64 bytes
offset 64                manifest              manifest_length bytes, UTF-8 JSON
offset 64+manifest_length   zero padding        up to the next multiple of 64
offset data_offset       data region           data_length bytes
end of file                                    data_offset + data_length == file length
```

### 1.1 Header (64 bytes)

| Offset | Size | Field | Value |
|---|---|---|---|
| 0 | 8 | magic | the ASCII bytes `ONEMODEL` |
| 8 | 4 | version | u32, `1` |
| 12 | 4 | flags | u32, `0`. A reader rejects any other value |
| 16 | 8 | manifest_offset | u64, `64` |
| 24 | 8 | manifest_length | u64, bytes of manifest |
| 32 | 8 | data_offset | u64, a multiple of 64, at least `64 + manifest_length`, at most the file length |
| 40 | 8 | data_length | u64, `file length - data_offset` |
| 48 | 8 | file_length | u64, the exact length of the file |
| 56 | 8 | reserved | zero. A reader rejects anything else |

All u64 fields must be at most 2^63 − 1. `file_length` lets a reader notice a truncated or extended file before it trusts any other
field. A reader checks every field above against the real file length before it reads the manifest or allocates from any value.

### 1.2 Manifest

UTF-8 JSON (RFC 8259) without a byte-order mark, one object. Reading it never needs more than a streaming JSON parser. Readers
limit its size (the C# reader: 512 MiB) and its nesting depth (1024 JSON levels).

**Integers** in the manifest are JSON numbers written without a fraction or exponent and fit in a signed 64-bit integer. A reader
must parse them as integers: a JSON parser that turns every number into a double loses bits above 2^53 (offsets and lengths stay
below that in any real model, but `int64` attributes may not).

**Floats** are never written as decimal numbers. A 32-bit float attribute is the unsigned integer of its IEEE-754 bit pattern
(`1.0` is `1065353216`), so NaN payloads, infinities and signed zeros survive.

**Bytes** (text attributes, string tensor elements) are base64 (RFC 4648, standard alphabet, padded) in a JSON string.

**Absent fields.** A field marked *optional* defaults to `""` (strings), `0` (integers) or empty (arrays). Unknown fields are
ignored by readers, so later versions can add fields and keep the version number only if old readers still see the same model.

### 1.3 Data region

The raw bytes of tensors, each starting at a multiple of 64 from the start of the file (`data_offset` is a multiple of 64, so
tensor offsets are multiples of 64 within the region too), with zero bytes between. Tensors are stored in the layout of ONNX's
`TensorProto.raw_data` for their element type, little endian: row-major, no stride, no header. Several tensors may point at the
same bytes. The writer in this library gives each distinct tensor its own bytes and places them in this order: graph
initializers, then each sparse initializer's values and indices, then, for each node in order, the tensors in its attributes
(descending into subgraphs in place), then the same for each function's nodes and attribute defaults. A reader does not depend on
the order.

## 2. Manifest schema

### 2.1 Model (the top-level object)

| Field | JSON | Notes |
|---|---|---|
| `format` | string | required, `"onemodel"` |
| `version` | integer | required, equal to the header's version |
| `ir_version` | integer | required, at least 3 (ONNX IR version) |
| `producer_name`, `producer_version`, `domain`, `doc` | string | optional |
| `model_version` | integer | optional |
| `opsets` | array of `{domain: string, version: integer}` | operator set imports. Domain `""` and `"ai.onnx"` are the default domain |
| `metadata` | array of `{key: string, value: string}` | optional, order kept |
| `graph` | Graph | required |
| `functions` | array of Function | optional |

### 2.2 Graph

| Field | JSON | Notes |
|---|---|---|
| `name`, `doc` | string | optional |
| `inputs` | array of TypedName | graph inputs in order. An input that also has an initializer is listed here and in `initializers` |
| `outputs` | array of TypedName | graph outputs in order |
| `value_info` | array of TypedName | declared types of other values |
| `initializers` | array of Tensor | |
| `sparse_initializers` | array of SparseTensor | |
| `nodes` | array of Node | in execution order when the writer sorted them; a reader must not rely on it (section 3) |

**TypedName**: `{name: string (required), type: Type (optional)}`.

### 2.3 Node

`{name, op (required), domain, overload, doc: string, inputs: [string], outputs: [string], attributes: [Attribute]}`.
An empty string in `inputs` or `outputs` is an omitted optional operand, as in ONNX. A node's operator set version is the
version the model's `opsets` give for the node's domain.

### 2.4 Attribute

`{name (required), type (required), doc, ref, value}`.

`type` is ONNX's `AttributeProto.AttributeType` code, and `value` depends on it:

| type | code | `value` |
|---|---|---|
| FLOAT | 1 | unsigned integer, the float's bits |
| INT | 2 | integer |
| STRING | 3 | base64 string |
| TENSOR | 4 | Tensor |
| GRAPH | 5 | Graph |
| FLOATS | 6 | array of unsigned integers (bits) |
| INTS | 7 | array of integers |
| STRINGS | 8 | array of base64 strings |
| TENSORS | 9 | array of Tensor |
| GRAPHS | 10 | array of Graph |
| SPARSE_TENSOR | 11 | SparseTensor |
| SPARSE_TENSORS | 12 | array of SparseTensor |
| TYPE_PROTO | 13 | Type |
| TYPE_PROTOS | 14 | array of Type |

If `ref` is a non-empty string the attribute is a reference to an attribute of the enclosing function (ONNX
`ref_attr_name`) and has no `value`; `type` is then the type of the referenced attribute. Otherwise `type` 0 is an error and
`value` is required.

### 2.5 Tensor

`{name, dtype (required), dims (required), doc, data | strings}`.

* `dtype` is ONNX's `TensorProto.DataType` code. Every code except 0 (undefined) and the string type 8 has a fixed size in bits:

  | codes | bits per element |
  |---|---|
  | 1 float, 6 int32, 12 uint32 | 32 |
  | 2 uint8, 3 int8, 9 bool, 17–20 float8 types, 24 float8e8m0 | 8 |
  | 4 uint16, 5 int16, 10 float16, 16 bfloat16 | 16 |
  | 7 int64, 11 double, 13 uint64, 14 complex64 | 64 |
  | 15 complex128 | 128 |
  | 21 uint4, 22 int4, 23 float4e2m1 | 4 |
  | 25 uint2, 26 int2 | 2 |
  | 27, 28 | 6 |

  Any other code is rejected as unsupported.
* `dims` is an array of non-negative integers (empty for a scalar). The element count is their product, which must stay below 2^60.
* `data` (every dtype except 8) is `{offset: integer, length: integer}`: the bytes are `length` bytes at `data_offset + offset`.
  `length` must equal `ceil(count × bits / 8)`, `offset` must be a multiple of 64 when `length > 0`, and
  `offset + length` must not exceed `data_length`. Elements narrower than a byte are packed as in ONNX `raw_data`. This library
  supports tensors of up to 2^31 − 1 bytes and rejects larger ones.
* `strings` (dtype 8) is an array of base64 strings, one per element, `count` of them.

### 2.6 SparseTensor

`{values: Tensor, indices: Tensor, dims: [integer]}` as ONNX `SparseTensorProto`. The `values` tensor's `name` is the sparse
initializer's name.

### 2.7 Type

An object with a `kind`:

| kind | other fields |
|---|---|
| `"tensor"`, `"sparse_tensor"` | `elem`: integer element type code (required). `shape`: optional array; present (even empty) means the rank is known. Each entry is a non-negative integer (a fixed size), a string (a named dimension), or `null` (unknown) |
| `"sequence"`, `"optional"` | `inner`: Type |
| `"map"` | `key`: integer element type code; `inner`: Type (the value type) |
| `"opaque"` | `domain`, `name`: strings |

Any other kind is rejected as unsupported.

### 2.8 Function

`{name, domain, overload, doc, inputs: [string], outputs: [string], attribute_names: [string], attribute_defaults: [Attribute], nodes: [Node], opsets: [{domain, version}]}`.
Functions are carried, not expanded.

## 3. Reading rules

After the manifest is parsed, names are resolved exactly as for an ONNX file:

1. Within one graph every name defines at most one value. Graph inputs, initializers, sparse initializers (by their values'
   name) and non-empty node outputs define values; a second definition of a name is an error. An initializer whose name matches
   a graph input attaches to that input instead.
2. A node input or graph output names a value defined in the same graph or in an enclosing graph (subgraphs read the values of
   the graphs around them). An undefined name is an error.
3. Nodes are put in dependency order (stable: nodes already in order keep their positions). A cycle is an error.
4. A node whose domain has no entry in `opsets` is an error.
5. `value_info` and the `type` of inputs and outputs set the declared type of the value they name.

### 3.1 Limits and checks

A reader checks, before it allocates or slices from any value in the file:

* the header against the real file length (section 1.1);
* the manifest size;
* every tensor range against the data region, in 64-bit arithmetic, and the byte length against dtype and dims;
* element counts: no negative dimension, no product at 2^60 or more;
* nesting: graphs and types are counted together; more than 64 open at once is rejected (a graph inside a node's attribute
  counts one level, a type inside a graph or a type one more).

Every failure is reported as one of four kinds, with the manifest path of the field (such as `manifest.graph.nodes[3].inputs[1]`)
or the header field name, and the file offset when there is one:

| Kind | Meaning |
|---|---|
| Malformed | not a well-formed file: bad magic, truncation, inconsistent header, invalid JSON, a value of the wrong JSON type, a range outside the file |
| InvalidModel | well formed but impossible: duplicate or undefined names, a byte length that does not match dims and dtype, a negative dimension |
| Unsupported | something this reader does not handle: a newer version, nonzero flags, an unknown dtype, attribute type or type kind, too much nesting, a tensor over 2^31 − 1 bytes |
| Io | the file could not be read or written |

## 4. Writing rules

A writer produces the same bytes for the same model: a single pass assigns offsets in the order of section 1.3, then writes the
manifest with no insignificant whitespace, then the header, the padding and the tensors. The C# writer emits fields in the order
of section 2's tables, writes optional strings `doc` only when non-empty, and always writes `name`, `domain`, `overload`,
`inputs`, `outputs` and `attributes` of a node. Other writers need not match its manifest byte for byte; the data region, the
header's fields and the meaning of the manifest are what the format fixes.

Weights that an ONNX file keeps in external data files are copied into the data region.

## 5. Loading

`Model.Load`, `Session.Load` and `ModelFormat.Load` read the file. `Model.Load` and `Session.Load` look at the first eight bytes to
choose between this format and ONNX. Files of 1 MiB or more are memory-mapped (where the platform allows it) and each tensor is a
view of the mapping, copied only when a session builds its weights; smaller files, streams and platforms without mapping read the
file into memory. Alignment to 64 bytes lets a consumer use mapped weights in place with any vector width.

Saving: `ModelFormat.Save(model, path | stream)` and `ModelFormat.Convert(onnxPath, outputPath)`.

## 6. Checks that back this text

* `tests/OneModel.Tests/Format/`: round trips (dtypes, strings, subgraphs, sparse, external data, mapped files), a hand-assembled
  file written from this text alone, and 64 hostile files in `testdata/format/hostile`, each named `<Kind>-<case>`.
* `tools/format_read.py`: a reader in Python with numpy only, written from this text; `tools/format_check.py` compares its view of a
  converted model with the ONNX file byte for byte.
* A fuzz target (`format`) in `fuzz/`.


<!-- https://recursiveai.org/onemodel/docs/graph-passes/ -->

# Graph passes

Rewrites applied to the compiled graph when a session is created, selected by `SessionOptions.OptimizationLevel` and the mode. Interface and ownership rules: [graph-runtime.md](https://recursiveai.org/onemodel/docs/graph-runtime/) section 6.

## Levels and exact mode

| Level | Passes |
|---|---|
| `None` | none; the graph runs as imported (plans still evaluate constant subgraphs) |
| `Basic` | `constant-fold` |
| `All` (default) | everything below |

Every pass states whether it **preserves bits**: results identical with and without the pass, in both modes. In `ExecutionMode.Exact` only passes that preserve bits run; the documented exact pipeline (and so the golden result) is that subset. A pass that does not preserve bits may still change fast-mode results only in the last bits (checked against the unfused graph, see the tests); fast mode stays deterministic for any thread count either way.

## Passes, in order

| Pass | Level | Preserves bits | What it does |
|---|---|---|---|
| `qdq` | All | no | Rewrites quantize-dequantize Conv/Gemm patterns to `QLinearConv` / `QLinearMatMul`; skipped in exact mode. Runs first, before `constant-fold` would turn the weights' `DequantizeLinear` into float constants. See [int8.md](https://recursiveai.org/onemodel/docs/int8/). |
| `matmul-integer-fusion` | All | yes | `MatMulInteger -> Cast -> Mul [-> Add]` becomes one dynamic-quantization product node. See [int8.md](https://recursiveai.org/onemodel/docs/int8/). |
| `constant-fold` | Basic | yes | Evaluates every node whose inputs are all constants, once, through its own kernel in the session's mode. Skips data-dependent shapes, nodes feeding a graph output, and results larger than both the inputs and 64 KiB. A view of a constant becomes a copy. Shape-dependent folding (Shape, Size of a runtime input) stays in plans. |
| `conv-batchnorm-fold` | All | **no** | `Conv → BatchNormalization` (constant statistics) becomes one Conv: `W' = W·s` per output channel, `b' = (b − mean)·s + beta`, `s = scale / sqrt(var + epsilon)`, in double precision, rounded once. The scale moves before the sum, so the last bits differ. |
| `constant-fold` | Basic | yes | Again, so the folded weights and bias are plain constants. |
| `gelu` | All | **no** | `Mul(Mul(x, 1 + Erf(Div(x, √2))), 0.5)` (the opset 17 export of Gelu) becomes one Gelu node (erf form). One pass over memory instead of five; the fused erf is not the separate operators' arithmetic. |
| `attention` | All | **no** | `MatMul(Q, Kt) → [Mul or Div by a constant] → [Add mask] → Softmax(last axis) → MatMul(P, V)` becomes one `AttentionOp` running the fused kernel ([kernels-attention.md](https://recursiveai.org/onemodel/docs/kernels-attention/)). Head-split transposes before Q, K, V (`[b,s,h,d]→[b,h,s,d]`; for K `(0,2,3,1)` or `(0,1,3,2)`) and the head-merge transpose after are absorbed: the kernel reads and writes the strided layouts. Any broadcastable mask is accepted; an odd head dimension takes a plain fallback. |
| `linear-pack-fuse` | All | yes | `MatMul` or `Gemm` with a constant 2-D float32 B becomes `LinearOp`: B is packed once into the kernel's panel layout (below); a following `Add` of a constant bias vector and a following activation (Relu, Clip with constant bounds, HardSwish, HardSigmoid, Sigmoid, Gelu, Tanh, LeakyRelu) run on each finished tile of C. Gemm needs alpha = 1 and no transA; `beta·C` is formed once at pass time with the multiply the kernel would do per call. |
| `conv-activation` | All | yes | `Conv → activation` (same list): the convolution's bias and the activation run in the product's tile epilogue (after the direct depthwise kernel: per plane). |

**Why the "yes" rows preserve bits.** Packing is a copy. The bias is one rounded addition per element (the stand-alone Add), and in exact mode a NaN becomes `0x7FC00000` as Add does. The activation is the operator's own kernel applied to the same values; exact-mode kernels do not depend on lane width or on how a row is cut, so the tile-row cuts change nothing. In fast mode the one possible difference is where a vector body hands over to its scalar tail inside an activation (rows are cut at tile edges), so fast-mode activations may differ from the separate operator in the last bits; fast results are still independent of the thread count because the tiles are fixed by the shape.

## Packed B

`Gemm.PackedBLength(path, n, k)`, `Gemm.PackB` and `Gemm.RunPacked` ([kernels-gemm.md](https://recursiveai.org/onemodel/docs/kernels-gemm/)): B (k × n) is cut into blocks of Kc = 256 rows; within a block, strips of nr columns (nr is the path's register tile width: 32, 16, 12, 8 or 4), each strip a kc × nr panel stored contiguously, ragged strips padded with +0. Block `pc/Kc` starts at `pc · stripCount · nr`, strip `s` of a block at `s · kc · nr` within it. The layout depends only on the path. `RunPacked` skips the per-call packing and computes the same products in the same order, so its results are bit-identical to `Run` on the unpacked B.

## Tile epilogue

`IGemmEpilogue.Apply(tile, ldc, row0, col0, rows, cols, mode)` is called once per finished mc × nc tile of C (after the exact-mode NaN rewrite), from whichever thread computed the tile. It writes only the tile. Tile boundaries are fixed by the path's blocking and the shape, never by the thread count.

## Memory

Fusion removes intermediates: the fused chains above keep one output tensor instead of two to five, and the in-place reuse of the plan applies to what remains. Packed weights replace the originals (the constant is released once nothing reads it), so resident weight memory is unchanged apart from padding of ragged strips.

## Half-precision weights

* A model that contains half-precision values of its own (types, initializers, casts) is not rewritten: its graph runs as imported, because the passes' kernels are binary32.
* With `WeightStorage` set to `Binary16` or `BFloat16`, the weights that qualify are kept compact and are **not packed**: the passes pack and fold only binary32 constants, and the half-precision adapter widens a compact weight on each run. Everything around them is still rewritten (constant folding of other nodes, a Conv's batch-norm fold needs binary32 weights and is skipped for compact ones, a Conv's activation still fuses, attention and Gelu chains fuse). Packing exists to save the per-call copy of B; a compact weight is widened per call anyway. Compact and packed storage therefore never apply to the same weight. Weights below the compaction threshold (1024 elements) are binary32 and are packed.


<!-- https://recursiveai.org/onemodel/docs/exact-mode/ -->

# Exact mode

Exact mode gives bit-identical results on every operating system, CPU, instruction set, thread count and runtime (including
WebAssembly), so a port to another language or platform can reproduce them from this text alone. Fast mode (the default) gives
up that promise for speed; its rules are in `kernels-gemm.md`.

## Rules every exact kernel follows

1. **IEEE binary32 arithmetic with round-to-nearest-even.** Only `+`, `-`, `*`, `/` and square root are used. Each operation is
   rounded to binary32 on its own: **no fused multiply-add** in any form (no `fma`, no multiply-accumulate instructions that skip
   the intermediate rounding, no estimate instructions). Flush-to-zero and denormals-are-zero modes must be off, so subnormals are
   computed, not dropped.
2. **One accumulator per output, in a stated order.** Each kernel below states the order in which the terms of an output are added.
   Vector lanes always hold different outputs, never different terms of one output, so the vector width cannot change a result.
3. **Start from +0.** An accumulator starts as positive zero. `+0 + (-0)` is `+0`, so an output is never `-0` unless it is the
   result of a later operation that yields `-0` (it cannot: a round-to-nearest sum is `-0` only when both operands are `-0`).
4. **NaN is canonical.** Hardware disagrees on the sign and payload of the NaN that `inf - inf`, `0 * inf` or a NaN input produces.
   Every NaN an exact kernel stores is replaced by the quiet NaN with bits `0x7FC00000`. Infinities, signed zeros and subnormals
   follow IEEE and need no special handling.
5. **No dependence on blocking or threads.** Cache blocking, tile shapes and chunking may differ between paths and machines; the
   order of additions inside one output may not.
6. **Transcendental functions** (exp, log, tanh, erf, sigmoid) are our own fixed polynomials, defined in
   `exact-transcendentals.md`.

Kernels that cannot meet these rules are not exact; selecting exact mode for an operator without an exact kernel is reported, not
silently approximated.

## Matrix product (`C = A * B`, row-major, A is m x k, B is k x n)

For every output `(i, j)`:

```
acc = +0
for p = 0, 1, ..., k-1 in increasing order:
    prod = round(A[i][p] * B[p][j])
    acc  = round(acc + prod)
C[i][j] = isNaN(acc) ? 0x7FC00000 : acc
```

With `k = 0` the output is `+0`. This is the whole definition: tile sizes, packing, the split of `k` into blocks (a block adds its
terms to the accumulator left by the previous block, in the same order), the assignment of tiles to threads and the vector width
do not appear in it. The product `A[i][p] * B[p][j]` is commutative, so operand order does not matter; the sum is not
associative, so the order of `p` does.

The same definition covers the score product `q . k` of attention (`exact-transcendentals.md` and `kernels-attention.md` use the
same order: one accumulator, increasing index, unfused, from +0), so attention can call the matrix product.

## Convolution (2-D, NCHW, weights `[Cout, Cin/G, KH, KW]`)

For output `(n, oc, y, x)` of group `g = oc / (Cout/G)`:

```
acc = +0
for c = 0 .. Cin/G-1, then ky = 0 .. KH-1, then kx = 0 .. KW-1 (row-major, in that nesting):
    iy = y*strideH - padTop  + ky*dilationH
    ix = x*strideW - padLeft + kx*dilationW
    v  = (0 <= iy < H and 0 <= ix < W) ? input[n][g*Cin/G + c][iy][ix] : +0
    acc = round(acc + round(weight[oc][c][ky][kx] * v))
out = isNaN(acc) ? 0x7FC00000 : acc
```

Padding taps are real products with `+0`, not skipped, so an infinite or NaN weight meeting padding gives NaN. This is the matrix
product of the weight matrix with the im2col matrix (rows ordered `c, ky, kx`), and the depthwise kernel (`Cin/G = 1`) computes
the same expression directly. Output size per axis is `floor((size + padBefore + padAfter - dilation*(K-1) - 1) / stride) + 1`.
No bias is added by these kernels.

## Checking it

* Every path (scalar, 4-lane, AVX2, AVX-512, NEON) is compared bit for bit with a naive implementation of the pseudo-code above
  for every size 0..70 in each dimension, with random strides and offsets, and with NaN, infinity, signed-zero and subnormal
  inputs (`tests/OneModel.Tests/Kernels`).
* `spike hashes` (in `bench/Micro`) hashes the outputs of sweeps over sizes 0..256, of random shapes and of the reference
  convolution and matrix shapes with SHA-256, for each path at 1 thread and at all threads, and prints one line per run; the exact
  lines must be equal on every machine. They were equal on x64 (default, AVX-512 off, AVX2 off, all hardware intrinsics off) and
  on arm64 NEON.
* Setting `DOTNET_EnableHWIntrinsic=0` selects the scalar path; `DOTNET_EnableAVX2=0` and `DOTNET_EnableAVX512=0` remove those
  paths (the .NET 10 switch for AVX-512 is `DOTNET_EnableAVX512`; `DOTNET_EnableAVX512F` no longer has an effect).


<!-- https://recursiveai.org/onemodel/docs/golden-hashes/ -->

# Golden hashes

Exact mode (see [exact-mode.md](https://recursiveai.org/onemodel/docs/exact-mode/)) promises the same bits on every OS, CPU, instruction set, thread count and runtime. This file records the check of that promise: the SHA-256 of every output of the reference models, run in `ExecutionMode.Exact`, and how the rows compared. Only hashes appear here.

## Method

`tools/GoldenModels` loads a model, binds the stored inputs (`.npy` files; the input hash is printed too, so a row that was fed different bytes shows up), runs it once in exact mode and prints `<model>/<output> <sha256>`. The hash covers the element type, the rank and dimensions as little-endian 64-bit integers, then the elements row-major. `tools/golden_hash.py` runs the same command under each row's environment and compares line by line. `tools/golden.sh` does it for the whole set on one machine; `tools/golden_wasm.sh` runs the same code on the browser-wasm runtime under node; `--merge` joins saved results from several machines into one table.

The ONNX backend node cases (`--cases`) are hashed the same way, one hash per case (output tensors in order). A case the library cannot run in exact mode prints a fixed marker hash instead, so "skipped" is compared across rows like any other result.

Rows (each must equal every other):

| Row | How |
|---|---|
| x64 JIT, default threads and `ONEMODEL_THREADS=1` | AVX-512 kernel path |
| x64 JIT, `DOTNET_EnableAVX512=0` | AVX2 kernels |
| x64 JIT, `DOTNET_EnableAVX2=0` (and AVX-512 off with it) | 4-lane SSE kernels |
| x64 JIT, `DOTNET_EnableHWIntrinsic=0` | scalar kernels |
| x64 NativeAOT (built for `x86-64-v3`), default threads and one thread | |
| arm64 JIT (macOS), default threads, one thread, `DOTNET_EnableArm64Dp=0`, `DOTNET_EnableHWIntrinsic=0` | NEON, with and without dot product, scalar |
| arm64 NativeAOT (macOS), default threads and one thread | |
| browser-wasm, interpreter | node, single thread, `PackedSimd` |
| browser-wasm, ahead-of-time compiled | node, single thread |

The wasm rows hold 11 of the 14 files: the fp32 BERT-base and the fp32 Whisper-small encoder and decoder are left out because the runtime has a 32-bit address space and the files are copied into its in-memory file system. All 1765 node cases ran on wasm.

Windows x64 and Linux arm64 run on GitHub-hosted runners: `.github/workflows/ci.yml` runs `tools/golden_ci.sh` on every platform in its matrix (JIT, one thread, hardware intrinsics off, NativeAOT, and wasm), comparing five committed test graphs with the expected hashes in `testdata/graphs/golden-exact.txt`. A mismatch on any runner fails the job.

## Result

All rows equal, for every model and for the node-case set. The reference models and the int8 variants all run in exact mode; no operator they use is flagged "not exact". (Fusion passes that change bits do not run in exact mode, see [graph-passes.md](https://recursiveai.org/onemodel/docs/graph-passes/); the QDQ rewrite is one of them, so the QDQ models run as dequantize, float convolution, quantize.)

Batch 1 inputs from the reference set (`reference-models.md`). The hash of the first output:

| Model | Output | SHA-256 |
|---|---|---|
| resnet50 fp32 | logits | `89f74a216bdf17bd450d0a74ed64e751b8c36ea314fc2a4f66d59576a327eaad` |
| mobilenet_v3_large fp32 | logits | `dbb86f523fb1323dd4d88f9b2b9d782f231bcc06a4774113e23cb710f8bedc37` |
| bert_base fp32 | last_hidden_state | `a2ec1187598fcdeffbb5802fc762765f7ade1ad0a2568a3ea16da27f10c3e962` |
| minilm_l6 fp32 | last_hidden_state | `ef19e17c1851b2124d844cbfa7255aeaad08c97d2f6d53e4c1995ff996eb1c53` |
| whisper_small encoder fp32 | encoder_hidden_states | `285cb006ab4f9840d5ae851841a9d173b7ba20a4cee847f462563815f2c62835` |
| whisper_small decoder fp32 | logits | `ab12ed9d247c8e4a6c93a89f0c40d3f15db891102bb5a0814d21d11b53ec5cd1` |
| crnn fp32 | logits | `d2a360cf1b37c911a7560b83562135dfbfca5fd8514d23bf330acbc10685425d` |
| dbnet fp32 | layout | `1302d16bf52f8c52c48179dbdd11ddb50b9c7a2a1ad79df325f465f8f5f3b338` |
| dbnet fp32 | orient | `9022e8cfc054257e6eecff03c01a4b05e22248e05d3c9cfc27933a3573d6fec1` |
| dbnet fp32 | prob | `1d5d519c988e5d91f7197fce32baf56eee80ebb10447c613ee616fe43391d508` |
| resnet50 int8 QDQ | logits | `6eb104959d4e5177bb587ab429cc078c775dd64917fb9f13ab859e1ea8fdc797` |
| mobilenet_v3_large int8 QDQ | logits | `0b54098f8cdfcb2ca03db41f4b0c375cc8dcfa54f676f5e20123371bf3b15ab3` |
| bert_base int8 dynamic | last_hidden_state | `a172050ebfbdde931fb3df08e833792f555740cda8be9fc4d7b7a3ba995a8790` |
| minilm_l6 int8 dynamic | last_hidden_state | `4cd90837755ebea1af067bd03ef631f8dd2779ef660b19f97b5bcf60f935615b` |
| whisper_small encoder int8 dynamic | encoder_hidden_states | `d2cb6f442450770ee80e2420331e2c24ea949d207efda3fbc07e125dc0757549` |
| whisper_small decoder int8 dynamic | logits | `745f0d6bfc63693a7815f4aeb7779402fda70a35f7bdda8befda7a13cd942764` |

The model files are the fetch-and-export results pinned in `tools/models.json`. A re-export that is not byte-identical (ResNet-50 and MobileNetV3 are not reproducible across machines) changes these hashes; the macOS host used copies of the exported files.

Node cases (ONNX 1.22.0 frozen set, 1765 cases): 1005 run in exact mode; 760 are skipped, all because the library does not implement an operator the case uses or stores a type it lacks (int4, float8), except 16 `Pow` cases: **`Pow` has no exact kernel and is the only implemented operator flagged "not exact"**. The SHA-256 over the sorted `case/<name> <hash>` lines is `87bfc594948556efe83691dc65e5da0cd89ed0d39aff391ead6844ac9d34a90a`.

## A bug the wasm row found

The first wasm rows differed on the two QDQ models. Cause: narrowing sixteen quantized values from `ushort` to `byte` (`Vector128.Narrow`) is truncating in the interpreter and in .NET on x64 and arm64, but saturating once the browser-wasm runtime has compiled the code (after about 2200 calls); a negative int8 value then became 0. `Int8Quantize.Pack16` now narrows through types that hold every value (int to short, then short to sbyte or byte), so truncating and saturating narrowing agree. A narrowing of `uint` to `ushort` (used by the fp16 conversions) showed no such difference in a 200000-call probe.

## Exact versus fast, indicative

Best of 5 warm runs, batch 1, one run per row on a loaded shared host, so read the ratios to about ±15%: exact time divided by fast time. Not a bar measurement.

| Model | x64 (AVX-512) 1 thread | x64 8 threads | arm64 (NEON) 1 thread | arm64 6 threads |
|---|---|---|---|---|
| resnet50 fp32 | 1.18 | 1.14 | 1.57 | 1.44 |
| mobilenet_v3_large fp32 | 1.59 | 1.25 | 0.75 | 1.43 |
| bert_base fp32 | 1.49 | 1.87 | 1.76 | 2.31 |
| minilm_l6 fp32 | 1.50 | 2.41 | 1.72 | 2.70 |
| whisper_small encoder fp32 | 1.60 | 2.35 | 1.53 | 1.97 |
| whisper_small decoder fp32 | 1.44 | 1.89 | 1.83 | 2.17 |
| crnn fp32 | 1.29 | 1.13 | 1.29 | 1.23 |
| dbnet fp32 | 1.51 | 1.57 | 1.36 | 1.27 |
| resnet50 int8 QDQ | 1.81 | 1.20 | 3.61 | 2.14 |
| mobilenet_v3_large int8 QDQ | 0.51 | 1.06 | 0.55 | 0.47 |
| bert_base int8 dynamic | 1.09 | 1.51 | 1.13 | 1.49 |
| minilm_l6 int8 dynamic | 1.31 | 1.48 | 1.21 | 1.60 |
| whisper_small encoder int8 dynamic | 1.50 | 1.81 | 1.14 | 1.49 |
| whisper_small decoder int8 dynamic | 1.29 | 1.84 | 1.10 | 1.76 |

Ratios below 1 are models whose fast run is slower than the exact one (the fast QDQ path is the integer kernels, the exact one is float). Exact is slower mainly because the fused attention, linear/GELU and QDQ rewrites do not run in exact mode, and because exact GEMM cannot fuse the multiply and add.

## Reproducing

```
tools/golden.sh --aot            # this machine's rows, needs the fetched models (tools/prepare.sh)
tools/golden_wasm.sh [--aot]     # browser-wasm rows under node (set NODE), needs the wasm-tools workload
tools/golden_ci.sh jit|aot|wasm  # the committed-hash check CI runs
tools/golden_cost.sh "1 8"       # exact-versus-fast timings
```


<!-- https://recursiveai.org/onemodel/docs/exact-transcendentals/ -->

# Exact fp32 transcendental functions

Normative. A port that follows this text reproduces the exact results bit for bit. Source: `src/OneModel/Kernels/Exact/Transcendental.cs`
(the algorithms) and `TranscendentalOps.cs` (the lane operations). Coefficients come from `tools/transcendentals/fit.py`
(minimax fits in relative error, Lawson iteration; rounded to fp32 afterwards).

## Rules

* Only IEEE-754 binary32 `+ - * /`, comparisons, bit operations and 32-bit integer add/subtract/shift on the float's bit pattern.
  No fused multiply-add, no reciprocal or square-root estimates, no library math. Every operation rounds to nearest-even, with
  subnormals enabled.
* Every expression below is evaluated exactly as written, left to right, one rounding per operator. `acc + a*b` means a product
  rounding, then a sum rounding. Horner's rule `c_k + (...)*x` runs from the highest coefficient down.
* Special values are selected by comparisons at the end, so the lane that computed garbage on the way (for example a shift of an
  infinity) never shows. Any NaN result is the canonical quiet NaN `0x7FC00000`; NaN payloads and signs of inputs are not kept.
* "Round to nearest integer" is `((x*c) + M) - M` with `M = 12582912 (1.5*2^23)`; the integer is read back as
  `bits(x*c + M) - bits(M)`. It needs |x*c| < 2^22 and is not used outside the thresholds below.
* The same operation sequence runs for scalar, 128-, 256- and 512-bit lanes, so the width never changes a bit.

## exp(x)

1. `t = x*1.44269502 + M`; `n = t - M` (as float); `n_int = bits(t) - bits(M)`.
2. `r = (x - n*0.693145751953125) - n*1.4286068e-6`. (`0.693145751953125` has 14 significant bits, so `n*ln2Hi` is exact for |n| < 256.)
3. `q = ((((0.0013814613*r + 0.00836871)*r + 0.04166839)*r + 0.16666521)*r + 0.49999994)`, written as `c + q*r` per step;
   `p = 1 + (r + (r*r)*q)`.
4. `a = n_int >> 1` (arithmetic), `b = n_int - a`; `result = (p * float(bits((a+127)<<23))) * float(bits((b+127)<<23))`
   (two exact power-of-two scalings, so a subnormal result rounds once).
5. `x > 88.72283172607422` gives `+inf`; `x < -103.97207641601562` gives `+0`; NaN gives the canonical NaN.

`ExpNonPositive` (used by softmax and attention, argument <= 0 or NaN) runs steps 1-3, then `result = p * float(bits((n_int+127)<<23))`,
`+0` for `x < -87.33654`, canonical NaN for NaN. For x in [-87.33654, 0] it equals `exp` bit for bit (tested); below that it flushes
to zero instead of producing subnormals.

## log(x)

1. `x < 1.17549435e-38` (subnormal or zero): `xs = x * 16777216`, exponent correction `-24`; otherwise `xs = x`.
2. `ix = bits(xs) - 0x3F3504F3`; `e = ix >> 23` (arithmetic); `m = float((ix & 0x007FFFFF) + 0x3F3504F3)` in [sqrt(1/2), sqrt(2)).
   `ef = float(e) - 24*[subnormal]`, with `float(e)` formed as `float(e + 0x4B400000) - M`.
3. `f = m - 1` (exact); `s = f/(2+f)`; `z = s*s`; `w = ((((1/11)*z + 1/9)*z + 1/7)*z + 1/5)*z + 1/3` with the fp32 constants
   `0.0909090936, 0.111111112, 0.142857149, 0.200000003, 0.333333343`; `s2 = s+s`; `logm = s2 + s2*(z*w)`.
4. `result = ef*0.693145751953125 + (logm + ef*1.4286068e-6)`.
5. `x > FLT_MAX` gives `+inf`; `x < 0` gives the canonical NaN; `x == 0` (either zero) gives `-inf`; NaN gives the canonical NaN.

## sigmoid(x)

`s = exp(0 - |x|)`; `num = (x >= 0) ? 1 : s`; `result = num / (1 + s)`. NaN gives the canonical NaN.

## tanh(x)

`ax = |x|`, `z = ax*ax`. For `ax < 0.6`: `P = ((((-0.0058951583*z + 0.020795971)*z - 0.05378383)*z + 0.13331924)*z - 0.333333)`
(Horner, `c + P*z` steps), `result = ax + ax*(z*P)`. Otherwise `E = exp(ax+ax)`, `result = 1 - 2/(E+1)`. The sign bit of `x` is OR-ed
in last (so `tanh(-0) = -0`). NaN gives the canonical NaN.

## erf(x), and 1 + erf(x) for the erf form of GELU

`ax = |x|`, `z = ax*ax`.

* `ax < 0.5`: `P = (((((-0.0007692853*z + 0.0052003083)*z - 0.02686312)*z + 0.11283774)*z - 0.37612638)*z + 1.1283792)`; `erf = ax*P`;
  `1 + erf(x) = 1 + (erf OR sign)`.
* `ax >= 0.5`: `t = 1/ax`, `s = t - 1`, `G` = the degree-16 Horner polynomial in `s` with coefficients (highest first)
  `-0.00057763467, 0.0017893383, -0.00010074016, -0.004373198, 0.0034639547, 0.0015023724, -0.0009413088, -0.0018891813, -0.0002586738,
  0.0052144383, -0.0103428485, 0.015788378, -0.018338649, 0.008167262, 0.035529688, -0.1543716, 0.42758358`.
  `x*x` is formed exactly as `z + zLo`: `split = ax*4097`, `hi = split - (split - ax)`, `lo = ax - hi`,
  `zLo = (((hi*hi - z) + 2*(hi*lo)) + lo*lo)`, set to 0 when `ax >= 64`. `expNegZ = exp(0 - z) * (1 - zLo)`; `erfc = expNegZ * (t*G)`;
  `erf = 1 - erfc`; `1 + erf(x) = erfc` for `x < 0`, `2 - erfc` otherwise.

The sign bit is OR-ed onto `erf`. `erf(+-inf) = +-1`; NaN gives the canonical NaN.

## GELU

* erf form: `y = x*0.70710677`; `gelu = (0.5*x) * (1 + erf(y))` with `1 + erf` from the section above (no cancellation for negative `y`).
* tanh form: `u = 0.79788456 * (x + 0.044715*((x*x)*x))`; `gelu = x * sigmoid(u + u)`. This is `0.5x(1+tanh u)` through the exact identity
  `1 + tanh(u) = 2 sigmoid(2u)`.
* Both: `gelu(-inf) = -0`, `gelu(+inf) = +inf`, NaN gives the canonical NaN.

## Measured error

Error is `|got - reference| / spacing`, where the reference is computed in double (`Math.Exp/Log/Tanh`, erf by series and continued
fraction, accurate to about 1e-13 relative) and the spacing is the fp32 spacing at the correctly rounded reference. Sweeps use every
fp32 value in the hot range (stride 1) and every 67th value of the wide range. Produced by
`ONEMODEL_ULP_TABLE=<file> ONEMODEL_ULP_STRIDE=1 ONEMODEL_ULP_THREADS=8` with the `WriteUlpTable` test.
"Fast" is the same code with fused accumulation (and `FastMath.Exp` for `exp`); it is not bit-reproducible across CPUs.

| function | variant | sweep | values | max ulp | at x |
|---|---|---|---|---|---|
| Exp | exact | hot [-16, 16] stride 1 | 2197815297 | 0.976 | -5.8606462 |
| Exp | exact | wide [-110, 90] stride 67 | 33444880 | 0.953 | 15.626762 |
| Exp | fast | hot [-16, 16] stride 1 | 2197815297 | 2.254 | 5.2451344 |
| Exp | fast | wide [-87, 88] stride 67 | 33395972 | 2.232 | 29.654173 |
| Log | exact | hot [0.0625, 16] stride 1 | 67108865 | 1.968 | 1.0038618 |
| Log | exact | wide [0, FLT_MAX] stride 67 | 31926792 | 1.929 | 1.0077533 |
| Log | fast | same as exact (the polynomial term is too small for fusing to matter) | | 1.968 | 1.0038618 |
| Tanh | exact | hot [-10, 10] stride 1 | 2185232385 | 1.332 | -0.6310442 |
| Tanh | exact | wide [-50, 50] stride 67 | 33194474 | 1.315 | -0.6016164 |
| Tanh | fast | same as exact | | 1.332 | -0.6310442 |
| Erf | exact | hot [-6, 6] stride 1 | 2172649473 | 2.762 | -0.47693625 |
| Erf | exact | wide [-10, 10] stride 67 | 32615409 | 2.484 | -0.50197357 |
| Erf | fast | hot [-6, 6] stride 1 | 2172649473 | 2.762 | -0.47693625 |
| Sigmoid | exact | hot [-16, 16] stride 1 | 2197815297 | 2.402 | -4.157294 |
| Sigmoid | exact | wide [-110, 90] stride 67 | 33444880 | 2.264 | -2.7397301 |
| Sigmoid | fast | same as exact | | 2.402 | -4.157294 |
| GeluErf | exact | hot [-3, 10] stride 1 | 2170552321 | 11.943 | -2.8339381 |
| GeluErf | fast | hot [-3, 10] stride 1 | 2170552321 | 11.583 | -2.843721 |
| GeluTanh | exact | hot [-3, 10] stride 1 | 2170552321 | 16.996 | -2.9897327 |
| GeluTanh | fast | same as exact | | 16.996 | -2.9897327 |
| GeluErf | exact | negative tail [-6, -3] stride 1 | 8388609 | 46.145 | -5.962 |
| GeluErf | exact | negative tail [-12, -6] stride 1 | 8388609 | 179.769 | -11.988144 |
| GeluTanh | exact | negative tail [-6, -3] stride 1 | 8388609 | 61.997 | -5.8887057 |
| GeluTanh | exact | negative tail [-12, -6] stride 1 | 8388609 | 218.405 | -9.859064 |
| MathF.Exp / Log / Tanh (platform library, for comparison) | | hot, stride 8 | | 0.502 / 0.817 / 2.168 | |

The GELU errors grow in the negative tail because the relative sensitivity of GELU to a rounding of its argument is about x^2
there: even a perfect evaluation of the fp32 formula from the fp32 input cannot do better. The absolute error stays below
about 2e-7 of the input scale for x >= -3. The `ExactAccuracyIsWithinTheDocumentedBounds` test pins the bounds (exp 1.2, log 2.1, tanh 1.5,
erf 3.0, sigmoid 2.5, GELU erf 13, GELU tanh 18 ulp).

## Fast exponential

`FastMath.Exp` (fast attention and softmax): clamp x to [-87.3, 88.3]; `t = fma(x, 1.44269502, M)`, `n = t - M`,
`r = fma(n, -ln2Hi, x)` then `fma(n, -ln2Lo, r)`; `q` a degree-3 fit (`0.008312525, 0.041890115, 0.16667114, 0.4999923`),
`p = 1 + fma(r*r, q, r)`; scale by one `2^n` built in the exponent field; `+0` for `x < -87.3`. About 2.3 ulp.

## Cross-platform check

`TranscendentalTests.ExactBitsMatchTheGoldenHashes` hashes (SHA-256) the exact outputs of all seven functions over every 4099th fp32
bit pattern plus a list of special values. The same hashes were reproduced on x64 with AVX-512 / AVX2 only / SSE only /
`DOTNET_EnableHWIntrinsic=0`, on arm64 (Apple M2, NEON) and with hardware intrinsics off there, and in a NativeAOT build.


<!-- https://recursiveai.org/onemodel/docs/fp16/ -->

# Half-precision storage (binary16 and bfloat16)

The library computes in binary32. A model stored in binary16 or bfloat16 runs with half-precision storage between operators and
binary32 arithmetic inside them. The conversion rules are in [tensors.md](https://recursiveai.org/onemodel/docs/tensors/) section 10; this note says how they are used.

## Models stored in half precision

Applies to a model whose weights (initializers), constants or intermediate values are `float16` or `bfloat16`, or that casts to one of
them. Inputs and outputs of the graph keep the types the file declares; a float32 model with a `Cast` to float16 at each end works.

At session creation the graph is scanned; if any such type occurs, every node whose kernel can be wrapped gets an adapter. For a node
with at least one half-precision input:

1. each half-precision input is widened to binary32 (exact);
2. the kernel runs on binary32, in the same fast or exact flavour as always;
3. each output the kernel produces as binary32 is rounded back to the storage type, to nearest even.

The storage type of the outputs is the type of the node's first half-precision input that is not a compact weight (below); a node with
only compact weights and binary32 activations stays in binary32. Only element-wise `Add`, `Sub`, `Mul` and `Div` with all inputs
binary16 skip the adapter (they compute in binary16 themselves, a correctly rounded operation either way). `Cast`, `CastLike`,
`Constant`, `ConstantOfShape`, `Shape`, `Size`, views (`Reshape`, `Squeeze`, ...) and nodes whose output shape depends on values are
never wrapped; they see the real element types.

Constant half-precision inputs (weights) are widened once when a plan is made and kept as binary32 in the plan, so a model stored in
half precision uses its binary32 size in memory while it runs; the file is half the size. Constant folding applies as for binary32.

Because every conversion is exact (widening) or the specified round-to-nearest-even, exact mode stays bit-identical across machines and
thread counts: a half-precision model is an exact-mode model whenever its operators have exact kernels. The vector conversions
(`Kernels/HalfConvert.cs`) are the same integer arithmetic as the scalar ones, eight values at a time (a test compares them with the
scalar code over every binary16 and bfloat16 pattern and several million float32 patterns, including every rounding boundary).

### Accuracy against ONNX Runtime

ONNX Runtime runs such files on its CPU provider too. The files in `tools/models.json`
(`fp16` entries, made by `tools/export_models.py --fp16` from the float32 files: weights and activations float16, float32 graph inputs
and outputs, finite values beyond the float16 range clamped to ±65504) are compared by scaled error (max |ours − reference| / max
|reference| per output):

* Gate for float16 files: **scaled error against ONNX Runtime's output on the same file ≤ 4e-3**, and ours against the float32
  model's output no worse than 1.5 times ONNX Runtime's own float16-versus-float32 error. Half precision has an 11-bit significand
  (2^-11 = 4.9e-4 per rounding), and the errors of the models are a few roundings; two correct engines that round at slightly different
  places differ by that much, and the gate allows about twice what was measured.
* Measured (x64, batch 1 and the model's batch): see the table in [oracles.md](https://recursiveai.org/onemodel/docs/oracles/).

## Keeping binary32 weights compact (`SessionOptions.WeightStorage`)

`WeightStorage.Binary16` or `BFloat16` rounds the constant weights of `MatMul`, `Gemm` and `Conv` (input 1, at least 1024 values, read
by nothing else) when the session is created, frees the binary32 copy, and widens the weights into run scratch for each use. The weight
memory halves; the cost is one widening pass per weight per run, and the results change by the rounding of the weights only: activations
stay binary32, so the result equals a binary32 model whose weights were rounded to the same format beforehand, bit for bit (a test
checks this at 1 and 4 threads). Embedding tables (`Gather`), biases and norm parameters are not compacted.

bfloat16 keeps the binary32 range with 8 bits of significand (2^-8 = 3.9e-3 per rounding); binary16 has more precision but ends at 65504,
so a weight tensor with a value beyond that (or NaN or infinity) is left as it is.

## Rules

* Round to nearest, ties to even, on every narrowing; NaN stays NaN and is made quiet; overflow becomes infinity.
* Widening is exact.
* The input and output types of a graph are not changed by the options; `Session.Run` takes and returns what the file declares.


<!-- https://recursiveai.org/onemodel/docs/int8/ -->

# int8 inference

Operators, kernels and graph rewrites for 8-bit quantized models. Integer arithmetic is exact, so every int8 kernel gives the same bits on every instruction set and every thread count. The float parts around it (scales, dequantization) follow the order written here.

## Operators

| Operator | Notes |
|---|---|
| `QuantizeLinear` | uint8, int8, uint16, int16 outputs; per-tensor and per-axis scale. `y = clamp(round_half_even(x / scale) + zp)`; the division is a float32 division. NaN quantizes to the smallest value of the type. |
| `DequantizeLinear` | uint8, int8, uint16, int16 and int32 inputs; `y = float(x - zp) * scale` in float32. A node with constant inputs is folded at plan time. |
| `DynamicQuantizeLinear` | uint8 output. `lo = min(0, min x)`, `hi = max(0, max x)`, `scale = (hi - lo) / 255` (float32), `zp = clamp(round_half_even(0 - lo / scale), 0, 255)`. An all-zero input gives scale 1, zero point 0, output 0 (as ONNX Runtime does). |
| `MatMulInteger`, `ConvInteger` | int32 results, uint8 or int8 operands in any mix. The zero point of the second operand of `ConvInteger` must be per-tensor. |
| `QLinearMatMul`, `QLinearConv` | Requantize with `y = clamp(round_half_even(float(acc) * mult) + zp_y)`, `mult = (sx * sw) / sy` computed in float32 (per output channel when `sw` is a vector). `QLinearConv` adds an int32 bias to `acc` first. `QLinearMatMul` also accepts a ninth input, an int32 per-column bias (an extension used by the QDQ pass; not in ONNX). |
| `MatMulIntegerToFloat` (internal) | `MatMulInteger -> Cast(float) -> Mul(scale)[-> Add(bias)]` evaluated in one kernel: `float(acc) * scale (+ bias)`, the same float32 operations in the same order. |

Not supported: blocked quantization (`block_size`), int4/uint4, 3-D convolution. All other dtypes combinations the ONNX specification allows for these operators are supported.

## Product kernel (`Int8Gemm`)

`C[m,n] = sum_k (A[m,k] - zA) * (B[k,n] - zB)` is computed on the raw integers and corrected afterwards (Jacob et al. 2018, section 2.3):

```
S[m,n]  = sum_k A'[m,k] * B'[k,n]                          (A' = A as int16 shifted so that uint8 becomes int8 range: A' = A - 128 for uint8)
Y[m,n]  = S - zA' * colSum[n] - zB' * (rowSum[m] - K * zA')
```

where primes denote the shifted domain (`zA' = zA - 128` for uint8 A, `zA` for int8; B is stored as int8 and a uint8 B is shifted by 128 the same way). Sums are int32; they cannot overflow while `K * 255 * 128 < 2^31`, i.e. K up to about 65,000 (not checked; the largest K among the reference models is 4608).

* Goto and van de Geijn blocking: `Kc = 512`; work chunks of `Mc = 48` rows by `Nc = 64` columns, constants of the kernel, so results never depend on the thread count. A is packed once per product (in parallel over row blocks), B once per weight when the weight is constant (prepacked at session creation) and per product otherwise.
* x86 (AVX2 and AVX-512): A is widened to int16, B kept int8 and widened in registers; pairs of k are multiplied with `vpmaddwd` (`G = 2`). This never saturates, unlike `vpmaddubsw`, which ONNX Runtime's AVX2 kernels use; it is half the multiply-adds per instruction of a VNNI kernel but exact. Tiles: AVX2 6x16, AVX-512 12x32. Packed A: panels of `mr` rows, k in pairs, int16; packed B: panels of `nr` columns, k in pairs, int8.
* Arm with dot product (`SDOT`): groups of four k (`G = 4`), tile 4x16, int8 both sides.
* Scalar, 128-bit portable and WASM: scalar loops over the same packed layout of the x86 path (`G = 2`).
* The epilogue (zero-point correction, optional bias, requantization or float conversion) is vectorized and uses the formulas above, so it is identical on every path.

The kernel is checked against a 64-bit naive reference on every supported path, random shapes, zero points and epilogues (`Int8GemmTests`), and against ONNX Runtime 1.31.1 bit for bit on 400 operator cases (`OracleQuantTests`, cases from `tools/ops_quant_oracle.py`). One exception: ONNX Runtime's own `Add` of two quantized tensors differs by 1 LSB between its x64 and arm64 builds, so the `qdq_add` cases carry a tolerance of 1.

## Convolution (`Int8Conv`)

im2col is fused into the packing of A (no column buffer); weights are prepacked per group; depthwise convolutions have a direct kernel. The result and the epilogue are the product kernel's.

## Graph rewrites

| Pass | Bits | Does |
|---|---|---|
| `qdq` | not preserved | `DequantizeLinear (input), DequantizeLinear (weights) [, DequantizeLinear (bias)] -> Conv or Gemm -> QuantizeLinear` becomes `QLinearConv` / `QLinearMatMul`. Conditions: constant 8-bit weights, constant scales and zero points, scalar activation scale and zero point, an optional int32 bias carrying the scale `sx * sw`, and the product feeding only the `QuantizeLinear`; anything else is left as written. The rewrite replaces float arithmetic by integer arithmetic, so results change by rounding; the pass is skipped in exact mode, where the QDQ graph runs as written (float simulation on the exact kernels). |
| `matmul-integer-fusion` | preserved | The three or four nodes of the `MatMulIntegerToFloat` row above become one node, placed at the position of the last absorbed node. |

## Noise floor of the model-level metric

`oracles.md` defines the int8 metric: our output must be at least as close to ONNX Runtime 1.31.1's `ORT_ENABLE_ALL` output as ONNX Runtime's own sensitivity allows, the floor `f`. `tools/check_int8.py` measures `f` as the smallest value of each statistic (1st-percentile row cosine, pooled cosine, decision count) over three sets of ONNX Runtime runs, all compared with the `ORT_ENABLE_ALL` reference:

1. `ORT_DISABLE_ALL` on the same inputs.
2. 16 `ORT_ENABLE_ALL` runs with relative Gaussian noise of 1e-7 (about one float32 ulp) on every float graph input.
3. For dynamically quantized models, 16 `ORT_ENABLE_ALL` runs with the same noise on the float tensor that feeds the first `DynamicQuantizeLinear`. Encoders whose inputs are token ids have no float input to perturb, so set 2 is empty for them; this set perturbs the embedding output (the `LayerNormalization` that ends the embeddings) instead. The tool adds a graph input `noise_tap` to that tensor with an `Add` (zeros leave the result unchanged) and feeds `base * 1e-7 * N(0,1)`. It is skipped when that tensor is itself a graph input (QDQ networks, the Whisper encoder), because set 2 already covers it. The `onnx` package is used only to edit the graph; `onnxruntime` only runs it.

Why. One float ulp upstream moves a few 8-bit roundings, and the flips spread through the network, so ONNX Runtime does not reproduce its own int8 outputs under perturbations this small. Sets 1 and 2 leave id-input encoders with only the level spread, which for MiniLM at batch 32 is optimistic (0.9895, against 0.9838 once the embedding is perturbed).

Why 16 runs. The floor is a minimum, so it falls as runs are added. Four runs gave 0.9525 for the Whisper encoder; 16 gave 0.8748 (the dynamic scheme is sensitive to outliers, so the spread is wide). A gate that depends on which four draws came up is not a gate, so the draw count is fixed at 16 (`--trials N` changes it). The noise generator is seeded, so the floor is repeatable.

### Rows (x64, 1 thread, ONNX Runtime 1.31.1, `tools/check_int8.sh`)

Row cosine is the 1st percentile over rows (the minimum below 100 rows); pooled is the minimum cosine of the mean-pooled vectors. "Old" is the floor of the earlier method (level spread and 4 input-noise runs); "new" is the one above. Decisions apply to task rows only (class per image, token per position); a floor is the number of positions where ONNX Runtime's worst run agrees with its reference.

| Row | Row cos (ours) | Row floor old / new | Pooled (ours) | Pooled floor old / new | Decisions (ours; floor old / new) | Result |
|---|---|---|---|---|---|---|
| ResNet-50 QDQ b1 | 0.9998 | 0.9999 / 0.9999 | n/a | n/a | 1/1; 1 / 1 | pass |
| ResNet-50 QDQ b8 | 0.9998 | 0.9997 / 0.9997 | n/a | n/a | 8/8; 8 / 8 | pass |
| MobileNetV3 QDQ b1 | 0.9975 | 0.9963 / 0.9963 | n/a | n/a | 1/1; 1 / 1 | pass |
| MobileNetV3 QDQ b8 | 0.9959 | 0.9913 / 0.9913 | n/a | n/a | 8/8; 6 / 6 | pass |
| BERT-base dynamic b1 | 0.9957 | 0.9974 / 0.9918 | 0.9989 | 0.9997 / 0.9982 | none | pass |
| BERT-base dynamic b32 | 0.9921 | 0.9876 / 0.9853 | 0.9978 | 0.9973 / 0.9959 | none | pass |
| MiniLM dynamic b1 | 0.9982 | 0.9923 / 0.9922 | 0.9997 | 0.9976 / 0.9976 | none | pass |
| MiniLM dynamic b32 | 0.9851 | 0.9895 / 0.9838 | 0.9931 | 0.9951 / 0.9921 | none | pass (a known gap before) |
| Whisper decoder dynamic b1 | 0.9915 | 0.9536 / 0.9536 | 0.9999 | 0.9997 / 0.9997 | 16/16; 14 / 14 | pass |
| Whisper encoder dynamic b1 | 0.9502 | 0.9525 / 0.8748 | 1.0000 | 0.9999 / 0.9999 | none | pass at 16 runs (a known gap before; fails by 0.0023 at 4 runs) |

The gate is min(0.99, floor), so a row whose ours exceeds 0.99 passes whatever the floor.

The two former gaps. MiniLM b32 passes because perturbing the embedding lowers ONNX Runtime's own floor to 0.9838 while ours is 0.9851, a margin of 0.0013 (with 4 draws per set it also passes, at 0.9840). The Whisper encoder passes only because 16 draws find a worse ONNX Runtime run than 4 did; its 0.9502 lies between the 4-run floor (0.9525) and the 16-run floor (0.8748), so the verdict depends on the draw count, and ONNX Runtime's own scatter on that model is wide (scaled error against its reference: 0.62 for us, 0.90 for its worst noisy run). The Whisper encoder's first quantized input is the float graph input (the mel features), so there is no embedding tap for it. Neither row shows a defect under this method; the Whisper encoder's margin is a judgement about how many draws make a fair minimum.

## Metric against ONNX Runtime

Defined in `oracles.md`. Operator-level results must equal ONNX Runtime's bit for bit on the oracle cases; model-level results are compared by cosine similarity and decision agreement, because the integer rounding chaos amplifies float noise (see there).


<!-- https://recursiveai.org/onemodel/docs/ops-coverage/ -->

# Operators added for backend-test coverage

Semantics the ONNX text leaves open, as settled by ONNX's backend-test data or, where that is silent, by ONNX Runtime 1.31.1's output on crafted inputs. Operators are checked against the backend tests (`docs/conformance.md`) and against ONNX Runtime on random inputs (`tools/ops_coverage_oracle.py`, 340 cases, run by `CoverageOracleTests` in both modes and on every kernel path).

## Exactness

An operator is exact (results bit-identical on every machine) when it moves elements, uses only `+ - * /` and square root in a fixed order, or uses the library's own `exp`/`log` polynomials (`docs/exact-transcendentals.md`). Not exact, refused in exact mode with a message naming the operator: the trigonometric and hyperbolic functions and their inverses, LRN, LpPool, Attention (opset 23 and later), DFT, STFT, the window functions, MelWeightMatrix, GridSample, RoiAlign, SoftmaxCrossEntropyLoss, and the float64 log reductions.

## Settled semantics

- **Attention (opset 23, 24).** `softcap` is applied before the mask; `is_causal` lets query i see keys j <= i + past_sequence_length; an attention mask shorter than the key length masks the remainder.
- **MaxUnpool.** With `output_shape`, an index is a position in the shape the pooling implies and is re-based into `output_shape` (this is what the ONNX test data expects; ONNX Runtime reads it as a flat position in `output_shape` and differs, so there is no oracle case for it). An index outside either shape is an error.
- **RoiAlign.** `coordinate_transformation_mode` defaults to `half_pixel` from opset 16 and `output_half_pixel` before. `sampling_ratio` 0 uses ceil(region size / bins). In max mode each of the four weighted neighbours competes on its own and the largest wins (the backend cases expect this).
- **LpNormalization, MeanVarianceNormalization.** A zero norm gives zeros.
- **Unique.** Sorted order compares slices element by element; NaN is never equal to NaN; the three index outputs are optional.
- **Resize with `antialias` = 1** (linear and cubic). On an axis whose scale is below 1 the filter's reach grows to 1/scale times its half-width (1 for linear, 2 for cubic). Every integer position i with a non-zero weight filter((i - x) * scale) around the sample position x gets that weight; positions outside the input read the nearest edge sample, so their weights add to it (with `exclude_outside` = 1 they are dropped); weights are divided by their sum. Axes that grow, and nearest mode, are unchanged. This is ONNX Runtime's behaviour, found by reading the weight matrix off impulse inputs. Fast and exact give the same bits.
- **ConvTranspose with three spatial axes.** Direct loops in a fixed order (input element, output channel, kernel tap), no matrix product; exact.
- **MelWeightMatrix.** num_mel_bins + 2 edge points evenly spaced in mel (2595 log10(1 + f / 700)) between the lower and upper edge, converted back to hertz, then to bins as floor((dft_length + 1) f / sample_rate), in double precision; filter i rises from edge i to edge i + 1 and falls to edge i + 2, peak 1. Only float and double outputs.
- **Squeeze, Unsqueeze.** A scalar `axes` input is accepted as one axis.

## Not covered

Element types float8, float4, 4-bit and 2-bit integers, complex and string; the `ai.onnx.ml` and training domains; DeformConv, ImageDecoder; blocked quantization; Bernoulli and the Random* family; Dropout in training mode; BatchNormalization in training mode; LinearAttention, CausalConvWithState and FlexAttention (opset 27 and preview); float64 `Exp`, `Log` and `Softmax`. Models below opset 7 are outside the import range.


<!-- https://recursiveai.org/onemodel/docs/ops-elementwise/ -->

# Element-wise operators, activations, comparisons, Where and Cast

Normative notes for the operators in `src/OneModel/Graph/Operators/Elementwise/`. Semantics come from the ONNX operator
specification; where the specification leaves a case open, the choice below was settled by running ONNX Runtime 1.31.1 on a crafted
input (cases in `tools/ops_elementwise_oracle.py`, outputs in `testdata/ops-elementwise/cases.json.gz`).

## Operators and modes

"Exact" means the operator has an exact kernel (bit-identical on every machine, `exact-mode.md`). Every operator here except Pow is
exact: each output element is one correctly rounded IEEE operation, a comparison, a bit operation, a documented conversion, or one of
the exact transcendentals of `exact-transcendentals.md`.

| Operator | Opsets | Element types | Mode |
|---|---|---|---|
| Sub, Div | 7+ | f16, f32, f64, i8..i64, u8..u64 | exact |
| Mod | 10+ | fmod=1: f16, f32, f64, all integers. fmod=0 (default): integers only | exact |
| Pow | 7+ | base f32, f64, i32, i64; exponent those or u32, u64 | **not exact** |
| Max, Min | 6+ | all numeric incl. f16 | exact |
| Sum, Mean | 6+ | f16, f32, f64 | exact |
| Exp, Log, Tanh, Sigmoid, Erf | 6+ (Erf 9+) | f32 | exact |
| Gelu | 20+ | f32; `approximate` = `none` or `tanh` | exact |
| LeakyRelu, HardSigmoid, HardSwish | 6+ (HardSwish 14+) | f32 | exact |
| Neg | 6+ | signed numeric | exact |
| Abs, Sign | 6+ (Sign 9+) | all numeric | exact |
| Reciprocal, Sqrt, Floor, Ceil | 6+ | f16, f32, f64 | exact |
| Round | 11+ | f16, f32, f64 | exact |
| Clip | 6+ | all numeric | exact |
| Equal | 7+ | all numeric and bool | exact |
| Less, Greater | 7+ | all numeric | exact |
| LessOrEqual, GreaterOrEqual | 12+ | all numeric | exact |
| Not | 1+ | bool | exact |
| And, Or, Xor | 7+ | bool | exact |
| Where | 9+ | any stored type | exact |
| Cast, CastLike | 6+, 15+ | any stored type to any stored type | exact |

"All numeric" is f16, f32, f64, i8, i16, i32, i64, u8, u16, u32, u64. Add and Mul take f16, f32, f64 and the integers. Half-precision arithmetic is done in single precision and rounded to nearest even once, which is exact for + - * /.

Not implemented (reported as `Unsupported` when the model is loaded): arithmetic on bf16;
string tensors; the float8 and 4-bit types (Cast to or from them); double-precision Exp, Log, Tanh, Sigmoid, Erf, Gelu, LeakyRelu,
HardSigmoid and HardSwish (they take f32 only); Pow with integer types narrower than 32 bits.
Operators of opsets before 6 (and the broadcast attributes of Add before 7) are not served.

## Broadcasting and in-place use

Binary operators broadcast NumPy style (`Shape.TryBroadcast`); Max, Min, Sum, Mean fold their inputs left to right, so the result
never depends on anything but the input order. Operators whose output has the type and shape of input 0 may overwrite it (the plan
decides); comparisons, Where, Cast, Pow with a different exponent shape, and the variadic operators never do.

## NaN, signed zero and special values

- **NaN results in exact mode** are rewritten to the quiet NaN with no payload and positive sign (0x7FC00000 for f32). Fast mode
  leaves what the hardware produced.
- **Max, Min:** a NaN in either operand gives NaN (the first operand's if both are). When the operands compare equal the second is
  returned, so `Max(-0, +0)` is `+0` and `Max(+0, -0)` is `-0`; ONNX Runtime agrees on the crafted cases.
- **Comparisons:** NaN compares false with everything, including itself (so `Equal(NaN, NaN)` is false and `LessOrEqual` is false).
- **Sign:** -1, 0 or +1; both zeros give `+0` (ONNX Runtime does too); NaN gives NaN.
- **Round:** half to even, keeps the sign of zero (`Round(-0.4)` is `-0`). **Floor, Ceil, Sqrt, Reciprocal:** IEEE.
- **Clip:** `min(max(x, lo), hi)` with the upper bound winning if `lo > hi`. A bound that is omitted is the type's finite extreme,
  not infinity: for floats `Clip(+inf)` with no bounds gives `+MaxValue` (ONNX Runtime does this). NaN passes through. Before opset 11
  the bounds are the attributes `min` and `max` (defaults: lowest and highest finite float); from 11 they are optional scalar inputs.
- **Gelu(-inf):** the library returns `-0` (the limit); ONNX Runtime returns NaN (it computes `-inf * 0`). Not changed: the exact
  functions are specified in `exact-transcendentals.md`.
- **Pow:** a float base uses single-precision `pow` with the exponent converted to float, a double base uses double `pow`, an integer
  base computes in double and converts back, saturating. An exponent of 2 is `x * x`. `Pow(x, 0)` is 1 for every x including NaN, as in C.
  Not exact because the platform `pow` is not bit-reproducible.

## Integers

All integer arithmetic wraps around (two's complement) for Sub, Add, Mul and Neg. `Div` truncates toward zero. Cases that would trap
in C# are defined: `x / 0` and `x % 0` give 0 (ONNX leaves them undefined), `MinValue / -1` wraps to `MinValue` and `MinValue % -1`
is 0. `Mod` with `fmod = 0` takes the sign of the divisor (floor modulus); with `fmod = 1` the sign of the dividend (C `fmod` for
floats). `Mod` with `fmod = 0` on a float type is rejected, as the specification requires `fmod = 1` there.

## Cast

Conversions go through a wide intermediate in chunks of 256: floating-point sources (f32, f64, f16, bf16) through double, which holds
all four exactly; integer and bool sources through a 64-bit integer.

| From | To | Rule |
|---|---|---|
| float | float | round to nearest even (f32 to f16 and bf16 included; overflow gives infinity) |
| float | integer | truncate toward zero, saturate at the target's range, NaN gives 0 |
| float | bool | `x != 0` (NaN is true; -0 is false) |
| integer | integer | wrap modulo 2^n (so `int32 300` to `uint8` is 44 and `-1` to `uint8` is 255) |
| integer | float | round to nearest even; u64 is read as unsigned |
| integer | bool | `x != 0` |
| bool | any | 0 or 1 |

The float-to-integer saturation and NaN rule is the library's choice where C leaves the out-of-range case undefined; the in-range
cases and the integer wrap match ONNX Runtime on all oracle cases. `CastLike` casts to the element type of its second input and never
reads that input's values. Cast to a type the library does not store is an `Unsupported` error naming the ONNX type.

## Function definitions (f32)

The transcendental functions are the ones in `exact-transcendentals.md`, run on the exact lane operations in exact mode and on the
fast lane operations (fused multiply-add where the CPU has it) in fast mode; they agree to a few ulp.

- LeakyRelu: `x < 0 ? alpha * x : x` (default alpha 0.01); NaN passes through.
- HardSigmoid: `clamp(alpha * x + beta, 0, 1)` with a rounded product and then a rounded sum (defaults 0.2 and 0.5).
- HardSwish: `x * clamp(x * 0.16666667f + 0.5f, 0, 1)`.
- Gelu: erf form `0.5 * x * (1 + erf(x / sqrt 2))` or the tanh form, both from `exact-transcendentals.md`.

Lane widths: the widest group the kernel path allows (512, 256 or 128 bits) runs first and a one-lane instantiation of the same
code finishes the tail, so an element's result does not depend on its position. There is no threading in these kernels yet; they
are memory-bound and a fixed chunk size would be added with the first measurement that wants it.

## Verification

`testdata/ops-elementwise/cases.json.gz` holds 700+ cases (graph, inputs, ONNX Runtime output) over shapes of size 0 and 1, scalars,
broadcasting of rank 0 to 4, odd lengths around the vector widths, NaN, infinities, signed zeros and subnormals. The tests run every
case on every kernel path of the machine in both modes: results must match within a scaled error (largest difference over the largest
magnitude, with that magnitude floored at 1e-3) of 0 for pure IEEE operations and 1e-4 for the transcendentals, and exact-mode bits
must be equal across paths. ONNX Runtime's own Sigmoid and tanh-form Gelu are off by up to 1e-4 relative for inputs below about -10,
where the library is closer to the true value.


<!-- https://recursiveai.org/onemodel/docs/ops-norms/ -->

# Normalisation, softmax and reduction operators

Normative for the exact flavour; the fast flavour is the same unless a section says otherwise. Source: `src/OneModel/Kernels/Exact/NormKernels.cs`,
`Exact/ReduceKernels.cs`, `Kernels/Softmax.cs`, `src/OneModel/Graph/Operators/Norms/`. Operators and opsets served:

| Operator | Opsets | Notes |
|---|---|---|
| LayerNormalization | 17+ | float32; optional `Mean` and `InvStdDev` outputs |
| RMSNormalization | 23+ | float32 |
| InstanceNormalization | 6+ | float32 |
| GroupNormalization | 18-20 (scale and bias per group), 21+ (per channel) | float32 |
| Softmax, LogSoftmax, Hardmax | 1-12 (coerce to 2-D, axis default 1), 13+ (one axis, default -1) | float32 |
| ReduceSum | 1-12 (`axes` attribute), 13+ (`axes` input) | float32, float64, int32/64, uint32/64, int8, uint8 |
| ReduceMean, ReduceMax, ReduceMin, ReduceProd, ReduceL1, ReduceL2, ReduceSumSquare | 1-17 (attribute), 18+ (input) | same types; L2 float32 and float64 |
| ReduceLogSum, ReduceLogSumExp | 1-17, 18+ | float32 |
| ArgMax, ArgMin | 1+ (`select_last_index` from 12) | float32/64, int8-64, uint8-64 |
| CumSum | 11+ | float32, float64, int32/64, uint32/64 |

Not served: float16 and bfloat16 inputs, bool reductions (opset 20), `stash_type` other than float32 (the attribute is read and ignored: every
sum is accumulated in float32), and `ReduceLogSum*` on float64. The session reports these as `Unsupported` naming the operator.

## Exactness

All of these are exact kernels: only IEEE `+ - * /` and `sqrt`, no fused multiply-add, the orders below, and every NaN result rewritten to
`0x7FC00000` (also in fast mode: the kernels share one code path). The fast and exact flavours therefore give the same bits, on every
kernel path and at every thread count, except LogSoftmax and Softmax, whose fast flavour uses the fast exponential (see
`kernels-attention.md` and `exact-transcendentals.md`). The log in LogSoftmax, ReduceLogSum and ReduceLogSumExp, and the exponential in
ReduceLogSumExp, are the library's own (`exact-transcendentals.md`), evaluated one element at a time.

## The canonical sum

Every sum of `n` terms `t[0..n)` below (mean, variance, L1, L2, sum of squares, log-sum) is the sum of `RowReduce.Sum16`: 16 partial sums,
term `j` added to partial `j mod 16` in increasing `j` starting from `+0`, then `a[t] = P[t] + P[t+8]` (t < 8), `b[t] = a[t] + a[t+4]`
(t < 4), `c[t] = b[t] + b[t+2]` (t < 2), result `c[0] + c[1]`. Vector lanes hold different partials, so the lane width does not matter.

## LayerNormalization, RMSNormalization

The input is viewed as `rows x cols`: `cols` is the product of the dimensions from `axis` on (default -1; negative axes count from the end).
For each row, with `n = cols` converted to float32:

1. `mean = Sum16(x) / n` (RMSNormalization: `mean = 0`).
2. `v = Sum16((x[j] - mean)^2) / n`: the difference and the square are separate rounded operations.
3. `inv = 1 / sqrt(v + epsilon)` (RMSNormalization: `v` is the mean of squares).
4. `y[j] = (((x[j] - mean) * inv) * scale[j]) + bias[j]`, four separate operations; without `B` the last addition is skipped.

`Scale` and `B` broadcast to the normalized dimensions (right-aligned; missing leading dimensions are ones). When one has fewer elements than the
normalized extent it is expanded into a scratch array first. `Mean` and `InvStdDev` hold `mean` and `inv` per row, shaped like the input with
the normalized dimensions replaced by ones. Rows are processed in chunks of `max(1, 16384 / cols)` rows; a row is never split.

## InstanceNormalization, GroupNormalization

A row is one (sample, channel) plane, or one (sample, group) block of `C / num_groups` channels times the spatial positions. Step 1 to 4 are the
same with the scale and bias taken per row: InstanceNormalization and GroupNormalization (opsets 18-20) use one scalar scale and bias per
channel or group for the whole row; GroupNormalization at opset 21 and later has per-channel parameters, spread across the block. The
opset-18 definition indexed scale and bias by group and was corrected in 21; both are implemented as their opset defines them.

## Softmax, LogSoftmax, Hardmax

Up to opset 12 the input is flattened to two dimensions at `axis` (rows = product of the dimensions before it, columns = the rest) and each row
is normalised. From opset 13 the operator works along `axis` of the original shape (default -1). When the axis is not the last, the data is moved
to rows in a scratch copy and moved back. Rows are processed in chunks of `max(1, 16384 / cols)` rows.

- **Softmax**: `kernels-attention.md`. A row holding NaN or +inf, or consisting only of -inf, gives NaN throughout (as ONNX Runtime does).
- **LogSoftmax**: `d[j] = x[j] - max`, `l = Sum16(exp(d[j]))` (exponential for non-positive arguments), `y[j] = d[j] - log(l)`; the same NaN rows
  as Softmax; a `-inf` element in an otherwise normal row stays `-inf`.
- **Hardmax**: 1 at the first maximum of the row, 0 elsewhere; a row containing NaN is all zeros (the one probed case of ONNX Runtime).

## Reductions

**Axes.** Opsets before the change read `axes` from the attribute; later ones read the second input (int64), which may be computed at run
time (the node is then shaped at run time). Negative axes count from the end. An absent or empty `axes` reduces every axis, unless
`noop_with_empty_axes` is set: then no axis is reduced and each output is the operation applied to its single element. ONNX Runtime does it
this way (ReduceSumSquare squares, ReduceLogSum takes the log, ReduceL1 and ReduceL2 take the absolute value), so we do too. `keepdims`
(default 1) keeps reduced axes as extent 1.

**Geometry.** Dimensions of extent one are dropped and neighbouring dimensions that are both reduced or both kept are merged, leaving
alternating runs of kept and reduced extents. Two layouts follow from the shape and axes alone:

- *Column layout* (the last run is kept): the last run is `inner` contiguous outputs. Each output is accumulated sequentially over the reduced
  index tuples in row-major order, starting from the identity (`+0` for sums, `1` for products). Neighbouring outputs share a vector, so
  reducing a leading axis costs one pass over the data, not a strided gather. Work is split into tiles of 256 outputs.
- *Row layout* (the last run is reduced): each output reduces a sequence made of the reduced index tuples in row-major order, innermost run
  contiguous. Sums use the canonical sum over that sequence (element `q` of the sequence goes to partial `q mod 16`). Products multiply
  sequentially from `1`.

**Chunks.** A chunk covers a whole number of tiles (column layout) or outputs (row layout): `max(1, 32768 / work)` of them, where `work` is the
elements one unit reads. It depends on the geometry only. Every output is computed by one chunk, so no partial result is combined across chunks
and thread count cannot matter. (A reduction to a few outputs over a huge extent therefore runs on few threads; splitting it would need a
chunked combine in a fixed order, which can be added later without changing results at the sizes where it does not apply.)

**Terms and finishing.** Sum: `x`. Mean: `x`, then `sum / count` (`count` as float32; an empty reduction gives 0; integers divide, truncating
toward zero). L1: `|x|`. L2: `x*x`, then `sqrt`. SumSquare: `x*x`. LogSum: `x`, then the library's `log`. Prod: multiply. LogSumExp: `m` = the
maximum (NaN if any element is NaN), then `m + log(sum of exp(x - m))` in sequence order; `+-inf` maxima are returned as they are.

**Maximum and minimum and NaN.** The result is that of a sequential scan: the running value starts as the first element and is replaced only
by a strictly better one. NaN compares false, so it is ignored unless it is the first element, in which case the result is NaN. An empty
reduction gives `-inf` (max) or `+inf` (min). *ONNX Runtime's result for NaN inputs depends on its internal code path.* Probed with random integer-valued data with 20% NaN, 150 trials each: for reduced axes that are not innermost this rule matched every trial; for an innermost reduced axis it matched 150 of 150 trials at extents 3 and 300, 146 of 150 at extent 7, 68 of 150 at extent 50 and 0 of 150 at extent 6. So NaN in ReduceMax and ReduceMin is documented, not matched. The spec leaves NaN open.

**Other types.** Integer sums, products and sums of squares wrap around; L1 of the minimum value wraps. float64 follows the same order with
float64 arithmetic one output at a time (no vector kernel).

## ArgMax, ArgMin

Scan the axis from index 0; a later element wins when strictly greater (ArgMax) or smaller (ArgMin), or, with `select_last_index`, when greater
or equal. NaN never wins unless it is first. ONNX Runtime's behaviour with NaN depends on its code path (a 1-D input made a NaN win in the probe;
a 3x3 input did not), so NaN is documented, not matched; ties and ordinary values match. Output is int64; `keepdims` default 1. An axis of
extent 0 gives zeros. This runs on one thread.

## CumSum

Sequential along the axis in the direction of accumulation: inclusive `y[i] = y[i-1] + x[i]` with `y[0] = x[0]`; `exclusive` shifts so
`y[0] = 0` and `y[i] = y[i-1] + x[i-1]`; `reverse` runs from the last element. Floats are added in exactly that order. The axis input is read at run
time; an axis outside `[-rank, rank)` is an `InvalidInput` error. This runs on one thread.

## Conformance

Per-operator results of the `onnx/backend/test` node cases (ONNX 1.22.0 data, fast and exact mode) and of the randomized comparison with ONNX Runtime
1.31.1 (`tools/ops_norms.py`, scaled error at most 1e-4, integers exact) are recorded in the repository's conformance table when it is
published; the randomized cases cover every operator above at the opsets listed, with axis subsets, keepdims, odd widths and special values.


<!-- https://recursiveai.org/onemodel/docs/ops-conv/ -->

# Convolution, pooling, normalisation and resize operators

Behaviour of the operators of this family beyond what the ONNX operator text says: which cases are supported, how the open edges
were settled, and which bits each operator promises. The kernels' arithmetic is in `kernels-gemm.md` (Conv) and below. Edge
behaviour was settled by running ONNX Runtime 1.31.1 as a black box (`tools/ops_conv_oracle.py` records its outputs for randomised
cases; the probe inputs are quoted below), never by reading its code.

## Operators

| Operator | Opsets | Supported | Not supported yet | Exact mode |
|---|---|---|---|---|
| Conv | 1 and later | float32; 1 or 2 spatial axes; group, dilation, strides, pads, `auto_pad`; optional bias | 3 spatial axes; other types | yes |
| ConvTranspose | 1 and later | float32; 1 or 2 spatial axes; group, dilation, strides, pads, `output_padding`, `output_shape`, `auto_pad`; optional bias | 3 spatial axes | yes |
| MaxPool | 1 and later | float32, float64, int8, uint8; 1 to 3 spatial axes; `ceil_mode`, dilations, pads, `auto_pad`; the Indices output, either `storage_order` | | yes |
| AveragePool | 1 and later | float32; 1 to 3 spatial axes; `ceil_mode`, `count_include_pad`, dilations (opset 19), pads, `auto_pad` | other types | yes |
| GlobalAveragePool, GlobalMaxPool | 1 and later | float32; any rank of at least 2 | | yes |
| BatchNormalization | 7 and later | float32, inference only | `training_mode = 1`, the running-statistics outputs, `spatial = 0` | yes |
| Resize | 10 and later | float32; any rank; nearest, linear, cubic; every coordinate mode and nearest mode; `cubic_coeff_a`, `exclude_outside`, `extrapolation_value`, `axes`, `keep_aspect_ratio_policy`; scales or sizes, constant or computed | `antialias = 1` | yes |
| Upsample | 7 to 10 | float32; nearest, linear (`bilinear`) | | yes |

"Exact" means the operator follows `exact-mode.md`: every sum is in the order written here, nothing is fused, and NaN results are
rewritten as `0x7FC00000`. Pooling, BatchNormalization and Resize use only separately rounded IEEE operations, so their fast and exact
results are the same bits; Conv and ConvTranspose follow `kernels-gemm.md`.

## Output sizes

For one spatial axis with input `n`, effective kernel `e = dilation * (kernel - 1) + 1`, stride `s`, pads `b`, `a` (before, after):

* `auto_pad` NOTSET: `floor((n + b + a - e) / s) + 1`, or with `ceil_mode` `ceil((n + b + a - e) / s) + 1`. With `ceil_mode`, if
  `(out - 1) * s >= n + b` the last window would start entirely in the end padding, and `out` is reduced by one. A window that
  does not fit (`n + b + a < e`) is an error.
* `SAME_UPPER`, `SAME_LOWER`: `out = ceil(n / s)`, total padding `max(0, (out - 1) * s + e - n)`; the odd element goes after the input
  for UPPER and before it for LOWER. `VALID`: no padding.
* **Settled against ONNX Runtime (pooling, `SAME_*` with dilation):** ONNX Runtime computes the padding of a pooling window with the
  undilated kernel and then applies the dilated size formula, so its output size differs from `ceil(n / s)` and its padding from ours.
  We follow the operator text (the formula above). No reference model uses it; `ops_conv_oracle.py` leaves those combinations out,
  and also those where ONNX Runtime's size disagrees with the text (reported as a count when it runs).
* ConvTranspose: `s * (n - 1) + output_padding + e - b - a`. With `output_shape`, the total padding is the difference to the natural
  size and is split with the extra element **before** the output (the operator text's "otherwise" branch, confirmed by ONNX Runtime's
  values); a size larger than the kernel reaches is allowed and the surplus is zero (backend case `convtranspose_output_shape`). With `SAME_*` the output is `n * s` when the kernel reaches that far, else as large as it gets (padding 0), as ONNX Runtime
  does; `SAME_UPPER` puts the extra element after, `SAME_LOWER` before.

## Conv, ConvTranspose

* One-dimensional operators run as two-dimensional ones of height 1.
* Conv's bias is added after the taps, once per output (`acc + bias`), as a separate pass over the output; in exact mode NaN results
  are canonicalised. ConvTranspose adds its bias last as well.
* ConvTranspose is the matrix product `Wg' * Xg` (rows: output channel, kernel row, kernel column; columns: input position) followed
  by col2im. An output element receives the kernel taps that land on it in increasing (kernel row, kernel column) order, starting
  from +0; the sum over input channels is inside the matrix product (`kernels-gemm.md`). A constant weight is rearranged once per plan.

## Pooling

All loops run window taps in depth, height, width order, one scalar at a time. One, two or three spatial axes are lifted to three.

* **MaxPool and NaN (settled against ONNX Runtime; partly open).** Our rule is IEEE `maxNum`: NaN taps are ignored unless every tap
  inside the input is NaN, in which case the result is NaN. The first tap inside the input that is not NaN starts the maximum, a later
  tap replaces it only if strictly greater, so the first of equal taps wins and decides the index. `-inf` data gives `-inf`.
  ONNX Runtime has no single rule. Probes with one window of the values `1..k` and a NaN at each position in turn (54 probes over
  1-D windows of 2, 3 and 5 and 2-D windows 1x2, 1x3, 1x4, 2x2, 2x3, 3x3, 4x4): the result is NaN for exactly one position per
  window shape from 2x2 up (and for 1-D 5), at a position that changes with the shape, and never otherwise; `[nan, 1, 2, 3]` with
  kernel 1x2 and stride 2 gives `[nan, 3]` while `[nan, 2]` as a single window gives `2`. The output depends on the window's size
  and position, not on a rule we could state, so exact agreement on NaN inputs is not a goal. The rule above agreed with ONNX
  Runtime in 48 of the 54 probes (propagating NaN would agree in 6), and it is the same as oneocr's. Generated comparison
  cases therefore carry infinities and `-0` but no NaN for MaxPool.
* **GlobalMaxPool:** the maximum starting from the lowest float and replaced by strictly greater values, so NaNs are ignored and a
  plane of NaNs or of `-inf` gives `-3.4028235e38`. That is what ONNX Runtime 1.31.1 does on x64; on arm64 it propagates a NaN
  instead (the same generated case gave NaN there), so the NaN behaviour of its max operators is a property of the CPU it runs on.
* **MaxPool Indices:** the flat index of the chosen element over the whole input tensor (`((n * C + c) * D + d) * H * W + ...` for
  `storage_order` 0); for `storage_order` 1 each plane is flattened with the first spatial axis fastest (for 2-D: `c * plane + w * H + h`).
  Confirmed on a 2x2x3x3 input.
* **AveragePool divisor:** the window's taps inside the input, summed in order from +0 and divided (not multiplied by a reciprocal).
  With `count_include_pad` the divisor also counts taps in the declared padding, but never taps beyond it (the overhang that
  `ceil_mode` can create). Confirmed by ONNX Runtime on 5x5 inputs with kernel 3, stride 2, pad 1, `ceil_mode` and dilation 2.
* **GlobalAveragePool:** each plane's sum in index order from +0 divided by the plane size.
* ONNX Runtime refuses pads that are not smaller than the kernel; we do not (a window with no tap inside the input gives the type's
  lowest value for MaxPool and NaN for AveragePool).

## BatchNormalization

Per channel `a = scale / sqrt(var + epsilon)`, `b = bias - mean * a`, then `y = x * a + b`; each operation rounded on its own, no
fused multiply-add. This differs from `(x - mean) / sqrt(var + epsilon) * scale + bias` by a few rounding errors, within the 1e-4
scaled metric of `oracles.md`. The operator may run in place.

## Resize

Axes are resampled one at a time (shrinking axes first, then growing ones, each in axis order), through per-output tap tables; the
result of a separable interpolation equals the multi-dimensional one up to rounding. Coordinates are computed in float32:

* output size per axis: `floor(in * scale)` for scales, `size` for sizes; `keep_aspect_ratio_policy` `not_larger` / `not_smaller`
  takes the smaller / larger of `size / in` over the listed axes as the scale and rounds `in * scale` half away from zero.
* input coordinate `x` of output position `j` (scale `s`, `in`, `out`): `half_pixel` `(j + 0.5)/s - 0.5`; `pytorch_half_pixel` the same
  unless `out == 1` (then 0); `align_corners` `j * (in - 1) / (out - 1)` with the rounded-down output size (0 if `out == 1`). **Open:** the backend cases `downsample_scales_{linear,cubic}_align_corners` expect the unrounded size `in * s` in the divisor (scales 0.6 on 4 elements); ONNX Runtime 1.31.1 uses the output size, as here, and so do we, so those two cases fail by design; `asymmetric` `j / s`;
  `half_pixel_symmetric` `offset + (j + 0.5)/s - 0.5` with `offset = in/2 * (1 - out / (s * in))`; `tf_crop_and_resize` from `roi`.
  Opset 10 and Upsample are `asymmetric` with nearest mode `floor`.
* nearest: `floor`, `ceil`, `round_prefer_floor` (`x - floor(x) > 0.5` rounds up), `round_prefer_ceil` (`>= 0.5` rounds up); the index
  is clamped to `[0, in - 1]`. Nearest copies elements, so any value (NaN, infinities) moves unchanged.
* linear: `x` is clamped to `[0, in - 1]`, taps `floor(x)`, `floor(x) + 1` (clamped) with weights `1 - t`, `t`.
* cubic (Keys 1981): taps `floor(x) - 1 .. floor(x) + 2`; with `exclude_outside` taps outside the input get weight 0 and the others
  are divided by their sum, otherwise tap indices are clamped.
* `tf_crop_and_resize`: output positions whose coordinate lies outside `[0, in - 1]` on any axis are `extrapolation_value`.
* A tap with weight exactly zero is left out of the sum, so an exactly hit sample next to an infinity or NaN stays finite. ONNX
  Runtime's own results for interpolation near infinities or NaN vary from one case to the next (it gave `-inf` where multiplying by
  the zero weight gives NaN, and NaN where leaving the tap out gives a number), so generated linear and cubic comparison cases contain
  only finite data.
* Scales or sizes that are not constants make the node shape at run time; `roi` is read only for `tf_crop_and_resize`.
* An axis with scale exactly 1 and equal sizes is skipped (identity), except under `tf_crop_and_resize`.


<!-- https://recursiveai.org/onemodel/docs/ops-shape-indexing/ -->

# Shape and indexing operators

Notes on the operators that move, select or describe elements without computing on them: what each one does where the ONNX text leaves room, and how each choice was settled. Semantics come from ONNX's `Operators.md` and `Changelog.md`; open points were settled by running ONNX Runtime 1.31.1 as a black box on crafted inputs (`tools/ops_shape_indexing.py` generates the cases and records its outputs).

## 1. Common rules

- **Types.** Operators that only move elements accept every stored element type (a gather moves 4-byte elements the same way for float32, int32 and uint32). Operators that compare or count (TopK, NonZero, ScatterND reductions, Range, OneHot) list their types below.
- **Exact by construction.** None of these operators does arithmetic on floating-point values, so each is bit-identical in fast and exact mode, on every instruction set and thread count (the kernels are scalar span loops and `memmove`; none uses vector intrinsics). All are flagged as having an exact kernel. NaN payloads and signed zeros pass through untouched. Exceptions in a *value* sense: ScatterND `add` and `mul` do one float add or multiply per element (correctly rounded, so also identical everywhere).
- **Views.** Squeeze, Unsqueeze, Flatten, Reshape, Identity and Dropout (without a mask output) are views: the session builds them and no byte moves. Everything else that returns existing elements copies them.
- **Plan-time values.** Shape and Size are constants of the plan. A node whose shape depends on an input value (Slice's starts, Squeeze's axes, Expand's shape, Tile's repeats, Pad's pads, Range, OneHot's depth, TopK's k, ConstantOfShape's shape, Split's sizes) reads it at plan time when it is a constant, which is what turns `Shape → Gather → Unsqueeze → Concat → Reshape` chains into a literal target shape. When it is not a constant the node asks for run-time values and is shaped again in every run. Only NonZero is data dependent in the contract's sense (its output extent is a count of values).
- **Errors.** Shape and attribute problems are `SessionException(InvalidModel)` naming the node when the session builds the plan. An index outside its extent at run time is `SessionException(InvalidInput)`. Nothing indexes outside a buffer on bad input.
- **Threads.** The kernels are single-threaded: they are memory-bound copies and the result cannot depend on the thread count.

## 2. Operators

| Operator | Opsets | Kind | Notes |
|---|---|---|---|
| Shape | 1+ | plan constant | `start`/`end` (15+): negative values count from the end, both clamp to [0, rank]; start past end gives an empty vector. |
| Size | 1+ | plan constant | element count as an int64 scalar. |
| Squeeze | 1+ | view | axes: attribute before 13, optional input from 13, none = every extent-1 axis. An axis whose extent is not 1, or listed twice, is an error. |
| Unsqueeze | 1+ | view | axes index the *output*, negative allowed; duplicates are an error. Output rank limit 8. |
| Flatten | 1+ | view | `axis` in [-rank, rank]; axis 0 gives `[1, size]`. |
| Dropout | 1+ | view, or copy when a mask output is used | Inference only: output = input, mask (if used) all true. `ratio` is ignored. A `training_mode` that is true is `Unsupported`; one fed at run time is read at run time. |
| Transpose | 1+ | copy | `perm` default reverses the axes. Axes that stay adjacent in both layouts are merged; a real transpose runs in 32x32 tiles. |
| Concat | 1+ | copy | `axis` required from opset 4 (default 1 before). Empty inputs are allowed. |
| Split | 1+ | copy | Sizes from the `split` attribute (before 13) or input (13+). With no sizes: equal parts (the extent must divide) before opset 18; from 18, parts of `ceil(extent / n)` with the last part taking the rest (`num_outputs`, default the output count). |
| Slice | 1+ | copy | See 3. |
| Expand | 8+ | copy | Output = bidirectional broadcast of the input and the shape input (so an input extent may exceed a shape entry of 1). |
| Tile | 6+ | copy | A repeat of 0 gives an empty axis. |
| Gather | 1+ | copy | Indices int32 or int64, negative allowed (counted from the end). |
| GatherElements | 11+ | copy | Indices int32 or int64; the indices' extents may be smaller than the data's off the axis. |
| GatherND | 11+ | copy | `batch_dims` supported; indices int64. |
| ScatterND | 11+ | copy | Output = data, then the updates applied in index order (duplicates: the last write wins, or the reduction folds them in order). `reduction` none/add/mul/max/min on float32, float64, int32, int64. For max and min the result is `max(u, o)` / `min(u, o)` written as `u < o ? o : u` and `o < u ? o : u`, so a NaN in either operand makes the update win; this is what ONNX Runtime does. |
| Pad | 1+ | copy | See 4. |
| ConstantOfShape | 9+ | fill | Value attribute of one element of any stored type, default float32 0. A negative extent is an error. |
| Range | 11+ | generate | float32, float64, int16, int32, int64, and float16 / bfloat16 (computed in float32, rounded to nearest even). Count = max(ceil((limit - start) / delta), 0), with the quotient taken **in double precision** from the (float) values (settled by black box: `Range(0, 0.3, 0.1)` in float32 has 4 elements); element i = start + i*delta in the element type. Delta 0 and a non-finite count are errors. |
| OneHot | 9+ | generate | Indices of any numeric type (a NaN index selects nothing), depth any numeric scalar, values any stored type. An index outside [-depth, depth) gives an all-off row. Rank-0 indices work (ONNX Runtime refuses them). |
| Trilu | 14+ | copy | `k` is optional. Batched over leading axes. |
| NonZero | 9+ | data dependent | Output `[max(rank,1), count]`; -0.0 is zero, NaN is not. A scalar input counts as a 1-element vector (ONNX Runtime agrees: output `[1, count]`). |
| TopK | 1+ | copy | See 5. |

## 3. Slice

Starts, ends, axes and steps come from attributes (opset 1-9) or inputs (10+). Per sliced axis of extent n:

- A negative start or end adds n first. For a positive step both then clamp to [0, n]; for a negative step the start clamps to [0, n-1] and the end to [-1, n-1] (the end -1 means "through element 0"). The step is clamped to the int32 range first (a larger step selects at most one element anyway), so INT64_MAX steps work. Step 0 is an error. An axis of extent 0 gives an empty result.
- Count = `ceil((end - start) / step)` for a positive step, `ceil((start - end) / -step)` for a negative one, at least 0.
- **ONNX Runtime's reading of the end sentinels (black box).** With a negative step an end of exactly INT32_MAX or INT64_MAX means "to the beginning", the same as INT_MIN, not "clamped to n-1" (`x[INT64_MAX : INT32_MAX : -1]` of five elements gives all five, reversed; an end of 100 gives nothing). The spec's clamping alone would give an empty slice; models rely on the sentinel, so we follow ONNX Runtime. The 214 slice cases of the comparison suite, which include random start/end/step combinations with these sentinels, agree.
- Axes may be negative (11+) and must not repeat. Unlisted axes are taken whole.

## 4. Pad

`pads = [begin_0 .. begin_{k-1}, end_0 .. end_{k-1}]` for the listed axes (all axes unless an `axes` input is given, opset 18+); other axes get 0. Attributes before opset 11 (`paddings` in opset 1, `pads` in 2 to 10, plus `value`); `pads`, an optional `constant_value` and the optional `axes` are inputs from 11. A negative pad crops.

- **constant:** the output is filled with the value (default 0, any type) and the overlap with the input is copied.
- **edge:** an index outside the input reads the nearest edge element.
- **reflect:** mirrors without repeating the edge element. **Beyond one reflection we continue periodically** (period `2*(n-1)`; an extent of 1 repeats its only element). ONNX Runtime refuses a reflect pad of `n` or more on an axis, and refuses an axis of extent 1; we accept both. The accepted range agrees with ONNX Runtime wherever it runs.
- **wrap** (opset 19): periodic with period n.
- Non-constant modes on an empty axis that would need elements are an error.

## 5. TopK

`k` is an attribute before opset 10 and an input from then on (a constant k gives plan-time shapes; a computed k is read at run time). `largest` (default 1) picks the order; `sorted` is accepted and ignored (the output is always sorted, which is a valid reading of both values).

- **Ties** keep input order: of equal values the lower index comes first, in both directions. This matches ONNX Runtime on every tie case tried, for k up to 64 and axis extents up to 64.
- **NaN** counts as greater than every number (first when `largest`, last when not). The spec does not define it, and ONNX Runtime's result for NaN inputs differs between its own code paths (the same input gives different placements for k = 2 and k = extent), so the NaN cases of the oracle comparison are skipped. This is a deliberate, documented choice, not agreement.
- Selection per row: insertion into a sorted list of k for k <= 16, otherwise a stable merge sort of the row's indices. Either way the result is the stable order, so the paths agree.

## 6. Not covered here

Scatter and ScatterElements, the sequence operators (SplitToSequence and friends), Compress, Reshape before opset 5 (the shape was an attribute), and `Identity` on sequence types.


<!-- https://recursiveai.org/onemodel/docs/ops-recurrent/ -->

# Recurrent operators: RNN, GRU, LSTM

Semantics are those of `Operators.md` (opsets 7, 14, 22 are one implementation: the attributes and inputs did not change in a way
that needs a second one; `layout` arrived in opset 14). This note records what the specification leaves open, how each point was
settled, and the order of operations that makes exact mode bit-identical everywhere.

## Support

| | |
|---|---|
| Operators | `RNN`, `GRU`, `LSTM`, default domain, opset 7 and later |
| Element types | float32 (X, W, R, B, initial state, P, Y, Y_h, Y_c); `sequence_lens` is int32 |
| Directions | `forward`, `reverse`, `bidirectional` |
| `layout` | 0 (`[seq, batch, ...]`) and 1 (`[batch, seq, ...]`); state tensors `[dirs, batch, hidden]` or `[batch, dirs, hidden]` |
| Inputs | X, W, R, B, sequence_lens, initial_h; LSTM also initial_c and P (peepholes), in the specification's order; any optional one may be omitted |
| Outputs | Y, Y_h; LSTM also Y_c. Any may be unused |
| Attributes | `direction`, `hidden_size`, `layout`, `activations`, `activation_alpha`, `activation_beta`, `clip`; LSTM `input_forget`; GRU `linear_before_reset` |
| Not supported | float16 and double recurrent weights |
| Exact mode | supported for all three |

Gate order inside W, R and B follows the specification: LSTM `i, o, f, c`; GRU `z, r, h`; RNN one gate. `B` is `[dirs, 2 * gates * hidden]`:
the W biases then the R biases. `P` is `[dirs, 3 * hidden]` in the order `i, o, f`.

## Computation

For one direction, with `G` gates and `H` hidden units:

1. The input projections of **all** time steps are one matrix product, `XW = X · Wᵀ` of shape `[seq * batch, G * H]`, on the shared
   GEMM (accumulation order of `kernels-gemm.md`, one accumulator per output, increasing k from +0).
2. For step `s` the recurrent product `Rec = H · Rᵀ` (`[batch, G * H]`, also the shared GEMM) is formed for the whole batch. For GRU with
   `linear_before_reset = 0` only the `z` and `r` columns are formed at this point, and the candidate's product `(r ⊙ H) · Rhᵀ` is a second
   small product after `r` is known.
3. The gate arithmetic below runs per batch row.

Weights are transposed once per plan when they are constants; otherwise on each run into scratch.

**Pre-activation (all cells), in this order:** `pre = ((xw + rec) + Wb) + Rb`, each `+` one rounding, bias terms skipped when `B` is absent.
For LSTM the peephole term is added after that: `pre_i += (Pi * c)`, `pre_f += (Pf * c)` with the previous cell, and `pre_o += (Po * c)` with the
new cell. `clip`, when present and positive, then clamps the pre-activation to `[-clip, clip]` (a NaN passes through); it applies to every
gate's input, as the specification says ("applied to the input of activations").

**LSTM:** `i = f(pre_i)`, `fg = f(pre_f)` (or `1 - i` with `input_forget`), `ct = g(pre_c)`, then
`c = (fg * c_prev) + (i * ct)` as two products and one sum, `o = f(pre_o)`, `h = o * h_act(c)`.

**GRU:** `z = f(pre_z)`, `r = f(pre_r)`.
`linear_before_reset = 0`: `pre_h = ((xw_h + (r ⊙ H_prev) · Rhᵀ) + Wbh) + Rbh`.
`linear_before_reset = 1`: `pre_h = (xw_h + (r * (rec_h + Rbh))) + Wbh`.
`ht = g(pre_h)`, `H = ((1 - z) * ht) + (z * H_prev)`, again as separate roundings.

**RNN:** `H = f(pre)`.

No step fuses a multiply with an add. The only fused arithmetic in fast mode is inside the GEMM (`kernels-gemm.md`).

## Activations

Names are matched case-insensitively. `f`, `g`, `h` are the first, second and third entries of `activations` per direction (defaults:
LSTM `Sigmoid, Tanh, Tanh`; GRU `Sigmoid, Tanh`; RNN `Tanh`). For a bidirectional layer the list may name either one set (used for both
directions) or both sets in order forward, reverse.

Supported: `Sigmoid`, `Tanh`, `Relu`, `Affine`, `LeakyRelu`, `ThresholdedRelu`, `ScaledTanh`, `HardSigmoid`, `Elu`, `Softsign`, `Softplus`.
`Sigmoid`, `Tanh`, `Elu`, `ScaledTanh` and `Softplus` use the library's own polynomials (`exact-transcendentals.md`); the others are
compares, selects and IEEE operations. In exact mode every one is bit-identical on every machine.

`activation_alpha` and `activation_beta` are consumed in activation order, **one value per activation that takes the parameter**, with separate
cursors for alpha and beta (`Affine`, `ScaledTanh`, `HardSigmoid` take both; `LeakyRelu`, `ThresholdedRelu`, `Elu` take alpha). Missing values default
to the activation's own defaults (`LeakyRelu` 0.01, `ThresholdedRelu` 1, `Elu` 1, `HardSigmoid` 0.2 and 0.5, `Affine` and `ScaledTanh` 1 and 0).
Settled by black box: recorded ONNX Runtime outputs for a `HardSigmoid, LeakyRelu, ScaledTanh` LSTM with `activation_alpha = [0.3, 0.2, 1.2]` and
`activation_beta = [0.4, 0.7]` (separate cursors) match.

Definitions: `Affine(x) = alpha * x + beta`; `LeakyRelu(x) = x < 0 ? alpha * x : x`; `ThresholdedRelu(x) = x > alpha ? x : 0`;
`ScaledTanh(x) = alpha * tanh(beta * x)`; `HardSigmoid(x) = min(1, max(0, alpha * x + beta))`; `Elu(x) = x >= 0 ? x : alpha * (exp(x) - 1)`;
`Softsign(x) = x / (1 + |x|)`; `Softplus(x) = x > 20 ? x : log(1 + exp(x))`; `Relu` keeps a NaN.

## Choices the specification leaves open (settled by recorded ONNX Runtime output, tools/ops_recurrent_oracle.py)

- **`sequence_lens`.** Row `b` runs its first `len_b` steps only; in the reverse direction it starts at step `len_b - 1`. Y is zero at
  steps `t >= len_b`. `Y_h` and `Y_c` are the state after the row's last valid step. **A row of length 0 has `Y_h = Y_c = 0`** (not the initial state);
  the reference does this, and the specification is silent. A length outside `[0, seq]` is an `InvalidInput` error at run time.
- **`input_forget`.** Couples the gates as `f = 1 - i`; recorded outputs match.
- **`layout = 1`.** The recorded ONNX Runtime CPU kernels refuse `layout = 1`, so the layout-1 cases are the layout-0 case with every tensor transposed as
  `Operators.md` defines (X and the initial states batch-first; Y `[batch, seq, dirs, hidden]`; Y_h/Y_c `[batch, dirs, hidden]`).
- **Initial state when only `initial_c` or `P` is given.** Omitted inputs are zeros (no peepholes).
- **Zero-size.** `seq = 0` or `batch = 0` runs nothing; Y is empty and the final states are the initial states (or zeros).
- **`hidden_size`.** Optional; when present it must equal R's last dimension. Zero hidden size is an error.

## Exact mode

All three operators are exact for float32. The canonical order is the one above; the matrix products follow `kernels-gemm.md`; the activation
functions follow `exact-transcendentals.md`; every NaN in Y, Y_h and Y_c is written as `0x7FC00000`. The scalar gate loops are IEEE
operations in the written order, so they are identical on every path. Tests run each recorded case on every kernel path the machine supports and
with 1 and 3 threads and require equal hashes.

## Fast mode

Same order; the GEMM may fuse multiply-adds and the elementwise functions use the fast polynomials. Results do not depend on the thread count
(the GEMM's chunking is fixed).

## Limits

The per-direction scratch (`seq * batch * G * H` floats for the projections plus a few `batch * H` buffers and the transposed weights when not
constant) must stay below 2^30 floats; a larger layer is refused with `Unsupported` at plan time.


<!-- https://recursiveai.org/onemodel/docs/ctc/ -->

# CTC decoding

A helper for consumers of recognisers and speech models (not an ONNX operator): `RecursiveAI.OneModel.Tensors.Ctc`. It turns per-step class
scores into label sequences. Scores are row-major `[steps, classes]` (a batch is `[batch, steps, classes]`), either logits or natural-log
probabilities that already normalise over the classes of each step. The blank class is configurable (default 0). Every function is pure and
gives the same result on every machine: probabilities come from the exact softmax and exponential (`exact-transcendentals.md`), and the beam
search uses the double-precision exp and log defined below.

## Greedy (best path)

Graves et al. 2006, section 3.1.

1. At each step choose the class with the largest score, the **lowest index on ties**. A NaN is never chosen over a number.
2. Merge consecutive equal classes into runs and drop the blank. Two equal labels separated by a blank are two labels.
3. The probability of a step's class `p(t)` is
   - logits: `exp(z_best - max) / sum_j exp(z_j - max)`, taken from the exact softmax kernel (`kernels-attention.md`); it equals `1 / sum_j exp(z_j - z_best)`;
   - log-probabilities: `exp(value)` with the exact exponential, capped at 1.

**Confidences.** A *run* is the steps of one emitted label; its confidence is the **largest `p(t)` of its class over the run**. The sequence confidence
is the **mean of the run confidences** in order (float32 sum, then divide by the run count); 0 when nothing is emitted. This is the usual
line-recogniser convention: the mean over emitted characters of each character's largest softmax probability. Per-step
confidence (`stepProbabilities`) is `p(t)` of the step's best class, blank included.

Each run also carries its first and last step, which consumers use for approximate alignment (a character occupies `[first, last]`).

The span entry point allocates nothing in steady state (it borrows a scratch row from the shared array pool). Output room: `steps` runs.

## Prefix beam search

Graves et al. 2006; Hannun et al. 2014, Algorithm 1. No language model.

State: a set of at most `beamWidth` prefixes, each with `pb` and `pnb`, the log probabilities of all alignments of the steps so far that give the prefix and
end in blank / in a non-blank. Start: the empty prefix with `pb = 0`, `pnb = -inf`.

For each step `t`, with `lp(j)` the log probability of class `j`, and for each prefix `p` with last label `c` (none for the empty prefix) and
`total = logadd(pb, pnb)`:

- `p` gets `pb += total + lp(blank)`;
- if `c` exists: `p` gets `pnb += pnb + lp(c)` (the run of `c` continues);
- for every class `j` other than the blank, the prefix `p·j` gets `pnb += (j == c ? pb : total) + lp(j)`; contributions equal to probability 0 are not added.

Contributions to one prefix are combined with `logadd`; new prefixes are listed in creation order. After each step the prefixes are ranked by
`logadd(pb, pnb)`, best first, equal scores in creation order, and the first `beamWidth` are kept. After the last step the best `topN` are returned,
each with `LogProbability = logadd(pb, pnb)` and `Confidence = exp(LogProbability / n)` for `n` labels (the per-label geometric mean; 0 for the empty
text). With a beam at least as wide as the number of distinct prefixes the search is exact: the best text over the sum of all alignments.

`lp(j)` for logits is `(z_j - max) - ln(S)` with `S = sum_i exp(z_i - max)`, accumulated in increasing `i` order in double precision (the float values are widened
first, so the subtraction is exact); for log-probabilities it is the given value.

Limits: `beamWidth` in `[1, 4096]`; `steps * beamWidth * (classes - 1) + 1` at most 2^24 (the most prefixes the search can create); NaN scores are refused.
The search allocates (a prefix tree and the result); it is a decoding step, not a per-sample kernel.

### Deterministic exp and ln (double)

Only `+ - * /`, comparisons and exact `floor`, in the order written; no fused multiply-add. Relative error below 1e-15 on the ranges used.

- `exp(x)`: `x <= -700` (or NaN) gives 0, `x > 700` is clamped to 700; `k = floor(x * 1.4426950408889634 + 0.5)`;
  `r = (x - k * 6.93147180369123816490e-01) - k * 1.90821492927058770002e-10`; the degree-13 Taylor polynomial of exp in Horner form with coefficients
  `1/n!` (`n = 13 … 0`), each step `p = p * r + c`; the result is `p * 2^k` built from the exponent bits.
- `ln(x)`: `x = m * 2^e` with `m` in `[1, sqrt 2)` (halve `m` and add 1 to `e` when it is above `sqrt 2`; subnormals are first scaled by 2^54);
  `s = (m - 1) / (m + 1)`, `z = s * s`, `q` the Horner form of `1 + z/3 + z²/5 + … + z⁹/19` (coefficients `1/(2j+1)`), result `e * 0.6931471805599453 + 2 * s * q`.
  `ln 0 = -inf`.
- `logadd(a, b)`: with `a >= b`, `a` if `b` is minus infinity or `b - a < -40`, else `a + ln(1 + exp(b - a))`.

## Checks

`tools/ctc_reference.py` is a straightforward double-precision Python implementation (numpy only): greedy, a dictionary-based prefix beam search, and for
small sizes the exact best text by enumerating every alignment. `testdata/ctc/cases.json` holds its output for 40 seeded random cases; the tests require the
same labels, run and step confidences within 2e-6, beam log probabilities within 1e-9, and the wide beam equal to the exhaustive result.


<!-- https://recursiveai.org/onemodel/docs/tensors/ -->

# Tensors and memory

Normative description of the tensor layer: element types, shapes and strides, views, layouts, buffer ownership, version counters,
the backward record, the kernel seam and the numeric conversions. Source: `src/OneModel/Tensors/`. A port to another language
must reproduce the rules in sections 2, 3, 4, 8 and 10 bit for bit; the rest is behaviour a port should match.

## 1. Model in one paragraph

A **buffer** is a 64-byte-aligned block of memory with a length in bytes, a generation number and a version counter. A **tensor** is
a value (no heap object) holding a buffer reference, the buffer generation it was created in, an element type, a logical shape,
strides, and an offset. Every view operation makes a new tensor value over the same buffer. Nothing is copied unless asked.

## 2. Element types

The numeric value of `DType` is the ONNX `TensorProto.DataType` code, so no mapping table exists.

| DType | Code | Bytes | Meaning |
|---|---|---|---|
| F32 | 1 | 4 | IEEE 754 binary32 |
| U8 | 2 | 1 | unsigned |
| I8 | 3 | 1 | signed, two's complement |
| U16 | 4 | 2 | unsigned |
| I16 | 5 | 2 | signed |
| I32 | 6 | 4 | signed |
| I64 | 7 | 8 | signed |
| Bool | 9 | 1 | 0 or 1 |
| F16 | 10 | 2 | IEEE 754 binary16 |
| F64 | 11 | 8 | IEEE 754 binary64 |
| U32 | 12 | 4 | unsigned |
| U64 | 13 | 8 | unsigned |
| BF16 | 16 | 2 | bfloat16 |

All multi-byte values are stored little-endian. Other ONNX codes (STRING 8, COMPLEX64/128 14–15, FLOAT8 variants 17–20, 4-bit and
2-bit integers 21–22 and 25–26, FLOAT4E2M1 23, FLOAT8E8M0 24, FLOAT6 27–28) are named by `DTypes.OnnxName` and rejected by
`DTypes.FromOnnx` with a message that names the type. Only floating-point tensors can require gradients.

## 3. Shapes, strides, limits

- Rank 0 to 8. Rank 0 is a scalar with one element.
- Each extent is a non-negative 32-bit integer. The element count (the product of extents) is at most 2^31−1, and each **prefix**
  product (left to right) must also fit, so `[100000, 100000, 0]` is rejected.
- A buffer holds at most 2^31−1 bytes, so `count × element size` must fit. Violations throw `TensorShapeException` before anything is
  allocated.
- Strides are signed 32-bit integers counting **elements** (not bytes). Zero means broadcast; negative means reversed.
- Element `(i0, …, ik)` lives at element offset `offset + Σ i_d · stride_d`. A view is valid only if, for a non-empty tensor, the
  lowest reachable offset (`offset + Σ over negative strides of (extent−1)·stride`) is ≥ 0 and the highest is below the buffer's
  element count. The check runs when the view is made.
- **Contiguous** means: the stride of every dimension with extent ≠ 1 equals the product of the extents to its right (empty tensors
  are contiguous). Strides of extent-1 dimensions are ignored. `Contiguous()` returns the tensor itself when contiguous, else a copy
  in row-major order.
- Broadcasting (`Shape.Broadcast`) is numpy's multidirectional rule: right-align, each pair of extents equal or one of them 1. The
  result must also satisfy the size limit.

## 4. Views (no data moves)

| Operation | Result |
|---|---|
| `Permute(p)` | dimension i of the result is dimension p[i] of the source; shape and strides permuted |
| `Transpose(a, b)`, `T()` | swap two dimensions |
| `Slice(axis, start, end, step)` | `0 ≤ start ≤ end ≤ size`, `step ≥ 1`; extent `ceil((end−start)/step)`; offset += `start·stride` (only if the result is non-empty); stride ×= step |
| `Select(axis, i)` | removes the axis; offset += `i·stride` |
| `Flip(axis)` | stride negated; offset += `(extent−1)·stride` (if extent > 0) |
| `Unsqueeze(a)` | inserts an extent-1 dimension (stride = `extent·stride` of the dimension it precedes, or 1 at the end) |
| `Squeeze(a)` / `Squeeze()` | removes one extent-1 dimension / all of them |
| `Expand(shape)` | right-aligned; extent 1 or a missing leading dimension gets stride 0 and the target's extent |
| `TryReshape(shape)` | see below |

**Reshape viewability.** Drop source dimensions of extent 1. Merge neighbouring source dimensions into a *run* while
`stride[i] == stride[i+1] · extent[i+1]`. Each run has a length (product of its extents) and the stride of its innermost
dimension. The new shape is viewable iff its dimensions, taken left to right, partition into groups whose extent products equal the
run lengths in order, followed only by extent-1 dimensions. Inside a group, strides are contiguous starting from the run's inner
stride. Empty tensors are always viewable with contiguous strides. `Reshape` falls back to `Contiguous()` + view when
`TryReshape` fails; row-major element order is the same either way. `Reshape(-1, …)` infers one extent.

## 5. Layouts

The logical shape of an image tensor is always `(N, C, H, W)`. Layout says where the elements live.

| Kind | Storage | How it is expressed |
|---|---|---|
| Nchw | row-major N, C, H, W | contiguous strides |
| Nhwc | row-major N, H, W, C | strides `(H·W·C, 1, W·C, C)` over the NCHW logical shape |
| NchwC(B) | `[N][ceil(C/B)][H][W][B]`, channel `c` at block `c/B`, lane `c%B`; lanes past `C` in the last block are **zero** | not strided: an opaque tensor flagged with block size B (1…256) |

`Layout` of a plain rank-4 tensor is derived from its strides (extent-1 dimensions ignored; when both patterns match, Nchw wins).
Other ranks and other strides report Strided. `ToLayout` converts between the three; it is a pure data movement (values and logical
shape are unchanged). NCHW and NHWC conversions are differentiable (backward is the identity on the logical tensor); blocked tensors
are kernel-internal: only `ToLayout`, `Data<T>()` (the physical span of `ceil(C/B)·N·H·W·B` elements) and metadata work on them, and
they cannot require gradients. Further blocked families (for convolution weights) are added the same way, as a flagged kind.

## 6. Buffers and the pool

- Every buffer starts on a 64-byte boundary.
- **Pool** (`BufferPool`). A request of `n` bytes (0 ≤ n ≤ 2^31−1) is served from a size class. Class size for `n ≤ 256`:
  `max(64, n rounded up to a multiple of 64)`. For `n > 256`: let `b` be the largest power of two below `n` (`b = 2^floor(log2(n−1))`),
  `step = b/4`, class = `b + ceil((n − b)/step)·step`. So classes are `b·{1.25, 1.5, 1.75, 2}`; waste is below 25 %. Released
  buffers wait in a per-class free list; the next request of the class takes one. Idle memory is capped (`MaxIdleBytes`, default
  2 GiB); beyond the cap a released block is freed. Contents of a rented buffer are unspecified unless zeroing was requested.
  Nothing in the pool depends on thread count or timing; it is safe to use from many threads.
- **Arena** (`Arena`). Bump allocation, 64-byte aligned, for the state of one run. Chunks (default 1 MiB, or the request if larger)
  come from a pool. `Reset()` reclaims everything; every tensor allocated since is then stale. Not thread-safe: one arena per
  concurrent run. After a warm-up run, a run with the same allocation sequence allocates no managed or native memory.
- **Lifetime.** `Tensor.Dispose()` returns the buffer for the tensor and all its views and copies of the handle. A buffer
  carries a generation; it increases when the buffer is returned or its arena reset. Every access compares the tensor's remembered
  generation to the buffer's and throws `StaleTensorException` on mismatch, so a tensor can never read memory re-rented to someone
  else, and releasing twice throws. A leaked (never disposed) pooled buffer is freed by the GC's finalizer.
- **`CHECKED` builds.** New and returned memory is filled with 0xFF (a NaN for floats, −1 for integers), so reads of
  unwritten or released memory stand out. 64 canary bytes (0xA5) follow every buffer's length and are verified when it is released
  or its arena reset; a kernel that wrote past the end makes release throw.

## 7. Version counters

Every buffer has a 64-bit version, 0 at rent. All views share it. It increases by one on: `MutableData`, `MutableStorage`, `Set`,
`Fill`, `CopyFrom`. (Kernels that write an output through a writable span obtain it through these accessors.) A write through a span
obtained earlier is not tracked. `SavedTensor` remembers the version at save time and `Unpack()` throws `TensorVersionException`
if it differs, so an in-place update to anything the backward pass needs is caught at backward time rather than yielding a wrong
gradient.

## 8. Gradient tracking and the backward record

- `WithRequiresGrad()` returns a handle that is a gradient leaf. Because a tensor is a value, the flag is on the returned handle.
- An operation records only if `GradMode.IsEnabled` (per thread; `GradMode.NoGrad()` scopes it off) and at least one input requires
  gradients. Otherwise it does no extra work and allocates nothing; for example, `MaxPool2d` then does not compute argmax.
- A `BackwardRecord` holds: the op (`OpKind`), per input the autograd node (null if that input needs no gradient), shape and type;
  positional `Saved` slots (tensors that share storage with the original, with the version at save time; a slot is empty when the
  operation did not need it); `Ints` and `Floats` attribute vectors. The result tensor points to the record (`GradFn`).
  Tensors the record created itself (index tensors, intermediates) are *owned* and freed by `Release()`.
- In-place `CopyFrom` on a tensor that requires gradients (while recording is on) throws.

| Op | Saved slots | Ints | Backward uses |
|---|---|---|---|
| Add, Sub | none | | gradient summed to each input shape (Sub negates b's) |
| Mul | 0 = a if b needs grad; 1 = b if a needs grad | | da = g·b, db = g·a, each summed to its input shape |
| Div | 0 = a if b needs grad; 1 = b | | da = g/b; db = −g·a/b² |
| Neg | none | | −g |
| Relu | 0 = output | | g where output > 0 |
| MatMul | 0 = a if b needs grad; 1 = b if a needs grad | | dA = g·bᵀ, dB = aᵀ·g, the transposes being strided views of the saved tensors |
| MaxPool2d | 0 = Int32 flat indices of the winners (logical row-major NCHW position in the input) | kernelH, kernelW, strideH, strideW, padH, padW | each output gradient is added at its index; input shape from InputShape(0) |
| Cast | none | | cast g back to InputDType(0) |
| Slice | none | axis, start, end, step | zeros of the input shape with g written into the slice |
| Permute | none | the permutation | inverse permutation |
| Reshape (also Squeeze, Unsqueeze, Contiguous, NHWC conversion, and Reshape-with-copy) | none | | reshape g to InputShape(0) |
| Expand | none | | sum g down to InputShape(0) |
| Flip | none | axis | flip g |
| Select | none | axis, index | zeros of the input shape with g written at the index |

Conventions for operations added later, fixed now so kernels and the engine agree: *normalization* saves the normalized rows and
the inverse standard deviation as owned slots plus the scale input; *GELU* saves the input and the tanh of its inner term;
*softmax* saves its output and keeps the scale in `Floats`; *convolution* saves the input and the weights (or the unfolded input
as an owned slot) and keeps kernel, stride, padding and padded K in `Ints`; *embedding lookup* saves the index tensor and keeps the
skip id in `Ints`, the table shape coming from InputShape(0); fused ReLU+pooling saves Int32 indices with −1 meaning "no positive
element, no gradient".

## 9. The kernel seam

`ITensorKernels` is the one place eager operations call kernels: `Binary`, `Unary`, `MatMul`, `MaxPool2d`. Inputs are float32
tensors with arbitrary strides (the caller expands broadcasts to stride 0 first, so all operands of `Binary` have one shape); the
output is allocated by the caller. An implementation is selected once, at startup or session creation (`KernelSet.SetDefault`, or
the `kernels` argument), from the machine's instruction sets and the fast-or-exact choice. `ReferenceKernels` defines the meaning:
scalar loops, `double` accumulation in `MatMul`, no threads. Pooling semantics: the window starts at `y = oy·stride − pad` and
includes only in-bounds positions (padding never wins); the first element wins ties; a NaN in the window wins, the first NaN is
reported; output extent `floor((size + 2·pad − kernel)/stride) + 1`, with `pad < kernel`.

## 10. Numeric conversions

**float → binary16** (round to nearest, ties to even, IEEE 754-2008 §4.3.1): with `a` the bits of |x|:
`a > 0x7F800000` (NaN): sign, `0x7E00`, top nine payload bits (`(a >> 13) & 0x1FF`); `a = 0x7F800000` or `a ≥ 0x477FF000`
(|x| ≥ 65520): signed infinity; `a ≥ 0x38800000` (normal): `m = a − (112 << 23)`, result bits `(m + 0xFFF + ((m >> 13) & 1)) >> 13`
(a carry into the exponent is correct); `a ≤ 0x33000000` (|x| ≤ 2^−25): signed zero; otherwise subnormal: with `e = a >> 23`,
`mant = (a & 0x7FFFFF) | 0x800000`, `shift = 126 − e`, `r = mant >> shift`, round up when the remainder exceeds half, or equals
half and `r` is odd.
**binary16 → float** is exact: subnormals as `man · 2^−24`, NaN keeps its payload shifted left 13 with the quiet bit forced.
**float → bfloat16**: NaN → `(bits >> 16) | 0x40`; otherwise `(bits + 0x7FFF + ((bits >> 16) & 1)) >> 16`. **bfloat16 → float**:
`bits << 16`, quiet bit forced for NaN. Both are verified over all 65536 half and bfloat16 patterns and 2 million random and
tie-constructed floats against an independent nearest-neighbour reference.

**Cast** (reference rules): float → float by the conversions above (float64 → 16-bit goes through float32 and so rounds twice);
integer or bool → float rounds to nearest even; float → integer truncates toward zero, saturates to the target range, NaN → 0;
integer → integer wraps modulo 2^bits; any → bool is "≠ 0" (NaN is true); bool → number is 0 or 1. Where ONNX leaves
out-of-range behaviour open, these rules are provisional and are checked against ONNX Runtime's output by the operator work.


<!-- https://recursiveai.org/onemodel/docs/kernels-gemm/ -->

# Matrix product and convolution kernels

fp32, row-major, `C = A * B` with `A` m x k, `B` k x n and row strides `lda`, `ldb`, `ldc`; convolution in NCHW. Both modes
exist for every kernel; the exact mode's definition is in `exact-mode.md`.

## Structure (Goto and van de Geijn 2008, "Anatomy of high-performance matrix multiplication", section 4)

1. **Pack B** once per call, for all of `k` and `n`: for each block of `KC = 256` rows of B, strips of `NR` columns, element
   `(p, j)` of a strip at `p*NR + j`; a ragged last strip is padded with +0. Work is split into chunks of 8 strips per k-block.
2. **Tiles.** C is cut into `MC x NC` tiles (the constants below). A tile is one chunk of work. For each k-block the tile packs its
   `MC x KC` block of A into strips of `MR` rows (element `(p, r)` at `p*MR + r`, padded with +0), then for each `NR`-wide strip of
   B it runs the microkernel down the A strips, so the B strip (`KC x NR x 4` bytes, 16 KB for AVX2) stays in the first-level
   cache while the A block (about 100 KB) streams from the second level.
3. **Microkernel.** An `MR x NR` tile of accumulators in registers; each step of k loads one A column of `MR` values and one
   B row of `NR` values and updates every accumulator. A later k-block starts from the C values the earlier blocks stored; the
   first block starts from +0. Ragged edge tiles run the full microkernel on a scratch tile and copy out the valid part.

| Path | MR x NR | MC x NC | Accumulators |
|---|---|---|---|
| AVX2 + FMA | 6 x 16 | 96 x 128 | 12 of 16 ymm |
| AVX-512 | 12 x 32 | 96 x 128 | 24 of 32 zmm |
| NEON (arm64) | 8 x 12 | 96 x 120 | 24 of 32 q (multiply by element) |
| 4-lane `Vector128` (SSE2, WebAssembly SIMD) | 4 x 8 | 64 x 128 | 8 |
| Scalar | 4 x 4 | 64 x 64 | 16 |

The scalar and 4-lane paths run wherever .NET runs; the 4-lane path is what browser WebAssembly uses (the JIT lowers
`Vector128` to WebAssembly SIMD).

## Convolution

* **Regular and grouped:** im2col (Chellapilla, Puri and Simard 2006): per image and group, the input patches become the columns
  of a `(Cin/G*KH*KW) x (OutH*OutW)` matrix, rows ordered channel, kernel row, kernel column; the group's weights are A and
  the group's output channels are C. Rows of the im2col matrix are built in chunks of 16. A 1x1, stride-1, unpadded
  convolution uses the input directly as B.
* **Depthwise** (`Groups == Cin`, any channel multiplier): a direct kernel, described in [kernels-depthwise.md](https://recursiveai.org/onemodel/docs/kernels-depthwise/).

## Fixed chunks and thread independence

Chunk sizes (8 strips of B per packing chunk, one `MC x NC` tile per compute chunk, 16 im2col rows, planes per depthwise chunk from the plane size) are
constants of the kernel and the shape. They are never derived from the number of threads or processors. Every output is computed
by exactly one chunk, with the whole of `k` in one order, so there are no partial results to combine and a result cannot depend on
which thread ran a chunk or on how many threads there were. `k` is never split across threads. A fixed-chunk parallel-for
(`Threading/ParallelFor.cs`) hands out chunk indices through an atomic counter; if the pool is busy serving another caller the
chunks run on the calling thread, with the same result.

How many threads a call uses is a separate matter: the caller passes an estimate of the work (in nanoseconds on one thread), and the pool
wakes at most one thread per 80 000 ns of it, so a small matrix product or convolution stays on the calling thread instead of paying
for wake-ups. The estimates are about 40 multiply-adds per nanosecond for the matrix product tiles, half a nanosecond per element
packed or copied, and a fixed rate per tap for depthwise. The number of threads only decides who runs a chunk, never what a chunk
computes, so it cannot change a result.

## Fast mode

Fast mode uses fused multiply-add where the hardware has it (AVX2+FMA, AVX-512, NEON) and `acc + a*b` elsewhere. The order of
additions is the same as exact mode's: one accumulator per output, increasing k, from +0, so the fused products are summed
sequentially. Consequences measured here: the output does not depend on the thread count, the k-block size, the tile shape or the
lane width; every fused path (scalar `MathF.FusedMultiplyAdd`, AVX2, AVX-512, NEON) produced the same bits on every size tested.
This is a property of the current kernels, not a promise: fast mode is only promised to be repeatable on one machine and path.
NaN payloads are whatever the hardware produced.

**Accuracy.** For a sum of `k` products accumulated sequentially in binary32 with unit roundoff `u = 2^-24`,
`|c - exact| <= k * u * sum_p |a_p * b_p|` (Higham, *Accuracy and Stability of Numerical Algorithms*, section 3.1). The tests check
fast outputs against a double-precision reference with exactly this bound plus 1e-30 for the subnormal range.

## Measuring exact against fast

`bench/Micro` command `spike table` (script: `tools/spike-x64.sh`): for each shape, each ISA path the CPU has, 1 thread and all
threads, it times fast and exact with the same inputs; a measurement is the median over three repetitions of the mean time of
calls run for at least 80 ms after two warm-up calls. Buffers are allocated once, outside the timing; packing and im2col are
inside it. "% of peak" divides by the nominal fused-multiply-add rate: `GHz x ports x lanes x 2` per core
(2 ports of 8 lanes for AVX2, 1 or 2 ports of 16 lanes for AVX-512 depending on the core, 4 pipes of 4 lanes for the Apple M2),
times the number of physical cores in use; it is a ceiling for orientation, not a measurement of the part. Model rows sum, over
the distinct GEMM and convolution shapes of one model, `count x time`, and divide the exact sum by the fast sum; they are
weighted by each shape's measured time (so by operation count and efficiency), and include only the matrix products and convolutions.
Shapes: BERT-base and MiniLM-L6 at 32 x 128 rows (and BERT at 128 rows), the Whisper-small encoder at 1500 frames, ResNet-50 at
224 x 224 batch 1, MobileNetV3-large at 224 x 224 batch 1 (squeeze-excite and classifier layers left out; attention
products are in `kernels-attention.md`).


<!-- https://recursiveai.org/onemodel/docs/kernels-attention/ -->

# Fused attention and softmax: fast and exact

Normative for the exact flavour. Source: `src/OneModel/Kernels/Attention.cs`, `Softmax.cs`, `Exact/Transcendental*.cs`.

## What is computed

For every (batch b, head h, query row i): `O[i] = softmax(scale * Q[i] K^T + mask[i]) V`, all fp32. Q is `[qLen, d]`, K and V `[kLen, d]`,
the optional additive mask `[qLen, kLen]` (strides are free; a stride of 0 broadcasts). `d` must be even.

## Work items and constants

A work item is one (b, h, block of `QB = 32` query rows). Items are independent and write disjoint output rows, so any distribution
of items over threads gives the same bits. `KB = 64` keys per block. `QB` and `KB` are part of the exact definition (they fix where
the online softmax rescales). They are constants of the kernel, never derived from the thread count or the CPU.

## The exact definition

For one query row, state `m = -inf`, `l = +0`, `acc[0..d) = +0`. For each key block (keys `kb .. kb+n`, `n <= 64`, in increasing order):

1. **Scores.** `dot[j] = (((+0 + q[0]*k[j][0]) + q[1]*k[j][1]) + ...) + q[d-1]*k[j][d-1]`: one accumulator, increasing index,
   product rounded, then sum rounded. `s[j] = mask[j] + dot[j]*scale`, with `mask = +0` when there is no mask. Lanes of a vector hold
   different `j`, so the lane width does not matter.
2. **Maximum.** `bm = max over j of s[j]`, ignoring NaN (`-inf` if none). `mNew = max(m, bm)`.
   `shift = (mNew == -inf) ? 0 : mNew`. Max is exact, so order is irrelevant.
3. **Weights.** `p[j] = expNonPositive(s[j] - shift)` (see `exact-transcendentals.md`).
4. **Block sum.** `bs` = the 16-partial-sum order: partial `t` adds `p[j]` for `j mod 16 = t` in increasing `j`, starting from `+0`
   (padding positions contribute `+0`, which changes nothing); then `a[t] = P[t] + P[t+8]` (t < 8), `b[t] = a[t] + a[t+4]` (t < 4),
   `c[t] = b[t] + b[t+2]` (t < 2), `bs = c[0] + c[1]`.
5. **Rescale.** `alpha = 0` if `m == -inf`, `1` if `m == shift`, else `expNonPositive(m - shift)`. `l = l*alpha + bs`.
   `acc[x] = acc[x]*alpha` for every x (skipped when `alpha == 1`, which changes nothing). `m = mNew`.
6. **Values.** For `j` in increasing order within the block: `acc[x] = acc[x] + p[j]*v[j][x]` (product rounded, then sum rounded).

After the last block `out[x] = acc[x] / l`. Any NaN in the output becomes `0x7FC00000`. A query row whose scores are all `-inf`
(fully masked) gives `l = 0` and therefore NaN in every output of the row; a row with a NaN score or a `+inf` score also gives
all-NaN. A fully masked *first block* followed by visible keys is fine (`alpha = 0`, `shift = 0`).

The fast flavour runs the same code with fused `acc + a*b` (FMA on x64 with FMA3, AVX-512 or Arm FMLA), `FastMath.Exp`,
`out[x] = acc[x] * (1/l)`, and is exactly as deterministic: same answer on the same machine for any thread count and any run. It differs from
the exact flavour in the last bits only.

## Softmax

`Softmax.Rows`: `m = max(x)` (NaN ignored), `e[j] = expNonPositive(x[j] - m)`, `l` by the 16-partial-sum order above over the whole row,
`y[j] = e[j] / l` (fast: `e[j] * (1/l)`); NaN becomes `0x7FC00000`.

## Layout used by the code (not part of the definition)

Keys of a block are packed transposed (`d x 64`, zero padded) so a micro-tile of 4 query rows x 2 vectors of keys streams contiguous
lanes (Goto & van de Geijn 2008, packing); `P V` uses a 4 x 2-vector tile over the head dimension. Query blocks shorter than a multiple of 4 rows
compute duplicate rows that are discarded. When `d` is not a multiple of twice the lane count the kernel narrows the lane width.
Vector loads and stores go through `IVecOps.Load/Store`; the Checked build asserts their range.

## Accuracy

Versus a double-precision naive reference, max absolute error on outputs of magnitude up to about 3 (inputs q, k ~ N(0, 1.5^2)):
fast and exact are both between 1.1e-6 and 3.4e-6, set by the fp32 accumulation of the scores (a logit error of ~1e-6 is a
relative weight error of ~1e-6). Tests use a bound of 4e-6.

## Cross-platform check

`AttentionTests.ExactOutputsMatchTheGoldenHashes` fixes SHA-256 hashes of exact attention (with and without mask, d = 32 and 64, 1500 keys) and
softmax outputs. They match at scalar, 128-, 256- and 512-bit lanes, 1 or many threads, on x64 with `DOTNET_EnableHWIntrinsic=0`, on arm64 (M2, NEON) and in a
NativeAOT build.

## Exact versus fast, measured (indicative: the shared worker carried 7 to 10 other busy threads, so these are not bar-grade)

Batch 1, fp32, medians of 3, milliseconds, AMD Ryzen 9 7950X3D (Zen 4, 32 logical cores), .NET 10.0.11. "32" is `Parallel.For` over the work items on 32 threads (bench code only).

| shape | lanes | threads | fast | exact | exact/fast |
|---|---|---|---|---|---|
| BERT-base seq 128 | AVX2 (256) | 1 / 32 | 0.964 / 0.092 | 1.410 / 0.115 | 1.46 / 1.25 |
| BERT-base seq 128 | AVX-512 | 1 / 32 | 0.741 / 0.078 | 1.435 / 0.093 | 1.93 / 1.19 |
| BERT-base seq 512 | AVX2 | 1 / 32 | 12.66 / 1.52 | 14.31 / 1.73 | 1.13 / 1.13 |
| BERT-base seq 512 | AVX-512 | 1 / 32 | 12.81 / 1.26 | 14.62 / 1.53 | 1.14 / 1.22 |
| MiniLM-L6 seq 128 (12 x 32) | AVX2 | 1 / 32 | 0.460 / 0.061 | 0.558 / 0.069 | 1.21 / 1.14 |
| MiniLM-L6 seq 128 | AVX-512 | 1 / 32 | 0.421 / 0.053 | 0.555 / 0.067 | 1.32 / 1.27 |
| Whisper-small encoder self 1500 | AVX2 | 1 / 32 | 102.8 / 10.64 | 122.1 / 12.69 | 1.19 / 1.19 |
| Whisper-small encoder self 1500 | AVX-512 | 1 / 32 | 102.3 / 9.11 | 124.0 / 11.92 | 1.21 / 1.31 |
| Whisper decoder cross, 1 step | AVX2 | 1 / 32 | 0.643 / 0.123 | 0.703 / 0.124 | 1.09 / 1.01 |
| Whisper decoder cross, 1 step | AVX-512 | 1 / 32 | 0.620 / 0.107 | 0.641 / 0.114 | 1.03 / 1.06 |
| Whisper decoder cross, 448 steps | AVX2 | 1 / 32 | 32.77 / 3.28 | 39.91 / 4.08 | 1.22 / 1.24 |
| Whisper decoder cross, 448 steps | AVX-512 | 1 / 32 | 33.39 / 2.86 | 39.44 / 3.69 | 1.18 / 1.29 |

Apple M2 (NEON, 128-bit lanes, some other load present), 1 / 8 threads: BERT-base 512 22.4 / 4.67 fast vs 27.9 / 6.16 exact (1.25 / 1.32);
MiniLM 128 0.832 / 0.190 vs 1.042 / 0.238 (1.25 / 1.25); Whisper encoder 1500 194.2 / 43.8 vs 244.4 / 55.0 (1.26 / 1.25);
decoder cross 1 step 1.032 / 0.284 vs 1.118 / 0.315 (1.08 / 1.11); 448 steps 57.8 / 11.9 vs 71.4 / 17.9 (1.24 / 1.50).
The exact outputs hash identically on both machines (e.g. BERT-base 128: `7fccb21b1fcc4d3b`).

Attention exact/fast stays between about 1.0 and 1.5 (AVX-512 BERT-128 at one thread, 1.9, is a short 0.7 ms call and noisy). Zen 4 has separate multiply and add pipes, so an unfused multiply-add costs it
little in the matrix products; the exponential's extra polynomial term, the unfused polynomial steps and the divide at the end are most of the gap. On a CPU whose
multiplier and adder share the FMA ports (Intel) the products would cost more, so this should be re-measured there.

Share of a model: counting multiply-accumulates per layer, attention (`Q K^T` and `P V`) is 2.7% of BERT-base at seq 128, 10% at seq 512,
5.3% of MiniLM-L6 at seq 128 and 25% of the Whisper-small encoder at seq 1500. If the rest of the model costs the same exact or fast, the model-level exact/fast from attention alone is
1 + share x (ratio - 1): about 1.01 (BERT 128), 1.02 (BERT 512), 1.01 (MiniLM), 1.05 (Whisper encoder). Attention is less efficient than a matrix product, so its time share is larger than its operation share;
even at three times the share the contribution stays below 1.15. The model-level ratio is then set by the GEMM and convolution exact/fast ratio and by the elementwise functions
(GELU: exact/fast 1.1 to 1.4 at AVX2; exp 2.3 to 2.6).

## Matrix kernels

The score and value products are the micro-tiles written here (no dependence on the GEMM unit). When the shared GEMM exists the packed-key
score product and the `P V` product can call it, provided its exact flavour accumulates each output in increasing reduction index from `+0`, one accumulator, unfused.


<!-- https://recursiveai.org/onemodel/docs/kernels-depthwise/ -->

# Depthwise convolution

Applies to a `Conv` with `Groups == Cin` (one input channel per group; `Cout = Cin * multiplier`). Other convolutions are lowered to the
matrix product ([kernels-gemm.md](https://recursiveai.org/onemodel/docs/kernels-gemm/)), which would be all overhead here because `k = KH * KW`.

## What each output is

For output plane `q` (image `n`, output channel `oc`; input channel `oc / multiplier`) and output position `(oy, ox)`:

```
acc = +0
for ky in 0 .. KH-1:            # row-major tap order
  for kx in 0 .. KW-1:
    acc = acc + w[oc, ky, kx] * x_padded[oy*StrideH + ky*DilH, ox*StrideW + kx*DilW]
```

`x_padded` is the input plane with the padding written as real zeros, so a padding tap is a multiplication by zero (an infinite or NaN
weight meets padding as NaN). Exact mode: the product is rounded, then the sum is rounded (no fused operation), and a NaN result is
rewritten as the canonical NaN `0x7FC00000`. Fast mode: one fused multiply-add per tap where the hardware has it (AVX2+FMA, AVX-512,
NEON), `acc + w*x` elsewhere. The bias, when present, is a separate pass: one rounded addition per output.

Every output is computed independently with this one order. The lane count of the vector code therefore never changes a result:
scalar, 4-lane, 8-lane and 16-lane paths give the same bits in exact mode, and in fast mode too wherever the fused operation is the same
(the fused scalar twin `MathF.FusedMultiplyAdd` is used when the hardware has a fused instruction, so the scalar path equals the vector
paths on that machine). Output columns that a path computes twice (see the last vector below) produce identical values both times.

## Layout of the work

* **Lanes are neighbouring output columns of one row.** A tap is one contiguous load of `W` input values at column `ox + dx / StrideW`
  (see phases) and one multiply-add with the broadcast weight.
* **Phase planes.** The zero-padded input plane is split by column phase into `StrideW` planes: plane `p` holds padded columns
  `p, p + StrideW, p + 2*StrideW, ...`. The tap at padded column `ox*StrideW + dx` (with `dx = kx * DilW`) is then at column
  `ox + dx / StrideW` of plane `dx % StrideW`, which is contiguous in `ox` for any stride. A stride-1 plane without padding is read in
  place; with padding it is copied into one zero-filled padded plane. For stride 2 the split is done with 4-lane shuffles (SSE
  `shufps`, NEON `uzp1/uzp2`) on eight source values at a time; only data moves.
* **Vectors per row.** A row is processed four vectors at a time, then one vector at a time; the last vector of a row is moved back to
  end at the last column (it recomputes up to `W-1` outputs with identical results) instead of a scalar tail. The lane type is the
  widest of 16 (AVX-512), 8 (AVX2), 4 (NEON, SSE, WebAssembly SIMD), 1 (scalar twin) that fits in the output width.
* **Narrow rows.** When the output is narrower than four vectors and at least four rows tall, four rows are computed together for one
  vector column, so that each tap's weight is shared by four multiply-adds. Remaining rows use the row-wise form.
* **Fallback.** Kernels with more than 64 taps or a column stride above 8 take a plain per-output loop in the same tap order.

## Chunks

A chunk is `clamp(4096 / (OutH * OutW), 1, 32)` output planes: a constant of the shape, never of the thread count. A chunk writes only
its own planes, so results do not depend on which thread ran it. Phase-plane scratch is per thread. The work estimate given to the pool
is `planes * (OutH*OutW*KH*KW/8 + PaddedH*PaddedW/2)` nanoseconds.

## Measured (indicative, one thread, one x64 machine with AVX2 and AVX-512, shared host)

GFLOP/s of the MobileNetV3-large depthwise layers, before (im2col-free kernel with scalar tails and strided gathers) and after:

| Layer | Before (AVX2, fast) | After (AVX2, fast) |
|---|---|---|
| 16->16 3x3/1 @112 | 13-17 | 41 |
| 64->64 3x3/2 @112 | 7.5-8 | 17-20 |
| 72->72 3x3/1 @56 | 13-16 | 34 |
| 72->72 5x5/2 @56 | 7 | 38 |
| 120->120 5x5/1 @28 | 10-12 | 50 |
| 240->240 3x3/2 @28 | 3-4 | 14 |
| 200->200 3x3/1 @14 | 4 | 27 |
| 672->672 3x3/1 @14 | 5 | 26 |
| 672->672 5x5/2 @14 | 2.8 | 9 |
| 960->960 5x5/1 @7 | 2-3 | 13 |


<!-- https://recursiveai.org/onemodel/docs/conformance/ -->

# Conformance: onnx/backend/test

Generated by `tools/conformance.sh` (it runs `OnnxSweep conformance` over the frozen ONNX 1.22.0 backend-test data, then `tools/conformance_table.py`). Nothing here is edited by hand. The method is at the end.

## Summary

| Mode | Cases | pass | fail | unsupported-op | unsupported-type | not-exact | known-diff |
|---|---:|---:|---:|---:|---:|---:|---:|
| fast | 1905 | 1580 | 0 | 145 | 176 | 0 | 4 |
| exact | 1905 | 1382 | 0 | 145 | 176 | 200 | 2 |

A **gap** is an `unsupported-op` or `unsupported-type` case: the library says so, loudly, when the session is created or first run. A **fail** is a case that ran (or threw) and is wrong: a defect. A **known-diff** is a failing case whose difference is understood and listed in the runner with its reason (Resize align_corners follows ONNX Runtime; the DFT inverse cases' expected data is noisier than their tolerance). A **not-exact** case, in exact mode only, uses an operator that has no exact kernel yet; its result is not computed.

| Group | Cases | pass | fail | gap | not-exact (exact mode) |
|---|---:|---:|---:|---:|---:|
| node | 1765 | 1475 | 0 | 286 | 200 |
| simple | 23 | 15 | 0 | 8 | 0 |
| pytorch-converted | 82 | 66 | 0 | 16 | 0 |
| pytorch-operator | 35 | 24 | 0 | 11 | 0 |

## Operators (node cases)

`Opsets` lists the opset versions of the cases that pass. `exact` is the pass count of the same cases in exact mode; `not-exact` counts those that need an operator without an exact kernel. Gap causes are the most frequent reasons.

| Operator | Opsets passing | Cases | pass | fail | gap | exact pass | not-exact | Gap and fail causes |
|---|---|---:|---:|---:|---:|---:|---:|---|
| Abs | 13 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| Acos | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Acosh | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Add | 14 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| AffineGrid | 20 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| ai.onnx.ml.ArrayFeatureExtractor | - | 1 | 0 | 0 | 1 | 0 | 0 | no kernel: ai.onnx.ml.ArrayFeatureExtractor (opset 1) |
| ai.onnx.ml.Binarizer | - | 1 | 0 | 0 | 1 | 0 | 0 | no kernel: ai.onnx.ml.Binarizer (opset 1) |
| ai.onnx.ml.LabelEncoder | - | 4 | 0 | 0 | 4 | 0 | 0 | value 'X' has element type string ×4 |
| ai.onnx.ml.TreeEnsemble | - | 2 | 0 | 0 | 2 | 0 | 0 | no kernel: ai.onnx.ml.TreeEnsemble (opset 5) ×2 |
| ai.onnx.preview.FlexAttention | 26 | 22 | 10 | 0 | 12 | 10 | 0 | no kernel: ai.onnx.preview.FlexAttention (opset 1) ×11; Softmax: element type F64 is not supported yet; only float32 is |
| ai.onnx.preview.training.Adagrad | - | 2 | 0 | 0 | 2 | 0 | 0 | no kernel: ai.onnx.preview.training.Adagrad (opset 1) ×2 |
| ai.onnx.preview.training.Adam | - | 2 | 0 | 0 | 2 | 0 | 0 | no kernel: ai.onnx.preview.training.Adam (opset 1) ×2 |
| ai.onnx.preview.training.Momentum | - | 3 | 0 | 0 | 3 | 0 | 0 | no kernel: ai.onnx.preview.training.Momentum (opset 1) ×3 |
| And | 7 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| ArgMax | 13 | 16 | 16 | 0 | 0 | 16 | 0 |  |
| ArgMin | 13 | 16 | 16 | 0 | 0 | 16 | 0 |  |
| Asin | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Asinh | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Atan | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Atanh | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Attention | 23, 24 | 128 | 128 | 0 | 0 | 64 | 64 |  |
| AveragePool | 22 | 20 | 20 | 0 | 0 | 20 | 0 |  |
| BatchNormalization | 15 | 4 | 2 | 0 | 2 | 2 | 0 | BatchNormalization: training_mode = 1 is not supported ×2 |
| Bernoulli | - | 6 | 0 | 0 | 6 | 0 | 0 | no kernel: Bernoulli (opset 22) ×3; no kernel: RandomUniformLike (opset 22) ×3 |
| BitCast | 26 | 10 | 10 | 0 | 0 | 10 | 0 |  |
| BitShift | 11 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| BitwiseAnd | 18 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| BitwiseNot | 18 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| BitwiseOr | 18 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| BitwiseXor | 18 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| BlackmanWindow | 17 | 4 | 4 | 0 | 0 | 0 | 4 |  |
| Cast | 25 | 116 | 16 | 0 | 100 | 16 | 0 | value 'input' has element type int2 ×6; value 'input' has element type int4 ×6; value 'input' has element type uint2 ×6; +26 more |
| CastLike | 25 | 56 | 8 | 0 | 48 | 8 | 0 | value 'like' has element type float8e4m3fn ×4; value 'like' has element type float8e4m3fnuz ×4; value 'like' has element type float8e5m2 ×4; +15 more |
| CausalConvWithState | 27 | 26 | 13 | 0 | 13 | 13 | 0 | no kernel: CausalConvWithState (opset 27) ×13 |
| Ceil | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Celu | 12 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| CenterCropPad | 18 | 12 | 12 | 0 | 0 | 12 | 0 |  |
| Clip | 13 | 22 | 22 | 0 | 0 | 22 | 0 |  |
| Col2Im | 18 | 5 | 5 | 0 | 0 | 5 | 0 |  |
| Compress | 11 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| Concat | 13 | 12 | 12 | 0 | 0 | 12 | 0 |  |
| Constant | 25 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| ConstantOfShape | 25 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| Conv | 22 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| ConvInteger | 10 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| ConvTranspose | 22 | 11 | 11 | 0 | 0 | 11 | 0 |  |
| Cos | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Cosh | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| CumProd | 26 | 9 | 9 | 0 | 0 | 9 | 0 |  |
| CumSum | 14 | 9 | 9 | 0 | 0 | 9 | 0 |  |
| DeformConv | - | 4 | 0 | 0 | 4 | 0 | 0 | no kernel: DeformConv (opset 22) ×4 |
| DepthToSpace | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| DequantizeLinear | 25 | 14 | 4 | 0 | 10 | 4 | 0 | value 'x' has element type float8e4m3fn ×3; DequantizeLinear: block_size (blocked quantization) is not supported; value 'x' has element type float4e2m1; +5 more |
| Det | 22 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| DFT | 19, 20 | 10 | 8 | 0 | 0 | 0 | 10 | expected data carries 1.9e-7 float32 FFT noise, above the case's 1e-7 absolute tolerance ×2 |
| Div | 14 | 10 | 10 | 0 | 0 | 10 | 0 |  |
| Dropout | 11, 22 | 12 | 6 | 0 | 6 | 6 | 0 | Dropout: training mode is not supported ×6 |
| DynamicQuantizeLinear | 11 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| Einsum | 12 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| Elu | 18, 22 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| Equal | 19 | 10 | 8 | 0 | 2 | 8 | 0 | value 'x' has element type string ×2 |
| Erf | 13 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| Exp | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Expand | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| EyeLike | 22 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| Flatten | 25 | 9 | 9 | 0 | 0 | 9 | 0 |  |
| Floor | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Gather | 13 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| GatherElements | 13 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| GatherND | 13 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| Gelu | 20 | 8 | 8 | 0 | 0 | 6 | 2 |  |
| Gemm | 13 | 11 | 11 | 0 | 0 | 11 | 0 |  |
| GlobalAveragePool | 22 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| GlobalMaxPool | 22 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Greater | 13 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| GreaterOrEqual | 16 | 16 | 16 | 0 | 0 | 16 | 0 |  |
| GridSample | 22 | 18 | 18 | 0 | 0 | 0 | 18 |  |
| GroupNormalization | 21 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| GRU | 22 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| HammingWindow | 17 | 4 | 4 | 0 | 0 | 0 | 4 |  |
| HannWindow | 17 | 4 | 4 | 0 | 0 | 0 | 4 |  |
| Hardmax | 13 | 7 | 7 | 0 | 0 | 7 | 0 |  |
| HardSigmoid | 18, 22 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| HardSwish | 22 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Identity | 13, 16, 25 | 5 | 5 | 0 | 0 | 5 | 0 |  |
| If | 11, 13, 16 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| ImageDecoder | - | 9 | 0 | 0 | 9 | 0 | 0 | no kernel: ImageDecoder (opset 20) ×9 |
| InstanceNormalization | 22 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| IsInf | 20 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| IsNaN | 20 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| LayerNormalization | 17, 18 | 57 | 57 | 0 | 0 | 57 | 0 |  |
| LeakyRelu | 16 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| Less | 13 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| LessOrEqual | 16 | 16 | 16 | 0 | 0 | 16 | 0 |  |
| LinearAttention | 27 | 28 | 14 | 0 | 14 | 14 | 0 | no kernel: LinearAttention (opset 27) ×14 |
| Log | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| LogSoftmax | 13, 18 | 21 | 21 | 0 | 0 | 21 | 0 |  |
| Loop | 11, 13, 16 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| LpNormalization | 22 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| LpPool | 22 | 8 | 8 | 0 | 0 | 0 | 8 |  |
| LRN | 13 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| LSTM | 22 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| MatMul | 13 | 7 | 7 | 0 | 0 | 7 | 0 |  |
| MatMulInteger | 10 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| Max | 13 | 14 | 14 | 0 | 0 | 14 | 0 |  |
| MaxPool | 22 | 19 | 19 | 0 | 0 | 19 | 0 |  |
| MaxUnpool | 22 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Mean | 13 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| MeanVarianceNormalization | 13, 18 | 3 | 3 | 0 | 0 | 1 | 2 |  |
| MelWeightMatrix | 17 | 1 | 1 | 0 | 0 | 0 | 1 |  |
| Min | 13 | 14 | 14 | 0 | 0 | 14 | 0 |  |
| Mish | 22 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Mod | 13 | 13 | 13 | 0 | 0 | 13 | 0 |  |
| Mul | 14 | 9 | 9 | 0 | 0 | 9 | 0 |  |
| Neg | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| NegativeLogLikelihoodLoss | 22 | 36 | 36 | 0 | 0 | 36 | 0 |  |
| NonMaxSuppression | 11 | 10 | 10 | 0 | 0 | 10 | 0 |  |
| NonZero | 13 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| Not | 1 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| OneHot | 11 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| OptionalGetElement | 18 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| OptionalHasElement | 18 | 7 | 7 | 0 | 0 | 7 | 0 |  |
| Or | 7 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| Pad | 25 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| Pow | 15 | 12 | 12 | 0 | 0 | 0 | 12 |  |
| PRelu | 16 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| QLinearConv | 10 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| QLinearMatMul | 21 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| QuantizeLinear | 25 | 13 | 4 | 0 | 9 | 4 | 0 | QuantizeLinear: block_size (blocked quantization) is not supported ×2; value 'y_zero_point' has element type float4e2m1; value 'y_zero_point' has element type float8e4m3fn; +5 more |
| Range | 27 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| Reciprocal | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| ReduceL1 | 18 | 18 | 18 | 0 | 0 | 18 | 0 |  |
| ReduceL2 | 18 | 18 | 18 | 0 | 0 | 18 | 0 |  |
| ReduceLogSum | 18 | 10 | 10 | 0 | 0 | 10 | 0 |  |
| ReduceLogSumExp | 18 | 18 | 9 | 0 | 9 | 1 | 8 | Exp: element type F64 is not supported yet ×9 |
| ReduceMax | 18, 20 | 10 | 10 | 0 | 0 | 10 | 0 |  |
| ReduceMean | 18 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| ReduceMin | 18, 20 | 10 | 10 | 0 | 0 | 10 | 0 |  |
| ReduceProd | 18 | 9 | 9 | 0 | 0 | 9 | 0 |  |
| ReduceSum | 13 | 12 | 12 | 0 | 0 | 12 | 0 |  |
| ReduceSumSquare | 18 | 18 | 18 | 0 | 0 | 18 | 0 |  |
| RegexFullMatch | - | 3 | 0 | 0 | 3 | 0 | 0 | value 'X' has element type string ×3 |
| Relu | 14, 18 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Reshape | 25 | 10 | 10 | 0 | 0 | 10 | 0 |  |
| Resize | 19 | 39 | 37 | 0 | 0 | 37 | 0 | Resize align_corners follows ONNX Runtime where the two backend cases disagree (docs/ops-conv.md) ×2 |
| ReverseSequence | 10 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| RMSNormalization | 23 | 38 | 38 | 0 | 0 | 38 | 0 |  |
| RNN | 22 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| RoiAlign | 22 | 3 | 3 | 0 | 0 | 0 | 3 |  |
| RotaryEmbedding | 23 | 16 | 16 | 0 | 0 | 16 | 0 |  |
| Round | 22 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| Scan | 8, 9 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| Scatter | 10 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| ScatterElements | 18 | 7 | 7 | 0 | 0 | 7 | 0 |  |
| ScatterND | 18 | 5 | 5 | 0 | 0 | 5 | 0 |  |
| Selu | 18, 22 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| SequenceInsert | 11 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| SequenceMap | 17 | 12 | 12 | 0 | 0 | 12 | 0 |  |
| Shape | 25 | 11 | 11 | 0 | 0 | 11 | 0 |  |
| Shrink | 9, 18 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| Sigmoid | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Sign | 13 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| Sin | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Sinh | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Size | 25 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Slice | 13 | 8 | 8 | 0 | 0 | 8 | 0 |  |
| Softmax | 13, 18 | 21 | 21 | 0 | 0 | 21 | 0 |  |
| SoftmaxCrossEntropyLoss | 13 | 68 | 68 | 0 | 0 | 34 | 34 |  |
| Softplus | 18, 22 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| Softsign | 18, 22 | 4 | 4 | 0 | 0 | 4 | 0 |  |
| SpaceToDepth | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Split | 13, 18 | 16 | 16 | 0 | 0 | 16 | 0 |  |
| SplitToSequence | 24 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| Sqrt | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Squeeze | 25 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| STFT | 17 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| StringConcat | - | 5 | 0 | 0 | 5 | 0 | 0 | value 'x' has element type string ×5 |
| StringNormalizer | - | 6 | 0 | 0 | 6 | 0 | 0 | value 'x' has element type string ×6 |
| StringSplit | - | 6 | 0 | 0 | 6 | 0 | 0 | value 'x' has element type string ×6 |
| Sub | 14 | 9 | 9 | 0 | 0 | 9 | 0 |  |
| Sum | 13 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| Swish | 24 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Tan | 22 | 2 | 2 | 0 | 0 | 0 | 2 |  |
| Tanh | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| TensorScatter | 24 | 3 | 3 | 0 | 0 | 3 | 0 |  |
| TfIdfVectorizer | - | 7 | 0 | 0 | 7 | 0 | 0 | no kernel: TfIdfVectorizer (opset 9) ×7 |
| ThresholdedRelu | 18, 22 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| Tile | 13 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| TopK | 24 | 7 | 7 | 0 | 0 | 7 | 0 |  |
| Transpose | 25 | 7 | 7 | 0 | 0 | 7 | 0 |  |
| Trilu | 14 | 18 | 18 | 0 | 0 | 18 | 0 |  |
| Unique | 11 | 6 | 6 | 0 | 0 | 6 | 0 |  |
| Unsqueeze | 25 | 7 | 7 | 0 | 0 | 7 | 0 |  |
| Upsample | 9 | 1 | 1 | 0 | 0 | 1 | 0 |  |
| Where | 16 | 2 | 2 | 0 | 0 | 2 | 0 |  |
| Xor | 7 | 8 | 8 | 0 | 0 | 8 | 0 |  |

## Model cases (simple, pytorch-converted, pytorch-operator)

| Group | Case | Opset | Fast | Exact | Reason |
|---|---|---:|---|---|---|
| simple | test_gradient_of_add | 12 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Gradient (opset 1) |
| simple | test_gradient_of_add_and_mul | 12 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Gradient (opset 1) |
| simple | test_strnorm_model_monday_casesensintive_lower | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| simple | test_strnorm_model_monday_casesensintive_nochangecase | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| simple | test_strnorm_model_monday_casesensintive_upper | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| simple | test_strnorm_model_monday_empty_output | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| simple | test_strnorm_model_monday_insensintive_upper_twodim | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| simple | test_strnorm_model_nostopwords_nochangecase | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| pytorch-converted | test_BatchNorm1d_3d_input_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| pytorch-converted | test_BatchNorm2d_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| pytorch-converted | test_BatchNorm2d_momentum_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| pytorch-converted | test_BatchNorm3d_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| pytorch-converted | test_BatchNorm3d_momentum_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| pytorch-converted | test_GLU | 6 | unsupported-op | unsupported-op | no kernel: Mul (opset 6) |
| pytorch-converted | test_GLU_dim | 6 | unsupported-op | unsupported-op | no kernel: Mul (opset 6) |
| pytorch-converted | test_Linear | 6 | unsupported-op | unsupported-op | no kernel: Gemm (opset 6) |
| pytorch-converted | test_PReLU_1d | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| pytorch-converted | test_PReLU_1d_multiparam | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| pytorch-converted | test_PReLU_2d | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| pytorch-converted | test_PReLU_2d_multiparam | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| pytorch-converted | test_PReLU_3d | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| pytorch-converted | test_PReLU_3d_multiparam | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| pytorch-converted | test_PoissonNLLLLoss_no_reduce | 6 | unsupported-op | unsupported-op | no kernel: Mul (opset 6), Sub (opset 6) |
| pytorch-converted | test_Softsign | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Div (opset 6) |
| pytorch-operator | test_operator_add_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| pytorch-operator | test_operator_add_size1_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| pytorch-operator | test_operator_add_size1_right_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| pytorch-operator | test_operator_add_size1_singleton_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| pytorch-operator | test_operator_addconstant | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| pytorch-operator | test_operator_addmm | 6 | unsupported-op | unsupported-op | no kernel: Gemm (opset 6) |
| pytorch-operator | test_operator_basic | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Mul (opset 6) |
| pytorch-operator | test_operator_mm | 6 | unsupported-op | unsupported-op | no kernel: Gemm (opset 6) |
| pytorch-operator | test_operator_non_float_params | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Mul (opset 6) |
| pytorch-operator | test_operator_params | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Mul (opset 6) |
| pytorch-operator | test_operator_pow | 6 | unsupported-op | unsupported-op | no kernel: Pow (opset 6) |

105 further model cases pass in both modes and are not listed.

## Operators with no kernel

Operators named by `unsupported-op` cases, with the number of cases that need them (a case can need several).

| Operator | Cases |
|---|---:|
| LinearAttention | 14 |
| CausalConvWithState | 13 |
| ai.onnx.preview.FlexAttention | 11 |
| Add | 9 |
| ImageDecoder | 9 |
| TfIdfVectorizer | 7 |
| Mul | 6 |
| PRelu | 6 |
| BatchNormalization | 5 |
| DeformConv | 4 |
| Bernoulli | 3 |
| Gemm | 3 |
| RandomUniformLike | 3 |
| ai.onnx.preview.training.Momentum | 3 |
| ai.onnx.ml.TreeEnsemble | 2 |
| ai.onnx.preview.training.Adagrad | 2 |
| ai.onnx.preview.training.Adam | 2 |
| ai.onnx.preview.training.Gradient | 2 |
| Div | 1 |
| Pow | 1 |
| Sub | 1 |
| ai.onnx.ml.ArrayFeatureExtractor | 1 |
| ai.onnx.ml.Binarizer | 1 |

## Every case that does not pass

One line per case: `case` (opset), fast status, exact status, reason.

<details><summary>List</summary>

| Case | Opset | Fast | Exact | Reason |
|---|---:|---|---|---|
| test_acos | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Acos. Use ExecutionMode.Fast or remove these operators |
| test_acos_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Acos. Use ExecutionMode.Fast or remove these operators |
| test_acosh | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Acosh. Use ExecutionMode.Fast or remove these operator |
| test_acosh_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Acosh. Use ExecutionMode.Fast or remove these operator |
| test_adagrad | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Adagrad (opset 1) |
| test_adagrad_multiple | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Adagrad (opset 1) |
| test_adam | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Adam (opset 1) |
| test_adam_multiple | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Adam (opset 1) |
| test_ai_onnx_ml_array_feature_extractor | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.ml.ArrayFeatureExtractor (opset 1) |
| test_ai_onnx_ml_binarizer | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.ml.Binarizer (opset 1) |
| test_ai_onnx_ml_label_encoder_string_int | 0 | unsupported-op | unsupported-op | value 'X' has element type string |
| test_ai_onnx_ml_label_encoder_string_int_no_default | 0 | unsupported-op | unsupported-op | value 'X' has element type string |
| test_ai_onnx_ml_label_encoder_tensor_mapping | 0 | unsupported-op | unsupported-op | value 'X' has element type string |
| test_ai_onnx_ml_label_encoder_tensor_value_only_mapping | 0 | unsupported-op | unsupported-op | value 'X' has element type string |
| test_ai_onnx_ml_tree_ensemble_set_membership | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.ml.TreeEnsemble (opset 5) |
| test_ai_onnx_ml_tree_ensemble_single_tree | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.ml.TreeEnsemble (opset 5) |
| test_asin | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Asin. Use ExecutionMode.Fast or remove these operators |
| test_asin_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Asin. Use ExecutionMode.Fast or remove these operators |
| test_asinh | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Asinh. Use ExecutionMode.Fast or remove these operator |
| test_asinh_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Asinh. Use ExecutionMode.Fast or remove these operator |
| test_atan | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Atan. Use ExecutionMode.Fast or remove these operators |
| test_atan_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Atan. Use ExecutionMode.Fast or remove these operators |
| test_atanh | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Atanh. Use ExecutionMode.Fast or remove these operator |
| test_atanh_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Atanh. Use ExecutionMode.Fast or remove these operator |
| test_attention_3d | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_attn_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_diff_heads_sizes | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_diff_heads_sizes_attn_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_diff_heads_sizes_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_diff_heads_sizes_scaled | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_diff_heads_sizes_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_diff_heads_with_past_and_present | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_gqa | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_gqa_attn_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_gqa_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_gqa_scaled | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_gqa_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_gqa_with_past_and_present | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_scaled | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_transpose_verification | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_with_past_and_present | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_with_past_and_present_qk_matmul | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_with_past_and_present_qk_matmul_bias | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_with_past_and_present_qk_matmul_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_3d_with_past_and_present_qk_matmul_softmax | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_attn_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_attn_mask_3d | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_attn_mask_3d_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_attn_mask_4d | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_attn_mask_4d_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_attn_mask_bool | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_attn_mask_bool_4d | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_mask4d_padded_kv | 24 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_sizes | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_sizes_attn_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_sizes_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_sizes_scaled | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_sizes_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_with_past_and_present | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_with_past_and_present_mask3d | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_diff_heads_with_past_and_present_mask4d | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_fp16 | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_gqa | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_gqa_attn_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_gqa_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_gqa_scaled | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_gqa_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_gqa_with_past_and_present | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_gqa_with_past_and_present_fp16 | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_scaled | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_softcap_neginf_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_softcap_neginf_mask_poison | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_past_and_present | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_past_and_present_qk_matmul | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_past_and_present_qk_matmul_bias | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_past_and_present_qk_matmul_bias_3d_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_past_and_present_qk_matmul_bias_3d_mask_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_past_and_present_qk_matmul_bias_4d_mask | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_past_and_present_qk_matmul_bias_4d_mask_causal | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_qk_matmul | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_qk_matmul_bias | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_qk_matmul_softcap | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_attention_4d_with_qk_matmul_softmax | 23 | pass | not-exact | Exact mode is not available: no exact kernel exists for Attention. Use ExecutionMode.Fast or remove these oper |
| test_basic_deform_conv_with_padding | 22 | unsupported-op | unsupported-op | no kernel: DeformConv (opset 22) |
| test_basic_deform_conv_without_padding | 22 | unsupported-op | unsupported-op | no kernel: DeformConv (opset 22) |
| test_batchnorm_epsilon_training_mode | 15 | unsupported-op | unsupported-op | BatchNormalization: training_mode = 1 is not supported |
| test_batchnorm_example_training_mode | 15 | unsupported-op | unsupported-op | BatchNormalization: training_mode = 1 is not supported |
| test_bernoulli | 22 | unsupported-op | unsupported-op | no kernel: Bernoulli (opset 22) |
| test_bernoulli_double | 22 | unsupported-op | unsupported-op | no kernel: Bernoulli (opset 22) |
| test_bernoulli_double_expanded | 22 | unsupported-op | unsupported-op | no kernel: RandomUniformLike (opset 22) |
| test_bernoulli_expanded | 22 | unsupported-op | unsupported-op | no kernel: RandomUniformLike (opset 22) |
| test_bernoulli_seed | 22 | unsupported-op | unsupported-op | no kernel: Bernoulli (opset 22) |
| test_bernoulli_seed_expanded | 22 | unsupported-op | unsupported-op | no kernel: RandomUniformLike (opset 22) |
| test_blackmanwindow | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for BlackmanWindow. Use ExecutionMode.Fast or remove these |
| test_blackmanwindow_expanded | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_blackmanwindow_symmetric | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for BlackmanWindow. Use ExecutionMode.Fast or remove these |
| test_blackmanwindow_symmetric_expanded | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_cast_FLOAT16_to_FLOAT4E2M1 | 25 | unsupported-type | unsupported-type | value 'output' has element type float4e2m1 |
| test_cast_FLOAT16_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fn |
| test_cast_FLOAT16_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fnuz |
| test_cast_FLOAT16_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2 |
| test_cast_FLOAT16_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2fnuz |
| test_cast_FLOAT16_to_INT2 | 25 | unsupported-type | unsupported-type | value 'output' has element type int2 |
| test_cast_FLOAT16_to_INT4 | 25 | unsupported-type | unsupported-type | value 'output' has element type int4 |
| test_cast_FLOAT16_to_UINT2 | 25 | unsupported-type | unsupported-type | value 'output' has element type uint2 |
| test_cast_FLOAT16_to_UINT4 | 25 | unsupported-type | unsupported-type | value 'output' has element type uint4 |
| test_cast_FLOAT4E2M1_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float4e2m1 |
| test_cast_FLOAT4E2M1_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float4e2m1 |
| test_cast_FLOAT8E4M3FNUZ_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fnuz |
| test_cast_FLOAT8E4M3FNUZ_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fnuz |
| test_cast_FLOAT8E4M3FN_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fn |
| test_cast_FLOAT8E4M3FN_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fn |
| test_cast_FLOAT8E5M2FNUZ_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2fnuz |
| test_cast_FLOAT8E5M2FNUZ_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2fnuz |
| test_cast_FLOAT8E5M2_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2 |
| test_cast_FLOAT8E5M2_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2 |
| test_cast_FLOAT_to_FLOAT4E2M1 | 25 | unsupported-type | unsupported-type | value 'output' has element type float4e2m1 |
| test_cast_FLOAT_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fn |
| test_cast_FLOAT_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fnuz |
| test_cast_FLOAT_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2 |
| test_cast_FLOAT_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2fnuz |
| test_cast_FLOAT_to_INT2 | 25 | unsupported-type | unsupported-type | value 'output' has element type int2 |
| test_cast_FLOAT_to_INT4 | 25 | unsupported-type | unsupported-type | value 'output' has element type int4 |
| test_cast_FLOAT_to_UINT2 | 25 | unsupported-type | unsupported-type | value 'output' has element type uint2 |
| test_cast_FLOAT_to_UINT4 | 25 | unsupported-type | unsupported-type | value 'output' has element type uint4 |
| test_cast_INT2_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_cast_INT2_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_cast_INT2_to_INT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_cast_INT4_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_cast_INT4_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_cast_INT4_to_INT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_cast_UINT2_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_cast_UINT2_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_cast_UINT2_to_UINT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_cast_UINT4_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_cast_UINT4_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_cast_UINT4_to_UINT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_cast_e8m0_FLOAT16_to_FLOAT8E8M0 | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e8m0 |
| test_cast_e8m0_FLOAT8E8M0_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e8m0 |
| test_cast_e8m0_FLOAT8E8M0_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e8m0 |
| test_cast_e8m0_FLOAT_to_FLOAT8E8M0 | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e8m0 |
| test_cast_no_saturate_FLOAT16_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fn |
| test_cast_no_saturate_FLOAT16_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fnuz |
| test_cast_no_saturate_FLOAT16_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2 |
| test_cast_no_saturate_FLOAT16_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2fnuz |
| test_cast_no_saturate_FLOAT_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fn |
| test_cast_no_saturate_FLOAT_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e4m3fnuz |
| test_cast_no_saturate_FLOAT_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2 |
| test_cast_no_saturate_FLOAT_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'output' has element type float8e5m2fnuz |
| test_castlike_FLOAT16_to_FLOAT4E2M1 | 25 | unsupported-type | unsupported-type | value 'like' has element type float4e2m1 |
| test_castlike_FLOAT16_to_FLOAT4E2M1_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float4e2m1 |
| test_castlike_FLOAT16_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_FLOAT16_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_FLOAT16_to_FLOAT8E4M3FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_FLOAT16_to_FLOAT8E4M3FN_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_FLOAT16_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_castlike_FLOAT16_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_FLOAT16_to_FLOAT8E5M2FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_FLOAT16_to_FLOAT8E5M2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_castlike_FLOAT16_to_INT2 | 25 | unsupported-type | unsupported-type | value 'like' has element type int2 |
| test_castlike_FLOAT16_to_INT2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type int2 |
| test_castlike_FLOAT16_to_INT4 | 25 | unsupported-type | unsupported-type | value 'like' has element type int4 |
| test_castlike_FLOAT16_to_INT4_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type int4 |
| test_castlike_FLOAT16_to_UINT2 | 25 | unsupported-type | unsupported-type | value 'like' has element type uint2 |
| test_castlike_FLOAT16_to_UINT2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type uint2 |
| test_castlike_FLOAT16_to_UINT4 | 25 | unsupported-type | unsupported-type | value 'like' has element type uint4 |
| test_castlike_FLOAT16_to_UINT4_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type uint4 |
| test_castlike_FLOAT4E2M1_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float4e2m1 |
| test_castlike_FLOAT4E2M1_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float4e2m1 |
| test_castlike_FLOAT4E2M1_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float4e2m1 |
| test_castlike_FLOAT4E2M1_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float4e2m1 |
| test_castlike_FLOAT8E4M3FNUZ_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fnuz |
| test_castlike_FLOAT8E4M3FNUZ_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fnuz |
| test_castlike_FLOAT8E4M3FNUZ_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fnuz |
| test_castlike_FLOAT8E4M3FNUZ_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fnuz |
| test_castlike_FLOAT8E4M3FN_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fn |
| test_castlike_FLOAT8E4M3FN_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fn |
| test_castlike_FLOAT8E4M3FN_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fn |
| test_castlike_FLOAT8E4M3FN_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e4m3fn |
| test_castlike_FLOAT8E5M2FNUZ_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2fnuz |
| test_castlike_FLOAT8E5M2FNUZ_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2fnuz |
| test_castlike_FLOAT8E5M2FNUZ_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2fnuz |
| test_castlike_FLOAT8E5M2FNUZ_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2fnuz |
| test_castlike_FLOAT8E5M2_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2 |
| test_castlike_FLOAT8E5M2_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2 |
| test_castlike_FLOAT8E5M2_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2 |
| test_castlike_FLOAT8E5M2_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type float8e5m2 |
| test_castlike_FLOAT_to_FLOAT4E2M1 | 25 | unsupported-type | unsupported-type | value 'like' has element type float4e2m1 |
| test_castlike_FLOAT_to_FLOAT4E2M1_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float4e2m1 |
| test_castlike_FLOAT_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_FLOAT_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_FLOAT_to_FLOAT8E4M3FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_FLOAT_to_FLOAT8E4M3FN_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_FLOAT_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_castlike_FLOAT_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_FLOAT_to_FLOAT8E5M2FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_FLOAT_to_FLOAT8E5M2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_castlike_FLOAT_to_INT2 | 25 | unsupported-type | unsupported-type | value 'like' has element type int2 |
| test_castlike_FLOAT_to_INT2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type int2 |
| test_castlike_FLOAT_to_INT4 | 25 | unsupported-type | unsupported-type | value 'like' has element type int4 |
| test_castlike_FLOAT_to_INT4_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type int4 |
| test_castlike_FLOAT_to_UINT2 | 25 | unsupported-type | unsupported-type | value 'like' has element type uint2 |
| test_castlike_FLOAT_to_UINT2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type uint2 |
| test_castlike_FLOAT_to_UINT4 | 25 | unsupported-type | unsupported-type | value 'like' has element type uint4 |
| test_castlike_FLOAT_to_UINT4_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type uint4 |
| test_castlike_INT2_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_castlike_INT2_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_castlike_INT2_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_castlike_INT2_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_castlike_INT2_to_INT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_castlike_INT2_to_INT8_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type int2 |
| test_castlike_INT4_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_castlike_INT4_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_castlike_INT4_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_castlike_INT4_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_castlike_INT4_to_INT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_castlike_INT4_to_INT8_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type int4 |
| test_castlike_UINT2_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_castlike_UINT2_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_castlike_UINT2_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_castlike_UINT2_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_castlike_UINT2_to_UINT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_castlike_UINT2_to_UINT8_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type uint2 |
| test_castlike_UINT4_to_FLOAT | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_castlike_UINT4_to_FLOAT16 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_castlike_UINT4_to_FLOAT16_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_castlike_UINT4_to_FLOAT_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_castlike_UINT4_to_UINT8 | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_castlike_UINT4_to_UINT8_expanded | 25 | unsupported-type | unsupported-type | value 'input' has element type uint4 |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E4M3FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E4M3FN_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E5M2FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_no_saturate_FLOAT16_to_FLOAT8E5M2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E4M3FN | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E4M3FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E4M3FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fnuz |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E4M3FN_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e4m3fn |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E5M2 | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E5M2FNUZ | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E5M2FNUZ_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2fnuz |
| test_castlike_no_saturate_FLOAT_to_FLOAT8E5M2_expanded | 25 | unsupported-type | unsupported-type | value 'like' has element type float8e5m2 |
| test_causal_conv_with_state_b1_c1_degenerate | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_basic | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_decode_step | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_fp16 | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_kernel_size_one | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_short_input_no_past_state | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_silu | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_silu_fp16 | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_silu_with_past_state | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_swish_alias | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_with_bias | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_with_bias_and_past_state | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_causal_conv_with_state_with_past_state | 27 | unsupported-op | unsupported-op | no kernel: CausalConvWithState (opset 27) |
| test_cos | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_cos_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_cosh | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cosh. Use ExecutionMode.Fast or remove these operators |
| test_cosh_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cosh. Use ExecutionMode.Fast or remove these operators |
| test_deform_conv_with_mask_bias | 22 | unsupported-op | unsupported-op | no kernel: DeformConv (opset 22) |
| test_deform_conv_with_multiple_offset_groups | 22 | unsupported-op | unsupported-op | no kernel: DeformConv (opset 22) |
| test_dequantizelinear_blocked | 25 | unsupported-op | unsupported-op | DequantizeLinear: block_size (blocked quantization) is not supported |
| test_dequantizelinear_e4m3fn | 25 | unsupported-type | unsupported-type | value 'x' has element type float8e4m3fn |
| test_dequantizelinear_e4m3fn_float16 | 25 | unsupported-type | unsupported-type | value 'x' has element type float8e4m3fn |
| test_dequantizelinear_e4m3fn_zero_point | 25 | unsupported-type | unsupported-type | value 'x' has element type float8e4m3fn |
| test_dequantizelinear_e5m2 | 25 | unsupported-type | unsupported-type | value 'x' has element type float8e5m2 |
| test_dequantizelinear_float4e2m1 | 25 | unsupported-type | unsupported-type | value 'x' has element type float4e2m1 |
| test_dequantizelinear_int2 | 25 | unsupported-type | unsupported-type | value 'x' has element type int2 |
| test_dequantizelinear_int4 | 25 | unsupported-type | unsupported-type | value 'x' has element type int4 |
| test_dequantizelinear_uint2 | 25 | unsupported-type | unsupported-type | value 'x' has element type uint2 |
| test_dequantizelinear_uint4 | 25 | unsupported-type | unsupported-type | value 'x' has element type uint4 |
| test_dft | 20 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_dft_axis | 20 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_dft_axis_opset19 | 19 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_dft_inverse | 20 | known-diff | not-exact | expected data carries 1.9e-7 float32 FFT noise, above the case's 1e-7 absolute tolerance |
| test_dft_inverse_opset19 | 19 | known-diff | not-exact | expected data carries 1.9e-7 float32 FFT noise, above the case's 1e-7 absolute tolerance |
| test_dft_irfft | 20 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_dft_irfft_opset19 | 19 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_dft_opset19 | 19 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_dft_rfft | 20 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_dft_rfft_opset19 | 19 | pass | not-exact | Exact mode is not available: no exact kernel exists for DFT. Use ExecutionMode.Fast or remove these operators. |
| test_equal_string | 19 | unsupported-type | unsupported-type | value 'x' has element type string |
| test_equal_string_broadcast | 19 | unsupported-type | unsupported-type | value 'x' has element type string |
| test_flexattention | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_causal_mask | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_diff_head_sizes | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_double | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_double_expanded_ver26 | 26 | unsupported-type | unsupported-type | Softmax: element type F64 is not supported yet; only float32 is |
| test_flexattention_fp16 | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_gqa | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_prob_mod | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_relative_positional | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_scaled | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_score_mod | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_flexattention_soft_cap | 26 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.FlexAttention (opset 1) |
| test_gelu_tanh_1_expanded | 20 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_gelu_tanh_2_expanded | 20 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_gridsample | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_aligncorners_true | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_bicubic | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_bicubic_align_corners_0_additional_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_bicubic_align_corners_1_additional_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_bilinear | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_bilinear_align_corners_0_additional_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_bilinear_align_corners_1_additional_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_border_padding | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_nearest | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_nearest_align_corners_0_additional_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_nearest_align_corners_1_additional_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_reflection_padding | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_volumetric_bilinear_align_corners_0 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_volumetric_bilinear_align_corners_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_volumetric_nearest_align_corners_0 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_volumetric_nearest_align_corners_1 | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_gridsample_zeros_padding | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for GridSample. Use ExecutionMode.Fast or remove these ope |
| test_hammingwindow | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for HammingWindow. Use ExecutionMode.Fast or remove these  |
| test_hammingwindow_expanded | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_hammingwindow_symmetric | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for HammingWindow. Use ExecutionMode.Fast or remove these  |
| test_hammingwindow_symmetric_expanded | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_hannwindow | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for HannWindow. Use ExecutionMode.Fast or remove these ope |
| test_hannwindow_expanded | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_hannwindow_symmetric | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for HannWindow. Use ExecutionMode.Fast or remove these ope |
| test_hannwindow_symmetric_expanded | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators. |
| test_image_decoder_decode_bmp_rgb | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_jpeg2k_rgb | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_jpeg_bgr | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_jpeg_grayscale | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_jpeg_rgb | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_png_rgb | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_pnm_rgb | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_tiff_rgb | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_image_decoder_decode_webp_rgb | 20 | unsupported-op | unsupported-op | no kernel: ImageDecoder (opset 20) |
| test_linear_attention_decode_step | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_delta | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_explicit_scale | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_fp16 | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_gated | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_gated_delta | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_gated_delta_beta_scalar | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_gated_delta_gqa | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_gated_delta_mqa | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_gated_per_head_decay | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_linear | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_linear_t1_no_past | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_no_past_explicit_zeros | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_linear_attention_prefill_with_past | 27 | unsupported-op | unsupported-op | no kernel: LinearAttention (opset 27) |
| test_lppool_1d_default | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lppool_2d_default | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lppool_2d_dilations | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lppool_2d_pads | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lppool_2d_same_lower | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lppool_2d_same_upper | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lppool_2d_strides | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lppool_3d_default | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for LpPool. Use ExecutionMode.Fast or remove these operato |
| test_lrn | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for LRN. Use ExecutionMode.Fast or remove these operators. |
| test_lrn_default | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for LRN. Use ExecutionMode.Fast or remove these operators. |
| test_melweightmatrix | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for MelWeightMatrix. Use ExecutionMode.Fast or remove thes |
| test_momentum | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Momentum (opset 1) |
| test_momentum_multiple | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Momentum (opset 1) |
| test_mvn_expanded | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_mvn_expanded_ver18 | 18 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_nesterov_momentum | 0 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Momentum (opset 1) |
| test_pow | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_bcast_array | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_bcast_scalar | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_example | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_float32_int32 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_float32_int64 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_float32_uint32 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_float32_uint64 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_int32_float32 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_int32_int32 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_int64_float32 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_pow_types_int64_int64 | 15 | pass | not-exact | Exact mode is not available: no exact kernel exists for Pow. Use ExecutionMode.Fast or remove these operators. |
| test_quantizelinear_blocked_asymmetric | 25 | unsupported-op | unsupported-op | QuantizeLinear: block_size (blocked quantization) is not supported |
| test_quantizelinear_blocked_symmetric | 25 | unsupported-op | unsupported-op | QuantizeLinear: block_size (blocked quantization) is not supported |
| test_quantizelinear_e4m3fn | 25 | unsupported-type | unsupported-type | value 'y_zero_point' has element type float8e4m3fn |
| test_quantizelinear_e5m2 | 25 | unsupported-type | unsupported-type | value 'y_zero_point' has element type float8e5m2 |
| test_quantizelinear_float4e2m1 | 25 | unsupported-type | unsupported-type | value 'y_zero_point' has element type float4e2m1 |
| test_quantizelinear_int2 | 25 | unsupported-type | unsupported-type | value 'y_zero_point' has element type int2 |
| test_quantizelinear_int4 | 25 | unsupported-type | unsupported-type | value 'y_zero_point' has element type int4 |
| test_quantizelinear_uint2 | 25 | unsupported-type | unsupported-type | value 'y_zero_point' has element type uint2 |
| test_quantizelinear_uint4 | 25 | unsupported-type | unsupported-type | value 'y_zero_point' has element type uint4 |
| test_reduce_log_sum_exp_default_axes_keepdims_example | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_default_axes_keepdims_example_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_default_axes_keepdims_random | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_default_axes_keepdims_random_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_do_not_keepdims_example | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_do_not_keepdims_example_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_do_not_keepdims_random | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_do_not_keepdims_random_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_empty_set_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_keepdims_example | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_keepdims_example_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_keepdims_random | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_keepdims_random_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_negative_axes_keepdims_example | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_negative_axes_keepdims_example_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_reduce_log_sum_exp_negative_axes_keepdims_random | 18 | pass | not-exact | Exact mode is not available: float64 ReduceLogSum and ReduceLogSumExp have no exact kernel. |
| test_reduce_log_sum_exp_negative_axes_keepdims_random_expanded | 18 | unsupported-type | unsupported-type | Exp: element type F64 is not supported yet |
| test_regex_full_match_basic | 20 | unsupported-op | unsupported-op | value 'X' has element type string |
| test_regex_full_match_email_domain | 20 | unsupported-op | unsupported-op | value 'X' has element type string |
| test_regex_full_match_empty | 20 | unsupported-op | unsupported-op | value 'X' has element type string |
| test_resize_downsample_scales_cubic_align_corners | 19 | known-diff | known-diff | Resize align_corners follows ONNX Runtime where the two backend cases disagree (docs/ops-conv.md) |
| test_resize_downsample_scales_linear_align_corners | 19 | known-diff | known-diff | Resize align_corners follows ONNX Runtime where the two backend cases disagree (docs/ops-conv.md) |
| test_roialign_aligned_false | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for RoiAlign. Use ExecutionMode.Fast or remove these opera |
| test_roialign_aligned_true | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for RoiAlign. Use ExecutionMode.Fast or remove these opera |
| test_roialign_mode_max | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for RoiAlign. Use ExecutionMode.Fast or remove these opera |
| test_sce_NCd1_mean_weight_negative_ii | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1_mean_weight_negative_ii_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3_none_no_weight_negative_ii | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3_none_no_weight_negative_ii_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3_sum_weight_high_ii | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3_sum_weight_high_ii_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3d4d5_mean_weight | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3d4d5_mean_weight_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3d4d5_none_no_weight | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_NCd1d2d3d4d5_none_no_weight_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_3d | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_3d_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_no_weight_ii | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_no_weight_ii_3d | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_no_weight_ii_3d_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_no_weight_ii_4d | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_no_weight_ii_4d_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_no_weight_ii_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight_ii | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight_ii_3d | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight_ii_3d_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight_ii_4d | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight_ii_4d_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight_ii_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_mean_weight_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_none | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_none_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_none_weights | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_none_weights_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_sum | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sce_sum_log_prob | 13 | pass | not-exact | Exact mode is not available: no exact kernel exists for SoftmaxCrossEntropyLoss. Use ExecutionMode.Fast or rem |
| test_sin | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Sin. Use ExecutionMode.Fast or remove these operators. |
| test_sin_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Sin. Use ExecutionMode.Fast or remove these operators. |
| test_sinh | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Sinh. Use ExecutionMode.Fast or remove these operators |
| test_sinh_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Sinh. Use ExecutionMode.Fast or remove these operators |
| test_stft | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for STFT. Use ExecutionMode.Fast or remove these operators |
| test_stft_with_window | 17 | pass | not-exact | Exact mode is not available: no exact kernel exists for STFT. Use ExecutionMode.Fast or remove these operators |
| test_string_concat | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_concat_broadcasting | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_concat_empty_string | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_concat_utf8 | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_concat_zero_dimensional | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_split_basic | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_split_consecutive_delimiters | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_split_empty_string_delimiter | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_split_empty_tensor | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_split_maxsplit | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_string_split_no_delimiter | 20 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnormalizer_export_monday_casesensintive_lower | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnormalizer_export_monday_casesensintive_nochangecase | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnormalizer_export_monday_casesensintive_upper | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnormalizer_export_monday_empty_output | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnormalizer_export_monday_insensintive_upper_twodim | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnormalizer_nostopwords_nochangecase | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_tan | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Tan. Use ExecutionMode.Fast or remove these operators. |
| test_tan_example | 22 | pass | not-exact | Exact mode is not available: no exact kernel exists for Tan. Use ExecutionMode.Fast or remove these operators. |
| test_tfidfvectorizer_tf_batch_onlybigrams_skip0 | 9 | unsupported-op | unsupported-op | no kernel: TfIdfVectorizer (opset 9) |
| test_tfidfvectorizer_tf_batch_onlybigrams_skip5 | 9 | unsupported-op | unsupported-op | no kernel: TfIdfVectorizer (opset 9) |
| test_tfidfvectorizer_tf_batch_uniandbigrams_skip5 | 9 | unsupported-op | unsupported-op | no kernel: TfIdfVectorizer (opset 9) |
| test_tfidfvectorizer_tf_only_bigrams_skip0 | 9 | unsupported-op | unsupported-op | no kernel: TfIdfVectorizer (opset 9) |
| test_tfidfvectorizer_tf_onlybigrams_levelempty | 9 | unsupported-op | unsupported-op | no kernel: TfIdfVectorizer (opset 9) |
| test_tfidfvectorizer_tf_onlybigrams_skip5 | 9 | unsupported-op | unsupported-op | no kernel: TfIdfVectorizer (opset 9) |
| test_tfidfvectorizer_tf_uniandbigrams_skip5 | 9 | unsupported-op | unsupported-op | no kernel: TfIdfVectorizer (opset 9) |
| test_training_dropout | 22 | unsupported-op | unsupported-op | Dropout: training mode is not supported |
| test_training_dropout_default | 22 | unsupported-op | unsupported-op | Dropout: training mode is not supported |
| test_training_dropout_default_mask | 22 | unsupported-op | unsupported-op | Dropout: training mode is not supported |
| test_training_dropout_mask | 22 | unsupported-op | unsupported-op | Dropout: training mode is not supported |
| test_training_dropout_zero_ratio | 22 | unsupported-op | unsupported-op | Dropout: training mode is not supported |
| test_training_dropout_zero_ratio_mask | 22 | unsupported-op | unsupported-op | Dropout: training mode is not supported |
| test_gradient_of_add | 12 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Gradient (opset 1) |
| test_gradient_of_add_and_mul | 12 | unsupported-op | unsupported-op | no kernel: ai.onnx.preview.training.Gradient (opset 1) |
| test_strnorm_model_monday_casesensintive_lower | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnorm_model_monday_casesensintive_nochangecase | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnorm_model_monday_casesensintive_upper | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnorm_model_monday_empty_output | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnorm_model_monday_insensintive_upper_twodim | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_strnorm_model_nostopwords_nochangecase | 10 | unsupported-op | unsupported-op | value 'x' has element type string |
| test_BatchNorm1d_3d_input_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| test_BatchNorm2d_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| test_BatchNorm2d_momentum_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| test_BatchNorm3d_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| test_BatchNorm3d_momentum_eval | 6 | unsupported-op | unsupported-op | no kernel: BatchNormalization (opset 6) |
| test_GLU | 6 | unsupported-op | unsupported-op | no kernel: Mul (opset 6) |
| test_GLU_dim | 6 | unsupported-op | unsupported-op | no kernel: Mul (opset 6) |
| test_Linear | 6 | unsupported-op | unsupported-op | no kernel: Gemm (opset 6) |
| test_PReLU_1d | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| test_PReLU_1d_multiparam | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| test_PReLU_2d | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| test_PReLU_2d_multiparam | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| test_PReLU_3d | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| test_PReLU_3d_multiparam | 6 | unsupported-op | unsupported-op | no kernel: PRelu (opset 6) |
| test_PoissonNLLLLoss_no_reduce | 6 | unsupported-op | unsupported-op | no kernel: Mul (opset 6), Sub (opset 6) |
| test_Softsign | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Div (opset 6) |
| test_operator_add_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| test_operator_add_size1_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| test_operator_add_size1_right_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| test_operator_add_size1_singleton_broadcast | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| test_operator_addconstant | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6) |
| test_operator_addmm | 6 | unsupported-op | unsupported-op | no kernel: Gemm (opset 6) |
| test_operator_basic | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Mul (opset 6) |
| test_operator_mm | 6 | unsupported-op | unsupported-op | no kernel: Gemm (opset 6) |
| test_operator_non_float_params | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Mul (opset 6) |
| test_operator_params | 6 | unsupported-op | unsupported-op | no kernel: Add (opset 6), Mul (opset 6) |
| test_operator_pow | 6 | unsupported-op | unsupported-op | no kernel: Pow (opset 6) |

</details>

## Method

- **Data.** The frozen ONNX 1.22.0 `onnx/backend/test/data` directories `node`, `simple`, `pytorch-converted` and `pytorch-operator`, fetched by `tools/fetch_backend_test.py` at a pinned SHA-256 (the data is never committed). ONNX 1.23 and later no longer ship frozen node cases, so cases for operators newer than that release are not in the data; see the gaps above.
- **Run.** Each case's `model.onnx` is imported and a `Session` is created in the mode under test (`Fast` or `Exact`, one intra-op thread, default optimization level). Every `test_data_set_*` is fed in and every output compared with the recorded output.
- **Tolerance.** Floating-point outputs (float16, bfloat16, float32, float64) pass when each element satisfies `|got - want| <= 1e-7 + 1e-3 * |want|` or both are NaN or equal (this includes infinities); integer and boolean outputs must be equal; shapes and element types must be equal.
- **Status.** `unsupported-op`: no kernel for the operator at that opset, or the operator rejects an attribute value or rank it does not handle. `unsupported-type`: an element type or value kind the library does not have (string, 4-bit, 8-bit float, map; sequences and optionals of tensors are supported). `not-exact`: exact mode only, an operator without an exact kernel. `fail`: anything else that is not a pass.
- **Reading the gaps.** Cases at opset 6 (the legacy pytorch models and a few operator versions) are below the supported range, which starts at opset 7. The rest are listed by cause in the operator table: element types the tensor layer does not have, the `ai.onnx.ml` and training domains, random-number operators and training modes, and the newest preview operators. `known-diff` cases are explained in the runner (`ConformanceRunner.KnownDifferences`) and count as neither pass nor fail.
- **Attribution.** A node case counts under its only operator; a case with several operators (the `_expanded` function bodies) counts under the longest operator name its case name starts with.


<!-- https://recursiveai.org/onemodel/docs/reference-models/ -->

# Reference models

The seven reference models run through `Model.Load(path).Run(...)` or `Session.Run(...)` and are compared with ONNX Runtime's outputs stored by `tools/make_reference.py` (opset-17 exports, files and pins in `tools/models.json`, method in [oracles.md](https://recursiveai.org/onemodel/docs/oracles/)).

## Method

* `tools/check_reference.sh` builds `bench/Bar` and runs `Bar check` per model and per thread count (1 and every core). It is a correctness run, not a timing run.
* **Scaled error** per output: `max|ours − ONNX Runtime| / max|ONNX Runtime|`. The gate is 1e-4. **Element-wise error** is `|ours − ref| / max(|ref|, 1e-6)`; it is reported, not gated, because ONNX Runtime itself differs from its other optimization levels by more than that on tiny elements.
* **Hash**: SHA-256 (first 16 hex digits) of each output's bytes in fast mode, from two runs at each thread count. All must be equal: fast mode does not depend on the thread count or the run.
* `tools/op_inventory.py` lists the operators, attributes and node counts of a set of models and marks those missing from the operator table.

## Operators used (opset 17 exports)

Add, And, Cast, Concat, Constant, ConstantOfShape, Conv, Div, Equal, Erf, Expand, Flatten, Gather, Gemm, GlobalAveragePool, GreaterOrEqual, HardSigmoid, HardSwish, LSTM, LayerNormalization, LessOrEqual, MatMul, MaxPool, Mul, Range, ReduceMean, Relu, Reshape, Resize, Shape, Slice, Softmax, Squeeze, Transpose, Unsqueeze, Where (35 operators; the default domain only).

## Results

Fast mode against ONNX Runtime 1.31.1 (`ORT_ENABLE_ALL`) on the stored reference inputs. Scaled error is the gate (1e-4). "Hash equal" means the output hashes were equal at 1 thread and at every core, and from run to run.

| Model (batch) | Outputs | Max scaled error x64 (AVX-512, 32 cores) | Max scaled error arm64 (NEON, 8 cores) | Max element-wise | Hash equal | Status |
|---|---|---|---|---|---|---|
| ResNet-50 (1) | logits | 1.5e-6 | 1.6e-6 | 1.1e-4 | yes (x64 and arm64 hashes also agree) | pass |
| MobileNetV3-large (1) | logits | 3.7e-6 | 3.0e-6 | 1.6e-3 | yes | pass |
| BERT-base (1, 32) | last_hidden_state | 1.3e-6, 7.9e-6 | not run | 1.3 (tiny elements) | yes | pass |
| MiniLM-L6 (1, 32) | last_hidden_state | 7.6e-7, 1.1e-6 | 7.6e-7, 1.1e-6 | 1.0 (tiny elements) | yes (x64 and arm64 hashes also agree) | pass |
| Whisper-small encoder (1) | encoder_hidden_states | 3.4e-5 | not run | 2.0 (tiny elements) | yes | pass |
| Whisper-small decoder (1, 16 tokens) | logits | 3.5e-6 | not run | 0.64 (tiny elements) | yes | pass |
| CRNN (1, 8) | logits | 6.9e-7, 8.5e-7 | not run | 3.9e-4 | yes | pass |
| DBNet (1, two inputs) | prob, layout, orient | 3.7e-7 | not run | 4.1e-4 | yes | pass |

The Whisper encoder is the largest scaled error (3.4e-5, a third of the gate) and is deterministic; the cause (accumulation order over its 12 attention blocks versus ONNX Runtime's) was not investigated. Element-wise errors above 1e-3 come from outputs near zero, as described in [oracles.md](https://recursiveai.org/onemodel/docs/oracles/).

### Allocation

With one thread, a steady-state `Session.Run` of MiniLM (reused output tensor) allocates 0 bytes (checked by a test when `ONEMODEL_MINILM` names the file). The one-line `Model.Run(tokens)` allocates the output array and, for models with a mask and segment inputs, small input tensors per call.

### Indicative single-thread latency (not a published measurement)

One arm64 laptop-class machine (8 cores, other work running), batch 1, 1 thread, both engines in the same harness, median of 2 repetitions; no weight prepacking, fusion or folding yet.

| Model | Ours (ms) | ONNX Runtime (ms) | Ratio |
|---|---|---|---|
| MiniLM-L6 | 48.8 | 32.1 | 1.52 |
| ResNet-50 | 129.6 | 89.8 | 1.44 |
| MobileNetV3-large | 16.7 | 11.0 | 1.52 |
| BERT-base | 330 | 228 | 1.45 |
| Whisper-small encoder | 4898 | 3684 | 1.33 |

## The one-line happy path on a text encoder

`m.Run(tokens)` works on a model whose inputs are the token ids plus `attention_mask` and `token_type_ids`: the mask is filled with ones (nothing is padding) and the segment ids with zeros, both in the ids' shape; token ids declared `[batch, sequence]` with no shape given are taken as one sequence. Anything else with several inputs goes through `Model.Session.Run` with named tensors.


<!-- https://recursiveai.org/onemodel/docs/oracles/ -->

# Oracles, models and reference data

ONNX Runtime and PyTorch are used as black boxes: they produce reference outputs, export models and quantize them. Their code is never read, copied or linked into the library. Everything below is reproduced by the scripts in `tools/`; paths come from `ONEMODEL_HOME` (default `~/onemodel`), and nothing generated is committed.

## Pins

| Package | Version | Used for |
|---|---|---|
| `onnxruntime` (Python) | 1.31.1 | reference outputs, `onnxruntime.quantization` as a tool |
| `Microsoft.ML.OnnxRuntime` (NuGet, `bench/Bar` only) | 1.31.1 | the bar runner's ONNX Runtime side |
| `onnx` | 1.23.2 | checker, model loading |
| `torch` (CPU wheels) | 2.15.0 | export of the torchvision and Hugging Face models |
| `torchvision` | 0.29.2 | ResNet-50 and MobileNetV3 definitions |
| `transformers` | 5.22.0 | BERT, MiniLM and Whisper definitions |
| `onnxscript` | 0.7.2 | required by the torch exporter package |
| `numpy` | 2.5.4 | array files |
| Python | 3.13 (via `uv`) | |

`tools/requirements-oracles.txt` holds the pins and `tools/setup-oracles.sh` creates the venv (`$ONEMODEL_HOME/venvs/oracles`) on Linux x64 and macOS arm64. It is idempotent, installs `uv` user-locally if missing, uses the CPU wheel index for torch on Linux, and writes the full `pip freeze` next to the venv.

## Models

`tools/models.json` is the manifest: for every model the source (URL or repository plus commit revision), the licence as stated there, the SHA-256 of the source weights, the SHA-256 and size of each produced file. `tools/verify_models.py` checks the cache against it.

| Model | Source | Files |
|---|---|---|
| ResNet-50 | torchvision `IMAGENET1K_V1` weights (pinned URL and hash) | fp32, int8 static QDQ |
| MobileNetV3-Large | torchvision `IMAGENET1K_V1` weights | fp32, int8 static QDQ |
| BERT-base | `google-bert/bert-base-uncased` at a pinned commit | fp32, int8 dynamic |
| MiniLM-L6 | `sentence-transformers/all-MiniLM-L6-v2` at a pinned commit | fp32, int8 dynamic |
| Whisper-small | `openai/whisper-small` at a pinned commit | encoder and decoder, each fp32 and int8 dynamic |
| CRNN recogniser `crnn-v0` 0.7.3 | fp16 `.onm` container (family-owned; fetch-only) | fp32 |
| DBNet-style detector `dbnet-lite-v1` 0.3.2 | fp16 `.onm` container (family-owned; fetch-only) | fp32 |

Export (`tools/export_models.py`): TorchScript-based `torch.onnx.export`, opset 17, weights embedded in one file, eval mode, dynamic batch (and sequence for the text models). Encoders output `last_hidden_state` only, with inputs `input_ids`, `attention_mask`, `token_type_ids` (attention is the explicit, eager form). The Whisper encoder takes `input_features` `[B,80,3000]`; the decoder takes `input_ids` and `encoder_hidden_states` and returns logits for the whole sequence without a key/value cache. These are fixed choices of this repository, not the upstream defaults.

### Reproducibility across machines

Exported on x64 Linux and on arm64 macOS with the same pins, BERT-base, MiniLM and Whisper (fp32 and int8) are **byte-identical**. The ResNet-50 and MobileNetV3 exports are **not** (same size, different bytes: the cause was not investigated; the likely one is constant folding of batch-norm into the convolution weights, which is floating-point arithmetic done on the exporting CPU), and their static int8 files follow. The manifest pins the x64 file, and other machines copy it instead of re-exporting; `export_models.py` prints a note when its output differs from the pinned hash.

### The OCR models (`tools/export_oneocr.py`)

Built from their containers with the family's training code imported read-only (the paths come from arguments or `ONEOCR_*` environment variables; the detector module's unused post-processing imports are stubbed so only torch and NumPy are needed). Opset 17, fp32, no CTC in the graph.

* Recogniser: input `[B,1,32,W]` (dynamic B and W), output `logits [B, W//4, 622]`. Ops: Conv 6, Relu 6, MaxPool 5, Transpose 5, LSTM 2 (bidirectional), MatMul 1, Add 1, plus Shape/Gather/ConstantOfShape/Slice/Squeeze/Unsqueeze/Reshape/Concat/Constant plumbing.
* Detector: input `[1,1,H,W]` (dynamic H, W, multiples of 32), outputs `prob [1,1,H/2,W/2]`, `layout [1,8,H/8,W/8]`, `orient [1,4]`. Ops: Conv 61 (depthwise ones with `group = C`), Relu 53, Add 5, Resize 5 (nearest ×2), ReduceMean 2, Concat 1, Gemm 1.

Against the fixtures' fp32 reference tensors (50 recogniser lines; 6 detector inputs; metrics defined in the script's header), maximum over all inputs:

| Output | PyTorch module: abs / rel | ONNX Runtime on the file: abs / rel | Fixtures' own fp32-vs-fp64 floor: abs / rel |
|---|---|---|---|
| recogniser logits | 3.6e-5 / 1.3e-3 | 3.8e-5 / 1.7e-3 | 9.4e-5 / 2.0e-3 |
| detector prob | 2.9e-6 / 1.1e-4 | 6.7e-6 / 2.1e-4 | 7.1e-6 / 1.3e-4 |
| detector layout | 3.8e-5 / 2.6e-5 | 5.3e-5 / 3.2e-5 | 4.2e-5 / 2.1e-5 |
| detector orient | 7.6e-6 / 4.4e-7 | 1.9e-5 / 3.8e-6 | 1.5e-5 / 3.9e-6 |

ONNX Runtime stays within 1.7× of the fixtures' floors on every output. Reference outputs are written for the recogniser at batch 1 on five fixture crops (`b1-<crop id>`) and at batch 8 (`b8`: eight crops right-padded with white to width 128), and for the detector at batch 1 on `tile0` and `wide` (`b1-tile0`, `b1-wide`), so a bar row is e.g. `crnn:fp32:8` or `dbnet:fp32:1-wide`. The exports are byte-stable on repeat runs on the same machine.

## int8 variants (`tools/quantize_models.py`)

* Encoders and Whisper: `quantize_dynamic`, int8 weights, per-channel, activations quantized at run time.
* ResNet-50 and MobileNetV3: `quantize_static`, QDQ format, int8 weights per-channel, int8 activations, MinMax calibration over 32 images. No licensed image set is fetched: the calibration images are seeded synthetic fields (a 7×7×3 normal field expanded 32× plus noise of 0.25), generated by `calibration_images()`. They exercise the quantized kernels but do not calibrate for accuracy.

## ONNX backend test data (`tools/fetch_backend_test.py`)

Only `onnx/backend/test/data/` is extracted from the archive. ONNX 1.23 stopped shipping the node cases (they are generated in memory by the installed package, with expected values from the package itself), so the pinned source is the **1.22.0** release archive, the last with frozen node data. Cases added after 1.22.0 are therefore not covered.

| Directory | Cases |
|---|---|
| `node` | 1765 |
| `pytorch-converted` | 82 |
| `pytorch-operator` | 35 |
| `simple` | 23 |
| `light` (flat model files) | 9 |
| `real` (need model-zoo downloads, not fetched) | 9 |

## Reference outputs (`tools/make_reference.py`)

For every model file and batch size (1 and the model's N: 32 text encoders, 8 conv models, 1 for Whisper) it writes, under `reference/<os>-<arch>/<model>/<variant>/b<batch>/`:

* `inputs/*.npy`, `outputs/*.npy`: NumPy format 1.0, little-endian, C order;
* `index.json`: dtypes, shapes, SHA-256 of each file, the model's SHA-256, the ONNX Runtime version and session options.

Outputs are ONNX Runtime `ORT_ENABLE_ALL`, sequential execution, 1 intra-op thread. Inputs are seeded per sample from a SHA-256 of `(model, kind, sample index)` through PCG64, so batch 1 is sample 0 of batch N on every machine. Conv inputs are standard normal `[3,224,224]`; text inputs are random token ids in `[1000,30000)` with sequence length 128, sample 0 unpadded and the others padded to a random length in `[64,128]`; Whisper uses a normal mel input scaled by 0.5, and the decoder 16 random token ids with a random hidden state.

Reference outputs are per machine class: ONNX Runtime's results differ in the last bits between x64 and arm64.

### ONNX Runtime's own noise: `ORT_ENABLE_ALL` against `ORT_DISABLE_ALL`

`noise.csv` has, per output, three numbers, with `a` = enabled and `b` = disabled: max |a−b|; that divided by max |b| ("scaled"); and max |a−b| / max(|b|, 1e-3·max|b|) ("floored relative"). Results (1 thread):

| Model / variant | Batch | x64 abs | x64 scaled | x64 floored rel | arm64 scaled | arm64 floored rel |
|---|---|---|---|---|---|---|
| ResNet-50 fp32 | 1 / 8 | 1.7e-6 / 1.9e-6 | 2.0e-7 / 2.2e-7 | 3.6e-5 / 6.9e-5 | 2.9e-7 / 2.8e-7 | 2.4e-5 / 5.8e-5 |
| MobileNetV3 fp32 | 1 / 8 | 5.1e-6 / 8.2e-6 | 9.6e-7 / 1.4e-6 | 4.2e-4 / 3.8e-4 | 8.6e-7 / 1.5e-6 | 6.0e-4 / 7.2e-4 |
| BERT-base fp32 | 1 / 32 | 7.3e-6 / 1.3e-5 | 1.1e-6 / 1.5e-6 | 3.9e-4 / 3.3e-4 | 8.5e-7 / 1.9e-6 | 1.1e-4 / 4.3e-4 |
| MiniLM-L6 fp32 | 1 / 32 | 1.3e-6 / 3.8e-6 | 2.2e-7 / 6.1e-7 | 1.8e-4 / 2.5e-4 | 2.8e-7 / 5.8e-7 | 1.2e-4 / 1.9e-4 |
| Whisper encoder fp32 | 1 | 2.1e-4 | 6.0e-6 | 7.4e-4 | 2.2e-5 | 3.2e-3 |
| Whisper decoder fp32 | 1 | 4.9e-5 | 1.9e-6 | 1.1e-3 | 2.5e-6 | 1.2e-3 |

So for fp32, **a 1e-4 relative bound is only meaningful against the scale of the output** (scaled error): ONNX Runtime itself is within ~2e-6 of its unoptimized self on that metric, but an element-wise relative error floored at 1e-3 of the maximum already reaches 1e-4 to 3e-3 between its two optimization levels, because small elements are computed as differences of large terms. The fp32 conformance metric should therefore be the scaled one (or an element-wise one with a floor at least 1e-2 of the maximum).

int8 outputs are much noisier. Between ONNX Runtime's own levels the scaled difference is 7e-3 to 2e-2 (ResNet-50), 8e-2 to 1e-1 (MobileNetV3), 6e-2 to 2e-1 (BERT-base dynamic), 3e-2 to 7e-2 (MiniLM dynamic, x64), and 0.7 for the dynamically quantized Whisper encoder (the dynamic scheme is sensitive to activation outliers). Comparing int8 against ONNX Runtime's int8 output element by element is not a stable metric, and neither is "at most ONNX Runtime's own difference": ONNX Runtime's optimization levels agree to 2e-7 on the dynamically quantized Whisper decoder only because their float kernels are identical, yet rerunning ONNX Runtime itself with 1e-7 relative noise on one input moves that output by 0.10 to 0.21 scaled, with minimum row cosines of 0.95 to 0.99 and 14 to 16 of 16 token decisions unchanged (`tools/int8_sensitivity.py`). One float32 ulp upstream flips a few 8-bit roundings, and the flips spread.

### The int8 metric

A row (model, variant, batch) passes when, for every float output compared with ONNX Runtime's `ORT_ENABLE_ALL` output:

1. **Cosine.** The 1st-percentile cosine similarity over the rows of the last axis (the minimum when there are fewer than 100 rows) is at least `min(0.99, f)`, and for outputs of rank 3 or more (encoders) so is the minimum cosine of the mean-pooled vectors (mean over the second-to-last axis). `f` is the floor defined below. A single extreme row is not gated: with thousands of rows it is an extreme-value statistic that ONNX Runtime itself does not hold from run to run.
2. **Decisions**, for models with a task head only: class per image for classifiers, token per position for the Whisper decoder and the text recognizer. The argmax over the last axis agrees on at least 95% of positions, or on at least as many as the floor agrees. A single image (one decision) must agree. Argmax over a hidden dimension is not a decision, so encoders have no part 2.

The floor `f` of each statistic (and of the decision count) is the smallest value, measured against `ORT_ENABLE_ALL`, over: ONNX Runtime's `ORT_DISABLE_ALL` output; 16 `ORT_ENABLE_ALL` runs with 1e-7 relative noise (about one float32 ulp) on the float inputs; and, for dynamic models whose first quantized input is not a graph input (token-id encoders), 16 runs with the same noise on that tensor. All of them measure ONNX Runtime's own sensitivity, and the worst one is the honest floor. See `int8.md`, "Noise floor of the model-level metric". (For the Whisper decoder the two levels agree to 2e-7, so the noise runs set the floor.)

Scaled error and ONNX Runtime's own difference are printed, not gated. Operator-level int8 (the oracle cases) must match bit for bit. `tools/check_int8.sh` runs `Bar check --dump` and `tools/check_int8.py` on any rows; `Bar check` itself prints the per-output row-cosine minimum and top-1 agreement.

## float16 files

`tools/export_models.py --fp16` writes `<model>_fp16.onnx` next to the float32 file (MiniLM-L6 and ResNet-50): float16 weights and
float16 activations, float32 graph inputs and outputs with a `Cast` at each end, finite values beyond the float16 range clamped to
±65504. `make_reference.py --variant fp16` stores ONNX Runtime's output on the same file and inputs. Both engines run it on the CPU.

Gate: scaled error of ours against ONNX Runtime on the same file ≤ 4e-3, and our error against the float32 model's output ≤ 1.5 times
ONNX Runtime's. Reason: half precision has 11 significant bits (4.9e-4 per rounding), the models chain dozens of roundings, and two
correct engines that round in slightly different places differ by that much. Measured on x64 (AVX-512), fast mode, scaled errors:

| Model (batch) | Ours vs ORT fp16 | ORT fp16 vs ORT fp32 | Ours fp16 vs ORT fp32 | ORT fp16, enabled vs disabled |
|---|---|---|---|---|
| MiniLM-L6 (1) | 9.8e-4 | 7.9e-4 | 1.05e-3 | 6.6e-4 |
| MiniLM-L6 (32) | 1.9e-3 | 1.33e-3 | 1.72e-3 | 1.26e-3 |
| ResNet-50 (1) | 3.6e-4 | 1.37e-3 | 1.02e-3 | 2.9e-7 |
| ResNet-50 (8) | 4.8e-4 | 1.49e-3 | 1.75e-3 | 2.7e-7 |

Output hashes are equal at 1 and 8 threads.


<!-- https://recursiveai.org/onemodel/docs/bench-method/ -->

# Benchmark and golden-hash method

## The bar runner (`bench/Bar`)

A row compares two engines on one model file with the same inputs in the same harness: ONNX Runtime (the `Microsoft.ML.OnnxRuntime` package pinned in `docs/oracles.md`) and this library (a placeholder until its session exists).

**What is timed.** One `Run` call with inputs already bound and outputs left in the engine's own buffers. Session creation is timed separately and reported as `load_ms` (no gate). The first call is reported as `first_run_ms`.

**Procedure.** One process per row and engine. Load, 1 cold call, 3 untimed warm-up calls, then `reps` repetitions (default 3) of as many back-to-back calls as fit in about one second (at most 1000). A repetition's value is its mean time per call; the row reports the median of the repetitions, their min and max, and `spread_pct = (max − min) / median`. Timing uses `Stopwatch`; `alloc_bytes_per_iter` is the managed allocation on the timing thread per call (native allocations inside ONNX Runtime are not counted, which is what that column means for that engine).

**Matching settings.** ONNX Runtime runs at `ORT_ENABLE_ALL`, sequential execution, intra-op threads = N, inter-op = 1. The other engine gets the same N. N is 1 or "all", where "all" is the number of logical processors the OS reports (including efficiency cores and SMT siblings). Batch sizes are 32 for the text encoders, 8 for the conv models, and 1 or as the brief says otherwise.

**Output check.** After timing, the engine's last outputs are compared with the reference outputs for the same inputs (`check_scaled_err`: max |a−b| / max |b|). A fast row that gives wrong answers is not a result.

**Peak RSS.** The process of one row is wrapped by `/usr/bin/time -v` (Linux, "Maximum resident set size") or `/usr/bin/time -l` (macOS, bytes); the number goes into `peak_rss_mb`. Each engine is therefore measured in its own process, harness included.

**Machine record.** Every row carries: machine name, CPU model, the ISA features the runtime reports (`sse4.2 avx2 fma avx512f avx512bw avx512dq avx512vbmi avxvnni avx512vnni`, or `neon dotprod`), logical core count, OS, SDK, runtime, engine versions, and the 1-minute load average before the run (`load_before`), at the end of timing (`load_during`) and right after the process exits (`load_after`). The one-minute average lags real load by tens of seconds, so a short row's three readings are close to each other even if something started halfway; the benchmark lock below is the real guard. The lock's label is stored in `lock_label`.

**Pass criteria** (for rows with both engines): time ≤ 1.5× ONNX Runtime's passes, ≤ 1.15× meets the target; peak RSS ≤ 1.5×; load time is reported. A row measured while another heavy job was running on the machine is not published.

**Driver.** `bench/Bar/run-bar.sh` builds the runner, then claims the benchmark lock file atomically from inside the same script (`noclobber`, label `onemodel bar <what> <start> <expected end>`) and removes it on exit; it refuses to start if the lock exists or the 1-minute load exceeds half the cores. It takes `--rows "model:variant:batch ..."` and `--threads "1 all"` and appends to `bar.csv` and `bar.md`. The lock path comes from `ONEMODEL_LOCK` (default `~/WORKER-BENCH.lock`). Model files and reference directories are resolved from `tools/models.json` by `tools/model_path.py`.

**Kernel micro-rows.** `Timing.Measure` in `bench/Bar/Timing.cs` is the same warm-up/repetition/median routine as a reusable function, so kernel spikes report numbers from the same method.

## The golden-hash runner (`tools/golden_hash.py`)

Runs one command under a set of configuration rows and compares the SHA-256 lines it prints (`<name> <64 hex>` per output):

| Row | Setting |
|---|---|
| `jit-default` | the runtime's default ISA selection, all cores |
| `jit-1thread` | `ONEMODEL_THREADS=1` (the program decides what that means) |
| `jit-no-avx512` / `jit-no-avx2` (x64) | `DOTNET_EnableAVX512F=0` / `DOTNET_EnableAVX2=0` |
| `jit-no-dotprod` (arm64) | `DOTNET_EnableArm64Dp=0` |
| `jit-no-hwintrinsic` | `DOTNET_EnableHWIntrinsic=0` (scalar and software fallbacks) |
| `aot-default`, `aot-1thread` | a NativeAOT build of the same program (`--aot-cmd`) |
| anything else | `--row NAME:ENV:COMMAND`, for example a browser-WASM host |

`--save` stores one machine's results as JSON and `--merge` combines several machines into one table with an `equal`/`UNEQUAL` verdict per output; the exit status is 1 if any output differs. `tools/GoldenDemo` is a tiny program with two outputs that must agree everywhere (a four-lane dot product without FMA, a fixed-chunk sum combined in chunk order) and two deliberate failures (a fused multiply-add dot product and a sum whose chunk size depends on the thread count), to show the runner catches both.

NativeAOT note: an AOT build targets the baseline instruction set unless told otherwise, so it takes the non-AVX2 path of the demo (`aot-default` matched `jit-no-avx2`, not `jit-default`). Anything that depends on the ISA path therefore differs between the JIT and AOT rows by construction; exact kernels must not.


<!-- https://recursiveai.org/onemodel/docs/docs-map/ -->

# Documentation

Normative, language-neutral notes live here, one file per topic, so a port can follow them without the C# code.

| Topic | File |
|---|---|
| Tensors, memory, layouts, fp16/bf16 rounding, gradient records | [tensors.md](https://recursiveai.org/onemodel/docs/tensors/) |
| Exact mode: definition and accumulation order | [exact-mode.md](https://recursiveai.org/onemodel/docs/exact-mode/) |
| Exact mode golden hashes: rows, results, cost of exactness | [golden-hashes.md](https://recursiveai.org/onemodel/docs/golden-hashes/) |
| Matrix product and convolution kernels | [kernels-gemm.md](https://recursiveai.org/onemodel/docs/kernels-gemm/) |
| Attention and softmax kernels | [kernels-attention.md](https://recursiveai.org/onemodel/docs/kernels-attention/) |
| exp, log, tanh, erf, sigmoid, GELU: polynomials and error | [exact-transcendentals.md](https://recursiveai.org/onemodel/docs/exact-transcendentals/) |
| Reference oracles, pinned versions, comparison metrics | [oracles.md](https://recursiveai.org/onemodel/docs/oracles/) |
| Benchmark method | [bench-method.md](https://recursiveai.org/onemodel/docs/bench-method/) |
| Depthwise convolution: phase planes, lane layout, bits | [kernels-depthwise.md](https://recursiveai.org/onemodel/docs/kernels-depthwise/) |
| Half-precision storage: conversions, the widening adapter, compact weights | [fp16.md](https://recursiveai.org/onemodel/docs/fp16/) |
| ONNX import: protobuf reading, tensors, graph resolution, external data, errors | [onnx-import.md](https://recursiveai.org/onemodel/docs/onnx-import/) |
| Operator contract, plans, sessions, memory reuse | [graph-runtime.md](https://recursiveai.org/onemodel/docs/graph-runtime/) |
| Reference models: method, operators used, results | [reference-models.md](https://recursiveai.org/onemodel/docs/reference-models/) |
| Graph passes: what each rewrites, levels, bit preservation | [graph-passes.md](https://recursiveai.org/onemodel/docs/graph-passes/) |
| Normalisation, softmax, reduction, arg-reduction and cumulative-sum operators | [ops-norms.md](https://recursiveai.org/onemodel/docs/ops-norms/) |
| Convolution, pooling, normalisation and resize operators: supported cases, edge rules, bits | [ops-conv.md](https://recursiveai.org/onemodel/docs/ops-conv/) |
| Shape and indexing operators: semantics, ONNX Runtime findings | [ops-shape-indexing.md](https://recursiveai.org/onemodel/docs/ops-shape-indexing/) |
| Recurrent operators (RNN, GRU, LSTM): order of operations, open choices | [ops-recurrent.md](https://recursiveai.org/onemodel/docs/ops-recurrent/) |
| CTC decoding: greedy and prefix beam search, confidences | [ctc.md](https://recursiveai.org/onemodel/docs/ctc/) |
| Element-wise operators, activations, comparisons, Where, Cast | [ops-elementwise.md](https://recursiveai.org/onemodel/docs/ops-elementwise/) |
| int8: operators, product kernel, requantization, QDQ rewrite | [int8.md](https://recursiveai.org/onemodel/docs/int8/) |
| The OneModel model file: header, JSON manifest, aligned tensors, reading and writing rules (version 1, unstable until 1.0) | [format.md](https://recursiveai.org/onemodel/docs/format/) |
| Conformance: onnx/backend/test results per operator, method, gaps | [conformance.md](https://recursiveai.org/onemodel/docs/conformance/) |
| Operators added for coverage: settled semantics, exactness | [ops-coverage.md](https://recursiveai.org/onemodel/docs/ops-coverage/) |


<!-- https://recursiveai.org/onemodel/docs/changelog/ -->

# Changelog

All notable changes are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow [Semantic Versioning](https://semver.org/).

## [Unreleased]

## [0.3.0-preview.1] - 2026-10-30

Our own model file, ONNX backend-test conformance with control flow, and exact-mode results checked bit for bit across platforms. Of the 1905 `onnx/backend/test` cases, 1580 pass in fast mode and 1382 in exact mode, none fail, and the rest are listed gaps (`docs/conformance.md`). Exact-mode output hashes are equal on Windows, Linux and macOS (x64 and arm64, JIT and NativeAOT) and on browser WebAssembly (`docs/golden-hashes.md`).

### Added

- Exact-mode golden hashes: `tools/GoldenModels`, `tools/golden*.sh`, and a CI check on every platform against `testdata/graphs/golden-exact.txt`.
- The OneModel model file (`ModelFormat`, `docs/format.md`; format version 1, unstable until 1.0): a 64-byte header, a JSON manifest of the graph and 64-byte-aligned raw weights that load as memory-mapped views. `ModelFormat.Convert` turns an ONNX file into one, `ModelFormat.Save` writes an imported model, and `Model.Load` / `Session.Load` open either kind by its first eight bytes. A converted model gives bit-identical outputs. Bad files fail with `ModelFormatException` naming the field.
- `OnnxSweep conformance` and `tools/conformance.sh`: one runner over `onnx/backend/test` (node, simple, pytorch-converted, pytorch-operator) in fast and exact mode, writing per-case status and `docs/conformance.md`.
- About 70 operators for backend-test coverage (activations and trigonometry, bitwise, scatter and set operators, loss functions, LRN, Einsum, DFT/STFT and windows, GridSample, RoiAlign, NonMaxSuppression, Attention opset 23 and others; `docs/ops-coverage.md`), ConvTranspose and Conv with three spatial axes, Resize `antialias`, MelWeightMatrix and IsNaN.
- Control flow and collections: `If`, `Loop`, `Scan`, the sequence operators, `SequenceMap` and the optional operators run as ONNX defines them. Sessions carry sequences and optionals as packed `uint8` tensors (`PackedValue`, `TensorInfo.Kind`); bodies run as nested sessions (`docs/graph-runtime.md` section 8).

### Changed

- int8 packing narrows through value-preserving types, so results agree on WebAssembly, where vector narrowing saturates in compiled code and truncates in the interpreter.
- The int8 model-level metric adds a noise floor on the first quantized tensor of token-input encoders (`docs/int8.md`).
- RoiAlign without `coordinate_transformation_mode` defaults to `half_pixel` from opset 16; Squeeze and Unsqueeze accept a scalar `axes`.

## [0.2.0-preview.1] - 2026-10-30

Graph optimizations, int8 and half-precision storage. All reference models still match ONNX Runtime within 1e-4 scaled error in float32 with outputs independent of thread count; exact-mode results are unchanged by the new passes.

### Added

- Graph optimizations, chosen by `SessionOptions.OptimizationLevel` (`None`, `Basic`, `All`; `All` is the default): constant folding, matrix-product weights packed once per session with bias and activation applied per tile, convolution with a following activation, Conv+BatchNormalization folding, Gelu and attention fusion. Passes that change rounding are skipped in exact mode (`docs/graph-passes.md`).
- int8 inference: `QuantizeLinear`, `DequantizeLinear`, `DynamicQuantizeLinear`, `MatMulInteger`, `ConvInteger`, `QLinearMatMul`, `QLinearConv`; exact int32 product kernels for AVX2, AVX-512 and Arm dot product with a scalar fallback; the `qdq` and `matmul-integer-fusion` passes. See `docs/int8.md`.
- Half-precision storage: models whose weights or values are float16 or bfloat16 run (each operator widens its inputs to float32 and rounds its results back; `docs/fp16.md`). `SessionOptions.WeightStorage` keeps large MatMul, Gemm and Conv weights of float32 models as float16 or bfloat16 to halve their memory.
- `tools/export_models.py --fp16` makes float16 variants of MiniLM-L6 and ResNet-50; `Bar` takes `--weights original|fp16|bf16`.
- A direct depthwise convolution (phase planes for strides, 16/8/4/1-lane paths, four rows at a time for narrow maps) replaces the first one: 3-8x faster on MobileNetV3's layers (`docs/kernels-depthwise.md`).

### Changed

- Matrix products, convolutions and the normalisation operators use no more threads than their work can use (about 80 microseconds of work per thread), so small shapes no longer run slower on many threads; the parallel-for allocates nothing per call.

## [0.1.0-preview.1] - 2026-10-29

First preview: ONNX models load and run on the CPU through `Model.Load(...).Run(...)` or a `Session`, with no native dependencies. The reference set (ResNet-50, MobileNetV3, BERT-base, MiniLM-L6, Whisper-small encoder and decoder, and two text-recognition models) matches ONNX Runtime within 1e-4 scaled error, with outputs independent of thread count. No graph optimizations, int8 or fp16 compute yet; speed is not tuned.

### Added

- Shape and indexing operators: Shape, Size, Squeeze, Unsqueeze, Flatten, Dropout (inference), Transpose, Concat, Split, Slice, Expand, Tile, Gather, GatherElements, GatherND, ScatterND, Pad, ConstantOfShape, Range, OneHot, Trilu, NonZero and TopK. All move or compare elements only, so they are exact in both modes.
- Recurrent operators RNN, GRU and LSTM (all directions, both layouts, sequence lengths, initial state, peepholes, clipping, activation lists; fast and exact) and `Ctc`, greedy and prefix-beam CTC decoding for consumers of recognisers and speech models.
- Element-wise operators: Sub, Div, Mod, Pow, Max, Min, Sum, Mean, Exp, Log, Tanh, Sigmoid, Erf, Gelu, HardSigmoid, HardSwish, LeakyRelu, Neg, Abs, Sign, Reciprocal, Sqrt, Floor, Ceil, Round, Clip, the comparisons, And, Or, Xor, Not, Where, Cast and CastLike; all but Pow have exact kernels.
- The graph runtime: operator contract and static operator table, shape inference and constant evaluation per set of input shapes, execution plans with liveness-based buffer reuse, thread-safe sessions with fast and exact modes, and the `Model` happy path. Operators so far: Add, Mul, Relu, MatMul, Gemm, Reshape, Identity, Constant.
- `Bar check` and `tools/check_reference.sh` compare the library with stored ONNX Runtime outputs and hash outputs across thread counts; `tools/op_inventory.py` lists the operators a set of models uses. The happy path fills in `attention_mask` and `token_type_ids` for text encoders, so `m.Run(tokens)` works on them.
- Convolution and pooling operators: Conv (1-D and 2-D, groups, depthwise, dilation, `auto_pad`, bias), ConvTranspose, MaxPool (with Indices), AveragePool, GlobalAveragePool, GlobalMaxPool, BatchNormalization (inference), Resize and Upsample. `Conv` kernel jobs no longer allocate per call.
- The bar runner can time this library (`--side ours --mode fast|exact`).
- Operators: LayerNormalization, RMSNormalization, InstanceNormalization, GroupNormalization, Softmax, LogSoftmax, Hardmax, the Reduce* family, ArgMax, ArgMin and CumSum, all with exact kernels (`docs/ops-norms.md`).
- The solution skeleton: library, tests, NativeAOT and browser-wasm smoke apps, micro-benchmarks and a fuzz harness.
- Tensor layer: data types with fp16/bf16 conversion, strided views, NCHW/NHWC/blocked layouts, a pooled allocator and arena, and gradient records (public, unstable until the first preview).
- Fast and exact fp32 matrix-product, convolution, attention and softmax kernels, exact transcendental functions, and a fixed-chunk thread pool (internal for now).
- Tooling: reference-model export and pinning, reference outputs from ONNX Runtime, cross-machine output hashing, and a benchmark runner against ONNX Runtime.
- ONNX import: a protobuf reader and ONNX model reader of our own (IR versions 3 and later, any opset, external data confined to the model's directory, memory-mapped weights), and the graph representation it builds. Bad input produces `OnnxImportException`.


<!-- https://recursiveai.org/onemodel/cookbook/exact-mode/ -->

# Exact mode: the same bits everywhere

Run a model in OneModel’s exact mode for results that are bit-identical on every OS, CPU, instruction set, thread count and WebAssembly, and see how it differs from the default fast mode.

The model is `wide.onnx` from [the sample models](https://recursiveai.org/onemodel/docs/getting-started/#the-sample-models): 256 inputs, a 512-wide hidden layer and 64 outputs. The program runs it in both modes at three thread counts and hashes the output bytes:

```csharp
using System.Security.Cryptography;
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;

float[] input = new float[256];
for (int i = 0; i < input.Length; i++) input[i] = MathF.Sin(i * 0.37f);

foreach (var mode in new[] { ExecutionMode.Fast, ExecutionMode.Exact })
{
    foreach (int threads in new[] { 1, 4, 16 })
    {
        var options = new SessionOptions { Mode = mode, IntraOpThreads = threads };
        using var model = Model.Load("wide.onnx", options);
        float[] y = model.Run(input);
        string hash = Convert.ToHexString(SHA256.HashData(MemoryMarshalBytes(y)))[..16];
        Console.WriteLine($"{mode,-5} threads={threads,-2} y[0]={y[0]:R}  sha256={hash}");
    }
}

static byte[] MemoryMarshalBytes(float[] v) =>
    System.Runtime.InteropServices.MemoryMarshal.AsBytes(v.AsSpan()).ToArray();
```

```text
Fast  threads=1  y[0]=-0.4711962  sha256=53778A7B3BF644BF
Fast  threads=4  y[0]=-0.4711962  sha256=53778A7B3BF644BF
Fast  threads=16 y[0]=-0.4711962  sha256=53778A7B3BF644BF
Exact threads=1  y[0]=-0.47119626  sha256=FF8C37256201B5B4
Exact threads=4  y[0]=-0.47119626  sha256=FF8C37256201B5B4
Exact threads=16 y[0]=-0.47119626  sha256=FF8C37256201B5B4
```

## What the two modes promise

- **Fast** (the default) is deterministic on one machine and kernel path: the thread count never changes a result, because reductions use fixed chunk sizes. Another CPU, with other kernels or fused multiply-add, can round differently in the last bits.
- **Exact** gives the same bits everywhere. Every exact kernel uses only IEEE `+ - * /` and square root, one accumulation order per output, no fused multiply-add, OneModel’s own `exp`, `log`, `tanh` and `erf`, and one canonical NaN. The rules are in [exact mode](https://recursiveai.org/onemodel/docs/exact-mode/), and the cross-platform check (Windows, Linux and macOS on x64 and arm64, JIT and NativeAOT, and browser WebAssembly) in [golden hashes](https://recursiveai.org/onemodel/docs/golden-hashes/).

The two modes agree here to seven digits, but they are not the same bits: choose one and keep it for every machine that has to agree.

## Good to know

- Exact mode costs time: on the float32 reference models it takes up to about 2.7 times fast mode’s time, mostly because fused rewrites don’t run and the matrix product can’t fuse multiply and add ([golden hashes](https://recursiveai.org/onemodel/docs/golden-hashes/#exact-versus-fast-indicative)). That is why fast is the default.
- An operator with no exact kernel makes `Model.Load` fail in exact mode with `SessionException` (`ExactModeUnavailable`) naming it, rather than silently approximating: see [When a model won’t load](https://recursiveai.org/onemodel/cookbook/missing-operators/).
- Graph passes that would change rounding are skipped in exact mode ([graph passes](https://recursiveai.org/onemodel/docs/graph-passes/)), so `OptimizationLevel` never moves an exact result.


<!-- https://recursiveai.org/onemodel/cookbook/convert-to-onemodel-format/ -->

# Convert ONNX to the OneModel format

Convert an ONNX file to OneModel’s own format once, load it with the same call, and get bit-identical outputs, with the weights read as memory-mapped views.

```csharp
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Format;

ModelFormat.Convert("wide.onnx", "wide.onemodel");
Console.WriteLine($"wide.onnx      {new FileInfo("wide.onnx").Length,8:N0} bytes");
Console.WriteLine($"wide.onemodel  {new FileInfo("wide.onemodel").Length,8:N0} bytes");
Console.WriteLine($"OneModel format? {ModelFormat.IsModelFormatFile("wide.onemodel")}");

float[] input = Enumerable.Range(0, 256).Select(i => i / 256f).ToArray();
using var fromOnnx = Model.Load("wide.onnx");
using var fromOwn = Model.Load("wide.onemodel");
float[] a = fromOnnx.Run(input), b = fromOwn.Run(input);

bool same = a.Zip(b).All(p => BitConverter.SingleToInt32Bits(p.First) == BitConverter.SingleToInt32Bits(p.Second));
Console.WriteLine($"Bit-identical outputs: {same}");

File.WriteAllBytes("broken.onemodel", File.ReadAllBytes("wide.onemodel")[..4000]);
try { Model.Load("broken.onemodel"); }
catch (ModelFormatException e) { Console.WriteLine($"{e.Kind}: {e.Message}"); }
```

```text
wide.onnx       655,516 bytes
wide.onemodel   656,320 bytes
OneModel format? True
Bit-identical outputs: True
Malformed: the header says the file is 656320 bytes; it is 4000 (truncated or extended) (broken.onemodel at byte 48, file_length)
```

## The format

A `.onemodel` file is a 64-byte header, a JSON manifest of the graph, and the raw weights, each tensor aligned to 64 bytes so it loads as a view of a memory map instead of being copied out of protobuf. What is saved is the imported graph before any optimization, so a converted model optimizes and runs exactly as the ONNX file it came from. The layout is in [the OneModel format](https://recursiveai.org/onemodel/docs/format/).

- `Model.Load` and `Session.Load` tell the two kinds apart by their first eight bytes (`ONEMODEL`), so the loading code doesn’t change.
- `ModelFormat.Save(model, path)` writes a model you already imported; `ModelFormat.Load` returns its `IrModel`.
- A damaged file fails with `ModelFormatException`, whose `Kind`, `Offset` and `FieldPath` say what is wrong and where.

## Good to know

- The format is version 1 and unstable until OneModel 1.0: a version 1 file may stop loading in a later preview. Keep the ONNX file and convert again after you update.
- On a model this small the file sizes are about the same. The gain is in loading: aligned weights are mapped, not parsed and copied.


<!-- https://recursiveai.org/onemodel/cookbook/named-inputs-and-outputs/ -->

# Named inputs and outputs

Run an ONNX model with more than one input or output through OneModel’s `Session`: list its inputs and outputs, pass named tensors in any order, and get the outputs back as tensors.

The model is `scorer.onnx` from [the sample models](https://recursiveai.org/onemodel/docs/getting-started/#the-sample-models): `score = features × W + bias` and `best`, the largest score of each row.

```csharp
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Tensors;

using var session = Session.Load("scorer.onnx");
foreach (var i in session.Inputs) Console.WriteLine($"in  {i.Name}: {i.DType} [{string.Join(", ", i.Dims)}]");
foreach (var o in session.Outputs) Console.WriteLine($"out {o.Name}: {o.DType} [{string.Join(", ", o.Dims)}]");

using var features = Tensor.FromSpan<float>([1f, 2f, 3f, 4f, 5f, 6f], Shape.Create(2, 3));
using var bias = Tensor.FromSpan<float>([0.5f, 1.5f], Shape.Create(1, 2));

var outputs = new Tensor[2];   // empty: the session allocates them, and we dispose them
session.Run(["bias", "features"], [bias, features], ["score", "best"], outputs);
using Tensor score = outputs[0], best = outputs[1];

Console.WriteLine($"score {score.Shape}: {string.Join(", ", score.Data<float>().ToArray())}");
Console.WriteLine($"best  {best.Shape}: {string.Join(", ", best.Data<float>().ToArray())}");
```

```text
in  features: F32 [batch, 3]
in  bias: F32 [1, 2]
out score: F32 [batch, 2]
out best: F32 [batch]
score [2,2]: 4.5, 6.5, 10.5, 12.5
best  [2]: 6.5, 12.5
```

## How `Session.Run` works

- Every model input is passed once, by name, in any order. Inputs are read, never changed and not kept.
- Each element of `outputs` receives the output named at the same position. An empty `Tensor` is replaced by a new one that you own and dispose; a tensor of the right type and shape is overwritten in place, so a loop can reuse its outputs.
- `batch` is a symbolic dimension: the session makes a plan for each set of input shapes it sees and caches it (16 by default, `SessionOptions.MaxCachedPlans`).
- A wrong name, element type or shape is a `SessionException` (`InvalidInput`) before anything runs.

For a model with one input and one output, the span form `session.Run<float, float>(inputName, input, shape, outputName, output)` copies in and out of your buffers and allocates nothing once warm. Sessions are thread-safe. More in [sessions and plans](https://recursiveai.org/onemodel/docs/graph-runtime/).


<!-- https://recursiveai.org/onemodel/cookbook/half-precision-weights/ -->

# Half-precision weights

Keep the large weights of a float32 ONNX model as binary16 or bfloat16 with `SessionOptions.WeightStorage`, which halves their memory, and measure what it costs in accuracy.

The model is `wide.onnx` from [the sample models](https://recursiveai.org/onemodel/docs/getting-started/#the-sample-models).

```csharp
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;

float[] input = Enumerable.Range(0, 256).Select(i => MathF.Cos(i * 0.1f)).ToArray();

using var full = Model.Load("wide.onnx");
using var half = Model.Load("wide.onnx", new SessionOptions { WeightStorage = WeightStorage.Binary16 });
using var bf16 = Model.Load("wide.onnx", new SessionOptions { WeightStorage = WeightStorage.BFloat16 });

float[] a = full.Run(input), b = half.Run(input), c = bf16.Run(input);
Console.WriteLine($"float32  y[0..3] = {Show(a)}");
Console.WriteLine($"Binary16 y[0..3] = {Show(b)}  max |diff| = {MaxDiff(a, b):E2}");
Console.WriteLine($"BFloat16 y[0..3] = {Show(c)}  max |diff| = {MaxDiff(a, c):E2}");

static string Show(float[] v) => string.Join(", ", v[..3].Select(x => x.ToString("F5")));
static float MaxDiff(float[] x, float[] y) => x.Zip(y).Max(p => MathF.Abs(p.First - p.Second));
```

```text
float32  y[0..3] = -0.52048, -0.56874, 0.46823
Binary16 y[0..3] = -0.52028, -0.56858, 0.46820  max |diff| = 2.71E-004
BFloat16 y[0..3] = -0.52041, -0.57004, 0.46990  max |diff| = 2.35E-003
```

## What changes

- `WeightStorage.Binary16` and `WeightStorage.BFloat16` round the constant weights of `MatMul`, `Gemm` and `Conv` (1024 values or more) to 16 bits when the session is created and free the float32 copy. Each run widens them into scratch memory. Activations stay float32, so the only change is the rounding of the weights.
- Embedding tables, biases and normalization parameters stay as they are.
- binary16 keeps more significand bits, so it is closer to float32 here. bfloat16 keeps float32’s range; binary16 ends at 65504, and a weight tensor with a larger value is left in float32.
- Models whose weights are already float16 or bfloat16 run as they are, with no option needed.

Conversions and the widening rule are in [half-precision storage](https://recursiveai.org/onemodel/docs/fp16/). For int8 models (QDQ, `MatMulInteger`, `QLinearConv` and the rest), see [int8 inference](https://recursiveai.org/onemodel/docs/int8/).


<!-- https://recursiveai.org/onemodel/cookbook/missing-operators/ -->

# When a model won’t load

What OneModel reports when an ONNX model uses an operator it doesn’t implement, or one without an exact kernel in exact mode, and how to read the error.

Two small models, made with the `onnx` Python package: `unsupported.onnx` uses `DeformConv` and `TfIdfVectorizer`, which the 0.3.0 preview doesn’t implement, and `wave.onnx` uses `Cos`, which has a fast kernel but no exact one. `broken_models.py`:

```python
import numpy as np, onnx
from onnx import helper as h, TensorProto as T, numpy_helper as nh
# unsupported.onnx: two operators the 0.3.0 preview has no kernel for
g = h.make_graph(
    [h.make_node("DeformConv", ["x", "w", "offset"], ["y"]),
     h.make_node("TfIdfVectorizer", ["ids"], ["tfidf"], mode="TF", min_gram_length=1, max_gram_length=1,
                 max_skip_count=0, ngram_counts=[0], ngram_indexes=[0], pool_int64s=[1])],
    "unsupported",
    [h.make_tensor_value_info("x", T.FLOAT, [1, 1, 3, 3]), h.make_tensor_value_info("w", T.FLOAT, [1, 1, 2, 2]),
     h.make_tensor_value_info("offset", T.FLOAT, [1, 8, 2, 2]), h.make_tensor_value_info("ids", T.INT64, [4])],
    [h.make_tensor_value_info("y", T.FLOAT, None), h.make_tensor_value_info("tfidf", T.FLOAT, None)])
onnx.save(h.make_model(g, opset_imports=[h.make_opsetid("", 22)], ir_version=10), "unsupported.onnx")
# wave.onnx: Cos has a fast kernel but no exact one in 0.3.0
g = h.make_graph([h.make_node("Cos", ["t"], ["y"])], "wave",
    [h.make_tensor_value_info("t", T.FLOAT, [4])], [h.make_tensor_value_info("y", T.FLOAT, [4])])
onnx.save(h.make_model(g, opset_imports=[h.make_opsetid("", 17)], ir_version=8), "wave.onnx")
print("ok")
```

```csharp
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;

try
{
    Model.Load("unsupported.onnx");
}
catch (SessionException e)
{
    Console.WriteLine($"{e.Kind}: {e.Message}");
    foreach (string op in e.Operators) Console.WriteLine($"  missing: {op}");
}

try
{
    Model.Load("wave.onnx", new SessionOptions { Mode = ExecutionMode.Exact });
}
catch (SessionException e)
{
    Console.WriteLine($"{e.Kind}: {e.Message}");
}

using var fast = Model.Load("wave.onnx");   // the same model runs in fast mode
Console.WriteLine(string.Join(", ", fast.Run([0f, 1f, 2f, 3f]).Select(x => x.ToString("F6"))));
```

```text
UnsupportedOperator: The model uses operators this library does not implement: DeformConv (opset 22), TfIdfVectorizer (opset 22).
  missing: DeformConv (opset 22)
  missing: TfIdfVectorizer (opset 22)
ExactModeUnavailable: Exact mode is not available: no exact kernel exists for Cos. Use ExecutionMode.Fast or remove these operators.
1.000000, 0.540302, -0.416147, -0.989992
```

## Reading the error

- `SessionException.Kind` is `UnsupportedOperator` or `ExactModeUnavailable`; `Operators` lists every operator involved, each with its opset, so one load tells you everything a model needs.
- The check happens in `Model.Load`, before any input is read. Nothing half-runs.
- Other kinds: `InvalidModel` (the graph is malformed, with the node name), `InvalidInput` (a wrong name, type or shape at `Run`), `Unsupported` and `Disposed`. A file that isn’t valid ONNX at all is an `OnnxImportException`.

Which operators pass the ONNX backend tests, in fast and in exact mode, and the reason for every gap, is in [conformance](https://recursiveai.org/onemodel/docs/conformance/).