Recursive AI

Reproducible GPU inference in C#

OneModel is a free, open-source neural-network runtime for .NET from Recursive AI that runs ONNX models and trains models. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

Exact GPU mode repeats the same output bits on the same device and driver, not across different GPUs or against the CPU.

Terminal
dotnet add package RecursiveAI.OneModel.Gpu

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
 
// Input file is in the example zip.
OneModelGpu.Register(new()
    { Adapter = "discrete" });
using var model = Model.Load("square.onnx", new()
{
    Device = ComputeDevice.Gpu,
    Mode = ArithmeticMode.Exact,
    RequireGpu = true
});
float[] input = [0.1f, 0.2f, 0.3f, 0.4f];
var a = model.Run(input, shape: [2, 2]);
var b = model.Run(input, shape: [2, 2]);
var aBits = a.Select(
    BitConverter.SingleToInt32Bits);
var bBits = b.Select(
    BitConverter.SingleToInt32Bits);
Console.WriteLine(aBits.SequenceEqual(bBits));

Prints:

Text
True

Unzip the example, then dotnet run.

square.onnx is in the zip. Its whole graph is y = MatMul(x, x), with float32 input x and output y, both shaped [2, 2]. The test input is [[0.1, 0.2], [0.3, 0.4]].

Adapter = "discrete" requests a discrete graphics card, not an integrated GPU. On an integrated-only system, that explicit selection cannot find a matching adapter and session creation fails with DeviceUnsupported; use Adapter = "integrated" or omit the selection to use the platform’s default.

The full program

Model is the array-based convenience wrapper; its underlying Session (also available as model.Session) accepts named tensors and provides GetDiagnostics and the session argument for OneModelGpu.GetInfo.

The full program prints each float32 value’s hexadecimal bit pattern, not rounded decimal text. This run used an NVIDIA GeForce RTX 4070 through Vulkan.

C#
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
OneModelGpu.Register(new GpuOptions
{
    Adapter = "discrete"
});
using var session = Session.Load(
    "square.onnx", new ModelOptions
{
    Device = ComputeDevice.Gpu,
    Mode = ArithmeticMode.Exact,
    RequireGpu = true
});
using var input = Tensor.FromSpan<float>(
    [0.1f, 0.2f, 0.3f, 0.4f], 2, 2);
var inputs = new Dictionary<string, Tensor>
    { ["x"] = input };
using var first = session.Run(inputs);
using var second = session.Run(inputs);
int[] firstBits = first["y"].ToArray<float>()
    .Select(BitConverter.SingleToInt32Bits).ToArray();
int[] secondBits = second["y"].ToArray<float>()
    .Select(BitConverter.SingleToInt32Bits).ToArray();
Console.WriteLine("Run 1: " + string.Join(" ",
    firstBits.Select(x => $"{x:X8}")));
Console.WriteLine("Run 2: " + string.Join(" ",
    secondBits.Select(x => $"{x:X8}")));
Console.WriteLine("Same bits: "
    + firstBits.SequenceEqual(secondBits));
var info = OneModelGpu.GetInfo(session);
Console.WriteLine($"Adapter: {info?.Adapter}; "
    + $"API: {info?.Api}");
var placement = session.GetDiagnostics(inputs);
Console.WriteLine(
    $"GPU nodes: {placement.NodesOnDevice}");

dotnet run prints:

Text
Run 1: 3D8F5C29 3DCCCCCE 3E19999A 3E6147AF
Run 2: 3D8F5C29 3DCCCCCE 3E19999A 3E6147AF
Same bits: True
Adapter: NVIDIA GeForce RTX 4070; API: Vulkan
GPU nodes: 1

With RequireGpu, an operator lacking a deterministic GPU kernel is rejected with DeviceUnsupported rather than silently kept on the CPU.

Variations

Use the same arithmetic mode on an explicitly selected CPU session by replacing the full program’s options with the following. Also remove the GetInfo call and adapter-print line, which apply only to GPU sessions:

C#
new ModelOptions
{
    Device = ComputeDevice.Cpu,
    Mode = ArithmeticMode.Exact
}

This variant also prints Same bits: True; its diagnostics report GPU nodes: 0. CPU and GPU exact mode have different reproducibility scopes.

The short answer

Set Mode = ArithmeticMode.Exact together with Device = ComputeDevice.Gpu, and keep the model, inputs, device and driver fixed. OneModel documents identical bits for repeated runs on that device and driver. This is not a promise of matching bits on another GPU, another driver, or the CPU.

The snippet at the top compares the integer bit representations of the output floats. The full program prints those representations in hexadecimal so you can see the comparison. It also uses RequireGpu = true and checks placement: the matrix product must actually run on the GPU for this to demonstrate GPU repeatability.

How do I make GPU inference deterministic?

In OneModel, use exact-mode GPU kernels rather than relying on the default fast mode. According to the GPU arithmetic contract, exact kernels use a fixed summation order, no atomics and no scheduling dependence. Matrix products, reductions, normalization and softmax have deterministic variants. Elementwise and layout operators have no reduction order, so their device kernel serves both modes.

Keep the inputs identical as well as the execution settings. A deterministic kernel does not make two different input tensors equivalent. This example spells out all four float32 inputs and ships a model with no external weights. For a deployed model, retain the exact model file and the tensors used for the comparison, not just a description of how they were prepared.

Compare bits rather than formatted numbers. Printing a few decimal places can hide differences in the last bits. Here BitConverter.SingleToInt32Bits preserves the float32 representation, and SequenceEqual compares the resulting integer arrays element by element. The displayed hexadecimal values are those integers written with eight hexadecimal digits. A matching printout is meaningful for these outputs because no decimal rounding is involved.

An operator without a deterministic GPU kernel stays on the CPU unless GPU execution is required. Add RequireGpu when mixed placement would make your check misleading, and inspect diagnostics with the actual input shapes. An error can appear at session creation or when the first plan for a shape is made. The full program’s strict requirement avoids silently presenting a CPU result as a GPU result.

Why can GPU inference be non-deterministic across devices?

OneModel’s GPU exact contract is narrower than its CPU exact contract. GPU kernels are repeatable on one device and driver, but another adapter or driver can differ in the last bits. Device transcendental functions such as Exp, Tanh and Erf are not the CPU exact mode’s fixed polynomials. Consequently, the docs describe GPU-to-CPU agreement in float32 tolerances, not bit identity.

Do not turn a successful two-run comparison into a cross-hardware guarantee. It shows that these inputs and this model repeated on the selected setup. If you change the GPU or driver, record that as a different execution environment and apply the numerical comparison your application requires. Preserve the adapter and API information with your result; the program prints them for this reason.

Likewise, exact inference is not a training seed setting or a text-generation sampling policy. This recipe executes an existing graph twice. It does not initialize weights, shuffle a dataset or choose output tokens. Keep those separate sources of variation out of a kernel repeatability check. For a CPU deployment requiring OneModel’s cross-machine exact behavior, choose the CPU device explicitly and follow the exact-mode documentation for supported operators.

All onemodel recipes