Run int8 ONNX inference on the GPU in C#

Run quantized matrix multiplication with int8 weights. Requires Vulkan on Windows or Linux; unsupported operators stay on the CPU.

dotnet add package \
    RecursiveAI.OneModel.Gpu

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

Use quantized.onnx from the example; its inputs and quantization constants are shown below.

using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
OneModelGpu.Register();
using var session = Session.Load("quantized.onnx",
    new ModelOptions
    { Device = ComputeDevice.Gpu, RequireGpu = true });
using var x = Tensor.FromSpan<float>(
    [1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor>
    { ["x"] = x };
using var y = session.Run(inputs);
Console.WriteLine(string.Join(", ",
    y["y"].ToArray<float>()));
foreach (var node in
    session.GetDiagnostics(inputs).Nodes)
    Console.WriteLine(
        $"{node.OpType}: {node.Placement}");

Prints the dequantized output and placement of each operator:

1.75, 2.5, 3.75, 5.5
QuantizeLinear: gpu
QLinearMatMul: gpu
DequantizeLinear: gpu

Unzip the example, then dotnet run.

The full program

using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
OneModelGpu.Register(new GpuOptions { Adapter = "discrete" });
using var gpu = Session.Load("quantized.onnx",
    new ModelOptions { Device = ComputeDevice.Gpu, RequireGpu = true });
using var cpu = Session.Load("quantized.onnx",
    new ModelOptions { Device = ComputeDevice.Cpu });
using var input = Tensor.FromSpan<float>([1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor> { ["x"] = input };
using var result = gpu.Run(inputs);
using var reference = cpu.Run(inputs);
var actual = result["y"].ToArray<float>();
Console.WriteLine($"GPU: {string.Join(", ", actual)}");
Console.WriteLine($"CPU: {string.Join(", ", reference["y"].ToArray<float>())}");
Console.WriteLine($"Equal: {actual.SequenceEqual(reference["y"].ToArray<float>())}");
foreach (var node in gpu.GetDiagnostics(inputs).Nodes)
    Console.WriteLine($"{node.OpType}: {node.Placement}{node.FallbackReason}");
var device = OneModelGpu.GetInfo(gpu)!;
Console.WriteLine($"{device.Adapter} / {device.Api}");
if (!actual.SequenceEqual(reference["y"].ToArray<float>()))
    throw new InvalidOperationException("CPU/GPU mismatch");

Output on Linux with an NVIDIA GeForce RTX 4070 via Vulkan:

GPU: 1.75, 2.5, 3.75, 5.5
CPU: 1.75, 2.5, 3.75, 5.5
Equal: True
QuantizeLinear: gpu
QLinearMatMul: gpu
DequantizeLinear: gpu
NVIDIA GeForce RTX 4070 / Vulkan

Adapter = "discrete" in GpuOptions selects the first matching discrete GPU; if none is available, creating the GPU session fails with SessionException.

Scale and zero point convert a stored integer to a real value: real = scale × (q − zero point), where q is the stored integer.

The example zip includes the original quantized.onnx graph, which uses opset 13 and these complete inputs and constants:

The nodes are a = QuantizeLinear(x, a_scale, a_zero), q = QLinearMatMul(a, a_scale, a_zero, w, b_scale, b_zero, y_scale, y_zero), and y = DequantizeLinear(q, y_scale, y_zero). The model file is freely redistributable and comes in the example zip; no separate pretrained-model download is needed. A device failure raises SessionException; unsupported operators can remain on the CPU unless RequireGpu is set.

Variations

To allow unsupported operators to fall back to the CPU, replace the GPU session’s ModelOptions with:

new ModelOptions
{
    Device = ComputeDevice.Gpu,
    RequireGpu = false
}

This graph produces the same output with fallback allowed: all three nodes still run on the GPU.

Related recipes

• Run an ONNX model on the GPU
• Inspect GPU session memory
• Use half-precision weights

The short answer

Run the quantized ONNX graph in a OneModel GPU session with RequireGpu = true on ModelOptions to reject CPU fallback.

Install RecursiveAI.OneModel.Gpu, register it, load the quantized ONNX graph with Device = ComputeDevice.Gpu, and inspect GetDiagnostics(inputs) after running. This example places QuantizeLinear, QLinearMatMul and DequantizeLinear on the GPU through Vulkan. Its matrix weights are signed int8, but the quantized product’s activations and result are uint8. The final output is dequantized float32.

The full program also loads the same model into a CPU session and compares every output element for the same input. Here the dequantized outputs match exactly. That observation is about this small graph, these constants and this input; it is not a statement that every quantized ONNX model has GPU kernels or produces the same result under every transformation.

How do you run int8 ONNX GPU inference in C#?

Register OneModelGpu, load the graph with GPU ModelOptions, and run it with named input tensors.

First check the model’s operators and tensor types against the released GPU operator table. “Int8 model” does not specify one execution path. A model may store signed weights, quantize activations, use integer convolutions, or put quantize/dequantize pairs around float operations. Those forms do not all select the same kernels.

For this QLinearMatMul path, OneModel’s GPU binder accepts uint8 activations and a constant uint8 or int8 weight matrix. It requires a uint8 output. The scales and output zero point must be constants when the plan is prepared; scales must be float32. The activation and output scales have one value, while the weight scale can have one value or one per output column. The activation zero point is uint8, and the weight zero point must have the weight’s type. The binder also accepts its documented row/column zero-point forms; the sample uses scalars throughout.

Each scale and zero point belongs to a particular tensor. Here quantizing x with scale 0.5 and zero point zero gives [[2, 4], [6, 8]]. The signed weight bytes represent real values [[0.25, 0.5], [0.75, 1]] because their scale is 0.25. The product is requantized with output scale 0.125, then the last node converts it back to float32. The printed result is therefore [[1.75, 2.5], [3.75, 5.5]], not the unscaled integer matrix product.

Read placement for every node, including quantization and dequantization. Successfully creating a GPU session is not the same as placing all operators on it. By default, an unsupported operator stays on the CPU with a fallback reason in NodeDiagnostic.FallbackReason. For example, the pinned binder rejects a QLinearMatMul whose activation type is not uint8, whose weight is not a constant uint8 or int8 matrix, or whose scales are not in the supported constant form.

Use RequireGpu = true when CPU fallback is unacceptable. A rejected placement then fails instead of silently leaving those operators on the host. It does not convert unsupported nodes into supported ones. In particular, the released table says QLinearConv is not mapped, while it lists ConvInteger with its own supported forms. Check the actual graph before applying the example to a larger model.