---
title: "Run int8 ONNX inference on the GPU in C#"
url: https://recursiveai.org/onemodel/cookbook/run-int8-model-on-gpu/
description: "Run a tiny int8-weight ONNX graph on the GPU in C#, inspect operator placement and compare dequantized output with a CPU reference using OneModel."
version: "1.3.0"
---

# Run int8 ONNX inference on the GPU in C#

**OneModel** is a free, open-source neural-network runtime for .NET from Recursive AI that runs ONNX models and trains models. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneModel.Gpu) · [GitHub](https://github.com/recursive-ai-oss/onemodel-csharp) · [Docs](https://recursiveai.org/onemodel/docs/) · [Download example (.zip)](https://recursiveai.org/onemodel/cookbook/run-int8-model-on-gpu/run-int8-model-on-gpu.zip)

Run quantized matrix multiplication with int8 weights. Requires Vulkan on Windows or Linux; unsupported operators stay on the CPU.

```shell
dotnet add package \
    RecursiveAI.OneModel.Gpu
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

Use `quantized.onnx` from the example; its inputs and quantization constants are shown below.

```csharp
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;

OneModelGpu.Register();
using var session = Session.Load("quantized.onnx",
    new ModelOptions
    { Device = ComputeDevice.Gpu, RequireGpu = true });
using var x = Tensor.FromSpan<float>(
    [1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor>
    { ["x"] = x };
using var y = session.Run(inputs);
Console.WriteLine(string.Join(", ",
    y["y"].ToArray<float>()));
foreach (var node in
    session.GetDiagnostics(inputs).Nodes)
    Console.WriteLine(
        $"{node.OpType}: {node.Placement}");
```

Prints the dequantized output and placement of each operator:

```text
1.75, 2.5, 3.75, 5.5
QuantizeLinear: gpu
QLinearMatMul: gpu
DequantizeLinear: gpu
```

Unzip [the example](https://recursiveai.org/onemodel/cookbook/run-int8-model-on-gpu/run-int8-model-on-gpu.zip), then `dotnet run`.

## The full program

```csharp
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;

OneModelGpu.Register(new GpuOptions { Adapter = "discrete" });
using var gpu = Session.Load("quantized.onnx",
    new ModelOptions { Device = ComputeDevice.Gpu, RequireGpu = true });
using var cpu = Session.Load("quantized.onnx",
    new ModelOptions { Device = ComputeDevice.Cpu });
using var input = Tensor.FromSpan<float>([1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor> { ["x"] = input };
using var result = gpu.Run(inputs);
using var reference = cpu.Run(inputs);
var actual = result["y"].ToArray<float>();
Console.WriteLine($"GPU: {string.Join(", ", actual)}");
Console.WriteLine($"CPU: {string.Join(", ", reference["y"].ToArray<float>())}");
Console.WriteLine($"Equal: {actual.SequenceEqual(reference["y"].ToArray<float>())}");
foreach (var node in gpu.GetDiagnostics(inputs).Nodes)
    Console.WriteLine($"{node.OpType}: {node.Placement}{node.FallbackReason}");
var device = OneModelGpu.GetInfo(gpu)!;
Console.WriteLine($"{device.Adapter} / {device.Api}");
if (!actual.SequenceEqual(reference["y"].ToArray<float>()))
    throw new InvalidOperationException("CPU/GPU mismatch");
```

Output on Linux with an NVIDIA GeForce RTX 4070 via Vulkan:

```text
GPU: 1.75, 2.5, 3.75, 5.5
CPU: 1.75, 2.5, 3.75, 5.5
Equal: True
QuantizeLinear: gpu
QLinearMatMul: gpu
DequantizeLinear: gpu
NVIDIA GeForce RTX 4070 / Vulkan
```

`Adapter = "discrete"` in `GpuOptions` selects the first matching discrete GPU; if none is available, creating the GPU session fails with `SessionException`.

Scale and zero point convert a stored integer to a real value: `real = scale × (q − zero point)`, where `q` is the stored integer.

The example zip includes the original `quantized.onnx` graph, which uses opset 13 and these complete inputs and constants:

| Value | Type and shape | Values |
|---|---|---|
| `x` | float32 `[2, 2]` input | `[[1, 2], [3, 4]]` |
| `w` | int8 `[2, 2]` constant | `[[1, 2], [3, 4]]` |
| `a_scale` | float32 scalar | `0.5` |
| `b_scale` | float32 scalar | `0.25` |
| `y_scale` | float32 scalar | `0.125` |
| `a_zero`, `y_zero` | uint8 scalars | `0` |
| `b_zero` | int8 scalar | `0` |

The nodes are `a = QuantizeLinear(x, a_scale, a_zero)`, `q = QLinearMatMul(a, a_scale, a_zero, w, b_scale, b_zero, y_scale, y_zero)`, and `y = DequantizeLinear(q, y_scale, y_zero)`. The model file is freely redistributable and comes in the example zip; no separate pretrained-model download is needed. A device failure raises `SessionException`; unsupported operators can remain on the CPU unless `RequireGpu` is set.

## Variations

To allow unsupported operators to fall back to the CPU, replace the GPU session’s `ModelOptions` with:

```csharp
new ModelOptions
{
    Device = ComputeDevice.Gpu,
    RequireGpu = false
}
```

This graph produces the same output with fallback allowed: all three nodes still run on the GPU.

## Related recipes

- [Run an ONNX model on the GPU](https://recursiveai.org/onemodel/cookbook/run-onnx-model-on-gpu/)
- [Inspect GPU session memory](https://recursiveai.org/onemodel/cookbook/inspect-gpu-session-memory/)
- [Use half-precision weights](https://recursiveai.org/onemodel/cookbook/half-precision-weights/)

## The short answer

Run the quantized ONNX graph in a OneModel GPU session with `RequireGpu = true` on `ModelOptions` to reject CPU fallback.

Install `RecursiveAI.OneModel.Gpu`, register it, load the quantized ONNX graph with `Device = ComputeDevice.Gpu`, and inspect `GetDiagnostics(inputs)` after running. This example places `QuantizeLinear`, `QLinearMatMul` and `DequantizeLinear` on the GPU through Vulkan. Its matrix weights are signed int8, but the quantized product’s activations and result are uint8. The final output is dequantized float32.

The full program also loads the same model into a CPU session and compares every output element for the same input. Here the dequantized outputs match exactly. That observation is about this small graph, these constants and this input; it is not a statement that every quantized ONNX model has GPU kernels or produces the same result under every transformation.

## How do you run int8 ONNX GPU inference in C#?

Register `OneModelGpu`, load the graph with GPU `ModelOptions`, and run it with named input tensors.

First check the model’s operators and tensor types against the [released GPU operator table](https://recursiveai.org/onemodel/docs/gpu/). “Int8 model” does not specify one execution path. A model may store signed weights, quantize activations, use integer convolutions, or put quantize/dequantize pairs around float operations. Those forms do not all select the same kernels.

For this `QLinearMatMul` path, OneModel’s GPU binder accepts uint8 activations and a constant uint8 or int8 weight matrix. It requires a uint8 output. The scales and output zero point must be constants when the plan is prepared; scales must be float32. The activation and output scales have one value, while the weight scale can have one value or one per output column. The activation zero point is uint8, and the weight zero point must have the weight’s type. The binder also accepts its documented row/column zero-point forms; the sample uses scalars throughout.

Each scale and zero point belongs to a particular tensor. Here quantizing `x` with scale `0.5` and zero point zero gives `[[2, 4], [6, 8]]`. The signed weight bytes represent real values `[[0.25, 0.5], [0.75, 1]]` because their scale is `0.25`. The product is requantized with output scale `0.125`, then the last node converts it back to float32. The printed result is therefore `[[1.75, 2.5], [3.75, 5.5]]`, not the unscaled integer matrix product.

Read placement for every node, including quantization and dequantization. Successfully creating a GPU session is not the same as placing all operators on it. By default, an unsupported operator stays on the CPU with a fallback reason in `NodeDiagnostic.FallbackReason`. For example, the pinned binder rejects a `QLinearMatMul` whose activation type is not uint8, whose weight is not a constant uint8 or int8 matrix, or whose scales are not in the supported constant form.

Use `RequireGpu = true` when CPU fallback is unacceptable. A rejected placement then fails instead of silently leaving those operators on the host. It does not convert unsupported nodes into supported ones. In particular, the released table says `QLinearConv` is not mapped, while it lists `ConvInteger` with its own supported forms. Check the actual graph before applying the example to a larger model.