Run an ONNX model on the GPU in C# · onemodel cookbook

Run ONNX inference on a discrete GPU. This desktop example uses Vulkan, not CUDA; no CUDA toolkit needed. It needs a working Vulkan graphics driver.

Terminal

dotnet add package RecursiveAI.OneModel.Gpu

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
 
// Input file is in the example zip.
OneModelGpu.Register(new()
    { Adapter = "discrete" });
using var model = Model.Load("square.onnx",
    new() { Device = ComputeDevice.Gpu });
Console.WriteLine(string.Join(", ",
    model.Run([1f, 2f, 3f, 4f],
        shape: [2, 2])));

Prints:

Text

7, 10, 15, 22

Unzip the example, then dotnet run.

square.onnx is in the zip. Its whole graph is y = MatMul(x, x), with float32 input x and output y, both shaped [2, 2]. The snippet’s input is the matrix [[1, 2], [3, 4]].

The full program

Model is the array-based convenience wrapper; its underlying Session (also available as model.Session) accepts named tensors and provides GetDiagnostics and the session argument for OneModelGpu.GetInfo.

The full program also prints the actual adapter, API and GPU node count. This run used an NVIDIA GeForce RTX 4070 through Vulkan.

using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
OneModelGpu.Register(new GpuOptions
{
    Adapter = "discrete"
});
using var session = Session.Load("square.onnx",
    new ModelOptions
    {
        Device = ComputeDevice.Gpu
    });
using var input = Tensor.FromSpan<float>(
    [1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor>
    { ["x"] = input };
using var outputs = session.Run(inputs);
Console.WriteLine(string.Join(", ",
    outputs["y"].ToArray<float>()));
var info = OneModelGpu.GetInfo(session);
Console.WriteLine($"Adapter: {info?.Adapter}");
Console.WriteLine($"API: {info?.Api}");
var placement = session.GetDiagnostics(inputs);
Console.WriteLine(
    $"GPU nodes: {placement.NodesOnDevice}");

dotnet run prints:

Text

7, 10, 15, 22
Adapter: NVIDIA GeForce RTX 4070
API: Vulkan
GPU nodes: 1

If no usable GPU exists, session creation throws SessionException with DeviceUnsupported; requesting GPU execution does not silently create a CPU-only session.

Variations

To require every remaining runtime operator to use the GPU, replace the options in the full program with:

new ModelOptions
{
    Device = ComputeDevice.Gpu,
    RequireGpu = true
}

For square.onnx, the output stays the same and the GPU node count remains 1.

The short answer

Register RecursiveAI.OneModel.Gpu once, then load the model with Device = ComputeDevice.Gpu. Registration makes the GPU backend available; it does not change sessions that have already been created. Set Adapter = "discrete" when you want the discrete graphics card rather than an integrated adapter.

The example deliberately uses a tiny ONNX graph with no external weights, image decoding or preprocessing. You can check its arithmetic by hand: the first row of the matrix product is [1 × 1 + 2 × 3, 1 × 2 + 2 × 4], or [7, 10]. Replace both the model and input preparation for your real application. ONNX describes the graph, but its input names, element types and dimensions still have to match the tensors you supply.

Can ONNX run on GPU?

Yes. OneModel’s GPU package runs supported ONNX operators through Vulkan on this desktop. The package also has Metal and browser WebGPU paths. It is an optional package: the core runtime does not discover or enable it by itself. The GPU documentation lists the supported operator families and the attribute, shape and type restrictions that decide placement.

GPU selection is not a promise that every operator runs there. OneModel offers operators to the backend individually and groups accepted consecutive operators into device regions. Operators the GPU cannot handle remain on the CPU unless you request RequireGpu. Constant calculations may be evaluated while making the plan rather than becoming runtime work.

Use session.GetDiagnostics(inputs) after supplying representative inputs. It reports the operators left after graph optimization, their placement and any fallback reason. NodesOnDevice is therefore a count for the resulting execution plan, not necessarily the original ONNX node count. If your model has symbolic dimensions, inspect the shapes you will actually run rather than drawing conclusions from one example batch.

OneModelGpu.GetInfo(session) supplies the adapter name and graphics API. The full program prints these separately from placement: successfully opening a graphics device and actually assigning operators to it are different facts. The output above establishes both for this small matrix-product graph.

Does ONNX use CUDA?

ONNX is the model format; the runtime and backend decide how to execute it. This OneModel recipe uses Vulkan, not CUDA. Installing a CUDA toolkit is not the setup step for this path. You need a usable Vulkan device and its graphics driver. The browser recipe instead uses WebGPU and asynchronous calls, so do not copy the desktop blocking run into a browser app.

If no usable device exists, requesting Device = Gpu fails rather than silently replacing the session with a CPU session. To offer an explicit CPU option in your application, create a separate session with Device = ComputeDevice.Cpu and prepare the same named tensors. Decide whether that is acceptable for your application instead of treating a missing device as the same condition as an unsupported operator.

Keep the session alive for repeated inference and dispose it when finished. Each GPU session owns its device, compiled kernels and buffers. Runs of one session take turns on the GPU. Dispose each SessionOutputs after reading its tensors; the result owns them. See tensors and ownership for the distinction between tensor values and the objects that own their storage.

The full program

Variations

Related recipes

The short answer

Can ONNX run on GPU?

Does ONNX use CUDA?