Recursive AI

Choose CPU or GPU for ONNX inference in C#

OneModel is a free, open-source neural-network runtime for .NET from Recursive AI that runs ONNX models and trains models. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

GPU sessions can leave individual operators on the CPU. Inspect placement instead of assuming the whole model moved to the GPU. Set RequireGpu = true to disallow CPU fallback.

Terminal
dotnet add package RecursiveAI.OneModel.Gpu

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using System;
using System.Collections.Generic;
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
// Input file is in the example zip.
OneModelGpu.Register(new()
    { Adapter = "discrete" });
using var session = Session.Load("fallback.onnx",
    new() { Device = ComputeDevice.Gpu });
using var x = Tensor.FromSpan<float>(
        [1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor>
        { ["x"] = x };
using var output = session.Run(inputs);
var plan = session.GetDiagnostics(inputs);
Console.WriteLine($"GPU {plan.NodesOnDevice}, "
    + $"CPU {plan.NodesOnHost}");

Prints:

Text
GPU 1, CPU 1

Unzip the example, then dotnet run.

fallback.onnx is in the zip. Its whole graph is product = MatMul(x, x); y = LogSoftmax(product), with default last-axis normalization. Input x and output y are float32 [2, 2] tensors; the snippet’s input is [[1, 2], [3, 4]].

The full program

The program runs the same graph on the CPU and GPU, prints placement and transfer bytes, then tries RequireGpu. The GPU run used an NVIDIA GeForce RTX 4070 through Vulkan.

C#
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
OneModelGpu.Register(new GpuOptions
{
    Adapter = "discrete"
});
using var input = Tensor.FromSpan<float>(
    [1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor>
    { ["x"] = input };
var devices = new[]
{
    ComputeDevice.Cpu, ComputeDevice.Gpu
};
foreach (var device in devices)
{
    using var session = Session.Load(
        "fallback.onnx",
        new ModelOptions { Device = device });
    using var outputs = session.Run(inputs);
    var placement = session.GetDiagnostics(inputs);
    Console.WriteLine($"{device}: "
        + $"GPU {placement.NodesOnDevice}, "
        + $"CPU {placement.NodesOnHost}");
    Console.WriteLine(string.Join(", ",
        outputs["y"].ToArray<float>()
            .Select(x => $"{x:F6}")));
    foreach (var node in placement.Nodes)
        Console.WriteLine($"  {node.OpType}: "
            + $"{node.Placement}; "
            + node.FallbackReason);
    Console.WriteLine(
        $"Regions: {placement.DeviceRegions}; "
        + $"upload: {placement.BytesToDevicePerRun}; "
        + "readback: "
        + $"{placement.BytesFromDevicePerRun} "
        + "bytes/run");
    if (device == ComputeDevice.Gpu)
    {
        var info = OneModelGpu.GetInfo(session);
        Console.WriteLine(
            $"Adapter: {info?.Adapter}; "
            + $"API: {info?.Api}");
    }
}
try
{
    using var required = Session.Load(
        "fallback.onnx", new ModelOptions
    {
        Device = ComputeDevice.Gpu,
        RequireGpu = true
    });
    using var outputs = required.Run(inputs);
}
catch (SessionException error) when
    (error.Kind == SessionErrorKind.DeviceUnsupported)
{
    Console.WriteLine($"RequireGpu: {error.Kind}");
    Console.WriteLine(error.Message);
}

dotnet run prints:

Text
Cpu: GPU 0, CPU 2
-3.048587, -0.048587, -7.000911, -0.000911
  MatMul: cpu; 
  LogSoftmax: cpu; 
Regions: 0; upload: 0; readback: 0 bytes/run
Gpu: GPU 1, CPU 1
-3.048587, -0.048587, -7.000911, -0.000911
  MatMul: gpu; 
  LogSoftmax: cpu; no GPU kernel for LogSoftmax
Regions: 1; upload: 16; readback: 16 bytes/run
Adapter: NVIDIA GeForce RTX 4070; API: Vulkan
RequireGpu: DeviceUnsupported
The device was required but 1 node(s) cannot run on it: log-probabilities (LogSoftmax): no GPU kernel for LogSoftmax.

RequireGpu throws SessionException with DeviceUnsupported, naming every rejected operator; shape-dependent rejection can also occur on the first run or diagnostics request for a shape.

The short answer

Choose ComputeDevice.Cpu for CPU execution. Choose ComputeDevice.Gpu after registering the GPU package to allow a mixture of GPU and CPU operators. Add RequireGpu = true if CPU fallback is not acceptable. Inspect GetDiagnostics(inputs) for the shapes you intend to use; selecting a GPU is not the same as placing every operator on it.

The sample makes that distinction visible with only two operators. MatMul runs on the RTX 4070, while LogSoftmax stays on the CPU because this release has no GPU kernel for it. That is a supported mixed execution plan, not a missing GPU or a failed inference. The full program also runs a CPU-only session so you can see the two placements independently.

How do I choose CPU or GPU for ONNX inference?

Start with the execution requirement. If the application must work without a graphics device, make the CPU path an explicit choice. If a GPU is required, fail clearly when it is unavailable. OneModel does not silently create a CPU-only session when you request a nonexistent GPU. The GPU contract distinguishes that failure from per-operator fallback on an available device.

Next check coverage for your real model and input shapes. The kernel table considers operator type, attributes, opset, tensor type and concrete shape. An operator may be supported in one form and remain on the CPU in another. A declaration such as “this model uses convolution” is not enough to determine placement. The reason attached to each CPU node tells you what actually prevented GPU placement.

Read DeviceRegions alongside node counts. A region is a consecutive sequence of operators on the device. A model can have many GPU nodes but several separate regions, with tensors crossing the CPU/device boundary between them. The transfer fields report bytes per run; constant weights, uploaded once, are excluded. In this example one float32 matrix is uploaded and one is read back, which explains each 16-byte count: four elements at four bytes each.

Those values describe data movement, not a speed result. They help you understand the plan before deciding which device to offer in your application. Keep your preprocessing and output interpretation identical when checking the CPU and GPU paths. The six-decimal outputs shown here agree, but rounded text is not a bitwise-equivalence test for arbitrary models or arithmetic modes.

How do I disable CPU fallback for an ONNX model?

Set both Device = ComputeDevice.Gpu and RequireGpu = true. The full program includes the complete call and error handling. For this graph it throws SessionException with kind DeviceUnsupported and names log-probabilities (LogSoftmax) plus the missing-kernel reason. This is useful when a mixed plan would violate your deployment requirement.

Do not check only that session creation succeeds. Some restrictions depend on the concrete shape or on what remains after constant evaluation. OneModel may reject the first Run or GetDiagnostics for a new shape even though creation succeeded. Keep those calls inside the same error boundary as creation when enforcing GPU-only execution.

The exception names every rejected runtime operator rather than asking you to discover them one at a time. It does not mean the ONNX file is necessarily invalid: the CPU session in this recipe runs the same graph. Conversely, disabling the requirement does not add a missing CPU implementation or repair a malformed model. Model import support and GPU placement are separate questions.

All onemodel recipes