---
title: "Choose CPU or GPU for ONNX inference in C#"
url: https://recursiveai.org/onemodel/cookbook/choose-cpu-or-gpu/
description: "Choose CPU or GPU for an ONNX model in C#, inspect per-operator CPU fallback, and require GPU execution with a runnable .NET 10 example."
version: "1.3.0"
---

# Choose CPU or GPU for ONNX inference in C#

**OneModel** is a free, open-source neural-network runtime for .NET from Recursive AI that runs ONNX models and trains models. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneModel.Gpu) · [GitHub](https://github.com/recursive-ai-oss/onemodel-csharp) · [Docs](https://recursiveai.org/onemodel/docs/) · [Download example (.zip)](https://recursiveai.org/onemodel/cookbook/choose-cpu-or-gpu/choose-cpu-or-gpu.zip)

GPU sessions can leave individual operators on the CPU. Inspect placement instead of assuming the whole model moved to the GPU. Set `RequireGpu = true` to disallow CPU fallback.

```shell
dotnet add package RecursiveAI.OneModel.Gpu
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using System;
using System.Collections.Generic;
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;

// Input file is in the example zip.
OneModelGpu.Register(new()
    { Adapter = "discrete" });
using var session = Session.Load("fallback.onnx",
    new() { Device = ComputeDevice.Gpu });
using var x = Tensor.FromSpan<float>(
        [1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor>
        { ["x"] = x };
using var output = session.Run(inputs);
var plan = session.GetDiagnostics(inputs);
Console.WriteLine($"GPU {plan.NodesOnDevice}, "
    + $"CPU {plan.NodesOnHost}");
```

Prints:

```text
GPU 1, CPU 1
```

Unzip [the example](https://recursiveai.org/onemodel/cookbook/choose-cpu-or-gpu/choose-cpu-or-gpu.zip), then `dotnet run`.

`fallback.onnx` is in the zip. Its whole graph is `product = MatMul(x, x); y = LogSoftmax(product)`, with default last-axis normalization. Input `x` and output `y` are float32 `[2, 2]` tensors; the snippet’s input is `[[1, 2], [3, 4]]`.

## The full program

The program runs the same graph on the CPU and GPU, prints placement and transfer bytes, then tries `RequireGpu`. The GPU run used an NVIDIA GeForce RTX 4070 through Vulkan.

```csharp
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;

OneModelGpu.Register(new GpuOptions
{
    Adapter = "discrete"
});
using var input = Tensor.FromSpan<float>(
    [1, 2, 3, 4], 2, 2);
var inputs = new Dictionary<string, Tensor>
    { ["x"] = input };
var devices = new[]
{
    ComputeDevice.Cpu, ComputeDevice.Gpu
};
foreach (var device in devices)
{
    using var session = Session.Load(
        "fallback.onnx",
        new ModelOptions { Device = device });
    using var outputs = session.Run(inputs);
    var placement = session.GetDiagnostics(inputs);
    Console.WriteLine($"{device}: "
        + $"GPU {placement.NodesOnDevice}, "
        + $"CPU {placement.NodesOnHost}");
    Console.WriteLine(string.Join(", ",
        outputs["y"].ToArray<float>()
            .Select(x => $"{x:F6}")));
    foreach (var node in placement.Nodes)
        Console.WriteLine($"  {node.OpType}: "
            + $"{node.Placement}; "
            + node.FallbackReason);
    Console.WriteLine(
        $"Regions: {placement.DeviceRegions}; "
        + $"upload: {placement.BytesToDevicePerRun}; "
        + "readback: "
        + $"{placement.BytesFromDevicePerRun} "
        + "bytes/run");
    if (device == ComputeDevice.Gpu)
    {
        var info = OneModelGpu.GetInfo(session);
        Console.WriteLine(
            $"Adapter: {info?.Adapter}; "
            + $"API: {info?.Api}");
    }
}
try
{
    using var required = Session.Load(
        "fallback.onnx", new ModelOptions
    {
        Device = ComputeDevice.Gpu,
        RequireGpu = true
    });
    using var outputs = required.Run(inputs);
}
catch (SessionException error) when
    (error.Kind == SessionErrorKind.DeviceUnsupported)
{
    Console.WriteLine($"RequireGpu: {error.Kind}");
    Console.WriteLine(error.Message);
}
```

`dotnet run` prints:

```text
Cpu: GPU 0, CPU 2
-3.048587, -0.048587, -7.000911, -0.000911
  MatMul: cpu; 
  LogSoftmax: cpu; 
Regions: 0; upload: 0; readback: 0 bytes/run
Gpu: GPU 1, CPU 1
-3.048587, -0.048587, -7.000911, -0.000911
  MatMul: gpu; 
  LogSoftmax: cpu; no GPU kernel for LogSoftmax
Regions: 1; upload: 16; readback: 16 bytes/run
Adapter: NVIDIA GeForce RTX 4070; API: Vulkan
RequireGpu: DeviceUnsupported
The device was required but 1 node(s) cannot run on it: log-probabilities (LogSoftmax): no GPU kernel for LogSoftmax.
```

`RequireGpu` throws `SessionException` with `DeviceUnsupported`, naming every rejected operator; shape-dependent rejection can also occur on the first run or diagnostics request for a shape.

## Related recipes

- [Run an ONNX model on the GPU](https://recursiveai.org/onemodel/cookbook/run-onnx-model-on-gpu/)
- [Reproducible GPU inference](https://recursiveai.org/onemodel/cookbook/reproducible-gpu-inference/)

## The short answer

Choose `ComputeDevice.Cpu` for CPU execution. Choose `ComputeDevice.Gpu` after registering the GPU package to allow a mixture of GPU and CPU operators. Add `RequireGpu = true` if CPU fallback is not acceptable. Inspect `GetDiagnostics(inputs)` for the shapes you intend to use; selecting a GPU is not the same as placing every operator on it.

The sample makes that distinction visible with only two operators. `MatMul` runs on the RTX 4070, while `LogSoftmax` stays on the CPU because this release has no GPU kernel for it. That is a supported mixed execution plan, not a missing GPU or a failed inference. The full program also runs a CPU-only session so you can see the two placements independently.

## How do I choose CPU or GPU for ONNX inference?

Start with the execution requirement. If the application must work without a graphics device, make the CPU path an explicit choice. If a GPU is required, fail clearly when it is unavailable. OneModel does not silently create a CPU-only session when you request a nonexistent GPU. The [GPU contract](https://recursiveai.org/onemodel/docs/gpu/) distinguishes that failure from per-operator fallback on an available device.

Next check coverage for your real model and input shapes. The kernel table considers operator type, attributes, opset, tensor type and concrete shape. An operator may be supported in one form and remain on the CPU in another. A declaration such as “this model uses convolution” is not enough to determine placement. The reason attached to each CPU node tells you what actually prevented GPU placement.

Read `DeviceRegions` alongside node counts. A region is a consecutive sequence of operators on the device. A model can have many GPU nodes but several separate regions, with tensors crossing the CPU/device boundary between them. The transfer fields report bytes per run; constant weights, uploaded once, are excluded. In this example one float32 matrix is uploaded and one is read back, which explains each `16`-byte count: four elements at four bytes each.

Those values describe data movement, not a speed result. They help you understand the plan before deciding which device to offer in your application. Keep your preprocessing and output interpretation identical when checking the CPU and GPU paths. The six-decimal outputs shown here agree, but rounded text is not a bitwise-equivalence test for arbitrary models or arithmetic modes.

## How do I disable CPU fallback for an ONNX model?

Set both `Device = ComputeDevice.Gpu` and `RequireGpu = true`. The full program includes the complete call and error handling. For this graph it throws `SessionException` with kind `DeviceUnsupported` and names `log-probabilities (LogSoftmax)` plus the missing-kernel reason. This is useful when a mixed plan would violate your deployment requirement.

Do not check only that session creation succeeds. Some restrictions depend on the concrete shape or on what remains after constant evaluation. OneModel may reject the first `Run` or `GetDiagnostics` for a new shape even though creation succeeded. Keep those calls inside the same error boundary as creation when enforcing GPU-only execution.

The exception names every rejected runtime operator rather than asking you to discover them one at a time. It does not mean the ONNX file is necessarily invalid: the CPU session in this recipe runs the same graph. Conversely, disabling the requirement does not add a missing CPU implementation or repair a malformed model. Model import support and GPU placement are separate questions.