Recursive AI

Inspect ONNX GPU memory allocations in C#

OneModel is a free, open-source neural-network runtime for .NET from Recursive AI that runs ONNX models and trains models. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

Track GPU weights and intermediate-tensor memory across input shapes. Requires a Vulkan driver on Windows or Linux.

Terminal
dotnet add package \
    RecursiveAI.OneModel.Gpu

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

Use linear.onnx from the example: y = Relu(x × [[1, 0], [0, 2]]), with float32 x shaped [batch, 2].

C#
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
OneModelGpu.Register();
using var session = Session.Load("linear.onnx",
    new ModelOptions
    { Device = ComputeDevice.Gpu });
foreach (int rows in new[] { 4, 4, 2 })
{
    using var x = Tensor.FromSpan<float>(
        new float[rows * 2], rows, 2);
    using var y = session.Run(
        new Dictionary<string, Tensor>
        { ["x"] = x });
    var info = OneModelGpu.GetInfo(session)!;
    Console.WriteLine(
        $"{info.ConstantBytes}, " +
        $"{info.ArenaBytes}, " +
        $"{info.DeviceAllocations}");
}

Prints weight bytes, arena bytes and cumulative device allocations:

Text
16, 1048576, 2
16, 1048576, 2
16, 1048576, 2

Unzip the example, then dotnet run.

The full program

C#
using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
using RecursiveAI.OneModel.Gpu;
using RecursiveAI.OneModel.Tensors;
 
OneModelGpu.Register(new GpuOptions { Adapter = "discrete" });
using var session = Session.Load("linear.onnx",
    new ModelOptions { Device = ComputeDevice.Gpu });
Console.WriteLine($"Before run: arena={OneModelGpu.GetInfo(session)!.ArenaBytes}");
foreach (int rows in new[] { 4, 4, 2, 262144, 262144, 2 })
{
    using var input = Tensor.FromSpan<float>(
        Enumerable.Repeat(1f, rows * 2).ToArray(), rows, 2);
    var inputs = new Dictionary<string, Tensor> { ["x"] = input };
    using var outputs = session.Run(inputs);
    var info = OneModelGpu.GetInfo(session)!;
    Console.WriteLine($"Rows {rows}: weights={info.ConstantBytes}, arena={info.ArenaBytes}, allocations={info.DeviceAllocations}");
    if (outputs["y"].ToArray<float>()[1] != 2f)
        throw new InvalidOperationException("Unexpected output");
}
var device = OneModelGpu.GetInfo(session)!;
Console.WriteLine($"{device.Adapter} / {device.Api}");

Output on Linux with an NVIDIA GeForce RTX 4070 via Vulkan:

Text
Before run: arena=0
Rows 4: weights=16, arena=1048576, allocations=2
Rows 4: weights=16, arena=1048576, allocations=2
Rows 2: weights=16, arena=1048576, allocations=2
Rows 262144: weights=16, arena=4194304, allocations=3
Rows 262144: weights=16, arena=4194304, allocations=3
Rows 2: weights=16, arena=4194304, allocations=3
NVIDIA GeForce RTX 4070 / Vulkan

linear.onnx is an original, redistributable sample graph: float32 input x[batch, 2], constant weight w[2, 2] = [[1, 0], [0, 2]], MatMul(x, w), then Relu, producing y[batch, 2]. The full program supplies ones and checks that each run’s second output is two. Device failures surface as SessionException; these counters do not include process-wide VRAM use.

Variations

Inside the loop, after obtaining info, add the arena-only allocation counter:

C#
Console.WriteLine($"Arena allocations: {info.ArenaAllocations}");

For the six runs above, it prints 1, 1, 1, 2, 2, 2 on separate labeled lines. The other allocation in the total is the constant-weight buffer.

The short answer

Keep one GPU session alive and call OneModelGpu.GetInfo(session) after each run. ConstantBytes reports device memory for constant weights. ArenaBytes reports the session’s intermediate-tensor arena. DeviceAllocations is the cumulative number of device buffers allocated by that session for its arena and weights, not a byte count or a live allocation count.

Compare snapshots across repeated shapes before deciding that changing memory use is a leak. In this example, the arena starts at zero, grows on first use, stays unchanged for the repeated and smaller inputs, grows for a much larger input, and remains at that size afterward. The full program uses the same session throughout; creating a new session for every row would test a different lifecycle.

How do you inspect ONNX GPU memory allocations in C#?

Register the GPU package before loading the session and request ComputeDevice.Gpu. After running with named input tensors, read the snapshot from OneModelGpu.GetInfo. The GPU memory documentation describes one arena per session, shared by the plans that session has prepared. Tensor lifetimes determine when its storage can be reused: once the last operator reading a tensor has run, another tensor can occupy that space.

The first small run here holds 16 bytes of constant weights and an arena of 1,048,576 bytes. Those figures describe this model and this run, not a minimum VRAM requirement for every model. The arena reserves more than this tiny graph’s four-row intermediates require. That is why increasing the input from four to eight rows would not necessarily cause another allocation. The example instead increases to 262,144 rows so the growth is visible.

After that larger shape, the observed arena is 4,194,304 bytes and the cumulative device-allocation count is three. Repeating that shape changes neither value. Returning to two rows also leaves both values unchanged. The arena grows when a plan needs more than any previous plan, but does not shrink simply because the next input is smaller. Retaining that capacity is the documented reuse behavior, not by itself evidence of a leak.

Use ArenaAllocations to separate arena growth from constant storage. Its sequence here is one allocation for the initial runs, then two after growth. ConstantBuffers separately reports the number of buffers holding uploaded constant weights. These counters give different views of session-owned buffers; do not add the cumulative allocation count to a byte total or interpret it as the number of currently live tensors.

They also are not a graphics-driver memory monitor. They do not promise to account for another session, another application, every driver allocation, or C# tensor arrays. A system tool’s VRAM total therefore need not equal ConstantBytes + ArenaBytes. Use these values to explain the session’s arena and weight behavior, with the adapter and API recorded alongside them, rather than presenting them as the whole device’s free-memory figure.

Dispose each SessionOutputs after reading it, dispose input tensors according to their ownership, and dispose the session when its work is finished. Session reuse and result disposal solve different problems: reusing the session preserves its arena and prepared weights, while disposing outputs releases the result objects. The example does both, so repeated runs measure retained session capacity rather than retained result collections.

All onemodel recipes