Recursive AI

Sum or average tensor axes on the GPU in C#

OneGpu is a free, open-source GPU compute library for .NET from Recursive AI. MIT or Apache-2.0, no bundled native binaries.

Use EncodeReduce with Sum or Mean to combine values along selected tensor axes. On Windows or Linux, use a Vulkan 1.3 driver and loader.

Terminal
dotnet add package RecursiveAI.OneGpu.Backends
dotnet add package RecursiveAI.OneGpu.Kernels

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneGpu;
using RecursiveAI.OneGpu.Kernels;
 
using var gpu =
    GpuBackends.CreateDevice();
using var reduce = new ReduceKernels(gpu);
using var input = gpu.CreateBuffer<float>(
    [1, 3, 5, 7, 9, 11, 13, 15]);
// Output size: multiply the kept dimensions.
// Only axis 1 is kept, so the size is 2.
using var sum = gpu.CreateBuffer<float>(2);
using var mean = gpu.CreateBuffer<float>(2);
var cmd = gpu.BeginCommands();
reduce.EncodeReduce(cmd, GpuReduceOp.Sum,
    input, 0, [2, 2, 2], [0, 2], sum, 0);
reduce.EncodeReduce(cmd, GpuReduceOp.Mean,
    input, 0, [2, 2, 2], [0, 2], mean, 0);
gpu.Submit(cmd);
var values = new float[2];
await sum.ReadAsync(values);
Console.WriteLine("Sum: " +
    string.Join(", ", values));
await mean.ReadAsync(values);
Console.WriteLine("Mean: " +
    string.Join(", ", values));

Unzip the example, then dotnet run.

Prints a sum and mean for each position along the middle axis:

Text
Sum: 24, 40
Mean: 6, 10

The full program

The download checks the results against a CPU loop over both reduced axes. It requests deterministic mode for both GPU passes.

C#
using RecursiveAI.OneGpu;
using RecursiveAI.OneGpu.Kernels;
 
using var gpu = GpuBackends.CreateDevice();
float[] sample = [1, 3, 5, 7, 9, 11, 13, 15];
int[] shape = [2, 2, 2];
int[] axes = [0, 2];
 
using var input = gpu.CreateBuffer<float>(sample);
using var reduce = new ReduceKernels(gpu);
using var sum = gpu.CreateBuffer<float>(2);
using var mean = gpu.CreateBuffer<float>(2);
var commands = gpu.BeginCommands();
reduce.EncodeReduce(commands, GpuReduceOp.Sum,
    input, 0, shape, axes, sum, 0, GpuKernelMode.Deterministic);
reduce.EncodeReduce(commands, GpuReduceOp.Mean,
    input, 0, shape, axes, mean, 0, GpuKernelMode.Deterministic);
gpu.Submit(commands);
 
var sums = new float[2];
var means = new float[2];
await sum.ReadAsync(sums);
await mean.ReadAsync(means);
 
// Keep axis 1; visit both axis-0 and axis-2 coordinates.
for (int row = 0; row < shape[1]; row++)
{
    double cpuSum = 0;
    for (int batch = 0; batch < shape[0]; batch++)
        for (int col = 0; col < shape[2]; col++)
            cpuSum += sample[(batch * shape[1] + row) * shape[2] + col];
    double cpuMean = cpuSum / (shape[0] * shape[2]);
    if (sums[row] != cpuSum || means[row] != cpuMean)
        throw new InvalidOperationException($"GPU and CPU differ at row {row}.");
}
Console.WriteLine($"Sum: {string.Join(", ", sums)}");
Console.WriteLine($"Mean: {string.Join(", ", means)}");
Console.WriteLine("Both reductions matched the CPU.");
Console.WriteLine($"{gpu.Backend}: {gpu.Adapter.Name}");
Text
Sum: 24, 40
Mean: 6, 10
Both reductions matched the CPU.
Vulkan: NVIDIA GeForce RTX 4070

A repeated axis or an axis outside [-rank, rank) throws GpuKernelShapeException before the GPU pass is recorded.

Variations

Count axes from the end. Replace the Sum call with this. At rank three, -3 and -1 select axes 0 and 2, so the sum still prints 24, 40.

C#
reduce.EncodeReduce(cmd, GpuReduceOp.Sum,
    input, 0, [2, 2, 2], [-3, -1], sum, 0);

Sum all axes. An empty axes list reduces to one value. Replace the corresponding lines for sum in the snippet at the top; leave the mean and its two-element readback unchanged. This prints Sum: 64.

C#
using var sum = gpu.CreateBuffer<float>(1);
reduce.EncodeReduce(cmd, GpuReduceOp.Sum,
    input, 0, [2, 2, 2], [], sum, 0);
await sum.ReadAsync(values.AsMemory(0, 1));
Console.WriteLine($"Sum: {values[0]}");

The short answer

Use ReduceKernels.EncodeReduce with GpuReduceOp.Sum or GpuReduceOp.Mean, the input shape, and the axes to reduce. For [2, 2, 2] reduced over [0, 2], both operations write two values: one for each position along axis 1. Keep the outputs in GPU buffers if the next pass consumes them; use ReadAsync when the CPU needs the results.

How do you sum or average over multiple tensor axes?

An axis is a dimension, numbered from zero. Keeping axis 1 fixes the middle coordinate and lets the first and last coordinates vary. In the example, the first output combines 1, 3, 9, 11, and the second combines 5, 7, 13, 15. Sum adds each group. Mean divides each sum by four, the product of the two reduced dimensions. The divisor isn’t the number of output elements or the number of axes.

You can give the kernel nonadjacent axes in one call, as this example does. The release API supports up to four runs of kept dimensions and four runs of reduced dimensions. A run is a stretch of adjacent axes of the same kind. If your axes alternate through too many runs, the call rejects that shape.

What does keepdims do to the reduction output shape?

Keeping dimensions means retaining each reduced axis at size 1, a convention used by ONNX ReduceSum. The same output values in this recipe can be described as [2] without keepdims or [1, 2, 1] with it. OneGpu writes the same dense values either way. It doesn’t return a shape or take a keepdims argument; supply the shape your next operation expects.

To size your buffer, multiply only the kept dimensions. If none are kept, reserve one element for the scalar. That is different from reducing a dimension with length zero: an empty group has no values to average, so Mean returns NaN for that group.

Why can parallel floating-point sums differ from CPU sums?

A floating-point addition rounds, so grouping additions differently can change the result. By default, OneGpu’s Fast mode uses the device’s subgroup arithmetic when available and a shared-memory tree otherwise. Deterministic uses a fixed reduction tree instead, independent of subgroup size. It doesn’t turn the GPU reduction into a sequential CPU loop or change the single-precision accumulation.

The integers in this sample and their means are exactly representable as floats. For general input, compare to a CPU reference with a tolerance appropriate to your values, rather than using the exact equality in this example. The kernel docs explain how the library measures error, including when positive and negative terms nearly cancel.

All onegpu recipes