Recursive AI

Run GPU compute with float16 buffers in C#

OneGpu is a free, open-source GPU compute library for .NET from Recursive AI. MIT or Apache-2.0, no bundled native binaries.

Upload float16 (Half) values, run ReLU to clamp negatives to zero, and read the result back. Requires Vulkan 1.3 on Windows or Linux.

Terminal
dotnet add package RecursiveAI.OneGpu.Backends
dotnet add package RecursiveAI.OneGpu.Kernels

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

The kernel packs two halves into each 32-bit word, so an odd count needs one padding slot.

C#
using System;
using RecursiveAI.OneGpu.Kernels;
using RecursiveAI.OneGpu.Vulkan;
 
using var gpu = VulkanGpu.CreateDevice();
int size =
    GpuKernelBuffers.PaddedHalfLength(3);
ulong bytes = (ulong)size * 2; // 2 bytes per half
var caps = gpu.Capabilities.Compute;
if (caps.MaxWorkgroupInvocations < 256 ||
    caps.MaxStorageBufferBindingBytes < bytes)
    throw new NotSupportedException();
using var src = gpu.CreateBuffer<Half>(
    [(Half)(-2), (Half)0.1f, (Half)3.5f,
     (Half)0]); // Padding, not a fourth input
using var dst = gpu.CreateBuffer<Half>(size);
using var ops = new ElementwiseKernels(gpu);
var cmd = gpu.BeginCommands();
ops.EncodeUnary(cmd, GpuUnaryOp.Relu,
    src, 0, dst, 0, 3);
gpu.Submit(cmd);
var result = new Half[size];
await dst.ReadAsync(result);
foreach (var value in result.AsSpan(0, 3))
    Console.WriteLine(FormattableString
        .Invariant($"{(float)value}"));

Unzip the example, then dotnet run.

Prints three results, not the padding element:

Text
0
0.099975586
3.5

The full program

The full program checks device limits before creating or uploading the buffers. The packed shader uses two storage buffers and a workgroup of 256 threads. It does not require native float16 arithmetic. The CPU computes ReLU on the original float32 values, then the program rounds those results to Half for an exact comparison. OneGpu throws GpuKernelShapeException if a half buffer ends in the middle of a 32-bit word.

C#
using System;
using System.Globalization;
using RecursiveAI.OneGpu.Kernels;
using RecursiveAI.OneGpu.Vulkan;
 
CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
using var gpu = VulkanGpu.CreateDevice(
    new() { Adapter = "discrete" });
float[] input = [-2f, 0.1f, 3.5f];
int count = input.Length;
int size = GpuKernelBuffers.PaddedHalfLength(count);
var caps = gpu.Capabilities.Compute;
ulong bytes = (ulong)size * 2;
// The packed ReLU shader uses 256 threads per workgroup.
if (caps.MaxWorkgroupSizeX < 256 ||
    caps.MaxWorkgroupInvocations < 256 ||
    caps.MaxWorkgroupCount < 1 ||
    caps.MaxStorageBuffersPerStage < 2 ||
    caps.MaxStorageBufferBindingBytes < bytes ||
    caps.MaxBufferBytes < bytes)
    throw new NotSupportedException("Packed ReLU does not fit this GPU.");
 
var packed = new Half[size];
for (int i = 0; i < count; i++)
    packed[i] = (Half)input[i];
using var src = gpu.CreateBuffer<Half>(packed);
using var dst = gpu.CreateBuffer<Half>(size);
using var ops = new ElementwiseKernels(gpu);
var cmd = gpu.BeginCommands();
ops.EncodeUnary(cmd, GpuUnaryOp.Relu,
    src, 0, dst, 0, count);
gpu.Submit(cmd);
var result = new Half[size];
await dst.ReadAsync(result);
 
float[] reference = new float[count];
float[] actual = new float[count];
for (int i = 0; i < count; i++)
{
    reference[i] = MathF.Max(0, input[i]);
    actual[i] = (float)result[i];
    if (result[i] != (Half)reference[i])
        throw new InvalidOperationException("GPU and CPU results differ.");
}
Console.WriteLine(gpu.Adapter.Name);
Console.WriteLine($"FP16 GPU: {string.Join(", ", actual)}");
Console.WriteLine($"FP32 CPU: {string.Join(", ", reference)}");

Output on Linux with Vulkan:

Text
NVIDIA GeForce RTX 4070
FP16 GPU: 0, 0.099975586, 3.5
FP32 CPU: 0, 0.1, 3.5

Variations

Keep part of a negative input

Replace the EncodeUnary call in the snippet at the top with Leaky ReLU and a scale of 0.25:

C#
ops.EncodeUnary(cmd, GpuUnaryOp.LeakyRelu,
    src, 0, dst, 0, 3, p0: 0.25f);
Text
-0.5
0.099975586
3.5

The short answer

Create GpuBuffer<Half> buffers for the input and output. Allocate GpuKernelBuffers.PaddedHalfLength(count) elements in each buffer but pass the original count to EncodeUnary. The storage holds pairs of half values in 32-bit words. For three values, allocate four slots, run the kernel over three, and ignore the fourth on readback.

Does Vulkan need native float16 support for half-precision buffers?

Not for this packed OneGpu kernel. The kernel docs describe a shader that reads words containing two halves and unpacks them with integer operations. ReLU runs in single precision inside the shader and the result is rounded to the nearest half, with ties rounded to even. The type of the buffer describes the input and output storage, not the precision of the arithmetic. For a different shader that actually declares 16-bit storage or does arithmetic in float16, check the device’s shader capabilities instead of assuming a Half type on the C# side enables those features.

The example checks the padded byte size against the per-buffer and storage binding limits, and the shader’s workgroup width against the device limits. Those checks establish that this dispatch fits before sending data; checking for native float16 would reject devices unnecessarily. Keep the buffer layout and the logical shape separate if you switch kernels: for example, the half softmax kernel also requires an even row length, even element offsets, and the softmax axis last.

Why does float16 give a different result from float32?

The 0.1f input is rounded when cast to Half. ReLU doesn’t change a positive input, so reading that half back and widening it to float shows 0.099975586, not the original float32 0.1f. The negative and 3.5f inputs are exactly representable as halves, so their ReLU results match without that difference.

For ReLU, comparing the GPU result with the CPU result rounded to half works because a positive value passes through and a negative value becomes zero. For an operation such as a sum or a dot product, form your CPU reference from the same rounded input values the GPU receives, then compare with a tolerance appropriate to the operation. Otherwise, a difference caused by input conversion can look like a difference caused by the kernel.

All onegpu recipes