Normalize tensor rows on a GPU; a working backend and driver are required.
dotnet add package RecursiveAI.OneGpu.Backends dotnet add package RecursiveAI.OneGpu.Kernels
Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var gpu = GpuBackends.CreateDevice(); using var x = gpu.CreateBuffer<float>( [1, 2, 3, 4]); using var y = gpu.CreateBuffer<float>(4); using var norm = new NormKernels(gpu); var commands = gpu.BeginCommands(); norm.Encode(commands, false, x, 0, 1, 4, null, 0, null, 0, y, 0); gpu.Submit(commands); var values = new float[4]; await y.ReadAsync(values); Console.WriteLine(string.Join(", ", values.Select(v => $"{v:F6}")));
false selects layer normalization, 1 and 4 mean one row of four values, null means no scale or bias buffer, and each 0 is an element offset.
-1.341635, -0.447212, 0.447212, 1.341635Unzip the example, then dotnet run.
The full program
The full program normalizes the row [1, 2, 3, 4], reads the result and compares it with a double-precision CPU formula using the documented normalized-error tolerance. With no ONEGPU_ADAPTER override, Adapter = "discrete" throws GpuUnsupportedException if only an integrated GPU is available; omit the adapter option to use the default device (adapter selection).
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var device = GpuBackends.CreateDevice( new GpuDeviceOptions { Adapter = "discrete" }); using var input = device.CreateBuffer<float>([1, 2, 3, 4]); using var output = device.CreateBuffer<float>(4); using var norm = new NormKernels(device); var commands = device.BeginCommands(); norm.Encode(commands, false, input, 0, 1, 4, null, 0, null, 0, output, 0); device.Submit(commands); var result = new float[4]; await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result.Select(x => $"{x:F6}"))); for (int i = 0; i < result.Length; i++) { double expected = (i + 1 - 2.5) / Math.Sqrt(1.25 + 1e-5f); if (Math.Abs(result[i] - expected) / (1.5 / Math.Sqrt(1.25 + 1e-5f)) > 1e-5) throw new Exception("Layer normalization mismatch."); } Console.WriteLine("CPU normalized error <= 1e-5: passed"); Console.WriteLine($"{device.Backend}: {device.Adapter.Name}");
dotnet run prints:
-1.341635, -0.447212, 0.447212, 1.341635 CPU normalized error <= 1e-5: passed Vulkan: NVIDIA GeForce RTX 4070
Invalid dimensions or out-of-buffer ranges throw GpuKernelShapeException; device creation also requires an available backend and compatible driver.
Variations
Use RMS normalization instead. Replace the norm.Encode call in the full program with this call and omit the layer-normalization CPU comparison loop and its success message:
norm.Encode(commands, true, input, 0, 1, 4, null, 0, null, 0, output, 0);
0.365148, 0.730296, 1.095444, 1.460593 Vulkan: NVIDIA GeForce RTX 4070
Related recipes
The short answer
Use NormKernels.Encode with rms: false to perform layer normalization. Supply the row count and row length, encode into a command buffer, submit it, and read the output buffer asynchronously. Optional scale and bias buffers apply per column. Passing null for them means a scale of one and a bias of zero.
This is an actual GPU dispatch, not a CPU fallback. The printed result above came from Vulkan on an NVIDIA GeForce RTX 4070. OneGPU also documents Metal and browser WebGPU backends, but those were not exercised for this recipe. Check that your deployment has its required backend and driver rather than treating a successful NuGet restore as evidence that a GPU is available.
How do I run layer normalization on the GPU from C#?
Create device buffers for your input and output, construct NormKernels on the same device, and encode the normalization. The shape is rows of consecutive values, with normalization over the last axis. For a larger tensor, rows is the product of the preceding dimensions and length is the size of that last dimension. Offsets are element offsets, not byte offsets.
For each row, layer normalization subtracts the row mean and divides by the square root of the row variance plus epsilon. The default epsilon is 1e-5f. The sample row has mean 2.5 and variance 1.25, which is why its output is symmetric around zero. Scale and bias, if supplied, follow that normalization rather than changing the mean and variance calculation.
Encoding records work; Submit dispatches it. ReadAsync copies the finished result into the C# array. Keep the device, kernel family and buffers alive until the readback completes. The using declarations in the complete program dispose them afterward. When integrating the operation into a larger GPU pipeline, leave the result in its GPU buffer until another CPU operation genuinely needs it.
The kernel documentation defines the comparison measure as the maximum absolute difference divided by the maximum absolute reference value over the output. For float layer normalization, its test tolerance is 1e-5; for half with float accumulation it is 2e-3. The complete program uses the float tolerance against a double-precision reference, not bitwise equality. These are documented test tolerances, not a promise about every tensor’s conditioning.
How do I switch between layer normalization and RMS normalization?
The boolean immediately after the encoder selects the operation: false for layer normalization and true for RMS normalization. RMS normalization scales by the root mean square without subtracting the mean. Consequently, the positive sample row stays positive in the variation, rather than producing negative and positive values around zero. Bias is ignored for RMS normalization; scale still applies.
Both operations offer fast and deterministic modes. The default is fast. Deterministic mode fixes the accumulation order as described in the kernel docs; choosing it does not change RMS normalization into layer normalization or remove the need for a numerical error check. Compare against the formula for the operation you actually requested.
Float inputs are the simplest place to start. Half tensors have additional packing requirements: normalization rows must have even lengths and even element offsets. Buffer ranges must cover all rows and each optional scale or bias must cover a row’s columns. The public API checks shapes and ranges, but cannot decide whether your chosen axis matches the model you intended. Write down the row layout before flattening a tensor into buffers.