Softmax turns scores into probabilities. This example requires Vulkan 1.3.
dotnet add package RecursiveAI.OneGpu.Backends dotnet add package RecursiveAI.OneGpu.Kernels
Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var device = GpuBackends.CreateDevice( new GpuDeviceOptions { Adapter = "discrete" }); using var input = device.CreateBuffer<float>([1, 2, 3, 1, 2, 3]); using var output = device.CreateBuffer<float>(6); using var softmax = new SoftmaxKernels(device); var commands = device.BeginCommands(); softmax.Encode(commands, input, 0, [2, 3], 1, output, 0, mode: GpuKernelMode.Deterministic); device.Submit(commands); var result = new float[6]; await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result.Select(x => $"{x:F6}")));
Unzip the example, then dotnet run.
Prints:
0.090031, 0.244728, 0.665241, 0.090031, 0.244728, 0.665241The full program
The input is a [2, 3] matrix with both rows [1, 2, 3]. The program checks row sums and a CPU reference. Invalid shapes or axes throw GpuKernelShapeException before recording.
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var device = GpuBackends.CreateDevice( new GpuDeviceOptions { Adapter = "discrete" }); using var input = device.CreateBuffer<float>([1, 2, 3, 1, 2, 3]); using var output = device.CreateBuffer<float>(6); using var softmax = new SoftmaxKernels(device); var commands = device.BeginCommands(); softmax.Encode(commands, input, 0, [2, 3], 1, output, 0, mode: GpuKernelMode.Deterministic); device.Submit(commands); var result = new float[6]; await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result.Select(x => $"{x:F6}"))); for (int row = 0; row < 2; row++) { var sum = result.Skip(row * 3).Take(3).Sum(); Console.WriteLine($"Row {row} sum: {sum:F6}"); for (int col = 0; col < 3; col++) { double expected = Math.Exp(col - 2) / (Math.Exp(-2) + Math.Exp(-1) + 1); if (Math.Abs(result[row * 3 + col] - expected) > 1e-5) throw new Exception("Softmax mismatch"); } } Console.WriteLine("CPU tolerance 1e-5: passed"); Console.WriteLine($"{device.Backend}: {device.Adapter.Name}");
Output from an NVIDIA GeForce RTX 4070 using Vulkan:
0.090031, 0.244728, 0.665241, 0.090031, 0.244728, 0.665241 Row 0 sum: 1.000000 Row 1 sum: 1.000000 CPU tolerance 1e-5: passed Vulkan: NVIDIA GeForce RTX 4070
Variations
Log-softmax along the last axis
Append this to the full program:
var logCommands = device.BeginCommands(); softmax.Encode(logCommands, input, 0, [2, 3], -1, output, 0, logarithm: true, mode: GpuKernelMode.Deterministic); device.Submit(logCommands); await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result.Select(x => $"{x:F6}")));
Additional output:
-2.407606, -1.407606, -0.407606, -2.407606, -1.407606, -0.407606Related recipes
- Max and average pooling on the GPU in C#
- Gather embedding rows on the GPU in C#
- Dynamic tensor quantization on the GPU in C#
The short answer
These console examples use Vulkan on Linux or Windows with a Vulkan 1.3 driver.
Use SoftmaxKernels.Encode with an explicit shape and axis, submit the command encoder, then read the output buffer. The float32 tensor stays the same shape. For [2, 3] and axis 1, each group of three scores becomes a probability distribution independently. Axis -1 means the last dimension and selects the same operation here.
This example uses deterministic mode and an absolute CPU comparison tolerance of 1e-5. It checks every output against a double-precision reference, not just the row sums. A sum near one alone would not detect a row whose probabilities were permuted. The tolerance is this example’s check; the library’s kernel tolerance table defines its own scaled-error tests.
How do I implement softmax in C# on the GPU?
Upload dense row-major scores into a GpuBuffer<float>. Create a second buffer with the same element count, and create one SoftmaxKernels family for the device. The snippet at the top records the operation; Submit sends it to the device. ReadAsync waits for the result and copies it into the C# array.
The first row in this example is [1, 2, 3], and the second is identical. Both produce approximately [0.090031, 0.244728, 0.665241]. No shader source file or model file is needed: the Kernels package provides the compiled operation. Keep the family and buffers alive until the work and readback finish. Reuse them for repeated calls rather than recreating the family for every tensor; families cache pipelines and buffer bindings and are not thread-safe.
Changing the axis changes which values share a normalizing sum. On this matrix, axis zero would normalize pairs down the columns instead of three values across each row. Do not flatten a batch and normalize all its scores together unless that is actually the operation your model requires.
How do I prevent overflow in a C# softmax implementation?
The library uses a numerically stable form that subtracts the row maximum and computes statistics with an online normalizer. This avoids directly exponentiating large positive logits. Its softmax implementation documents the exceptional cases: a row containing NaN, positive infinity, or only negative infinities produces NaN throughout. A negative infinity beside finite entries produces probability zero.
Deterministic mode fixes the reduction tree for the workgroup form. It does not turn float32 into exact real arithmetic, and it does not remove the need for a numerical tolerance when comparing with a CPU formula. Short rows use a single thread; other row lengths select grouped or full-workgroup paths automatically.
For Half buffers, accumulation is single precision and the result is rounded on output. The softmax axis must be last, the row length must be even, and element offsets must be even. Float32 is the simpler starting point when the axis is not last.
What is log-softmax?
Log-softmax returns log probabilities directly. Set logarithm: true as in the variation instead of applying a separate logarithm to already rounded probabilities. On the example rows its outputs are approximately [-2.407606, -1.407606, -0.407606]. Exponentiating these values recovers the probabilities, within floating-point rounding. Choose this output when the next operation expects log probabilities; choose ordinary softmax when it expects probabilities.