Recursive AI

Run scaled dot-product attention on the GPU in C#

OneGpu is a free, open-source GPU compute and rendering library for .NET from Recursive AI, on Vulkan, Metal and WebGPU. MIT or Apache-2.0, no bundled native binaries.

NuGet GitHub Docs Download example (.zip)

Attention combines values according to query–key scores. This example uses float32 on Vulkan; Linux and Windows need a Vulkan 1.3 driver.

Terminal
dotnet add package RecursiveAI.OneGpu.Backends
dotnet add package RecursiveAI.OneGpu.Kernels

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneGpu;
using RecursiveAI.OneGpu.Kernels;
 
using var device = GpuBackends.CreateDevice(
    new GpuDeviceOptions { Adapter = "discrete" });
using var q = device.CreateBuffer<float>([1, 0, 0, 1]);
using var k = device.CreateBuffer<float>([1, 0, 0, 1]);
using var v = device.CreateBuffer<float>([2, 4, 6, 8]);
using var mask = device.CreateBuffer<float>([0, float.NegativeInfinity]);
using var output = device.CreateBuffer<float>(4);
using var attention = new AttentionKernels(device);
var layout = GpuAttentionLayout.PositionsFirst(1, 2, 2);
var problem = new AttentionProblem(batch: 1, heads: 1, qLen: 2, kLen: 2, headDim: 2)
{
    Q = layout, K = layout, V = layout, O = layout,
    Mask = GpuAttentionLayout.ForMask([1, 1, 1, 2], 1, 1, 2, 2)
};
var commands = device.BeginCommands();
attention.Encode(commands, problem, q, k, v, output,
    mask: mask, mode: GpuKernelMode.Deterministic);
device.Submit(commands);
var result = new float[4];
await output.ReadAsync(result);
Console.WriteLine(string.Join(", ", result));

Unzip the example, then dotnet run.

Prints:

Text
2, 4, 2, 4

The full program

The snippet above, plus the CPU check and device name below, is the complete Program.cs in the zip:

C#
// CPU softmax reference, additive mask broadcast over both query rows.
float[] queries = [1, 0, 0, 1], keys = [1, 0, 0, 1];
float[] values = [2, 4, 6, 8], maskValues = [0, float.NegativeInfinity];
var expected = new double[4];
for (int row = 0; row < 2; row++)
{
    var scores = new double[2];
    for (int key = 0; key < 2; key++)
        scores[key] = (queries[row * 2] * keys[key * 2]
            + queries[row * 2 + 1] * keys[key * 2 + 1]) / Math.Sqrt(2)
            + maskValues[key];
    double max = scores.Max();
    var probabilities = scores.Select(s => Math.Exp(s - max)).ToArray();
    double sum = probabilities.Sum();
    for (int col = 0; col < 2; col++)
        for (int key = 0; key < 2; key++)
            expected[row * 2 + col] += probabilities[key] / sum * values[key * 2 + col];
}
Console.WriteLine($"CPU match: {result.Zip(expected).All(p => Math.Abs(p.First - p.Second) < 1e-6)}");
Console.WriteLine($"{device.Backend}: {device.Adapter.Name}");

The example has batch 1, one head, two queries, two keys and head dimension 2. Each Q and K row is a basis vector. V’s rows are [2, 4] and [6, 8]. The additive mask [0, −∞] hides the second key for both queries, so both output rows equal the first V row. All inputs are literal arrays; no model or file is needed.

The full program includes the softmax calculation on the CPU and compares within an absolute tolerance of 1e-6. It ran on an NVIDIA GeForce RTX 4070 through Vulkan. Encode records the kernel; Submit sends it to the device, and the awaited readback completes before disposal.

Invalid shapes or ranges throw GpuKernelShapeException before work is recorded; each tensor must fit one storage-buffer binding.

Text
2, 4, 2, 4
CPU match: True
Vulkan: NVIDIA GeForce RTX 4070

Without an adapter override, Adapter = "discrete" throws GpuUnsupportedException on an integrated-only machine; macOS defaults to Metal (Apple arm64 only), so omit the adapter filter or use "integrated" on Apple silicon.

Variations

Keep both keys visible

Replace the mask allocation in the program. Zero adds nothing to either score:

C#
using var mask = device.CreateBuffer<float>([0, 0]);
Text
3.3209538, 5.320954, 4.679046, 6.679046

The short answer

Use AttentionKernels.Encode to compute O = softmax(Scale · QKᵀ + Mask)V, then submit the encoder and read O. The kernel performs the normalization and value weighting, not just a matrix product. This is one attention operation on tensors you supply, not a complete language-model session. The attention docs describe the supported layouts and masking rules.

The example deliberately hides one key. That makes the expected result easy to inspect without removing softmax from the operation. With one visible key, its probability is one for every query. The variation makes both keys visible, so the scores determine a mixture of the two V rows instead.

What is the scaled dot-product attention formula?

For each query row, take its dot product with every key, multiply by the scale, add the mask, and normalize those scores with softmax. Multiply each value row by its probability and sum. The default scale is 1 / sqrt(HeadDim), so it is 1 / sqrt(2) here. The CPU loop in the full program spells out these steps, including subtracting the largest score before exponentiation.

Scaling happens before adding the mask. The mask is an additive float tensor, not a Boolean “keep” array. A zero leaves a score alone; negative infinity hides a key. A finite negative number lowers a key’s weight rather than necessarily removing it. The kernel’s documented implementation keeps a running normalization and value accumulator without writing the full score matrix to memory.

How do I implement scaled dot-product attention in C#?

Choose the logical dimensions before allocating buffers. PositionsFirst(1, 2, 2) describes [batch, positions, heads, headDim]; it does not transpose or copy your data. Here Q, K, V and O each occupy four floats. Offsets and strides are in elements. When using slices from a packed projection, describe where rows really sit rather than assuming a dense allocation.

ForMask([1, 1, 1, 2], 1, 1, 2, 2) broadcasts a two-key mask over batch, heads and query rows. A separate per-query mask would need a query dimension of two and enough stored elements. Query and key lengths can differ for cross-attention; allocate Q and O for query rows and K and V for key rows.

A row with no visible keys produces NaN in every output by default. MaskedRows = GpuAttentionMaskedRows.Zero selects zeros instead. A score that is NaN or positive infinity also produces NaN. These are arithmetic semantics, not evidence that a readback failed. Keep at least one visible key per row when that is what your model expects.

The sample requests GpuKernelMode.Deterministic. That does not promise matching last bits on different GPU vendors or driver versions. Compare against a numerical reference with a tolerance suited to your tensors, rather than treating this tiny exact result as a guarantee for every input.

All onegpu recipes