An embedding lookup selects table rows by integer ID. This example requires Vulkan 1.3.
dotnet add package RecursiveAI.OneGpu.Backends dotnet add package RecursiveAI.OneGpu.Kernels
Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var device = GpuBackends.CreateDevice( new GpuDeviceOptions { Adapter = "discrete" }); using var table = device.CreateBuffer<float>([10, 11, 20, 21, 30, 31]); using var indices = device.CreateBuffer<int>([2, 0, -1]); using var output = device.CreateBuffer<float>(6); using var gather = new GatherKernels(device); using var status = new GpuIndexStatus(device); var commands = device.BeginCommands(); gather.EncodeGather(commands, table, 0, [3, 2], 0, indices, 0, 3, output, 0, status); device.Submit(commands); await status.ThrowIfFailedAsync(); var result = new float[6]; await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result));
Unzip the example, then dotnet run.
Prints:
30, 31, 10, 11, 30, 31The full program
The table is [3, 2] and the indices are [2, 0, -1]. Invalid IDs write zeros for affected elements; checking the supplied status throws GpuKernelIndexException after execution.
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var device = GpuBackends.CreateDevice( new GpuDeviceOptions { Adapter = "discrete" }); using var table = device.CreateBuffer<float>([10, 11, 20, 21, 30, 31]); using var indices = device.CreateBuffer<int>([2, 0, -1]); using var output = device.CreateBuffer<float>(6); using var gather = new GatherKernels(device); using var status = new GpuIndexStatus(device); var commands = device.BeginCommands(); gather.EncodeGather(commands, table, 0, [3, 2], 0, indices, 0, 3, output, 0, status); device.Submit(commands); await status.ThrowIfFailedAsync(); var result = new float[6]; await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result)); if (!result.SequenceEqual(new float[] { 30, 31, 10, 11, 30, 31 })) throw new Exception("Gather mismatch"); Console.WriteLine("Expected rows: passed"); Console.WriteLine($"{device.Backend}: {device.Adapter.Name}");
Output from an NVIDIA GeForce RTX 4070 using Vulkan:
30, 31, 10, 11, 30, 31 Expected rows: passed Vulkan: NVIDIA GeForce RTX 4070
Variations
Int64 indices and an invalid row ID
Append this to the full program:
using var longIndices = device.CreateBuffer<long>([-1, 1, 0]); status.Reset(); var pass = device.BeginCommands(); gather.EncodeGather(pass, table, 0, [3, 2], 0, longIndices, 0, 3, output, 0, status); device.Submit(pass); await status.ThrowIfFailedAsync(); await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result)); // Now deliberately ask for row 3 in a table with rows 0..2. indices.Write([3, 0, 1]); status.Reset(); var invalid = device.BeginCommands(); gather.EncodeGather(invalid, table, 0, [3, 2], 0, indices, 0, 3, output, 0, status); device.Submit(invalid); try { await status.ThrowIfFailedAsync(); } catch (GpuKernelIndexException error) { Console.WriteLine(error.Message); }
Additional output:
30, 31, 20, 21, 10, 11 Gather: the index at position 0 of the indices is outside its axis
Related recipes
- Compute softmax on the GPU in C#
- Max and average pooling on the GPU in C#
- Dynamic tensor quantization on the GPU in C#
The short answer
These console examples use Vulkan on Linux or Windows with a Vulkan 1.3 driver.
Use GatherKernels.EncodeGather with a table shape [rowCount, embeddingWidth] and axis zero. Upload integer row IDs, allocate indexCount * embeddingWidth output elements, and pass a GpuIndexStatus so invalid IDs are reported after submission. This selects existing vectors; it does not generate embeddings from text.
The table in this recipe has rows [10, 11], [20, 21] and [30, 31]. IDs [2, 0, -1] return the third row, first row and third row again. The full program checks every selected value exactly. There is no floating-point arithmetic in the selection itself, so no tolerance is needed for this fixture.
How does gather implement an embedding lookup?
A row ID chooses a vector from a dense table. The GPU copies that row’s payload to the next output row. In this example, each vector contains two float32 elements, so each ID produces two adjacent output values. Repeated IDs intentionally repeat their vectors. Their order in the output is the order of the IDs, not a sorted order of table rows.
More generally, Gather replaces the selected data axis with the shape of the indices. Around that axis, the library treats the data as outer dimensions, the axis length and inner dimensions. For an embedding table gathered along axis zero, the inner extent is the embedding width. The API takes the number of indices; your application tracks their shape when interpreting the flat result.
All tensors are dense and row-major. Element offsets are measured in elements of the buffer’s type, not bytes. The output must not overlap the data. Keep the table resident when selecting several batches of IDs, and reuse the family so its cached pipelines and bindings can be reused. Families are not thread-safe; arrange their use accordingly. The kernel guide describes this buffer-based model.
Can a GPU gather use negative or int64 indices?
Yes. An index of -1 means the final entry on the selected axis; -rowCount means its first entry. For a three-row table, valid IDs are from -3 through 2. A negative index is not a padding ID or a request for a zero vector.
Both C# int and long index buffers are supported. The variation uses GpuBuffer<long> and returns rows for [-1, 1, 0]. You do not have to change the float payload type when changing the index type. Int64 indices are represented with pairs of 32-bit words, so they do not require native 64-bit integer operations on the GPU.
Float is not the only payload option: the documented family also moves Half, int, uint, long, byte and sbyte payloads. Packed narrow types have word-alignment rules, so check the buffer padding requirements before replacing this float32 fixture.
What happens when a gather index is out of range?
An invalid index never reads outside the table. The kernel writes zero for the affected output element and records the smallest offending index position in the supplied status. ThrowIfFailedAsync waits for completion, reads that record and throws GpuKernelIndexException. In the variation, ID 3 is invalid and the exception identifies position zero.
Do not consume the result as a successful embedding batch before checking the status. Passing no status makes invalid indices produce zeros silently, which is usually not the error behavior wanted for an embedding lookup. Call Reset before reusing a status for another independent batch; otherwise a previous failure remains recorded. If the IDs are already on the CPU, the library also exposes host-side index validation before upload. Shape mistakes, such as an invalid axis or insufficient output capacity, are separate GpuKernelShapeException errors checked before recording.