Recursive AI

Use HLSL append and consume buffers in C#

OneShader is a free, open-source HLSL compiler library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Add or remove shader elements through a counter. This host needs Vulkan 1.2; compilation also supports Metal, not WebGPU.

Terminal
dotnet add package RecursiveAI.OneShader.Build
dotnet add package Silk.NET.Vulkan

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

The zip supplies VulkanHost and Stream.hlsl; the build generates Stream. Running needs a Vulkan loader and compute queue.

C#
using Stream = project.Shaders.Stream;
var result = VulkanHost.Run(
    Stream.CSMain.SpirV,
    [0, 1, 2, 3],
    [[10, 20, 30, 40], [4],
     new uint[4], [0]]);
Array.Sort(result[2]);
Console.WriteLine(
    string.Join(", ", result[2]));
Console.WriteLine($"Count: {result[3][0]}");
Text
11, 21, 31, 41
Count: 4

Unzip the example, then dotnet run.

The full program

Input: Stream.hlsl. Save this file beside Program.cs.

HLSL
[[vk::counter_binding(1)]]
ConsumeStructuredBuffer<uint> Source : register(u0);
[[vk::counter_binding(3)]]
AppendStructuredBuffer<uint> Hits : register(u2);
 
[shader("compute")]
[numthreads(4, 1, 1)]
void CSMain(uint3 id : SV_DispatchThreadID)
{
    Hits.Append(Source.Consume() + 1u);
}

The build package generates the project.Shaders.Stream class, including Bindings.Source.CounterBinding and Bindings.Hits.CounterBinding. The project in the zip pins the packages and enables unsafe code for its Vulkan host:

XML
<PropertyGroup>
  <TargetFramework>net10.0</TargetFramework>
  <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
</PropertyGroup>
<ItemGroup>
  <PackageReference Include="RecursiveAI.OneShader.Build" Version="1.7.1" />
  <PackageReference Include="Silk.NET.Vulkan" Version="2.22.0" />
</ItemGroup>

The full program uses generated constants rather than numeric bindings. VulkanHost.cs in the zip creates a Vulkan device, uploads four storage buffers, binds them in set 0 and dispatches one group. It inserts a compute-write-to-host-read barrier and waits for completion before mapping results.

C#
using Stream = project.Shaders.Stream;
 
foreach (var binding in Stream.Bindings.All)
    Console.WriteLine($"{binding.Name}: set={binding.Set}, binding={binding.Binding}");
uint[] bindings = [Stream.Bindings.Source.Binding,
    Stream.Bindings.Source.CounterBinding, Stream.Bindings.Hits.Binding,
    Stream.Bindings.Hits.CounterBinding];
var output = VulkanHost.Run(Stream.CSMain.SpirV, bindings,
    [[10, 20, 30, 40], [4], new uint[4], [0]]);
Array.Sort(output[2]);
if (output[1][0] != 0 || output[3][0] != 4 ||
    !output[2].SequenceEqual(new uint[] { 11, 21, 31, 41 }))
    throw new InvalidOperationException("Unexpected count or payload.");
Console.WriteLine($"Source count: {output[1][0]}");
Console.WriteLine($"Hits count: {output[3][0]}");
Console.WriteLine($"Sorted hits: {string.Join(", ", output[2])}");
Text
Source: set=0, binding=0
Source.counter: set=0, binding=1
Hits: set=0, binding=2
Hits.counter: set=0, binding=3
Source count: 0
Hits count: 4
Sorted hits: 11, 21, 31, 41

Shader errors fail the build; the host throws if Vulkan setup fails or if readback has an unexpected count or payload.

Variations

Compile the same file to Metal source instead of dispatching it. The runtime compiler is available through the build package’s dependency:

C#
using OneShader;
var result = ShaderCompiler.CompileFile("Stream.hlsl",
    new CompileOptions { Targets = [CompileTarget.Msl] });
if (!result.Success)
    throw new Exception(string.Join("\n", result.Diagnostics));
File.WriteAllBytes("stream.metal",
    result.Find("CSMain", CompileTarget.Msl)!.ToArray());
Console.WriteLine("Wrote stream.metal");
Text
Wrote stream.metal

This variation compiles Metal source; it does not run a Metal dispatch.

The short answer

Declare AppendStructuredBuffer<T> or ConsumeStructuredBuffer<T> in HLSL and bind two resources for each: the element storage buffer and a counter storage buffer in the same descriptor set. OneShader generates both descriptors and a CounterBinding constant. Initialize the append counter to zero; initialize the consume counter to the number of valid input elements.

The sample consumes four integers and appends each value plus one. Four threads consume exactly four inputs and produce exactly four outputs. It sorts the readback before comparing it, so the check does not depend on the order in which threads claim counter slots. OneShader compiles the shader; the included Vulkan host creates resources and submits the GPU work.

How do I use an HLSL append and consume buffer from C#?

Add the build package and place the HLSL file in the project. It compiles the shader during dotnet build and generates a C# class named after the file. The namespace is the project’s root namespace followed by .Shaders; this example’s project is named project.

Use the generated element and counter bindings when writing the host’s descriptor set. In this shader, Source uses binding 0 and its counter uses binding 1. Hits uses binding 2 and its counter uses binding 3. All are in set 0. The full program prints this generated metadata before dispatch, making a mismatched host layout visible without guessing the compiler’s numbering.

The input arrays passed to the host are the complete initial buffer contents: four source integers, a one-word source counter containing four, four empty output slots, and a one-word append counter containing zero. These counters are actual storage buffers, not ordinary CPU variables. Reset or reinitialize them before reusing a list for a new independent dispatch.

The supplied host is deliberately bounded: one dispatch group, four storage buffers, host-visible coherent memory and a compute queue. It is not a general rendering framework. In your own renderer, retain the compiler’s generated bindings but integrate allocation, synchronization and resource lifetime into your existing Vulkan code.

How does the AppendStructuredBuffer counter work?

The counter stores the number of elements. Append atomically adds one and stores at the old counter value. Consume subtracts one and reads the element below the old value. OneShader’s generated bindings guide specifies a counter buffer of at least four bytes in the same set as its element buffer.

GetDimensions reports capacity and stride, not the number of appended elements. Read the counter when you need the current list length. Do not use capacity as the number of safe consumes: this example has four valid elements because the host writes four into the source counter, not because the element buffer happens to have four slots.

The host must also bound the work. The sample dispatches one four-thread group, consumes four inputs and appends into four output slots. Increasing the dispatch count without increasing valid input and output capacity would break those bounds. A counter alone does not make an arbitrarily large dispatch safe.

The shader sets counter bindings explicitly with [[vk::counter_binding(n)]]. Without that attribute, the compiler chooses the next free binding in the resource’s set after resources declared so far. Prefer generated constants either way. Arrays of append or consume buffers are rejected, and RWStructuredBuffer.IncrementCounter and DecrementCounter are not supported. The WGSL backend rejects append and consume buffers; choose SPIR-V or Metal for this pattern.

All oneshader recipes