Dynamic quantization maps floats to bytes using their range. This example requires Vulkan 1.3.
dotnet add package RecursiveAI.OneGpu.Backends dotnet add package RecursiveAI.OneGpu.Kernels
Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var device = GpuBackends.CreateDevice( new GpuDeviceOptions { Adapter = "discrete" }); using var input = device.CreateBuffer<float>([-1, 0, 1, 2]); using var output = device.CreateBuffer<byte>(4); using var scale = device.CreateBuffer<float>(1); using var zero = device.CreateBuffer<byte>(4); using var quantize = new DynamicQuantizeKernels(device); var commands = device.BeginCommands(); quantize.Encode(commands, input, 0, 4, output, 0, scale, 0, zero, 0); device.Submit(commands); var result = new byte[4]; await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result));
Unzip the example, then dotnet run.
Prints:
0, 85, 170, 255The full program
The input is [-1, 0, 1, 2]. The program reads scale and zero point, reconstructs the floats and checks the error. Invalid counts or buffer ranges throw GpuKernelShapeException before recording.
using RecursiveAI.OneGpu; using RecursiveAI.OneGpu.Kernels; using var device = GpuBackends.CreateDevice( new GpuDeviceOptions { Adapter = "discrete" }); using var input = device.CreateBuffer<float>([-1, 0, 1, 2]); using var output = device.CreateBuffer<byte>(4); using var scale = device.CreateBuffer<float>(1); using var zero = device.CreateBuffer<byte>(4); using var quantize = new DynamicQuantizeKernels(device); var commands = device.BeginCommands(); quantize.Encode(commands, input, 0, 4, output, 0, scale, 0, zero, 0); device.Submit(commands); var result = new byte[4]; await output.ReadAsync(result); Console.WriteLine(string.Join(", ", result)); var scales = new float[1]; var zeros = new byte[4]; await scale.ReadAsync(scales); await zero.ReadAsync(zeros); Console.WriteLine($"Scale: {scales[0]:F9}; zero point: {zeros[0]}"); using var restored = device.CreateBuffer<float>(4); using var dequantize = new DequantizeKernels(device); var decode = device.BeginCommands(); dequantize.Encode(decode, output, 0, [4], 0, false, scale, 0, zero, 0, restored, 0); device.Submit(decode); var reconstructed = new float[4]; await restored.ReadAsync(reconstructed); float[] original = [-1, 0, 1, 2]; float error = reconstructed.Zip(original, (a, b) => Math.Abs(a - b)).Max(); Console.WriteLine($"Reconstructed: {string.Join(", ", reconstructed)}"); if (error > scales[0] / 2 + 1e-6f) throw new Exception("Error too large"); Console.WriteLine("Half-scale error bound: passed"); Console.WriteLine($"{device.Backend}: {device.Adapter.Name}");
Output from an NVIDIA GeForce RTX 4070 using Vulkan:
0, 85, 170, 255 Scale: 0.011764706; zero point: 85 Reconstructed: -1, 0, 1, 2 Half-scale error bound: passed Vulkan: NVIDIA GeForce RTX 4070
Variations
An all-zero constant tensor
Append this to the full program:
input.Write([0, 0, 0, 0]); var constant = device.BeginCommands(); quantize.Encode(constant, input, 0, 4, output, 0, scale, 0, zero, 0); device.Submit(constant); await output.ReadAsync(result); await scale.ReadAsync(scales); await zero.ReadAsync(zeros); Console.WriteLine(string.Join(", ", result)); Console.WriteLine($"Scale: {scales[0]}; zero point: {zeros[0]}");
Additional output:
0, 0, 0, 0 Scale: 1; zero point: 0
Related recipes
- Compute softmax on the GPU in C#
- Max and average pooling on the GPU in C#
- Gather embedding rows on the GPU in C#
The short answer
These console examples use Vulkan on Linux or Windows with a Vulkan 1.3 driver.
DynamicQuantizeKernels.Encode takes float32 values, finds their range on the GPU and writes uint8 values plus a float32 scale and uint8 zero point into device buffers. The full program feeds those buffers to DequantizeKernels and checks reconstruction against the original fixture. No ONNX graph or calibration file is needed.
For [-1, 0, 1, 2], the result is [0, 85, 170, 255], with scale approximately 0.011764706 and zero point 85. The metadata matters: the bytes alone do not specify the original real-valued tensor. Keep scale and zero point alongside the quantized buffer for subsequent operations.
Can you explain quantization in a simple way?
This operation places float values on a grid of 256 unsigned integer codes. A scale says how far apart adjacent codes are in real units. A zero point says which code represents real zero. Dequantization computes (code - zeroPoint) * scale and produces a floating-point approximation to the original value.
OneGpu’s dynamic quantization implementation defines the range as lo = min(0, min x) and hi = max(0, max x). Scale is (hi - lo) / 255. The zero point is the rounded value of -(lo / scale), clamped into [0, 255]. Each output rounds x / scale, adds the zero point and clamps again.
Rounding is nearest-even, not always away from zero. NaN maps to code zero. The library uses integer arithmetic for correctly rounded float32 division rather than relying on the GPU’s native division. Both kernel modes therefore give the same bits for this operation. It records two passes and keeps a small scratch buffer inside the family; the family is not thread-safe.
What is the difference between static and dynamic quantization?
Dynamic quantization derives parameters from the current input at runtime. Static quantization uses parameters established beforehand. The ONNX DynamicQuantizeLinear definition describes the runtime range and output parameters used here. OneGpu’s dynamic operation computes one scale and one zero point for the entire input tensor, not one per channel.
If your application already has scales and zero points, DequantizeKernels.EncodeQuantize provides QuantizeLinear instead. That API accepts per-tensor or per-axis parameters and supports uint8 and int8 output. Do not use a newly measured dynamic scale when the next layer expects a fixed scale supplied with trained weights. Matching the next operation’s data type and parameter convention is part of connecting quantized layers correctly.
Does quantization reduce accuracy?
It can: many float values map to the same code. The four values in this fixture happen to reconstruct exactly on the tested device, but that is not a general promise. The full program checks an absolute reconstruction error bound of half the measured scale plus 1e-6 for float rounding. Treat that as a check on this finite fixture, not a universal bound for arbitrary magnitudes or nonfinite inputs.
An all-zero tensor is a special case: the operation returns scale one, zero point zero and all-zero codes. The variation exercises it explicitly. Constant nonzero input is different because the range still includes real zero; a constant positive tensor does not have a zero-width quantization range.
Byte buffers must end on a whole 32-bit word. This example has four payload bytes and allocates four bytes for the zero-point buffer even though only its first value is used. For other lengths, use the library’s padded-buffer sizing rules and retain the logical element count separately. Readback is only for inspection here; scale, zero point and codes already reside on the GPU for the next kernel.