---
title: "Dynamic tensor quantization on the GPU in C#"
url: https://recursiveai.org/onegpu/cookbook/quantize-tensors-on-gpu/
description: "Dynamic tensor quantization on the GPU in C#. Includes buffer setup, real Vulkan output and a runnable .NET 10 example."
version: "1.0.0"
---

# Dynamic tensor quantization on the GPU in C#

**OneGpu** is a free, open-source GPU compute library for .NET from Recursive AI. MIT or Apache-2.0, no bundled native binaries.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneGpu.Backends) · [GitHub](https://github.com/recursive-ai-oss/onegpu-csharp) · [Docs](https://recursiveai.org/onegpu/docs/) · [Download example (.zip)](https://recursiveai.org/onegpu/cookbook/quantize-tensors-on-gpu/quantize-tensors-on-gpu.zip)

Dynamic quantization maps floats to bytes using their range. This example requires Vulkan 1.3.

```shell
dotnet add package RecursiveAI.OneGpu.Backends
dotnet add package RecursiveAI.OneGpu.Kernels
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneGpu;
using RecursiveAI.OneGpu.Kernels;

using var device = GpuBackends.CreateDevice(
    new GpuDeviceOptions { Adapter = "discrete" });
using var input = device.CreateBuffer<float>([-1, 0, 1, 2]);
using var output = device.CreateBuffer<byte>(4);
using var scale = device.CreateBuffer<float>(1);
using var zero = device.CreateBuffer<byte>(4);
using var quantize = new DynamicQuantizeKernels(device);
var commands = device.BeginCommands();
quantize.Encode(commands, input, 0, 4, output, 0, scale, 0, zero, 0);
device.Submit(commands);
var result = new byte[4];
await output.ReadAsync(result);
Console.WriteLine(string.Join(", ", result));
```

Unzip [the example](https://recursiveai.org/onegpu/cookbook/quantize-tensors-on-gpu/quantize-tensors-on-gpu.zip), then `dotnet run`.

Prints:

```text
0, 85, 170, 255
```

## The full program

The input is `[-1, 0, 1, 2]`. The program reads scale and zero point, reconstructs the floats and checks the error. Invalid counts or buffer ranges throw `GpuKernelShapeException` before recording.

```csharp
using RecursiveAI.OneGpu;
using RecursiveAI.OneGpu.Kernels;

using var device = GpuBackends.CreateDevice(
    new GpuDeviceOptions { Adapter = "discrete" });
using var input = device.CreateBuffer<float>([-1, 0, 1, 2]);
using var output = device.CreateBuffer<byte>(4);
using var scale = device.CreateBuffer<float>(1);
using var zero = device.CreateBuffer<byte>(4);
using var quantize = new DynamicQuantizeKernels(device);
var commands = device.BeginCommands();
quantize.Encode(commands, input, 0, 4, output, 0, scale, 0, zero, 0);
device.Submit(commands);
var result = new byte[4];
await output.ReadAsync(result);
Console.WriteLine(string.Join(", ", result));
var scales = new float[1];
var zeros = new byte[4];
await scale.ReadAsync(scales);
await zero.ReadAsync(zeros);
Console.WriteLine($"Scale: {scales[0]:F9}; zero point: {zeros[0]}");
using var restored = device.CreateBuffer<float>(4);
using var dequantize = new DequantizeKernels(device);
var decode = device.BeginCommands();
dequantize.Encode(decode, output, 0, [4], 0, false,
    scale, 0, zero, 0, restored, 0);
device.Submit(decode);
var reconstructed = new float[4];
await restored.ReadAsync(reconstructed);
float[] original = [-1, 0, 1, 2];
float error = reconstructed.Zip(original, (a, b) => Math.Abs(a - b)).Max();
Console.WriteLine($"Reconstructed: {string.Join(", ", reconstructed)}");
if (error > scales[0] / 2 + 1e-6f) throw new Exception("Error too large");
Console.WriteLine("Half-scale error bound: passed");
Console.WriteLine($"{device.Backend}: {device.Adapter.Name}");
```

Output from an NVIDIA GeForce RTX 4070 using Vulkan:

```text
0, 85, 170, 255
Scale: 0.011764706; zero point: 85
Reconstructed: -1, 0, 1, 2
Half-scale error bound: passed
Vulkan: NVIDIA GeForce RTX 4070
```

## Variations

### An all-zero constant tensor

Append this to the full program:

```csharp
input.Write([0, 0, 0, 0]);
var constant = device.BeginCommands();
quantize.Encode(constant, input, 0, 4, output, 0, scale, 0, zero, 0);
device.Submit(constant);
await output.ReadAsync(result);
await scale.ReadAsync(scales);
await zero.ReadAsync(zeros);
Console.WriteLine(string.Join(", ", result));
Console.WriteLine($"Scale: {scales[0]}; zero point: {zeros[0]}");
```

Additional output:

```text
0, 0, 0, 0
Scale: 1; zero point: 0
```

## Related recipes

- [Compute softmax on the GPU in C#](https://recursiveai.org/onegpu/cookbook/compute-softmax-on-gpu/)
- [Max and average pooling on the GPU in C#](https://recursiveai.org/onegpu/cookbook/pool-tensors-on-gpu/)
- [Gather embedding rows on the GPU in C#](https://recursiveai.org/onegpu/cookbook/gather-embedding-rows-on-gpu/)

## The short answer

These console examples use Vulkan on Linux or Windows with a Vulkan 1.3 driver.

`DynamicQuantizeKernels.Encode` takes float32 values, finds their range on the GPU and writes uint8 values plus a float32 scale and uint8 zero point into device buffers. The full program feeds those buffers to `DequantizeKernels` and checks reconstruction against the original fixture. No ONNX graph or calibration file is needed.

For `[-1, 0, 1, 2]`, the result is `[0, 85, 170, 255]`, with scale approximately `0.011764706` and zero point `85`. The metadata matters: the bytes alone do not specify the original real-valued tensor. Keep scale and zero point alongside the quantized buffer for subsequent operations.

## Can you explain quantization in a simple way?

This operation places float values on a grid of 256 unsigned integer codes. A scale says how far apart adjacent codes are in real units. A zero point says which code represents real zero. Dequantization computes `(code - zeroPoint) * scale` and produces a floating-point approximation to the original value.

OneGpu’s [dynamic quantization implementation](https://github.com/recursive-ai-oss/onegpu-csharp/blob/5d3317caecb82fedd96642e54b0ab33bcb9759c5/src/RecursiveAI.OneGpu.Kernels/DynamicQuantizeKernels.cs) defines the range as `lo = min(0, min x)` and `hi = max(0, max x)`. Scale is `(hi - lo) / 255`. The zero point is the rounded value of `-(lo / scale)`, clamped into `[0, 255]`. Each output rounds `x / scale`, adds the zero point and clamps again.

Rounding is nearest-even, not always away from zero. NaN maps to code zero. The library uses integer arithmetic for correctly rounded float32 division rather than relying on the GPU’s native division. Both kernel modes therefore give the same bits for this operation. It records two passes and keeps a small scratch buffer inside the family; the family is not thread-safe.

## What is the difference between static and dynamic quantization?

Dynamic quantization derives parameters from the current input at runtime. Static quantization uses parameters established beforehand. The [ONNX DynamicQuantizeLinear definition](https://onnx.ai/onnx/operators/onnx__DynamicQuantizeLinear.html) describes the runtime range and output parameters used here. OneGpu’s dynamic operation computes one scale and one zero point for the entire input tensor, not one per channel.

If your application already has scales and zero points, `DequantizeKernels.EncodeQuantize` provides QuantizeLinear instead. That API accepts per-tensor or per-axis parameters and supports uint8 and int8 output. Do not use a newly measured dynamic scale when the next layer expects a fixed scale supplied with trained weights. Matching the next operation’s data type and parameter convention is part of connecting quantized layers correctly.

## Does quantization reduce accuracy?

It can: many float values map to the same code. The four values in this fixture happen to reconstruct exactly on the tested device, but that is not a general promise. The full program checks an absolute reconstruction error bound of half the measured scale plus `1e-6` for float rounding. Treat that as a check on this finite fixture, not a universal bound for arbitrary magnitudes or nonfinite inputs.

An all-zero tensor is a special case: the operation returns scale one, zero point zero and all-zero codes. The variation exercises it explicitly. Constant nonzero input is different because the range still includes real zero; a constant positive tensor does not have a zero-width quantization range.

Byte buffers must end on a whole 32-bit word. This example has four payload bytes and allocates four bytes for the zero-point buffer even though only its first value is used. For other lengths, use the library’s padded-buffer sizing rules and retain the logical element count separately. Readback is only for inspection here; scale, zero point and codes already reside on the GPU for the next kernel.
