Half-precision weights · onemodel cookbook

Keep the large weights of a float32 ONNX model as binary16 or bfloat16 with SessionOptions.WeightStorage, which halves their memory, and measure what it costs in accuracy.

The model is wide.onnx from the sample models.

using RecursiveAI.OneModel;
using RecursiveAI.OneModel.Graph;
 
float[] input = Enumerable.Range(0, 256).Select(i => MathF.Cos(i * 0.1f)).ToArray();
 
using var full = Model.Load("wide.onnx");
using var half = Model.Load("wide.onnx", new SessionOptions { WeightStorage = WeightStorage.Binary16 });
using var bf16 = Model.Load("wide.onnx", new SessionOptions { WeightStorage = WeightStorage.BFloat16 });
 
float[] a = full.Run(input), b = half.Run(input), c = bf16.Run(input);
Console.WriteLine($"float32  y[0..3] = {Show(a)}");
Console.WriteLine($"Binary16 y[0..3] = {Show(b)}  max |diff| = {MaxDiff(a, b):E2}");
Console.WriteLine($"BFloat16 y[0..3] = {Show(c)}  max |diff| = {MaxDiff(a, c):E2}");
 
static string Show(float[] v) => string.Join(", ", v[..3].Select(x => x.ToString("F5")));
static float MaxDiff(float[] x, float[] y) => x.Zip(y).Max(p => MathF.Abs(p.First - p.Second));

Text

float32  y[0..3] = -0.52048, -0.56874, 0.46823
Binary16 y[0..3] = -0.52028, -0.56858, 0.46820  max |diff| = 2.71E-004
BFloat16 y[0..3] = -0.52041, -0.57004, 0.46990  max |diff| = 2.35E-003

What changes

WeightStorage.Binary16 and WeightStorage.BFloat16 round the constant weights of MatMul, Gemm and Conv (1024 values or more) to 16 bits when the session is created and free the float32 copy. Each run widens them into scratch memory. Activations stay float32, so the only change is the rounding of the weights.
Embedding tables, biases and normalization parameters stay as they are.
binary16 keeps more significand bits, so it is closer to float32 here. bfloat16 keeps float32’s range; binary16 ends at 65504, and a weight tensor with a larger value is left in float32.
Models whose weights are already float16 or bfloat16 run as they are, with no option needed.

Conversions and the widening rule are in half-precision storage. For int8 models (QDQ, MatMulInteger, QLinearConv and the rest), see int8 inference.