The library computes in binary32. A model stored in binary16 or bfloat16 runs with half-precision storage between operators and binary32 arithmetic inside them. The conversion rules are in tensors.md section 10; this note says how they are used.
Models stored in half precision
Applies to a model whose weights (initializers), constants or intermediate values are float16 or bfloat16, or that casts to one of
them. Inputs and outputs of the graph keep the types the file declares; a float32 model with a Cast to float16 at each end works.
At session creation the graph is scanned; if any such type occurs, every node whose kernel can be wrapped gets an adapter. For a node with at least one half-precision input:
- each half-precision input is widened to binary32 (exact);
- the kernel runs on binary32, in the same fast or exact flavour as always;
- each output the kernel produces as binary32 is rounded back to the storage type, to nearest even.
The storage type of the outputs is the type of the node's first half-precision input that is not a compact weight (below); a node with
only compact weights and binary32 activations stays in binary32. Only element-wise Add, Sub, Mul and Div with all inputs
binary16 skip the adapter (they compute in binary16 themselves, a correctly rounded operation either way). Cast, CastLike,
Constant, ConstantOfShape, Shape, Size, views (Reshape, Squeeze, ...) and nodes whose output shape depends on values are
never wrapped; they see the real element types.
Constant half-precision inputs (weights) are widened once when a plan is made and kept as binary32 in the plan, so a model stored in half precision uses its binary32 size in memory while it runs; the file is half the size. Constant folding applies as for binary32.
Because every conversion is exact (widening) or the specified round-to-nearest-even, exact mode stays bit-identical across machines and
thread counts: a half-precision model is an exact-mode model whenever its operators have exact kernels. The vector conversions
(Kernels/HalfConvert.cs) are the same integer arithmetic as the scalar ones, eight values at a time (a test compares them with the
scalar code over every binary16 and bfloat16 pattern and several million float32 patterns, including every rounding boundary).
Accuracy against ONNX Runtime
ONNX Runtime runs such files on its CPU provider too. The files in tools/models.json
(fp16 entries, made by tools/export_models.py --fp16 from the float32 files: weights and activations float16, float32 graph inputs
and outputs, finite values beyond the float16 range clamped to ±65504) are compared by scaled error (max |ours − reference| / max
|reference| per output):
- Gate for float16 files: scaled error against ONNX Runtime's output on the same file ≤ 4e-3, and ours against the float32 model's output no worse than 1.5 times ONNX Runtime's own float16-versus-float32 error. Half precision has an 11-bit significand (2^-11 = 4.9e-4 per rounding), and the errors of the models are a few roundings; two correct engines that round at slightly different places differ by that much, and the gate allows about twice what was measured.
- Measured (x64, batch 1 and the model's batch): see the table in oracles.md.
Keeping binary32 weights compact (SessionOptions.WeightStorage)
WeightStorage.Binary16 or BFloat16 rounds the constant weights of MatMul, Gemm and Conv (input 1, at least 1024 values, read
by nothing else) when the session is created, frees the binary32 copy, and widens the weights into run scratch for each use. The weight
memory halves; the cost is one widening pass per weight per run, and the results change by the rounding of the weights only: activations
stay binary32, so the result equals a binary32 model whose weights were rounded to the same format beforehand, bit for bit (a test
checks this at 1 and 4 threads). Embedding tables (Gather), biases and norm parameters are not compacted.
bfloat16 keeps the binary32 range with 8 bits of significand (2^-8 = 3.9e-3 per rounding); binary16 has more precision but ends at 65504, so a weight tensor with a value beyond that (or NaN or infinity) is left as it is.
Rules
- Round to nearest, ties to even, on every narrowing; NaN stays NaN and is made quiet; overflow becomes infinity.
- Widening is exact.
- The input and output types of a graph are not changed by the options;
Session.Runtakes and returns what the file declares.