Speed at 1.0

Note: in the README’s words, these figures were “Measured through onemodel’s RecursiveAI.OneModel.Gpu”. That package isn’t released yet; OneModel’s GPU support is coming in a later OneModel release, so the figures can’t be reproduced with the released packages.

The harness and the full method are in onemodel-csharp, tools/bar.sh and docs/bench.md. The models are onemodel's reference set: MiniLM-L6, BERT-base, ResNet-50, MobileNetV3, DBNet, CRNN and Whisper-small (encoder and decoder), in float32, float16 and dynamically quantized 8-bit forms.

Speed is judged against the reference provider on the same machine and the same model file; ratio is ours divided by the reference, so lower is better. The bar is 1.5 on a desktop (target 1.15) and 2.5 in a browser (target 1.5). met is a ratio above the target and within the bar, target is a ratio within the target, missed is above the bar. Only gate rows are judged (the reference ran every compute node on its GPU, read from its own placement log and, for Core ML, its compute plan); information rows are listed with their ratio and never judged. Every row runs a model through onemodel's RecursiveAI.OneModel.Gpu on these packages at their 1.0.0 code, fast mode, validation layers off, host arrays in and out, the median of warm runs (Linux 20, Apple M2 10, Chrome 10 per page) in one benchmark-queue job per group of rows, each group alone on its machine. Load average sampled every 5 s before, during and after is in every row of the jsonl; the one-minute load average at the start of a group still carries the harness's own CPU reference runs of the group before it. Reference versions: ONNX Runtime 1.31.1 with the onnxruntime-ep-webgpu 0.4.0 plugin (Dawn over Vulkan on Linux, over Metal on macOS), Core ML through the same wheel on GPU compute units, onnxruntime-web 1.31.1.

Linux, RTX 4070 (Vulkan, driver 595.99.02), Ryzen 9 7950X3D, 32 threads; reference: WebGPU provider

Three groups of rows, each alone on the machine. Gate rows 14 of 20: all met, worst ResNet-50 float32 at batch 8 (1.40) and BERT float32 at batch 8 (1.35); the other twelve are at or below the 1.15 target. The int8 rows are information rows: the reference leaves MatMulInteger, DynamicQuantizeLinear, QuantizeLinear or ConvInteger to its CPU provider, so those rows are not a like-for-like comparison. CUDA and onemodel CPU columns were not run for the gate.

Apple M2 (Metal, macOS 27.0.1); reference: whichever gate configuration of Core ML (GPU compute units) and the WebGPU provider on Metal took less time

Three groups in the benchmark queue, 10 runs per row. Gate rows 14 of 20: twelve are met, two are misses for 1.0: ResNet-50 float32 at batch 1 (2.16) and batch 8 (2.21) against Core ML on the GPU (6.07 ms and 35.27 ms; Core ML MLProgram on the GPU computes in float32 to 1e-6, so it is a fair reference). Against the WebGPU provider on Metal the same rows are 0.82 (13.09 against 15.94 ms) and 0.81 (77.94 against 95.71 ms). The miss is documented and ships: the levers left are per-model weight preparation for contiguous 3x3 reads and a Winograd F(2x2,3x3) kernel, estimated at about 1.8 times Core ML at batch 8, so not enough to close it. MobileNetV3 float32 at batch 8 (1.16) is just above the target. The int8 rows are information rows (MatMulInteger, DynamicQuantizeLinear and ConvInteger on the reference's CPU provider).

Chrome 155 on the RTX 4070, WebGPU (ahead-of-time compiled page); reference: onnxruntime-web WebGPU provider

Two groups, each alone on the machine, ten warm runs per page, same adapter. The twelve float32 rows; every one is within the 2.5 bar and eleven within the 1.5 target (ResNet-50 batch 8, 1.54, is the only one above). The float16 rows decline in this browser (no shader-f16 on this Chrome) and the int8 rows are information, so none is a bar row here; they run (fast and exact, two-run output identical) in the device matrix below.

Windows, Intel Iris Xe (Windows 11, driver 101.7085), indicative only

Not publishable: one run on a shared laptop, ONNX Runtime 1.24.4 DirectML wheel, tools/bar.ps1. The float32 transformer rows (BERT, MiniLM), ResNet-50 (float32 and float16), MobileNetV3 at batch 8 and Whisper miss the 1.5 bar against DirectML (1.53 to 2.89): that GPU has no cooperative matrix, so the portable tiled kernels run. They ship as documented misses. NativeAOT on Windows is untested.

Correctness of the same runs

OneModel.Gpu.Tests (383 tests, 0 failed, 0 skipped): RTX 4070 and llvmpipe and RADV (Mesa 25.2.8) with the Khronos validation layer on, 0 validation messages; Apple M2 with Metal API and shader validation fatal. All 21 rows (20 above plus the greedy decode) with RequireGpu, fast and exact: 100 % of nodes in one region on every device, outputs identical between two runs in each mode (40 of 40 model-mode pairs per device), 0 validation messages. float32 rows are within a scaled error of 1.03e-5 of the CPU's exact mode on every device (Whisper-small encoder 8.0e-5), float16 within 1.2e-3, the 8-bit rows within the int8 policy (model-level error against the CPU exact mode 3.9e-4 to 8.8e-1, reported next to the CPU's own fast-versus-exact drift). CRNN greedy ids and the Whisper decoder's greedy tokens (8 of 8) equal the CPU's on every device. The Whisper-small encoder at batch 4 against a float64 evaluation, 16 inputs: mean error 1.14 (fast) and 1.21 (exact) times the CPU exact mode's on the RTX 4070 (max 0.90 and 0.97 times), 1.11 and 1.22 on the M2 (max 1.17 and 1.00), within 1.5 times the CPU exact mode's error. Chrome (RTX 4070): all rows fast and exact with RequireGpu, one region, identical runs, no page errors; SwiftShader subset (MiniLM, DBNet, CRNN, ResNet-50, Whisper decoder) likewise. OneModel.Tests 999 passed, 4 skipped, golden table equal to the base, NativeAOT smoke publish without trim or AOT warnings, run on RTX 4070 and llvmpipe with validation (all ConvInteger and model checks ok).