x64 release speed rows

Measured 2026-11-08 on an AMD Ryzen 9 7950X3D (16 physical cores, 32 logical), x64 Linux 7.0.0-38-generic, .NET SDK 10.0.403, runtime 10.0.14. Available ISA: SSE4.2, AVX2, AVX-512F/BW/VL, FMA and BMI2. CPU inference only; no GPU used. OneNLP 0.4.0-preview.1 source build, OneFont 1.4.0, OneModel and OneModel.Training 1.0.0. Comparators: Microsoft.ML.Tokenizers 2.0.1, SearchPioneer.Lingua 2.0.0, Microsoft.Recognizers.Text 1.8.13, Catalyst 26.10.6514 with Catalyst.Models.English 26.9.5316.

Three exclusive benchmark-queue jobs, each bounded to 20 minutes, ran once after prebuilding the tools. Actual timed-group wall times: tokenizers 2m23s, language ID/recognizers 4m40s, Catalyst 1m16s (about 8m19s total). No training, builds or other queued heavy work overlapped these jobs. The methods are in measurement.md and the repository's bench scripts. Values below are medians, not best-case samples.

Tokenizers

Every cell includes time, managed allocation per operation and extra resident high-water growth during the samples. All 47 matched encode/count/decode rows pass (1.13–48.55× the managed comparator). Seven operation rows are not timed because the comparator produces different IDs or decoded text: WordPiece medium decode and long encode/count/decode; Unigram long encode/count/decode. They are not passes. Six load rows are information only; WordPiece, TinyLlama, gpt2 and cl100k_base load more slowly. Total process peak RSS was 400 MiB. Native HF timing was not requested here; exactness against HF is reported separately in parity.md.

Each operation warms up for about 500 ms, then measures 5–25 batches of about 40 ms (1,000 ms sampling budget); the reported time is the median batch's per-operation time. Allocation uses the thread counter. The six model families use the pinned files in tools/parity/families.json and Families.All in the benchmark. Short (59 characters), medium (513), and long (990,287 UTF-8 bytes) inputs are byte-identical to the arm64 run:

Language ID

Single-thread scoring, every 15th held-out main-set line per bucket. Warm-up 200 inputs, then three repetitions. Median microseconds and bytes per text:

Load: ours 12.6 ms, high 3,670.6 ms, low 282.0 ms. Process peak RSS: 95, 5,042, and 360 MiB respectively. Our model is 3,588,959 bytes, SHA-256 9d06cc77632083274ae5ca5fbb0f3e676abfc63aa6a2d702cbdf9d558cba9d6d. Lingua's model size was not separately measured (it is delivered by its package). Input SHA-256:

• words: c878bfb1403c9f70b4616771397674fe46d1dbbc6008b4cfb24a7e9e61d7a421
• pairs: f5123bbf28cf9423d3f179c208f3a427c8902fbb4150a29b93fbdaf1b6817473
• sentences: 956a081df9d16a2d1b825b782857dd13e3bb0ac680250ef7deedee8c6aecd127

English recognizers

Same generated input lines and reference date for both systems (2026-03-18 10:30 UTC). Three repetitions; managed allocations below are the median total per repetition, not bytes per expression. UTF-16 character throughput excludes line terminators. Model size: data rules, no trained weights; rule byte size not separately measured. Different result counts reflect the quality differences reported in results-table.md, not different input work.

Date/time uses the first quarter of the full text to keep the group bounded; both systems receive those same lines. Turkish and Hindi timing is not a gate.

English fast tier against Catalyst

One thread, batch 1; UD English-EWT r2.18 test raw documents, repeated four times as one input workload. Each tool makes 12 passes; discard the first two and take the median of the remaining ten (internal samples, not repeated benchmark jobs). Ours emits 100,524 tokens per pass, Catalyst 101,724, because tokenizers differ.

The tagging row counts under matched or better measured accuracy on raw EWT (test F1/accuracy): Tokens 99.00 vs 94.47, Sentences 71.08 vs 64.57, UPOS 92.90 vs 80.70, Lemmas 95.03 vs 83.84. UNER character-span entity F1 is 63.09 vs 57.77. The NER speed row is not a gate: Catalyst's NER needs its tagging pipeline, whereas ours runs just the sentencizer and entity component.

Process peak RSS: ours 1,240 MiB (whole multi-path job); Catalyst 383 MiB (316 MiB at the end of tagging alone). English model file: 117,451,750 bytes, SHA-256 ac0080adcb1a9832fe4c5c7b80e22a54da51bfcdac88e0d53986c80e6ce7ed9b. Cold English model load was not measured in this run. The quality rows are unchanged; no model was retrained or selected using these timings.

Accurate tiers: indicative, not a speed gate

A separate heavy, four-core-reservation job (007347), not an exclusive benchmark, measured one English pipeline from each package. TensorCompute.Threads is 1 and DOTNET_PROCESSOR_COUNT is 4. Other heavy jobs were running; load was 13.39 / 13.20 / 9.11 before, 12.62–13.77 (1-min) during, and 13.77 / 13.25 / 9.44 after. These figures must not be compared as publishable speed gates or as a no-load regression measurement.

Input: the first 100 # text sentences of UD English-EWT r2.18 test, one per line, 2,206 emitted word tokens including punctuation; SHA-256 1d719e08e95c2d852234bcb59f284b0306a60a6358ac932b7adbf337e30fbaa6. Each throughput process warms up on 20 documents and takes the median of three passes. Cold load uses a separate fresh process and the first sentence, with an explicit English language (no language-detection work). Models are the exact manifest-verified package payloads; no quantization or tensors were changed.

Only English heads load. Allocation per token was not instrumented for this information run. The quality tables and known published-reference gaps are unchanged; these timings do not select a model or relax any quality bar.