Results

Measured rows. The tokenizer speed rows below are final: they were taken in one benchmark-queue group per machine, alone on the machine, with bench/scripts/tokenizer-rows.sh. Speed numbers elsewhere on this page are indicative until they are re-measured alone on the machine.

Tokenizer speed against Microsoft.ML.Tokenizers

Both libraries encode the same text to token IDs, count tokens and decode IDs, one thread. Before a row is timed the harness checks that the IDs from both are identical on that input (and that both decode to the same text); a row whose outputs differ is shown as such and not timed. Our correctness reference is Hugging Face tokenizers and tiktoken (see parity.md); where the comparator differs from them that is information, not a failure of ours.

• Gate: ours at or faster than the managed reference on every timed row. Result: all 47 timed rows on x64 and all 47 on arm64 pass (speedup 1.00x or more). Load time is reported, not gated.
• Speedup is comparator time divided by ours: the median per operation (5 to 25 samples after a half-second warm-up).
• Alloc is bytes allocated per operation on the calling thread. Ours writes into a caller's buffer (EncodeToIds(text, ids)); the comparator returns a list (EncodeToIds(string)), which is the call each library offers for the job. Decode allocates the result string in both.
• Inputs (from the Tatoeba sentence export, fixed by the first run and reused on the second machine): short = one English sentence (59 chars); medium = eight English sentences (513 chars); long = every 600th sentence of the export, 990,287 bytes of UTF-8 across all the languages it holds.
• Operations: encode-to-ids without special tokens; count = token count; decode of the IDs without special tokens.
• Families and files: WordPiece = bert-base-uncased (tokenizer.json; the comparator reads vocab.txt with lowercasing and accent stripping on, as the Hugging Face file has them); Unigram = xlm-roberta-base (tokenizer.json; the comparator reads sentencepiece.bpe.model); SentencePiece BPE with byte fallback = TinyLlama (tokenizer.json); byte-level BPE = gpt2 (tokenizer.json); Tiktoken = cl100k_base and o200k_base (.tiktoken files, comparator TiktokenTokenizer.CreateForModel).
• Peak extra RSS (growth of the resident-set high-water mark during the samples) is in the harness output (rows.md); it follows the garbage collector's timing and is not tabulated here. It is only measured on Linux.

Rows not timed, and why

• WordPiece, medium decode: the comparator's decoded text differs from ours at character 74 (lengths 525 and 519); it does not apply the Hugging Face cleanup rules to spaces before punctuation and contractions. Ours matches Hugging Face (see parity.md).
• WordPiece, long (encode, count, decode): the IDs differ from the first difference at token 159 (ours 1456, the comparator 100, its unknown token): the comparator gives the unknown token for text that the Hugging Face tokenizer, and ours, split into pieces. 357,483 tokens (ours) against 356,166.
• Unigram, long (encode, count, decode): the IDs differ at token 30,076 (ours 49,729, the comparator 1,759); 250,378 tokens against 251,115. On short and medium the IDs are identical once the comparator's numbering is mapped to the Hugging Face file's: the comparator numbers the pieces of sentencepiece.bpe.model, and the Hugging Face file puts four control tokens first, so a piece ID p is token p + 1 (the three control pieces swap places).
• t5-small is not a Unigram row: the comparator throws IndexOutOfRangeException while building its Unigram model from t5-small's spiece.model.

Machines and pins

• x64: AMD Ryzen 9 7950X3D (16 cores, 32 threads), Linux 7.0.0-38-generic, .NET SDK 10.0.403, Release, non-concurrent GC, ICU on. Run on 2026-11-05 at 03:27 UTC. Load average before the run 3.6 (1 min), 15.5 (5 min), 17.4 (15 min), the tail of an earlier job; during the run (10 s samples) it fell from 3.6 to about 1.5; the machine was otherwise idle (the queue checked that the CPU was idle before starting).
• arm64: Apple M2 (8 cores), macOS (Darwin 27.0.0), .NET SDK 10.0.403 (user-local install), Release, non-concurrent GC, ICU on. Run on 2026-11-05 at 04:40 UTC. Load average before the run 2.7 / 3.9 / 12.4; during the run 2.4 to 3.3 (the harness's own thread and the OS).
• Versions: Microsoft.ML.Tokenizers 2.0.1 with Microsoft.ML.Tokenizers.Data.{Gpt2, Cl100kBase, O200kBase} 2.0.1; OneNLP at the commit of this page; information column: tokenizers 0.22.1 and tiktoken 0.14.0 (Python, one thread, x64 only).
• Tokenizer files: the Hugging Face model files named in parity.md; the .tiktoken files from the published encodings.
• Run with bench/scripts/tokenizer-rows.sh MODELS INPUTS OUT; INPUTS is the Tatoeba export or a directory with short.txt, medium.txt and long.txt.

Rows

Times are medians per operation. native is the information column: the Hugging Face tokenizers / tiktoken Python package on the x64 machine, same text.

x64

arm64

Language-ID and recognizer speed

Single thread, one process per system and input set, after a warm-up over 200 inputs; three timed repetitions, and the table shows the median repetition (the first repetition of every system is slower while the JIT tiers up, and the comparators' first recognizer repetition is slower still). Run with bench/scripts/langid-recognizer-rows.sh; each machine ran alone in the benchmark queue. These are publishable rows for the two components.

• Speedup is comparator time divided by ours; 1.00x or more passes. Alloc is bytes allocated on the calling thread. Peak RSS is the process's peak resident size (one process per system, so it includes that system's model load).
• Language ID inputs: every 15th line of the held-out main evaluation set per bucket (tools/bench/make-langid-bench.sh): 4,970 words, 4,872 pairs, 5,000 sentences. Ours: the 75-language model (3.6 MB). Comparator: SearchPioneer.Lingua 2.0.0 built for the same 75 languages with its models preloaded, in high and low accuracy mode. Accuracy rows are in recipes/langid/results/.
• Recognizers inputs: English text of 11,558 lines, 9,979,226 characters, built from the development split only (tools/recognizers-gold/make_bench_text.py). Row 1 compares the comparator's number, ordinal, percentage, currency, dimension, temperature, age, e-mail, URL, IP, phone, hashtag, mention and GUID models with our rules for the same types (the rule files without date/time). Row 2 adds the date/time model on both sides; because the comparator runs it at about 0.1 M chars/s, that row uses the first quarter of the same text (2,889 lines, 2,492,312 characters) so a benchmark group fits its time limit. Both systems see the same lines; the comparator is Microsoft.Recognizers.Text 1.8.13 (en-US). The two systems return different numbers of results (conventions differ); this table compares cost, and the quality rows compare results.

Machines and pins

OneModel is not used by either component. Language-ID weights: langid.bin, sha256 9d06cc77632083274ae5ca5fbb0f3e676abfc63aa6a2d702cbdf9d558cba9d6d. The runs queue behind other work, then hold the machine alone; the load averages above are the 1-minute values read between rows.

Language ID (median repetition)

Our repetitions of 15 to 50 ms vary more than the comparator's (x64 pairs: 8.3, 5.1 and 3.0 µs/text as the JIT tiers up); the median is shown and the speedup is at least 7.6x on every row either way.

Recognizers (median repetition)

Pass/fail: every row is at or faster than its managed comparator, on both machines.

German fast tier

UD German-GSD r2.18 test and GermEval 2014 test against spaCy 3.8.18 trained with its default configuration on the same data; the table, the settings and the weights are in recipes/de/fast/PROVENANCE.md. Raw text: every row ahead of spaCy or within noise (tokens 99.91 vs 98.03, UPOS 94.22 vs 90.13, UFeats 85.32 vs 80.11, LAS 76.75 vs 70.77). Gold tokens: UPOS (94.56 vs 94.59), XPOS (95.35 vs 95.21), lemmas (95.95 vs 95.65), UAS, LAS and entities (66.28 vs 65.77) within noise; UFeats (86.25 vs 84.71) and AllTags (82.78 vs 81.29) ahead. The tagging heads sit on an encoder with one bidirectional LSTM layer after one window layer (recipes/de/fast/PROVENANCE.md). The package is 109 MB; the consumer app runs on JIT and NativeAOT.

French fast tier

UD French-GSD r2.18 test and WikiNER French (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in recipes/fr/PROVENANCE.md. Gold tokens: UPOS (96.77 vs 97.07), UFeats (96.34 vs 96.41), lemmas (97.51 vs 97.49), UAS (90.22 vs 89.10) and LAS (86.69 vs 85.98) within noise. Raw text: every row ahead of spaCy, whose tokenizer keeps du and au as one token where the treebank has two words (tokens 99.88 vs 98.85, words 99.48 vs 94.64, UPOS 96.06 vs 91.80, LAS 85.97 vs 78.10); sentences 94.79 vs 91.82, within noise. Entities, gold tokens: micro F1 82.48 vs 75.50 (+6.97 [6.12, 7.78]) and each of LOC, MISC, ORG, PER ahead. French-GSD has no XPOS. The package is 95 MB.

Portuguese fast tier

UD Portuguese-Bosque r2.18 test and Universal NER Portuguese-Bosque test against spaCy 3.8.18 trained with its default configuration on the same data; the table, the settings and the weights are in recipes/pt/fast/PROVENANCE.md. Bosque has no XPOS, so there is no tagger and no XPOS or AllTags row. Raw text: every row ahead of spaCy except tokens (99.86 vs 99.87, within noise) (words 99.74 vs 88.51, UPOS 95.35 vs 84.15, LAS 82.09 vs 65.85, sentences 92.72 vs 84.48). Gold tokens: UPOS (95.89 vs 95.92), UFeats (94.51 vs 94.11), lemmas (97.30 vs 96.90), UAS (86.93 vs 86.05) and LAS (82.74 vs 81.96) within noise. The tagging heads sit on an encoder with one bidirectional LSTM layer after one window layer (recipes/pt/fast/PROVENANCE.md). Entities 73.13 vs 75.27 (paired difference -2.14 [-4.26, 0.39], within noise).

Fast-tier tagging speed in the other languages (indicative)

The tagging model of each language alone (sentencizer, tagger where present, morphologizer, lemmatizer), raw test text of the language's gating treebank, one thread, one document per call, three runs of four passes; build before the faster kernels and the LSTM-free encoder prediction against the current build, in the same job, x64 worker with load 16 to 25. Predictions of the current build equal the earlier build's on every file (dumps of the tagging, parser and entity files of each language: 0 differing lines, plus the Dutch entity confidence noted under English).

German, Portuguese and Dutch (the recurrent-layer models) gain 15 to 40 percent; the rest of their time is in the three wide heads and the 256-wide encoders, which need the same retraining as English to be faster. Allocation went up for them because the recurrent layers now return one array per call. First-call times of the tagging models: German 1,419 ms, Dutch 1,258 ms, Portuguese 226 ms (load 125 to 171 ms). A French and a Turkish narrow student were trained on web text labelled by the shipped models; their text had not been checked against the evaluation sets, so they are not used.

English fast tier (shipped: self-trained tagging and entity models, earlier parser)

Model english.onenlp in RecursiveAI.OneNlp.Models.English (126.2 MB, sha256 1414ec79...1e48): the tagging model (tagger, morphologizer, lemmatizer; 96-wide encoder with no window layer), the parser model (recipes/en/parser, unchanged) and the entity model (128-wide encoder with no window layer), each with its own encoder, put into one pipeline file by recipes/en/assemble. The tagging and entity models are trained by recipes/en/selftrain: the treebank (or UNER) plus web text labelled by the previous packaged model, after removing every sentence that matches a train, development or test sentence of the evaluation data (recipes/en/selftrain/results/dedup-report.jsonl). Provenance: recipes/en/selftrain/PROVENANCE.md.

UD English-EWT r2.18 test (2,077 sentences, 25,131 tokens) and UNER English-EWT test, each read once for the final configuration; the development sets chose every setting. Paired bootstrap 95% CI over sentences (1,000 resamples, seed 1) from tools/OneNlp.Eval; reference spaCy 3.8.18 trained on the same data with its default configuration, same input mode, with its own interval. "Within noise" means the intervals overlap (docs/measurement.md). The rows are in recipes/en/selftrain/results/.

Quality, gold tokens and sentences

Quality, raw text

Tagging and dependency rows below: ten consecutive sentences per call. The parser file is the one of the earlier model and its heads and relations are identical to that model's, so its whole-document rows are unchanged: Sentences 76.15 [74.25, 78.08] against spaCy 72.66, UAS 81.80, LAS 78.13 [77.18, 79.06] against 78.53, CLAS 73.04 against 74.34 (all within noise or ahead).

MLAS, information: 65.61 [64.46, 66.72] over ten-sentence calls, against spaCy's 69.05 [67.81, 70.21]; it is outside the interval, as for every earlier English model, and is not one of the gated rows.

Entities (UNER English-EWT test, gold tokens)

Every row is within noise.

Speed (indicative, not a gate row)

Raw EWT test text (25,131 tokens, repeated four times per pass, 100,524 tokens), one document per call, twelve passes after the process starts, medians of the last ten; Catalyst 26.10.6514 with its English models on the same documents, run back to back in the same job. Machine: x64 worker (Ryzen 9 7950X3D, 32 threads) with other jobs running (load average 22 to 30 before and after, so the numbers are noisy), .NET SDK 10.0.403, OneModel 1.0.0. The gate run is made once, on a quiet machine, at the 1.0 gate.

The ratio on the tagging path is 0.67 in this job (the same model against Catalyst in an earlier job with the same method: 0.94 for a 96-wide variant of the same recipe before the final retraining, 0.06 for the previous packaged model), which meets the 0.5 bar. Peak working set 1,241 MB (1 thread) and 1,360 MB (8 threads) for the full pipeline; model file 126.2 MB; load 416 ms and the first call 365 ms (previous model: 328 ms and 434 ms). Full pipeline in the single-file mode used by Nlp.Analyze (one document per call): 5,100 to 5,500 tokens/s against 3,000 to 3,500 for the previous model. The parser is now the limit of the full pipeline (about 190 us per token, an LSTM over a 192-wide state), and its prediction is serialized per head, so eight threads do not help it; the tagging and entity paths scale (4.0 and 1.6 times at eight threads).

How the speed was reached without changing a row: the encoders are narrower and trained on more text (above); GELU, the dense layers and the parser's LSTM and label scores run on dedicated row kernels (RowKernels, LstmInference) with bit-identical or last-digit-identical results (prediction dumps of every shipped file before and after: 0 differing lines on English, German, Portuguese, Dutch, Turkish and French except one Dutch entity confidence that rounds 0.901 to 0.902); the encoder runs outside the head lock.

English fast tier, option: one shared encoder (not shipped)

A single encoder shared by every head, trained by recipes/en/joint (23.7 MB, sha256 6bb03663...50b9): four times smaller and more than ten times faster on the tagging path, with the rows below. It is not the packaged model because its UFeats, AllTags, entity and raw CLAS and MLAS rows are weaker. Rows measured on the model file english.onenlp of that recipe. UD English-EWT r2.18 test, 2,077 sentences, 25,131 tokens; UNER English-EWT test. Paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); reference spaCy 3.8.18 trained on the same data with its default configuration, same input mode. "Within noise" means the intervals overlap. The development sets chose the configuration; the test files were read once.

Quality, gold tokens and sentences

Quality, raw text

Entities (UNER English-EWT test, gold tokens)

The intervals overlap on every entity row; the paired differences for micro F1 and LOC exclude zero.

Speed (indicative, not a gate row)

Raw EWT test text (25,131 tokens, repeated four times per pass), one thread, sentences + tokens + UPOS/UFeats + lemmas; Catalyst 26.10.6514 with its English models on the same documents, run back to back in the same job. Machine: x64 worker (Ryzen 9 7950X3D) while other jobs were running (load 9 to 15), .NET SDK 10.0.403. The gate run is made once, on a quiet machine, at the 1.0 gate.

The ratio on the tagging path is about 0.4 (misses). Peak working set 640 MB for the full pipeline; model file 23.7 MB; first Nlp.Analyze (loads the weights) 325 ms on x64. Where the time goes (microseconds per token, one document per call): tokenizer 0.2, sentencizer 0.05, encoder about 2 (lookups 0.6, window-layer product 1.2), lemmatizer 0.9, morphologizer 1.0. A smaller configuration of the same recipe (48 wide, no window layer) ran the same path at 500,000 to 690,000 tokens/s (about 1.6 to 2.3 times Catalyst in the same conditions) with a development tagging mean 0.6 points lower; it is not the shipped model.

Dutch fast tier

UD Dutch-Alpino r2.18 test and WikiNER Dutch (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in recipes/nl/fast/PROVENANCE.md. Every row is within noise of spaCy in both modes. Gold tokens (the treebank's own words and sentences): UPOS 93.83 vs 94.42, XPOS 91.52 vs 91.73, UFeats 92.94 vs 93.13, AllTags 90.30 vs 90.51, lemmas 91.61 vs 91.83, UAS 85.18 vs 84.09, LAS 80.28 vs 79.85. Raw text: tokens 99.66 vs 99.82, sentences 91.31 vs 89.60, UPOS 93.09 vs 94.36, XPOS 90.94 vs 91.63, UFeats 92.47 vs 92.97, AllTags 89.52 vs 90.39, lemmas 91.19 vs 91.65, UAS and LAS (79.48 vs 79.57). The tagging heads sit on an encoder with one bidirectional LSTM layer after one window layer (recipes/nl/fast/PROVENANCE.md). Entities (gold tokens): micro F1 87.47 vs 86.21 (difference +1.26 [0.76, 1.72], ahead), every type within noise or ahead.

Turkish fast tier

UD Turkish-BOUN r2.18 test and Turkish-WikiNER test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in recipes/tr/fast/PROVENANCE.md. The tokenizer is a trained multiword-token splitter (NeuralMultiwordSplitter, docs/segmentation.md). Gold words: UPOS (88.97 vs 88.04), UFeats (78.15 vs 78.30) and MLAS (40.47 vs 38.85) within noise; XPOS, AllTags and lemmas (85.11 vs 80.24) ahead. Raw text: tokens (99.56 vs 99.42), words (96.55 vs 97.06), UPOS (86.02 vs 86.82), UFeats, UAS (65.73 vs 63.96), LAS (57.53 vs 55.86), CLAS (54.83 vs 53.43), MLAS (38.10 vs 38.02) and BLEX within noise, sentences, XPOS, AllTags and lemmas ahead. The parser was retrained on the tokenizer's words and on 60,239 sentences of web text parsed by earlier parsers where two of them agree (it was CLAS 49.84 and MLAS 34.87 before); gold words: UAS, LAS, CLAS (59.08 vs 55.03) and BLEX ahead. Entities (gold tokens): 67.40 vs 65.82, within noise. The package is about 148 MB; the consumer app runs on JIT and NativeAOT.

Spanish fast tier

UD Spanish-AnCora r2.18 test and WikiNER Spanish (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in recipes/es/fast/PROVENANCE.md. Gold words and sentences: UPOS (98.11 vs 98.18), UFeats (97.71 vs 97.74) and lemmas (98.84 vs 98.72) within noise; XPOS (86.73 vs 84.97), AllTags (86.09 vs 84.47), UAS (90.05 vs 89.04) and LAS (87.00 vs 85.95) ahead. Raw text: every row within noise or ahead (spaCy's tokenizer keeps del and verlo as one token where the treebank has several words; words 99.83 vs 96.60, UPOS 97.86 vs 94.81, LAS 86.70 vs 81.82); tokens 99.84 vs 99.90 and sentences 98.14 vs 97.70 within noise. Entities, gold tokens: micro F1 82.88 vs 83.35 (difference -0.46 [-1.14, 0.20], within noise); ORG ahead (+4.96), PER behind (-1.96 [-2.83, -1.13]). The entity model reads the tokenizer's pieces of del and al: trained on whole tokens it scored 81.71 (-1.64, outside the interval). Indicative cost on one thread of the shared worker: tagging heads 44 us/token, whole pipeline about 190 us/token. The package is 109 MB; the consumer app runs on JIT and NativeAOT.

Italian fast tier

UD Italian-ParlaMint r2.18 test (gating) and WikiNER Italian (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the tables, the settings and the weights are in recipes/it/fast/PROVENANCE.md. The training treebank is small: 897 sentences (ParlaMint and MarkIT; the larger Italian treebanks are non-commercial), so parser rows are low in absolute terms (LAS 72.6) for both systems. Gold words and sentences: UPOS (94.44 vs 94.10), XPOS (93.74 vs 93.59), UAS (78.52 vs 77.08) and LAS (72.57 vs 71.39) within noise; UFeats (89.52 vs 87.42), AllTags (86.31 vs 84.08) and lemmas (95.18 vs 93.13) ahead. Raw text: tokens 99.86 vs 99.75 within noise; every other row ahead (spaCy's tokenizer keeps del and farlo as one token where the treebank has several words; words 99.67 vs 89.73, UPOS 94.01 vs 83.53, LAS 72.77 vs 58.38, sentences 96.00 vs 87.55). Entities, gold tokens: micro F1 86.35 vs 83.54 (+2.80 [2.18, 3.42]); PER, ORG and MISC ahead, LOC within noise. No row misses. Indicative cost on one thread of the shared worker: the whole pipeline 7.2k tokens/s. The package is 103 MB; the consumer app runs on JIT and NativeAOT.

Chinese fast tier

UD Chinese-GSDSimp r2.18 test and UNER Chinese-GSDSIMP test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the settings and weights are in recipes/zh/fast/PROVENANCE.md. spaCy's default Chinese tokenizer cuts single characters, so its raw-text pipeline cannot reach word-level scores (Tokens 39.43); the gold-token reference is the gold_preproc training run described in tools/baselines/spacy/lang/PROVENANCE.md, which trains and scores on gold words. The pipeline cuts words with a trained character tagger (docs/segmentation.md), tags, lemmatizes, parses and finds entities.

Gold words and sentences (spaCy gold_preproc):

Raw text (spaCy's default pipeline, character tokens):

Entities, UNER Chinese-GSDSIMP test, gold tokens (paired with spaCy gold_preproc): micro F1 62.80 [58.46, 67.23] vs 61.86 [57.66, 66.12] (difference 0.94 [-2.18, 4.40], within noise); precision 62.44 vs 67.08 (-4.64 [-8.52, -0.50], inside the noise rule); recall 63.17 vs 57.40. Per type: LOC 69.86 vs 68.15, ORG 47.91 vs 45.00, PER 57.69 vs 59.08, all within noise. With the shipped entity confidence threshold of 0.6 (recipes/zh/fast/PROVENANCE.md, "Entity confidence threshold"): micro F1 63.79 [59.39, 68.27] (difference +1.93 [-1.48, 5.33]); precision 69.28 vs 67.08 (+2.20 [-2.02, 6.59]); recall 59.11 vs 57.40 (+1.70 [-1.61, 5.12]); LOC 70.38, ORG 51.28, PER 57.95, all within noise. On raw text (our segmenter's words, character-span F1) the micro F1 is 52.88 [48.60, 57.61]; spaCy's default model reaches 13.30 on gold tokens only, and no raw-text spaCy row exists.

For information (docs/references.md): Stanza 1.5.1 raw text UPOS 88.82, LAS 70.65; UDPipe 2 gold tokens UPOS 95.73, LAS 83.60; Trankit base raw UPOS 94.17, LAS 81.96 (an older annotation). Our fast tier is below them, as expected of a fast tier with no pretrained embeddings; the gate is spaCy's.

Misses: none. Everything is within noise or ahead; MLAS (-1.93) is inside the noise rule.

Indicative speed (1 thread, load average 8-13 on the shared worker, raw development text; Catalyst has no Chinese model, so there is no managed row): segmenter alone 22-23k words/s (3.7 kB allocated per word); segmenter and tagging 16-17k words/s; the whole pipeline (segmenter, tagging, parser, entities) 4.5-4.6k words/s.

Parser speed (one LSTM layer, product label features)

Every fast-tier parser was retrained with the setting in docs/parser.md ("Speed configuration"): English english.onenlp is now 117.5 MB (sha256 ac0080ad...ed9b; the tagging and entity models are unchanged, only the parser part differs). Training data, seeds and licences are those of the earlier parsers (UD treebanks, CC BY-SA 4.0 / CC BY 4.0; the models keep share-alike lineage); selection used each language's development file with the earlier early-stopping rule (patience 6), and each test file was read for the final models (see the note below for the two languages where a smaller setting was scored on test before the larger one was chosen on dev).

Speed (indicative: 1 thread, batches of 32 documents, raw test text, x64 worker, load average 14 to 23)

The English parser alone: 137 us/token by component before (LSTM 81, label scorer 52, projections 9, arcs and tree 8) and 20 us/token after in the per-component profile. Peak working set of the English full pipeline 939 MB (was 1,083 MB). The other languages' rows are the parser file alone (its own encoder 10 to 13 us/token is included in the full-pipeline figure, not in the head column).

English, UD English-EWT r2.18 test (hash fa024f43...c494a), paired bootstrap 95% CI, 1,000 resamples, seed 1, reference spaCy 3.8.18 default configuration on the same data

Tagging rows are those of the unchanged tagging model (gold UPOS 94.16, UFeats 95.15, lemmas 96.11; raw UPOS 92.90).

Other languages, gating treebank test, LAS (UAS, CLAS, MLAS in the committed result files), paired CI against spaCy 3.8.18

Every UAS, LAS and CLAS row is within noise or ahead in both modes. MLAS, which also depends on the tagging model, is within noise or ahead in every row of these languages (Chinese gold words: 55.78 [53.92, 57.68] against 58.87). Raw-text rows compare our tokenizer against spaCy's, which is the reason for the large margins.

Note on selection: for Italian and Chinese the 128-unit setting was scored on test before the 192-unit setting was chosen on dev (dev LAS 72.43 and 64.82 against 73.97 and 66.81, and 74.81 and 68.22 for the earlier parsers); the larger setting was chosen because the smaller lost more than 1 point of dev LAS, the rule used for every language.