Results table

One table of what is measured, what it is measured against, and whether it holds. Every number is copied from a file in this repository, named in the "Source" column or the section heading; nothing here was measured for this page. Rows that lose, and gaps, are listed in their own sections below, not left out.

How to read it.

• Fast-tier and text-classification rows are scored on official test splits, read once for the final configuration (the development splits chose every setting). The reference is spaCy 3.8.18, trained as a black box with its default configuration on the same data, in the same input mode (tools/baselines/spacy/). Intervals are paired bootstrap 95% over sentences or documents (1,000 resamples; seeds are in each source file). Within noise means the two intervals overlap (measurement.md); ahead means they do not and ours is higher; behind means they do not and ours is lower.
• Gold rows give the treebank's own words and sentences as input; raw rows give only the text, so tokenization and sentence splitting are scored too. spaCy's tokenizers keep contractions such as del and zum whole, which is why several raw rows are far ahead.
• Within noise is not a claim of equal. With a few hundred sentences an overlap is a weak statement. Where the paired difference excludes zero while the intervals overlap, the source file says so, and the row is also named under "Misses and caveats".
• Speed rows are in results.md. The tokenizer, language-ID and recognizer rows there were taken alone on the machine and are repeated at the release gate; the fast-tier speed rows are indicative, and the publishable run against Catalyst is made at the release gate.

Tokenizers and segmentation

Tokenizer speed against Microsoft.ML.Tokenizers 2.0.1, release gate, arm64 (Apple M2, 8 cores, .NET 10.0.403, load 2.0 to 3.3 before and during, idle after; run alone on the machine): 47 timed rows, none below 1.00x. Seven operation rows are not timed because the comparator's output differs from ours (WordPiece medium decode and long x3, Unigram long x3; six load rows are reported separately); they are not counted as passes. x64 release run: 47/47 matched operation rows pass, 1.13–48.55x; seven differing-output rows are not timed. Four x64 load rows are slower (WordPiece, TinyLlama, gpt2, cl100k_base; information only). Every x64 cell and machine record. On arm64, load time is reported, not gated, and is slower than the comparator on three rows: WordPiece from tokenizer.json 28.16 ms vs 19.17 ms (0.68x), SentencePiece-BPE TinyLlama 49.60 ms vs 11.66 ms (0.24x), cl100k_base 19.93 ms vs 19.00 ms (0.95x); ours is faster on Unigram (3.47x), gpt2 (1.02x) and o200k_base (1.31x).

Language identification

Held-out Leipzig texts ("main") and a Tatoeba slice, macro accuracy over 75 languages (every language weighs the same), against lingua-language-detector 2.2.0 and SearchPioneer.Lingua 2.0.0 in high-accuracy mode, built for the same 75 languages. Source: language-id.md, recipes/langid/results/.

The macro averages hide languages where we are below Lingua: on the main set 14 languages on words (ar da ga hr ko lg mk nl ru so ur xh yo zh), 8 on pairs (hr ja lg ms ru xh yo zh) and 2 on sentences (en hr). The closely related Latin-script pairs are the hard rows. The Leipzig texts come from the collection Lingua says its models were built from (this may favour Lingua), and our training includes other Tatoeba sentences (this may favour us); both are reported. Per-language rows: recipes/langid/results/per-language.*.csv.

Language-ID speed (alone on the machine, x64 and arm64): at least 6.9x Lingua on every row (the minimum is arm64 word pairs against low accuracy; on x64 at least 7.8x); 10.7 ms load against 326 ms (low accuracy) and 3,850 ms (high accuracy) on x64 (results.md).

Language-ID speed, release gate, arm64 (Apple M2, 8 cores; median microseconds per text, ours / SearchPioneer.Lingua 2.0.0 high accuracy / low accuracy): words 7.74 / 76.42 / 63.54 (9.9x and 8.2x faster), pairs 8.09 / 88.65 / 55.80 (11.0x and 6.9x), sentences 10.77 / 300.58 / 216.10 (27.9x and 20.1x). Allocation 1.4 KB per text against 75 to 126 KB; RSS 66 MB against 2,575 and 480 MB; load 9.3 ms against 4,329 and 328 ms. No row is slower. x64 release run: words 8.83 / 81.62 / 69.05 µs (9.24x / 7.82x), pairs 3.58 / 96.29 / 65.58 (26.90x / 18.32x), sentences 9.06 / 392.54 / 246.83 (43.33x / 27.24x). All rows pass. Ours allocates 1.4 KB/text vs 75–125 KB; RSS 95 vs 5,042 / 360 MiB; load 12.6 vs 3,670.6 / 282.0 ms. Inputs and machine record.

Recognizers

Our hand-labelled gold sets, one per language, written from the type definitions and real text (recognizer-gold.md), test split. Reference: Microsoft.Recognizers.Text 1.8.13 in the same culture, run as a black box. Exact-span F1, all types pooled, with the paired bootstrap 95% interval (1,000 resamples over sentences, seed 20260101). Source: testdata/recognizers/<lang>/comparators/ours-vs-recognizers.test.md.

No type in any language is behind Recognizers.Text (the behind verdict appears in none of the ten comparison files). "All types" pools types with ten to thirty items each, so per-type intervals are wide (the per-type rows are in the source files, and Duckling's, which are information, beside them). The comparator uses its own conventions for range ends and drops stated time zones; the gold uses ours, and the differences are listed in recognizer-gold.md. The Chinese test file was read twice after a rule change; the published row is the first read (recognizer-gold.md).

Recognizer speed against Microsoft.Recognizers.Text 1.8.13, release gate, arm64 (English): numbers, units, currency and sequences 0.605 s against 31.874 s (52.7x faster; 402 MB allocated against 8,798 MB); with date/time 0.721 s against 27.085 s (37.6x; 616 MB against 4,879 MB); RSS 85 and 68 MB against 310 and 292 MB. x64 release run: 0.542 vs 26.140 s (48.23x), 402.3 vs 8,794.7 MB allocated without date/time; 0.621 vs 21.948 s (35.34x), 615.7 vs 4,877.2 MB with date/time. Both pass. RSS 113 / 103 MiB vs 344 / 325 MiB. Load times, input counts and machine record. The comparator is far slower in Turkish and Hindi (0.01 to 0.03 million characters per second); those speed rows are not part of the gate run.

Fast tier: tagging, lemmas, parsing, entities

Source for each language: its section in results.md and the recipe's PROVENANCE.md (recipes/<lang>/...). Treebanks are UD r2.18 test (English-EWT, German-GSD, French-GSD, Spanish-AnCora, Portuguese-Bosque, Italian-ParlaMint, Dutch-Alpino, Turkish-BOUN, Chinese-GSDSimp). Entity data is named per row. Ours / spaCy, each with its interval. Only the headline columns are shown here; XPOS, AllTags, UAS, CLAS, MLAS, sentences and per-type entity rows are in the sources.

English

Other languages

Hindi (self-labelled; information, not a gated comparison)

There is no allowed Hindi treebank or entity corpus (ud-licences.md: Hindi-HDTB and the Samanantar-sourced NER sets are non-commercial), so the Hindi student learns from Hindi web text (FineWeb-2) labelled by the accurate tier: the heads of five large languages and of heads we trained on the same frozen encoder for Marathi, Sindhi and Punjabi (UD, CC BY-SA), combined by agreement, with explicit adult content filtered out of the text first (recipes/hi/fast/README.md). Hindi-PUD (1,000 sentences, CC BY-SA 3.0, sha256 3562b7f4..., test only) was read once for the final model, which is the second round of this recipe; the first round (labels from nine unrelated languages) is shown as "before". Intervals are 95% bootstrap (1,000 resamples, seed 20261107). The "labeller" is the unfiltered ensemble of the source heads on the same input, the ceiling of what the student was shown. spaCy 3.8.18 is the black-box baseline trained on the same labelled sentences with its default configuration and the pipeline morphologizer,parser (the labels have no XPOS or lemmas). The published Stanza, UDPipe 2 and Trankit numbers are on Hindi-HDTB with non-commercial training data and are not comparable (references.md, "Hindi"). Hindi cannot be brought within noise of the published references with the training data that may be used.

The labeller's own gold-token scores rose from UPOS 72.77 and LAS 26.43 in the first round to 85.55 and 49.04. The student is 2.4 points below the labeller on UPOS and 3.8 above it on LAS (a parser trained on the medoid trees of many sources is more consistent than any one source). Against spaCy trained on the same labels it is within noise on every gold-token row and on raw UFeats, UAS, LAS, CLAS, MLAS and Sentences, and behind on raw UPOS (1.2 points; the tokenizer differs, 99.38 against 99.82). Lemmas: the model has no lemmatizer, and an edit-tree lemmatizer trained on Marathi was not clearly better than copying the form on 315 nouns, verbs and adjectives that we lemmatized by hand (62.5% against 61.0%), so the package fills Token.Lemma with the token text; the Lemmas row above is that, 72.7% on Hindi-PUD, and wrong for inflected words. Words are not scored separately from Tokens: Hindi has no multiword tokens.

Entities (information only). There is no Hindi entity gold we may use. A spot check on 91 held-out web sentences (61 entities) that we labelled by hand (testdata/hi/nerspot.iob2; PER, ORG, LOC): student 70.48 [57.14, 82.05] F1 (precision 84.09, recall 60.66), labeller 77.48 [65.35, 86.27], spaCy on the same labels 70.48 [56.47, 82.05]; the first round's student scored 67.92. On held-out labelled text the student reaches 79.36 F1 against the labeller's own entities. This is not a gated row.

Hand-annotated development sentences. 100 sentences (931 tokens, testdata/hi/selfdev.conllu: 17 web sentences and 83 that we wrote), gold tokens, UPOS / UAS / LAS: first-round student 80.56 [78.16, 82.80] / 70.46 / 55.85; second-round student 91.30 [89.43, 92.99] / 83.46 / 75.83; labeller (used set) 92.16 [90.64, 93.62] / 82.38 / 74.44; the nine large languages alone 83.03 / 71.64 / 59.51. These sentences ranked the source languages and the labelling model (recipes/hi/fast/README.md), so they are a development set, not a test.

Indicative speed (x64 Ryzen 9 7950X3D, 1 thread, load 7 to 9 from other jobs): full pipeline about 4.1k to 4.6k tokens/s over 20 repetitions on the development sentences (3.5k to 4.0k in the first round at higher load; the architecture and sizes are unchanged), 6.5 kB allocated per token.

Text classification and span categorization (English)

Reference: spaCy 3.8.18 textcat, textcat_multilabel and spancat, default configurations, same data. Sources: recipes/en/textcat/README.md, recipes/en/textcat-multilabel/README.md, recipes/en/spancat/README.md.

For information, the LexGLUE paper's EUR-LEX numbers on the same test split (TF-IDF with an SVM: micro-F1 71.3, macro-F1 51.4; BERT-base: 71.4 and 57.2) are above ours; ours is a from-scratch fast-tier model with a 512-word cut.

Embeddings

Two shipped models, scored with mteb 2.27.1 as a black box. "Ours minus sentence-transformers" is 0.00 on every task, because the vectors agree to 4e-7 and 3e-7 (maximum absolute difference, 2,103 texts, zero token-ID mismatches). "Published" is the model card's number. The gate is runtime fidelity (same model, same harness); the card gaps below are shown, not hidden. Source: embeddings.md.

The four bold cells are below the card for sentence-transformers itself, unchanged across mteb 1.1.1, 1.12.94 and 2.27.1; the card's numbers could not be reproduced from the published model with the released harness, whoever computes the vectors. The int8 variants are information only (0.3 to 3.8 points lower).

Open-type NER

Release measurement through packed Models.Multilingual/FrozenEncoder 0.3.0-preview.1, core/Neural 0.4.0-preview.1, frozen student-v1 and default threshold 0.5. The newer scratch feed had no multilingual package: this is not a test of a newly packed 0.4.0-preview.1 model. The embedded head, frozen encoder and tokenizer match the current weight manifests byte for byte. Every score and interval below reproduces recipes/openner/results.md to its reported precision. Strict token-span-and-type micro F1; 1,000 sentence-bootstrap resamples, seed 1.

Published reference: Zaratiana et al., GLiNER, arXiv:2311.08526v1, Table 1, §4.1. Model sizes and encoders (§3.2, Table 1): GLiNER-S 50M (primary, smallest), M 90M, L 0.3B, DeBERTa-v3 family (as reported for S/M/L, with no exact checkpoint identifiers or revisions specified). Ours is a frozen 6×384 student-v1, 35.5M encoder weights + 595,969 head parameters, about 36.1M total. The head parameter count is measured from the loaded package.

Position is relative to our CI, since the paper has no CI: published S above our upper bound = behind, below our lower bound = ahead, otherwise within noise. It is not a paired test. All five CrossNER rows lose numerically to S, M and L; the gaps to S are politics 20.23, science 19.10, music 34.89, literature 29.62, AI 29.60 points. None proves the published-small-model quality target is met. “Not comparable” is not a passing verdict.

The fixed-type black-box spaCy 3.8.18 comparison on UNER English-EWT remains 62.32 [59.45, 65.20], trained on the same UNER data with documented defaults. Ours is ahead by the point-vs-CI convention, but intervals overlap and this is not a new paired test or a zero-shot reference.

Protocol caveats (full audit): our unchanged scorer uses gold tokens and skips empty/>120-token sentences and whitespace-containing tokens (none excluded in UNER/CrossNER; 11 of 37,648 excluded in Few-NERD); asks the retained file's whole type inventory using our expanded label names; and keeps scores ≥0.5, whereas paper §2.3 says >0.5. Paper §3.4 names the OOD benchmark and exact-match entity F1 but supplies no split hashes, retained sentence counts, full label-name lists or matching filters. Equivalence cannot be established from the paper alone. Few-NERD includes both fine and coarse gold labels on the same mention, with flat predictions: it is not the conventional fine-only benchmark. The model/protocol notes and tools/OneNlp.OpenNerGate/README.md give provenance and reproduction commands.

Accurate tier

The distilled package covers nine languages; the base-size package covers 84 for syntax and 19 for entities. The gate is the paired comparison with spaCy 3.8.18 trained on the same data: 144 rows, 117 ahead, 14 within noise, 13 behind (listed in accurate-tier-rows.md). Published research systems (Stanza, UDPipe 2, Trankit) are information: they are ahead on most rows (1,205 of 1,314 best-published comparisons, 534 of 650 against Stanza 1.5.1), typically by 3–9 LAS points. Every row is in accurate-tier-rows.md.

The following English rows are historical information, not a summary of the latest 84-language weights. Their values are unchanged. They come from accurate-encoder.md, recipes/accurate/student-v1/RESULTS.md and, for the references, references.md (published numbers on other UD releases: compare the direction, not the last digit).

Accurate-tier speed and load (indicative, x64)

One-thread English, first 100 EWT r2.18 test texts (2,206 word tokens), median of three passes in a non-exclusive heavy job at load 12.6–13.8: .Multilingual 462 word tokens/s, model load 3,512 ms, cold/throughput peak RSS 809/814 MiB; .Large 150 word tokens/s, model load 3,134 ms, peak 3,904/4,055 MiB. Installed weights 529,158,395 and 3,508,907,198 bytes. These are information only, not idle-machine speed gates. Input hash, first-call times and method. Existing quality rows are unchanged.

Misses and caveats

• English raw MLAS 65.30 against spaCy's 69.05: outside the interval; information, not one of the gated rows.
• Span categorizer: at spaCy's threshold of 0.5 precision was 70.33 against 75.38 (paired -5.05 [-8.63, -1.42]); the shipped default threshold is 0.8, chosen on the development split (recipes/en/spancat/README.md).
• Embeddings: the five cells below the model cards, shared with sentence-transformers (above).
• Language ID: the languages named under the table are below Lingua on short text.
• Entities per type: per-type rows are in each source file; several have intervals that overlap with a paired difference that excludes zero (for example English LOC with the single-encoder option); the packaged English model has none of those.
• Gaps: Hindi has a fast tier but only on transferred labels (see "Hindi", not a gated comparison); no German, Spanish, Portuguese, Italian, Dutch, Turkish or Chinese rows against Catalyst (it has English models only); many accurate-tier published-reference rows remain below the reference (see accurate-tier-rows.md); open-type NER rows are above, with all five CrossNER rows numerically behind GLiNER-S and protocol equivalence unverified.
• Speed against Catalyst 26.10.6514 (English, 1 thread, arm64 Apple M2, release gate; median tokens/s of the last 10 of 12 passes): tagging path (Catalyst tagging 1,007,344 tokens/s, 307 B/token; ours sentences, UPOS/UFeats and lemmas 670,783, 330 B/token) is 0.666x, above the 0.5x bar. It counts under matched or better accuracy on raw EWT test: Tokens 99.00 vs 94.47, Sentences 71.08 vs 64.57, UPOS 92.90 vs 80.70, Lemmas 95.03 vs 83.84. NER path: Catalyst with WikiNER 478,220 vs ours sentences and entities 333,620 (0.70x), not matched work (ours excludes tagging; Catalyst's NER needs it), and so indicative only; entities on UNER character spans 63.09 vs 57.77. Ours parser path 25,535 and full pipeline 22,634 tokens/s (Catalyst has no parser). RSS 311 MB against 1,401 MB. English model load time not measured in this run. x64 release run: tagging 833,483 vs 1,049,762.5 tokens/s (0.794x, passes ≥0.5x), 330 vs 307 B/token. NER 443,997.5 vs 471,738 (0.941x, information only); parser 45,235.5 and full pipeline 38,313 tokens/s. Process peak RSS ours 1,240 vs Catalyst 383 MiB; English load not measured. Same accuracy condition as arm64. Full paths, samples and machine record.
• Spanish gold LAS is within noise at the release gate (86.41 [85.90, 86.89] vs 85.95); an earlier version of this table showed an older parser.
• Rows behind at the gate: English raw MLAS (information); Hindi raw Tokens 99.38 vs 99.82 and raw UPOS 82.41 vs 83.65 (Hindi is not a gated comparison); Language ID below Lingua on short text for the languages named under that table; embeddings cells below the model cards (shared with sentence-transformers); tokenizer load time (three arm64 rows and four x64 rows, above). The current Spanish PER row is ahead, not behind. All five open-type CrossNER rows are numerically behind GLiNER-S (protocol caveats above). No recognizer, fast-tier gated or tokenizer exactness row is behind.
• Not re-run at the gate: Duckling and Lingua predictions were reused from pinned earlier runs; English entities have no paired spaCy file at the gate; embeddings MTEB rows stand because the package manifests are unchanged since those runs (parity re-run on the packaged files: bge max abs 4.03e-7, e5 3.20e-7, 0 id mismatches).