# onenlp docs, all pages (OneNLP 1.0.0) Every page of https://recursiveai.org/onenlp/docs/ as markdown, getting started first. Index: https://recursiveai.org/onenlp/llms.txt # Getting started with OneNLP Install OneNLP for .NET 10, find the language, sentences, numbers and dates of a text with the core package alone, then add a model package for entities, lemmas and the parse, and another for sentence embeddings. Every program below was run on the 1.0.0 packages from NuGet, and the output shown is what it printed. ## Install OneNLP targets `net10.0`, so you need the .NET 10 SDK. ```shell dotnet new console -o NlpStart cd NlpStart dotnet add package RecursiveAI.OneNlp ``` The core package depends on the .NET base class library and `RecursiveAI.OneFont` only, and contains no native code. It holds the tokenizers, word and sentence segmentation, rule matching, the recognizers and language identification for 75 languages. Trained models come in their own packages, one per language or task. Nothing is ever downloaded: a model is delivered with its package. ## With the core package alone `Program.cs`: ```csharp using RecursiveAI.OneNlp; string text = "Angela Merkel met Emmanuel Macron in Paris on 5 March 2026. The deal cost $5m."; // Language ID, sentences and tokens: in the core package, no model needed. Console.WriteLine(Nlp.DetectLanguage(text)); Console.WriteLine(Nlp.DetectLanguage("Der Vertrag wurde gestern in Berlin unterschrieben.")); foreach (Span s in Nlp.Sentences(text)) Console.WriteLine($"[{s.StartChar}..{s.EndChar}) {s.Text}"); Console.WriteLine(string.Join(" | ", Nlp.Tokens(text).Take(8))); // Recognizers: numbers, money, dates, resolved to values. foreach (Quantity q in Nlp.Quantities(text)) Console.WriteLine($"{q.Kind} {q.Value} {q.Currency}"); DateTimeOffset today = new(2026, 3, 18, 10, 30, 0, TimeSpan.Zero); DateRange? when = Nlp.ParseDate("the first Monday after Easter 2027", today); Console.WriteLine(when?.Start.ToString("yyyy-MM-dd")); // A verb that needs a model throws and names the package to add. try { Nlp.Entities(text); } catch (InvalidOperationException e) { Console.WriteLine(e.Message); } ``` `dotnet run` prints: ```text en de [0..59) Angela Merkel met Emmanuel Macron in Paris on 5 March 2026. [60..78) The deal cost $5m. Angela | Merkel | met | Emmanuel | Macron | in | Paris | on Money 5000000 USD 2027-03-29 No model is available for language 'en'. Add the NuGet package RecursiveAI.OneNlp.Models.English (or RecursiveAI.OneNlp.Models.Multilingual); models are never downloaded. ``` Offsets are UTF-16 offsets into the string you passed in. Relative dates need a reference date, which you pass: Easter 2027 is 28 March, so the first Monday after it is 29 March. The exception is a `ModelNotAvailableException`, an `InvalidOperationException` that names the package to add. ## Add a model: entities, lemmas and the parse ```shell dotnet add package RecursiveAI.OneNlp.Models.English dotnet add package RecursiveAI.OneNlp.Models.Embeddings ``` `RecursiveAI.OneNlp.Models.English` is the English fast tier: tags, morphology, lemmas, the dependency parse and entities. `RecursiveAI.OneNlp.Models.Embeddings` is the default English sentence embedder for `Nlp.Embed` (bge-small-en-v1.5, MIT). There is nothing to register: add the package and the `Nlp` verbs use it for text in its language. `Program.cs`: ```csharp using RecursiveAI.OneNlp; string text = "Angela Merkel met Emmanuel Macron in Paris on 5 March 2026. The deal cost $5m."; // With RecursiveAI.OneNlp.Models.English: entities, lemmas and the full analysis. foreach (Span e in Nlp.Entities(text)) Console.WriteLine($"{e.Label} {e.Text} [{e.StartChar}..{e.EndChar})"); Console.WriteLine(string.Join(" ", Nlp.Lemmas("The cats were running."))); Doc doc = Nlp.Analyze(text); foreach (Token t in doc.Take(5)) Console.WriteLine($"{t.Text} {t.Pos} {t.Lemma} {t.Dep} -> {t.Head.Text}"); // With RecursiveAI.OneNlp.Models.Embeddings: sentence vectors. float[] a = Nlp.Embed("how do I reset my password"); float[] b = Nlp.Embed("I forgot my password and can't log in"); float[] c = Nlp.Embed("what time does the bakery open"); Console.WriteLine($"{a.Length} dimensions"); Console.WriteLine($"password / login: {Cosine(a, b):F2}"); Console.WriteLine($"password / bakery: {Cosine(a, c):F2}"); static float Cosine(float[] x, float[] y) { float dot = 0, nx = 0, ny = 0; for (int i = 0; i < x.Length; i++) { dot += x[i] * y[i]; nx += x[i] * x[i]; ny += y[i] * y[i]; } return dot / MathF.Sqrt(nx * ny); } ``` `dotnet run` prints: ```text PER Angela Merkel [0..13) PER Emmanuel [18..26) LOC Paris [37..42) the cat be run . Angela PROPN Angela compound -> Merkel Merkel PROPN Merkel nsubj -> met met VERB meet root -> met Emmanuel PROPN Emmanuel xcomp -> met Macron PROPN Macron obj -> met 384 dimensions password / login: 0.86 password / bakery: 0.49 ``` This is the model’s real output on this sentence, mistakes included: it marks “Emmanuel” alone as a person and attaches “Macron” as the object. `Nlp.Analyze` returns a `Doc` with the tokens, sentences, tags, lemmas, the parse and the entities from one call. ## Another language Each fast-tier language is its own package: `RecursiveAI.OneNlp.Models.French`, `.German`, `.Spanish`, `.Portuguese`, `.Dutch`, `.Italian`, `.Chinese`, `.Turkish` and `.Hindi`. With `RecursiveAI.OneNlp.Models.German` in place of the English model: ```csharp using RecursiveAI.OneNlp; string text = "Die Bäckerei Sonnenschein in Hamburg eröffnet am 3. Mai eine zweite Filiale. Der Umbau kostete 12.500 Euro."; Console.WriteLine(Nlp.DetectLanguage(text)); foreach (Span e in Nlp.Entities(text)) Console.WriteLine($"{e.Label} {e.Text}"); Console.WriteLine(string.Join(" ", Nlp.Lemmas(text).Take(8))); foreach (Quantity q in Nlp.Quantities(text, Language.German)) Console.WriteLine($"{q.Kind} {q.Value} {q.Currency}"); DateTimeOffset today = new(2026, 4, 20, 9, 0, 0, TimeSpan.Zero); Console.WriteLine(Nlp.ParseDate("am 3. Mai", today, Language.German)?.Start.ToString("yyyy-MM-dd")); ``` `dotnet run` prints: ```text de ORG Bäckerei Sonnenschein LOC Hamburg MISC Euro der Bäckerei Sonnenschein in Hamburg eröffnen an der Ordinal 3 Ordinal 2 Money 12500 EUR 2026-05-03 ``` The recognizers take the language as an argument (`Nlp.Quantities(text)` without one reads English). The German model splits “am” into two words, with the lemmas “an” and “der”. ## The accurate tiers Two packages cover many languages with one model: - `RecursiveAI.OneNlp.Models.Multilingual` is the distilled accurate tier: syntax and entities for English, German, French, Spanish, Portuguese, Italian, Dutch, Turkish and Chinese, plus open-type entities (`Nlp.Entities(text, types)`, labels named at runtime). - `RecursiveAI.OneNlp.Models.Multilingual.Large` is the base-size accurate tier: syntax for 84 languages and entities for 19. It is about 3.5 GB across 19 packages; reference the root package only, and deploy the complete publish directory. ```shell dotnet add package RecursiveAI.OneNlp.Models.Multilingual.Large ``` When a fast-tier package for the text’s language is referenced, the `Nlp` verbs use it; otherwise they use the multilingual accurate tier. See [Model packages](https://recursiveai.org/onenlp/docs/model-packages/) for the order and for loading one package’s pipeline directly. ## Where next - [How the API reads](https://recursiveai.org/onenlp/docs/reads-like/): the happy path, the low-level `Pipeline` and `Tokenizer`. - [Recognizers](https://recursiveai.org/onenlp/docs/recognizers/) and [Language identification](https://recursiveai.org/onenlp/docs/language-id/). - [Sentence embeddings](https://recursiveai.org/onenlp/docs/embeddings/), including the multilingual embedder. - [Training and recipes](https://recursiveai.org/onenlp/docs/training/): train every neural part with the API that trained the shipped models. - [Results table](https://recursiveai.org/onenlp/docs/results-table/): every measured row, including the rows behind. # OneNLP **spaCy-class NLP for .NET.** A clean-room library for .NET 10 with no native code, NativeAOT and trim clean, built on [OneModel](https://www.nuget.org/packages/RecursiveAI.OneModel). OneNLP is unaffiliated with other projects named OneNLP. Text in, structure out: tokens, sentences, tags, morphology, lemmas, dependency parses and entities through fast and accurate model tiers; token-exact Hugging Face tokenizers; numbers, units, currency, e-mail and URLs, and dates and times resolved to values in ten languages; language identification for 75 languages; text classification, span categorization and open-type entity recognition; sentence embeddings. You can train every one of the neural parts yourself with the API that trained the shipped models. > Every quality and speed claim here is a row in [docs/results-table.md](https://recursiveai.org/onenlp/docs/results-table/), with its reference, its interval and its verdict, including the rows we lose. ## What is in it | Package | Contents | |---|---| | `RecursiveAI.OneNlp` | The core; depends on the .NET base class library and [`RecursiveAI.OneFont`](https://www.nuget.org/packages/RecursiveAI.OneFont) only. `Doc`, `Token`, `Span` with character offsets into your string; `Pipeline` with batch and streaming processing; tokenizers that match Hugging Face `tokenizers` byte for byte (WordPiece, BPE, byte-level BPE, Unigram/SentencePiece, Tiktoken; `tokenizer.json`, SentencePiece `.model`, `vocab.txt` and `.tiktoken` loaders); UAX #29 word and sentence segmentation; `Matcher`, `PhraseMatcher`, `EntityRuler`; recognizers for numbers, ordinals, percentages, units, currency, e-mail, URL, IP, phone-like sequences, hashtags, mentions, GUIDs and dates and times, with rules as data, in English, French, German, Spanish, Portuguese, Italian, Dutch, Turkish, Hindi and Chinese; language ID for 75 languages (3.6 MB); JSON and binary serialization of documents; the `Nlp` statics. | | `RecursiveAI.OneNlp.Neural` | Depends on the core, `RecursiveAI.OneModel` and `RecursiveAI.OneModel.Training`. The encoders (the fast hash-embedding encoder, transformer encoders loaded from exported ONNX weights, fine-tunable), the heads (tagger, morphologizer, edit-tree lemmatizer, biaffine dependency parser, entity recognizer, text classifier, span categorizer, and, as trained word tokenizers, a multiword splitter and a word segmenter), `Train` with checkpoints and early stopping, `PipelineFile`, and the sentence embedder. | | `RecursiveAI.OneNlp.Models.English`, `.French`, `.German`, `.Spanish`, `.Portuguese`, `.Dutch`, `.Italian`, `.Chinese`, `.Turkish`, `.Hindi` | Ten fast-tier languages: tokenization, tags, morphology, lemmas, dependency parse and entities (Hindi has no learned lemmas or XPOS; see below). Add the package and `Nlp.Analyze` and `Nlp.Entities` use it. | | `RecursiveAI.OneNlp.Models.Embeddings` | The default English sentence embedder (bge-small-en-v1.5, MIT) for `Nlp.Embed`. | | `RecursiveAI.OneNlp.Models.Embeddings.Multilingual` | multilingual-e5-small (MIT), shipped in shards that depend on one another; reference this one package. | | `RecursiveAI.OneNlp.Models.Multilingual` | Distilled accurate tier: syntax and entities for English, German, French, Spanish, Portuguese, Italian, Dutch, Turkish and Chinese (nine languages), plus open-type NER (labels named at runtime). | | `RecursiveAI.OneNlp.Models.Multilingual.Large` | Base-size accurate tier: syntax for 84 languages, entities for 19, with trained word segmentation for unspaced scripts. About 3.5 GB across 19 packages; reference the root package. Deploy the complete publish directory, including the copied head assets. | **Hindi** has no treebank we may train on (Hindi-HDTB is non-commercial), so `RecursiveAI.OneNlp.Models.Hindi` is trained on Hindi web text that the accurate tier labelled by transfer from other languages, Marathi and Sindhi above all. It has UPOS, features, a parse and PER/ORG/LOC entities, no real lemmas (`Token.Lemma` is the token text) and no XPOS, and its scores (UPOS 83.2, LAS 52.8 on Hindi-PUD, gold tokens) are below the other languages'; see [docs/results-table.md](https://recursiveai.org/onenlp/docs/results-table/). Hindi UD is not available for an accurate-tier model. Nothing is ever downloaded. A model is delivered with its package; without the package, a verb that needs it throws an exception that names the package to add (see [docs/model-packages.md](https://recursiveai.org/onenlp/docs/model-packages/)). ## Install dotnet add package RecursiveAI.OneNlp dotnet add package RecursiveAI.OneNlp.Models.English // a model package per language you analyze dotnet add package RecursiveAI.OneNlp.Models.Embeddings // for Nlp.Embed ## The happy path Static task verbs, no options objects; the language is detected for you. The code below is `samples/OneNlp.Samples/HappyPath.cs`, which builds and runs. ```csharp using RecursiveAI.OneNlp; string text = "Angela Merkel met Emmanuel Macron in Paris on 5 March 2026. The deal cost $5m."; Language language = Nlp.DetectLanguage(text); // Language.English IReadOnlyList sentences = Nlp.Sentences(text); // sentence spans, offsets into text Token[] words = Nlp.Tokens(text); // word tokens string[] lemmas = Nlp.Lemmas("The cats were running."); // "the", "cat", "be", "run", "." IReadOnlyList entities = Nlp.Entities(text); // "Angela Merkel" PER, "Paris" LOC, ... DateTimeOffset today = new(2026, 3, 18, 10, 30, 0, TimeSpan.Zero); DateRange? when = Nlp.ParseDate("the first Monday after Easter 2027", today); IEnumerable money = Nlp.Quantities(text); // 5000000 USD Doc doc = Nlp.Analyze(text); // tokens, sentences, tags, lemmas, the parse and the entities in one call ``` Offsets are always UTF-16 offsets into the string you passed in. Relative dates need a reference date, which you pass; time zones are explicit. [docs/reads-like.md](https://recursiveai.org/onenlp/docs/reads-like/) shows the happy path next to Apple's NaturalLanguage, the low-level `Pipeline` next to spaCy, and `Tokenizer` next to Hugging Face `tokenizers`. ## Moving from another library Call-by-call maps, with no compatibility shims: - [Catalyst](https://recursiveai.org/onenlp/docs/migrate-catalyst/) - [Microsoft.Recognizers.Text](https://recursiveai.org/onenlp/docs/migrate-recognizers-text/) - [Lingua (SearchPioneer.Lingua)](https://recursiveai.org/onenlp/docs/migrate-lingua/) - [Microsoft.ML.Tokenizers](https://recursiveai.org/onenlp/docs/migrate-ml-tokenizers/) ## Measured against the leaders [docs/results-table.md](https://recursiveai.org/onenlp/docs/results-table/) has one table: component, language, metric, our score with its interval, the reference, and whether the difference is within noise. In short: - **Tokenizers** are byte-exact against Hugging Face `tokenizers` and `tiktoken` on twelve model families, and at least as fast as Microsoft.ML.Tokenizers on all 47 timed rows (rows the comparator cannot reproduce are listed in the results). - **Language ID** is more accurate than Lingua on words, word pairs and sentences, and faster on every row (a few rows below it per language are listed). - **Recognizers** score ahead of Microsoft.Recognizers.Text on our hand-labelled sets in ten languages, and are faster on English, the language the speed rows measure. - **The fast tier** is within noise of spaCy 3.8.18 trained on the same data on the gated UD and entity rows, or ahead of it; the few rows behind are listed by name. - **The base-size accurate tier** (84 languages) is measured against spaCy 3.8.18 trained on the same data: 117 of 144 rows ahead, 14 within noise, 13 behind (listed under known gaps). Published research systems such as Stanza, UDPipe 2 and Trankit, which train a larger model per treebank, are ahead on most rows; their numbers are shown beside ours as information. - **Text classification and span categorization** are ahead of spaCy's defaults on accuracy and F1; span precision is within noise at the default threshold. - **Embeddings** equal sentence-transformers in the same harness; the model cards' own numbers are shown beside them, with the cells neither reproduces. The English tagging path is compared with Catalyst at matched or better measured accuracy; see the platform-specific speed rows in the results table. ## Known gaps Stated plainly; each is also in [docs/results-table.md](https://recursiveai.org/onenlp/docs/results-table/). - **Accurate-tier coverage:** no model for Ancient Hebrew, Basque, Belarusian, Bulgarian, Chintang, Classical Armenian, Estonian, Gothic, Hungarian, Kadiweu, Khoekhoe, Maghrebi Arabic French, Manx, Mbya Guarani, Naga, Nheengatu, Occitan, Old Church Slavonic, Pomak, Ruuli, Shanghainese, South Levantine Arabic, Turkish German, Urdu or Xibe (25 languages). Their available training data is non-commercial, platform-restricted or has unresolved terms; see [the treebank licence inventory](https://recursiveai.org/onenlp/docs/ud-licences/). Hindi is a separate test-only UD case, with the transferred-label fast tier described above. - **Accurate-tier quality:** 13 of 144 rows are behind spaCy trained on the same data: Chinese UPOS, UFeats, lemmas, UAS and LAS (LAS 61.42 against 68.24); Spanish UPOS, UFeats and lemmas (by 0.2–1.0); Dutch WikiNER entity F1, precision and recall (F1 83.70 against 86.64); German raw tokenization; Turkish raw sentence boundaries. Published research systems are ahead on most rows, typically by 3–9 LAS points (1,205 of 1,314 best-published comparisons; 534 of 650 against Stanza 1.5.1). See [the full accurate-tier rows](https://recursiveai.org/onenlp/docs/accurate-tier-rows/). - **Open-type NER:** all five CrossNER zero-shot rows are 19–35 F1 points below the published GLiNER-S model, and protocol equivalence is unverified. No zero-shot quality pass is claimed. - **Hindi:** measured on Hindi-PUD only; gold UPOS 83.19 and LAS 52.79, raw UPOS behind the same-label spaCy baseline. No learned lemmatizer or XPOS, and entity scores are information from a small hand-labelled sample, not a treebank gate. - English raw-text MLAS is behind spaCy (65.30 against 69.05; an information row). - Embeddings: five MTEB cells are below the model cards for sentence-transformers itself, and ours equal it. Language ID is below Lingua on some languages and short-text buckets (listed in the results). - Italian's fast tier is trained on 897 sentences, since the larger Italian treebanks are non-commercial. Catalyst has English models only, so the other fast-tier languages have no managed speed comparator. ## Documentation - Start with [docs/reads-like.md](https://recursiveai.org/onenlp/docs/reads-like/); the contracts every component follows are in [docs/contracts.md](https://recursiveai.org/onenlp/docs/contracts/). - Components: [tokenizers](https://recursiveai.org/onenlp/docs/parity/), [segmentation](https://recursiveai.org/onenlp/docs/segmentation/), [Matcher and EntityRuler](https://recursiveai.org/onenlp/docs/matching/), [serialization](https://recursiveai.org/onenlp/docs/json-format/), [recognizers](https://recursiveai.org/onenlp/docs/recognizers/), [language ID](https://recursiveai.org/onenlp/docs/language-id/), [the parser](https://recursiveai.org/onenlp/docs/parser/), [embeddings](https://recursiveai.org/onenlp/docs/embeddings/), [the accurate-tier encoder](https://recursiveai.org/onenlp/docs/accurate-encoder/), [model packages](https://recursiveai.org/onenlp/docs/model-packages/), [model files](https://recursiveai.org/onenlp/docs/model-files/), [data formats](https://recursiveai.org/onenlp/docs/data-formats/) and [training and recipes](https://recursiveai.org/onenlp/docs/training/). - How things are measured: [docs/measurement.md](https://recursiveai.org/onenlp/docs/measurement/), [docs/references.md](https://recursiveai.org/onenlp/docs/references/), [docs/provenance.md](https://recursiveai.org/onenlp/docs/provenance/). ## Building You need the .NET SDK pinned in `global.json`. dotnet build -c Release dotnet test --solution OneNlp.slnx -c Release --no-build Build output goes to `artifacts/` in the repository (ignored by git). Set `ONENLP_ARTIFACTS` to a directory to put it elsewhere; each checkout gets its own folder under it. Machine-specific MSBuild overrides can go in an ignored `Directory.Build.local.props`. Weights are not in git: the model projects embed them from a directory named by an environment variable (for example `ONENLP_ENGLISH_WEIGHTS`) and check each file against a committed hash; see each project's `PROVENANCE.md`. The NativeAOT smoke app is `tests/OneNlp.AotSmoke`; publish it with `dotnet publish tests/OneNlp.AotSmoke -c Release -r ` (it needs a native C linker: on Linux clang by default, or gcc with `-p:CppCompilerAndLinker=gcc`, and the zlib development package). Run the samples with `dotnet run --project samples/OneNlp.Samples -- all `. ## Layout - `src/`: the packages (`OneNlp`, `OneNlp.Neural`, one project per model package). - `recipes/`: the training recipes; each is a small console project over the public `Train` API, with its data manifest, provenance and results. - `samples/`: the code shown in the docs. - `tests/`, `bench/`, `fuzz/`: tests, the AOT smoke app, benchmarks and fuzz targets. - `tools/`: scripts for data, oracles and evaluation ([tools/README.md](https://github.com/recursive-ai-oss/onenlp-csharp/blob/59bd14caa0e08a13494d74e29c396d1444730911/tools/README.md)). - `docs/`: behaviour written down. ## Licence `MIT OR Apache-2.0`, at your option. See `LICENSE-MIT` and `LICENSE-APACHE`. Copyright (c) Recursive AI, LLC. Third-party notices: `THIRD-PARTY-NOTICES.md`. # Reads like One sample per layer, each next to the library or API it will feel familiar from. The C# is the code in [`samples/OneNlp.Samples`](https://github.com/recursive-ai-oss/onenlp-csharp/blob/59bd14caa0e08a13494d74e29c396d1444730911/samples/OneNlp.Samples), which builds and runs (`dotnet run --project samples/OneNlp.Samples -- all `); the other side is that library's own documented usage. The samples need the English model package, the embeddings package and the language-ID weights, as an application that references the packages would. Three layers, three shapes: - **Happy path (`Nlp`).** Static task verbs with no options objects, like Apple's NaturalLanguage framework: ask a question of a string, get an answer. The language is detected for you and the model comes from the packages you reference. - **Low level (`Pipeline`, `Doc`, `Matcher`, `Train`).** spaCy's documented shape and terms: a pipeline of components over a `Doc` of tokens, spans and attributes, with rule matchers and training. - **Tokenizers (`Tokenizer`).** The .NET span style: your buffers, an `OperationStatus`, nothing allocated; and the Hugging Face `Encoding` fields when you want them. ## Happy path: Apple NaturalLanguage and `Nlp` Apple's side (Swift): ```swift import NaturalLanguage let text = "Angela Merkel met Emmanuel Macron in Paris on 5 March 2026." let recognizer = NLLanguageRecognizer() recognizer.processString(text) let language = recognizer.dominantLanguage // .english let words = NLTokenizer(unit: .word) words.string = text words.enumerateTokens(in: text.startIndex.. sentences = Nlp.Sentences(text); // sentence spans, offsets into text Token[] words = Nlp.Tokens(text); // word tokens string[] lemmas = Nlp.Lemmas("The cats were running."); // "the", "cat", "be", "run", "." IReadOnlyList entities = Nlp.Entities(text); // "Angela Merkel" PER, "Paris" LOC, ... DateTimeOffset today = new(2026, 3, 18, 10, 30, 0, TimeSpan.Zero); DateRange? when = Nlp.ParseDate("the first Monday after Easter 2027", today); IEnumerable money = Nlp.Quantities(text); // 5000000 USD ``` ```csharp float[] question = Nlp.Embed("how do I reset my password"); // 384 floats, length 1 (normalized) float[] related = Nlp.Embed("I forgot my login credentials"); float[] unrelated = Nlp.Embed("the best way to bake sourdough bread"); static float Dot(float[] a, float[] b) => a.Zip(b, (x, y) => x * y).Sum(); ``` - `Nlp.Analyze(text)` returns a `Doc` with everything at once: tokens, sentences, tags, lemmas, the dependency parse and the entities. - Entity labels are strings (`"PER"`, `"ORG"`, `"LOC"`, `"MISC"`); `EntityLabels` has the names as constants. - Relative dates need a reference you pass in. There is no hidden clock, and time zones are explicit. - A verb that needs a model you did not add throws an exception that names the package ([model-packages.md](https://recursiveai.org/onenlp/docs/model-packages/)); nothing is downloaded. - What the verbs do not do (yet): Apple's `NLTagger` also gives lexical classes and sentiment; the equivalents here are `Token.Pos` on the analyzed `Doc` and, for classification, `TextClassifier`. ## Low level: spaCy and `Pipeline` spaCy's side (Python): ```python import spacy from spacy.matcher import Matcher nlp = spacy.blank("en") ruler = nlp.add_pipe("entity_ruler", config={"overwrite_ents": True}) ruler.add_patterns([ {"label": "CASE_NO", "pattern": [{"SHAPE": "dd-dddd"}]}, {"label": "ORG", "pattern": "Recursive AI"}, ]) doc = nlp("Recursive AI filed case 21-1234 on Monday.") for ent in doc.ents: print(ent.label_, ent.text, ent.start_char, ent.end_char) matcher = Matcher(nlp.vocab) matcher.add("GREETING", [[{"LOWER": "hello"}, {"IS_PUNCT": True, "OP": "?"}, {"LOWER": "world"}]]) matches = matcher(nlp("Hello, world! hello world")) for d in nlp.pipe(["First text.", "Second text."]): print(len(list(d.sents))) ``` OneNLP: ```csharp // A blank English pipeline: the Unicode word tokenizer, then the components in the order they are added. using Pipeline nlp = Pipeline.Blank(Language.English); // Rule entities, from token patterns (the vocabulary of spaCy's rule-based matching) or from phrases. EntityRuler ruler = new() { OverwriteEnts = true }; ruler.Add("CASE_NO", Pattern.Shape("dd-dddd")); ruler.AddPhrase("ORG", "Recursive AI"); nlp.Add(ruler); Doc doc = nlp.Process("Recursive AI filed case 21-1234 on Monday."); foreach (Span e in doc.Ents) { Console.WriteLine($"{e.Label} {e} {e.StartChar}..{e.EndChar}"); } // The Matcher finds token patterns in any Doc. Matcher matcher = new(); matcher.Add("GREETING", Pattern.Token(new TokenSpec().Lower("hello"), new TokenSpec().Is(TokenProperty.IsPunct, true).Optional(), new TokenSpec().Lower("world"))); IReadOnlyList matches = matcher.Match(nlp.Process("Hello, world! hello world")); // Many texts at once, lazily, in batches. foreach (Doc d in nlp.ProcessMany(["First text.", "Second text."])) { Console.WriteLine(d.Sentences.Count); } ``` Training uses the same shape as spaCy's `nlp.update` loop, wrapped as one call ([training.md](https://recursiveai.org/onenlp/docs/training/)): ```csharp TrainResult result = nlp.Train(examples, new TrainOptions { Epochs = 30, Seed = 7, Dropout = 0f, DevExamples = examples }); PipelineFile.Save(nlp, path); // one file: tokenizer, encoder, heads, labels ``` How the terms map: | spaCy | OneNLP | Note | |---|---|---| | `spacy.blank("en")` | `Pipeline.Blank(Language.English)` | `UaxWordTokenizer` (UAX #29 words, English exceptions); add components with `Add` | | `spacy.load("en_core_web_sm")` | reference `RecursiveAI.OneNlp.Models.English`; `Nlp.Analyze`, or `PipelineFile.Load(path)` for a file | models are NuGet packages, never downloaded | | `nlp(text)` / `nlp.pipe(texts)` | `nlp.Process(text)` / `nlp.ProcessMany(texts)` (and `ProcessManyAsync`) | synchronous; batches are read lazily | | `doc.ents`, `ent.label_`, `ent.start_char` | `doc.Ents`, `span.Label`, `span.StartChar` | offsets are UTF-16 units in the string you passed | | `token.pos_`, `token.tag_`, `token.lemma_`, `token.dep_`, `token.head` | `token.Pos`, `token.Tag`, `token.Lemma`, `token.Dep`, `token.Head` | `Token` and `Span` are struct views over the `Doc` | | `Matcher`, `PhraseMatcher`, `entity_ruler` | `Matcher`, `PhraseMatcher`, `EntityRuler` | the pattern vocabulary is spaCy's ([matching.md](https://recursiveai.org/onenlp/docs/matching/)) | | `doc.cats`, `textcat`, `spancat` | `doc.Cats`, `TextClassifier`, `SpanCategorizer` | | | `spacy train` with a config file | `nlp.Train(examples, new TrainOptions {...})` and the recipes in `recipes/` | | ## Tokenizers: Hugging Face `tokenizers` and `Tokenizer` Hugging Face `tokenizers` (Python): ```python from tokenizers import Tokenizer tok = Tokenizer.from_file("tokenizer.json") enc = tok.encode("Hello, world! Tokenizers count in UTF-16 units.") for i, t in enumerate(enc.tokens): print(enc.ids[i], t, enc.offsets[i]) count = len(tok.encode("How long is this text, in tokens?").ids) text = tok.decode(enc.ids) # skip_special_tokens=True by default ``` OneNLP: ```csharp Tokenizer tok = Tokenizer.FromFile(tokenizerJson); // tokenizer.json, SentencePiece .model, vocab.txt, .tiktoken // Ids, tokens and character offsets into the input, as the Hugging Face Encoding has them. TokenEncoding enc = tok.Encode("Hello, world! Tokenizers count in UTF-16 units."); for (int i = 0; i < enc.Length; i++) { Console.WriteLine($"{enc.Ids[i],6} {enc.Tokens[i],-12} {enc.Offsets[i].Start}..{enc.Offsets[i].End}"); } // The span form: ids go into your buffer, nothing is allocated, and the status says whether it all fit. int[] buffer = new int[8]; OperationStatus status = tok.EncodeToIds("Hello, world! Tokenizers count in UTF-16 units.", buffer, out int written, out int consumed); int count = tok.CountTokens("How long is this text, in tokens?"); string back = tok.Decode(enc.Ids); // special tokens skipped ReadOnlySpan budget = tok.TrimToTokenBudget("one two three four five six seven eight", 4); ``` - The ids, tokens, offsets, special-token mask, type ids, attention mask and word ids equal the Python library's, byte for byte, on every family in [parity.md](https://recursiveai.org/onenlp/docs/parity/). Offsets count UTF-16 units for `char` input and UTF-8 bytes for byte input; special tokens carry `(0, 0)`. - `EncodeToIds` writes into your buffer and returns `Done` or `DestinationTooSmall`; after `DestinationTooSmall` the ids written are an exact prefix of the full encoding. - `Truncation` and `Padding` are properties of the tokenizer (loaded from the file's own settings) and apply to `Encode` and `EncodeBatch`; `EncodeToIds`, `CountTokens` and `TrimToTokenBudget` ignore them. - `TrimToTokenBudget` returns the slice of your text covered by the first (or last) *n* tokens, for fitting text to a model's context. # Contracts Behaviour that every component and tokenizer family relies on. Language-neutral and normative. ## Offsets - Every offset a `Doc`, `Token` or `Span` reports is a UTF-16 code unit offset into the string the caller passed in (`Doc.Text`). Nothing is reported against a normalized copy. - A token has `Idx` (first character) and `EndIdx` (just past the last). Spans cover whole tokens; `StartChar` is the first token's `Idx` and `EndChar` the last token's `EndIdx`. An empty span has `StartChar == EndChar`. - `Token.Text` and `Span.Text` are strings cut from `Doc.Text` on each call, so `token.Text == "x"` compares contents. `TextSpan` is the same text as a `ReadOnlySpan` over `Doc.Text` and copies nothing; the library's own loops use it. A span's text includes the whitespace between its tokens. - `Doc.TryGetCharSpan` makes a span from character offsets only when both fall on token boundaries. Nothing is rounded. - Tokenizer encodings (`TokenOffset`) count UTF-16 units for text passed as `char`, and bytes for text passed as UTF-8. Tokens that stand for no input text (added special tokens, padding) have the offsets `(0, 0)`. ## Doc - Token boundaries are fixed when the document is created. They are non-empty, ordered and non-overlapping; what lies between tokens is the caller's whitespace. - Annotations are columns indexed by token: strings as IDs in a shared `StringStore` (ID 0 is the empty string, meaning "not set"), heads as token indexes (a token that is its own head is a root), sentence boundaries as `Unset`/`Start`/`Inside`, entities as an ordered list of non-overlapping spans with a per-token IOB and type, and a confidence per token. - A new document is unannotated: empty strings, every token its own head, boundaries unset, `EntIob.Unset`, confidence 1. - A document with no sentence boundaries set is one sentence. After `SetEntities` (even with an empty list) every token has an `EntIob`. - `Token.Confidence` is the confidence of the token's tag annotation. Entity and span confidences live on the span. Rule-based results have confidence 1. - Reads may run concurrently; writes need exclusive use. The `StringStore` is thread-safe and only grows. ## Pipeline and components - A pipeline is a word tokenizer followed by components in the order they were added. Component names are unique within a pipeline. - A component only fills annotations; it never changes token boundaries or the text. - `Pipeline.Process` may be called from several threads when the components allow it: per-call state stays in locals, shared state is read-only. - `ProcessMany` reads its source lazily, tokenizes one batch, runs each component over the whole batch (`IPipelineComponent.ProcessBatch`, which defaults to calling `Process` per document), then yields the batch in input order. `ProcessManyAsync` is the same over an `IAsyncEnumerable`; only reading the source awaits. - Components are created by name from `ComponentRegistry`. Registration is by explicit call (a package's module initializer or a public `Register` method), never by scanning assemblies. An unknown name throws and lists the registered names. ## Tokenizers - `Tokenizer` is the one type callers hold; the families (WordPiece, BPE, byte-level BPE, Unigram, Tiktoken) derive from it and implement `EncodeCore`, `Decode`, `IdToToken`, `TokenToId` and `VocabularySize`. - `Encode` returns a `TokenEncoding`: ids, token strings, offsets, special-token mask, type ids, attention mask and word ids, all one entry per token. Truncation is the family's job (it must count special tokens against the limit); padding is applied by the base class afterwards. - `Truncation` and `Padding` apply to `Encode` and `EncodeBatch`. `EncodeToIds`, `CountTokens` and `TrimToTokenBudget` ignore them. - `EncodeToIds` writes IDs into the caller's buffer and returns `Done` or `DestinationTooSmall`. After `DestinationTooSmall` the written IDs are an exact prefix of the full encoding, and `consumed` is the end offset of the last written token; encoding the remainder separately is not guaranteed to continue the same sequence. - `TrimToTokenBudget` returns the slice of the text covered by the first (or last) *n* tokens of the full encoding without special tokens. - Invalid UTF-8 is read as U+FFFD, one per invalid sequence, and offsets still count the original bytes. - `Tokenizer.FromFile` picks the format from the extension (`.json`, `.model`, `.txt`, `.tiktoken`); `FromStream` takes the format explicitly. A format without a loader throws `NotSupportedException`. ## Tokenizer.json pipeline (internal seam) `Tokenizer.FromFile`/`FromStream(HuggingFaceJson)` returns a `HuggingFaceTokenizer`, composed as the published `tokenizer.json` format describes: **normalizer → pre-tokenizer → model → post-processor**, with a **decoder** for decoding, **added tokens** beside the model vocabulary, and the file's **truncation** and **padding** copied to `Tokenizer.Truncation` and `Padding`. All of it lives in `src/OneNlp/Tokenizers/Hf/` and is internal. - **Alignment.** `NormalizedString` holds the current text and, for each UTF-16 unit, the range of the original input it came from. Normalizers rewrite it with `NormalizedStringBuilder` (inserted or replaced units share the origin of the unit they stand for). `PreTokenizedString` is a list of such pieces; a piece's index is the word index of its tokens. A range of a piece maps back to the original input from the origin start of its first unit to the origin end of its last. - **Stages.** `INormalizer.Normalize(NormalizedString)`, `IPreTokenizer.PreTokenize(PreTokenizedString)`, `IPostProcessor.AddedTokenCount(isPair)` and `Process(encoding, pair, addSpecialTokens)`, `ITokenDecoder.DecodeChain(List)`. Stages are immutable and thread-safe. - **Model seam.** `ITokenizerModel.Tokenize(ReadOnlySpan piece, List output)` appends `(Id, Value, Start, End)` with offsets relative to the piece, plus `TokenToId`, `IdToToken` and `VocabularySize` (model entries only). The piece is the text after normalization and pre-tokenization (for byte-level BPE, the byte-mapped characters). Byte fallback, `ignore_merges`, unknown-token fusing and rank lookups are the model's business. - **Registry.** `StageRegistry` maps a `type` string to a factory per kind. A model or stage is added from its own file by implementing the matching optional partial method of `StageRegistry` and calling `AddModel`/`AddNormalizer`/`AddPreTokenizer`/`AddPostProcessor`/`AddDecoder`: `RegisterWordPiece`, `RegisterUnigram`, `RegisterBpe`, `RegisterTiktoken`, `RegisterNormalizers`, `RegisterPreTokenizers`, `RegisterPostProcessors`, `RegisterDecoders`. Registering an existing type replaces it. Every documented type is registered first as a stub that throws `NotSupportedException` ("... is not implemented yet"); an undocumented type throws "... is not supported". A model object without `type` (older files leave it out) is typed from its fields, as the reference does: `merges` present means `BPE`, otherwise a `vocab` list of pairs means `Unigram` and a `vocab` object means `WordPiece`; with no `vocab` it is read as `BPE`. - **JSON access.** Factories read their object through `StageConfig` (typed accessors that throw `InvalidDataException` naming the entry and field; `Element` for bulk reads such as a vocabulary). The loader reads at most 512 MiB and parses with a nesting limit of 32. - **Truncation and pair encoding** in the composed tokenizer: the room reserved for the template's special tokens is `IPostProcessor.AddedTokenCount`, counted only when special tokens are added; pairs are truncated longest-first. - **Stages implemented.** Normalizers: BertNormalizer, NFC, NFD, NFKC, NFKD, Lowercase, Strip, StripAccents, Replace, Prepend, Precompiled (SentencePiece character map), ByteLevel, Sequence. Pre-tokenizers: BertPreTokenizer, Whitespace, WhitespaceSplit, Punctuation, Metaspace, ByteLevel, Split (regular expression or literal), Digits, UnicodeScripts, CharDelimiterSplit, Sequence. Post-processors: TemplateProcessing, BertProcessing, RobertaProcessing, ByteLevel, Sequence. Decoders: WordPiece, Metaspace, ByteLevel, ByteFallback, Fuse, Strip, Replace, BPEDecoder, CTC, Sequence. - **Regular expressions.** `Split` patterns run on a small engine over code points (`CodePointRegex`): classes, Unicode properties (general category and script), lookahead, lazy and possessive quantifiers, case-insensitive groups. A match is cut off after a step budget (2,000,000 plus 500 per input character) and a stack-depth limit; exceeding either throws `InvalidOperationException`, and a pattern that does not parse throws `InvalidDataException`. Tiktoken encodings keep their `[GeneratedRegex]` split patterns, which are faster; they give the same pieces on the parity corpus. - **Normalization data.** NFC/NFD/NFKC/NFKD use tables generated from the Unicode Character Database by `tools/gen_normalization_tables.py`. The reference library behaves like an older Unicode version, so the tables keep decompositions and combining classes only for characters assigned up to Unicode 9.0 (read from `DerivedAge.txt`). The sets that BertNormalizer removes (control and format characters) and that accent stripping removes (nonspacing marks up to Unicode 8.0) are generated from a sweep of every scalar value against the reference (`tools/gen_bert_clean_table.py`); characters unassigned at that time are kept. - **Alignment rules observed in the reference** (each probed through its public `NormalizedString` API and recorded in comments): the pieces of a decomposed character keep that character's origin; canonical reordering moves characters but not origins; when composition shortens a run, the first character keeps the first origin and the rest line up with the end of the run; a run is a starter with the marks and the second members of composable pairs that follow it. The SentencePiece character map is applied per grapheme cluster (looked up as a whole when it is under 6 UTF-8 bytes, otherwise per code point); the k-th character of a replacement covers the k-th code point it replaces, and characters it deletes at the very start of the text pull all later origins back by that many code points. - **Lone surrogates** in a `string` input are replaced by U+FFFD (one unit each) before normalization, so offsets still count the original units. - **Loader errors.** Invalid UTF-8 in the file, a missing or mistyped member of a stage, or a value out of range raises `InvalidDataException` naming the stage; nothing else escapes. The files are never executed or fetched. - **Checks that need the reference.** `tools/stage_goldens/make_real_goldens.py` and `make_sweep.py` write reference answers for real tokenizer files and for every scalar value; `RealTokenizerGoldenTests` and `NormalizerSweepTests` read them from the locations named by environment variables and skip when those are unset. The full corpus comparison is `tools/OneNlp.Parity` (`docs/parity.md`). ## Encoders, heads and training (`RecursiveAI.OneNlp.Neural`) - **Encoder seam.** `IEncoder.Encode(ReadOnlySpan)` returns an `EncodedBatch`: one float32 row per token, the documents' rows stacked in order (`DocCount`, `Start(i)`, `Length(i)`, `Rows(i)`). One call serves both modes: with recording on and trainable `Weights` the matrix carries the autodiff graph into the encoder; inside `GradMode.NoGrad()` (inference, a frozen encoder) it is plain values. `Weights` (an `NnModule`, or `null` for an encoder with nothing to train or save, such as an imported ONNX model) fixes the state order for checkpoints. `Kind` and `WriteSettings` name the encoder in a saved pipeline (`EncoderRegistry`). `Dispose` is idempotent; heads that share an encoder may each dispose it. - **Head contract.** `ITrainableComponent : IPipelineComponent` has `Encoder`, `Initialize(examples, seed)` (fixes the label set from the training examples' reference documents, in an order that depends only on their content, and builds the weights), `Loss(EncodedBatch, examples)` (one rank-0 tensor; the heads' losses are added and backpropagated once), `Predict(EncodedBatch, docs)` (writes into the documents), `Weights`, `Kind` and `WriteConfig` (labels and settings; rebuilt by `HeadRegistry`). `TrainableHead` implements the pipeline side: it skips empty documents, runs the encoder and `Predict` without gradients in evaluation mode, and serializes calls on one head. A head sees whole batches, so a softmax tagger, a transition or biaffine parser, span or transition NER and a pooled classifier all fit. - **`Example`** (`RecursiveAI.OneNlp.Data`) pairs a predicted `Doc` with a reference `Doc`, plus gold `Cats` and `Spans`. `Alignment` maps tokens when the tokenizations differ (rules in `docs/data-formats.md`); a loss uses `GetAligned(column)`, where `null` means "no gold label". - **`Train`.** `pipeline.Train(examples, TrainOptions, CancellationToken)` trains every trainable head and the encoders they read with AdamW. Batches are made by token count from a shuffle that depends only on the seed and the epoch. Dropout is applied to the encoder output. The dev set is `DevExamples` or a deterministic split of `DevFraction`; the dev score is `Score` (a delegate over the dev examples with their predicted documents annotated; `DevScores.Ud(metric)` and `DevScores.Entities()` wrap the scorers) or else the negative mean dev loss. `Patience` ends the run after that many evaluations without a new best; the best weights are put back. With `CheckpointDirectory`, `last.ckpt` (weights, optimizer, dropout counter, position, best score, patience) and `best.ckpt` are written atomically, and a later call resumes from `last.ckpt` to exactly the weights an uninterrupted run reaches (same machine and thread count). Progress goes to `IProgress`; the library never writes to the console. - **Sharing one encoder between heads.** Heads that hold the same `IEncoder` instance share its work. - *Training.* `Train` encodes each batch once per distinct encoder instance, adds the heads' losses and runs `Backward` once, so the gradients of all heads sum into the encoder (this is how the tagger, morphologizer and lemmatizer train together). A head must therefore read only `EncodedBatch` and never call the encoder itself. - *Inference.* Each head's `ProcessBatch` calls `Encoder.Encode`. An encoder may keep its output per document so that the second and later heads find it: `FastEncoder` stores each document's rows in a `ConditionalWeakTable` keyed by the `Doc`, used only when recording is off and the weights are in evaluation mode, and gives the same values as encoding the document alone or in any batch (nothing crosses a document edge). The cache assumes the weights do not change while a document is alive, and ends with the document. Encoders without a cache are correct, only slower. - A new head kind joins a pipeline by `HeadRegistry.Register` (or an entry in the built-in table) and reads `encoded.Vectors`/`Rows(i)`; it needs nothing else from the encoder. - **Fast encoder** (`FastEncoder`, kind `fast`; settings `width`, `depth`, `normRows`, `prefixRows`, `suffixRows`, `shapeRows`, `seed`, all recorded in the saved pipeline). Hash embeddings (two hashes per token, summed) of the lower-cased text, the first character, the last three characters and the shape, mixed by a linear layer with GELU and layer norm, then `depth` residual window layers over the previous, current and next vector. Document edges are zero padded and nothing crosses documents. The hash is FNV-1a over UTF-16 units, seeded per attribute and finished with SplitMix64 (low half for hash 0, high half for hash 1, each modulo the table's rows); the shape maps upper case to `X`, lower case to `x`, digits to `d`, and cuts runs of the same character after 4. A model rebuilt from its file hashes every token to the same rows on every platform. - **Fast-tier heads.** `Tagger` (one string column), `Morphologizer` (kind `morphologizer`: joint labels `POS=|`, fills `Pos` and `Morph`) and `EditTreeLemmatizer` (kind `lemmatizer`; classes are an edit tree, Müller et al. 2015, plus a flag saying the form is lower-cased first, serialized as `l|r` followed by `M,()()` or `S::`; a token takes its most probable class among the best 16 whose tree fits its text, else its own text). The tagger and the morphologizer write the token confidence (the later component wins); the lemmatizer leaves it alone. - **Saved pipelines** are one file (`docs/model-files.md`). - **Readers** (`RecursiveAI.OneNlp.Data`): `ConllUReader`/`ConllUWriter` and `JsonlReader` (`docs/data-formats.md`). `UdCorpus.Read` in `Scoring` reads through `ConllUReader`. # Model packages A model package carries trained weights for one language (or one task) and makes the `Nlp` verbs use them. Nothing is ever downloaded: the weights are inside the package, embedded as resources. ## The packages | Package | Language or task | Registration class | Used by | |---|---|---|---| | `RecursiveAI.OneNlp.Models.English` | English fast tier | `EnglishModel` | `Nlp.Analyze`, `Nlp.Entities`, `Nlp.Lemmas` for English text | | `RecursiveAI.OneNlp.Models.French`, `.German`, `.Spanish`, `.Portuguese`, `.Dutch`, `.Italian`, `.Turkish`, `.Chinese` | the fast tier of that language | `FrenchModel`, `GermanModel`, `SpanishModel`, `PortugueseModel`, `DutchModel`, `ItalianModel`, `TurkishModel`, `ChineseModel` | the same verbs, for text of that language | | `RecursiveAI.OneNlp.Models.Embeddings` | English sentence embeddings (bge-small-en-v1.5) | `EmbeddingsModel` | `Nlp.Embed`, `Nlp.EmbedMany` | | `RecursiveAI.OneNlp.Models.Embeddings.Multilingual` | multilingual sentence embeddings (multilingual-e5-small); its three `...Shard1..3` packages hold the weights and come with it | `MultilingualEmbeddingsModel` | `Nlp.Embed`, `Nlp.EmbedMany` (wins over the English embedder when both are referenced) | | `RecursiveAI.OneNlp.Models.Multilingual` | the distilled accurate tier for English, German, French, Spanish, Portuguese, Italian, Dutch, Turkish and Chinese, and open-type entities (`Nlp.Entities(text, types)`); its parts `.Encoder`, `.FrozenEncoder` and `.Segmenters` come with it (`distilled-tier.md`) | `MultilingualModel` | `Nlp.Analyze`, `Nlp.Entities`, `Nlp.Lemmas` where no fast-tier package is referenced | | `RecursiveAI.OneNlp.Models.Multilingual.FrozenEncoder` | the frozen distilled encoder alone, for your own heads | `FrozenStudentEncoder.Create()` | an `IEncoder` | | `RecursiveAI.OneNlp.Models.Multilingual.Large` | the base-size accurate tier (`accurate-tier.md`) | `MultilingualLargeModel` | as above, preferred over `.Multilingual` when both are referenced | `RecursiveAI.OneNlp.Models.Hindi` (`HindiModel`) is the fast tier for Hindi: UPOS, features, parse and PER/ORG/LOC entities, with no real lemmas (`Token.Lemma` is the token text) and no XPOS. There is no allowed Hindi treebank (`ud-licences.md`), so it is trained on web text labelled by the accurate tier and by heads for related languages (`recipes/hi/fast/README.md`) and is markedly weaker than the other languages. Hindi text is also identified, tokenized, split into sentences and run through the Hindi recognizers without it. ## What the user does Add the package. That is all. ```csharp Doc doc = Nlp.Analyze(text); // tokens, sentences, tags, morphology, lemmas, parse, entities IReadOnlyList ents = Nlp.Entities(text); ``` The language of the text is detected with the built-in language identifier, and the package registered for that language is used. A text with no letters is treated as English. Without a package for the text's language, `Nlp.Analyze` returns tokens and sentence boundaries only, and the model verbs (`Nlp.Entities`, `Nlp.Lemmas`) throw `InvalidOperationException`. The message names the exact package to add: > No model is available for language 'de'. Add the NuGet package RecursiveAI.OneNlp.Models.German (or RecursiveAI.OneNlp.Models.Multilingual); models are never downloaded. The same for embeddings, when no embedding package is referenced: > No embedding model is available. Add the NuGet package RecursiveAI.OneNlp.Models.Embeddings; models are never downloaded. Resolution order for a text: the fast-tier package of its language if referenced, otherwise an accurate-tier package that covers it: `RecursiveAI.OneNlp.Models.Multilingual.Large` if referenced, else `RecursiveAI.OneNlp.Models.Multilingual`. To use a particular package regardless of detection, load its pipeline (`EnglishModel.Load()`, ...) or any saved pipeline file (`PipelineFile.Load`) and call `Process` on it; the low level takes any file or stream, including models you trained yourself or fine-tuned from another source. ## How core reaches a pipeline that lives in a package Core (`RecursiveAI.OneNlp`) knows nothing about `.Neural` or any model. It holds `ModelPackages`, a table from language to a registered factory (`Func`). A model package registers its factory; the `Nlp` verbs look the language up. The registration runs from a module initializer, and the module initializer lives in the **referencing project**, not in the package assembly: - The package ships a C# file under `contentFiles/cs/any/` (build action `Compile`). NuGet adds it to every C# project that references the package, directly or through another package. It contains one `[ModuleInitializer]` that calls `EnglishModel.Register()`. - Because the file is compiled into the application, the call is a normal, statically visible reference to the model assembly. Nothing has to load the assembly by name, scan assemblies or use reflection, and the trimmer and the NativeAOT compiler see the model code, its embedded weights and everything it calls as reachable. (A module initializer placed in the package assembly itself would not run in a JIT program that never touches a type from it, and could be trimmed away in one that does not reference it.) - `Register` is idempotent, so several projects (an application and a library, both referencing the package) may add the file. - A project that cannot compile the file (F#, Visual Basic) calls `EnglishModel.Register()` once at start-up. - Registration stores a factory only. The weights are read on first use, once, under a lock, and the resulting pipeline is shared by all callers (components are safe for concurrent `Process`). **Why this is AOT- and trim-safe:** there is no `Assembly.Load`, no `Type.GetType`, no `Activator`, no IL emission and no reflection-based discovery. The weights are `EmbeddedResource`s read with `Assembly.GetManifestResourceStream` of a type's own assembly, which NativeAOT supports; head kinds and components are created through explicit tables (`HeadRegistry`, `ComponentRegistry`). The consumer app in `tests/OneNlp.ModelConsumer` publishes with NativeAOT with no warnings and runs the verbs. ## The English pipeline One pipeline from one saved pipeline (`PipelineFile`), `english.onenlp`, assembled by `recipes/en/assemble` from three separately trained models, each with its own encoder: tagging (tagger for XPOS, morphologizer for UPOS and UFeats, edit-tree lemmatizer; `recipes/en/fast-tagging`), the dependency parser, which also predicts the sentence starts (`recipes/en/parser`), and the entity recognizer for PER, ORG and LOC (`recipes/en/ner`). The rule sentencizer saved with the tagging model is left out, because the parser sets the sentence boundaries. The assembled file predicts what each part predicted alone (the scores are in [results.md](https://recursiveai.org/onenlp/docs/results/)). A single shared encoder for all heads (`recipes/en/joint`, 23.7 MB) is much faster and has weaker rows; it is documented in [results.md](https://recursiveai.org/onenlp/docs/results/) as an option, and a user can load it with `PipelineFile.Load` like any saved pipeline. The tokenizer is the UAX #29 word tokenizer with English data, shared with the string table; the pipeline owns and disposes every component. ## Package layout and the weights ``` src/OneNlp.Models.English/ OneNlp.Models.English.csproj references OneNlp.Neural only EnglishModel.cs Register(), Load() contentFiles/...Registration.cs the file compiled into referencing projects weights.manifest.props name, size, sha256, recipe, data of each weights file PROVENANCE.md, THIRD-PARTY-NOTICES.md, PACKAGE.md ``` - Weights never enter git. Building or packing with the environment variable `ONENLP_ENGLISH_WEIGHTS` (or the property `EnglishWeightsDir`) set to a directory holding the files named in the manifest embeds them as resources `weights/`. - Every file is hashed (SHA-256) and compared with the manifest at build. A missing or different file fails the build with the file name and both hashes. Without the variable the project builds (for development) but `dotnet pack` fails, saying what to set. - **Re-pack with retrained weights:** put the new files in the directory, change the `Size` and `Sha256` of their rows in `weights.manifest.props`, update `PROVENANCE.md` (datasets, recipe, seed), and pack. No code changes. - The package contains: the assembly (with the weights embedded), the manifest, `PROVENANCE.md`, `THIRD-PARTY-NOTICES.md`, `PACKAGE.md`, both licences and the registration file. `tests/OneNlp.ModelConsumer/run.sh ` packs the core, neural and model packages into a throwaway folder feed, then builds, runs and NativeAOT-publishes a consumer app against that feed with and without the model package. ## Writing another model package Same shape: a project referencing `RecursiveAI.OneNlp.Neural`, a manifest with hashes, a loader that builds a `Pipeline` from its resources, a `Register` that calls `ModelPackages.Register(language, packageId, factory)`, and the contentFiles registration. The package for language X names itself `RecursiveAI.OneNlp.Models.`. ## Pipeline.Create, Add<T>() and tiers `Pipeline.Create(language, tier)` makes a pipeline from the model package referenced for the language, with the package's tokenizer and no components. `.Add()` then takes the package's trained component of type `T` (`Tagger`, `Morphologizer`, `Lemmatizer`, `DependencyParser`, `EntityRecognizer`, ...) and appends it; `.Add(component)` appends one you built (an `EntityRuler`). Components run in the order added. Components never added are released when the pipeline is disposed. The `Tagger` of the fast-tier packages sets fine-grained tags (`Token.Tag`); add the `Morphologizer` for `Pos` and morphology, which noun chunks need together with the parser. - `Tier.Fast`: the language's fast-tier package, else `RecursiveAI.OneNlp.Models.Multilingual`. - `Tier.Accurate`: `RecursiveAI.OneNlp.Models.Multilingual.Large` if referenced, else `RecursiveAI.OneNlp.Models.Multilingual`; never the fast model. - With nothing referenced the call throws `InvalidOperationException` naming the package to add. Nothing is downloaded and nothing is found by reflection: packages register from a module initializer. - Each `Create` reads the weights into a new copy owned by the pipeline; the `Nlp` verbs keep their own shared one. - `Doc.NounChunks` returns the base noun phrases of a parsed document as spans; it throws for a document without a parse. - `pipeline.Train(examples, epochs: 10)` needs only `using RecursiveAI.OneNlp;`. # Moving from Catalyst Catalyst is a managed spaCy-style pipeline for .NET. This page maps its calls to OneNLP's. It is written from Catalyst's published README and from running its released packages as a black box (versions 26.10.6514 and English models 26.9.5316); nothing here comes from its source. There are no compatibility shims: the calls differ, and the table says how. ## At a glance | | Catalyst | OneNLP | |---|---|---| | Models | `Catalyst.Models.English.Register()` and a storage folder; model packages per language | reference the model package (`RecursiveAI.OneNlp.Models.English`, ...); nothing to register, nothing downloaded | | Pipeline | `Pipeline.ForAsync(Language.English)` | `Nlp.Analyze(text)`, or `Pipeline.Blank(language).Add(...)`, or `PipelineFile.Load(path)` | | Text object | `new Document(text, Language.English)` | `Doc` made by the pipeline; it keeps your string | | Processing | `nlp.ProcessSingle(doc)` | `nlp.Process(text)`; `nlp.ProcessMany(texts)` for batches | | Sentences | `doc.Spans` | `doc.Sentences` (`Span` values) | | Tokens | `span.Tokens` | `foreach (Token t in doc)` | | Token text | `token.Value` (may be normalized: `'ve` as `have`) | `token.Text` (a string cut from your input; `token.TextSpan` is the copy-free view); `token.Norm` for a normalized form | | Offsets | `token.Begin`, `token.End` (the end is inclusive) | `token.Idx`, `token.EndIdx` (the end is exclusive); UTF-16 units into the string you passed | | Part of speech | `token.POS` (an enum) | `token.Pos` (a Universal Dependencies tag string such as `PROPN`) and `token.Tag` (the treebank's own tag) | | Lemma | `token.Lemma` | `token.Lemma` | | Entities | an extra model (`AveragePerceptronEntityRecognizer`) and `token.EntityTypes` | `doc.Ents` or `Nlp.Entities(text)`; `span.Label` is `PER`, `ORG`, `LOC` or `MISC` | | Dependency parse | none | `token.Dep`, `token.Head` | | Morphology | none | `token.Morph` | ## Before and after Catalyst, as its README shows it: ```csharp Catalyst.Models.English.Register(); Storage.Current = new DiskStorage("catalyst-models"); var nlp = await Pipeline.ForAsync(Language.English); var doc = new Document("Marie Curie won the Nobel Prize in Paris.", Language.English); nlp.ProcessSingle(doc); foreach (var span in doc.Spans) { foreach (var token in span.Tokens) { Console.WriteLine($"{token.Begin}..{token.End} {token.Value} {token.POS} {token.Lemma}"); } } ``` OneNLP (`samples/OneNlp.Samples/MigrationSamples.cs`, which builds and runs): ```csharp // Reference the model package, then call the verb. Nothing is registered by hand and nothing is downloaded. Doc doc = Nlp.Analyze("Marie Curie won the Nobel Prize in Paris."); foreach (Token token in doc) { Console.WriteLine($"{token.Idx} {token} {token.Pos} {token.Lemma}"); } foreach (Span entity in doc.Ents) { Console.WriteLine($"{entity.Label} {entity}"); } ``` ## Differences that matter when you port - **Tokenization** follows Universal Dependencies and UAX #29: contractions are split as treebanks split them (`don't` becomes `do` and `n't`), and in French, Spanish, Portuguese, Italian, German, Dutch and Turkish the contractions that are multiword tokens in the treebanks are cut into pieces the model tags. Expect different token counts than Catalyst's. - **Sentence splitting** is set by the parser when the model has one, otherwise by the rule sentencizer. - **Languages.** The Catalyst English models are the only managed comparison for English; the other eight fast-tier languages have no Catalyst counterpart here. - **Entity labels** are the UNER scheme (`PER`, `ORG`, `LOC`), plus `MISC` where the training data has it, not Catalyst's WikiNER types. The recognizers add `CARDINAL`, `ORDINAL`, `PERCENT`, `MONEY` and `QUANTITY` through `Nlp.Quantities` ([recognizers](https://recursiveai.org/onenlp/docs/recognizers/)). - **Training.** Catalyst trains averaged perceptrons; OneNLP trains neural heads with `Train` ([training.md](https://recursiveai.org/onenlp/docs/training/)). A model trained for Catalyst does not load here; retrain on the same data. - **Threading.** A `Pipeline` may be shared between threads for `Process`; documents are independent. How the two compare on accuracy and speed is in [results.md](https://recursiveai.org/onenlp/docs/results/) and [results-table.md](https://recursiveai.org/onenlp/docs/results-table/): the English UPOS and lemma rows (on raw text, 92.90 and 95.03 against Catalyst's 80.70 and 83.84 in its black-box run, `tools/oracles/Catalyst/README.md`) and the speed run, which is repeated at the release gate. # Moving from Microsoft.Recognizers.Text Microsoft.Recognizers.Text finds numbers, units, currency, sequences and date/time expressions and returns `ModelResult` values. OneNLP's recognizers do the same job with one engine and rules kept as data per language. This page maps the calls. It is written from the package's published usage and from running version 1.8.13 as a black box on our own gold sentences ([recognizer-gold.md](https://recursiveai.org/onenlp/docs/recognizer-gold/)); nothing comes from its source or its spec files. There is no shim. ## At a glance | | Microsoft.Recognizers.Text | OneNLP | |---|---|---| | Entry point | one static class per area: `NumberRecognizer`, `NumberWithUnitRecognizer`, `SequenceRecognizer`, `DateTimeRecognizer`; one method per model | `Recognizer.For(language).Recognize(text, context)` for everything; or the `Nlp` verbs | | Culture | a culture string, `Culture.English` (`"en-us"`), `"fr-fr"`, ... | a `Language`, `Recognizer.English`, `Recognizer.For(Language.French)`: English, French, German, Spanish, Portuguese, Italian, Dutch, Turkish, Hindi, Chinese | | Result | `ModelResult`: `Text`, `Start`, `End`, `TypeName`, `Resolution` (a dictionary of strings) | a typed record: `Quantity`, `SequenceMatch` or `TemporalMatch`, each with `Start`, `End`, `Text`, `Label`, `Confidence` | | End offset | **inclusive** (`End` is the last character) | **exclusive** (`End - Start == Text.Length`) | | Reference time | the `refTime` argument of `RecognizeDateTime` | `RecognitionContext(referenceDate)`; relative expressions without one are not reported | | Resolved values | strings in `Resolution` (`"value"`, `"unit"`, `"timex"`, `"start"`, `"end"`) | a `decimal Value`, `Unit`, `Dimension`, `Currency`; a `DateRange` with `DateTimeOffset` `Start` and `End`, a `Granularity` and an `Approximate` flag | ## Which call becomes which | Recognizers.Text call | OneNLP result | |---|---| | `NumberRecognizer.RecognizeNumber` | `Quantity` with `Kind == QuantityKind.Cardinal` (`Nlp.Numbers(text)`) | | `NumberRecognizer.RecognizeOrdinal` | `Quantity`, `Kind == Ordinal` | | `NumberRecognizer.RecognizePercentage` | `Quantity`, `Kind == Percent`; `Value` is in percent (5, not 0.05) | | `NumberWithUnitRecognizer.RecognizeCurrency` | `Quantity`, `Kind == Money`; `Currency` is the ISO 4217 code and `Value` is in that currency (50 cents is 0.5) | | `NumberWithUnitRecognizer.RecognizeDimension`, `RecognizeTemperature`, `RecognizeAge` | `Quantity`, `Kind == Measure`; `Dimension` is `length`, `mass`, `volume`, `area`, `speed`, `information`, `temperature` or `age`; `Unit` is the canonical English unit name in every language | | `SequenceRecognizer.RecognizeEmail`, `RecognizeURL`, `RecognizeIpAddress`, `RecognizePhoneNumber`, `RecognizeHashtag`, `RecognizeMention`, `RecognizeGUID` | `SequenceMatch` with `Kind` `Email`, `Url`, `IpAddress`, `PhoneLike`, `Hashtag`, `Mention`, `Uuid` (`Nlp.Sequences(text)`); phone-like is syntax only and unvalidated | | `DateTimeRecognizer.RecognizeDateTime` | `TemporalMatch` with `Kind` `Date`, `Time`, `DateTime`, `DateRange`, `TimeRange`, `DateTimeRange`, `Duration` or `Set` (`Nlp.Dates(text, reference)`, `Nlp.ParseDate(text, reference)`) | ## Before and after Recognizers.Text: ```csharp using Microsoft.Recognizers.Text; using Microsoft.Recognizers.Text.DateTime; using Microsoft.Recognizers.Text.NumberWithUnit; using Microsoft.Recognizers.Text.Sequence; var reference = new DateTime(2026, 3, 18, 10, 30, 0); string text = "Pay $20 to bob@example.com by 5 March 2026, or within two weeks."; var money = NumberWithUnitRecognizer.RecognizeCurrency(text, Culture.English); var email = SequenceRecognizer.RecognizeEmail(text, Culture.English); var dates = DateTimeRecognizer.RecognizeDateTime(text, Culture.English, DateTimeOptions.None, reference); foreach (var r in money.Concat(email).Concat(dates)) { Console.WriteLine($"{r.TypeName} [{r.Start},{r.End}] {r.Text}"); // End is inclusive } ``` OneNLP (`samples/OneNlp.Samples/MigrationSamples.cs`, which builds and runs): ```csharp Recognizer english = Recognizer.English; // Recognizer.For(Language.French), ... for the other nine DateTimeOffset reference = new(2026, 3, 18, 10, 30, 0, TimeSpan.Zero); foreach (Recognition r in english.Recognize("Pay $20 to bob@example.com by 5 March 2026, or within two weeks.", new RecognitionContext(reference))) { string detail = r switch { Quantity q => $"{q.Kind} {q.Value} {q.Unit ?? q.Currency}", // ModelResult.Resolution["value"], ["unit"] TemporalMatch t => $"{t.Kind} {t.Value} {t.Range?.Start:yyyy-MM-dd}", // Resolution["values"][..]["timex"], ["start"], ["end"] SequenceMatch s => $"{s.Kind}", _ => string.Empty, }; Console.WriteLine($"{r.Label,-10} [{r.Start},{r.End}) {r.Text} {detail}"); // TypeName, Start, End (inclusive in the old library) } ``` ## Differences that matter when you port - **Spans do not overlap.** One call returns every type in text order, and where two readings cover the same characters the engine keeps one ([recognizers.md](https://recursiveai.org/onenlp/docs/recognizers/), "Overlaps"). Recognizers.Text returns each model's results separately, so the same characters can appear as a number and as part of a date. - **Ranges end where they say.** A `DateRange` has an inclusive `Start` and an **exclusive** `End`: `from April 6 to April 10` ends at the start of April 11. Recognizers.Text states the end of such day ranges as the end day itself. Both conventions are named in [recognizer-gold.md](https://recursiveai.org/onenlp/docs/recognizer-gold/). - **Several candidates.** For an ambiguous expression Recognizers.Text lists several values (past and future Friday, am and pm). OneNLP resolves to one, by rules written in [recognizers.md](https://recursiveai.org/onenlp/docs/recognizers/) (a bare weekday is the first such day on or after the reference day; `next Friday` is Friday of next week). - **Time zones are explicit.** A stated zone (`PST`, `UTC+2`, a trailing `Z`) is kept in `Value`, `UtcOffset` and the offset of `Range`. There is no time zone database, so no daylight-saving arithmetic. - **Approximate expressions** (`spring 2021`, `early 2021`, `late 2023`) resolve to a range with a granularity and an `Approximate` flag. - **Region.** `RecognitionContext.Region` (`us` or `gb`) picks the order of numeric dates and the holidays. - **Rules are data.** The words, symbols, units, currencies, holidays and patterns of a language are rule files, not code; the grammar that combines them is code ([recognizers.md](https://recursiveai.org/onenlp/docs/recognizers/), "Rule data" says where the line falls). `Recognizer.FromFiles(...)` loads rules from your own files, so you can add a unit, a currency or a holiday without a new build. - **Values are exact.** `Quantity.Value` is a `decimal`; money does not round-trip through `double`. On our hand-labelled test sets, in each of eight languages, every type is within noise of or ahead of Recognizers.Text, and the engine is faster; the rows are in [results-table.md](https://recursiveai.org/onenlp/docs/results-table/) and [results.md](https://recursiveai.org/onenlp/docs/results/). # Moving from Lingua (SearchPioneer.Lingua) Lingua is a language detector for short and long text; SearchPioneer.Lingua is its .NET port. OneNLP's `LanguageDetector` (namespace `RecursiveAI.OneNlp.LanguageId`) does the same jobs. This page maps the calls. It is written from Lingua's README and from running `SearchPioneer.Lingua` 2.0.0 and `lingua-language-detector` 2.2.0 as black boxes ([language-id.md](https://recursiveai.org/onenlp/docs/language-id/)); nothing comes from any Lingua implementation's source. There is no shim. ## At a glance | | Lingua | OneNLP | |---|---|---| | Build a detector | `LanguageDetectorBuilder.FromAllLanguages().Build()`, then `WithPreloadedLanguageModels()` to load eagerly | `LanguageDetector.Default`: 75 languages in one 3.6 MB model embedded in the core package, loaded in about 10 ms (`results.md`) | | A subset of languages | `FromLanguages(English, French, German)` | `LanguageDetector.Default.Restrict([Language.English, Language.French, Language.German])` | | Accuracy mode | `WithLowAccuracyMode()` | none: one model, one mode | | Detect | `detector.DetectLanguageOf(text)`; `Language.Unknown` when it cannot tell | `detector.Detect(text)`; `Language.Unknown` when the text has no letter of a supported script | | Confidence values | `ComputeLanguageConfidenceValues(text)`, sorted | `detector.Confidences(text)`; probabilities sum to 1 | | Mixed text | `DetectMultipleLanguagesOf(text)`: results with a language, a word count and a character `Range` | `detector.DetectSpans(text, switchPenalty = 9)`: runs that tile the text from 0 to its length | | Minimum relative distance | `WithMinimumRelativeDistance(d)` | not needed: restrict the languages, or read `Confidences` | | Language names | an enum with a member per language | `Language` is a code: `Language.English`, ...; the others by `Language.FromCode("sv")` (two-letter codes of `tools/fetch/langid/languages.tsv`) | | Happy path | none | `Nlp.DetectLanguage(text)` | ## Before and after Lingua: ```csharp LanguageDetector detector = LanguageDetectorBuilder .FromLanguages(Language.English, Language.French, Language.German) .WithPreloadedLanguageModels() .Build(); Language language = detector.DetectLanguageOf("languages are awesome"); var confidences = detector.ComputeLanguageConfidenceValues("languages are awesome"); var runs = detector.DetectMultipleLanguagesOf("Parlez-vous français? Ich spreche Französisch nur ein bisschen."); ``` OneNLP (`samples/OneNlp.Samples/MigrationSamples.cs`, which builds and runs): ```csharp // LanguageDetectorBuilder.FromLanguages(...).Build() becomes Restrict(...) on the detector that ships with the library. LanguageDetector detector = LanguageDetector.Default.Restrict([Language.English, Language.French, Language.German]); Language language = detector.Detect("languages are awesome"); // DetectLanguageOf IReadOnlyList ranked = detector.Confidences("languages are awesome"); // ComputeLanguageConfidenceValues IReadOnlyList runs = detector.DetectSpans( // DetectMultipleLanguagesOf "Parlez-vous français? Ich spreche Französisch nur ein bisschen."); foreach (LanguageSpan run in runs) { Console.WriteLine($"{run.Language} {run.Start}..{run.End} {run.Confidence:F2}"); } ``` Or, with no setup, `Language language = Nlp.DetectLanguage(text);`. ## Differences that matter when you port - **Two `Language` types.** If a file uses both libraries, alias one (`using LinguaLanguage = Lingua.Language;`). OneNLP's is a small struct of the language code, so any language in the model can be named without an enum member. - **Languages.** OneNLP identifies 75 languages (the list is in `tools/fetch/langid/languages.tsv`); SearchPioneer.Lingua 2.0.0 has 79, and our comparison runs it restricted to the same 75. Restricting to your languages helps most on short text. - **Short text.** On single words, word pairs and sentences from the held-out Leipzig and Tatoeba text, OneNLP's accuracy is above Lingua's high-accuracy mode in every bucket; the hard cases are the closely related Latin-script pairs (Croatian and Bosnian, Indonesian and Malay, Norwegian Bokmål, Nynorsk and Danish), where some languages score below Lingua. The per-language rows are in [results-table.md](https://recursiveai.org/onenlp/docs/results-table/). - **Cost.** One model of 3.6 MB that loads in about 10 ms, about 1.4 KB allocated per text, and no per-language models to preload or accuracy modes to choose. The load, memory and throughput rows against Lingua are in [results.md](https://recursiveai.org/onenlp/docs/results/). - **Offsets** in `DetectSpans` are UTF-16 offsets into your string; a run begins at the first word of a new language, so spaces and punctuation after a run belong to it. - **Thread safety.** A detector is immutable and safe to share. # Moving from Microsoft.ML.Tokenizers Microsoft.ML.Tokenizers loads one tokenizer type per model family. OneNLP's `Tokenizer` loads Hugging Face `tokenizer.json`, SentencePiece `.model`, `vocab.txt` and `.tiktoken` files through one type and matches Hugging Face `tokenizers` and `tiktoken` byte for byte. This page maps the calls. It is written from Microsoft.ML.Tokenizers 2.0.1's published API and from running it as a black box in the speed comparison ([results.md](https://recursiveai.org/onenlp/docs/results/)); nothing comes from its source. There is no shim. ## At a glance | Microsoft.ML.Tokenizers | OneNLP | |---|---| | `BertTokenizer.Create(vocabStream, options)` (WordPiece; options for lower-casing and accent stripping) | `Tokenizer.FromFile("tokenizer.json")`; the normalizer settings come from the file. For a bare `vocab.txt`, say the casing: `Tokenizer.FromWordPieceVocabulary("vocab.txt", new WordPieceVocabularyOptions { LowerCase = true })`. `Tokenizer.FromFile("vocab.txt")` guesses it from the vocabulary | | `SentencePieceTokenizer.Create(stream, addBeginningOfSentence, addEndOfSentence)`; `LlamaTokenizer.Create` | `Tokenizer.FromFile("tokenizer.json")` or `Tokenizer.FromFile("tokenizer.model")`; the special tokens come from the post-processor of the `tokenizer.json` | | `TiktokenTokenizer.CreateForModel("gpt-4o")` (the ranks come with the `Microsoft.ML.Tokenizers.Data.*` packages) | `Tokenizer.FromFile("o200k_base.tiktoken")`: you supply the ranks file; the encoding is taken from the file name or rank count (`r50k_base`, `p50k_base`, `cl100k_base`, `o200k_base`) | | `tokenizer.EncodeToIds(text)` returns `IReadOnlyList` | `tokenizer.EncodeToIds(text)` returns an `int[]` (pass `addSpecialTokens: false` to match the old call, see below); `EncodeToIds(text, buffer, out written, out consumed)` writes into your buffer and returns an `OperationStatus` (`Done` or `DestinationTooSmall`: always check it, a short buffer cuts the text); `Encode(text)` returns a `TokenEncoding` with ids, tokens, offsets, the special-token mask and word ids | | `tokenizer.CountTokens(text)` (no special tokens) | `tokenizer.CountTokens(text, addSpecialTokens: false)`; without the flag `[CLS]` and `[SEP]` are counted too, so a short text counts two more | | `tokenizer.GetIndexByTokenCount(text, n, out normalized, out count)` and `GetIndexByTokenCountFromEnd` | `tokenizer.TrimToTokenBudget(text, n, TokenizerSide.Left or Right)` returns the slice of your text covered by the first or last *n* tokens | | `tokenizer.Decode(ids)` | `tokenizer.Decode(ids)`, or `Decode(ids, destination, out idsConsumed, out charsWritten)` into your buffer | | `EncodedToken` values with `Offset` | `TokenEncoding.Offsets` (`Start`, `End`) | ## Before and after Microsoft.ML.Tokenizers (the calls its README shows): ```csharp Tokenizer tokenizer = BertTokenizer.Create(File.OpenRead("vocab.txt")); IReadOnlyList ids = tokenizer.EncodeToIds("a text to encode"); int count = tokenizer.CountTokens("a text to encode"); string text = tokenizer.Decode(ids); int index = tokenizer.GetIndexByTokenCount("a text to encode", 2, out string? normalized, out int tokenCount); ``` OneNLP (`samples/OneNlp.Samples/MigrationSamples.cs`, which builds and runs; it was run on a BERT `tokenizer.json`): ```csharp // BertTokenizer.Create / TiktokenTokenizer.CreateForModel / SentencePieceTokenizer.Create become one type that reads the file's own format. Tokenizer tokenizer = Tokenizer.FromFile(tokenizerJson); int[] ids = tokenizer.EncodeToIds("a text to encode", addSpecialTokens: false); // EncodeToIds(text): a new array, nothing cut off int count = tokenizer.CountTokens("a text to encode", addSpecialTokens: false); // CountTokens(text) string text = tokenizer.Decode(ids); // Decode(ids) ReadOnlySpan head = tokenizer.TrimToTokenBudget("a text to encode", 2); // GetIndexByTokenCount // For a fixed buffer with no allocation, check the status: DestinationTooSmall means the buffer cut the text short. int[] buffer = new int[512]; OperationStatus status = tokenizer.EncodeToIds("a text to encode", buffer, out int written, out int consumed); ``` ## Differences that matter when you port - **Special tokens.** `EncodeToIds`, `Encode`, `CountTokens` add the model's special tokens by default (`addSpecialTokens: true`), as Hugging Face `tokenizers` does; pass `false` for none. Check counts against your old code with the flag set the same way. - **Equality with the Hugging Face reference.** On the parity corpus the ids, tokens, offsets and decoded text equal Hugging Face `tokenizers` and `tiktoken`. Where Microsoft.ML.Tokenizers differs from those references, ours follows them. The speed run lists the rows where the two libraries' outputs differ and so were not timed: WordPiece text with characters the comparator maps to its unknown token, long Unigram text, and the cleanup of spaces before punctuation when decoding WordPiece ([results.md](https://recursiveai.org/onenlp/docs/results/), "Rows not timed, and why"). - **Unigram numbering.** SentencePiece `.model` files and Hugging Face `tokenizer.json` files number the control tokens differently; ours follows the file it loads. If you compare ids with Microsoft.ML.Tokenizers on a SentencePiece model, map the numbering first (the same page explains the offset). - **Offsets** count UTF-16 units for `char` input and UTF-8 bytes for byte input; special tokens carry `(0, 0)`. - **Truncation and padding** are properties of the `Tokenizer` (loaded from `tokenizer.json`) and apply to `Encode` and `EncodeBatch`. - **Allocation.** The span forms (`EncodeToIds` into your buffer, `CountTokens`) allocate nothing for WordPiece, Unigram, SentencePiece-BPE and byte-level BPE inputs in the speed run; Tiktoken allocates a little (408 B for a short sentence, 1.3 KB for eight). - **Speed.** At or faster than Microsoft.ML.Tokenizers 2.0.1 on every timed row on x64 and arm64; load time is slower for WordPiece and SentencePiece-BPE (31.88 ms against 19.66 ms, and 48.44 ms against 10.91 ms on x64) and faster for Unigram and byte-level BPE; it is reported, not gated ([results.md](https://recursiveai.org/onenlp/docs/results/)). # Segmentation Normative behaviour of the word tokenizer and the sentence segmenter. Language-neutral except where a language's data is named. ## Word tokens 1. The text is cut into word segments by the default word boundary rules of Unicode UAX #29 section 4 (the Unicode version OneFont pins). No tailoring. 2. A segment made only of white space (`char.IsWhiteSpace`) is dropped. Every other character belongs to exactly one token; white space belongs to none. `Token.HasSpaceAfter` recovers it. 3. With `SegmentationData.None` the tokens are exactly those segments. The conformance tests run every line of `WordBreakTest.txt` and `SentenceBreakTest.txt` through the layer in this mode. 4. With language data, each segment may be adjusted. All adjustments are data in `SegmentationData`, looked up by exact (case-insensitive) text or by one-character class, never by a pattern language: - **URLs and e-mail addresses** stay whole: `http://`, `https://`, `ftp://` or `www.` up to white space (at most 2048 characters), without trailing sentence punctuation or an unbalanced closing bracket; `local@domain.tld` with an ASCII local part (at most 256 characters) and a domain of at most 255 characters whose last label is at least two letters. - **Abbreviations** keep their full stop: a listed word (`Mr.`, `Inc.`) or dotted single letters (`U.S.`, `e.g.`). - **Hyphens**: a hyphen between two letters or digits joins them when the word before it is a listed prefix (`e-mail`, `anti-war`) and, for any language that lists none, always. Digits joined by a hyphen (`303-832-8160`) always join. Other hyphenated words split at the hyphen, as UD English does. - **Digit joiners** (`:` `/`) between two digits do not split (`3:30`, `1/2`). - **Mark runs**: a run of the marks `. - ! ? * = _ ~ < > # $ + / \` is one token (`...`, `--`, `?!`, `$$$`); a full stop that closes a run of other marks is its own token. - **Emoticons** from a short list stay whole. - **Contractions** (English): `n't` and the suffixes `'s 're 've 'll 'd 'm` after a letters-or-digits stem split off (`do` `n't`, `it` `'s`); the apostrophe may be `'` or U+2019. Fused forms split at fixed lengths (`can` `not`, `gon` `na`). `o'clock` and `rock'n'roll` stay whole. - **Elision** (French): when the text before the first apostrophe is a listed word (`c d j l m n s t qu jusqu lorsqu puisqu quoiqu`), the elided word, apostrophe included, is a token and the rest is segmented again: `l'homme` is `l'` `homme`, `qu'est-ce` is `qu'` `est` `-ce`. Words whose prefix is not listed, and listed whole forms, stay whole (`aujourd'hui`, `quelqu'un`, `l'on`, `c'est-à-dire`). - **Hyphenated pronouns** (French): a token that ends in a hyphen and a listed pronoun splits at the hyphen, the hyphen going with the pronoun (`dis-moi` is `dis` `-moi`, `a-t-il` is `a` `-t-il`, `donne-le-lui` is `donne` `-le` `-lui`). `-ci` and `-là` split off except after `celui celle celles ceux jusque par`. Other hyphenated words, and listed whole forms such as `rendez-vous`, stay whole (`peut-être`, `lui-même`, `week-end`). - **Contractions that are multiword tokens** (French): `du au aux duquel auquel desquels desquelles auxquels auxquelles` are cut at fixed lengths into the pieces of the multiword token they are (`du` is `d` `u`); the pieces are what a model tags. `des` is cut the same way (`de` `s`) after a word that is not on a list of verbs, prepositions and conjunctions (`tools/fetch/fr/des_context.py` draws it from the training split) and not after punctuation, and is one token (the plural article) otherwise. `SegmentationData.TryMatchMultiword` gives the words a group of pieces stands for (`de` `le`), so that output can be written as a CoNLL-U multiword token. - **Numbers** (French): digits in groups of three separated by a space (`25 785`, `1 234 567`) are one token, which holds the space; a minus sign after white space and before a digit belongs to the number; `20h30` is `20` `h` `30`; a capital letter and its full stop before another capitalized word (`V. Prebeg`) is one token; `n°` is one token. 5. Offsets are UTF-16 offsets into the original string. Nothing is normalized. A lone surrogate is a one-character token. 6. Work is linear in the text. The look-ahead for a URL, e-mail local part or domain is capped as above, and there is no regular expression. ## Sentences The `sentencizer` component writes sentence starts from the sentence boundaries of UAX #29 section 5, snapped to tokens: a sentence starts at the first token that begins at or after a Unicode boundary. Token 0 always starts one. Data may list tokens after which no sentence starts (English `Mr.`, `Dr.`). `Pipeline.Add("sentencizer")` takes the data of the pipeline's language unless its settings name `data` or `language`; with an unknown language the boundaries are Unicode's. A full stop after an abbreviation that can also end a sentence (`the U.S. He said`) is left to Unicode's rules. It is rule-based: UAX #29 does not break before a lower-case word after a full stop, nor split web text that has no end punctuation, so it scores below a trained segmenter. ## Measurement `tools/eval/segmentation.sh` scores both against a UD split with `tools/OneNlp.Eval` (`segment` writes the CoNLL-U, `ud` scores it with the CoNLL 2018 metric definitions; the shared-task script can be run beside it as a cross-check), and against spaCy's rule `sentencizer` on `spacy.blank("en")`. The raw text is the `# text` lines joined by single spaces. The `mwt` variant writes the pieces of a split contraction as a multi-word token line, as English UD treebanks annotate them. The English data was tuned on the English-EWT development split only, the French data on the French-GSD development split only, and the Dutch data on the Dutch-Alpino development split only; each test split was scored once. ## German data (`SegmentationData.German`, name `german`) Written from the UD German guidelines and the German-GSD treebank; the choices below were made on its training and development files only. - **Preposition-article contractions** (`zum`, `zur`, `im`, `am`, `vom`, `beim`, `ins`, `ans`, `ums`, `aufs`, `übers`, `durchs`, `fürs`, `hinterm`, `überm`, `unterm`, `vorm`, any case) are cut into two pieces of nearly equal length, longer first (`zu`|`m`, `be`|`im`), which is how the CoNLL-U reader cuts the surface form of a multiword token whose words do not spell it (`docs/data-formats.md`). A consumer that needs UD words writes the words back (`zu` `dem`); the recipe's `predict` does, from a table of the treebank's words. - **Abbreviations** keep their full stop: the listed words, dotted single letters (`z.B.`, `u.a.`, `d.h.`) and single letters (`Johann S. Bach`, `z. B.`). The list holds the words GSD keeps joined to their stop more often than not (`Dr.`, `St.`, `ca.`, `etc.`, `Prof.`); `usw.` and `bzw.` are split there and so are not listed. - **Ordinals**: one to four digits and a full stop are one token (`3.`) when the next word is a month name, one of a few listed nouns (`Quartal`, `Jahrhundert`, `Weltkrieg`), or starts in lower case; otherwise the stop ends a sentence and is its own token (`im Jahr 2010.`). No sentence ends after an ordinal token. - **Hyphens** between letters split (`US-Soldaten` is `US` `-` `Soldaten`); between digits they stay (`1-2`). - **Marks**: a run of one repeated mark is one token (`...`, `--`, ` `` `, `''`), but `!` and `?` are always single tokens, as GSD writes them; mixed runs do not form (`.''` is a stop and a quote). `'s` after a stem splits off (`geht's`). - Sentences do not break after the listed titles and abbreviations (`Dr.`, `Prof.`, `St.`, `Nr.`, `z.B.`). - Dates and numbers with `.` or `,` between digits (`24.12.2020`, `1.234,56`) stay whole by UAX #29. Development split of UD German-GSD, raw text (`recipes/de/fast segment`): tokens 99.81 [99.69, 99.91], words 99.81, sentences 94.64. ## Spanish data (`SegmentationData.Spanish`, name `spanish`) Written from the UD Spanish guidelines and the AnCora treebank; the choices below were made on its training and development files only. - **Contractions** `del` (`de` + `el`) and `al` (`a` + `el`), any case, are cut into two pieces, which is how the CoNLL-U reader cuts a surface form its words do not spell (`de`|`l`, `a`|`l`; `docs/data-formats.md`). - **Verbs with enclitic pronouns** are multiword tokens in AnCora (`conseguirlo` = `conseguir` + `lo`, `situándose` = `situando` + `se`). A token is cut when it ends in one to three of `me te se lo la le nos les las los`, what is left is an infinitive (`-ar -er -ir`) or a gerund (`-ando -iendo -yendo`) once the accent written for the stress is removed, and it is not on a short list of words the training file writes whole although they look like a verb and a pronoun (`parte`, `fuerte`, `muerte`, `Carlos`; the 47 words that occur more often whole than split). The pieces are the words when they spell the surface form (`conseguir`|`lo`), otherwise nearly equal parts (`situá`|`ndose`), as the reader cuts them. `os` is not an enclitic here: `-ros` and `-tros` words are far more common than second-person-plural forms. Imperative and subjunctive forms (`dámelo`, `véase`: about 45 of 1,540 clitic tokens in the training file) are not recognized. `SegmentationData.TryMatchMultiword` gives the words (`encontrárnoslas` is `encontrar` `nos` `las`; `reírse` keeps its accent). - **Abbreviations** keep their full stop: single letters (`W.`, `A.`), `fig.`, `EE.UU.`, `Co.`, `kg.` and the usual titles (`Sr.`, `Dra.`, `Prof.`, `Avda.`, `núm.`). Newswire AnCora has few of them. Sentences do not break after the titles. - **Inverted `¿` and `¡`** are tokens of their own (UAX #29 already separates them). Numbers with `.` or `,` between digits (`2.000`, `3,5`) stay whole; hyphenated names (`Ruiz-Mateos`) stay whole, as in the neutral defaults; apostrophe names (`d'Hebron`) stay whole by UAX #29. Development split of UD Spanish-AnCora, raw text (`recipes/es/fast segment`): tokens 99.92 [99.88, 99.95], words 99.91 [99.87, 99.94], sentences 94.89 [93.87, 95.90] (rules alone; the parser sets sentences in the pipeline). What is left: team names joined by a slash (`Reynard/Honda`, 11 tokens), quotes glued to a word, and `km.24`. ## Italian data (`SegmentationData.Italian`, name `italian`) Written from the UD Italian guidelines; the tables and lists below were counted on the training and development files of UD Italian-ParlaMint and Italian-MarkIT only (`tools/fetch/it/mwt_counts.py`, `partitive_context.py`). - **Articulated prepositions** (`del dello della dell' dei degli delle`, likewise with `a`, `da`, `in`, `su`, `con`) are multiword tokens of the preposition and the article (`di` + `il`). They are cut into two pieces like the other contractions (`de`|`l`, `nell'` = `nel`|`l'`), the apostrophe also in its typographic form. `dei`, `delle` and `degli` are the partitive article, one word, after a verb or a preposition (`ci sono dei bambini`) and a contraction after a noun (`l'elenco dei bambini`): the words that precede the article as a single word in the training files are listed (`PartitiveContext`), punctuation and the start of a text count as before an article, and every other word counts as a noun. - **Elision:** an elided article, preposition or demonstrative keeps its apostrophe and the next word is a token of its own (`l'` `amico`, `un'` `altra`, `c'` `è`, `quest'` `anno`, `dell'` `altro`). The text of the Senate transcripts also sets the apostrophe before a space (`l' idea`): the apostrophe stays with the word then too. `po'` stays whole. - **Verbs with enclitic pronouns** are multiword tokens (`farlo` = `far` + `lo`, `vedersi` = `veder` + `si`, `essendoci` = `essendo` + `ci`, `accorgercene` = `accorger` + `ce` + `ne`, `affrontiamolo`, `credetemi`, `sviluppatasi`). A token is cut when it ends in a pronoun (`mi ti ci vi si lo la li le ne gli`, or `me te ce ve se` before `lo la li le ne`) and what is left is an infinitive without its final `e` (`-ar -er -ir -por`), a gerund, an imperative in `-iamo -ete -ate -ite`, or a participle before `si`. Plural nouns and adjectives end the same way (`diversi`, `parti`, `esterne`), so the pronouns that end such plurals (`mi ti ci vi ne`) need a stem of five (`-ar`) or six letters (`-er -ir`), a few short infinitives are listed (`farci`, `dirmi`), and a list of look-alikes taken from the training files stays whole. - **Abbreviations** keep their full stop: single letters (`J.`, `A.`), `etc.` and the usual titles (`Sig.`, `Dott.`, `Prof.`, `art.`, `pag.`). Sentences do not break after the titles. What is left on the development split (raw text, `recipes/it/fast segment`, rules alone): quotes glued to a word (`'x'`), `In` written for `nel` in the treebank, participles with a pronoun (`trasmessaci`), and partitive articles after a noun. ## Portuguese data (`SegmentationData.Portuguese`, name `portuguese`) Written from the UD Portuguese guidelines and the Portuguese-Bosque treebank; the lists below were counted on its training and development files only. - **Contractions** of a preposition with an article, pronoun or demonstrative (`do`, `da`, `no`, `na`, `ao`, `pelo`, `nesse`, `dele`, `daquilo`, `num`, any case; 78 forms) are cut into two pieces. A form is listed when the treebank writes it as a multiword token far more often than as a single word. The pieces are cut as the CoNLL-U reader cuts the surface form of a multiword token (`docs/data-formats.md`): the words when they spell it (`ao` is `a`|`o`), else nearly equal lengths, longer first (`do` is `d`|`o`, `pelo` is `pe`|`lo`, `nesse` is `nes`|`se`). A consumer that needs UD words writes them back (`de` `o`); the recipe's `predict` does, from the table `Contractions`. `nos` is listed although the pronoun `nos` also occurs (29 of 308 in the training file). - **`à`** (`a` + `a`) is one character for two words, so it stays one token; the recipe trains on it as one word and writes the two words back. - **Hyphenated words** stay whole (`quinta-feira`, `norte-americano`), except that a pronoun after a verb's hyphen (`se`, `me`, `te`, `lhe`, `lhes`, `nos`, `vos`, `o`, `a`, `os`, `as`, `lo`, `la`, `los`, `las`) is split off with its hyphen: `trata-se` is `trata` `-se`, `dá-lo` is `dá` `-lo`. Every single-hyphen form ending in one of these is a two-word multiword token in the treebank (counted on the training and development files). Forms with two hyphens (`dar-lhe-á`) stay whole. The recipe writes `-lo` back as the word `lo` or `o`, as the training file spells it. - **Currency**: `US$`, `R$` and `CR$` are one token (`DollarWords`); the amount is the next token. - **Abbreviations** keep their full stop: the listed words (`Dr.`, `Sra.`, `art.`, `pág.`, `etc.`, `Av.`, `Cia.`), dotted initials (`J.F.K.`) and single letters except `a`, `e` and `o`. Sentences do not break after the titles and abbreviations that precede a name or a noun. - **Marks**: a run of one repeated mark is one token (`...`, `--`); `!` and `?` are single tokens. Decimal commas and thousands points stay inside the number (`5,50`, `2.200,00`). Development split of UD Portuguese-Bosque, raw text (`recipes/pt/fast segment`, rule sentencizer): tokens 99.89 [99.83, 99.94], words 99.81 [99.74, 99.87], sentences 80.55 [78.27, 82.61]. ## Dutch data (`SegmentationData.Dutch`, name `dutch`) Written from the Dutch-Alpino treebank; the choices below were made on its training and development files only (`tools/fetch/nl/token_stats.py` prints the counts, `tools/fetch/nl/seg_diff.py` the differences against a gold file). - **Apostrophes**: a word with an apostrophe inside stays whole by UAX #29 (`zo'n`, `z'n`, `auto's`, `Jan's`). The short forms that start with an apostrophe (`'t`, `'s`, `'n`, `'m`, `'ns`) and years without their century (`'69`) are one token when the apostrophe follows white space, punctuation or the start of the text and the word after it is exactly one of these; any other apostrophe is a quotation mark of its own (`'ja'` is `'` `ja` `'`). A hyphen after such a form continues the token (`'s-avonds`). - **Hyphens** between letters or digits stay (`Zuid-Afrika`, `21-jarige`, `2-0`); a hyphen with white space around it is a token. - **Abbreviations** keep their full stop: the listed words (`mr.`, `drs.`, `prof.`, `pct.`, `mln.`, `enz.`, `bijv.`), dotted single letters (`o.a.`, `b.v.`, `t.o.v.`, `N.V.`) and single letters (initials: `J. Bakker`). Sentences do not break after the titles, dotted abbreviations and initials that name a person or place (`mr. Van Dam`, `J. Bakker`). - **Numbers**: a thousands point or decimal comma between digits stays inside the number by UAX #29 (`130.000`, `2,5`); `/` between digits does too (`1/2`); an amount in whole guilders with a trailing dash (`5,-`, `130.000,-`, `11.-`) is one token. - **Marks**: a run of one repeated mark is one token (`...`, `,,` and `''` as quotation marks), but `!` and `?` are always single tokens. - **Contractions**: none are cut. UD Dutch-Alpino has one multiword token (`nogeens`) in its training and development files, and the tokenizer leaves it whole. Development split of UD Dutch-Alpino, raw text (`recipes/nl/fast segment`): tokens 99.78 [99.65, 99.89], words 99.78, sentences 81.60 (the rule sentencizer alone; the parser sets sentence starts in the full pipeline). The remaining differences are a few company names with `&` (`AZ&PC`), hyphens left open before a coordination (`mond- en keelholte`) and `,,`. ## Turkish data (`SegmentationData.Turkish`, name `turkish`) Written from the tokenization of UD Turkish-BOUN; every choice below was made on its training and development files only (`tools/fetch/tr/`). - **Apostrophes** stay inside the token: a suffix written after a name or a number is part of it (`Ankara'da`, `Ali'nin`, `1882'den`). UAX #29 already keeps a letter, apostrophe, letter run together; between a digit and a letter it cuts, so the data joins that case. - **Hyphens** are always tokens of their own, between letters (`Türk-Arap`) and between digits (`1-2`), as BOUN writes them. - **A full stop that closes a quotation** is one token with the closing quote (`gel."`), the more common form in BOUN (91 against 42 separate). - **Ordinals**: one to four digits and a full stop are one token before a lower-case word or a month name (`3. Ocak`); otherwise the stop is its own token (`2001.`). - **Multiword tokens.** BOUN writes a copula or a suffix such as `-ki`, `-ken`, `-ydi` that is spelled onto its word as a word of its own: `vardır` is `var` + `dır`, `yaparken` is `yapar` + `ken`. Whether it is a word of its own depends on the stem and the treebank is not consistent about it, so no word list decides. The tokenizer reads a table of word endings (`src/OneNlp/Segmentation/Data/tr.mwt.tsv`, 119 rows, written by `tools/fetch/tr/mwt_table.py` from the training split): the last two to nine letters of a lower-cased token (Turkish case rules: `I` is `ı`, `İ` is `i`) map to the length of the final piece, the longest ending that matches decides, and a token that is not all letters is never cut. A row is kept when the ending occurs at least three times in the training split and at least 90% of those tokens are cut the same way (`tools/fetch/tr/mwt_sweep.py` chose the two thresholds on the development split by the Words F1 they give). On the development split the table makes 119 of the treebank's 266 cuts and 87 cuts the treebank does not make. A cut word is written back as one multiword token with its two words (`SegmentationData.TryMatchMultiword` finds the pieces). - **Learned splitter.** `SegmentationData.TurkishWhole` (name `turkish-whole`) is the same data without the ending table: a word is never cut. It is the inner tokenizer of `NeuralMultiwordSplitter`, which the packaged pipeline uses instead of the table. UD tokenization has two steps (Zeman et al. 2018, section 2): cut the text into surface tokens, then expand multiword tokens; the splitter does the second step by labelling characters. Each sentence (at most 40 tokens per window, cut after a lone `.`, `!`, `?`) is one document of units: one unit per character of a candidate token (3 to 48 letters, nothing else), one unit for any other token and one for each space. A tagger over a fast encoder with a bidirectional LSTM labels each unit `B` (a UD word starts here) or `I`; a `B` after the first character of a candidate token is a cut, so the pieces are slices of the surface token and every label sequence is a valid expansion. Because the LSTM reads the neighbouring words, the decision can depend on the sentence, which the table cannot. Trained on the BOUN training split with `Train` (`recipes/tr/fast`, `train-splitter`); the multiword tokens whose words do not spell the surface form are cut after the letters they share (`işçiydi` = `işçi` + `di`), and about 6% of them (a word replaced by another) are labelled as one word. On the development split the splitter makes 186 of the treebank's 266 cuts and 152 it does not make (the table: 120 and 87); the treebank's own choices are inconsistent, and the confidence of a cut does not separate the right ones from the wrong ones (`MinCutPercent` from 50 to 97 leaves the precision at 55 to 60%), so the default threshold stays. `TryMatch` finds the pieces again (touching, letters-only tokens). - No abbreviation list: only six tokens of the training split keep a full stop inside or after letters (`Dr.` twice), so every full stop after a word is a token of its own. ## Chinese: trained word segmentation (`NeuralWordSegmenter`) Chinese is written without spaces and UAX #29 gives one word per ideograph, so the words of the Chinese pipeline come from a trained model, not from rules, and not from `SegmentationData`. The segmentation is the one of the training treebank (UD Chinese-GSDSimp), which writes personal names as a surname token and a given-name token and cuts compounds finely. - **Model.** Character tagging (Xue 2003; Peng, Feng and McCallum 2004): each unit is labelled `B` (a word starts here) or `I` (it continues the word before it). Two labels instead of the usual four (B, I, E, S) mean that every label sequence is a segmentation, so the most likely label of each unit is the answer and there is nothing to decode. The tagger is the ordinary `Tagger` over a `FastEncoder` whose tokens are the units; the encoder's neighbour, word-pair and wider-context hash tables give each unit its surrounding characters and its bidirectional LSTM gives it the sentence. - **Units.** Each character that is not white space is a unit (a surrogate pair is one character), except that a run of Latin, Greek or Cyrillic letters, and a run of digits (a full stop or comma between two digits stays inside), is one unit, because the treebank keeps such runs whole. White space is never inside a word, and the first unit after it always begins a word. Gold word boundaries inside a unit cannot be represented; there are none in the training and development files. - **Long texts** are cut into pieces of at most 1,024 units, after a sentence-ending mark (`。`, `！`, `？`, `；`, `!`, `?`, `;`, a line feed) in the last quarter of the piece when there is one, so the cost stays linear. - **Where it plugs in.** The segmenter is the `IWordTokenizer` of the pipeline; a saved pipeline records it as the tokenizer kind `segmenter` (`docs/model-files.md`). Sentence boundaries come from the rule sentencizer's UAX #29 rules, which break after `。！？` and their closing quotes, or from the parser. - **Training.** Reference documents have the units as tokens and `B`/`I` in the tag column (`NeuralWordSegmenter.Units`, `NeuralWordSegmenter.Tags`); eight consecutive treebank sentences, joined by a space, make one document; the tagger trains with the pipeline's `Train` extension; the recipe is `recipes/zh/fast` (`train-segmenter`). - **Result.** UD Chinese-GSDSimp r2.18 test, raw text: tokens and words F1 89.62 [88.87, 90.42] (spaCy 3.8.18's default Chinese tokenizer, which cuts single characters: 39.43); sentences 99.40 [98.70, 99.90] with the parser. Published, for information: Stanza 92.14, UDPipe 2 90.13 (words), Trankit base 96.94 (older annotation of the same treebank) (`docs/references.md`). # Tokenizer parity OneNLP's tokenizers must equal the reference libraries byte for byte: Hugging Face `tokenizers` for the WordPiece, Unigram, SentencePiece-BPE and byte-level BPE families, and `tiktoken` for the Tiktoken encodings. One mismatch is a bug, never a tolerance. This page defines what is compared, and how to run the harness. The reference libraries are only ever run as installed packages through their documented Python APIs. ## What is compared For every record of the parity corpus the harness compares our results with a **golden** record written by the reference library. | Check | Our call | Compared with | |---|---|---| | `encode.special`, `encode.plain` | `Encode(char span, addSpecialTokens: true/false)` | `ids`, `tokens`, `offsets` (UTF-16), `special_mask`, `type_ids`, `word_ids` | | `encode.utf8.special`, `encode.utf8.plain` | `Encode(UTF-8 span, …)` | the same, with `offsets` in UTF-8 bytes of the caller's input | | `encode.trunc`, `encode.utf8.trunc` | `Encode` with the truncation and padding below | the same plus `attention_mask`, over the padded length | | `ids.char.*`, `ids.utf8.*` | `EncodeToIds` into an exact-size buffer, then into a buffer of half the size | `ids`, `status` (`Done`, then `DestinationTooSmall`), `written`, and `consumed` = end offset of the last written token | | `ids.*` (count) | `CountTokens` | number of ids | | `decode.skip`, `decode.keep` | `Decode(ids, skipSpecialTokens: true/false)` on the reference's ids | the reference's `decode` | | `pair.special`, `pair.trunc` | `Encode(first, second, …)` (BERT and RoBERTa families) | the same fields as a single text, offsets into each text | `Tokenizer.Truncation` and `Padding` are `null` for every check except the `trunc` ones. ### Offsets Offsets are into the caller's original input, after any normalisation the tokenizer applied internally. Hugging Face reports offsets as Python code-point indices. The oracle driver converts every offset to the unit of our API: **UTF-16 code units** for the char API and **UTF-8 bytes** for the UTF-8 API. A code point above U+FFFF counts 2 units and 1 to 4 bytes. A padding position has offsets `(0, 0)`. `tiktoken` reports no offsets, so they are derived from the exact bytes of each token: the token covers the code points its bytes touch, and a token that begins or ends inside a multi-byte character covers the whole character. The same rule applies to the byte-level BPE families of Hugging Face, which report the span of the whole character for a partial-character token. ### Invalid input Python strings cannot hold invalid UTF-8 or lone surrogates, so the oracle sees a **defined** replacement of the record: - **Invalid UTF-8** (UTF-8 API only): each *maximal subpart* of an ill-formed sequence becomes one U+FFFD, as in the Unicode Standard section 3.9, "U+FFFD Substitution of Maximal Subparts". This is also what .NET's `UTF8Encoding` does, and what `Tokenizer`'s UTF-8 methods do. The offsets of the oracle's tokens are mapped back to byte positions in the *original* bytes, so a U+FFFD that stands for three invalid bytes spans those three bytes. - **Lone surrogates** (char API only): each lone UTF-16 surrogate becomes one U+FFFD. It is one UTF-16 unit before and after, so offsets are unchanged. ### Special tokens - Hugging Face: `add_special_tokens` is the post-processor's flag. A special token that appears *in the text* is encoded as that token and gets `special_mask` 0 and its own word index (observed in the output of `tokenizers` 0.22.1; only the tokens the post-processor adds are marked special). - Tiktoken: `addSpecialTokens: true` means special-token text in the input is encoded as the special token (`allowed_special="all"`); `false` reads it as ordinary text (`encode_ordinary`). `special_mask` is 1 for ids in the encoding's special-token set. `decode` with `skipSpecialTokens: true` removes those ids first (tiktoken has no such flag). - Tiktoken has no tokens strings, word ids, pair encoding, truncation or padding, so those checks do not exist for it. ### Truncation and padding One configuration per family, alternating so both sides are exercised: truncate to 16 tokens **including** the special tokens, from the right and pad on the left to length 24 with id 0 and token ``, or the reverse; the family list order picks which. For pairs the strategy is `longest_first`, and the golden header records the configuration. Observed in the output of `tokenizers` 0.22.1, for a pair with budget *n* (max length minus the 3 special tokens): the shorter sequence keeps at most ⌊n/2⌋ tokens and the longer one the rest, and when the two are equal the second counts as the longer. ## Inputs The corpus is built deterministically (fixed seed) from two inputs and synthetic classes. | Class | Source | |---|---| | `udhr-full`, `udhr-para` | Universal Declaration of Human Rights in Unicode, every language: the whole text, and four sampled paragraphs. Unicode licence. | | `tatoeba` | Up to 200 sentences for every Tatoeba language with at least 200 (reservoir sample). Mostly CC BY 2.0 FR; fetched, never committed. | | `empty`, `single-char` | the empty string; ~200 single characters across scripts, controls and planes | | `cjk-run`, `hangul`, `emoji`, `combining`, `rtl-bidi` | CJK without spaces; jamo and syllables; ZWJ, skin tone, flag and keycap sequences; mark stacks; bidi controls | | `whitespace`, `control`, `numbers`, `url-code` | whitespace runs (tab, NBSP, U+3000, CRLF…); controls, noncharacters, tags, private use; digits in several scripts; URLs, paths, code | | `special-lookalike` | `[CLS]`, ``, `<|endoftext|>`, partial and doubled forms, alone and inside text | | `lone-surrogate` | char API only | | `invalid-utf8` | UTF-8 API only | | `long-line` | single lines of 1,000,000 characters: words, CJK, mixed scripts, one repeated character, whitespace, emoji | | `tatoeba-pair`, `synthetic-pair` | pairs for the pair checks | Every record has a class tag. Records longer than 100,000 code points are compared by **digest**: the golden stores, per field, the count and the SHA-256 of the values (int32 little-endian; strings as an int32 length plus UTF-8), and the diff tool computes the same. A digest mismatch names the field but not the position. ## Running it All scripts take directories as arguments. Python is standard-library only for fetching and building; the oracle driver runs in an environment that has the reference package installed (`pip install tokenizers`, `pip install tiktoken`). ```sh # 1. fetch: tokenizer files at pinned revisions (tools/fetch/tokenizer-models.lock.json), then the raw text python tools/fetch/fetch_tokenizers.py --out $MODELS python tools/fetch/fetch_parity_inputs.py --out $RAW # 2. corpus: deterministic; two builds give the same corpus_sha256 in manifest.json python tools/parity/build_corpus.py --udhr $RAW/udhr --tatoeba $RAW/sentences.csv --out $CORPUS # 3. goldens: one .golden.jsonl.gz each; the header records the oracle's self-reported version python tools/parity/oracle.py --families tools/parity/families.json --models $MODELS --corpus $CORPUS --out $GOLDEN \ --family bert-base-uncased --family gpt2 # omit --family for all; run tiktoken families with the tiktoken environment # 4. diff dotnet run --project tools/OneNlp.Parity -c Release -- --families tools/parity/families.json --models $MODELS \ --goldens $GOLDEN --corpus $CORPUS --out $RESULTS [--family ID]... [--load vocab|sentencepiece|…] ``` The diff tool loads each family with `Tokenizer.FromFile`. A family whose loader does not exist yet reports `not supported` and does not fail the run (`--require-supported` makes it fail). Per family it writes to `$RESULTS`: - `.summary.json`: records and pairs compared, comparisons made, mismatches by `check.field`, by input class, and by both, and the first few mismatches with details; - `.mismatch-tests.cs`: xunit tests built from the *shortest* failing inputs (one per distinct check and field before repeating), each stating the reference's value. They load the tokenizer through `ParityTokenizers.Load(familyId)`, which reads `ONENLP_TOKENIZER_MODELS` and skips when it is unset. Inputs are the corpus's own, not further shrunk: the reference is not available to the diff tool to produce expected values for a reduced input; - `summary.txt`: one line per family. The exit code is 0 when every family matched or has no loader, 1 on any mismatch or failed load, 2 on a usage error. A full run is mostly the tokenizers' own time; run it per family (`--family`) to shard, and under the heavy queue on shared hosts. `--load` picks another file of a family (`vocab` for `vocab.txt`, `sentencepiece` for the `.model` file, …), listed in `tools/parity/families.json`. Those files do not carry the post-processing of `tokenizer.json`, so they are compared only where the family's loader defines the same behaviour. ## Self-check `tests/OneNlp.Tests/Parity/` holds a toy character-level tokenizer (`toy/tokenizer.json`, and a C# class written from it), a small synthetic corpus and the golden file the reference library wrote for it. The tests show that the harness passes when the tokenizer matches (all classes, pairs, truncation, padding, digests) and reports each injected defect: wrong ids, offsets, special mask, decode, `consumed`, truncation side and the pair tie-break. To regenerate: ```sh python tools/parity/build_corpus.py --profile toy --out tests/OneNlp.Tests/Parity/toy/corpus python tools/parity/oracle.py --adhoc toy=tests/OneNlp.Tests/Parity/toy/tokenizer.json \ --corpus tests/OneNlp.Tests/Parity/toy/corpus --out tests/OneNlp.Tests/Parity/toy --digest-min-chars 10000 ``` ## Golden file format `.golden.jsonl.gz` is gzip (mtime 0, so equal input gives equal bytes) of JSON lines. Line 1 is `{"header": …}`: `format`, `family`, `oracle`, `oracle_version` (the library's self-reported `__version__`), `oracle_dist_version` (package metadata), `python`, `tokenizer_sha256`, `vocab_size`, `corpus_sha256`, `digest_min_chars`, `trunc_pad` (the configuration, or null), `has_pairs`, `has_word_ids`, `has_tokens`. Then one line per corpus record, in corpus order: `{"id", "cls", "special", "plain", "trunc", "dec", "oracle_text"?}`, where each encoding holds `ids`, `tok`, `o16`, `o8` (absent for records with no UTF-8 form), `sm`, `ty`, `wi`, and `am` (in `trunc`); `dec` holds `skip` and `keep`; `oracle_text` is the text the oracle saw when it differs from the corpus record. Pair lines follow: `{"pair": id, "special", "trunc"}`. ## Release run Full sweep over the golden files produced by the parity harness, for the release gate: 12 families, 47,981 corpus records each, 0 mismatches. Oracles as they report themselves: `tokenizers` 0.22.1, `tiktoken` 0.14.0. | Family | Comparisons | |---|---:| | bert-base-uncased | 3,451,629 | | bert-base-multilingual-cased | 3,451,661 | | xlm-roberta-base | 3,451,681 | | t5-small | 3,449,575 | | mmBERT-base | 3,449,935 | | TinyLlama | 3,449,935 | | gpt2 | 3,449,919 | | roberta-base | 3,452,041 | | Qwen2.5-0.5B | 3,449,919 | | ModernBERT-base | 3,449,935 | | cl100k_base | 2,395,778 | | o200k_base | 2,395,778 | UAX #29 (Unicode 17.0.0): `WordBreakTest.txt` 1944 lines and `SentenceBreakTest.txt` 512 lines, all pass. # BPE, byte-level BPE and Tiktoken Behaviour of the byte-pair families. Language-neutral and normative. Pinned references: Hugging Face `tokenizers` 0.22.1 and `tiktoken` 0.14.0. ## BPE model (`tokenizer.json` `"type": "BPE"`) - **Algorithm** (Sennrich, Haddow and Birch 2016, §3.2). A piece starts as one symbol per character (a surrogate pair is one character). The adjacent pair with the lowest merge rank is replaced by its merged symbol, leftmost first on equal ranks, until no adjacent pair has a rank. The result is the symbols' vocabulary IDs, with offsets into the piece. - **Merges** come as `"left right"` strings (exactly one space) or `["left","right"]` pairs. The rank is the index. A pair listed twice keeps its last rank. The merged token is `left + right`, where the right half first loses `continuing_subword_prefix` if it starts with it. A merge whose halves or joined token are not in the vocabulary makes the file invalid. - **Options:** `unk_token` (must be in the vocabulary), `continuing_subword_prefix` (added to every symbol after the first when looking a character up), `end_of_word_suffix` (added to the last symbol), `fuse_unk` (a run of unknown characters is one token), `byte_fallback` (an unknown character is spelled by its UTF-8 bytes as `<0xNN>` tokens when all of them exist, before the unknown token is tried), `ignore_merges` (a piece that is a whole vocabulary entry is one token), `dropout` (each merge is skipped with that probability; 0 or absent is the deterministic algorithm). With no `unk_token`, unknown characters are dropped. - **Offsets.** A symbol covers the characters it was built from. Byte-fallback tokens and fused unknowns cover the whole characters they stand for, so the bytes of one character share its range. - **Byte-level BPE** (GPT-2 and relatives) is this model over text that the ByteLevel pre-tokenizer has mapped to printable characters, one per byte. It needs nothing extra. - **Vocabulary limits.** IDs must be non-negative and no sparser than 4× the entry count plus 65,536. - **Cache.** Results for pieces up to 128 characters are cached (20,000 entries, then no more are added). Dropout disables the cache and is not reproducible between runs. - **SentencePiece BPE** (`.model` files of the BPE type): the merge for a pair is the piece the two join into, ranked by that piece's score (higher score, lower rank; equal scores share a rank, so the leftmost wins). `BpeModel.FromSentencePieces` builds the model from pieces, scores and a per-piece "ordinary piece" flag. ## Tiktoken - **Ranks file (`.tiktoken`).** One line per token: the token's bytes in standard padded base64, one space, the rank in decimal, then `\n` (a preceding `\r` is tolerated, the last newline may be missing). The rank is the token ID and the merge priority. All 256 single bytes must be present; token bytes and ranks are unique. Ranks may skip numbers (`p50k_base` has a gap at the position of its special token) but not by more than 4× the count plus 65,536. Limits: 64 MiB, 8 KiB per line, 4 million lines, rank at most 100 million. Violations throw `InvalidDataException`. - **Encoding.** Special tokens are cut out first when allowed; the rest is split by the encoding's pattern into pieces; each piece's UTF-8 bytes are looked up whole, else byte-pair merged by rank (lowest rank first, leftmost on ties). `Encode(text, addSpecialTokens: true)` is `encode(allowed_special="all")` and `false` is `encode_ordinary`. Text that no match of the pattern covers is dropped. - **Patterns and special tokens** are not in the file. They are those of the Python library's `Encoding` objects (`_pat_str`, `_special_tokens`, read from the running object) with possessive quantifiers written as atomic groups and a final `$` as `\z`, because the .NET engine has no possessive quantifiers and the engine tiktoken runs on reads `$` as the end of the text only. - `r50k_base`, `p50k_base`: `<|endoftext|>` 50256. - `cl100k_base`: `<|endoftext|>` 100257, `<|fim_prefix|>` 100258, `<|fim_middle|>` 100259, `<|fim_suffix|>` 100260, `<|endofprompt|>` 100276. - `o200k_base`: `<|endoftext|>` 199999, `<|endofprompt|>` 200018. - **Characters outside the BMP.** The .NET engine matches UTF-16 units, so the named patterns would split such a character in two and not see it as a letter. For the named encodings each such character is swapped for one BMP character of the same general category while matching, and the matches are mapped back. A pattern you supply yourself is matched against raw UTF-16 units. - **Offsets** are UTF-16 units (or bytes for UTF-8 input). A token that starts or ends inside a multi-byte character reports the whole character. The Python library reports start offsets in code points; the two agree below U+10000. - **Token strings.** The token's UTF-8 text when its bytes are valid UTF-8, otherwise each byte as `<0xNN>`. `DecodeBytes` gives exact bytes; `Decode` replaces invalid UTF-8 with U+FFFD. - **File names.** `Tokenizer.FromFile("x.tiktoken")` takes the encoding from the stem (`r50k_base`, `p50k_base`, `cl100k_base`, `o200k_base`), else from the rank count (50,256, 50,281, 100,256, 199,998); otherwise it throws and asks for `TiktokenTokenizer.Create` with a pattern. - **Long runs.** Pieces over 48 bytes merge with a heap, so a long run of one letter costs O(n log n). # WordPiece, WordLevel and Unigram models, and the `vocab.txt` and `.model` readers Normative behaviour of the three model families behind the `tokenizer.json` pipeline and of the two file readers that build a tokenizer without a `tokenizer.json`. Offsets are UTF-16 units of the piece the model is given. A piece is never empty. Where the published papers and docs leave a detail open, it was settled by observing the output of Hugging Face `tokenizers` 0.22.1 (its model objects, called through the documented Python API on crafted vocabularies); those details are marked *observed*. ## WordPiece Schuster and Nakajima 2012; Devlin et al. 2019 (appendix A.3). Settings: `unk_token` (default `[UNK]`), `continuing_subword_prefix` (default `##`), `max_input_chars_per_word` (default 100). - A word of more than `max_input_chars_per_word` *Unicode scalar values* (not UTF-16 units) is one unknown token covering the whole word. *Observed.* - Otherwise the word is cut greedily from the left: at each start the longest vocabulary entry that matches is taken, with the prefix added in front of every entry that does not start the word. Shortening goes one scalar value at a time, so a surrogate pair is never split. - If any start has no match, the whole word is one unknown token (the pieces found so far are dropped), with offsets `(0, word length)`. - The token value is the vocabulary entry, prefix included. When the unknown token is not in the vocabulary and a word needs it, tokenizing throws `InvalidDataException`. - Vocabulary size is the number of distinct entries. IDs need not be contiguous; an ID or entry that repeats keeps its last use. ## WordLevel The piece is looked up whole. Not found: the unknown token (default `[UNK]`) with offsets `(0, piece length)`, or `InvalidDataException` when the vocabulary has none. ## Unigram Kudo 2018 (subword regularization, section 2) and Kudo and Richardson 2018 (section 2.2): Viterbi search over the lattice of vocabulary pieces for the highest total score. Settings: `unk_id` (may be absent), `byte_fallback` (default false). The vocabulary is a list of `[piece, score]`; the ID is the list position; scores are 64-bit. - A position with no one-character piece gets a synthesized one-character (one scalar value) unknown node scored ten below the lowest score in the vocabulary. *Observed.* - Ties between segmentations go to the one found first: the earliest start, then the shortest piece. *Observed.* - Consecutive unknown nodes fuse into one token whose value is the text it covers and whose ID is `unk_id`. A vocabulary piece that has the unknown ID counts as unknown and fuses too. *Observed.* - With `byte_fallback`, a fused unknown run becomes one `<0xXX>` token per UTF-8 byte of the run, each reporting the whole run's range, when every byte has a piece; otherwise it stays one unknown token. A run that is a single vocabulary piece with the unknown ID is left as it is. *Observed.* - Without `unk_id`, a character that has no one-character piece is an error (`InvalidDataException`) unless some matched piece ends right after that character, even when the best path never needs the missing piece. *Observed* (checked against 40,000 random cases). - A repeated piece keeps its last ID for lookups and matching; vocabulary size is the list length. ## Untyped model objects Older `tokenizer.json` files leave out the model's `type`. It is inferred: `merges` present means BPE; otherwise a `vocab` list means Unigram and a `vocab` object means WordPiece. *Observed* in the files of bert-base-uncased, t5-small and xlm-roberta-base. ## `vocab.txt` reader One token per line; the ID is the line number. Read as the reference reads it (*observed*): lines end at `\n` only (not at `\r`, U+0085 or U+2028); trailing Unicode White_Space of a line is dropped (leading white space is kept); blank lines are empty tokens that keep their IDs; there is no byte-order-mark handling; a repeated token keeps its last ID; invalid UTF-8 is an error. The file must contain `[UNK]`. Files are limited to 64 MiB. The tokenizer is composed as the BERT model cards describe it: `BertNormalizer` (clean text, Chinese characters spaced, lowercasing and accent stripping when the vocabulary is uncased), `BertPreTokenizer`, WordPiece (`##`, 100), the `[CLS] A [SEP]` / `[CLS] A [SEP] B [SEP]` template when `[CLS]` and `[SEP]` exist, and the WordPiece decoder with cleanup. `[PAD]`, `[UNK]`, `[CLS]`, `[SEP]` and `[MASK]` that are in the vocabulary are special added tokens. A `vocab.txt` does not say whether the model lowercases. `Tokenizer.FromFile` guesses: a vocabulary with any token that has an uppercase letter *which has a lowercase form* (outside bracketed special tokens) is cased. Letters without a case pair do not count (`bert-base-uncased` contains `ℝ`, which lowercasing leaves unchanged). `Tokenizer.FromWordPieceVocabulary(path, new WordPieceVocabularyOptions { LowerCase = …, StripAccents = … })` states the casing instead (`StripAccents` null follows `LowerCase`); prefer it, or the model's `tokenizer.json`, whenever the model's configuration is known. ## SentencePiece `.model` reader The file is a protocol-buffer message; the reader follows the published schema (pieces with score and type; the trainer's model type, unknown ID and byte fallback; the normalizer's name, compiled table and whitespace flags) and skips other fields. Every length is checked against the data before anything is allocated. Limits: 128 MiB, 4 million pieces, 4096 bytes per piece. - Unigram models only. A BPE-type model throws `NotSupportedException` until the BPE model is available; word and character models are refused. Models that keep runs of spaces or do not escape spaces are refused. - The pipeline is the one the converted `tokenizer.json` files of such models have: the model's compiled table as the `Precompiled` normalizer, `WhitespaceSplit` then `Metaspace` (U+2581, prefix space per the model), the Unigram model, and the `Metaspace` decoder. - The lattice holds ordinary and user-defined pieces; control, unknown, unused and byte pieces are never matched in text (the byte pieces serve byte fallback). User-defined pieces are also added tokens (not special, not normalized), so they are never split. - No end-of-sequence token is added and control symbols written in the text are ordinary text, as in the SentencePiece library; a converted `tokenizer.json` differs there. # Rule matching How `Matcher`, `PhraseMatcher` and `EntityRuler` behave. The token-pattern vocabulary (attributes, operators, predicates) follows the one spaCy documents for its rule-based matching; where that description is silent, behaviour was settled by running spaCy 3.8.18 on probe inputs, and the cases are kept as a differential test (see Verification). ## Token patterns A pattern is a list of positions. A run of tokens matches when each position, in order, accepts the tokens assigned to it. A position is a set of attribute tests that must all hold (an empty set accepts any token) and an operator that says how many tokens it covers. ### Operators | Operator | Tokens covered | |---|---| | none | exactly one | | `!` | exactly one token that does **not** satisfy the tests | | `?` | zero or one | | `+` | one or more | | `*` | zero or more | | `{n}` `{n,m}` `{n,}` `{,m}` | exactly n; n to m; n or more; up to m | - Bounds are at most 1000, `m >= n`, `m >= 1`. A pattern may expand to at most 10,000 token slots and has at most 1000 positions. - `OP` must be left out for exactly one; an empty `OP` is an error. - Matching is exhaustive: every start and every end that the pattern can reach is a match, so `a+` over `a a a` gives six matches. A match covers at least one token; a pattern that could match nothing yields nothing at that point. - Patterns run as a non-deterministic automaton without backtracking over tokens, so time is linear in the document for a given pattern. ### Attributes | Name (any letter case in files) | Value | Meaning | |---|---|---| | `ORTH`, `TEXT` | string | the text as written | | `LOWER` | string | the text lower-cased with the invariant culture | | `NORM` | string | the document's norm column; the lower-cased text where none is set | | `LENGTH` | integer | number of Unicode code points | | `LEMMA`, `POS`, `TAG`, `DEP`, `MORPH` | string | the document's columns; empty where no component set them | | `ENT_TYPE` | string | label of the entity containing the token, empty outside entities | | `ENT_IOB` | `B` `I` `O` or empty (or 3 1 2 0) | position in an entity; empty when the document has no entity annotation | | `SHAPE` | string | word shape, below | | `IS_ALPHA` `IS_ASCII` `IS_DIGIT` `IS_LOWER` `IS_UPPER` `IS_TITLE` `IS_PUNCT` `IS_SPACE` `IS_CURRENCY` `IS_LEFT_PUNCT` `IS_RIGHT_PUNCT` `IS_QUOTE` `IS_BRACKET` | boolean | orthographic flags (`TokenText`) | | `LIKE_NUM` `LIKE_URL` `LIKE_EMAIL` | boolean | resemblance flags | | `IS_STOP` | boolean | lower-cased text is in `Matcher.StopWords` (empty by default) | | `IS_SENT_START`, `SENT_START` | boolean | the token starts a sentence; token 0 always does | | `SPACY` | boolean | the token is followed by white space | Not supported: extension attributes (`_`), `ENT_ID`, `ENT_KB_ID`, dependency patterns. ### Predicates A value may be an object of predicates instead of a plain value: - `REGEX` (string attributes): a .NET regular expression searched anywhere in the value. The linear-time engine runs it when the expression allows; look-around and back-references fall back to the backtracking engine with a one-second timeout, and a timed-out test counts as no match. - `IN`, `NOT_IN`: a list of strings (string attributes except `ENT_IOB`), or of integers (`LENGTH`). - `IS_SUBSET`, `IS_SUPERSET`, `INTERSECTS` (`MORPH` only): the token's features, split on `|`, compared as a set with the listed `Key=Value` features. - `==`, `!=`, `>=`, `<=`, `>`, `<` (`LENGTH` only): integer comparisons. A plain `MORPH` string must equal the whole feature string; the empty string never matches (use `NOT_IN` or `IS_SUBSET` to find tokens without features). ### Word shape Each letter becomes `X` (upper case) or `x`, each digit `d`, every other character stays, and a run of the same output character is cut after four. `Hello` is `Xxxxx`, `12-3456` is `dd-dddd`, `123456` is `dddd`. Letters, case and digits are Unicode properties of code points; title-case letters count as `x`. ### Several patterns, keys, greedy filtering, callbacks - Patterns are added under a key. A key with several patterns matches when any does; the same range found by two patterns of one key is reported once. Keys do not affect each other. - `GreedyMode.First` and `GreedyMode.Longest` thin one key's overlapping matches to a non-overlapping set. First prefers the earliest start, then the longer match; Longest prefers the longer match, then the earlier start. Adjacent matches do not overlap. - Results are ordered by end token, then start token, then the order keys were added. (spaCy's documentation does not define an order; the differential test compares sets.) - A callback registered for a key runs once per match of that key, after matching, with the whole result list. ## PhraseMatcher Compares one attribute (`Orth` by default; `Lower`, `Norm`, `Shape`, `Lemma`, `Pos`, `Tag`, `Dep`, `Morph`) between phrase and document tokens, with phrases stored in a trie. All occurrences are reported, overlapping and nested ones included, ordered like `Matcher` results. A phrase added under two keys is reported for each key. Phrases given as text are split by the matcher's word tokenizer, so use the one that tokenizes your documents. Attributes other than `Orth`, `Lower`, `Norm` and `Shape` need phrase documents carrying that annotation; an unannotated phrase token is an error. ## EntityRuler A pipeline component, registered as `entity_ruler`, with settings `overwrite_ents`, `phrase_matcher_attr` (`ORTH`, `LOWER`, `NORM`, `SHAPE`) and `patterns_path`. 1. Every token pattern and phrase is matched. 2. Matches are taken longest first; ties go to the earlier start, then to the rule added first. A match overlapping one already taken is dropped. 3. A match that overlaps an entity already in the document is dropped, unless `OverwriteEnts` is set, in which case it is kept and each existing entity it overlaps is removed. 4. The document's entities become the surviving existing entities plus the new ones, in text order. Rule entities have confidence 1. A rule's ID is not stored in the document; `FindEntities` returns it with each entity. When two rules with different labels or IDs match exactly the same tokens, the rule added first wins (spaCy's documentation does not say, and its choice is not stable enough to copy). ## Pattern files Token pattern: a JSON array of objects, e.g. `[{"LOWER": "hello"}, {"IS_PUNCT": true, "OP": "?"}, {"LOWER": {"IN": ["world", "there"]}}]`. Entity rules: a JSON array of objects or JSON Lines, each `{"label": "ORG", "pattern": "Acme Corp", "id": "acme"}` where `pattern` is a phrase string or a token pattern and `id` is optional. Other keys are ignored. `EntityRuler.SavePatterns` writes JSON Lines. Readers reject, with `PatternFormatException` and before allocating for them: nesting deeper than 16, more than 1000 positions in a pattern, more than 64 tests on a position, regular expressions longer than 4096 characters or with a counted repeat above 999, repeat bounds above 1000, unknown attributes or predicates, wrong value types, invalid UTF-8, truncated input, and (for streams) input over a byte limit (default 256 MiB). JSON Lines errors name the line. ## Differences from spaCy - **Regular expressions** use the .NET dialect and run over UTF-16 units: `.` and `{n}` count a character outside the Basic Multilingual Plane as two, and `\w` does not include numeric characters such as `½`. - **Lower-casing** uses simple per-character invariant mapping, so Python's context rules (final sigma, `İ`) do not apply. - **Lexical flags** follow the Unicode data of the .NET runtime, which can be newer than the Python build's. `LIKE_URL`, `LIKE_EMAIL` and the number-word list of `LIKE_NUM` are written from observed behaviour and agree on every probe string in the test set, but are not claimed identical for all inputs. - **Unset annotation:** spaCy raises when `LEMMA`, `POS`, `TAG` or `DEP` is tested on an unannotated document; here the value is the empty string. - **Stop words:** no list ships; set `Matcher.StopWords`. - **Order of results** is defined above; spaCy's is not documented. ## Verification `tests/OneNlp.Tests/Matching/` holds the generated documents, patterns, phrase lists and rule sets with spaCy 3.8.18's answers (`cases.json.gz`, `expected.json.gz`); `tools/oracles/spacy_matcher` regenerates them. Tokens come from the cases, so tokenization does not enter the comparison. The test compares match sets for every pattern, phrase list and rule set, and the lexical flags and shapes for every case string and, for the single-character properties, every code point below U+20000 that the Python build assigns. # Recognizers Behaviour of the recognizers for numbers, ordinals, percentages, measures, money, address-like sequences, and dates and times. Language-neutral where it can be; the English rules are the first data set. Rules are data: adding a language means writing rule files (see "Rule data"), not code. ## Results `Recognizer.Recognize(text)` returns the results in text order without overlaps. Each result has UTF-16 offsets `Start` and `End` into the string passed in, its exact `Text`, a `Label`, and a `Confidence` from 0 to 1. | Type | Label | Meaning | |---|---|---| | `Quantity`, `Kind = Cardinal` | `CARDINAL` | A plain number: `42`, `3.5 million`, `two hundred and five`, `a dozen`, `3/4`. | | `Quantity`, `Kind = Ordinal` | `ORDINAL` | `1st`, `twenty-third`. `Value` is the position. | | `Quantity`, `Kind = Percent` | `PERCENT` | `5%`, `fifty percent`. `Value` is in percent: 5, not 0.05. `Unit` and `Dimension` are `percent`. | | `Quantity`, `Kind = Money` | `MONEY` | `$5m`, `five bucks`, `USD 20`, `50 cents`. `Currency` is the ISO 4217 code. `Value` is in that currency: 50 cents is 0.5. | | `Quantity`, `Kind = Measure` | `QUANTITY` | A number with a unit: `5 km`, `20°C`, `3-year-old`. `Unit` is the canonical unit name, `Dimension` what it measures: `length`, `mass`, `volume`, `area`, `speed`, `information`, `temperature`, `age`. | | `SequenceMatch`, `Email` | `EMAIL` | | | `SequenceMatch`, `Url` | `URL` | | | `SequenceMatch`, `IpAddress` | `IP_ADDRESS` | IPv4 or IPv6. | | `SequenceMatch`, `PhoneLike` | `PHONE_LIKE` | **Phone-like sequences, unvalidated; use a dedicated contact-parsing library for validated, canonical phones.** | | `SequenceMatch`, `Hashtag` | `HASHTAG` | | | `SequenceMatch`, `Mention` | `MENTION` | | | `SequenceMatch`, `Uuid` | `GUID` | | | `TemporalMatch`, `Kind = Date` or `DateRange` | `DATE` | A day or a period of days: `March 5, 2026`, `next Friday`, `spring 2021`. | | `TemporalMatch`, `Kind = Time` or `TimeRange` | `TIME` | A time of day or a range of times of day: `3pm`, `from 9am to 5pm`. | | `TemporalMatch`, `Kind = DateTime` or `DateTimeRange` | `DATETIME` | A day and a time, or a part of a day: `tomorrow at 3pm`, `Saturday evening`. | | `TemporalMatch`, `Kind = Duration` | `DURATION` | An amount of time: `two weeks`, `1 year and 6 months`. | | `TemporalMatch`, `Kind = Set` | `SET` | A recurrence: `every Monday at 9am`, `daily`. | `CARDINAL`, `ORDINAL`, `PERCENT`, `MONEY` and `QUANTITY` are the names OntoNotes uses for numbers, and spaCy's English models use them too. A NER model that predicts the UNER types (`PER`, `ORG`, `LOC`) does not use these names, so recognizer entities and model entities coexist in one `Doc`. **`Value` is a `decimal`.** Money must be exact (`19.99` is `19.99`), and decimal holds 28 digits. Numbers beyond ±7.9e28 do not fit and are not reported (`one octillion`, 1e27, still fits). Values are normalized so that `2.50` and `2.5` are the same value. A `double` would have given a wider range and lost the cents. Values are not converted between units: `5 km` is 5 with unit `kilometer`. ## Component `RecognizerComponent` (registered as `recognizers`) writes the results into `Doc.Ents`. Entities cover whole tokens; a result that starts or ends inside a token (`5km` as one token) is widened to the tokens it touches. When two results touch the same token the earlier one stays. Existing entities (from a model) stay and win over overlapping results unless `overwrite_ents` is set, in which case the recognizer results win and the entities they overlap are removed. ## Overlaps Every rule proposes results independently. Overlaps are settled the same way every time: 1. the longest result wins; 2. of equal length, the type earlier in this order: `URL`, `EMAIL`, `MENTION`, `HASHTAG`, `GUID`, `IP_ADDRESS`, `DATE`/`TIME`/`DATETIME`/`DURATION`/`SET`, `PHONE_LIKE`, `MONEY`, `PERCENT`, `QUANTITY`, `ORDINAL`, `CARDINAL`; 3. then the earlier start, then the higher confidence. The date and time labels rank after `IP_ADDRESS` and before `PHONE_LIKE`, so `2031` is a year, not a cardinal, and `30 minutes` is a duration, not a measure. A result overlapping one already taken is dropped. So `$5 million` (money) beats `5 million` (a number), and `a@b.com` beats the `b` and `com` inside it. ## Numbers ### Digits - Integer and decimal numbers with `.` as the decimal separator and `,` as the group separator (set in the rule data). A group is exactly three digits after a first run of one to three digits: `1,234,567.89` is one number; `1,2345` is two; `12,34` is two. - Indian digit grouping is read too: a first run of one or two digits, then groups of two digits, then a last group of three (`1,00,000`, `12,34,567`, `1,23,45,678`). `1,23,45` has no three-digit group at the end, so it is three numbers. - Scientific notation: a mantissa followed by `e` or `E` and a signed exponent of up to three digits (`6.02e23`, `2E-3`), or by a times sign (`×`, `x`, `X`, `*`) and `10` with the exponent after `^` or `**` (`6.02 × 10^23`, `1.5×10^3`) or in superscript digits (`2.5×10³`). The exponent must not run into a letter or digit (`7e5x` is no number). A value that does not fit a `decimal`, or underflows to zero (`9.1×10⁻³¹`), is not reported. `3 x 10 apples` is two numbers: a times sign counts only with an exponent. - A leading decimal point is a number: `.5` is 0.5. - A dotted run of more than one decimal point (`1.2.3`, `25.12.2020`) is not a number: it is a version or a date. - A sign: `-5` is negative when the hyphen follows white space, an opening bracket, a quote, `:`, `;`, `=`, `<`, `>`, `~`, `,` or a currency sign, or starts the text. After a digit it is a range (`3-5` is 3 and 5). After a letter it is part of a name (`COVID-19`, `B-52`: no number). `−` (U+2212) is treated like `-`. `minus` and `negative` before a number also negate it. - A number touching letters is not a number unless the letters are an ordinal ending, a multiplier suffix, or the start of a unit: `x86`, `H2O`, `5abc` give no number. - A proper fraction in digits: `1/2`, `3/4` (numerator below denominator, denominator up to 1000). `3/4/2020` is three numbers. `1/2` is also a date to some readers; a date recognizer takes precedence by being longer or earlier in the type order. - Multipliers touching digits: `3.5k` is 3500. Others are only multipliers next to a currency (`$5m`, `5m dollars`), because `5m` is also five metres. - A scale word after digits multiplies them: `3.5 million`, `2 dozen`, `5 hundred`. ### Words - Units, teens, tens, `hundred`, and the scale words of the short scale (`thousand` to `octillion`). A group of hundreds is built by addition inside the group; scale words close a group and must descend (`one million two thousand` is fine, `one thousand million` is not read as one number). - Words may be joined by white space (at most two white-space characters, none a line break) or a hyphen touching both words. `and` joins `hundred` or a scale word to what follows: `one hundred and five`, `two thousand and ten`. - `a`/`an` is one before `hundred`, a scale word, `dozen` and a fraction word (`a hundred`, `a dozen`, `a half`). It is not a number elsewhere. - `twelve hundred` is 1200 (a group below 100 times a hundred). - `zero` stands alone. - Fractions: `three quarters` (0.75), `a half`, `two thirds`; a fraction word needs a numerator, so `half` alone is not a number. Mixed numbers: `two and three quarters`, `one and a half`. - Decimals: `three point one four` is 3.14: every word after `point` is one digit. A scale word may follow a decimal or a mixed number: `two and a half million`. - The longest well-formed prefix wins: `one two` is two numbers; `one hundred hundred` is 100 followed by an unread `hundred`. - A lone word that is also a pronoun or an idiom is not a number when a neighbour says so (`number.not-number`): `no one`, `the one`, `which one`, `one of them`, `one another`, `one's own`, and a word ordinal before `time` (`for the first time`, `for the second time`). `one apple` and `I have one` stay numbers; `one hundred and one` is untouched, since only a word standing alone is judged. ### Numerals repeated in brackets Drafted text repeats an amount: `sixty (60) days`, `five thousand dollars ($5,000)`, `fifty percent (50%)`, `ten (10) business days`. - A number written in words followed (one white-space character at most) by a bracketed number written in digits with the same value is one number: `sixty (60)` is a CARDINAL of 60, and it takes the unit that follows it (`sixty (60) days` is a duration of 60 days). - When an amount with a unit, a currency or a percent sign is followed by a bracketed result of the same kind and unit or currency (or by a plain number), and the values agree, the result grows to include the brackets (`five thousand dollars ($5,000)`). - When the values disagree (`sixty (90) days`, `five thousand dollars ($4,000)`), the words' value stands, its confidence is multiplied by 0.6, and nothing is merged: the bracketed number is reported by itself. - Only a number written in words starts this; `60 (sixty) days` is not merged. ### Ordinals - Digits with an ending that fits the number: `1st`, `2nd`, `3rd`, `4th`, `11th`, `12th`, `13th`, `21st`, `22nd`, and grouped integers (`1,000th`). A wrong ending (`21th`) is not an ordinal and not a number either. - Words: `first` to `nineteenth`, `twentieth` to `ninetieth`, `hundredth`, `thousandth`, ..., and compounds `twenty-third`, `one hundred and first`. - `second` is also a unit of time, so on its own right after a number it is not an ordinal (`one second` is the number one); after tens it is (`twenty second` is 22nd). ## Percent A number followed (at most one white-space character between) by `%` or a percent word: `5%`, `5 %`, `fifty percent`, `12.5 per cent`. ## Measures A number, then (touching it, after one white-space character, or after one hyphen touching both) a unit from the data: `5km`, `5 km`, `5-foot`, `two miles per hour`, `100 km/h`, `20°C`, `68 degrees Fahrenheit`. The longest unit form wins. A unit must not be followed by a letter or digit touching it (`5 kg2` is no unit). Unit forms are matched without regard to case unless the data marks them case-sensitive (`MB` is megabytes, `Mb` megabits, `m` metres). Short forms that are everyday words (`in`, `a`, `at`, `min`) are not units. **Compound measures.** Two measures of related units written one after the other are one: `5 feet 3 inches`, `5 feet and 3 inches`, `6 pounds, 4 ounces`, `3 miles 200 yards`. The pairs and their ratios are data (`unit.compound`: `foot | inch | 12`). The result is in the smaller unit (`5 feet 3 inches` is 63 inches), the first number must be a whole number and the second below the ratio, so `5 feet 15 inches` stays two measures. **Age** is a number and a time unit followed by `old`: `5 years old`, `a 3-year-old boy` (Unit `year`, Dimension `age`). A bare duration (`5 years`) is not recognized here. ## Money - A currency sign or code before the number, touching it or after one white-space character: `$5`, `$ 5`, `USD 5`, `€20`, `US$5`; a leading minus before the sign: `-$5`; a minus after it: `$-5`. - A sign, code or name after the number: `20€`, `100 USD`, `5 dollars`, `five bucks`, `50 cents`, `10 zł`. - Currency names may have several words (`new zealand dollars`). A name may carry a factor: `cents` is 0.01 of its currency. - Codes are the alphabetic codes of ISO 4217 in capitals. Lower-case `usd` is not read because many codes are English words (`ALL`, `TRY`, `TOP`). - Ambiguous signs get the common reading and a confidence below 1: `$` is USD (0.9), `¥` is JPY (0.7), `dollars` is USD (0.9). Signs that pin down a currency (`US$`, `A$`, `CA$`, `€`, `£`) have confidence 1. - `pounds` is a unit of mass; `pound sterling` and `pounds sterling` are currency. A name that is both (`currency.context`) after a number is money, at confidence 0.8, when a money word (`fine`, `cost`, `paid`, `price`, ...) stands in the clause within five words before the number or four after the name and no weight word (`weighs`, `lost`, `of`, ...) does; otherwise it is the unit. The window stops at the clause end and at another number. So `The fine was 1,000 pounds` is GBP, `He weighs 180 pounds` and `10 pounds of flour` are mass. - The alphabetic codes were checked on 2026-11-05 against List One of ISO 4217 as published by SIX, the maintenance agency (dated 2026-09-17, 178 codes). All are present except `XTS` (reserved for testing) and `XXX` (no currency). `ANG` and `BGN`, no longer on List One, are kept because older text uses them. - A multiplier touching the number counts next to a currency: `$5m` is 5,000,000; `$1.2 billion`; `5m dollars`. ## Sequences The shape of each sequence is fixed by a standard, so it lives in code; what varies by deployment (URL schemes, domain endings) is rule data. ### E-mail A simplified addr-spec of RFC 5322 §3.4.1: a dot-atom local part of at most 64 characters (RFC 5321 §4.5.3.1.1) (no quoted strings, comments or folding), `@`, and a domain of dot-separated labels (up to 63 characters each, 253 in all, RFC 1035 §2.3.1), with at least two labels, the last a known domain ending (or an `xn--` ending). Non-ASCII letters and digits are accepted in both parts (RFC 6531). The local part starts at a letter, digit or `_`, and ends before the `@` without a dot; two dots in a row end the match. Trailing dots and hyphens after the domain are not part of it. `Normalized` lower-cases the domain. ### URL RFC 3986 §3 (`scheme://[userinfo@]host[:port]/path?query#fragment`) with the WHATWG URL Standard's leniency for addresses written without a scheme: - With `scheme://`: the scheme must be one listed in the rule data (so `note://abc` in prose is left alone). The host is a registered name, an IPv4 address, or a bracketed IPv6 address; the port is 1 to 5 digits; the authority may be empty (`file:///tmp/x`). - Starting with `www.`: the host needs at least three labels and a known domain ending. - Bare (`example.com/page`): the last host label must be in the shorter bare list of the rule data (`com`, `org`, `uk`, ...; not `md`, `py`, `sh`, `it`, which are also file extensions and words), so that `README.md`, `node.js` and `e.g.` are not URLs. - Path, query and fragment run over RFC 3986 characters and non-ASCII letters, digits and symbols, up to 2048 characters in all. - Trailing `! ' * , . : ; ?` are dropped (RFC 3986 Appendix C), and a closing `)` or `]` is dropped unless its opening bracket is inside the match. - An address is not resolved or checked to exist. ### IP address - IPv4 (RFC 791 dotted decimal): four decimal parts 0–255 without leading zeros (`010.1.1.1` and `256.1.1.1` are not addresses), not part of a longer dotted run (`1.2.3.4.5`). - IPv6 (RFC 4291 §2.2): eight 16-bit hex groups, one `::`, an IPv4 tail allowed. `Normalized` is the RFC 5952 §4 form (lower case, no leading zeros, the longest run of at least two zero groups compressed, the first on a tie). Trailing sentence punctuation is dropped. `::` alone, text where a hex word touches letters (`std::vector`) and runs with too many groups are not addresses. Zone identifiers (`%eth0`) are not read. ### Phone-like Syntax only, nothing verified, confidence 0.7. `Normalized` is an optional `+` and the digits. Accepted shapes (digits in all 7 to 15; E.164 caps a number at 15): - `+` then a country code of one to three digits and more groups: `+44 20 7183 8750`, `+1 (555) 123-4567`; or `+` and 8–15 digits together; - groups with a parenthesized group: `(415) 555-0132` (9 or more digits); - groups joined by one hyphen or dot, three to five groups of two to four digits (a leading group of one digit is allowed with four or more groups: `1-800-555-0199`): `555-123-4567`, `01.23.45.67.89`; or exactly `ddd-dddd`; - space-separated groups only when they start with `0` (national prefix): `020 7183 8750`. Not accepted: dates (`2020-01-15`, `15.01.2020`), decimals (`3.14159265`), plain digit runs, groups of digits separated by spaces without a prefix (`4111 1111 1111 1111`), anything touching letters. ### Hashtag, mention, GUID - `#` and one or more letters, digits or `_` (Unicode letters and marks included), not touching a preceding letter, digit, `_`, `&`, `#` or `/`; all-digit tags (`#2024`) are numbers, not hashtags; at most 100 characters. `Normalized` is the tag lower-cased without the `#`. - `@` and a user name of letters, digits and `_`, with `.` and `-` inside, not touching a preceding letter, digit, `_`, `@`, `+` or `.`; an `@host.example` after the name is part of a fediverse handle. `Normalized` is the name lower-cased without the `@`. - GUID (RFC 9562 §4): 8-4-4-4-12 hexadecimal digits, optionally in braces or after `urn:uuid:`. Version and variant bits are not checked. Not touching a letter, digit or hyphen. `Normalized` is 36 lower-case characters. ## Dates and times `Recognizer.Recognize(text, new RecognitionContext(reference) { Region = "us" })` resolves dates and times to values. `Nlp.Dates(text, reference)` lists them, and `Nlp.ParseDate(text, reference)` returns the first as a `DateRange`. The English rules are `Data/en/datetime.rules` (vocabulary) and `Data/en/holidays.rules` (holiday table); the grammar that combines the words is code. ### Result `TemporalMatch` has the `Kind` (below), a `Value` string, a `Granularity` (null for durations), an `Approximate` flag, the `UtcOffset` the text names, and a `Range` (`DateRange`: `Start` inclusive, `End` exclusive, `Granularity`, `Approximate`) on the calendar. A day runs from its midnight to the next; an instant such as `3pm` runs for its precision (an hour; `3:05pm` a minute). A time of day with no date is placed on the reference day; with no reference date its `Range` is null. Durations and sets have no `Range`; a fixed-length duration has `Amount` as a `TimeSpan`. A duration also has `Count` and `Unit` (a `DurationUnit`): `60 days` is 60 `Day`, `18 months` is 18 `Month`, `ten business days` is 10 `BusinessDay`. One term keeps its unit as written, with a quarter as three months, a decade as ten years and a fortnight as two weeks, and `1.5 hours` is 1.5 `Hour`. Several terms are counted in the smallest unit when they are all clock and day units or all month and year units (`2 hours and 30 minutes` is 150 `Minute`, `1 year and 6 months` is 18 `Month`); `1 year and 3 days` cannot be added, so its `Unit` and `Count` are null, as they are for every other kind. | Kind | Examples | `Value` | |---|---|---| | `Date` | `March 5, 2026`, `tomorrow`, `the 4th of July`, `Easter 2027`, `in 3 days`, `a week ago` | `2026-03-05` | | `Time` | `3pm`, `7:05 AM`, `15:45`, `noon`, `5 in the afternoon`, `3.30pm`, `9am JST` | `T15:00`, `T09:00+09:00`; `Z` for UTC and GMT | | `DateTime` | `tomorrow at 3pm`, `3pm on March 22`, `in 45 minutes`, `2026-06-30T09:15:00Z` | `2026-03-19T15:00` | | `DateRange` | `next week`, `last winter`, `the first quarter of 2026`, `late 2023`, `the 1990s`, `from April 6 to April 10`, `May 4-8`, `1990-2000`, `the past three days` | `2026-03-23/2026-03-30` | | `TimeRange` | `9am-5pm`, `from 6 pm until 9 pm`, `between 12 and 1 pm` | `T09:00/T17:00` | | `DateTimeRange` | `tomorrow morning`, `Saturday evening`, `Monday from 9am to 11am`, `from Monday 8am to Monday 6pm` | `2026-03-23T09:00/2026-03-23T11:00` | | `Duration` | `two weeks`, `45 seconds`, `an hour and a half`, `2 hours and 30 minutes`, `sixty (60) days`, `ten business days` | seconds as an integer (`1209600`); ISO 8601 `P1Y6M` when it names months or years; `P10BD` for business days and `PT8BH` for business hours (not ISO 8601; they have no length in seconds) | | `Set` | `every Monday`, `every Monday at 9am`, `daily`, `every 3 hours`, `every other day`, `every day at noon` | TIMEX3: `XXXX-WXX-1`, `XXXX-WXX-1T09:00`, `P1D`, `PT3H`, `P2D`, `T12:00` | `Granularity` is the unit the expression is stated in: for an instant its precision (`Second`, `Minute`, `Hour`, `Day`); for a range the kind of period (`Week`, `Month`, `Quarter`, `Half`, `Season`, `Year`, `Decade`, or `Day` for explicit day ranges and weekends, `Hour` or `Minute` for ranges of times); for a set the recurrence unit. This is the gold-set convention (`docs/recognizer-gold.md`). ### Reference date, zones and region - **Relative expressions need a reference date** (`today`, `next Friday`, `in 3 days`, `Christmas` with no year, `the 25th`, `March 5` with no year). Without one (`RecognitionContext.ReferenceDate` null) they are not reported; absolute ones are (`March 5, 2027`, `3pm`, `two weeks`, `every Monday`). Wall-clock expressions are read in the reference's offset. - **Zones are explicit.** `PST`, `UTC+2`, `GMT-05:00` and a trailing `Z` or `+hh:mm` after a time are kept in `Value` and `UtcOffset` and become the offset of `Range`. The recognizer carries no time zone database: a named zone is a fixed offset, so there is no daylight-saving arithmetic (`PST` is always -08:00; `PDT` is -07:00). A reference date in `America/New_York` is just its `DateTimeOffset`. The abbreviation table is `time.zone`; ambiguous abbreviations (`CST`, `BST`, `IST`) take the reading English text usually means, and `IST` is left out. - **Region** (`RecognitionContext.Region`, `us` default or `gb`) picks the holiday rows and the order of numeric dates: `04/05/2026` is April 5 in `us` and 4 May in `gb`; a first number above 12 and a dotted date are always day first. ### How expressions resolve - **Weekdays.** A bare weekday or `this` weekday is the first such day on or after the reference day (today counts). `next Friday` is Friday of the next week (weeks run Monday to Sunday). `last Friday` is the latest one before today. `Wednesday of next week` is that day in that week. `Monday, March 23` is the date; the weekday is not checked. - **Missing year.** A month and day, a holiday or `the 25th` is the first such day on or after the reference day, so a date that has passed this year is next year's. - **Ranges are half-open.** The end of a range of days or longer periods is the start of the period after the stated end (`from April 6 to April 10` ends at `2026-04-11`; `through 2027` at `2028-01-01`). A clock end point is the instant stated (`from 3pm to 5pm` is `T15:00/T17:00`). A second weekday or month-day counts from the first (`from Monday to Friday` is the week of the Monday it names; `December 28 to January 3` crosses the year). - **Times of day.** `3pm` and `3:30 pm` have am/pm; `15:45` and `09:00` are 24-hour; `H:MM` with no am/pm is read as a clock time; a bare hour (`at 9`) is not a time, because it has no am/pm. In a range the first end takes the second's am/pm (`between 12 and 1 pm`, `from 9 to 5 pm`), moving to am when that would put it after the end (`between 11 and 1 pm`). `5 in the afternoon` and `10 at night` use the day part. `noon` and `midnight` are `12:00` and `00:00`. - **Parts of a day** are approximate ranges: morning 06:00-12:00, afternoon 12:00-18:00, evening 18:00-24:00, night 20:00-24:00. `tonight` is the night of today, `last night` the night of yesterday, `this morning` the morning of today. - **Periods.** Weeks run Monday to Sunday; `this weekend` is Saturday and Sunday of the current week and `next weekend` those of the next week. Seasons are meteorological, northern hemisphere (spring March-May, summer June-August, fall September-November, winter December-February), approximate, and `winter 2024` starts in December 2024. `early`, `mid` and `late` split a year into January-April, May-August and September-December and a month into days 1-10, 11-20 and 21-end, approximate. `H1`/`Q3` and `the second half of ...` give halves and quarters; with no year they are in the reference year. `last winter` is the latest season that has ended; `this summer` the one that contains today or comes next. - **Offsets.** `in 3 days`, `3 days ago`, `a year from now`, `the day after tomorrow`, `two days before Christmas`, `the third Thursday of April`, `the first Monday after Easter 2027`. Units of a day or more give a date; hours, minutes and seconds give a date-time at that precision. - **Durations** are an amount and a unit (`two weeks`, `a day`, `half an hour`, `1.5 hours`, `an hour and a half`); several terms join with `and` or a comma when each is smaller than the one before (`2 hours and 30 minutes`; `seven days a week` is not one duration). `a second` is a duration of one second only when what follows cannot be the thing that is second: the end of the text, punctuation, or a word listed as `secondfollow` (`wait a second, please`, `a second ago`, `give me a second to think`); otherwise it is an ordinal (`a second opinion`). `the second floor`, `second place` and `the second half of 2026` are ordinals and a half-year, as before. - **Business and calendar units.** `business day(s)`, `working day(s)`, `work day(s)`, `workday(s)` and `business hour(s)` are units with no length in seconds: a duration of business days stands alone (it does not add to other terms) and its `Value` is `P10BD` or `PT8BH`. `calendar day(s)`, `week(s)`, `month(s)` and `year(s)` are the plain units. `in 3 business days` and `5 business days ago` are dates, counting Monday to Friday and skipping Saturdays and Sundays; public holidays are not considered. Counting from a weekend day starts from the weekday on the other side of the weekend (Saturday plus one business day is Monday). - **Sets** are `daily`, `weekly`, `monthly`, `quarterly`, `yearly`, `every`/`each` with a weekday or an optional count and a unit, and `every other`. With a time (`every Monday at 9am`, `daily at noon`) the value carries it. ### Holidays The table is `Data/en/holidays.rules`: forms | rule | regions (`us`, `gb`, `all`). Rules are `fixed MM-DD`, `nth N|last WEEKDAY MONTH [+-DAYS]`, `easter +-DAYS` and `nearest WEEKDAY MM-DD`. Easter is the Western (Gregorian) date from the "Anonymous Gregorian" computus (Meeus, *Astronomical Algorithms* ch. 8, after Butcher 1876), valid for every Gregorian year; the other moveable feasts count from it (Good Friday is -2, Ash Wednesday -46, Pentecost +49). The dates are the calendar days of the observances, not the weekday a holiday is observed in lieu of a weekend. | Region | Holidays | |---|---| | `us` | the United States federal holidays of 5 U.S.C. 6103 (New Year's Day, Martin Luther King Jr. Day, Washington's Birthday/Presidents' Day, Memorial Day, Juneteenth, Independence Day, Labor Day, Columbus Day, Veterans Day, Thanksgiving, Christmas), Election Day, Tax Day, Flag Day, Groundhog Day, Mother's Day (second Sunday of May) | | `gb` | the bank holidays of England and Wales as published at gov.uk/bank-holidays (New Year's Day, Good Friday, Easter Monday, Early May, Spring and Summer bank holidays, Christmas Day, Boxing Day), Mothering Sunday (three weeks before Easter, also "Mother's Day"), Remembrance Sunday, Bonfire Night, St David's, St George's and St Andrew's days, Burns Night, Whit Monday | | `all` | Valentine's Day, St Patrick's Day, April Fools' Day, Halloween, Thanksgiving, Black Friday, Cyber Monday, Christmas Eve and Day, Boxing Day, New Year's Eve, Easter and its feasts, Father's Day, Pi Day, May Day, Cinco de Mayo, Earth Day, Epiphany, Armistice Day, Leap Day | Holidays that follow a lunar calendar (Eid, Diwali, Passover, Hanukkah, Ramadan, Lunar New Year) are not in the table: they need a calendar table per year, which is data a later set of rules can add. ### Rule data for dates and times | Section | Fields | Meaning | |---|---|---| | `time.month` | `1-12 \| forms` | Month names and abbreviations. Forms starting `=` are case-sensitive, which keeps `May` and `March` from matching the verbs | | `time.weekday` | `1-7 \| forms [\| plural forms]` | Monday is 1; plurals make a weekly set | | `time.unit` | `name \| forms \| seconds \| months` | `day \| day, days \| 86400 \| 0` | | `time.season` | `name \| forms \| first month \| months` | | | `time.daypart` | `name \| forms \| start hour \| end hour` | End 24 is midnight | | `time.meridiem` | `am or pm \| forms` | `a.m.`, `p.m.` included | | `time.word` | `role \| forms` | The words the grammar uses: `today`, `next`, `ago`, `from`, `rangeto` (to, until, `-`), `early`, `approx`, `zonebase`, ... The roles are listed in `datetime.rules` | | `time.recurrence` | `TIMEX3 period \| unit \| forms` | `P1W \| week \| weekly` | | `time.zone` | `forms \| offset minutes [\| z]` | `z` writes the zone as `Z` | | `time.holiday` | `forms \| rule \| regions` | See above | A malformed temporal row throws `RecognizerRulesException` with the line, like the others. ### Limits - English only. Day-first order for slashes is by region and by the numbers (a first number above 12), not by learned usage. - No time zone database, no daylight saving, no calendars other than Gregorian, and no lunar holidays. - No fuzzy or vague quantities (`a couple of days`, `a few weeks`), no `end of the month`, `the weekend of March 21`, `Friday the 13th`, `every second Tuesday`, `twice a week`, or fiscal periods. - A bare weekday is the next one, even in past-tense text (`said on Tuesday` resolves forward); a bare month or season (`in March`, `in summer`) and `M/D` without a year are not reported, because they need context to mean one thing. - A bare four-digit number from 1000 to 2999 is read as a year (`2031`, `1969`) at confidence 0.8, so a count such as `2000 people` is also reported as a year; a measure or money amount of the same length wins when the number has a unit. - `in 45 minutes` is a time ahead, also in text such as `ran 5 miles in 45 minutes`. ## Rule data Rule files are UTF-8 text. `#` starts a comment. `[section]` starts a section; each other line is a rule whose fields are separated by `|`. A section may appear in several files; a language is a set of files (`Recognizers/Data//*.rules` embedded in the assembly, or files given to `Recognizer.FromFiles`). In a list of forms, forms are separated by commas, and may have several words. A form starting with `=` is matched case-sensitively; others are matched without regard to case. Forms are cut into words, digit runs and single symbols by the same lexer as the text, so `km/h` in the data is three items touching each other, and matches `km/h` but not `km / h`. | Section | Fields | Meaning | |---|---|---| | `number.digits` | `key \| value` | `decimal`, `group` (one character each), `negative` (characters read as a minus) | | `number.cardinal` | `word \| kind \| value` | `kind` is `unit` (0–9), `teen` (10–19), `tens`, `scale` (100 and up; 100 is "hundred") or `multiplier` ("dozen") | | `number.ordinal` | `word \| value [\| not-after-number]` | An ordinal word; the flag keeps `second` out right after a number | | `number.fraction` | `word \| denominator` | `half`, `thirds`, ... | | `number.article`, `number.connector`, `number.negative`, `number.point` | `word` | `a`/`an`; `and`; `minus`; `point` | | `number.suffix` | `form \| multiplier \| any or money` | `k`, `m`, `bn` touching digits | | `number.ordinal-suffix` | `ending \| last digits \| any, teen or not-teen` | `st \| 1 \| not-teen`: `1st`, not `11st` | | `percent` | `forms` | `%`, `percent`, `per cent` | | `unit` | `canonical \| dimension \| forms [\| confidence]` | | | `unit.compound` | `larger \| smaller \| ratio` | `foot \| inch \| 12`: `5 feet 3 inches` is one length | | `number.not-number` | `word \| before or after \| words` | Words that, next to the lone `word`, show it is no number: `one \| before \| no, the, which` | | `currency.context` | `forms \| ISO code \| value of one \| money words \| weight words` | `pounds` is money or mass by its neighbours | | `age.unit`, `age.suffix` | `canonical \| forms`; `forms` | Combined as `5 years old` and `5-year-old` | | `currency.symbol` | `form \| ISO code \| prefix, suffix or both [\| confidence]` | | | `currency.name` | `forms \| ISO code [\| factor [\| confidence]]` | | | `currency.code` | `codes` | Alphabetic ISO 4217 codes, in capitals, separated by commas or spaces | | `url.scheme`, `url.tld`, `url.tld.bare` | `items` | Schemes read with `://`; all domain endings; the endings allowed for bare addresses | Limits, checked while loading: 8 MB per file, 200,000 lines, 8192 characters per line, 8 fields per rule, 16 words or symbols per form. A malformed file throws `RecognizerRulesException` naming the line: invalid UTF-8, a rule before any section, an unknown section, a wrong field count, a number that does not parse, an unknown kind or position. Words and symbols (not numbers or dates) are what the rules recognize; the sequence syntaxes and the arithmetic of the number grammar are code. A language whose numbers are not built from additive and multiplicative words in the short scale (French `quatre-vingts`, German `einundzwanzig`) needs engine support in addition to data; that is a known limit, and a language of the same shape as English needs data only. ## Limits and decisions - Compound measures cover pairs in the data only (two terms); `1 yard 2 feet 3 inches` gives the first pair and a separate `3 inches`. - A number whose exponent leaves the range of a `decimal` (`9.1×10⁻³¹`) is not reported. - Separators are per language: English takes `.` as the decimal point and `,` as the group separator; French and German read `18.000 EUR` as eighteen thousand and `3,5` as three and a half. - Business-day arithmetic ignores public holidays. - Number words glued to digits (`5 hundred thousand`) read the first scale word only. - Phone-like detection will miss shapes outside the list above and will accept some number groups that are not phones: it is a guess, with confidence 0.7. - The engine takes a `RecognitionContext` with a reference date and region; the number, unit, currency and sequence rules ignore it, the date and time rules use it. ## French and German French (`fr-FR`) and German (`de-DE`) are data sets on the same engine: `Data//{numbers,units,currency,datetime,holidays}.rules`, with the language-neutral sequence rules in `Data/shared/`. Use `Recognizer.For(Language.French)`, or `Nlp.Quantities`, `Numbers`, `Sequences`, `Dates` and `ParseDate` with a `Language` argument. English results are unchanged. ### Numbers - `number.digits` also takes `system` (`english`, `french`, `german`) and `ordinal-dot` (`yes` makes `3.` an ordinal). `group` takes several separators by name or character: `space nbsp nnbsp .` for French (space, no-break space, narrow no-break space, dot), `.` for German. A group separator counts only between groups of exactly three digits, so `3.5` stays a decimal in English and `2 400` is 2400 in French. - **French** counting is the `french` system: `quatre-vingt-dix-sept` (four twenties and seventeen), `soixante et onze`, `vingt et un`, `septante`, `huitante`, `octante` and `nonante` as plain tens, hundreds with a plural `cents`, `mille` without a count. Cardinals of kind `bare-scale` (`cent`, `mille`, `hundert`, `tausend`) may stand for one of themselves. - **German** counting is the `german` system: a whole number below a million is one word. The word is cut into the parts listed in `number.cardinal` (units, teens, tens, `hundert`, `tausend`) and the connector `und`, in the order units before tens (`dreihundertvierundzwanzig`). Ordinals are stems (`number.ordinal` rows whose word ends in `~`, as `dritt~`) plus the endings of `number.ordinal-ending`; `einundzwanzigste` is 21. `number.ordinal-dot-word` lists capitalized nouns that make the dot after a number an ordinal (`5. Stock`); a month name or a lower-case word does it already. - `number.lone-skip` lists words that count only inside a longer number (`un`, `ein`), so the article is not a number but `vingt et un` is. - `number.ordinal-suffix` rows serve French endings (`1er`, `1re`, `2e`, `2nd`). - `age.prefix` (French `âgé de`) puts words before the number into the span of an age; `age.word` (German `jährig`, `4-jähriges`) is an age unit that needs no `alt`. ### Dates and times Dates are day first wherever `time.region` is not `us`. The extra word roles of `time.word`: | Role | Meaning | |---|---| | `determiner` | an article that belongs to the expression when it starts it: `le 3 mars`, `la semaine prochaine` | | `agoprefix` | the word before an amount that means "ago": `il y a`, `vor` | | `dayaftertomorrow`, `daybeforeyesterday` | `après-demain`, `übermorgen`, `avant-hier`, `vorgestern` | | `nextpost`, `lastpost` | `next` and `last` after the noun: `samedi prochain`, `mardi dernier` | | `hourmark`, `hourword` | the mark of the 24-hour clock: `15h30`, `15 Uhr` (strong), and `heures` (weak: also a length of time, a time only after an at-word or `clockcue`, with minutes or a part of the day, or with a clock tail) | | `clockcue` | a word before an hour that makes it a time: `il est huit heures` | | `plus15`, `plus30`, `minus15`, `minus` | the tail of an hour: `et quart`, `et demie`, `moins le quart`, `moins dix` | | `halfbefore`, `plusquarter`, `minusquarter` | words before an hour: `halb drei` (2:30), `Viertel nach drei`, `Viertel vor drei` | | `andhalf` | `et demi` after a unit: `trois jours et demi` | | `daypartprefix` | words between an hour and a part of the day: `huit heures du soir` | | `decadeprefix`, `decadesuffix`, `decadeword` | `les années 1990`, `1990er Jahre` | `half` also covers `demi-heure` and `eine halbe Stunde` as 30 minutes. `time.flag` switches on behaviours: `daypart-direct` (German `acht Uhr abends`), `no-lone-day` (`la 2e place` is an ordinal, not the next 2nd), `bare-hour-24` (`von 16 bis 18:30 Uhr`), `determiner-inside-only` (German: a determiner stays out of the span except in `Dienstag, den 24. März`). `time.region` names the default region (`fr`, `de`). Holidays: `fr` is the French public holidays of the Code du travail L3133-1 (1 January, Easter Monday, 1 May, 8 May, Ascension, Whit Monday, 14 July, 15 August, 1 November, 11 November, 25 December) with Easter itself and the customary feasts; `de` is the nationwide German holidays (New Year, Good Friday, Easter Monday, 1 May, Ascension, Whit Monday, 3 October, Christmas Day and Boxing Day) with the customary and regional feasts (Rosenmontag, Fronleichnam, Allerheiligen, Reformationstag) listed without regional limits. ### Limits (French and German) - Written-out numbers glued to an age word (`sechsjährig`) and `zweieinhalb`, `anderthalb`, `une quinzaine` are not read. - `midi`, `Mittag`, `minuit` and `Mitternacht` are not times on their own; the gold sets do not label them. - `ce mois-ci` is not read. - Belgian and Swiss counting (`septante`, `nonante`, `huitante`) is read; Swiss `Franken` are francs. - The French and German unit tables are shorter than the English one. ## Spanish and Portuguese Spanish (`es`, Spain by default) and Portuguese (`pt`, Brazil by default) are data sets on the same engine: `Data//{numbers,units,currency,datetime}.rules`, the holidays inside `datetime.rules`, and the shared sequence rules. Use `Recognizer.For(Language.Spanish)` or the `Nlp` recognizer verbs with a `Language` argument. English, French and German results are unchanged. ### Numbers - `number.digits` takes `system | iberian` and per-region separators: `decimal.mx | .` and `group.mx | space nbsp nnbsp ,` replace the defaults (`decimal | ,`, `group | space nbsp nnbsp .`) when `RecognitionContext.Region` is `mx`, so `1,234.56` is read in Mexican Spanish and `1.234,56` in Spain's. - **Iberian counting.** Tens above twenty join a unit with the connector (`treinta y cinco`, `trinta e cinco`; Spanish `veintiuno` to `veintinueve` are teens of their own). Hundreds are words of kind `hundreds` (`doscientos`, `quinhentos`, `cien`, `ciento`, `cento`) that open a group; `mil` is a `bare-scale`. In Portuguese the connector also links a hundred to what follows (`cento e vinte`) and a thousand to its rest (`dois mil e quinhentos`). `mil millones` as one scale word is not read; write `7.900 millones`. - **Ordinals.** An ordinal tens word may be followed by an ordinal unit (`vigésimo tercero`, `décimo quinto`, `décima segunda`); one-word forms (`decimoquinto`) are rows. `number.ordinal-suffix` rows may start with a dot: `.º`, `.ª`, `.er` read `1.º`, `3.ª`, `1.er`. `segundo` after `por` (`por segundo`) and before an object pronoun (Portuguese `segundo ele`) is not an ordinal, and `primera hora` is not one (`number.not-number`). - `age.suffix` makes `82 años de edad` / `82 anos de idade` an age; `82 años` alone is a length of time. - `phone.space-start | digits | min | max` lets a phone-like number be written as space-separated groups when it starts with one of the digits and has that many in all (`913 45 67 89` in Spain: `6789 | 9 | 9`). Without the section only a leading 0 and 9 to 11 digits count. ### Dates and times Dates are day first. New data for `time.word` and `time.flag`: | Item | Meaning | |---|---| | `time.flag of-before-year` | the `of` word may stand between a month and its year or a holiday and its year: `12 de septiembre de 2026`, `abril de 1987`, `Navidad de 2026`; and between the last day and the month of a day range: `del 12 al 20 de agosto` | | `hourarticle` | the article or fused preposition and article that opens a clock time and is part of it: `las 15:00`, `la una`, `às 14:30`. With it (or a `determiner`) a spoken or digit hour is a time when a part of the day or a tail follows (`las diez de la mañana`, `las tres y media`, `às oito da noite`), or when it is above twelve (`a las 15`) | | `decadeprefix` | also reads two digits: `los años 80` is 1980-1990, `anos 20` is 2020-2030 (30 to 90 are the 1900s, 00 to 20 the 2000s) | | `daypartprefix` | also joins a date and a part of the day when it is one word: `amanhã à noite` | `noche` and `noite` are 18:00 to 24:00, `tarde` 12:00 to 18:00, `mañana` and `manhã` 06:00 to 12:00 (`mañana` alone is tomorrow). Regions for `time.region` and `RecognitionContext.Region`: `es` (default) and `mx` for Spanish, `br` (default) and `pt` for Portuguese; the region also picks the holiday rows. Holiday sources are statute facts: Spain's national holidays (Estatuto de los Trabajadores art. 37.2 and the yearly BOE list), Mexico's mandatory rest days (Ley Federal del Trabajo art. 74; the Monday holidays are `nth` rules), Brazil's (Leis 662/1949, 10.607/2002, 14.759/2023) and Portugal's (Código do Trabalho art. 234), with the Easter-based feasts (`Carnaval` -48, `Corpus Christi` +60) from the computus. ### Limits (Spanish and Portuguese) - Weekdays in Portuguese are the full `segunda-feira` forms and the dotted abbreviations; a bare `quarta` or `sexta` is not a weekday. - `mil millones`, `una docena y media` and compound measures past two terms are not read. - The seasons are the northern-hemisphere meteorological ones, as for English. ## Italian and Dutch Italian (`it`) and Dutch (`nl`) are data sets on the same engine: `Data//{numbers,units,currency,datetime}.rules`, the holidays inside `datetime.rules`, and the shared sequence rules. Use `Recognizer.For(Language.Italian)` or the `Nlp` recognizer verbs with a `Language` argument. The other languages' results are unchanged. ### Numbers - **`number.digits | system`** takes `italian` and `dutch` besides the earlier names. `dutch` is an alias of `german` (one word below a million, units before tens: `vierentwintig`, `tweeduizend vijfhonderd`). `italian` is one word too, but the tens come before the units (`ventitré`, `trecentoventiquattro`). - **`[number.part]`** lists the elided stems that only occur inside a compound: `vent` (twenty, in `ventuno`, `ventotto`), `cent` (in `centotto`), `trent`, `quarant`, and so on, each with its kind and value. A stem is never read alone. - A compound word may follow a round thousand (`tweeduizend vijfhonderd`, `duemilacinquecento`), and a teen may precede `honderd` (`twaalfhonderd` is 1200). - Decimal comma and dot or space grouping, as for French and German: the CLDR conventions, written as data. ### Dates and times Dates are day first. New sections and flags: | Item | Meaning | |---|---| | `[time.glue]` | splits glued words before reading them: `zaterdagavond` is `zaterdag` + `avond`. Rows are `first words | second words` | | `[time.decade]` | names decades (`Trenta | 30`); a two-digit decade after `anni '90` or `anni 90` is read too | | `time.flag bare-range-24` | a range of bare hours is a time range when the hours are plausible clock times: `dalle 8 alle 12` | | `time.flag word-hour` | a spoken hour with a part of the day is a time: `sei del mattino` | | `hourword` | also after the second end of a range: `van 8 tot 12 uur` | | holiday rule `fixed-not-sunday MM-DD` | the date, moved to the day before when it falls on a Sunday. Used for Koningsdag (27 April; 26 April when 27 April is a Sunday) | Regions: `it` and `nl` (they pick the holiday rows). Holiday sources are statute facts: the Italian public holidays (Legge 27 maggio 1949, n. 260, as amended by later laws) and the Dutch general holidays (Algemene termijnenwet art. 3), with the Easter-based feasts from the computus. Customary dates (Ferragosto, Sinterklaasavond, Festa della mamma) are `fixed` or `nth` rows. ### Limits (Italian and Dutch) - `di N anni` always reads as an age in Italian, so `un contratto di 5 anni` is an age; write `per 5 anni` for a duration. - Dutch `uur` is a weak hour word: `8 uur` is a time only with a part of the day, a tail or a range around it. - `vanmiddag`, `vanavond` and `vannacht` are cut as `van` (this) plus `middag`, `avond`, `nacht`. - Compound measures past two terms are not read. ## Chinese Chinese (`zh`, Mainland conventions, simplified characters) is a data set on the same engine plus a scanner for text without spaces: `Data/zh/{numbers,units,currency,datetime}.rules` (the `zh.*` sections), the shared sequence rules, and the code in `Scan.Chinese*.cs`. Use `Recognizer.Chinese`, `Recognizer.For(Language.Chinese)` or the `Nlp` recognizer verbs with `Language.Chinese`. The other languages' results are unchanged. ### Numbers, measures and money - **`number.digits | system chinese`** reads numerals built from digits (`一二三`, `零`, `〇`, `两`) and place words (`十`, `百`, `千`, `万`, `亿`), with `零` as the filler (`一百零一`, `两千零二十`), mixed forms (`3万`, `2.5亿`) and decimals (`零点零五`). Fractions and percentages: `百分之五`, `三分之一`, `5%`. - **Ordinals** are `第` plus a number (`第三`, `第一百零一`). - **A lone numeral needs a counting word.** `一个`, `一杯` and `一次` are not numbers on their own; a numeral is a number when it is two or more characters, carries a digit, or stands in a construct (a measure, money, a date). The measure word stays outside the span. - **Measures** (`zh.unit`): length, weight, area, volume, speed, temperature (`零下二十度`, `摄氏`), age (`岁`, `周岁`, inside the span). `斤` is the jin and `亩` the mu (Chinese market units, named as such in the values). - **Money** (`zh.currency`): `元`, `块`, `角`, `分`, `人民币`, `美元`, symbols and ISO codes, and compounds (`三块五`, `两元五角`). - **Sequences** are found in a masked pass: text outside ASCII word characters is treated as a boundary, so e-mail, URL, IP and phone numbers are found directly beside Chinese characters. A hashtag is `#x#` (a pair) or `#x`. ### Dates and times Rules in `zh.*` sections of `datetime.rules`: day words (`今天`, `后天`), weekdays (`周一`, `星期天`, `礼拜三`), dates in `2026年3月18日`, `3月18号` and `2026-03-18` forms, relative periods (`下个月`, `去年`, `三天前`, `两周后`), parts of the day (`早上` is 06:00-09:00, `下午`, `晚上`), clock times (`3点半`, `下午三点十分`, `15:30`), ranges (`从周一到周五`, `3点到5点`; the `从` is inside the span), durations (`两年零三个月` is `P2Y3M`; `零` between terms is a gap word, `半` is a half), sets (`每天`, `每周三`, `每个月`) and holidays. `北京时间` is inside a time span and gives the +08:00 zone. Holidays (`[zh.holiday]`): fixed ones (`国庆节`, `劳动节`, `元旦`, `儿童节`), nth-weekday ones (`母亲节`), `复活节` from the computus, and the lunar ones (`春节`, `大年初一`, `除夕`, `元宵节`, `端午节`, `中秋节`) plus `清明节`, which read the pinned table `[zh.lunar]` of Gregorian dates for **2022 to 2030**; other years do not resolve. Statutory holidays follow the State Council's Measures for National Holidays; the compensatory working days of a year's notice are not modelled. ### Limits (Chinese) - `5号线` (line 5) and `三块` (three pieces or three yuan) are ambiguous; the readings follow the context words in the rules, and money is read only with a currency construct. - A bare `3点` is read as a clock hour as written, with no part of the day. - `年初` and `年底` are not read for years; `90年代` is read, a two-digit decade (`九十年代` as a number of years) is not. - Lunar festivals and `清明节` resolve only for 2022-2030. The 2028-2030 rows come from the astronomical calendar, not from a published notice. - Only simplified characters are covered by the rules and the gold set. ## Turkish and Hindi Turkish (`tr`) and Hindi (`hi`) are data sets on the same engine (`Data//{numbers,units,currency,datetime}.rules`). Use `Recognizer.For(Language.Turkish)` or `Language.Hindi`, or the `Nlp` recognizer verbs with a `Language` argument. The other languages' results are unchanged (their test predictions are byte-identical to before). ### Numbers - **`number.digits | system`** takes `devanagari` digits (read as ASCII digits of the same value; Indian grouping `12,34,567`, lakh and crore as scale words) and `native-digits`. **`case | turkic`** folds `İ`/`I` to `i`/`ı` before matching, with offsets unchanged. - Number words follow the language: Turkish compounds are separate words (`yüz yirmi üç bin dört yüz elli`); `on bir` (tens plus unit) is accepted. Hindi has an irregular table for 1 to 100 (generated into `numbers.rules`) and the scale words `सौ`, `हज़ार`, `लाख`, `करोड़`, `अरब`, `खरब`. A decimal number before a scale word multiplies (`1.5 करोड़`). - **Digit ordinals** take an apostrophe ending in Turkish (`5'inci`); the ending is outside the number. - **`[percent.prefix]`**: percent before the number (Turkish `%12`, `yüzde on iki`). ### Dates and times Day first. Case endings after an apostrophe (`3 Mart'ta`, `20.03.2026'da`) are outside the span. New items: | Item | Meaning | |---|---| | holiday rule `table YYYY-MM-DD ...` | a pinned list of dates, for observances that follow the lunar calendar; years not listed have no date | | regions `tr`, `in` | pick the holiday rows (national days; Ramazan and Kurban Bayramı 2025-2030 from the Diyanet lists; Hindi gazetted holidays 2026 and 2027) | | roles `earlypost`, `midpost`, `latepost` | the part of a period named after it: `2021 başında`, `Mayıs ortasında`, `2022 के मध्य`, `मई की शुरुआत`: a year or month, an optional `of` word, then the part. They give an approximate range, as `early 2021` does | | a season after the year | `2021 baharında`, `2025 के पतझड़` | | role `nodate` | phrases that must not be read as a weekday (`pazar yerinde`, the market) | | roles `plushalf`, `plusquarter`, `minusquarter` | Hindi `साढ़े तीन बजे`, `सवा चार`, `पौने पाँच` | | role `rangetail` | the words that close a range after its second end (`10 Nisan'a kadar`, `16:00 तक`, `arasında`) | | flags `daypart-first`, `dot-clock`, `bare-hour-range`, `spoken-hour-mark` | `akşam 7`, `saat 15.30`, a range of bare hours (`9-17`), a spoken hour with its mark | | `phone.space-start` | Hindi: ten-digit mobile numbers in two groups, and landlines with a trunk 0 (`0141 234 5678`) | ### Limits (Turkish and Hindi) - Ramazan and Kurban Bayramı are listed for 2025-2030 only. Hindi lunar holidays are listed for 2026 and 2027 only (Bakrid is left out: the published lists disagree). - Inflected forms of Turkish month and weekday names are listed in the rule data, not derived: a rare case ending is not read. - Hindi `रात` reads as evening to midnight (18:00-24:00) and `कल` as tomorrow unless a past word is nearby. - Early, middle and late parts are only read after a four-digit year or a month name. # Recognizer gold set (English) A hand-labelled set of numbers, quantities, sequences and dates/times in short sentences. It judges recognizers: ours and two black-box comparators. We wrote every sentence and label ourselves from the type definitions, ISO 4217, the RFCs and SI, and everyday text. No comparator's specification or test data was used. ## Files | File | Role | |---|---| | `testdata/recognizers/en/gold.jsonl` | **Development split** (245 sentences, 237 items). Open for everyone. | | `testdata/recognizers/en/test.jsonl` | **Held-out test split** (313 sentences, 312 items). | | `tools/recognizers-gold/en.src.txt` | The authoring source both files are built from (it holds both splits). | | `tools/recognizers-gold/build.py` | Validates the source and writes the two files. | | `testdata/recognizers/en/comparators/` | Scores of the comparators on both splits. | SHA-256 at freeze (second batch): `gold.jsonl` `6ad9be4e3f9fd8aa25c1565769b95adfc97e02c006d64e6893a34a9f03f09cd2`, `test.jsonl` `dd81db526c51e91404e256b0804c7bc1a361a610876db142a70e3c027b7b3bef`. The first freeze (220 + 160 sentences, test `7e985dc1…`) is a subset of both: no sentence changed split. **The separation rule.** Work on a recognizer reads and tunes against `gold.jsonl` only. Nobody who builds or tunes a recognizer opens `test.jsonl`, `en.src.txt` or the comparators' scores on `test`, and test items are never copied into rules, vocabularies, unit tests or training data. The test split is scored once per milestone and reported as it comes. The files are committed, so the rule rests on discipline. Adding sentences later is allowed, but never moves one between splits: re-run `build.py` only after reading its stratification rule below, and compare the test hash. **Split.** The first 380 sentences were assigned by a stratified hash rule: they are grouped by the type of their first item (sentences with no item form their own group). Within a group they are ordered by the SHA-256 of their id and the first 40% (rounded up) are test. The second batch (a second set of test sentences that raises every type to at least 10 test items, and a few development sentences with the same patterns) is assigned explicitly by `@split test` and `@split dev` lines in the source, so adding it moved no earlier sentence. The id is the first 12 hex digits of the SHA-256 of `reference + "\n" + text`. ## Line schema (JSON Lines, UTF-8) ```json {"id":"…","text":"…","reference":"2026-03-18T10:30:00","timezone":"UTC", "items":[{"start":21,"end":27,"type":"duration","text":"a week","value":"604800"}]} ``` - `reference` is the instant relative expressions resolve against, as a UTC wall-clock time without an offset; `timezone` is always `UTC`. Almost every sentence uses `2026-03-18T10:30:00` (a Wednesday). Three groups of sentences use other references: `2026-12-31T23:30:00` (year boundary), `2028-02-28T09:00:00` (leap day), `2026-01-05T08:00:00` (a Monday). - `start`/`end` are offsets into `text` in UTF-16 code units, `end` exclusive. `text` is the covered text. - `type` is one of the 22 types below. `value` is the normalized resolution (absent for sequences). `grain` and `approx` (default false) apply to the temporal types. `alt` lists other accepted values for an ambiguous expression. - A prediction file has the same item shape (`id`, then `items` with `start`, `end`, `type`, `value`, `grain`, `approx`) and no text. ## Types and value forms All numbers are the shortest decimal string: no thousands separators, no exponent, no trailing zeros (`1250`, `0.75`, `-7`, `7900000000`). | Type | Covers | Value | |---|---|---| | `number` | cardinals in digits or words, fractions of a million etc. (`a dozen`, `2.5 thousand`, `minus twelve`) | the number | | `ordinal` | `third`, `21st`, `1,000th` | the integer | | `percentage` | `12%`, `twelve percent`, `7 per cent` | the number without `%` | | `dimension` | length, mass, volume, area, speed, information, power | ` ` | | `temperature` | `25 degrees Celsius`, `72°F`, `20 degrees` | ` celsius\|fahrenheit\|kelvin\|degree` | | `age` | `5 years old`, `3-year-old`, `90 years of age` | ` year\|month\|week\|day` | | `currency` | symbol, ISO code or word, with the amount | ` ` | | `email`, `url`, `ip`, `phone`, `hashtag`, `mention`, `guid` | the Sequence set | none | | `date` | one calendar day | `YYYY-MM-DD` | | `time` | a time of day, no date | `Thh:mm[:ss]` + zone | | `datetime` | a day and a time of day | `YYYY-MM-DDThh:mm[:ss]` + zone | | `daterange` | a named period (week, month, quarter, year, season…) or an explicit range of days | `start/end` dates | | `timerange` | a range of times of day | `Thh:mm/Thh:mm` | | `datetimerange` | a range between date-times | `…T…/…T…` | | `duration` | an amount of time | seconds as an integer; if it names years or months, ISO 8601 `P…Y…M` (`P1Y6M`, `P3M`) | | `set` | a recurrence | TIMEX3-style: `P1D` (daily), `P1W`, `P2W`, `P1M`, `P1Y`, `PT3H`, `XXXX-WXX-1` (every Monday), `XXXX-WXX-1T09:00`, `T12:00` (every day at noon) | - **Units** are lower-case singular English names: `kilometer`, `foot`, `mile per hour`, `kilometer per hour`, `square meter`, `gigabyte`, `kilowatt`. `degree` is a temperature with no scale. - **Zones.** A zone is written only when the expression names one: `Z` for UTC and GMT, otherwise the offset of the abbreviation (`PST` -08:00, `EST` -05:00, `CST` -06:00, `CET` +01:00, `JST` +09:00). The abbreviation is part of the span (`3pm PST`). - **Temporal values.** An instant (`date`, `time`, `datetime`) has a `grain` (`second`, `minute`, `hour`, `day`) and the value is its start; precision follows the expression (`3pm` is `T15:00` with grain `hour`; `10:15:30` has seconds). A value that spells out zero seconds equals one that does not. A range (`…range`) is half-open `start/end`. `grain` is the unit the range is stated in (`week`, `month`, `quarter`, `half`, `season`, `year`, `decade`, or `day` for explicit day ranges, `hour`/`minute` for time ranges). For `set`, the unit of the recurrence. - **Range ends.** The end of a range of days or longer periods is the start of the period after the stated end point, so "from April 6 to April 10" ends at `2026-04-11` and "through 2027" ends at `2028-01-01`. A clock end point is the stated instant: "from 3pm to 5pm" is `T15:00/T17:00`. - **Time of day only.** A gold `time` or `timerange` value starts with `T`. A prediction that adds a date (as a system does when it resolves "3pm" to a day) is compared by its time of day alone. - **`approx`** is true for expressions that name a loose part of a period: `early/mid/late` + period, seasons, `around 3pm`, and day parts (`tomorrow morning`). Early/mid/late split a year into Jan–Apr, May–Aug, Sep–Dec; a month into days 1–10, 11–20, 21–end. Seasons are the meteorological ones in the northern hemisphere: spring Mar–May, summer Jun–Aug, fall Sep–Nov, winter Dec–Feb. Day parts: morning 06:00–12:00, afternoon 12:00–18:00, evening 18:00–24:00. ## Labelling rules 1. **Span.** The smallest span that carries the value, including the unit, currency symbol, zone, the connectives of a range (`from … to`, `between … and`, `-`) and sign words (`minus`), but not a leading preposition (`on`, `at`, `in`, `by`) or trailing punctuation. `3pm on March 22` is one `datetime`; `tomorrow at 3pm` is one `datetime`. 2. **One label per span.** The outermost expression wins. The `3` of `3 km`, the `2021` of `early 2021` and the `22` of `3pm on March 22` are not separate numbers. The scorer ignores a prediction strictly inside a gold item of another type; a prediction with the same span but another type is a false positive. 3. **Number words.** Labelled when they are numbers (`three pizzas`), not when pronouns (`one of them`). No sentence of the set depends on that distinction. 4. **Sequences.** A URL needs a scheme, `www.` or a recognisable host with a path (`example.com/pricing`) and does not include trailing punctuation. An e-mail address contains no separate URL or mention. Phone-like means a syntactic phone number (unvalidated); the numbers are in the fictional `555-01xx` range or documented examples. 5. **Sentences with nothing.** Fifteen sentences contain nothing to find; any prediction on them is a false positive. 6. **Missing year.** An expression with a month and day but no year is in the reference year; the set keeps those after the reference date. A holiday or weekday with no further words is the next occurrence on or after the reference date. ### Ambiguity policy An expression with two defensible readings gets the primary value and the other readings under `alt`. A system is right if it matches any of them. - `next `: if the weekday is later in the ISO week than the reference (Thu–Sun for a Wednesday), the next occurrence this week and next week's both count (`next Friday`: `2026-03-27`, alt `2026-03-20`). For earlier weekdays it is the one in the following week. - `last `: the most recent past occurrence; for a weekday earlier in the reference week, last week's also counts (`last Monday`: `2026-03-16`, alt `2026-03-09`). - `from Monday to Friday`, when Monday has passed this week: the coming week, alt this week. - Day-first or month-first dates: month first (en-US) for `04/15/1990`. The set has no sentence where day-first would change the value. - Weeks run Monday to Sunday. `this weekend` is Saturday and Sunday. - Time of day without am/pm: only 24-hour forms (`15:45`, `09:00`) or forms with a day part (`5 in the afternoon`) are used. ## Scoring (`tools/OneNlp.Eval recognizers`) ``` dotnet run --project tools/OneNlp.Eval -- recognizers --gold testdata/recognizers/en/test.jsonl --pred PRED.jsonl \ --component NAME --dataset NAME [--resamples 1000] [--seed N] [--json OUT] [--markdown OUT] [--errors OUT.tsv] ``` Per type, and for two pooled views, **temporal (any subtype)** (matches regardless of which of the six date/time types is used, because a system with no notion of the split cannot be faulted for it) and **all types**: - **Precision, recall, F1:** exact match of start, end and type. Duplicate predictions of one span and type count once. - **Relaxed F1:** a gold item is found, and a prediction is right, if it overlaps an item of the same type. Read this next to the exact row to separate span conventions from missing items. - **Value accuracy:** among exactly matched gold items that have a value, the fraction whose predicted value equals the gold value or an `alt`. **Value+grain+approx** also needs the same `grain` and `approx`. **Value accuracy on overlapping spans** uses the widest-overlapping prediction of the same type instead of an exact match. - **Intervals:** 95% percentile bootstrap over sentences (1000 resamples, seed 20260101), the method of `docs/measurement.md`. With a handful of items the interval is degenerate (`100.0 [100.0, 100.0]`) or very wide; the table gives the item counts. - `--errors` lists misses, wrong values and spurious predictions with the sentence text. Use it on the development split only. ## Comparators Both run as black boxes: released builds, called through their documented public interfaces, with the same reference time per sentence. | System | Version | Licence | How it runs | |---|---|---|---| | Microsoft.Recognizers.Text | `Microsoft.Recognizers.Text`, `.Number`, `.NumberWithUnit`, `.DateTime`, `.Sequence` 1.8.13 (NuGet) | MIT (NuGet metadata) | `tools/oracles/Recognizers`: English culture; the per-model `Recognize…` calls; date/time with `DateTimeOptions.None` and the sentence's reference time | | Duckling | `rasa/duckling:0.2.0.2-r4`, image digest `sha256:55e7adfb6d63eb0d1c1bdb8a28080a74d4d281871258c2ca2d7cd0eaf29b2d11` (linux/amd64) | BSD (Duckling's README) | `tools/oracles/duckling`: `fetch.sh` downloads the published image's layers; the released `duckling-example-exe` runs as a local server; `POST /parse` with `locale=en_US`, `reftime`, `tz=UTC` | ### Comparator mappings Both comparators are mapped into this schema by scripts of ours (`tools/oracles/Recognizers/map_raw.py`, `tools/oracles/duckling/map_duckling.py`); the headers of those files are the normative description. The choices that move scores: - **Candidates.** Recognizers.Text lists several values for ambiguous input (past and future Friday, am and pm). We take the first that starts on or after the reference date, else the last. Duckling's primary value is taken. - **Types.** Recognizers.Text has the six date/time types and `duration` and `set`. Duckling has one `time` dimension: an instant with a grain of day becomes `date`; a longer grain becomes a `daterange` of one period; a finer grain becomes `time` or `datetime` depending on whether the matched text names a day; intervals become the matching range type. Duckling has no percentage, age, IP, hashtag, mention, GUID or recurrence dimension, so those rows score 0 by construction. Its interval results for money and quantities are dropped. - **Spans.** Recognizers.Text reports an inclusive end (we add 1); both report their own span conventions. Duckling includes leading prepositions (`on Christmas Day 2027`, `at 3pm`) and some trailing words in its spans, so its exact-span rows are low where its relaxed rows are not. We do not trim spans. - **Range ends.** Duckling's interval ends are exclusive at the end point's grain; for ranges with clock end points we subtract one grain so that "3pm to 5pm" ends at 17:00. Recognizers.Text states the end of "from A to B" day ranges as B itself, not the day after, and we take it as reported; those items count against it. - **Years.** Both put a past month and day in the next year. The development set avoids such expressions; this is why `from April 6 to April 10` replaced a March example. - **Zones.** Neither keeps a stated zone as an offset in the form of the gold (Recognizers.Text drops it, Duckling converts to UTC); items with a zone are scored as they come. - **Not mapped.** Duckling's `quantity` products and credit-card or other dimensions; Recognizers.Text's relative ordinals (`next`, `last`). ## Our recognizers `tools/oracles/score_ours.sh` runs `OneNlp.Eval recognize` over the sentences, maps its output with `tools/recognizers-gold/map_ours.py` and scores it with the unchanged scorer; `--ref-pred` adds the paired comparison with Microsoft.Recognizers.Text. Results are in `testdata/recognizers/en/comparators/`: `ours.{test,gold}.{md,json}` are the scores, `ours-vs-recognizers.{test,gold}.{md,json}` the comparisons. - **Mapping.** `CARDINAL`, `ORDINAL` and `PERCENT` become `number`, `ordinal` and `percentage`; `MONEY` becomes `currency` (` `); `QUANTITY` becomes `temperature`, `age` or `dimension` by its dimension (` `); the sequence kinds keep their names (`PHONE_LIKE` is `phone`, `IP_ADDRESS` is `ip`). Offsets are the engine's UTF-16 offsets. - **Not emitted yet:** `date`, `time`, `datetime`, `daterange`, `timerange`, `datetimerange`, `duration` and `set`. Their rows score 0 and the comparison calls them "behind" by construction. The row **all but date/time, duration, set** pools the 14 types the engine emits and is the one to read; the **all types** row includes the types not yet built and is not a verdict. - **Verdict.** Per type and per metric (exact-span F1, value accuracy on exact matches), the two systems are scored on the same sentences and the 95% bootstrap intervals compared by the within-noise rule of `docs/measurement.md`: overlapping intervals are "within noise", otherwise "ahead" or "behind". The paired difference (ours minus the reference) and its interval are printed beside it, because with 10 to 20 items per type an overlapping interval is a weak statement. - **Convention differences** are the comparator's, not ours: it reports the end of "from A to B" day ranges as B and drops stated time zones. They affect only date/time rows. - **Throughput** is measured by `tools/oracles/bench_recognizers.sh` (about 10 MB made from the development split, 3 runs each, comparator with its number, unit, currency and sequence models). Indicative only, on a shared machine. ## French and German `testdata/recognizers/{fr,de}/` hold the gold sets in the same schema, written for these languages (never copied from a comparator): `test.jsonl` and `gold.jsonl` (development). Sources: `tools/recognizers-gold/{fr,de}.src.txt`, built by `build.py --lang fr|de`; the test split was fixed and hashed before any rule for the language was written, and the development split is the only one the rules were tuned on. | File | Items | Sentences | sha256 | |---|---:|---:|---| | `fr/test.jsonl` | 347 | 357 | `7b697b11cdd5893bb6f7316a451e15fc47489e39b0fdd98cd8a14ec4b72fb1b9` | | `de/test.jsonl` | 349 | 359 | `640fb591d83dfc96b1048024511d01929ca68fbe6acee3bb47ae0a3dfe1ebf0d` | Every type has at least 10 test items. Conventions beyond the English ones: a preposition is outside the span (`à`, `en`, `am`, `im`, `um`, `zum`) except where it belongs to the construct (`dans 3 jours`, `il y a 3 jours`, `vor 3 Tagen`, `de 9h à 17h`, `vom 6. bis 10. April`); a French age includes `âgé de`; a German age is `N Jahre alt` or `40-jährige`; `halb eins` is 12:30 (`alt`: 00:30); unit names in values are the English canonical ones. Run the comparators with `tools/oracles/comparators_lang.sh ` (Microsoft.Recognizers.Text 1.8.13 in `fr-fr` or `de-de`; its French and German unit names are mapped to the canonical ones in `map_raw.py`), Duckling with `tools/oracles/duckling_lang.sh`, and ours with `tools/oracles/score_dev.sh [gold|test]`. Results, with paired 95% intervals and Duckling beside them, are in `testdata/recognizers/{fr,de}/comparators/`. ## Spanish and Portuguese `testdata/recognizers/{es,pt}/` hold the gold sets in the same schema, written for these languages (never copied from a comparator): `test.jsonl` and `gold.jsonl` (development). Sources: `tools/recognizers-gold/{es,pt}.src.txt`, built by `build.py --lang es|pt`; the test split was fixed and hashed before any rule for the language was written, and only the development split guided the rules (extra development items were added later; the test file is unchanged). | File | Items | Sentences | sha256 | |---|---:|---:|---| | `es/test.jsonl` | 348 | 358 | `d3036d0d47b0d2e0a781b4bed8f1b35157859061f0eb1df5fe9b16cb7bb3506a` | | `pt/test.jsonl` | 349 | 359 | `822eaecb3fe9104cee8c31fa5dbb71d68b3f1696ea11209e3794b514ae4cde5b` | Every type has at least 10 test items. Conventions beyond the English ones: a preposition, also fused with an article (`no`, `na`, `do`, `pelo`), is outside the span, except where it belongs to the construct (`dentro de 5 días`, `hace 10 días`, `daqui a 5 dias`, `de 8 h a 12 h`, `das 8h às 12h`); an article that starts the expression is inside it (`el 20 de marzo`, `las 15:00`, `o Dia das Mães`); the fused `às`/`à` before a clock time is inside; ages are `N años de edad` / `N anos de idade` only; `noche`/`noite` is 18:00-24:00. The Spanish gold is Spain's (decimal comma, dot or space grouping); the Portuguese gold is Brazil's, with a few European forms. Run the comparators with `tools/oracles/comparators_lang.sh ` (Microsoft.Recognizers.Text 1.8.13; its unit names are mapped in `map_raw.py`), Duckling with `tools/oracles/duckling_lang.sh ...` (information only), and ours with `tools/oracles/score_dev.sh [gold|test]`. Results, with paired 95% intervals: `testdata/recognizers/{es,pt}/comparators/`. ## Italian and Dutch `testdata/recognizers/{it,nl}/` hold the gold sets in the same schema, written for these languages (never copied from a comparator): `test.jsonl` and `gold.jsonl` (development). Sources: `tools/recognizers-gold/{it,nl}.src.txt`, built by `build.py --lang it|nl`; only the development split guided the rules. | File | Items | Sentences | sha256 | |---|---:|---:|---| | `it/test.jsonl` | 368 | 370 | `e4fadcf947a4032543205e7948a006e09f6e2cd3453d609ce2de6602e782d865` | | `nl/test.jsonl` | 369 | 372 | `0d78115506a09c2261050f0530647b80c400b0428100e745b74d8aa8816fed1b` | | `it/gold.jsonl` (dev) | | | `61f613054ce3c69f429be959343ea11640ff7b3b76d75baf7f1d534f261ceb27` | | `nl/gold.jsonl` (dev) | | | `d196c8d32d99ed7edffcdac60bb44a0c8e53bd32b76faf48002b1bd51776a8aa` | The test files were edited once after their first hash (it `3d15d452…`, nl `e1a04260…`), before any score on them was read and without system output: three Italian durations were reworded because `di N anni` reads as an age, incidental dates, durations and sets were labelled or reworded, and the Dutch determiner `de` was moved outside the span in `de derde donderdag van april`, `de laatste vrijdag van maart` and `de eerste maandag van mei`. Conventions beyond the English ones: a preposition, also fused with an article (`nel`, `dal`, `alle`, `in`, `op`), is outside the span, except where it belongs to the construct (`tra 3 giorni`, `3 giorni fa`, `over 3 dagen`, `dalle 8 alle 12`, `van 8 tot 12 uur`); an Italian age includes `di` (`di 82 anni`); a Dutch age is `N jaar oud` or `N-jarige`. Run the comparators with `tools/oracles/comparators_lang.sh ` (Microsoft.Recognizers.Text 1.8.13), Duckling with `tools/oracles/duckling_lang.sh ...` (information only), and ours with `tools/oracles/score_dev.sh [gold|test]`. Results, with paired 95% intervals: `testdata/recognizers/{it,nl}/comparators/`. ## Chinese `testdata/recognizers/zh/` holds the gold set in the same schema, written for Chinese (never copied from a comparator): `test.jsonl` and `gold.jsonl` (development). Source: `tools/recognizers-gold/zh.src.txt`, built by `build.py --lang zh`; only the development split guided the rules. The same author wrote the gold set and the rules, which the results below do not correct for. | File | Items | sha256 | |---|---:|---| | `zh/test.jsonl` | 338 | `4a50aa15f954234f4b800fee45fe5cfeb17642670fa60334553a2c9f2819949b` | | `zh/gold.jsonl` (dev) | 189 | `285eac0361893ae6defd14c2bad16c0f68f54a2777507cb81442d08a4832b922` | The test file was edited once after its first hash (`e728c157…`), before any score on it was read and without system output: incidental lone numerals (`一个`, `一杯`, `一部`, `一口气`, `一次`) were reworded in 12 sentences, because a lone numeral with a counting word is not a number in our convention. **Test reads.** The first read of the test split gave all types F1 98.8 [97.4, 99.9] against Microsoft.Recognizers.Text 49.8 [46.4, 53.0]; that read is the published row (`comparators/ours.test.score.*`, `ours-vs-recognizers.test.*`). It showed one error that the development split had no item for (`两年零三个月`, a gap word before a duration term), and a rule change followed. The post-change run (all types 99.1, duration 100) is in `comparators/post-fix/`; it is not an independent read, since the change was made after seeing the test error. Conventions beyond the English ones: measure words stay outside spans; `从` is inside a range; `北京时间` is inside a time; `岁` and `周岁` are inside an age; `斤` is the jin and `亩` the mu. Run the comparators with `tools/oracles/comparators_lang.sh zh zh-cn ` (Microsoft.Recognizers.Text 1.8.13), Duckling with `tools/oracles/duckling_lang.sh zh zh_CN ...` (information only), and ours with `tools/oracles/score_dev.sh zh [gold|test]`. ## Turkish and Hindi `testdata/recognizers/{tr,hi}/` hold the gold sets in the same schema, written for these languages (never copied from a comparator): `test.jsonl` and `gold.jsonl` (development). Sources: `tools/recognizers-gold/{tr,hi}.src.txt`, built by `build.py --lang tr|hi`. The test splits were hashed before any rule was written and have not changed; only the development split guided the rules. | File | sha256 | |---|---| | `tr/test.jsonl` (316 items, 327 sentences) | `4876a7eb3dd2af542a00da5f682414102c1fb176ed8a6958cec96a3aa42be19a` | | `hi/test.jsonl` (324 items, 335 sentences) | `f4358d9849ea40f25d93e73fe4b698415aa84b29efd541d54ba3b372b006a68f` | | `tr/gold.jsonl` (dev) | `3cc32a3de4f4fbcb090901b98623b43a8d6e42b0719b535cee4f9e6b9f9c8756` | | `hi/gold.jsonl` (dev) | `896be443c66c8a3cea3c739e916243d7d63b843500807df2b31ea91bd73cd559` | Conventions beyond the English ones: a case ending after an apostrophe (`'da`, `'inci`) is outside the span; a Hindi postposition (`को`, `में`, `पर`, `से`) is outside the span, except the ones that frame a range (`से ... तक`) and the age endings (`का`, `की`, `के`), which are inside; `बजे` is inside a clock time; the closing words of a worded range (`arasında`, `kadar`, `तक`) are inside the range only when the range is worded; `रात` is evening 18:00-24:00; `कल` is tomorrow. Early, middle and late parts and seasons are approximate ranges. Comparators: `tools/oracles/comparators_lang.sh `; Duckling `tools/oracles/duckling_lang.sh ...` (information only); ours `tools/oracles/score_dev.sh [gold|test]`; `tools/oracles/regress_check.sh ` compares test predictions with another build byte for byte. # Language identification `LanguageDetector` (namespace `RecursiveAI.OneNlp.LanguageId`) names the language of a text from one word up to a document, among the 75 languages listed in `tools/fetch/langid/languages.tsv`. The model is one file of 3.6 MB that is embedded in the core assembly at build time. This page is normative for the method, the model file and the way it is measured. ## Behaviour - `Detect(text)` returns the most likely language, or `Language.Unknown` when the text has no letter of a supported script. Digits, punctuation and white space are ignored. - `Confidences(text)` returns every candidate with a probability; the values sum to 1. Only languages that use the text's script are candidates. - `Restrict(languages)` returns a detector that never answers outside the set. A text whose script none of the allowed languages use is `Unknown`. - `DetectSpans(text, switchPenalty = 9)` returns runs of one language that tile the text from index 0 to its length. See "Mixed text". - The happy path is `Nlp.DetectLanguage(text)`. `Nlp.Analyze`, `Nlp.Tokens` and `Nlp.Sentences` detect the language first and use its segmentation data (the language's own exceptions and multiword-token rules where it has them, plain UAX #29 otherwise). - A detector is immutable and thread-safe. ## Where the weights come from The weights are not in git. `src/OneNlp/LanguageId/weights.manifest.props` lists the file with its size, sha256 and recipe. A build embeds it when the environment variable `ONENLP_LANGID_WEIGHTS` (or the property `LangIdWeightsDir`) names a directory holding `langid.bin`; the build fails if the file is missing or its hash differs from the manifest, and packing fails without it. A build without the variable still compiles and runs, but carries no model: `LanguageDetector.Default` throws `InvalidOperationException` naming the variable, `LanguageDetector.Load` still reads any model file, and the `Nlp` verbs treat every text as English. The recipe in `recipes/langid` rebuilds the file. ## Method 1. **Words.** The text is cut into runs of letters and combining marks (Unicode general categories L\* and M\*), lowercased with the invariant simple case mapping. A letter of a different script than the run so far ends the run. At most 65,536 letters of one text are used. 2. **Script and group.** Each word has a script (`Script.cs`: Latin, Cyrillic, Greek, Armenian, Hebrew, Arabic, Devanagari, Bengali, Gurmukhi, Gujarati, Tamil, Telugu, Thai, Georgian, Hangul, and one CJK group for Han and Kana). The model has one group per script, listing the languages written in it. A language belongs to every group whose script makes up at least 3% of its training letters (Serbian is in Cyrillic and Latin). The text goes to the group that holds most of its letters. 3. **Single-language scripts.** Greek, Armenian, Hebrew, Gujarati, Gurmukhi, Tamil, Telugu, Thai, Georgian, Bengali and Hangul each have one language in the list: the answer follows from the script, with confidence 1. 4. **Features.** For each word padded with a blank on each side, every character n-gram of length 1 to 5 (Cavnar and Trenkle 1994, §3), plus the whole word when it has at most 15 letters. Each feature is hashed to 32 bits and the top bits are the bucket (Weinberger et al. 2009). Group bucket counts: 2^16 for Latin, 2^14 for the other groups. 5. **Scores.** Per group, a table of signed 8-bit weights, `2^bits × classes`. The score of a class is `bias + scale × Σ weights / n^0.5`, with `n` the number of features. The probabilities are the softmax of `temperature × score` (temperature 1.218, fitted on held-out text). This is a linear softmax classifier as in Joulin et al. 2016, without the low-rank layer. 6. **Training** (`tools/OneNlp.LangId`): sparse AdaGrad (Duchi et al. 2011) on the cross-entropy loss. A sample is a random view of a training sentence: one word (22%), two adjacent words (28%), three to five words (15%) or the whole sentence (35%), so the model sees the lengths it is judged on. After training, the weights are quantized to int8 with a clip at the 99.95th percentile of magnitudes. The trainer and the detector call the same featurizer. ## Mixed text `DetectSpans` scores every word on its own (the softmax of its group, floored at e^-12), then finds the language sequence that maximizes the sum of word log-probabilities minus `switchPenalty` for every change of language (Viterbi 1967). Words of an unsupported script say nothing. Runs begin at the first word of a new language, so the spaces and punctuation after a run belong to it. Confidence of a run is the probability of its language computed from the words of the run together. The default penalty 9 was chosen on synthetic documents of two or three held-out sentences in different languages (2,000 documents, `recipes/langid/results/mixed.txt`): | switch penalty | characters in the right language | documents with exactly the right runs | |---|---|---| | 2 | 92.35% | 31.80% | | 4 | 95.96% | 65.85% | | 6 | 97.11% | 82.00% | | 9 | 97.53% | 89.30% | ## Model file All integers little-endian. `"ONLI"` (4 bytes), version (1 byte, 1), language count (u16), per language a length byte (2 or 3) and the ASCII code, exponent (f32), temperature (f32), group count (u8), then per group: script mask (u32, bit per `Script` value), bucket bits (u8), class count (u8), the class language indexes (u16 each), weight scale (f32), class biases (f32 each) and, when the group has more than one class, `2^bits × classes` signed bytes, row-major by bucket. A group with one class carries no weights. The reader checks every count before allocating, limits the weights to 96 MB, and throws `InvalidDataException` for anything else. `LanguageDetector.Load(Stream)` reads a model file; the shipped one is the embedded resource `langid.bin`. ## Data Permissive and attribution data only; `recipes/langid/PROVENANCE.md` has the full log. - Leipzig Corpora Collection downloads, offered under CC BY 4.0 (their usage page): up to three corpora per language (newest news, then Wikipedia text, then older news), 100,000 sentences each at most. One corpus per language is held out for evaluation and never used for training (68 languages). For the 7 languages that have too little other text (Maori, Shona, Southern Sotho, Tswana, Tsonga, Xhosa, Yoruba) the evaluation texts are a hash slice of the one corpus, marked `same-corpus` in `dedup-report.tsv`. - Tatoeba, default licence CC BY 2.0 FR: at most 20,000 sentences per language for training; a 5% hash slice is a second evaluation set (64 languages). - Southern Sotho, Tswana and Tsonga have no Leipzig corpus. Their training and held-out text is their Wikipedia text (CC BY-SA 4.0). - Share-alike lineage: yes. Some Leipzig corpora are Wikipedia text and the three Wikipedia dumps above are CC BY-SA 4.0; the provenance log lists them. Sentences are identified by their letters only (lowercase, no digits, punctuation or spaces); a sentence whose identity occurs in an evaluation set is removed from training. ## Evaluation Buckets follow the way Lingua's README describes its own: single words of at least 5 letters, word pairs of at least 10 characters, and whole sentences. For Chinese, Japanese and Thai, which have no spaces, a word is a run of 5 to 7 letters and a pair a run of 10 to 14. Up to 1,000 texts per bucket per language, drawn by a fixed seed from the held-out pool. The set is listed with SHA-256 in `recipes/langid/MANIFEST.sha256` and was written before any model was trained. The score is the macro average of per-language accuracy (every language weighs the same). Intervals are percentile bootstrap (1,000 resamples, seed 20260101) over test texts; differences between systems are paired. The references are run only through their public APIs: `lingua-language-detector` 2.2.0 (Python, native core) and `SearchPioneer.Lingua` 2.0.0 (.NET), each in the high-accuracy and low-accuracy modes the README documents, built from the same 75 languages (`tools/oracles/Lingua`). Results are in `recipes/langid/results/` (tables, per-language CSVs with paired intervals, and the confusions of every language below 90%). Summary on the held-out Leipzig texts (main) and the Tatoeba slice, macro accuracy %, difference to Lingua high accuracy with its 95% interval: | set / bucket | ours | Lingua Python high | Lingua .NET high | ours − Python high | |---|---|---|---|---| | main / words | 77.14 | 75.59 | 75.35 | +1.56 [+1.31, +1.86] | | main / pairs | 89.53 | 85.66 | 85.49 | +3.87 [+3.65, +4.08] | | main / sentences | 98.03 | 95.98 | 95.81 | +2.05 [+1.95, +2.15] | | Tatoeba / words | 79.14 | 75.03 | 74.79 | +4.10 [+3.39, +4.77] | | Tatoeba / pairs | 91.14 | 85.79 | 85.75 | +5.35 [+4.66, +6.18] | | Tatoeba / sentences | 96.15 | 93.27 | 93.19 | +2.88 [+2.55, +3.20] | Caveats: the Leipzig held-out texts come from the same collection Lingua's README says its models were built from, so the main set may favour Lingua, not us. The Tatoeba slice is a different style (short, conversational) and our training includes other Tatoeba sentences, so it may favour us. Both are reported. Languages where we are significantly below Lingua Python high on the main set (paired interval of the difference entirely below zero): words 14 (ar da ga hr ko lg mk nl ru so ur xh yo zh), pairs 8 (hr ja lg ms ru xh yo zh), sentences 2 (en hr). The closely related Latin-script pairs are the hard rows: Croatian/Bosnian, Indonesian/Malay, Norwegian Bokmål/Nynorsk/Danish. ## Speed and memory (indicative) One thread, the 75,000 main sentences, a 32-thread host that also ran other jobs (`recipes/langid/results/timing-indicative.txt`): ours 1.0 s; SearchPioneer.Lingua high accuracy 53 s, low accuracy 63 s. Words: ours 0.38 s, Lingua .NET 7.6 s. Model load 12 ms, process peak RSS 127 MB (includes the .NET runtime and the test harness). Not a publishable run. # The fast tier The fast tier is a per-language pipeline built on a small hash-embedding encoder: each head reads vectors made from the hashed lower-cased form, prefix, suffix and shape of every token, mixed by a few window layers (and, for some languages, one bidirectional LSTM layer). It runs on one CPU thread with no GPU, loads in a fraction of a second and is the model behind `Nlp.Analyze` and `Nlp.Entities` when a language package is referenced. Every part is trained in C# with `Train` ([training.md](https://recursiveai.org/onenlp/docs/training/)); the encoder and heads are specified in [contracts.md](https://recursiveai.org/onenlp/docs/contracts/), the file layout in [model-files.md](https://recursiveai.org/onenlp/docs/model-files/). ## What a language package contains One saved pipeline (`PipelineFile`) of these parts: - **Tokenizer.** UAX #29 words plus the language's exception data ([segmentation.md](https://recursiveai.org/onenlp/docs/segmentation/)): elision and clitics, contractions cut into the pieces the treebank counts as words, abbreviations. Turkish uses a trained multiword splitter and Chinese a trained word segmenter. - **Tagging heads.** `Tagger` (the treebank's own tag, `Token.Tag`), `Morphologizer` (universal POS and features, `Token.Pos` and `Token.Morph`) and the edit-tree `EditTreeLemmatizer` (`Token.Lemma`). - **`Parser`.** A biaffine dependency parser (Dozat and Manning 2017) that also sets sentence starts (`Token.Dep`, `Token.Head`). It reads its own encoder. - **`EntityRecognizer`.** Entities with the labels of its training data (`PER`, `ORG`, `LOC`, and `MISC` where the data has it). Where a treebank has no XPOS column (French-GSD, Portuguese-Bosque) there is no tagger and `Token.Tag` stays empty. ## Languages | Package | Treebank (UD r2.18) | Entity data | Language-specific tokenization | |---|---|---|---| | `.Models.English` | English-EWT | UNER English-EWT | English exceptions; the packaged tagging and entity encoders are also trained on web text labelled by the previous model | | `.Models.French` | French-GSD | WikiNER French | elision, inversion clitics; `du`, `au`, `aux`, `des` as multiword tokens | | `.Models.German` | German-GSD | GermEval 2014 | contractions (`zum`, `im`) as multiword tokens, ordinals, abbreviations | | `.Models.Spanish` | Spanish-AnCora | WikiNER Spanish | `del`, `al`, enclitic pronouns, `¿` and `¡` | | `.Models.Portuguese` | Portuguese-Bosque | UNER Portuguese-Bosque | contractions as pieces, enclitics after a verb's hyphen | | `.Models.Italian` | Italian-ParlaMint and Italian-MarkIT (897 sentences in all: the larger Italian treebanks are non-commercial) | WikiNER Italian | `del`, `farlo` and similar | | `.Models.Dutch` | Dutch-Alpino | WikiNER Dutch | Dutch abbreviations and compounds | | `.Models.Turkish` | Turkish-BOUN | Turkish-WikiNER | apostrophe suffixes kept; a trained multiword splitter; Turkic case folding | | `.Models.Chinese` | Chinese-GSDSimp | UNER Chinese-GSDSIMP | a trained character-tagging word segmenter | | `.Models.Hindi` | none (no allowed treebank): FineWeb-2 Hindi text labelled by the accurate tier and by Marathi, Sindhi and Punjabi heads on its encoder, 32,276 sentences kept by agreement | the same text, 50,380 sentences | none; the danda is a token. No lemmas (the token text stands in), no XPOS | Every training source is credited in the package's `THIRD-PARTY-NOTICES.md`, and `PROVENANCE.md` lists datasets, recipes, seeds and hashes. Most of the data is CC BY-SA, so the weights carry that lineage (credited), and they ship under `MIT OR Apache-2.0` ([ud-licences.md](https://recursiveai.org/onenlp/docs/ud-licences/) lists every treebank considered and why some are not used). There is no Hindi package: Hindi has no allowed UD or named-entity training data. ## Measured Per language, against spaCy 3.8.18 trained on the same data with its default configuration, in gold-token and raw-text modes: [results-table.md](https://recursiveai.org/onenlp/docs/results-table/). The details and the recipes' own tables are in [results.md](https://recursiveai.org/onenlp/docs/results/) and in each recipe's `PROVENANCE.md`. Speed is indicative until the release gate. ## Train your own The same recipes run on your data; for a language we do not ship, start from the nearest one: `recipes//...` has the commands, and `Pipeline.Blank(Language.FromCode("xx"))` takes any language code (the tokenizer falls back to plain UAX #29 words). # The distilled accurate tier `RecursiveAI.OneNlp.Models.Multilingual`: one six-layer, width-384 student of mmBERT-base (35.5 M weights, distilled in PyTorch from mmBERT-base and exported once; `recipes/accurate/student-v1`), **fine-tuned** with per-language heads for English, German, French, Spanish, Portuguese, Italian, Dutch, Turkish and Chinese (tagger, morphologizer, edit-tree lemmatizer, biaffine parser, entity recognizer), plus an open-type entity head on the **frozen** student. The frozen student is also published alone (`RecursiveAI.OneNlp.Models.Multilingual.FrozenEncoder`, `FrozenStudentEncoder.Create()`). - **Recipe:** `recipes/accurate/student-multilingual` (`train`, with `--export DIR` writing the encoder and one head file per language; `export-open` re-saves the open-type head over the frozen encoder). The languages are data: `--langs` names codes, `--languages tools/fetch/ud/languages.tsv` supplies the gating treebank of any other language whose data has been fetched, so adding a language is a line in that file and a fetch. - **Fine-tuned for the heads, frozen for the open-type head.** On English-EWT UPOS a fine-tuned student scored 95.88 and a frozen one 93.01 (`recipes/accurate/student-v1/RESULTS.md`); the open-type head was trained, and checked, on the frozen student and keeps it. - **Training run:** seed 1, 2,000 steps of about 3,000 words over the nine languages (language-balanced), learning rate 2e-4 chosen on development against 1e-4 (80.40 against 77.85 mean development score at step 1,250); the development score was still rising at the end, so the model is under-trained, not converged. About 2 hours on four cores. Provenance: the package's `PROVENANCE.md`. - **Size and speed (indicative, one thread, a loaded 16-core x64 machine, load about 9):** the packages total 482 MB (main 172, Encoder 131, FrozenEncoder 136, Segmenters 43). Full pipeline (tokenize, tag, lemmatize, parse, entities): 1,153 tokens/s on English and 1,162 on German test sentences (400 sentences each, best of three), peak RSS about 0.8 GB. The first `Nlp.Analyze` of a short sentence took 37 ms and the process stayed at 47 MB (the weights are mapped from the assembly, so the pages a sentence does not touch are not read yet). - **Training cost against PyTorch (indicative):** one fine-tuning step of the student, 24 sentences of 32 pieces, 8 threads, same machine: 0.41 s in C# (forward 0.11, backward 0.24, clipping 0.03, AdamW 0.04) against 0.13 s in PyTorch 2.11 on CPU, 3.2 times. Gradient clipping and AdamW account for a disproportionate part (about 0.07 s against 0.01 s); both are on the OneModel backlog. ## Results Every row is information: the base-size tier is the one that gates. Per language and input mode, with the fast tier's rows and the published Stanza, UDPipe 2 and Trankit numbers beside ours: `distilled-tier-rows.md`. Test splits were read once for this model; development sets chose the learning rate and the checkpoint. ### Entities (micro F1, test, gold tokens, bootstrap 95% interval over sentences, 1,000 resamples, seed 20261106) Entity test sets are capped at 3,000 sentences sampled evenly, so rows are not the fast tier's full-test rows. | Language | Corpus | Distilled tier | Fast tier (full test) | |---|---|---|---| | English | UNER EWT | 68.89 [66.01, 71.52] | 63.09 [60.01, 65.89] | | German | GermEval 2014 | 71.78 [70.07, 73.51] | 66.28 | | French | WikiNER | 74.99 [73.65, 76.20] | 82.48 [81.74, 83.28] | | Spanish | WikiNER | 78.62 [77.36, 79.77] | 82.88 [82.12, 83.64] | | Portuguese | UNER Bosque | 80.87 [78.42, 83.26] | 73.13 [70.58, 75.60] | | Italian | WikiNER | 77.04 [75.70, 78.31] | 86.35 [85.73, 87.01] | | Dutch | WikiNER | 77.47 [76.14, 78.70] | 87.47 [86.91, 87.98] | | Turkish | Turkish-WikiNER | 71.93 [69.74, 74.38] | 67.40 [64.90, 69.81] | | Chinese | UNER GSDSimp | 72.78 [68.50, 76.90] | 62.80 [58.46, 67.23] | ### Where it stands - It is ahead of the fast tier on UPOS in English raw text, Dutch and Turkish and on entities in English, German, Portuguese, Turkish and Chinese; it trails it on lemmas everywhere (the lemmatizer here has less capacity than the fast tier's), on parsing (LAS 4 to 15 points), on French, Spanish, Italian and Dutch entities, and on Chinese tagging and parsing (UPOS 85.56 against 90.41, LAS 54.05 against 68.76). - It trails the published UDPipe 2 and Stanza numbers by a wide margin on every row: a short run of a six-layer student. The base-size tier (`accurate-tier.md`) is the model for accuracy. - Sentence splitting comes from the parser and is weakest in English (65.4 on EWT raw text) and Chinese. - Hindi has no allowed training data and is not in this package. # The base-size accurate tier `RecursiveAI.OneNlp.Models.Multilingual.Large` shares one fine-tuned mmBERT-base encoder across **84 UD languages**, with entity heads for **19**. It is distinct from `.Models.Multilingual`, the distilled model covering nine languages. The full language list is `MultilingualLargeModel.Languages`; annotation availability and data gaps are in `docs/ud-licences.md`. A referenced fast-tier package is preferred by the plain `Nlp` verbs; explicit `Tier.Accurate` prefers this base-size package over the distilled one. **Gate:** the accurate tier is measured against spaCy 3.8.18 trained on the same data: of 144 paired rows, 117 are ahead, 14 within noise and **13 behind** (Chinese UPOS, UFeats, lemmas, UAS and LAS; Spanish UPOS, UFeats and lemmas; Dutch WikiNER entity F1, precision and recall; German raw tokens; Turkish raw sentences). **Information, not a gate:** published research systems (Stanza, UDPipe 2, Trankit) are ahead on most rows, typically by 3–9 LAS points: 1,205 of 1,314 best-published comparisons and 534 of 650 Stanza 1.5.1 comparisons are misses. Every metric, language, input mode and confidence interval is reported in [accurate-tier-rows.md](https://recursiveai.org/onenlp/docs/accurate-tier-rows/), including separate counts for the best published system per row and Stanza 1.5.1 pretrained/charlm without a transformer. Missing references are not passes. The table also includes the previous nine-language model and a paired comparison with the same-data spaCy 3.8.18 baselines. No multilingual average substitutes for those rows. The complete results cover **72,685 UD test sentences / 1,202,477 gold words**, in both gold-token and raw-text modes, plus 19 entity samples. There are **1,719 metric rows**: **1,205/1,314 best-published comparisons miss**, and **534/650 Stanza-only comparisons miss** (different view populations, not comparable pass percentages). The paired spaCy information view has 117 rows ahead, 14 overlapping and 13 behind, out of 144. ## Reproducible recipe Recipe: `recipes/accurate/multilingual`, shared reader/sampler: `recipes/accurate/common`. The packed `PROVENANCE.md`, `corpus.json`, `lexicon-dev.tsv`, manifests and notices identify the exact inputs and selected weights. - **Base:** mmBERT-base revision `c5955035435e2bf121cde7f3c8863ef52ff35d82` (MIT), continued from the previous nine-language checkpoint. Encoder tensors and matching heads are copied by name; new heads and all optimizer moments start fresh. The frozen token table is unchanged. - **Data:** UD r2.18, 109 training treebanks; UNER, GermEval, WikiNER, Turkish-WikiNER, KPWr-NER and RONEC entity data. Only train/dev roles train or select anything. Test-only treebanks never train. Extra-treebank training examples repeating a gating test text are excluded. Official split overlap in the original nine is documented in their manifests. ShareAlike lineage: **yes**, with all dataset credits, including Ligurian-GLT under C-UDA 1.0, in the notices. - **Training:** C# on OneModel and OneModel.Training 1.0.0; seed 3, AdamW, batch target 3,000 words, language-temperature exponent 0.3, parser loss weight 1.5. Peak learning rate 6e-5, a 1,200-step cosine schedule. Training stopped at **step 1,050**, also the best development checkpoint; no test-driven continuation or language boosts. - **Selection:** every 150 steps, mean of available per-language development metrics. The selection curve was 59.00, 69.63, 73.35, 75.38, 76.81, 77.79, **78.24**. This is a development selection statistic, not a test-quality claim. Small ordered dev samples are domain-biased and noisy; exact sampling limits are in provenance. - **Heads:** XPOS tagger, UPOS/features morphologizer, edit-tree lemmatizer, biaffine dependency parser and entity recognizer, where annotations exist. One shared encoder receives each language's documents, with updates restricted to that language's heads. - **Lemma tables:** after checkpoint selection, tune training-only `(lower-cased form, predicted UPOS)` tables on dev: minimum count 1/2, minimum share 0.6/0.8/0.9/1.0. A strict improvement is required. Of 79 lemma heads, 68 select a table and 11 retain edit trees alone; five languages have no lemma head. No-dev languages do not select a table. - **Raw text:** Chinese uses the trained Chinese fast-tier word segmenter; Japanese, Classical Chinese, Thai and Vietnamese have segmenters trained from their allowed UD training splits. Other languages use the documented word and sentence rules. Multiword-token pieces carry their expanded word forms in `Token.Norm`, matching the encoder's training input. **Finnish is FTB**, not TDT: FTB has more training sentences in the selected corpus. Its dev slice begins with colloquial/dialectal material, and FTB/TDT XPOS and compound-lemma conventions differ. FTB scores are never compared with a TDT reference. Italian trains on the combined ParlaMint/MarkIT data but its gating test is MarkIT; the table names the actual test treebank. ## Parser-only recovery on the frozen encoder The seven original languages whose full-dev evidence supports adaptation now use recovered parser heads: **de, fr, es, pt, nl, tr, zh**. English and Italian keep their exact jointly trained files. Only parser parameters changed; the graph, token table, tokenizer, all tag/morphology/lemma/entity parameters and the other 75 languages are unchanged. Recipe and complete provenance: `recipes/accurate/multilingual/PARSER-RECOVERY.md` and the package's `PROVENANCE.md`. Each candidate was selected on the **full** development split with a paired sentence-bootstrap LAS comparison against the jointly trained parser (1,000 resamples, seed 66). The lower difference bound must exceed zero. Best full-dev epochs are restored after wall-time cancellation; where no epoch completes, the final partial epoch is evaluated in full. No test-driven continuation or selection follows. The seven selected heads were then evaluated once in gold-token and raw-text modes on 6,356 official test sentences. All seven improve LAS in **both** modes, with paired 95% difference intervals excluding zero against the joint 84-language model. English and Italian reuse the previous test results because their exact files and encoder are retained. Complete before/joint/after intervals for all nine, all metrics and both published-reference views are in [accurate-tier-rows.md](https://recursiveai.org/onenlp/docs/accurate-tier-rows/). | Language | Gold LAS: earlier nine → joint → recovered | Raw LAS: joint → recovered | |---|---|---| | en | 85.43 → 85.12 → 85.12 (unchanged) | 86.55 → 86.55 (unchanged) | | de | 83.32 → 82.17 → 83.49 | 81.05 → 82.38 | | fr | 86.55 → 85.83 → 86.89 | 88.96 → 89.78 | | es | 85.37 → 84.66 → 86.97 | 88.35 → 89.70 | | pt | 84.01 → 81.83 → 84.54 | 86.86 → 87.98 | | it | 77.27 → 77.63 → 77.63 (unchanged) | 82.22 → 82.22 (unchanged) | | nl | 87.63 → 83.40 → 87.24 | 89.09 → 91.11 | | tr | 67.92 → 64.97 → 67.13 | 62.33 → 64.31 | | zh | 63.29 → 57.55 → 61.42 | 58.91 → 60.98 | **Sentence-boundary trade-off:** Turkish raw sentence F1 falls from 79.89 to 78.82. Its paired difference against the joint model is **-1.073 [-1.998, -0.134]**, despite raw LAS improving by 1.978 [1.413, 2.578]. This is the only significantly negative paired row among the 139 recomputed UD metric rows. All 76 checked non-parser metric intervals are identical; sentence classification is part of the adapted parser. The selection rule was fixed to full-dev gold LAS before test, and no post-test weight selection was made. This is not a regression-free update. This is recovery relative to joint training, **not a published-system gate pass**. Chinese gold LAS remains 1.88 points below the earlier nine-language point; the original and recovered marginal intervals overlap, and both rows remain visible. Published-reference misses are still reported without relaxation. Save/reload checks reproduce the selected parsers' actual full-dev head/label predictions and preserve every non-parser head's configuration and tensor hash. Package part boundaries and payload sizes are unchanged. The affected root and Chinese packages have been built and size-checked; validation through the packed JIT consumer remains pending after its restore ran out of disk space. ## Serving and numerical checks The selected encoder is exported into the original ONNX graph and runs on OneModel, not an external native runtime. The graph has **1,228,334,485 bytes**, SHA-256 `f1528693273734e7c91273f33098cabbaf2e09e44449b63768204114cf7e230c`. Development export checks compared **11,869 word rows** across 149 files: zero form differences and two Egyptian dependency-head differences. Feeding identical serialized heads with the module and graph outputs reproduces those two punctuation-arc changes: hidden-state maximum absolute difference **2.98023224e-5**, RMSE **3.04120242e-6** (94 × 768 values; module eight threads, graph two). This is a recorded floating-point difference, not exact prediction parity. Saved-pipeline checks for English, Czech, Chinese, Japanese, Classical Chinese, Thai and Vietnamese compare 695 raw words with identical annotations. Debug/Release checks compare 3,116 rows in 38 files with zero differences. ## Packages and NativeAOT Reference the root package only. It depends on **18 model parts**: six consecutive, byte-preserving encoder pieces (`Encoder1/2`, `TokenTable1–4`, historical names), Chinese heads/segmenter, and eleven additional head groups. There are **91 payload files totalling 3,508,907,198 bytes**. No tensor is quantized, reduced or dropped to fit a package. The encoder and original nine-language heads are embedded. The other 75 heads are package assets copied transitively by `buildTransitive` targets to `weights/multilingual-large/heads/` in both build and publish output. **Deploy the complete publish directory, not just the executable.** No runtime download or separate model installation occurs. Assigned-code switches select files; arbitrary paths are not accepted. An all-embedded layout failed NativeAOT `ilc` with “Format of the executable (.exe) or library (.dll) is invalid.” A clean-cache no-model publish succeeded; a clean-cache all-embedded model publish still failed with free disk. The copied-assets layout passes both JIT and NativeAOT, loading all 84 language heads, checking six happy-path texts and processing Czech, Japanese and Finnish through explicitly selected trained components. Both output directories contain all 75 additional head files. NativeAOT/trim warnings: **zero**. Every model package is below 250,000,000 bytes; exact payload hashes and sizes are in the package manifests. ### Indicative one-language memory and startup `tests/OneNlp.ModelConsumer.Large` against the packed packages, `--single-language`, German first `Nlp.Analyze` including language detection, model load and the first inference; `/usr/bin/time -v` in a fresh process: | Runtime | First Analyze | Peak RSS | Process elapsed | |---|---:|---:|---:| | JIT | 3,694 ms | 4,028,732 KiB | 3.94 s | | NativeAOT | 2,874 ms | 3,993,404 KiB | 3.22 s | These are **shared-host indicative observations, not throughput benchmarks or a speed-gate result**. Linux x64, AMD Ryzen 9 7950X3D (16 cores/32 logical CPUs, AVX2/AVX-512 available), four logical processors exposed to the process; .NET SDK 10.0.403/runtime 10.0.14, OneModel 1.0.0, CPU only. One-minute load before/during/after was 8.04/8.04/7.96 for JIT and 9.83/9.36/9.36 for AOT. No swapping occurred. These are fresh processes with warm filesystem caches. Materializing every language in the same process peaked at 8,304 MiB in the JIT smoke and 8,323 MiB in AOT; applications loading one language need not materialize all heads. ### OneModel runtime update: memory and exactness OneModel **1.2.0** replaces 1.0.0 in the neural package. Single-resource embedded graphs now load in place: the distilled encoders and the English sentence embedder transfer their model streams to the encoder/embedder, which keeps them open until disposal. The existing stream-loading overloads retain their buffered lifetime contract; the new `leaveOpen` overloads explicitly select the borrowed/owned in-place path. Tokenizer streams remain caller-owned. Training checkpoints use OneModel's streamed loader rather than a local checkpoint-format implementation. A follow-up **controlled, interleaved indicative comparison** supersedes the earlier single-process observations, which compared different machine loads and suggested a 2.5–3× slowdown. Each version uses the same console harness over the source model projects, loading one German pipeline with `Load("de")` and calling `Process` on “Angela Merkel besuchte gestern die Universität in Heidelberg.” The first-process interval includes graph/head loading and the first inference, but **not language detection**; it is not the `Nlp.Analyze` workload in the preceding table. Both builds embed the same current weight files, verified against the same manifests, including the recovered German parser head. Filesystem caches were warm. Each cell below is the **median of three fresh processes [minimum, maximum]**. The versions alternate within each model's pair, reversing order in the second repetition. Loading is timed separately from first inference; RSS and total process elapsed time come from `/usr/bin/time -v`. Builds finish before the measurements start. | Model | OneModel | Load pipeline | First process | Peak RSS | Process elapsed | |---|---|---:|---:|---:|---:| | `.Multilingual.Large`, German | 1.0.0 | 2,189 [2,104, 2,520] ms | 2,401 [2,324, 2,734] ms | 4,027,688 [4,026,976, 4,035,176] KiB | 2.70 [2.62, 3.02] s | | `.Multilingual.Large`, German | 1.2.0 | 2,319 [2,082, 2,467] ms | 2,554 [2,293, 2,713] ms | 4,001,736 [3,995,144, 4,008,996] KiB | 2.88 [2.57, 3.01] s | | `.Multilingual`, German | 1.0.0 | 959 [885, 1,203] ms | 1,116 [1,037, 1,363] ms | 816,136 [813,972, 830,876] KiB | 1.21 [1.10, 1.45] s | | `.Multilingual`, German | 1.2.0 | 959 [767, 1,251] ms | 1,116 [915, 1,412] ms | 660,048 [658,404, 660,796] KiB | 1.18 [0.97, 1.46] s | The earlier 2.5–3× startup difference **does not reproduce**: median first-process ratios (1.2.0 / 1.0.0) are **1.064** for the base-size model and **1.000** for the distilled model, with overlapping ranges. Three repetitions do not establish equivalence or rule out smaller differences. Temporary load-boundary timestamps in a separate diagnostic batch found German head loading at 67–83 ms before and 61–100 ms after; the much larger variation was in encoder loading, and its direction varied between repetitions. There is no evidence here for a checkpoint-loading fix. No loading code, model bytes, public API or format was changed for this follow-up. The distilled model's median peak falls by **152.4 MiB**. The base-size model shows no substantial memory saving: **its six resource pieces still require a full serialized-graph buffer**. OneModel's in-place stream API recognizes a single `UnmanagedMemoryStream`, not a concatenation of resources. The three-shard multilingual E5 embedder has the same limitation. No tensors, weights or package layout changed to work around it. These observations were made on 2026-11-08, Linux x64, AMD Ryzen 9 7950X3D (16 cores/32 logical CPUs; AVX2 and AVX-512 available), SDK 10.0.403/runtime 10.0.14, CPU only. Four logical processors were exposed; graph sessions used their default four intra-op threads and eager components used one. The measurement job reserved four cores, **not the whole machine**: unrelated queued jobs overlapped it. The following one-minute load samples were taken before/during/after each process (during sampled every 250 ms). Interleaving matches the load much better than the old observations, but **this is not an exclusive-machine benchmark or an isolated latency claim**. An attempted fully idle window was unavailable; no startup speed improvement is claimed. | Model | Repetition | 1.0.0 load: before / during / after | 1.2.0 load: before / during / after | |---|---:|---|---| | `.Multilingual` | 1 | 8.51 / 8.51 / 8.51 | 8.51 / 8.51 / 8.51 | | `.Multilingual` | 2 | 8.43 / 8.43 / 8.43 | 8.47 / 8.43–8.47 / 8.43 | | `.Multilingual` | 3 | 8.64 / 8.64 / 8.64 | 8.64 / 8.64 / 8.64 | | `.Multilingual.Large` | 1 | 8.51 / 8.47–8.51 / 8.47 | 8.47 / 8.47 / 8.47 | | `.Multilingual.Large` | 2 | 8.43 / 8.43–8.64 / 8.64 | 8.43 / 8.43 / 8.43 | | `.Multilingual.Large` | 3 | 8.64 / 8.64–8.75 / 8.75 | 8.75 / 8.75–8.93 / 8.93 | The German annotation JSON, including confidences, is byte-identical across all six processes per model. SHA-256: distilled `e7bc02dfa37b9ab61feeda8cfdbf9ccc31b00f2d5da3003717655b2243d2e0e4`; base-size `34bcb371f68193c4a9f5b92c8f611cfce0058a496939fdfd5b67c0a98460d363`. A JIT consumer of the packed distilled model, with direct OneModel/OneModel.Training 1.2.0 references, gives the same distilled hash. Stream ownership, checkpoint, training/resume and embedder tests pass (51 tests, no failures or skips). Exact-output checks across these runtime versions found: - English fast-tier predictions on **2,077 EWT r2.18 test sentences / 25,094 gold words**: byte-identical JSON in gold-token/sentence and raw-text modes, including tags, morphology, lemmas, dependency heads/labels, entities and confidences. Thus UPOS and LAS inputs are unchanged, not merely within tolerance. - Both German accurate pipelines on the nine-word probe above: byte-identical annotation JSON, including confidences. - BGE sentence embeddings on **2,103 fixed texts**: byte-identical float32 vectors across versions, zero token-ID mismatches against cached sentence-transformers **6.2.0** references, and the same maximum absolute reference difference, **4.02797e-7**, before and after. The English embedded model's process peak was 454,104 → 323,116 KiB (four single-text probes; process elapsed 0.40 → 0.77 s, likewise not isolated). - Multilingual E5 on the same **2,103-text** corpus: float32 bytes and token IDs match the cached pre-update packaged-model run exactly. The unchanged maximum absolute difference from sentence-transformers 6.2.0 / PyTorch 2.15.0 CPU is **3.203749656677246e-7**; zero token-ID mismatches. Its resource shards remain on the buffered loading path. - The fixed-seed toy training recipe retained its 1.0.0 checkpoint SHA-256, `50ecab9ecc658026aba8868555f658010006f59d6093685b347073f742764d25`, at one and four processors. Resume still reproduces the uninterrupted checkpoint bytes; warm start and restore-best paths pass. The source-linked sample consumer and the base-size consumer pass on JIT and NativeAOT with no trim/AOT warnings. The latter loads all 84 language heads and publishes all **75 copied head assets**. These checks do not replace validation of the final packed release. ## Evaluation conventions and gaps - Gold-token and raw-text rows are distinct. UD scoring uses the same public metrics tool, 1,000 sentence/document bootstrap resamples, seed 20261106. Published references have no paired predictions: their value is compared with our upper interval bound, not a paired difference. - Stanza 1.5.1, UDPipe 2 and Trankit published rows use older UD releases; each cell records the release and split-change flag. A changed split is information, not a claim of identical evaluation data. Only published numbers are used for non-commercial reference models. - Entity scores use deterministic evenly spaced samples of at most 3,001 test sentences per language. The paired spaCy comparison recreates the exact sample indices and checks sentence/token alignment. Corpora differ by language; there is no pooled entity gate. - XPOS and AllTags are reported when a treebank contains any XPOS annotation; Turkish-BOUN has only partial XPOS annotation. Metrics are not removed because they score poorly. - Entirely blank gold columns are not scored: lemmas in Maltese-MUDT, Old Occitan-CorAG, Telugu English-TECT, Telugu-MTG and Thai-TUD; features in Korean-Kaist and Vietnamese-VTB. The score tool filters those absent-column rows without changing any retained interval or model prediction. - Missing annotations/heads and missing references are not perfect accuracy or implicit passes. Hindi has no allowed accurate-tier UD training split; the complete excluded-language list and reasons remain in the licence/coverage documentation. - The original nine were not protected by per-language resampling boosts in this expansion. Their before/after intervals, including regressions, are retained alongside all newly covered languages. # The accurate tier's encoder: transformers on OneModel How an exported transformer becomes an `IEncoder`, what was measured, and what can and cannot be trained. Tools are in `tools/accurate/` and `tools/OneNlp.Accurate/`; the recipe is `recipes/accurate/en-tagging/`. ## Export `tools/accurate/export_onnx.py` runs PyTorch and Hugging Face `transformers` through their documented APIs (`AutoModel.from_pretrained`, `torch.onnx.export`) and writes one ONNX file per model with - inputs `input_ids` and `attention_mask`, `int64`, `[batch, sequence]` (both axes dynamic); - output `last_hidden_state`, `float32`, `[batch, sequence, width]`; - opset 17, the TorchScript-based exporter (`dynamo=False`), eager attention so attention is plain `MatMul`/`Softmax`; - a `.export.json` beside it with the repository revision, library versions, file sizes and SHA-256. `--tiny` builds a small random model from a repository's configuration, and `make_tiny_fixture.py` generates the committed test fixture (`testdata/accurate/`: a random 2-layer BERT-style encoder of width 16, a hand-made WordPiece `tokenizer.json`, and PyTorch's hidden states for fixed ids). The fixture is ours; it is not derived from a released model. Pins: torch 2.15.0 (CPU), transformers 5.22.1, onnx 1.23.2, onnxruntime 1.31.1. | Model | Repository revision | ONNX size | SHA-256 | |---|---|---|---| | `xlm-roberta-base` | `e73636d4f797dec63c3081bb6ed5c7b0bb3f2089` | 1,110,013,637 | `fb0644d3c12460c1e7dc7c9797224aafb01be59f9e0f9cd94a6e221ffed77b7b` | | `jhu-clsp/mmBERT-base` | `c5955035435e2bf121cde7f3c8863ef52ff35d82` | 1,228,334,485 | `6a9f485066d95de7eec7fb29c0e3839a9c4ddbefd6f87d46ece33b2cb52a550b` | ## Inference parity OneModel (fast mode, AVX-512 kernel path) against PyTorch float32 on 200 multilingual texts (150 Tatoeba sentences, 50 UDHR paragraphs, cut at 400 characters; the same records for both models, chosen by `select_texts.py`), tokenized by our own `tokenizer.json` loader, one text per run. Hidden states compared for every token. | Model | Tokens | max abs | mean abs | min cosine | mean cosine | |---|---:|---:|---:|---:|---:| | xlm-roberta-base | 4,764 | 6.3e-5 | 2.9e-7 | 1.00000000 | 1.00000000 | | mmBERT-base | 5,255 | 3.5e-3 | 2.3e-6 | 0.99999996 | 1.00000000 | mmBERT's hidden states reach 40.7 in magnitude (XLM-R's 18.6), so its largest difference is about 1e-4 relative. Neither graph needed an operator OneModel lacks. Indicative latency, 8 threads, sequence length 64, median of 20 after 3 warm-ups, against ONNX Runtime 1.31.1 on the same file (a shared host, load average 2 to 5; not a gated row): | Model | Batch | OneModel | ONNX Runtime | |---|---:|---:|---:| | xlm-roberta-base | 1 | 25.0 ms | 21.8 ms | | xlm-roberta-base | 8 | 122 ms | 135 ms | | mmBERT-base | 1 | 42.7 ms | 40.4 ms | | mmBERT-base | 8 | 268 ms | 203 ms | ## `TransformerEncoder` `TransformerEncoder(modelPath, tokenizerPath, pooling = Mean, maxLength = 512)` loads the ONNX file and `tokenizer.json`. 1. The whole document text is tokenized once without special tokens; pieces carry UTF-16 offsets into `Doc.Text`. 2. The special tokens the tokenizer wraps a text in are found by encoding a probe text, then added around every window. 3. A document longer than `maxLength` (default 512) is cut into windows that overlap by half. A piece takes its vector from the window whose centre is nearest, so no piece is read near a window edge when another window reads it better. 4. A piece belongs to every token whose character range it overlaps. `Mean` (the default) averages the vectors of those pieces; `FirstPiece` takes the first one. On English-EWT development data a frozen mmBERT tagger scored 96.39 UPOS with `Mean` and 95.62 with `FirstPiece`. A token no piece overlaps (a character the normalizer drops) gets a zero vector. 5. Windows from all documents of a call run together in batches of `WindowBatch` (default 8), longest first so padding stays small. 6. Outputs are cached per `Doc` instance, so repeated epochs over the same documents run the transformer once. The encoder is **frozen**: `Weights` is `null`, `Train` leaves it alone, and heads train on its output. A saved pipeline **references** the encoder files by name and optional SHA-256 (`docs/model-files.md`). ## Head training on a frozen encoder `recipes/accurate/en-tagging` trains the existing `Tagger` (UPOS, one linear layer) on UD English-EWT r2.18 (train for training, dev for selection) over the frozen encoder; see its `PROVENANCE.md` for the results. ## Fine-tuning the encoder The graph runtime and the autodiff tensor layer are separate: `Session.Run` returns tensors with `RequiresGrad == false`, so an imported graph cannot be backpropagated. `TrainableTransformerEncoder` therefore rebuilds the encoder from the exported file's weights in modules written on OneModel's tensor ops, so `Train` updates the encoder together with the head. `TransformerEncoder` (above) stays the frozen, cached inference path. ```csharp using var encoder = TrainableTransformerEncoder.LoadOnnx("mmbert-base.onnx", "tokenizer.json"); // family detected from the weights using var pipeline = Pipeline.Blank(Language.English).Add(new Tagger(encoder, TokenColumn.Pos)); pipeline.Train(examples, new TrainOptions { LearningRate = 5e-5f, WarmupSteps = 40, BatchWords = 500 }); PipelineFile.Save(pipeline, "tagger.onenlp"); // kind "transformer-trainable": settings plus the layers' weights ``` ### Encoder kinds in a saved pipeline | Kind | Class | Trains | Weights in the pipeline file | |---|---|---|---| | `transformer` | `TransformerEncoder` | no (frozen ONNX graph) | none; file names and hash only | | `transformer-trainable` | `TrainableTransformerEncoder` | yes | the layers (not the frozen token table); `PipelineFile.Save` to a path replaces it by `transformer` over a written graph (see below) | Both reference the exported model and tokenizer by name (`TransformerEncoder.ModelDirectory`). The trainable kind reads the token table from the exported model again on load and overwrites the layers from the checkpoint, so the model file must still be there. ### Families The family is read from the weight names (`TransformerFamily.Auto`) or given. Weights come from graphs written by the PyTorch exporter without dynamo (`torch.onnx.export(..., dynamo=False)`), where each linear layer's weight is the constant of the `MatMul` node named after the module; other graphs are rejected with a message. - **`ModernBert`** (mmBERT, ModernBERT; `ModernBertModule`). Warner et al. 2024 (arXiv 2412.13663) §2 and the mmBERT paper (arXiv 2509.06888): token embedding, layer norm, then pre-norm layers `x = x + Wo(attention(rotary(Wqkv(norm(x)))))` and `x = x + Wo2(gelu(a) * b)` with `[a, b] = Wi(norm(x))` (GeGLU, Shazeer 2020, arXiv 2002.05202), a final norm. No biases, norms without shift, epsilon 1e-5; the first layer has no attention norm. Layer `i` is global when `i % 3 == 0`; the others see tokens at most 64 positions away (a 128-token window). Rotary embeddings (Su et al. 2021, arXiv 2104.09864) pair coordinate `i` with `i + d/2`, base 160,000 in mmBERT's global and local layers (ModernBERT: 160,000 and 10,000). Details the papers leave open were settled against the exported graph of mmBERT-base and checked against PyTorch: `Wqkv` is laid out `[3, heads, d]`, the first half of `Wi` goes through GELU, the window test is `|i - j| <= 64`, attention scale `1/sqrt(d)`. - **`PostNorm`** (XLM-R, RoBERTa, BERT; `PostNormModule`). Token and learned position embeddings (the single token-type vector is added into every position row), layer norm, and post-norm layers with exact GELU. XLM-R starts positions at 2 (`PositionOffset`), BERT at 0; epsilon is read from the graph (1e-5 and 1e-12). The head count is not in the weights: it defaults to width / 64 (set `Heads` otherwise). A graph with two different rotary bases does not say which table is the global one, so `GlobalRopeTheta` and `LocalRopeTheta` must be given for it (the loader refuses to guess); a graph with one base uses it for both. ### Pooling and the embedding table Pieces become tokens as in `TransformerEncoder` (same tokenizer, windows and `Mean`/`FirstPiece`), but as a differentiable gather: each token's vector is the sum of the rows of its pieces, scaled by `1/count`, so gradients reach every piece. Windows of up to `MaxBatchPieces` padded pieces (default 8,192) run together; padding keys are masked. `FreezeEmbeddings` (default `true`) keeps the 256,000 x 768 token table (197 million values in mmBERT) out of training and out of the saved weights. Fine-tuning a tagger barely moves it, and dense AdamW over it would add two moment tensors and a 786 MB gradient to every step. With `false` the table trains, and a pipeline holding it cannot be saved (the pipeline file's weights are capped at 1 GiB, and `WriteSettings` says so). `Train` has one learning rate for the encoder and the head. 5e-5 with a short warm-up is what the recipe uses. ### Tests `tests/OneNlp.Tests/TrainableTransformerTests.cs` uses two random tiny encoders exported from PyTorch (`tools/accurate/make_tiny_fixture.py`, `make_tiny_modernbert_fixture.py`; the second has alternating global and local layers with a window of 8, two rotary bases and padding): modules against PyTorch's hidden states (max abs below 2e-5), padded batches against single sequences, token vectors against `TransformerEncoder`'s for both poolings and with windowing, finite-difference gradients through rotary, the gated feed-forward and the window mask, a tagger trained through `Train` and reloaded from a saved pipeline, and a resumed run reproducing an uninterrupted one bit for bit. ### Measured Forward parity of the trainable modules (one text per run, fast mode, the same 200 texts as above; `tools/OneNlp.Accurate trainable-parity`). "Graph runtime" is `TransformerEncoder`'s inference path on the exported file. | Model | Tokens | Comparison | max abs | mean abs | min cosine | |---|---:|---|---:|---:|---:| | mmBERT-base (`ModernBert`) | 5,255 | trainable vs PyTorch | 1.0e-3 (2.6e-5 of the largest value, 40.7) | 2.2e-6 | 1.00000000 | | | | trainable vs graph runtime | 4.6e-3 | 1.9e-6 | 0.99999993 | | | | graph runtime vs PyTorch | 3.5e-3 | 2.3e-6 | 0.99999996 | | xlm-roberta-base (`PostNorm`) | 4,764 | trainable vs PyTorch | 7.3e-5 | 2.7e-7 | 1.00000000 | | | | trainable vs graph runtime | 4.0e-5 | 2.1e-7 | 1.00000000 | | | | graph runtime vs PyTorch | 6.3e-5 | 2.9e-7 | 1.00000000 | Batches of 8 right-padded sequences gave exactly the rows of the sequences run alone. The trainable module is closer to PyTorch than the imported graph is. One fine-tuning step (24 sequences of 32 pieces, 8 threads, 110,344,721 trainable parameters, frozen token table, AdamW, gradient clipping), C# against plain PyTorch 2.15.0 CPU on the same batch (`step-time` and `tools/accurate/torch_step_time.py`; indicative, shared host at load average about 25, steady-state steps 1 to 3): | | forward | backward | clip | AdamW | total | |---|---:|---:|---:|---:|---:| | OneModel | 0.70 s | 1.27 s | 0.33 s | 0.34 s | 2.62 s | | PyTorch | 0.47 s | 0.80 s | 0.06 s | 0.20 s | 1.52 s | The C# step takes 1.7 times the PyTorch step, inside the 3x rule. Gradient clipping (about 6 times) and the AdamW update (1.7 times) are the slowest parts relative to PyTorch. The full recipe ran at about 6.6 s per step on batches of 500 words (about 700 pieces), peak resident memory 11 GB. Fine-tuning proof (`recipes/accurate/en-tagging`, `--finetune`; `PROVENANCE.md`): one 26-minute chunk, 236 steps (0.58 epoch of UD English-EWT r2.18 train), development UPOS 96.894 at step 200 (the frozen encoder reached 96.39 on development), test UPOS 97.07 [96.79, 97.35], 0.84 [0.62, 1.07] above the frozen encoder's 96.23 and 2.78 [2.44, 3.16] above spaCy's 94.29. ## Fine-tuned weights written back to ONNX The trainable modules are about three times slower at inference than OneModel's graph runtime on the same weights, so a fine-tuned encoder is not served by them. `TrainableTransformerEncoder.ExportOnnx(path)` writes the fine-tuned weights into an ONNX file of the graph the encoder was loaded from, and `PipelineFile.Save(pipeline, path)` does it for every fine-tuned transformer in the pipeline. ```csharp PipelineFile.Save(pipeline, "tagger.onenlp"); // also writes tagger.encoder0.onnx beside it using var loaded = PipelineFile.Load("tagger.onenlp"); // the encoder is a TransformerEncoder on the written graph ``` - **What is written.** A copy of the original export in which only the values of the layers' weights (and the token table, if it trained) differ. The graph, its operators, constants and every other byte are unchanged. The file is as large as the original (about 1.2 GB for mmBERT-base, 142 MB for the student), is written beside the pipeline file as `.encoder.onnx` and is referred to by that plain file name, looked up in the pipeline file's directory. The two files travel together. Loading from a stream cannot find it and says so. - **How.** `OnnxWeightPatcher` (no dependency) indexes the file's protobuf fields with reads at offsets (`onnx.proto3`: ModelProto.graph, GraphProto.node and .initializer, TensorProto dims, data_type, name, raw_data; the protobuf encoding specification), then writes the output field by field: untouched fields are copied as byte ranges, a replaced initializer is re-encoded with its new `raw_data`. Linear weights go back to the graph's `[in, out]` layout. A BERT-family graph adds one token-type vector to every position row; the position table is written with it folded in and the token-type table zeroed, which gives the graph the same sums. The PyTorch exporter stores equal tensors once and names the others with `Identity` nodes (the zero biases of a fresh model): a replacement for such a name becomes an initializer and the node is dropped; a tensor shared with a name that is not being replaced is refused. Tensors in external data files, non-float32 weights and results over 2 GiB are refused with `NotSupportedException`. - **Bounds.** Every length is compared with the bytes left in its parent before it is read; names are at most 4 KiB; a node larger than 64 KiB is never read; at most 4 million fields per message; the output is written under a temporary name and renamed, and nothing is left behind on failure. - **Keeping the weights in the pipeline file.** `PipelineFile.Save(pipeline, path, exportEncoderGraphs: false)`, or saving to a stream, stores them inside the file as before. The pipeline then loads as a trainable encoder: it can be trained further, and it runs on the slower modules. Pipeline files written before this existed still load. - **Token table.** An encoder that trained its token table can be saved this way too (the table goes into the graph, not the pipeline file), which the 1 GiB cap of the pipeline file previously forbade. ### Measured Hidden states of the written graph against the trainable modules the weights came from, on the 200 parity texts above (5,255 tokens, one text per run, 8 threads, fast mode), and ONNX Runtime 1.31.1 (CPU, black box) on the same written file against the modules' states (`tools/OneNlp.Accurate export-parity`, `tools/accurate/ort_hidden_compare.py`, `tools/accurate/export-rows.sh`): | Encoder | Comparison | max abs | largest value | min cosine | |---|---|---:|---:|---:| | student, fine-tuned (`r3-upos-en`) | OneModel graph vs modules | 6.5e-5 | 4.5 | 1.00000000 | | | ONNX Runtime vs modules | 1.9e-4 | 4.5 | 0.99999976 | | mmBERT-base, fine-tuned 25 steps | OneModel graph vs modules | 6.9e-4 | 40.3 | 0.99999999 | | | ONNX Runtime vs modules | 1.2e-3 | 40.3 | 0.99999976 | UPOS on EWT r2.18 test (gold tokens): the student scores 95.88 [95.55, 96.19] through the written graph, as before, with no token whose prediction differs from the inline pipeline's (25,094 tokens). The mmBERT run, 25 steps from the frozen start (a smoke fine-tune, not a recipe), scores 90.67 [90.22, 91.09] through the written graph and also has no differing token. Indicative speed, tagger over the first 400 sentences of the EWT test split (6,305 tokens), batches of 32 documents, median of 2 repetitions after a warm-up, shared host (load 10 to 16), not a gated row: | Encoder | Threads | Trainable modules | Written graph | Ratio | |---|---:|---:|---:|---:| | student | 1 | 811 tok/s | 2,690 | 3.3 | | student | 8 | 1,914 | 11,474 | 6.0 | | mmBERT-base | 1 | 99 | 270 | 2.7 | | mmBERT-base | 8 | 268 | 983 | 3.7 | Tests: `tests/OneNlp.Tests/OnnxExportTests.cs` (the three fixture families: perturbed weights give the modules' vectors from the graph, unchanged weights give the original's, a trained table, hostile files, the paired pipeline file and its inline alternative). Fuzz target `onnx-patch` (`fuzz/README.md`). ## The distilled student The default small encoder is distilled from mmBERT-base (MIT) by `tools/distill/`, with PyTorch as development tooling only. Nothing there ships; the result is one ONNX file, its `.config.json` and mmBERT's `tokenizer.json`, treated by the library like any other exported encoder. **Architecture** (`tools/distill/student.py`): six post-norm transformer layers of width 384 (12 heads, feed-forward 1536, exact GELU), learned absolute positions (512), and a factorised token embedding: a 256,000 x 96 table lifted to 384 by a linear map (Lan et al. 2020, section 3.1), so the vocabulary costs 24.6 M of the 35.5 M weights. It uses mmBERT's tokenizer unchanged, so token ids equal the teacher's for every text and no vocabulary is pruned. **Objective** (`tools/distill/distill.py`): hidden-state distillation (Jiao et al. 2020, section 3.1, Eq. 8). Each student layer's output goes through its own learned linear map and is matched by mean squared error to a teacher layer: the uniform mapping (section 4.6) sends student layers 1 to 5 to teacher hidden states 4, 7, 11, 15 and 18, and the last layer to the teacher's final hidden state, standardised per feature, with a cosine term. Intermediate targets are layer-normalised per token because the teacher's residual stream has a few very large dimensions. The token table starts from the teacher's, reduced to 96 columns by principal components. Data: FineWeb-2 and Wikipedia text in 113 languages, sampled with language weights proportional to the square root of the amount of text, windows of 128, 256 and 512 pieces (`recipes/accurate/student-v1/PROVENANCE.md`). **Export** (`tools/distill/export_student.py`): the same graph shape as the other exported encoders (inputs `input_ids`, `attention_mask`; output `last_hidden_state`; opset 17, attention as MatMul/Softmax), written from the exponential moving average of the weights. Every linear weight is a named initializer (`model.encoder.layer.N.attention.query.weight` and so on, stored `[in, out]`) and the head count is in `.config.json`. **Fine-tuning.** `TrainableTransformerEncoder` (family `PostNorm`) reads this layout as well as the PyTorch exporter's: when the graph has `embeddings.projection.weight` the table is factorised (`PostNormConfig.EmbeddingWidth` = 96, a `projection` linear layer), the weights are read by name, positions start at 0, and the head count comes from `Heads` or from the `.config.json` beside the model (a student graph without either is refused; a saved pipeline stores it). `tools/OneNlp.Accurate trainable-parity --family postnorm --position-offset 0` on the 200 parity texts (5,255 tokens): trainable modules against PyTorch max abs 2.6e-5 (largest value 4.4), graph runtime against PyTorch 2.4e-5, padded batches identical to single sequences. # Accurate-tier corpus The languages the accurate tier is trained on, as `recipes/accurate/common/corpus.json` describes them (written by `tools/accurate/make_corpus_spec.py` from the fetched files; licence verdicts in `ud-licences.md`). Release r2.18 of Universal Dependencies. - **84 languages** with allowed training data; 9 of them keep the data of the earlier nine-language run, so their rows stay comparable. - **605,763 training sentences, 9,882,945 words** (before the dev hold-out). - **19 languages with an entity corpus** (21 sets). - **5 languages need a trained word segmenter** in front for raw text: Japanese (ja), Classical Chinese (lzh), Thai (th), Vietnamese (vi), Chinese (zh). - **17 small languages** (under 500 training sentences in all; the sampler caps their repetition). - **Dev split:** 67 languages have one in the release; 7 hold out every 10th training sentence; 10 (fewer than 200 sentences, no dev split) train on all of it and are not seen by model selection. ## Languages Words are syntactic words of the training files. Extra treebanks are trained on and never scored; the language's test set is its gating treebank's test split. *Leaks* are extra-treebank training sentences that repeat a gating test sentence; the reader drops them. | Code | Language | Gating treebank (train sentences / words) | Extra treebanks (train sentences) | Dev | Entities (train sentences) | Segmenter | Small | Lemma table (dev) | |---|---|---|---|---|---|---|---|---| | af | Afrikaans | Afrikaans-AfriBooms (1,315 / 33,878) | | release | | | | 1:0.6 (86.7% at 99.9%) | | as | Assamese | Assamese-AiW (51 / 597) | | release | | | yes | 1:0.6 (45.5% at 97.7%) | | bor | Bororo | Bororo-BDT (17,107 / 128,418) | | release | | | | none | | bxr | Buryat | Buryat-BDT (20 / 158) | | none | | | yes | none | | ca | Catalan | Catalan-AnCora (13,123 / 429,579) | | release | | | | 1:0.6 (94.9% at 99.6%) | | ckb | Central Kurdish | Central_Kurdish-Mukri (92 / 539) | | release | | | yes | 2:1.0 (42.3% at 97.7%) | | cop | Coptic | Coptic-Scriptorium (1,467 / 37,639) | Coptic-Bohairic (477) | release | | | | 1:0.8 (79.0% at 98.5%) | | cs | Czech | Czech-CAC (23,478 / 472,609) | Czech-CLTT (467) | release | | | | 1:0.6 (91.0% at 99.9%) | | cy | Welsh | Welsh-CCG (1,355 / 28,563) | | release | | | | 1:0.6 (87.0% at 99.0%) | | da | Danish | Danish-DDT (4,383 / 80,378) | | release | uner-da-ddt (4,383) | | | 1:0.6 (86.3% at 99.8%) | | de | German | German-GSD (13,813 / 263,777) | | release | germeval-2014 (24,000) | | | 1:0.6 (89.1% at 99.2%) | | egy | Egyptian | Egyptian-PC (1,619 / 19,486) | | release | | | | 1:0.6 (89.2% at 99.5%) | | el | Greek | Greek-GUD (1,071 / 14,859) | Greek-Cretan (180), Greek-Messinian (60) | release | | | | 1:0.6 (76.1% at 98.1%) | | en | English | English-EWT (12,544 / 204,578) | | release | uner-en-ewt (12,543) | | | 1:0.6 (91.5% at 99.4%) | | es | Spanish | Spanish-AnCora (14,287 / 453,039) | | release | wikiner-es-wp3 (113,959) | | | 1:0.6 (94.5% at 99.8%) | | fa | Persian | Persian-Seraji (4,798 / 121,067) | Persian-PerDT (26,196, 1 leaks) | release | | | | 1:0.6 (95.3% at 98.7%) | | fi | Finnish | Finnish-FTB (14,981 / 127,613) | Finnish-TDT (12,217) | release | | | | 1:0.6 (79.9% at 98.4%) | | fo | Faroese | Faroese-FarPaHC (1,020 / 23,094) | | release | | | | 1:0.6 (12.9% at 100.0%) | | fr | French | French-GSD (14,450 / 354,647) | | release | wikiner-fr-wp3 (119,460) | | | 1:0.6 (92.2% at 99.8%) | | ga | Irish | Irish-IDT (4,005 / 95,881) | | release | | | | 1:0.6 (87.2% at 99.2%) | | gd | Scottish Gaelic | Scottish_Gaelic-ARCOSG (3,544 / 68,775) | | release | | | | 1:0.6 (90.0% at 99.6%) | | gl | Galician | Galician-TreeGal (600 / 15,436) | | held out | | | | 1:0.6 (81.9% at 99.7%) | | grc | Ancient Greek | Ancient_Greek-PTNK (727 / 18,440) | | release | | | | 1:0.6 (81.1% at 100.0%) | | he | Hebrew | Hebrew-IAHLTwiki (4,298 / 120,832) | Hebrew-IAHLTknesset (2,362) | release | | | | 1:0.6 (91.6% at 99.5%) | | hr | Croatian | Croatian-SET (6,914 / 152,857) | | release | uner-hr-set (6,914) | | | 1:0.6 (85.5% at 99.5%) | | ht | Haitian Creole | Haitian_Creole-Adolphe (2,605 / 55,527) | | release | | | | 1:0.6 (97.5% at 99.3%) | | hy | Armenian | Armenian-ArmTDP (4,352 / 82,871) | Armenian-BSUT (1,451, 1 leaks) | release | | | | 1:0.6 (82.6% at 99.7%) | | hyw | Western Armenian | Western_Armenian-ArmTDP (5,274 / 95,823) | | release | | | | 1:0.6 (79.3% at 99.6%) | | id | Indonesian | Indonesian-GSD (4,482 / 97,602) | Indonesian-CSUI (656) | release | uner-id-gsd (4,482), uner-id-csui (656) | | | 1:0.6 (86.8% at 99.4%) | | is | Icelandic | Icelandic-IcePaHC (34,007 / 705,747) | Icelandic-GC (3,960), Icelandic-Modern (2,703, 1 leaks) | release | | | | 1:0.6 (93.7% at 98.3%) | | it | Italian | Italian-ParlaMint (897 / 28,881) | | release | wikiner-it-wp3 (115,965) | | | 1:0.6 (88.1% at 98.2%) | | ja | Japanese | Japanese-GSD (7,050 / 168,333) | | release | | character-tagger | | 1:0.6 (92.0% at 99.7%) | | ka | Georgian | Georgian-GLC (2,214 / 40,530) | Georgian-GNC (918) | release | | | | 1:0.6 (75.1% at 97.7%) | | kk | Kazakh | Kazakh-KTB (31 / 529) | | none | | | yes | none | | kmr | Northern Kurdish | Northern_Kurdish-Kurmanji (20 / 242) | | none | | | yes | none | | ko | Korean | Korean-Kaist (23,010 / 296,446) | Korean-GSD (4,400), Korean-KSL (14,076) | release | | | | 1:0.6 (75.2% at 98.0%) | | koi | Komi Permyak | Komi_Permyak-UH (34 / 248) | | none | | | yes | none | | kpv | Komi Zyrian | Komi_Zyrian-Lattice (20 / 158) | | none | | | yes | none | | ky | Kyrgyz | Kyrgyz-KTMU (1,308 / 12,357) | | held out | | | | 1:0.8 (71.1% at 97.5%) | | la | Latin | Latin-LLCT (7,289 / 194,143) | Latin-CIRCSE (918) | release | | | | 1:0.6 (94.4% at 99.7%) | | lij | Ligurian | Ligurian-GLT (20 / 226) | | none | | | yes | none | | lt | Lithuanian | Lithuanian-ALKSNIS (2,341 / 47,641) | Lithuanian-HSE (153) | release | | | | 1:0.6 (72.5% at 99.4%) | | lv | Latvian | Latvian-LVTB (15,059 / 258,266) | | release | | | | 1:0.6 (85.4% at 99.0%) | | lzh | Classical Chinese | Classical_Chinese-Kyoto (74,609 / 374,560) | | release | | character-tagger | | 1:0.6 (95.5% at 100.0%) | | mdf | Moksha | Moksha-JR (22 / 226) | | none | | | yes | none | | mr | Marathi | Marathi-UFAL (373 / 2,997) | Marathi-CMUPAN (7,616) | release | | | | 1:0.9 (68.0% at 97.2%) | | mt | Maltese | Maltese-MUDT (1,123 / 22,880) | | release | | | | none | | myv | Erzya | Erzya-JR (1,217 / 10,294) | | held out | | | | 1:0.6 (73.9% at 98.8%) | | nds | Low Saxon | Low_Saxon-LSDC (500 / 11,666) | | held out | | | | 1:0.6 (79.3% at 99.5%) | | nl | Dutch | Dutch-Alpino (12,289 / 186,027) | | release | wikiner-nl-wp3 (163,725) | | | 1:0.6 (86.4% at 99.0%) | | no | Norwegian | Norwegian-Bokmaal (15,696 / 243,886) | Norwegian-Nynorsk (14,174, 4 leaks) | release | uner-no-bokmaal (15,696), uner-no-nynorsk (14,174) | | | 1:0.6 (85.7% at 99.6%) | | olo | Livvi | Livvi-KKPP (19 / 144) | | none | | | yes | none | | orv | Old East Slavic | Old_East_Slavic-RNC (3,778 / 101,125) | Old_East_Slavic-Birchbark (1,113), Old_East_Slavic-Ruthenian (4,120) | release | | | | 1:0.8 (72.1% at 97.1%) | | ota | Ottoman Turkish | Ottoman_Turkish-BOUN (114 / 1,997) | Ottoman_Turkish-DUDU (1,117) | release | | | | 1:0.6 (63.2% at 99.3%) | | pa | Punjabi | Punjabi-Rang (81 / 1,212) | Punjabi-CS (73) | release | | | yes | 2:0.9 (50.6% at 97.1%) | | pcm | Naija | Naija-NSC (7,279 / 111,916) | | release | | | | 1:0.6 (96.9% at 99.7%) | | pl | Polish | Polish-MPDT (1,433 / 33,520) | | release | kpwr-ner-n82 (12,570) | | | 1:0.6 (74.8% at 99.6%) | | pro | Old Occitan | Old_Occitan-CorAG (1,092 / 37,926) | | release | | | | none | | pt | Portuguese | Portuguese-Bosque (7,018 / 171,776) | | release | uner-pt-bosque (7,018) | | | 1:0.6 (90.3% at 99.6%) | | qte | Telugu English | Telugu_English-TECT (61 / 290) | | none | | | yes | none | | ro | Romanian | Romanian-RRT (8,043 / 185,106) | Romanian-SiMoNERo (3,747) | release | ronec-2.0 (9,000) | | | 1:0.6 (91.6% at 99.7%) | | ru | Russian | Russian-GSD (3,850 / 74,900) | Russian-Poetry (3,568) | release | | | | 1:0.6 (77.0% at 99.7%) | | sa | Sanskrit | Sanskrit-Vedic (21,477 / 161,959) | | release | | | | 1:0.6 (85.0% at 99.2%) | | say | Zaar | Zaar-Autogramm (978 / 10,712) | | held out | | | | 1:0.8 (85.4% at 97.6%) | | sd | Sindhi | Sindhi-Isra (4,096 / 74,589) | | release | | | | 1:0.6 (89.7% at 100.0%) | | sdh | Southern Kurdish | Southern_Kurdish-Garrusi (152 / 1,177) | | release | | | yes | none | | sk | Slovak | Slovak-SNK (8,483 / 80,874) | | release | uner-sk-snk (8,483) | | | 1:0.6 (65.5% at 99.5%) | | sl | Slovenian | Slovenian-SSJ (10,903 / 215,155) | Slovenian-SST (5,094) | release | uner-sl-ssj (10,903) | | | 1:0.6 (88.6% at 99.8%) | | sme | North Sami | North_Sami-Giella (2,257 / 16,835) | | held out | | | | 1:0.6 (75.9% at 99.6%) | | sms | Skolt Sami | Skolt_Sami-Giellagas (20 / 153) | | none | | | yes | none | | sq | Albanian | Albanian-STAF (160 / 2,879) | | release | | | yes | 2:0.6 (53.2% at 97.6%) | | sr | Serbian | Serbian-SET (3,328 / 74,259) | | release | uner-sr-set (3,328) | | | 1:0.6 (83.9% at 99.5%) | | sv | Swedish | Swedish-Talbanken (4,315 / 66,646) | | release | uner-sv-talbanken (4,303) | | | 1:0.6 (87.6% at 99.8%) | | te | Telugu | Telugu-MTG (1,051 / 5,082) | | release | | | | none | | th | Thai | Thai-TUD (2,902 / 62,011) | | release | | character-tagger | | none | | tr | Turkish | Turkish-BOUN (7,803 / 100,713) | | release | turkish-wikiner (17,967) | | | 1:0.6 (75.9% at 97.8%) | | ug | Uyghur | Uyghur-UDT (1,656 / 19,262) | | release | | | | 1:0.6 (57.2% at 99.8%) | | uk | Ukrainian | Ukrainian-ParlaMint (5,611 / 87,369) | | release | | | | 1:0.6 (87.0% at 99.9%) | | uz | Uzbek | Uzbek-UzUDT (483 / 5,442) | | held out | | | yes | 1:0.6 (62.6% at 97.4%) | | vi | Vietnamese | Vietnamese-VTB (1,400 / 20,215) | | release | | character-tagger | | 1:0.6 (80.8% at 99.5%) | | wo | Wolof | Wolof-WTB (1,188 / 23,561) | | release | | | | 1:0.6 (84.5% at 97.9%) | | yi | Yiddish | Yiddish-YiTB (1,877 / 16,458) | | release | | | | 1:0.6 (82.7% at 98.8%) | | zh | Chinese | Chinese-GSDSimp (3,997 / 98,614) | | release | uner-zh-gsdsimp (3,997) | character-tagger | | 1:0.6 (86.4% at 100.0%) | | zza | Zazaki | Zazaki-ZSD (138 / 978) | | release | | | yes | 1:0.6 (71.8% at 100.0%) | Left out of training: Japanese: Japanese-GSDLUW (the same sentences as Japanese-GSD, cut into other word units). ## Languages without an accurate-tier model Every treebank of these languages is ruled out (`ud-licences.md`, "Languages without an accurate-tier model"), so there is no allowed training data: Ancient Hebrew, Basque, Belarusian, Bulgarian, Chintang, Classical Armenian, Estonian, Gothic, Hungarian, Kadiweu, Khoekhoe, Maghrebi Arabic French, Manx, Mbya Guarani, Naga, Nheengatu, Occitan, Old Church Slavonic, Pomak, Ruuli, Shanghainese, South Levantine Arabic, Turkish German, Urdu, Xibe. Hindi has only the test-only PUD treebank. ## Evaluation-only data Test-only treebanks of the trained languages are scored as extra rows and never trained on: Czech-PUD, Czech-Poetry, Greek-Lesbian, Finnish-PUD, Faroese-OFT, Irish-Cadhan, Galician-PUD, Hebrew-PostRab, Haitian_Creole-Autogramm, Indonesian-PUD, Icelandic-PUD, Japanese-PUD, Japanese-PUDLUW, Korean-LittlePrince, Korean-PUD, Komi_Zyrian-IKDP, Kyrgyz-TueCL, Latvian-Cairo, Classical_Chinese-TueCL, Ottoman_Turkish-TueCL, Polish-PUD, Romanian-ArT, Romanian-MolDoRo, Russian-PUD, Sanskrit-UFAL, Albanian-TSA, Swedish-Old, Swedish-PUD, Swedish-SweLL, Thai-PUD, Uzbek-TueCL, Uzbek-UT, Vietnamese-TueCL. # Dependency parser and noun chunks Normative for the `parser` head (`RecursiveAI.OneNlp.Neural.Components.Parser`) and `Doc.NounChunks`. ## Parser A graph-based parser: Dozat and Manning 2017, "Deep Biaffine Attention for Neural Dependency Parsing" (§2), with maximum-spanning-tree decoding (McDonald et al. 2005, using the Chu-Liu/Edmonds algorithm of Chu and Liu 1965 and Edmonds 1967). Chosen over a transition-based parser (Goldberg and Nivre 2012, with pseudo-projective lifting, Nivre and Nilsson 2005) because the loss is a plain differentiable function of the whole sentence (no oracle runs inside the training step), the batch is a few large matrix products, and the decoder returns non-projective trees without a lifting step. The transition-based design was not implemented, so there is no measured comparison. - **Input.** One vector per token from the encoder. Optionally `ContextLayers` residual window layers (`x <- x + ReLU(W [x(i-1); x(i); x(i+1)])`, the ends of a document read zeros), then optionally a bidirectional LSTM (`RecurrentSize` hidden units per direction, `RecurrentLayers` layers) run over each document. The documents of a batch form one padded LSTM batch with their lengths (packed-sequence semantics), so a document's parse does not depend on which others share the batch. The recipe uses no window layers and a 2-layer LSTM of 192 per direction: on the same encoder, window layers alone reached 74.9 dev LAS and the LSTM 79.7 (see the recipe's results). - **Arcs.** Two ReLU projections (of the LSTM output) to `ArcSize` (default 256). The score of token *j* as the head of token *i* is `d(i)' U h(j) + b . h(j)`; column 0 is an artificial root with a learned vector; a token cannot head itself (a fixed -10000 on the diagonal). Training uses the softmax cross entropy over the heads of each token. - **Labels.** Two ReLU projections to `LabelSize` (default 128). The score of relation *c* for the arc (*i*, *j*) is `d(i)' U_c h(j) + W [d(i); h(j)] + b`, trained on the gold arcs with cross entropy. Two options trade accuracy for speed. `labelBilinear: false` drops the matrix `U_c` per relation, which costs `LabelSize` squared times the number of relations multiply-adds per token and was most of the label work. `labelProduct: true` then adds the element-wise product to the linear map, `W [d(i); h(j); d(i) * h(j)] + b`; the product is the diagonal of the bilinear form, so the scorer keeps a multiplicative interaction at three times the width instead of a full matrix per relation. Both are written to the model file only when set, so older files load unchanged. - **Sentences.** A tree is built per sentence. If the document already has sentence boundaries they are used. Otherwise a two-class classifier on the same vectors marks sentence starts and the boundaries are written into the document. Training examples that hold several sentences are what teach it (the recipe groups ten). A sentence longer than 400 tokens is parsed in pieces of 400. - **Decoding.** The maximum spanning arborescence over the arc scores with exactly one token attached to the root: if the free solution has several, each candidate root is forced in turn and the best total score wins. Labels: only the token attached to the root can take the relation `root`, and it takes no other (when the label set holds `root`). - **Labels set.** Every dependency relation in the training examples' reference documents, in ordinal order, subtypes included. - **Use in training.** An example is used only when predicted and reference tokens match one to one. The loss is the sum of arc, label and boundary cross entropies over the batch, divided by the number of tokens. ### Speed configuration The first English parser (two LSTM layers of 192, arc 256, label 128 with a matrix per relation) took 137 microseconds per token on one thread, of which the LSTM was about 80 and the label scorer about 50; the arc scorer, the projections and the tree decoder together were under 20. The shipped parsers use `labelBilinear: false`, `labelProduct: true` and one LSTM layer: | Languages | LSTM per direction | Arc, label size | Why | |---|---|---|---| | English, German, Dutch, Portuguese, Spanish, French | 128 | 128, 64 | the cheapest setting whose dev LAS stayed within about 1 point of the larger parser | | Italian, Chinese | 192 | 192, 96 | the smaller setting lost 2.4 (Italian) and 3.4 (Chinese) dev LAS on these small treebanks; this one lost 0.8 and 1.4 | English dev LAS (UD English-EWT r2.18, gold tokens, selection by dev only), same encoder, seed 7 unless noted: 2 x 192 with a bilinear label scorer 79.66; 1 x 128, arc 128, label 64, product 79.40 (seed 8: 79.06); 1 x 192, arc 192, label 96, product 79.74; 2 x 128 78.74; 2 x 96 78.73; 1 x 160 78.89; 1 x 128 with one window layer 78.49. Seed-to-seed spread on dev is about 0.5, so the differences among the one-layer settings are not resolved by the dev set. The LSTM step is memory-bound at 192 units (a step reads 590 KB of weights per direction) and compute-bound at 128; prediction code is in `LstmInference` and `RowKernels`, unchanged by this configuration. Speed numbers and test rows are in `docs/results.md`. ## Noun chunks `Doc.NounChunks` returns base noun phrases from heads, relations and parts of speech: a NOUN, PROPN or PRON token that is a subject, object, indirect object, oblique, nominal modifier, apposition, nominal predicate (`xcomp`) or the root, or is `conj` in a chain that ends in one of those, plus the words before it in its phrase. Relation subtypes `tmod`, `npmod` and `poss` do not make a phrase head. Phrases never overlap; the next is skipped when its left edge falls inside the previous. The rule was written from spaCy's published description (a flat phrase with a noun as its head, with no nested noun phrase) and its exact conditions settled by black-box comparison with spaCy 3.8.18 on trees built with `Doc(vocab, words, pos, heads, deps)`: - spaCy's English rule answers to a fixed set of relation names (`nsubj`, `nsubjpass`, `dobj`, `dative`, `pobj`, `appos`, `attr`, `pcomp`, `oprd`, `ROOT`, plus `conj`), not to UD names, so the core rule is checked in a profile that uses those names (`NounChunkProfile.ClearStyle`) and the UD profile maps onto it; - a phrase runs from its left edge to the head token; the left edge is the smallest index in the head's subtree; - the second and later `conj` of a chain qualify when the first conjunct's relation does. The Universal Dependencies profile (the one `NounChunks` uses) differs from spaCy's English rule in two ways: a case marker, conjunction, subordinator, copula, auxiliary or clause (`case`, `cc`, `mark`, `cop`, `aux`, `advcl`, `acl`, `ccomp`, `csubj`, `parataxis`, ...) does not extend a phrase to the left, so *of the staff* gives *the staff*; and the phrase extends to the right over `flat`, `goeswith` and `fixed` dependents, so a name written head-first (*Ada Lovelace*) is one phrase. Differential result: `tests/OneNlp.Tests/NounChunkTests.cs` compares with spaCy on 4,000 random trees (crossing arcs included, seed recorded in the fixture) and the 2,001 tree shapes of the EWT r2.18 dev set with relations mapped. Known residue (not in the committed fixture, found in larger runs on other seeds: 3 in 90,000 random trees): spaCy under-reports the left edge of a phrase when arcs cross several times in a row; this occurs in about 1 in 30,000 of the random trees and in none of the treebank trees. Regenerate the fixture with `tools/oracles/spacy_chunks/make_cases.py`. # Open-type entities Entity types named at run time: `Nlp.Entities(text, "person", "organization")`, with a threshold `Nlp.Entities(text, ["contract party"], 0.1f)`, or `OpenEntityRecognizer.Recognize(doc, types)` on a document you tokenized. ## Model - The encoder reads the text once. Candidate spans (up to 12 tokens, flat in the output) get a vector from the span's first, last and mean token vectors and its width; every type name is run through the same encoder and mean-pooled and passed through a feed-forward layer to one vector. A span scores a type with a dot product of the two (scaled by 1/sqrt(width), plus a learned offset) in a shared 128-wide space, and a sigmoid turns it into a probability (Zaratiana et al. 2023, sections 3.2 to 3.4; the paper only). - Training uses binary cross-entropy per token-type cell. A type an example lists with no spans is a negative; a type it does not list teaches nothing, since one data set may label "person" and not "actor". Up to `negativeTypes` negatives are drawn per example. - Prediction keeps spans at or above the threshold (0.5 by default, per call with `threshold`), and where two overlap the more probable wins. The label is the type string as you gave it. Type vectors are cached, so a fixed list of types costs one encoder pass per type, once. - Types: 1 to 256 distinct strings of at most 200 characters, in any language the encoder reads (the shipped head is trained on English). ## Happy path `Nlp.Entities(string text, params string[] types)` finds the registered open-type pipeline (`ModelPackages.RegisterOpenEntities`, called by the model package) and throws, naming `RecursiveAI.OneNlp.Models.Multilingual`, when there is none. Nothing is downloaded. A pipeline's `Recognize` is thread-safe. The default threshold of 0.5 was the best on the Few-NERD and UNER dev sets, and it suits common type names: "organization" finds "Acme Corp." (0.76) and "Beta LLC" (0.80). An unusual name such as "contract party" scores far lower on the same text (0.15 to 0.17), so it returns nothing at 0.5. Pass a lower threshold to `Nlp.Entities(text, types, threshold)`, and check the results, or use a type name the model has seen. To hold the recognizer yourself, `MultilingualModel.LoadOpenEntities()` returns a pipeline whose `Components` include an `IOpenEntityRecognizer`; its `Recognize(doc, types, threshold)` takes the same threshold (null for the component's default). ## Data and limits The recipe (`recipes/openner`) trains on Few-NERD and UNER English (CC BY-SA 4.0) and on entity labels written by Qwen2.5-7B-Instruct-AWQ (Apache-2.0) for FineWeb and Wikipedia passages, after a precision filter. The model therefore has share-alike lineage. CrossNER (MIT) only measures. Results, hand-check numbers and every hash are in `recipes/openner/PROVENANCE.md` and `results.md`. The model is English only and well below the best published open-type recognizers on zero-shot sets (CrossNER 21 to 41 F1 on our run; recall is the weak side). When the types are fixed and you have labels, train the ordinary `EntityRecognizer`. A saved head names its encoder, as every head does; load it with the encoder's directory available. ## Published comparison Strict token-span-and-type micro F1, threshold **0.5**, frozen student-v1, shipped human + Qwen head. Intervals are sentence-bootstrap percentile 95% CIs (1,000 resamples, seed 1). All seven packed-model scores and intervals reproduce `recipes/openner/results.md` to the reported precision; neither weights nor thresholds were changed. The consumer `tools/OneNlp.OpenNerGate` links the recipe's scorer but loads `MultilingualModel.LoadOpenEntities()` from packages. The measured combination is **Models.Multilingual and FrozenEncoder 0.3.0-preview.1**, core and Neural **0.4.0-preview.1**. The newer scratch feed contained no multilingual package; the previously packed model was used, not repacked. Its embedded head (2,385,609 bytes, SHA-256 `25d7cd3f8f3cc67b73aaca5d49db00dc88dc1758ffb34f43658aeb2fefff1d68`), frozen encoder (`aee56231d97134e455fe80a79738eb294c7974c61311412256fa9cd854eeeae6`) and tokenizer (`197d4cc5406ee12cc50c8b5511f2393cc32d9db321545979ce041c1199178356`) were checked against the current manifests **before inference**. A newly packed multilingual 0.4.0-preview.1 is not verified by this run. ### Models and reference Reference: Zaratiana et al., *GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer*, [arXiv:2311.08526v1](https://arxiv.org/abs/2311.08526v1), 14 November 2023, **Table 1, §4.1** (zero-shot OOD benchmark); encoder and training in **§3.2**, evaluation in **§3.4**, decoding in **§2.3**. The paper is [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); PDF SHA-256 `2b6efdc8337d12814308e480378587059a34a960d1da816f192b8cc0aaea04d0`. Only the paper was read; no GLiNER implementation or weights were used. | Model | Parameter count | Encoder | |---|---:|---| | OneNLP open-type | 35.5M encoder weights + **595,969 head parameters** (about 36.1M total) | Frozen student-v1, 6 layers × 384, distilled from mmBERT-base | | GLiNER-S (smallest, primary published reference) | **50M**, as reported | DeBERTa-v3 family, S size in Table 1 | | GLiNER-M (information) | **90M**, as reported | DeBERTa-v3 family, M size in Table 1 | | GLiNER-L (information) | **0.3B**, as reported | DeBERTa-v3 family, L size in Table 1 | These are the paper's counts, not measurements of downloaded GLiNER weights. Table 1 and §3.2 identify the encoder family and size variants, not exact checkpoint revisions. Their encoders are trained on Pile-NER; ours is frozen, and our head trains on Few-NERD, UNER and filtered Qwen labels. This is a comparison to other small open-type models, not a controlled architecture study. ### Rows “Position” compares the published **S** point estimate with our CI: above our upper bound is **behind**, below our lower bound is **ahead**, otherwise **within noise**. It is descriptive, not a paired significance test. Published GLiNER numbers have no CI. “Not comparable” means protocol equivalence is not established; it is not a passing quality verdict. | Evaluation row | Sentences scored | OneNLP F1 [95% CI] | GLiNER-S | GLiNER-M (info) | GLiNER-L (info) | Position vs S | Protocol verdict | |---|---:|---|---:|---:|---:|---|---| | UNER English-EWT test, fixed PER/ORG/LOC | 2,077 | **67.14 [64.34, 69.97]** | — | — | — | — | Not comparable: no UNER row in the paper | | Few-NERD supervised test, seen types | 37,637 | **49.89 [49.63, 50.15]** | — | — | — | — | Not comparable: no Few-NERD row in the paper | | CrossNER politics test, zero-shot | 651 | **41.27 [38.58, 44.09]** | 61.5 | 68.6 | 72.6 | **Behind** | Not comparable (protocol caveats below) | | CrossNER science test, zero-shot | 543 | **36.50 [33.09, 40.06]** | 55.6 | 58.1 | 62.6 | **Behind** | Not comparable (protocol caveats below) | | CrossNER music test, zero-shot | 465 | **26.01 [23.67, 28.35]** | 60.9 | 69.4 | 69.6 | **Behind** | Not comparable (protocol caveats below) | | CrossNER literature test, zero-shot | 416 | **30.38 [27.88, 33.05]** | 60.0 | 59.7 | 64.4 | **Behind** | Not comparable (protocol caveats below) | | CrossNER AI test, zero-shot | 431 | **21.10 [17.79, 24.52]** | 50.7 | 51.8 | 57.2 | **Behind** | Not comparable (protocol caveats below) | All five CrossNER rows are numerically behind **even the smallest** published GLiNER model, and behind M and L as well. The gaps to S are respectively 20.23, 19.10, 34.89, 29.62 and 29.60 F1 points. There is no evidence here that the published-small-model quality target is met; the protocol caveats do not turn these losses into passes. UNER and Few-NERD lack a same-dataset published GLiNER comparison in this paper. The fixed-type black-box reference remains **spaCy 3.8.18, 62.32 [59.45, 65.20]** on UNER English-EWT test, trained with the documented default configuration on the same UNER training data. Its point estimate is below our lower CI bound (**ahead** by the point-vs-CI convention), but the intervals overlap and no paired spaCy comparison was run here. It is not an open-type or zero-shot baseline. ### Protocol audit - **Splits:** we read the fetched CrossNER domain test files, never train on them. The paper §3.4 names the OOD benchmark comprising CrossNER and MIT; it does not give file hashes, sentence counts or exact split revisions. Identical input rows cannot be certified from the paper alone. - **Filtering:** the unchanged recipe excludes empty sentences, sentences over 120 tokens and tokens containing whitespace. It scores gold tokens, joined by single spaces, not a raw-text tokenization pipeline. Counts in the table are after filtering: all fetched UNER and CrossNER sentences remain; Few-NERD retains 37,637 of 37,648 (11 excluded). The paper does not specify these filters or enough input fingerprints to certify the same CrossNER inputs. - **Labels:** our scorer asks for the union of types in the retained file on every sentence, including absent types as negatives, ordered ordinally. `tools/openner/fetch.py` expands labels to words, e.g. `politicalparty` → `political party`, `programlang` → `programming language`, `otherperson` → `person`, `organization` → `organisation`. The paper does not enumerate the exact strings or inventories requested for each CrossNER row, so label equivalence is unverified. Our type counts are politics 9, science 17, music 13, literature 12, AI 14; UNER 3; Few-NERD 67 derived labels (not the 66 original fine labels). “Zero-shot” here means no CrossNER examples used to train or select the head, not that every label word was unseen in training. - **Threshold and decoding:** paper §2.3 keeps scores **>0.5**, with greedy flat non-overlap; ours keeps **≥0.5**, also greedy flat non-overlap, up to 12 tokens. These have the same nominal threshold, but the exact-boundary convention differs. No threshold was retuned on test. - **Metric:** paper §3.4 says exact-match entity F1. Ours is micro F1 over exact token range **and type**, sentence-bootstrapped, not relaxed matching. The paper supplies no bootstrap intervals or precise aggregation details. - **Few-NERD:** our converter adds both fine and coarse labels for a mention where they differ, while prediction is flat and non-overlapping. Thus this seen-types row is the recipe's own two-level target, **not** a conventional fine-only Few-NERD benchmark score. It must not be compared to a published fine-only score without a separate, matching evaluation. See the [results table](https://recursiveai.org/onenlp/docs/results-table/#open-type-ner) for the same rows, and `tools/OneNlp.OpenNerGate/README.md` for the command. This CPU-only measurement uses Linux x64, four threads; no training, GPU inference or speed claim is made. ### Measurement environment and package fingerprints Measured 8 November 2026 on Linux x64, AMD Ryzen 9 7950X3D (16 physical cores, 32 logical; AVX2, AVX-512F and FMA available), four reserved execution threads. .NET SDK 10.0.403, runtime 10.0.14, OneModel and OneModel.Training 1.0.0. No GPU was used. Load averages (1/5/15 min) were 2.97/3.72/7.08 before and 7.58/5.77/7.11 after; during-run load was not sampled. These are quality results, not timings. This measurement does not add a managed speed row: no managed open-type comparator is available. Package SHA-256 fingerprints of the inputs to restore: | Package | SHA-256 | |---|---| | Models.Multilingual 0.3.0-preview.1 | `0ce6681c4ec43e66990ed06b274a2a366cc9cd4b9eb44710e1d4adad5880a2ad` | | Models.Multilingual.FrozenEncoder 0.3.0-preview.1 | `2aa706a43562471d0870654eda6d14c1511b76f5652bfd07ed5305f84eda775b` | | OneNlp 0.4.0-preview.1 | `bb1824efb015ffd09b696233f3b6ae85d3a189d85c001e8dba5d7f8df5f162bd` | | OneNlp.Neural 0.4.0-preview.1 | `93bc0858c9fcae426424f60903377955403e8aa2307e68feda1d52a987d1abd5` | Data fingerprints are in `recipes/openner/PROVENANCE.md` and the fetched `MANIFEST.tsv`. The loaded dependency graph was checked to use the core, Neural and OneModel versions above, rather than the older transitive runtime. # Text classification and span categorization Two heads for labels you define: `TextClassifier` gives a document one or several categories, and `SpanCategorizer` labels spans of a document, including spans that overlap or nest. Both sit on any encoder (the fast encoder by default), train with `Train` ([training.md](https://recursiveai.org/onenlp/docs/training/)) and are scored in [results-table.md](https://recursiveai.org/onenlp/docs/results-table/). ## TextClassifier `TextClassifier` (kind `textcat`) averages the encoder's vectors over the document's tokens and applies a linear layer, with a softmax for exclusive categories (multi-class) or independent sigmoids for any number of categories per document (`multiLabel: true`). A hidden ReLU layer is available (`hiddenSize`) and off by default. Scores land in `doc.Cats` (label to probability); multi-class scores sum to 1. The design follows the sentence-classification setups of Kim 2014 and Joulin et al. 2016. ```csharp Pipeline nlp = Pipeline.Blank(Language.English) .Add(new TextClassifier(FastEncoder.English(7)) { OwnsEncoder = true }); // multiLabel: true for several labels per document nlp.Train(examples, new TrainOptions { DevExamples = dev, Score = DevScores.Accuracy(), Seed = 7 }); Doc doc = nlp.Process("my card has not arrived"); // doc.Cats: label -> probability ``` Data is JSON Lines, one `{"text": "...", "cats": {"label_a": 1, "label_b": 0}}` per document (a label missing from a line counts as 0); see [data-formats.md](https://recursiveai.org/onenlp/docs/data-formats/). The recipes `recipes/en/textcat` (BANKING77) and `recipes/en/textcat-multilabel` (EUR-LEX) run it end to end, with `train`, `predict` and `bench`. ## SpanCategorizer `SpanCategorizer` (kind `spancat`) has a suggester and a classifier, as spaCy documents its `spancat`. The suggester proposes every n-gram of the configured sizes (1 to 3 tokens by default). The classifier represents a span as the concatenation of the mean of its token vectors, its first token's vector and its last token's vector (boundary-plus-pooled span embeddings after Lee et al. 2017), applies a hidden ReLU layer and gives one independent sigmoid per label; a span is kept for every label whose probability reaches the threshold (default 0.8; `SpanCategorizer.Threshold` can be changed on a trained categorizer). Results are in `doc.Spans[key]`, each a `Span` with a label and a confidence. ```csharp Pipeline nlp = Pipeline.Blank(Language.English) .Add(new SpanCategorizer(FastEncoder.English(7), "sc", sizes: [1, 2, 3], hiddenSize: 256) { OwnsEncoder = true }); nlp.Train(examples, new TrainOptions { DevExamples = dev, Score = DevScores.SpanGroup("sc"), Seed = 7 }); Doc doc = nlp.Process("Hamdan v. Rumsfeld was decided in Washington."); foreach (Span span in doc.Spans["sc"]) { /* span.Label, span.StartChar, span.EndChar, span.Confidence */ } ``` Data: `{"text": "...", "spans": {"sc": [[start, end, "LABEL"], ...]}}`, character offsets, with `tokens` optional. The default threshold, 0.8, was chosen on the development split of the recipe's data: at spaCy's 0.5 precision was 70.3 against spaCy's 75.4, at 0.8 it is 76.0 and recall and F1 stay above spaCy's ([results-table.md](https://recursiveai.org/onenlp/docs/results-table/)). It trades precision for recall, so tune it on your own development set. ## Entities from the entity recognizer, and rules `EntityRecognizer` labels non-overlapping entities by tagging tokens; use the span categorizer when spans can overlap. Entities from rules (`EntityRuler`, the recognizers) coexist in `doc.Ents` with model entities ([matching.md](https://recursiveai.org/onenlp/docs/matching/), [recognizers.md](https://recursiveai.org/onenlp/docs/recognizers/)). # Sentence embeddings `IEmbedder` (core) turns text into a fixed-length vector; `SentenceEmbedder` (`RecursiveAI.OneNlp.Neural`) runs a sentence-transformers-format model on OneModel. `Nlp.Embed` is the happy path over the default model in `RecursiveAI.OneNlp.Models.Embeddings`. ```csharp float[] v = Nlp.Embed("how do I reset my password"); // needs RecursiveAI.OneNlp.Models.Embeddings using SentenceEmbedder e = SentenceEmbedder.Load("my-model-dir"); // any sentence-transformers export float[][] batch = e.EmbedBatch(texts); // sorted by length internally, results in input order foreach (float[] x in e.EmbedMany(lazyTexts)) { } // streaming, in batches of e.BatchSize ``` ## Multilingual: `RecursiveAI.OneNlp.Models.Embeddings.Multilingual` `intfloat/multilingual-e5-small` (MIT, about 100 languages, 384 dimensions, mean pooling). Its ONNX file is 470 MB, over the NuGet size limit, so it is cut into three equal byte ranges carried by `...Embeddings.Multilingual.Shard1`, `.Shard2` and `.Shard3`; the thin package (tokenizer, configuration, registration, the API below) depends on all three. The ranges are joined in memory by a read-only seekable stream and handed to the loader, with no temporary file and no joined copy on disk (`tools/embeddings/make_shards.sh` cuts the file and prints the manifest rows). - **Default for `Nlp.Embed`.** The multilingual package registers with a higher priority than the English one, so when it is referenced `Nlp.Embed` uses it, also when `RecursiveAI.OneNlp.Models.Embeddings` (bge) is referenced too. To use bge anyway, call `EmbeddingsModel.Load()` from the English package and keep the embedder. - **Prefixes.** e5 models were trained with `query: ` in front of a search query (and of both texts of a similarity task) and `passage: ` in front of a passage. `Nlp.Embed` and `E5Embedder.Embed` use `query: `, which is right for similarity. For retrieval, `MultilingualEmbeddingsModel.Load()` returns an `E5Embedder` with `EmbedQuery`, `EmbedPassage` and their batch forms `EmbedQueries`, `EmbedPassages`. The helpers live on the model, not on `Nlp`. - **Checks.** Parity with sentence-transformers on the joined shards: see the table below. The consumer application (JIT and NativeAOT) works with the multilingual package alone and with both packages. ## Model directory `SentenceEmbedder.Load(dir, modelFile = "model.onnx", prefix = "")` reads, as data, what sentence-transformers publishes: | File | Used for | |---|---| | `model.onnx` | the encoder: inputs `input_ids`, `attention_mask` (and `token_type_ids`, fed zeros), first output `[batch, sequence, width]` | | `tokenizer.json` | the tokenizer, with its post-processor's special tokens | | `modules.json` | the module chain. Supported: Transformer, Pooling, Normalize. Anything else (a Dense projection, say) throws `NotSupportedException` naming the module | | `1_Pooling/config.json` | exactly one of CLS, mean, max, mean-sqrt-length. Several at once (concatenation) is not supported | | `sentence_bert_config.json` | `max_seq_length` (2 to 65,536; default 512 when absent) | All files are bounded (1 MiB each for the JSON files), the pooling folder must lie inside the model directory, and a damaged file throws `InvalidDataException`. Export a PyTorch model to ONNX with `tools/accurate/export_onnx.py` (first output = last hidden states). `Load(Stream, Stream, EmbeddingConfig)` takes weights from memory or an embedded resource. ## Behaviour - **Pooling** follows Reimers and Gurevych 2019 (Sentence-BERT) §3 over the tokens the attention mask keeps. **Normalization** scales to unit L2 length (a floor of 1e-12 on the norm) when `modules.json` has a Normalize module. - **Truncation:** inputs are cut to `max_seq_length` tokens including the special tokens, keeping the closing special token. Longer text is never an error. - **Prefix:** `prefix` is put in front of every input (E5-family models need `query: ` or `passage: `). The default is none, which is what sentence-transformers does. - **Batching:** a batch is sorted longest first and run in groups of `BatchSize` (default 32); padding never changes a text's vector beyond float rounding. Calls are serialized per embedder. - **int8 (opt-in, never the default):** pass an int8-quantized ONNX file of the same model as `modelFile`. `tools/embeddings/quantize_int8.py` makes one with ONNX Runtime's dynamic quantization. OneModel runs the integer operators exactly. Deltas are below. ### SentencePiece models and white space For models whose tokenizer has a lone `▁` piece (XLM-R and other SentencePiece Unigram vocabularies) sentence-transformers 6.2.0 (transformers 5.22.1) does not give the same pieces as the bare `tokenizer.json` pipeline in a few white-space cases. Observed in its output with `multilingual-e5-small` and reproduced here: 1. Characters the normalizer maps to a space are collapsed to one space (`ab` is `a b`). 2. White space at the end of the text, or before a special token written in the text (``, ``, ``, ``), gives no `▁` piece. White space after a special token is kept. 3. U+0085 (next line) acts as a space. `SentenceEmbedder` applies these three rules for such tokenizers; WordPiece models are unaffected. Without them 91 of 2,103 parity texts differed in token ids (all texts containing trailing or exotic white space or special-token look-alikes); with them none do. ## Parity with sentence-transformers Reference: sentence-transformers 6.2.0, torch 2.15.0+cpu, CPU, float32. Inputs: 2,103 texts from the multilingual parity corpus (all scripts of the Universal Declaration of Human Rights, Tatoeba sentences, and the synthetic classes: CJK, emoji and ZWJ, combining marks, whitespace runs, special-token look-alikes, URLs and code; 62 to 65 of them longer than 512 tokens). Selection is deterministic (`tools/embeddings/parity_reference.py`, seed in the script). Token ids come from `SentenceTransformer.tokenize`. | Model (revision) | Texts | Token-id mismatches | Max abs diff | Min cosine | Mean cosine | |---|---|---|---|---|---| | bge-small-en-v1.5 (5c38ec7c) | 2,103 | 0 | 4.0e-7 | 0.99999988 | 1.0000000 | | multilingual-e5-small (614241f6), `query: ` prefix | 2,103 | 0 | 3.2e-7 | 0.99999988 | 1.0000000 | | multilingual-e5-small, through the shipped package (three shards joined in memory) | 2,103 | 0 | 3.2e-7 | 0.99999988 | 1.0000000 | The vectors match to float tolerance (differences are float32 summation order). int8 against the float reference, for information: bge-small-en-v1.5 max abs diff 0.124, min cosine 0.848, mean cosine 0.966 on this deliberately hostile set. ## MTEB Harness: `mteb` 2.27.1, run black-box through `mteb.evaluate` with a small custom model class (`tools/embeddings/mteb_run.py`); the same class wraps sentence-transformers for the same-harness reference, so the two columns differ only in who computes the vectors. Scores are main scores x 100. "Published" is the model card's `model-index` (the metric each task type uses as its main score; for the multilingual tasks the mean over the subsets that were run). Prompts: bge-small-en-v1.5 uses its retrieval query instruction on SciFact queries and nothing elsewhere; multilingual-e5-small uses `query: ` for queries and symmetric tasks and `passage: ` for documents. Multilingual runs cover STS22 in en, de, es, fr, it, tr, zh and the cross-lingual pairs among them, and MTOPDomain in en, de, es, fr. ### bge-small-en-v1.5 | Task | Type | Published | sentence-transformers (same harness) | Ours | Ours − ST | Ours − published | int8 | int8 − ours | |---|---|---|---|---|---|---|---|---| | STSBenchmark | STS | 85.86 | 85.86 | 85.86 | 0.00 | 0.00 | 85.53 | −0.33 | | Banking77Classification | Classification | 85.74 | 81.75 | 81.75 | 0.00 | **−3.99** | 80.71 | −1.04 | | TwentyNewsgroupsClustering | Clustering | 48.52 | 48.63 | 48.63 | 0.00 | +0.10 | 45.72 | −2.91 | | TwitterSemEval2015 | Pair classification | 73.33 | 73.24 | 73.24 | 0.00 | −0.09 | 71.59 | −1.65 | | AskUbuntuDupQuestions | Reranking | 62.59 | 62.58 | 62.58 | 0.00 | 0.00 | 61.71 | −0.87 | | SciFact | Retrieval | 71.28 | 71.27 | 71.27 | 0.00 | 0.00 | 69.65 | −1.62 | ### multilingual-e5-small | Task | Type | Published | sentence-transformers (same harness) | Ours | Ours − ST | Ours − published | int8 | int8 − ours | |---|---|---|---|---|---|---|---|---| | STSBenchmark | STS | 84.01 | 84.11 | 84.11 | 0.00 | +0.10 | 83.64 | −0.47 | | STS22 (multilingual) | STS | 66.01 | 65.13 | 65.13 | 0.00 | −0.88 | 63.93 | −1.20 | | Banking77Classification | Classification | 79.42 | 70.44 | 70.44 | 0.00 | **−8.97** | 69.72 | −0.72 | | MTOPDomainClassification (en, de, es, fr) | Classification | 88.01 | 85.59 | 85.59 | 0.00 | **−2.42** | 83.94 | −1.65 | | TwentyNewsgroupsClustering | Clustering | 33.22 | 32.81 | 32.81 | 0.00 | −0.41 | 31.67 | −1.14 | | TwitterSemEval2015 | Pair classification | 70.95 | 70.75 | 70.75 | 0.00 | −0.20 | 69.33 | −1.42 | | AskUbuntuDupQuestions | Reranking | 57.88 | 56.42 | 56.42 | 0.00 | **−1.46** | 55.25 | −1.18 | | SciFact | Retrieval | 67.70 | 67.70 | 67.70 | 0.00 | 0.00 | 63.86 | −3.84 | Reading the tables: - **Ours equals sentence-transformers** in the same harness on every task to two decimals (the vectors agree to 1e-7, so the scores cannot differ). - **Published numbers:** STS, pair classification, clustering, retrieval and (for bge) reranking reproduce within 0.5. The bold rows do not, **and they do not for sentence-transformers either**. See the next section for what was tried. - **int8** is smaller (bge 133 MB to 34 MB) and about as fast; quality drops 0.3 to 3.8 points depending on the task (clustering and retrieval most). It is information only; fp32 is what ships. ### Card-era harness (mteb 1.x), for the cells that miss The cells that miss under mteb 2.27.1 were re-run, ours and sentence-transformers 6.2.0 side by side, under two 1.x releases of `mteb` (each in its own venv; `tools/embeddings/mteb1_run.py`, `mteb1_chunk.sh`): 1.1.1 (the first release with these tasks, the closest to the era of the cards) and 1.12.94 (the last 1.12 release), both with datasets 3.2.0 and scikit-learn 1.9.1. The cards state no mteb version, so "the version that produced the card" is not knowable from the card; the datasets' revisions in the cards equal the ones these releases use for Banking77. Per-subset scores are in the result files. Under 1.1.1 STS22 and MTOPDomain would not load (the pinned dataset revisions are loader scripts, which were not run); they were run under 1.12.94 only. | Model and cell | Published | ST 2.27.1 | Ours 2.27.1 | ST 1.1.1 | Ours 1.1.1 | ST 1.12.94 | Ours 1.12.94 | Ours (1.12.94) − published | |---|---|---|---|---|---|---|---|---| | bge Banking77Classification | 85.74 | 81.75 | 81.75 | 81.75 | 81.75 | 81.75 | 81.75 | **−3.99** | | e5 Banking77Classification | 79.42 | 70.44 | 70.44 | 70.44 | 70.44 | 70.44 | 70.44 | **−8.97** | | e5 AskUbuntuDupQuestions | 57.88 | 56.42 | 56.42 | 56.42 | 56.42 | 56.42 | 56.42 | **−1.46** | | e5 STS22 (12 subsets) | 66.01 | 65.13 | 65.13 | not loadable | not loadable | 66.10 | 66.10 | +0.09 | | e5 MTOPDomainClassification (en, de, es, fr) | 88.01 | 85.59 | 85.59 | not loadable | not loadable | 85.54 | 85.54 | **−2.47** | - **STS22 now reproduces** (66.10 against 66.01): the per-subset scores agree with the card to about 0.1 on most subsets; the 2.x difference came from the harness. **Ours equals sentence-transformers** on every cell and release (STS22 differs by 7e-5 in the mean, from one subset). - **The other four cells do not move with the mteb release** (identical to four decimals across 1.1.1, 1.12.94 and 2.27.1 for the three classification and reranking cells). The harness release is therefore not the cause of the gap, and these cells stay below the card for sentence-transformers itself. MTOPDomain per subset, ours (ST identical): en 88.99, de 86.15, es 85.53, fr 81.50 against the card's 91.07, 87.47, 89.27, 84.23. Not investigated further: the card's numbers cannot be reproduced from the published model with the released harness, whoever computes the vectors. ## Speed (indicative) One run on the shared worker (AMD Ryzen 9 7950X3D, 4 threads for both sides, load average about 35 to 41 during the run, so treat as a rough guide), 204 English sentences of 30 to 250 characters, tokenization included, the same exported model. Reference: ONNX Runtime 1.31.1 with `tokenizers` and numpy pooling. There is no managed comparator, so this row passes with that reason recorded; it is not a gate. | Model | Batch | Ours (texts/s, two runs) | ONNX Runtime (texts/s) | |---|---|---|---| | bge-small-en-v1.5 | 1 | 81, 82 | 100 | | bge-small-en-v1.5 | 32 | 119, 111 | 76 | | multilingual-e5-small | 1 | 59, 82 | 112 | | multilingual-e5-small | 32 | 83, 80 | 81 | | bge-small-en-v1.5 int8 | 32 | 121 | | ## Reproducing `tools/embeddings/`: `setup_venv.sh` (sentence-transformers, mteb, ONNX Runtime), `fetch_model.py` and `tools/accurate/export_onnx.py` (pinned revision to ONNX), `parity.sh`, `make_shards.sh`, `mteb_chunk.sh`, `mteb1_chunk.sh` (venvs: mteb 1.1.1 or 1.12.94 with `pyarrow==18.1.0 datasets==3.2.0`), `speed_ort.py`, `quantize_int8.py`, and the `OneNlp.EmbedTool` console project that embeds a text file or serves vectors over a pipe. # Training and recipes Everything a user can train, the shipped models were trained with the same public `Train` API, on [OneModel](https://www.nuget.org/packages/RecursiveAI.OneModel) (automatic differentiation, AdamW, checkpoints), in C#. A recipe is a small console project over that API; running ours on your data is "retraining our model". Behaviour of `Train`, the heads and the encoders is written down in [contracts.md](https://recursiveai.org/onenlp/docs/contracts/); file formats in [data-formats.md](https://recursiveai.org/onenlp/docs/data-formats/) and [model-files.md](https://recursiveai.org/onenlp/docs/model-files/). ## The call The code below is `samples/OneNlp.Samples/TrainSample.cs`, which builds and runs. ```csharp string[] lines = [ "{\"text\": \"my card has not arrived\", \"cats\": {\"card\": 1, \"pin\": 0}}", "{\"text\": \"where is my new card\", \"cats\": {\"card\": 1, \"pin\": 0}}", "{\"text\": \"the card never came in the post\", \"cats\": {\"card\": 1, \"pin\": 0}}", "{\"text\": \"I forgot my pin\", \"cats\": {\"card\": 0, \"pin\": 1}}", "{\"text\": \"how do I change my pin\", \"cats\": {\"card\": 0, \"pin\": 1}}", "{\"text\": \"my pin is not working\", \"cats\": {\"card\": 0, \"pin\": 1}}", ]; using Pipeline nlp = Pipeline.Blank(Language.English) .Add(new TextClassifier(FastEncoder.English(7)) { OwnsEncoder = true }); List examples = [.. JsonlReader.Read(new StringReader(string.Join('\n', lines)), nlp)]; // A fixed seed makes the run repeatable; with CheckpointDirectory set, an interrupted run resumes from its last checkpoint. TrainResult result = nlp.Train(examples, new TrainOptions { Epochs = 30, Seed = 7, Dropout = 0f, DevExamples = examples }); string path = Path.Combine(Path.GetTempPath(), "onenlp-sample-" + Guid.NewGuid().ToString("N") + ".onenlp"); PipelineFile.Save(nlp, path); // one file: tokenizer, encoder, heads, labels using Pipeline loaded = PipelineFile.Load(path); Doc doc = loaded.Process("I lost my pin"); // doc.Cats maps each label to a probability ``` (The sample uses its training set as the development set so that it finishes in seconds; real runs keep them apart.) - **Adding labels.** A head without weights is built from the labels of the training examples. A head that already has weights (a packaged pipeline) keeps them: an `EntityRecognizer` appends the labels the examples add, with fresh weights for the new outputs, so `Pipeline.Create(Language.English).Add()` followed by `Train` on examples with an `ORDER` label learns `ORDER` beside `PER`, `ORG` and `LOC` (include examples of the old labels to keep them). A `Tagger`, `Morphologizer`, `EditTreeLemmatizer`, `Parser`, `SpanCategorizer` or `TextClassifier` has a fixed label set and throws `ArgumentException` naming the values it does not have (for the lemmatizer, edit trees seen at least `minClassCount` times in the new data; rarer ones are left out as they are at the start of training). Their outputs are shared by structure (frequency-ordered classes, pairwise label scoring), so growing them would move trained weights; to change the label set, train a new head. - **Data.** `JsonlReader` reads JSON Lines (`text`, optional `tokens`, `entities`, `spans`, `cats`); `ConllUReader` reads Universal Dependencies files. `Example` pairs a predicted `Doc` with a reference `Doc`; when the tokenizations differ it aligns them. For data already in memory, `Example.Create(nlp, text, [(start, end, "LABEL")])` takes character offsets and tokenizes with the pipeline. A training program needs `using RecursiveAI.OneNlp;` (pipeline, `Example`, `TrainOptions`), `.Neural` (`PipelineFile`), `.Neural.Components` and `.Neural.Encoders`. - **Heads.** `Tagger`, `Morphologizer`, `EditTreeLemmatizer`, `Parser`, `EntityRecognizer`, `TextClassifier`, `SpanCategorizer`, on `FastEncoder` or a transformer encoder; and, as trained word tokenizers, `NeuralMultiwordSplitter` (Turkish and similar) and `NeuralWordSegmenter` (Chinese). Heads that share one encoder instance share its work and add their losses. - **Options.** `TrainOptions` has epochs, batch size in words, learning rate and schedule, warm-up, weight decay, clipping, dropout, seed, a frozen-encoder switch, the development set (`DevExamples` or a split), the score function (`DevScores.Ud(...)`, `DevScores.Entities()`, `DevScores.Accuracy()`), evaluation interval, patience, `CheckpointDirectory` and a progress callback. The library never writes to the console. - **Repeatable.** The batches depend only on the seed and the epoch. A resumed run reaches the same weights as an uninterrupted one (same machine and thread count). The best development checkpoint is restored at the end. - **Fine-tuning a transformer.** `TrainableTransformerEncoder` loads exported ONNX weights (ModernBERT and BERT/XLM-R families) as trainable modules and fine-tunes them with the same call; the written-back weights run in the graph runtime. See [accurate-encoder.md](https://recursiveai.org/onenlp/docs/accurate-encoder/). ## Recipes `recipes//...` and `recipes/en/...` hold one console project per trained part. Each has a `README.md` with its commands, a `PROVENANCE.md` and the results it produced: | Recipe | Trains | Data | |---|---|---| | `recipes/en/fast-tagging`, `parser`, `ner`, `assemble`, `selftrain` | the English tagging, parsing and entity models, assembled into one pipeline file; the self-training that made the packaged encoders narrower | UD English-EWT, UNER English-EWT, self-labelled web text | | `recipes/en/joint` | one shared encoder for all English heads (an option; smaller and faster, weaker rows) | UD English-EWT, UNER English-EWT | | `recipes/en/textcat`, `textcat-multilabel`, `spancat` | the text classifier (multi-class and multi-label) and the span categorizer | BANKING77, EUR-LEX, UNER English-EWT | | `recipes/fr`, `de/fast`, `es/fast`, `pt/fast`, `nl/fast`, `it/fast`, `tr/fast`, `zh/fast` | the fast tier of each language: tagging, parsing, entities, and the multiword splitter or word segmenter where the language needs one | UD treebanks and the named-entity sets listed per recipe | | `recipes/langid` | the language-ID model | held-out multilingual text, listed in its manifest | | `recipes/accurate/en-tagging`, `probe`, `student-v1` | heads on a frozen or fine-tuned transformer encoder; the distilled student's results | UD English-EWT, UNER English-EWT | Conventions every recipe follows: - **Data is fetched by script and checked by hash.** `tools/fetch/` fetches treebanks and pins them; the manifest of each recipe lists every dataset with its role (`train`, `dev` or `eval`) and licence. A recipe reads only its `train` and `dev` files; the test files are read by `predict` and the scorer. - **Resumable.** Every `train` writes checkpoints and continues when run again with the same arguments, so a limited job is chained until the model file appears. `--budget-seconds` stops cleanly. - **Provenance.** `PROVENANCE.md` records the datasets (version, URL, hash, licence), the label generator if any, the recipe and seed, the evaluation sets and the output weights' hash. - **Scoring.** `predict` writes CoNLL-U or entity files that `tools/OneNlp.Eval` scores with the CoNLL 2018 definitions and paired bootstrap intervals ([measurement.md](https://recursiveai.org/onenlp/docs/measurement/)). A model you train this way loads with `PipelineFile.Load` and plugs into `Pipeline` like ours. Models from elsewhere load too: any exported ONNX encoder through `TransformerEncoder` and `TrainableTransformerEncoder`, and any sentence-embedding model through `SentenceEmbedder.Load`. The licence rules for what we ship (permissive weights only) do not limit what you load. ## Speed of training The C# recipes were timed against the same recipes in plain PyTorch on CPU (same data, architecture, batch size and threads); the ratios are in `tools/ratio/RESULTS.md` and the step timing of a transformer fine-tune in [accurate-encoder.md](https://recursiveai.org/onenlp/docs/accurate-encoder/). # Saved pipelines Normative for `PipelineFile` (`RecursiveAI.OneNlp.Neural`). One file holds a trained pipeline: its structure and label sets as JSON, and the weights of its encoders and heads in OneModel's checkpoint encoding. ## Layout (little endian) | Offset | Size | Field | |---|---|---| | 0 | 8 | ASCII `ONENLPPL` | | 8 | 4 | version, 1 | | 12 | 4 | flags, 0 | | 16 | 8 | config length `C` | | 24 | 8 | weights length `W` | | 32 | `C` | config, UTF-8 JSON | | 32 + `C` | `W` | weights, a OneModel checkpoint (`docs/training.md` section 6 of that package) of the state of every encoder and head | The file length is exactly `32 + C + W`. ## Config ```json {"format": "onenlp-pipeline", "version": 1, "language": "en", "tokenizer": {"kind": "uax", "data": "english"}, "encoders": [{"kind": "hash-embedding", "settings": {"width": 64, "buckets": 16384, "seed": 1}}], "components": [{"name": "tagger", "type": "head", "kind": "tagger", "encoder": 0, "config": {"column": "Tag", "labels": ["NN", "VB"]}}, {"name": "sentencizer", "type": "builtin", "settings": {"data": "english"}}]} ``` - `encoders` are the distinct encoders the heads read, in order of first use. Each is rebuilt by `EncoderRegistry.Create(kind, settings)`. - `components` are in pipeline order. A `head` is rebuilt by `HeadRegistry.Create(kind, name, encoder, config)`; a `builtin` is created by name from `ComponentRegistry`. A component that is neither a trainable head nor registered cannot be saved. - Weights are named `encoder.…` (encoders in order, those with weights) and `head.…` (trainable heads in pipeline order). Loading checks every name, type and shape against the shapes the config implies. - `tokenizer` is `{"kind": "rule"}`, `{"kind": "uax", "data": "none"|"neutral"|"english"|...}` (the segmentation data set the tokenizer uses) or `{"kind": "segmenter", "encoder": i, "head": {...}}`. Other tokenizers cannot be saved yet. - **`segmenter`** is a trained word segmenter (`NeuralWordSegmenter`, for text written without spaces). `encoder` is the index of one of the file's `encoders`; `head` is the config of a `tagger` over the `Tag` column with the labels `B` and `I`, exactly as a tagger component's `config`. The segmenter's tagger and encoder are not in `components`: their weights are saved under the first names, `encoder0`/`head0` when the segmenter's encoder is the first encoder, and the components' encoders and heads are numbered after them. A file with another tokenizer is numbered as before, so every file written before this kind existed loads unchanged; a reader that predates it rejects a `segmenter` file with "Unsupported tokenizer kind". The format version stays 1. A pipeline file that only carries a segmenter has an empty `components` list. - **`mwt`** is a multiword-token splitter (`NeuralMultiwordSplitter`): `{"kind": "mwt", "inner": {"kind": "uax", "data": "turkish-whole"}, "encoder": i, "head": {...}}`. `inner` is the surface tokenizer, `rule` or `uax` (with its `data`); `encoder` and `head` are as for `segmenter` (a `tagger` over the `Tag` column with the labels `B` and `I`, one decision per character of a letters-only token of 3 to 48 units: `B` starts a UD word, so a `B` after the first character is a cut). Its tagger and encoder are numbered first, exactly as a segmenter's are, `minCutPercent` (50 to 99, default 50 and then absent) is the confidence a `B` needs to count as a cut. Every file of another tokenizer numbers its weights as before and loads unchanged; a reader that predates the kind rejects an `mwt` file with "Unsupported tokenizer kind". The format version stays 1. - A `builtin` component may carry `settings`, an object of at most 64 short strings, written by components that implement `IConfigurableComponent` and passed to `ComponentRegistry.Create` on load (the sentencizer writes `data`). ## Kinds Encoder kinds (`encoders[i].kind`, with `settings`) and head kinds (`components[j].kind`, with `config`) that the loader knows. Weights of each are named under `encoder.` and `head.`. | Kind | Settings or config | Weights | |---|---|---| | encoder `hash-embedding` | `width`, `buckets`, `seed` | `embed.weight` `[buckets, width]`; `project.weight`, `project.bias` | | encoder `transformer` | `model`, `tokenizer` (file names of the exported ONNX encoder and its `tokenizer.json`, at most 1,024 characters each), `modelSha256` (optional, 64 hex characters), `pooling` (`first` or `mean`), `maxLength` (2 to 65,536) | none: the encoder is frozen and lives in its own files, which the pipeline file references and does not embed | | encoder `fast` | `width` (8 to 2048), `depth` (0 to 32), `normRows`, `prefixRows`, `suffixRows`, `shapeRows` (each 1 to 4,194,304, and rows times embedding width over all tables at most 33,554,432 values together), `seed`; optional, absent meaning the first file version: `prefixLength` (1 to 8, default 1), `suffixLength` (1 to 8, default 3), `casedRows` (0 to 4,194,304, default 0: a fifth table for the case-preserving text), `embedWidth` (8 to 2048, default `width`), `hashes` (1 to 4, default 2), `plain` (0 or 1, default 0), `relu` (0 or 1, default 0: ReLU instead of GELU), `neighbourRows` (0 to 4,194,304, default 0: four tables for the lower-cased text and the suffix of the previous and next token: `embed_prev_norm`, `embed_next_norm`, `embed_prev_suffix`, `embed_next_suffix`), `contextRows` (same range, default 0: four tables for the text two tokens before and after and the shape of the previous and next token: `embed_prev2_norm`, `embed_next2_norm`, `embed_prev_shape`, `embed_next_shape`), `bigramRows` (same range, default 0: `embed_bigram_prev`, `embed_bigram_next`), `turkicCase` (0 or 1, default 0: lower-case the first attribute with the Turkish rules, `I` to `ı` and `İ` to `i`, before hashing; Unicode SpecialCasing, Turkish and Azeri; the other attributes are unchanged), `recurrentLayers` (0 to 4, default 0: bidirectional LSTM layers over each document after the window layers; the output is added to the window output) and `recurrentSize` (0 to 2048, default 0 meaning half the width: hidden size per direction; when it is not half the width a linear layer `lstm_out` maps the two directions back to `width`); every table counts in `mix.weight`'s input width | `embed_norm.weight`, `embed_prefix.weight`, `embed_suffix.weight`, `embed_shape.weight` (and `embed_cased.weight` when `casedRows` is set) (`[rows, embedWidth]`); `mix.weight` `[width, tables * embedWidth]`, `mix.bias`, and, unless `plain`, `mix_norm.weight`, `mix_norm.bias`; for each layer `i`: `window.weight` `[width, 3 * width]`, `window.bias`, `window_norm.weight`, `window_norm.bias`; with `recurrentLayers`, after them: the `lstm` module (`Recurrent`, bidirectional, input `width`) and, when its size is not half the width, `lstm_out.weight` `[width, 2 * recurrentSize]`, `lstm_out.bias` | | encoder `transformer-trainable` | `family` (`modernbert` or `postnorm`), `model`, `tokenizer`, `modelSha256` (optional), `pooling`, `maxLength`, `heads` (1 to 1,024), `globalEvery`, `localWindow`, `ropeThetaGlobal`, `ropeThetaLocal`, `normEpsilon`, `positionOffset`, `maxBatchPieces` | the layers: `embedding_norm`, `layers.*`, `final_norm` for `modernbert`; `positions`, `embedding_norm`, `layers.*` for `postnorm`. The token table is not stored: it is read again from the exported model (a pipeline that trains it cannot be saved) | | head `tagger` | `column` (`Tag`, `Pos` or `Morph`), `labels` | `output.weight` `[labels, width]`, `output.bias` | | head `morphologizer` | `labels`: joint labels, each `POS=` or `POS=\|`, at most 65,536 | `output.weight`, `output.bias` as the tagger | | head `lemmatizer` | `labels`: classes, each `l` or `r` followed by a serialized edit tree, at most 262,144 | `output.weight`, `output.bias` as the tagger | | head `textcat` | `multi_label` (true or false), `hidden` (0 to 4,096: width of the optional hidden layer, 0 for none), `labels` (the categories, at most 65,536, distinct) | with `hidden` > 0: `hidden.weight` `[hidden, width]`, `hidden.bias`; then `output.weight` `[labels, hidden or width]`, `output.bias` | | head `spancat` | `spans_key` (the span group, at most 256 characters), `sizes` (1 to 32 distinct span lengths in tokens, each 1 to 256), `threshold` (above 0, at most 1), `hidden` (0 to 4,096), `labels` (at most 16,384, distinct) | with `hidden` > 0: `hidden.weight` `[hidden, 3 * width]`, `hidden.bias`; then `output.weight` `[labels, hidden or 3 * width]`, `output.bias` | The `textcat` and `spancat` kinds were added after the first file version and change nothing else: a file without them loads as before, the format version stays 1, and a reader that predates them rejects a file that holds one with "No head kind ... is registered". A `textcat` head scores the mean of the document's token vectors; a `spancat` head scores `[mean; first; last]` of the token vectors of every n-gram of the listed sizes. Both read the label list in output order. The `fast` encoder's optional settings are written only when they differ from the defaults, so a file made with the first-version settings is read and written unchanged. With one hash function only the low half of the 64-bit hash is used; with three or four, the further functions use a re-mixed hash (SplitMix64 of the hash plus the function number). With `plain`, the joined attributes go through a ReLU instead of a GELU and layer normalization. With `turkicCase`, the lower-cased form (the first attribute) maps `I` to `ı` and `İ` to `i` before hashing (Unicode SpecialCasing, Turkish and Azeri) instead of the invariant mapping; the other attributes are unchanged. The `fast` encoder's hash is part of the format: FNV-1a over UTF-16 units, seeded per attribute and finished with SplitMix64 (`docs/contracts.md`), so a saved model hashes every token to the same rows on every platform. A serialized edit tree is `M,()()` for a match node and `S::` for a substitution, where `` and `` are the lengths in UTF-16 units of the strings that follow them; at most 4,096 characters and 64 levels deep. A label list with a repeated entry, a morphologizer label not starting with `POS=`, a lemmatizer class not starting with `l` or `r`, or a tree that does not parse is an `InvalidDataException`. ## Bounds on read Config at most 16 MiB with JSON nesting at most 32; weights at most 1 GiB; at most 64 encoders and 1,024 components; label sets at most 1,048,576 entries of at most 4,096 characters; an embedding table at most 67,108,864 values and a head's output layer at most 67,108,864 weights. The declared lengths are compared with the real length of the data before anything is allocated. Any failure is an `InvalidDataException`. **Fine-tuned encoders.** `PipelineFile.Save(pipeline, path)` writes each `transformer-trainable` encoder as an ONNX file `.encoder.onnx` beside the pipeline and records the encoder as `transformer` with `model` set to that plain file name and `relativeTo` set to `pipeline` (the name is looked up in the pipeline file's directory, not in `ModelDirectory`; it must not contain a path separator or `..`). No `encoder` weights are stored for it. A reader that does not know `relativeTo` ignores it. Loading such a pipeline from a stream that is not a file fails. `Save(pipeline, path, exportEncoderGraphs: false)` and saving to a stream keep the `transformer-trainable` layout. See `accurate-encoder.md`. **Referenced encoder files.** A `transformer` encoder is a multi-hundred-megabyte ONNX file plus a `tokenizer.json`. A pipeline file stores their names (and optionally the model's SHA-256), not their bytes: the same encoder serves many pipelines, and a pipeline file stays a few kilobytes. A relative name is looked up in `TransformerEncoder.ModelDirectory` (the current directory when unset); an absolute name is used as is. Loading fails with the file's own `FileNotFoundException` when a file is missing, and `VerifyModelFile` checks the hash on request. The model packages put the encoder files next to the pipeline files they belong to. ## Heads with a two-level output The morphologizer and lemmatizer may have `headClasses` (0 to 4,096) in their config. With a value above 0, labels are stored in falling order of frequency and the output layer is two Linear layers, `head` (`headClasses + 1` outputs, the last meaning "other") and `tail` (the remaining classes); with 0 or absent it is the single layer `output`. The parser's `labelBilinear` (0 when the label biaffine is left out, absent meaning 1) and the shared-encoder pipelines of `recipes/en/joint` use the same file layout otherwise. ## Inference At inference the fast encoder with `relu` computes from derived tables (each embedding table multiplied by its slice of `mix.weight`, each window layer as one product); they are rebuilt when a weight changes and are not stored. The encoder keeps, per thread, the result for the last documents it encoded, valid until that thread encodes other documents or any training-mode encoding happens, so the heads of a pipeline share one encoding. # Data formats Normative for the readers in `RecursiveAI.OneNlp.Data`. Both readers stream (one sentence or line at a time), bound every size before allocating, and reject malformed input with a `DataFormatException` that carries the 1-based line number (`LineNumber`; the message starts `line N:`). ## CoNLL-U Per the CoNLL-U format specification (universaldependencies.org/format.html). `ConllUReader.Read` yields a `ConllUSentence` per sentence; `ConllUReader.ReadExamples` yields `Example`s. **Input.** UTF-8 (a leading byte-order mark is skipped; invalid bytes are an error). Lines end at LF or CR LF. A blank line ends a sentence; the last sentence needs no blank line after it. Limits (`ConllUOptions`): line 1,048,576 UTF-16 units, 100,000 word and multiword-token lines per sentence, 10,000 comment lines per sentence. **Lines.** - `#` lines are comments. They must come before the first word of a sentence. They are kept as written (`Comments`); `GetComment("sent_id")` reads `# key = value`. - A word line has exactly 10 tab-separated columns. IDs are `1, 2, 3, ...` in order. FORM must not be empty. HEAD is an integer from 0 to the number of words, or `_`; a word cannot be its own head. `_` in LEMMA, UPOS, XPOS, FEATS, DEPREL, DEPS and MISC means "not set". - A multiword token line has an ID range `a-b` where `a` is the next word and `b > a`; its words follow immediately. Ranges do not nest or overlap. - An empty node line has an ID `N.M`. It is kept (position and text) and has no token in the document. - A sentence of only comments, or an empty node with no word, is an error. **The document.** One token per syntactic word. `Doc.Text` is the surface text: the surface forms (a multiword token's range form, else the word's form) joined by one space, except after a token whose MISC has `SpaceAfter=No` (on the range line for a multiword token). Columns: LEMMA to `Lemma`, UPOS to `Pos`, XPOS to `Tag`, FEATS to `Morph` (as written), DEPREL to `Dep`; HEAD 0 and `_` both make the word its own head (a root); the first token is a sentence start and the rest are inside the sentence. **Words of a multiword token** need characters of their own, since a `Doc` token is a non-empty slice of the text. If the word forms concatenate to the surface form (`do` + `n't` = `don't`), each word covers its own form. Otherwise (`de` + `el` for `del`) the surface form is cut into pieces of nearly equal length, longer pieces first. When the surface form has fewer characters than the multiword token has words (Portuguese `à` = `a` + `a`), the words cannot share it, so `Doc.Text` spells the word forms separated by one space in place of the surface form for that token; the range line still keeps the surface form for writing. The word's own form stays in `ConllUSentence.Forms`. **Kept beside the document** so that `ConllUWriter.Write` gives the sentence back exactly: comments, `Forms`, `Deps`, `Misc` (as written), `MultiwordTokens`, `EmptyNodes`. Reading and writing a UD file reproduces it byte for byte when it uses LF and a blank line after each sentence (checked on English-EWT train, dev and test). `ConllUWriter.Write(writer, doc)` writes any document, for example a prediction. ## JSON Lines `JsonlReader.Read` yields one `Example` per non-blank line. A line is one JSON object (nesting depth at most 16; at most 16,777,216 UTF-16 units by default); unknown properties are ignored. | Property | Type | Meaning | |---|---|---| | `text` | string, required | The text. | | `tokens` | optional array of `[start, end]` | Token boundaries, UTF-16 offsets into `text`, `end` exclusive, in order, non-empty, not overlapping. Without it the reader's tokenizer (default: the tokenizer `Pipeline.Blank` uses; the overloads that take a `Pipeline` use that pipeline's) splits the text. | | `entities` | optional array of `[start, end, "LABEL"]` | Gold entities, character offsets. Must not overlap. Their presence, even as `[]`, marks the document as entity-annotated (every other token is outside an entity); absence leaves the entity column unset. Stored in the reference `Doc`. | | `spans` | optional object, name to array of `[start, end, "LABEL"]` | Span groups; spans may overlap. Stored as token ranges in `Example.Spans`. | | `cats` | optional object, label to number (or boolean) in `[0, 1]` | Text categories (multi-class or multi-label). Stored in `Example.Cats`. | A span or entity whose edges are not token boundaries is an error, or is dropped when `JsonlOptions.Misaligned` is `Skip`. Labels are non-empty. Offsets must lie inside `text` with `start < end`. ```json {"text": "Ada met Bob in Paris.", "entities": [[0, 3, "PER"], [8, 11, "PER"], [15, 20, "LOC"]], "cats": {"travel": 1}} {"text": "Hi there", "tokens": [[0, 2], [3, 8]]} ``` ## Example alignment `Example.Alignment` aligns the predicted and reference documents when they are tokenized differently. The texts must be equal after whitespace is removed and case is ignored. Each token covers a range of the remaining characters; two tokens are aligned when their ranges overlap. `Example.GetAligned(column)` gives the reference value to a predicted token only when it overlaps exactly one reference token; `GetAlignedEntities()` keeps a reference entity only when both its edges are predicted token boundaries. # JSON document format Normative and language-neutral. A document is a JSON object in the shape spaCy documents for `Doc.to_json`, so documents can move between the two. Fields of ours are additions that other readers can ignore (spaCy 3.8.18's `Doc.from_json` does). ```json { "text": "Ada Lovelace wrote notes in London.", "ents": [{"start": 0, "end": 12, "label": "PERSON"}], "sents": [{"start": 0, "end": 35}], "tokens": [{"id": 0, "start": 0, "end": 3, "tag": "NNP", "pos": "PROPN", "morph": "Number=Sing", "lemma": "Ada", "dep": "compound", "head": 1}] } ``` ## Fields | Field | Meaning | |---|---| | `text` | the original text | | `tokens[]` | one object per token, in order: `id` (its index), `start`, `end` (end exclusive), and `tag`, `pos`, `morph`, `lemma`, `dep`, `head` | | `ents[]` | `start`, `end`, `label`; written only when entities have been set (an empty list means "set, none found") | | `sents[]` | `start` (first token's start) and `end` (last token's end) of each sentence; written only when sentence boundaries have been set | The writer emits `tag`, `pos`, `morph`, `lemma` and `dep` for every token when any token has the value, `head` when any token has a dependency or a head other than itself, and omits an annotation nobody set. A token that is its own head is a root. The reader treats a missing or empty annotation as not set and a missing `head` as the token itself. **Categories and span groups.** `cats` is an object from label to score (0 to 1, in the order the component wrote them), written when a classifier has set categories. `spans` is an object from group name to a list of `{start, end, label}` with code point offsets as for `ents`; a span may overlap or nest in another of its group, and keeps `conf` when not 1. spaCy's extra span fields (`kb_id`, `token_start`, `token_end`) are skipped on reading. A document without them writes neither field. **Our additions.** Token `norm` (the normalized form), token `conf` (the confidence of the token's tag; written when not 1, read as 1 when missing), token `sb` (the sentence boundary: 1 starts a sentence, 2 is inside one; a token with no `sb` has an unset boundary) and entity `conf`. When no token has `sb` and `sents` is present (a document from spaCy), the first token of each sentence is a start and every other token is inside. ## Offsets `start` and `end` count Unicode code points, as spaCy's do (Python string indexes). A `Doc` counts UTF-16 code units; the two differ only after a character outside the Basic Multilingual Plane, and the writer and reader convert. A character offset in `ents` and `sents` must fall on a token boundary. ## Not carried JSON text cannot hold an unpaired surrogate, so writing a document whose text or a label has one throws `InvalidOperationException`; use the binary form. A token boundary inside a surrogate pair has no code point offset and throws the same way. `user_data` and unknown fields are skipped on reading, because a `Doc` does not store them. ## Reading limits The reader accepts UTF-8 with an optional byte-order mark. It rejects, with `InvalidDataException`: nesting deeper than 32, trailing data, wrong types, a token `id` that is not its index, a `head` outside the document, a `conf` outside 0 to 1, an `sb` other than 0, 1 or 2, and any offset beyond the text. From a stream it takes at most 512 MiB. ## With System.Text.Json `JsonSerializer.Serialize(doc)` writes exactly `doc.ToJson()`, and `Deserialize` reads it (an invalid document throws `JsonException`), so a `Doc` can be a property of a DTO or an endpoint's return value. This goes through a converter named by an attribute on `Doc`, with no reflection; under NativeAOT, list the DTO in a source-generated `JsonSerializerContext` as usual. A `Span` is written as `{"start", "end", "label", "text", "confidence"}` and a `Token` as `{"i", "start", "end", "text", "lemma", "pos", "tag", "morph", "dep", "head"}` (annotations only when set). Their `start` and `end` are code points, the same unit and values as `ents` and `tokens` in the document form, so one response never mixes units (`Span.StartChar` and `Token.Idx` stay UTF-16 offsets into `Doc.Text`; they differ only after a character outside the Basic Multilingual Plane). Neither can be read on its own. # Binary document format Normative and language-neutral. Two containers share one body encoding: a single document (`ONDC`) and a bin of many documents (`ONDB`) that share a string table. Version 1. ## Conventions - Integers are little-endian. `u16` and `u32` are fixed width. - A **varint** is LEB128: seven bits per byte, low bits first, the high bit set while more bytes follow. A value is at most 32 bits and at most 5 bytes; a longer or larger one is invalid. - A **signed varint** is the zig-zag mapping `(v << 1) ^ (v >> 31)` written as a varint. - A **float** is an IEEE 754 binary32 stored as its `u32` bit pattern, so confidences round-trip bit for bit. - A **string** is a varint `h` followed by `h >> 1` bytes. If `h & 1` is 0 the bytes are UTF-8 and must be valid; if 1 they are UTF-16LE code units (an even byte count) and may hold unpaired surrogates. A writer uses UTF-8 whenever the string is well-formed. - A **section** is `u8 id`, `u32 length`, then `length` bytes. Sections inside one body appear in increasing id order, each at most once. A reader skips an id it does not know. ## Header (12 bytes, both containers) | Offset | Size | Field | |---|---|---| | 0 | 4 | magic: `ONDC` (one document) or `ONDB` (a bin) | | 4 | 2 | format version, 1 | | 6 | 2 | flags: bit 0 = a CRC-32 trailer follows the body; every other bit must be 0 | | 8 | 4 | body length: the bytes after the header, not counting the trailer | The optional trailer is a `u32` CRC-32 (IEEE 802.3, polynomial 0xEDB88320, reflected, as in zip and PNG) of every byte before it, header included. The body length makes a single document self-delimiting, so documents written back to back can be read in turn from a stream. ## String table (section 1) `varint count`, then `count` strings. Table index 0 is the empty string and is not stored; entry *i* (1-based) is the *i*-th stored string. The table holds only strings the documents use: attribute values and entity labels, in order of first use. ## Document body (sections 2–4) All string-valued fields are table indexes (varints); 0 means "not set". **Text (section 2).** One encoding byte (0 = UTF-8, 1 = UTF-16LE) and the text. UTF-16 is used only when the text has an unpaired surrogate. **Tokens (section 3).** In this order: 1. `varint n`, the token count. 2. For each token, two varints: the gap from the end of the previous token (for the first token, from the start of the text) to this token's start, and its length. Both count UTF-16 code units. Lengths are at least 1 and every token ends inside the text. Whether a space follows a token is derived from the text, not stored. 3. Six attribute columns, each `n` string indexes, in `TokenColumn` order: norm, lemma, pos, tag, dep, morph. 4. Heads: `n` signed varints, each the head's token index minus the token's own index (0 for a root). 5. Sentence boundaries: `ceil(n / 4)` bytes, two bits per token, low bits first: 0 unset, 1 start, 2 inside (3 is invalid). 6. Confidences: a mode byte. 0: every confidence is exactly 1.0 and nothing follows. 1: `n` floats follow. The section must end exactly there. **Entities (section 4).** Present only when entity annotation is set (a document with tokens whose `EntIob` is set; an empty list is a document with no entities). `varint count`, then per entity in text order: the gap in tokens from the previous entity's end (or from 0), its length in tokens (at least 1), the label's table index (not 0) and a float confidence. Entities do not overlap. ## Single document (`ONDC`) **Categories (section 6).** Present only when categories have been set. `varint count`, then per category in the order written: the label's table index (not 0) and a float score. Labels are distinct. **Span groups (section 7).** Present only when a document has a span group. `varint groups`, then per group: the name's table index (not 0), `varint count`, and per span in the order kept: the first token, the length in tokens (at least 1), the label's table index (not 0) and a float confidence. Spans of a group may overlap. Sections 6 and 7 were added to version 1 after the first release: a reader that predates them skips them by length, so every older reader still reads the rest of the document. Header, string table section, then sections 2–4, then the optional trailer. ## Bin (`ONDB`) Header, string table section (shared by every document), then section 5 and the optional trailer. Section 5 holds `u32 count` and `count` records, each a `u32 length` followed by a document body (sections 2–4). The section's length must equal the body bytes that remain, and the records must fill it exactly. A reader may take the records one at a time (`DocBinReader`); it verifies the checksum after the last. ## Reader limits A reader checks every length and count against the bytes that remain before it allocates, and rejects the input with `InvalidDataException` when: - the magic, version or flags are wrong, or the input is shorter than the length it declares, or has bytes after the end; - a section is out of order or repeated, or does not end where its contents do; - the string count exceeds the bytes in the table, a string is invalid UTF-8, or an index is beyond the table; - the token count exceeds 33,554,432 or needs more than 9 bytes per token of the section; - a token is empty or outside the text, a head is outside the document, an entity is empty, unordered, overlapping or unlabelled, a confidence is outside 0 to 1, or a boundary value is 3; - the checksum does not match. From a stream, a reader takes at most a declared length of 1 GiB and reads it in 1 MiB steps, so a false length cannot make it allocate more than the stream delivers. # Results table One table of what is measured, what it is measured against, and whether it holds. Every number is copied from a file in this repository, named in the "Source" column or the section heading; nothing here was measured for this page. Rows that lose, and gaps, are listed in their own sections below, not left out. **How to read it.** - **Fast-tier and text-classification rows** are scored on official test splits, read once for the final configuration (the development splits chose every setting). The reference is spaCy 3.8.18, trained as a black box with its default configuration on the same data, in the same input mode (`tools/baselines/spacy/`). Intervals are paired bootstrap 95% over sentences or documents (1,000 resamples; seeds are in each source file). **Within noise** means the two intervals overlap (`measurement.md`); **ahead** means they do not and ours is higher; **behind** means they do not and ours is lower. - **Gold** rows give the treebank's own words and sentences as input; **raw** rows give only the text, so tokenization and sentence splitting are scored too. spaCy's tokenizers keep contractions such as `del` and `zum` whole, which is why several raw rows are far ahead. - **Within noise is not a claim of equal.** With a few hundred sentences an overlap is a weak statement. Where the paired difference excludes zero while the intervals overlap, the source file says so, and the row is also named under "Misses and caveats". - **Speed rows** are in [results.md](https://recursiveai.org/onenlp/docs/results/). The tokenizer, language-ID and recognizer rows there were taken alone on the machine and are repeated at the release gate; the fast-tier speed rows are indicative, and the publishable run against Catalyst is made at the release gate. ## Tokenizers and segmentation | Component | Metric | Ours | Reference | Verdict | Source | |---|---|---|---|---|---| | Tokenizers: WordPiece, Unigram, SentencePiece-BPE, byte-level BPE, Tiktoken (12 model families) | IDs, tokens, offsets, special-token mask, decoded text vs Hugging Face `tokenizers` 0.22.1 and `tiktoken` 0.14.0 on the multilingual parity corpus | byte-exact (a mismatch is a bug, never a tolerance) | the two libraries, run as installed packages | holds: 0 mismatches on all 12 families (bert-base-uncased, bert-base-multilingual-cased, xlm-roberta-base, t5-small, mmBERT-base, TinyLlama, gpt2, roberta-base, Qwen2.5, ModernBERT, cl100k_base, o200k_base), each over 47,981 corpus records (about 3.45 million comparisons per family); release gate run on 2026-11-07 (tokenizers 0.22.1, tiktoken 0.14.0): 0 mismatches on every family; comparisons per family: bert-base-uncased 3,451,629; bert-base-multilingual-cased 3,451,661; xlm-roberta-base 3,451,681; t5-small 3,449,575; mmBERT-base 3,449,935; TinyLlama 3,449,935; gpt2 3,449,919; roberta-base 3,452,041; Qwen2.5 3,449,919; ModernBERT 3,449,935; cl100k_base 2,395,778; o200k_base 2,395,778 | [parity.md](https://recursiveai.org/onenlp/docs/parity/), `tools/parity/families.json` | | UAX #29 word and sentence boundaries | every line of Unicode 17.0.0 `WordBreakTest.txt` and `SentenceBreakTest.txt` through our layer with no language data | conformance tests pass | the Unicode test files | holds: WordBreakTest 1944/1944, SentenceBreakTest 512/512 (release gate run, both conformance tests pass) | [segmentation.md](https://recursiveai.org/onenlp/docs/segmentation/) | | English tokenization, raw EWT r2.18 test | Tokens F1 | 99.00 [98.80, 99.19] | spaCy 95.70 [95.14, 96.21] | ahead | [results.md](https://recursiveai.org/onenlp/docs/results/) | Tokenizer speed against Microsoft.ML.Tokenizers 2.0.1, release gate, arm64 (Apple M2, 8 cores, .NET 10.0.403, load 2.0 to 3.3 before and during, idle after; run alone on the machine): 47 timed rows, none below 1.00x. Seven operation rows are not timed because the comparator's output differs from ours (WordPiece medium decode and long x3, Unigram long x3; six load rows are reported separately); they are not counted as passes. **x64 release run:** 47/47 matched operation rows pass, 1.13–48.55x; seven differing-output rows are not timed. Four x64 load rows are slower (WordPiece, TinyLlama, gpt2, cl100k_base; information only). [Every x64 cell and machine record](https://recursiveai.org/onenlp/docs/speed-x64/). On arm64, load time is reported, not gated, and is slower than the comparator on three rows: WordPiece from tokenizer.json 28.16 ms vs 19.17 ms (0.68x), SentencePiece-BPE TinyLlama 49.60 ms vs 11.66 ms (0.24x), cl100k_base 19.93 ms vs 19.00 ms (0.95x); ours is faster on Unigram (3.47x), gpt2 (1.02x) and o200k_base (1.31x). ## Language identification Held-out Leipzig texts ("main") and a Tatoeba slice, macro accuracy over 75 languages (every language weighs the same), against `lingua-language-detector` 2.2.0 and SearchPioneer.Lingua 2.0.0 in high-accuracy mode, built for the same 75 languages. Source: [language-id.md](https://recursiveai.org/onenlp/docs/language-id/), `recipes/langid/results/`. | Set / bucket | Ours | Lingua (Python) high | Lingua (.NET) high | Ours minus Lingua Python | Verdict | |---|---|---|---|---|---| | main / words | 77.14 [76.86, 77.46] | 75.59 [75.30, 75.86] | 75.35 | +1.56 [+1.31, +1.86] | ahead | | main / pairs | 89.53 | 85.66 | 85.49 | +3.87 [+3.65, +4.08] | ahead | | main / sentences | 98.03 | 95.98 | 95.81 | +2.05 [+1.95, +2.15] | ahead | | Tatoeba / words | 79.14 | 75.03 | 74.79 | +4.10 [+3.39, +4.77] | ahead | | Tatoeba / pairs | 91.14 | 85.79 | 85.75 | +5.35 [+4.66, +6.18] | ahead | | Tatoeba / sentences | 96.15 | 93.27 | 93.19 | +2.88 [+2.55, +3.20] | ahead | The macro averages hide languages where we are below Lingua: on the main set 14 languages on words (ar da ga hr ko lg mk nl ru so ur xh yo zh), 8 on pairs (hr ja lg ms ru xh yo zh) and 2 on sentences (en hr). The closely related Latin-script pairs are the hard rows. The Leipzig texts come from the collection Lingua says its models were built from (this may favour Lingua), and our training includes other Tatoeba sentences (this may favour us); both are reported. Per-language rows: `recipes/langid/results/per-language.*.csv`. Language-ID speed (alone on the machine, x64 and arm64): at least 6.9x Lingua on every row (the minimum is arm64 word pairs against low accuracy; on x64 at least 7.8x); 10.7 ms load against 326 ms (low accuracy) and 3,850 ms (high accuracy) on x64 (`results.md`). Language-ID speed, release gate, arm64 (Apple M2, 8 cores; median microseconds per text, ours / SearchPioneer.Lingua 2.0.0 high accuracy / low accuracy): words 7.74 / 76.42 / 63.54 (9.9x and 8.2x faster), pairs 8.09 / 88.65 / 55.80 (11.0x and 6.9x), sentences 10.77 / 300.58 / 216.10 (27.9x and 20.1x). Allocation 1.4 KB per text against 75 to 126 KB; RSS 66 MB against 2,575 and 480 MB; load 9.3 ms against 4,329 and 328 ms. No row is slower. **x64 release run:** words 8.83 / 81.62 / 69.05 µs (9.24x / 7.82x), pairs 3.58 / 96.29 / 65.58 (26.90x / 18.32x), sentences 9.06 / 392.54 / 246.83 (43.33x / 27.24x). All rows pass. Ours allocates 1.4 KB/text vs 75–125 KB; RSS 95 vs 5,042 / 360 MiB; load 12.6 vs 3,670.6 / 282.0 ms. [Inputs and machine record](https://recursiveai.org/onenlp/docs/speed-x64/). ## Recognizers Our hand-labelled gold sets, one per language, written from the type definitions and real text (`recognizer-gold.md`), test split. Reference: Microsoft.Recognizers.Text 1.8.13 in the same culture, run as a black box. Exact-span F1, all types pooled, with the paired bootstrap 95% interval (1,000 resamples over sentences, seed 20260101). Source: `testdata/recognizers//comparators/ours-vs-recognizers.test.md`. | Language | Test items | Ours, all types | Recognizers.Text, all types | Value accuracy on exact matches (ours) | Verdict | |---|---:|---|---|---|---| | English | 312 | 97.1 [95.3, 98.9] | 59.9 [56.2, 63.1] | 100.0 | ahead | | French | 347 | 95.6 [93.7, 97.5] | 45.2 [41.6, 48.7] | 99.6 | ahead | | German | 349 | 96.0 [93.8, 97.9] | 53.9 [50.9, 57.0] | 99.6 | ahead | | Spanish | 348 | 97.3 [95.7, 98.6] | 46.7 [43.1, 50.4] | 99.6 | ahead | | Portuguese | 349 | 94.4 [91.8, 96.8] | 45.3 [41.8, 48.8] | 99.6 | ahead | | Italian | 368 | 96.5 [94.5, 98.2] | 51.0 [48.0, 54.2] | 99.6 | ahead | | Dutch | 369 | 97.2 [95.0, 98.8] | 54.6 [51.7, 57.6] | 99.0 | ahead | | Chinese | 338 | 99.3 [98.2, 100.0] | 49.8 [46.4, 53.0] | 100.0 | ahead | | Turkish | 316 | 95.2 [92.9, 97.1] | 56.3 [53.1, 59.9] | 100.0 | ahead | | Hindi | 324 | 96.9 [95.2, 98.6] | 49.3 [45.8, 52.5] | 100.0 | ahead | No type in any language is behind Recognizers.Text (the `behind` verdict appears in none of the ten comparison files). "All types" pools types with ten to thirty items each, so per-type intervals are wide (the per-type rows are in the source files, and Duckling's, which are information, beside them). The comparator uses its own conventions for range ends and drops stated time zones; the gold uses ours, and the differences are listed in [recognizer-gold.md](https://recursiveai.org/onenlp/docs/recognizer-gold/). The Chinese test file was read twice after a rule change; the published row is the first read (`recognizer-gold.md`). Recognizer speed against Microsoft.Recognizers.Text 1.8.13, release gate, arm64 (English): numbers, units, currency and sequences 0.605 s against 31.874 s (52.7x faster; 402 MB allocated against 8,798 MB); with date/time 0.721 s against 27.085 s (37.6x; 616 MB against 4,879 MB); RSS 85 and 68 MB against 310 and 292 MB. **x64 release run:** 0.542 vs 26.140 s (48.23x), 402.3 vs 8,794.7 MB allocated without date/time; 0.621 vs 21.948 s (35.34x), 615.7 vs 4,877.2 MB with date/time. Both pass. RSS 113 / 103 MiB vs 344 / 325 MiB. [Load times, input counts and machine record](https://recursiveai.org/onenlp/docs/speed-x64/). The comparator is far slower in Turkish and Hindi (0.01 to 0.03 million characters per second); those speed rows are not part of the gate run. ## Fast tier: tagging, lemmas, parsing, entities Source for each language: its section in [results.md](https://recursiveai.org/onenlp/docs/results/) and the recipe's `PROVENANCE.md` (`recipes//...`). Treebanks are UD r2.18 test (English-EWT, German-GSD, French-GSD, Spanish-AnCora, Portuguese-Bosque, Italian-ParlaMint, Dutch-Alpino, Turkish-BOUN, Chinese-GSDSimp). Entity data is named per row. Ours / spaCy, each with its interval. Only the headline columns are shown here; XPOS, AllTags, UAS, CLAS, MLAS, sentences and per-type entity rows are in the sources. ### English | Input | Metric | Ours | spaCy 3.8.18 | Verdict | |---|---|---|---|---| | gold | UPOS | 94.16 [93.79, 94.54] | 94.29 [93.89, 94.66] | within noise | | gold | UFeats | 95.15 [94.86, 95.47] | 95.18 [94.88, 95.45] | within noise | | gold | Lemmas | 96.11 [95.77, 96.44] | 95.80 [95.42, 96.13] | within noise | | gold | LAS | 81.53 [80.62, 82.43] | 81.96 [81.03, 82.83] | within noise | | raw | UPOS | 92.90 [92.42, 93.35] | 92.45 [91.84, 93.06] | within noise | | raw | Lemmas | 95.03 [94.61, 95.43] | 93.69 [93.08, 94.29] | ahead | | raw | LAS | 77.26 [76.28, 78.25] | 78.53 [77.46, 79.55] | within noise | | raw | Sentences (parser) | 71.08 [69.06, 72.98] | 72.66 [70.48, 74.75] | within noise | | raw | MLAS | 65.30 [64.12, 66.48] | 69.05 [67.81, 70.21] | **behind** (information row, not gated) | | gold, UNER EWT test | Entities micro F1 | 63.09 [60.01, 65.89] | 62.32 [59.42, 65.33] | within noise | ### Other languages | Language | Input | Metric | Ours | spaCy 3.8.18 | Verdict | |---|---|---|---|---|---| | German | gold | UPOS | 94.56 [94.18, 94.94] | 94.59 [94.22, 94.95] | within noise | | German | gold | UFeats | 86.25 [85.60, 86.96] | 84.71 [84.03, 85.39] | ahead | | German | gold | Lemmas | 95.95 [95.67, 96.25] | 95.65 [95.34, 95.98] | within noise | | German | gold | LAS | 76.66 [75.63, 77.72] | 77.31 [76.29, 78.36] | within noise | | German | raw | UPOS | 94.22 [93.83, 94.61] | 90.13 [89.57, 90.67] | ahead | | German | raw | LAS | 74.87 [73.79, 75.92] | 70.77 [69.59, 71.95] | ahead | | German | gold, GermEval 2014 | Entities micro F1 | 66.28 [64.94, 67.61] | 65.77 [64.49, 67.19] | within noise | | French | gold | UPOS | 96.77 [96.33, 97.19] | 97.07 [96.68, 97.47] | within noise | | French | gold | UFeats | 96.34 [95.89, 96.81] | 96.41 [95.91, 96.88] | within noise | | French | gold | Lemmas | 97.51 [97.16, 97.84] | 97.49 [97.14, 97.84] | within noise | | French | gold | LAS | 86.96 [85.86, 87.96] | 85.98 [84.80, 87.12] | within noise | | French | raw | UPOS | 96.06 [95.60, 96.53] | 91.80 [91.07, 92.55] | ahead | | French | raw | LAS | 86.07 [84.92, 87.11] | 78.10 [76.77, 79.43] | ahead | | French | gold, WikiNER test | Entities micro F1 | 82.48 [81.74, 83.28] | 75.50 [74.68, 76.46] | ahead | | Spanish | gold | UPOS | 98.11 [97.98, 98.25] | 98.18 [98.04, 98.32] | within noise | | Spanish | gold | UFeats | 97.71 [97.56, 97.86] | 97.74 [97.58, 97.89] | within noise | | Spanish | gold | Lemmas | 98.84 [98.73, 98.95] | 98.72 [98.59, 98.83] | within noise | | Spanish | gold | LAS | 86.41 [85.90, 86.89] | 85.95 [85.44, 86.44] | within noise | | Spanish | raw | UPOS | 97.86 [97.71, 98.01] | 94.81 [94.56, 95.02] | ahead | | Spanish | raw | LAS | 86.15 [85.61, 86.64] | 81.82 [81.25, 82.37] | ahead | | Spanish | gold, WikiNER test | Entities micro F1 | 84.81 [84.08, 85.54] | 83.35 [82.63, 84.08] | ahead (paired +1.46 [0.84, 2.14]; PER 91.57 vs 90.45, ahead) | | Portuguese | gold | UPOS | 95.89 [95.62, 96.14] | 95.92 [95.66, 96.22] | within noise | | Portuguese | gold | UFeats | 94.51 [94.17, 94.80] | 94.11 [93.82, 94.42] | within noise | | Portuguese | gold | Lemmas | 97.30 [97.10, 97.51] | 96.90 [96.70, 97.11] | within noise | | Portuguese | gold | LAS | 81.80 [81.10, 82.49] | 81.96 [81.31, 82.61] | within noise | | Portuguese | raw | UPOS | 95.35 [95.05, 95.64] | 84.15 [83.59, 84.73] | ahead | | Portuguese | raw | LAS | 81.14 [80.44, 81.82] | 65.85 [65.02, 66.75] | ahead | | Portuguese | gold, UNER Bosque test | Entities micro F1 | 73.48 [70.95, 75.86] | 75.27 [72.73, 77.60] | within noise (precision 75.46 vs 76.96, paired -1.50 [-3.76, 0.83]) | | Italian | gold | UPOS | 94.44 [93.92, 94.96] | 94.10 [93.58, 94.60] | within noise | | Italian | gold | UFeats | 89.52 [88.85, 90.16] | 87.42 [86.67, 88.14] | ahead | | Italian | gold | Lemmas | 95.18 [94.73, 95.63] | 93.13 [92.58, 93.64] | ahead | | Italian | gold | LAS | 71.55 [70.26, 72.97] | 71.39 [70.26, 72.58] | within noise | | Italian | raw | UPOS | 94.01 [93.45, 94.51] | 83.53 [82.50, 84.48] | ahead | | Italian | raw | LAS | 71.55 [70.27, 72.90] | 58.38 [56.92, 59.86] | ahead | | Italian | gold, WikiNER test | Entities micro F1 | 86.35 [85.73, 87.01] | 83.54 [82.85, 84.26] | ahead | | Dutch | gold | UPOS | 93.83 [93.16, 94.49] | 94.42 [93.77, 95.04] | within noise | | Dutch | gold | UFeats | 92.94 [92.24, 93.53] | 93.13 [92.47, 93.72] | within noise | | Dutch | gold | Lemmas | 91.61 [90.99, 92.14] | 91.83 [91.21, 92.39] | within noise | | Dutch | gold | LAS | 79.12 [77.87, 80.39] | 79.85 [78.58, 81.16] | within noise | | Dutch | raw | UPOS | 93.09 [92.40, 93.77] | 94.36 [93.67, 94.98] | within noise | | Dutch | raw | LAS | 78.18 [76.87, 79.46] | 79.57 [78.26, 80.89] | within noise | | Dutch | gold, WikiNER test | Entities micro F1 | 87.47 [86.91, 87.98] | 86.21 [85.70, 86.78] | ahead (paired +1.26 [0.76, 1.72]) | | Turkish | gold | UPOS | 88.97 [88.31, 89.58] | 88.04 [87.37, 88.64] | within noise | | Turkish | gold | UFeats | 78.15 [77.40, 78.90] | 78.30 [77.46, 79.05] | within noise | | Turkish | gold | Lemmas | 85.11 [84.38, 85.74] | 80.24 [79.47, 80.92] | ahead | | Turkish | gold | LAS | 62.80 [61.37, 64.19] | 57.87 [56.52, 59.18] | ahead | | Turkish | raw | UPOS | 86.02 [85.28, 86.68] | 86.82 [86.14, 87.48] | within noise | | Turkish | raw | LAS | 57.53 [56.08, 59.01] | 55.86 [54.53, 57.27] | within noise | | Turkish | gold, Turkish-WikiNER test | Entities micro F1 | 67.40 [64.90, 69.81] | 65.82 [63.16, 68.28] | within noise | | Chinese | gold (spaCy `gold_preproc`) | UPOS | 90.41 [89.70, 91.07] | 90.35 [89.70, 90.99] | within noise | | Chinese | gold | UFeats | 98.38 [98.13, 98.59] | 98.43 [98.19, 98.66] | within noise | | Chinese | gold | Lemmas | 99.87 [99.79, 99.94] | 99.91 [99.85, 99.96] | within noise | | Chinese | gold | LAS | 67.49 [65.94, 69.14] | 68.24 [66.47, 69.94] | within noise | | Chinese | gold | MLAS | 55.78 [53.92, 57.68] | 58.87 [56.85, 60.72] | within noise | | Chinese | raw | Words | 89.62 [88.87, 90.42] | 39.43 [38.60, 40.23] | ahead (spaCy's default Chinese tokenizer cuts single characters) | | Chinese | raw | UPOS | 81.90 [80.99, 82.86] | 34.92 [34.11, 35.68] | ahead | | Chinese | raw | LAS | 56.93 [55.23, 58.73] | 8.59 [7.94, 9.28] | ahead | | Chinese | gold, UNER GSDSIMP test | Entities micro F1 | 63.79 [59.39, 68.27] | 61.86 [57.66, 66.12] | within noise (precision 69.28 vs 67.08, recall 59.11 vs 57.40, all within noise) | | Chinese | raw, UNER GSDSIMP test | Entities micro F1 (character spans) | 53.71 [49.28, 58.49] | no reference | no raw spaCy row: the stored spaCy raw predictions are tokenized as single characters and do not align with the gold character spans, so the scorer gives a meaningless 13.30. Re-measured at the release gate (was 52.88, taken before the 0.6 threshold) | ### Hindi (self-labelled; information, not a gated comparison) There is no allowed Hindi treebank or entity corpus (`ud-licences.md`: Hindi-HDTB and the Samanantar-sourced NER sets are non-commercial), so the Hindi student learns from Hindi web text (FineWeb-2) labelled by the accurate tier: the heads of five large languages and of heads we trained on the same frozen encoder for Marathi, Sindhi and Punjabi (UD, CC BY-SA), combined by agreement, with explicit adult content filtered out of the text first (`recipes/hi/fast/README.md`). Hindi-PUD (1,000 sentences, CC BY-SA 3.0, sha256 `3562b7f4...`, test only) was read once for the final model, which is the second round of this recipe; the first round (labels from nine unrelated languages) is shown as "before". Intervals are 95% bootstrap (1,000 resamples, seed 20261107). The "labeller" is the unfiltered ensemble of the source heads on the same input, the ceiling of what the student was shown. spaCy 3.8.18 is the black-box baseline trained on the same labelled sentences with its default configuration and the pipeline `morphologizer,parser` (the labels have no XPOS or lemmas). The published Stanza, UDPipe 2 and Trankit numbers are on Hindi-HDTB with non-commercial training data and are not comparable (`references.md`, "Hindi"). **Hindi cannot be brought within noise of the published references with the training data that may be used.** | Mode | Metric | Student | Labeller | spaCy, same labels | Verdict against spaCy | Student before | |---|---|---|---|---|---|---| | gold | UPOS | 83.19 [82.65, 83.75] | 85.55 [85.05, 86.08] | 83.78 [83.24, 84.31] | within noise | 70.93 | | gold | UFeats | 34.28 [33.67, 34.90] | 34.63 [34.00, 35.27] | 34.29 [33.66, 34.90] | within noise | 34.01 | | gold | Lemmas (form) | 72.68 [72.15, 73.24] | none | none | n/a | 72.68 | | gold | UAS | 65.00 [64.01, 66.09] | 60.88 [59.84, 61.93] | 65.00 [64.01, 66.12] | within noise | 42.57 | | gold | LAS | 52.79 [51.85, 53.77] | 49.04 [48.06, 50.05] | 53.44 [52.48, 54.47] | within noise | 31.90 | | gold | CLAS | 45.60 [44.56, 46.69] | 44.52 [43.49, 45.56] | 46.46 [45.41, 47.57] | within noise | 29.62 | | gold | MLAS | 7.72 [7.30, 8.17] | 8.16 [7.74, 8.62] | 8.36 [7.91, 8.81] | within noise | 5.14 | | raw | Tokens | 99.38 [99.23, 99.53] | 99.38 | 99.82 [99.72, 99.89] | behind (spaCy's tokenizer) | 99.38 | | raw | Sentences | 89.68 [87.47, 91.82] | 98.31 [97.22, 99.25] (rule sentencizer) | 91.30 [89.07, 93.19] | within noise | 90.75 | | raw | UPOS | 82.41 [81.85, 82.95] | 84.99 [84.45, 85.54] | 83.65 [83.11, 84.18] | behind (1.2 points) | 70.49 | | raw | UFeats | 34.03 [33.43, 34.65] | 34.23 [33.61, 34.85] | 34.22 [33.60, 34.82] | within noise | 33.72 | | raw | UAS | 65.10 [64.09, 66.14] | 58.47 [57.47, 59.55] | 64.77 [63.77, 65.88] | within noise | 42.73 | | raw | LAS | 52.87 [51.93, 53.84] | 46.76 [45.81, 47.78] | 53.32 [52.35, 54.36] | within noise | 32.00 | | raw | CLAS | 45.54 [44.52, 46.67] | 42.60 [41.58, 43.68] | 46.40 [45.38, 47.57] | within noise | 29.80 | | raw | MLAS | 7.56 [7.16, 8.00] | 7.95 [7.55, 8.39] | 8.32 [7.86, 8.77] | within noise | 5.23 | The labeller's own gold-token scores rose from UPOS 72.77 and LAS 26.43 in the first round to 85.55 and 49.04. The student is 2.4 points below the labeller on UPOS and 3.8 above it on LAS (a parser trained on the medoid trees of many sources is more consistent than any one source). Against spaCy trained on the same labels it is within noise on every gold-token row and on raw UFeats, UAS, LAS, CLAS, MLAS and Sentences, and behind on raw UPOS (1.2 points; the tokenizer differs, 99.38 against 99.82). **Lemmas:** the model has no lemmatizer, and an edit-tree lemmatizer trained on Marathi was not clearly better than copying the form on 315 nouns, verbs and adjectives that we lemmatized by hand (62.5% against 61.0%), so the package fills `Token.Lemma` with the token text; the Lemmas row above is that, 72.7% on Hindi-PUD, and wrong for inflected words. Words are not scored separately from Tokens: Hindi has no multiword tokens. **Entities (information only).** There is no Hindi entity gold we may use. A spot check on 91 held-out web sentences (61 entities) that we labelled by hand (`testdata/hi/nerspot.iob2`; PER, ORG, LOC): student 70.48 [57.14, 82.05] F1 (precision 84.09, recall 60.66), labeller 77.48 [65.35, 86.27], spaCy on the same labels 70.48 [56.47, 82.05]; the first round's student scored 67.92. On held-out labelled text the student reaches 79.36 F1 against the labeller's own entities. This is not a gated row. **Hand-annotated development sentences.** 100 sentences (931 tokens, `testdata/hi/selfdev.conllu`: 17 web sentences and 83 that we wrote), gold tokens, UPOS / UAS / LAS: first-round student 80.56 [78.16, 82.80] / 70.46 / 55.85; second-round student 91.30 [89.43, 92.99] / 83.46 / 75.83; labeller (used set) 92.16 [90.64, 93.62] / 82.38 / 74.44; the nine large languages alone 83.03 / 71.64 / 59.51. These sentences ranked the source languages and the labelling model (`recipes/hi/fast/README.md`), so they are a development set, not a test. Indicative speed (x64 Ryzen 9 7950X3D, 1 thread, load 7 to 9 from other jobs): full pipeline about 4.1k to 4.6k tokens/s over 20 repetitions on the development sentences (3.5k to 4.0k in the first round at higher load; the architecture and sizes are unchanged), 6.5 kB allocated per token. ## Text classification and span categorization (English) Reference: spaCy 3.8.18 `textcat`, `textcat_multilabel` and `spancat`, default configurations, same data. Sources: `recipes/en/textcat/README.md`, `recipes/en/textcat-multilabel/README.md`, `recipes/en/spancat/README.md`. | Task, data | Metric | Ours | spaCy | Verdict | |---|---|---|---|---| | Multi-class, BANKING77 test (3,080) | Accuracy | 89.87 [88.80, 90.94] | 86.69 [85.42, 87.92] | ahead (paired +3.18 [2.08, 4.35]) | | | Macro-F1 | 89.87 [88.71, 90.81] | 86.17 [84.78, 87.22] | ahead | | Multi-label, EUR-LEX test (5,000) | Micro-F1 | 59.60 [58.92, 60.30] | 44.86 [44.07, 45.69] | ahead | | | Macro-F1 | 40.93 [39.78, 41.78] | 15.66 [15.27, 16.05] | ahead | | | Macro ROC-AUC | 91.81 [91.47, 92.15] | 67.29 [66.65, 67.93] | ahead | | Spans, UNER English-EWT test (threshold 0.8, chosen on dev) | Span F1 | 63.79 [60.78, 66.63] | 60.19 [57.02, 63.40] | ahead (paired +3.60 [1.14, 6.07]) | | | Precision | 75.98 [72.83, 79.18] | 75.38 [72.01, 78.86] | within noise (paired +0.60 [-3.02, 4.17]) | | | Recall | 54.96 [51.67, 58.24] | 50.09 [46.64, 53.48] | ahead (paired +4.87 [2.44, 7.10]) | For information, the LexGLUE paper's EUR-LEX numbers on the same test split (TF-IDF with an SVM: micro-F1 71.3, macro-F1 51.4; BERT-base: 71.4 and 57.2) are above ours; ours is a from-scratch fast-tier model with a 512-word cut. ## Embeddings Two shipped models, scored with `mteb` 2.27.1 as a black box. "Ours minus sentence-transformers" is 0.00 on every task, because the vectors agree to 4e-7 and 3e-7 (maximum absolute difference, 2,103 texts, zero token-ID mismatches). "Published" is the model card's number. The gate is runtime fidelity (same model, same harness); the card gaps below are shown, not hidden. Source: [embeddings.md](https://recursiveai.org/onenlp/docs/embeddings/). | Model | Task | Published | sentence-transformers 6.2.0 | Ours | Ours minus published | |---|---|---|---|---|---| | bge-small-en-v1.5 | STSBenchmark | 85.86 | 85.86 | 85.86 | 0.00 | | | Banking77Classification | 85.74 | 81.75 | 81.75 | **-3.99** | | | TwentyNewsgroupsClustering | 48.52 | 48.63 | 48.63 | +0.10 | | | TwitterSemEval2015 | 73.33 | 73.24 | 73.24 | -0.09 | | | AskUbuntuDupQuestions | 62.59 | 62.58 | 62.58 | 0.00 | | | SciFact | 71.28 | 71.27 | 71.27 | 0.00 | | multilingual-e5-small | STSBenchmark | 84.01 | 84.11 | 84.11 | +0.10 | | | STS22 (multilingual) | 66.01 | 65.13 | 65.13 | -0.88 (66.10 under `mteb` 1.12.94: +0.09) | | | Banking77Classification | 79.42 | 70.44 | 70.44 | **-8.97** | | | MTOPDomainClassification (en, de, es, fr) | 88.01 | 85.59 | 85.59 | **-2.42** | | | TwentyNewsgroupsClustering | 33.22 | 32.81 | 32.81 | -0.41 | | | TwitterSemEval2015 | 70.95 | 70.75 | 70.75 | -0.20 | | | AskUbuntuDupQuestions | 57.88 | 56.42 | 56.42 | **-1.46** | | | SciFact | 67.70 | 67.70 | 67.70 | 0.00 | The four bold cells are below the card for sentence-transformers itself, unchanged across `mteb` 1.1.1, 1.12.94 and 2.27.1; the card's numbers could not be reproduced from the published model with the released harness, whoever computes the vectors. The int8 variants are information only (0.3 to 3.8 points lower). ## Open-type NER Release measurement through packed `Models.Multilingual`/`FrozenEncoder` **0.3.0-preview.1**, core/Neural **0.4.0-preview.1**, frozen student-v1 and default threshold **0.5**. The newer scratch feed had no multilingual package: this is not a test of a newly packed 0.4.0-preview.1 model. The embedded head, frozen encoder and tokenizer match the current weight manifests byte for byte. Every score and interval below reproduces `recipes/openner/results.md` to its reported precision. Strict token-span-and-type micro F1; 1,000 sentence-bootstrap resamples, seed 1. Published reference: Zaratiana et al., [GLiNER, arXiv:2311.08526v1](https://arxiv.org/abs/2311.08526v1), **Table 1, §4.1**. Model sizes and encoders (§3.2, Table 1): **GLiNER-S 50M** (primary, smallest), **M 90M**, **L 0.3B**, DeBERTa-v3 family (as reported for S/M/L, with no exact checkpoint identifiers or revisions specified). Ours is a frozen **6×384 student-v1, 35.5M encoder weights + 595,969 head parameters**, about 36.1M total. The head parameter count is measured from the loaded package. | Row (test) | Sentences scored | Ours F1 [95% CI] | GLiNER-S | M (info) | L (info) | Position vs S point | Protocol verdict | |---|---:|---|---:|---:|---:|---|---| | UNER English-EWT, fixed PER/ORG/LOC | 2,077 | **67.14 [64.34, 69.97]** | — | — | — | — | Not comparable: no UNER row in paper | | Few-NERD, seen types | 37,637 | **49.89 [49.63, 50.15]** | — | — | — | — | Not comparable: no Few-NERD row in paper | | CrossNER politics, zero-shot | 651 | **41.27 [38.58, 44.09]** | 61.5 | 68.6 | 72.6 | **Behind** | Not comparable: protocol not identical/verified | | CrossNER science, zero-shot | 543 | **36.50 [33.09, 40.06]** | 55.6 | 58.1 | 62.6 | **Behind** | Not comparable: protocol not identical/verified | | CrossNER music, zero-shot | 465 | **26.01 [23.67, 28.35]** | 60.9 | 69.4 | 69.6 | **Behind** | Not comparable: protocol not identical/verified | | CrossNER literature, zero-shot | 416 | **30.38 [27.88, 33.05]** | 60.0 | 59.7 | 64.4 | **Behind** | Not comparable: protocol not identical/verified | | CrossNER AI, zero-shot | 431 | **21.10 [17.79, 24.52]** | 50.7 | 51.8 | 57.2 | **Behind** | Not comparable: protocol not identical/verified | Position is relative to our CI, since the paper has no CI: published S above our upper bound = behind, below our lower bound = ahead, otherwise within noise. It is not a paired test. **All five CrossNER rows lose numerically to S, M and L**; the gaps to S are politics **20.23**, science **19.10**, music **34.89**, literature **29.62**, AI **29.60** points. None proves the published-small-model quality target is met. “Not comparable” is not a passing verdict. The fixed-type black-box spaCy 3.8.18 comparison on UNER English-EWT remains **62.32 [59.45, 65.20]**, trained on the same UNER data with documented defaults. Ours is **ahead** by the point-vs-CI convention, but intervals overlap and this is not a new paired test or a zero-shot reference. Protocol caveats ([full audit](https://recursiveai.org/onenlp/docs/open-ner/#protocol-audit)): our unchanged scorer uses gold tokens and skips empty/>120-token sentences and whitespace-containing tokens (none excluded in UNER/CrossNER; 11 of 37,648 excluded in Few-NERD); asks the retained file's whole type inventory using our expanded label names; and keeps scores **≥0.5**, whereas paper §2.3 says **>0.5**. Paper §3.4 names the OOD benchmark and exact-match entity F1 but supplies no split hashes, retained sentence counts, full label-name lists or matching filters. Equivalence cannot be established from the paper alone. Few-NERD includes both fine and coarse gold labels on the same mention, with flat predictions: it is not the conventional fine-only benchmark. The [model/protocol notes](https://recursiveai.org/onenlp/docs/open-ner/#published-comparison) and `tools/OneNlp.OpenNerGate/README.md` give provenance and reproduction commands. ## Accurate tier The distilled package covers nine languages; the base-size package covers 84 for syntax and 19 for entities. The gate is the paired comparison with spaCy 3.8.18 trained on the same data: 144 rows, 117 ahead, 14 within noise, **13 behind** (listed in [accurate-tier-rows.md](https://recursiveai.org/onenlp/docs/accurate-tier-rows/#paired-spacy-3818-comparison-gate)). Published research systems (Stanza, UDPipe 2, Trankit) are information: they are ahead on most rows (1,205 of 1,314 best-published comparisons, 534 of 650 against Stanza 1.5.1), typically by 3–9 LAS points. Every row is in [accurate-tier-rows.md](https://recursiveai.org/onenlp/docs/accurate-tier-rows/). The following English rows are historical information, not a summary of the latest 84-language weights. Their values are unchanged. They come from [accurate-encoder.md](https://recursiveai.org/onenlp/docs/accurate-encoder/), `recipes/accurate/student-v1/RESULTS.md` and, for the references, [references.md](https://recursiveai.org/onenlp/docs/references/) (published numbers on other UD releases: compare the direction, not the last digit). | Model | Data (test) | Metric | Ours | Reference | Reading | |---|---|---|---|---|---| | Distilled student, fine-tuned (6 layers, 35.5 M weights) | UD English-EWT r2.18, gold tokens | UPOS | 95.88 [95.55, 96.19] | English fast tier 93.89; UDPipe 2 (gold, UD 2.17) 97.61 | above the fast tier (+1.99 [1.63, 2.36]); below the published UDPipe 2 | | Distilled student, frozen, tuned head | UNER English-EWT | Entities micro F1 | 75.48 [72.66, 78.03] | English fast tier 61.52; published XLM-R large 85.8 | above the fast tier; below the published large model | | mmBERT-base, fine-tuned 0.58 epoch (one chunk) | UD English-EWT r2.18, gold tokens | UPOS | 97.07 [96.79, 97.35] | spaCy 94.29; UDPipe 2 (gold, UD 2.17) 97.61 | above spaCy by 2.78 [2.44, 3.16]; below the published UDPipe 2 | ### Accurate-tier speed and load (indicative, x64) One-thread English, first 100 EWT r2.18 test texts (2,206 word tokens), median of three passes in a non-exclusive heavy job at load 12.6–13.8: `.Multilingual` 462 word tokens/s, model load 3,512 ms, cold/throughput peak RSS 809/814 MiB; `.Large` 150 word tokens/s, model load 3,134 ms, peak 3,904/4,055 MiB. Installed weights 529,158,395 and 3,508,907,198 bytes. These are information only, not idle-machine speed gates. [Input hash, first-call times and method](https://recursiveai.org/onenlp/docs/speed-x64/#accurate-tiers-indicative-not-a-speed-gate). Existing quality rows are unchanged. ## Misses and caveats - **English raw MLAS** 65.30 against spaCy's 69.05: outside the interval; information, not one of the gated rows. - **Span categorizer:** at spaCy's threshold of 0.5 precision was 70.33 against 75.38 (paired -5.05 [-8.63, -1.42]); the shipped default threshold is 0.8, chosen on the development split (`recipes/en/spancat/README.md`). - **Embeddings:** the five cells below the model cards, shared with sentence-transformers (above). - **Language ID:** the languages named under the table are below Lingua on short text. - **Entities per type:** per-type rows are in each source file; several have intervals that overlap with a paired difference that excludes zero (for example English LOC with the single-encoder option); the packaged English model has none of those. - **Gaps:** Hindi has a fast tier but only on transferred labels (see "Hindi", not a gated comparison); no German, Spanish, Portuguese, Italian, Dutch, Turkish or Chinese rows against Catalyst (it has English models only); many accurate-tier published-reference rows remain below the reference (see [accurate-tier-rows.md](https://recursiveai.org/onenlp/docs/accurate-tier-rows/)); open-type NER rows are above, with all five CrossNER rows numerically behind GLiNER-S and protocol equivalence unverified. - **Speed against Catalyst 26.10.6514** (English, 1 thread, arm64 Apple M2, release gate; median tokens/s of the last 10 of 12 passes): tagging path (Catalyst tagging 1,007,344 tokens/s, 307 B/token; ours sentences, UPOS/UFeats and lemmas 670,783, 330 B/token) is **0.666x**, above the 0.5x bar. It counts under matched or better accuracy on raw EWT test: Tokens 99.00 vs 94.47, Sentences 71.08 vs 64.57, UPOS 92.90 vs 80.70, Lemmas 95.03 vs 83.84. NER path: Catalyst with WikiNER 478,220 vs ours sentences and entities 333,620 (0.70x), not matched work (ours excludes tagging; Catalyst's NER needs it), and so indicative only; entities on UNER character spans 63.09 vs 57.77. Ours parser path 25,535 and full pipeline 22,634 tokens/s (Catalyst has no parser). RSS 311 MB against 1,401 MB. English model load time not measured in this run. **x64 release run:** tagging 833,483 vs 1,049,762.5 tokens/s (**0.794x, passes ≥0.5x**), 330 vs 307 B/token. NER 443,997.5 vs 471,738 (0.941x, information only); parser 45,235.5 and full pipeline 38,313 tokens/s. Process peak RSS ours 1,240 vs Catalyst 383 MiB; English load not measured. Same accuracy condition as arm64. [Full paths, samples and machine record](https://recursiveai.org/onenlp/docs/speed-x64/). - **Spanish gold LAS** is within noise at the release gate (86.41 [85.90, 86.89] vs 85.95); an earlier version of this table showed an older parser. - **Rows behind at the gate:** English raw MLAS (information); Hindi raw Tokens 99.38 vs 99.82 and raw UPOS 82.41 vs 83.65 (Hindi is not a gated comparison); Language ID below Lingua on short text for the languages named under that table; embeddings cells below the model cards (shared with sentence-transformers); tokenizer load time (three arm64 rows and four x64 rows, above). The current Spanish PER row is ahead, not behind. All five open-type CrossNER rows are numerically behind GLiNER-S (protocol caveats above). No recognizer, fast-tier gated or tokenizer exactness row is behind. - **Not re-run at the gate:** Duckling and Lingua predictions were reused from pinned earlier runs; English entities have no paired spaCy file at the gate; embeddings MTEB rows stand because the package manifests are unchanged since those runs (parity re-run on the packaged files: bge max abs 4.03e-7, e5 3.20e-7, 0 id mismatches). # Results Measured rows. The tokenizer speed rows below are final: they were taken in one benchmark-queue group per machine, alone on the machine, with `bench/scripts/tokenizer-rows.sh`. Speed numbers elsewhere on this page are indicative until they are re-measured alone on the machine. ## Tokenizer speed against Microsoft.ML.Tokenizers Both libraries encode the same text to token IDs, count tokens and decode IDs, one thread. Before a row is timed the harness checks that the IDs from both are identical on that input (and that both decode to the same text); a row whose outputs differ is shown as such and not timed. Our correctness reference is Hugging Face `tokenizers` and `tiktoken` (see `parity.md`); where the comparator differs from them that is information, not a failure of ours. - **Gate:** ours at or faster than the managed reference on every timed row. **Result: all 47 timed rows on x64 and all 47 on arm64 pass** (speedup 1.00x or more). Load time is reported, not gated. - **Speedup** is comparator time divided by ours: the median per operation (5 to 25 samples after a half-second warm-up). - **Alloc** is bytes allocated per operation on the calling thread. Ours writes into a caller's buffer (`EncodeToIds(text, ids)`); the comparator returns a list (`EncodeToIds(string)`), which is the call each library offers for the job. Decode allocates the result string in both. - **Inputs** (from the Tatoeba sentence export, fixed by the first run and reused on the second machine): short = one English sentence (59 chars); medium = eight English sentences (513 chars); long = every 600th sentence of the export, 990,287 bytes of UTF-8 across all the languages it holds. - **Operations:** `encode-to-ids` without special tokens; `count` = token count; `decode` of the IDs without special tokens. - **Families and files:** WordPiece = bert-base-uncased (`tokenizer.json`; the comparator reads `vocab.txt` with lowercasing and accent stripping on, as the Hugging Face file has them); Unigram = xlm-roberta-base (`tokenizer.json`; the comparator reads `sentencepiece.bpe.model`); SentencePiece BPE with byte fallback = TinyLlama (`tokenizer.json`); byte-level BPE = gpt2 (`tokenizer.json`); Tiktoken = `cl100k_base` and `o200k_base` (`.tiktoken` files, comparator `TiktokenTokenizer.CreateForModel`). - **Peak extra RSS** (growth of the resident-set high-water mark during the samples) is in the harness output (`rows.md`); it follows the garbage collector's timing and is not tabulated here. It is only measured on Linux. ### Rows not timed, and why - **WordPiece, medium decode:** the comparator's decoded text differs from ours at character 74 (lengths 525 and 519); it does not apply the Hugging Face `cleanup` rules to spaces before punctuation and contractions. Ours matches Hugging Face (see `parity.md`). - **WordPiece, long (encode, count, decode):** the IDs differ from the first difference at token 159 (ours 1456, the comparator 100, its unknown token): the comparator gives the unknown token for text that the Hugging Face tokenizer, and ours, split into pieces. 357,483 tokens (ours) against 356,166. - **Unigram, long (encode, count, decode):** the IDs differ at token 30,076 (ours 49,729, the comparator 1,759); 250,378 tokens against 251,115. On short and medium the IDs are identical once the comparator's numbering is mapped to the Hugging Face file's: the comparator numbers the pieces of `sentencepiece.bpe.model`, and the Hugging Face file puts four control tokens first, so a piece ID p is token p + 1 (the three control pieces swap places). - **t5-small** is not a Unigram row: the comparator throws `IndexOutOfRangeException` while building its Unigram model from t5-small's `spiece.model`. ### Machines and pins - **x64:** AMD Ryzen 9 7950X3D (16 cores, 32 threads), Linux 7.0.0-38-generic, .NET SDK 10.0.403, Release, non-concurrent GC, ICU on. Run on 2026-11-05 at 03:27 UTC. Load average before the run 3.6 (1 min), 15.5 (5 min), 17.4 (15 min), the tail of an earlier job; during the run (10 s samples) it fell from 3.6 to about 1.5; the machine was otherwise idle (the queue checked that the CPU was idle before starting). - **arm64:** Apple M2 (8 cores), macOS (Darwin 27.0.0), .NET SDK 10.0.403 (user-local install), Release, non-concurrent GC, ICU on. Run on 2026-11-05 at 04:40 UTC. Load average before the run 2.7 / 3.9 / 12.4; during the run 2.4 to 3.3 (the harness's own thread and the OS). - **Versions:** Microsoft.ML.Tokenizers 2.0.1 with Microsoft.ML.Tokenizers.Data.{Gpt2, Cl100kBase, O200kBase} 2.0.1; OneNLP at the commit of this page; information column: `tokenizers` 0.22.1 and `tiktoken` 0.14.0 (Python, one thread, x64 only). - **Tokenizer files:** the Hugging Face model files named in `parity.md`; the `.tiktoken` files from the published encodings. - **Run with** `bench/scripts/tokenizer-rows.sh MODELS INPUTS OUT`; `INPUTS` is the Tatoeba export or a directory with `short.txt`, `medium.txt` and `long.txt`. ### Rows Times are medians per operation. `native` is the information column: the Hugging Face `tokenizers` / `tiktoken` Python package on the x64 machine, same text. #### x64 | family | operation | size | ours | Microsoft.ML.Tokenizers | speedup | alloc ours | alloc comparator | native (info, x64) | |---|---|---|---:|---:|---:|---:|---:|---:| | WordPiece (bert-base-uncased) | load | - | 31.88 ms | 19.66 ms | 0.62x | | | | | WordPiece (bert-base-uncased) | encode-to-ids | short | 281 ns | 873 ns | 3.10x | 0 B | 456 B | 15.00 µs | | WordPiece (bert-base-uncased) | count | short | 283 ns | 835 ns | 2.96x | 0 B | 240 B | 14.82 µs | | WordPiece (bert-base-uncased) | decode | short | 70 ns | 161 ns | 2.31x | 144 B | 176 B | 5.52 µs | | WordPiece (bert-base-uncased) | encode-to-ids | medium | 2.85 µs | 7.81 µs | 2.74x | 0 B | 2.3 KB | 108.94 µs | | WordPiece (bert-base-uncased) | count | medium | 2.80 µs | 7.51 µs | 2.68x | 0 B | 1.1 KB | 108.24 µs | | WordPiece (bert-base-uncased) | decode | medium | not timed | not timed | - | | | | | WordPiece (bert-base-uncased) | encode-to-ids | long | not timed | not timed | - | | | | | WordPiece (bert-base-uncased) | count | long | not timed | not timed | - | | | | | WordPiece (bert-base-uncased) | decode | long | not timed | not timed | - | | | | | Unigram (xlm-roberta-base) | load | - | 226.30 ms | 705.90 ms | 3.12x | | | | | Unigram (xlm-roberta-base) | encode-to-ids | short | 199 ns | 1.35 µs | 6.79x | 0 B | 216 B | 12.97 µs | | Unigram (xlm-roberta-base) | count | short | 200 ns | 1.43 µs | 7.14x | 0 B | 0 B | 13.25 µs | | Unigram (xlm-roberta-base) | decode | short | 69 ns | 146 ns | 2.11x | 144 B | 176 B | 1.02 µs | | Unigram (xlm-roberta-base) | encode-to-ids | medium | 1.67 µs | 11.31 µs | 6.76x | 0 B | 2.2 KB | 106.73 µs | | Unigram (xlm-roberta-base) | count | medium | 1.59 µs | 12.12 µs | 7.63x | 0 B | 0 B | 104.94 µs | | Unigram (xlm-roberta-base) | decode | medium | 651 ns | 1.38 µs | 2.13x | 1.0 KB | 1.1 KB | 10.47 µs | | Unigram (xlm-roberta-base) | encode-to-ids | long | not timed | not timed | - | | | | | Unigram (xlm-roberta-base) | count | long | not timed | not timed | - | | | | | Unigram (xlm-roberta-base) | decode | long | not timed | not timed | - | | | | | SentencePiece BPE, byte fallback (TinyLlama) | load | - | 48.44 ms | 10.91 ms | 0.23x | | | | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | short | 89 ns | 4.14 µs | 46.76x | 0 B | 1.6 KB | 7.37 µs | | SentencePiece BPE, byte fallback (TinyLlama) | count | short | 88 ns | 4.11 µs | 46.67x | 0 B | 1.4 KB | 7.48 µs | | SentencePiece BPE, byte fallback (TinyLlama) | decode | short | 83 ns | 148 ns | 1.78x | 144 B | 176 B | 2.97 µs | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | medium | 12.05 µs | 42.03 µs | 3.49x | 0 B | 12.3 KB | 77.08 µs | | SentencePiece BPE, byte fallback (TinyLlama) | count | medium | 12.28 µs | 42.40 µs | 3.45x | 0 B | 11.1 KB | 78.32 µs | | SentencePiece BPE, byte fallback (TinyLlama) | decode | medium | 671 ns | 1.52 µs | 2.27x | 1.0 KB | 1.1 KB | 24.45 µs | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | long | 86.57 ms | 243.17 ms | 2.81x | 1.8 MB | 21.5 MB | 359.33 ms | | SentencePiece BPE, byte fallback (TinyLlama) | count | long | 88.74 ms | 241.62 ms | 2.72x | 1.8 MB | 17.5 MB | 358.47 ms | | SentencePiece BPE, byte fallback (TinyLlama) | decode | long | 4.48 ms | 6.70 ms | 1.49x | 1.6 MB | 1.6 MB | 71.05 ms | | Byte-level BPE (gpt2) | load | - | 20.22 ms | 26.44 ms | 1.31x | | | | | Byte-level BPE (gpt2) | encode-to-ids | short | 438 ns | 677 ns | 1.55x | 0 B | 312 B | 11.05 µs | | Byte-level BPE (gpt2) | count | short | 441 ns | 633 ns | 1.43x | 0 B | 96 B | 11.08 µs | | Byte-level BPE (gpt2) | decode | short | 67 ns | 113 ns | 1.70x | 144 B | 176 B | 1.23 µs | | Byte-level BPE (gpt2) | encode-to-ids | medium | 3.99 µs | 6.21 µs | 1.56x | 0 B | 1.2 KB | 86.19 µs | | Byte-level BPE (gpt2) | count | medium | 3.93 µs | 5.68 µs | 1.45x | 0 B | 96 B | 83.95 µs | | Byte-level BPE (gpt2) | decode | medium | 450 ns | 978 ns | 2.17x | 1.0 KB | 1.1 KB | 9.14 µs | | Byte-level BPE (gpt2) | encode-to-ids | long | 28.97 ms | 63.26 ms | 2.18x | 0 B | 14.0 MB | 382.68 ms | | Byte-level BPE (gpt2) | count | long | 28.90 ms | 61.19 ms | 2.12x | 0 B | 10.0 MB | 380.77 ms | | Byte-level BPE (gpt2) | decode | long | 4.59 ms | 6.78 ms | 1.48x | 1.6 MB | 1.6 MB | 43.94 ms | | Tiktoken cl100k_base | load | - | 18.31 ms | 19.22 ms | 1.05x | | | | | Tiktoken cl100k_base | encode-to-ids | short | 475 ns | 704 ns | 1.48x | 408 B | 312 B | 2.07 µs | | Tiktoken cl100k_base | count | short | 520 ns | 644 ns | 1.24x | 408 B | 96 B | 2.03 µs | | Tiktoken cl100k_base | decode | short | 67 ns | 111 ns | 1.65x | 144 B | 176 B | 364 ns | | Tiktoken cl100k_base | encode-to-ids | medium | 5.02 µs | 6.32 µs | 1.26x | 1.3 KB | 1.2 KB | 16.57 µs | | Tiktoken cl100k_base | count | medium | 5.15 µs | 6.05 µs | 1.17x | 1.3 KB | 96 B | 16.66 µs | | Tiktoken cl100k_base | decode | medium | 508 ns | 1.01 µs | 1.99x | 1.0 KB | 1.1 KB | 1.49 µs | | Tiktoken cl100k_base | encode-to-ids | long | 47.85 ms | 65.20 ms | 1.36x | 8.8 MB | 12.5 MB | 51.54 ms | | Tiktoken cl100k_base | count | long | 47.77 ms | 63.95 ms | 1.34x | 8.8 MB | 8.5 MB | 52.25 ms | | Tiktoken cl100k_base | decode | long | 4.47 ms | 6.07 ms | 1.36x | 1.6 MB | 1.6 MB | 7.22 ms | | Tiktoken o200k_base | load | - | 27.60 ms | 43.85 ms | 1.59x | | | | | Tiktoken o200k_base | encode-to-ids | short | 564 ns | 793 ns | 1.41x | 408 B | 312 B | 1.21 µs | | Tiktoken o200k_base | count | short | 555 ns | 740 ns | 1.33x | 408 B | 96 B | 1.22 µs | | Tiktoken o200k_base | decode | short | 71 ns | 111 ns | 1.56x | 144 B | 176 B | 386 ns | | Tiktoken o200k_base | encode-to-ids | medium | 5.25 µs | 6.83 µs | 1.30x | 1.3 KB | 1.2 KB | 9.38 µs | | Tiktoken o200k_base | count | medium | 5.26 µs | 6.48 µs | 1.23x | 1.3 KB | 96 B | 9.45 µs | | Tiktoken o200k_base | decode | medium | 519 ns | 943 ns | 1.82x | 1.0 KB | 1.1 KB | 1.45 µs | | Tiktoken o200k_base | encode-to-ids | long | 44.38 ms | 68.68 ms | 1.55x | 8.8 MB | 10.2 MB | 37.04 ms | | Tiktoken o200k_base | count | long | 43.92 ms | 65.53 ms | 1.49x | 8.8 MB | 6.2 MB | 37.88 ms | | Tiktoken o200k_base | decode | long | 3.99 ms | 5.38 ms | 1.35x | 1.6 MB | 1.6 MB | 5.63 ms | #### arm64 | family | operation | size | ours | Microsoft.ML.Tokenizers | speedup | alloc ours | alloc comparator | native (info, x64) | |---|---|---|---:|---:|---:|---:|---:|---:| | WordPiece (bert-base-uncased) | load | - | 33.32 ms | 19.15 ms | 0.57x | | | | | WordPiece (bert-base-uncased) | encode-to-ids | short | 305 ns | 971 ns | 3.18x | 0 B | 456 B | | | WordPiece (bert-base-uncased) | count | short | 306 ns | 955 ns | 3.12x | 0 B | 240 B | | | WordPiece (bert-base-uncased) | decode | short | 89 ns | 165 ns | 1.85x | 144 B | 176 B | | | WordPiece (bert-base-uncased) | encode-to-ids | medium | 2.77 µs | 10.49 µs | 3.79x | 0 B | 2.3 KB | | | WordPiece (bert-base-uncased) | count | medium | 2.77 µs | 10.42 µs | 3.77x | 0 B | 1.1 KB | | | WordPiece (bert-base-uncased) | decode | medium | not timed | not timed | - | | | | | WordPiece (bert-base-uncased) | encode-to-ids | long | not timed | not timed | - | | | | | WordPiece (bert-base-uncased) | count | long | not timed | not timed | - | | | | | WordPiece (bert-base-uncased) | decode | long | not timed | not timed | - | | | | | Unigram (xlm-roberta-base) | load | - | 206.07 ms | 757.73 ms | 3.68x | | | | | Unigram (xlm-roberta-base) | encode-to-ids | short | 171 ns | 1.45 µs | 8.51x | 0 B | 216 B | | | Unigram (xlm-roberta-base) | count | short | 168 ns | 1.57 µs | 9.38x | 0 B | 0 B | | | Unigram (xlm-roberta-base) | decode | short | 98 ns | 148 ns | 1.51x | 144 B | 176 B | | | Unigram (xlm-roberta-base) | encode-to-ids | medium | 1.32 µs | 13.04 µs | 9.90x | 0 B | 2.2 KB | | | Unigram (xlm-roberta-base) | count | medium | 1.30 µs | 14.68 µs | 11.25x | 0 B | 0 B | | | Unigram (xlm-roberta-base) | decode | medium | 751 ns | 1.57 µs | 2.10x | 1.0 KB | 1.1 KB | | | Unigram (xlm-roberta-base) | encode-to-ids | long | not timed | not timed | - | | | | | Unigram (xlm-roberta-base) | count | long | not timed | not timed | - | | | | | Unigram (xlm-roberta-base) | decode | long | not timed | not timed | - | | | | | SentencePiece BPE, byte fallback (TinyLlama) | load | - | 48.96 ms | 11.15 ms | 0.23x | | | | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | short | 108 ns | 3.63 µs | 33.63x | 0 B | 1.6 KB | | | SentencePiece BPE, byte fallback (TinyLlama) | count | short | 116 ns | 3.65 µs | 31.42x | 0 B | 1.4 KB | | | SentencePiece BPE, byte fallback (TinyLlama) | decode | short | 98 ns | 174 ns | 1.78x | 144 B | 176 B | | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | medium | 13.70 µs | 44.19 µs | 3.22x | 0 B | 12.3 KB | | | SentencePiece BPE, byte fallback (TinyLlama) | count | medium | 13.64 µs | 43.78 µs | 3.21x | 0 B | 11.1 KB | | | SentencePiece BPE, byte fallback (TinyLlama) | decode | medium | 687 ns | 1.99 µs | 2.89x | 1.0 KB | 1.1 KB | | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | long | 102.25 ms | 265.79 ms | 2.60x | 1.8 MB | 21.5 MB | | | SentencePiece BPE, byte fallback (TinyLlama) | count | long | 102.62 ms | 263.03 ms | 2.56x | 1.8 MB | 17.5 MB | | | SentencePiece BPE, byte fallback (TinyLlama) | decode | long | 4.48 ms | 7.12 ms | 1.59x | 1.6 MB | 1.6 MB | | | Byte-level BPE (gpt2) | load | - | 26.90 ms | 32.47 ms | 1.21x | | | | | Byte-level BPE (gpt2) | encode-to-ids | short | 396 ns | 638 ns | 1.61x | 0 B | 312 B | | | Byte-level BPE (gpt2) | count | short | 392 ns | 620 ns | 1.58x | 0 B | 96 B | | | Byte-level BPE (gpt2) | decode | short | 74 ns | 120 ns | 1.63x | 144 B | 176 B | | | Byte-level BPE (gpt2) | encode-to-ids | medium | 3.72 µs | 5.84 µs | 1.57x | 0 B | 1.2 KB | | | Byte-level BPE (gpt2) | count | medium | 3.71 µs | 5.66 µs | 1.53x | 0 B | 96 B | | | Byte-level BPE (gpt2) | decode | medium | 426 ns | 1.13 µs | 2.66x | 1.0 KB | 1.1 KB | | | Byte-level BPE (gpt2) | encode-to-ids | long | 24.74 ms | 66.39 ms | 2.68x | 0 B | 14.0 MB | | | Byte-level BPE (gpt2) | count | long | 24.92 ms | 65.01 ms | 2.61x | 0 B | 10.0 MB | | | Byte-level BPE (gpt2) | decode | long | 4.28 ms | 7.94 ms | 1.85x | 1.6 MB | 1.6 MB | | | Tiktoken cl100k_base | load | - | 19.96 ms | 19.20 ms | 0.96x | | | | | Tiktoken cl100k_base | encode-to-ids | short | 439 ns | 675 ns | 1.54x | 408 B | 312 B | | | Tiktoken cl100k_base | count | short | 430 ns | 622 ns | 1.45x | 408 B | 96 B | | | Tiktoken cl100k_base | decode | short | 61 ns | 116 ns | 1.92x | 144 B | 176 B | | | Tiktoken cl100k_base | encode-to-ids | medium | 4.17 µs | 6.01 µs | 1.44x | 1.3 KB | 1.2 KB | | | Tiktoken cl100k_base | count | medium | 4.05 µs | 5.88 µs | 1.45x | 1.3 KB | 96 B | | | Tiktoken cl100k_base | decode | medium | 447 ns | 1.12 µs | 2.51x | 1.0 KB | 1.1 KB | | | Tiktoken cl100k_base | encode-to-ids | long | 51.07 ms | 68.65 ms | 1.34x | 8.8 MB | 12.5 MB | | | Tiktoken cl100k_base | count | long | 50.61 ms | 67.49 ms | 1.33x | 8.8 MB | 8.5 MB | | | Tiktoken cl100k_base | decode | long | 3.80 ms | 6.42 ms | 1.69x | 1.6 MB | 1.6 MB | | | Tiktoken o200k_base | load | - | 23.05 ms | 50.26 ms | 2.18x | | | | | Tiktoken o200k_base | encode-to-ids | short | 501 ns | 816 ns | 1.63x | 408 B | 312 B | | | Tiktoken o200k_base | count | short | 493 ns | 762 ns | 1.55x | 408 B | 96 B | | | Tiktoken o200k_base | decode | short | 60 ns | 116 ns | 1.92x | 144 B | 176 B | | | Tiktoken o200k_base | encode-to-ids | medium | 4.86 µs | 7.56 µs | 1.56x | 1.3 KB | 1.2 KB | | | Tiktoken o200k_base | count | medium | 4.80 µs | 7.41 µs | 1.54x | 1.3 KB | 96 B | | | Tiktoken o200k_base | decode | medium | 432 ns | 1.09 µs | 2.52x | 1.0 KB | 1.1 KB | | | Tiktoken o200k_base | encode-to-ids | long | 45.61 ms | 70.04 ms | 1.54x | 8.8 MB | 10.2 MB | | | Tiktoken o200k_base | count | long | 45.55 ms | 67.84 ms | 1.49x | 8.8 MB | 6.2 MB | | | Tiktoken o200k_base | decode | long | 3.54 ms | 5.78 ms | 1.63x | 1.6 MB | 1.6 MB | | ## Language-ID and recognizer speed Single thread, one process per system and input set, after a warm-up over 200 inputs; three timed repetitions, and the table shows the median repetition (the first repetition of every system is slower while the JIT tiers up, and the comparators' first recognizer repetition is slower still). Run with `bench/scripts/langid-recognizer-rows.sh`; each machine ran alone in the benchmark queue. These are publishable rows for the two components. - **Speedup** is comparator time divided by ours; 1.00x or more passes. **Alloc** is bytes allocated on the calling thread. **Peak RSS** is the process's peak resident size (one process per system, so it includes that system's model load). - **Language ID** inputs: every 15th line of the held-out `main` evaluation set per bucket (`tools/bench/make-langid-bench.sh`): 4,970 words, 4,872 pairs, 5,000 sentences. Ours: the 75-language model (3.6 MB). Comparator: SearchPioneer.Lingua 2.0.0 built for the same 75 languages with its models preloaded, in high and low accuracy mode. Accuracy rows are in `recipes/langid/results/`. - **Recognizers** inputs: English text of 11,558 lines, 9,979,226 characters, built from the development split only (`tools/recognizers-gold/make_bench_text.py`). Row 1 compares the comparator's number, ordinal, percentage, currency, dimension, temperature, age, e-mail, URL, IP, phone, hashtag, mention and GUID models with our rules for the same types (the rule files without date/time). Row 2 adds the date/time model on both sides; because the comparator runs it at about 0.1 M chars/s, that row uses the first quarter of the same text (2,889 lines, 2,492,312 characters) so a benchmark group fits its time limit. Both systems see the same lines; the comparator is Microsoft.Recognizers.Text 1.8.13 (en-US). The two systems return different numbers of results (conventions differ); this table compares cost, and the quality rows compare results. ### Machines and pins | | x64 | arm64 | |---|---|---| | CPU | AMD Ryzen 9 7950X3D, 16 cores / 32 threads (avx2, avx512f, fma, bmi2) | Apple M2, 8 cores | | OS, SDK | Linux 7.0, .NET SDK 10.0.403, Release | macOS, .NET SDK 10.0.403, Release | | Load average (1/5/15 min) before | 1.22 / 2.48 / 6.93 | 1.81 / 1.82 / 1.81 | | 1-minute load after the Lingua rows, after the first recognizer row, at the end | 1.5, 3.1, 4.8 | 2.3, 2.2, 2.3 | | Run | 2026-11-05 | 2026-11-05 | OneModel is not used by either component. Language-ID weights: `langid.bin`, sha256 `9d06cc77632083274ae5ca5fbb0f3e676abfc63aa6a2d702cbdf9d558cba9d6d`. The runs queue behind other work, then hold the machine alone; the load averages above are the 1-minute values read between rows. ### Language ID (median repetition) | bucket | system | x64 µs/text | x64 texts/s | x64 speedup | arm64 µs/text | arm64 texts/s | arm64 speedup | alloc per text | |---|---|---:|---:|---:|---:|---:|---:|---:| | words | ours | 7.88 | 126,970 | | 7.96 | 125,657 | | 1.4 KB | | words | Lingua high | 76.28 | 13,110 | 9.7x | 84.37 | 11,853 | 10.6x | 79 KB | | words | Lingua low | 68.79 | 14,536 | 8.7x | 60.72 | 16,468 | 7.6x | 75 KB | | pairs | ours | 5.05 | 198,104 | | 8.08 | 123,636 | | 1.4 KB | | pairs | Lingua high | 90.87 | 11,005 | 18.0x | 85.99 | 11,629 | 10.6x | 84 KB | | pairs | Lingua low | 64.42 | 15,524 | 12.8x | 55.67 | 17,964 | 6.9x | 80 KB | | sentences | ours | 9.06 | 110,422 | | 10.75 | 93,058 | | 1.4 KB | | sentences | Lingua high | 375.10 | 2,666 | 41.4x | 294.61 | 3,394 | 27.4x | 126 KB | | sentences | Lingua low | 250.14 | 3,998 | 27.6x | 212.59 | 4,704 | 19.8x | 123 KB | Our repetitions of 15 to 50 ms vary more than the comparator's (x64 pairs: 8.3, 5.1 and 3.0 µs/text as the JIT tiers up); the median is shown and the speedup is at least 7.6x on every row either way. | | ours | Lingua high | Lingua low | |---|---:|---:|---:| | Load, x64 / arm64 | 10.7 ms / 8.2 ms | 3,850 ms / 4,238 ms | 326 ms / 321 ms | | Peak RSS, x64 / arm64 | 95 MB / 65 MB | 4,791 MB / 2,747 MB | 367 MB / 253 MB | | Model on disk | 3.6 MB | n/a (embedded in the package) | | ### Recognizers (median repetition) | row | system | x64 time | x64 M chars/s | x64 speedup | arm64 time | arm64 M chars/s | arm64 speedup | alloc | peak RSS x64 / arm64 | load x64 / arm64 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | number, unit, currency, sequence (10.0 M chars) | ours | 0.497 s | 20.08 | | 0.559 s | 17.86 | | 387 MB | 113 / 84 MB | 21 / 18 ms | | number, unit, currency, sequence (10.0 M chars) | Recognizers.Text 1.8.13 | 27.12 s | 0.37 | 54.6x | 32.18 s | 0.31 | 57.6x | 8,795 MB (x64), 8,827 MB (arm64) | 345 / 299 MB | 266 / 259 ms | | with date/time (2.5 M chars) | ours | 0.620 s | 4.02 | | 0.558 s | 4.47 | | 556 MB | 101 / 67 MB | 39 / 31 ms | | with date/time (2.5 M chars) | Recognizers.Text 1.8.13 | 27.16 s | 0.09 | 43.8x | 28.18 s | 0.09 | 50.5x | 5,024 MB (x64), 5,026 MB (arm64) | 334 / 303 MB | 630 / 424 ms | Pass/fail: every row is at or faster than its managed comparator, on both machines. ## German fast tier UD German-GSD r2.18 test and GermEval 2014 test against spaCy 3.8.18 trained with its default configuration on the same data; the table, the settings and the weights are in `recipes/de/fast/PROVENANCE.md`. Raw text: every row ahead of spaCy or within noise (tokens 99.91 vs 98.03, UPOS 94.22 vs 90.13, UFeats 85.32 vs 80.11, LAS 76.75 vs 70.77). Gold tokens: UPOS (94.56 vs 94.59), XPOS (95.35 vs 95.21), lemmas (95.95 vs 95.65), UAS, LAS and entities (66.28 vs 65.77) within noise; UFeats (86.25 vs 84.71) and AllTags (82.78 vs 81.29) ahead. The tagging heads sit on an encoder with one bidirectional LSTM layer after one window layer (`recipes/de/fast/PROVENANCE.md`). The package is 109 MB; the consumer app runs on JIT and NativeAOT. ## French fast tier UD French-GSD r2.18 test and WikiNER French (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in `recipes/fr/PROVENANCE.md`. Gold tokens: UPOS (96.77 vs 97.07), UFeats (96.34 vs 96.41), lemmas (97.51 vs 97.49), UAS (90.22 vs 89.10) and LAS (86.69 vs 85.98) within noise. Raw text: every row ahead of spaCy, whose tokenizer keeps `du` and `au` as one token where the treebank has two words (tokens 99.88 vs 98.85, words 99.48 vs 94.64, UPOS 96.06 vs 91.80, LAS 85.97 vs 78.10); sentences 94.79 vs 91.82, within noise. Entities, gold tokens: micro F1 82.48 vs 75.50 (+6.97 [6.12, 7.78]) and each of LOC, MISC, ORG, PER ahead. French-GSD has no XPOS. The package is 95 MB. ## Portuguese fast tier UD Portuguese-Bosque r2.18 test and Universal NER Portuguese-Bosque test against spaCy 3.8.18 trained with its default configuration on the same data; the table, the settings and the weights are in `recipes/pt/fast/PROVENANCE.md`. Bosque has no XPOS, so there is no tagger and no XPOS or AllTags row. Raw text: every row ahead of spaCy except tokens (99.86 vs 99.87, within noise) (words 99.74 vs 88.51, UPOS 95.35 vs 84.15, LAS 82.09 vs 65.85, sentences 92.72 vs 84.48). Gold tokens: UPOS (95.89 vs 95.92), UFeats (94.51 vs 94.11), lemmas (97.30 vs 96.90), UAS (86.93 vs 86.05) and LAS (82.74 vs 81.96) within noise. The tagging heads sit on an encoder with one bidirectional LSTM layer after one window layer (`recipes/pt/fast/PROVENANCE.md`). Entities 73.13 vs 75.27 (paired difference -2.14 [-4.26, 0.39], within noise). ## Fast-tier tagging speed in the other languages (indicative) The tagging model of each language alone (sentencizer, tagger where present, morphologizer, lemmatizer), raw test text of the language's gating treebank, one thread, one document per call, three runs of four passes; build before the faster kernels and the LSTM-free encoder prediction against the current build, in the same job, x64 worker with load 16 to 25. Predictions of the current build equal the earlier build's on every file (dumps of the tagging, parser and entity files of each language: 0 differing lines, plus the Dutch entity confidence noted under English). | Language, file | Before, tokens/s | Now, tokens/s | Allocation now | Peak working set now | |---|---|---|---|---| | German (GSD) | 13,300 to 15,800 | 16,100 to 17,200 | 3,227 B/token | 1,150 MB | | Portuguese (Bosque) | 13,500 to 17,400 | 16,000 to 21,500 | 1,893 B/token | 691 MB | | Dutch (Alpino) | 10,300 to 11,200 | 12,900 to 15,100 | 5,102 B/token | 1,055 MB | | French (GSD) | 23,900 to 24,000 | 30,500 to 31,100 | 2,238 B/token | 745 MB | | Turkish (BOUN) | 3,150 to 3,170 | 4,280 to 4,290 | 6,310 B/token | 1,900 MB | German, Portuguese and Dutch (the recurrent-layer models) gain 15 to 40 percent; the rest of their time is in the three wide heads and the 256-wide encoders, which need the same retraining as English to be faster. Allocation went up for them because the recurrent layers now return one array per call. First-call times of the tagging models: German 1,419 ms, Dutch 1,258 ms, Portuguese 226 ms (load 125 to 171 ms). A French and a Turkish narrow student were trained on web text labelled by the shipped models; their text had not been checked against the evaluation sets, so they are not used. ## English fast tier (shipped: self-trained tagging and entity models, earlier parser) Model `english.onenlp` in `RecursiveAI.OneNlp.Models.English` (126.2 MB, sha256 `1414ec79...1e48`): the tagging model (tagger, morphologizer, lemmatizer; 96-wide encoder with no window layer), the parser model (`recipes/en/parser`, unchanged) and the entity model (128-wide encoder with no window layer), each with its own encoder, put into one pipeline file by `recipes/en/assemble`. The tagging and entity models are trained by `recipes/en/selftrain`: the treebank (or UNER) plus web text labelled by the previous packaged model, after removing every sentence that matches a train, development or test sentence of the evaluation data (`recipes/en/selftrain/results/dedup-report.jsonl`). Provenance: `recipes/en/selftrain/PROVENANCE.md`. UD English-EWT r2.18 test (2,077 sentences, 25,131 tokens) and UNER English-EWT test, each read once for the final configuration; the development sets chose every setting. Paired bootstrap 95% CI over sentences (1,000 resamples, seed 1) from `tools/OneNlp.Eval`; reference spaCy 3.8.18 trained on the same data with its default configuration, same input mode, with its own interval. "Within noise" means the intervals overlap (`docs/measurement.md`). The rows are in `recipes/en/selftrain/results/`. ### Quality, gold tokens and sentences | Metric | Ours | spaCy | Within noise | |---|---|---|---| | UPOS | 94.16 [93.79, 94.54] | 94.29 [93.89, 94.66] | yes | | XPOS | 93.21 [92.79, 93.61] | 93.43 [93.00, 93.82] | yes | | UFeats | 95.15 [94.86, 95.47] | 95.18 [94.88, 95.45] | yes | | AllTags | 91.29 [90.83, 91.75] | 91.53 [91.05, 91.97] | yes | | Lemmas | 96.11 [95.77, 96.44] | 95.80 [95.42, 96.13] | yes | | UAS | 85.77 [84.91, 86.55] | 85.40 [84.50, 86.26] | yes | | LAS | 81.68 [80.82, 82.49] | 81.96 [81.03, 82.83] | yes | | CLAS | 76.76 [75.76, 77.75] | 77.28 [76.18, 78.29] | yes | ### Quality, raw text Tagging and dependency rows below: ten consecutive sentences per call. The parser file is the one of the earlier model and its heads and relations are identical to that model's, so its whole-document rows are unchanged: Sentences 76.15 [74.25, 78.08] against spaCy 72.66, UAS 81.80, LAS 78.13 [77.18, 79.06] against 78.53, CLAS 73.04 against 74.34 (all within noise or ahead). | Metric | Ours | spaCy | Within noise | |---|---|---|---| | Tokens | 99.00 [98.80, 99.19] | 95.70 [95.14, 96.21] | ahead | | UPOS | 92.90 [92.42, 93.35] | 92.45 [91.84, 93.06] | yes | | XPOS | 91.91 [91.38, 92.38] | 91.72 [91.09, 92.34] | yes | | UFeats | 93.87 [93.48, 94.26] | 93.37 [92.81, 93.92] | yes | | AllTags | 90.04 [89.49, 90.57] | 89.99 [89.32, 90.62] | yes | | Lemmas | 95.03 [94.61, 95.43] | 93.69 [93.08, 94.29] | ahead | | UAS | 81.40 [80.41, 82.30] | 81.34 [80.25, 82.37] | yes | | LAS | 77.76 [76.82, 78.70] | 78.53 [77.46, 79.55] | yes | | CLAS | 72.63 [71.53, 73.75] | 74.34 [73.14, 75.47] | yes | MLAS, information: 65.61 [64.46, 66.72] over ten-sentence calls, against spaCy's 69.05 [67.81, 70.21]; it is outside the interval, as for every earlier English model, and is not one of the gated rows. ### Entities (UNER English-EWT test, gold tokens) | Row | Ours | spaCy | Paired difference | |---|---|---|---| | micro F1 | 63.09 [60.01, 65.89] | 62.32 [59.42, 65.33] | 0.77 [-1.92, 3.33] | | precision | 70.95 [67.54, 74.27] | 67.97 [64.80, 71.11] | 2.98 [-0.18, 6.06] | | recall | 56.80 [53.58, 60.13] | 57.54 [54.26, 60.91] | -0.74 [-3.42, 1.83] | | LOC | 71.56 [67.09, 75.67] | 71.43 [66.56, 76.16] | 0.13 [-3.83, 3.97] | | ORG | 43.87 [38.29, 48.99] | 44.33 [39.12, 49.82] | -0.46 [-5.77, 4.70] | | PER | 69.40 [64.96, 73.26] | 68.05 [63.91, 71.83] | 1.35 [-2.51, 5.16] | Every row is within noise. ### Speed (indicative, not a gate row) Raw EWT test text (25,131 tokens, repeated four times per pass, 100,524 tokens), one document per call, twelve passes after the process starts, medians of the last ten; Catalyst 26.10.6514 with its English models on the same documents, run back to back in the same job. Machine: x64 worker (Ryzen 9 7950X3D, 32 threads) with other jobs running (load average 22 to 30 before and after, so the numbers are noisy), .NET SDK 10.0.403, OneModel 1.0.0. The gate run is made once, on a quiet machine, at the 1.0 gate. | | 1 thread, tokens/s (min to max) | allocation | 8 threads, tokens/s | |---|---|---|---| | Catalyst, sentences + tokens + POS + lemmas | 545,000 (409,000 to 733,000) | not measured in this job (307 B/token before) | not run | | Ours, sentences + tokens + UPOS/UFeats + lemmas | 366,000 (336,000 to 464,000) | 329 B/token | 1,460,000 | | Ours, sentences + entities | 221,000 (184,000 to 314,000) | 322 B/token | 361,000 | | Ours, parser path (the parser sets sentences) | 5,200 (4,500 to 5,400) | 4,940 B/token | 5,700 | | Ours, full pipeline | 5,600 (4,800 to 5,800) | 5,480 B/token | 5,600 | The ratio on the tagging path is **0.67** in this job (the same model against Catalyst in an earlier job with the same method: 0.94 for a 96-wide variant of the same recipe before the final retraining, 0.06 for the previous packaged model), which meets the 0.5 bar. Peak working set 1,241 MB (1 thread) and 1,360 MB (8 threads) for the full pipeline; model file 126.2 MB; load 416 ms and the first call 365 ms (previous model: 328 ms and 434 ms). Full pipeline in the single-file mode used by `Nlp.Analyze` (one document per call): 5,100 to 5,500 tokens/s against 3,000 to 3,500 for the previous model. The parser is now the limit of the full pipeline (about 190 us per token, an LSTM over a 192-wide state), and its prediction is serialized per head, so eight threads do not help it; the tagging and entity paths scale (4.0 and 1.6 times at eight threads). How the speed was reached without changing a row: the encoders are narrower and trained on more text (above); GELU, the dense layers and the parser's LSTM and label scores run on dedicated row kernels (`RowKernels`, `LstmInference`) with bit-identical or last-digit-identical results (prediction dumps of every shipped file before and after: 0 differing lines on English, German, Portuguese, Dutch, Turkish and French except one Dutch entity confidence that rounds 0.901 to 0.902); the encoder runs outside the head lock. ## English fast tier, option: one shared encoder (not shipped) A single encoder shared by every head, trained by `recipes/en/joint` (23.7 MB, sha256 `6bb03663...50b9`): four times smaller and more than ten times faster on the tagging path, with the rows below. It is not the packaged model because its UFeats, AllTags, entity and raw CLAS and MLAS rows are weaker. Rows measured on the model file `english.onenlp` of that recipe. UD English-EWT r2.18 test, 2,077 sentences, 25,131 tokens; UNER English-EWT test. Paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); reference spaCy 3.8.18 trained on the same data with its default configuration, same input mode. "Within noise" means the intervals overlap. The development sets chose the configuration; the test files were read once. ### Quality, gold tokens and sentences | Metric | Ours | spaCy | Within noise | |---|---|---|---| | UPOS | 93.58 [93.17, 93.99] | 94.29 [93.91, 94.68] | yes | | XPOS | 92.81 [92.36, 93.24] | 93.43 [93.00, 93.87] | yes | | UFeats | 94.40 [94.07, 94.73] | 95.18 [94.86, 95.48] | **no** | | AllTags | 90.45 [89.98, 90.95] | 91.53 [91.05, 91.99] | **no** | | Lemmas | 95.84 [95.50, 96.17] | 95.80 [95.46, 96.13] | yes | | UAS | 84.98 [84.12, 85.80] | 85.40 [84.48, 86.28] | yes | | LAS | 81.31 [80.47, 82.16] | 81.96 [81.00, 82.91] | yes | ### Quality, raw text | Metric | Ours | spaCy | Within noise | |---|---|---|---| | Tokens | 99.00 [98.81, 99.19] | 95.70 [95.16, 96.17] | ahead | | Sentences | 72.76 [70.97, 74.66] | 72.66 [70.46, 74.70] | yes | | UPOS | 92.53 [92.09, 93.01] | 92.45 [91.83, 93.05] | yes | | XPOS | 91.60 [91.11, 92.11] | 91.72 [91.10, 92.34] | yes | | UFeats | 93.39 [92.97, 93.79] | 93.37 [92.84, 93.91] | yes | | AllTags | 89.35 [88.80, 89.89] | 89.99 [89.34, 90.64] | yes | | Lemmas | 94.68 [94.29, 95.07] | 93.69 [93.07, 94.28] | ahead | | UAS | 80.41 [79.47, 81.36] | 81.34 [80.25, 82.34] | yes | | LAS | 76.93 [75.98, 77.92] | 78.53 [77.49, 79.54] | yes | | CLAS | 71.81 [70.67, 72.97] | 74.34 [73.19, 75.49] | **no** | | MLAS | 65.14 [63.97, 66.36] | 69.05 [67.85, 70.21] | **no** | ### Entities (UNER English-EWT test, gold tokens) | Row | Ours | spaCy | Paired difference | |---|---|---|---| | micro F1 | 59.38 [56.06, 62.60] | 62.32 [59.42, 65.33] | -2.93 [-5.81, -0.15] | | LOC | 65.55 [60.45, 70.29] | 71.43 [66.56, 76.16] | -5.88 [-9.82, -1.94] | | ORG | 45.19 [39.35, 50.48] | 44.33 [39.12, 49.82] | 0.86 [-3.94, 5.38] | | PER | 64.19 [59.35, 68.55] | 68.05 [63.91, 71.83] | -3.86 [-8.32, 0.72] | The intervals overlap on every entity row; the paired differences for micro F1 and LOC exclude zero. ### Speed (indicative, not a gate row) Raw EWT test text (25,131 tokens, repeated four times per pass), one thread, sentences + tokens + UPOS/UFeats + lemmas; Catalyst 26.10.6514 with its English models on the same documents, run back to back in the same job. Machine: x64 worker (Ryzen 9 7950X3D) while other jobs were running (load 9 to 15), .NET SDK 10.0.403. The gate run is made once, on a quiet machine, at the 1.0 gate. | | tokens/s | allocation | note | |---|---|---|---| | Catalyst | 530,000 to 590,000 | 307 B/token | UPOS 80.7, lemmas 83.8 on this text | | Ours, one document per call | 215,000 to 270,000 | 413 B/token | UPOS 92.5, lemmas 94.7 | | Ours, `ProcessMany` batches of 64 | 226,000 to 259,000 | 327 B/token | | | Ours, entities path | 209,000 to 257,000 | 393 B/token | | | Ours, parser path | 5,300 to 7,900 | 856 B/token | the bidirectional LSTM is the cost | | Ours, full pipeline | 5,500 to 7,900 | 1,377 B/token | | The ratio on the tagging path is about 0.4 (**misses**). Peak working set 640 MB for the full pipeline; model file 23.7 MB; first `Nlp.Analyze` (loads the weights) 325 ms on x64. Where the time goes (microseconds per token, one document per call): tokenizer 0.2, sentencizer 0.05, encoder about 2 (lookups 0.6, window-layer product 1.2), lemmatizer 0.9, morphologizer 1.0. A smaller configuration of the same recipe (48 wide, no window layer) ran the same path at 500,000 to 690,000 tokens/s (about 1.6 to 2.3 times Catalyst in the same conditions) with a development tagging mean 0.6 points lower; it is not the shipped model. ## Dutch fast tier UD Dutch-Alpino r2.18 test and WikiNER Dutch (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in `recipes/nl/fast/PROVENANCE.md`. Every row is within noise of spaCy in both modes. Gold tokens (the treebank's own words and sentences): UPOS 93.83 vs 94.42, XPOS 91.52 vs 91.73, UFeats 92.94 vs 93.13, AllTags 90.30 vs 90.51, lemmas 91.61 vs 91.83, UAS 85.18 vs 84.09, LAS 80.28 vs 79.85. Raw text: tokens 99.66 vs 99.82, sentences 91.31 vs 89.60, UPOS 93.09 vs 94.36, XPOS 90.94 vs 91.63, UFeats 92.47 vs 92.97, AllTags 89.52 vs 90.39, lemmas 91.19 vs 91.65, UAS and LAS (79.48 vs 79.57). The tagging heads sit on an encoder with one bidirectional LSTM layer after one window layer (`recipes/nl/fast/PROVENANCE.md`). Entities (gold tokens): micro F1 87.47 vs 86.21 (difference +1.26 [0.76, 1.72], ahead), every type within noise or ahead. ## Turkish fast tier UD Turkish-BOUN r2.18 test and Turkish-WikiNER test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in `recipes/tr/fast/PROVENANCE.md`. The tokenizer is a trained multiword-token splitter (`NeuralMultiwordSplitter`, `docs/segmentation.md`). Gold words: UPOS (88.97 vs 88.04), UFeats (78.15 vs 78.30) and MLAS (40.47 vs 38.85) within noise; XPOS, AllTags and lemmas (85.11 vs 80.24) ahead. Raw text: tokens (99.56 vs 99.42), words (96.55 vs 97.06), UPOS (86.02 vs 86.82), UFeats, UAS (65.73 vs 63.96), LAS (57.53 vs 55.86), CLAS (54.83 vs 53.43), MLAS (38.10 vs 38.02) and BLEX within noise, sentences, XPOS, AllTags and lemmas ahead. The parser was retrained on the tokenizer's words and on 60,239 sentences of web text parsed by earlier parsers where two of them agree (it was CLAS 49.84 and MLAS 34.87 before); gold words: UAS, LAS, CLAS (59.08 vs 55.03) and BLEX ahead. Entities (gold tokens): 67.40 vs 65.82, within noise. The package is about 148 MB; the consumer app runs on JIT and NativeAOT. ## Spanish fast tier UD Spanish-AnCora r2.18 test and WikiNER Spanish (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the table, the settings and the weights are in `recipes/es/fast/PROVENANCE.md`. Gold words and sentences: UPOS (98.11 vs 98.18), UFeats (97.71 vs 97.74) and lemmas (98.84 vs 98.72) within noise; XPOS (86.73 vs 84.97), AllTags (86.09 vs 84.47), UAS (90.05 vs 89.04) and LAS (87.00 vs 85.95) ahead. Raw text: every row within noise or ahead (spaCy's tokenizer keeps `del` and `verlo` as one token where the treebank has several words; words 99.83 vs 96.60, UPOS 97.86 vs 94.81, LAS 86.70 vs 81.82); tokens 99.84 vs 99.90 and sentences 98.14 vs 97.70 within noise. Entities, gold tokens: micro F1 82.88 vs 83.35 (difference -0.46 [-1.14, 0.20], within noise); ORG ahead (+4.96), PER behind (-1.96 [-2.83, -1.13]). The entity model reads the tokenizer's pieces of `del` and `al`: trained on whole tokens it scored 81.71 (-1.64, outside the interval). Indicative cost on one thread of the shared worker: tagging heads 44 us/token, whole pipeline about 190 us/token. The package is 109 MB; the consumer app runs on JIT and NativeAOT. ## Italian fast tier UD Italian-ParlaMint r2.18 test (gating) and WikiNER Italian (wp3) test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the tables, the settings and the weights are in `recipes/it/fast/PROVENANCE.md`. The training treebank is small: 897 sentences (ParlaMint and MarkIT; the larger Italian treebanks are non-commercial), so parser rows are low in absolute terms (LAS 72.6) for both systems. Gold words and sentences: UPOS (94.44 vs 94.10), XPOS (93.74 vs 93.59), UAS (78.52 vs 77.08) and LAS (72.57 vs 71.39) within noise; UFeats (89.52 vs 87.42), AllTags (86.31 vs 84.08) and lemmas (95.18 vs 93.13) ahead. Raw text: tokens 99.86 vs 99.75 within noise; every other row ahead (spaCy's tokenizer keeps `del` and `farlo` as one token where the treebank has several words; words 99.67 vs 89.73, UPOS 94.01 vs 83.53, LAS 72.77 vs 58.38, sentences 96.00 vs 87.55). Entities, gold tokens: micro F1 86.35 vs 83.54 (+2.80 [2.18, 3.42]); PER, ORG and MISC ahead, LOC within noise. No row misses. Indicative cost on one thread of the shared worker: the whole pipeline 7.2k tokens/s. The package is 103 MB; the consumer app runs on JIT and NativeAOT. ## Chinese fast tier UD Chinese-GSDSimp r2.18 test and UNER Chinese-GSDSIMP test against spaCy 3.8.18 trained with its default configuration on the same data, paired bootstrap 95% CI over sentences (1,000 resamples, seed 1); the settings and weights are in `recipes/zh/fast/PROVENANCE.md`. spaCy's default Chinese tokenizer cuts single characters, so its raw-text pipeline cannot reach word-level scores (Tokens 39.43); the gold-token reference is the `gold_preproc` training run described in `tools/baselines/spacy/lang/PROVENANCE.md`, which trains and scores on gold words. The pipeline cuts words with a trained character tagger (`docs/segmentation.md`), tags, lemmatizes, parses and finds entities. Gold words and sentences (spaCy `gold_preproc`): | Metric | Ours | spaCy | Within noise | |---|---|---|---| | UPOS | 90.41 [89.70, 91.07] | 90.35 [89.70, 90.99] | yes | | XPOS | 91.37 [90.75, 91.98] | 91.11 [90.45, 91.70] | yes | | UFeats | 98.38 [98.13, 98.59] | 98.43 [98.19, 98.66] | yes | | AllTags | 89.49 [88.76, 90.17] | 89.52 [88.79, 90.20] | yes | | Lemmas | 99.87 [99.79, 99.94] | 99.91 [99.85, 99.96] | yes | | UAS | 74.94 [73.46, 76.38] | 73.11 [71.46, 74.68] | yes | | LAS | 68.76 [67.12, 70.32] | 68.24 [66.47, 69.94] | yes | | CLAS | 65.85 [64.15, 67.51] | 65.49 [63.52, 67.27] | yes | | MLAS | 56.94 [55.08, 58.69] | 58.87 [56.85, 60.72] | yes | Raw text (spaCy's default pipeline, character tokens): | Metric | Ours | spaCy | Within noise | |---|---|---|---| | Tokens, Words | 89.62 [88.87, 90.42] | 39.43 [38.60, 40.23] | no (ahead) | | Sentences | 99.40 [98.70, 99.90] | 46.93 [42.22, 51.79] | no (ahead) | | UPOS | 81.90 [80.99, 82.86] | 34.92 [34.11, 35.68] | no (ahead) | | XPOS | 82.73 [81.85, 83.67] | 35.69 [34.86, 36.47] | no (ahead) | | UFeats | 88.24 [87.49, 89.01] | 38.38 [37.56, 39.16] | no (ahead) | | AllTags | 81.00 [80.09, 81.99] | 34.29 [33.48, 35.06] | no (ahead) | | Lemmas | 89.56 [88.82, 90.35] | 39.43 [38.60, 40.23] | no (ahead) | | UAS | 62.50 [60.81, 64.17] | 9.12 [8.44, 9.79] | no (ahead) | | LAS | 57.88 [56.21, 59.63] | 8.59 [7.94, 9.28] | no (ahead) | Entities, UNER Chinese-GSDSIMP test, gold tokens (paired with spaCy `gold_preproc`): micro F1 62.80 [58.46, 67.23] vs 61.86 [57.66, 66.12] (difference 0.94 [-2.18, 4.40], within noise); precision 62.44 vs 67.08 (-4.64 [-8.52, -0.50], inside the noise rule); recall 63.17 vs 57.40. Per type: LOC 69.86 vs 68.15, ORG 47.91 vs 45.00, PER 57.69 vs 59.08, all within noise. With the shipped entity confidence threshold of 0.6 (`recipes/zh/fast/PROVENANCE.md`, "Entity confidence threshold"): micro F1 63.79 [59.39, 68.27] (difference +1.93 [-1.48, 5.33]); precision 69.28 vs 67.08 (+2.20 [-2.02, 6.59]); recall 59.11 vs 57.40 (+1.70 [-1.61, 5.12]); LOC 70.38, ORG 51.28, PER 57.95, all within noise. On raw text (our segmenter's words, character-span F1) the micro F1 is 52.88 [48.60, 57.61]; spaCy's default model reaches 13.30 on gold tokens only, and no raw-text spaCy row exists. For information (`docs/references.md`): Stanza 1.5.1 raw text UPOS 88.82, LAS 70.65; UDPipe 2 gold tokens UPOS 95.73, LAS 83.60; Trankit base raw UPOS 94.17, LAS 81.96 (an older annotation). Our fast tier is below them, as expected of a fast tier with no pretrained embeddings; the gate is spaCy's. **Misses:** none. Everything is within noise or ahead; MLAS (-1.93) is inside the noise rule. Indicative speed (1 thread, load average 8-13 on the shared worker, raw development text; Catalyst has no Chinese model, so there is no managed row): segmenter alone 22-23k words/s (3.7 kB allocated per word); segmenter and tagging 16-17k words/s; the whole pipeline (segmenter, tagging, parser, entities) 4.5-4.6k words/s. ## Parser speed (one LSTM layer, product label features) Every fast-tier parser was retrained with the setting in `docs/parser.md` ("Speed configuration"): English `english.onenlp` is now 117.5 MB (sha256 `ac0080ad...ed9b`; the tagging and entity models are unchanged, only the parser part differs). Training data, seeds and licences are those of the earlier parsers (UD treebanks, CC BY-SA 4.0 / CC BY 4.0; the models keep share-alike lineage); selection used each language's development file with the earlier early-stopping rule (patience 6), and each test file was read for the final models (see the note below for the two languages where a smaller setting was scored on test before the larger one was chosen on dev). ### Speed (indicative: 1 thread, batches of 32 documents, raw test text, x64 worker, load average 14 to 23) | Model | Parser head, before | Parser head, after | Full pipeline, before | Full pipeline, after | |---|---|---|---|---| | English | 137 to 145 us/token (5.3k tokens/s), 4,974 B/token | 30 us/token (32.5k tokens/s), 2,563 B/token | 4.9k to 5.0k tokens/s, 5,516 B/token | 26.3k tokens/s, 3,105 B/token | | German | about 150 us/token | 32 us/token | | | | Dutch | 151 | 30 | | | | Portuguese | 169 | 29 | | | | Spanish | 156 | 28 | | | | French | 126 | 20 | | | | Italian | 80 | 46 | | | | Chinese | 142 | 48 | | | The English parser alone: 137 us/token by component before (LSTM 81, label scorer 52, projections 9, arcs and tree 8) and 20 us/token after in the per-component profile. Peak working set of the English full pipeline 939 MB (was 1,083 MB). The other languages' rows are the parser file alone (its own encoder 10 to 13 us/token is included in the full-pipeline figure, not in the head column). ### English, UD English-EWT r2.18 test (hash `fa024f43...c494a`), paired bootstrap 95% CI, 1,000 resamples, seed 1, reference spaCy 3.8.18 default configuration on the same data | Mode | Metric | Ours | spaCy | Earlier parser | Within noise | |---|---|---|---|---|---| | gold tokens | UAS | 85.26 [84.38, 86.13] | 85.40 | 85.77 | yes | | | LAS | 81.53 [80.62, 82.43] | 81.96 | 81.68 | yes | | | CLAS | 76.78 [75.68, 77.86] | 77.28 | 76.76 | yes | | | MLAS | 69.36 [68.23, 70.47] | 71.14 | 69.17 | yes | | raw text | UAS | 80.74 [79.78, 81.71] | 81.34 | 81.80 | yes | | | LAS | 77.26 [76.28, 78.25] | 78.53 | 78.13 | yes | | | CLAS | 72.19 [71.00, 73.35] | 74.34 | 73.04 | yes | | | MLAS | 65.30 [64.12, 66.48] | 69.05 | 65.61 | no (behind; not a gated row) | | raw text | Sentences | 71.08 [69.06, 72.98] | 72.66 | 76.15 | yes | Tagging rows are those of the unchanged tagging model (gold UPOS 94.16, UFeats 95.15, lemmas 96.11; raw UPOS 92.90). ### Other languages, gating treebank test, LAS (UAS, CLAS, MLAS in the committed result files), paired CI against spaCy 3.8.18 | Language | Mode | Ours (earlier parser) | spaCy | Within noise | |---|---|---|---|---| | German GSD | gold | 76.66 [75.63, 77.72] (78.25) | 77.31 | yes | | | raw | 74.87 [73.79, 75.92] (76.75) | 70.77 | ahead | | Dutch Alpino | gold | 79.12 [77.87, 80.39] (80.28) | 79.85 | yes | | | raw | 78.18 [76.87, 79.46] (79.48) | 79.57 | yes | | Portuguese Bosque | gold | 81.80 [81.10, 82.49] (82.74) | 81.96 | yes | | | raw | 81.14 [80.44, 81.82] (82.09) | 65.85 | ahead | | Spanish AnCora | gold | 86.41 [85.90, 86.89] (87.00) | 85.95 | yes | | | raw | 86.15 [85.61, 86.64] (86.70) | 81.82 | ahead | | French GSD | gold | 86.96 [85.86, 87.96] (86.62) | 85.98 | yes | | | raw | 86.07 [84.92, 87.11] (85.92) | 78.10 | ahead | | Italian ParlaMint | gold | 71.55 [70.26, 72.97] (72.57) | 71.39 | yes | | | raw | 71.55 [70.27, 72.90] (72.77) | 58.38 | ahead | | Chinese GSDSimp | gold words (spaCy trained with `gold_preproc`) | 67.49 [65.94, 69.14] (68.76) | 68.24 | yes | | | raw (our segmenter) | 56.93 [55.23, 58.73] (57.88) | 8.59 | ahead | Every UAS, LAS and CLAS row is within noise or ahead in both modes. MLAS, which also depends on the tagging model, is within noise or ahead in every row of these languages (Chinese gold words: 55.78 [53.92, 57.68] against 58.87). Raw-text rows compare our tokenizer against spaCy's, which is the reason for the large margins. Note on selection: for Italian and Chinese the 128-unit setting was scored on test before the 192-unit setting was chosen on dev (dev LAS 72.43 and 64.82 against 73.97 and 66.81, and 74.81 and 68.22 for the earlier parsers); the larger setting was chosen because the smaller lost more than 1 point of dev LAS, the rule used for every language. # Base-size accurate tier: per-row results **The gate** is the [paired comparison with spaCy 3.8.18 trained on the same data](#paired-spacy-3818-comparison-gate): 144 rows, 117 ahead, 14 within the paired interval, **13 behind** (listed there and in the README's known gaps). **Information:** published research systems (Stanza, UDPipe 2, Trankit) train a separate, larger model per treebank and are not a gate for this package. Three views of the same rows (no averaging over languages or metrics): - **Best published system per row: 1314 comparisons, 1205 misses**, 109 within the interval or ahead. - **Stanza 1.5.1 pretrained/charlm, no transformer: 650 comparisons, 534 misses**, 116 within the interval or ahead. - **All published system comparisons: 2678 comparisons, 2346 misses**, 332 within the interval or ahead. Intervals: 95% bootstrap, 1,000 resamples, seed 20261106. The published reference passes when it is no higher than our upper bound; these are not paired intervals against the reference. Gold and raw input are never crossed. A missing reference is not a pass. The reference UD version and test-split change flag accompany every comparison; changed splits are information, not a claim of equal evaluation data. Entirely absent gold columns are not scored. Before is the previous nine-language model, where available. | Language / treebank | Input | Metric | Before [95% CI] | Current [95% CI] | Best published view | Stanza-only view | |---|---|---|---|---|---|---| | af: UD_Afrikaans-AfriBooms | gold | AllTags | not measured | 84.60 [83.80, 85.35] | 95.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | gold | CLAS | not measured | 71.68 [70.24, 73.17] | not published | not published | | af: UD_Afrikaans-AfriBooms | gold | LAS | not measured | 77.91 [76.68, 79.12] | 87.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | gold | Lemmas | not measured | 96.86 [96.44, 97.24] | 98.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | gold | MLAS | not measured | 61.10 [59.33, 62.75] | 79.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | gold | UAS | not measured | 83.28 [82.11, 84.33] | 90.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | gold | UFeats | not measured | 94.08 [93.53, 94.58] | 98.32 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | gold | UPOS | not measured | 95.81 [95.37, 96.24] | 98.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | gold | XPOS | not measured | 86.44 [85.68, 87.14] | 95.94 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | af: UD_Afrikaans-AfriBooms | raw | AllTags | not measured | 82.91 [82.17, 83.62] | 96.54 (Trankit large, UD 2.5; test changed) — **MISS** | 95.66 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | CLAS | not measured | 75.53 [73.96, 77.02] | 85.30 (Trankit large, UD 2.5; test changed) — **MISS** | 80.10 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | LAS | not measured | 79.24 [77.98, 80.45] | 89.35 (Trankit large, UD 2.5; test changed) — **MISS** | 86.14 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | Lemmas | not measured | 93.98 [93.46, 94.48] | 98.29 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 98.29 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | MLAS | not measured | 65.29 [63.55, 66.93] | 82.64 (Trankit large, UD 2.5; test changed) — **MISS** | 76.71 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | Sentences | not measured | 95.74 [93.63, 97.51] | 100.00 (Trankit large, UD 2.5; test changed) — **MISS** | 99.65 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | Tokens | not measured | 96.77 [96.31, 97.18] | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | UAS | not measured | 83.39 [82.21, 84.57] | 91.61 (Trankit large, UD 2.5; test changed) — **MISS** | 89.54 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | UFeats | not measured | 92.19 [91.62, 92.77] | 98.68 (Trankit large, UD 2.5; test changed) — **MISS** | 98.41 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | UPOS | not measured | 93.68 [93.13, 94.18] | 98.87 (Trankit large, UD 2.5; test changed) — **MISS** | 98.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | Words | not measured | 96.77 [96.31, 97.18] | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | af: UD_Afrikaans-AfriBooms | raw | XPOS | not measured | 84.46 [83.73, 85.13] | 96.63 (Trankit large, UD 2.5; test changed) — **MISS** | 95.77 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | as: UD_Assamese-AiW | gold | CLAS | not measured | 54.29 [43.90, 64.79] | not published | not published | | as: UD_Assamese-AiW | gold | LAS | not measured | 58.82 [47.62, 69.88] | not published | not published | | as: UD_Assamese-AiW | gold | Lemmas | not measured | 67.06 [54.55, 79.52] | not published | not published | | as: UD_Assamese-AiW | gold | MLAS | not measured | 24.29 [17.65, 32.12] | not published | not published | | as: UD_Assamese-AiW | gold | UAS | not measured | 76.47 [61.25, 88.10] | not published | not published | | as: UD_Assamese-AiW | gold | UFeats | not measured | 47.06 [37.65, 58.54] | not published | not published | | as: UD_Assamese-AiW | gold | UPOS | not measured | 75.29 [66.27, 82.14] | not published | not published | | as: UD_Assamese-AiW | raw | CLAS | not measured | 56.74 [42.96, 71.76] | not published | not published | | as: UD_Assamese-AiW | raw | LAS | not measured | 60.00 [47.73, 73.49] | not published | not published | | as: UD_Assamese-AiW | raw | Lemmas | not measured | 65.88 [54.55, 76.67] | not published | not published | | as: UD_Assamese-AiW | raw | MLAS | not measured | 21.28 [10.61, 33.59] | not published | not published | | as: UD_Assamese-AiW | raw | Sentences | not measured | 100.00 [100.00, 100.00] | not published | not published | | as: UD_Assamese-AiW | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | as: UD_Assamese-AiW | raw | UAS | not measured | 78.82 [68.60, 87.80] | not published | not published | | as: UD_Assamese-AiW | raw | UFeats | not measured | 45.88 [39.77, 54.02] | not published | not published | | as: UD_Assamese-AiW | raw | UPOS | not measured | 64.71 [56.32, 74.03] | not published | not published | | as: UD_Assamese-AiW | raw | Words | not measured | 100.00 [100.00, 100.00] | not published | not published | | bor: UD_Bororo-BDT | gold | AllTags | not measured | 51.11 [50.40, 51.90] | 84.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | gold | CLAS | not measured | 42.27 [41.09, 43.51] | not published | not published | | bor: UD_Bororo-BDT | gold | LAS | not measured | 46.49 [45.35, 47.72] | 72.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | gold | Lemmas | not measured | 83.28 [82.59, 83.87] | 91.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | gold | MLAS | not measured | 19.95 [19.11, 20.80] | 59.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | gold | UAS | not measured | 56.91 [55.72, 58.22] | 79.11 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | gold | UFeats | not measured | 66.97 [66.22, 67.69] | 89.82 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | gold | UPOS | not measured | 71.89 [71.13, 72.67] | 91.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | gold | XPOS | not measured | 71.09 [70.35, 71.87] | 90.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | AllTags | not measured | 51.81 [51.09, 52.61] | 84.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | CLAS | not measured | 47.37 [46.01, 48.70] | not published | not published | | bor: UD_Bororo-BDT | raw | LAS | not measured | 51.41 [50.20, 52.63] | 72.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | Lemmas | not measured | 84.21 [83.57, 84.80] | 91.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | MLAS | not measured | 23.75 [22.80, 24.70] | 59.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | Sentences | not measured | 79.78 [78.05, 81.40] | 97.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | Tokens | not measured | 99.79 [99.69, 99.88] | not published | not published | | bor: UD_Bororo-BDT | raw | UAS | not measured | 61.94 [60.73, 63.13] | 78.80 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | UFeats | not measured | 66.81 [66.07, 67.53] | 89.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | UPOS | not measured | 73.84 [73.10, 74.60] | 91.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | Words | not measured | 99.79 [99.69, 99.88] | 99.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bor: UD_Bororo-BDT | raw | XPOS | not measured | 72.25 [71.52, 73.02] | 90.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | bxr: UD_Buryat-BDT | gold | AllTags | not measured | 8.16 [7.52, 8.78] | not published | not published | | bxr: UD_Buryat-BDT | gold | CLAS | not measured | 20.67 [19.71, 21.64] | not published | not published | | bxr: UD_Buryat-BDT | gold | LAS | not measured | 25.01 [23.98, 25.97] | not published | not published | | bxr: UD_Buryat-BDT | gold | Lemmas | not measured | 57.00 [56.07, 58.05] | not published | not published | | bxr: UD_Buryat-BDT | gold | MLAS | not measured | 1.15 [0.92, 1.39] | not published | not published | | bxr: UD_Buryat-BDT | gold | UAS | not measured | 47.48 [46.02, 48.87] | not published | not published | | bxr: UD_Buryat-BDT | gold | UFeats | not measured | 39.30 [38.32, 40.29] | not published | not published | | bxr: UD_Buryat-BDT | gold | UPOS | not measured | 20.69 [19.98, 21.43] | not published | not published | | bxr: UD_Buryat-BDT | gold | XPOS | not measured | 99.99 [99.97, 100.00] | not published | not published | | bxr: UD_Buryat-BDT | raw | AllTags | not measured | 8.25 [7.61, 8.89] | 67.97 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 67.97 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | CLAS | not measured | 22.31 [21.34, 23.32] | 31.58 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 31.58 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | LAS | not measured | 26.11 [25.14, 27.17] | 40.52 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 40.52 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | Lemmas | not measured | 55.90 [54.95, 56.93] | 77.78 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 77.78 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | MLAS | not measured | 1.39 [1.11, 1.67] | 24.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 24.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | Sentences | not measured | 18.61 [16.41, 20.73] | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | Tokens | not measured | 99.54 [99.31, 99.74] | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | UAS | not measured | 48.91 [47.69, 50.08] | 56.21 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 56.21 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | UFeats | not measured | 37.35 [36.44, 38.25] | 74.51 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 74.51 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | UPOS | not measured | 23.55 [22.73, 24.34] | 75.16 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 75.16 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | Words | not measured | 99.54 [99.31, 99.74] | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | bxr: UD_Buryat-BDT | raw | XPOS | not measured | 99.53 [99.28, 99.73] | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | gold | AllTags | not measured | 75.95 [75.36, 76.56] | 96.98 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | gold | CLAS | not measured | 78.17 [77.49, 78.84] | not published | not published | | ca: UD_Catalan-AnCora | gold | LAS | not measured | 83.64 [83.10, 84.16] | 93.46 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | gold | Lemmas | not measured | 98.06 [97.92, 98.19] | 99.49 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | gold | MLAS | not measured | 64.60 [63.86, 65.35] | 88.08 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | gold | UAS | not measured | 87.28 [86.79, 87.77] | 94.98 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | gold | UFeats | not measured | 93.07 [92.81, 93.32] | 98.75 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | gold | UPOS | not measured | 95.95 [95.74, 96.14] | 99.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | gold | XPOS | not measured | 77.80 [77.23, 78.38] | 97.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ca: UD_Catalan-AnCora | raw | AllTags | not measured | 69.68 [69.10, 70.32] | 98.24 (Trankit large, UD 2.5; test changed) — **MISS** | 95.67 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | CLAS | not measured | 71.60 [70.83, 72.41] | 91.41 (Trankit large, UD 2.5; test changed) — **MISS** | 87.24 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | LAS | not measured | 73.13 [72.51, 73.80] | 93.83 (Trankit large, UD 2.5; test changed) — **MISS** | 91.66 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | Lemmas | not measured | 87.71 [87.37, 88.04] | 99.43 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.06 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | MLAS | not measured | 56.53 [55.72, 57.32] | 89.31 (Trankit large, UD 2.5; test changed) — **MISS** | 84.72 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | Sentences | not measured | 89.47 [88.25, 90.72] | 100.00 (Trankit base, UD 2.5; test changed) — **MISS** | 99.57 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | Tokens | not measured | 93.67 [93.42, 93.91] | 99.97 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.97 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | UAS | not measured | 75.62 [75.01, 76.27] | 95.15 (Trankit large, UD 2.5; test changed) — **MISS** | 93.55 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | UFeats | not measured | 83.14 [82.75, 83.53] | 98.67 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 98.27 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | UPOS | not measured | 86.15 [85.78, 86.50] | 99.11 (Trankit large, UD 2.5; test changed) — **MISS** | 98.70 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | Words | not measured | 89.32 [88.99, 89.64] | 99.95 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.95 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ca: UD_Catalan-AnCora | raw | XPOS | not measured | 71.10 [70.53, 71.72] | 99.07 (Trankit large, UD 2.5; test changed) — **MISS** | 95.93 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ckb: UD_Central_Kurdish-Mukri | gold | CLAS | not measured | 23.08 [14.91, 31.43] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | gold | LAS | not measured | 34.43 [26.12, 42.96] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | gold | Lemmas | not measured | 64.75 [57.98, 70.87] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | gold | MLAS | not measured | 8.97 [2.72, 15.69] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | gold | UAS | not measured | 47.54 [35.07, 59.84] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | gold | UFeats | not measured | 54.92 [46.40, 62.77] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | gold | UPOS | not measured | 54.10 [47.46, 60.17] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | CLAS | not measured | 19.05 [9.30, 29.46] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | LAS | not measured | 27.83 [18.10, 37.44] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | Lemmas | not measured | 55.65 [49.54, 61.80] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | MLAS | not measured | 5.44 [0.00, 11.76] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | Sentences | not measured | 93.33 [82.61, 100.00] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | UAS | not measured | 33.91 [21.70, 46.28] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | UFeats | not measured | 49.57 [42.62, 56.56] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | UPOS | not measured | 51.30 [43.88, 58.72] | not published | not published | | ckb: UD_Central_Kurdish-Mukri | raw | Words | not measured | 81.74 [75.00, 88.11] | not published | not published | | cop: UD_Coptic-Scriptorium | gold | AllTags | not measured | 62.73 [61.51, 63.94] | 96.10 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | gold | CLAS | not measured | 38.27 [36.78, 40.01] | not published | not published | | cop: UD_Coptic-Scriptorium | gold | LAS | not measured | 51.38 [50.04, 52.70] | 88.27 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | gold | Lemmas | not measured | 88.04 [87.22, 88.88] | 97.67 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | gold | MLAS | not measured | 20.89 [19.66, 22.19] | 78.68 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | gold | UAS | not measured | 62.23 [60.95, 63.61] | 91.07 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | gold | UFeats | not measured | 72.24 [71.18, 73.33] | 97.94 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | gold | UPOS | not measured | 76.21 [75.18, 77.17] | 97.42 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | gold | XPOS | not measured | 73.58 [72.49, 74.68] | 97.16 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | cop: UD_Coptic-Scriptorium | raw | AllTags | not measured | 15.13 [14.46, 15.83] | 84.77 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 84.77 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | CLAS | not measured | 0.61 [0.37, 0.89] | 56.39 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 56.39 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | LAS | not measured | 0.80 [0.55, 1.10] | 64.43 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 64.43 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | Lemmas | not measured | 22.46 [21.49, 23.51] | 85.95 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 85.95 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | MLAS | not measured | 0.24 [0.11, 0.40] | 53.08 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 53.08 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | Sentences | not measured | 5.15 [2.78, 8.19] | 34.09 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 34.09 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | Tokens | not measured | 99.49 [98.85, 99.94] | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | UAS | not measured | 0.97 [0.68, 1.32] | 66.37 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 66.37 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | UFeats | not measured | 16.99 [16.30, 17.69] | 86.37 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 86.37 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | UPOS | not measured | 16.78 [16.00, 17.54] | 85.77 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 85.77 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | Words | not measured | 23.13 [22.10, 24.19] | 88.63 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 88.63 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cop: UD_Coptic-Scriptorium | raw | XPOS | not measured | 16.82 [16.08, 17.59] | 85.69 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 85.69 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | cs: UD_Czech-CAC | gold | AllTags | not measured | 68.03 [67.08, 68.96] | 97.42 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | gold | CLAS | not measured | 80.85 [79.61, 82.10] | not published | not published | | cs: UD_Czech-CAC | gold | LAS | not measured | 83.93 [82.92, 84.99] | 94.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | gold | Lemmas | not measured | 94.33 [93.83, 94.84] | 99.28 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | gold | MLAS | not measured | 48.87 [47.38, 50.51] | 90.46 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | gold | UAS | not measured | 87.40 [86.41, 88.27] | 96.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | gold | UFeats | not measured | 73.92 [73.03, 74.80] | 97.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | gold | UPOS | not measured | 95.08 [94.62, 95.50] | 99.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | gold | XPOS | not measured | 76.79 [75.91, 77.66] | 98.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cs: UD_Czech-CAC | raw | AllTags | not measured | 70.84 [70.01, 71.68] | 97.65 (Trankit large, UD 2.5; test same size) — **MISS** | 92.92 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cs: UD_Czech-CAC | raw | CLAS | not measured | 86.79 [85.63, 87.85] | 93.06 (Trankit large, UD 2.5; test same size) — **MISS** | 86.78 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | cs: UD_Czech-CAC | raw | LAS | not measured | 88.56 [87.59, 89.47] | 94.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 88.71 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | cs: UD_Czech-CAC | raw | Lemmas | not measured | 95.06 [94.57, 95.50] | 99.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.46 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cs: UD_Czech-CAC | raw | MLAS | not measured | 55.33 [53.88, 56.80] | 90.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.27 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cs: UD_Czech-CAC | raw | Sentences | not measured | 97.60 [96.39, 98.72] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cs: UD_Czech-CAC | raw | Tokens | not measured | 99.92 [99.82, 99.98] | 99.99 (Trankit large, UD 2.5; test same size) — **MISS** | 99.82 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | cs: UD_Czech-CAC | raw | UAS | not measured | 91.21 [90.28, 92.08] | 96.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 91.09 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | cs: UD_Czech-CAC | raw | UFeats | not measured | 75.72 [74.89, 76.56] | 98.01 (Trankit large, UD 2.5; test same size) — **MISS** | 93.64 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cs: UD_Czech-CAC | raw | UPOS | not measured | 95.77 [95.32, 96.18] | 99.64 (Trankit large, UD 2.5; test same size) — **MISS** | 98.52 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cs: UD_Czech-CAC | raw | Words | not measured | 99.67 [99.53, 99.78] | 99.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.57 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | cs: UD_Czech-CAC | raw | XPOS | not measured | 79.17 [78.37, 79.96] | 98.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.59 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | gold | AllTags | not measured | 74.20 [73.55, 74.85] | 88.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | gold | CLAS | not measured | 70.34 [69.27, 71.39] | not published | not published | | cy: UD_Welsh-CCG | gold | LAS | not measured | 75.92 [75.04, 76.75] | 82.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | gold | Lemmas | not measured | 91.15 [90.67, 91.63] | 95.32 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | gold | MLAS | not measured | 44.09 [42.90, 45.16] | 65.14 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | gold | UAS | not measured | 82.71 [81.92, 83.51] | 88.45 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | gold | UFeats | not measured | 79.27 [78.69, 79.90] | 90.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | gold | UPOS | not measured | 91.91 [91.45, 92.36] | 95.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | gold | XPOS | not measured | 91.03 [90.54, 91.45] | 94.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | cy: UD_Welsh-CCG | raw | AllTags | not measured | 67.37 [66.61, 68.13] | 87.83 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 87.83 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | CLAS | not measured | 62.93 [61.67, 64.17] | 74.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 74.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | LAS | not measured | 64.89 [63.74, 65.95] | 81.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.25 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | Lemmas | not measured | 81.89 [81.22, 82.52] | 94.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | MLAS | not measured | 37.17 [36.08, 38.25] | 64.32 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 62.81 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | Sentences | not measured | 81.61 [79.19, 83.90] | 97.79 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 97.79 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | Tokens | not measured | 92.22 [91.66, 92.72] | 99.90 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.90 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | UAS | not measured | 70.58 [69.41, 71.73] | 87.52 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 86.32 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | UFeats | not measured | 70.70 [69.97, 71.41] | 90.11 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 90.11 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | UPOS | not measured | 84.08 [83.38, 84.71] | 95.86 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 95.86 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | Words | not measured | 89.93 [89.38, 90.50] | 99.68 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.68 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | cy: UD_Welsh-CCG | raw | XPOS | not measured | 83.40 [82.70, 84.08] | 94.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 94.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | gold | AllTags | not measured | 1.00 [0.83, 1.19] | 96.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | gold | CLAS | not measured | 70.27 [68.75, 71.83] | not published | not published | | da: UD_Danish-DDT | gold | LAS | not measured | 74.84 [73.49, 76.18] | 87.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | gold | Lemmas | not measured | 94.41 [93.87, 94.92] | 97.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | gold | MLAS | not measured | 55.74 [54.13, 57.42] | 80.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | gold | UAS | not measured | 78.94 [77.63, 80.25] | 89.84 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | gold | UFeats | not measured | 90.63 [89.97, 91.21] | 97.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | gold | UPOS | not measured | 94.66 [94.09, 95.18] | 98.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | gold | XPOS | not measured | 1.00 [0.83, 1.19] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | da: UD_Danish-DDT | raw | AllTags | not measured | 0.99 [0.82, 1.18] | 97.87 (Trankit large, UD 2.5; test same size) — **MISS** | 97.14 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | CLAS | not measured | 77.30 [75.79, 78.87] | 88.81 (Trankit large, UD 2.5; test same size) — **MISS** | 83.80 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | LAS | not measured | 80.60 [79.35, 81.96] | 90.33 (Trankit large, UD 2.5; test same size) — **MISS** | 86.61 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | Lemmas | not measured | 93.97 [93.43, 94.52] | 97.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 97.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | MLAS | not measured | 65.32 [63.63, 67.11] | 85.66 (Trankit large, UD 2.5; test same size) — **MISS** | 80.06 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | Sentences | not measured | 93.90 [91.83, 95.75] | 95.46 (Trankit large, UD 2.5; test same size) — within or ahead | 91.81 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | da: UD_Danish-DDT | raw | Tokens | not measured | 98.69 [98.37, 99.02] | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | UAS | not measured | 83.84 [82.62, 85.19] | 91.75 (Trankit large, UD 2.5; test same size) — **MISS** | 88.61 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | UFeats | not measured | 91.29 [90.67, 91.88] | 98.33 (Trankit large, UD 2.5; test same size) — **MISS** | 97.87 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | UPOS | not measured | 94.32 [93.75, 94.85] | 98.71 (Trankit large, UD 2.5; test same size) — **MISS** | 98.30 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | Words | not measured | 98.69 [98.37, 99.02] | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | da: UD_Danish-DDT | raw | XPOS | not measured | 0.99 [0.82, 1.18] | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | de: UD_German-GSD | gold | AllTags | 83.34 [82.57, 84.04] | 83.09 [82.34, 83.75] | 89.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | gold | CLAS | not measured | 78.67 [77.54, 79.74] | not published | not published | | de: UD_German-GSD | gold | LAS | 83.32 [82.43, 84.17] | 83.49 [82.59, 84.31] | 86.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | gold | Lemmas | 96.25 [95.95, 96.53] | 96.64 [96.37, 96.92] | 97.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | gold | MLAS | not measured | 59.58 [58.38, 60.86] | 69.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | gold | UAS | 87.01 [86.18, 87.75] | 87.09 [86.26, 87.84] | 89.24 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | gold | UFeats | 86.81 [86.11, 87.44] | 86.58 [85.91, 87.19] | 91.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | gold | UPOS | 96.16 [95.83, 96.49] | 95.78 [95.43, 96.12] | 97.11 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | gold | XPOS | 96.07 [95.71, 96.43] | 95.84 [95.48, 96.19] | 97.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | de: UD_German-GSD | raw | AllTags | 81.04 [80.35, 81.70] | 81.69 [80.99, 82.33] | 88.82 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.19 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | de: UD_German-GSD | raw | CLAS | not measured | 81.12 [80.13, 82.07] | 81.49 (Trankit large, UD 2.5; test changed) — within or ahead | 77.08 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | de: UD_German-GSD | raw | LAS | 81.17 [80.27, 82.00] | 82.38 [81.55, 83.18] | 85.20 (Trankit large, UD 2.5; test changed) — **MISS** | 81.80 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | de: UD_German-GSD | raw | Lemmas | 93.36 [92.90, 93.78] | 94.07 [93.64, 94.48] | 97.23 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 97.23 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | de: UD_German-GSD | raw | MLAS | not measured | 60.36 [59.24, 61.44] | 67.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 62.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | de: UD_German-GSD | raw | Sentences | 88.70 [86.70, 90.74] | 88.65 [86.73, 90.58] | 92.72 (Trankit base, UD 2.5; test changed) — **MISS** | 83.84 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | de: UD_German-GSD | raw | Tokens | 97.40 [97.09, 97.69] | 97.40 [97.09, 97.69] | 99.75 (Trankit base, UD 2.5; test changed) — **MISS** | 99.62 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | de: UD_German-GSD | raw | UAS | 84.71 [83.86, 85.52] | 85.50 [84.64, 86.25] | 89.01 (Trankit large, UD 2.5; test changed) — **MISS** | 85.80 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | de: UD_German-GSD | raw | UFeats | 84.39 [83.77, 85.03] | 85.16 [84.51, 85.74] | 91.68 (Trankit large, UD 2.5; test changed) — **MISS** | 89.78 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | de: UD_German-GSD | raw | UPOS | 93.27 [92.79, 93.69] | 93.33 [92.87, 93.75] | 96.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.61 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | de: UD_German-GSD | raw | Words | 97.06 [96.70, 97.39] | 97.06 [96.70, 97.39] | 99.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.60 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | de: UD_German-GSD | raw | XPOS | 93.38 [92.91, 93.83] | 93.35 [92.91, 93.79] | 97.71 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.36 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | egy: UD_Egyptian-PC | gold | AllTags | not measured | 0.00 [0.00, 0.00] | not published | not published | | egy: UD_Egyptian-PC | gold | CLAS | not measured | 37.45 [36.31, 38.59] | not published | not published | | egy: UD_Egyptian-PC | gold | LAS | not measured | 42.41 [41.30, 43.51] | not published | not published | | egy: UD_Egyptian-PC | gold | Lemmas | not measured | 84.54 [83.73, 85.28] | not published | not published | | egy: UD_Egyptian-PC | gold | MLAS | not measured | 19.05 [18.22, 19.98] | not published | not published | | egy: UD_Egyptian-PC | gold | UAS | not measured | 57.76 [56.65, 58.80] | not published | not published | | egy: UD_Egyptian-PC | gold | UFeats | not measured | 52.85 [51.87, 53.87] | not published | not published | | egy: UD_Egyptian-PC | gold | UPOS | not measured | 73.69 [72.75, 74.56] | not published | not published | | egy: UD_Egyptian-PC | gold | XPOS | not measured | 0.00 [0.00, 0.00] | not published | not published | | egy: UD_Egyptian-PC | raw | AllTags | not measured | 0.00 [0.00, 0.00] | not published | not published | | egy: UD_Egyptian-PC | raw | CLAS | not measured | 9.42 [8.82, 10.07] | not published | not published | | egy: UD_Egyptian-PC | raw | LAS | not measured | 12.52 [11.81, 13.22] | not published | not published | | egy: UD_Egyptian-PC | raw | Lemmas | not measured | 38.59 [37.53, 39.58] | not published | not published | | egy: UD_Egyptian-PC | raw | MLAS | not measured | 5.15 [4.72, 5.61] | not published | not published | | egy: UD_Egyptian-PC | raw | Sentences | not measured | 3.26 [2.26, 4.34] | not published | not published | | egy: UD_Egyptian-PC | raw | Tokens | not measured | 46.31 [45.19, 47.45] | not published | not published | | egy: UD_Egyptian-PC | raw | UAS | not measured | 16.51 [15.72, 17.36] | not published | not published | | egy: UD_Egyptian-PC | raw | UFeats | not measured | 25.94 [25.02, 26.83] | not published | not published | | egy: UD_Egyptian-PC | raw | UPOS | not measured | 32.00 [31.01, 32.89] | not published | not published | | egy: UD_Egyptian-PC | raw | Words | not measured | 45.07 [43.98, 46.14] | not published | not published | | egy: UD_Egyptian-PC | raw | XPOS | not measured | 0.00 [0.00, 0.00] | not published | not published | | el: UD_Greek-GUD | gold | CLAS | not measured | 81.01 [79.90, 82.11] | not published | not published | | el: UD_Greek-GUD | gold | LAS | not measured | 82.87 [81.91, 83.75] | 90.79 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | el: UD_Greek-GUD | gold | Lemmas | not measured | 90.86 [90.34, 91.41] | 95.97 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | el: UD_Greek-GUD | gold | MLAS | not measured | 49.27 [47.87, 50.55] | 77.89 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | el: UD_Greek-GUD | gold | UAS | not measured | 89.65 [88.81, 90.43] | 94.05 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | el: UD_Greek-GUD | gold | UFeats | not measured | 78.37 [77.65, 79.14] | 94.41 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | el: UD_Greek-GUD | gold | UPOS | not measured | 92.10 [91.62, 92.62] | 98.13 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | el: UD_Greek-GUD | raw | CLAS | not measured | 83.78 [82.69, 84.84] | 80.28 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 80.28 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | el: UD_Greek-GUD | raw | LAS | not measured | 83.08 [82.11, 84.02] | 90.37 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 85.33 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | Lemmas | not measured | 87.84 [87.21, 88.52] | 95.93 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 92.63 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | MLAS | not measured | 52.10 [50.79, 53.43] | 77.62 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 68.93 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | Sentences | not measured | 92.76 [90.99, 94.36] | 98.16 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 98.16 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | Tokens | not measured | 99.51 [99.34, 99.69] | 99.89 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.89 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | UAS | not measured | 86.89 [85.95, 87.74] | 93.61 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 89.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | UFeats | not measured | 77.20 [76.41, 78.02] | 94.36 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 92.50 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | UPOS | not measured | 90.62 [90.00, 91.21] | 98.08 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 96.52 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | el: UD_Greek-GUD | raw | Words | not measured | 95.55 [95.09, 96.04] | 99.94 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 99.89 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | en: UD_English-EWT | gold | AllTags | 93.66 [93.25, 94.02] | 93.47 [93.07, 93.83] | 96.34 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | gold | CLAS | not measured | 81.48 [80.57, 82.45] | not published | not published | | en: UD_English-EWT | gold | LAS | 85.43 [84.61, 86.19] | 85.12 [84.40, 85.87] | 92.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | gold | Lemmas | 97.09 [96.81, 97.36] | 97.26 [97.01, 97.52] | 97.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | gold | MLAS | not measured | 75.10 [74.10, 76.11] | 86.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | gold | UAS | 88.56 [87.81, 89.26] | 88.34 [87.65, 89.01] | 94.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | gold | UFeats | 96.44 [96.17, 96.69] | 96.37 [96.09, 96.62] | 98.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | gold | UPOS | 96.58 [96.26, 96.86] | 96.49 [96.19, 96.76] | 97.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | gold | XPOS | 96.02 [95.69, 96.32] | 95.73 [95.40, 96.04] | 97.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | en: UD_English-EWT | raw | AllTags | 93.14 [92.69, 93.57] | 93.39 [92.93, 93.80] | 95.51 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.64 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | en: UD_English-EWT | raw | CLAS | not measured | 83.38 [82.51, 84.30] | 87.41 (Trankit large, UD 2.5; test changed) — **MISS** | 84.11 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | en: UD_English-EWT | raw | LAS | 85.02 [84.25, 85.77] | 86.55 [85.82, 87.26] | 89.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 86.77 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | en: UD_English-EWT | raw | Lemmas | 96.14 [95.81, 96.48] | 96.39 [96.04, 96.74] | 97.25 (Trankit large, UD 2.5; test changed) — **MISS** | 96.63 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | en: UD_English-EWT | raw | MLAS | not measured | 78.46 [77.50, 79.41] | 83.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.27 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | en: UD_English-EWT | raw | Sentences | 75.28 [73.54, 77.14] | 75.85 [74.20, 77.60] | 90.49 (Trankit large, UD 2.5; test changed) — **MISS** | 89.16 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | en: UD_English-EWT | raw | Tokens | 97.13 [96.83, 97.43] | 97.13 [96.83, 97.43] | 99.17 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.17 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | en: UD_English-EWT | raw | UAS | 87.32 [86.58, 88.01] | 88.78 [88.07, 89.47] | 91.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 88.90 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | en: UD_English-EWT | raw | UFeats | 95.76 [95.42, 96.07] | 95.94 [95.61, 96.27] | 97.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.39 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | en: UD_English-EWT | raw | UPOS | 95.95 [95.59, 96.29] | 95.95 [95.59, 96.27] | 96.75 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.20 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | en: UD_English-EWT | raw | Words | 98.68 [98.45, 98.90] | 98.68 [98.45, 98.90] | 99.07 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 98.92 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | en: UD_English-EWT | raw | XPOS | 95.22 [94.83, 95.58] | 95.18 [94.78, 95.54] | 96.47 (Trankit large, UD 2.5; test changed) — **MISS** | 95.75 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | gold | AllTags | 83.67 [83.20, 84.13] | 83.21 [82.74, 83.67] | 95.64 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | gold | CLAS | not measured | 82.53 [81.95, 83.14] | not published | not published | | es: UD_Spanish-AnCora | gold | LAS | 85.37 [84.89, 85.88] | 86.97 [86.51, 87.42] | 92.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | gold | Lemmas | 97.72 [97.58, 97.85] | 98.53 [98.42, 98.64] | 99.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | gold | MLAS | not measured | 75.87 [75.23, 76.53] | 87.03 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | gold | UAS | 88.14 [87.68, 88.63] | 89.50 [89.08, 89.94] | 94.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | gold | UFeats | 96.50 [96.33, 96.66] | 96.74 [96.58, 96.91] | 98.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | gold | UPOS | 98.10 [97.98, 98.24] | 98.00 [97.87, 98.14] | 99.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | gold | XPOS | 84.88 [84.41, 85.31] | 84.37 [83.90, 84.82] | 96.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | es: UD_Spanish-AnCora | raw | AllTags | 84.50 [84.05, 84.94] | 84.84 [84.40, 85.29] | 98.36 (Trankit large, UD 2.5; test changed) — **MISS** | 95.65 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | CLAS | not measured | 86.36 [85.78, 86.89] | 91.10 (Trankit large, UD 2.5; test changed) — **MISS** | 87.80 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | LAS | 87.42 [86.90, 87.92] | 89.70 [89.24, 90.11] | 93.29 (Trankit large, UD 2.5; test changed) — **MISS** | 91.30 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | Lemmas | 97.75 [97.61, 97.89] | 98.58 [98.46, 98.70] | 99.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.24 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | MLAS | not measured | 80.63 [79.97, 81.23] | 89.23 (Trankit large, UD 2.5; test changed) — **MISS** | 85.49 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | Sentences | 96.66 [95.79, 97.49] | 97.05 [96.24, 97.78] | 99.39 (Trankit large, UD 2.5; test changed) — **MISS** | 98.49 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | Tokens | 99.84 [99.76, 99.90] | 99.84 [99.76, 99.90] | 99.97 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.97 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | UAS | 90.08 [89.60, 90.51] | 91.93 [91.49, 92.31] | 94.77 (Trankit large, UD 2.5; test changed) — **MISS** | 93.09 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | UFeats | 96.57 [96.40, 96.73] | 96.83 [96.66, 96.99] | 98.85 (Trankit large, UD 2.5; test changed) — **MISS** | 98.54 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | UPOS | 98.09 [97.96, 98.24] | 98.20 [98.07, 98.34] | 99.06 (Trankit large, UD 2.5; test changed) — **MISS** | 98.96 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | Words | 99.83 [99.75, 99.89] | 99.83 [99.75, 99.89] | 99.96 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.96 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | es: UD_Spanish-AnCora | raw | XPOS | 85.63 [85.17, 86.09] | 85.87 [85.43, 86.33] | 99.00 (Trankit large, UD 2.5; test changed) — **MISS** | 96.10 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | fa: UD_Persian-Seraji | gold | AllTags | not measured | 1.58 [1.38, 1.79] | 97.82 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | gold | CLAS | not measured | 61.56 [60.30, 62.91] | not published | not published | | fa: UD_Persian-Seraji | gold | LAS | not measured | 66.70 [65.63, 67.81] | 89.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | gold | Lemmas | not measured | 96.14 [95.87, 96.43] | 98.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | gold | MLAS | not measured | 29.39 [28.32, 30.42] | 85.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | gold | UAS | not measured | 73.22 [72.20, 74.32] | 92.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | gold | UFeats | not measured | 72.36 [71.68, 73.09] | 98.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | gold | UPOS | not measured | 85.15 [84.41, 85.81] | 98.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | gold | XPOS | not measured | 3.41 [3.09, 3.75] | 98.28 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fa: UD_Persian-Seraji | raw | AllTags | not measured | 4.75 [4.35, 5.14] | 97.49 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.31 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | CLAS | not measured | 65.75 [64.46, 66.96] | 86.86 (Trankit large, UD 2.5; test same size) — **MISS** | 85.43 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | LAS | not measured | 70.39 [69.29, 71.48] | 89.58 (Trankit large, UD 2.5; test same size) — **MISS** | 87.96 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | Lemmas | not measured | 95.47 [95.12, 95.82] | 98.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 98.18 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | MLAS | not measured | 32.32 [31.28, 33.28] | 84.97 (Trankit large, UD 2.5; test same size) — **MISS** | 83.76 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | Sentences | not measured | 90.42 [88.10, 92.57] | 99.25 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.25 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | Tokens | not measured | 100.00 [100.00, 100.00] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | fa: UD_Persian-Seraji | raw | UAS | not measured | 76.52 [75.54, 77.59] | 92.24 (Trankit large, UD 2.5; test same size) — **MISS** | 90.98 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | UFeats | not measured | 72.04 [71.38, 72.80] | 97.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.70 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | UPOS | not measured | 86.26 [85.61, 86.83] | 97.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.69 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | Words | not measured | 99.08 [98.88, 99.28] | 99.72 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.72 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fa: UD_Persian-Seraji | raw | XPOS | not measured | 7.17 [6.71, 7.66] | 97.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | gold | AllTags | not measured | 59.08 [58.24, 59.90] | 94.23 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | gold | CLAS | not measured | 72.18 [71.04, 73.24] | not published | not published | | fi: UD_Finnish-FTB | gold | LAS | not measured | 72.93 [71.91, 73.94] | 89.42 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | gold | Lemmas | not measured | 81.17 [80.50, 81.86] | 95.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | gold | MLAS | not measured | 48.49 [47.40, 49.63] | 82.72 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | gold | UAS | not measured | 79.34 [78.40, 80.27] | 92.38 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | gold | UFeats | not measured | 78.17 [77.47, 78.83] | 96.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | gold | UPOS | not measured | 87.31 [86.69, 87.87] | 97.05 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | gold | XPOS | not measured | 62.13 [61.30, 62.92] | 95.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fi: UD_Finnish-FTB | raw | AllTags | not measured | 49.32 [48.47, 50.14] | 96.87 (Trankit large, UD 2.5; test same size) — **MISS** | 95.29 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | CLAS | not measured | 75.32 [74.16, 76.42] | 93.54 (Trankit large, UD 2.5; test same size) — **MISS** | 86.24 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | LAS | not measured | 77.41 [76.42, 78.35] | 94.53 (Trankit large, UD 2.5; test same size) — **MISS** | 88.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | Lemmas | not measured | 82.30 [81.70, 83.00] | 97.10 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 97.10 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | MLAS | not measured | 52.18 [51.07, 53.35] | 90.68 (Trankit large, UD 2.5; test same size) — **MISS** | 82.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | Sentences | not measured | 87.32 [85.87, 88.72] | 97.36 (Trankit large, UD 2.5; test same size) — **MISS** | 90.13 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | Tokens | not measured | 99.59 [99.44, 99.72] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | UAS | not measured | 82.60 [81.70, 83.50] | 95.84 (Trankit large, UD 2.5; test same size) — **MISS** | 90.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | UFeats | not measured | 79.28 [78.59, 79.94] | 98.09 (Trankit large, UD 2.5; test same size) — **MISS** | 97.12 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | UPOS | not measured | 87.62 [87.05, 88.17] | 98.32 (Trankit large, UD 2.5; test same size) — **MISS** | 96.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | Words | not measured | 99.36 [99.20, 99.51] | 99.98 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.98 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fi: UD_Finnish-FTB | raw | XPOS | not measured | 51.48 [50.63, 52.30] | 97.29 (Trankit large, UD 2.5; test same size) — **MISS** | 96.09 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | gold | AllTags | not measured | 77.78 [76.95, 78.63] | 92.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fo: UD_Faroese-FarPaHC | gold | CLAS | not measured | 66.42 [64.64, 68.31] | not published | not published | | fo: UD_Faroese-FarPaHC | gold | LAS | not measured | 71.92 [70.54, 73.47] | 83.58 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fo: UD_Faroese-FarPaHC | gold | Lemmas | not measured | 100.00 [100.00, 100.00] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | fo: UD_Faroese-FarPaHC | gold | MLAS | not measured | 47.80 [46.06, 49.66] | 69.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fo: UD_Faroese-FarPaHC | gold | UAS | not measured | 77.08 [75.67, 78.58] | 87.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fo: UD_Faroese-FarPaHC | gold | UFeats | not measured | 86.06 [85.32, 86.77] | 94.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fo: UD_Faroese-FarPaHC | gold | UPOS | not measured | 94.59 [94.09, 95.06] | 97.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fo: UD_Faroese-FarPaHC | gold | XPOS | not measured | 81.33 [80.53, 82.21] | 93.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fo: UD_Faroese-FarPaHC | raw | AllTags | not measured | 77.85 [76.98, 78.74] | 92.46 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 90.86 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | CLAS | not measured | 72.03 [70.17, 73.86] | 72.28 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 72.28 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | fo: UD_Faroese-FarPaHC | raw | LAS | not measured | 76.88 [75.50, 78.29] | 82.71 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.07 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | Lemmas | not measured | 99.39 [99.20, 99.58] | 99.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.64 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | MLAS | not measured | 53.79 [52.01, 55.62] | 68.94 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 64.48 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | Sentences | not measured | 91.84 [88.85, 94.59] | 93.94 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 93.94 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | fo: UD_Faroese-FarPaHC | raw | Tokens | not measured | 99.41 [99.21, 99.59] | 99.65 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.65 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | UAS | not measured | 82.00 [80.73, 83.34] | 86.48 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 84.39 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | UFeats | not measured | 86.72 [85.97, 87.46] | 94.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 92.92 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | UPOS | not measured | 94.17 [93.65, 94.68] | 97.48 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.51 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | Words | not measured | 99.39 [99.20, 99.58] | 99.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.64 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fo: UD_Faroese-FarPaHC | raw | XPOS | not measured | 81.38 [80.53, 82.24] | 92.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 91.56 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | fr: UD_French-GSD | gold | CLAS | not measured | 82.47 [81.26, 83.85] | not published | not published | | fr: UD_French-GSD | gold | LAS | 86.55 [85.57, 87.49] | 86.89 [85.93, 87.88] | 93.11 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fr: UD_French-GSD | gold | Lemmas | 97.56 [97.23, 97.88] | 98.03 [97.72, 98.31] | 98.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fr: UD_French-GSD | gold | MLAS | not measured | 75.00 [73.45, 76.63] | 86.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fr: UD_French-GSD | gold | UAS | 89.36 [88.49, 90.23] | 89.52 [88.61, 90.43] | 94.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fr: UD_French-GSD | gold | UFeats | 96.24 [95.75, 96.67] | 96.06 [95.58, 96.49] | 98.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fr: UD_French-GSD | gold | UPOS | 97.21 [96.82, 97.58] | 96.99 [96.60, 97.39] | 98.55 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | fr: UD_French-GSD | raw | CLAS | not measured | 86.65 [85.41, 87.83] | 90.17 (Trankit large, UD 2.5; test changed) — **MISS** | 87.44 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | fr: UD_French-GSD | raw | LAS | 88.07 [87.07, 89.05] | 89.78 [88.84, 90.63] | 93.37 (Trankit large, UD 2.5; test changed) — **MISS** | 91.28 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | fr: UD_French-GSD | raw | Lemmas | 97.42 [97.09, 97.77] | 97.96 [97.64, 98.25] | 98.20 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 98.20 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | fr: UD_French-GSD | raw | MLAS | not measured | 80.57 [79.06, 81.91] | 86.35 (Trankit large, UD 2.5; test changed) — **MISS** | 84.45 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | fr: UD_French-GSD | raw | Sentences | 92.40 [89.74, 94.66] | 92.65 [90.07, 94.98] | 98.80 (Trankit large, UD 2.5; test changed) — **MISS** | 95.03 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | fr: UD_French-GSD | raw | Tokens | 99.88 [99.79, 99.96] | 99.88 [99.79, 99.96] | 99.77 (Trankit large, UD 2.5; test changed) — within or ahead | 99.71 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | fr: UD_French-GSD | raw | UAS | 90.67 [89.74, 91.58] | 91.97 [91.12, 92.76] | 94.97 (Trankit large, UD 2.5; test changed) — **MISS** | 93.27 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | fr: UD_French-GSD | raw | UFeats | 96.07 [95.59, 96.52] | 96.26 [95.82, 96.68] | 97.74 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 97.74 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | fr: UD_French-GSD | raw | UPOS | 97.06 [96.71, 97.42] | 97.10 [96.71, 97.46] | 97.89 (Trankit large, UD 2.5; test changed) — **MISS** | 97.63 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | fr: UD_French-GSD | raw | Words | 99.48 [99.29, 99.65] | 99.48 [99.29, 99.65] | 99.72 (Trankit large, UD 2.5; test changed) — **MISS** | 99.50 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ga: UD_Irish-IDT | gold | AllTags | not measured | 60.83 [59.34, 62.05] | 88.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | gold | CLAS | not measured | 64.78 [62.68, 66.81] | not published | not published | | ga: UD_Irish-IDT | gold | LAS | not measured | 71.66 [69.97, 73.24] | 82.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | gold | Lemmas | not measured | 91.11 [90.18, 91.90] | 95.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | gold | MLAS | not measured | 32.12 [30.49, 33.73] | 65.80 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | gold | UAS | not measured | 78.67 [77.14, 80.11] | 87.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | gold | UFeats | not measured | 68.36 [67.18, 69.47] | 91.43 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | gold | UPOS | not measured | 90.83 [89.85, 91.65] | 96.46 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | gold | XPOS | not measured | 85.82 [84.83, 86.77] | 95.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ga: UD_Irish-IDT | raw | AllTags | not measured | 61.92 [60.46, 63.32] | 88.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 86.52 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | CLAS | not measured | 68.12 [65.79, 70.27] | 76.34 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 76.34 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | LAS | not measured | 73.90 [71.88, 75.71] | 81.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 81.51 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | Lemmas | not measured | 89.93 [88.98, 90.82] | 95.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.32 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | MLAS | not measured | 34.25 [32.40, 35.98] | 65.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 63.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | Sentences | not measured | 86.06 [82.70, 89.19] | 98.24 (Trankit large, UD 2.5; test changed) — **MISS** | 97.24 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | Tokens | not measured | 98.09 [97.47, 98.59] | 99.75 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.75 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | UAS | not measured | 80.66 [78.83, 82.33] | 87.31 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 86.81 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | UFeats | not measured | 68.07 [66.72, 69.32] | 91.32 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.41 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | UPOS | not measured | 90.27 [89.27, 91.12] | 96.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.01 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | Words | not measured | 98.09 [97.47, 98.59] | 99.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.75 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ga: UD_Irish-IDT | raw | XPOS | not measured | 87.25 [86.27, 88.17] | 95.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.55 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | gold | AllTags | not measured | 65.11 [63.95, 66.15] | 90.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | CLAS | not measured | 64.97 [63.30, 66.43] | not published | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | LAS | not measured | 71.21 [69.81, 72.39] | 83.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | Lemmas | not measured | 93.80 [93.19, 94.32] | 97.70 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | MLAS | not measured | 42.12 [40.61, 43.54] | 72.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | UAS | not measured | 78.53 [77.26, 79.68] | 87.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | UFeats | not measured | 77.00 [76.03, 77.94] | 93.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | UPOS | not measured | 90.89 [90.16, 91.57] | 96.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | gold | XPOS | not measured | 70.33 [69.21, 71.31] | 92.03 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gd: UD_Scottish_Gaelic-ARCOSG | raw | AllTags | not measured | 54.95 [53.83, 55.93] | 89.42 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 89.42 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | CLAS | not measured | 58.12 [56.23, 60.02] | 72.86 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 72.86 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | LAS | not measured | 59.34 [57.76, 60.86] | 77.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 77.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | Lemmas | not measured | 80.82 [79.94, 81.74] | 95.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 95.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | MLAS | not measured | 34.30 [32.91, 35.65] | 66.53 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 66.53 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | Sentences | not measured | 63.02 [59.09, 67.21] | 64.54 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 64.54 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | gd: UD_Scottish_Gaelic-ARCOSG | raw | Tokens | not measured | 90.18 [89.42, 90.90] | 99.65 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.65 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | UAS | not measured | 64.70 [63.06, 66.21] | 82.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 82.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | UFeats | not measured | 65.18 [64.18, 66.18] | 91.91 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 91.91 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | UPOS | not measured | 78.37 [77.49, 79.18] | 94.70 (Trankit large, UD 2.5; test changed) — **MISS** | 94.62 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | Words | not measured | 85.57 [84.70, 86.41] | 99.42 (Trankit large, UD 2.5; test changed) — **MISS** | 98.86 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gd: UD_Scottish_Gaelic-ARCOSG | raw | XPOS | not measured | 59.64 [58.48, 60.65] | 90.24 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 90.24 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | gl: UD_Galician-TreeGal | gold | AllTags | not measured | 81.94 [80.86, 82.94] | 94.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | gold | CLAS | not measured | 72.40 [70.85, 74.02] | not published | not published | | gl: UD_Galician-TreeGal | gold | LAS | not measured | 76.69 [75.42, 77.87] | 82.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | gold | Lemmas | not measured | 95.39 [94.95, 95.82] | 98.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | gold | MLAS | not measured | 54.87 [53.08, 56.73] | 71.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | gold | UAS | not measured | 82.48 [81.42, 83.50] | 86.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | gold | UFeats | not measured | 87.74 [86.84, 88.54] | 96.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | gold | UPOS | not measured | 91.85 [91.17, 92.52] | 97.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | gold | XPOS | not measured | 85.46 [84.56, 86.30] | 95.42 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | gl: UD_Galician-TreeGal | raw | AllTags | not measured | 72.91 [71.91, 73.96] | 94.97 (Trankit large, UD 2.5; test same size) — **MISS** | 90.64 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | CLAS | not measured | 71.90 [70.19, 73.59] | 80.31 (Trankit large, UD 2.5; test same size) — **MISS** | 64.38 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | gl: UD_Galician-TreeGal | raw | LAS | not measured | 66.34 [65.07, 67.80] | 83.90 (Trankit large, UD 2.5; test same size) — **MISS** | 71.86 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | Lemmas | not measured | 83.92 [83.08, 84.76] | 97.31 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.52 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | MLAS | not measured | 49.73 [48.04, 51.36] | 74.67 (Trankit large, UD 2.5; test same size) — **MISS** | 58.14 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | Sentences | not measured | 79.95 [76.05, 83.67] | 95.52 (Trankit large, UD 2.5; test same size) — **MISS** | 88.02 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | Tokens | not measured | 99.28 [98.99, 99.53] | 99.68 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.68 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | UAS | not measured | 70.24 [68.98, 71.51] | 87.17 (Trankit large, UD 2.5; test same size) — **MISS** | 77.27 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | UFeats | not measured | 78.13 [77.16, 79.07] | 96.50 (Trankit large, UD 2.5; test same size) — **MISS** | 93.11 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | UPOS | not measured | 81.47 [80.65, 82.36] | 97.62 (Trankit large, UD 2.5; test same size) — **MISS** | 93.89 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | Words | not measured | 87.31 [86.48, 88.11] | 99.22 (Trankit large, UD 2.5; test same size) — **MISS** | 98.53 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | gl: UD_Galician-TreeGal | raw | XPOS | not measured | 75.29 [74.29, 76.34] | 95.68 (Trankit large, UD 2.5; test same size) — **MISS** | 91.51 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | grc: UD_Ancient_Greek-PTNK | gold | CLAS | not measured | 67.13 [65.47, 68.59] | not published | not published | | grc: UD_Ancient_Greek-PTNK | gold | LAS | not measured | 73.13 [71.87, 74.27] | 87.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | gold | Lemmas | not measured | 85.61 [84.97, 86.33] | 94.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | gold | MLAS | not measured | 36.20 [34.57, 37.58] | 70.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | gold | UAS | not measured | 77.95 [76.78, 79.06] | 92.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | gold | UFeats | not measured | 71.83 [70.76, 72.88] | 91.80 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | gold | UPOS | not measured | 93.42 [92.84, 93.97] | 98.55 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | CLAS | not measured | 70.73 [69.01, 72.43] | not published | not published | | grc: UD_Ancient_Greek-PTNK | raw | LAS | not measured | 75.45 [74.11, 76.71] | 84.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | Lemmas | not measured | 84.76 [84.05, 85.46] | 94.84 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | MLAS | not measured | 38.16 [36.46, 39.77] | 66.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | Sentences | not measured | 46.79 [41.96, 52.07] | 55.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | Tokens | not measured | 99.03 [98.78, 99.26] | not published | not published | | grc: UD_Ancient_Greek-PTNK | raw | UAS | not measured | 80.23 [78.90, 81.53] | 88.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | UFeats | not measured | 71.42 [70.28, 72.44] | 91.75 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | UPOS | not measured | 92.10 [91.47, 92.68] | 98.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | grc: UD_Ancient_Greek-PTNK | raw | Words | not measured | 98.96 [98.72, 99.19] | 99.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | AllTags | not measured | 73.33 [72.42, 74.24] | 91.52 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | CLAS | not measured | 73.52 [72.25, 74.92] | not published | not published | | he: UD_Hebrew-IAHLTwiki | gold | LAS | not measured | 76.97 [75.90, 78.16] | 91.23 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | Lemmas | not measured | 94.16 [93.65, 94.62] | 98.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | MLAS | not measured | 43.05 [41.54, 44.44] | 75.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | UAS | not measured | 81.92 [80.92, 82.90] | 93.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | UFeats | not measured | 77.07 [76.20, 77.92] | 92.55 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | UPOS | not measured | 88.46 [87.69, 89.21] | 97.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | gold | XPOS | not measured | 90.72 [90.05, 91.35] | 97.11 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | he: UD_Hebrew-IAHLTwiki | raw | AllTags | not measured | 41.59 [40.42, 42.81] | 86.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 86.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | CLAS | not measured | 27.76 [26.30, 29.28] | 75.58 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 75.58 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | LAS | not measured | 29.34 [27.85, 30.73] | 81.19 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 81.19 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | Lemmas | not measured | 51.58 [50.39, 52.82] | 92.51 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 92.51 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | MLAS | not measured | 16.88 [15.80, 17.93] | 64.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 64.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | Sentences | not measured | 42.17 [37.22, 47.57] | 97.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.78 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | Tokens | not measured | 97.18 [96.65, 97.72] | 99.64 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.64 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | UAS | not measured | 30.34 [28.78, 31.81] | 83.23 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 83.23 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | UFeats | not measured | 42.73 [41.55, 43.94] | 86.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 86.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | UPOS | not measured | 50.85 [49.71, 52.06] | 91.69 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 91.69 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | Words | not measured | 54.72 [53.59, 55.98] | 94.61 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 94.61 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | he: UD_Hebrew-IAHLTwiki | raw | XPOS | not measured | 51.98 [50.83, 53.18] | 91.69 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 91.69 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hr: UD_Croatian-SET | gold | AllTags | not measured | 76.51 [75.83, 77.15] | 95.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | gold | CLAS | not measured | 75.54 [74.70, 76.39] | not published | not published | | hr: UD_Croatian-SET | gold | LAS | not measured | 79.15 [78.36, 79.95] | 89.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | gold | Lemmas | not measured | 92.20 [91.78, 92.59] | 97.71 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | gold | MLAS | not measured | 54.16 [53.21, 55.13] | 82.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | gold | UAS | not measured | 84.29 [83.63, 85.02] | 92.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | gold | UFeats | not measured | 81.27 [80.65, 81.85] | 96.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | gold | UPOS | not measured | 93.74 [93.41, 94.10] | 98.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | gold | XPOS | not measured | 80.52 [79.83, 81.12] | 95.90 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hr: UD_Croatian-SET | raw | AllTags | not measured | 80.47 [79.88, 81.06] | 95.99 (Trankit large, UD 2.5; test same size) — **MISS** | 94.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | CLAS | not measured | 83.78 [83.02, 84.46] | 87.77 (Trankit large, UD 2.5; test same size) — **MISS** | 84.14 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | hr: UD_Croatian-SET | raw | LAS | not measured | 86.11 [85.44, 86.69] | 89.74 (Trankit large, UD 2.5; test same size) — **MISS** | 86.95 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | Lemmas | not measured | 92.60 [92.21, 92.96] | 97.64 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | MLAS | not measured | 62.72 [61.75, 63.63] | 82.19 (Trankit large, UD 2.5; test same size) — **MISS** | 78.19 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | Sentences | not measured | 94.43 [93.04, 95.73] | 99.16 (Trankit base, UD 2.5; test same size) — **MISS** | 97.08 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | Tokens | not measured | 98.83 [98.63, 99.02] | 99.96 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.96 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | UAS | not measured | 89.62 [89.01, 90.20] | 93.86 (Trankit large, UD 2.5; test same size) — **MISS** | 90.29 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | UFeats | not measured | 83.30 [82.78, 83.82] | 96.85 (Trankit large, UD 2.5; test same size) — **MISS** | 95.27 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | UPOS | not measured | 94.15 [93.80, 94.48] | 98.58 (Trankit large, UD 2.5; test same size) — **MISS** | 98.21 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | Words | not measured | 98.83 [98.63, 99.02] | 99.96 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.96 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | hr: UD_Croatian-SET | raw | XPOS | not measured | 83.44 [82.88, 83.98] | 96.55 (Trankit large, UD 2.5; test same size) — **MISS** | 94.89 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ht: UD_Haitian_Creole-Adolphe | gold | CLAS | not measured | 51.35 [49.48, 53.37] | not published | not published | | ht: UD_Haitian_Creole-Adolphe | gold | LAS | not measured | 54.20 [52.50, 56.13] | 60.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | gold | Lemmas | not measured | 98.70 [98.42, 98.96] | 99.05 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | gold | MLAS | not measured | 40.26 [38.56, 42.04] | 50.10 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | gold | UAS | not measured | 59.00 [57.26, 60.95] | 64.83 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | gold | UFeats | not measured | 97.12 [96.65, 97.52] | 98.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | gold | UPOS | not measured | 93.12 [92.47, 93.71] | 95.55 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | raw | CLAS | not measured | 55.04 [53.35, 56.98] | not published | not published | | ht: UD_Haitian_Creole-Adolphe | raw | LAS | not measured | 57.76 [56.19, 59.57] | 60.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | raw | Lemmas | not measured | 98.91 [98.66, 99.15] | 99.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ht: UD_Haitian_Creole-Adolphe | raw | MLAS | not measured | 45.00 [43.41, 46.76] | 50.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | raw | Sentences | not measured | 97.28 [95.80, 98.64] | 98.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | ht: UD_Haitian_Creole-Adolphe | raw | UAS | not measured | 62.04 [60.34, 63.85] | 64.82 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | raw | UFeats | not measured | 97.75 [97.40, 98.07] | 97.98 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ht: UD_Haitian_Creole-Adolphe | raw | UPOS | not measured | 94.15 [93.65, 94.59] | 95.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ht: UD_Haitian_Creole-Adolphe | raw | Words | not measured | 100.00 [100.00, 100.00] | 99.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | hy: UD_Armenian-ArmTDP | gold | CLAS | not measured | 72.45 [70.80, 74.17] | not published | not published | | hy: UD_Armenian-ArmTDP | gold | LAS | not measured | 76.44 [75.04, 77.94] | 87.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hy: UD_Armenian-ArmTDP | gold | Lemmas | not measured | 92.16 [91.51, 92.79] | 97.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hy: UD_Armenian-ArmTDP | gold | MLAS | not measured | 43.37 [41.73, 45.16] | 76.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hy: UD_Armenian-ArmTDP | gold | UAS | not measured | 82.51 [81.17, 83.84] | 90.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hy: UD_Armenian-ArmTDP | gold | UFeats | not measured | 71.29 [70.21, 72.45] | 94.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hy: UD_Armenian-ArmTDP | gold | UPOS | not measured | 93.30 [92.71, 93.86] | 98.06 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hy: UD_Armenian-ArmTDP | raw | CLAS | not measured | 69.55 [67.66, 71.50] | 80.91 (Trankit large, UD 2.5; test changed) — **MISS** | 52.52 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | hy: UD_Armenian-ArmTDP | raw | LAS | not measured | 72.59 [70.91, 74.32] | 86.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 61.80 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | hy: UD_Armenian-ArmTDP | raw | Lemmas | not measured | 89.43 [88.45, 90.37] | 96.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 91.35 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hy: UD_Armenian-ArmTDP | raw | MLAS | not measured | 41.47 [39.92, 43.13] | 76.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 29.58 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | hy: UD_Armenian-ArmTDP | raw | Sentences | not measured | 75.25 [72.09, 78.44] | 99.46 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.46 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hy: UD_Armenian-ArmTDP | raw | Tokens | not measured | 97.68 [96.98, 98.30] | 99.92 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.92 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hy: UD_Armenian-ArmTDP | raw | UAS | not measured | 77.81 [76.28, 79.54] | 89.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 74.04 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | hy: UD_Armenian-ArmTDP | raw | UFeats | not measured | 68.74 [67.65, 69.79] | 94.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 66.22 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | hy: UD_Armenian-ArmTDP | raw | UPOS | not measured | 90.12 [89.22, 90.98] | 97.67 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 83.05 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | hy: UD_Armenian-ArmTDP | raw | Words | not measured | 96.34 [95.57, 97.03] | 99.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.48 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | gold | CLAS | not measured | 68.29 [66.88, 69.56] | not published | not published | | hyw: UD_Western_Armenian-ArmTDP | gold | LAS | not measured | 75.20 [74.07, 76.31] | 85.14 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hyw: UD_Western_Armenian-ArmTDP | gold | Lemmas | not measured | 91.15 [90.60, 91.67] | 97.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hyw: UD_Western_Armenian-ArmTDP | gold | MLAS | not measured | 33.62 [32.35, 34.93] | 70.46 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hyw: UD_Western_Armenian-ArmTDP | gold | UAS | not measured | 82.46 [81.40, 83.53] | 89.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hyw: UD_Western_Armenian-ArmTDP | gold | UFeats | not measured | 68.53 [67.63, 69.40] | 92.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hyw: UD_Western_Armenian-ArmTDP | gold | UPOS | not measured | 90.67 [90.07, 91.19] | 96.90 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | hyw: UD_Western_Armenian-ArmTDP | raw | CLAS | not measured | 64.78 [63.25, 66.27] | 77.33 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 77.33 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | LAS | not measured | 71.14 [69.87, 72.46] | 84.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 82.83 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | Lemmas | not measured | 88.68 [88.02, 89.34] | 96.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.70 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | MLAS | not measured | 32.06 [30.64, 33.36] | 70.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 68.89 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | Sentences | not measured | 89.14 [86.80, 91.23] | 99.56 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.56 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | Tokens | not measured | 97.81 [97.49, 98.16] | 99.95 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.95 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | UAS | not measured | 77.90 [76.61, 79.29] | 89.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | UFeats | not measured | 66.33 [65.43, 67.22] | 92.70 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 92.70 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | UPOS | not measured | 87.57 [86.89, 88.26] | 96.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.42 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | hyw: UD_Western_Armenian-ArmTDP | raw | Words | not measured | 96.53 [96.10, 96.98] | 99.91 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.91 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | id: UD_Indonesian-GSD | gold | AllTags | not measured | 74.40 [73.51, 75.36] | 89.24 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | gold | CLAS | not measured | 63.61 [62.21, 65.22] | not published | not published | | id: UD_Indonesian-GSD | gold | LAS | not measured | 67.40 [66.10, 68.75] | 82.72 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | gold | Lemmas | not measured | 95.84 [95.43, 96.23] | 98.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | gold | MLAS | not measured | 50.56 [49.26, 52.04] | 73.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | gold | UAS | not measured | 75.84 [74.67, 77.07] | 88.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | gold | UFeats | not measured | 89.32 [88.69, 89.93] | 95.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | gold | UPOS | not measured | 91.42 [90.75, 92.11] | 94.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | gold | XPOS | not measured | 80.13 [79.27, 80.99] | 94.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | id: UD_Indonesian-GSD | raw | AllTags | not measured | 75.52 [74.56, 76.51] | 89.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 89.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | CLAS | not measured | 69.72 [68.23, 71.40] | 78.64 (Trankit base, UD 2.5; test changed) — **MISS** | 76.98 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | LAS | not measured | 73.87 [72.45, 75.36] | 81.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.83 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | Lemmas | not measured | 94.33 [93.77, 94.84] | 99.53 (Trankit large, UD 2.5; test changed) — **MISS** | 98.28 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | MLAS | not measured | 59.56 [57.98, 61.27] | 72.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 71.78 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | Sentences | not measured | 78.47 [74.98, 82.32] | 95.54 (Trankit large, UD 2.5; test changed) — **MISS** | 93.04 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | Tokens | not measured | 99.73 [99.59, 99.85] | 99.89 (Trankit large, UD 2.5; test changed) — **MISS** | 99.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | UAS | not measured | 80.64 [79.38, 82.02] | 87.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.05 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | UFeats | not measured | 88.20 [87.53, 88.85] | 96.11 (Trankit large, UD 2.5; test changed) — **MISS** | 95.58 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | UPOS | not measured | 90.19 [89.45, 90.92] | 94.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.16 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | Words | not measured | 96.96 [96.51, 97.36] | 99.89 (Trankit large, UD 2.5; test changed) — **MISS** | 99.64 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | id: UD_Indonesian-GSD | raw | XPOS | not measured | 80.07 [79.20, 80.92] | 95.06 (Trankit large, UD 2.5; test changed) — **MISS** | 94.69 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | is: UD_Icelandic-IcePaHC | gold | AllTags | not measured | 66.58 [66.30, 66.85] | 87.48 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | gold | CLAS | not measured | 60.44 [59.91, 60.99] | not published | not published | | is: UD_Icelandic-IcePaHC | gold | LAS | not measured | 65.74 [65.29, 66.19] | 83.82 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | gold | Lemmas | not measured | 92.47 [92.31, 92.64] | 96.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | gold | MLAS | not measured | 34.68 [34.22, 35.16] | 67.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | gold | UAS | not measured | 71.65 [71.19, 72.11] | 87.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | gold | UFeats | not measured | 74.61 [74.33, 74.88] | 92.24 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | gold | UPOS | not measured | 92.14 [91.99, 92.32] | 97.08 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | gold | XPOS | not measured | 82.13 [81.89, 82.37] | 93.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | is: UD_Icelandic-IcePaHC | raw | AllTags | not measured | 66.85 [66.56, 67.12] | 87.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 85.48 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | CLAS | not measured | 65.66 [65.14, 66.17] | 77.27 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 77.27 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | LAS | not measured | 71.34 [70.88, 71.81] | 83.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 82.85 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | Lemmas | not measured | 92.34 [92.17, 92.52] | 96.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.78 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | MLAS | not measured | 38.66 [38.17, 39.16] | 66.86 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 64.78 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | Sentences | not measured | 89.93 [89.02, 90.80] | 93.07 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 93.07 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | Tokens | not measured | 99.76 [99.69, 99.81] | 99.90 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.90 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | UAS | not measured | 77.49 [77.06, 77.95] | 87.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 86.79 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | UFeats | not measured | 74.75 [74.44, 75.02] | 92.10 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 90.42 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | UPOS | not measured | 91.85 [91.68, 92.02] | 96.92 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.36 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | Words | not measured | 99.63 [99.56, 99.69] | 99.84 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.84 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | is: UD_Icelandic-IcePaHC | raw | XPOS | not measured | 82.26 [82.02, 82.51] | 93.34 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 92.27 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | it: UD_Italian-MarkIT | gold | AllTags | 89.51 [88.83, 90.21] | 89.79 [89.12, 90.43] | 92.60 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | gold | CLAS | not measured | 72.13 [70.70, 73.64] | not published | not published | | it: UD_Italian-MarkIT | gold | LAS | 77.27 [76.19, 78.38] | 77.63 [76.52, 78.78] | 85.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | gold | Lemmas | 86.37 [85.60, 87.06] | 86.38 [85.63, 87.08] | 88.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | gold | MLAS | not measured | 57.58 [56.03, 59.25] | 70.84 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | gold | UAS | 81.82 [80.77, 82.86] | 82.05 [81.01, 83.15] | 89.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | gold | UFeats | 91.91 [91.27, 92.50] | 92.12 [91.51, 92.73] | 94.28 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | gold | UPOS | 95.69 [95.26, 96.12] | 95.81 [95.39, 96.26] | 97.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | gold | XPOS | 96.31 [95.90, 96.68] | 96.26 [95.87, 96.66] | 97.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | it: UD_Italian-MarkIT | raw | AllTags | 90.13 [89.50, 90.80] | 90.50 [89.88, 91.10] | 92.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 92.34 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | it: UD_Italian-MarkIT | raw | CLAS | not measured | 77.84 [76.30, 79.29] | 74.48 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 74.48 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | it: UD_Italian-MarkIT | raw | LAS | 81.12 [79.93, 82.18] | 82.22 [81.04, 83.28] | 84.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.63 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | it: UD_Italian-MarkIT | raw | Lemmas | 86.46 [85.69, 87.17] | 86.65 [85.93, 87.35] | 88.31 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.08 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | it: UD_Italian-MarkIT | raw | MLAS | not measured | 64.34 [62.76, 65.86] | 70.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 65.35 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | it: UD_Italian-MarkIT | raw | Sentences | 90.61 [87.84, 93.57] | 90.11 [87.27, 92.93] | 99.12 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.12 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | it: UD_Italian-MarkIT | raw | Tokens | 99.93 [99.86, 99.98] | 99.93 [99.86, 99.98] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | it: UD_Italian-MarkIT | raw | UAS | 85.40 [84.26, 86.44] | 86.39 [85.36, 87.29] | 88.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 85.58 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | it: UD_Italian-MarkIT | raw | UFeats | 92.40 [91.81, 93.01] | 92.84 [92.29, 93.42] | 94.28 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 94.28 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | it: UD_Italian-MarkIT | raw | UPOS | 95.98 [95.54, 96.36] | 96.12 [95.75, 96.52] | 97.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.29 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | it: UD_Italian-MarkIT | raw | Words | 99.73 [99.58, 99.86] | 99.73 [99.58, 99.86] | 99.67 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.67 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | it: UD_Italian-MarkIT | raw | XPOS | 96.56 [96.18, 96.92] | 96.45 [96.05, 96.80] | 97.03 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.34 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | ja: UD_Japanese-GSD | gold | AllTags | not measured | 55.45 [54.50, 56.44] | 97.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | gold | CLAS | not measured | 48.32 [46.78, 49.96] | not published | not published | | ja: UD_Japanese-GSD | gold | LAS | not measured | 52.94 [51.62, 54.32] | 94.03 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | gold | Lemmas | not measured | 88.79 [88.18, 89.43] | 98.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | gold | MLAS | not measured | 31.47 [30.14, 32.98] | 88.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | gold | UAS | not measured | 60.33 [59.00, 61.70] | 94.94 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | gold | UFeats | not measured | 99.24 [99.08, 99.39] | 99.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | gold | UPOS | not measured | 77.54 [76.68, 78.44] | 98.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | gold | XPOS | not measured | 60.54 [59.69, 61.44] | 97.69 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ja: UD_Japanese-GSD | raw | AllTags | not measured | 52.40 [51.39, 53.38] | 95.45 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 95.45 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | CLAS | not measured | 52.15 [50.37, 53.93] | 84.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 84.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | LAS | not measured | 56.76 [55.25, 58.32] | 89.29 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 89.29 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | Lemmas | not measured | 87.06 [86.44, 87.73] | 96.02 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 96.02 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | MLAS | not measured | 37.05 [35.26, 38.80] | 83.86 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 83.86 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | Sentences | not measured | 75.90 [72.12, 79.31] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.72 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | Tokens | not measured | 96.34 [95.91, 96.79] | 97.37 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 97.37 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | UAS | not measured | 62.62 [61.04, 64.19] | 90.01 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 90.01 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | UFeats | not measured | 95.62 [95.16, 96.08] | 97.36 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 97.36 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | UPOS | not measured | 75.11 [74.17, 76.05] | 96.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 96.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | Words | not measured | 96.34 [95.91, 96.79] | 97.37 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 97.37 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ja: UD_Japanese-GSD | raw | XPOS | not measured | 58.15 [57.31, 59.00] | 95.72 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 95.72 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ka: UD_Georgian-GLC | gold | AllTags | not measured | 71.95 [70.53, 73.31] | 92.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | gold | CLAS | not measured | 74.32 [72.92, 75.66] | not published | not published | | ka: UD_Georgian-GLC | gold | LAS | not measured | 75.62 [74.36, 76.88] | 81.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | gold | Lemmas | not measured | 86.00 [85.23, 86.81] | 94.86 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | gold | MLAS | not measured | 47.86 [46.15, 49.53] | 71.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | gold | UAS | not measured | 80.80 [79.59, 82.00] | 85.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | gold | UFeats | not measured | 75.66 [74.50, 76.73] | 92.52 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | gold | UPOS | not measured | 92.83 [92.27, 93.36] | 97.19 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | gold | XPOS | not measured | 91.99 [91.29, 92.73] | 97.08 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | AllTags | not measured | 62.53 [60.91, 64.07] | 91.43 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | CLAS | not measured | 58.39 [56.54, 60.36] | not published | not published | | ka: UD_Georgian-GLC | raw | LAS | not measured | 57.09 [55.21, 59.04] | 79.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | Lemmas | not measured | 75.84 [74.68, 76.94] | 94.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | MLAS | not measured | 38.45 [36.66, 40.33] | 70.43 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | Sentences | not measured | 87.88 [85.15, 90.48] | 95.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | Tokens | not measured | 99.11 [98.75, 99.40] | not published | not published | | ka: UD_Georgian-GLC | raw | UAS | not measured | 61.15 [59.26, 63.07] | 83.58 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | UFeats | not measured | 65.88 [64.51, 67.20] | 91.80 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | UPOS | not measured | 81.28 [80.19, 82.37] | 96.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | Words | not measured | 87.47 [86.58, 88.31] | 99.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ka: UD_Georgian-GLC | raw | XPOS | not measured | 80.93 [79.77, 82.01] | 96.34 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | kk: UD_Kazakh-KTB | gold | AllTags | not measured | 24.52 [23.64, 25.46] | not published | not published | | kk: UD_Kazakh-KTB | gold | CLAS | not measured | 45.66 [44.25, 47.10] | not published | not published | | kk: UD_Kazakh-KTB | gold | LAS | not measured | 49.32 [47.99, 50.61] | not published | not published | | kk: UD_Kazakh-KTB | gold | Lemmas | not measured | 58.91 [57.97, 59.84] | not published | not published | | kk: UD_Kazakh-KTB | gold | MLAS | not measured | 10.68 [9.94, 11.46] | not published | not published | | kk: UD_Kazakh-KTB | gold | UAS | not measured | 67.55 [66.13, 68.93] | not published | not published | | kk: UD_Kazakh-KTB | gold | UFeats | not measured | 46.86 [45.91, 47.77] | not published | not published | | kk: UD_Kazakh-KTB | gold | UPOS | not measured | 50.89 [49.90, 51.85] | not published | not published | | kk: UD_Kazakh-KTB | gold | XPOS | not measured | 58.02 [56.89, 59.15] | not published | not published | | kk: UD_Kazakh-KTB | raw | AllTags | not measured | 32.17 [31.28, 33.00] | 74.50 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 74.50 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | CLAS | not measured | 32.15 [30.90, 33.51] | 49.45 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 49.45 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | LAS | not measured | 33.79 [32.50, 35.13] | 51.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 51.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | Lemmas | not measured | 56.21 [55.20, 57.18] | 89.21 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 89.21 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | MLAS | not measured | 8.37 [7.67, 9.03] | 38.40 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 38.40 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | Sentences | not measured | 18.15 [15.63, 20.76] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | Tokens | not measured | 97.45 [96.91, 97.93] | 99.42 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.42 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | UAS | not measured | 45.81 [44.34, 47.37] | 60.55 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 60.55 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | UFeats | not measured | 45.88 [44.91, 46.76] | 78.70 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 78.70 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | UPOS | not measured | 53.41 [52.43, 54.35] | 85.58 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 85.58 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kk: UD_Kazakh-KTB | raw | Words | not measured | 96.99 [96.44, 97.50] | 96.85 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 96.85 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | kk: UD_Kazakh-KTB | raw | XPOS | not measured | 63.99 [62.97, 65.01] | 86.15 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 86.15 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | kmr: UD_Northern_Kurdish-Kurmanji | gold | AllTags | not measured | 13.72 [13.02, 14.37] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | CLAS | not measured | 44.92 [43.24, 46.64] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | LAS | not measured | 49.91 [48.41, 51.34] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | Lemmas | not measured | 62.32 [61.26, 63.37] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | MLAS | not measured | 4.98 [4.42, 5.54] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | UAS | not measured | 63.32 [61.80, 64.85] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | UFeats | not measured | 42.46 [41.64, 43.30] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | UPOS | not measured | 49.90 [48.87, 50.85] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | gold | XPOS | not measured | 45.00 [44.06, 45.88] | not published | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | AllTags | not measured | 21.91 [21.19, 22.61] | 52.27 (Trankit large, UD 2.5; test same size) — **MISS** | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | CLAS | not measured | 41.35 [39.52, 43.23] | 25.57 (Trankit large, UD 2.5; test same size) — within or ahead | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | LAS | not measured | 46.20 [44.40, 47.97] | 29.89 (Trankit large, UD 2.5; test same size) — within or ahead | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | Lemmas | not measured | 63.13 [62.13, 64.19] | 57.63 (Trankit large, UD 2.5; test same size) — within or ahead | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | MLAS | not measured | 5.53 [4.93, 6.12] | 9.04 (Trankit large, UD 2.5; test same size) — **MISS** | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | Sentences | not measured | 47.71 [43.89, 51.46] | 91.50 (Trankit large, UD 2.5; test same size) — **MISS** | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | Tokens | not measured | 96.59 [96.08, 97.08] | 94.95 (Trankit large, UD 2.5; test same size) — within or ahead | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | UAS | not measured | 55.20 [53.33, 57.16] | 37.12 (Trankit large, UD 2.5; test same size) — within or ahead | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | UFeats | not measured | 43.71 [42.81, 44.58] | 57.15 (Trankit large, UD 2.5; test same size) — **MISS** | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | UPOS | not measured | 53.90 [52.84, 54.98] | 75.07 (Trankit large, UD 2.5; test same size) — **MISS** | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | Words | not measured | 96.09 [95.57, 96.57] | 94.63 (Trankit large, UD 2.5; test same size) — within or ahead | not published | | kmr: UD_Northern_Kurdish-Kurmanji | raw | XPOS | not measured | 53.43 [52.40, 54.38] | 74.16 (Trankit large, UD 2.5; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | gold | AllTags | not measured | 50.87 [50.24, 51.48] | 87.60 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | gold | CLAS | not measured | 64.84 [64.16, 65.57] | not published | not published | | ko: UD_Korean-Kaist | gold | LAS | not measured | 68.13 [67.42, 68.83] | 86.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | gold | Lemmas | not measured | 82.01 [81.55, 82.51] | 94.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | gold | MLAS | not measured | 54.48 [53.71, 55.22] | 82.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | gold | UAS | not measured | 76.81 [76.13, 77.49] | 88.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | gold | UPOS | not measured | 83.77 [83.31, 84.20] | 96.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | gold | XPOS | not measured | 53.61 [53.00, 54.24] | 87.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ko: UD_Korean-Kaist | raw | AllTags | not measured | 55.15 [54.56, 55.79] | 88.07 (Trankit large, UD 2.5; test same size) — **MISS** | 85.81 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | CLAS | not measured | 72.10 [71.40, 72.85] | 86.37 (Trankit large, UD 2.5; test same size) — **MISS** | 83.65 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | LAS | not measured | 74.17 [73.47, 74.90] | 88.22 (Trankit large, UD 2.5; test same size) — **MISS** | 85.50 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | Lemmas | not measured | 82.88 [82.41, 83.38] | 94.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.10 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | MLAS | not measured | 62.12 [61.35, 62.90] | 83.56 (Trankit large, UD 2.5; test same size) — **MISS** | 80.93 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | Sentences | not measured | 92.51 [91.52, 93.49] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.93 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | Tokens | not measured | 99.90 [99.84, 99.95] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | UAS | not measured | 80.66 [79.98, 81.38] | 90.00 (Trankit large, UD 2.5; test same size) — **MISS** | 87.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | UPOS | not measured | 85.53 [85.09, 85.97] | 96.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.74 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | Words | not measured | 99.90 [99.84, 99.95] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ko: UD_Korean-Kaist | raw | XPOS | not measured | 57.88 [57.27, 58.51] | 88.07 (Trankit large, UD 2.5; test same size) — **MISS** | 85.81 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | koi: UD_Komi_Permyak-UH | gold | AllTags | not measured | 17.39 [15.54, 19.26] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | CLAS | not measured | 23.79 [19.78, 28.52] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | LAS | not measured | 30.79 [26.88, 35.53] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | Lemmas | not measured | 60.58 [56.73, 63.70] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | MLAS | not measured | 4.98 [3.51, 6.81] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | UAS | not measured | 50.61 [45.71, 56.22] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | UFeats | not measured | 44.96 [41.69, 47.92] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | UPOS | not measured | 54.71 [51.54, 57.78] | not published | not published | | koi: UD_Komi_Permyak-UH | gold | XPOS | not measured | 17.39 [15.54, 19.26] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | AllTags | not measured | 17.39 [15.54, 19.26] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | CLAS | not measured | 27.03 [22.60, 32.05] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | LAS | not measured | 33.78 [29.41, 38.81] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | Lemmas | not measured | 61.35 [57.52, 64.60] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | MLAS | not measured | 4.10 [2.64, 5.81] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | Sentences | not measured | 75.38 [66.34, 82.90] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | UAS | not measured | 54.60 [49.09, 60.45] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | UFeats | not measured | 45.63 [42.91, 48.31] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | UPOS | not measured | 50.50 [47.91, 52.99] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | Words | not measured | 100.00 [100.00, 100.00] | not published | not published | | koi: UD_Komi_Permyak-UH | raw | XPOS | not measured | 17.39 [15.54, 19.26] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | AllTags | not measured | 6.93 [6.30, 7.55] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | CLAS | not measured | 20.80 [19.60, 22.09] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | LAS | not measured | 25.75 [24.50, 27.15] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | Lemmas | not measured | 60.89 [59.87, 61.84] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | MLAS | not measured | 2.24 [1.85, 2.67] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | UAS | not measured | 45.10 [43.31, 46.85] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | UFeats | not measured | 12.17 [11.45, 12.95] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | UPOS | not measured | 26.84 [25.76, 27.89] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | gold | XPOS | not measured | 32.46 [31.47, 33.42] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | AllTags | not measured | 12.96 [12.23, 13.72] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | CLAS | not measured | 9.99 [9.02, 11.00] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | LAS | not measured | 12.68 [11.68, 13.80] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | Lemmas | not measured | 60.23 [59.20, 61.18] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | MLAS | not measured | 0.89 [0.64, 1.16] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | Sentences | not measured | 1.45 [0.47, 2.66] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | Tokens | not measured | 99.91 [99.82, 99.98] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | UAS | not measured | 22.17 [20.60, 23.81] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | UFeats | not measured | 20.18 [19.30, 21.14] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | UPOS | not measured | 36.32 [35.14, 37.42] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | Words | not measured | 99.24 [98.96, 99.51] | not published | not published | | kpv: UD_Komi_Zyrian-Lattice | raw | XPOS | not measured | 37.42 [36.41, 38.43] | not published | not published | | ky: UD_Kyrgyz-KTMU | gold | AllTags | not measured | 61.92 [60.81, 63.04] | 73.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | gold | CLAS | not measured | 61.76 [60.54, 62.98] | not published | not published | | ky: UD_Kyrgyz-KTMU | gold | LAS | not measured | 65.35 [64.24, 66.42] | 74.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | gold | Lemmas | not measured | 80.51 [79.69, 81.38] | 89.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | gold | MLAS | not measured | 38.51 [37.17, 39.79] | 54.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | gold | UAS | not measured | 78.82 [77.89, 79.75] | 84.78 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | gold | UFeats | not measured | 66.43 [65.37, 67.55] | 77.98 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | gold | UPOS | not measured | 88.87 [88.18, 89.53] | 91.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | gold | XPOS | not measured | 88.72 [88.07, 89.39] | 91.06 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ky: UD_Kyrgyz-KTMU | raw | AllTags | not measured | 61.82 [60.78, 62.86] | 72.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 57.91 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ky: UD_Kyrgyz-KTMU | raw | CLAS | not measured | 64.75 [63.38, 66.06] | 52.60 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 52.60 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ky: UD_Kyrgyz-KTMU | raw | LAS | not measured | 67.79 [66.50, 68.97] | 73.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 58.77 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ky: UD_Kyrgyz-KTMU | raw | Lemmas | not measured | 79.55 [78.75, 80.43] | 88.71 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 88.09 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ky: UD_Kyrgyz-KTMU | raw | MLAS | not measured | 40.79 [39.41, 42.11] | 54.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 28.47 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ky: UD_Kyrgyz-KTMU | raw | Sentences | not measured | 92.70 [91.28, 94.09] | 98.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 98.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ky: UD_Kyrgyz-KTMU | raw | Tokens | not measured | 97.59 [97.23, 97.93] | 99.06 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.06 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ky: UD_Kyrgyz-KTMU | raw | UAS | not measured | 79.63 [78.60, 80.52] | 84.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 74.95 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ky: UD_Kyrgyz-KTMU | raw | UFeats | not measured | 66.26 [65.21, 67.25] | 77.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 60.22 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ky: UD_Kyrgyz-KTMU | raw | UPOS | not measured | 87.18 [86.43, 87.88] | 90.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.65 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | ky: UD_Kyrgyz-KTMU | raw | Words | not measured | 97.59 [97.23, 97.93] | 99.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.06 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ky: UD_Kyrgyz-KTMU | raw | XPOS | not measured | 87.55 [86.88, 88.18] | 90.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.94 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | la: UD_Latin-LLCT | gold | AllTags | not measured | 70.96 [70.03, 71.97] | 96.84 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | gold | CLAS | not measured | 77.30 [75.92, 78.81] | not published | not published | | la: UD_Latin-LLCT | gold | LAS | not measured | 80.93 [79.78, 82.10] | 94.07 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | gold | Lemmas | not measured | 95.42 [94.99, 95.85] | 97.80 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | gold | MLAS | not measured | 51.35 [49.65, 53.15] | 88.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | gold | UAS | not measured | 83.95 [82.93, 84.98] | 95.05 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | gold | UFeats | not measured | 75.63 [74.78, 76.57] | 97.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | gold | UPOS | not measured | 96.38 [96.09, 96.67] | 99.78 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | gold | XPOS | not measured | 76.59 [75.69, 77.53] | 97.05 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | la: UD_Latin-LLCT | raw | AllTags | not measured | 72.20 [71.18, 73.34] | 96.83 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.47 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | CLAS | not measured | 79.91 [78.42, 81.41] | 93.53 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 93.53 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | LAS | not measured | 83.30 [82.10, 84.52] | 94.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 94.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | Lemmas | not measured | 95.64 [95.22, 96.06] | 98.09 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 98.09 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | MLAS | not measured | 53.58 [51.84, 55.42] | 89.49 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 89.49 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | Sentences | not measured | 88.69 [86.82, 90.62] | 99.49 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.32 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | Tokens | not measured | 99.73 [99.62, 99.82] | 99.99 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | UAS | not measured | 86.01 [84.90, 87.13] | 96.13 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 96.13 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | UFeats | not measured | 75.55 [74.62, 76.55] | 97.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.79 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | UPOS | not measured | 96.72 [96.43, 97.01] | 99.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.57 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | Words | not measured | 99.71 [99.60, 99.81] | 99.99 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | la: UD_Latin-LLCT | raw | XPOS | not measured | 77.82 [76.97, 78.83] | 97.05 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.65 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lij: UD_Ligurian-GLT | gold | CLAS | not measured | 25.63 [23.69, 27.63] | not published | not published | | lij: UD_Ligurian-GLT | gold | LAS | not measured | 36.41 [34.59, 38.31] | not published | not published | | lij: UD_Ligurian-GLT | gold | Lemmas | not measured | 72.29 [70.97, 73.49] | not published | not published | | lij: UD_Ligurian-GLT | gold | MLAS | not measured | 4.68 [3.83, 5.43] | not published | not published | | lij: UD_Ligurian-GLT | gold | UAS | not measured | 54.80 [52.79, 56.88] | not published | not published | | lij: UD_Ligurian-GLT | gold | UFeats | not measured | 45.14 [43.85, 46.38] | not published | not published | | lij: UD_Ligurian-GLT | gold | UPOS | not measured | 56.57 [55.42, 57.72] | not published | not published | | lij: UD_Ligurian-GLT | raw | CLAS | not measured | 19.82 [18.05, 21.74] | 60.33 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 60.33 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | LAS | not measured | 25.76 [24.05, 27.47] | 69.18 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 69.18 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | Lemmas | not measured | 61.84 [60.48, 63.20] | 92.68 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 92.68 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | MLAS | not measured | 3.31 [2.61, 3.97] | 51.24 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 51.24 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | Sentences | not measured | 34.74 [28.95, 40.53] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | Tokens | not measured | 89.31 [88.32, 90.26] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | UAS | not measured | 37.84 [35.73, 39.98] | 77.61 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 77.61 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | UFeats | not measured | 38.11 [36.98, 39.19] | 88.25 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 88.25 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | UPOS | not measured | 47.86 [46.61, 49.07] | 89.14 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 89.14 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lij: UD_Ligurian-GLT | raw | Words | not measured | 81.74 [80.53, 83.00] | 98.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 98.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | gold | AllTags | not measured | 67.88 [66.97, 68.88] | 89.46 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | gold | CLAS | not measured | 69.77 [68.26, 71.06] | not published | not published | | lt: UD_Lithuanian-ALKSNIS | gold | LAS | not measured | 72.74 [71.48, 73.89] | 79.92 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | gold | Lemmas | not measured | 80.46 [79.60, 81.34] | 93.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | gold | MLAS | not measured | 41.85 [40.64, 43.11] | 68.90 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | gold | UAS | not measured | 78.38 [77.13, 79.56] | 83.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | gold | UFeats | not measured | 72.11 [71.26, 73.06] | 91.19 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | gold | UPOS | not measured | 88.95 [88.27, 89.65] | 96.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | gold | XPOS | not measured | 72.57 [71.68, 73.49] | 90.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lt: UD_Lithuanian-ALKSNIS | raw | AllTags | not measured | 69.48 [68.60, 70.43] | 93.30 (Trankit large, UD 2.5; test same size) — **MISS** | 85.54 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | CLAS | not measured | 74.53 [73.23, 75.77] | 86.66 (Trankit large, UD 2.5; test same size) — **MISS** | 70.23 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lt: UD_Lithuanian-ALKSNIS | raw | LAS | not measured | 76.39 [75.20, 77.55] | 87.67 (Trankit large, UD 2.5; test same size) — **MISS** | 72.75 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lt: UD_Lithuanian-ALKSNIS | raw | Lemmas | not measured | 81.55 [80.75, 82.31] | 94.30 (Trankit large, UD 2.5; test same size) — **MISS** | 92.95 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | MLAS | not measured | 46.90 [45.62, 48.09] | 79.86 (Trankit large, UD 2.5; test same size) — **MISS** | 60.53 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | Sentences | not measured | 81.03 [77.95, 83.57] | 95.72 (Trankit large, UD 2.5; test same size) — **MISS** | 89.93 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | Tokens | not measured | 99.71 [99.57, 99.82] | 99.94 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.94 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | UAS | not measured | 81.67 [80.47, 82.76] | 90.48 (Trankit large, UD 2.5; test same size) — **MISS** | 77.82 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lt: UD_Lithuanian-ALKSNIS | raw | UFeats | not measured | 73.62 [72.76, 74.51] | 94.46 (Trankit large, UD 2.5; test same size) — **MISS** | 88.16 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | UPOS | not measured | 89.59 [88.93, 90.24] | 97.45 (Trankit large, UD 2.5; test same size) — **MISS** | 93.71 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | Words | not measured | 99.71 [99.57, 99.82] | 99.94 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.94 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lt: UD_Lithuanian-ALKSNIS | raw | XPOS | not measured | 74.08 [73.23, 74.96] | 93.98 (Trankit large, UD 2.5; test same size) — **MISS** | 86.48 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lv: UD_Latvian-LVTB | gold | AllTags | not measured | 62.76 [62.26, 63.23] | 92.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | gold | CLAS | not measured | 74.70 [73.92, 75.41] | not published | not published | | lv: UD_Latvian-LVTB | gold | LAS | not measured | 78.42 [77.72, 79.07] | 87.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | gold | Lemmas | not measured | 90.81 [90.48, 91.14] | 97.48 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | gold | MLAS | not measured | 50.21 [49.42, 51.02] | 80.23 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | gold | UAS | not measured | 83.67 [83.00, 84.27] | 90.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | gold | UFeats | not measured | 78.48 [78.06, 78.91] | 95.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | gold | UPOS | not measured | 92.03 [91.73, 92.32] | 98.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | gold | XPOS | not measured | 65.93 [65.43, 66.42] | 92.69 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lv: UD_Latvian-LVTB | raw | AllTags | not measured | 64.02 [63.51, 64.51] | 91.60 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.29 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | CLAS | not measured | 79.91 [79.15, 80.62] | 89.78 (Trankit large, UD 2.5; test changed) — **MISS** | 83.37 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | LAS | not measured | 82.85 [82.18, 83.48] | 91.25 (Trankit large, UD 2.5; test changed) — **MISS** | 85.77 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | Lemmas | not measured | 90.83 [90.49, 91.16] | 96.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.12 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | MLAS | not measured | 56.14 [55.35, 56.98] | 82.69 (Trankit large, UD 2.5; test changed) — **MISS** | 76.71 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | Sentences | not measured | 94.39 [93.46, 95.24] | 99.10 (Trankit base, UD 2.5; test changed) — **MISS** | 98.15 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | Tokens | not measured | 98.79 [98.60, 98.97] | 99.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | UAS | not measured | 87.56 [86.95, 88.15] | 93.63 (Trankit large, UD 2.5; test changed) — **MISS** | 88.91 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | UFeats | not measured | 79.94 [79.53, 80.40] | 95.28 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | UPOS | not measured | 91.98 [91.63, 92.29] | 97.61 (Trankit large, UD 2.5; test changed) — **MISS** | 96.70 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | Words | not measured | 98.79 [98.60, 98.97] | 99.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.81 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lv: UD_Latvian-LVTB | raw | XPOS | not measured | 66.51 [65.99, 66.99] | 92.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.72 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | lzh: UD_Classical_Chinese-Kyoto | gold | AllTags | not measured | 54.93 [54.32, 55.53] | 89.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | CLAS | not measured | 63.68 [62.84, 64.54] | not published | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | LAS | not measured | 63.79 [62.96, 64.62] | 79.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | Lemmas | not measured | 98.51 [98.36, 98.66] | 99.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | MLAS | not measured | 53.02 [52.23, 53.89] | 75.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | UAS | not measured | 72.31 [71.47, 73.16] | 84.45 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | UFeats | not measured | 85.19 [84.71, 85.62] | 94.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | UPOS | not measured | 84.48 [84.01, 84.93] | 93.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | gold | XPOS | not measured | 58.82 [58.24, 59.41] | 92.07 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | lzh: UD_Classical_Chinese-Kyoto | raw | AllTags | not measured | 54.83 [54.26, 55.39] | 89.98 (Trankit large, UD 2.5; test changed) — **MISS** | 85.89 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lzh: UD_Classical_Chinese-Kyoto | raw | CLAS | not measured | 63.95 [63.12, 64.79] | 75.60 (Trankit large, UD 2.5; test changed) — **MISS** | 64.54 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lzh: UD_Classical_Chinese-Kyoto | raw | LAS | not measured | 64.12 [63.30, 64.96] | 76.82 (Trankit large, UD 2.5; test changed) — **MISS** | 65.80 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lzh: UD_Classical_Chinese-Kyoto | raw | Lemmas | not measured | 96.66 [96.40, 96.91] | 99.67 (Trankit large, UD 2.5; test changed) — **MISS** | 97.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lzh: UD_Classical_Chinese-Kyoto | raw | MLAS | not measured | 54.27 [53.49, 55.12] | 73.16 (Trankit large, UD 2.5; test changed) — **MISS** | 61.94 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lzh: UD_Classical_Chinese-Kyoto | raw | Sentences | not measured | 94.40 [93.81, 94.96] | 70.58 (Trankit large, UD 2.5; test changed) — within or ahead | 49.57 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lzh: UD_Classical_Chinese-Kyoto | raw | Tokens | not measured | 97.90 [97.67, 98.11] | 99.70 (Trankit large, UD 2.5; test changed) — **MISS** | 97.85 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lzh: UD_Classical_Chinese-Kyoto | raw | UAS | not measured | 71.96 [71.14, 72.75] | 82.03 (Trankit large, UD 2.5; test changed) — **MISS** | 71.32 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lzh: UD_Classical_Chinese-Kyoto | raw | UFeats | not measured | 84.25 [83.79, 84.68] | 94.43 (Trankit large, UD 2.5; test changed) — **MISS** | 91.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lzh: UD_Classical_Chinese-Kyoto | raw | UPOS | not measured | 84.19 [83.73, 84.68] | 92.89 (Trankit large, UD 2.5; test changed) — **MISS** | 89.16 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | lzh: UD_Classical_Chinese-Kyoto | raw | Words | not measured | 97.90 [97.67, 98.11] | 99.70 (Trankit large, UD 2.5; test changed) — **MISS** | 97.85 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | lzh: UD_Classical_Chinese-Kyoto | raw | XPOS | not measured | 58.63 [58.06, 59.17] | 91.90 (Trankit large, UD 2.5; test changed) — **MISS** | 88.17 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mdf: UD_Moksha-JR | gold | AllTags | not measured | 12.50 [11.26, 13.69] | not published | not published | | mdf: UD_Moksha-JR | gold | CLAS | not measured | 19.82 [18.27, 21.49] | not published | not published | | mdf: UD_Moksha-JR | gold | LAS | not measured | 25.61 [23.95, 27.43] | not published | not published | | mdf: UD_Moksha-JR | gold | Lemmas | not measured | 54.49 [53.07, 55.74] | not published | not published | | mdf: UD_Moksha-JR | gold | MLAS | not measured | 3.51 [2.85, 4.21] | not published | not published | | mdf: UD_Moksha-JR | gold | UAS | not measured | 46.61 [44.25, 49.02] | not published | not published | | mdf: UD_Moksha-JR | gold | UFeats | not measured | 33.08 [31.94, 34.17] | not published | not published | | mdf: UD_Moksha-JR | gold | UPOS | not measured | 48.38 [47.03, 49.73] | not published | not published | | mdf: UD_Moksha-JR | gold | XPOS | not measured | 33.10 [31.59, 34.54] | not published | not published | | mdf: UD_Moksha-JR | raw | AllTags | not measured | 21.77 [20.35, 23.14] | not published | not published | | mdf: UD_Moksha-JR | raw | CLAS | not measured | 25.35 [23.57, 27.09] | not published | not published | | mdf: UD_Moksha-JR | raw | LAS | not measured | 30.66 [28.78, 32.54] | not published | not published | | mdf: UD_Moksha-JR | raw | Lemmas | not measured | 55.83 [54.40, 57.02] | not published | not published | | mdf: UD_Moksha-JR | raw | MLAS | not measured | 5.54 [4.72, 6.43] | not published | not published | | mdf: UD_Moksha-JR | raw | Sentences | not measured | 51.73 [46.72, 57.11] | not published | not published | | mdf: UD_Moksha-JR | raw | Tokens | not measured | 99.93 [99.82, 100.00] | not published | not published | | mdf: UD_Moksha-JR | raw | UAS | not measured | 51.48 [49.02, 53.81] | not published | not published | | mdf: UD_Moksha-JR | raw | UFeats | not measured | 38.54 [37.34, 39.79] | not published | not published | | mdf: UD_Moksha-JR | raw | UPOS | not measured | 52.79 [51.42, 54.27] | not published | not published | | mdf: UD_Moksha-JR | raw | Words | not measured | 99.79 [99.58, 99.95] | not published | not published | | mdf: UD_Moksha-JR | raw | XPOS | not measured | 41.83 [40.35, 43.45] | not published | not published | | mr: UD_Marathi-UFAL | gold | CLAS | not measured | 57.49 [49.18, 65.92] | not published | not published | | mr: UD_Marathi-UFAL | gold | LAS | not measured | 63.59 [55.41, 71.76] | 64.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | mr: UD_Marathi-UFAL | gold | Lemmas | not measured | 73.30 [69.70, 77.23] | 87.86 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mr: UD_Marathi-UFAL | gold | MLAS | not measured | 4.93 [2.60, 7.48] | 40.50 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mr: UD_Marathi-UFAL | gold | UAS | not measured | 74.76 [66.75, 82.95] | 73.06 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | mr: UD_Marathi-UFAL | gold | UFeats | not measured | 38.59 [34.34, 42.44] | 76.94 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mr: UD_Marathi-UFAL | gold | UPOS | not measured | 80.58 [76.30, 84.38] | 88.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mr: UD_Marathi-UFAL | raw | CLAS | not measured | 47.72 [40.09, 55.63] | 59.67 (Trankit large, UD 2.5; test same size) — **MISS** | 54.73 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | mr: UD_Marathi-UFAL | raw | LAS | not measured | 51.52 [42.56, 60.16] | 63.36 (Trankit large, UD 2.5; test same size) — **MISS** | 59.56 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | mr: UD_Marathi-UFAL | raw | Lemmas | not measured | 69.54 [65.28, 74.07] | 84.67 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 82.84 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mr: UD_Marathi-UFAL | raw | MLAS | not measured | 6.22 [3.70, 9.22] | 38.10 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 37.04 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mr: UD_Marathi-UFAL | raw | Sentences | not measured | 83.15 [72.09, 92.31] | 92.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.36 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | mr: UD_Marathi-UFAL | raw | Tokens | not measured | 100.00 [100.00, 100.00] | 99.20 (Trankit large, UD 2.5; test same size) — within or ahead | 97.99 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | mr: UD_Marathi-UFAL | raw | UAS | not measured | 60.91 [51.46, 69.66] | 72.79 (Trankit large, UD 2.5; test same size) — **MISS** | 68.87 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | mr: UD_Marathi-UFAL | raw | UFeats | not measured | 38.32 [33.85, 43.07] | 74.45 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 73.53 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mr: UD_Marathi-UFAL | raw | UPOS | not measured | 74.11 [69.57, 78.53] | 87.79 (Trankit large, UD 2.5; test same size) — **MISS** | 85.78 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mr: UD_Marathi-UFAL | raw | Words | not measured | 86.29 [81.97, 90.71] | 97.22 (Trankit large, UD 2.5; test same size) — **MISS** | 92.89 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | gold | AllTags | not measured | 88.74 [88.04, 89.38] | 95.43 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mt: UD_Maltese-MUDT | gold | CLAS | not measured | 71.21 [69.65, 72.69] | not published | not published | | mt: UD_Maltese-MUDT | gold | LAS | not measured | 78.33 [77.05, 79.52] | 80.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mt: UD_Maltese-MUDT | gold | MLAS | not measured | 63.68 [62.10, 65.29] | 68.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mt: UD_Maltese-MUDT | gold | UAS | not measured | 83.92 [82.73, 85.04] | 85.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mt: UD_Maltese-MUDT | gold | UFeats | not measured | 100.00 [100.00, 100.00] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | mt: UD_Maltese-MUDT | gold | UPOS | not measured | 93.68 [93.10, 94.18] | 95.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mt: UD_Maltese-MUDT | gold | XPOS | not measured | 90.00 [89.37, 90.64] | 95.90 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | mt: UD_Maltese-MUDT | raw | AllTags | not measured | 65.68 [64.63, 66.79] | 95.35 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 95.35 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | CLAS | not measured | 42.42 [40.64, 44.15] | 70.76 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 70.76 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | LAS | not measured | 45.83 [44.35, 47.41] | 80.04 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 78.22 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | MLAS | not measured | 35.62 [34.05, 37.16] | 68.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 67.20 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | Sentences | not measured | 64.56 [60.41, 69.05] | 86.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 85.35 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | Tokens | not measured | 74.36 [73.25, 75.49] | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | UAS | not measured | 49.18 [47.60, 50.89] | 84.86 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 83.62 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | UFeats | not measured | 74.36 [73.25, 75.49] | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | UPOS | not measured | 69.86 [68.71, 70.99] | 95.78 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.72 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | Words | not measured | 74.36 [73.25, 75.49] | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | mt: UD_Maltese-MUDT | raw | XPOS | not measured | 66.72 [65.56, 67.86] | 95.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.61 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | myv: UD_Erzya-JR | gold | AllTags | not measured | 51.29 [50.32, 52.26] | 74.86 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | gold | CLAS | not measured | 40.87 [39.41, 42.54] | not published | not published | | myv: UD_Erzya-JR | gold | LAS | not measured | 47.25 [45.94, 48.65] | 64.70 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | gold | Lemmas | not measured | 70.08 [69.19, 70.92] | 84.87 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | gold | MLAS | not measured | 19.12 [18.03, 20.26] | 42.76 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | gold | UAS | not measured | 60.64 [59.17, 62.12] | 74.25 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | gold | UFeats | not measured | 57.01 [56.11, 57.93] | 79.93 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | gold | UPOS | not measured | 75.05 [74.17, 75.92] | 89.25 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | gold | XPOS | not measured | 76.79 [75.93, 77.59] | 88.73 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | myv: UD_Erzya-JR | raw | AllTags | not measured | 52.66 [51.72, 53.53] | 74.38 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 69.52 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | CLAS | not measured | 47.96 [46.52, 49.39] | 52.10 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 52.10 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | LAS | not measured | 54.23 [52.86, 55.57] | 63.92 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 57.35 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | Lemmas | not measured | 71.48 [70.56, 72.36] | 84.72 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 84.72 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | MLAS | not measured | 22.89 [21.77, 24.10] | 42.32 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 36.09 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | Sentences | not measured | 88.78 [86.67, 90.57] | 97.02 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 96.92 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | Tokens | not measured | 99.54 [99.36, 99.69] | 99.63 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 99.63 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | myv: UD_Erzya-JR | raw | UAS | not measured | 68.25 [66.78, 69.66] | 73.33 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 70.08 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | UFeats | not measured | 58.35 [57.40, 59.22] | 79.36 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 75.88 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | UPOS | not measured | 76.37 [75.55, 77.17] | 88.50 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 84.84 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | myv: UD_Erzya-JR | raw | Words | not measured | 98.98 [98.69, 99.22] | 99.12 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | 99.08 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | myv: UD_Erzya-JR | raw | XPOS | not measured | 78.29 [77.49, 79.07] | 88.04 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 83.29 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | nds: UD_Low_Saxon-LSDC | gold | CLAS | not measured | 53.25 [51.60, 54.90] | not published | not published | | nds: UD_Low_Saxon-LSDC | gold | LAS | not measured | 60.37 [58.92, 61.90] | 65.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | gold | Lemmas | not measured | 80.89 [79.99, 81.80] | 84.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | gold | MLAS | not measured | 24.10 [22.94, 25.34] | 36.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | gold | UAS | not measured | 69.57 [68.00, 71.18] | 75.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | gold | UFeats | not measured | 62.06 [61.10, 63.09] | 72.23 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | gold | UPOS | not measured | 83.92 [83.02, 84.85] | 90.80 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | raw | CLAS | not measured | 63.87 [62.41, 65.69] | not published | not published | | nds: UD_Low_Saxon-LSDC | raw | LAS | not measured | 69.59 [68.18, 71.11] | 64.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | nds: UD_Low_Saxon-LSDC | raw | Lemmas | not measured | 81.66 [80.81, 82.53] | 83.84 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | raw | MLAS | not measured | 30.05 [28.72, 31.52] | 36.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | raw | Sentences | not measured | 91.08 [88.59, 93.39] | 90.23 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | nds: UD_Low_Saxon-LSDC | raw | Tokens | not measured | 99.65 [99.48, 99.81] | not published | not published | | nds: UD_Low_Saxon-LSDC | raw | UAS | not measured | 77.79 [76.34, 79.26] | 73.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | nds: UD_Low_Saxon-LSDC | raw | UFeats | not measured | 62.89 [61.93, 63.94] | 71.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | raw | UPOS | not measured | 86.22 [85.42, 87.04] | 90.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nds: UD_Low_Saxon-LSDC | raw | Words | not measured | 99.41 [99.14, 99.65] | 99.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | nl: UD_Dutch-Alpino | gold | AllTags | 94.31 [93.48, 95.00] | 94.13 [93.32, 94.87] | 96.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | gold | CLAS | not measured | 82.88 [81.34, 84.21] | not published | not published | | nl: UD_Dutch-Alpino | gold | LAS | 87.63 [86.63, 88.59] | 87.24 [86.15, 88.24] | 93.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | gold | Lemmas | 91.88 [91.32, 92.47] | 92.34 [91.80, 92.92] | 96.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | gold | MLAS | not measured | 76.21 [74.34, 77.73] | 86.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | gold | UAS | 90.41 [89.40, 91.32] | 90.23 [89.24, 91.11] | 95.30 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | gold | UFeats | 96.73 [96.16, 97.22] | 96.62 [96.06, 97.12] | 97.75 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | gold | UPOS | 96.89 [96.24, 97.44] | 96.91 [96.21, 97.44] | 98.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | gold | XPOS | 95.09 [94.32, 95.77] | 95.05 [94.26, 95.72] | 97.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | nl: UD_Dutch-Alpino | raw | AllTags | 94.31 [93.58, 95.01] | 94.50 [93.79, 95.22] | 96.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.97 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | CLAS | not measured | 87.99 [86.70, 89.11] | 89.56 (Trankit large, UD 2.5; test same size) — **MISS** | 83.94 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | LAS | 89.06 [87.93, 89.96] | 91.11 [90.20, 91.92] | 92.49 (Trankit large, UD 2.5; test same size) — **MISS** | 88.67 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | Lemmas | 91.54 [90.97, 92.14] | 92.05 [91.49, 92.67] | 96.39 (Trankit large, UD 2.5; test same size) — **MISS** | 95.33 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | nl: UD_Dutch-Alpino | raw | MLAS | not measured | 82.24 [80.73, 83.63] | 85.14 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 79.65 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | Sentences | 94.81 [92.78, 96.42] | 95.90 [94.35, 97.34] | 91.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | 90.08 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | Tokens | 99.66 [99.46, 99.82] | 99.66 [99.46, 99.82] | 99.87 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.87 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | nl: UD_Dutch-Alpino | raw | UAS | 91.67 [90.65, 92.52] | 93.37 [92.55, 94.08] | 94.41 (Trankit large, UD 2.5; test same size) — **MISS** | 91.42 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | UFeats | 96.78 [96.26, 97.27] | 96.80 [96.29, 97.28] | 97.52 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.00 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | UPOS | 96.66 [96.01, 97.23] | 96.74 [96.11, 97.27] | 97.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.75 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | nl: UD_Dutch-Alpino | raw | Words | 99.66 [99.46, 99.82] | 99.66 [99.46, 99.82] | 99.87 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.87 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | nl: UD_Dutch-Alpino | raw | XPOS | 95.03 [94.32, 95.67] | 95.27 [94.61, 95.94] | 96.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.50 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | no: UD_Norwegian-Bokmaal | gold | AllTags | not measured | 79.66 [79.14, 80.14] | 97.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | gold | CLAS | not measured | 77.86 [77.09, 78.70] | not published | not published | | no: UD_Norwegian-Bokmaal | gold | LAS | not measured | 82.98 [82.32, 83.67] | 93.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | gold | Lemmas | not measured | 94.34 [94.06, 94.60] | 98.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | gold | MLAS | not measured | 54.53 [53.68, 55.36] | 87.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | gold | UAS | not measured | 86.61 [85.96, 87.26] | 94.69 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | gold | UFeats | not measured | 82.49 [82.00, 82.96] | 97.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | gold | UPOS | not measured | 95.93 [95.54, 96.29] | 98.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | gold | XPOS | not measured | 96.63 [96.25, 96.98] | 99.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | no: UD_Norwegian-Bokmaal | raw | AllTags | not measured | 79.29 [78.77, 79.79] | 97.61 (Trankit large, UD 2.5; test same size) — **MISS** | 96.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | CLAS | not measured | 85.08 [84.24, 85.81] | 92.82 (Trankit large, UD 2.5; test same size) — **MISS** | 89.11 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | LAS | not measured | 88.26 [87.58, 88.85] | 94.33 (Trankit large, UD 2.5; test same size) — **MISS** | 91.85 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | Lemmas | not measured | 93.79 [93.49, 94.10] | 98.67 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 98.61 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | MLAS | not measured | 59.86 [58.97, 60.75] | 90.15 (Trankit large, UD 2.5; test same size) — **MISS** | 85.13 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | Sentences | not measured | 92.95 [91.84, 94.03] | 99.20 (Trankit base, UD 2.5; test same size) — **MISS** | 97.70 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | Tokens | not measured | 99.50 [99.38, 99.60] | 99.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | UAS | not measured | 90.80 [90.15, 91.37] | 95.54 (Trankit large, UD 2.5; test same size) — **MISS** | 93.26 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | UFeats | not measured | 81.74 [81.25, 82.21] | 98.07 (Trankit large, UD 2.5; test same size) — **MISS** | 97.09 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | UPOS | not measured | 95.96 [95.58, 96.30] | 98.85 (Trankit large, UD 2.5; test same size) — **MISS** | 98.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | Words | not measured | 99.50 [99.38, 99.60] | 99.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | no: UD_Norwegian-Bokmaal | raw | XPOS | not measured | 96.89 [96.51, 97.22] | 99.88 (Trankit large, UD 2.5; test same size) — **MISS** | 99.04 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | olo: UD_Livvi-KKPP | gold | AllTags | not measured | 3.83 [2.79, 4.84] | not published | not published | | olo: UD_Livvi-KKPP | gold | CLAS | not measured | 24.64 [21.78, 27.57] | not published | not published | | olo: UD_Livvi-KKPP | gold | LAS | not measured | 29.37 [26.68, 32.19] | not published | not published | | olo: UD_Livvi-KKPP | gold | Lemmas | not measured | 47.98 [45.38, 50.23] | not published | not published | | olo: UD_Livvi-KKPP | gold | MLAS | not measured | 1.41 [0.71, 2.23] | not published | not published | | olo: UD_Livvi-KKPP | gold | UAS | not measured | 48.05 [44.69, 51.76] | not published | not published | | olo: UD_Livvi-KKPP | gold | UFeats | not measured | 10.35 [8.70, 11.98] | not published | not published | | olo: UD_Livvi-KKPP | gold | UPOS | not measured | 36.96 [34.50, 39.59] | not published | not published | | olo: UD_Livvi-KKPP | gold | XPOS | not measured | 35.69 [32.79, 38.82] | not published | not published | | olo: UD_Livvi-KKPP | raw | AllTags | not measured | 10.52 [9.33, 11.82] | not published | not published | | olo: UD_Livvi-KKPP | raw | CLAS | not measured | 29.94 [26.69, 33.16] | not published | not published | | olo: UD_Livvi-KKPP | raw | LAS | not measured | 32.98 [29.67, 36.26] | not published | not published | | olo: UD_Livvi-KKPP | raw | Lemmas | not measured | 46.98 [44.31, 49.50] | not published | not published | | olo: UD_Livvi-KKPP | raw | MLAS | not measured | 1.92 [1.11, 2.81] | not published | not published | | olo: UD_Livvi-KKPP | raw | Sentences | not measured | 26.94 [19.19, 35.54] | not published | not published | | olo: UD_Livvi-KKPP | raw | Tokens | not measured | 98.86 [98.15, 99.49] | not published | not published | | olo: UD_Livvi-KKPP | raw | UAS | not measured | 51.34 [47.43, 55.19] | not published | not published | | olo: UD_Livvi-KKPP | raw | UFeats | not measured | 14.75 [13.25, 16.50] | not published | not published | | olo: UD_Livvi-KKPP | raw | UPOS | not measured | 44.37 [41.50, 47.23] | not published | not published | | olo: UD_Livvi-KKPP | raw | Words | not measured | 98.86 [98.15, 99.49] | not published | not published | | olo: UD_Livvi-KKPP | raw | XPOS | not measured | 42.23 [39.39, 45.42] | not published | not published | | orv: UD_Old_East_Slavic-RNC | gold | AllTags | not measured | 0.18 [0.13, 0.23] | 82.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | gold | CLAS | not measured | 59.04 [57.82, 60.34] | not published | not published | | orv: UD_Old_East_Slavic-RNC | gold | LAS | not measured | 66.62 [65.53, 67.76] | 78.58 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | gold | Lemmas | not measured | 79.36 [78.68, 80.01] | 90.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | gold | MLAS | not measured | 25.71 [24.47, 26.88] | 60.21 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | gold | UAS | not measured | 71.64 [70.28, 72.92] | 82.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | gold | UFeats | not measured | 64.39 [63.59, 65.16] | 90.38 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | gold | UPOS | not measured | 90.98 [90.54, 91.45] | 98.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | gold | XPOS | not measured | 0.77 [0.61, 0.96] | 91.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | orv: UD_Old_East_Slavic-RNC | raw | AllTags | not measured | 0.18 [0.13, 0.23] | 82.43 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 64.40 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | CLAS | not measured | 58.45 [56.77, 60.09] | 51.90 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 51.90 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | orv: UD_Old_East_Slavic-RNC | raw | LAS | not measured | 65.14 [63.69, 66.63] | 77.10 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 60.81 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | orv: UD_Old_East_Slavic-RNC | raw | Lemmas | not measured | 76.43 [75.40, 77.44] | 90.60 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 81.04 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | MLAS | not measured | 26.36 [25.18, 27.55] | 58.83 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 33.25 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | Sentences | not measured | 77.71 [74.93, 80.37] | 94.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.29 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | Tokens | not measured | 94.29 [93.17, 95.25] | 99.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | UAS | not measured | 69.06 [67.51, 70.59] | 80.58 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 68.10 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | orv: UD_Old_East_Slavic-RNC | raw | UFeats | not measured | 62.63 [61.64, 63.62] | 90.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 72.39 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | UPOS | not measured | 87.23 [86.10, 88.24] | 97.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 91.04 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | Words | not measured | 94.29 [93.17, 95.25] | 99.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | orv: UD_Old_East_Slavic-RNC | raw | XPOS | not measured | 0.74 [0.59, 0.92] | 91.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.53 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ota: UD_Ottoman_Turkish-BOUN | gold | AllTags | not measured | 22.38 [21.42, 23.47] | 71.24 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | CLAS | not measured | 49.18 [47.57, 51.01] | not published | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | LAS | not measured | 53.30 [51.69, 55.05] | 53.06 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | Lemmas | not measured | 71.93 [70.73, 73.16] | 82.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | MLAS | not measured | 24.74 [23.36, 26.21] | 33.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | UAS | not measured | 63.25 [61.59, 65.00] | 64.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | UFeats | not measured | 65.77 [64.62, 66.99] | 80.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | UPOS | not measured | 87.06 [86.18, 87.91] | 86.18 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | gold | XPOS | not measured | 36.50 [35.42, 37.62] | 90.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | AllTags | not measured | 22.74 [21.81, 23.78] | 70.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | CLAS | not measured | 56.78 [55.03, 58.62] | not published | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | LAS | not measured | 60.16 [58.55, 61.82] | 52.37 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | Lemmas | not measured | 71.73 [70.53, 72.92] | 81.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | MLAS | not measured | 29.74 [28.17, 31.21] | 32.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | Sentences | not measured | 86.01 [82.52, 89.41] | 87.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | Tokens | not measured | 99.52 [99.24, 99.77] | not published | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | UAS | not measured | 70.12 [68.62, 71.73] | 63.60 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | UFeats | not measured | 68.04 [66.90, 69.24] | 80.51 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | UPOS | not measured | 86.51 [85.54, 87.45] | 85.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | Words | not measured | 99.32 [99.02, 99.61] | 99.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | ota: UD_Ottoman_Turkish-BOUN | raw | XPOS | not measured | 36.82 [35.72, 37.92] | 90.42 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pa: UD_Punjabi-Rang | gold | CLAS | not measured | 45.92 [39.05, 53.05] | not published | not published | | pa: UD_Punjabi-Rang | gold | LAS | not measured | 49.00 [42.61, 55.25] | not published | not published | | pa: UD_Punjabi-Rang | gold | Lemmas | not measured | 79.00 [74.77, 82.86] | not published | not published | | pa: UD_Punjabi-Rang | gold | MLAS | not measured | 12.24 [8.85, 16.26] | not published | not published | | pa: UD_Punjabi-Rang | gold | UAS | not measured | 66.33 [60.36, 72.85] | not published | not published | | pa: UD_Punjabi-Rang | gold | UFeats | not measured | 58.33 [53.18, 62.91] | not published | not published | | pa: UD_Punjabi-Rang | gold | UPOS | not measured | 74.33 [69.86, 79.44] | not published | not published | | pa: UD_Punjabi-Rang | raw | CLAS | not measured | 43.55 [36.02, 51.61] | not published | not published | | pa: UD_Punjabi-Rang | raw | LAS | not measured | 47.67 [41.55, 54.04] | not published | not published | | pa: UD_Punjabi-Rang | raw | Lemmas | not measured | 79.00 [74.52, 82.86] | not published | not published | | pa: UD_Punjabi-Rang | raw | MLAS | not measured | 10.75 [5.78, 16.62] | not published | not published | | pa: UD_Punjabi-Rang | raw | Sentences | not measured | 48.78 [28.57, 68.29] | not published | not published | | pa: UD_Punjabi-Rang | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | pa: UD_Punjabi-Rang | raw | UAS | not measured | 62.33 [54.87, 69.20] | not published | not published | | pa: UD_Punjabi-Rang | raw | UFeats | not measured | 61.67 [57.29, 66.08] | not published | not published | | pa: UD_Punjabi-Rang | raw | UPOS | not measured | 71.33 [66.67, 75.68] | not published | not published | | pa: UD_Punjabi-Rang | raw | Words | not measured | 100.00 [100.00, 100.00] | not published | not published | | pcm: UD_Naija-NSC | gold | CLAS | not measured | 72.92 [71.48, 74.35] | not published | not published | | pcm: UD_Naija-NSC | gold | LAS | not measured | 77.80 [76.59, 79.04] | 90.03 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pcm: UD_Naija-NSC | gold | Lemmas | not measured | 98.18 [97.90, 98.45] | 99.31 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pcm: UD_Naija-NSC | gold | MLAS | not measured | 62.01 [60.37, 63.61] | 84.13 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pcm: UD_Naija-NSC | gold | UAS | not measured | 83.76 [82.61, 84.95] | 92.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pcm: UD_Naija-NSC | gold | UFeats | not measured | 94.78 [94.33, 95.23] | 98.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pcm: UD_Naija-NSC | gold | UPOS | not measured | 93.41 [92.94, 93.86] | 98.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pcm: UD_Naija-NSC | raw | CLAS | not measured | 70.47 [68.80, 71.95] | 88.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 88.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | LAS | not measured | 70.52 [69.09, 71.90] | 89.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.56 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | Lemmas | not measured | 91.69 [91.06, 92.29] | 99.28 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.14 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | MLAS | not measured | 59.40 [57.82, 60.86] | 86.43 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 86.43 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | Sentences | not measured | 79.29 [76.93, 81.60] | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 100.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | Tokens | not measured | 92.84 [92.26, 93.41] | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | UAS | not measured | 75.89 [74.49, 77.33] | 92.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 92.39 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | UFeats | not measured | 87.98 [87.21, 88.74] | 98.90 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 98.84 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | UPOS | not measured | 87.23 [86.44, 88.00] | 98.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.55 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pcm: UD_Naija-NSC | raw | Words | not measured | 92.84 [92.26, 93.41] | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pl: UD_Polish-MPDT | gold | AllTags | not measured | 58.11 [57.06, 59.12] | 88.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | gold | CLAS | not measured | 59.62 [57.97, 61.33] | not published | not published | | pl: UD_Polish-MPDT | gold | LAS | not measured | 65.29 [63.82, 66.70] | 78.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | gold | Lemmas | not measured | 84.84 [84.10, 85.62] | 96.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | gold | MLAS | not measured | 29.31 [27.92, 30.67] | 63.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | gold | UAS | not measured | 72.40 [70.93, 73.72] | 82.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | gold | UFeats | not measured | 68.09 [67.09, 69.07] | 91.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | gold | UPOS | not measured | 87.29 [86.49, 88.03] | 96.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | gold | XPOS | not measured | 66.19 [65.19, 67.25] | 90.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | AllTags | not measured | 59.11 [58.14, 60.11] | 86.51 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | CLAS | not measured | 63.52 [61.84, 65.04] | not published | not published | | pl: UD_Polish-MPDT | raw | LAS | not measured | 67.83 [66.27, 69.25] | 75.06 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | Lemmas | not measured | 83.03 [82.17, 83.94] | 94.70 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | MLAS | not measured | 31.42 [30.02, 32.70] | 60.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | Sentences | not measured | 75.73 [71.70, 79.64] | 87.46 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | Tokens | not measured | 98.93 [98.65, 99.20] | not published | not published | | pl: UD_Polish-MPDT | raw | UAS | not measured | 73.27 [71.78, 74.66] | 78.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | UFeats | not measured | 68.13 [67.15, 69.12] | 89.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | UPOS | not measured | 86.08 [85.21, 86.88] | 94.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | Words | not measured | 96.80 [96.29, 97.28] | 98.22 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pl: UD_Polish-MPDT | raw | XPOS | not measured | 66.98 [66.01, 68.05] | 88.43 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | gold | CLAS | not measured | 58.36 [56.05, 60.78] | not published | not published | | pro: UD_Old_Occitan-CorAG | gold | LAS | not measured | 66.97 [64.71, 69.20] | 80.85 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | gold | MLAS | not measured | 42.81 [40.62, 44.80] | 69.04 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | gold | UAS | not measured | 72.61 [70.28, 74.89] | 83.90 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | gold | UFeats | not measured | 88.11 [87.08, 89.06] | 97.02 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | gold | UPOS | not measured | 88.64 [87.74, 89.47] | 96.83 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | raw | CLAS | not measured | 57.05 [54.28, 59.82] | not published | not published | | pro: UD_Old_Occitan-CorAG | raw | LAS | not measured | 62.08 [59.46, 64.73] | 74.80 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | raw | MLAS | not measured | 40.76 [38.51, 43.02] | 63.84 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | raw | Sentences | not measured | 57.74 [51.69, 63.62] | 83.64 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | raw | Tokens | not measured | 95.35 [94.57, 96.17] | not published | not published | | pro: UD_Old_Occitan-CorAG | raw | UAS | not measured | 66.36 [63.69, 69.18] | 77.79 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | raw | UFeats | not measured | 80.84 [79.60, 82.01] | 95.37 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | raw | UPOS | not measured | 82.51 [81.42, 83.58] | 95.25 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pro: UD_Old_Occitan-CorAG | raw | Words | not measured | 91.16 [90.34, 91.98] | 98.55 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | pt: UD_Portuguese-Bosque | gold | CLAS | not measured | 80.24 [79.51, 81.07] | not published | not published | | pt: UD_Portuguese-Bosque | gold | LAS | 84.01 [83.39, 84.63] | 84.54 [83.90, 85.14] | 90.93 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pt: UD_Portuguese-Bosque | gold | Lemmas | 97.49 [97.30, 97.70] | 97.59 [97.39, 97.79] | 98.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pt: UD_Portuguese-Bosque | gold | MLAS | not measured | 70.29 [69.36, 71.22] | 81.55 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pt: UD_Portuguese-Bosque | gold | UAS | 87.64 [87.07, 88.25] | 88.22 [87.63, 88.77] | 93.31 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pt: UD_Portuguese-Bosque | gold | UFeats | 94.80 [94.49, 95.09] | 94.48 [94.16, 94.80] | 97.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pt: UD_Portuguese-Bosque | gold | UPOS | 96.56 [96.32, 96.80] | 96.10 [95.82, 96.36] | 98.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | pt: UD_Portuguese-Bosque | raw | CLAS | not measured | 85.27 [84.54, 85.99] | 87.75 (Trankit large, UD 2.5; test changed) — **MISS** | 82.66 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | pt: UD_Portuguese-Bosque | raw | LAS | 86.47 [85.84, 87.04] | 87.98 [87.39, 88.55] | 90.91 (Trankit large, UD 2.5; test changed) — **MISS** | 87.75 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | pt: UD_Portuguese-Bosque | raw | Lemmas | 96.96 [96.73, 97.19] | 97.29 [97.09, 97.50] | 98.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 98.04 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pt: UD_Portuguese-Bosque | raw | MLAS | not measured | 76.55 [75.69, 77.41] | 81.09 (Trankit large, UD 2.5; test changed) — **MISS** | 77.07 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | pt: UD_Portuguese-Bosque | raw | Sentences | 92.27 [90.68, 93.82] | 93.50 [92.14, 94.81] | 97.48 (Trankit base, UD 2.5; test changed) — **MISS** | 93.10 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | pt: UD_Portuguese-Bosque | raw | Tokens | 99.86 [99.79, 99.92] | 99.86 [99.79, 99.92] | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.85 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | pt: UD_Portuguese-Bosque | raw | UAS | 89.38 [88.77, 89.91] | 90.68 [90.16, 91.19] | 93.31 (Trankit large, UD 2.5; test changed) — **MISS** | 90.59 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | pt: UD_Portuguese-Bosque | raw | UFeats | 95.06 [94.75, 95.34] | 95.19 [94.90, 95.48] | 96.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.44 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pt: UD_Portuguese-Bosque | raw | UPOS | 96.59 [96.32, 96.83] | 96.75 [96.50, 96.98] | 97.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.45 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | pt: UD_Portuguese-Bosque | raw | Words | 99.26 [99.14, 99.38] | 99.26 [99.14, 99.38] | 99.76 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.76 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | qte: UD_Telugu_English-TECT | gold | AllTags | not measured | 67.47 [60.24, 74.85] | not published | not published | | qte: UD_Telugu_English-TECT | gold | CLAS | not measured | 64.31 [55.17, 73.90] | not published | not published | | qte: UD_Telugu_English-TECT | gold | LAS | not measured | 68.67 [61.11, 76.54] | not published | not published | | qte: UD_Telugu_English-TECT | gold | MLAS | not measured | 37.65 [29.43, 46.81] | not published | not published | | qte: UD_Telugu_English-TECT | gold | UAS | not measured | 86.14 [79.29, 92.36] | not published | not published | | qte: UD_Telugu_English-TECT | gold | UFeats | not measured | 99.40 [98.18, 100.00] | not published | not published | | qte: UD_Telugu_English-TECT | gold | UPOS | not measured | 68.07 [61.08, 75.00] | not published | not published | | qte: UD_Telugu_English-TECT | gold | XPOS | not measured | 68.07 [60.95, 75.61] | not published | not published | | qte: UD_Telugu_English-TECT | raw | AllTags | not measured | 80.12 [72.88, 87.72] | not published | not published | | qte: UD_Telugu_English-TECT | raw | CLAS | not measured | 62.90 [51.38, 73.90] | not published | not published | | qte: UD_Telugu_English-TECT | raw | LAS | not measured | 69.88 [60.00, 79.53] | not published | not published | | qte: UD_Telugu_English-TECT | raw | MLAS | not measured | 55.65 [44.44, 67.78] | not published | not published | | qte: UD_Telugu_English-TECT | raw | Sentences | not measured | 83.58 [71.64, 94.12] | not published | not published | | qte: UD_Telugu_English-TECT | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | qte: UD_Telugu_English-TECT | raw | UAS | not measured | 85.54 [76.79, 92.99] | not published | not published | | qte: UD_Telugu_English-TECT | raw | UFeats | not measured | 99.40 [98.18, 100.00] | not published | not published | | qte: UD_Telugu_English-TECT | raw | UPOS | not measured | 80.72 [73.84, 87.80] | not published | not published | | qte: UD_Telugu_English-TECT | raw | Words | not measured | 100.00 [100.00, 100.00] | not published | not published | | qte: UD_Telugu_English-TECT | raw | XPOS | not measured | 80.12 [72.88, 87.72] | not published | not published | | ro: UD_Romanian-RRT | gold | AllTags | not measured | 73.10 [72.40, 73.85] | 97.19 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | gold | CLAS | not measured | 76.60 [75.57, 77.61] | not published | not published | | ro: UD_Romanian-RRT | gold | LAS | not measured | 80.28 [79.46, 81.10] | 89.53 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | gold | Lemmas | not measured | 94.09 [93.64, 94.54] | 98.36 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | gold | MLAS | not measured | 48.34 [47.20, 49.55] | 82.89 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | gold | UAS | not measured | 85.60 [84.82, 86.33] | 92.77 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | gold | UFeats | not measured | 78.09 [77.38, 78.83] | 97.64 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | gold | UPOS | not measured | 93.35 [92.92, 93.78] | 98.14 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | gold | XPOS | not measured | 83.58 [82.95, 84.21] | 97.48 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | ro: UD_Romanian-RRT | raw | AllTags | not measured | 78.66 [77.97, 79.39] | 97.11 (Trankit large, UD 2.5; test changed) — **MISS** | 96.74 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | CLAS | not measured | 79.77 [78.67, 80.94] | 86.61 (Trankit large, UD 2.5; test changed) — **MISS** | 82.99 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | LAS | not measured | 82.32 [81.33, 83.35] | 89.50 (Trankit large, UD 2.5; test changed) — **MISS** | 86.86 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | Lemmas | not measured | 92.22 [91.71, 92.76] | 98.10 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | 97.42 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | MLAS | not measured | 57.93 [56.55, 59.29] | 82.78 (Trankit large, UD 2.5; test changed) — **MISS** | 79.90 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | Sentences | not measured | 89.05 [86.86, 91.15] | 98.49 (Trankit large, UD 2.5; test changed) — **MISS** | 95.72 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | Tokens | not measured | 96.96 [96.60, 97.31] | 99.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | UAS | not measured | 86.02 [85.08, 86.94] | 93.60 (Trankit large, UD 2.5; test changed) — **MISS** | 90.70 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | UFeats | not measured | 81.68 [81.06, 82.39] | 97.44 (Trankit base, UD 2.5; test changed) — **MISS** | 97.10 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | UPOS | not measured | 91.67 [91.16, 92.16] | 97.90 (Trankit large, UD 2.5; test changed) — **MISS** | 97.50 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | Words | not measured | 96.96 [96.60, 97.31] | 99.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 99.73 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ro: UD_Romanian-RRT | raw | XPOS | not measured | 85.00 [84.38, 85.66] | 97.35 (Trankit large, UD 2.5; test changed) — **MISS** | 96.80 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | ru: UD_Russian-GSD | gold | AllTags | not measured | 66.51 [65.46, 67.52] | 92.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | gold | CLAS | not measured | 73.03 [70.98, 74.74] | not published | not published | | ru: UD_Russian-GSD | gold | LAS | not measured | 75.27 [72.81, 77.18] | 90.19 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | gold | Lemmas | not measured | 87.93 [87.13, 88.70] | 97.58 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | gold | MLAS | not measured | 41.80 [40.05, 43.56] | 81.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | gold | UAS | not measured | 80.60 [78.19, 82.47] | 93.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | gold | UFeats | not measured | 70.83 [69.82, 71.76] | 94.18 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | gold | UPOS | not measured | 91.68 [91.13, 92.27] | 98.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | gold | XPOS | not measured | 92.12 [91.33, 92.87] | 97.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ru: UD_Russian-GSD | raw | AllTags | not measured | 65.75 [63.92, 67.53] | 94.73 (Trankit large, UD 2.5; test same size) — **MISS** | 92.76 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | CLAS | not measured | 82.94 [80.87, 84.72] | 88.71 (Trankit large, UD 2.5; test same size) — **MISS** | 83.91 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | ru: UD_Russian-GSD | raw | LAS | not measured | 78.76 [75.58, 81.39] | 89.75 (Trankit large, UD 2.5; test same size) — **MISS** | 85.89 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | Lemmas | not measured | 83.84 [81.63, 85.79] | 97.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 95.51 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | MLAS | not measured | 51.94 [49.99, 53.69] | 82.92 (Trankit large, UD 2.5; test same size) — **MISS** | 77.00 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | Sentences | not measured | 92.26 [90.07, 94.24] | 99.25 (Trankit large, UD 2.5; test same size) — **MISS** | 97.33 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | Tokens | not measured | 92.86 [90.99, 94.49] | 99.79 (Trankit large, UD 2.5; test same size) — **MISS** | 99.58 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | UAS | not measured | 82.15 [78.95, 84.74] | 92.70 (Trankit large, UD 2.5; test same size) — **MISS** | 89.56 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | UFeats | not measured | 69.12 [67.53, 70.70] | 95.78 (Trankit large, UD 2.5; test same size) — **MISS** | 93.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | UPOS | not measured | 86.36 [84.48, 88.05] | 98.25 (Trankit large, UD 2.5; test same size) — **MISS** | 97.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | Words | not measured | 92.86 [90.99, 94.49] | 99.79 (Trankit large, UD 2.5; test same size) — **MISS** | 99.58 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ru: UD_Russian-GSD | raw | XPOS | not measured | 87.73 [85.60, 89.58] | 97.89 (Trankit large, UD 2.5; test same size) — **MISS** | 97.41 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sa: UD_Sanskrit-Vedic | gold | CLAS | not measured | 32.81 [31.76, 33.95] | not published | not published | | sa: UD_Sanskrit-Vedic | gold | LAS | not measured | 34.05 [33.06, 35.15] | 68.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sa: UD_Sanskrit-Vedic | gold | Lemmas | not measured | 75.77 [75.04, 76.50] | 93.50 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sa: UD_Sanskrit-Vedic | gold | MLAS | not measured | 17.76 [17.00, 18.55] | 60.40 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sa: UD_Sanskrit-Vedic | gold | UAS | not measured | 50.53 [49.31, 51.78] | 77.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sa: UD_Sanskrit-Vedic | gold | UFeats | not measured | 51.29 [50.48, 52.14] | 90.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sa: UD_Sanskrit-Vedic | gold | UPOS | not measured | 70.59 [69.80, 71.37] | 93.83 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sa: UD_Sanskrit-Vedic | raw | CLAS | not measured | 19.56 [18.68, 20.43] | 49.90 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 49.90 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | LAS | not measured | 21.48 [20.62, 22.37] | 56.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 50.64 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | Lemmas | not measured | 75.57 [74.82, 76.32] | 93.45 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 88.92 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | MLAS | not measured | 11.55 [10.94, 12.20] | 49.07 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 43.27 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | Sentences | not measured | 7.22 [6.04, 8.52] | 32.93 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | 32.93 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | Tokens | not measured | 100.00 [100.00, 100.00] | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | sa: UD_Sanskrit-Vedic | raw | UAS | not measured | 32.91 [31.88, 33.99] | 65.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 62.52 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | UFeats | not measured | 51.03 [50.19, 51.82] | 89.24 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.70 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | UPOS | not measured | 70.54 [69.75, 71.32] | 93.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.19 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | sa: UD_Sanskrit-Vedic | raw | Words | not measured | 100.00 [100.00, 100.00] | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | 100.00 (Stanza 1.5.1, UD 2.12; test changed) — within or ahead | | say: UD_Zaar-Autogramm | gold | CLAS | not measured | 37.35 [35.92, 38.67] | not published | not published | | say: UD_Zaar-Autogramm | gold | LAS | not measured | 49.53 [48.16, 50.82] | 73.34 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | gold | Lemmas | not measured | 76.93 [76.12, 77.76] | 87.42 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | gold | MLAS | not measured | 17.13 [16.12, 18.16] | 53.71 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | gold | UAS | not measured | 67.81 [66.22, 69.22] | 83.04 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | gold | UFeats | not measured | 75.96 [75.11, 76.84] | 91.37 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | gold | UPOS | not measured | 66.51 [65.55, 67.43] | 90.67 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | CLAS | not measured | 24.67 [23.21, 26.09] | not published | not published | | say: UD_Zaar-Autogramm | raw | LAS | not measured | 36.26 [34.74, 37.74] | 62.81 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | Lemmas | not measured | 68.34 [67.29, 69.42] | 82.47 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | MLAS | not measured | 12.24 [11.27, 13.22] | 43.14 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | Sentences | not measured | 63.56 [60.64, 66.19] | 98.93 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | Tokens | not measured | 97.49 [97.07, 97.92] | not published | not published | | say: UD_Zaar-Autogramm | raw | UAS | not measured | 49.09 [47.38, 50.77] | 70.99 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | UFeats | not measured | 66.79 [65.80, 67.82] | 85.78 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | UPOS | not measured | 62.38 [61.35, 63.50] | 84.94 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | say: UD_Zaar-Autogramm | raw | Words | not measured | 87.48 [86.71, 88.26] | 94.06 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | sd: UD_Sindhi-Isra | gold | AllTags | not measured | 54.53 [53.45, 55.53] | 69.11 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | gold | CLAS | not measured | 62.26 [60.63, 63.73] | not published | not published | | sd: UD_Sindhi-Isra | gold | LAS | not measured | 68.88 [67.40, 70.23] | 76.42 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | gold | Lemmas | not measured | 95.63 [95.19, 96.02] | 94.88 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sd: UD_Sindhi-Isra | gold | MLAS | not measured | 25.46 [24.16, 26.65] | 41.22 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | gold | UAS | not measured | 79.54 [78.22, 80.86] | 85.51 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | gold | UFeats | not measured | 58.35 [57.29, 59.41] | 71.23 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | gold | UPOS | not measured | 88.18 [87.42, 88.98] | 93.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | gold | XPOS | not measured | 87.16 [86.35, 87.93] | 91.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | AllTags | not measured | 56.13 [55.01, 57.17] | 68.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | CLAS | not measured | 61.01 [59.53, 62.49] | not published | not published | | sd: UD_Sindhi-Isra | raw | LAS | not measured | 68.19 [66.86, 69.52] | 74.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | Lemmas | not measured | 95.89 [95.44, 96.28] | 94.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sd: UD_Sindhi-Isra | raw | MLAS | not measured | 27.07 [25.83, 28.33] | 39.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | Sentences | not measured | 71.46 [68.49, 74.43] | 88.28 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | Tokens | not measured | 99.99 [99.96, 100.00] | not published | not published | | sd: UD_Sindhi-Isra | raw | UAS | not measured | 78.38 [77.10, 79.68] | 83.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | UFeats | not measured | 60.15 [59.02, 61.21] | 71.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | UPOS | not measured | 88.87 [88.14, 89.57] | 93.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sd: UD_Sindhi-Isra | raw | Words | not measured | 99.99 [99.96, 100.00] | 99.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sd: UD_Sindhi-Isra | raw | XPOS | not measured | 87.00 [86.22, 87.72] | 91.83 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sdh: UD_Southern_Kurdish-Garrusi | gold | CLAS | not measured | 48.57 [38.78, 58.56] | not published | not published | | sdh: UD_Southern_Kurdish-Garrusi | gold | LAS | not measured | 54.41 [43.84, 65.00] | 68.38 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sdh: UD_Southern_Kurdish-Garrusi | gold | Lemmas | not measured | 59.56 [51.64, 66.43] | 75.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sdh: UD_Southern_Kurdish-Garrusi | gold | MLAS | not measured | 23.81 [16.41, 32.22] | 45.63 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sdh: UD_Southern_Kurdish-Garrusi | gold | UAS | not measured | 72.79 [60.51, 84.80] | 76.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sdh: UD_Southern_Kurdish-Garrusi | gold | UFeats | not measured | 61.03 [54.17, 69.05] | 78.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sdh: UD_Southern_Kurdish-Garrusi | gold | UPOS | not measured | 62.50 [55.40, 70.97] | 83.82 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | CLAS | not measured | 59.70 [49.09, 73.26] | not published | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | LAS | not measured | 63.43 [52.99, 76.52] | 62.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | Lemmas | not measured | 58.96 [50.22, 65.49] | 67.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | MLAS | not measured | 46.77 [37.17, 58.76] | 40.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | Sentences | not measured | 100.00 [100.00, 100.00] | 92.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | UAS | not measured | 72.39 [61.09, 83.40] | 68.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | UFeats | not measured | 72.39 [64.79, 81.27] | 72.92 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | UPOS | not measured | 74.63 [68.10, 83.33] | 77.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sdh: UD_Southern_Kurdish-Garrusi | raw | Words | not measured | 95.52 [90.99, 98.92] | 90.25 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sk: UD_Slovak-SNK | gold | AllTags | not measured | 62.68 [61.73, 63.55] | 89.19 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | gold | CLAS | not measured | 77.40 [76.24, 78.53] | not published | not published | | sk: UD_Slovak-SNK | gold | LAS | not measured | 80.00 [79.04, 80.95] | 91.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | gold | Lemmas | not measured | 87.30 [86.65, 87.95] | 96.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | gold | MLAS | not measured | 45.69 [44.38, 46.86] | 81.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | gold | UAS | not measured | 84.02 [83.09, 84.86] | 93.75 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | gold | UFeats | not measured | 70.16 [69.26, 71.01] | 93.31 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | gold | UPOS | not measured | 91.25 [90.73, 91.76] | 97.83 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | gold | XPOS | not measured | 67.51 [66.60, 68.37] | 90.18 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sk: UD_Slovak-SNK | raw | AllTags | not measured | 68.26 [67.41, 69.12] | 90.20 (Trankit large, UD 2.5; test changed) — **MISS** | 87.35 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sk: UD_Slovak-SNK | raw | CLAS | not measured | 87.18 [86.26, 88.16] | 95.04 (Trankit large, UD 2.5; test changed) — **MISS** | 84.43 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sk: UD_Slovak-SNK | raw | LAS | not measured | 88.48 [87.67, 89.31] | 95.60 (Trankit large, UD 2.5; test changed) — **MISS** | 86.58 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sk: UD_Slovak-SNK | raw | Lemmas | not measured | 91.29 [90.75, 91.76] | 96.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.08 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sk: UD_Slovak-SNK | raw | MLAS | not measured | 55.91 [54.76, 57.19] | 87.64 (Trankit large, UD 2.5; test changed) — **MISS** | 74.72 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sk: UD_Slovak-SNK | raw | Sentences | not measured | 86.76 [84.86, 88.51] | 98.49 (Trankit large, UD 2.5; test changed) — **MISS** | 84.31 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sk: UD_Slovak-SNK | raw | Tokens | not measured | 99.97 [99.93, 100.00] | 99.96 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.96 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sk: UD_Slovak-SNK | raw | UAS | not measured | 90.75 [90.04, 91.51] | 96.97 (Trankit large, UD 2.5; test changed) — **MISS** | 89.13 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sk: UD_Slovak-SNK | raw | UFeats | not measured | 75.33 [74.50, 76.16] | 95.22 (Trankit large, UD 2.5; test changed) — **MISS** | 90.95 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sk: UD_Slovak-SNK | raw | UPOS | not measured | 92.17 [91.70, 92.63] | 97.90 (Trankit large, UD 2.5; test changed) — **MISS** | 96.34 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sk: UD_Slovak-SNK | raw | Words | not measured | 99.93 [99.88, 99.97] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.96 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sk: UD_Slovak-SNK | raw | XPOS | not measured | 71.71 [70.86, 72.58] | 90.82 (Trankit large, UD 2.5; test changed) — **MISS** | 88.19 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | gold | AllTags | not measured | 74.45 [73.88, 75.05] | 96.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | gold | CLAS | not measured | 77.48 [76.61, 78.37] | not published | not published | | sl: UD_Slovenian-SSJ | gold | LAS | not measured | 81.88 [81.19, 82.57] | 93.10 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | gold | Lemmas | not measured | 92.93 [92.58, 93.27] | 98.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | gold | MLAS | not measured | 50.13 [49.19, 51.12] | 87.50 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | gold | UAS | not measured | 86.44 [85.81, 87.04] | 94.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | gold | UFeats | not measured | 78.15 [77.59, 78.70] | 97.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | gold | UPOS | not measured | 93.28 [92.91, 93.60] | 98.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | gold | XPOS | not measured | 78.26 [77.75, 78.80] | 97.10 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sl: UD_Slovenian-SSJ | raw | AllTags | not measured | 77.13 [76.59, 77.70] | 97.60 (Trankit large, UD 2.5; test changed) — **MISS** | 95.37 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | CLAS | not measured | 85.70 [84.98, 86.50] | 94.88 (Trankit large, UD 2.5; test changed) — **MISS** | 88.28 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | LAS | not measured | 88.30 [87.67, 89.00] | 96.06 (Trankit large, UD 2.5; test changed) — **MISS** | 90.80 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | Lemmas | not measured | 93.06 [92.68, 93.44] | 98.69 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.69 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | MLAS | not measured | 58.69 [57.81, 59.66] | 91.78 (Trankit large, UD 2.5; test changed) — **MISS** | 83.80 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | Sentences | not measured | 96.78 [95.80, 97.68] | 100.00 (Trankit large, UD 2.5; test changed) — **MISS** | 99.26 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | Tokens | not measured | 99.20 [98.99, 99.41] | 99.97 (Trankit large, UD 2.5; test changed) — **MISS** | 99.93 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | UAS | not measured | 91.24 [90.64, 91.85] | 96.91 (Trankit large, UD 2.5; test changed) — **MISS** | 92.45 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | UFeats | not measured | 80.00 [79.46, 80.54] | 97.99 (Trankit large, UD 2.5; test changed) — **MISS** | 96.09 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | UPOS | not measured | 93.49 [93.10, 93.84] | 99.24 (Trankit large, UD 2.5; test changed) — **MISS** | 98.41 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | Words | not measured | 99.20 [98.99, 99.41] | 99.97 (Trankit large, UD 2.5; test changed) — **MISS** | 99.93 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sl: UD_Slovenian-SSJ | raw | XPOS | not measured | 80.09 [79.60, 80.64] | 97.83 (Trankit large, UD 2.5; test changed) — **MISS** | 95.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | gold | AllTags | not measured | 62.16 [61.15, 63.19] | 85.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | gold | CLAS | not measured | 52.19 [50.64, 53.68] | not published | not published | | sme: UD_North_Sami-Giella | gold | LAS | not measured | 55.43 [53.91, 56.87] | 71.52 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | gold | Lemmas | not measured | 73.86 [72.93, 74.78] | 87.05 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | gold | MLAS | not measured | 30.82 [29.58, 32.05] | 60.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | gold | UAS | not measured | 64.22 [62.61, 65.69] | 76.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | gold | UFeats | not measured | 69.28 [68.34, 70.28] | 89.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | gold | UPOS | not measured | 81.63 [80.70, 82.42] | 91.94 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | gold | XPOS | not measured | 84.71 [83.93, 85.40] | 93.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sme: UD_North_Sami-Giella | raw | AllTags | not measured | 65.57 [64.57, 66.61] | 85.48 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 83.69 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | CLAS | not measured | 60.76 [59.17, 62.39] | 64.19 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 64.19 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | LAS | not measured | 63.12 [61.54, 64.64] | 71.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 66.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | Lemmas | not measured | 74.79 [73.90, 75.67] | 88.54 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 88.54 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | MLAS | not measured | 38.33 [36.89, 39.78] | 60.71 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 57.21 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | Sentences | not measured | 96.12 [94.45, 97.45] | 99.65 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.65 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | Tokens | not measured | 98.83 [98.49, 99.14] | 99.71 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.71 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | UAS | not measured | 70.90 [69.24, 72.42] | 76.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 72.26 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sme: UD_North_Sami-Giella | raw | UFeats | not measured | 71.50 [70.52, 72.50] | 89.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 88.08 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | UPOS | not measured | 84.08 [83.28, 84.85] | 91.83 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 90.73 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | Words | not measured | 98.83 [98.49, 99.14] | 99.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 99.71 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sme: UD_North_Sami-Giella | raw | XPOS | not measured | 87.64 [86.90, 88.36] | 93.50 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 92.77 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sms: UD_Skolt_Sami-Giellagas | gold | AllTags | not measured | 13.36 [11.90, 14.74] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | CLAS | not measured | 17.59 [15.73, 19.70] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | LAS | not measured | 22.10 [20.21, 24.31] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | Lemmas | not measured | 60.88 [59.19, 62.36] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | MLAS | not measured | 1.20 [0.68, 1.78] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | UAS | not measured | 38.47 [35.74, 41.39] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | UFeats | not measured | 42.75 [41.38, 44.19] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | UPOS | not measured | 35.15 [33.65, 36.84] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | gold | XPOS | not measured | 35.67 [33.71, 37.69] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | AllTags | not measured | 16.68 [15.23, 18.12] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | CLAS | not measured | 12.62 [10.80, 14.79] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | LAS | not measured | 16.40 [14.47, 18.50] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | Lemmas | not measured | 61.05 [59.43, 62.51] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | MLAS | not measured | 0.97 [0.58, 1.41] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | Sentences | not measured | 16.62 [11.32, 22.16] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | UAS | not measured | 28.87 [26.11, 31.77] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | UFeats | not measured | 42.27 [40.76, 43.69] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | UPOS | not measured | 39.33 [37.74, 41.08] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | Words | not measured | 100.00 [100.00, 100.00] | not published | not published | | sms: UD_Skolt_Sami-Giellagas | raw | XPOS | not measured | 41.30 [39.05, 43.34] | not published | not published | | sq: UD_Albanian-STAF | gold | CLAS | not measured | 60.05 [52.05, 69.94] | not published | not published | | sq: UD_Albanian-STAF | gold | LAS | not measured | 67.74 [62.23, 74.29] | 64.52 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | gold | Lemmas | not measured | 80.65 [73.78, 86.16] | 81.99 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | gold | MLAS | not measured | 25.67 [19.54, 32.65] | 33.09 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sq: UD_Albanian-STAF | gold | UAS | not measured | 79.03 [73.76, 85.76] | 77.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | gold | UFeats | not measured | 61.02 [52.66, 68.93] | 70.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sq: UD_Albanian-STAF | gold | UPOS | not measured | 84.68 [80.45, 89.31] | 88.71 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | raw | CLAS | not measured | 60.78 [53.28, 70.55] | not published | not published | | sq: UD_Albanian-STAF | raw | LAS | not measured | 68.30 [61.94, 75.59] | 65.86 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | raw | Lemmas | not measured | 77.55 [70.55, 83.11] | 80.65 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | raw | MLAS | not measured | 27.45 [19.69, 36.47] | 33.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | raw | Sentences | not measured | 84.21 [66.67, 97.44] | 92.68 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | raw | Tokens | not measured | 99.59 [98.53, 100.00] | not published | not published | | sq: UD_Albanian-STAF | raw | UAS | not measured | 78.10 [73.22, 83.78] | 77.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | sq: UD_Albanian-STAF | raw | UFeats | not measured | 60.68 [51.60, 69.45] | 70.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sq: UD_Albanian-STAF | raw | UPOS | not measured | 82.18 [77.75, 86.79] | 87.90 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sq: UD_Albanian-STAF | raw | Words | not measured | 95.51 [92.25, 97.95] | 98.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | AllTags | not measured | 77.26 [76.37, 78.21] | 95.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | CLAS | not measured | 78.75 [77.56, 79.97] | not published | not published | | sr: UD_Serbian-SET | gold | LAS | not measured | 81.81 [80.86, 82.86] | 91.77 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | Lemmas | not measured | 91.91 [91.34, 92.48] | 97.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | MLAS | not measured | 56.11 [54.67, 57.59] | 84.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | UAS | not measured | 85.84 [84.96, 86.71] | 94.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | UFeats | not measured | 80.98 [80.12, 81.87] | 96.19 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | UPOS | not measured | 93.67 [93.13, 94.19] | 99.06 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | gold | XPOS | not measured | 81.84 [81.03, 82.68] | 96.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sr: UD_Serbian-SET | raw | AllTags | not measured | 81.57 [80.81, 82.38] | 95.84 (Trankit large, UD 2.5; test same size) — **MISS** | 93.76 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | CLAS | not measured | 86.02 [84.98, 87.10] | 91.92 (Trankit large, UD 2.5; test same size) — **MISS** | 85.92 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sr: UD_Serbian-SET | raw | LAS | not measured | 88.01 [87.10, 88.97] | 93.13 (Trankit large, UD 2.5; test same size) — **MISS** | 88.36 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sr: UD_Serbian-SET | raw | Lemmas | not measured | 92.61 [92.05, 93.13] | 97.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 96.30 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | MLAS | not measured | 66.16 [64.76, 67.54] | 85.76 (Trankit large, UD 2.5; test same size) — **MISS** | 78.82 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | Sentences | not measured | 92.20 [90.01, 94.20] | 100.00 (Trankit large, UD 2.5; test same size) — **MISS** | 98.46 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | Tokens | not measured | 98.80 [98.53, 99.06] | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | UAS | not measured | 90.90 [90.04, 91.74] | 95.57 (Trankit large, UD 2.5; test same size) — **MISS** | 91.44 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sr: UD_Serbian-SET | raw | UFeats | not measured | 84.64 [83.87, 85.41] | 96.40 (Trankit large, UD 2.5; test same size) — **MISS** | 94.31 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | UPOS | not measured | 94.92 [94.49, 95.34] | 99.06 (Trankit large, UD 2.5; test same size) — **MISS** | 98.44 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | Words | not measured | 98.80 [98.53, 99.06] | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.99 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sr: UD_Serbian-SET | raw | XPOS | not measured | 84.12 [83.36, 84.87] | 96.22 (Trankit large, UD 2.5; test same size) — **MISS** | 94.16 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | gold | AllTags | not measured | 86.03 [85.49, 86.53] | 96.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | gold | CLAS | not measured | 79.60 [78.66, 80.53] | not published | not published | | sv: UD_Swedish-Talbanken | gold | LAS | not measured | 82.89 [82.06, 83.67] | 91.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | gold | Lemmas | not measured | 96.25 [95.98, 96.50] | 98.66 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | gold | MLAS | not measured | 64.57 [63.52, 65.53] | 85.23 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | gold | UAS | not measured | 86.74 [85.99, 87.46] | 93.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | gold | UFeats | not measured | 90.07 [89.62, 90.48] | 97.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | gold | UPOS | not measured | 96.74 [96.45, 97.00] | 98.79 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | gold | XPOS | not measured | 90.35 [89.90, 90.79] | 97.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | sv: UD_Swedish-Talbanken | raw | AllTags | not measured | 87.64 [87.17, 88.10] | 97.43 (Trankit large, UD 2.5; test same size) — **MISS** | 96.24 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | CLAS | not measured | 85.59 [84.73, 86.44] | 92.08 (Trankit large, UD 2.5; test same size) — **MISS** | 85.91 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sv: UD_Swedish-Talbanken | raw | LAS | not measured | 87.66 [86.83, 88.46] | 92.97 (Trankit large, UD 2.5; test same size) — **MISS** | 87.85 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sv: UD_Swedish-Talbanken | raw | Lemmas | not measured | 96.46 [96.20, 96.73] | 98.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 97.92 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | MLAS | not measured | 72.70 [71.71, 73.66] | 88.67 (Trankit large, UD 2.5; test same size) — **MISS** | 81.88 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | Sentences | not measured | 95.84 [94.86, 96.79] | 99.38 (Trankit base, UD 2.5; test same size) — **MISS** | 97.49 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | Tokens | not measured | 99.36 [99.22, 99.50] | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | UAS | not measured | 90.67 [89.94, 91.38] | 94.41 (Trankit large, UD 2.5; test same size) — **MISS** | 90.32 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | sv: UD_Swedish-Talbanken | raw | UFeats | not measured | 91.02 [90.62, 91.42] | 98.04 (Trankit large, UD 2.5; test same size) — **MISS** | 97.29 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | UPOS | not measured | 96.83 [96.55, 97.10] | 99.06 (Trankit large, UD 2.5; test same size) — **MISS** | 98.28 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | Words | not measured | 99.36 [99.22, 99.50] | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.97 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | sv: UD_Swedish-Talbanken | raw | XPOS | not measured | 90.73 [90.30, 91.15] | 98.09 (Trankit large, UD 2.5; test same size) — **MISS** | 97.13 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | te: UD_Telugu-MTG | gold | AllTags | not measured | 94.45 [92.55, 96.23] | 94.04 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | te: UD_Telugu-MTG | gold | CLAS | not measured | 77.67 [73.36, 82.27] | not published | not published | | te: UD_Telugu-MTG | gold | LAS | not measured | 81.41 [77.48, 85.31] | 85.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | te: UD_Telugu-MTG | gold | MLAS | not measured | 73.09 [68.09, 78.28] | 77.86 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | te: UD_Telugu-MTG | gold | UAS | not measured | 90.71 [87.60, 93.74] | 91.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | te: UD_Telugu-MTG | gold | UFeats | not measured | 98.89 [97.83, 99.72] | 98.89 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | te: UD_Telugu-MTG | gold | UPOS | not measured | 94.45 [92.55, 96.23] | 94.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | te: UD_Telugu-MTG | gold | XPOS | not measured | 94.45 [92.55, 96.23] | 94.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | te: UD_Telugu-MTG | raw | AllTags | not measured | 93.42 [91.41, 95.40] | 94.32 (Trankit base, UD 2.5; test same size) — within or ahead | 93.97 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | CLAS | not measured | 81.14 [76.68, 85.38] | 84.46 (Trankit large, UD 2.5; test same size) — within or ahead | 78.13 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | LAS | not measured | 84.13 [80.47, 87.77] | 87.08 (Trankit large, UD 2.5; test same size) — within or ahead | 81.64 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | MLAS | not measured | 75.81 [71.10, 80.64] | 80.65 (Trankit large, UD 2.5; test same size) — **MISS** | 74.50 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | Sentences | not measured | 89.68 [85.09, 94.04] | 98.62 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 98.62 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | te: UD_Telugu-MTG | raw | Tokens | not measured | 99.79 [99.35, 100.00] | 99.79 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.79 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | UAS | not measured | 91.20 [88.35, 94.05] | 93.19 (Trankit large, UD 2.5; test same size) — within or ahead | 90.09 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | UFeats | not measured | 98.54 [97.49, 99.57] | 99.10 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.10 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | UPOS | not measured | 93.42 [91.41, 95.40] | 94.32 (Trankit base, UD 2.5; test same size) — within or ahead | 93.97 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | Words | not measured | 99.79 [99.35, 100.00] | 99.79 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.79 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | te: UD_Telugu-MTG | raw | XPOS | not measured | 93.42 [91.41, 95.40] | 94.32 (Trankit base, UD 2.5; test same size) — within or ahead | 93.97 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | th: UD_Thai-TUD | gold | CLAS | not measured | 57.88 [56.17, 59.81] | not published | not published | | th: UD_Thai-TUD | gold | LAS | not measured | 61.77 [60.27, 63.62] | 71.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | th: UD_Thai-TUD | gold | MLAS | not measured | 48.08 [46.44, 49.91] | 61.35 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | th: UD_Thai-TUD | gold | UAS | not measured | 73.57 [72.28, 74.97] | 81.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | th: UD_Thai-TUD | gold | UFeats | not measured | 100.00 [100.00, 100.00] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | th: UD_Thai-TUD | gold | UPOS | not measured | 85.27 [84.38, 86.17] | 89.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | th: UD_Thai-TUD | raw | CLAS | not measured | 42.94 [40.91, 44.91] | not published | not published | | th: UD_Thai-TUD | raw | LAS | not measured | 47.69 [45.84, 49.59] | 50.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | th: UD_Thai-TUD | raw | MLAS | not measured | 34.82 [32.93, 36.61] | 41.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | th: UD_Thai-TUD | raw | Sentences | not measured | 26.10 [20.93, 31.02] | 11.36 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | th: UD_Thai-TUD | raw | Tokens | not measured | 88.26 [87.23, 89.24] | not published | not published | | th: UD_Thai-TUD | raw | UAS | not measured | 57.38 [55.53, 59.28] | 58.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | th: UD_Thai-TUD | raw | UFeats | not measured | 88.26 [87.23, 89.24] | 87.14 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | th: UD_Thai-TUD | raw | UPOS | not measured | 73.95 [72.71, 75.22] | 77.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | th: UD_Thai-TUD | raw | Words | not measured | 88.26 [87.23, 89.24] | 87.14 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | tr: UD_Turkish-BOUN | gold | AllTags | not measured | 37.77 [37.03, 38.61] | 73.41 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | gold | CLAS | not measured | 65.35 [63.77, 66.74] | not published | not published | | tr: UD_Turkish-BOUN | gold | LAS | 67.92 [66.56, 69.24] | 67.13 [65.67, 68.45] | 74.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | gold | Lemmas | 86.40 [85.75, 87.00] | 86.00 [85.31, 86.61] | 93.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | gold | MLAS | not measured | 45.80 [44.46, 47.03] | 52.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | gold | UAS | 75.25 [73.92, 76.56] | 74.44 [73.05, 75.76] | 80.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | gold | UFeats | 80.57 [79.80, 81.33] | 80.29 [79.54, 81.01] | 83.03 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | gold | UPOS | 91.50 [90.98, 92.05] | 91.28 [90.72, 91.81] | 93.14 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | gold | XPOS | not measured | 44.67 [43.81, 45.55] | 89.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | tr: UD_Turkish-BOUN | raw | AllTags | not measured | 37.44 [36.68, 38.26] | 73.03 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 73.03 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | tr: UD_Turkish-BOUN | raw | CLAS | not measured | 63.21 [61.68, 64.58] | 63.79 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 63.79 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | tr: UD_Turkish-BOUN | raw | LAS | 63.03 [61.61, 64.41] | 64.31 [62.89, 65.78] | 67.27 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 65.17 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | tr: UD_Turkish-BOUN | raw | Lemmas | 84.51 [83.78, 85.20] | 84.42 [83.71, 85.09] | 90.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 90.37 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | tr: UD_Turkish-BOUN | raw | MLAS | not measured | 46.89 [45.51, 48.21] | 49.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 48.49 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | tr: UD_Turkish-BOUN | raw | Sentences | 81.28 [78.81, 83.58] | 78.82 [76.14, 81.22] | 90.08 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 90.08 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | tr: UD_Turkish-BOUN | raw | Tokens | 99.56 [99.41, 99.71] | 99.56 [99.41, 99.71] | 99.58 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.58 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | tr: UD_Turkish-BOUN | raw | UAS | 69.75 [68.39, 71.06] | 71.15 [69.82, 72.44] | 73.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 71.53 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | tr: UD_Turkish-BOUN | raw | UFeats | 80.24 [79.44, 81.03] | 80.23 [79.44, 81.03] | 81.66 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 81.66 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | tr: UD_Turkish-BOUN | raw | UPOS | 89.08 [88.46, 89.72] | 88.62 [87.99, 89.32] | 90.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.92 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | tr: UD_Turkish-BOUN | raw | Words | 96.71 [96.34, 97.08] | 96.71 [96.34, 97.08] | 96.57 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | 96.32 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | tr: UD_Turkish-BOUN | raw | XPOS | not measured | 43.51 [42.62, 44.33] | 86.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 85.74 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | gold | AllTags | not measured | 54.59 [53.68, 55.49] | 81.17 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | gold | CLAS | not measured | 46.41 [45.08, 47.78] | not published | not published | | ug: UD_Uyghur-UDT | gold | LAS | not measured | 51.62 [50.37, 52.94] | 66.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | gold | Lemmas | not measured | 91.36 [90.75, 91.99] | 95.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | gold | MLAS | not measured | 23.01 [21.96, 24.06] | 51.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | gold | UAS | not measured | 65.46 [64.08, 66.72] | 77.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | gold | UFeats | not measured | 65.89 [65.03, 66.75] | 88.51 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | gold | UPOS | not measured | 78.97 [78.20, 79.76] | 90.26 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | gold | XPOS | not measured | 81.13 [80.38, 81.97] | 92.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | ug: UD_Uyghur-UDT | raw | AllTags | not measured | 56.36 [55.37, 57.27] | 80.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 80.06 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | CLAS | not measured | 53.55 [52.19, 55.00] | 66.06 (Trankit large, UD 2.5; test same size) — **MISS** | 58.96 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | LAS | not measured | 58.25 [56.97, 59.60] | 70.14 (Trankit large, UD 2.5; test same size) — **MISS** | 63.42 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | Lemmas | not measured | 91.17 [90.51, 91.81] | 95.30 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 95.30 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | MLAS | not measured | 27.61 [26.39, 28.77] | 51.63 (Trankit large, UD 2.5; test same size) — **MISS** | 48.01 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | Sentences | not measured | 77.41 [74.83, 80.10] | 88.95 (Trankit large, UD 2.5; test same size) — **MISS** | 84.69 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | Tokens | not measured | 99.57 [99.35, 99.75] | 99.42 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.42 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | ug: UD_Uyghur-UDT | raw | UAS | not measured | 72.12 [70.81, 73.47] | 80.19 (Trankit large, UD 2.5; test same size) — **MISS** | 74.80 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | UFeats | not measured | 67.20 [66.28, 68.12] | 88.03 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 87.58 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | UPOS | not measured | 80.12 [79.30, 80.95] | 89.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 89.26 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | ug: UD_Uyghur-UDT | raw | Words | not measured | 99.57 [99.35, 99.75] | 99.54 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | 99.42 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | ug: UD_Uyghur-UDT | raw | XPOS | not measured | 82.75 [82.01, 83.52] | 91.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 91.19 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | uk: UD_Ukrainian-ParlaMint | gold | AllTags | not measured | 74.30 [73.28, 75.29] | 93.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | gold | CLAS | not measured | 79.88 [78.76, 81.01] | not published | not published | | uk: UD_Ukrainian-ParlaMint | gold | LAS | not measured | 81.97 [80.94, 82.94] | 91.92 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | gold | Lemmas | not measured | 95.04 [94.60, 95.51] | 99.12 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | gold | MLAS | not measured | 52.19 [50.73, 53.66] | 82.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | gold | UAS | not measured | 86.02 [85.07, 86.91] | 93.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | gold | UFeats | not measured | 76.60 [75.66, 77.59] | 95.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | gold | UPOS | not measured | 94.32 [93.84, 94.79] | 98.60 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | gold | XPOS | not measured | 95.96 [95.55, 96.34] | 98.49 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | AllTags | not measured | 76.62 [75.63, 77.58] | 93.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | CLAS | not measured | 84.71 [83.72, 85.70] | not published | not published | | uk: UD_Ukrainian-ParlaMint | raw | LAS | not measured | 87.15 [86.25, 87.96] | 91.91 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | Lemmas | not measured | 95.83 [95.46, 96.22] | 99.07 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | MLAS | not measured | 57.47 [56.07, 58.80] | 82.96 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | Sentences | not measured | 91.06 [89.16, 92.92] | 99.81 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | Tokens | not measured | 99.96 [99.91, 100.00] | not published | not published | | uk: UD_Ukrainian-ParlaMint | raw | UAS | not measured | 90.59 [89.79, 91.34] | 93.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | UFeats | not measured | 78.33 [77.39, 79.27] | 95.01 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | UPOS | not measured | 94.91 [94.48, 95.32] | 98.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uk: UD_Ukrainian-ParlaMint | raw | Words | not measured | 99.96 [99.91, 100.00] | 99.95 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | uk: UD_Ukrainian-ParlaMint | raw | XPOS | not measured | 96.74 [96.40, 97.08] | 98.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | gold | AllTags | not measured | 52.94 [50.05, 55.77] | 60.21 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | gold | CLAS | not measured | 47.89 [44.82, 50.61] | not published | not published | | uz: UD_Uzbek-UzUDT | gold | LAS | not measured | 52.10 [49.27, 54.95] | 57.97 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | gold | Lemmas | not measured | 78.41 [76.68, 80.28] | 29.22 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | gold | MLAS | not measured | 21.68 [19.53, 24.18] | 30.19 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | gold | UAS | not measured | 67.80 [65.04, 70.66] | 73.63 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | gold | UFeats | not measured | 61.87 [59.49, 64.40] | 70.49 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | gold | UPOS | not measured | 84.77 [83.18, 86.33] | 86.24 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | gold | XPOS | not measured | 83.36 [80.32, 86.10] | 85.58 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | AllTags | not measured | 54.86 [52.05, 57.56] | 60.26 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | raw | CLAS | not measured | 52.43 [49.35, 55.39] | not published | not published | | uz: UD_Uzbek-UzUDT | raw | LAS | not measured | 56.59 [53.81, 59.44] | 55.78 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | Lemmas | not measured | 80.14 [78.47, 81.87] | 29.27 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | MLAS | not measured | 26.44 [23.96, 29.29] | 28.87 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | Sentences | not measured | 70.95 [64.41, 76.45] | 72.26 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | uz: UD_Uzbek-UzUDT | raw | UAS | not measured | 71.68 [68.90, 74.38] | 71.16 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | UFeats | not measured | 63.97 [61.65, 66.26] | 70.68 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — **MISS** | not published | | uz: UD_Uzbek-UzUDT | raw | UPOS | not measured | 86.54 [85.03, 88.04] | 86.15 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | Words | not measured | 100.00 [100.00, 100.00] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | uz: UD_Uzbek-UzUDT | raw | XPOS | not measured | 85.33 [82.30, 88.01] | 85.25 (UDPipe 2 2.17 models 251125, UD 2.17; test changed) — within or ahead | not published | | vi: UD_Vietnamese-VTB | gold | AllTags | not measured | 86.23 [85.62, 86.85] | 88.59 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | vi: UD_Vietnamese-VTB | gold | CLAS | not measured | 57.91 [56.54, 59.19] | not published | not published | | vi: UD_Vietnamese-VTB | gold | LAS | not measured | 61.70 [60.51, 62.95] | 65.51 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | vi: UD_Vietnamese-VTB | gold | Lemmas | not measured | 99.30 [99.15, 99.44] | 99.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | vi: UD_Vietnamese-VTB | gold | MLAS | not measured | 48.86 [47.61, 50.15] | 55.15 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | vi: UD_Vietnamese-VTB | gold | UAS | not measured | 73.55 [72.37, 74.73] | 75.84 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | vi: UD_Vietnamese-VTB | gold | UPOS | not measured | 89.63 [89.04, 90.16] | 89.92 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | vi: UD_Vietnamese-VTB | gold | XPOS | not measured | 87.77 [87.21, 88.33] | 88.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | vi: UD_Vietnamese-VTB | raw | AllTags | not measured | 66.08 [65.06, 67.23] | 88.07 (Trankit large, UD 2.5; test changed) — **MISS** | 80.56 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | CLAS | not measured | 34.08 [32.58, 35.61] | 63.67 (Trankit large, UD 2.5; test changed) — **MISS** | 46.60 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | LAS | not measured | 38.80 [37.32, 40.34] | 65.37 (Trankit large, UD 2.5; test changed) — **MISS** | 51.49 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | Lemmas | not measured | 74.75 [73.78, 75.75] | 95.22 (Trankit base, UD 2.5; test changed) — **MISS** | 81.66 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | MLAS | not measured | 29.54 [28.16, 30.91] | 60.22 (Trankit large, UD 2.5; test changed) — **MISS** | 42.58 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | Sentences | not measured | 90.38 [88.34, 92.39] | 99.25 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.25 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | Tokens | not measured | 75.25 [74.26, 76.27] | 95.22 (Trankit base, UD 2.5; test changed) — **MISS** | 88.02 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | UAS | not measured | 44.71 [43.10, 46.33] | 71.07 (Trankit large, UD 2.5; test changed) — **MISS** | 58.82 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | UPOS | not measured | 68.68 [67.63, 69.75] | 89.70 (Trankit large, UD 2.5; test changed) — **MISS** | 81.48 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | Words | not measured | 75.25 [74.26, 76.27] | 95.22 (Trankit base, UD 2.5; test changed) — **MISS** | 88.02 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | vi: UD_Vietnamese-VTB | raw | XPOS | not measured | 67.17 [66.14, 68.31] | 88.14 (Trankit large, UD 2.5; test changed) — **MISS** | 80.61 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | gold | AllTags | not measured | 65.36 [64.21, 66.41] | 92.39 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | gold | CLAS | not measured | 44.54 [42.87, 46.19] | not published | not published | | wo: UD_Wolof-WTB | gold | LAS | not measured | 52.57 [51.05, 54.17] | 80.16 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | gold | Lemmas | not measured | 87.49 [86.82, 88.16] | 95.94 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | gold | MLAS | not measured | 27.49 [26.14, 28.82] | 68.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | gold | UAS | not measured | 63.82 [62.37, 65.41] | 85.75 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | gold | UFeats | not measured | 72.32 [71.33, 73.19] | 94.33 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | gold | UPOS | not measured | 79.61 [78.71, 80.53] | 95.29 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | gold | XPOS | not measured | 79.43 [78.38, 80.41] | 95.14 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | wo: UD_Wolof-WTB | raw | AllTags | not measured | 64.26 [63.16, 65.31] | 91.44 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 91.38 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | CLAS | not measured | 48.57 [46.94, 50.19] | 71.22 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 71.22 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | LAS | not measured | 56.10 [54.60, 57.59] | 78.02 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 77.32 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | Lemmas | not measured | 84.43 [83.62, 85.26] | 95.18 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.63 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | MLAS | not measured | 30.16 [28.77, 31.48] | 66.05 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 64.89 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | Sentences | not measured | 91.55 [88.99, 93.80] | 94.33 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 94.33 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | Tokens | not measured | 99.93 [99.84, 99.99] | 99.95 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | 99.95 (Stanza 1.5.1, UD 2.12; test same size) — within or ahead | | wo: UD_Wolof-WTB | raw | UAS | not measured | 66.01 [64.54, 67.58] | 83.49 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 83.28 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | UFeats | not measured | 70.25 [69.32, 71.19] | 93.47 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 93.19 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | UPOS | not measured | 78.63 [77.67, 79.69] | 94.36 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 94.36 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | Words | not measured | 95.18 [94.64, 95.76] | 99.35 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | 99.35 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | wo: UD_Wolof-WTB | raw | XPOS | not measured | 78.48 [77.53, 79.42] | 94.24 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | 94.16 (Stanza 1.5.1, UD 2.12; test same size) — **MISS** | | yi: UD_Yiddish-YiTB | gold | CLAS | not measured | 75.56 [73.82, 77.36] | not published | not published | | yi: UD_Yiddish-YiTB | gold | LAS | not measured | 81.00 [79.64, 82.38] | 86.48 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | gold | Lemmas | not measured | 91.14 [90.52, 91.75] | 93.87 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | gold | MLAS | not measured | 65.50 [63.65, 67.37] | 76.74 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | gold | UAS | not measured | 86.40 [85.18, 87.61] | 90.34 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | gold | UFeats | not measured | 100.00 [100.00, 100.00] | 100.00 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — within or ahead | not published | | yi: UD_Yiddish-YiTB | gold | UPOS | not measured | 91.95 [91.35, 92.53] | 94.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | CLAS | not measured | 76.89 [75.13, 78.55] | not published | not published | | yi: UD_Yiddish-YiTB | raw | LAS | not measured | 80.77 [79.38, 82.11] | 85.56 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | Lemmas | not measured | 88.37 [87.63, 89.06] | 93.53 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | MLAS | not measured | 68.47 [66.73, 70.19] | 75.85 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | Sentences | not measured | 94.73 [93.41, 96.01] | 97.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | Tokens | not measured | 98.32 [97.95, 98.70] | not published | not published | | yi: UD_Yiddish-YiTB | raw | UAS | not measured | 85.44 [84.17, 86.74] | 89.38 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | UFeats | not measured | 96.17 [95.63, 96.70] | 99.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | UPOS | not measured | 89.78 [89.11, 90.49] | 94.50 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | yi: UD_Yiddish-YiTB | raw | Words | not measured | 96.17 [95.63, 96.70] | 99.61 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | AllTags | 85.15 [84.47, 85.82] | 83.66 [82.95, 84.39] | 95.20 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | CLAS | not measured | 59.72 [58.32, 61.19] | not published | not published | | zh: UD_Chinese-GSDSimp | gold | LAS | 63.29 [61.88, 64.70] | 61.42 [60.21, 62.70] | 83.60 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | Lemmas | 99.80 [99.71, 99.88] | 99.78 [99.67, 99.86] | 99.90 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | MLAS | not measured | 47.94 [46.40, 49.45] | 77.62 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | UAS | 69.15 [67.81, 70.55] | 67.22 [66.03, 68.48] | 86.76 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | UFeats | 96.03 [95.69, 96.39] | 95.86 [95.48, 96.23] | 99.28 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | UPOS | 88.77 [88.18, 89.35] | 87.55 [86.92, 88.22] | 95.73 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | gold | XPOS | 87.86 [87.24, 88.47] | 87.14 [86.47, 87.79] | 95.97 (UDPipe 2 2.17 models 251125, UD 2.17; test same size) — **MISS** | not published | | zh: UD_Chinese-GSDSimp | raw | AllTags | 77.97 [77.04, 78.87] | 77.47 [76.54, 78.38] | 94.32 (Trankit large, UD 2.5; test changed) — **MISS** | 87.95 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | CLAS | not measured | 58.05 [56.40, 59.67] | 83.23 (Trankit large, UD 2.5; test changed) — **MISS** | 67.39 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | LAS | 58.84 [57.23, 60.35] | 60.98 [59.49, 62.45] | 84.63 (Trankit large, UD 2.5; test changed) — **MISS** | 70.65 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | Lemmas | 89.45 [88.61, 90.27] | 89.41 [88.61, 90.21] | 97.66 (Trankit large, UD 2.5; test changed) — **MISS** | 92.07 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | MLAS | not measured | 49.33 [47.69, 50.97] | 79.84 (Trankit large, UD 2.5; test changed) — **MISS** | 63.96 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | Sentences | 85.31 [82.10, 88.41] | 83.77 [80.32, 87.30] | 99.70 (Trankit base, UD 2.5; test changed) — **MISS** | 99.10 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | Tokens | 89.62 [88.81, 90.43] | 89.62 [88.81, 90.43] | 97.66 (Trankit large, UD 2.5; test changed) — **MISS** | 92.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | UAS | 62.78 [61.24, 64.26] | 64.42 [62.89, 65.87] | 87.01 (Trankit large, UD 2.5; test changed) — **MISS** | 73.41 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | UFeats | 85.94 [85.11, 86.79] | 85.98 [85.14, 86.82] | 97.24 (Trankit large, UD 2.5; test changed) — **MISS** | 91.47 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | UPOS | 80.96 [80.07, 81.81] | 80.66 [79.76, 81.48] | 94.99 (Trankit large, UD 2.5; test changed) — **MISS** | 88.82 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | Words | 89.62 [88.81, 90.43] | 89.62 [88.81, 90.43] | 97.66 (Trankit large, UD 2.5; test changed) — **MISS** | 92.14 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zh: UD_Chinese-GSDSimp | raw | XPOS | 80.11 [79.28, 80.98] | 79.75 [78.91, 80.63] | 94.78 (Trankit large, UD 2.5; test changed) — **MISS** | 88.61 (Stanza 1.5.1, UD 2.12; test changed) — **MISS** | | zza: UD_Zazaki-ZSD | gold | CLAS | not measured | 32.89 [25.34, 41.83] | not published | not published | | zza: UD_Zazaki-ZSD | gold | LAS | not measured | 39.60 [31.87, 48.95] | not published | not published | | zza: UD_Zazaki-ZSD | gold | Lemmas | not measured | 76.00 [69.52, 81.85] | not published | not published | | zza: UD_Zazaki-ZSD | gold | MLAS | not measured | 8.05 [3.88, 13.74] | not published | not published | | zza: UD_Zazaki-ZSD | gold | UAS | not measured | 56.00 [46.58, 66.49] | not published | not published | | zza: UD_Zazaki-ZSD | gold | UFeats | not measured | 51.20 [45.45, 57.41] | not published | not published | | zza: UD_Zazaki-ZSD | gold | UPOS | not measured | 63.20 [57.20, 69.49] | not published | not published | | zza: UD_Zazaki-ZSD | raw | CLAS | not measured | 33.00 [23.68, 41.45] | not published | not published | | zza: UD_Zazaki-ZSD | raw | LAS | not measured | 36.10 [26.64, 45.28] | not published | not published | | zza: UD_Zazaki-ZSD | raw | Lemmas | not measured | 67.63 [62.40, 72.44] | not published | not published | | zza: UD_Zazaki-ZSD | raw | MLAS | not measured | 8.08 [4.65, 11.95] | not published | not published | | zza: UD_Zazaki-ZSD | raw | Sentences | not measured | 83.33 [70.00, 93.55] | not published | not published | | zza: UD_Zazaki-ZSD | raw | Tokens | not measured | 100.00 [100.00, 100.00] | not published | not published | | zza: UD_Zazaki-ZSD | raw | UAS | not measured | 51.04 [39.65, 61.45] | not published | not published | | zza: UD_Zazaki-ZSD | raw | UFeats | not measured | 44.81 [40.29, 49.63] | not published | not published | | zza: UD_Zazaki-ZSD | raw | UPOS | not measured | 56.85 [52.63, 61.44] | not published | not published | | zza: UD_Zazaki-ZSD | raw | Words | not measured | 88.80 [83.68, 93.88] | not published | not published | ## Entity samples (gold tokens) Deterministic evenly spaced samples, at most 3,001 sentences per corpus. The published UD references above do not supply comparable entity predictions; these rows are information. The paired spaCy view below uses identical sampled sentences. | Language | Metric | Before [95% CI] | Current [95% CI] | |---|---|---|---| | da | micro-F1 | not measured | 82.86 [78.88, 86.39] | | da | micro-precision | not measured | 83.91 [79.61, 88.03] | | da | micro-recall | not measured | 81.84 [77.26, 85.71] | | de | micro-F1 | 82.68 [81.18, 84.04] | 80.96 [79.47, 82.28] | | de | micro-precision | 85.82 [84.28, 87.23] | 81.02 [79.37, 82.39] | | de | micro-recall | 79.77 [78.03, 81.32] | 80.89 [79.30, 82.45] | | en | micro-F1 | 79.07 [76.69, 81.48] | 77.46 [74.74, 79.96] | | en | micro-precision | 86.13 [83.63, 88.42] | 81.17 [77.98, 84.08] | | en | micro-recall | 73.07 [70.21, 76.07] | 74.08 [71.08, 76.90] | | es | micro-F1 | 85.58 [84.44, 86.59] | 84.78 [83.64, 85.84] | | es | micro-precision | 85.97 [84.88, 87.00] | 84.98 [83.82, 86.02] | | es | micro-recall | 85.19 [84.02, 86.29] | 84.59 [83.41, 85.69] | | fr | micro-F1 | 81.56 [80.48, 82.75] | 80.96 [79.84, 82.17] | | fr | micro-precision | 82.11 [80.98, 83.29] | 80.92 [79.71, 82.07] | | fr | micro-recall | 81.02 [79.97, 82.24] | 81.01 [79.87, 82.22] | | hr | micro-F1 | not measured | 81.87 [79.66, 84.04] | | hr | micro-precision | not measured | 82.03 [79.67, 84.46] | | hr | micro-recall | not measured | 81.71 [79.41, 83.67] | | id | micro-F1 | not measured | 78.39 [74.93, 81.32] | | id | micro-precision | not measured | 82.68 [79.44, 85.53] | | id | micro-recall | not measured | 74.52 [70.15, 78.59] | | it | micro-F1 | 86.30 [85.27, 87.25] | 84.74 [83.65, 85.74] | | it | micro-precision | 86.57 [85.54, 87.48] | 84.84 [83.74, 85.84] | | it | micro-recall | 86.03 [84.96, 87.02] | 84.64 [83.48, 85.69] | | nl | micro-F1 | 85.18 [84.06, 86.17] | 83.70 [82.61, 84.70] | | nl | micro-precision | 84.85 [83.74, 85.88] | 83.16 [82.06, 84.23] | | nl | micro-recall | 85.52 [84.42, 86.50] | 84.24 [83.13, 85.25] | | no | micro-F1 | not measured | 73.18 [70.65, 75.53] | | no | micro-precision | not measured | 69.24 [66.62, 71.93] | | no | micro-recall | not measured | 77.60 [75.15, 79.85] | | pl | micro-F1 | not measured | 67.67 [65.13, 70.05] | | pl | micro-precision | not measured | 72.38 [69.44, 75.06] | | pl | micro-recall | not measured | 63.54 [60.98, 66.21] | | pt | micro-F1 | 88.28 [86.43, 90.13] | 88.30 [86.58, 90.01] | | pt | micro-precision | 90.14 [88.16, 92.00] | 89.04 [87.14, 90.98] | | pt | micro-recall | 86.50 [84.43, 88.59] | 87.57 [85.56, 89.54] | | ro | micro-F1 | not measured | 67.70 [66.54, 68.89] | | ro | micro-precision | not measured | 70.80 [69.39, 72.13] | | ro | micro-recall | not measured | 64.87 [63.55, 66.15] | | sk | micro-F1 | not measured | 65.10 [61.62, 68.69] | | sk | micro-precision | not measured | 63.51 [59.46, 67.53] | | sk | micro-recall | not measured | 66.78 [63.39, 70.22] | | sl | micro-F1 | not measured | 68.03 [64.66, 71.79] | | sl | micro-precision | not measured | 70.97 [67.36, 74.92] | | sl | micro-recall | not measured | 65.32 [61.72, 69.26] | | sr | micro-F1 | not measured | 89.92 [87.89, 91.79] | | sr | micro-precision | not measured | 91.79 [89.78, 93.63] | | sr | micro-recall | not measured | 88.12 [85.75, 90.43] | | sv | micro-F1 | not measured | 69.35 [61.24, 76.73] | | sv | micro-precision | not measured | 88.19 [81.97, 93.48] | | sv | micro-recall | not measured | 57.14 [47.95, 65.96] | | tr | micro-F1 | 83.55 [81.59, 85.40] | 82.84 [80.81, 84.79] | | tr | micro-precision | 85.68 [83.58, 87.77] | 82.65 [80.52, 84.71] | | tr | micro-recall | 81.51 [79.30, 83.56] | 83.03 [80.86, 85.07] | | zh | micro-F1 | 84.84 [81.89, 87.94] | 84.04 [81.02, 86.97] | | zh | micro-precision | 86.51 [83.49, 89.63] | 84.59 [81.42, 87.57] | | zh | micro-recall | 83.22 [79.76, 86.69] | 83.49 [79.86, 86.79] | ## Paired spaCy 3.8.18 comparison (gate) 144 rows: 117 ahead, 14 within the paired interval, 13 behind. Same gold, mode and sampled entity sentences; 1,000 paired resamples, seed 20261106. spaCy runs its documented default configuration on the same training data. A row passes when it is ahead or within the paired interval; the 13 rows behind are gaps. Positive differences favor this model. | Language / dataset | Input | Metric | Current [95% CI] | spaCy [95% CI] | Difference [95% CI] | Verdict | |---|---|---|---|---|---|---| | de: de_gsd-ud-test | gold | UPOS | 95.78 [95.43, 96.12] | 94.59 [94.22, 94.97] | 1.19 [0.80, 1.55] | ahead | | de: de_gsd-ud-test | gold | UFeats | 86.58 [85.91, 87.19] | 84.71 [84.07, 85.40] | 1.87 [1.16, 2.60] | ahead | | de: de_gsd-ud-test | gold | Lemmas | 96.64 [96.37, 96.92] | 95.65 [95.32, 95.97] | 0.99 [0.71, 1.27] | ahead | | de: de_gsd-ud-test | gold | UAS | 87.09 [86.26, 87.84] | 81.88 [80.78, 82.88] | 5.21 [4.40, 6.16] | ahead | | de: de_gsd-ud-test | gold | LAS | 83.49 [82.59, 84.31] | 77.31 [76.19, 78.32] | 6.18 [5.35, 7.14] | ahead | | de: de_gsd-ud-test | raw | Tokens | 97.40 [97.09, 97.69] | 98.03 [97.67, 98.36] | -0.63 [-1.04, -0.21] | behind | | de: de_gsd-ud-test | raw | Sentences | 88.65 [86.73, 90.58] | 78.73 [76.35, 81.37] | 9.91 [7.13, 12.72] | ahead | | de: de_gsd-ud-test | raw | Words | 97.06 [96.70, 97.39] | 95.51 [95.03, 95.96] | 1.55 [1.22, 1.89] | ahead | | de: de_gsd-ud-test | raw | UPOS | 93.33 [92.87, 93.75] | 90.13 [89.52, 90.69] | 3.20 [2.76, 3.67] | ahead | | de: de_gsd-ud-test | raw | UFeats | 85.16 [84.51, 85.74] | 80.11 [79.31, 80.89] | 5.05 [4.33, 5.79] | ahead | | de: de_gsd-ud-test | raw | Lemmas | 94.07 [93.64, 94.48] | 91.28 [90.71, 91.79] | 2.80 [2.36, 3.25] | ahead | | de: de_gsd-ud-test | raw | UAS | 85.50 [84.64, 86.25] | 75.30 [74.13, 76.35] | 10.19 [9.36, 11.13] | ahead | | de: de_gsd-ud-test | raw | LAS | 82.38 [81.55, 83.18] | 70.77 [69.60, 71.89] | 11.61 [10.73, 12.52] | ahead | | en: en_ewt-ud-test | gold | UPOS | 96.49 [96.19, 96.76] | 94.29 [93.88, 94.65] | 2.20 [1.86, 2.57] | ahead | | en: en_ewt-ud-test | gold | UFeats | 96.37 [96.09, 96.62] | 95.18 [94.87, 95.49] | 1.19 [0.89, 1.50] | ahead | | en: en_ewt-ud-test | gold | Lemmas | 97.26 [97.01, 97.52] | 95.80 [95.45, 96.12] | 1.47 [1.21, 1.76] | ahead | | en: en_ewt-ud-test | gold | UAS | 88.34 [87.65, 89.01] | 85.40 [84.51, 86.30] | 2.93 [2.07, 3.81] | ahead | | en: en_ewt-ud-test | gold | LAS | 85.12 [84.40, 85.87] | 81.96 [81.00, 82.88] | 3.16 [2.30, 4.00] | ahead | | en: en_ewt-ud-test | raw | Tokens | 97.13 [96.83, 97.43] | 95.70 [95.17, 96.16] | 1.43 [1.00, 1.93] | ahead | | en: en_ewt-ud-test | raw | Sentences | 75.85 [74.20, 77.60] | 72.66 [70.73, 74.67] | 3.19 [1.22, 5.36] | ahead | | en: en_ewt-ud-test | raw | Words | 98.68 [98.45, 98.90] | 97.48 [96.99, 97.96] | 1.21 [0.78, 1.69] | ahead | | en: en_ewt-ud-test | raw | UPOS | 95.95 [95.59, 96.27] | 92.45 [91.86, 93.05] | 3.49 [2.96, 4.08] | ahead | | en: en_ewt-ud-test | raw | UFeats | 95.94 [95.61, 96.27] | 93.37 [92.80, 93.92] | 2.57 [2.07, 3.12] | ahead | | en: en_ewt-ud-test | raw | Lemmas | 96.39 [96.04, 96.74] | 93.69 [93.08, 94.28] | 2.70 [2.18, 3.28] | ahead | | en: en_ewt-ud-test | raw | UAS | 88.78 [88.07, 89.47] | 81.34 [80.35, 82.35] | 7.44 [6.58, 8.27] | ahead | | en: en_ewt-ud-test | raw | LAS | 86.55 [85.82, 87.26] | 78.53 [77.47, 79.56] | 8.01 [7.20, 8.84] | ahead | | es: es_ancora-ud-test | gold | UPOS | 98.00 [97.87, 98.14] | 98.18 [98.03, 98.32] | -0.18 [-0.33, -0.03] | behind | | es: es_ancora-ud-test | gold | UFeats | 96.74 [96.58, 96.91] | 97.74 [97.58, 97.89] | -1.00 [-1.17, -0.83] | behind | | es: es_ancora-ud-test | gold | Lemmas | 98.53 [98.42, 98.64] | 98.72 [98.60, 98.83] | -0.19 [-0.30, -0.07] | behind | | es: es_ancora-ud-test | gold | UAS | 89.50 [89.08, 89.94] | 89.04 [88.53, 89.51] | 0.46 [0.02, 0.91] | ahead | | es: es_ancora-ud-test | gold | LAS | 86.97 [86.51, 87.42] | 85.95 [85.40, 86.43] | 1.03 [0.56, 1.50] | ahead | | es: es_ancora-ud-test | raw | Tokens | 99.84 [99.76, 99.90] | 99.90 [99.85, 99.94] | -0.06 [-0.13, 0.01] | within | | es: es_ancora-ud-test | raw | Sentences | 97.05 [96.24, 97.78] | 97.70 [96.94, 98.43] | -0.66 [-1.59, 0.28] | within | | es: es_ancora-ud-test | raw | Words | 99.83 [99.75, 99.89] | 96.60 [96.38, 96.80] | 3.23 [3.02, 3.45] | ahead | | es: es_ancora-ud-test | raw | UPOS | 98.20 [98.07, 98.34] | 94.81 [94.55, 95.05] | 3.40 [3.15, 3.64] | ahead | | es: es_ancora-ud-test | raw | UFeats | 96.83 [96.66, 96.99] | 94.57 [94.32, 94.81] | 2.25 [2.01, 2.50] | ahead | | es: es_ancora-ud-test | raw | Lemmas | 98.58 [98.46, 98.70] | 95.37 [95.13, 95.59] | 3.21 [2.98, 3.45] | ahead | | es: es_ancora-ud-test | raw | UAS | 91.93 [91.49, 92.31] | 84.87 [84.27, 85.43] | 7.06 [6.56, 7.55] | ahead | | es: es_ancora-ud-test | raw | LAS | 89.70 [89.24, 90.11] | 81.82 [81.22, 82.40] | 7.89 [7.37, 8.38] | ahead | | fr: fr_gsd-ud-test | gold | UPOS | 96.99 [96.60, 97.39] | 97.07 [96.68, 97.47] | -0.09 [-0.49, 0.34] | within | | fr: fr_gsd-ud-test | gold | UFeats | 96.06 [95.58, 96.49] | 96.41 [95.93, 96.85] | -0.35 [-0.90, 0.19] | within | | fr: fr_gsd-ud-test | gold | Lemmas | 98.03 [97.72, 98.31] | 97.49 [97.13, 97.81] | 0.54 [0.22, 0.88] | ahead | | fr: fr_gsd-ud-test | gold | UAS | 89.52 [88.61, 90.43] | 89.10 [88.14, 90.12] | 0.42 [-0.58, 1.43] | within | | fr: fr_gsd-ud-test | gold | LAS | 86.89 [85.93, 87.88] | 85.98 [84.84, 87.07] | 0.91 [-0.11, 1.95] | within | | fr: fr_gsd-ud-test | raw | Tokens | 99.88 [99.79, 99.96] | 98.85 [98.49, 99.16] | 1.02 [0.74, 1.37] | ahead | | fr: fr_gsd-ud-test | raw | Sentences | 92.65 [90.07, 94.98] | 91.82 [89.08, 94.23] | 0.83 [-1.20, 3.09] | within | | fr: fr_gsd-ud-test | raw | Words | 99.48 [99.29, 99.65] | 94.64 [94.01, 95.17] | 4.83 [4.31, 5.44] | ahead | | fr: fr_gsd-ud-test | raw | UPOS | 97.10 [96.71, 97.46] | 91.80 [91.06, 92.46] | 5.30 [4.66, 6.01] | ahead | | fr: fr_gsd-ud-test | raw | UFeats | 96.26 [95.82, 96.68] | 91.16 [90.40, 91.86] | 5.11 [4.40, 5.91] | ahead | | fr: fr_gsd-ud-test | raw | Lemmas | 97.96 [97.64, 98.25] | 92.20 [91.44, 92.88] | 5.76 [5.13, 6.48] | ahead | | fr: fr_gsd-ud-test | raw | UAS | 91.97 [91.12, 92.76] | 81.64 [80.34, 82.90] | 10.33 [9.24, 11.53] | ahead | | fr: fr_gsd-ud-test | raw | LAS | 89.78 [88.84, 90.63] | 78.10 [76.76, 79.31] | 11.68 [10.48, 12.84] | ahead | | it: it_markit-ud-test | gold | UPOS | 95.81 [95.39, 96.26] | 92.20 [91.61, 92.77] | 3.61 [3.05, 4.19] | ahead | | it: it_markit-ud-test | gold | UFeats | 92.12 [91.51, 92.73] | 90.11 [89.46, 90.77] | 2.01 [1.47, 2.63] | ahead | | it: it_markit-ud-test | gold | Lemmas | 86.38 [85.63, 87.08] | 83.80 [82.97, 84.52] | 2.58 [2.15, 3.03] | ahead | | it: it_markit-ud-test | gold | UAS | 82.05 [81.01, 83.15] | 75.39 [74.12, 76.63] | 6.66 [5.42, 7.96] | ahead | | it: it_markit-ud-test | gold | LAS | 77.63 [76.52, 78.78] | 69.27 [67.94, 70.57] | 8.36 [7.06, 9.75] | ahead | | it: it_markit-ud-test | raw | Tokens | 99.93 [99.86, 99.98] | 95.76 [95.28, 96.24] | 4.17 [3.69, 4.66] | ahead | | it: it_markit-ud-test | raw | Sentences | 90.11 [87.27, 92.93] | 91.62 [88.51, 94.91] | -1.51 [-5.49, 1.94] | within | | it: it_markit-ud-test | raw | Words | 99.73 [99.58, 99.86] | 88.66 [87.84, 89.43] | 11.07 [10.30, 11.88] | ahead | | it: it_markit-ud-test | raw | UPOS | 96.12 [95.75, 96.52] | 81.37 [80.37, 82.39] | 14.75 [13.79, 15.79] | ahead | | it: it_markit-ud-test | raw | UFeats | 92.84 [92.29, 93.42] | 81.05 [80.07, 82.01] | 11.79 [10.85, 12.68] | ahead | | it: it_markit-ud-test | raw | Lemmas | 86.65 [85.93, 87.35] | 73.64 [72.57, 74.71] | 13.01 [12.18, 13.89] | ahead | | it: it_markit-ud-test | raw | UAS | 86.39 [85.36, 87.29] | 62.66 [61.07, 64.19] | 23.73 [22.07, 25.38] | ahead | | it: it_markit-ud-test | raw | LAS | 82.22 [81.04, 83.28] | 56.71 [55.14, 58.22] | 25.51 [23.86, 27.23] | ahead | | de: germeval-2014 | entities (gold) | micro-F1 | 80.96 [79.47, 82.28] | 65.30 [63.38, 67.02] | 15.66 [13.73, 17.43] | ahead | | de: germeval-2014 | entities (gold) | micro-precision | 81.02 [79.37, 82.39] | 70.48 [68.42, 72.31] | 10.54 [8.39, 12.66] | ahead | | de: germeval-2014 | entities (gold) | micro-recall | 80.89 [79.30, 82.45] | 60.82 [58.79, 62.78] | 20.07 [18.06, 21.89] | ahead | | en: en_ewt-ud | entities (gold) | micro-F1 | 77.46 [74.74, 79.96] | 62.32 [59.32, 65.10] | 15.14 [12.00, 18.48] | ahead | | en: en_ewt-ud | entities (gold) | micro-precision | 81.17 [77.98, 84.08] | 67.97 [64.61, 70.86] | 13.20 [9.69, 17.10] | ahead | | en: en_ewt-ud | entities (gold) | micro-recall | 74.08 [71.08, 76.90] | 57.54 [54.28, 60.60] | 16.54 [12.93, 20.15] | ahead | | es: wikiner-es-wp3 | entities (gold) | micro-F1 | 84.78 [83.64, 85.84] | 83.21 [82.13, 84.22] | 1.57 [0.47, 2.70] | ahead | | es: wikiner-es-wp3 | entities (gold) | micro-precision | 84.98 [83.82, 86.02] | 83.79 [82.71, 84.80] | 1.19 [0.01, 2.35] | ahead | | es: wikiner-es-wp3 | entities (gold) | micro-recall | 84.59 [83.41, 85.69] | 82.64 [81.49, 83.69] | 1.95 [0.84, 3.08] | ahead | | fr: wikiner-fr-wp3 | entities (gold) | micro-F1 | 80.96 [79.84, 82.17] | 75.52 [74.20, 76.84] | 5.44 [4.14, 6.87] | ahead | | fr: wikiner-fr-wp3 | entities (gold) | micro-precision | 80.92 [79.71, 82.07] | 76.75 [75.45, 78.02] | 4.17 [2.76, 5.63] | ahead | | fr: wikiner-fr-wp3 | entities (gold) | micro-recall | 81.01 [79.87, 82.22] | 74.33 [72.91, 75.76] | 6.67 [5.35, 8.12] | ahead | | it: wikiner-it-wp3 | entities (gold) | micro-F1 | 84.74 [83.65, 85.74] | 83.58 [82.47, 84.52] | 1.16 [0.02, 2.24] | ahead | | it: wikiner-it-wp3 | entities (gold) | micro-precision | 84.84 [83.74, 85.84] | 84.08 [83.00, 85.06] | 0.76 [-0.43, 1.90] | within | | it: wikiner-it-wp3 | entities (gold) | micro-recall | 84.64 [83.48, 85.69] | 83.09 [81.90, 84.07] | 1.55 [0.41, 2.64] | ahead | | nl: wikiner-nl-wp3 | entities (gold) | micro-F1 | 83.70 [82.61, 84.70] | 86.64 [85.63, 87.59] | -2.94 [-4.05, -1.85] | behind | | nl: wikiner-nl-wp3 | entities (gold) | micro-precision | 83.16 [82.06, 84.23] | 86.28 [85.24, 87.33] | -3.12 [-4.21, -2.02] | behind | | nl: wikiner-nl-wp3 | entities (gold) | micro-recall | 84.24 [83.13, 85.25] | 87.00 [85.96, 87.99] | -2.76 [-3.93, -1.65] | behind | | pt: pt_bosque-ud | entities (gold) | micro-F1 | 88.30 [86.58, 90.01] | 75.27 [72.77, 77.68] | 13.03 [10.84, 15.42] | ahead | | pt: pt_bosque-ud | entities (gold) | micro-precision | 89.04 [87.14, 90.98] | 76.96 [74.27, 79.75] | 12.08 [9.54, 14.63] | ahead | | pt: pt_bosque-ud | entities (gold) | micro-recall | 87.57 [85.56, 89.54] | 73.66 [70.89, 76.30] | 13.91 [11.49, 16.67] | ahead | | tr: turkish-wikiner | entities (gold) | micro-F1 | 82.84 [80.81, 84.79] | 65.82 [63.46, 68.33] | 17.01 [14.38, 19.32] | ahead | | tr: turkish-wikiner | entities (gold) | micro-precision | 82.65 [80.52, 84.71] | 68.49 [65.67, 71.28] | 14.15 [11.16, 16.97] | ahead | | tr: turkish-wikiner | entities (gold) | micro-recall | 83.03 [80.86, 85.07] | 63.36 [60.70, 65.94] | 19.67 [16.92, 22.08] | ahead | | zh: zh_gsdsimp-ud | entities (gold) | micro-F1 | 84.04 [81.02, 86.97] | 61.86 [57.51, 66.06] | 22.17 [18.04, 26.66] | ahead | | zh: zh_gsdsimp-ud | entities (gold) | micro-precision | 84.59 [81.42, 87.57] | 67.08 [62.38, 71.86] | 17.52 [13.00, 22.28] | ahead | | zh: zh_gsdsimp-ud | entities (gold) | micro-recall | 83.49 [79.86, 86.79] | 57.40 [52.59, 62.05] | 26.08 [21.24, 31.29] | ahead | | nl: nl_alpino-ud-test | gold | UPOS | 96.91 [96.21, 97.44] | 94.42 [93.78, 95.02] | 2.49 [2.00, 3.00] | ahead | | nl: nl_alpino-ud-test | gold | UFeats | 96.62 [96.06, 97.12] | 93.13 [92.47, 93.70] | 3.49 [3.04, 3.95] | ahead | | nl: nl_alpino-ud-test | gold | Lemmas | 92.34 [91.80, 92.92] | 91.83 [91.24, 92.37] | 0.52 [0.15, 0.91] | ahead | | nl: nl_alpino-ud-test | gold | UAS | 90.23 [89.24, 91.11] | 84.09 [82.87, 85.31] | 6.14 [5.08, 7.25] | ahead | | nl: nl_alpino-ud-test | gold | LAS | 87.24 [86.15, 88.24] | 79.85 [78.57, 81.13] | 7.40 [6.20, 8.55] | ahead | | nl: nl_alpino-ud-test | raw | Tokens | 99.66 [99.46, 99.82] | 99.82 [99.70, 99.93] | -0.16 [-0.34, 0.01] | within | | nl: nl_alpino-ud-test | raw | Sentences | 95.90 [94.35, 97.34] | 89.60 [86.73, 92.03] | 6.30 [3.78, 9.05] | ahead | | nl: nl_alpino-ud-test | raw | Words | 99.66 [99.46, 99.82] | 99.82 [99.70, 99.93] | -0.16 [-0.34, 0.01] | within | | nl: nl_alpino-ud-test | raw | UPOS | 96.74 [96.11, 97.27] | 94.36 [93.70, 94.95] | 2.38 [1.88, 2.89] | ahead | | nl: nl_alpino-ud-test | raw | UFeats | 96.80 [96.29, 97.28] | 92.97 [92.32, 93.58] | 3.83 [3.33, 4.36] | ahead | | nl: nl_alpino-ud-test | raw | Lemmas | 92.05 [91.49, 92.67] | 91.65 [91.06, 92.23] | 0.40 [0.01, 0.86] | ahead | | nl: nl_alpino-ud-test | raw | UAS | 93.37 [92.55, 94.08] | 83.70 [82.50, 84.92] | 9.67 [8.60, 10.84] | ahead | | nl: nl_alpino-ud-test | raw | LAS | 91.11 [90.20, 91.92] | 79.57 [78.38, 80.82] | 11.54 [10.38, 12.58] | ahead | | pt: pt_bosque-ud-test | gold | UPOS | 96.10 [95.82, 96.36] | 95.92 [95.66, 96.19] | 0.18 [-0.12, 0.45] | within | | pt: pt_bosque-ud-test | gold | UFeats | 94.48 [94.16, 94.80] | 94.11 [93.79, 94.41] | 0.37 [0.04, 0.68] | ahead | | pt: pt_bosque-ud-test | gold | Lemmas | 97.59 [97.39, 97.79] | 96.90 [96.69, 97.10] | 0.69 [0.50, 0.90] | ahead | | pt: pt_bosque-ud-test | gold | UAS | 88.22 [87.63, 88.77] | 86.05 [85.48, 86.64] | 2.17 [1.54, 2.81] | ahead | | pt: pt_bosque-ud-test | gold | LAS | 84.54 [83.90, 85.14] | 81.96 [81.32, 82.63] | 2.58 [1.95, 3.26] | ahead | | pt: pt_bosque-ud-test | raw | Tokens | 99.86 [99.79, 99.92] | 99.87 [99.80, 99.93] | -0.01 [-0.06, 0.05] | within | | pt: pt_bosque-ud-test | raw | Sentences | 93.50 [92.14, 94.81] | 84.48 [82.18, 86.78] | 9.02 [6.41, 11.50] | ahead | | pt: pt_bosque-ud-test | raw | Words | 99.26 [99.14, 99.38] | 88.51 [88.07, 88.99] | 10.76 [10.24, 11.20] | ahead | | pt: pt_bosque-ud-test | raw | UPOS | 96.75 [96.50, 96.98] | 84.15 [83.63, 84.67] | 12.60 [12.06, 13.09] | ahead | | pt: pt_bosque-ud-test | raw | UFeats | 95.19 [94.90, 95.48] | 82.16 [81.65, 82.69] | 13.03 [12.45, 13.58] | ahead | | pt: pt_bosque-ud-test | raw | Lemmas | 97.29 [97.09, 97.50] | 85.35 [84.89, 85.85] | 11.94 [11.47, 12.38] | ahead | | pt: pt_bosque-ud-test | raw | UAS | 90.68 [90.16, 91.19] | 70.23 [69.40, 71.06] | 20.45 [19.54, 21.32] | ahead | | pt: pt_bosque-ud-test | raw | LAS | 87.98 [87.39, 88.55] | 65.85 [65.01, 66.74] | 22.12 [21.20, 23.03] | ahead | | tr: tr_boun-ud-test | gold | UPOS | 91.28 [90.72, 91.81] | 88.04 [87.40, 88.67] | 3.24 [2.56, 3.86] | ahead | | tr: tr_boun-ud-test | gold | UFeats | 80.29 [79.54, 81.01] | 78.30 [77.53, 79.14] | 2.00 [1.32, 2.67] | ahead | | tr: tr_boun-ud-test | gold | Lemmas | 86.00 [85.31, 86.61] | 80.24 [79.46, 80.94] | 5.76 [5.11, 6.36] | ahead | | tr: tr_boun-ud-test | gold | UAS | 74.44 [73.05, 75.76] | 67.44 [66.03, 68.71] | 7.00 [5.68, 8.35] | ahead | | tr: tr_boun-ud-test | gold | LAS | 67.13 [65.67, 68.45] | 57.87 [56.54, 59.16] | 9.25 [7.98, 10.53] | ahead | | tr: tr_boun-ud-test | raw | Tokens | 99.56 [99.41, 99.71] | 99.42 [99.26, 99.60] | 0.14 [0.00, 0.27] | ahead | | tr: tr_boun-ud-test | raw | Sentences | 78.82 [76.14, 81.22] | 82.09 [79.45, 84.67] | -3.28 [-6.27, -0.42] | behind | | tr: tr_boun-ud-test | raw | Words | 96.71 [96.34, 97.08] | 97.06 [96.71, 97.41] | -0.35 [-0.70, 0.02] | within | | tr: tr_boun-ud-test | raw | UPOS | 88.62 [87.99, 89.32] | 86.82 [86.12, 87.52] | 1.80 [1.14, 2.48] | ahead | | tr: tr_boun-ud-test | raw | UFeats | 80.23 [79.44, 81.03] | 77.20 [76.40, 78.07] | 3.02 [2.35, 3.70] | ahead | | tr: tr_boun-ud-test | raw | Lemmas | 84.42 [83.71, 85.09] | 78.83 [78.04, 79.58] | 5.59 [4.98, 6.21] | ahead | | tr: tr_boun-ud-test | raw | UAS | 71.15 [69.82, 72.44] | 63.96 [62.51, 65.35] | 7.19 [5.92, 8.56] | ahead | | tr: tr_boun-ud-test | raw | LAS | 64.31 [62.89, 65.78] | 55.86 [54.49, 57.22] | 8.45 [7.26, 9.75] | ahead | | zh: zh_gsdsimp-ud-test | gold | UPOS | 87.55 [86.92, 88.22] | 90.35 [89.68, 91.02] | -2.80 [-3.57, -1.97] | behind | | zh: zh_gsdsimp-ud-test | gold | UFeats | 95.86 [95.48, 96.23] | 98.43 [98.21, 98.64] | -2.57 [-2.98, -2.17] | behind | | zh: zh_gsdsimp-ud-test | gold | Lemmas | 99.78 [99.67, 99.86] | 99.91 [99.84, 99.96] | -0.13 [-0.22, -0.05] | behind | | zh: zh_gsdsimp-ud-test | gold | UAS | 67.22 [66.03, 68.48] | 73.11 [71.58, 74.68] | -5.89 [-7.60, -4.07] | behind | | zh: zh_gsdsimp-ud-test | gold | LAS | 61.42 [60.21, 62.70] | 68.24 [66.60, 69.90] | -6.83 [-8.61, -4.99] | behind | | zh: zh_gsdsimp-ud-test | raw | Tokens | 89.62 [88.81, 90.43] | 39.43 [38.65, 40.19] | 50.19 [48.99, 51.37] | ahead | | zh: zh_gsdsimp-ud-test | raw | Sentences | 83.77 [80.32, 87.30] | 46.93 [42.54, 51.47] | 36.84 [31.24, 42.38] | ahead | | zh: zh_gsdsimp-ud-test | raw | Words | 89.62 [88.81, 90.43] | 39.43 [38.65, 40.19] | 50.19 [48.99, 51.37] | ahead | | zh: zh_gsdsimp-ud-test | raw | UPOS | 80.66 [79.76, 81.48] | 34.92 [34.21, 35.61] | 45.73 [44.63, 46.91] | ahead | | zh: zh_gsdsimp-ud-test | raw | UFeats | 85.98 [85.14, 86.82] | 38.38 [37.61, 39.14] | 47.60 [46.37, 48.85] | ahead | | zh: zh_gsdsimp-ud-test | raw | Lemmas | 89.41 [88.61, 90.21] | 39.43 [38.65, 40.19] | 49.97 [48.77, 51.15] | ahead | | zh: zh_gsdsimp-ud-test | raw | UAS | 64.42 [62.89, 65.87] | 9.12 [8.48, 9.79] | 55.29 [53.62, 57.00] | ahead | | zh: zh_gsdsimp-ud-test | raw | LAS | 60.98 [59.49, 62.45] | 8.59 [7.96, 9.24] | 52.38 [50.74, 54.05] | ahead | ## Frozen-encoder parser recovery: original nine languages English and Italian retain their exact previous files and reuse the corresponding previous test results. The other seven final heads were read once in each input mode after development selection; no test-driven continuation. Other heads, the encoder and all other languages are unchanged. Intervals use 1,000 resamples, seed 20261106. | Dataset | Input | Metric | Earlier nine-language model [95% CI] | Joint 84-language model [95% CI] | After parser recovery [95% CI] | |---|---|---|---|---|---| | de_gsd-ud-test | gold | UPOS | 96.16 [95.83, 96.49] | 95.78 [95.43, 96.12] | 95.78 [95.43, 96.12] | | de_gsd-ud-test | gold | XPOS | 96.07 [95.71, 96.43] | 95.84 [95.48, 96.19] | 95.84 [95.48, 96.19] | | de_gsd-ud-test | gold | AllTags | 83.34 [82.57, 84.04] | 83.09 [82.34, 83.75] | 83.09 [82.34, 83.75] | | de_gsd-ud-test | gold | UFeats | 86.81 [86.11, 87.44] | 86.58 [85.91, 87.19] | 86.58 [85.91, 87.19] | | de_gsd-ud-test | gold | Lemmas | 96.25 [95.95, 96.53] | 96.64 [96.37, 96.92] | 96.64 [96.37, 96.92] | | de_gsd-ud-test | gold | UAS | 87.01 [86.18, 87.75] | 85.90 [84.99, 86.68] | 87.09 [86.26, 87.84] | | de_gsd-ud-test | gold | LAS | 83.32 [82.43, 84.17] | 82.17 [81.24, 83.02] | 83.49 [82.59, 84.31] | | de_gsd-ud-test | gold | CLAS | not measured | 77.15 [76.03, 78.29] | 78.67 [77.54, 79.74] | | de_gsd-ud-test | gold | MLAS | not measured | 58.17 [56.95, 59.56] | 59.58 [58.38, 60.86] | | de_gsd-ud-test | raw | Tokens | 97.40 [97.09, 97.69] | 97.40 [97.09, 97.69] | 97.40 [97.09, 97.69] | | de_gsd-ud-test | raw | Sentences | 88.70 [86.70, 90.74] | 88.63 [86.65, 90.56] | 88.65 [86.73, 90.58] | | de_gsd-ud-test | raw | Words | 97.06 [96.70, 97.39] | 97.06 [96.70, 97.39] | 97.06 [96.70, 97.39] | | de_gsd-ud-test | raw | UPOS | 93.27 [92.79, 93.69] | 93.33 [92.87, 93.75] | 93.33 [92.87, 93.75] | | de_gsd-ud-test | raw | XPOS | 93.38 [92.91, 93.83] | 93.35 [92.91, 93.79] | 93.35 [92.91, 93.79] | | de_gsd-ud-test | raw | AllTags | 81.04 [80.35, 81.70] | 81.69 [80.99, 82.33] | 81.69 [80.99, 82.33] | | de_gsd-ud-test | raw | UFeats | 84.39 [83.77, 85.03] | 85.16 [84.51, 85.74] | 85.16 [84.51, 85.74] | | de_gsd-ud-test | raw | Lemmas | 93.36 [92.90, 93.78] | 94.07 [93.64, 94.48] | 94.07 [93.64, 94.48] | | de_gsd-ud-test | raw | UAS | 84.71 [83.86, 85.52] | 84.31 [83.43, 85.14] | 85.50 [84.64, 86.25] | | de_gsd-ud-test | raw | LAS | 81.17 [80.27, 82.00] | 81.05 [80.15, 81.88] | 82.38 [81.55, 83.18] | | de_gsd-ud-test | raw | CLAS | not measured | 79.72 [78.63, 80.79] | 81.12 [80.13, 82.07] | | de_gsd-ud-test | raw | MLAS | not measured | 59.29 [58.13, 60.48] | 60.36 [59.24, 61.44] | | en_ewt-ud-test | gold | UPOS | 96.58 [96.26, 96.86] | 96.49 [96.19, 96.76] | 96.49 [96.19, 96.76] | | en_ewt-ud-test | gold | XPOS | 96.02 [95.69, 96.32] | 95.73 [95.40, 96.04] | 95.73 [95.40, 96.04] | | en_ewt-ud-test | gold | AllTags | 93.66 [93.25, 94.02] | 93.47 [93.07, 93.83] | 93.47 [93.07, 93.83] | | en_ewt-ud-test | gold | UFeats | 96.44 [96.17, 96.69] | 96.37 [96.09, 96.62] | 96.37 [96.09, 96.62] | | en_ewt-ud-test | gold | Lemmas | 97.09 [96.81, 97.36] | 97.26 [97.01, 97.52] | 97.26 [97.01, 97.52] | | en_ewt-ud-test | gold | UAS | 88.56 [87.81, 89.26] | 88.34 [87.65, 89.01] | 88.34 [87.65, 89.01] | | en_ewt-ud-test | gold | LAS | 85.43 [84.61, 86.19] | 85.12 [84.40, 85.87] | 85.12 [84.40, 85.87] | | en_ewt-ud-test | gold | CLAS | not measured | 81.48 [80.57, 82.45] | 81.48 [80.57, 82.45] | | en_ewt-ud-test | gold | MLAS | not measured | 75.10 [74.10, 76.11] | 75.10 [74.10, 76.11] | | en_ewt-ud-test | raw | Tokens | 97.13 [96.83, 97.43] | 97.13 [96.83, 97.43] | 97.13 [96.83, 97.43] | | en_ewt-ud-test | raw | Sentences | 75.28 [73.54, 77.14] | 75.85 [74.20, 77.60] | 75.85 [74.20, 77.60] | | en_ewt-ud-test | raw | Words | 98.68 [98.45, 98.90] | 98.68 [98.45, 98.90] | 98.68 [98.45, 98.90] | | en_ewt-ud-test | raw | UPOS | 95.95 [95.59, 96.29] | 95.95 [95.59, 96.27] | 95.95 [95.59, 96.27] | | en_ewt-ud-test | raw | XPOS | 95.22 [94.83, 95.58] | 95.18 [94.78, 95.54] | 95.18 [94.78, 95.54] | | en_ewt-ud-test | raw | AllTags | 93.14 [92.69, 93.57] | 93.39 [92.93, 93.80] | 93.39 [92.93, 93.80] | | en_ewt-ud-test | raw | UFeats | 95.76 [95.42, 96.07] | 95.94 [95.61, 96.27] | 95.94 [95.61, 96.27] | | en_ewt-ud-test | raw | Lemmas | 96.14 [95.81, 96.48] | 96.39 [96.04, 96.74] | 96.39 [96.04, 96.74] | | en_ewt-ud-test | raw | UAS | 87.32 [86.58, 88.01] | 88.78 [88.07, 89.47] | 88.78 [88.07, 89.47] | | en_ewt-ud-test | raw | LAS | 85.02 [84.25, 85.77] | 86.55 [85.82, 87.26] | 86.55 [85.82, 87.26] | | en_ewt-ud-test | raw | CLAS | not measured | 83.38 [82.51, 84.30] | 83.38 [82.51, 84.30] | | en_ewt-ud-test | raw | MLAS | not measured | 78.46 [77.50, 79.41] | 78.46 [77.50, 79.41] | | es_ancora-ud-test | gold | UPOS | 98.10 [97.98, 98.24] | 98.00 [97.87, 98.14] | 98.00 [97.87, 98.14] | | es_ancora-ud-test | gold | XPOS | 84.88 [84.41, 85.31] | 84.37 [83.90, 84.82] | 84.37 [83.90, 84.82] | | es_ancora-ud-test | gold | AllTags | 83.67 [83.20, 84.13] | 83.21 [82.74, 83.67] | 83.21 [82.74, 83.67] | | es_ancora-ud-test | gold | UFeats | 96.50 [96.33, 96.66] | 96.74 [96.58, 96.91] | 96.74 [96.58, 96.91] | | es_ancora-ud-test | gold | Lemmas | 97.72 [97.58, 97.85] | 98.53 [98.42, 98.64] | 98.53 [98.42, 98.64] | | es_ancora-ud-test | gold | UAS | 88.14 [87.68, 88.63] | 87.61 [87.18, 88.08] | 89.50 [89.08, 89.94] | | es_ancora-ud-test | gold | LAS | 85.37 [84.89, 85.88] | 84.66 [84.20, 85.15] | 86.97 [86.51, 87.42] | | es_ancora-ud-test | gold | CLAS | not measured | 79.49 [78.89, 80.12] | 82.53 [81.95, 83.14] | | es_ancora-ud-test | gold | MLAS | not measured | 72.39 [71.73, 73.14] | 75.87 [75.23, 76.53] | | es_ancora-ud-test | raw | Tokens | 99.84 [99.76, 99.90] | 99.84 [99.76, 99.90] | 99.84 [99.76, 99.90] | | es_ancora-ud-test | raw | Sentences | 96.66 [95.79, 97.49] | 96.54 [95.70, 97.31] | 97.05 [96.24, 97.78] | | es_ancora-ud-test | raw | Words | 99.83 [99.75, 99.89] | 99.83 [99.75, 99.89] | 99.83 [99.75, 99.89] | | es_ancora-ud-test | raw | UPOS | 98.09 [97.96, 98.24] | 98.20 [98.07, 98.34] | 98.20 [98.07, 98.34] | | es_ancora-ud-test | raw | XPOS | 85.63 [85.17, 86.09] | 85.87 [85.43, 86.33] | 85.87 [85.43, 86.33] | | es_ancora-ud-test | raw | AllTags | 84.50 [84.05, 84.94] | 84.84 [84.40, 85.29] | 84.84 [84.40, 85.29] | | es_ancora-ud-test | raw | UFeats | 96.57 [96.40, 96.73] | 96.83 [96.66, 96.99] | 96.83 [96.66, 96.99] | | es_ancora-ud-test | raw | Lemmas | 97.75 [97.61, 97.89] | 98.58 [98.46, 98.70] | 98.58 [98.46, 98.70] | | es_ancora-ud-test | raw | UAS | 90.08 [89.60, 90.51] | 90.92 [90.49, 91.33] | 91.93 [91.49, 92.31] | | es_ancora-ud-test | raw | LAS | 87.42 [86.90, 87.92] | 88.35 [87.88, 88.78] | 89.70 [89.24, 90.11] | | es_ancora-ud-test | raw | CLAS | not measured | 84.50 [83.92, 85.06] | 86.36 [85.78, 86.89] | | es_ancora-ud-test | raw | MLAS | not measured | 78.68 [78.03, 79.32] | 80.63 [79.97, 81.23] | | fr_gsd-ud-test | gold | UPOS | 97.21 [96.82, 97.58] | 96.99 [96.60, 97.39] | 96.99 [96.60, 97.39] | | fr_gsd-ud-test | gold | UFeats | 96.24 [95.75, 96.67] | 96.06 [95.58, 96.49] | 96.06 [95.58, 96.49] | | fr_gsd-ud-test | gold | Lemmas | 97.56 [97.23, 97.88] | 98.03 [97.72, 98.31] | 98.03 [97.72, 98.31] | | fr_gsd-ud-test | gold | UAS | 89.36 [88.49, 90.23] | 88.98 [88.06, 89.81] | 89.52 [88.61, 90.43] | | fr_gsd-ud-test | gold | LAS | 86.55 [85.57, 87.49] | 85.83 [84.90, 86.72] | 86.89 [85.93, 87.88] | | fr_gsd-ud-test | gold | CLAS | not measured | 80.98 [79.72, 82.20] | 82.47 [81.26, 83.85] | | fr_gsd-ud-test | gold | MLAS | not measured | 73.78 [72.25, 75.26] | 75.00 [73.45, 76.63] | | fr_gsd-ud-test | raw | Tokens | 99.88 [99.79, 99.96] | 99.88 [99.79, 99.96] | 99.88 [99.79, 99.96] | | fr_gsd-ud-test | raw | Sentences | 92.40 [89.74, 94.66] | 92.18 [89.49, 94.59] | 92.65 [90.07, 94.98] | | fr_gsd-ud-test | raw | Words | 99.48 [99.29, 99.65] | 99.48 [99.29, 99.65] | 99.48 [99.29, 99.65] | | fr_gsd-ud-test | raw | UPOS | 97.06 [96.71, 97.42] | 97.10 [96.71, 97.46] | 97.10 [96.71, 97.46] | | fr_gsd-ud-test | raw | UFeats | 96.07 [95.59, 96.52] | 96.26 [95.82, 96.68] | 96.26 [95.82, 96.68] | | fr_gsd-ud-test | raw | Lemmas | 97.42 [97.09, 97.77] | 97.96 [97.64, 98.25] | 97.96 [97.64, 98.25] | | fr_gsd-ud-test | raw | UAS | 90.67 [89.74, 91.58] | 91.34 [90.43, 92.18] | 91.97 [91.12, 92.76] | | fr_gsd-ud-test | raw | LAS | 88.07 [87.07, 89.05] | 88.96 [88.02, 89.84] | 89.78 [88.84, 90.63] | | fr_gsd-ud-test | raw | CLAS | not measured | 85.67 [84.45, 86.80] | 86.65 [85.41, 87.83] | | fr_gsd-ud-test | raw | MLAS | not measured | 79.66 [78.13, 81.14] | 80.57 [79.06, 81.91] | | it_markit-ud-test | gold | UPOS | 95.69 [95.26, 96.12] | 95.81 [95.39, 96.26] | 95.81 [95.39, 96.26] | | it_markit-ud-test | gold | XPOS | 96.31 [95.90, 96.68] | 96.26 [95.87, 96.66] | 96.26 [95.87, 96.66] | | it_markit-ud-test | gold | AllTags | 89.51 [88.83, 90.21] | 89.79 [89.12, 90.43] | 89.79 [89.12, 90.43] | | it_markit-ud-test | gold | UFeats | 91.91 [91.27, 92.50] | 92.12 [91.51, 92.73] | 92.12 [91.51, 92.73] | | it_markit-ud-test | gold | Lemmas | 86.37 [85.60, 87.06] | 86.38 [85.63, 87.08] | 86.38 [85.63, 87.08] | | it_markit-ud-test | gold | UAS | 81.82 [80.77, 82.86] | 82.05 [81.01, 83.15] | 82.05 [81.01, 83.15] | | it_markit-ud-test | gold | LAS | 77.27 [76.19, 78.38] | 77.63 [76.52, 78.78] | 77.63 [76.52, 78.78] | | it_markit-ud-test | gold | CLAS | not measured | 72.13 [70.70, 73.64] | 72.13 [70.70, 73.64] | | it_markit-ud-test | gold | MLAS | not measured | 57.58 [56.03, 59.25] | 57.58 [56.03, 59.25] | | it_markit-ud-test | raw | Tokens | 99.93 [99.86, 99.98] | 99.93 [99.86, 99.98] | 99.93 [99.86, 99.98] | | it_markit-ud-test | raw | Sentences | 90.61 [87.84, 93.57] | 90.11 [87.27, 92.93] | 90.11 [87.27, 92.93] | | it_markit-ud-test | raw | Words | 99.73 [99.58, 99.86] | 99.73 [99.58, 99.86] | 99.73 [99.58, 99.86] | | it_markit-ud-test | raw | UPOS | 95.98 [95.54, 96.36] | 96.12 [95.75, 96.52] | 96.12 [95.75, 96.52] | | it_markit-ud-test | raw | XPOS | 96.56 [96.18, 96.92] | 96.45 [96.05, 96.80] | 96.45 [96.05, 96.80] | | it_markit-ud-test | raw | AllTags | 90.13 [89.50, 90.80] | 90.50 [89.88, 91.10] | 90.50 [89.88, 91.10] | | it_markit-ud-test | raw | UFeats | 92.40 [91.81, 93.01] | 92.84 [92.29, 93.42] | 92.84 [92.29, 93.42] | | it_markit-ud-test | raw | Lemmas | 86.46 [85.69, 87.17] | 86.65 [85.93, 87.35] | 86.65 [85.93, 87.35] | | it_markit-ud-test | raw | UAS | 85.40 [84.26, 86.44] | 86.39 [85.36, 87.29] | 86.39 [85.36, 87.29] | | it_markit-ud-test | raw | LAS | 81.12 [79.93, 82.18] | 82.22 [81.04, 83.28] | 82.22 [81.04, 83.28] | | it_markit-ud-test | raw | CLAS | not measured | 77.84 [76.30, 79.29] | 77.84 [76.30, 79.29] | | it_markit-ud-test | raw | MLAS | not measured | 64.34 [62.76, 65.86] | 64.34 [62.76, 65.86] | | nl_alpino-ud-test | gold | UPOS | 96.89 [96.24, 97.44] | 96.91 [96.21, 97.44] | 96.91 [96.21, 97.44] | | nl_alpino-ud-test | gold | XPOS | 95.09 [94.32, 95.77] | 95.05 [94.26, 95.72] | 95.05 [94.26, 95.72] | | nl_alpino-ud-test | gold | AllTags | 94.31 [93.48, 95.00] | 94.13 [93.32, 94.87] | 94.13 [93.32, 94.87] | | nl_alpino-ud-test | gold | UFeats | 96.73 [96.16, 97.22] | 96.62 [96.06, 97.12] | 96.62 [96.06, 97.12] | | nl_alpino-ud-test | gold | Lemmas | 91.88 [91.32, 92.47] | 92.34 [91.80, 92.92] | 92.34 [91.80, 92.92] | | nl_alpino-ud-test | gold | UAS | 90.41 [89.40, 91.32] | 87.18 [86.11, 88.11] | 90.23 [89.24, 91.11] | | nl_alpino-ud-test | gold | LAS | 87.63 [86.63, 88.59] | 83.40 [82.24, 84.47] | 87.24 [86.15, 88.24] | | nl_alpino-ud-test | gold | CLAS | not measured | 78.51 [77.05, 79.84] | 82.88 [81.34, 84.21] | | nl_alpino-ud-test | gold | MLAS | not measured | 70.51 [68.69, 72.12] | 76.21 [74.34, 77.73] | | nl_alpino-ud-test | raw | Tokens | 99.66 [99.46, 99.82] | 99.66 [99.46, 99.82] | 99.66 [99.46, 99.82] | | nl_alpino-ud-test | raw | Sentences | 94.81 [92.78, 96.42] | 95.56 [94.03, 97.03] | 95.90 [94.35, 97.34] | | nl_alpino-ud-test | raw | Words | 99.66 [99.46, 99.82] | 99.66 [99.46, 99.82] | 99.66 [99.46, 99.82] | | nl_alpino-ud-test | raw | UPOS | 96.66 [96.01, 97.23] | 96.74 [96.11, 97.27] | 96.74 [96.11, 97.27] | | nl_alpino-ud-test | raw | XPOS | 95.03 [94.32, 95.67] | 95.27 [94.61, 95.94] | 95.27 [94.61, 95.94] | | nl_alpino-ud-test | raw | AllTags | 94.31 [93.58, 95.01] | 94.50 [93.79, 95.22] | 94.50 [93.79, 95.22] | | nl_alpino-ud-test | raw | UFeats | 96.78 [96.26, 97.27] | 96.80 [96.29, 97.28] | 96.80 [96.29, 97.28] | | nl_alpino-ud-test | raw | Lemmas | 91.54 [90.97, 92.14] | 92.05 [91.49, 92.67] | 92.05 [91.49, 92.67] | | nl_alpino-ud-test | raw | UAS | 91.67 [90.65, 92.52] | 91.93 [91.05, 92.69] | 93.37 [92.55, 94.08] | | nl_alpino-ud-test | raw | LAS | 89.06 [87.93, 89.96] | 89.09 [88.06, 89.97] | 91.11 [90.20, 91.92] | | nl_alpino-ud-test | raw | CLAS | not measured | 85.23 [83.88, 86.40] | 87.99 [86.70, 89.11] | | nl_alpino-ud-test | raw | MLAS | not measured | 79.56 [77.95, 81.03] | 82.24 [80.73, 83.63] | | pt_bosque-ud-test | gold | UPOS | 96.56 [96.32, 96.80] | 96.10 [95.82, 96.36] | 96.10 [95.82, 96.36] | | pt_bosque-ud-test | gold | UFeats | 94.80 [94.49, 95.09] | 94.48 [94.16, 94.80] | 94.48 [94.16, 94.80] | | pt_bosque-ud-test | gold | Lemmas | 97.49 [97.30, 97.70] | 97.59 [97.39, 97.79] | 97.59 [97.39, 97.79] | | pt_bosque-ud-test | gold | UAS | 87.64 [87.07, 88.25] | 85.88 [85.20, 86.50] | 88.22 [87.63, 88.77] | | pt_bosque-ud-test | gold | LAS | 84.01 [83.39, 84.63] | 81.83 [81.11, 82.53] | 84.54 [83.90, 85.14] | | pt_bosque-ud-test | gold | CLAS | not measured | 77.38 [76.47, 78.25] | 80.24 [79.51, 81.07] | | pt_bosque-ud-test | gold | MLAS | not measured | 67.49 [66.45, 68.46] | 70.29 [69.36, 71.22] | | pt_bosque-ud-test | raw | Tokens | 99.86 [99.79, 99.92] | 99.86 [99.79, 99.92] | 99.86 [99.79, 99.92] | | pt_bosque-ud-test | raw | Sentences | 92.27 [90.68, 93.82] | 92.57 [91.08, 93.99] | 93.50 [92.14, 94.81] | | pt_bosque-ud-test | raw | Words | 99.26 [99.14, 99.38] | 99.26 [99.14, 99.38] | 99.26 [99.14, 99.38] | | pt_bosque-ud-test | raw | UPOS | 96.59 [96.32, 96.83] | 96.75 [96.50, 96.98] | 96.75 [96.50, 96.98] | | pt_bosque-ud-test | raw | UFeats | 95.06 [94.75, 95.34] | 95.19 [94.90, 95.48] | 95.19 [94.90, 95.48] | | pt_bosque-ud-test | raw | Lemmas | 96.96 [96.73, 97.19] | 97.29 [97.09, 97.50] | 97.29 [97.09, 97.50] | | pt_bosque-ud-test | raw | UAS | 89.38 [88.77, 89.91] | 89.73 [89.18, 90.26] | 90.68 [90.16, 91.19] | | pt_bosque-ud-test | raw | LAS | 86.47 [85.84, 87.04] | 86.86 [86.27, 87.44] | 87.98 [87.39, 88.55] | | pt_bosque-ud-test | raw | CLAS | not measured | 83.83 [83.04, 84.61] | 85.27 [84.54, 85.99] | | pt_bosque-ud-test | raw | MLAS | not measured | 75.21 [74.33, 76.14] | 76.55 [75.69, 77.41] | | tr_boun-ud-test | gold | UPOS | 91.50 [90.98, 92.05] | 91.28 [90.72, 91.81] | 91.28 [90.72, 91.81] | | tr_boun-ud-test | gold | XPOS | not measured | 44.67 [43.81, 45.55] | 44.67 [43.81, 45.55] | | tr_boun-ud-test | gold | AllTags | not measured | 37.77 [37.03, 38.61] | 37.77 [37.03, 38.61] | | tr_boun-ud-test | gold | UFeats | 80.57 [79.80, 81.33] | 80.29 [79.54, 81.01] | 80.29 [79.54, 81.01] | | tr_boun-ud-test | gold | Lemmas | 86.40 [85.75, 87.00] | 86.00 [85.31, 86.61] | 86.00 [85.31, 86.61] | | tr_boun-ud-test | gold | UAS | 75.25 [73.92, 76.56] | 72.36 [70.98, 73.75] | 74.44 [73.05, 75.76] | | tr_boun-ud-test | gold | LAS | 67.92 [66.56, 69.24] | 64.97 [63.60, 66.26] | 67.13 [65.67, 68.45] | | tr_boun-ud-test | gold | CLAS | not measured | 62.95 [61.49, 64.24] | 65.35 [63.77, 66.74] | | tr_boun-ud-test | gold | MLAS | not measured | 43.83 [42.52, 45.02] | 45.80 [44.46, 47.03] | | tr_boun-ud-test | raw | Tokens | 99.56 [99.41, 99.71] | 99.56 [99.41, 99.71] | 99.56 [99.41, 99.71] | | tr_boun-ud-test | raw | Sentences | 81.28 [78.81, 83.58] | 79.89 [77.32, 82.20] | 78.82 [76.14, 81.22] | | tr_boun-ud-test | raw | Words | 96.71 [96.34, 97.08] | 96.71 [96.34, 97.08] | 96.71 [96.34, 97.08] | | tr_boun-ud-test | raw | UPOS | 89.08 [88.46, 89.72] | 88.62 [87.99, 89.32] | 88.62 [87.99, 89.32] | | tr_boun-ud-test | raw | XPOS | not measured | 43.51 [42.62, 44.33] | 43.51 [42.62, 44.33] | | tr_boun-ud-test | raw | AllTags | not measured | 37.44 [36.68, 38.26] | 37.44 [36.68, 38.26] | | tr_boun-ud-test | raw | UFeats | 80.24 [79.44, 81.03] | 80.23 [79.44, 81.03] | 80.23 [79.44, 81.03] | | tr_boun-ud-test | raw | Lemmas | 84.51 [83.78, 85.20] | 84.42 [83.71, 85.09] | 84.42 [83.71, 85.09] | | tr_boun-ud-test | raw | UAS | 69.75 [68.39, 71.06] | 69.30 [67.85, 70.58] | 71.15 [69.82, 72.44] | | tr_boun-ud-test | raw | LAS | 63.03 [61.61, 64.41] | 62.33 [60.88, 63.71] | 64.31 [62.89, 65.78] | | tr_boun-ud-test | raw | CLAS | not measured | 61.35 [59.79, 62.70] | 63.21 [61.68, 64.58] | | tr_boun-ud-test | raw | MLAS | not measured | 45.06 [43.56, 46.41] | 46.89 [45.51, 48.21] | | zh_gsdsimp-ud-test | gold | UPOS | 88.77 [88.18, 89.35] | 87.55 [86.92, 88.22] | 87.55 [86.92, 88.22] | | zh_gsdsimp-ud-test | gold | XPOS | 87.86 [87.24, 88.47] | 87.14 [86.47, 87.79] | 87.14 [86.47, 87.79] | | zh_gsdsimp-ud-test | gold | AllTags | 85.15 [84.47, 85.82] | 83.66 [82.95, 84.39] | 83.66 [82.95, 84.39] | | zh_gsdsimp-ud-test | gold | UFeats | 96.03 [95.69, 96.39] | 95.86 [95.48, 96.23] | 95.86 [95.48, 96.23] | | zh_gsdsimp-ud-test | gold | Lemmas | 99.80 [99.71, 99.88] | 99.78 [99.67, 99.86] | 99.78 [99.67, 99.86] | | zh_gsdsimp-ud-test | gold | UAS | 69.15 [67.81, 70.55] | 64.08 [62.81, 65.38] | 67.22 [66.03, 68.48] | | zh_gsdsimp-ud-test | gold | LAS | 63.29 [61.88, 64.70] | 57.55 [56.30, 58.83] | 61.42 [60.21, 62.70] | | zh_gsdsimp-ud-test | gold | CLAS | not measured | 56.22 [54.82, 57.68] | 59.72 [58.32, 61.19] | | zh_gsdsimp-ud-test | gold | MLAS | not measured | 44.42 [42.92, 45.86] | 47.94 [46.40, 49.45] | | zh_gsdsimp-ud-test | raw | Tokens | 89.62 [88.81, 90.43] | 89.62 [88.81, 90.43] | 89.62 [88.81, 90.43] | | zh_gsdsimp-ud-test | raw | Sentences | 85.31 [82.10, 88.41] | 83.64 [80.04, 86.90] | 83.77 [80.32, 87.30] | | zh_gsdsimp-ud-test | raw | Words | 89.62 [88.81, 90.43] | 89.62 [88.81, 90.43] | 89.62 [88.81, 90.43] | | zh_gsdsimp-ud-test | raw | UPOS | 80.96 [80.07, 81.81] | 80.66 [79.76, 81.48] | 80.66 [79.76, 81.48] | | zh_gsdsimp-ud-test | raw | XPOS | 80.11 [79.28, 80.98] | 79.75 [78.91, 80.63] | 79.75 [78.91, 80.63] | | zh_gsdsimp-ud-test | raw | AllTags | 77.97 [77.04, 78.87] | 77.47 [76.54, 78.38] | 77.47 [76.54, 78.38] | | zh_gsdsimp-ud-test | raw | UFeats | 85.94 [85.11, 86.79] | 85.98 [85.14, 86.82] | 85.98 [85.14, 86.82] | | zh_gsdsimp-ud-test | raw | Lemmas | 89.45 [88.61, 90.27] | 89.41 [88.61, 90.21] | 89.41 [88.61, 90.21] | | zh_gsdsimp-ud-test | raw | UAS | 62.78 [61.24, 64.26] | 62.91 [61.40, 64.43] | 64.42 [62.89, 65.87] | | zh_gsdsimp-ud-test | raw | LAS | 58.84 [57.23, 60.35] | 58.91 [57.38, 60.42] | 60.98 [59.49, 62.45] | | zh_gsdsimp-ud-test | raw | CLAS | not measured | 55.70 [53.91, 57.39] | 58.05 [56.40, 59.67] | | zh_gsdsimp-ud-test | raw | MLAS | not measured | 47.57 [45.78, 49.21] | 49.33 [47.69, 50.97] | ### Paired recovery regression Against the joint 84-language model, Turkish-BOUN raw **sentence F1** changes from **79.89 to 78.82**, difference **-1.073 [-1.998, -0.134]** (1,000 paired sentence resamples, seed 20261106). This is the only significantly negative difference among the 139 recomputed UD rows. Turkish raw LAS rises from 62.33 to 64.31, difference 1.978 [1.413, 2.578]. Selection was fixed to full-dev gold LAS before these test results; no test-driven reversion or additional training was performed. # Distilled accurate tier: per-row results ### English (UD_English-EWT), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 94.56 [94.20, 94.90] | 94.16 [93.79, 94.54] | | 97.61 | | | XPOS | 93.33 [92.93, 93.70] | | | 97.13 | | | UFeats | 94.27 [93.93, 94.59] | 95.15 [94.86, 95.47] | | 98.13 | | | AllTags | 90.51 [90.04, 90.93] | | | 96.34 | | | Lemmas | 94.97 [94.64, 95.33] | 96.11 [95.77, 96.44] | | 97.96 | | | UAS | 81.23 [80.40, 82.07] | | | 94.09 | | | LAS | 77.33 [76.48, 78.20] | 81.68 [80.82, 82.49] | | 92.36 | | | CLAS | 72.12 [71.11, 73.13] | | | | | | MLAS | 63.63 [62.52, 64.72] | | | 86.29 | | ### English (UD_English-EWT), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 97.13 [96.83, 97.43] | | 99.17 | | 98.48 | | Sentences | 65.36 [63.39, 67.34] | 76.15 [74.25, 78.08] | 89.16 | 88.42 | 88.35 | | Words | 98.68 [98.45, 98.90] | | 98.92 | 99.07 | 98.48 | | UPOS | 93.71 [93.26, 94.11] | 92.90 [92.42, 93.35] | 96.20 | 96.75 | 95.95 | | XPOS | 92.72 [92.27, 93.14] | | 95.75 | 96.27 | 95.71 | | UFeats | 93.58 [93.19, 93.95] | | 96.39 | 97.21 | 96.26 | | AllTags | 89.94 [89.43, 90.39] | | 94.64 | 95.51 | 94.62 | | Lemmas | 94.14 [93.75, 94.55] | 95.03 [94.61, 95.43] | 96.63 | 97.13 | 96.84 | | UAS | 76.95 [76.02, 77.90] | | 88.90 | 91.59 | 90.14 | | LAS | 73.61 [72.66, 74.54] | 77.76 [76.82, 78.70] | 86.77 | 89.91 | 87.96 | | CLAS | 68.17 [67.02, 69.30] | | 84.11 | | 85.75 | | MLAS | 60.63 [59.54, 61.80] | 65.61 [64.46, 66.72] | 80.27 | 83.87 | 81.02 | ### German (UD_German-GSD), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 94.55 [94.15, 94.93] | 94.56 [94.18, 94.94] | | 97.11 | | | XPOS | 94.11 [93.71, 94.49] | | | 97.93 | | | UFeats | 80.50 [79.78, 81.16] | 86.25 [85.60, 86.96] | | 91.53 | | | AllTags | 76.34 [75.53, 77.13] | | | 89.21 | | | Lemmas | 92.55 [92.16, 92.92] | 95.95 [95.67, 96.25] | | 97.37 | | | UAS | 78.02 [76.94, 78.96] | | | 89.24 | | | LAS | 72.87 [71.79, 73.81] | 78.25 [77.23, 79.26] | | 86.16 | | | CLAS | 65.24 [63.85, 66.47] | | | | | | MLAS | 43.66 [42.28, 44.84] | | | 69.30 | | ### German (UD_German-GSD), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 97.40 [97.09, 97.69] | | 99.62 | | 99.75 | | Sentences | 78.22 [75.90, 80.83] | | 83.84 | 83.32 | 92.72 | | Words | 97.06 [96.70, 97.39] | | 99.60 | 99.79 | 99.75 | | UPOS | 91.75 [91.27, 92.23] | 94.22 [93.83, 94.61] | 95.61 | 96.87 | 95.04 | | XPOS | 91.25 [90.75, 91.72] | | 97.36 | 97.71 | 97.57 | | UFeats | 77.75 [77.00, 78.44] | | 89.78 | 91.17 | 91.51 | | AllTags | 73.55 [72.75, 74.28] | | 87.19 | 88.82 | 86.86 | | Lemmas | 89.50 [89.01, 90.00] | | 97.23 | 97.19 | 96.60 | | UAS | 73.95 [72.91, 74.91] | | 85.80 | 87.44 | 88.73 | | LAS | 69.03 [68.00, 70.01] | 76.75 [75.74, 77.76] | 81.80 | 84.36 | 84.77 | | CLAS | 63.47 [62.19, 64.77] | | 77.08 | | 80.78 | | MLAS | 41.34 [40.09, 42.60] | | 62.81 | 67.63 | 64.76 | ### French (UD_French-GSD), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 96.07 [95.61, 96.46] | 96.77 [96.33, 97.19] | | 98.55 | | | UFeats | 94.38 [93.76, 94.92] | 96.34 [95.89, 96.81] | | 98.30 | | | Lemmas | 95.19 [94.71, 95.61] | 97.51 [97.16, 97.84] | | 98.85 | | | UAS | 83.49 [82.31, 84.73] | | | 94.99 | | | LAS | 79.63 [78.37, 80.97] | 86.69 [85.60, 87.81] | | 93.11 | | | CLAS | 71.21 [69.53, 72.85] | | | | | | MLAS | 64.08 [62.26, 65.75] | | | 86.95 | | ### French (UD_French-GSD), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 99.88 [99.79, 99.96] | | 99.71 | | 99.70 | | Sentences | 92.65 [90.35, 94.87] | | 95.03 | 93.82 | 96.63 | | Words | 99.48 [99.29, 99.65] | | 99.50 | 98.88 | 99.66 | | UPOS | 95.56 [95.09, 95.97] | 96.06 [95.60, 96.53] | 97.63 | 97.43 | 97.85 | | UFeats | 94.03 [93.41, 94.55] | | 97.74 | 97.15 | 97.16 | | Lemmas | 94.54 [94.04, 95.03] | | 98.20 | 97.73 | 97.80 | | UAS | 83.11 [81.88, 84.45] | | 93.27 | 93.26 | 94.00 | | LAS | 79.17 [77.86, 80.50] | 85.97 [84.73, 87.08] | 91.28 | 91.21 | 92.34 | | CLAS | 71.58 [69.86, 73.31] | | 87.44 | | 88.66 | | MLAS | 64.26 [62.40, 65.99] | | 84.45 | 85.02 | 84.76 | ### Spanish (UD_Spanish-AnCora), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 97.34 [97.17, 97.49] | 98.11 [97.98, 98.25] | | 99.09 | | | XPOS | 84.25 [83.80, 84.71] | | | 96.12 | | | UFeats | 95.90 [95.73, 96.09] | 97.71 [97.56, 97.86] | | 98.79 | | | AllTags | 83.20 [82.76, 83.64] | | | 95.64 | | | Lemmas | 95.27 [95.07, 95.45] | 98.84 [98.73, 98.95] | | 99.41 | | | UAS | 83.02 [82.46, 83.56] | | | 94.01 | | | LAS | 79.36 [78.79, 79.94] | 87.00 [86.50, 87.51] | | 92.26 | | | CLAS | 71.25 [70.52, 71.98] | | | | | | MLAS | 64.64 [63.89, 65.42] | | | 87.03 | | ### Spanish (UD_Spanish-AnCora), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 99.84 [99.76, 99.90] | | 99.97 | | 99.94 | | Sentences | 95.61 [94.59, 96.55] | | 98.49 | 98.69 | 99.13 | | Words | 99.83 [99.75, 99.89] | | 99.96 | 99.95 | 99.93 | | UPOS | 97.24 [97.07, 97.40] | 97.86 [97.71, 98.01] | 98.96 | 99.05 | 99.02 | | XPOS | 84.46 [84.02, 84.93] | | 96.10 | 96.08 | 98.94 | | UFeats | 95.73 [95.53, 95.93] | | 98.54 | 98.74 | 98.80 | | AllTags | 83.34 [82.90, 83.80] | | 95.65 | 95.61 | 98.27 | | Lemmas | 95.28 [95.09, 95.50] | | 99.24 | 99.37 | 99.17 | | UAS | 82.41 [81.83, 82.99] | | 93.09 | 93.81 | 94.11 | | LAS | 78.74 [78.14, 79.35] | 86.70 [86.15, 87.23] | 91.30 | 92.07 | 92.41 | | CLAS | 70.59 [69.87, 71.34] | | 87.80 | | 89.66 | | MLAS | 63.87 [63.10, 64.65] | | 85.49 | 86.86 | 87.60 | ### Portuguese (UD_Portuguese-Bosque), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 95.10 [94.82, 95.38] | 95.89 [95.62, 96.14] | | 98.09 | | | UFeats | 92.96 [92.62, 93.29] | 94.51 [94.17, 94.80] | | 97.15 | | | Lemmas | 94.80 [94.52, 95.09] | 97.30 [97.10, 97.51] | | 98.63 | | | UAS | 82.29 [81.62, 82.97] | | | 93.31 | | | LAS | 77.78 [77.03, 78.52] | 82.74 [82.07, 83.35] | | 90.93 | | | CLAS | 70.28 [69.38, 71.28] | | | | | | MLAS | 59.41 [58.45, 60.40] | | | 81.55 | | ### Portuguese (UD_Portuguese-Bosque), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 99.86 [99.79, 99.92] | | 99.85 | | 99.70 | | Sentences | 89.57 [87.82, 91.21] | | 93.10 | 89.73 | 97.48 | | Words | 99.26 [99.14, 99.38] | | 99.76 | 99.68 | 99.65 | | UPOS | 94.91 [94.60, 95.23] | 95.35 [95.05, 95.64] | 97.45 | 97.77 | 97.27 | | UFeats | 92.72 [92.35, 93.06] | | 96.44 | 96.89 | 96.50 | | Lemmas | 94.31 [94.01, 94.61] | | 98.04 | 98.33 | 97.89 | | UAS | 81.32 [80.58, 82.05] | | 90.59 | 92.16 | 92.76 | | LAS | 77.41 [76.62, 78.16] | 82.09 [81.38, 82.76] | 87.75 | 89.84 | 90.25 | | CLAS | 70.16 [69.22, 71.17] | | 82.66 | | 86.96 | | MLAS | 59.55 [58.52, 60.60] | | 77.07 | 80.52 | 80.03 | ### Italian (UD_Italian-ParlaMint), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 92.14 [91.57, 92.72] (it_markit-ud-test) | 94.44 [93.92, 94.96] | | 99.19 | | | UPOS | 94.71 [94.22, 95.19] (it_parlamint-ud-test) | | | 99.19 | | | XPOS | 93.38 [92.87, 93.87] (it_markit-ud-test) | | | 98.55 | | | XPOS | 94.35 [93.83, 94.87] (it_parlamint-ud-test) | | | 98.55 | | | UFeats | 88.71 [87.90, 89.51] (it_markit-ud-test) | 89.52 [88.85, 90.16] | | 98.49 | | | UFeats | 87.36 [86.54, 88.11] (it_parlamint-ud-test) | | | 98.49 | | | AllTags | 84.34 [83.40, 85.26] (it_markit-ud-test) | | | 97.61 | | | AllTags | 84.32 [83.48, 85.20] (it_parlamint-ud-test) | | | 97.61 | | | Lemmas | 80.25 [79.27, 81.08] (it_markit-ud-test) | 95.18 [94.73, 95.63] | | 99.20 | | | Lemmas | 89.49 [88.80, 90.14] (it_parlamint-ud-test) | | | 99.20 | | | UAS | 71.17 [69.80, 72.37] (it_markit-ud-test) | | | 93.24 | | | UAS | 76.09 [74.78, 77.45] (it_parlamint-ud-test) | | | 93.24 | | | LAS | 64.54 [63.15, 65.74] (it_markit-ud-test) | 72.57 [71.24, 74.06] | | 91.13 | | | LAS | 71.46 [70.08, 72.85] (it_parlamint-ud-test) | | | 91.13 | | | CLAS | 53.59 [51.96, 55.19] (it_markit-ud-test) | | | | | | CLAS | 60.50 [58.78, 62.22] (it_parlamint-ud-test) | | | | | | MLAS | 39.60 [38.08, 41.14] (it_markit-ud-test) | | | 85.52 | | | MLAS | 44.30 [42.65, 46.10] (it_parlamint-ud-test) | | | 85.52 | | ### Italian (UD_Italian-ParlaMint), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 99.93 [99.86, 99.98] (it_markit-ud-test) | | 99.59 | | | | Tokens | 99.86 [99.76, 99.96] (it_parlamint-ud-test) | | 99.59 | | | | Sentences | 90.41 [87.56, 93.19] (it_markit-ud-test) | | 95.83 | 94.12 | | | Sentences | 83.62 [79.88, 87.08] (it_parlamint-ud-test) | | 95.83 | 94.12 | | | Words | 99.73 [99.58, 99.86] (it_markit-ud-test) | | 99.01 | 99.42 | | | Words | 99.67 [99.52, 99.80] (it_parlamint-ud-test) | | 99.01 | 99.42 | | | UPOS | 93.43 [92.89, 93.91] (it_markit-ud-test) | 94.01 [93.45, 94.51] | 96.83 | 98.58 | | | UPOS | 94.73 [94.31, 95.16] (it_parlamint-ud-test) | | 96.83 | 98.58 | | | XPOS | 94.23 [93.75, 94.70] (it_markit-ud-test) | | 95.96 | 97.99 | | | XPOS | 94.75 [94.27, 95.25] (it_parlamint-ud-test) | | 95.96 | 97.99 | | | UFeats | 90.24 [89.54, 90.94] (it_markit-ud-test) | | 95.28 | 97.95 | | | UFeats | 88.14 [87.47, 88.83] (it_parlamint-ud-test) | | 95.28 | 97.95 | | | AllTags | 86.39 [85.62, 87.17] (it_markit-ud-test) | | 93.74 | 97.09 | | | AllTags | 85.14 [84.42, 85.94] (it_parlamint-ud-test) | | 93.74 | 97.09 | | | Lemmas | 81.77 [80.86, 82.58] (it_markit-ud-test) | | 95.64 | 98.69 | | | Lemmas | 90.48 [89.82, 91.11] (it_parlamint-ud-test) | | 95.64 | 98.69 | | | UAS | 74.20 [72.80, 75.50] (it_markit-ud-test) | | 85.19 | 91.79 | | | UAS | 74.63 [73.21, 76.04] (it_parlamint-ud-test) | | 85.19 | 91.79 | | | LAS | 68.64 [67.22, 69.91] (it_markit-ud-test) | 72.77 [71.46, 74.16] | 81.45 | 89.72 | | | LAS | 70.66 [69.25, 72.02] (it_parlamint-ud-test) | | 81.45 | 89.72 | | | CLAS | 59.56 [57.78, 61.13] (it_markit-ud-test) | | 74.76 | | | | CLAS | 59.25 [57.31, 61.17] (it_parlamint-ud-test) | | 74.76 | | | | MLAS | 45.47 [43.83, 47.16] (it_markit-ud-test) | | 68.88 | 83.94 | | | MLAS | 43.29 [41.49, 45.12] (it_parlamint-ud-test) | | 68.88 | 83.94 | | ### Dutch (UD_Dutch-Alpino), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 94.51 [93.84, 95.09] | 93.83 [93.16, 94.49] | | 98.17 | | | XPOS | 91.45 [90.69, 92.15] | | | 97.12 | | | UFeats | 93.62 [92.91, 94.22] | 92.94 [92.24, 93.53] | | 97.75 | | | AllTags | 89.76 [88.97, 90.46] | | | 96.73 | | | Lemmas | 87.76 [87.14, 88.41] | 91.61 [90.99, 92.14] | | 96.20 | | | UAS | 78.13 [76.85, 79.38] | | | 95.30 | | | LAS | 72.96 [71.59, 74.30] | 80.28 [79.12, 81.51] | | 93.37 | | | CLAS | 64.11 [62.50, 65.76] | | | | | | MLAS | 54.34 [52.68, 56.13] | | | 86.35 | | ### Dutch (UD_Dutch-Alpino), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 99.66 [99.46, 99.82] | | 99.87 | | 99.22 | | Sentences | 89.96 [87.08, 92.24] | | 90.08 | 91.16 | 89.88 | | Words | 99.66 [99.46, 99.82] | | 99.87 | 99.78 | 99.22 | | UPOS | 94.51 [93.95, 95.02] | 93.09 [92.40, 93.77] | 96.75 | 97.97 | 96.55 | | XPOS | 91.06 [90.31, 91.79] | | 95.50 | 96.89 | 94.92 | | UFeats | 93.36 [92.72, 93.95] | | 97.00 | 97.52 | 96.22 | | AllTags | 89.60 [88.80, 90.32] | | 94.97 | 96.56 | 94.56 | | Lemmas | 87.44 [86.77, 88.10] | | 95.33 | 95.99 | 96.23 | | UAS | 78.33 [77.02, 79.60] | | 91.42 | 94.12 | 93.28 | | LAS | 73.64 [72.28, 74.88] | 79.48 [78.28, 80.78] | 88.67 | 92.25 | 91.28 | | CLAS | 65.33 [63.66, 66.86] | | 83.94 | | 87.88 | | MLAS | 55.76 [54.04, 57.43] | | 79.65 | 85.14 | 82.58 | ### Turkish (UD_Turkish-BOUN), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 90.02 [89.50, 90.53] | 88.97 [88.31, 89.58] | | 93.14 | | | UFeats | 79.95 [79.18, 80.72] | 78.15 [77.40, 78.90] | | 83.03 | | | Lemmas | 78.58 [77.84, 79.34] | 85.11 [84.38, 85.74] | | 93.62 | | | UAS | 66.63 [65.34, 67.91] | | | 80.54 | | | LAS | 58.29 [57.07, 59.56] | 62.80 [61.37, 64.19] | | 74.00 | | | CLAS | 54.17 [52.77, 55.54] | | | | | | MLAS | 37.91 [36.65, 39.17] | | | 52.95 | | ### Turkish (UD_Turkish-BOUN), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 98.03 [97.71, 98.33] | | 99.58 | | | | Sentences | 74.99 [72.25, 77.87] | | 90.08 | 86.25 | | | Words | 96.55 [96.15, 96.92] | | 96.32 | 96.57 | | | UPOS | 87.01 [86.35, 87.66] | 86.02 [85.28, 86.68] | 89.92 | 90.01 | | | UFeats | 78.63 [77.87, 79.42] | | 81.66 | 81.08 | | | Lemmas | 77.22 [76.42, 77.96] | | 90.37 | 90.59 | | | UAS | 59.53 [58.08, 60.84] | | 71.53 | 73.16 | | | LAS | 52.31 [50.99, 53.62] | 57.53 [56.08, 59.01] | 65.17 | 67.27 | | | CLAS | 49.32 [47.89, 50.75] | | 63.79 | | | | MLAS | 35.62 [34.29, 36.84] | | 48.49 | 49.26 | | ### Chinese (UD_Chinese-GSDSimp), gold input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | UPOS | 85.56 [84.90, 86.24] | 90.41 [89.70, 91.07] | | 95.73 | | | XPOS | 85.13 [84.47, 85.84] | | | 95.97 | | | UFeats | 95.43 [95.03, 95.83] | 98.38 [98.13, 98.59] | | 99.28 | | | AllTags | 81.86 [81.08, 82.64] | | | 95.20 | | | Lemmas | 99.59 [99.46, 99.70] | 99.87 [99.79, 99.94] | | 99.90 | | | UAS | 60.43 [59.02, 61.82] | | | 86.76 | | | LAS | 54.05 [52.70, 55.43] | 68.76 [67.12, 70.32] | | 83.60 | | | CLAS | 52.37 [50.78, 53.79] | | | | | | MLAS | 41.59 [40.14, 43.08] | 56.94 [55.08, 58.69] | | 77.62 | | ### Chinese (UD_Chinese-GSDSimp), raw input | Metric | Distilled tier | Fast tier | Stanza 1.5.1 | UDPipe 2 | Trankit | |---|---|---|---|---|---| | Tokens | 89.62 [88.81, 90.43] | | 92.14 | | 96.94 | | Sentences | 83.49 [80.00, 86.92] | | 99.10 | 98.80 | 99.70 | | Words | 89.62 [88.81, 90.43] | 89.62 [88.87, 90.42] | 92.14 | 90.13 | 96.94 | | UPOS | 77.69 [76.76, 78.59] | 81.90 [80.99, 82.86] | 88.82 | 86.67 | 94.17 | | XPOS | 77.19 [76.29, 78.12] | | 88.61 | 86.83 | 93.98 | | UFeats | 85.54 [84.70, 86.37] | | 91.47 | 89.52 | 96.51 | | AllTags | 74.60 [73.61, 75.61] | | 87.95 | 86.16 | 93.52 | | Lemmas | 89.32 [88.51, 90.11] | | 92.07 | 90.09 | 96.94 | | UAS | 51.12 [49.71, 52.49] | | 73.41 | 71.74 | 84.64 | | LAS | 46.49 [45.03, 47.91] | 57.88 [56.21, 59.63] | 70.65 | 69.52 | 81.96 | | CLAS | 42.94 [41.24, 44.46] | | 67.39 | | 80.14 | | MLAS | 34.57 [33.00, 36.03] | | 63.96 | 62.23 | 76.30 | # x64 release speed rows Measured 2026-11-08 on an AMD Ryzen 9 7950X3D (16 physical cores, 32 logical), x64 Linux 7.0.0-38-generic, .NET SDK 10.0.403, runtime 10.0.14. Available ISA: SSE4.2, AVX2, AVX-512F/BW/VL, FMA and BMI2. CPU inference only; no GPU used. OneNLP 0.4.0-preview.1 source build, OneFont 1.4.0, OneModel and OneModel.Training 1.0.0. Comparators: Microsoft.ML.Tokenizers 2.0.1, SearchPioneer.Lingua 2.0.0, Microsoft.Recognizers.Text 1.8.13, Catalyst 26.10.6514 with Catalyst.Models.English 26.9.5316. Three exclusive benchmark-queue jobs, each bounded to 20 minutes, ran once after prebuilding the tools. Actual timed-group wall times: tokenizers 2m23s, language ID/recognizers 4m40s, Catalyst 1m16s (about 8m19s total). No training, builds or other queued heavy work overlapped these jobs. The methods are in [measurement.md](https://recursiveai.org/onenlp/docs/measurement/) and the repository's bench scripts. Values below are medians, not best-case samples. | Group | Queue job | Load before (1/5/15 min) | Load during | Load after | |---|---|---|---|---| | Tokenizers | 007311 | 1.39 / 1.74 / 3.38 | 1-min samples 1.34–2.05, every 10 s | 1.74 / 1.79 / 3.17 | | Language ID and recognizers | 007317 | 1.47 / 1.73 / 3.13 | Between operations: 1.47, 1.72, 1.66, 1.61 (1-min) | 1.71 / 1.68 / 2.74 | | Catalyst and English model | 007321 | 2.85 / 2.04 / 2.80 | 1-min samples 1.78–2.85, every 10 s | 1.71 / 1.87 / 2.68 | ## Tokenizers [Every cell](https://recursiveai.org/onenlp/docs/tokenizer-speed-x64/) includes time, managed allocation per operation and extra resident high-water growth during the samples. All **47 matched encode/count/decode rows pass** (1.13–48.55× the managed comparator). Seven operation rows are not timed because the comparator produces different IDs or decoded text: WordPiece medium decode and long encode/count/decode; Unigram long encode/count/decode. They are not passes. Six load rows are information only; WordPiece, TinyLlama, gpt2 and cl100k_base load more slowly. Total process peak RSS was 400 MiB. Native HF timing was not requested here; exactness against HF is reported separately in [parity.md](https://recursiveai.org/onenlp/docs/parity/). Each operation warms up for about 500 ms, then measures 5–25 batches of about 40 ms (1,000 ms sampling budget); the reported time is the median batch's per-operation time. Allocation uses the thread counter. The six model families use the pinned files in `tools/parity/families.json` and `Families.All` in the benchmark. Short (59 characters), medium (513), and long (990,287 UTF-8 bytes) inputs are byte-identical to the arm64 run: | Input | SHA-256 | |---|---| | short.txt | b15e7e0dd880d275e4d21144fc05f857eb0670669a9d61a55188cb751cdb0ed9 | | medium.txt | 8235943c7f5f0f8475095d4d99a5d5912eebe7bb7d8baae34133b9c8c986df85 | | long.txt | 1f6178675374c4ba5ea26a16556b1a05f605e6333585d9fe881d7a6dd8bbd5ce | ## Language ID Single-thread scoring, every 15th held-out main-set line per bucket. Warm-up 200 inputs, then three repetitions. Median microseconds and bytes per text: | Bucket | Texts | Ours µs | Lingua high µs | Low µs | Speedup high / low | Allocation ours / high / low B | Verdict | |---|---:|---:|---:|---:|---|---|---| | word | 4,970 | 8.83 | 81.62 | 69.05 | 9.24× / 7.82× | 1,454 / 79,278 / 75,495 | pass | | word pair | 4,872 | 3.58 | 96.29 | 65.58 | 26.90× / 18.32× | 1,467 / 84,283 / 80,446 | pass | | sentence | 5,000 | 9.06 | 392.54 | 246.83 | 43.33× / 27.24× | 1,454 / 125,503 / 123,283 | pass | Load: ours 12.6 ms, high 3,670.6 ms, low 282.0 ms. Process peak RSS: 95, 5,042, and 360 MiB respectively. Our model is 3,588,959 bytes, SHA-256 `9d06cc77632083274ae5ca5fbb0f3e676abfc63aa6a2d702cbdf9d558cba9d6d`. Lingua's model size was not separately measured (it is delivered by its package). Input SHA-256: - words: `c878bfb1403c9f70b4616771397674fe46d1dbbc6008b4cfb24a7e9e61d7a421` - pairs: `f5123bbf28cf9423d3f179c208f3a427c8902fbb4150a29b93fbdaf1b6817473` - sentences: `956a081df9d16a2d1b825b782857dd13e3bb0ac680250ef7deedee8c6aecd127` ## English recognizers Same generated input lines and reference date for both systems (2026-03-18 10:30 UTC). Three repetitions; managed allocations below are the median total per repetition, not bytes per expression. UTF-16 character throughput excludes line terminators. Model size: data rules, no trained weights; rule byte size not separately measured. Different result counts reflect the quality differences reported in [results-table.md](https://recursiveai.org/onenlp/docs/results-table/), not different input work. | Rules | Lines / characters | Ours s | Comparator s | Speedup | Allocated MB ours / comparator | Peak RSS MiB ours / comparator | Load ms ours / comparator | Verdict | |---|---|---:|---:|---:|---|---|---|---| | Number/unit/currency/sequence | 11,557 / 9,980,266 | 0.542 | 26.140 | 48.23× | 402.3 / 8,794.7 | 113 / 344 | 30.2 / 356.5 | pass | | All, including date/time | 2,889 / 2,492,392 | 0.621 | 21.948 | 35.34× | 615.7 / 4,877.2 | 103 / 325 | 47.7 / 642.4 | pass | Date/time uses the first quarter of the full text to keep the group bounded; both systems receive those same lines. Turkish and Hindi timing is not a gate. ## English fast tier against Catalyst One thread, batch 1; UD English-EWT r2.18 test raw documents, repeated four times as one input workload. Each tool makes 12 passes; discard the first two and take the median of the remaining ten (internal samples, not repeated benchmark jobs). Ours emits 100,524 tokens per pass, Catalyst 101,724, because tokenizers differ. | Path | Ours tokens/s | Catalyst tokens/s | Ratio | Allocation B/token ours / Catalyst | Verdict | |---|---:|---:|---:|---|---| | Sentences + UPOS/UFeats + lemmas vs tagging | 833,483 | 1,049,762.5 | **0.794×** | 330 / 307 | **pass ≥0.5×** | | With XPOS too vs tagging | 661,659.5 | 1,049,762.5 | 0.630× | 425 / 307 | information | | Sentences + entities vs tagging + WikiNER | 443,997.5 | 471,738 | 0.941× | 322 / 699 | information, not matched work | | Dependency parser, predicts sentences | 45,235.5 | — | — | 2,530 / — | no managed parser comparator | | Full pipeline | 38,313 | — | — | 3,070 / — | information | The tagging row counts under matched or better measured accuracy on raw EWT (test F1/accuracy): Tokens 99.00 vs 94.47, Sentences 71.08 vs 64.57, UPOS 92.90 vs 80.70, Lemmas 95.03 vs 83.84. UNER character-span entity F1 is 63.09 vs 57.77. The NER speed row is not a gate: Catalyst's NER needs its tagging pipeline, whereas ours runs just the sentencizer and entity component. Process peak RSS: ours 1,240 MiB (whole multi-path job); Catalyst 383 MiB (316 MiB at the end of tagging alone). English model file: 117,451,750 bytes, SHA-256 `ac0080adcb1a9832fe4c5c7b80e22a54da51bfcdac88e0d53986c80e6ce7ed9b`. Cold English model load was not measured in this run. The quality rows are unchanged; no model was retrained or selected using these timings. ## Accurate tiers: indicative, not a speed gate A separate **heavy**, four-core-reservation job (007347), not an exclusive benchmark, measured one English pipeline from each package. `TensorCompute.Threads` is 1 and `DOTNET_PROCESSOR_COUNT` is 4. Other heavy jobs were running; load was 13.39 / 13.20 / 9.11 before, 12.62–13.77 (1-min) during, and 13.77 / 13.25 / 9.44 after. These figures must not be compared as publishable speed gates or as a no-load regression measurement. Input: the first 100 `# text` sentences of UD English-EWT r2.18 test, one per line, 2,206 emitted word tokens including punctuation; SHA-256 `1d719e08e95c2d852234bcb59f284b0306a60a6358ac932b7adbf337e30fbaa6`. Each throughput process warms up on 20 documents and takes the median of three passes. Cold load uses a separate fresh process and the first sentence, with an explicit English language (no language-detection work). Models are the exact manifest-verified package payloads; no quantization or tensors were changed. | Package | Installed weights | Median word tokens/s | Model load ms | First result including load ms | Second call ms | Peak RSS cold / throughput MiB | |---|---:|---:|---:|---:|---:|---| | `.Models.Multilingual` | 529,158,395 bytes | 462 | 3,512 | 3,992 | 151.8 | 809 / 814 | | `.Models.Multilingual.Large` | 3,508,907,198 bytes | 150 | 3,134 | 3,406 | 54.1 | 3,904 / 4,055 | Only English heads load. Allocation per token was not instrumented for this information run. The quality tables and known published-reference gaps are unchanged; these timings do not select a model or relax any quality bar. # Tokenizer speed: x64 See [speed-x64.md](https://recursiveai.org/onenlp/docs/speed-x64/) for machine, input hashes, versions and method. Speedup is comparator time divided by ours. Load is information, not gated. Inputs: short 59 chars, medium 513 chars, long 990287 bytes (UTF-8). | family | operation | size | ours | comparator | speedup | alloc ours | alloc comparator | peak extra RSS ours | peak extra RSS comparator | note | |---|---|---|---:|---:|---:|---:|---:|---:|---:|---| | WordPiece (bert-base-uncased) | load | - | 37.15 ms | 22.89 ms | 0.62x | - | - | - | - | load time; ours from tokenizer.json | | WordPiece (bert-base-uncased) | encode-to-ids | short | 289 ns | 899 ns | 3.11x | 0 B | 456 B | 8.0 KB | 4.0 KB | | | WordPiece (bert-base-uncased) | count | short | 287 ns | 821 ns | 2.86x | 0 B | 240 B | 4.0 KB | 0 B | | | WordPiece (bert-base-uncased) | decode | short | 71 ns | 167 ns | 2.35x | 144 B | 176 B | 4.0 KB | 4.0 KB | | | WordPiece (bert-base-uncased) | encode-to-ids | medium | 2.85 µs | 7.63 µs | 2.68x | 0 B | 2.3 KB | 36.0 KB | 4.0 KB | | | WordPiece (bert-base-uncased) | count | medium | 2.92 µs | 7.59 µs | 2.60x | 0 B | 1.1 KB | 0 B | 64.0 KB | | | WordPiece (bert-base-uncased) | decode | medium | - | - | - | - | - | - | - | decoded text differs at char 74 (lengths 525 vs 519) | | WordPiece (bert-base-uncased) | encode-to-ids | long | - | - | - | - | - | - | - | ids differ: 357483 vs 356166 tokens, first difference at 159 (ours 1456, comparator 100) | | WordPiece (bert-base-uncased) | count | long | - | - | - | - | - | - | - | ids differ: 357483 vs 356166 tokens, first difference at 159 (ours 1456, comparator 100) | | WordPiece (bert-base-uncased) | decode | long | - | - | - | - | - | - | - | ids differ: 357483 vs 356166 tokens, first difference at 159 (ours 1456, comparator 100) | | Unigram (xlm-roberta-base) | load | - | 260.55 ms | 766.93 ms | 2.94x | - | - | - | - | load time; ours from tokenizer.json | | Unigram (xlm-roberta-base) | encode-to-ids | short | 206 ns | 1.40 µs | 6.80x | 0 B | 216 B | 0 B | 15.3 MB | | | Unigram (xlm-roberta-base) | count | short | 203 ns | 1.42 µs | 7.01x | 0 B | 0 B | 0 B | 0 B | | | Unigram (xlm-roberta-base) | decode | short | 73 ns | 140 ns | 1.92x | 144 B | 176 B | 0 B | 56.0 KB | | | Unigram (xlm-roberta-base) | encode-to-ids | medium | 1.66 µs | 11.47 µs | 6.93x | 0 B | 2.2 KB | 0 B | 32.0 KB | | | Unigram (xlm-roberta-base) | count | medium | 1.70 µs | 12.32 µs | 7.24x | 0 B | 0 B | 0 B | 440.0 KB | | | Unigram (xlm-roberta-base) | decode | medium | 658 ns | 1.50 µs | 2.28x | 1.0 KB | 1.1 KB | 4.0 MB | 0 B | | | Unigram (xlm-roberta-base) | encode-to-ids | long | - | - | - | - | - | - | - | ids differ: 250378 vs 251115 tokens, first difference at 30076 (ours 49729, comparator 1759) | | Unigram (xlm-roberta-base) | count | long | - | - | - | - | - | - | - | ids differ: 250378 vs 251115 tokens, first difference at 30076 (ours 49729, comparator 1759) | | Unigram (xlm-roberta-base) | decode | long | - | - | - | - | - | - | - | ids differ: 250378 vs 251115 tokens, first difference at 30076 (ours 49729, comparator 1759) | | SentencePiece BPE, byte fallback (TinyLlama) | load | - | 60.91 ms | 11.69 ms | 0.19x | - | - | - | - | load time; ours from tokenizer.json | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | short | 92 ns | 4.08 µs | 44.59x | 0 B | 1.6 KB | 0 B | 0 B | | | SentencePiece BPE, byte fallback (TinyLlama) | count | short | 87 ns | 4.21 µs | 48.55x | 0 B | 1.4 KB | 0 B | 0 B | | | SentencePiece BPE, byte fallback (TinyLlama) | decode | short | 79 ns | 148 ns | 1.88x | 144 B | 176 B | 0 B | 0 B | | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | medium | 12.26 µs | 43.60 µs | 3.56x | 0 B | 12.3 KB | 0 B | 12.0 KB | | | SentencePiece BPE, byte fallback (TinyLlama) | count | medium | 12.24 µs | 43.22 µs | 3.53x | 0 B | 11.1 KB | 12.0 KB | 0 B | | | SentencePiece BPE, byte fallback (TinyLlama) | decode | medium | 707 ns | 1.56 µs | 2.21x | 1.0 KB | 1.1 KB | 0 B | 0 B | | | SentencePiece BPE, byte fallback (TinyLlama) | encode-to-ids | long | 89.74 ms | 262.47 ms | 2.92x | 1.8 MB | 21.5 MB | 0 B | 55.9 MB | | | SentencePiece BPE, byte fallback (TinyLlama) | count | long | 87.95 ms | 252.61 ms | 2.87x | 1.8 MB | 17.5 MB | 0 B | 26.3 MB | | | SentencePiece BPE, byte fallback (TinyLlama) | decode | long | 4.50 ms | 6.54 ms | 1.45x | 1.6 MB | 1.6 MB | 87.6 MB | 56.6 MB | | | Byte-level BPE (gpt2) | load | - | 31.46 ms | 24.78 ms | 0.79x | - | - | - | - | load time; ours from tokenizer.json | | Byte-level BPE (gpt2) | encode-to-ids | short | 454 ns | 722 ns | 1.59x | 0 B | 312 B | 0 B | 0 B | | | Byte-level BPE (gpt2) | count | short | 442 ns | 650 ns | 1.47x | 0 B | 96 B | 0 B | 0 B | | | Byte-level BPE (gpt2) | decode | short | 62 ns | 120 ns | 1.93x | 144 B | 176 B | 0 B | 0 B | | | Byte-level BPE (gpt2) | encode-to-ids | medium | 3.96 µs | 6.53 µs | 1.65x | 0 B | 1.2 KB | 0 B | 0 B | | | Byte-level BPE (gpt2) | count | medium | 4.11 µs | 5.97 µs | 1.45x | 0 B | 96 B | 0 B | 0 B | | | Byte-level BPE (gpt2) | decode | medium | 444 ns | 990 ns | 2.23x | 1.0 KB | 1.1 KB | 4.0 KB | 0 B | | | Byte-level BPE (gpt2) | encode-to-ids | long | 28.81 ms | 63.16 ms | 2.19x | 0 B | 14.0 MB | 12.0 KB | 65.8 MB | | | Byte-level BPE (gpt2) | count | long | 28.69 ms | 63.88 ms | 2.23x | 0 B | 10.0 MB | 0 B | 0 B | | | Byte-level BPE (gpt2) | decode | long | 4.71 ms | 7.23 ms | 1.54x | 1.6 MB | 1.6 MB | 124.0 MB | 60.5 MB | | | Tiktoken cl100k_base | load | - | 30.39 ms | 23.48 ms | 0.77x | - | - | - | - | load time; ours from cl100k_base.tiktoken | | Tiktoken cl100k_base | encode-to-ids | short | 510 ns | 711 ns | 1.39x | 408 B | 312 B | 0 B | 0 B | | | Tiktoken cl100k_base | count | short | 510 ns | 664 ns | 1.30x | 408 B | 96 B | 0 B | 0 B | | | Tiktoken cl100k_base | decode | short | 71 ns | 108 ns | 1.51x | 144 B | 176 B | 0 B | 0 B | | | Tiktoken cl100k_base | encode-to-ids | medium | 5.16 µs | 6.68 µs | 1.30x | 1.3 KB | 1.2 KB | 0 B | 0 B | | | Tiktoken cl100k_base | count | medium | 5.21 µs | 5.90 µs | 1.13x | 1.3 KB | 96 B | 0 B | 96.0 KB | | | Tiktoken cl100k_base | decode | medium | 522 ns | 1.03 µs | 1.97x | 1.0 KB | 1.1 KB | 0 B | 0 B | | | Tiktoken cl100k_base | encode-to-ids | long | 47.19 ms | 72.39 ms | 1.53x | 8.8 MB | 12.5 MB | 53.2 MB | 20.4 MB | | | Tiktoken cl100k_base | count | long | 47.82 ms | 63.87 ms | 1.34x | 8.8 MB | 8.5 MB | 36.5 MB | 3.8 MB | | | Tiktoken cl100k_base | decode | long | 4.41 ms | 6.22 ms | 1.41x | 1.6 MB | 1.6 MB | 38.6 MB | 72.3 MB | | | Tiktoken o200k_base | load | - | 33.55 ms | 50.85 ms | 1.52x | - | - | - | - | load time; ours from o200k_base.tiktoken | | Tiktoken o200k_base | encode-to-ids | short | 576 ns | 772 ns | 1.34x | 408 B | 312 B | 0 B | 0 B | | | Tiktoken o200k_base | count | short | 592 ns | 741 ns | 1.25x | 408 B | 96 B | 0 B | 0 B | | | Tiktoken o200k_base | decode | short | 70 ns | 110 ns | 1.58x | 144 B | 176 B | 0 B | 0 B | | | Tiktoken o200k_base | encode-to-ids | medium | 5.46 µs | 6.84 µs | 1.25x | 1.3 KB | 1.2 KB | 0 B | 0 B | | | Tiktoken o200k_base | count | medium | 5.24 µs | 6.59 µs | 1.26x | 1.3 KB | 96 B | 8.0 KB | 0 B | | | Tiktoken o200k_base | decode | medium | 553 ns | 948 ns | 1.71x | 1.0 KB | 1.1 KB | 0 B | 0 B | | | Tiktoken o200k_base | encode-to-ids | long | 46.13 ms | 71.88 ms | 1.56x | 8.8 MB | 10.2 MB | 130.9 MB | 7.8 MB | | | Tiktoken o200k_base | count | long | 43.73 ms | 64.85 ms | 1.48x | 8.8 MB | 6.2 MB | 62.3 MB | 0 B | | | Tiktoken o200k_base | decode | long | 4.24 ms | 5.85 ms | 1.38x | 1.6 MB | 1.6 MB | 66.4 MB | 65.1 MB | | # Measurement How OneNLP scores its own output, how it puts confidence intervals on scores, and how a results row is judged against a reference. The scorers live in `src/OneNlp/Scoring/`; the command-line tool that turns predictions into results rows is `tools/OneNlp.Eval/`. ## Universal Dependencies metrics `UdScorer` computes the metrics of the CoNLL 2018 shared task on parsing from raw text to Universal Dependencies (Zeman et al., 2018, "CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies", §5; evaluation description at ). **Character stream.** A corpus's text is the concatenation of its token forms (a multiword token counts once, as its own form) with spaces removed. Gold and predicted corpora must have the same stream, otherwise scoring throws. Every token, word and sentence is a span over this stream. The words of a multiword token share the token's span. Empty nodes (IDs like `8.1`) are ignored. | Metric | Defined as | |---|---| | Tokens, Sentences | F1 of the spans: a predicted span is correct when a gold span has the same start and end. | | Words | F1 of the aligned words (below). | | UPOS, XPOS, UFeats, AllTags, Lemmas | Precision and recall over words; a gold word and its aligned predicted word agree on the column (AllTags: UPOS, XPOS and UFeats together). | | UAS | Aligned words whose head is the aligned counterpart of the gold head (both roots also count). | | LAS | UAS and the same relation. | | CLAS | LAS restricted to content words (below). | | MLAS | CLAS, and the same UPOS and UFeats, and the functional children of the word (below) agree one for one in attachment, relation, UPOS and UFeats. | | BLEX | CLAS, and the same lemma. | All of them report precision (correct / predicted words), recall (correct / gold words) and F1 = 2·correct / (gold + predicted). Where a metric restricts to content words, each side counts its own content words. `AlignedAccuracy` is correct over aligned gold words. The figure usually quoted, and the one the results table carries, is F1. **Word alignment.** Words with the same span align. Where a multiword token is involved on either side, the words of the group of overlapping tokens are aligned by the longest common subsequence of their lower-cased forms. A plain word that ends before a multiword token starts aligns with nothing. A group larger than 64 million word pairs is refused. **Relations.** Language-specific subtypes are dropped (`nmod:poss` is `nmod`). Content relations: `nsubj obj iobj csubj ccomp xcomp obl vocative expl dislocated advcl advmod discourse nmod appos nummod acl amod conj fixed flat compound list parataxis orphan goeswith reparandum root dep`. Functional relations: `aux cop mark det clf case cc`. **Lemmas.** A gold lemma of `_` accepts any predicted lemma. **Features.** UFeats compares only the 21 universal features of the UD guidelines (`Abbr Animacy Aspect Case Definite Degree Evident Foreign Gender Mood NumType Number Person Polarity Polite Poss PronType Reflex Tense VerbForm Voice`), in any order. Everything else (`Typo`, `NumForm`, `Style`, `ExtPos`, layered features such as `Number[psor]`) is ignored; a word with no universal feature has `_`. The shared-task description does not spell this out; it was settled by running the reference script, as a program, on crafted pairs (script SHA-256 `1072e02af00b1a56205b5e8216d51dee9b8944a104d80744afaccc78859fcb16`). **Heads.** A head out of range, or a cycle among the heads of a sentence, is an error for gold and predicted data alike. Inputs are CoNLL-U files (`UdCorpus.ReadFile`) or `Doc`s (`UdCorpus.FromDocs`; UPOS is `Token.Pos`, XPOS `Token.Tag`, features `Token.Morph`, relation `Token.Dep`; a token that is its own head is a root). `UdScorer.Score(predicted, gold, "LAS")` returns a `NamedScore` in points for use as a training loop's development score. ## Span F1 `SpanScorer` scores entities and other labelled spans by exact match: a predicted span is correct when its start, end and label equal a gold span's (Tjong Kim Sang and De Meulder, 2003, the CoNLL-2003 span-level definition as `conlleval` and `seqeval` compute it). Precision = correct / predicted, recall = correct / gold, F1 their harmonic mean, 0 where undefined. It reports micro counts, counts per label, and macro F1 (the mean of the per-label F1 over labels in the gold or the prediction). Tag sequences are read as strict IOB2: a span is `B-X` followed by any number of `I-X`. An `I-X` that does not continue such a span, and an `I-Y` after `B-X`, starts nothing and is ignored. This equals seqeval's `mode="strict", scheme=IOB2`. ## Text classification `ClassificationScorer`: accuracy; per-label counts, macro F1 (unweighted mean over labels in the gold or the prediction) and micro F1 (equal to accuracy for single-label tasks); for multi-label tasks, macro and micro F1 at a threshold (0.5 by default) and ROC-AUC. ROC-AUC per label is the Mann-Whitney statistic with ties counting half (Hanley and McNeil, 1982); the multi-label figure is the mean over labels, skipping labels with only positives or only negatives. ## Bootstrap confidence intervals `Bootstrap` implements the percentile bootstrap (Efron and Tibshirani, 1993, §13.3). - **Resampling unit.** UD: the gold sentence. Each gold item and its aligned predicted item count for the gold sentence in which the gold item starts; a predicted item without a gold counterpart counts for the gold sentence containing its first character. Span F1: the sentence or document of the input. Classification: the document. - **Statistic.** Counts add up across units, so a resample sums its units' counts and takes F1 (`Bootstrap.F1Points`); nothing is re-scored. - **Draws.** 1,000 resamples by default (at least 100), each drawing as many units as the test set has, with replacement. The generator is SplitMix64 from the caller's seed, so a seed gives the same draws on every platform; the default seed is recorded in each row. - **Interval.** The resampled values are sorted; the interval runs from the entry at `floor(0.025 · B)` to the entry at `ceil(0.975 · B) − 1` (0-based), so for 1,000 resamples entries 25 and 974. The estimate is the statistic on the full test set. - **Paired.** To compare two systems on the same test set, `Bootstrap.Paired` applies the same draw to both on each resample and reports each system's interval and the interval of the difference. The systems differ when the difference's interval excludes 0. ## The within-noise rule A quality row passes when our score is within noise of its reference. Scores are in points (0 to 100). - **The reference has an interval** (we ran it ourselves on the same test set): within noise when our interval and the reference's overlap. - **The reference is a published number** with no interval: within noise when it lies inside our interval, or is within 0.5 points of our score. `NoiseRule.WithinNoise` implements both. ## Producing a results row ``` dotnet OneNlp.Eval.dll ud --gold gold.conllu --pred ours.conllu \ --component "fast tagger" --language en --dataset "UD English-EWT test" \ --reference UPOS=96.4,LAS=88.1 --reference-source "published, " \ --metrics UPOS,LAS --resamples 1000 --seed 20260101 --json rows.json --markdown rows.md ``` With `--ref-pred reference.conllu` (a reference system's output on the same test set) the reference gets its own interval and the row carries the paired difference. `span --gold gold.iob2 --pred ours.iob2` does the same for entity F1 (micro F1, precision, recall; files have one token per line, the tag in the last column, blank lines between sentences). The Markdown row is `| Component | Language | Metric | Score | 95% CI | Reference | Within noise |` and the JSON row carries the same fields plus the dataset, the reference source, the resample count and the seed. ## Cross-check against the reference scorers `tools/eval/crosscheck.py` checks the scorers against the reference programs, which run as black boxes: - **UD:** the shared-task evaluation script (`conll18_ud_eval.py`, from the shared-task site; run through its public functions at full precision and once as a program for its two-decimal table) on UD English-EWT test (release r2.15) paired with predictions that `tools/eval/perturb_ud.py` makes by seeded errors: wrong tags, features, lemmas, relations and heads; split and merged words; multiword tokens added, removed and re-spelled; merged and split sentences; blanked columns. - **Span F1:** seqeval in strict IOB2 mode against `tools/eval/perturb_span.py`'s synthetic tag sets, including malformed predictions and gold. Every precision, recall, F1 and aligned accuracy must agree to four decimal places (in points). `tools/fetch/conll18_crosscheck.sh` fetches the data and tools and checks them against pinned SHA-256 values. # Published reference results Numbers for the accurate tier to be judged against, copied from the sources named in each row, nothing re-run. Every number is an F1 (or accuracy where the source says so) × 100. How to read the rows: - **Metric.** UD columns are the CoNLL 2018 shared-task definitions, produced by the official `conll18_ud_eval.py`. *Tokens* and *Words* score the system's tokenization against the gold; *Sents* is sentence-boundary F1; UPOS, XPOS, UFeats, AllTags, Lemmas, UAS and LAS are F1 over aligned words, so a tokenization error costs the later columns too. - **Raw** means the system received only the text (predicted tokenization and sentence splitting). **Gold** means gold tokens and sentences were given, which removes the first columns' errors from the rest. Only compare like with like. - **Release.** UD treebanks change between releases (EWT's annotations are corrected at most releases), so numbers from different releases differ slightly even for the same system. Our own test set is EWT r2.18. - Retrieved 2026-11-04. ## UD English-EWT, test split | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.17 | 89.16 | 98.92 | 96.20 | 95.75 | 96.39 | 94.64 | 96.63 | 88.90 | 86.77 | | UDPipe 2, `english-ewt-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 88.42 | 99.07 | 96.75 | 96.27 | 97.21 | 95.51 | 97.13 | 91.59 | 89.91 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 97.61 | 97.13 | 98.13 | 96.34 | 97.96 | 94.09 | 92.36 | | Trankit large, XLM-R large ([3]) | 2.5 | raw | 98.67 | 90.49 | 98.67 | 96.65 | 96.47 | 96.75 | 95.49 | 97.25 | 91.29 | 89.40 | | Trankit base, XLM-R base ([3]) | 2.5 | raw | 98.48 | 88.35 | 98.48 | 95.95 | 95.71 | 96.26 | 94.62 | 96.84 | 90.14 | 87.96 | | Stanza 1.1.1 ([3], same table) | 2.5 | raw | 99.01 | 81.13 | 99.01 | 95.40 | 95.12 | 96.11 | 93.90 | 97.21 | 86.22 | 83.59 | | UDPipe 1.2 ([3], same table) | 2.5 | raw | 98.90 | 77.40 | 98.90 | 93.26 | 92.75 | 94.23 | n/a | 95.45 | 80.22 | 77.03 | | spaCy 2.3 pretrained ([3], same table) | 2.5 | raw | 97.30 | 61.19 | 97.30 | 86.72 | 90.83 | n/a | n/a | 87.05 | n/a | n/a | Notes: - UDPipe 2 reports the column "Words" (not Tokens) and does not report Tokens for raw text on the page; its gold-tokenization row has no Tokens/Sents/Words by definition. Its UD 2.17 models are distributed under CC BY-NC-SA ([2]); we cite the published numbers only and do not use or redistribute the models. - The Trankit page's "spaCy" row is spaCy 2.3's public English model, trained on a different corpus; it is information, not a target. - Stanza's page states its scores are end-to-end from raw text on the official test sets with the CoNLL 2018 script, and are for models with charlm and without transformers. - Trankit's page lists the Stanza and UDPipe numbers as obtained by running their public pretrained models on UD 2.5. - Gates for the accurate tier use the row whose input mode and release match what we measure; where our run differs (release, mode), the comparison is information and says so. ## UNER v1, English-EWT, test split | System (source) | Setting | PER | ORG | LOC | Overall (micro F1) | |---|---|---|---|---|---| | XLM-R large fine-tuned on en_ewt ([4], Table 6) | gold tokens, entity-level micro F1 | .950 | .709 | .871 | .858 | Notes: [4] reports one fine-tuning configuration (XLM-R large, hyperparameters in its §5.1) and one number per tag; no confidence interval is published. Types are PER, ORG and LOC. ## The nine 0.5 languages: UD test sets The same reading rules as above. The treebank per language is the one chosen in `tools/fetch//MANIFEST.md` (r2.18). Retrieved 2026-11-05. Columns are Tokens, Sents, Words, UPOS, XPOS, UFeats, AllTags, Lemmas, UAS, LAS (CLAS, MLAS and BLEX are on the sources' pages). **Version mismatch applies to every row**: Stanza's numbers are on UD 2.12, UDPipe 2's on 2.17, Trankit's on 2.5, ours on 2.18; treebanks are corrected between releases, and a row from another release is information, not a gate. Where a source's table has no row for our treebank, the section says so. ### Chinese: UD Chinese-GSDSimp, test (r2.18 is our release) | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 92.14 | 99.10 | 92.14 | 88.82 | 88.61 | 91.47 | 87.95 | 92.07 | 73.41 | 70.65 | | UDPipe 2, `chinese-gsdsimp-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 98.80 | 90.13 | 86.67 | 86.83 | 89.52 | 86.16 | 90.09 | 71.74 | 69.52 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 95.73 | 95.97 | 99.28 | 95.20 | 99.90 | 86.76 | 83.60 | | Trankit large, XLM-R large ([3]) | 2.5 | raw | 97.66 | 98.20 | 97.66 | 94.99 | 94.78 | 97.24 | 94.32 | 97.66 | 87.01 | 84.63 | | Trankit base, XLM-R base ([3]) | 2.5 | raw | 96.94 | 99.70 | 96.94 | 94.17 | 93.98 | 96.51 | 93.52 | 96.94 | 84.64 | 81.96 | - Stanza names this model `zh-hans`. Raw-text rows depend on word segmentation (Tokens 92 for Stanza, 90 for UDPipe 2 on Words), so the gold-tokenization UDPipe row is the cleaner target for the heads. Trankit's Chinese-GSDSimp figures use an older annotation of the same treebank (r2.5). ### French: UD French-GSD, test | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.71 | 95.03 | 99.50 | 97.63 | 99.50 | 97.74 | 97.05 | 98.20 | 93.27 | 91.28 | | UDPipe 2, `french-gsd-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 93.82 | 98.88 | 97.43 | — | 97.15 | 96.70 | 97.73 | 93.26 | 91.21 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 98.55 | — | 98.30 | 97.83 | 98.85 | 94.99 | 93.11 | | Trankit large, XLM-R large ([3]) | 2.5 | raw | 99.77 | 98.80 | 99.72 | 97.89 | 99.72 | 97.34 | 96.66 | 97.90 | 94.97 | 93.37 | | Trankit base, XLM-R base ([3]) | 2.5 | raw | 99.70 | 96.63 | 99.66 | 97.85 | 99.66 | 97.16 | 96.60 | 97.80 | 94.00 | 92.34 | - French-GSD carries no XPOS: UDPipe 2 reports a dash, and the XPOS scores of Stanza and Trankit equal their Words scores, as they do when every XPOS is `_`. ### Spanish: UD Spanish-AnCora, test | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.97 | 98.49 | 99.96 | 98.96 | 96.10 | 98.54 | 95.65 | 99.24 | 93.09 | 91.30 | | UDPipe 2, `spanish-ancora-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 98.69 | 99.95 | 99.05 | 96.08 | 98.74 | 95.61 | 99.37 | 93.81 | 92.07 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 99.09 | 96.12 | 98.79 | 95.64 | 99.41 | 94.01 | 92.26 | | Trankit large, XLM-R large ([3]) | 2.5 | raw | 99.91 | 99.39 | 99.91 | 99.06 | 99.00 | 98.85 | 98.36 | 99.15 | 94.77 | 93.29 | | Trankit base, XLM-R base ([3]) | 2.5 | raw | 99.94 | 99.13 | 99.93 | 99.02 | 98.94 | 98.80 | 98.27 | 99.17 | 94.11 | 92.41 | ### Portuguese: UD Portuguese-Bosque, test | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.85 | 93.10 | 99.76 | 97.45 | 99.76 | 96.44 | 95.33 | 98.04 | 90.59 | 87.75 | | UDPipe 2, `portuguese-bosque-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 89.73 | 99.68 | 97.77 | — | 96.89 | 95.82 | 98.33 | 92.16 | 89.84 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 98.09 | — | 97.15 | 96.07 | 98.63 | 93.31 | 90.93 | | Trankit large, XLM-R large ([3]) | 2.5 | raw | 99.75 | 97.18 | 99.67 | 97.52 | 99.67 | 96.50 | 95.17 | 97.97 | 93.31 | 90.91 | | Trankit base, XLM-R base ([3]) | 2.5 | raw | 99.70 | 97.48 | 99.65 | 97.27 | 99.65 | 96.50 | 94.95 | 97.89 | 92.76 | 90.25 | - Bosque carries no XPOS; same reading as for French-GSD. ### German: UD German-GSD, test | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.62 | 83.84 | 99.60 | 95.61 | 97.36 | 89.78 | 87.19 | 97.23 | 85.80 | 81.80 | | UDPipe 2, `german-gsd-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 83.32 | 99.79 | 96.87 | 97.71 | 91.17 | 88.82 | 97.19 | 87.44 | 84.36 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 97.11 | 97.93 | 91.53 | 89.21 | 97.37 | 89.24 | 86.16 | | Trankit large, XLM-R large ([3]) | 2.5 | raw | 99.71 | 89.72 | 99.72 | 95.23 | 97.68 | 91.68 | 87.21 | 96.58 | 89.01 | 85.20 | | Trankit base, XLM-R base ([3]) | 2.5 | raw | 99.75 | 92.72 | 99.75 | 95.04 | 97.57 | 91.51 | 86.86 | 96.60 | 88.73 | 84.77 | ### Italian: UD Italian-ParlaMint and MarkIT, test (the standard Italian treebanks are NC) | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1 ParlaMint, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.59 | 95.83 | 99.01 | 96.83 | 95.96 | 95.28 | 93.74 | 95.64 | 85.19 | 81.45 | | UDPipe 2, `italian-parlamint-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 94.12 | 99.42 | 98.58 | 97.99 | 97.95 | 97.09 | 98.69 | 91.79 | 89.72 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 99.19 | 98.55 | 98.49 | 97.61 | 99.20 | 93.24 | 91.13 | | Stanza 1.5.1 MarkIT, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 100.00 | 99.12 | 99.67 | 96.29 | 96.34 | 94.28 | 92.34 | 87.08 | 85.58 | 80.63 | | UDPipe 2 MarkIT, `italian-markit-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 98.24 | 99.62 | 97.02 | 97.03 | 94.02 | 92.37 | 88.31 | 88.68 | 84.79 | | UDPipe 2 MarkIT, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 97.36 | 97.36 | 94.28 | 92.60 | 88.62 | 89.41 | 85.54 | - The rows are labelled by treebank where the table mixes two. Each system was trained on that treebank's own training split of its release; ParlaMint and MarkIT are young treebanks whose splits may differ between releases 2.12, 2.17 and 2.18, which is not checked sentence by sentence. Our fast tier trains on 897 sentences (`tools/fetch/it/MANIFEST.md`). - For information only (ISDT is CC BY-NC-SA and not usable by us): Stanza 1.5.1 on Italian-ISDT, raw, UD 2.12: 99.93 99.07 99.79 98.41 98.32 98.14 97.69 98.23 93.29 91.61 (Tokens, Sents, Words, UPOS, XPOS, UFeats, AllTags, Lemmas, UAS, LAS); UDPipe 2.17 Italian-ISDT raw: UPOS 98.49, Lemma 98.67, LAS 93.10. ### Turkish: UD Turkish-BOUN, test (IMST is NC) | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.58 | 90.08 | 96.32 | 89.92 | 85.74 | 81.66 | 73.03 | 90.37 | 71.53 | 65.17 | | UDPipe 2, `turkish-boun-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 86.25 | 96.57 | 90.01 | 86.01 | 81.08 | 71.65 | 90.59 | 73.16 | 67.27 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 93.14 | 89.02 | 83.03 | 73.41 | 93.62 | 80.54 | 74.00 | - Trankit (UD 2.5) predates BOUN; its Turkish model is IMST (NC), information only: Trankit large raw UPOS 96.20, Lemmas 96.21, UAS 78.94, LAS 73.14 (IMST, not our test set). - BOUN's "Words" differ from "Tokens" (96.32 vs 99.58 in Stanza's run) because the treebank splits agglutinated forms into syntactic words in multiword tokens. ### Hindi: no allowed training treebank; Hindi-PUD is eval-only | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| - Our own Hindi model is trained on self-labelled web text (labelled with the help of related-language heads) and scored on Hindi-PUD (`results-table.md`, "Hindi"); the black-box reference for it is spaCy 3.8.18 trained on the same labels. Every published system is trained and scored on Hindi-HDTB, which is CC BY-NC-SA 4.0 and not usable by us. No published number is on Hindi-PUD, so there is no reference row for the test set we can fetch. For information only, on HDTB: Stanza 1.5.1 raw, UD 2.12: 100.00 99.52 100.00 97.95 97.53 94.41 92.74 96.88 95.26 92.46 (Tokens, Sents, Words, UPOS, XPOS, UFeats, AllTags, Lemmas, UAS, LAS); UDPipe 2.17 raw: UPOS 97.61, Lemma 98.86, LAS 92.30; Trankit large raw, UD 2.5: UPOS 98.01, Lemmas 96.54, UAS 95.95, LAS 92.96. ### Dutch: UD Dutch-Alpino, test | System (source) | UD release | Input | Tokens | Sents | Words | UPOS | XPOS | UFeats | AllTags | Lemmas | UAS | LAS | |---|---|---|---|---|---|---|---|---|---|---|---|---| | Stanza 1.5.1, pretrained, charlm, no transformer ([1]) | 2.12 | raw | 99.87 | 90.08 | 99.87 | 96.75 | 95.50 | 97.00 | 94.97 | 95.33 | 91.42 | 88.67 | | UDPipe 2, `dutch-alpino-ud-2.17-251125`, mBERT embeddings ([2]) | 2.17 | raw | n/a | 91.16 | 99.78 | 97.97 | 96.89 | 97.52 | 96.56 | 95.99 | 94.12 | 92.25 | | UDPipe 2, same model ([2]) | 2.17 | gold | n/a | n/a | n/a | 98.17 | 97.12 | 97.75 | 96.73 | 96.20 | 95.30 | 93.37 | | Trankit large, XLM-R large ([3]) | 2.5 | raw | 99.43 | 90.65 | 99.43 | 96.67 | 95.01 | 96.55 | 94.75 | 96.39 | 94.41 | 92.49 | | Trankit base, XLM-R base ([3]) | 2.5 | raw | 99.22 | 89.88 | 99.22 | 96.55 | 94.92 | 96.22 | 94.56 | 96.23 | 93.28 | 91.28 | - Alpino was reworked between UD 2.5 and 2.17 (lemmas and relations), which accounts for part of the gap between Trankit's and the later systems' rows. ## The nine 0.5 languages: named entities, gold tokens, entity-level micro F1 Only rows on our evaluation corpus, or on the corpus the nearest published row used, are listed; every row says which. | Language | Our corpus (test) | System (source) | F1 | Comparable? | |---|---|---|---|---| | zh | UNER Chinese-GSDSimp | XLM-R large fine-tuned on the same corpus ([4], Table 6) | .894 (LOC .906, ORG .802, PER .925) | same corpus, same scheme (UNER v1 commit differs from the r2.18 tokens in 5 sentences) | | zh | UNER Chinese-PUD | XLM-R large fine-tuned on English EWT, zero-shot ([4], Table 6) | .410 | information: cross-lingual, not a target | | pt | UNER Portuguese-Bosque | XLM-R large fine-tuned on the same corpus ([4], Table 6) | .904 (LOC .882, ORG .861, PER .966) | same corpus, same scheme | | pt | UNER Portuguese-PUD | XLM-R large fine-tuned on English EWT, zero-shot ([4], Table 6) | .848 | information | | de | GermEval 2014 | Stanza NER model, Stanza 1.x ([6]) | 85.2 | same corpus; the page lists four classes, as our baseline uses; it does not state the split | | de | GermEval 2014 | Trankit large / base ([7]) | 89.4 / 86.9 | same corpus; the page does not state the split or the classes | | de | UNER German-PUD | XLM-R large fine-tuned on English EWT, zero-shot ([4], Table 6) | .814 | information | | fr | WikiNER French | Stanza ([6]) / Trankit large / base ([7]) | 92.9 / 92.9 / 92.3 | same corpus, **different split**: ours cuts by article at fixed positions, theirs is their own; information, not a gate | | nl | WikiNER Dutch | Stanza ([6]) / Trankit large / base ([7]) | 94.8 / 95.0 / 94.8 | same corpus, different split, as for French | | es | WikiNER Spanish | none published on this corpus | | Stanza reports CoNLL-2002 88.1 and AnCora 88.6, Trankit CoNLL-2002 89.2 / 88.9: other corpora, not comparable | | it | WikiNER Italian | none published on this corpus | | Stanza reports FBK 87.92: another corpus | | tr | Turkish-WikiNER | none read | | Stanza reports Starlang 81.65: another corpus | | hi | Naamapadam Hindi | none read | | Stanza reports IL-NER 84.16: another corpus | So the reference for the NER rows of es, it, tr and hi is the black-box spaCy 3.8.18 baseline on the same data (`tools/baselines/spacy/lang/PROVENANCE.md`), as for English. ## Sources 1. Stanza: Qi, Zhang, Zhang, Bolton, Manning (2020), *Stanza: A Python Natural Language Processing Toolkit for Many Human Languages*, ACL System Demonstrations (arXiv 2003.07082). Performance page, "System Performance on UD Treebanks", Stanza 1.5.1 on UD 2.12: . 2. UDPipe 2: Straka (2018), *UDPipe 2.0 Prototype at CoNLL 2018 UD Shared Task*, CoNLL 2018; Straka, Straková, Hajič (2019), *Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing* (arXiv 1904.02099). Published model documentation with per-treebank results, "Universal Dependencies 2.17 Models", version 251125: . 3. Trankit: Nguyen, Lai, Veyseh, Nguyen (2021), *Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing*, EACL System Demonstrations (arXiv 2101.03289). Results page, "Universal Dependencies v2.5": . 4. Mayhew, Blevins, Liu, Šuppa, Gonen, Imperial, Karlsson, Lin, Ljubešić, Miranda, Plank, Riab, Pinter (2024), *Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark*, NAACL 2024, pp. 4322–4337, (Table 6 in the appendix). 5. CoNLL 2018 shared-task evaluation: . 6. Stanza NER models page, F1 per corpus: (retrieved 2026-11-05). 7. Trankit results page, "Named Entity Recognition" (Trankit large, Trankit base, Stanza v1.1.1; micro F1 on the test sets of 11 datasets) and the 90-treebank table (UD 2.5; columns Tokens, Sents, Words, UPOS, XPOS, UFeats, AllTags, Lemmas, UAS, LAS, CLAS, MLAS, BLEX): (retrieved 2026-11-05). 8. Nothman, Ringland, Radford, Murphy, Curran (2013), *Learning multilingual named entity recognition from Wikipedia*, Artificial Intelligence 194; data: (CC BY 4.0 per the record). 9. Benikova, Biemann, Reznicek (2014), *NoSta-D Named Entity Annotation for German: Guidelines and Dataset*, LREC 2014; GermEval 2014 NER shared task. 10. Mhaske, Kedia, Doddapaneni, Khapra, Kumar, Murthy, Kunchukuttan (2023), *Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages*, ACL 2023 (arXiv 2212.10168). 11. Turkish-WikiNER (turkish-nlp-suite): data card (CC BY-SA 4.0 per the card); no paper was read. ## Every gating treebank: Stanza, UDPipe 2 and Trankit `docs/references.tsv` holds every number as published, one row per system, treebank and input mode, for all treebanks the three sources list (655 rows). Columns: `system`, `system_version`, `model`, `treebank` (repository name), `ud_release`, `input` (`raw` or `gold` tokenization), `tokens`, `sents`, `words`, `upos`, `xpos`, `ufeats`, `alltags`, `lemmas`, `uas`, `las`, `clas`, `mlas`, `blex` (F1 × 100, empty when the source does not report it), `source`, then the comparison with r2.18: `in_r2_18`, `test_sents_ref`/`test_tokens_ref` (the test split at the source's release), `test_sents_r2_18`/`test_tokens_r2_18`, and `test_changed`. Sources (read 2026-11-06, copied, nothing re-run): - **Stanza 1.5.1**, pretrained models with charlm and no transformer, UD 2.12, end to end from raw text, CoNLL 2018 script: `stanfordnlp.github.io/stanza/performance.html` (137 treebanks). - **UDPipe 2**, models `*-ud-2.17-251125` on UD 2.17 (mBERT embeddings), raw text and gold tokenization: `ufal.mff.cuni.cz/udpipe/2/models`. The models are CC BY-NC-SA; only the published numbers are used. The page reports Words, not Tokens. - **Trankit** (large = XLM-R large, base = XLM-R base), UD 2.5, raw text: `trankit.readthedocs.io/en/latest/performance.html` (EACL 2021, Nguyen et al.). The page repeats runs of Stanza 1.1.1, UDPipe 1.2 and spaCy 2.3 on UD 2.5; those are not used. **Version caveat.** `test_changed` compares the number of sentences and tokens in the test split at the source's release tag with r2.18 (`stats.xml` of each repository at that tag): `same size` means the test set has the same size (annotations may still have been corrected: UD fixes labels in most releases, so scores move by a few tenths even then), `changed` means the test set itself differs in size and the comparison is information, not a gate. Rows whose treebank has no allowed data are marked `not checked`. Compare like with like: Stanza and Trankit are raw-text end-to-end; UDPipe 2 also gives gold-tokenization rows. The table lists UPOS / UFeats / Lemmas / UAS / LAS for each gating treebank (the treebank per language is in `docs/ud-licences.md`). Stanza: UD 2.12, raw. UDPipe 2: UD 2.17, raw and gold. Trankit large: UD 2.5, raw. `=` means the test split has the same size in r2.18 as in the source's release, `≠` that it differs, `-` that the source does not list the treebank. | Language | Treebank | Stanza 2.12 | UDPipe 2 raw (2.17) | UDPipe 2 gold (2.17) | Trankit large (2.5) | |---|---|---|---|---|---| | Afrikaans | Afrikaans-AfriBooms | 98.60 / 98.41 / 98.29 / 89.54 / 86.14 = | 98.64 / 98.23 / 98.29 / 90.57 / 87.56 = | 98.73 / 98.32 / 98.36 / 90.76 / 87.76 = | 98.87 / 98.68 / 97.47 / 91.61 / 89.35 ≠ | | Albanian | Albanian-STAF | - | 87.90 / 70.16 / 80.65 / 77.96 / 65.86 = | 88.71 / 70.16 / 81.99 / 77.15 / 64.52 = | - | | Ancient Greek | Ancient_Greek-PTNK | - | 98.57 / 91.75 / 94.84 / 88.44 / 84.30 = | 98.55 / 91.80 / 94.85 / 92.12 / 87.85 = | - | | Armenian | Armenian-ArmTDP | 83.05 / 66.22 / 91.35 / 74.04 / 61.80 ≠ | 97.67 / 94.17 / 96.76 / 89.91 / 86.17 = | 98.06 / 94.57 / 97.13 / 90.79 / 87.01 = | 95.03 / 90.42 / 93.45 / 86.47 / 83.01 ≠ | | Assamese | Assamese-AiW | - | - | - | - | | Bororo | Bororo-BDT | - | 91.68 / 89.76 / 91.56 / 78.80 / 72.57 = | 91.73 / 89.82 / 91.56 / 79.11 / 72.85 = | - | | Buryat | Buryat-BDT | 75.16 / 74.51 / 77.78 / 56.21 / 40.52 ≠ | - | - | - | | Catalan | Catalan-AnCora | 98.70 / 98.27 / 99.06 / 93.55 / 91.66 ≠ | 99.07 / 98.67 / 99.43 / 94.84 / 93.32 = | 99.15 / 98.75 / 99.49 / 94.98 / 93.46 = | 99.11 / 98.66 / 98.49 / 95.15 / 93.83 ≠ | | Central Kurdish | Central_Kurdish-Mukri | - | - | - | - | | Chinese | Chinese-GSDSimp | 88.82 / 91.47 / 92.07 / 73.41 / 70.65 ≠ | 86.67 / 89.52 / 90.09 / 71.74 / 69.52 = | 95.73 / 99.28 / 99.90 / 86.76 / 83.60 = | 94.99 / 97.24 / 97.66 / 87.01 / 84.63 ≠ | | Classical Chinese | Classical_Chinese-Kyoto | 89.16 / 91.38 / 97.63 / 71.32 / 65.80 = | 89.58 / 91.65 / 97.58 / 71.49 / 66.32 = | 93.21 / 94.56 / 99.53 / 84.45 / 79.13 = | 92.89 / 94.43 / 99.67 / 82.03 / 76.82 ≠ | | Coptic | Coptic-Scriptorium | 85.77 / 86.37 / 85.95 / 66.37 / 64.43 ≠ | 73.46 / 72.83 / 74.21 / 52.08 / 50.03 ≠ | 97.42 / 97.94 / 97.67 / 91.07 / 88.27 ≠ | - | | Croatian | Croatian-SET | 98.21 / 95.27 / 96.60 / 90.29 / 86.95 = | 98.46 / 96.36 / 97.64 / 92.36 / 89.46 = | 98.53 / 96.44 / 97.71 / 92.88 / 89.97 = | 98.58 / 96.85 / 96.71 / 93.86 / 89.74 = | | Czech | Czech-CAC | 98.52 / 93.64 / 97.46 / 91.09 / 88.71 = | 99.58 / 97.85 / 99.27 / 96.25 / 94.95 = | 99.59 / 97.85 / 99.28 / 96.25 / 94.95 = | 99.64 / 98.01 / 98.30 / 95.48 / 93.97 = | | Danish | Danish-DDT | 98.30 / 97.87 / 97.88 / 88.61 / 86.61 = | 97.99 / 97.44 / 97.52 / 88.59 / 86.60 = | 98.20 / 97.63 / 97.65 / 89.84 / 87.87 = | 98.71 / 98.33 / 97.35 / 91.75 / 90.33 = | | Dutch | Dutch-Alpino | 96.75 / 97.00 / 95.33 / 91.42 / 88.67 = | 97.97 / 97.52 / 95.99 / 94.12 / 92.25 = | 98.17 / 97.75 / 96.20 / 95.30 / 93.37 = | 96.67 / 96.55 / 96.39 / 94.41 / 92.49 = | | Egyptian | Egyptian-PC | - | - | - | - | | English | English-EWT | 96.20 / 96.39 / 96.63 / 88.90 / 86.77 ≠ | 96.75 / 97.21 / 97.13 / 91.59 / 89.91 = | 97.61 / 98.13 / 97.96 / 94.09 / 92.36 = | 96.65 / 96.75 / 97.25 / 91.29 / 89.40 ≠ | | Erzya | Erzya-JR | 84.84 / 75.88 / 84.72 / 70.08 / 57.35 ≠ | 88.50 / 79.36 / 84.27 / 73.33 / 63.92 ≠ | 89.25 / 79.93 / 84.87 / 74.25 / 64.70 ≠ | - | | Faroese | Faroese-FarPaHC | 96.51 / 92.92 / 99.64 / 84.39 / 80.07 = | 97.48 / 94.37 / 99.74 / 86.48 / 82.71 = | 97.65 / 94.59 / 100.00 / 87.41 / 83.58 = | - | | Finnish | Finnish-FTB | 96.88 / 97.12 / 97.10 / 90.99 / 88.63 = | 96.73 / 96.69 / 95.66 / 90.21 / 87.31 = | 97.05 / 96.85 / 95.81 / 92.38 / 89.42 = | 98.32 / 98.09 / 96.94 / 95.84 / 94.53 = | | French | French-GSD | 97.63 / 97.74 / 98.20 / 93.27 / 91.28 ≠ | 97.43 / 97.15 / 97.73 / 93.26 / 91.21 = | 98.55 / 98.30 / 98.85 / 94.99 / 93.11 = | 97.89 / 97.34 / 97.90 / 94.97 / 93.37 ≠ | | Galician | Galician-TreeGal | 93.89 / 93.11 / 94.52 / 77.27 / 71.86 = | 96.10 / 95.22 / 97.31 / 83.47 / 79.23 = | 97.35 / 96.30 / 98.56 / 86.85 / 82.33 = | 97.62 / 96.50 / 91.08 / 87.17 / 83.90 = | | Georgian | Georgian-GLC | - | 96.37 / 91.80 / 94.16 / 83.58 / 79.76 = | 97.19 / 92.52 / 94.86 / 85.33 / 81.35 = | - | | German | German-GSD | 95.61 / 89.78 / 97.23 / 85.80 / 81.80 ≠ | 96.87 / 91.17 / 97.19 / 87.44 / 84.36 = | 97.11 / 91.53 / 97.37 / 89.24 / 86.16 = | 95.23 / 91.68 / 96.58 / 89.01 / 85.20 ≠ | | Greek | Greek-GUD | 96.52 / 92.50 / 92.63 / 89.81 / 85.33 ≠ | 98.08 / 94.36 / 95.93 / 93.61 / 90.37 ≠ | 98.13 / 94.41 / 95.97 / 94.05 / 90.79 ≠ | - | | Haitian Creole | Haitian_Creole-Adolphe | - | 95.53 / 97.98 / 99.02 / 64.82 / 60.63 = | 95.55 / 98.02 / 99.05 / 64.83 / 60.63 = | - | | Hebrew | Hebrew-IAHLTwiki | 91.69 / 86.88 / 92.51 / 83.23 / 81.19 ≠ | 86.16 / 81.75 / 87.24 / 75.73 / 74.01 = | 97.12 / 92.55 / 98.30 / 93.63 / 91.23 = | - | | Icelandic | Icelandic-IcePaHC | 96.36 / 90.42 / 95.78 / 86.79 / 82.85 ≠ | 96.92 / 92.10 / 96.33 / 87.27 / 83.41 = | 97.08 / 92.24 / 96.47 / 87.73 / 83.82 = | - | | Indonesian | Indonesian-GSD | 94.16 / 95.58 / 98.28 / 87.05 / 80.83 = | 94.35 / 95.55 / 98.14 / 87.65 / 81.91 = | 94.79 / 95.99 / 98.59 / 88.53 / 82.72 = | 93.39 / 96.11 / 99.53 / 86.33 / 79.81 ≠ | | Irish | Irish-IDT | 96.01 / 89.41 / 95.32 / 86.81 / 81.51 = | 96.35 / 91.32 / 95.77 / 87.31 / 81.93 = | 96.46 / 91.43 / 95.88 / 87.61 / 82.20 = | 94.72 / 80.90 / 92.64 / 83.47 / 76.86 ≠ | | Italian | Italian-ParlaMint | 96.83 / 95.28 / 95.64 / 85.19 / 81.45 = | 98.58 / 97.95 / 98.69 / 91.79 / 89.72 = | 99.19 / 98.49 / 99.20 / 93.24 / 91.13 = | - | | Japanese | Japanese-GSD | 96.38 / 97.36 / 96.02 / 90.01 / 89.29 = | 95.07 / 96.16 / 95.10 / 88.04 / 87.28 = | 98.61 / 99.99 / 98.53 / 94.94 / 94.03 = | 93.66 / 95.23 / 94.68 / 86.67 / 85.56 ≠ | | Kazakh | Kazakh-KTB | 85.58 / 78.70 / 89.21 / 60.55 / 51.38 = | - | - | 77.94 / 63.01 / 50.60 / 47.46 / 37.98 ≠ | | Komi Permyak | Komi_Permyak-UH | - | - | - | - | | Komi Zyrian | Komi_Zyrian-Lattice | - | - | - | - | | Korean | Korean-Kaist | 95.74 / 100.00 / 94.10 / 87.63 / 85.50 = | 96.21 / - / 94.39 / 88.95 / 86.97 = | 96.21 / - / 94.39 / 88.95 / 86.97 = | 95.13 / 98.70 / 92.36 / 90.00 / 88.22 = | | Kyrgyz | Kyrgyz-KTMU | 87.65 / 60.22 / 88.09 / 74.95 / 58.77 ≠ | 90.96 / 77.29 / 88.71 / 84.01 / 73.36 = | 91.66 / 77.98 / 89.39 / 84.78 / 74.02 = | - | | Latin | Latin-LLCT | 99.57 / 96.79 / 98.09 / 96.13 / 94.88 ≠ | 99.77 / 97.12 / 97.80 / 95.03 / 94.06 = | 99.78 / 97.12 / 97.80 / 95.05 / 94.07 = | - | | Latvian | Latvian-LVTB | 96.70 / 94.73 / 96.12 / 88.91 / 85.77 ≠ | 97.43 / 95.28 / 96.81 / 89.35 / 86.48 = | 98.09 / 95.97 / 97.48 / 90.37 / 87.47 = | 97.61 / 95.18 / 95.83 / 93.63 / 91.25 ≠ | | Ligurian | Ligurian-GLT | 89.14 / 88.25 / 92.68 / 77.61 / 69.18 = | - | - | - | | Lithuanian | Lithuanian-ALKSNIS | 93.71 / 88.16 / 92.95 / 77.82 / 72.75 = | 95.96 / 91.02 / 93.40 / 82.45 / 78.63 = | 96.09 / 91.19 / 93.47 / 83.76 / 79.92 = | 97.45 / 94.46 / 94.30 / 90.48 / 87.67 = | | Livvi | Livvi-KKPP | - | - | - | - | | Low Saxon | Low_Saxon-LSDC | - | 90.25 / 71.74 / 83.84 / 73.91 / 64.88 = | 90.80 / 72.23 / 84.33 / 75.13 / 65.93 = | - | | Maltese | Maltese-MUDT | 95.72 / 99.85 / 99.85 / 83.62 / 78.22 = | 95.78 / 99.84 / - / 84.86 / 80.04 = | 95.88 / 100.00 / - / 85.36 / 80.47 = | - | | Marathi | Marathi-UFAL | 85.78 / 73.53 / 82.84 / 68.87 / 59.56 = | 83.94 / 74.45 / 84.67 / 67.15 / 59.85 = | 88.59 / 76.94 / 87.86 / 73.06 / 64.81 = | 87.79 / 70.62 / 81.50 / 72.79 / 63.36 = | | Moksha | Moksha-JR | - | - | - | - | | Naija | Naija-NSC | 97.55 / 98.84 / 99.14 / 92.39 / 89.56 = | 98.00 / 98.90 / 99.28 / 92.62 / 89.97 = | 98.02 / 98.93 / 99.31 / 92.68 / 90.03 = | - | | North Sami | North_Sami-Giella | 90.73 / 88.08 / 88.54 / 72.26 / 66.63 = | 91.83 / 89.47 / 86.94 / 76.20 / 71.26 = | 91.94 / 89.62 / 87.05 / 76.44 / 71.52 = | - | | Northern Kurdish | Northern_Kurdish-Kurmanji | - | - | - | 75.07 / 57.15 / 57.63 / 37.12 / 29.89 = | | Norwegian | Norwegian-Bokmaal | 98.38 / 97.09 / 98.61 / 93.26 / 91.85 = | 98.38 / 97.38 / 98.67 / 94.05 / 92.84 = | 98.57 / 97.59 / 98.87 / 94.69 / 93.47 = | 98.85 / 98.07 / 98.40 / 95.54 / 94.33 = | | Old East Slavic | Old_East_Slavic-RNC | 91.04 / 72.39 / 81.04 / 68.10 / 60.81 ≠ | 97.77 / 90.13 / 90.60 / 80.58 / 77.10 = | 98.02 / 90.38 / 90.81 / 82.12 / 78.58 = | - | | Old Occitan | Old_Occitan-CorAG | - | 95.25 / 95.37 / - / 77.79 / 74.80 ≠ | 96.83 / 97.02 / - / 83.90 / 80.85 ≠ | - | | Ottoman Turkish | Ottoman_Turkish-BOUN | - | 85.77 / 80.51 / 81.99 / 63.60 / 52.37 = | 86.18 / 80.96 / 82.36 / 64.41 / 53.06 = | - | | Persian | Persian-Seraji | 97.69 / 97.70 / 98.18 / 90.98 / 87.96 = | 97.97 / 97.95 / 98.57 / 92.00 / 89.17 = | 98.30 / 98.27 / 98.85 / 92.62 / 89.76 = | 97.78 / 97.70 / 97.35 / 92.24 / 89.58 = | | Polish | Polish-MPDT | - | 94.59 / 89.44 / 94.70 / 78.97 / 75.06 = | 96.29 / 91.02 / 96.35 / 82.66 / 78.54 = | - | | Portuguese | Portuguese-Bosque | 97.45 / 96.44 / 98.04 / 90.59 / 87.75 = | 97.77 / 96.89 / 98.33 / 92.16 / 89.84 = | 98.09 / 97.15 / 98.63 / 93.31 / 90.93 = | 97.52 / 96.50 / 97.97 / 93.31 / 90.91 ≠ | | Punjabi | Punjabi-Rang | - | - | - | - | | Romanian | Romanian-RRT | 97.50 / 97.10 / 97.42 / 90.70 / 86.86 ≠ | 97.86 / 97.38 / 98.10 / 92.01 / 88.84 ≠ | 98.14 / 97.64 / 98.36 / 92.77 / 89.53 ≠ | 97.90 / 97.43 / 97.98 / 93.60 / 89.50 ≠ | | Russian | Russian-GSD | 97.63 / 93.63 / 95.51 / 89.56 / 85.89 = | 98.06 / 93.77 / 97.20 / 92.04 / 89.11 = | 98.54 / 94.18 / 97.58 / 93.25 / 90.19 = | 98.25 / 95.78 / 95.75 / 92.70 / 89.75 = | | Sanskrit | Sanskrit-Vedic | 89.19 / 80.70 / 88.92 / 62.52 / 50.64 ≠ | 93.56 / 89.24 / 93.45 / 65.47 / 56.91 = | 93.83 / 90.36 / 93.50 / 77.99 / 68.85 = | - | | Scottish Gaelic | Scottish_Gaelic-ARCOSG | 94.62 / 91.91 / 95.81 / 82.73 / 77.81 ≠ | 93.87 / 90.71 / 95.19 / 81.04 / 76.86 = | 96.65 / 93.63 / 97.70 / 87.26 / 83.15 = | 94.70 / 90.71 / 95.48 / 81.60 / 76.46 ≠ | | Serbian | Serbian-SET | 98.44 / 94.31 / 96.30 / 91.44 / 88.36 = | 99.06 / 96.16 / 97.96 / 93.40 / 91.07 = | 99.06 / 96.19 / 97.95 / 94.12 / 91.77 = | 99.06 / 96.40 / 96.90 / 95.57 / 93.13 = | | Sindhi | Sindhi-Isra | - | 93.27 / 71.01 / 94.79 / 83.73 / 74.44 = | 93.41 / 71.23 / 94.88 / 85.51 / 76.42 = | - | | Skolt Sami | Skolt_Sami-Giellagas | - | - | - | - | | Slovak | Slovak-SNK | 96.34 / 90.95 / 94.08 / 89.13 / 86.58 = | 97.65 / 93.14 / 96.59 / 91.41 / 89.66 = | 97.83 / 93.31 / 96.66 / 93.75 / 91.91 = | 97.90 / 95.22 / 94.79 / 96.97 / 95.60 ≠ | | Slovenian | Slovenian-SSJ | 98.41 / 96.09 / 97.69 / 92.45 / 90.80 = | 98.85 / 97.10 / 98.69 / 94.47 / 92.92 = | 98.91 / 97.16 / 98.73 / 94.65 / 93.10 = | 99.24 / 97.99 / 97.55 / 96.91 / 96.06 ≠ | | Southern Kurdish | Southern_Kurdish-Garrusi | - | 77.26 / 72.92 / 67.15 / 68.59 / 62.09 = | 83.82 / 78.68 / 75.00 / 76.47 / 68.38 = | - | | Spanish | Spanish-AnCora | 98.96 / 98.54 / 99.24 / 93.09 / 91.30 ≠ | 99.05 / 98.74 / 99.37 / 93.81 / 92.07 = | 99.09 / 98.79 / 99.41 / 94.01 / 92.26 = | 99.06 / 98.85 / 99.15 / 94.77 / 93.29 ≠ | | Swedish | Swedish-Talbanken | 98.28 / 97.29 / 97.92 / 90.32 / 87.85 = | 98.67 / 97.24 / 98.53 / 92.89 / 90.52 = | 98.79 / 97.36 / 98.66 / 93.39 / 91.02 = | 99.06 / 98.04 / 97.87 / 94.41 / 92.97 = | | Telugu | Telugu-MTG | 93.97 / 99.10 / 99.79 / 90.09 / 81.64 = | 93.77 / 98.48 / - / 90.44 / 84.35 = | 94.17 / 98.89 / - / 91.26 / 85.16 = | 94.31 / 98.33 / 98.89 / 93.19 / 87.08 = | | Telugu English | Telugu_English-TECT | - | - | - | - | | Thai | Thai-TUD | - | 77.29 / 87.14 / - / 58.97 / 50.87 = | 89.39 / 100.00 / - / 81.27 / 71.00 = | - | | Turkish | Turkish-BOUN | 89.92 / 81.66 / 90.37 / 71.53 / 65.17 = | 90.01 / 81.08 / 90.59 / 73.16 / 67.27 = | 93.14 / 83.03 / 93.62 / 80.54 / 74.00 = | - | | Ukrainian | Ukrainian-ParlaMint | - | 98.56 / 95.01 / 99.07 / 93.95 / 91.91 = | 98.60 / 95.02 / 99.12 / 93.96 / 91.92 = | - | | Uyghur | Uyghur-UDT | 89.26 / 87.58 / 95.30 / 74.80 / 63.42 = | 89.87 / 88.03 / 94.64 / 75.68 / 64.94 = | 90.26 / 88.51 / 95.15 / 77.54 / 66.62 = | 88.50 / 86.56 / 94.45 / 80.19 / 70.14 = | | Uzbek | Uzbek-UzUDT | - | 86.15 / 70.68 / 29.27 / 71.16 / 55.78 ≠ | 86.24 / 70.49 / 29.22 / 73.63 / 57.97 ≠ | - | | Vietnamese | Vietnamese-VTB | 81.48 / 88.02 / 81.66 / 58.82 / 51.49 = | 78.40 / - / 85.77 / 56.34 / 49.57 = | 89.92 / - / 99.47 / 75.84 / 65.51 = | 89.70 / 94.64 / 94.88 / 71.07 / 65.37 ≠ | | Welsh | Welsh-CCG | 95.86 / 90.11 / 94.85 / 86.32 / 80.25 = | 95.50 / 89.93 / 94.89 / 87.52 / 81.76 = | 95.89 / 90.30 / 95.32 / 88.45 / 82.61 = | - | | Western Armenian | Western_Armenian-ArmTDP | 96.42 / 92.70 / 96.70 / 87.73 / 82.83 ≠ | 96.81 / 92.37 / 96.89 / 89.35 / 84.85 = | 96.90 / 92.47 / 97.01 / 89.65 / 85.14 = | - | | Wolof | Wolof-WTB | 94.36 / 93.19 / 94.63 / 83.28 / 77.32 = | 94.32 / 93.47 / 95.18 / 83.49 / 78.02 = | 95.29 / 94.33 / 95.94 / 85.75 / 80.16 = | - | | Yiddish | Yiddish-YiTB | - | 94.50 / 99.61 / 93.53 / 89.38 / 85.56 = | 94.85 / 100.00 / 93.87 / 90.34 / 86.48 = | - | | Zaar | Zaar-Autogramm | - | 84.94 / 85.78 / 82.47 / 70.99 / 62.81 ≠ | 90.67 / 91.37 / 87.42 / 83.04 / 73.34 ≠ | - | | Zazaki | Zazaki-ZSD | - | - | - | - | 73 of the 84 gating treebanks have published numbers from at least one source. No source lists: Assamese-AiW, Central_Kurdish-Mukri, Egyptian-PC, Komi_Permyak-UH, Komi_Zyrian-Lattice, Livvi-KKPP, Moksha-JR, Punjabi-Rang, Skolt_Sami-Giellagas, Telugu_English-TECT, Zazaki-ZSD. For these the accurate tier reports its own numbers with the bootstrap interval and no reference row (small treebanks, mostly under 100 training sentences, which the three systems do not cover). Of the rows on gating treebanks, 75 are on a test split whose size differs from r2.18 (marked ≠); most are Trankit's UD 2.5 and Stanza's UD 2.12 rows on treebanks that grew or were resplit since. # Model provenance logs Every trained model, ours or a baseline we train for comparison, ships with a provenance log. The log is a Markdown file named `PROVENANCE.md` next to the recipe that produced the model; it is packed into the model package's notices. It answers four questions: what data went in, who produced the labels, how to reproduce it, and why the evaluation can be trusted. ## Rules the log records - **Allowed training data:** permissive data (public domain, CC0, CC BY, MIT, Apache-2.0, BSD), CC BY-SA and ODbL data and crawled web text (with credit), our own labels, and synthetic labels from open-weight models. Out: non-commercial, research-only, paid or LDC data, and data under terms that forbid the use. - **Share-alike lineage.** Mark every model whose training data includes a CC BY-SA or ODbL source (`SA lineage: yes`), with the sources, so the credit and any later licensing question can be answered from the log alone. - **Evaluation data never trains.** Every data file has a role (`train`, `dev`, `eval`) in the data manifest. Training code reads `train` and `dev` files only. The log records the manifest hash and the result of the train/test separation check. ## Template Copy this block, fill every field, and write `none` or `n/a` where a field does not apply. Do not leave a field out. ````markdown # Provenance: - **Model:** ; - **Recipe:** ; config hash - **Seed(s):** - **Produced:** ; machine ; wall time ; software - **Weights:** , bytes, sha256 - **SA lineage:** ; - **Licence of the weights:** ## Training data | Dataset | Release | Role | Files | Licence (quoted from the dataset's own licence file) | URL | sha256 | |---|---|---|---|---|---|---| ## Label generator ## Pretrained components ## Evaluation data | Dataset | Release | Role | Files | sha256 | Used for | |---|---|---|---|---|---| ## Train/test separation - Manifest: sha256 - Text-level overlap of each evaluation file with the training and dev files: - Evaluation files were not used for early stopping, model selection or hyperparameter choice: ## Results ## Notes ```` ## Where the pieces live - The data manifest for a dataset family is written by its fetch script (for English, `tools/fetch/en/fetch.sh` writes `manifest-en.json` under the data root, and `tools/fetch/en/MANIFEST.md` keeps the hashes and licences in the repository). - Recipes and their logs: `recipes//PROVENANCE.md`. Baselines trained with other tools for comparison: next to the baseline scripts, for example `tools/baselines/spacy/PROVENANCE.md`. ## Data entries for the 0.5 languages One entry per language for the data our fast-tier models and their spaCy baselines train and are judged on. Each lists the datasets with release, role and licence; file-level hashes, split sizes and the train/test text overlap are in `tools/fetch//MANIFEST.md`; the baselines' own log is `tools/baselines/spacy/lang/PROVENANCE.md`. The licence verdict for every UD treebank of the language is in `docs/ud-licences.md`. **All of these are SA lineage** (CC BY-SA sources, or Wikipedia-derived text) except where a line says otherwise. Evaluation files never train: `fetch_lang.py` places them under `eval/`, and the roles in each manifest are `train`, `dev`, `eval`. | Language | Training and dev data (release, licence) | Evaluation data | Notes | |---|---|---|---| | Chinese | UD Chinese-GSDSimp r2.18 (CC BY-SA 4.0); UNER Chinese-GSDSIMP (CC BY-SA 4.0) | GSDSimp test; UNER GSDSIMP test; UD Chinese-PUD and UNER Chinese-PUD (CC BY-SA 3.0, test-only, Traditional script) | Never train on the Traditional twin GSD and test on GSDSimp, or the reverse | | French | UD French-GSD r2.18 (CC BY-SA 4.0); WikiNER French wp3 (CC BY 4.0, silver labels, Wikipedia text) | GSD test (416 sentences); UD French-PUD (CC BY-SA 3.0, test-only); WikiNER test (our article split) | Sequoia and FQB held (LGPL-LR) | | Spanish | UD Spanish-AnCora r2.18 (CC BY 4.0, not SA); WikiNER Spanish wp3 (CC BY 4.0) | AnCora test; UD Spanish-PUD; WikiNER test | The AnCora README still carries a stale GNU-licence sentence; `LICENSE.txt` and metadata say CC BY 4.0 | | Portuguese | UD Portuguese-Bosque r2.18 (CC BY-SA 4.0); UNER Portuguese-Bosque (CC BY-SA 4.0, same tokens) | Bosque test; UNER Bosque test; UD and UNER Portuguese-PUD (CC BY-SA 3.0, test-only) | | | German | UD German-GSD r2.18 (CC BY-SA 4.0); GermEval 2014 NER (CC BY 4.0, not SA) | GSD test; UD German-PUD; GermEval test; UNER German-PUD (test-only) | HDT no-go (text licensed for academic use) | | Italian | UD Italian-ParlaMint r2.18 train (CC BY-SA 4.0) and MarkIT (CC BY 4.0), derived into `it_gating`; WikiNER Italian wp3 (CC BY 4.0) | ParlaMint test (375 sentences); MarkIT test; UD Italian-PUD; WikiNER test | 897 training sentences: ISDT, VIT, ParTUT, PoSTWITA, KIParlaForest are NC; TWITTIRO no-go | | Turkish | UD Turkish-BOUN r2.18 (CC BY-SA 4.0); Turkish-WikiNER (CC BY-SA 4.0) | BOUN test; UD Turkish-PUD; Turkish-WikiNER test | IMST is NC; Penn no-go (translated Penn Treebank) | | Hindi | none (Naamapadam is out: its text comes from the CC BY-NC Samanantar corpus); web text labelled by transfer, with label heads trained on UD Marathi-UFAL and -CMUPAN, Sindhi-Isra and Punjabi-Rang and -CS (CC BY-SA 4.0) | UD Hindi-PUD (test-only); none | HDTB is NC. No allowed Hindi treebank exists | | Dutch | UD Dutch-Alpino r2.18 (CC BY-SA 4.0); WikiNER Dutch wp3 (CC BY 4.0) | Alpino test; WikiNER test | LassySmall (CC BY-SA 4.0) allowed, unused | The corpora with no official split (WikiNER train and validation) are cut by the fixed rules in each `HEADER.md`/`MANIFEST.md`; the Italian training set is derived by `derive_italian` in `tools/fetch/fetch_lang.py`. ## Data gaps - **Hindi:** no allowed UD training treebank (HDTB is non-commercial; Hindi-PUD is test-only). There is no UD baseline and no UD-trained Hindi model until allowed data exists. Hindi NER has no allowed data either: Naamapadam is out because its text comes from Samanantar (CC BY-NC 4.0), so the spaCy Hindi NER baseline run on it is withdrawn. - **Italian:** UD training is limited to 897 sentences (ParlaMint and MarkIT); ISDT, VIT and ParTUT are non-commercial. Rows are reported as they come. - **Ruled 2026-11-05:** French Sequoia and FQB (LGPL-LR) are allowed with credit, like CC BY-SA, and unused. WikiANN is out. Naamapadam is out (Samanantar text, CC BY-NC 4.0). AnCora is used under its `LICENSE.txt` (CC BY 4.0); its README still carries a stale GNU sentence. - **No-go by rule:** German-HDT (academic use only), Italian-TWITTIRO and Portuguese-DANTEStocks (tweets, platform terms), Turkish-Penn (translated from an LDC corpus). # Licences of the Universal Dependencies treebanks The first sections cover ZH, FR, ES, PT, DE, IT, TR, HI and NL in detail. The last section ("Every treebank of release 2.18", generated by `tools/fetch/ud_docs.py`) covers all 353 treebanks in 193 languages, with the summary counts, the per-language classification, the quotes behind every non-routine ruling and the named-entity data beyond UD. Every UD treebank of these nine languages in release 2.18 (15 May 2026; each repository at tag `r2.18`), with the licence as the treebank states it and the verdict for training and evaluating OneNLP models. Read on 2026-11-05 from each treebank's `README` (the `License:` metadata line and its licence section) and `LICENSE.txt`. Rule applied: CC BY-SA, CC BY and CC0 data may train models, with credit; non-commercial (NC), research-only, paid and platform-restricted text may not. A treebank inside a CC BY-SA release that is itself NC is out. - **go**: train and evaluate, with credit. Models trained on CC BY-SA data carry "SA lineage" in their provenance. - **eval-only**: test-only treebanks (they ship no training split) and treebanks kept for evaluation. - **hold**: unresolved terms or restricted source text; not used until the terms are resolved. - **no-go**: non-commercial or otherwise outside the rule; never fetched. - **chosen**: the treebank that trains (and gates) the language's fast-tier model. The release-wide notice in each README that applies to the Google-originated treebanks (GSD for German, Spanish, French and Portuguese) and to the Parallel UD (PUD) treebanks reads: the underlying sentences are ones "Google asserts no ownership over and no copyright over", provided "AS IS"; the annotations are CC BY-SA 4.0 (GSD) or CC BY-SA 3.0 (PUD). ## Chinese | Treebank | Licence line (README) | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | GSDSimp | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **go, chosen** (Simplified script) | | GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go; the Traditional-script twin of GSDSimp (same sentences): never train on one and test on the other. Not fetched | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched as a second test set; Traditional script) | | CFL | CC BY-SA 4.0 | CC BY-SA 4.0 | test | eval-only (learner essays); not fetched | | HK | CC BY-SA 4.0 | CC BY-SA 4.0 | test | eval-only (subtitles, legislative proceedings, Traditional script); not fetched | | Beginner | CC BY-NC-SA 3.0 | CC BY-NC-SA 3.0 | test | **no-go** (NC) | | PatentChar | CC BY-NC-SA 3.0 | CC BY-SA 4.0 (contradicts the README) | test | **no-go** until the maintainers resolve the contradiction (the stricter line is taken) | Classical Chinese (`lzh`) is a different language and is not covered. ## French | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **go, chosen** | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched) | | Rhapsodie | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (spoken French, small); not used | | ParisStories | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (oral French); not used | | Sequoia | LGPL-LR | none | train, dev, test | **go**: LGPL-LR is allowed with credit and treated like CC BY-SA (SA lineage). Not used for the baselines (not retrained) | | FQB | LGPL-LR | none | test | **go**, same ruling (test-only: eval) | | ParTUT | CC BY-NC-SA 4.0 | CC BY-NC-SA 4.0 | train, dev, test | **no-go** (NC) | | FTB | none in release 2.18 | none | not in the release | not available (last tagged r2.11) | ## Spanish | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | AnCora | CC BY 4.0 | CC BY 4.0 | train, dev, test | **go, chosen**. The README still carries an older sentence, "The GNU license is inherited from the original dataset", while its changelog says "The license changed to CC BY 4.0" in release 2.8 and `LICENSE.txt` and the metadata agree on CC BY 4.0. Recorded as a stale sentence, low risk | | GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go; not used | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched) | | COSER | CC BY-SA 4.0 | CC BY-SA 4.0 | test | eval-only (spoken rural Spanish); not fetched | | SVarT, Spanish-English Miami | not in release 2.18 | none | none | not available | Spanish Sign Language (`LSE`) is not Spanish text and is not covered. ## Portuguese | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | Bosque | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **go, chosen** (Universal NER annotates the same sentences) | | GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (the changelog: Google dropped the "NC" restriction from the annotations); not used | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched) | | PetroGold | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (oil and gas academic texts); not used | | Porttinari | CC BY 4.0 | CC BY 4.0 | train, dev, test | go on the authors' statement (newspaper text); not used | | DANTEStocks | CC BY 4.0 | CC BY 4.0 | train, dev, test | **no-go** (ruled): tweets, platform terms apply to the text | | CINTIL | CC BY-NC-ND 4.0 | CC BY-NC-ND 4.0 | train, dev, test | **no-go** (NC, ND) | | DHBB | not in release 2.18 | none | none | not available | ## German | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **go, chosen** | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched) | | HDT | CC BY-SA 4.0 (annotation) | "The annotation of the treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0" | train (in parts), dev, test | **no-go** (ruled): the README's licence section says "Heise gave permission to distribute the text for academic use", so the text is not cleared for commercial use; not used until resolved | | LIT | CC BY-NC-SA 4.0 | CC BY-NC-SA 4.0 | test | **no-go** (NC) | ## Italian | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | ISDT | CC BY-NC-SA 3.0 | "released by the authors under the Creative Commons" NC-SA 3.0 | train, dev, test | **no-go** (NC). The standard Italian treebank is not usable | | VIT | CC BY-NC-SA 3.0 | NC-SA 3.0 | train, dev, test | **no-go** (NC) | | ParTUT | CC BY-NC-SA 4.0 | NC-SA 4.0 | train, dev, test | **no-go** (NC) | | PoSTWITA | CC BY-NC-SA 4.0 | NC-SA 4.0 | train, dev, test | **no-go** (NC; also tweets) | | KIParlaForest | CC BY-NC-SA 4.0 | NC-SA 4.0 | test | **no-go** (NC) | | ParlaMint | CC BY-SA 4.0 | CC BY-SA 4.0 | train (326 sentences), test (375) | **go, chosen** by default: modern text, but tiny and with no dev split | | MarkIT | CC BY 4.0 | CC BY 4.0 | train, dev, test | go (611 grammar-example sentences), used with ParlaMint | | Old | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go; Dante's Comedy (Old Italian), not used: not modern text | | TWITTIRO | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **no-go** (ruled): ironic tweets, platform terms apply to the text | | Valico | CC BY-SA 4.0 | CC BY-SA 4.0 | test | eval-only (learner essays); not fetched | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched) | ## Turkish | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | BOUN | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **go, chosen** | | Kenet | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (dictionary examples); not used | | Tourism | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (customer reviews); not used | | FrameNet | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (grammar examples); not used | | Atis | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go on the maintainers' licence; the sentences translate the English ATIS corpus; not used | | Penn | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **no-go** (ruled): a Turkish version of the Penn Treebank, whose source sentences are LDC-licensed; not used | | GB | CC BY-SA 4.0 | CC BY-SA 4.0 | test | eval-only (grammar-book examples); not fetched | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched) | | IMST | CC BY-NC-SA 4.0 | CC BY-NC-SA 4.0 | train, dev, test | **no-go** (NC). The standard Turkish treebank is not usable | | ULU | not in release 2.18 | none | none | not available | Turkish-German `SAGT` is a code-switching treebank under CC BY-NC-SA 4.0 and is out of scope. ## Hindi | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | HDTB | CC BY-NC-SA 4.0 | "licensed under the Creative Commons License Attribution-NonCommercial-ShareAlike 4.0" | train, dev, test | **no-go** (NC). It is Hindi's only sizeable treebank | | PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | test | eval-only (fetched) | | Convers | not in release 2.18 | none | none | not available | | Hindi-English `HIENCS` | not in release 2.18 | none | none | not available (code-mixed) | **There is no allowed Hindi training treebank.** Hindi-PUD has 1,000 test sentences and no training split, so no tagger, morphologizer, lemmatizer or parser baseline can be trained for Hindi from UD data, and our Hindi models would have to learn from other data. The Hindi fast-tier model is therefore trained on web text labelled by the accurate tier and by heads trained on the Marathi, Sindhi and Punjabi treebanks above (all CC BY-SA 4.0; `recipes/hi/fast/README.md`); no Hindi treebank or Hindi-derived annotation of HDTB enters the labeller or the student. ## Dutch | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | Alpino | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | **go, chosen** (news) | | LassySmall | CC BY-SA 4.0 | CC BY-SA 4.0 | train, dev, test | go (Wikipedia; larger than Alpino); not used | ## Universal NER (v1) UNER v1 (Mayhew et al., NAACL 2024) is released under CC BY-SA 4.0 as `UniversalNER/UNER_-` repositories; the PUD-based ones keep PUD's CC BY-SA 3.0. Listing of the organisation's repositories, 2026-11-05: | Language | UNER datasets | Verdict | |---|---|---| | Chinese | GSD, GSDSIMP (train, dev, test), PUD (test) | go; PUD eval-only | | Portuguese | Bosque (train, dev, test), PUD (test) | go; PUD eval-only | | German | PUD (test) | eval-only; no training data | | French, Spanish, Italian, Turkish, Hindi, Dutch | none | gap: other sources below | Each UNER dataset is a separate corpus from its treebank: it was cut from an older UD commit. Checked against r2.18 sentence by sentence: Portuguese-Bosque has identical tokens in every sentence of every split (7,018 / 1,172 / 1,167), so its entity labels sit on r2.18's tokens; Chinese-GSDSimp has identical texts, with different tokens in 2 train, 1 dev and 2 test sentences. Scheme: PER, ORG, LOC. ## Named-entity corpora for the languages without UNER Terms read 2026-11-05 from the dataset's own page or card. "Chosen" means fetched by `tools/fetch//fetch.sh` and used for the baseline. | Corpus | Languages | Terms (as stated) | Verdict | |---|---|---|---| | WikiNER (Nothman et al. 2013), wp3 files, figshare 5462500 | fr, es, it, nl (also de, en, pt, pl, ru) | figshare record licence: CC BY 4.0. Silver labels projected from Wikipedia links; the underlying text is Wikipedia (CC BY-SA) | **go, chosen** for fr, es, it, nl; SA lineage. No official split: we split by article (see the manifests) | | GermEval 2014 NER (NoSta-D, Benikova et al. 2014) | de | "The data is released under the permissive CC-BY license" (LREC 2014 abstract); the shared-task page links the CC BY 4.0 text; HF card: cc-by-4.0 | **go, chosen** for de; gold labels, official split; 12 fine-grained types with embedded and derived spans | | Turkish-WikiNER (Altinok 2023), `turkish-nlp-suite/turkish-wikiNER` | tr | HF card: `license: cc-by-sa-4.0` | **go, chosen** for tr; SA lineage; 18 OntoNotes-style types | | Naamapadam (Mhaske et al. 2023), `ai4bharat/naamapadam` | hi | Card: "We license the actual packaging of the mined data under the Creative Commons CC0 license"; "We do not own any of the text from which this data has been extracted." Source data: "Samanantar dataset", whose card says "Creative Commons Attribution-NonCommercial 4.0 International" (`license: cc-by-nc-4.0`) | **no-go** the text comes from Samanantar, which is non-commercial, so the text sources are not permissive. Hindi NER is a logged gap | | WikiANN / PAN-X (Pan et al. 2017), `unimelb-nlp/wikiann` | all nine | HF card: `license: unknown`, "[More Information Needed]" | **no-go**: out | | WikiNEuRal, MultiNERD (Babelscape) | fr, es, it, de, nl, zh (MultiNERD) | HF cards: cc-by-nc-sa-4.0 | **no-go** (NC) | | MultiCoNER v1/v2 | hi, zh, tr, nl, de, es, fr, it, pt | HF card: cc-by-4.0 | not examined: a fine-grained scheme that does not map onto PER/ORG/LOC; candidate for a later look | | CoNLL-2002 (es, nl) | es, nl | HF card: `license: unknown`; newspaper text, no open terms stated | not used (at most evaluation, through published baselines) | | `ctoraman/atis-ner-turkish` | tr | HF: cc-by-nc-sa-4.0 | **no-go** (NC) | | Smaller Turkish NER sets on the HF hub (e-commerce, clinical, privacy, synthetic) | tr | assorted (mit, cc-by-4.0, cc0-1.0, cc-by-sa-4.0) | not used: single narrow domains, not a general-purpose gold set | NER labels follow each corpus: the baselines map them to PER, ORG, LOC (and MISC where the corpus has it) as the manifests state; Turkish maps PERSON, ORG, GPE and LOC and drops the other types to O. ## Rulings and data gaps (recorded 2026-11-05) Rulings of the project; the owner may revisit them. - **No-go by rule:** German-HDT (text licensed for academic use only, i.e. research-only); Italian-TWITTIRO and Portuguese-DANTEStocks (tweet text, platform-restricted); Turkish-Penn (translated from an LDC corpus). - **Ruled 2026-11-05:** French Sequoia and FQB (LGPL-LR) are allowed with credit, treated like CC BY-SA (go; unused, French baselines not retrained). WikiANN is out. Naamapadam is out: its text comes from Samanantar (CC BY-NC 4.0), so the Hindi NER baseline trained on it is withdrawn and Hindi NER is a logged gap beside Hindi UD. AnCora: the `LICENSE.txt` (CC BY 4.0) governs; the README still carries a stale sentence ("The GNU license is inherited ..."), which is recorded and not relied on. - **Hindi has no UD baseline and no UD-trained model.** Its only sizeable treebank (HDTB) is non-commercial and Hindi-PUD is test-only. A Hindi tagger, parser or NER model waits for allowed data (Naamapadam is out, see the NER table). - **Italian trains on 897 sentences** (ParlaMint train plus MarkIT), because ISDT, VIT and ParTUT are non-commercial. Italian rows are reported as they come and compared with spaCy trained on the same small set; they are not hidden or compared with systems trained on ISDT. ## Every treebank of release 2.18 Release 2.18 (15 May 2026) has 353 treebanks in 193 languages. Each repository was read at tag `r2.18` on 2026-11-06 (`tools/fetch/ud_survey.py`): the `License:` line of the README, the whole of `LICENSE.txt`, and the README text for the source of the sentences. The full table with sizes is `tools/fetch/ud/treebanks.tsv`; the per-language classification is `tools/fetch/ud/languages.tsv`. The sections above give the nine languages' detail; this one covers all languages with the same rules. ### Summary | Verdict | Treebanks | |---|---| | go | 129 | | eval-only | 145 | | no-go | 79 | | total | 353 | | Language class | Languages | |---|---| | **train+eval**: at least one `go` treebank with training and test splits | 84 (of which 60 have 1,000 or more training sentences in the gating treebank and 20 fewer than 500) | | **eval-only**: only test-only or evaluation data is allowed | 84 | | **gap**: every treebank is non-commercial, research-only or otherwise ruled out: the accurate tier has no UD data | 25 | | total | 193 | The nine languages with fast-tier work (zh fr es pt de it tr nl, plus en) keep their treebanks and manifests; Hindi stays a recorded gap for training (HDTB is NC), with Hindi-PUD as its test set. ### Rules applied - **go**: the README licence line is CC BY-SA, CC BY, CC0, PD, LGPL-LR or another permissive licence, `LICENSE.txt` does not contradict it, no README sentence restricts the text, and the treebank has a training split. SA lineage is recorded in provenance. Credit goes in THIRD-PARTY-NOTICES. - **eval-only**: allowed licence but no training split (PUD and other test-only treebanks). - **no-go**: non-commercial or no-derivatives licence (the README line and `LICENSE.txt` agree), research-only text, platform-restricted text (tweets, Reddit), LDC-derived text, or the rulings of 2026-11-05. - **Rulings of 2026-11-07 on the 22 treebanks that were `unclear`**: (1) the README says non-commercial and `LICENSE.txt` is the generic CC BY-SA template: no-go, the README statement wins; (2) CC BY-SA annotation over platform or terms-restricted text: no-go; (3) Polish-LFG (GNU GPL 3.0): no-go; (4) Ligurian-GLT (Computational Use of Data Agreement 1.0): go, credited. Nothing stays unclear. - The `LICENSE.txt` column abbreviates its first lines; `own text` means a bespoke document (read in full for the decision). ### Languages Gating treebank: among the `go` treebanks with training and test splits, the one the most published systems report (Stanza, UDPipe 2, Trankit), then the larger training set; the nine fast-tier languages keep theirs. Training sentences in brackets. *Extra training* are further `go` treebanks. *Test-only* are `eval-only` sets. *Not usable* lists `no-go` treebanks. | Language | Class | Gating treebank (train sentences) | Extra training | Test-only | Not usable | |---|---|---|---|---|---| | Abaza | eval-only | | | Abaza-ATB | | | Abkhaz | eval-only | | | Abkhaz-AbNC | | | Afrikaans | train+eval | Afrikaans-AfriBooms (1,315) | | | | | Akkadian | eval-only | | | Akkadian-PISANDUB, Akkadian-RIAO | | | Akuntsu | eval-only | | | Akuntsu-TuDeT | | | Albanian | train+eval | Albanian-STAF (160) | | Albanian-TSA | | | Alemannic | eval-only | | | Alemannic-DIVITAL, Alemannic-UZH | | | Amharic | eval-only | | | Amharic-ATT | | | Ancient Greek | train+eval | Ancient_Greek-PTNK (727) | | | no-go: Ancient_Greek-PROIEL, Ancient_Greek-Perseus | | Ancient Hebrew | gap | | | | no-go: Ancient_Hebrew-PTNK | | Apurina | eval-only | | | Apurina-UFPA | | | Arabic | eval-only | | | Arabic-PUD | no-go: Arabic-NYUAD, Arabic-PADT | | Armenian | train+eval | Armenian-ArmTDP (4,352) | Armenian-BSUT | | | | Assamese | train+eval | Assamese-AiW (51) | | | | | Assyrian | eval-only | | | Assyrian-AS | | | Azerbaijani | eval-only | | | Azerbaijani-TueCL | | | Bambara | eval-only | | | Bambara-CRB | | | Basque | gap | | | | no-go: Basque-BDT | | Bavarian | eval-only | | | Bavarian-MaiBaam | | | Beja | eval-only | | | Beja-Autogramm | | | Belarusian | gap | | | | no-go: Belarusian-HSE | | Bengali | eval-only | | | Bengali-BRU | | | Bhojpuri | eval-only | | | Bhojpuri-BHTB | | | Bokota | eval-only | | | Bokota-ChibErgIS | | | Bororo | train+eval | Bororo-BDT (17,107) | | | | | Brahui | eval-only | | | Brahui-Kholum | | | Breton | eval-only | | | Breton-KEB | | | Bulgarian | gap | | | | no-go: Bulgarian-BTB | | Buryat | train+eval | Buryat-BDT (20) | | | | | Cantonese | eval-only | | | Cantonese-HK | | | Cappadocian | eval-only | | | Cappadocian-AMGiC, Cappadocian-TueCL | | | Catalan | train+eval | Catalan-AnCora (13,123) | | | | | Cebuano | eval-only | | | Cebuano-GJA | | | Central Kurdish | train+eval | Central_Kurdish-Mukri (92) | | | | | Chinese | train+eval | Chinese-GSDSimp (3,997) | Chinese-GSD | Chinese-CFL, Chinese-HK, Chinese-PUD | no-go: Chinese-Beginner, Chinese-PatentChar | | Chintang | gap | | | | no-go: Chintang-CTNTB | | Chukchi | eval-only | | | Chukchi-HSE | | | Classical Armenian | gap | | | | no-go: Classical_Armenian-CAVaL | | Classical Chinese | train+eval | Classical_Chinese-Kyoto (74,609) | | Classical_Chinese-TueCL | | | Coptic | train+eval | Coptic-Scriptorium (1,467) | Coptic-Bohairic | | | | Croatian | train+eval | Croatian-SET (6,914) | | | | | Czech | train+eval | Czech-CAC (23,478) | Czech-CLTT | Czech-PUD, Czech-Poetry | no-go: Czech-FicTree, Czech-PDTC | | Danish | train+eval | Danish-DDT (4,383) | | | | | Dutch | train+eval | Dutch-Alpino (12,289) | Dutch-LassySmall | | | | Egyptian | train+eval | Egyptian-PC (1,619) | | | | | English | train+eval | English-EWT (12,544) | English-Atis, English-CHILDES, English-ESLSpok | English-CTeTex, English-LittlePrince, English-PUD, English-Pronouns | no-go: English-GENTLE, English-GUM, English-GUMReddit, English-LinES, English-ParTUT | | Erzya | train+eval | Erzya-JR (1,217) | | | | | Esperanto | eval-only | | | Esperanto-Cairo, Esperanto-Prago | | | Estonian | gap | | | | no-go: Estonian-EDT, Estonian-EWT | | Faroese | train+eval | Faroese-FarPaHC (1,020) | | Faroese-OFT | | | Finnish | train+eval | Finnish-FTB (14,981) | Finnish-TDT | Finnish-PUD | no-go: Finnish-OOD | | French | train+eval | French-GSD (14,450) | French-ALTS, French-ParisStories, French-Rhapsodie, French-Sequoia | French-FQB, French-PUD, French-PoitevinDIVITAL | no-go: French-ParTUT | | Frisian Dutch | eval-only | | | Frisian_Dutch-Fame | | | Galician | train+eval | Galician-TreeGal (600) | | Galician-PUD | no-go: Galician-CTG | | Georgian | train+eval | Georgian-GLC (2,214) | Georgian-GNC | | | | German | train+eval | German-GSD (13,813) | | German-PUD | no-go: German-HDT, German-LIT | | Gheg | eval-only | | | Gheg-GPS | | | Gorontalo | eval-only | | | Gorontalo-BungoLoLombi | | | Gothic | gap | | | | no-go: Gothic-PROIEL | | Greek | train+eval | Greek-GUD (1,071) | Greek-Cretan, Greek-Messinian | Greek-Lesbian | no-go: Greek-GDT, Greek-GLCII | | Guajajara | eval-only | | | Guajajara-TuDeT | | | Guarani | eval-only | | | Guarani-OldTuDeT | | | Gujarati | eval-only | | | Gujarati-GujTB | | | Gwichin | eval-only | | | Gwichin-TueCL | | | Haitian Creole | train+eval | Haitian_Creole-Adolphe (2,605) | | Haitian_Creole-Autogramm | | | Hausa | eval-only | | | Hausa-EasternAutogramm, Hausa-NorthernAutogramm, Hausa-SouthernAutogramm, Hausa-WesternAutogramm | | | Hebrew | train+eval | Hebrew-IAHLTwiki (4,298) | Hebrew-IAHLTknesset | Hebrew-PostRab | no-go: Hebrew-HTB | | Highland Puebla Nahuatl | eval-only | | | Highland_Puebla_Nahuatl-ITML | | | Hindi | eval-only | | | Hindi-PUD | no-go: Hindi-HDTB | | Hittite | eval-only | | | Hittite-HitTB | | | Hungarian | gap | | | | no-go: Hungarian-Szeged | | Icelandic | train+eval | Icelandic-IcePaHC (34,007) | Icelandic-GC, Icelandic-Modern | Icelandic-PUD | | | Ika | eval-only | | | Ika-ChibErgIS | | | Indonesian | train+eval | Indonesian-GSD (4,482) | Indonesian-CSUI | Indonesian-PUD | | | Irish | train+eval | Irish-IDT (4,005) | | Irish-Cadhan | no-go: Irish-TwittIrish | | Italian | train+eval | Italian-ParlaMint (326) | Italian-MarkIT, Italian-Old | Italian-PUD, Italian-Valico | no-go: Italian-ISDT, Italian-KIParlaForest, Italian-ParTUT, Italian-PoSTWITA, Italian-TWITTIRO, Italian-VIT | | Japanese | train+eval | Japanese-GSD (7,050) | Japanese-GSDLUW | Japanese-PUD, Japanese-PUDLUW | no-go: Japanese-BCCWJ, Japanese-BCCWJLUW | | Javanese | eval-only | | | Javanese-CSUI | | | Kaapor | eval-only | | | Kaapor-TuDeT | | | Kadiweu | gap | | | | no-go: Kadiweu-Unicamp | | Kangri | eval-only | | | Kangri-KDTB | | | Karelian | eval-only | | | Karelian-KKPP | | | Karo | eval-only | | | Karo-TuDeT | | | Kazakh | train+eval | Kazakh-KTB (31) | | | | | Khoekhoe | gap | | | | no-go: Khoekhoe-KDT | | Kiche | eval-only | | | Kiche-IU | | | Komi Permyak | train+eval | Komi_Permyak-UH (34) | | | | | Komi Zyrian | train+eval | Komi_Zyrian-Lattice (20) | | Komi_Zyrian-IKDP | | | Korean | train+eval | Korean-Kaist (23,010) | Korean-GSD, Korean-KSL | Korean-LittlePrince, Korean-PUD | | | Kyrgyz | train+eval | Kyrgyz-KTMU (1,308) | | Kyrgyz-TueCL | | | Latgalian | eval-only | | | Latgalian-Cairo | | | Latin | train+eval | Latin-LLCT (7,289) | Latin-CIRCSE | | no-go: Latin-ITTB, Latin-PROIEL, Latin-Perseus, Latin-UDante | | Latvian | train+eval | Latvian-LVTB (15,059) | | Latvian-Cairo | | | Ligurian | train+eval | Ligurian-GLT (20) | | | | | Lithuanian | train+eval | Lithuanian-ALKSNIS (2,341) | Lithuanian-HSE | | | | Livvi | train+eval | Livvi-KKPP (19) | | | | | Low Saxon | train+eval | Low_Saxon-LSDC (500) | | | | | Luxembourgish | eval-only | | | Luxembourgish-LuxBank | | | Macedonian | eval-only | | | Macedonian-MTB | | | Madi | eval-only | | | Madi-Jarawara | | | Maghrebi Arabic French | gap | | | | no-go: Maghrebi_Arabic_French-Arabizi | | Makurap | eval-only | | | Makurap-TuDeT | | | Malayalam | eval-only | | | Malayalam-UFAL | | | Maltese | train+eval | Maltese-MUDT (1,123) | | | | | Manx | gap | | | | no-go: Manx-Cadhan | | Marathi | train+eval | Marathi-UFAL (373) | Marathi-CMUPAN | | | | Mbya Guarani | gap | | | | no-go: Mbya_Guarani-Thomas | | Middle Armenian | eval-only | | | Middle_Armenian-ArmTDP | | | Middle French | eval-only | | | Middle_French-ALTM | no-go: Middle_French-PROFITEROLE | | Moksha | train+eval | Moksha-JR (22) | | | | | Munduruku | eval-only | | | Munduruku-TuDeT | | | Naga | gap | | | | no-go: Naga-Suansu | | Naija | train+eval | Naija-NSC (7,279) | | | | | Neapolitan | eval-only | | | Neapolitan-RB | | | Nenets | eval-only | | | Nenets-Tundra | | | Nepali | eval-only | | | Nepali-BK | | | Nheengatu | gap | | | | no-go: Nheengatu-CompLin | | North Sami | train+eval | North_Sami-Giella (2,257) | | | | | Northern Kurdish | train+eval | Northern_Kurdish-Kurmanji (20) | | | | | Northwest Gbaya | eval-only | | | Northwest_Gbaya-Autogramm | | | Norwegian | train+eval | Norwegian-Bokmaal (15,696) | Norwegian-Nynorsk | | | | Occitan | gap | | | | no-go: Occitan-TTB | | Odia | eval-only | | | Odia-ODTB | | | Old Church Slavonic | gap | | | | no-go: Old_Church_Slavonic-PROIEL | | Old East Slavic | train+eval | Old_East_Slavic-RNC (3,778) | Old_East_Slavic-Birchbark, Old_East_Slavic-Ruthenian | | no-go: Old_East_Slavic-TOROT | | Old English | eval-only | | | Old_English-Cairo | | | Old French | eval-only | | | Old_French-ALTM | no-go: Old_French-PROFITEROLE | | Old Georgian | eval-only | | | Old_Georgian-GLC | | | Old Irish | eval-only | | | Old_Irish-DipWBG | no-go: Old_Irish-DipSGG | | Old Occitan | train+eval | Old_Occitan-CorAG (1,092) | | | | | Old Turkish | eval-only | | | Old_Turkish-Clausal | | | Ottoman Turkish | train+eval | Ottoman_Turkish-BOUN (114) | Ottoman_Turkish-DUDU | Ottoman_Turkish-TueCL | | | Pashto | eval-only | | | Pashto-Prince, Pashto-Sikaram | | | Paumari | eval-only | | | Paumari-TueCL | | | Persian | train+eval | Persian-Seraji (4,798) | Persian-PerDT | | | | Pesh | eval-only | | | Pesh-ChibErgIS | | | Phrygian | eval-only | | | Phrygian-KUL | | | Polish | train+eval | Polish-MPDT (1,433) | | Polish-PUD | no-go: Polish-PDB, Polish-LFG | | Pomak | gap | | | | no-go: Pomak-Philotis | | Portuguese | train+eval | Portuguese-Bosque (7,018) | Portuguese-GSD, Portuguese-PetroGold, Portuguese-Porttinari | Portuguese-PUD | no-go: Portuguese-CINTIL, Portuguese-DANTEStocks | | Punjabi | train+eval | Punjabi-Rang (81) | Punjabi-CS | | | | Romanian | train+eval | Romanian-RRT (8,043) | Romanian-SiMoNERo | Romanian-ArT, Romanian-MolDoRo | no-go: Romanian-TueCL, Romanian-Nonstandard | | Russian | train+eval | Russian-GSD (3,850) | Russian-Poetry | Russian-PUD | no-go: Russian-SynTagRus, Russian-Taiga | | Ruuli | gap | | | | no-go: Ruuli-RDT | | Sanskrit | train+eval | Sanskrit-Vedic (21,477) | | Sanskrit-UFAL | | | Scottish Gaelic | train+eval | Scottish_Gaelic-ARCOSG (3,544) | | | | | Serbian | train+eval | Serbian-SET (3,328) | | | | | Shanghainese | gap | | | | no-go: Shanghainese-ShUD | | Sicilian | eval-only | | | Sicilian-STB | | | Sindhi | train+eval | Sindhi-Isra (4,096) | | | | | Sinhala | eval-only | | | Sinhala-Appuwa, Sinhala-STB | | | Skolt Sami | train+eval | Skolt_Sami-Giellagas (20) | | | | | Slovak | train+eval | Slovak-SNK (8,483) | | | | | Slovenian | train+eval | Slovenian-SSJ (10,903) | Slovenian-SST | | | | South Levantine Arabic | gap | | | | no-go: South_Levantine_Arabic-MADAR | | Southern Kurdish | train+eval | Southern_Kurdish-Garrusi (152) | | | | | Spanish | train+eval | Spanish-AnCora (14,287) | Spanish-GSD | Spanish-COSER, Spanish-PUD | | | Spanish Sign Language | eval-only | | | Spanish_Sign_Language-LSE | | | Swedish | train+eval | Swedish-Talbanken (4,315) | | Swedish-Old, Swedish-PUD, Swedish-SweLL | no-go: Swedish-LinES | | Swedish Sign Language | eval-only | | | Swedish_Sign_Language-SSLC | | | Tagalog | eval-only | | | Tagalog-TRG | no-go: Tagalog-Ugnayan | | Tamil | eval-only | | | Tamil-MWTT | no-go: Tamil-TTB | | Tatar | eval-only | | | Tatar-NMCTT | | | Teko | eval-only | | | Teko-TuDeT | | | Telugu | train+eval | Telugu-MTG (1,051) | | | | | Telugu English | train+eval | Telugu_English-TECT (61) | | | | | Thai | train+eval | Thai-TUD (2,902) | | Thai-PUD | | | Tswana | eval-only | | | Tswana-Popapolelo | | | Tupinamba | eval-only | | | Tupinamba-TuDeT | | | Turkish | train+eval | Turkish-BOUN (7,803) | Turkish-Atis, Turkish-FrameNet, Turkish-Kenet, Turkish-Tourism | Turkish-GB, Turkish-PUD, Turkish-TueCL | no-go: Turkish-IMST, Turkish-Penn | | Turkish English | eval-only | | | Turkish_English-BUTR | | | Turkish German | gap | | | | no-go: Turkish_German-SAGT | | Ukrainian | train+eval | Ukrainian-ParlaMint (5,611) | | | no-go: Ukrainian-IU | | Umbrian | eval-only | | | Umbrian-IKUVINA | | | Upper Sorbian | eval-only | | | Upper_Sorbian-UFAL | | | Urdu | gap | | | | no-go: Urdu-UDTB | | Uyghur | train+eval | Uyghur-UDT (1,656) | | | | | Uzbek | train+eval | Uzbek-UzUDT (483) | | Uzbek-TueCL, Uzbek-UT | | | Veps | eval-only | | | Veps-VWT | | | Vietnamese | train+eval | Vietnamese-VTB (1,400) | | Vietnamese-TueCL | | | Warlpiri | eval-only | | | Warlpiri-UFAL | | | Welsh | train+eval | Welsh-CCG (1,355) | | | | | Western Armenian | train+eval | Western_Armenian-ArmTDP (5,274) | | | | | Western Sierra Puebla Nahuatl | eval-only | | | Western_Sierra_Puebla_Nahuatl-MesoTree | | | Wolof | train+eval | Wolof-WTB (1,188) | | | | | Xavante | eval-only | | | Xavante-XDT | | | Xibe | gap | | | | no-go: Xibe-XDT | | Yakut | eval-only | | | Yakut-YKTDT | | | Yiddish | train+eval | Yiddish-YiTB (1,877) | | | | | Yoruba | eval-only | | | Yoruba-YTB | | | Yupik | eval-only | | | Yupik-SLI | | | Zaar | train+eval | Zaar-Autogramm (978) | | | | | Zazaki | train+eval | Zazaki-ZSD (138) | | | | ### Languages without an accurate-tier model Every treebank of these languages is ruled out, so there is no allowed training data. Hindi has UD data only in the test-only PUD set (HDTB is non-commercial); its fast-tier model is trained on labels from the accurate tier. | Language | Treebanks and why they are out | |---|---| | Ancient Hebrew | Ancient_Hebrew-PTNK: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC 4.0, but LICENSE | | Basque | Basque-BDT: README licence line CC BY-NC-SA 3.0; LICENSE.txt agrees | | Belarusian | Belarusian-HSE: ruled no-go (terms-restricted text): README: genre list includes "5) social media: messages from the Telegram | | Bulgarian | Bulgarian-BTB: README licence line CC BY-NC-SA 3.0; LICENSE.txt agrees | | Chintang | Chintang-CTNTB: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICE | | Classical Armenian | Classical_Armenian-CAVaL: README licence line CC BY-NC-SA 4.0; LICENSE.txt agrees | | Estonian | Estonian-EDT: README licence line CC BY-NC-SA 4.0; LICENSE.txt agrees; Estonian-EWT: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICE | | Gothic | Gothic-PROIEL: README licence line CC BY-NC-SA 3.0; LICENSE.txt agrees | | Hungarian | Hungarian-Szeged: README licence line CC BY-NC-SA 3.0; LICENSE.txt agrees | | Kadiweu | Kadiweu-Unicamp: README licence line CC BY-NC-SA 4.0; LICENSE.txt agrees | | Khoekhoe | Khoekhoe-KDT: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICE | | Maghrebi Arabic French | Maghrebi_Arabic_French-Arabizi: ruled no-go (terms-restricted text): README: sources include "Algerian newspaper’s web forums"; no statement a | | Manx | Manx-Cadhan: ruled no-go (terms-restricted text): README: sentences "came from a large web-crawled corpus of Manx"; genres | | Mbya Guarani | Mbya_Guarani-Thomas: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICE | | Naga | Naga-Suansu: ruled no-go (terms-restricted text): README: test set includes "film_Bridge: Begining of the subtitles from Br | | Nheengatu | Nheengatu-CompLin: README licence line CC BY-NC-SA 4.0; LICENSE.txt agrees | | Occitan | Occitan-TTB: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICE | | Old Church Slavonic | Old_Church_Slavonic-PROIEL: README licence line CC BY-NC-SA 4.0; LICENSE.txt agrees | | Pomak | Pomak-Philotis: README licence line CC BY-NC-SA 3.0; LICENSE.txt agrees | | Ruuli | Ruuli-RDT: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICE | | Shanghainese | Shanghainese-ShUD: README: the source corpus is "licensed under Creative Commons BY-NC-ND 4.0", with "additional permission for d | | South Levantine Arabic | South_Levantine_Arabic-MADAR: README: sentences come from the MADAR parallel corpus, built on BTEC; the MADAR corpus "is made available to t | | Turkish German | Turkish_German-SAGT: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICE | | Urdu | Urdu-UDTB: README licence line CC BY-NC-SA 4.0; LICENSE.txt agrees | | Xibe | Xibe-XDT: ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 3.0, but LICE | ### Treebanks Splits are sentence counts (train/dev/test) from `stats.xml`. "Licence line" is the README's `License:` metadata; the quote for any other decision is in the Rulings section. | Treebank | Licence line | LICENSE.txt | Splits | Verdict | |---|---|---|---|---| | Abaza-ATB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/98 | eval-only | | Abkhaz-AbNC | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1545 | eval-only | | Afrikaans-AfriBooms | CC BY-SA 4.0 | CC BY-SA 4.0 | 1315/194/425 | go | | Akkadian-PISANDUB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/101 | eval-only | | Akkadian-RIAO | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1874 | eval-only | | Akuntsu-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/343 | eval-only | | Albanian-STAF | CC BY-SA 4.0 | CC BY-SA 4.0 | 160/20/20 | go | | Albanian-TSA | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/60 | eval-only | | Alemannic-DIVITAL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/977 | eval-only | | Alemannic-UZH | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/100 | eval-only | | Amharic-ATT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1074 | eval-only | | Ancient_Greek-PROIEL | CC BY-NC-SA 3.0 | own text | 15016/1019/1047 | no-go | | Ancient_Greek-PTNK | CC BY-SA 4.0 | CC BY-SA 4.0 | 727/439/410 | go | | Ancient_Greek-Perseus | CC BY-NC-SA 2.5 | own text | 11476/1137/1306 | no-go | | Ancient_Hebrew-PTNK | CC BY-NC 4.0 | CC BY-SA 4.0 | 4417/639/552 | no-go (conflict) | | Apurina-UFPA | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/183 | eval-only | | Arabic-NYUAD | CC BY-SA 4.0 | CC BY-SA 4.0 | 15789/1986/1963 | no-go | | Arabic-PADT | CC BY-NC-SA 3.0 | own text | 6075/909/680 | no-go | | Arabic-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Armenian-ArmTDP | CC BY-SA 4.0 | CC BY-SA 4.0 | 4352/575/600 | go | | Armenian-BSUT | CC BY-SA 4.0 | CC BY-SA 4.0 | 1451/479/593 | go | | Assamese-AiW | CC BY-SA 4.0 | CC BY-SA 4.0 | 51/14/8 | go | | Assyrian-AS | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/57 | eval-only | | Azerbaijani-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/148 | eval-only | | Bambara-CRB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1026 | eval-only | | Basque-BDT | CC BY-NC-SA 3.0 | CC BY-NC-SA | 5396/1798/1799 | no-go | | Bavarian-MaiBaam | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1070 | eval-only | | Beja-Autogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/763 | eval-only | | Belarusian-HSE | CC BY-SA 4.0 | CC BY-SA 4.0 | 22853/1301/1077 | no-go | | Bengali-BRU | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/56 | eval-only | | Bhojpuri-BHTB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/357 | eval-only | | Bokota-ChibErgIS | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/406 | eval-only | | Bororo-BDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 17107/2138/2139 | go | | Brahui-Kholum | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/64 | eval-only | | Breton-KEB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/884 | eval-only | | Bulgarian-BTB | CC BY-NC-SA 3.0 | own text | 8907/1115/1116 | no-go | | Buryat-BDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 20/-/907 | go | | Cantonese-HK | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1004 | eval-only | | Cappadocian-AMGiC | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/70 | eval-only | | Cappadocian-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/350 | eval-only | | Catalan-AnCora | CC BY 4.0 | CC BY 4.0 | 13123/1709/1846 | go | | Cebuano-GJA | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/198 | eval-only | | Central_Kurdish-Mukri | CC BY-SA 4.0 | CC BY-SA 4.0 | 92/23/23 | go | | Chinese-Beginner | CC BY-NC-SA 3.0 | CC BY-NC-SA | -/-/2295 | no-go | | Chinese-CFL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/451 | eval-only | | Chinese-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 3997/500/500 | go | | Chinese-GSDSimp | CC BY-SA 4.0 | CC BY-SA 4.0 | 3997/500/500 | go | | Chinese-HK | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1004 | eval-only | | Chinese-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Chinese-PatentChar | CC BY-NC-SA 3.0 | CC BY-SA 4.0 | -/-/200 | no-go (conflict) | | Chintang-CTNTB | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | -/-/2289 | no-go (conflict) | | Chukchi-HSE | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1004 | eval-only | | Classical_Armenian-CAVaL | CC BY-NC-SA 4.0 | CC BY-NC-SA | 3795/423/563 | no-go | | Classical_Chinese-Kyoto | PD | CC BY-SA 4.0 | 74609/6102/5528 | go | | Classical_Chinese-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/100 | eval-only | | Coptic-Bohairic | CC BY 4.0 | CC BY 4.0 | 477/166/358 | go | | Coptic-Scriptorium | CC BY 4.0 | CC BY 4.0 | 1467/380/405 | go | | Croatian-SET | CC BY-SA 4.0 | CC BY-SA 4.0 | 6914/960/1136 | go | | Czech-CAC | CC BY-SA 4.0 | CC BY-SA 4.0 | 23478/603/628 | go | | Czech-CLTT | CC BY-SA 4.0 | CC BY-SA 4.0 | 467/316/338 | go | | Czech-FicTree | CC BY-NC-SA 4.0 | CC BY-NC-SA | 10160/1309/1291 | no-go | | Czech-PDTC | CC BY-NC-SA 4.0 | own text | 171044/22666/20187 | no-go | | Czech-PUD | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Czech-Poetry | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/297 | eval-only | | Danish-DDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 4383/564/565 | go | | Dutch-Alpino | CC BY-SA 4.0 | CC BY-SA 4.0 | 12289/718/596 | go | | Dutch-LassySmall | CC BY-SA 4.0 | own text | 13817/1542/1761 | go | | Egyptian-PC | CC BY-SA 4.0 | CC BY-SA 4.0 | 1619/230/1240 | go | | English-Atis | CC BY-SA 4.0 | CC BY-SA 4.0 | 4274/572/586 | go | | English-CHILDES | CC BY-SA 4.0 | CC BY-SA 4.0 | 34732/3860/9591 | go | | English-CTeTex | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/276 | eval-only | | English-ESLSpok | CC BY-SA 4.0 | CC BY-SA 4.0 | 1856/232/232 | go | | English-EWT | CC BY-SA 4.0 | CC BY-SA 4.0 | 12544/2001/2077 | go | | English-GENTLE | CC BY-NC-SA 4.0 | CC BY-NC-SA | -/-/1334 | no-go | | English-GUM | CC BY-NC-SA 4.0 | CC BY-NC-SA | 11314/1575/1464 | no-go | | English-GUMReddit | CC BY 4.0 | CC BY 4.0 | 686/104/105 | no-go | | English-LinES | CC BY-NC-SA 4.0 | own text | 3457/1118/1121 | no-go | | English-LittlePrince | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/500 | eval-only | | English-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | English-ParTUT | CC BY-NC-SA 4.0 | own text | 1781/156/153 | no-go | | English-Pronouns | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/285 | eval-only | | Erzya-JR | CC BY-SA 4.0 | CC BY-SA 4.0 | 1217/-/935 | go | | Esperanto-Cairo | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Esperanto-Prago | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/131 | eval-only | | Estonian-EDT | CC BY-NC-SA 4.0 | own text | 24601/3122/3207 | no-go | | Estonian-EWT | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | 5444/833/913 | no-go (conflict) | | Faroese-FarPaHC | CC BY-SA 4.0 | CC BY-SA 4.0 | 1020/300/301 | go | | Faroese-OFT | CC BY-SA 4.0 | own text | -/-/1208 | eval-only | | Finnish-FTB | CC BY 4.0 | own text | 14981/1875/1867 | go | | Finnish-OOD | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/2122 | no-go | | Finnish-PUD | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Finnish-TDT | CC BY-SA 4.0 | own text | 12217/1364/1555 | go | | French-ALTS | CC BY-SA 4.0 | CC BY-SA 4.0 | 1202/154/670 | go | | French-FQB | LGPL-LR | LGPL-LR | -/-/2289 | eval-only | | French-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 14450/1476/416 | go | | French-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | French-ParTUT | CC BY-NC-SA 4.0 | own text | 803/107/110 | no-go | | French-ParisStories | CC BY-SA 4.0 | CC BY-SA 4.0 | 1387/692/697 | go | | French-PoitevinDIVITAL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/239 | eval-only | | French-Rhapsodie | CC BY-SA 4.0 | CC BY-SA 4.0 | 1288/1081/840 | go | | French-Sequoia | LGPL-LR | LGPL-LR | 2231/412/456 | go | | Frisian_Dutch-Fame | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/400 | eval-only | | Galician-CTG | CC BY-NC-SA 3.0 | CC BY-NC-SA | 2272/860/861 | no-go | | Galician-PUD | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Galician-TreeGal | LGPL-LR | LGPL-LR | 600/-/400 | go | | Georgian-GLC | CC BY-SA 4.0 | CC BY-SA 4.0 | 2214/469/481 | go | | Georgian-GNC | CC BY-SA 4.0 | CC BY-SA 4.0 | 918/130/830 | go | | German-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 13813/799/977 | go | | German-HDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 153035/18434/18459 | no-go | | German-LIT | CC BY-NC-SA 4.0 | CC BY-NC-SA | -/-/1920 | no-go | | German-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Gheg-GPS | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/966 | eval-only | | Gorontalo-BungoLoLombi | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/39 | eval-only | | Gothic-PROIEL | CC BY-NC-SA 3.0 | own text | 3387/985/1029 | no-go | | Greek-Cretan | CC BY-SA 4.0 | CC BY-SA 4.0 | 180/60/30 | go | | Greek-GDT | CC BY-NC-SA 3.0 | own text | 1662/403/456 | no-go | | Greek-GLCII | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | -/-/671 | no-go (conflict) | | Greek-GUD | CC BY-SA 4.0 | CC BY-SA 4.0 | 1071/-/736 | go | | Greek-Lesbian | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/625 | eval-only | | Greek-Messinian | CC BY-SA 4.0 | CC BY-SA 4.0 | 60/20/30 | go | | Guajajara-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1182 | eval-only | | Guarani-OldTuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/59 | eval-only | | Gujarati-GujTB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/187 | eval-only | | Gwichin-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/313 | eval-only | | Haitian_Creole-Adolphe | CC BY-SA 4.0 | CC BY-SA 4.0 | 2605/265/444 | go | | Haitian_Creole-Autogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/144 | eval-only | | Hausa-EasternAutogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/335 | eval-only | | Hausa-NorthernAutogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1305 | eval-only | | Hausa-SouthernAutogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1927 | eval-only | | Hausa-WesternAutogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/775 | eval-only | | Hebrew-HTB | CC BY-NC-SA 4.0 | CC BY-NC-SA | 5168/484/491 | no-go | | Hebrew-IAHLTknesset | CC BY-SA 4.0 | CC BY-SA 4.0 | 2362/253/268 | go | | Hebrew-IAHLTwiki | CC BY-SA 4.0 | CC BY-SA 4.0 | 4298/348/393 | go | | Hebrew-PostRab | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/296 | eval-only | | Highland_Puebla_Nahuatl-ITML | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1260 | eval-only | | Hindi-HDTB | CC BY-NC-SA 4.0 | CC BY-NC-SA | 13306/1659/1684 | no-go | | Hindi-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Hittite-HitTB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/136 | eval-only | | Hungarian-Szeged | CC BY-NC-SA 3.0 | CC BY-NC-SA | 910/441/449 | no-go | | Icelandic-GC | CC BY-SA 4.0 | CC BY-SA 4.0 | 3960/500/540 | go | | Icelandic-IcePaHC | CC BY-SA 4.0 | CC BY-SA 4.0 | 34007/4865/5157 | go | | Icelandic-Modern | CC BY-SA 4.0 | CC BY-SA 4.0 | 2703/394/438 | go | | Icelandic-PUD | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Ika-ChibErgIS | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/628 | eval-only | | Indonesian-CSUI | CC BY-SA 4.0 | CC BY-SA 4.0 | 656/-/374 | go | | Indonesian-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 4482/559/557 | go | | Indonesian-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Irish-Cadhan | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/193 | eval-only | | Irish-IDT | CC BY-SA 3.0 | own text | 4005/451/454 | go | | Irish-TwittIrish | CC BY-SA 4.0 | CC BY-SA 4.0 | 866/864/866 | no-go | | Italian-ISDT | CC BY-NC-SA 3.0 | own text | 13121/564/482 | no-go | | Italian-KIParlaForest | CC BY-NC-SA 4.0 | CC BY-NC-SA | -/-/2221 | no-go | | Italian-MarkIT | CC BY 4.0 | CC BY 4.0 | 611/341/340 | go | | Italian-Old | CC BY-SA 4.0 | CC BY-SA 4.0 | 2774/308/337 | go | | Italian-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Italian-ParTUT | CC BY-NC-SA 4.0 | own text | 1781/156/153 | no-go | | Italian-ParlaMint | CC BY-SA 4.0 | CC BY-SA 4.0 | 326/-/375 | go | | Italian-PoSTWITA | CC BY-NC-SA 4.0 | CC BY-NC-SA | 5368/670/674 | no-go | | Italian-TWITTIRO | CC BY-SA 4.0 | CC BY-SA 4.0 | 1138/144/142 | no-go | | Italian-VIT | CC BY-NC-SA 3.0 | CC BY-NC-SA | 8277/743/1067 | no-go | | Italian-Valico | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/398 | eval-only | | Japanese-BCCWJ | CC BY-NC-SA 4.0 | own text | 40801/8427/7881 | no-go | | Japanese-BCCWJLUW | CC BY-NC-SA 4.0 | own text | 40801/8427/7881 | no-go | | Japanese-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 7050/507/543 | go | | Japanese-GSDLUW | CC BY-SA 4.0 | CC BY-SA 4.0 | 7050/507/543 | go | | Japanese-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Japanese-PUDLUW | CC BY-SA 3.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Javanese-CSUI | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Kaapor-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/83 | eval-only | | Kadiweu-Unicamp | CC BY-NC-SA 4.0 | CC BY-NC-SA | -/-/71 | no-go | | Kangri-KDTB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/288 | eval-only | | Karelian-KKPP | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/228 | eval-only | | Karo-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/674 | eval-only | | Kazakh-KTB | CC BY-SA 4.0 | CC BY-SA 4.0 | 31/-/1047 | go | | Khoekhoe-KDT | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | 2002/232/1355 | no-go (conflict) | | Kiche-IU | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1435 | eval-only | | Komi_Permyak-UH | CC BY-SA 4.0 | CC BY-SA 4.0 | 34/-/106 | go | | Komi_Zyrian-IKDP | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/214 | eval-only | | Komi_Zyrian-Lattice | CC BY-SA 4.0 | CC BY-SA 4.0 | 20/-/653 | go | | Korean-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 4400/950/989 | go | | Korean-KSL | CC BY-SA 4.0 | CC BY-SA 4.0 | 14076/1480/1813 | go | | Korean-Kaist | CC BY-SA 4.0 | CC BY-SA 4.0 | 23010/2066/2287 | go | | Korean-LittlePrince | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1551 | eval-only | | Korean-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Kyrgyz-KTMU | CC BY-SA 4.0 | CC BY-SA 4.0 | 1308/-/1222 | go | | Kyrgyz-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/173 | eval-only | | Latgalian-Cairo | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Latin-CIRCSE | CC BY-SA 4.0 | CC BY-SA 4.0 | 918/161/893 | go | | Latin-ITTB | CC BY-NC-SA 3.0 | own text | 22775/2101/2101 | no-go | | Latin-LLCT | CC BY-SA 4.0 | CC BY-SA 4.0 | 7289/850/884 | go | | Latin-PROIEL | CC BY-NC-SA 3.0 | own text | 16196/1233/1260 | no-go | | Latin-Perseus | CC BY-NC-SA 2.5 | own text | 1334/-/939 | no-go | | Latin-UDante | CC BY-NC-SA 3.0 | CC BY-SA 4.0 | 926/376/421 | no-go (conflict) | | Latvian-Cairo | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Latvian-LVTB | CC BY-SA 4.0 | CC BY-SA 4.0 | 15059/2109/2412 | go | | Ligurian-GLT | C-UDA 1.0 | C-UDA 1.0 | 20/-/296 | go | | Lithuanian-ALKSNIS | CC BY-SA 4.0 | CC BY-SA 4.0 | 2341/617/684 | go | | Lithuanian-HSE | CC BY-SA 4.0 | CC BY-SA 4.0 | 153/55/55 | go | | Livvi-KKPP | CC BY-SA 4.0 | CC BY-SA 4.0 | 19/-/106 | go | | Low_Saxon-LSDC | CC BY-SA 4.0 | CC BY-SA 4.0 | 500/-/500 | go | | Luxembourgish-LuxBank | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Macedonian-MTB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/155 | eval-only | | Madi-Jarawara | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Maghrebi_Arabic_French-Arabizi | CC BY-SA 4.0 | CC BY-SA 4.0 | 1003/139/145 | no-go | | Makurap-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/37 | eval-only | | Malayalam-UFAL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/218 | eval-only | | Maltese-MUDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 1123/433/518 | go | | Manx-Cadhan | CC BY-SA 4.0 | CC BY-SA 4.0 | 1172/-/1164 | no-go | | Marathi-CMUPAN | CC BY-SA 4.0 | CC BY-SA 4.0 | 7616/987/1003 | go | | Marathi-UFAL | CC BY-SA 4.0 | CC BY-SA 4.0 | 373/46/47 | go | | Mbya_Guarani-Thomas | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | -/-/98 | no-go (conflict) | | Middle_Armenian-ArmTDP | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/27 | eval-only | | Middle_French-ALTM | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/217 | eval-only | | Middle_French-PROFITEROLE | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | 4496/569/906 | no-go (conflict) | | Moksha-JR | CC BY-SA 4.0 | CC BY-SA 4.0 | 22/-/468 | go | | Munduruku-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/158 | eval-only | | Naga-Suansu | CC BY-SA 4.0 | CC BY-SA 4.0 | 20/-/564 | no-go | | Naija-NSC | CC BY-SA 4.0 | CC BY-SA 4.0 | 7279/990/972 | go | | Neapolitan-RB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Nenets-Tundra | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/170 | eval-only | | Nepali-BK | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/69 | eval-only | | Nheengatu-CompLin | CC BY-NC-SA 4.0 | CC BY-NC-SA | 1796/-/1043 | no-go | | North_Sami-Giella | CC BY-SA 4.0 | CC BY-SA 4.0 | 2257/-/865 | go | | Northern_Kurdish-Kurmanji | CC BY-SA 4.0 | CC BY-SA 4.0 | 20/-/734 | go | | Northwest_Gbaya-Autogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/403 | eval-only | | Norwegian-Bokmaal | CC BY-SA 4.0 | CC BY-SA 4.0 | 15696/2409/1939 | go | | Norwegian-Nynorsk | CC BY-SA 4.0 | CC BY-SA 4.0 | 14174/1890/1511 | go | | Occitan-TTB | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | 830/79/613 | no-go (conflict) | | Odia-ODTB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/456 | eval-only | | Old_Church_Slavonic-PROIEL | CC BY-NC-SA 4.0 | CC BY-NC-SA | 18327/2205/2096 | no-go | | Old_East_Slavic-Birchbark | CC BY-SA 4.0 | CC BY-SA 4.0 | 1113/1062/1009 | go | | Old_East_Slavic-RNC | CC BY-SA 4.0 | CC BY-SA 4.0 | 3778/1363/906 | go | | Old_East_Slavic-Ruthenian | CC BY-SA 4.0 | CC BY-SA 4.0 | 4120/855/390 | go | | Old_East_Slavic-TOROT | CC BY-NC-SA 4.0 | CC BY-NC-SA | 20480/3061/2955 | no-go | | Old_English-Cairo | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Old_French-ALTM | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/553 | eval-only | | Old_French-PROFITEROLE | CC BY-NC-SA 3.0 | CC BY-SA 4.0 | 15955/2143/2252 | no-go (conflict) | | Old_Georgian-GLC | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/151 | eval-only | | Old_Irish-DipSGG | CC BY-NC-SA 4.0 | CC BY-NC-SA | -/-/64 | no-go | | Old_Irish-DipWBG | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/34 | eval-only | | Old_Occitan-CorAG | CC BY-SA 4.0 | CC BY-SA 4.0 | 1092/118/276 | go | | Old_Turkish-Clausal | CC0 1.0 | CC0 | -/-/20 | eval-only | | Ottoman_Turkish-BOUN | CC BY-SA 4.0 | CC BY-SA 4.0 | 114/-/400 | go | | Ottoman_Turkish-DUDU | CC BY-SA 4.0 | CC BY-SA 4.0 | 1117/121/1008 | go | | Ottoman_Turkish-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/148 | eval-only | | Pashto-Prince | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/64 | eval-only | | Pashto-Sikaram | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/200 | eval-only | | Paumari-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/101 | eval-only | | Persian-PerDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 26196/1456/1455 | go | | Persian-Seraji | CC BY-SA 4.0 | CC BY-SA 4.0 | 4798/599/600 | go | | Pesh-ChibErgIS | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/524 | eval-only | | Phrygian-KUL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/203 | eval-only | | Polish-LFG | GNU GPL 3.0 | GPL | 13774/1745/1727 | no-go | | Polish-MPDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 1433/162/423 | go | | Polish-PDB | CC BY-NC-SA 4.0 | own text | 17722/2215/2215 | no-go | | Polish-PUD | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Pomak-Philotis | CC BY-NC-SA 3.0 | own text | 980/635/635 | no-go | | Portuguese-Bosque | CC BY-SA 4.0 | CC BY-SA 4.0 | 7018/1172/1167 | go | | Portuguese-CINTIL | CC BY-NC-ND 4.0 | CC BY-NC-ND | 30720/3840/3840 | no-go | | Portuguese-DANTEStocks | CC BY 4.0 | CC BY 4.0 | 3040/501/501 | no-go | | Portuguese-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 9616/1204/1200 | go | | Portuguese-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Portuguese-PetroGold | CC BY-SA 4.0 | CC BY-SA 4.0 | 7170/737/1039 | go | | Portuguese-Porttinari | CC BY 4.0 | CC BY 4.0 | 5893/842/1683 | go | | Punjabi-CS | CC BY-SA 4.0 | CC BY-SA 4.0 | 73/21/19 | go | | Punjabi-Rang | CC BY-SA 4.0 | CC BY-SA 4.0 | 81/30/26 | go | | Romanian-ArT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/50 | eval-only | | Romanian-MolDoRo | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/30 | eval-only | | Romanian-Nonstandard | CC BY-SA 4.0 | CC BY-SA 4.0 | 24121/1052/1052 | no-go | | Romanian-RRT | CC BY-SA 4.0 | CC BY-SA 4.0 | 8043/752/729 | go | | Romanian-SiMoNERo | CC BY-SA 4.0 | CC BY-SA 4.0 | 3747/443/491 | go | | Romanian-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/210 | no-go | | Russian-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 3850/579/601 | go | | Russian-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Russian-Poetry | CC BY-SA 4.0 | CC BY-SA 4.0 | 3568/789/728 | go | | Russian-SynTagRus | CC BY-NC-SA 4.0 | CC BY-NC-SA | 69631/8906/8800 | no-go | | Russian-Taiga | CC BY-SA 4.0 | CC BY-SA 4.0 | 119490/1260/1217 | no-go | | Ruuli-RDT | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | -/-/1234 | no-go (conflict) | | Sanskrit-UFAL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/230 | eval-only | | Sanskrit-Vedic | CC BY-SA 4.0 | CC BY-SA 4.0 | 21477/2996/2709 | go | | Scottish_Gaelic-ARCOSG | CC BY-SA 4.0 | CC BY-SA 4.0 | 3544/656/548 | go | | Serbian-SET | CC BY-SA 4.0 | CC BY-SA 4.0 | 3328/536/520 | go | | Shanghainese-ShUD | CC BY-SA 4.0 | CC BY-SA 4.0 | 786/98/99 | no-go | | Sicilian-STB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/505 | eval-only | | Sindhi-Isra | CC BY-SA 4.0 | CC BY-SA 4.0 | 4096/773/887 | go | | Sinhala-Appuwa | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/69 | eval-only | | Sinhala-STB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/100 | eval-only | | Skolt_Sami-Giellagas | CC BY-SA 4.0 | CC BY-SA 4.0 | 20/-/241 | go | | Slovak-SNK | CC BY-SA 4.0 | CC BY-SA 4.0 | 8483/1060/1061 | go | | Slovenian-SSJ | CC BY-SA 4.0 | CC BY-SA 4.0 | 10903/1250/1282 | go | | Slovenian-SST | CC BY-SA 4.0 | own text | 5094/595/432 | go | | South_Levantine_Arabic-MADAR | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/100 | no-go | | Southern_Kurdish-Garrusi | CC BY-SA 4.0 | CC BY-SA 4.0 | 152/49/20 | go | | Spanish-AnCora | CC BY 4.0 | CC BY 4.0 | 14287/1654/1721 | go | | Spanish-COSER | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/539 | eval-only | | Spanish-GSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 14186/1400/427 | go | | Spanish-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Spanish_Sign_Language-LSE | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/488 | eval-only | | Swedish-LinES | CC BY-NC-SA 4.0 | own text | 3457/1118/1121 | no-go | | Swedish-Old | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/40 | eval-only | | Swedish-PUD | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/1000 | eval-only | | Swedish-SweLL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/643 | eval-only | | Swedish-Talbanken | CC BY-SA 4.0 | own text | 4315/504/1219 | go | | Swedish_Sign_Language-SSLC | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/203 | eval-only | | Tagalog-TRG | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/128 | eval-only | | Tagalog-Ugnayan | CC BY-NC-SA 4.0 | CC BY-NC-SA | -/-/94 | no-go | | Tamil-MWTT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/534 | eval-only | | Tamil-TTB | CC BY-NC-SA 3.0 | own text | 400/80/120 | no-go | | Tatar-NMCTT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/148 | eval-only | | Teko-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/913 | eval-only | | Telugu-MTG | CC BY-SA 4.0 | CC BY-SA 4.0 | 1051/131/146 | go | | Telugu_English-TECT | CC BY-SA 4.0 | CC BY-SA 4.0 | 61/-/36 | go | | Thai-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Thai-TUD | CC BY-SA 4.0 | CC BY-SA 4.0 | 2902/362/363 | go | | Tswana-Popapolelo | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/20 | eval-only | | Tupinamba-TuDeT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/581 | eval-only | | Turkish-Atis | CC BY-SA 4.0 | CC BY-SA 4.0 | 4126/572/586 | go | | Turkish-BOUN | CC BY-SA 4.0 | CC BY-SA 4.0 | 7803/979/979 | go | | Turkish-FrameNet | CC BY-SA 4.0 | CC BY-SA 4.0 | 2288/205/205 | go | | Turkish-GB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/2880 | eval-only | | Turkish-IMST | CC BY-NC-SA 4.0 | own text | 3435/1100/1100 | no-go | | Turkish-Kenet | CC BY-SA 4.0 | CC BY-SA 4.0 | 15398/1646/1643 | go | | Turkish-PUD | CC BY-SA 3.0 | CC BY-SA 3.0 | -/-/1000 | eval-only | | Turkish-Penn | CC BY-SA 4.0 | CC BY-SA 4.0 | 14849/622/924 | no-go | | Turkish-Tourism | CC BY-SA 4.0 | CC BY-SA 4.0 | 15473/2166/2191 | go | | Turkish-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/148 | eval-only | | Turkish_English-BUTR | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/58 | eval-only | | Turkish_German-SAGT | CC BY-NC-SA 4.0 | CC BY-SA 4.0 | 578/801/805 | no-go (conflict) | | Ukrainian-IU | CC BY-NC-SA 4.0 | CC BY-NC-SA | 5521/673/898 | no-go | | Ukrainian-ParlaMint | CC BY-SA 4.0 | CC BY-SA 4.0 | 5611/739/792 | go | | Umbrian-IKUVINA | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/202 | eval-only | | Upper_Sorbian-UFAL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/644 | eval-only | | Urdu-UDTB | CC BY-NC-SA 4.0 | CC BY-NC-SA | 4043/552/535 | no-go | | Uyghur-UDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 1656/900/900 | go | | Uzbek-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/148 | eval-only | | Uzbek-UT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/500 | eval-only | | Uzbek-UzUDT | CC BY-SA 4.0 | CC BY-SA 4.0 | 483/-/201 | go | | Veps-VWT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/103 | eval-only | | Vietnamese-TueCL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/100 | eval-only | | Vietnamese-VTB | CC BY-SA 4.0 | CC BY-SA 4.0 | 1400/1123/800 | go | | Warlpiri-UFAL | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/55 | eval-only | | Welsh-CCG | CC BY-SA 4.0 | CC BY-SA 4.0 | 1355/409/953 | go | | Western_Armenian-ArmTDP | CC BY-SA 4.0 | CC BY-SA 4.0 | 5274/693/677 | go | | Western_Sierra_Puebla_Nahuatl-MesoTree | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/3024 | eval-only | | Wolof-WTB | CC BY-SA 4.0 | CC BY-SA 4.0 | 1188/449/470 | go | | Xavante-XDT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/248 | eval-only | | Xibe-XDT | CC BY-NC-SA 3.0 | CC BY-SA 4.0 | -/-/810 | no-go (conflict) | | Yakut-YKTDT | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/299 | eval-only | | Yiddish-YiTB | CC BY-SA 4.0 | CC BY-SA 4.0 | 1877/157/1079 | go | | Yoruba-YTB | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/317 | eval-only | | Yupik-SLI | CC BY-SA 4.0 | CC BY-SA 4.0 | -/-/309 | eval-only | | Zaar-Autogramm | CC BY-SA 4.0 | CC BY-SA 4.0 | 978/-/1127 | go | | Zazaki-ZSD | CC BY-SA 4.0 | CC BY-SA 4.0 | 138/30/32 | go | ### Rulings and quotes **no-go on the text source or the sources of terms** (licence line alone would pass): - **Arabic-NYUAD**: README: "Due to LDC licensing restrictions, the annotations are released without the data." (LDC text, not distributed) - **English-GUMReddit**: README: "This repository only contains annotations, without the underlying textual data from Reddit" (Reddit text; ToS-restricted) - **German-HDT**: README: "Heise gave permission to distribute the text for academic use" (research-only text; ruled) - **Irish-TwittIrish**: README: "A Universal Dependencies treebank of 2596 tweets in modern Irish." (platform text) - **Italian-TWITTIRO**: README: "a collection of ironic Italian tweets" (platform terms on the text; ruled) - **Portuguese-DANTEStocks**: tweets (platform terms on the text; ruled) - **Romanian-TueCL**: README: "It contains 210 annotated tweets selected from CoRoSeOf" (platform text) - **Shanghainese-ShUD**: README: the source corpus is "licensed under Creative Commons BY-NC-ND 4.0", with "additional permission for derivative research" (research-only) - **South_Levantine_Arabic-MADAR**: README: sentences come from the MADAR parallel corpus, built on BTEC; the MADAR corpus "is made available to the research community under a non-commercial license" (NYU Abu Dhabi CAMeL Lab resources page) - **Turkish-Penn**: sentences translated from the LDC Penn Treebank (ruled) **ruled no-go from `unclear`** (not fetched; the quotes behind the rulings): - **Ancient_Hebrew-PTNK** (CC BY-NC 4.0; 4417 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Belarusian-HSE** (CC BY-SA 4.0; 22853 train sentences): ruled no-go (terms-restricted text): README: genre list includes "5) social media: messages from the Telegram channels"; annotations CC BY-SA 4.0, no statement about the channel text - **Chinese-PatentChar** (CC BY-NC-SA 3.0; 0 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 3.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Chintang-CTNTB** (CC BY-NC-SA 4.0; 0 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Estonian-EWT** (CC BY-NC-SA 4.0; 5444 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Finnish-OOD** (CC BY-SA 4.0; 0 train sentences): ruled no-go (terms-restricted text): README: texts from "nursing narratives (hospital patient records), discussion forums, tweets, general web crawls and poetry"; "The underlying texts come from various sources collected from the Internet. These may hold different copyright owners." - **Greek-GLCII** (CC BY-NC-SA 4.0; 0 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Khoekhoe-KDT** (CC BY-NC-SA 4.0; 2002 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Latin-UDante** (CC BY-NC-SA 3.0; 926 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 3.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Maghrebi_Arabic_French-Arabizi** (CC BY-SA 4.0; 1003 train sentences): ruled no-go (terms-restricted text): README: sources include "Algerian newspaper’s web forums"; no statement about the forum text - **Manx-Cadhan** (CC BY-SA 4.0; 1172 train sentences): ruled no-go (terms-restricted text): README: sentences "came from a large web-crawled corpus of Manx"; genres include blog and social - **Mbya_Guarani-Thomas** (CC BY-NC-SA 4.0; 0 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Middle_French-PROFITEROLE** (CC BY-NC-SA 4.0; 4496 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Naga-Suansu** (CC BY-SA 4.0; 20 train sentences): ruled no-go (terms-restricted text): README: test set includes "film_Bridge: Begining of the subtitles from Bridge of Spies (2015)" (a commercial film’s subtitles); the rest is grammar examples - **Occitan-TTB** (CC BY-NC-SA 4.0; 830 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Old_French-PROFITEROLE** (CC BY-NC-SA 3.0; 15955 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 3.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Polish-LFG** (GNU GPL 3.0; 13774 train sentences): ruled no-go (GNU GPL 3.0 on a treebank): licence line and LICENSE.txt: GNU GPL 3.0 on a treebank (copyleft terms for data, and for what is trained on it, are untested); 3% of the text is forums and chatrooms - **Romanian-Nonstandard** (CC BY-SA 4.0; 24121 train sentences): ruled no-go (terms-restricted text): README: nonstandard genres "Old Romanian, Chat, Folklore"; the chat source is not stated - **Russian-Taiga** (CC BY-SA 4.0; 119490 train sentences): ruled no-go (terms-restricted text): README: "source: vk.com, instagram, facebook, twitter, youtube comments, questions & answers (otvet.mail.ru), reviews (reviews.yandex.ru)"; annotations CC BY-SA 4.0, no statement about the platform text - **Ruuli-RDT** (CC BY-NC-SA 4.0; 0 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Turkish_German-SAGT** (CC BY-NC-SA 4.0; 578 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 4.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" - **Xibe-XDT** (CC BY-NC-SA 3.0; 0 train sentences): ruled no-go (README statement over a template LICENSE.txt): README licence line says CC BY-NC-SA 3.0, but LICENSE.txt says: "The treebank is licensed under the Creative Commons License Attribution-ShareAlike 4.0 International. The complete" **go with a condition or note**: - **Bavarian-MaiBaam**: Wikipedia (CC BY-SA) and Tatoeba sentences; per-sentence source URLs given - **Cantonese-HK**: film subtitles by students of the authors' university and legislative proceedings - **Chinese-HK**: film subtitles by students of the authors' university and legislative proceedings - **Classical_Chinese-Kyoto**: README licence line says PD; LICENSE.txt says CC BY-SA 4.0; either passes - **Czech-CAC**: built on a corpus the LDC distributes; the UD release is stated to be "made freely available under the CreativeCommons license" - **Dutch-LassySmall**: LICENSE.txt states "distributed under the Creative Commons Attribution-ShareAlike 4.0 International license (CC BY-SA 4.0)"; its informal-reference links point at the by-nc-sa URL, which is read as a typo - **English-EWT**: built on the LDC English Web Treebank; the UD release is CC BY-SA 4.0 (kept as in the English units) - **Finnish-FTB**: LICENSE.txt offers LGPLv3+ or, alternatively, "CC BY 4.0 ... even commercially"; the GPL text it appends has the word "noncommercially" in an unrelated clause - **Indonesian-GSD**: same as Japanese-GSD - **Japanese-GSD**: Google: "Google gave permission to drop the NC restriction from the license" - **Japanese-GSDLUW**: same as Japanese-GSD - **Korean-GSD**: same as Japanese-GSD - **Ligurian-GLT**: Computational Use of Data Agreement 1.0 (C-UDA): "You agree that you will use the Data solely for Computational Use"; training is Computational Use and "The C-UDA does not impose any restriction with respect to the use, modification, or distribution of Results". Redistributing the data would bind recipients; we do not redistribute it ### Named-entity data beyond the nine languages UNER v1 (`github.com/UniversalNER`, listing read 2026-11-06) holds 31 datasets. Each labels the sentences of one UD treebank, so a UNER set is usable only where that treebank is. Verdicts: | UNER repository | UD treebank | UD verdict | UNER set | |---|---|---|---| | UNER_Cebuano-GJA | Cebuano-GJA | eval-only | eval-only | | UNER_Chinese-GSD | Chinese-GSD | go | go (train, dev, eval; SA lineage) | | UNER_Chinese-GSDSIMP | Chinese-GSDSimp | go | go (train, dev, eval; SA lineage) | | UNER_Chinese-PUD | Chinese-PUD | eval-only | eval-only | | UNER_Croatian-SET | Croatian-SET | go | go (train, dev, eval; SA lineage) | | UNER_Czech-PUD | Czech-PUD | eval-only | eval-only | | UNER_Danish-DDT | Danish-DDT | go | go (train, dev, eval; SA lineage) | | UNER_English-EWT | English-EWT | go | go (train, dev, eval; SA lineage) | | UNER_English-PUD | English-PUD | eval-only | eval-only | | UNER_German-PUD | German-PUD | eval-only | eval-only | | UNER_Greek-GDT | Greek-GDT | no-go | no-go (its text is NC) | | UNER_Hebrew-HTB | Hebrew-HTB | no-go | no-go (its text is NC) | | UNER_Indonesian-CSUI | Indonesian-CSUI | go | go (train, dev, eval; SA lineage) | | UNER_Indonesian-GSD | Indonesian-GSD | go | go (train, dev, eval; SA lineage) | | UNER_Indonesian-PUD | Indonesian-PUD | eval-only | eval-only | | UNER_Japanese-PUD | Japanese-PUD | eval-only | eval-only | | UNER_Korean-PUD | Korean-PUD | eval-only | eval-only | | UNER_Maghrebi_Arabic_French-Arabizi | Maghrebi_Arabic_French-Arabizi | no-go | no-go (its text is NC) | | UNER_Norwegian-NDT | Norwegian-Bokmaal | go | go (train, dev, eval; SA lineage) | | UNER_Portuguese-Bosque | Portuguese-Bosque | go | go (train, dev, eval; SA lineage) | | UNER_Portuguese-PUD | Portuguese-PUD | eval-only | eval-only | | UNER_Romanian-LegalNERo | Romanian-LegalNERo | not in r2.18 | out (ruled 2026-11-07), unclear terms: the UNER README says "Initially, the LegalNERo dataset was made available under the license CC BY-NC-ND 4.0 ... The Zenodo version remains under that license. This UNER version of the work is provided under the license CC BY-SA 4.0" | | UNER_Russian-PUD | Russian-PUD | eval-only | eval-only | | UNER_Serbian-SET | Serbian-SET | go | go (train, dev, eval; SA lineage) | | UNER_Slovak-SNK | Slovak-SNK | go | go (train, dev, eval; SA lineage) | | UNER_Slovenian-SSJ | Slovenian-SSJ | go | go (train, dev, eval; SA lineage) | | UNER_Swedish-Lines | Swedish-LinES | no-go | no-go (its text is NC) | | UNER_Swedish-PUD | Swedish-PUD | eval-only | eval-only | | UNER_Swedish-Talbanken | Swedish-Talbanken | go | go (train, dev, eval; SA lineage) | | UNER_Tagalog-TRG | Tagalog-TRG | eval-only | eval-only | | UNER_Tagalog-Ugnayan | Tagalog-Ugnayan | no-go | no-go (its text is NC) | Languages with a usable UNER training set beyond the nine: Croatian, Danish, Indonesian (CSUI, GSD), Norwegian, Serbian, Slovak, Slovenian, Swedish (Talbanken); eval-only: Cebuano, Czech, Japanese, Korean, Russian, Swedish (PUD), Tagalog (TRG). UNER Chinese and Portuguese are the nine-language sets. Each UNER set was cut from an older UD commit, so its tokens must be checked against the r2.18 files before training (the English-EWT set is not aligned with r2.18; Portuguese-Bosque is). Hebrew, Greek and Swedish-LinES carry NC text; Tagalog-Ugnayan is NC-SA. Other named-entity corpora checked at their cards (a card's licence is the dataset's, and the text source is read where the card states it): | Corpus | Languages | Terms (as stated) | Verdict | Use | |---|---|---|---|---| | UNER v1 repositories `UniversalNER/UNER_*` (Mayhew et al. 2024) | see the UNER table | labels CC BY-SA 4.0 (CC BY-SA 3.0 for PUD sets); a few repositories carry no LICENSE file and rely on the dataset card's "released under the terms of the Creative Commons Attribution-ShareAlike 4.0" | go where the underlying UD treebank is go; the text keeps the treebank's own licence | train and eval | | WikiNER wp3 (Nothman et al. 2013) | de en es fr it nl pl pt ru | figshare record: CC BY 4.0; silver labels projected from Wikipedia links, text is Wikipedia (CC BY-SA) | go (SA lineage) | silver NER for es fr it nl pt (already fetched for fr es it nl); de en pl ru not fetched | | DaNE `alexandrainst/dane` | da | card: cc-by-sa-4.0; text from Parole (the UD Danish-DDT text) | go (SA lineage) | Danish NER; the same sentences as UNER Danish-DDT | | KPWr-NER `clarin-pl/kpwr-ner` | pl | card: "Creative Commons Attribution 3.0 Unported Licence"; "various sources, covering many domains and genres" | go on the card's licence (text sources are not itemised on the card) | Polish NER | | hr500k `classla/hr500k` | hr | card: cc-by-sa-4.0; NER layer of the Croatian reference corpus | go (SA lineage) | Croatian NER | | RONEC `community-datasets/ronec` | ro | card: MIT; "sentences that are free of copyright" from SEETimes, Romanian Wikipedia and Common Crawl | go | Romanian NER (15 classes) | | SUCX 3.0 NER `KBLab/sucx3_ner` | sv | card: cc-by-4.0 (SUC 3.0, Språkbanken) | go | Swedish NER; the Swedish-Talbanken UNER set is the UD-aligned alternative | | MasakhaNER 2.0 `masakhane/masakhaner2` | 20 African languages (bm bbj ee fon ha ig rw lg luo mos ny pcm sn sw tn tw wo xh yo zu) | card: afl-3.0; text from news outlets: "The data is sourced from newspaper source" | out (ruled 2026-11-07), unclear terms: the annotations' licence is permissive, the news publishers' terms on the text are not stated | none | | MasakhaNER 1.0 `masakhane/masakhaner` | 10 African languages | card: license unknown | out (ruled 2026-11-07), no stated terms | none | | Hausa VOA NER `UdS-LSV/hausa_voa_ner` | ha | card: cc-by-4.0; "crawled from VOA Hausa news" | out (ruled 2026-11-07), unclear terms: crawled news site; the site's terms are not stated on the card | none | | HiNER original `cfilt/HiNER-original` | hi | card: cc-by-sa-4.0; "The dataset contains sentences from ILCI and other sources. ILCI dataset requires license from Indian Language Consortium due to which we do not distribute the ILCI portion of the data." | out (ruled 2026-11-07), unclear terms: the remaining sources are not named on the card (would be the first Hindi NER candidate) | none | | KLUE-NER `klue/klue` | ko | card: cc-by-sa-4.0; the card's examples show news, review and other web text | out (ruled 2026-11-07), unclear terms: the terms on those texts are not stated | none | | FiNER `mpsilfve/finer` | fi | card: mit; the card is a blank template and names no text source | out (ruled 2026-11-07), unclear terms: the text source is not stated | none | | Persian NER `HaniehPoostchi/persian_ner` | fa | card: cc-by-4.0 | out (ruled 2026-11-07), unclear terms: the card does not name the text source | none | | Catalan NER `Ugiat/ner-cat` | ca | card: apache-2.0; television transcriptions | out (ruled 2026-11-07), unclear terms: rights in the transcribed programmes are not stated | none | | fiNERweb `whoisjones/fiNERweb` | 91 languages | card: mit; "annotated text in multiple languages"; the card does not name the label generator | out (ruled 2026-11-07), unclear terms: the label generator (and so the labels' terms) is not stated | none | | Swedish NER corpus `klintan/swedish_ner_corpus` | sv | card: cc-by-4.0; "news from Swedish newspapers' websites" | out (ruled 2026-11-07), unclear terms: newspaper text, terms not stated | none | | Europeana Newspapers `community-datasets/euronews` | de fr nl | card: cc0-1.0; historical newspapers | go (public-domain scans, CC0 card) | silver-quality historical NER; not chosen (old orthography) | | WikiANN / PAN-X | many languages | card: `license: unknown` | no-go [ruled] | - | | WikiNEuRal, MultiNERD (Babelscape) | 9 languages | cards: cc-by-nc-sa-4.0 | no-go (NC) | - | | Naamapadam (ai4bharat) | 11 Indic languages | text from Samanantar, CC BY-NC 4.0 | no-go [ruled] | - | | MultiCoNER v1/v2 | several languages | cards: cc-by-4.0; fine-grained scheme, web and search-query text | not examined (scheme does not map onto PER/ORG/LOC) | - | **NER gaps.** Of the 84 train+eval languages, those with no allowed NER training data at all: Afrikaans, Albanian, Ancient Greek, Armenian, Assamese, Bororo, Buryat, Catalan, Central Kurdish, Classical Chinese, Coptic, Czech, Egyptian, Erzya, Faroese, Finnish, Galician, Georgian, Greek, Haitian Creole, Hebrew, Icelandic, Irish, Japanese, Kazakh, Komi Permyak, Komi Zyrian, Korean, Kyrgyz, Latin, Latvian, Ligurian, Lithuanian, Livvi, Low Saxon, Maltese, Marathi, Moksha, Naija, North Sami, Northern Kurdish, Old East Slavic, Old Occitan, Ottoman Turkish, Persian, Punjabi, Russian, Sanskrit, Scottish Gaelic, Sindhi, Skolt Sami, Southern Kurdish, Telugu, Telugu English, Thai, Ukrainian, Uyghur, Uzbek, Vietnamese, Welsh, Western Armenian, Wolof, Yiddish, Zaar, Zazaki. The nine languages are covered in the sections above (Hindi remains a gap). The base-size accurate tier has entity heads for the 19 languages with allowed entity training data. The other languages provide UD annotations only; a language without an entity head does not claim cross-lingual NER. # Third-party notices OneNLP's own code is licensed `MIT OR Apache-2.0`, Copyright (c) Recursive AI, LLC. Packages depend on other Recursive AI libraries (`RecursiveAI.OneFont`, `RecursiveAI.OneModel`, `RecursiveAI.OneModel.Training`), which carry the same licence and their own notices. Each third-party dataset, model or weight file that is packed is listed here with its licence and provenance. ## Language-identification model (packed in `RecursiveAI.OneNlp`) `LanguageId/langid.bin` is trained by `tools/OneNlp.LangId` on text from the following sources, credited as their licences require. The model is a table of numbers derived from the texts; it contains no sentence. Its provenance log is `recipes/langid/PROVENANCE.md`. - **Leipzig Corpora Collection** (Wortschatz Leipzig, Leipzig University; D. Goldhahn, T. Eckart, U. Quasthoff, "Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages", LREC 2012), news, web and Wikipedia text corpora, offered for download under Creative Commons Attribution 4.0 (https://creativecommons.org/licenses/by/4.0/). The list of corpora is `tools/fetch/langid/sources.tsv`. - **Tatoeba** (https://tatoeba.org), sentences contributed by the Tatoeba community under Creative Commons Attribution 2.0 France by default (https://creativecommons.org/licenses/by/2.0/fr/); individual sentences may carry other licences. - **Wikipedia** (Southern Sotho, Tswana and Tsonga editions, and the Wikipedia text inside the Leipzig corpora), Creative Commons Attribution-ShareAlike 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), by the Wikipedia contributors. The training data of this model includes share-alike text. - **FineWeb** (Hugging Face, sample-10BT of the English edition), released under the Open Data Commons Attribution License 1.0 (https://opendatacommons.org/licenses/by/1-0/) and subject to the Common Crawl terms of use, and **English Wikipedia** (20231101 dump), Creative Commons Attribution-ShareAlike 3.0 and GFDL, by the Wikipedia contributors. Sentences of this text, labelled by an earlier English model of this project, were part of the training data of the English tagging and entity models (`recipes/en/selftrain`). The training data of these models includes share-alike text. Packages depend on other Recursive AI libraries (`RecursiveAI.OneFont`, `RecursiveAI.OneModel`, `RecursiveAI.OneModel.Training`), which carry the same licence and their own notices. `RecursiveAI.OneNlp.Models.English` ships weights we trained on CC BY-SA 4.0 treebank and NER data; its notices are in that package (`src/OneNlp.Models.English/THIRD-PARTY-NOTICES.md`). `RecursiveAI.OneNlp.Models.Embeddings` redistributes the MIT-licensed `BAAI/bge-small-en-v1.5` embedding model, converted to ONNX; its notices are in that package (`src/OneNlp.Models.Embeddings/THIRD-PARTY-NOTICES.md`). `RecursiveAI.OneNlp.Models.Embeddings.Multilingual` (with three Shard packages that carry byte ranges of the file) redistributes the MIT-licensed `intfloat/multilingual-e5-small` embedding model, converted to ONNX; its notices are in that package (`src/OneNlp.Models.Embeddings.Multilingual/THIRD-PARTY-NOTICES.md`). No other third-party models or weights ship; each addition will be listed here with its licence and provenance before it is packed. The data it does ship is listed under "Data packed with the library". ## Data packed with the library - `src/OneNlp/Recognizers/Data/shared/sequence.rules` lists the domain endings (top-level domains) of the IANA Root Zone Database, `tlds-alpha-by-domain.txt` version 2026110304 (https://data.iana.org/TLD/tlds-alpha-by-domain.txt). IANA dedicates its registry data to the public domain under CC0 1.0 (https://www.iana.org/help/licensing-terms). - `src/OneNlp/Recognizers/Data/en/currency.rules` lists the alphabetic currency codes of ISO 4217. The codes are facts of the published standard (checked against List One as published by the maintenance agency, dated 2026-09-17); the file carries no text of the standard. - `src/OneNlp/Recognizers/Data/en/holidays.rules` lists the dates of holidays. The United States federal holidays follow 5 U.S.C. 6103(a) (federal law, public domain). The bank holidays of England and Wales follow the dates published at https://www.gov.uk/bank-holidays: contains public sector information licensed under the Open Government Licence v3.0 (https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/). The other observances are customary dates, and the file carries no text of any source. - `src/OneNlp/Tokenizers/Hf/NormalizationData.g.cs`, `ScriptData.g.cs` and `BertCleanData.g.cs` are generated from the Unicode Character Database 17.0.0 (`UnicodeData.txt`, `DerivedNormalizationProps.txt`, `DerivedAge.txt`, `Scripts.txt`), © Unicode, Inc., under the Unicode License v3 (https://www.unicode.org/license.txt): permission to deal in the Data Files without restriction, provided the copyright and permission notice appear with them. ## Distilled encoder (`student-v1`) The distilled multilingual encoder (`recipes/accurate/student-v1/PROVENANCE.md`) was trained to reproduce the hidden states of **mmBERT-base** (Marone et al. 2025; MIT licence, JHU CLSP) on text from **FineWeb-2** and **FineWeb** (Hugging Face; ODC-By 1.0, subject to the Common Crawl terms of use) and **Wikipedia** (the Wikipedia contributors; CC BY-SA 3.0 and GFDL). Its token table is initialised from mmBERT's, and it uses mmBERT's tokenizer. The training data includes share-alike text. ## Open-type entity recognizer (`recipes/openner`) The open-type entity head (`recipes/openner/PROVENANCE.md`) was trained on **Few-NERD** (Ding et al. 2021; CC BY-SA 4.0, text from Wikipedia, CC BY-SA), on **Universal NER** English-EWT (Mayhew et al. 2024; CC BY-SA 4.0) and on entity labels written by **Qwen2.5-7B-Instruct-AWQ** (Alibaba Cloud; Apache-2.0, run once to label passages; its weights are not redistributed) for passages of **FineWeb** (Hugging Face; ODC-By 1.0, subject to the Common Crawl terms of use) and **English Wikipedia** (CC BY-SA 3.0 and GFDL). The training data includes share-alike text. **CrossNER** (Liu et al. 2021; MIT licence, text from Wikipedia) is used only to measure the model and is not part of any package. ## Test data (not packed) `tests/testdata/unicode/WordBreakTest.txt` and `SentenceBreakTest.txt` are the Unicode 17.0.0 UAX #29 conformance files, © 2025 Unicode, Inc., redistributed under the Unicode License v3 (https://www.unicode.org/license.txt). They are used only by the test project and are not part of any package. ## Test fixtures `tests/OneNlp.Tests/Matching/expected.json.gz` records the output of spaCy 3.8.18 (MIT licence, Copyright Explosion) on documents and patterns written for this repository. It is used only by tests. `tests/OneNlp.Tests/NounChunks/cases.json.gz` records the noun chunks spaCy 3.8.18 (MIT licence, Copyright Explosion) returns on randomly generated trees and on the tree shapes (heads, relations and parts of speech, with placeholder words) of the Universal Dependencies English-EWT development set (CC BY-SA 4.0, credit: Universal Dependencies English Web Treebank). It is used only by tests. `testdata/accurate/` holds randomly initialised tiny encoders (a two-layer BERT-style one and a four-layer ModernBERT-style one, width 16) exported to ONNX, a hand-written WordPiece `tokenizer.json` and PyTorch hidden states for fixed ids, all generated by `tools/accurate/make_tiny_fixture.py` and `make_tiny_modernbert_fixture.py`; `tiny-student.*` is a random two-layer encoder with a factorised embedding from `tools/distill/make_tiny_student_fixture.py`. No released model or third-party text is in it. - `src/OneNlp/Recognizers/Data/{es,pt}/datetime.rules` list the dates of holidays: Spain (Estatuto de los Trabajadores art. 37.2), Mexico (Ley Federal del Trabajo art. 74), Brazil (Leis 662/1949, 10.607/2002 and 14.759/2023) and Portugal (Código do Trabalho art. 234), all public law, and the Christian feasts counted from Easter. The other observances are customary dates, and the files carry no text of any source. - `src/OneNlp/Recognizers/Data/{it,nl}/datetime.rules` list the dates of holidays: Italy (Legge 27 maggio 1949, n. 260, as amended) and the Netherlands (Algemene termijnenwet art. 3), both public law, and the Christian feasts counted from Easter. The other observances are customary dates, and the files carry no text of any source. - `src/OneNlp/Recognizers/Data/zh/datetime.rules` lists the dates of Chinese holidays: the statutory ones follow the Measures for National Holidays (State Council Order, 2013) and the State Council's yearly notices, and the lunar festivals for 2022-2030 are pinned dates taken from the published calendar (2028-2030 from the astronomical calendar rather than a notice). They are facts of public law and the calendar; the number and date conventions follow Unicode CLDR as facts (Unicode License v3), with no CLDR file shipped. - `src/OneNlp/Recognizers/Data/{tr,hi}/datetime.rules` list the dates of holidays: Turkey (Law 2429 on national and public holidays; the dates of Ramazan and Kurban Bayramı from the lists of the Presidency of Religious Affairs, Diyanet) and India (the Republic Day, Independence Day and Gandhi Jayanti dates, and the gazetted and restricted holiday lists of the Department of Personnel and Training for 2026 and 2027), all facts about dates, and the Christian feasts counted from Easter. The files carry no text of any source. ## Ligurian Genoese Ligurian Treebank (GLT) The base-size multilingual model was trained on the 20-sentence training split of **UD Ligurian-GLT r2.18**, contributed by **Stefano Lusito and Jean Maillard**, under the Computational Use of Data Agreement 1.0 (C-UDA). Source and acknowledgments: . The provider acknowledges De Ferrari Editore and the newspaper O Stafî. Only computational use was made; no treebank text is redistributed with the model. C-UDA §2.2: “The C-UDA does not impose any restriction with respect to the use, modification, or distribution of Results.” The training data retains its own terms. ## Hindi hand-annotated sentences (`testdata/hi`) `testdata/hi/selfdev.conllu` (100 sentences) and `testdata/hi/nerspot.iob2` (91 sentences): 17 of the first and all of the second are Hindi web sentences from FineWeb-2 (Hugging Face, `HuggingFaceFW/fineweb-2`, Open Data Commons Attribution License 1.0, subject to the Common Crawl terms of use); the other 83 sentences of the first, and `selfdev.lemmas.tsv`, are text we wrote. All part-of-speech tags, dependency relations, lemmas and entity labels were written by us. They are used to check the Hindi model, never to train it. # Release weights and packages Weights never enter git. Each `weights.manifest.props` records file sizes and SHA-256 digests; these committed values are the trust anchor, not the downloaded archive checksums. Runtime code never downloads anything. ## Prepare the weight assets Use Python 3.12 or newer and directories outside the checkout. The search roots must contain the trained/exported files named by the manifests, including the core language-ID model, every model package, and nested accurate-tier parts. ```sh python3 tools/gate/stage_weights.py "$STAGED" "$MODEL_ROOT" "$DISTILLED_ROOT" > "$ENV_FILE" python3 tools/gate/weight_assets.py create "$STAGED" "$ASSET_DIR" python3 tools/gate/weight_assets.py list ``` `STAGED` holds symlinks, so the search roots must remain in place until asset creation finishes. `ASSET_DIR` must not exist yet. Each archive contains the regular file bytes of one manifest (not symlinks). It must be strictly smaller than GitHub's 2 GiB asset limit; an oversized manifest fails rather than silently splitting or omitting files. Archive headers are deterministic. The current set has 39 archives and `SHA256SUMS`: ```text weights-LangId.tar weights-Chinese.tar weights-Dutch.tar weights-Embeddings.tar weights-Embeddings.Multilingual.tar weights-Embeddings.Multilingual.Shard1.tar weights-Embeddings.Multilingual.Shard2.tar weights-Embeddings.Multilingual.Shard3.tar weights-English.tar weights-French.tar weights-German.tar weights-Hindi.tar weights-Italian.tar weights-Multilingual.tar weights-Multilingual.Large.tar weights-Multilingual.Large.Chinese.tar weights-Multilingual.Large.Encoder1.tar weights-Multilingual.Large.Encoder2.tar weights-Multilingual.Large.Heads1.tar weights-Multilingual.Large.Heads10.tar weights-Multilingual.Large.Heads11.tar weights-Multilingual.Large.Heads2.tar weights-Multilingual.Large.Heads3.tar weights-Multilingual.Large.Heads4.tar weights-Multilingual.Large.Heads5.tar weights-Multilingual.Large.Heads6.tar weights-Multilingual.Large.Heads7.tar weights-Multilingual.Large.Heads8.tar weights-Multilingual.Large.Heads9.tar weights-Multilingual.Large.TokenTable1.tar weights-Multilingual.Large.TokenTable2.tar weights-Multilingual.Large.TokenTable3.tar weights-Multilingual.Large.TokenTable4.tar weights-Multilingual.Encoder.tar weights-Multilingual.FrozenEncoder.tar weights-Multilingual.Segmenters.tar weights-Portuguese.tar weights-Spanish.tar weights-Turkish.tar SHA256SUMS ``` Only a release maintainer performs the following upload, after approving the weights and selecting the existing source commit. The weights release is separate from version releases; its tag must **not** start with `v` (which triggers package publication). Use a new tag, never replace an existing asset set. ```sh REPO=recursive-ai-oss/onenlp-csharp WEIGHTS_TAG=weights-1.0.0 # SOURCE_COMMIT is the full reviewed source commit hash, not a branch name. gh release create "$WEIGHTS_TAG" "$ASSET_DIR"/weights-*.tar "$ASSET_DIR/SHA256SUMS" \ --repo "$REPO" --target "$SOURCE_COMMIT" --draft --latest=false \ --title "OneNLP 1.0.0 model weights" \ --notes "Model weights; file sizes and SHA-256 digests are pinned in the source manifests." ``` Review the asset list and checksums before publishing that draft. A read-only workflow token cannot download another draft's assets, so publish the weights release before creating the package version tag. Set the repository Actions variable `ONENLP_WEIGHTS_RELEASE` to that weights tag. This variable is required; there is no fallback to the latest release. Creating or publishing releases is not part of the preparation scripts. ## Verify and pack without uploading ```sh # Start with an empty download directory. GH_TOKEN needs contents:read only. mkdir -p "$DOWNLOAD_DIR" while IFS= read -r asset; do gh release download "$WEIGHTS_TAG" --repo "$REPO" --pattern "$asset" --dir "$DOWNLOAD_DIR" done < <(python3 tools/gate/weight_assets.py list) python3 tools/gate/weight_assets.py unpack "$DOWNLOAD_DIR" "$VERIFIED" > "$ENV_FILE" source "$ENV_FILE" python3 tools/gate/stage_weights.py "$VERIFIED" --verify rm -r "$DOWNLOAD_DIR" dotnet pack OneNlp.slnx -c Release -o "$PACKAGES_DIR" ``` Unpacking requires a new output directory. It rejects missing or extra assets, checksum mismatches, unexpected/duplicate tar members, links, traversal, wrong sizes and file-content mismatches. Files are hashed while writing, then checked against every manifest. Environment exports are produced only after all files pass. `--github-env FILE` appends those same variables for subsequent Actions steps. Shared parts use one root directory per weights variable: `ONENLP_MULTILINGUAL_WEIGHTS`, `ONENLP_MLLARGE_WEIGHTS` and `ONENLP_EMBEDDINGS_E5_WEIGHTS`; nested paths such as `heads/en.onenlp` are preserved. The language-ID and ten fast-tier variables are also set explicitly. `release.yml` downloads that exact list with the job's read-only `GITHUB_TOKEN`, verifies and stages it **before** restore/build/test/pack, and removes the archives to save disk. Existing pinned actions and publishing permissions are unchanged. Missing weights or any mismatched file stop the job before publication. Model projects also retain their own hash checks and reject packing without weights. Allow at least 35 GiB of free disk, in addition to SDK and package caches, for a full solution build and pack. Weights alone total about 5.5 GiB; dependent tools copy model assemblies and assets into their own outputs. A disk-bounded pack check can pack one project at a time and remove its packages after recording their hashes, but publication needs to retain the complete package set. After verification, delete scratch archives, staged weights, package outputs and build artifacts that are no longer needed. # Changelog All notable changes to this project are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] ## [1.0.0] - 2026-11-08 First release. ### Changed - The neural runtime and trainer use OneModel 1.2.0. Checkpoints stream through OneModel, including files over 2 GiB; single-resource embedded encoders and embeddings load in place and keep their resource streams open for the lifetime of the model. Split-resource graphs still use the buffered concatenation path. ### Added - `TransformerEncoder.FromStreams` and `SentenceEmbedder.Load` have stream-ownership overloads (`leaveOpen`). They opt into in-place loading for embedded graphs; the existing overloads keep their buffered lifetime contract. With `leaveOpen: false`, the model stream is closed on disposal or load failure; tokenizer streams remain caller-owned. - `EntityRecognizer.MinConfidence` drops decoded entities below a confidence (default 0, which changes nothing); it is saved in the model file only when above 0, so existing files load unchanged. The Chinese entity model ships with 0.6. ### Changed (breaking) - `Nlp.Entities(text)`, `Nlp.Lemmas(text)` and `Nlp.Analyze(text)`: when the language detector is unsure (confidence below 0.8) and has no model for its answer, the text is read in the most likely of the languages that have a model, so "Paris", "Ok" or a name is read by the model you referenced instead of throwing from one verb and silently losing entities in the other. Confident answers are unchanged: French text without a French model is still an error naming the package. - A missing model throws `ModelNotAvailableException` (still an `InvalidOperationException`) with `Language`, `Tier` and `PackageIds`; messages name only packages that exist. - `Nlp` verbs and `Pipeline.Process`/`ProcessMany` refuse text longer than `Nlp.MaxLength` / `Pipeline.MaxLength` (1,000,000 characters by default) with an `ArgumentOutOfRangeException`. Texts above that size that used to be accepted now need a raised limit. - `Tokenizer.FromFile("vocab.txt")` no longer reads letters without a case pair (such as U+211D, which `bert-base-uncased` contains) as evidence of a cased vocabulary: that vocabulary is now uncased, as it is. - `Recognizer.Recognize(text, context = default)` is two overloads, `Recognize(text)` and `Recognize(text, context)`; a `default(DateTimeOffset)` reference date throws `ArgumentException` here and in `Nlp.Dates` and `Nlp.ParseDate`. - `Language.Unknown.ToString()` is `"Unknown"` (its `Code` is still empty). - `PatternFormatException` and `RecognizerRulesException` carry `LineNumber` (and `BytePositionInLine` for patterns); pattern messages no longer end in `(Parameter 'name')`, and rule-line messages no longer repeat "Line N" inside the text they pass to the constructor. - `Token.Text` and `Span.Text` are now `string`s cut from `Doc.Text`, so `token.Text == "x"`, `doc.Select(t => t.Text)` and `ToDictionary(s => s.Text)` work. The copy-free view is `TextSpan` (a `ReadOnlySpan`). *Migration:* code that wanted the span, such as `token.Text.SequenceEqual(...)` or a `ReadOnlySpan` local, reads `token.TextSpan`; `token.Text.ToString()` becomes `token.Text`. - One default word tokenizer: `PhraseMatcher`, `EntityRuler` phrases and `JsonlOptions.Tokenizer` now default to the tokenizer `Pipeline.Blank` uses (`UaxWordTokenizer.Default`) instead of `RuleWordTokenizer`. An `EntityRuler` added to a pipeline splits its phrases with that pipeline's tokenizer unless it was given one. New: `new PhraseMatcher(pipeline)` and `JsonlReader.Read(path | stream | reader, pipeline)`. *Migration:* calls that passed `null` for the string table with options, `JsonlReader.Read(path, null, options)`, are now ambiguous; write `JsonlReader.Read(path, strings: null, options: options)`, or pass the pipeline. - `EntityRuler` has no indexer. It is a collection: `new EntityRuler { { "CASE_NO", Pattern.Shape("dd-dddd") } }` (or `ruler.Add(label, pattern)`). *Migration:* `new EntityRuler { ["X"] = p }` becomes `new EntityRuler { { "X", p } }`. - Serialization verbs: `Doc.WriteTo(Stream)` is `Doc.WriteBinary(Stream)`, `Doc.FromBytes(Stream)` is `Doc.FromStream(Stream)`, and `DocBin.WriteTo(Stream)` is `DocBin.WriteBinary(Stream)`. - `TrainOptions`, `TrainResult`, `TrainProgress`, `DevScores` (was `RecursiveAI.OneNlp.Neural.Training`) and `Example` (was `RecursiveAI.OneNlp.Data`) are in the `RecursiveAI.OneNlp` namespace. *Migration:* delete `using RecursiveAI.OneNlp.Neural.Training;`; `using RecursiveAI.OneNlp.Data;` is still needed for the readers and `Alignment`. - Training a head that already has weights (a packaged pipeline) on data with labels it lacks: an `EntityRecognizer` now appends the new labels and keeps its weights (spaCy's "add a label, then train"); before, the labels were silently dropped. A `Tagger`, `Morphologizer`, `EditTreeLemmatizer`, `Parser`, `SpanCategorizer` or `TextClassifier` throws `ArgumentException` naming the unknown values (part-of-speech and feature combinations, edit trees seen at least `minClassCount` times, dependency relations, span labels, or categories) instead of skipping them. `ITrainableComponent.ExtendLabels` is the new member (a default interface method). - `Pipeline.Add("sentencizer")` on a pipeline for a language uses that language's segmentation data (English `Mr.` no longer ends a sentence); it used the neutral data before. Settings that name `data` or `language` still win. - `SpanCategorizer` default threshold is 0.8 (was 0.5), and `Threshold` is settable. At 0.8, chosen on a development split, the English UNER precision is 76.0 (was 70.3; spaCy 75.4) with F1 63.8 (was 64.4). Saved models keep the threshold they were saved with. - `ModelNotAvailableException` for a language no published package covers (for example Basque) has an empty `PackageIds` and says so, instead of naming a package that cannot help; a package already referenced is never named. - `Span` and `Token` write their `start` and `end` in code points through `System.Text.Json`, as the `Doc` form does (they were UTF-16 offsets). The two differ only after a character outside the Basic Multilingual Plane; `StartChar` and `Idx` are unchanged. - `default(Token)` and `default(Span)` members throw `InvalidOperationException` (they threw `NullReferenceException`). `Example.Create` throws `ArgumentOutOfRangeException` for an offset outside the text. `Doc.FromJson` messages no longer end in `(Parameter 'name')`. - `Pipeline.ProcessMany` takes a trailing `CancellationToken` (source compatible; a method group or delegate bound to the old signature must be rebuilt). ### Added - `Nlp.Entities(text, types, threshold)` for open-type entities with a probability threshold (the default stays 0.5); `docs/open-ner.md` says where the recognizer comes from. - `Nlp.Entities`, `Nlp.Analyze`, `Nlp.AnalyzeMany` and `Nlp.Lemmas` overloads that take a `Language`; `Nlp.MaxLength`; `Pipeline.MaxLength`. - `Tokenizer.FromWordPieceVocabulary(path or stream, WordPieceVocabularyOptions)` states the casing of a `vocab.txt`; `Tokenizer.EncodeToIds(text, addSpecialTokens)` returns an `int[]`. - `ModelNotAvailableException`. - `System.Text.Json` works on `Doc` (it writes and reads the `Doc.ToJson` shape), and writes `Span` and `Token` as objects with character offsets, label and text; no reflection, so it also works under NativeAOT with a source-generated context. - `Example.Create(pipeline, text, entities, cats)` builds a training example from character offsets, tokenized by the pipeline. - `WriteJsonAsync`, `WriteBinaryAsync`, `FromJsonAsync` and `FromStreamAsync` on `Doc`, `WriteBinaryAsync` and `FromStreamAsync` on `DocBin`, and `DocBinReader` is an `IEnumerable` with `ReadAsync` and `ReadAllAsync`. - Documentation for the first release: a rewritten README and package readmes, `docs/results-table.md` (every quality row with its reference and interval), side-by-side "reads like" samples (`docs/reads-like.md`), migration pages for Catalyst, Microsoft.Recognizers.Text, Lingua and Microsoft.ML.Tokenizers, guides for training, the fast tier and classification, and `samples/OneNlp.Samples`, which builds and runs every sample the docs show. - Recognizer comparison files for Turkish and Hindi test sets, and refreshed English ones that include date/time. - Sentence embeddings: `IEmbedder`, `SentenceEmbedder` (sentence-transformers-format ONNX models, CLS/mean/max pooling, normalization, batching, opt-in int8), `Nlp.Embed`/`Nlp.EmbedMany`, `EmbedderPackages`, `Tokenizer.IsSpecialToken`, and the `RecursiveAI.OneNlp.Models.Embeddings` package (bge-small-en-v1.5, MIT). Parity and MTEB results: `docs/embeddings.md`. - French: `SegmentationData.French` (elision, inversion clitics, `du`/`au`/`aux`/`des` as multiword-token pieces, digit groups) with `TryMatchMultiword`, the recipes in `recipes/fr` and the `RecursiveAI.OneNlp.Models.French` package. - Dutch: `SegmentationData.Dutch` (apostrophe words and short forms, abbreviations, dash amounts), the recipe in `recipes/nl/fast` and the `RecursiveAI.OneNlp.Models.Dutch` package (tagging, morphology, lemmas, parser, entities on UD Dutch-Alpino and WikiNER Dutch). - Language identification: `LanguageDetector` names the language of a text (75 languages, a 3.6 MB model embedded at build time from `ONENLP_LANGID_WEIGHTS`) from one word to a document, with confidences, `Restrict(languages)` and mixed-language runs (`DetectSpans`). `Nlp.DetectLanguage` is the happy path, and `Nlp.Analyze`, `Nlp.Tokens` and `Nlp.Sentences` now use the detected language's segmentation data. Method, model file and measurements are in `docs/language-id.md`; the recipe and provenance log are in `recipes/langid`. - `RecursiveAI.OneNlp.Models.English`: the English fast-tier model package (tags, morphology, lemmas, dependency parse, PER/ORG/LOC entities) with weights embedded; `ModelPackages` (the registry model packages use), `Nlp.Entities`, `Nlp.Lemmas`, `Nlp.AnalyzeMany`, and `Nlp.Analyze` filling every annotation when a model package is referenced. See `docs/model-packages.md`. - Recognizers (`Recognizer`, `RecognizerComponent`, `Quantity`, `SequenceMatch`, `Nlp.Quantities`, `Nlp.Numbers`, `Nlp.Sequences`): numbers and ordinals in digits and words, percentages, measures (length, mass, volume, area, speed, information, temperature, age), money with ISO 4217 codes, e-mail, URL, IPv4/IPv6, phone-like sequences (unvalidated), hashtags, mentions and GUIDs, from per-language rule data that is embedded in the assembly or loaded from files. English rules are included; see `docs/recognizers.md`. - Recognizers: a number in words repeated in brackets is one result (`sixty (60) days`, `five thousand dollars ($5,000)`; a disagreeing numeral lowers the confidence and is not merged); durations of business days (`ten business days`, `P10BD`), `calendar` units, and `TemporalMatch.Count` and `TemporalMatch.Unit` (`DurationUnit`) on every duration; `in 3 business days` resolves to a date; compound measures (`5 feet 3 inches` is 63 inches); Indian digit grouping (`12,34,567`) and scientific notation (`6.02e23`, `1.5×10^3`); `1,000th` as an ordinal; `1,000 pounds` is money in a money context; `wait a second` is a duration; `no one`, `one of them` and `for the first time` are not numbers; the ISO 4217 code list was checked against the published List One. - Repository skeleton: `RecursiveAI.OneNlp` and `RecursiveAI.OneNlp.Neural` projects, tests, a NativeAOT smoke app, a benchmark project and a fuzz harness. - `Doc`, `Token` and `Span`: columnar documents with character offsets into the original string, interned attribute columns, entities, sentences and confidences. - `Pipeline`, `IPipelineComponent`, `IWordTokenizer` and `ComponentRegistry`: ordered components, batch and streaming processing, and reflection-free registration by name. - `Tokenizer`, `TokenEncoding` and the format loaders' entry points (`Tokenizer.FromFile`, `Tokenizer.FromStream`): the contract the tokenizer families will implement. - `RuleWordTokenizer` and the `Nlp.Analyze` static. - `IEncoder`, the first shape of the encoder seam, in `RecursiveAI.OneNlp.Neural`. - `Doc.ToJson`/`Doc.FromJson` in the shape spaCy documents for `Doc.to_json` (offsets in code points, extra fields for norm, confidences and sentence boundaries), and a compact, versioned, checksummed binary form (`Doc.ToBytes`/`Doc.FromBytes`), plus `DocBin` and a streaming `DocBinReader` that share one string table. Readers reject malformed, truncated and oversized input with `InvalidDataException`. Formats are specified in `docs/json-format.md` and `docs/binary-format.md`. ### Fixed - `bench/OneNlp.Bench` runs out of process under BenchmarkDotNet (the assembly name now matches the project file); `tools/oracles/bench_recognizers_lang.sh` builds its own tools, uses small inputs where the comparator is slow (Turkish, Hindi) and times out each run.