--- title: "onenlp docs" url: https://recursiveai.org/onenlp/docs/ description: "Everything about OneNLP 1.0.0: getting started, the model packages, tokenizers and segmentation, rule matching, recognizers and language ID, the fast…" version: "1.0.0" --- # onenlp docs Everything about OneNLP 1.0.0: getting started, the model packages, tokenizers and segmentation, rule matching, recognizers and language ID, the fast, distilled and base-size accurate tiers, embeddings, training, file formats and every measured result. These docs are for OneNLP 1.0.0, read from main at commit 59bd14c after the v1.0.0 tag (documentation changes only). ## Start - [Getting started](https://recursiveai.org/onenlp/docs/getting-started/index.md): Install OneNLP for .NET 10, find the language, sentences, numbers and dates of a text with the core package alone, then add a model package for entities… - [Overview](https://recursiveai.org/onenlp/docs/overview/index.md): Text in, structure out: tokens, sentences, tags, morphology, lemmas, dependency parses and entities through fast and accurate model tiers; token-exact… - [How the API reads](https://recursiveai.org/onenlp/docs/reads-like/index.md): One sample per layer, each next to the library or API it will feel familiar from. The C# is the code in samples/OneNlp.Samples, which builds and runs… - [Contracts](https://recursiveai.org/onenlp/docs/contracts/index.md): Behaviour that every component and tokenizer family relies on. Language-neutral and normative. - [Model packages](https://recursiveai.org/onenlp/docs/model-packages/index.md): A model package carries trained weights for one language (or one task) and makes the Nlp verbs use them. Nothing is ever downloaded: the weights are… ## Coming from another library - [From Catalyst](https://recursiveai.org/onenlp/docs/migrate-catalyst/index.md): Catalyst is a managed spaCy-style pipeline for .NET. This page maps its calls to OneNLP's. It is written from Catalyst's published README and from running… - [From Microsoft.Recognizers.Text](https://recursiveai.org/onenlp/docs/migrate-recognizers-text/index.md): Microsoft.Recognizers.Text finds numbers, units, currency, sequences and date/time expressions and returns ModelResult values. OneNLP's recognizers do the… - [From Lingua](https://recursiveai.org/onenlp/docs/migrate-lingua/index.md): Lingua is a language detector for short and long text; SearchPioneer.Lingua is its .NET port. OneNLP's LanguageDetector (namespace… - [From Microsoft.ML.Tokenizers](https://recursiveai.org/onenlp/docs/migrate-ml-tokenizers/index.md): Microsoft.ML.Tokenizers loads one tokenizer type per model family. OneNLP's Tokenizer loads Hugging Face tokenizer.json, SentencePiece .model, vocab.txt… ## Text, tokens and rules - [Words and sentences](https://recursiveai.org/onenlp/docs/segmentation/index.md): Normative behaviour of the word tokenizer and the sentence segmenter. Language-neutral except where a language's data is named. - [Tokenizer parity](https://recursiveai.org/onenlp/docs/parity/index.md): OneNLP's tokenizers must equal the reference libraries byte for byte: Hugging Face tokenizers for the WordPiece, Unigram, SentencePiece-BPE and byte-level… - [BPE and Tiktoken](https://recursiveai.org/onenlp/docs/tokenizers-bpe-tiktoken/index.md): Behaviour of the byte-pair families. Language-neutral and normative. Pinned references: Hugging Face tokenizers 0.22.1 and tiktoken 0.14.0. - [WordPiece and Unigram](https://recursiveai.org/onenlp/docs/tokenizers-wordpiece-unigram/index.md): Normative behaviour of the three model families behind the tokenizer.json pipeline and of the two file readers that build a tokenizer without a… - [Matcher and EntityRuler](https://recursiveai.org/onenlp/docs/matching/index.md): How Matcher, PhraseMatcher and EntityRuler behave. The token-pattern vocabulary (attributes, operators, predicates) follows the one spaCy documents for… ## Recognizers and language ID - [Recognizers](https://recursiveai.org/onenlp/docs/recognizers/index.md): Behaviour of the recognizers for numbers, ordinals, percentages, measures, money, address-like sequences, and dates and times. Language-neutral where it… - [The recognizer gold sets](https://recursiveai.org/onenlp/docs/recognizer-gold/index.md): A hand-labelled set of numbers, quantities, sequences and dates/times in short sentences. It judges recognizers: ours and two black-box comparators. We… - [Language identification](https://recursiveai.org/onenlp/docs/language-id/index.md): LanguageDetector (namespace RecursiveAI.OneNlp.LanguageId) names the language of a text from one word up to a document, among the 75 languages listed in… ## Models - [The fast tier](https://recursiveai.org/onenlp/docs/fast-tier/index.md): The fast tier is a per-language pipeline built on a small hash-embedding encoder: each head reads vectors made from the hashed lower-cased form, prefix… - [The distilled accurate tier](https://recursiveai.org/onenlp/docs/distilled-tier/index.md): RecursiveAI.OneNlp.Models.Multilingual: one six-layer, width-384 student of mmBERT-base (35.5 M weights, distilled in PyTorch from mmBERT-base and… - [The base-size accurate tier](https://recursiveai.org/onenlp/docs/accurate-tier/index.md): RecursiveAI.OneNlp.Models.Multilingual.Large shares one fine-tuned mmBERT-base encoder across 84 UD languages, with entity heads for 19. It is distinct… - [The accurate-tier encoder](https://recursiveai.org/onenlp/docs/accurate-encoder/index.md): How an exported transformer becomes an IEncoder, what was measured, and what can and cannot be trained. Tools are in tools/accurate/ and… - [The accurate-tier corpus](https://recursiveai.org/onenlp/docs/accurate-corpus/index.md): The languages the accurate tier is trained on, as recipes/accurate/common/corpus.json describes them (written by tools/accurate/make_corpus_spec.py from… - [Parser and noun chunks](https://recursiveai.org/onenlp/docs/parser/index.md): Normative for the parser head (RecursiveAI.OneNlp.Neural.Components.Parser) and Doc.NounChunks. - [Open-type entities](https://recursiveai.org/onenlp/docs/open-ner/index.md): Entity types named at run time: Nlp.Entities(text, "person", "organization"), with a threshold Nlp.Entities(text, ["contract party"], 0.1f), or… - [Classification and span categories](https://recursiveai.org/onenlp/docs/classification/index.md): Two heads for labels you define: TextClassifier gives a document one or several categories, and SpanCategorizer labels spans of a document, including… - [Sentence embeddings](https://recursiveai.org/onenlp/docs/embeddings/index.md): IEmbedder (core) turns text into a fixed-length vector; SentenceEmbedder (RecursiveAI.OneNlp.Neural) runs a sentence-transformers-format model on… ## Training and files - [Training and recipes](https://recursiveai.org/onenlp/docs/training/index.md): Everything a user can train, the shipped models were trained with the same public Train API, on OneModel (automatic differentiation, AdamW, checkpoints)… - [Saved pipelines](https://recursiveai.org/onenlp/docs/model-files/index.md): Normative for PipelineFile (RecursiveAI.OneNlp.Neural). One file holds a trained pipeline: its structure and label sets as JSON, and the weights of its… - [Data formats](https://recursiveai.org/onenlp/docs/data-formats/index.md): Normative for the readers in RecursiveAI.OneNlp.Data. Both readers stream (one sentence or line at a time), bound every size before allocating, and reject… - [JSON document format](https://recursiveai.org/onenlp/docs/json-format/index.md): Normative and language-neutral. A document is a JSON object in the shape spaCy documents for Doc.to_json, so documents can move between the two. Fields of… - [Binary document format](https://recursiveai.org/onenlp/docs/binary-format/index.md): Normative and language-neutral. Two containers share one body encoding: a single document (ONDC) and a bin of many documents (ONDB) that share a string… ## Measured results - [Results table](https://recursiveai.org/onenlp/docs/results-table/index.md): One table of what is measured, what it is measured against, and whether it holds. Every number is copied from a file in this repository, named in the… - [Results](https://recursiveai.org/onenlp/docs/results/index.md): Measured rows. The tokenizer speed rows below are final: they were taken in one benchmark-queue group per machine, alone on the machine, with… - [Base-size accurate tier: every row](https://recursiveai.org/onenlp/docs/accurate-tier-rows/index.md): Intervals: 95% bootstrap, 1,000 resamples, seed 20261106. The published reference passes when it is no higher than our upper bound; these are not paired… - [Distilled accurate tier: every row](https://recursiveai.org/onenlp/docs/distilled-tier-rows/index.md): Every row of the distilled accurate tier, RecursiveAI.OneNlp.Models.Multilingual, per language and input mode, with the fast tier and the published Stanza, UDPipe 2 and Trankit numbers beside it. Every row is information: the base-size tier is the one that gates. - [x64 speed rows](https://recursiveai.org/onenlp/docs/speed-x64/index.md): Measured 2026-11-08 on an AMD Ryzen 9 7950X3D (16 physical cores, 32 logical), x64 Linux 7.0.0-38-generic, .NET SDK 10.0.403, runtime 10.0.14. Available… - [Tokenizer speed: x64](https://recursiveai.org/onenlp/docs/tokenizer-speed-x64/index.md): See speed-x64.md for machine, input hashes, versions and method. Speedup is comparator time divided by ours. Load is information, not gated. - [Measurement](https://recursiveai.org/onenlp/docs/measurement/index.md): How OneNLP scores its own output, how it puts confidence intervals on scores, and how a results row is judged against a reference. The scorers live in… - [Published reference results](https://recursiveai.org/onenlp/docs/references/index.md): Numbers for the accurate tier to be judged against, copied from the sources named in each row, nothing re-run. Every number is an F1 (or accuracy where… - [Model provenance](https://recursiveai.org/onenlp/docs/provenance/index.md): Every trained model, ours or a baseline we train for comparison, ships with a provenance log. The log is a Markdown file named PROVENANCE.md next to the… ## Inside the library - [Treebank licences](https://recursiveai.org/onenlp/docs/ud-licences/index.md): The first sections cover ZH, FR, ES, PT, DE, IT, TR, HI and NL in detail. The last section ("Every treebank of release 2.18", generated by… - [Third-party notices](https://recursiveai.org/onenlp/docs/third-party-notices/index.md): OneNLP's own code is licensed MIT OR Apache-2.0, Copyright (c) Recursive AI, LLC. - [Release weights and packages](https://recursiveai.org/onenlp/docs/release/index.md): Weights never enter git. Each weights.manifest.props records file sizes and SHA-256 digests; these committed values are the trust anchor, not the… - [Changelog](https://recursiveai.org/onenlp/docs/changelog/index.md): All notable changes to this project are documented in this file.