onenlpDocs
onenlp docs
Everything about OneNLP 1.0.0: getting started, the model packages, tokenizers and segmentation, rule matching, recognizers and language ID, the fast, distilled and base-size accurate tiers, embeddings, training, file formats and every measured result. These docs are for OneNLP 1.0.0, read from main at commit 59bd14c after the v1.0.0 tag (documentation changes only).
Start
Getting startedInstall OneNLP for .NET 10, find the language, sentences, numbers and dates of a text with the core package alone, then add a model package for entities, lemmas and the parse, and another for sentence embeddings. Every program below was run on the 1.0.0 packages from NuGet…OverviewText in, structure out: tokens, sentences, tags, morphology, lemmas, dependency parses and entities through fast and accurate model tiers; token-exact Hugging Face tokenizers; numbers, units, currency, e-mail and URLs, and dates and times resolved to values in ten languages…How the API readsOne sample per layer, each next to the library or API it will feel familiar from. The C# is the code in samples/OneNlp.Samples, which builds and runs (dotnet run --project samples/OneNlp.Samples -- all ); the other side is that library's own documented usage. The samples need…ContractsBehaviour that every component and tokenizer family relies on. Language-neutral and normative.Model packagesA model package carries trained weights for one language (or one task) and makes the Nlp verbs use them. Nothing is ever downloaded: the weights are inside the package, embedded as resources.
Coming from another library
From CatalystCatalyst is a managed spaCy-style pipeline for .NET. This page maps its calls to OneNLP's. It is written from Catalyst's published README and from running its released packages as a black box (versions 26.10.6514 and English models 26.9.5316); nothing here comes from its…From Microsoft.Recognizers.TextMicrosoft.Recognizers.Text finds numbers, units, currency, sequences and date/time expressions and returns ModelResult values. OneNLP's recognizers do the same job with one engine and rules kept as data per language. This page maps the calls. It is written from the package's…From LinguaLingua is a language detector for short and long text; SearchPioneer.Lingua is its .NET port. OneNLP's LanguageDetector (namespace RecursiveAI.OneNlp.LanguageId) does the same jobs. This page maps the calls. It is written from Lingua's README and from running…From Microsoft.ML.TokenizersMicrosoft.ML.Tokenizers loads one tokenizer type per model family. OneNLP's Tokenizer loads Hugging Face tokenizer.json, SentencePiece .model, vocab.txt and .tiktoken files through one type and matches Hugging Face tokenizers and tiktoken byte for byte. This page maps the…
Text, tokens and rules
Words and sentencesNormative behaviour of the word tokenizer and the sentence segmenter. Language-neutral except where a language's data is named.Tokenizer parityOneNLP's tokenizers must equal the reference libraries byte for byte: Hugging Face tokenizers for the WordPiece, Unigram, SentencePiece-BPE and byte-level BPE families, and tiktoken for the Tiktoken encodings. One mismatch is a bug, never a tolerance. This page defines what…BPE and TiktokenBehaviour of the byte-pair families. Language-neutral and normative. Pinned references: Hugging Face tokenizers 0.22.1 and tiktoken 0.14.0.WordPiece and UnigramNormative behaviour of the three model families behind the tokenizer.json pipeline and of the two file readers that build a tokenizer without a tokenizer.json. Offsets are UTF-16 units of the piece the model is given. A piece is never empty.Matcher and EntityRulerHow Matcher, PhraseMatcher and EntityRuler behave. The token-pattern vocabulary (attributes, operators, predicates) follows the one spaCy documents for its rule-based matching; where that description is silent, behaviour was settled by running spaCy 3.8.18 on probe inputs…
Recognizers and language ID
RecognizersBehaviour of the recognizers for numbers, ordinals, percentages, measures, money, address-like sequences, and dates and times. Language-neutral where it can be; the English rules are the first data set. Rules are data: adding a language means writing rule files (see "Rule…The recognizer gold setsA hand-labelled set of numbers, quantities, sequences and dates/times in short sentences. It judges recognizers: ours and two black-box comparators. We wrote every sentence and label ourselves from the type definitions, ISO 4217, the RFCs and SI, and everyday text. No…Language identificationLanguageDetector (namespace RecursiveAI.OneNlp.LanguageId) names the language of a text from one word up to a document, among the 75 languages listed in tools/fetch/langid/languages.tsv. The model is one file of 3.6 MB that is embedded in the core assembly at build time. This…
Models
The fast tierThe fast tier is a per-language pipeline built on a small hash-embedding encoder: each head reads vectors made from the hashed lower-cased form, prefix, suffix and shape of every token, mixed by a few window layers (and, for some languages, one bidirectional LSTM layer). It…The distilled accurate tierRecursiveAI.OneNlp.Models.Multilingual: one six-layer, width-384 student of mmBERT-base (35.5 M weights, distilled in PyTorch from mmBERT-base and exported once; recipes/accurate/student-v1), fine-tuned with per-language heads for English, German, French, Spanish, Portuguese…The base-size accurate tierRecursiveAI.OneNlp.Models.Multilingual.Large shares one fine-tuned mmBERT-base encoder across 84 UD languages, with entity heads for 19. It is distinct from .Models.Multilingual, the distilled model covering nine languages. The full language list is…The accurate-tier encoderHow an exported transformer becomes an IEncoder, what was measured, and what can and cannot be trained. Tools are in tools/accurate/ and tools/OneNlp.Accurate/; the recipe is recipes/accurate/en-tagging/.The accurate-tier corpusThe languages the accurate tier is trained on, as recipes/accurate/common/corpus.json describes them (written by tools/accurate/make_corpus_spec.py from the fetched files; licence verdicts in ud-licences.md). Release r2.18 of Universal Dependencies.Parser and noun chunksNormative for the parser head (RecursiveAI.OneNlp.Neural.Components.Parser) and Doc.NounChunks.Open-type entitiesEntity types named at run time: Nlp.Entities(text, "person", "organization"), with a threshold Nlp.Entities(text, ["contract party"], 0.1f), or OpenEntityRecognizer.Recognize(doc, types) on a document you tokenized.Classification and span categoriesTwo heads for labels you define: TextClassifier gives a document one or several categories, and SpanCategorizer labels spans of a document, including spans that overlap or nest. Both sit on any encoder (the fast encoder by default), train with Train (training.md) and are…Sentence embeddingsIEmbedder (core) turns text into a fixed-length vector; SentenceEmbedder (RecursiveAI.OneNlp.Neural) runs a sentence-transformers-format model on OneModel. Nlp.Embed is the happy path over the default model in RecursiveAI.OneNlp.Models.Embeddings.
Training and files
Training and recipesEverything a user can train, the shipped models were trained with the same public Train API, on OneModel (automatic differentiation, AdamW, checkpoints), in C#. A recipe is a small console project over that API; running ours on your data is "retraining our model". Behaviour…Saved pipelinesNormative for PipelineFile (RecursiveAI.OneNlp.Neural). One file holds a trained pipeline: its structure and label sets as JSON, and the weights of its encoders and heads in OneModel's checkpoint encoding.Data formatsNormative for the readers in RecursiveAI.OneNlp.Data. Both readers stream (one sentence or line at a time), bound every size before allocating, and reject malformed input with a DataFormatException that carries the 1-based line number (LineNumber; the message starts line N:).JSON document formatNormative and language-neutral. A document is a JSON object in the shape spaCy documents for Doc.to_json, so documents can move between the two. Fields of ours are additions that other readers can ignore (spaCy 3.8.18's Doc.from_json does).Binary document formatNormative and language-neutral. Two containers share one body encoding: a single document (ONDC) and a bin of many documents (ONDB) that share a string table. Version 1.
Measured results
Results tableOne table of what is measured, what it is measured against, and whether it holds. Every number is copied from a file in this repository, named in the "Source" column or the section heading; nothing here was measured for this page. Rows that lose, and gaps, are listed in their…ResultsMeasured rows. The tokenizer speed rows below are final: they were taken in one benchmark-queue group per machine, alone on the machine, with bench/scripts/tokenizer-rows.sh. Speed numbers elsewhere on this page are indicative until they are re-measured alone on the machine.Base-size accurate tier: every rowIntervals: 95% bootstrap, 1,000 resamples, seed 20261106. The published reference passes when it is no higher than our upper bound; these are not paired intervals against the reference. Gold and raw input are never crossed. A missing reference is not a pass. The reference UD…Distilled accurate tier: every rowEvery row of the distilled accurate tier, RecursiveAI.OneNlp.Models.Multilingual, per language and input mode, with the fast tier and the published Stanza, UDPipe 2 and Trankit numbers beside it. Every row is information: the base-size tier is the one that gates.x64 speed rowsMeasured 2026-11-08 on an AMD Ryzen 9 7950X3D (16 physical cores, 32 logical), x64 Linux 7.0.0-38-generic, .NET SDK 10.0.403, runtime 10.0.14. Available ISA: SSE4.2, AVX2, AVX-512F/BW/VL, FMA and BMI2. CPU inference only; no GPU used. OneNLP 0.4.0-preview.1 source build…Tokenizer speed: x64See speed-x64.md for machine, input hashes, versions and method. Speedup is comparator time divided by ours. Load is information, not gated.MeasurementHow OneNLP scores its own output, how it puts confidence intervals on scores, and how a results row is judged against a reference. The scorers live in src/OneNlp/Scoring/; the command-line tool that turns predictions into results rows is tools/OneNlp.Eval/.Published reference resultsNumbers for the accurate tier to be judged against, copied from the sources named in each row, nothing re-run. Every number is an F1 (or accuracy where the source says so) × 100.Model provenanceEvery trained model, ours or a baseline we train for comparison, ships with a provenance log. The log is a Markdown file named PROVENANCE.md next to the recipe that produced the model; it is packed into the model package's notices. It answers four questions: what data went…
Inside the library
Treebank licencesThe first sections cover ZH, FR, ES, PT, DE, IT, TR, HI and NL in detail. The last section ("Every treebank of release 2.18", generated by tools/fetch/ud_docs.py) covers all 353 treebanks in 193 languages, with the summary counts, the per-language classification, the quotes…Third-party noticesOneNLP's own code is licensed MIT OR Apache-2.0, Copyright (c) Recursive AI, LLC.Release weights and packagesWeights never enter git. Each weights.manifest.props records file sizes and SHA-256 digests; these committed values are the trust anchor, not the downloaded archive checksums. Runtime code never downloads anything.ChangelogAll notable changes to this project are documented in this file.