Contracts

Behaviour that every component and tokenizer family relies on. Language-neutral and normative.

Offsets

• Every offset a Doc, Token or Span reports is a UTF-16 code unit offset into the string the caller passed in (Doc.Text). Nothing is reported against a normalized copy.
• A token has Idx (first character) and EndIdx (just past the last). Spans cover whole tokens; StartChar is the first token's Idx and EndChar the last token's EndIdx. An empty span has StartChar == EndChar.
• Token.Text and Span.Text are strings cut from Doc.Text on each call, so token.Text == "x" compares contents. TextSpan is the same text as a ReadOnlySpan<char> over Doc.Text and copies nothing; the library's own loops use it. A span's text includes the whitespace between its tokens.
• Doc.TryGetCharSpan makes a span from character offsets only when both fall on token boundaries. Nothing is rounded.
• Tokenizer encodings (TokenOffset) count UTF-16 units for text passed as char, and bytes for text passed as UTF-8. Tokens that stand for no input text (added special tokens, padding) have the offsets (0, 0).

Doc

• Token boundaries are fixed when the document is created. They are non-empty, ordered and non-overlapping; what lies between tokens is the caller's whitespace.
• Annotations are columns indexed by token: strings as IDs in a shared StringStore (ID 0 is the empty string, meaning "not set"), heads as token indexes (a token that is its own head is a root), sentence boundaries as Unset/Start/Inside, entities as an ordered list of non-overlapping spans with a per-token IOB and type, and a confidence per token.
• A new document is unannotated: empty strings, every token its own head, boundaries unset, EntIob.Unset, confidence 1.
• A document with no sentence boundaries set is one sentence. After SetEntities (even with an empty list) every token has an EntIob.
• Token.Confidence is the confidence of the token's tag annotation. Entity and span confidences live on the span. Rule-based results have confidence 1.
• Reads may run concurrently; writes need exclusive use. The StringStore is thread-safe and only grows.

Pipeline and components

• A pipeline is a word tokenizer followed by components in the order they were added. Component names are unique within a pipeline.
• A component only fills annotations; it never changes token boundaries or the text.
• Pipeline.Process may be called from several threads when the components allow it: per-call state stays in locals, shared state is read-only.
• ProcessMany reads its source lazily, tokenizes one batch, runs each component over the whole batch (IPipelineComponent.ProcessBatch, which defaults to calling Process per document), then yields the batch in input order. ProcessManyAsync is the same over an IAsyncEnumerable<string>; only reading the source awaits.
• Components are created by name from ComponentRegistry. Registration is by explicit call (a package's module initializer or a public Register method), never by scanning assemblies. An unknown name throws and lists the registered names.

Tokenizers

• Tokenizer is the one type callers hold; the families (WordPiece, BPE, byte-level BPE, Unigram, Tiktoken) derive from it and implement EncodeCore, Decode, IdToToken, TokenToId and VocabularySize.
• Encode returns a TokenEncoding: ids, token strings, offsets, special-token mask, type ids, attention mask and word ids, all one entry per token. Truncation is the family's job (it must count special tokens against the limit); padding is applied by the base class afterwards.
• Truncation and Padding apply to Encode and EncodeBatch. EncodeToIds, CountTokens and TrimToTokenBudget ignore them.
• EncodeToIds writes IDs into the caller's buffer and returns Done or DestinationTooSmall. After DestinationTooSmall the written IDs are an exact prefix of the full encoding, and consumed is the end offset of the last written token; encoding the remainder separately is not guaranteed to continue the same sequence.
• TrimToTokenBudget returns the slice of the text covered by the first (or last) n tokens of the full encoding without special tokens.
• Invalid UTF-8 is read as U+FFFD, one per invalid sequence, and offsets still count the original bytes.
• Tokenizer.FromFile picks the format from the extension (.json, .model, .txt, .tiktoken); FromStream takes the format explicitly. A format without a loader throws NotSupportedException.

Tokenizer.json pipeline (internal seam)

Tokenizer.FromFile/FromStream(HuggingFaceJson) returns a HuggingFaceTokenizer, composed as the published tokenizer.json format describes: normalizer → pre-tokenizer → model → post-processor, with a decoder for decoding, added tokens beside the model vocabulary, and the file's truncation and padding copied to Tokenizer.Truncation and Padding. All of it lives in src/OneNlp/Tokenizers/Hf/ and is internal.

• Alignment. NormalizedString holds the current text and, for each UTF-16 unit, the range of the original input it came from. Normalizers rewrite it with NormalizedStringBuilder (inserted or replaced units share the origin of the unit they stand for). PreTokenizedString is a list of such pieces; a piece's index is the word index of its tokens. A range of a piece maps back to the original input from the origin start of its first unit to the origin end of its last.

• Stages. INormalizer.Normalize(NormalizedString), IPreTokenizer.PreTokenize(PreTokenizedString), IPostProcessor.AddedTokenCount(isPair) and Process(encoding, pair, addSpecialTokens), ITokenDecoder.DecodeChain(List<string>). Stages are immutable and thread-safe.

• Model seam. ITokenizerModel.Tokenize(ReadOnlySpan<char> piece, List<ModelToken> output) appends (Id, Value, Start, End) with offsets relative to the piece, plus TokenToId, IdToToken and VocabularySize (model entries only). The piece is the text after normalization and pre-tokenization (for byte-level BPE, the byte-mapped characters). Byte fallback, ignore_merges, unknown-token fusing and rank lookups are the model's business.

• Registry. StageRegistry maps a type string to a factory per kind. A model or stage is added from its own file by implementing the matching optional partial method of StageRegistry and calling AddModel/AddNormalizer/AddPreTokenizer/AddPostProcessor/AddDecoder: RegisterWordPiece, RegisterUnigram, RegisterBpe, RegisterTiktoken, RegisterNormalizers, RegisterPreTokenizers, RegisterPostProcessors, RegisterDecoders. Registering an existing type replaces it. Every documented type is registered first as a stub that throws NotSupportedException ("... is not implemented yet"); an undocumented type throws "... is not supported". A model object without type (older files leave it out) is typed from its fields, as the reference does: merges present means BPE, otherwise a vocab list of pairs means Unigram and a vocab object means WordPiece; with no vocab it is read as BPE.

• JSON access. Factories read their object through StageConfig (typed accessors that throw InvalidDataException naming the entry and field; Element for bulk reads such as a vocabulary). The loader reads at most 512 MiB and parses with a nesting limit of 32.

• Truncation and pair encoding in the composed tokenizer: the room reserved for the template's special tokens is IPostProcessor.AddedTokenCount, counted only when special tokens are added; pairs are truncated longest-first.

• Stages implemented. Normalizers: BertNormalizer, NFC, NFD, NFKC, NFKD, Lowercase, Strip, StripAccents, Replace, Prepend, Precompiled (SentencePiece character map), ByteLevel, Sequence. Pre-tokenizers: BertPreTokenizer, Whitespace, WhitespaceSplit, Punctuation, Metaspace, ByteLevel, Split (regular expression or literal), Digits, UnicodeScripts, CharDelimiterSplit, Sequence. Post-processors: TemplateProcessing, BertProcessing, RobertaProcessing, ByteLevel, Sequence. Decoders: WordPiece, Metaspace, ByteLevel, ByteFallback, Fuse, Strip, Replace, BPEDecoder, CTC, Sequence.

• Regular expressions. Split patterns run on a small engine over code points (CodePointRegex): classes, Unicode properties (general category and script), lookahead, lazy and possessive quantifiers, case-insensitive groups. A match is cut off after a step budget (2,000,000 plus 500 per input character) and a stack-depth limit; exceeding either throws InvalidOperationException, and a pattern that does not parse throws InvalidDataException. Tiktoken encodings keep their [GeneratedRegex] split patterns, which are faster; they give the same pieces on the parity corpus.

• Normalization data. NFC/NFD/NFKC/NFKD use tables generated from the Unicode Character Database by tools/gen_normalization_tables.py. The reference library behaves like an older Unicode version, so the tables keep decompositions and combining classes only for characters assigned up to Unicode 9.0 (read from DerivedAge.txt). The sets that BertNormalizer removes (control and format characters) and that accent stripping removes (nonspacing marks up to Unicode 8.0) are generated from a sweep of every scalar value against the reference (tools/gen_bert_clean_table.py); characters unassigned at that time are kept.

• Alignment rules observed in the reference (each probed through its public NormalizedString API and recorded in comments): the pieces of a decomposed character keep that character's origin; canonical reordering moves characters but not origins; when composition shortens a run, the first character keeps the first origin and the rest line up with the end of the run; a run is a starter with the marks and the second members of composable pairs that follow it. The SentencePiece character map is applied per grapheme cluster (looked up as a whole when it is under 6 UTF-8 bytes, otherwise per code point); the k-th character of a replacement covers the k-th code point it replaces, and characters it deletes at the very start of the text pull all later origins back by that many code points.

• Lone surrogates in a string input are replaced by U+FFFD (one unit each) before normalization, so offsets still count the original units.

• Loader errors. Invalid UTF-8 in the file, a missing or mistyped member of a stage, or a value out of range raises InvalidDataException naming the stage; nothing else escapes. The files are never executed or fetched.

• Checks that need the reference. tools/stage_goldens/make_real_goldens.py and make_sweep.py write reference answers for real tokenizer files and for every scalar value; RealTokenizerGoldenTests and NormalizerSweepTests read them from the locations named by environment variables and skip when those are unset. The full corpus comparison is tools/OneNlp.Parity (docs/parity.md).

Encoders, heads and training (RecursiveAI.OneNlp.Neural)

• Encoder seam. IEncoder.Encode(ReadOnlySpan<Doc>) returns an EncodedBatch: one float32 row per token, the documents' rows stacked in order (DocCount, Start(i), Length(i), Rows(i)). One call serves both modes: with recording on and trainable Weights the matrix carries the autodiff graph into the encoder; inside GradMode.NoGrad() (inference, a frozen encoder) it is plain values. Weights (an NnModule, or null for an encoder with nothing to train or save, such as an imported ONNX model) fixes the state order for checkpoints. Kind and WriteSettings name the encoder in a saved pipeline (EncoderRegistry). Dispose is idempotent; heads that share an encoder may each dispose it.
• Head contract. ITrainableComponent : IPipelineComponent has Encoder, Initialize(examples, seed) (fixes the label set from the training examples' reference documents, in an order that depends only on their content, and builds the weights), Loss(EncodedBatch, examples) (one rank-0 tensor; the heads' losses are added and backpropagated once), Predict(EncodedBatch, docs) (writes into the documents), Weights, Kind and WriteConfig (labels and settings; rebuilt by HeadRegistry). TrainableHead implements the pipeline side: it skips empty documents, runs the encoder and Predict without gradients in evaluation mode, and serializes calls on one head. A head sees whole batches, so a softmax tagger, a transition or biaffine parser, span or transition NER and a pooled classifier all fit.
• Example (RecursiveAI.OneNlp.Data) pairs a predicted Doc with a reference Doc, plus gold Cats and Spans. Alignment maps tokens when the tokenizations differ (rules in docs/data-formats.md); a loss uses GetAligned(column), where null means "no gold label".
• Train. pipeline.Train(examples, TrainOptions, CancellationToken) trains every trainable head and the encoders they read with AdamW. Batches are made by token count from a shuffle that depends only on the seed and the epoch. Dropout is applied to the encoder output. The dev set is DevExamples or a deterministic split of DevFraction; the dev score is Score (a delegate over the dev examples with their predicted documents annotated; DevScores.Ud(metric) and DevScores.Entities() wrap the scorers) or else the negative mean dev loss. Patience ends the run after that many evaluations without a new best; the best weights are put back. With CheckpointDirectory, last.ckpt (weights, optimizer, dropout counter, position, best score, patience) and best.ckpt are written atomically, and a later call resumes from last.ckpt to exactly the weights an uninterrupted run reaches (same machine and thread count). Progress goes to IProgress<TrainProgress>; the library never writes to the console.
• Sharing one encoder between heads. Heads that hold the same IEncoder instance share its work.
  • Training. Train encodes each batch once per distinct encoder instance, adds the heads' losses and runs Backward once, so the gradients of all heads sum into the encoder (this is how the tagger, morphologizer and lemmatizer train together). A head must therefore read only EncodedBatch and never call the encoder itself.
  • Inference. Each head's ProcessBatch calls Encoder.Encode. An encoder may keep its output per document so that the second and later heads find it: FastEncoder stores each document's rows in a ConditionalWeakTable keyed by the Doc, used only when recording is off and the weights are in evaluation mode, and gives the same values as encoding the document alone or in any batch (nothing crosses a document edge). The cache assumes the weights do not change while a document is alive, and ends with the document. Encoders without a cache are correct, only slower.
  • A new head kind joins a pipeline by HeadRegistry.Register (or an entry in the built-in table) and reads encoded.Vectors/Rows(i); it needs nothing else from the encoder.
• Fast encoder (FastEncoder, kind fast; settings width, depth, normRows, prefixRows, suffixRows, shapeRows, seed, all recorded in the saved pipeline). Hash embeddings (two hashes per token, summed) of the lower-cased text, the first character, the last three characters and the shape, mixed by a linear layer with GELU and layer norm, then depth residual window layers over the previous, current and next vector. Document edges are zero padded and nothing crosses documents. The hash is FNV-1a over UTF-16 units, seeded per attribute and finished with SplitMix64 (low half for hash 0, high half for hash 1, each modulo the table's rows); the shape maps upper case to X, lower case to x, digits to d, and cuts runs of the same character after 4. A model rebuilt from its file hashes every token to the same rows on every platform.
• Fast-tier heads. Tagger (one string column), Morphologizer (kind morphologizer: joint labels POS=<upos>|<features>, fills Pos and Morph) and EditTreeLemmatizer (kind lemmatizer; classes are an edit tree, Müller et al. 2015, plus a flag saying the form is lower-cased first, serialized as l|r followed by M<prefix>,<suffix>(<left>)(<right>) or S<n>:<original><m>:<replacement>; a token takes its most probable class among the best 16 whose tree fits its text, else its own text). The tagger and the morphologizer write the token confidence (the later component wins); the lemmatizer leaves it alone.
• Saved pipelines are one file (docs/model-files.md).
• Readers (RecursiveAI.OneNlp.Data): ConllUReader/ConllUWriter and JsonlReader (docs/data-formats.md). UdCorpus.Read in Scoring reads through ConllUReader.