Saved pipelines

Normative for PipelineFile (RecursiveAI.OneNlp.Neural). One file holds a trained pipeline: its structure and label sets as JSON, and the weights of its encoders and heads in OneModel's checkpoint encoding.

Layout (little endian)

The file length is exactly 32 + C + W.

Config

{"format": "onenlp-pipeline", "version": 1, "language": "en", "tokenizer": {"kind": "uax", "data": "english"},
 "encoders": [{"kind": "hash-embedding", "settings": {"width": 64, "buckets": 16384, "seed": 1}}],
 "components": [{"name": "tagger", "type": "head", "kind": "tagger", "encoder": 0,
                 "config": {"column": "Tag", "labels": ["NN", "VB"]}},
                {"name": "sentencizer", "type": "builtin", "settings": {"data": "english"}}]}

• encoders are the distinct encoders the heads read, in order of first use. Each is rebuilt by EncoderRegistry.Create(kind, settings).
• components are in pipeline order. A head is rebuilt by HeadRegistry.Create(kind, name, encoder, config); a builtin is created by name from ComponentRegistry. A component that is neither a trainable head nor registered cannot be saved.
• Weights are named encoder<i>.… (encoders in order, those with weights) and head<j>.… (trainable heads in pipeline order). Loading checks every name, type and shape against the shapes the config implies.
• tokenizer is {"kind": "rule"}, {"kind": "uax", "data": "none"|"neutral"|"english"|...} (the segmentation data set the tokenizer uses) or {"kind": "segmenter", "encoder": i, "head": {...}}. Other tokenizers cannot be saved yet.
• segmenter is a trained word segmenter (NeuralWordSegmenter, for text written without spaces). encoder is the index of one of the file's encoders; head is the config of a tagger over the Tag column with the labels B and I, exactly as a tagger component's config. The segmenter's tagger and encoder are not in components: their weights are saved under the first names, encoder0/head0 when the segmenter's encoder is the first encoder, and the components' encoders and heads are numbered after them. A file with another tokenizer is numbered as before, so every file written before this kind existed loads unchanged; a reader that predates it rejects a segmenter file with "Unsupported tokenizer kind". The format version stays 1. A pipeline file that only carries a segmenter has an empty components list.
• mwt is a multiword-token splitter (NeuralMultiwordSplitter): {"kind": "mwt", "inner": {"kind": "uax", "data": "turkish-whole"}, "encoder": i, "head": {...}}. inner is the surface tokenizer, rule or uax (with its data); encoder and head are as for segmenter (a tagger over the Tag column with the labels B and I, one decision per character of a letters-only token of 3 to 48 units: B starts a UD word, so a B after the first character is a cut). Its tagger and encoder are numbered first, exactly as a segmenter's are, minCutPercent (50 to 99, default 50 and then absent) is the confidence a B needs to count as a cut. Every file of another tokenizer numbers its weights as before and loads unchanged; a reader that predates the kind rejects an mwt file with "Unsupported tokenizer kind". The format version stays 1.
• A builtin component may carry settings, an object of at most 64 short strings, written by components that implement IConfigurableComponent and passed to ComponentRegistry.Create on load (the sentencizer writes data).

Kinds

Encoder kinds (encoders[i].kind, with settings) and head kinds (components[j].kind, with config) that the loader knows. Weights of each are named under encoder<i>. and head<j>..

| head tagger | column (Tag, Pos or Morph), labels | output.weight [labels, width], output.bias |

| head morphologizer | labels: joint labels, each POS=<upos> or POS=<upos>\|<features>, at most 65,536 | output.weight, output.bias as the tagger |

| head lemmatizer | labels: classes, each l or r followed by a serialized edit tree, at most 262,144 | output.weight, output.bias as the tagger |

| head textcat | multi_label (true or false), hidden (0 to 4,096: width of the optional hidden layer, 0 for none), labels (the categories, at most 65,536, distinct) | with hidden > 0: hidden.weight [hidden, width], hidden.bias; then output.weight [labels, hidden or width], output.bias |

| head spancat | spans_key (the span group, at most 256 characters), sizes (1 to 32 distinct span lengths in tokens, each 1 to 256), threshold (above 0, at most 1), hidden (0 to 4,096), labels (at most 16,384, distinct) | with hidden > 0: hidden.weight [hidden, 3 * width], hidden.bias; then output.weight [labels, hidden or 3 * width], output.bias |

The textcat and spancat kinds were added after the first file version and change nothing else: a file without them loads as before, the format version stays 1, and a reader that predates them rejects a file that holds one with "No head kind ... is registered". A textcat head scores the mean of the document's token vectors; a spancat head scores [mean; first; last] of the token vectors of every n-gram of the listed sizes. Both read the label list in output order.

The fast encoder's optional settings are written only when they differ from the defaults, so a file made with the first-version settings is read and written unchanged. With one hash function only the low half of the 64-bit hash is used; with three or four, the further functions use a re-mixed hash (SplitMix64 of the hash plus the function number). With plain, the joined attributes go through a ReLU instead of a GELU and layer normalization. With turkicCase, the lower-cased form (the first attribute) maps I to ı and İ to i before hashing (Unicode SpecialCasing, Turkish and Azeri) instead of the invariant mapping; the other attributes are unchanged. The fast encoder's hash is part of the format: FNV-1a over UTF-16 units, seeded per attribute and finished with SplitMix64 (docs/contracts.md), so a saved model hashes every token to the same rows on every platform. A serialized edit tree is M<prefix>,<suffix>(<left>)(<right>) for a match node and S<n>:<original><m>:<replacement> for a substitution, where <n> and <m> are the lengths in UTF-16 units of the strings that follow them; at most 4,096 characters and 64 levels deep. A label list with a repeated entry, a morphologizer label not starting with POS=, a lemmatizer class not starting with l or r, or a tree that does not parse is an InvalidDataException.

Bounds on read

Config at most 16 MiB with JSON nesting at most 32; weights at most 1 GiB; at most 64 encoders and 1,024 components; label sets at most 1,048,576 entries of at most 4,096 characters; an embedding table at most 67,108,864 values and a head's output layer at most 67,108,864 weights. The declared lengths are compared with the real length of the data before anything is allocated. Any failure is an InvalidDataException.

Fine-tuned encoders. PipelineFile.Save(pipeline, path) writes each transformer-trainable encoder as an ONNX file <file name without extension>.encoder<N>.onnx beside the pipeline and records the encoder as transformer with model set to that plain file name and relativeTo set to pipeline (the name is looked up in the pipeline file's directory, not in ModelDirectory; it must not contain a path separator or ..). No encoder<N> weights are stored for it. A reader that does not know relativeTo ignores it. Loading such a pipeline from a stream that is not a file fails. Save(pipeline, path, exportEncoderGraphs: false) and saving to a stream keep the transformer-trainable layout. See accurate-encoder.md.

Referenced encoder files. A transformer encoder is a multi-hundred-megabyte ONNX file plus a tokenizer.json. A pipeline file stores their names (and optionally the model's SHA-256), not their bytes: the same encoder serves many pipelines, and a pipeline file stays a few kilobytes. A relative name is looked up in TransformerEncoder.ModelDirectory (the current directory when unset); an absolute name is used as is. Loading fails with the file's own FileNotFoundException when a file is missing, and VerifyModelFile checks the hash on request. The model packages put the encoder files next to the pipeline files they belong to.

Heads with a two-level output

The morphologizer and lemmatizer may have headClasses (0 to 4,096) in their config. With a value above 0, labels are stored in falling order of frequency and the output layer is two Linear layers, head (headClasses + 1 outputs, the last meaning "other") and tail (the remaining classes); with 0 or absent it is the single layer output. The parser's labelBilinear (0 when the label biaffine is left out, absent meaning 1) and the shared-encoder pipelines of recipes/en/joint use the same file layout otherwise.

Inference

At inference the fast encoder with relu computes from derived tables (each embedding table multiplied by its slice of mix.weight, each window layer as one product); they are rebuilt when a weight changes and are not stored. The encoder keeps, per thread, the result for the last documents it encoded, valid until that thread encodes other documents or any training-mode encoding happens, so the heads of a pipeline share one encoding.