Training and recipes

Everything a user can train, the shipped models were trained with the same public Train API, on OneModel (automatic differentiation, AdamW, checkpoints), in C#. A recipe is a small console project over that API; running ours on your data is "retraining our model". Behaviour of Train, the heads and the encoders is written down in contracts.md; file formats in data-formats.md and model-files.md.

The call

The code below is samples/OneNlp.Samples/TrainSample.cs, which builds and runs.

string[] lines =
[
    "{\"text\": \"my card has not arrived\", \"cats\": {\"card\": 1, \"pin\": 0}}",
    "{\"text\": \"where is my new card\", \"cats\": {\"card\": 1, \"pin\": 0}}",
    "{\"text\": \"the card never came in the post\", \"cats\": {\"card\": 1, \"pin\": 0}}",
    "{\"text\": \"I forgot my pin\", \"cats\": {\"card\": 0, \"pin\": 1}}",
    "{\"text\": \"how do I change my pin\", \"cats\": {\"card\": 0, \"pin\": 1}}",
    "{\"text\": \"my pin is not working\", \"cats\": {\"card\": 0, \"pin\": 1}}",
];
using Pipeline nlp = Pipeline.Blank(Language.English)
    .Add(new TextClassifier(FastEncoder.English(7)) { OwnsEncoder = true });
List<Example> examples = [.. JsonlReader.Read(new StringReader(string.Join('\n', lines)), nlp)];
 
// A fixed seed makes the run repeatable; with CheckpointDirectory set, an interrupted run resumes from its last checkpoint.
TrainResult result = nlp.Train(examples, new TrainOptions { Epochs = 30, Seed = 7, Dropout = 0f, DevExamples = examples });
 
string path = Path.Combine(Path.GetTempPath(), "onenlp-sample-" + Guid.NewGuid().ToString("N") + ".onenlp");
PipelineFile.Save(nlp, path);                                  // one file: tokenizer, encoder, heads, labels
using Pipeline loaded = PipelineFile.Load(path);
Doc doc = loaded.Process("I lost my pin");
// doc.Cats maps each label to a probability

(The sample uses its training set as the development set so that it finishes in seconds; real runs keep them apart.)

• Adding labels. A head without weights is built from the labels of the training examples. A head that already has weights (a packaged pipeline) keeps them: an EntityRecognizer appends the labels the examples add, with fresh weights for the new outputs, so Pipeline.Create(Language.English).Add<EntityRecognizer>() followed by Train on examples with an ORDER label learns ORDER beside PER, ORG and LOC (include examples of the old labels to keep them). A Tagger, Morphologizer, EditTreeLemmatizer, Parser, SpanCategorizer or TextClassifier has a fixed label set and throws ArgumentException naming the values it does not have (for the lemmatizer, edit trees seen at least minClassCount times in the new data; rarer ones are left out as they are at the start of training). Their outputs are shared by structure (frequency-ordered classes, pairwise label scoring), so growing them would move trained weights; to change the label set, train a new head.
• Data. JsonlReader reads JSON Lines (text, optional tokens, entities, spans, cats); ConllUReader reads Universal Dependencies files. Example pairs a predicted Doc with a reference Doc; when the tokenizations differ it aligns them. For data already in memory, Example.Create(nlp, text, [(start, end, "LABEL")]) takes character offsets and tokenizes with the pipeline. A training program needs using RecursiveAI.OneNlp; (pipeline, Example, TrainOptions), .Neural (PipelineFile), .Neural.Components and .Neural.Encoders.
• Heads. Tagger, Morphologizer, EditTreeLemmatizer, Parser, EntityRecognizer, TextClassifier, SpanCategorizer, on FastEncoder or a transformer encoder; and, as trained word tokenizers, NeuralMultiwordSplitter (Turkish and similar) and NeuralWordSegmenter (Chinese). Heads that share one encoder instance share its work and add their losses.
• Options. TrainOptions has epochs, batch size in words, learning rate and schedule, warm-up, weight decay, clipping, dropout, seed, a frozen-encoder switch, the development set (DevExamples or a split), the score function (DevScores.Ud(...), DevScores.Entities(), DevScores.Accuracy()), evaluation interval, patience, CheckpointDirectory and a progress callback. The library never writes to the console.
• Repeatable. The batches depend only on the seed and the epoch. A resumed run reaches the same weights as an uninterrupted one (same machine and thread count). The best development checkpoint is restored at the end.
• Fine-tuning a transformer. TrainableTransformerEncoder loads exported ONNX weights (ModernBERT and BERT/XLM-R families) as trainable modules and fine-tunes them with the same call; the written-back weights run in the graph runtime. See accurate-encoder.md.

Recipes

recipes/<language>/... and recipes/en/... hold one console project per trained part. Each has a README.md with its commands, a PROVENANCE.md and the results it produced:

Conventions every recipe follows:

• Data is fetched by script and checked by hash. tools/fetch/ fetches treebanks and pins them; the manifest of each recipe lists every dataset with its role (train, dev or eval) and licence. A recipe reads only its train and dev files; the test files are read by predict and the scorer.
• Resumable. Every train writes checkpoints and continues when run again with the same arguments, so a limited job is chained until the model file appears. --budget-seconds stops cleanly.
• Provenance. PROVENANCE.md records the datasets (version, URL, hash, licence), the label generator if any, the recipe and seed, the evaluation sets and the output weights' hash.
• Scoring. predict writes CoNLL-U or entity files that tools/OneNlp.Eval scores with the CoNLL 2018 definitions and paired bootstrap intervals (measurement.md).

A model you train this way loads with PipelineFile.Load and plugs into Pipeline like ours. Models from elsewhere load too: any exported ONNX encoder through TransformerEncoder and TrainableTransformerEncoder, and any sentence-embedding model through SentenceEmbedder.Load. The licence rules for what we ship (permissive weights only) do not limit what you load.

Speed of training

The C# recipes were timed against the same recipes in plain PyTorch on CPU (same data, architecture, batch size and threads); the ratios are in tools/ratio/RESULTS.md and the step timing of a transformer fine-tune in accurate-encoder.md.