Train custom named entity recognition in C#

Named entity recognition labels ranges of text, such as customer names. Train your own labels from annotated examples.

dotnet add package RecursiveAI.OneNlp.Neural

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Neural.Components;
using RecursiveAI.OneNlp.Neural.Encoders;
 
using var nlp = Pipeline.Blank(Language.English)
    .Add(new EntityRecognizer(
        FastEncoder.English(7))
        { OwnsEncoder = true });
nlp.Train([Example.Create(nlp,
    "Ada", [(0, 3, "CUSTOMER")])],
    new TrainOptions { Epochs = 30,
        DevFraction = 0, Seed = 7 });
foreach (var e in nlp.Process("Ada").Ents)
    Console.WriteLine($"{e.Text}: {e.Label}");

Ada: CUSTOMER

Unzip the example, then dotnet run.

Full program

The full program trains on “Ada ordered tea” and “Ben ordered coffee”, with characters 0–3 labeled CUSTOMER. It holds out “Cara ordered tea” with characters 0–4 labeled CUSTOMER for development. These invented inputs are small API examples, not an accuracy evaluation.

using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Neural.Components;
using RecursiveAI.OneNlp.Neural.Encoders;
 
using var nlp = Pipeline.Blank(Language.English)
    .Add(new EntityRecognizer(FastEncoder.English(7))
        { OwnsEncoder = true });
Example[] examples = [
    Example.Create(nlp, "Ada ordered tea",
        [(0, 3, "CUSTOMER")]),
    Example.Create(nlp, "Ben ordered coffee",
        [(0, 3, "CUSTOMER")])
];
nlp.Train(examples, new TrainOptions
{ Epochs = 30, Seed = 7, DevExamples = [
    Example.Create(nlp, "Cara ordered tea",
        [(0, 4, "CUSTOMER")])] });
foreach (var entity in nlp.Process("Ada ordered tea").Ents)
    Console.WriteLine($"{entity.Text}: {entity.Label}");
 
Console.WriteLine("Held-out prediction:");
foreach (var entity in nlp.Process("Cara ordered tea").Ents)
    Console.WriteLine($"{entity.Text}: {entity.Label}");

dotnet run prints:

Ada: CUSTOMER
Held-out prediction:
Cara: CUSTOMER

Training annotations with overlapping entities or boundaries inside a token throw ArgumentException.

Variations

Save and load the trained pipeline. Add this namespace at the top and the remaining lines after training in the full program.

using RecursiveAI.OneNlp.Neural;
PipelineFile.Save(nlp, "customers.onenlp");
using var loaded = PipelineFile.Load("customers.onenlp");
foreach (var entity in loaded.Process("Ada ordered tea").Ents)
    Console.WriteLine($"{entity.Text}: {entity.Label}");

Ada: CUSTOMER

Related recipes

• Named entity recognition
• Categorize overlapping spans

The short answer

To train custom named entity recognition in C#, create a blank OneNLP pipeline, attach an EntityRecognizer backed by an encoder, and call Train with labeled examples. Example.Create accepts the text and entity character ranges. After training, Process returns predicted entities in Doc.Ents. The example trains a new CUSTOMER label from invented sentences, without a pretrained model package.

The download includes two training sentences and a separate development sentence, “Cara ordered tea”, annotated as CUSTOMER over characters 0 through 4, end exclusive. Development examples let training choose a checkpoint without training on those examples. This is still a demonstration dataset, not evidence of accuracy on customer messages. The program prints predictions on both a training sentence and the development sentence. Because development data selects the checkpoint, this is not an untouched final evaluation.

How do I label data for custom named entity recognition?

For custom labels, decide exactly what each label means, then annotate complete text examples with that definition. Here, CUSTOMER means the person placing an order, not every person mentioned in a document. The training guide describes the public training API and development scoring options.

An annotation is a start offset, an exclusive end offset and a label. For “Ada ordered tea”, (0, 3, "CUSTOMER") selects “Ada”; it does not include the following space. Example.Create uses UTF-16 character positions, the indexing units of C# strings. Every range must begin and end on the pipeline tokenizer’s token boundaries. Reusing the pipeline for annotation and prediction avoids silently training with a different tokenization.

Entities must not overlap. If your labels require a phrase inside another labeled phrase, use a span categorizer instead. A token-tagging entity recognizer and a span categorizer solve different output problems; adding more training examples cannot make the former express nested output.

How do I evaluate and save a custom entity recognizer?

Keep training and development data separate. The full program passes DevExamples explicitly; the shorter snippet uses no development split solely to fit a tiny API demonstration. For a real dataset, use representative development examples and leave a final evaluation set untouched while choosing settings. DevScores.Entities() is the documented entity-scoring option. The library restores the best development checkpoint at the end of training.

The random value passed to FastEncoder.English(7) initializes the encoder. It is not a model version. TrainOptions.Seed separately controls training’s random choices. A blank pipeline learns from the annotations you supply; it does not acquire general English entity knowledge just because the language is English.

A new entity head builds its labels from the examples. An already trained EntityRecognizer can append labels, retaining its existing weights and initializing new outputs. When extending an existing recognizer, include examples of its old labels too, as the training guide recommends. Otherwise new-label training does not demonstrate that the old behavior was retained.

Save the trained pipeline with PipelineFile.Save, as in the variation. That file holds the tokenizer, encoder, heads and labels together. Loading the saved file is different from serializing an analyzed document: the pipeline contains learned behavior, while document JSON contains a particular input and its annotations.