Recursive AI

Classify overlapping text spans in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

Span categorization labels phrases that can overlap or nest inside each other. This example learns labels for “New York” and “York”.

Terminal
dotnet add package RecursiveAI.OneNlp.Neural

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Data;
using RecursiveAI.OneNlp.Neural.Components;
using RecursiveAI.OneNlp.Neural.Encoders;
 
using var nlp = Pipeline.Blank(Language.English)
    .Add(new SpanCategorizer(FastEncoder.English(7),
        "sc", sizes: [1, 2], hiddenSize: 64)
        { OwnsEncoder = true });
var reference = nlp.Tokenizer.Tokenize(
    "New York", nlp.Strings);
var example = Example.FromReference(reference,
    spans: new Dictionary<string, IReadOnlyList<SpanSpec>>
    { ["sc"] = [new(0, 2, "CITY"), new(1, 2, "PART")] });
nlp.Train([example], new TrainOptions
{ Epochs = 50, DevFraction = 0, Seed = 7 });
foreach (var span in nlp.Process("New York").Spans["sc"])
    Console.WriteLine($"{span.Text}: {span.Label}");
Text
New York: CITY
York: PART

FastEncoder.English(7) creates an encoder with randomly initialized weights; 7 is its initialization seed, not a model version. TrainOptions.Seed separately controls training’s random choices.

The training ranges use token indexes, not character offsets: [0, 2) labels “New York” as CITY, and [1, 2) labels “York” as PART. Training and prediction use the same single invented input, so this demonstrates overlapping output rather than generalization.

Adding unknown labels to an initialized span head throws ArgumentException; train a new head to change its label set.

Unzip the example, then dotnet run.

Variations

Raise the confidence threshold. Run this after training in the full program.

C#
var head = nlp.Components.OfType<SpanCategorizer>().Single();
head.Threshold = 0.95f;
foreach (var span in nlp.Process("New York").Spans["sc"])
    Console.WriteLine($"{span.Text}: {span.Label}");
Text
New York: CITY
York: PART

Both predictions still reach the higher threshold, so this input’s labels do not change.

The short answer

Use OneNLP’s SpanCategorizer to classify overlapping text spans in C#. Train it with a named group of labeled token ranges, then read predictions from Doc.Spans under that same group name. In the example, the group is sc: “New York” receives CITY, while the nested span “York” receives PART. Both predictions can coexist.

This trains a new model on one invented example and predicts that same example. The output demonstrates the shape of overlapping results, not accuracy on unseen place names. The classification guide describes the categorizer’s proposal and classification stages.

What is a text span in NLP?

Here, a span is a contiguous range of tokens in a text document, not C#’s memory-oriented Span<T>. A span can have a label, text, character boundaries and confidence. “New York” covers two tokens; “York” covers only the second token. Because those ranges share a token, they overlap. The shorter range is also entirely inside the longer one, making it nested.

The training example uses SpanSpec(0, 2, "CITY") and SpanSpec(1, 2, "PART"). These numbers are token indexes, with the end exclusive. They are not character indexes. Example.FromReference takes gold span groups over the reference document’s tokens, so inspect that tokenization before assigning ranges. The JSON Lines training format instead accepts character offsets; see data formats.

Use the same group name in training and when reading predictions. sc is just this program’s chosen key, not a special built-in category. Labels such as CITY and PART are learned from examples; the blank pipeline has no pretrained knowledge of those labels.

How do I train a categorizer for nested text spans?

The full program is a minimal nested-span example. The categorizer first suggests candidate token ranges, then predicts labels for each candidate. sizes: [1, 2] proposes spans of one or two tokens, enough for both annotations. If your task needs longer phrases, configure sizes that can propose them. A range that is never proposed cannot be returned by the classifier, regardless of its training annotations.

For each candidate, the head combines its first token vector, last token vector and mean token vector, then applies a hidden layer and independent sigmoid scores. This allows more than one label to pass for a range. It differs from selecting a single highest-scoring label for an entire document.

A label is kept when its probability reaches the threshold. The documented default is 0.8; the variation raises it to 0.95. Raising the threshold can remove predictions, but does not establish their correctness. Choose a threshold using separate development examples, including overlapping ranges and negatives resembling your real inputs. Use DevScores.SpanGroup("sc") when scoring this group during training.

The span categorizer has a fixed label set after initialization. Training an existing head on previously unseen labels throws ArgumentException rather than growing that set. To change the set, train a new head. By contrast, an EntityRecognizer can extend labels, but produces non-overlapping entities. Choose the head by the structure your application must return, not just the labels’ names.

All onenlp recipes