Recursive AI

Create an NLP document from token offsets in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Pass TokenBoundary values to new Doc to keep the original text without retokenizing it.

Terminal
dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneNlp;
 
var doc = new Doc(new StringStore(),
    "red/blue", [new(0, 3), new(4, 4)]);
foreach (var token in doc)
    Console.WriteLine($"{token.Idx}.." +
        $"{token.EndIdx}: {token.Text}");
Text
0..3: red
4..8: blue

Unzip the example, then dotnet run.

new(0, 3) means start at 0 and take 3 UTF-16 code units (C# char values): red.

Full program

C#
using RecursiveAI.OneNlp;
 
const string text = "red/blue";
var strings = new StringStore();
var doc = new Doc(strings, text,
    [new(0, 3), new(4, 4)]);
foreach (var token in doc)
    Console.WriteLine($"{token.Idx}..{token.EndIdx}: {token.Text}");
Console.WriteLine($"Original: {doc.Text}");
 
try
{
    _ = new Doc(strings, text,
        [new(0, 3), new(2, 4)]);
}
catch (ArgumentException)
{
    Console.WriteLine("Overlapping offsets rejected.");
}

dotnet run prints:

Text
0..3: red
4..8: blue
Original: red/blue
Overlapping offsets rejected.

The constructor throws ArgumentException for an empty, out-of-range, out-of-order, or overlapping boundary.

Variations

Include the separator as a token. Replace the first Doc constructor in the full program:

C#
var doc = new Doc(strings, text,
    [new(0, 3), new(3, 1), new(4, 4)]);

The token loop now prints:

Text
0..3: red
3..4: /
4..8: blue

The short answer

Pass a StringStore, the original string, and a sequence of TokenBoundary values to the Doc constructor. Each boundary takes a start and a length, in UTF-16 code units. Use offsets in text order; each token must start at or after the previous token’s end. The constructor uses those boundaries without tokenizing the string again.

What do UTF-16 offsets mean in a C# string?

A UTF-16 offset counts .NET char values from the start of the string, not bytes or displayed characters. For example, in 🐂 red the emoji takes two char values, the space takes one, and red starts at offset 3. See .NET’s character encoding guide for why some characters take two values.

If your tokenizer exports UTF-8 byte offsets or Unicode code point offsets, convert them to UTF-16 before constructing the document. Use the exact string the positions refer to. If the upstream source gives an exclusive end rather than a length, the length is the end minus the start. In red/blue, blue covers [4, 8), so its constructor arguments are 4, 4, not 4, 8.

Do token offsets have to cover whitespace and punctuation?

No. A token covers only the range you supply. The two-token example leaves the slash unassigned, but doc.Text still contains red/blue. The variation makes that same slash a token without changing the original text. Adjacent tokens can touch, but they can’t share characters: a range starting at 2 overlaps the preceding red range, which ends at 3. This distinction matters when importing offsets from a format that allows nested or overlapping annotations: those ranges can’t all serve as the document’s token boundaries.

All onenlp recipes