Recursive AI

Tokenize text in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

Split prose into words and punctuation, not language-model token IDs.

Terminal
dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneNlp;
 
var tokens = Nlp.Tokens("I can't go.");
foreach (var token in tokens)
    Console.WriteLine(token.Text);

The snippet prints one token per line.

Text
I
ca
n't
go
.

The English contraction rule splits “can’t” into ca and n't; these are pieces of the original spelling, not a rewrite to “can not”.

Unzip the example, then dotnet run.

The full program

The download’s Program.cs uses the same visible input as the snippet at the top. No input files are needed.

The download’s Program.cs is the snippet above.

dotnet run prints:

Text
I
ca
n't
go
.

Null input raises ArgumentNullException; input beyond Nlp.MaxLength raises ArgumentOutOfRangeException. The default Nlp.MaxLength is 1,000,000 UTF-16 characters; the property can be changed.

Variations

Keep token offsets. Offsets refer to the unchanged original string. This is a standalone replacement for Program.cs.

C#
using RecursiveAI.OneNlp;
 
foreach (var token in Nlp.Tokens("Hello, world!"))
    Console.WriteLine($"{token.Idx}..{token.EndIdx}: {token.Text}");
Text
0..5: Hello
5..6: ,
7..12: world
12..13: !

The short answer

Call Nlp.Tokens(text) to get word tokens, including punctuation, with offsets into the original string. The core NuGet package is enough. The snippet at the top turns I can't go. into I, ca, n't, go and .. This is linguistic tokenization, not splitting a string at every space or encoding it as model-specific integer IDs.

How do I tokenize text in C#?

Pass a string directly to Nlp.Tokens and iterate the returned tokens. Each token exposes Text, Idx, Length and EndIdx. The variation above prints character ranges so a consumer can associate tokens with the input. Offsets are UTF-16 positions, matching .NET strings; they aren’t byte offsets into a UTF-8 file. The segmentation documentation states that nothing is normalized.

Whitespace-only segments are dropped, while punctuation remains. That makes the output useful when a later step needs both words and punctuation. Don’t count every token as a word without deciding how your application should treat punctuation, contractions and numbers. The five tokens in the example are an illustration of that distinction, not a five-word sentence.

Token objects retain access to their owning document. Use their offsets against the same original text. If your application lowercases, replaces quotes or collapses whitespace, do that before tokenizing or retokenize the transformed string. Positions from one version should not be reused against another.

The library starts from Unicode word boundaries and then applies language data. The high-level method detects the language and uses that language’s segmentation rules where available. English contractions split into grammatical pieces; listed abbreviations, URLs and email addresses have their own documented handling. These choices are useful for natural-language processing but may differ from a file format’s exact delimiter rules.

Is a C# tokenizer the same as a lexer?

Not for this task. The “C# tokenizer” search can mean either a tokenizer implemented in C# or a lexer that reads C# source code. This page does the former for human-language text. It does not parse C# syntax, construct an expression tree or validate a programming language.

For delimiter-based data, String.Split separates a string using the separators you supply. That is a different contract from language-aware word segmentation. Use explicit delimiters for a format that specifies them; use linguistic boundaries when the input is prose and you need words with offsets.

Subword encoding is different again. A model-specific tokenizer maps text to the vocabulary expected by that model, often breaking a single word into several pieces. OneNLP documents separate BPE and Tiktoken tokenizers. The Nlp.Tokens example does not supply those integer IDs. Decide which output your next stage expects before choosing a tokenization API.

All onenlp recipes