Load WordPiece and SentencePiece tokenizers in C#

Load a tokenizer vocabulary to turn text into model-specific token IDs and decode them again. SentencePiece .model loading supports Unigram models, not BPE, word or character models.

dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

using RecursiveAI.OneNlp.Tokenizers;
 
// Both files are in the example zip.
foreach (var file in
    new[] { "vocab.txt", "tiny.model" })
{
    var tokenizer = Tokenizer.FromFile(file);
    var ids = tokenizer.EncodeToIds(
        "hello worlds");
    Console.WriteLine(
        tokenizer.Decode(ids));
}

hello worlds
hello worlds

Unzip the example, then dotnet run.

Full program

The zip has two invented fixtures, not production model vocabularies: vocab.txt below and tiny.model. The Unigram model contains <unk> (unknown), <s> and </s> (control), and ordinary pieces ▁hello, ▁world, s with scores −1, −1, −2. It leaves letters unchanged and represents spaces with ▁, including an added space marker before the first word.

[PAD]
[UNK]
[CLS]
[SEP]
[MASK]
hello
world
##s

using RecursiveAI.OneNlp.Tokenizers;
 
foreach (var path in new[] { "vocab.txt", "tiny.model" })
{
    var tokenizer = Tokenizer.FromFile(path);
    var encoded = tokenizer.Encode("hello worlds");
    Console.WriteLine(path);
    for (int i = 0; i < encoded.Ids.Length; i++)
        Console.WriteLine($"{encoded.Ids[i]} {encoded.Tokens[i]} [{encoded.Offsets[i].Start},{encoded.Offsets[i].End}) special={encoded.SpecialTokensMask[i]}");
    Console.WriteLine(tokenizer.Decode(encoded.Ids));
}

dotnet run prints:

vocab.txt
2 [CLS] [0,0) special=True
5 hello [0,5) special=False
6 world [6,11) special=False
7 ##s [11,12) special=False
3 [SEP] [0,0) special=True
hello worlds
tiny.model
3 ▁hello [0,5) special=False
4 ▁world [6,11) special=False
5 s [11,12) special=False
hello worlds

An unsupported SentencePiece model kind throws NotSupportedException; malformed model data is rejected rather than guessed.

Variations

Choose casing and omit special tokens. This uses the supplied vocabulary with explicit lowercasing rather than letting FromFile infer casing from the vocabulary.

using RecursiveAI.OneNlp.Tokenizers;
 
var tokenizer = Tokenizer.FromWordPieceVocabulary("vocab.txt",
    new WordPieceVocabularyOptions { LowerCase = true });
Console.WriteLine(string.Join(",", tokenizer.EncodeToIds(
    "HELLO worlds", addSpecialTokens: false)));

5,6,7

Related recipes

• Count LLM tokens
• Tokenize text

The short answer

Use Tokenizer.FromFile for a BERT vocab.txt or a supported Unigram SentencePiece .model. Call Encode for IDs, vocabulary pieces and offsets, or EncodeToIds when you only need IDs. Decode using that same tokenizer: IDs are positions in a particular vocabulary, not universal word numbers.

How do I load a WordPiece tokenizer in C#?

The example vocabulary gives hello ID 5, world ID 6 and the continuation piece ##s ID 7. WordPiece splits worlds into world and ##s; the decoder joins them again. The full program prints both these pieces and their original UTF-16 ranges, making it possible to align model input with the source text.

Keep the vocabulary order exactly as distributed with your model. Removing a line, sorting the file or replacing it with a similar vocabulary changes the IDs. The invented fixture demonstrates the API; it is not a vocabulary suitable for an existing transformer’s embedding table.

Before looking up pieces, the reader cleans the text, spaces out Chinese characters and splits text into word and punctuation groups. For an uncased vocabulary, it also lowercases text and removes accents. A plain vocabulary does not state whether its model lowercases text. FromFile infers casing from the vocabulary, ignoring bracketed special tokens. When you know the model settings, use FromWordPieceVocabulary with WordPieceVocabularyOptions, as in the variation, or load the model’s tokenizer configuration instead. StripAccents can be set explicitly; when unspecified, it follows LowerCase. See the reader contract.

Why do CLS and SEP have empty offsets?

The vocabulary contains [CLS] and [SEP], so the default encoding adds them around the input. Their ranges are [0,0) because they represent no characters from that input. The special-token mask identifies them. Decoding skips special tokens by default, which is why the output is hello worlds rather than the vocabulary spellings of the surrounding markers.

The variation passes addSpecialTokens: false and returns only 5,6,7. Use the format your downstream model expects; removing special tokens is not a general way to make an encoding more correct. Offsets refer to UTF-16 when you pass a .NET string. Do not interpret these positions as UTF-8 byte locations.

How do I use a SentencePiece tokenizer in C#?

The second file uses the same entry point but follows its own model pipeline. The visible ▁ character marks spaces in the vocabulary pieces. It is not literally part of hello worlds. Decoding replaces those markers with normal spaces, and no end-of-sequence token is automatically appended by this .model reader.

OneNLP’s released reader accepts Unigram models. It rejects SentencePiece BPE, word and character model kinds; a .model suffix alone does not establish support. It also refuses models that preserve space runs or do not escape spaces. Check the model kind before choosing this entry point rather than assuming all SentencePiece files are interchangeable.

Unigram chooses a highest-scoring path through candidate vocabulary pieces. Unknown text and byte fallback depend on the vocabulary and model settings. The tiny fixture intentionally covers only its demonstration phrase, so successful round-tripping here says nothing about vocabulary coverage for other languages or inputs. For a real transformer, keep the tokenizer files and their normalization settings paired with the exact model that produced them.