---
title: "Load WordPiece and SentencePiece tokenizers in C#"
url: https://recursiveai.org/onenlp/cookbook/load-transformer-tokenizer-files/
description: "Load a tokenizer vocabulary to turn text into model-specific token IDs and decode them again."
version: "1.0.0"
---

# Load WordPiece and SentencePiece tokenizers in C#

**OneNLP** is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneNlp) · [GitHub](https://github.com/recursive-ai-oss/onenlp-csharp) · [Docs](https://recursiveai.org/onenlp/docs/) · [Download example (.zip)](https://recursiveai.org/onenlp/cookbook/load-transformer-tokenizer-files/load-transformer-tokenizer-files.zip)

Load a tokenizer vocabulary to turn text into model-specific token IDs and decode them again. SentencePiece `.model` loading supports Unigram models, not BPE, word or character models.

```shell
dotnet add package RecursiveAI.OneNlp
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneNlp.Tokenizers;

// Both files are in the example zip.
foreach (var file in
    new[] { "vocab.txt", "tiny.model" })
{
    var tokenizer = Tokenizer.FromFile(file);
    var ids = tokenizer.EncodeToIds(
        "hello worlds");
    Console.WriteLine(
        tokenizer.Decode(ids));
}
```

```text
hello worlds
hello worlds
```

Unzip [the example](https://recursiveai.org/onenlp/cookbook/load-transformer-tokenizer-files/load-transformer-tokenizer-files.zip), then `dotnet run`.

## Full program

The zip has two invented fixtures, not production model vocabularies: `vocab.txt` below and `tiny.model`. The Unigram model contains `<unk>` (unknown), `<s>` and `</s>` (control), and ordinary pieces `▁hello`, `▁world`, `s` with scores −1, −1, −2. It leaves letters unchanged and represents spaces with `▁`, including an added space marker before the first word.

```text
[PAD]
[UNK]
[CLS]
[SEP]
[MASK]
hello
world
##s
```

```csharp
using RecursiveAI.OneNlp.Tokenizers;

foreach (var path in new[] { "vocab.txt", "tiny.model" })
{
    var tokenizer = Tokenizer.FromFile(path);
    var encoded = tokenizer.Encode("hello worlds");
    Console.WriteLine(path);
    for (int i = 0; i < encoded.Ids.Length; i++)
        Console.WriteLine($"{encoded.Ids[i]} {encoded.Tokens[i]} [{encoded.Offsets[i].Start},{encoded.Offsets[i].End}) special={encoded.SpecialTokensMask[i]}");
    Console.WriteLine(tokenizer.Decode(encoded.Ids));
}
```

`dotnet run` prints:

```text
vocab.txt
2 [CLS] [0,0) special=True
5 hello [0,5) special=False
6 world [6,11) special=False
7 ##s [11,12) special=False
3 [SEP] [0,0) special=True
hello worlds
tiny.model
3 ▁hello [0,5) special=False
4 ▁world [6,11) special=False
5 s [11,12) special=False
hello worlds
```

An unsupported SentencePiece model kind throws `NotSupportedException`; malformed model data is rejected rather than guessed.

## Variations

**Choose casing and omit special tokens.** This uses the supplied vocabulary with explicit lowercasing rather than letting `FromFile` infer casing from the vocabulary.

```csharp
using RecursiveAI.OneNlp.Tokenizers;

var tokenizer = Tokenizer.FromWordPieceVocabulary("vocab.txt",
    new WordPieceVocabularyOptions { LowerCase = true });
Console.WriteLine(string.Join(",", tokenizer.EncodeToIds(
    "HELLO worlds", addSpecialTokens: false)));
```

```text
5,6,7
```

## Related recipes

- [Count LLM tokens](https://recursiveai.org/onenlp/cookbook/count-llm-tokens/)
- [Tokenize text](https://recursiveai.org/onenlp/cookbook/tokenize-text/)

## The short answer

Use `Tokenizer.FromFile` for a BERT `vocab.txt` or a supported Unigram SentencePiece `.model`. Call `Encode` for IDs, vocabulary pieces and offsets, or `EncodeToIds` when you only need IDs. Decode using that same tokenizer: IDs are positions in a particular vocabulary, not universal word numbers.

## How do I load a WordPiece tokenizer in C#?

The example vocabulary gives `hello` ID 5, `world` ID 6 and the continuation piece `##s` ID 7. WordPiece splits `worlds` into `world` and `##s`; the decoder joins them again. The full program prints both these pieces and their original UTF-16 ranges, making it possible to align model input with the source text.

Keep the vocabulary order exactly as distributed with your model. Removing a line, sorting the file or replacing it with a similar vocabulary changes the IDs. The invented fixture demonstrates the API; it is not a vocabulary suitable for an existing transformer’s embedding table.

Before looking up pieces, the reader cleans the text, spaces out Chinese characters and splits text into word and punctuation groups. For an uncased vocabulary, it also lowercases text and removes accents. A plain vocabulary does not state whether its model lowercases text. `FromFile` infers casing from the vocabulary, ignoring bracketed special tokens. When you know the model settings, use `FromWordPieceVocabulary` with `WordPieceVocabularyOptions`, as in the variation, or load the model’s tokenizer configuration instead. `StripAccents` can be set explicitly; when unspecified, it follows `LowerCase`. See the [reader contract](https://recursiveai.org/onenlp/docs/tokenizers-wordpiece-unigram/).

### Why do CLS and SEP have empty offsets?

The vocabulary contains `[CLS]` and `[SEP]`, so the default encoding adds them around the input. Their ranges are `[0,0)` because they represent no characters from that input. The special-token mask identifies them. Decoding skips special tokens by default, which is why the output is `hello worlds` rather than the vocabulary spellings of the surrounding markers.

The variation passes `addSpecialTokens: false` and returns only `5,6,7`. Use the format your downstream model expects; removing special tokens is not a general way to make an encoding more correct. Offsets refer to UTF-16 when you pass a .NET string. Do not interpret these positions as UTF-8 byte locations.

## How do I use a SentencePiece tokenizer in C#?

The second file uses the same entry point but follows its own model pipeline. The visible `▁` character marks spaces in the vocabulary pieces. It is not literally part of `hello worlds`. Decoding replaces those markers with normal spaces, and no end-of-sequence token is automatically appended by this `.model` reader.

OneNLP’s released reader accepts Unigram models. It rejects SentencePiece BPE, word and character model kinds; a `.model` suffix alone does not establish support. It also refuses models that preserve space runs or do not escape spaces. Check the model kind before choosing this entry point rather than assuming all SentencePiece files are interchangeable.

Unigram chooses a highest-scoring path through candidate vocabulary pieces. Unknown text and byte fallback depend on the vocabulary and model settings. The tiny fixture intentionally covers only its demonstration phrase, so successful round-tripping here says nothing about vocabulary coverage for other languages or inputs. For a real transformer, keep the tokenizer files and their normalization settings paired with the exact model that produced them.