Count tiktoken tokens in C#

Count tokens using a specific encoding, rather than counting words. This example uses cl100k_base, not a model-independent token estimate. It needs the cl100k_base.tiktoken ranks file, included in the download.

dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

using RecursiveAI.OneNlp.Tokenizers;
 
var tokenizer = TiktokenTokenizer.Cl100kBase(
    "cl100k_base.tiktoken");
var encoded = tokenizer.Encode("Hello, world!",
    addSpecialTokens: false);
Console.WriteLine(encoded.Ids.Length);

4

The input is “Hello, world!”. The download contains the cl100k_base.tiktoken ranks file from the encoding distribution, with the tiktoken MIT license. The file has one base64 token and decimal rank per line; its first three lines are shown below.

IQ== 0
Ig== 1
Iw== 2

Malformed encoding data throws InvalidDataException.

Unzip the example, then dotnet run.

Variations

Recognize special tokens explicitly. Run this after creating tokenizer in the full program.

string text = "<|endoftext|>";
Console.WriteLine(tokenizer.Encode(text,
    addSpecialTokens: true).Ids.Length);
Console.WriteLine(tokenizer.Encode(text,
    addSpecialTokens: false).Ids.Length);

1
7

Related recipes

• Tokenize text
• Save nlp results as json

The short answer

To get a tiktoken token count in C#, load the encoding’s ranks file with OneNLP, encode the exact text, and count Ids.Length. The example uses cl100k_base and prints 4 for “Hello, world!”. It counts encoded text tokens, not words, characters or a complete model request.

The ranks file is supplied with the download. OneNLP does not fetch an encoding at runtime. The tokenizer documentation specifies the file format, named encodings, patterns and special-token behavior. An encoding choice is part of the count: do not apply this result to every model. For GPT-4o, the pinned tokenizer guide shows o200k_base, not cl100k_base.

How to count the number of tokens?

Choose the encoding your application actually needs, load it, then encode the text without truncating it. Count the resulting IDs rather than counting whitespace-separated words. An ID represents a vocabulary entry selected by the encoding’s pattern and byte-pair merging rules. Punctuation and whitespace participate in that process.

For ordinary user text, the example explicitly sets addSpecialTokens: false. That treats strings resembling special tokens as ordinary text. The variation counts “<|endoftext|>” both ways: it is one recognized special token with special-token recognition enabled and seven ordinary encoded tokens with it disabled. The switch is not a general instruction to append a model’s chat-message framing.

The default for Encode is to recognize all allowed special tokens appearing in the input. If your application accepts arbitrary text, make this choice explicitly rather than inheriting the default by accident. The setting can change the count even though the input string has not changed.

The .tiktoken file contains base64-encoded token bytes and decimal ranks. It does not contain the split pattern or special-token definitions. The named Cl100kBase loader supplies those for this encoding. Tokenizer.FromFile can infer a supported named encoding from a recognized filename stem or ranks count; a custom encoding needs TiktokenTokenizer.Create with its own pattern.

How much text is 1000 tokens?

There is no fixed character or word length for that many encoded IDs. Count the actual text using the actual encoding instead of converting a word count with a constant. A text containing punctuation, unusual words or several languages need not split like an English sentence of the same character length.

This recipe counts only the string passed to Encode. If an API wraps messages with role markers, tool definitions or other structure, counting message text alone does not count that whole request. Match your provider’s documented request format and encoding before using a count to enforce a request budget. This example does not estimate billing or predict an output response’s length.

The result also should not be read as a token-to-character mapping. OneNLP’s tokenizer offsets use UTF-16 units for string input, and a token that begins or ends inside a multi-byte character reports the whole character. Neighboring tokens may consequently share a reported character range. Count IDs, not distinct offsets, for the encoded length.

Keep the file and encoding name together when packaging an application; replacing the ranks data while retaining an unrelated pattern would no longer describe the encoding this example uses.