Recursive AI

Trim text to a token limit in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Trim text by the tokenizer’s count, not by words or characters. The zip includes the small vocab.txt used below.

Terminal
dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneNlp.Tokenizers;
 
var tok = Tokenizer.FromFile("vocab.txt");
const string text = "red green blue";
var head = tok.TrimToTokenBudget(text, 2);
var tail = tok.TrimToTokenBudget(text, 2,
    TokenizerSide.Right);
string prefix = head.ToString();
Console.WriteLine($"Start: {prefix}");
Console.WriteLine($"End: {tail}");
Text
Start: red green
End: green blue

Unzip the example, then dotnet run.

TrimToTokenBudget returns a ReadOnlySpan<char> over the original string. Call .ToString() when you need a string, as shown for head.

Leave room for special tokens: markers like [CLS] and [SEP] that some tokenizers add but this trim doesn’t count.

Full program

C#
using RecursiveAI.OneNlp.Tokenizers;
 
var tokenizer = Tokenizer.FromFile("vocab.txt");
const string text = "red green blue";
var head = tokenizer.TrimToTokenBudget(text, 2);
var tail = tokenizer.TrimToTokenBudget(
    text, 2, TokenizerSide.Right);
Console.WriteLine($"Start: {head}");
Console.WriteLine($"End: {tail}");
Console.WriteLine($"Start tokens: {tokenizer.CountTokens(head, false)}");
Console.WriteLine($"End tokens: {tokenizer.CountTokens(tail, false)}");

dotnet run prints:

Text
Start: red green
End: green blue
Start tokens: 2
End tokens: 2

The example’s WordPiece vocabulary lists one token per line. Save these four lines as vocab.txt next to the project if you’re pasting the snippet:

Text
[UNK]
red
green
blue

A negative budget throws ArgumentOutOfRangeException.

Variations

Keep the whole string if it fits. Add this after the full program’s output lines:

C#
var whole = tokenizer.TrimToTokenBudget(text, 3);
Console.WriteLine($"Fits: {whole}");
Text
Fits: red green blue

Trim with the cl100k_base LLM encoding. Get cl100k_base.tiktoken from the token-counting example zip and place it next to the project. Add these lines after the full program’s output lines:

C#
var llm = TiktokenTokenizer.Cl100kBase("cl100k_base.tiktoken");
var clipped = llm.TrimToTokenBudget("Hello, world!", 2);
Console.WriteLine(clipped.ToString());
Text
Hello,

The short answer

With the tokenizer your model uses, call TrimToTokenBudget(text, maxTokens) to keep the start, or pass TokenizerSide.Right to keep the end. The method returns a ReadOnlySpan<char> over the original string. In the example, a budget of two keeps red green from the start or green blue from the end. The budget excludes special tokens added by a tokenizer’s post-processor.

How many words fit in a token limit?

There’s no fixed conversion from words to tokens. The example vocabulary has an entry for each color, so each color takes one token. A model vocabulary can instead split a word into several pieces. Use the tiny vocabulary to run the example, not to estimate the budget of another model.

The full program counts both outputs with CountTokens(..., addSpecialTokens: false) so the count uses the same special-token setting as the trim. Trimming ignores the tokenizer’s padding and truncation settings, as does CountTokens. A budget large enough for the content leaves the original text unchanged.

How do I truncate a prompt that exceeds a token limit?

Decide which end to keep before trimming. Keep the start if it contains the instructions you need; keep the end if the most recent passage matters more. If only one field in a prompt can be shortened, trim that field rather than cutting the combined prompt through its instructions or delimiters.

Allow room for any text you put around the trimmed field and for special tokens before sending the prompt. If the model’s limit includes the generated response, leave room for that too. Check the tokenizer count of the assembled input rather than treating the field’s budget as the total for the request.

All onenlp recipes