---
title: "Create an NLP document from token offsets in C#"
url: https://recursiveai.org/onenlp/cookbook/create-doc-from-existing-tokens/
description: "Build a OneNLP Doc from UTF-16 start offsets and lengths without retokenizing the original string. Keep gaps and reject overlapping ranges."
version: "1.0.0"
---

# Create an NLP document from token offsets in C#

**OneNLP** is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneNlp) · [GitHub](https://github.com/recursive-ai-oss/onenlp-csharp) · [Docs](https://recursiveai.org/onenlp/docs/) · [Download example (.zip)](https://recursiveai.org/onenlp/cookbook/create-doc-from-existing-tokens/create-doc-from-existing-tokens.zip)

Pass `TokenBoundary` values to `new Doc` to keep the original text without retokenizing it.

```shell
dotnet add package RecursiveAI.OneNlp
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneNlp;

var doc = new Doc(new StringStore(),
    "red/blue", [new(0, 3), new(4, 4)]);
foreach (var token in doc)
    Console.WriteLine($"{token.Idx}.." +
        $"{token.EndIdx}: {token.Text}");
```

```text
0..3: red
4..8: blue
```

Unzip [the example](https://recursiveai.org/onenlp/cookbook/create-doc-from-existing-tokens/create-doc-from-existing-tokens.zip), then `dotnet run`.

`new(0, 3)` means start at 0 and take 3 UTF-16 code units (C# `char` values): `red`.

## Full program

```csharp
using RecursiveAI.OneNlp;

const string text = "red/blue";
var strings = new StringStore();
var doc = new Doc(strings, text,
    [new(0, 3), new(4, 4)]);
foreach (var token in doc)
    Console.WriteLine($"{token.Idx}..{token.EndIdx}: {token.Text}");
Console.WriteLine($"Original: {doc.Text}");

try
{
    _ = new Doc(strings, text,
        [new(0, 3), new(2, 4)]);
}
catch (ArgumentException)
{
    Console.WriteLine("Overlapping offsets rejected.");
}
```

`dotnet run` prints:

```text
0..3: red
4..8: blue
Original: red/blue
Overlapping offsets rejected.
```

The constructor throws `ArgumentException` for an empty, out-of-range, out-of-order, or overlapping boundary.

## Variations

**Include the separator as a token.** Replace the first `Doc` constructor in the full program:

```csharp
var doc = new Doc(strings, text,
    [new(0, 3), new(3, 1), new(4, 4)]);
```

The token loop now prints:

```text
0..3: red
3..4: /
4..8: blue
```

## Related recipes

- [Tokenize text](https://recursiveai.org/onenlp/cookbook/tokenize-text/)
- [Read CoNLL-U training data](https://recursiveai.org/onenlp/cookbook/read-conllu-training-data/)

---

## The short answer

Pass a `StringStore`, the original string, and a sequence of `TokenBoundary` values to the `Doc` constructor. Each boundary takes a start and a length, in UTF-16 code units. Use offsets in text order; each token must start at or after the previous token’s end. The constructor uses those boundaries without tokenizing the string again.

## What do UTF-16 offsets mean in a C# string?

A UTF-16 offset counts .NET `char` values from the start of the string, not bytes or displayed characters. For example, in `🐂 red` the emoji takes two `char` values, the space takes one, and `red` starts at offset 3. See [.NET’s character encoding guide](https://learn.microsoft.com/en-us/dotnet/standard/base-types/character-encoding-introduction#the-string-and-char-types) for why some characters take two values.

If your tokenizer exports UTF-8 byte offsets or Unicode code point offsets, convert them to UTF-16 before constructing the document. Use the exact string the positions refer to. If the upstream source gives an exclusive end rather than a length, the length is the end minus the start. In `red/blue`, `blue` covers `[4, 8)`, so its constructor arguments are `4, 4`, not `4, 8`.

## Do token offsets have to cover whitespace and punctuation?

No. A token covers only the range you supply. The two-token example leaves the slash unassigned, but `doc.Text` still contains `red/blue`. The variation makes that same slash a token without changing the original text. Adjacent tokens can touch, but they can’t share characters: a range starting at 2 overlaps the preceding `red` range, which ends at 3. This distinction matters when importing offsets from a format that allows nested or overlapping annotations: those ranges can’t all serve as the document’s token boundaries.
