---
title: "Convert regex matches to token spans in C#"
url: https://recursiveai.org/onenlp/cookbook/align-regex-matches-to-tokens/
description: "Turn .NET regex match offsets into OneNLP token spans. Accept whole-token matches and reject partial matches without rounding."
version: "1.0.0"
---

# Convert regex matches to token spans in C#

**OneNLP** is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneNlp) · [GitHub](https://github.com/recursive-ai-oss/onenlp-csharp) · [Docs](https://recursiveai.org/onenlp/docs/) · [Download example (.zip)](https://recursiveai.org/onenlp/cookbook/align-regex-matches-to-tokens/align-regex-matches-to-tokens.zip)

Use `TryGetCharSpan` to turn a regex hit into a span when both ends line up with tokens.

```shell
dotnet add package RecursiveAI.OneNlp
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using System.Text.RegularExpressions;
using RecursiveAI.OneNlp;

var doc = Nlp.Analyze("Visit North Vale.");
var hits = Regex.Matches(doc.Text,
    "North Vale|isit");
foreach (Match hit in hits)
{
    int end = hit.Index + hit.Length;
    if (doc.TryGetCharSpan(hit.Index,
        end, out var span))
        Console.WriteLine($"{hit.Value}: " +
            $"tokens [{span.Start}, {span.End})");
    else
        Console.WriteLine($"{hit.Value}: " +
            "not aligned");
}
```

```text
isit: not aligned
North Vale: tokens [1, 3)
```

Unzip [the example](https://recursiveai.org/onenlp/cookbook/align-regex-matches-to-tokens/align-regex-matches-to-tokens.zip), then `dotnet run`.

`Nlp.Analyze` gives you tokens and sentence boundaries with just `RecursiveAI.OneNlp`; no model download is needed.

## Full program

```csharp
using System.Text.RegularExpressions;
using RecursiveAI.OneNlp;

var doc = Nlp.Analyze("Visit North Vale.");
foreach (Match match in Regex.Matches(doc.Text, "North Vale|isit"))
{
    int start = match.Index;
    int end = start + match.Length;
    if (doc.TryGetCharSpan(start, end, out var span))
        Console.WriteLine($"{match.Value}: tokens [{span.Start}, {span.End})");
    else
        Console.WriteLine($"{match.Value}: not aligned");
}
```

`dotnet run` prints:

```text
isit: not aligned
North Vale: tokens [1, 3)
```

When either edge falls inside a token, `TryGetCharSpan` returns `false`; it doesn’t move the edge.

## Variations

**Label the returned span.** Replace the `TryGetCharSpan` branch and its `else` in the full program with:

```csharp
if (doc.TryGetCharSpan(start, end,
    out var span, label: "PLACE"))
    Console.WriteLine($"{span.Label}: {span}");
```

```text
PLACE: North Vale
```

## Related recipes

- [Tokenize text](https://recursiveai.org/onenlp/cookbook/tokenize-text/)
- [Match token patterns](https://recursiveai.org/onenlp/cookbook/match-token-patterns/)

---

## The short answer

Run the regex on `doc.Text` and pass `Match.Index` and `Match.Index + Match.Length` to `doc.TryGetCharSpan`. If it returns `true`, `span.Start` and `span.End` are zero-based token positions, with the end excluded. In `Visit North Vale.`, `North Vale` covers tokens `[1, 3)`, so it includes tokens 1 and 2, not token 3.

## How do I convert regex character offsets to token indexes?

Keep two coordinate systems separate. [`Match.Index`](https://learn.microsoft.com/en-us/dotnet/api/system.text.regularexpressions.capture.index?view=net-10.0) is a position in the original string, and [`Match.Length`](https://learn.microsoft.com/en-us/dotnet/api/system.text.regularexpressions.capture.length?view=net-10.0) measures the length of the matched text. .NET strings and OneNLP document offsets both count UTF-16 code units. A token index, by contrast, is a position in the token sequence. The first character of `North` is at string offset 6 but `North` is token 1. Run the regex on the exact same string as the document; positions from a shortened or normalized copy may point elsewhere.

## Why doesn’t a regex hit line up with token boundaries?

A regex can find a substring inside a word: `isit` begins after the first character of `Visit`. It is a valid regex hit but not an aligned token span. A span may cover space between tokens, as `North Vale` does. It can’t start in a gap before its first token or end in a gap after its last token. An empty match also fails because the end must be after the start.

A regex word boundary (`\b`) is defined by the regex engine, not by the document’s tokenizer. Even a pattern that checks word boundaries still needs the alignment check if the result will be used as a token span. If your application needs inside-token hits as well, keep those as character ranges rather than treating them as token annotations.
