Recursive AI

Convert regex matches to token spans in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Use TryGetCharSpan to turn a regex hit into a span when both ends line up with tokens.

Terminal
dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using System.Text.RegularExpressions;
using RecursiveAI.OneNlp;
 
var doc = Nlp.Analyze("Visit North Vale.");
var hits = Regex.Matches(doc.Text,
    "North Vale|isit");
foreach (Match hit in hits)
{
    int end = hit.Index + hit.Length;
    if (doc.TryGetCharSpan(hit.Index,
        end, out var span))
        Console.WriteLine($"{hit.Value}: " +
            $"tokens [{span.Start}, {span.End})");
    else
        Console.WriteLine($"{hit.Value}: " +
            "not aligned");
}
Text
isit: not aligned
North Vale: tokens [1, 3)

Unzip the example, then dotnet run.

Nlp.Analyze gives you tokens and sentence boundaries with just RecursiveAI.OneNlp; no model download is needed.

Full program

C#
using System.Text.RegularExpressions;
using RecursiveAI.OneNlp;
 
var doc = Nlp.Analyze("Visit North Vale.");
foreach (Match match in Regex.Matches(doc.Text, "North Vale|isit"))
{
    int start = match.Index;
    int end = start + match.Length;
    if (doc.TryGetCharSpan(start, end, out var span))
        Console.WriteLine($"{match.Value}: tokens [{span.Start}, {span.End})");
    else
        Console.WriteLine($"{match.Value}: not aligned");
}

dotnet run prints:

Text
isit: not aligned
North Vale: tokens [1, 3)

When either edge falls inside a token, TryGetCharSpan returns false; it doesn’t move the edge.

Variations

Label the returned span. Replace the TryGetCharSpan branch and its else in the full program with:

C#
if (doc.TryGetCharSpan(start, end,
    out var span, label: "PLACE"))
    Console.WriteLine($"{span.Label}: {span}");
Text
PLACE: North Vale

The short answer

Run the regex on doc.Text and pass Match.Index and Match.Index + Match.Length to doc.TryGetCharSpan. If it returns true, span.Start and span.End are zero-based token positions, with the end excluded. In Visit North Vale., North Vale covers tokens [1, 3), so it includes tokens 1 and 2, not token 3.

How do I convert regex character offsets to token indexes?

Keep two coordinate systems separate. Match.Index is a position in the original string, and Match.Length measures the length of the matched text. .NET strings and OneNLP document offsets both count UTF-16 code units. A token index, by contrast, is a position in the token sequence. The first character of North is at string offset 6 but North is token 1. Run the regex on the exact same string as the document; positions from a shortened or normalized copy may point elsewhere.

Why doesn’t a regex hit line up with token boundaries?

A regex can find a substring inside a word: isit begins after the first character of Visit. It is a valid regex hit but not an aligned token span. A span may cover space between tokens, as North Vale does. It can’t start in a gap before its first token or end in a gap after its last token. An empty match also fails because the end must be after the start.

A regex word boundary (\b) is defined by the regex engine, not by the document’s tokenizer. Even a pattern that checks word boundaries still needs the alignment check if the result will be used as a token span. If your application needs inside-token hits as well, keep those as character ranges rather than treating them as token annotations.

All onenlp recipes