Use TryGetCharSpan to turn a regex hit into a span when both ends line up with tokens.
dotnet add package RecursiveAI.OneNlpRequires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using System.Text.RegularExpressions; using RecursiveAI.OneNlp; var doc = Nlp.Analyze("Visit North Vale."); var hits = Regex.Matches(doc.Text, "North Vale|isit"); foreach (Match hit in hits) { int end = hit.Index + hit.Length; if (doc.TryGetCharSpan(hit.Index, end, out var span)) Console.WriteLine($"{hit.Value}: " + $"tokens [{span.Start}, {span.End})"); else Console.WriteLine($"{hit.Value}: " + "not aligned"); }
isit: not aligned North Vale: tokens [1, 3)
Unzip the example, then dotnet run.
Nlp.Analyze gives you tokens and sentence boundaries with just RecursiveAI.OneNlp; no model download is needed.
Full program
using System.Text.RegularExpressions; using RecursiveAI.OneNlp; var doc = Nlp.Analyze("Visit North Vale."); foreach (Match match in Regex.Matches(doc.Text, "North Vale|isit")) { int start = match.Index; int end = start + match.Length; if (doc.TryGetCharSpan(start, end, out var span)) Console.WriteLine($"{match.Value}: tokens [{span.Start}, {span.End})"); else Console.WriteLine($"{match.Value}: not aligned"); }
dotnet run prints:
isit: not aligned North Vale: tokens [1, 3)
When either edge falls inside a token, TryGetCharSpan returns false; it doesn’t move the edge.
Variations
Label the returned span. Replace the TryGetCharSpan branch and its else in the full program with:
if (doc.TryGetCharSpan(start, end, out var span, label: "PLACE")) Console.WriteLine($"{span.Label}: {span}");
PLACE: North ValeRelated recipes
The short answer
Run the regex on doc.Text and pass Match.Index and Match.Index + Match.Length to doc.TryGetCharSpan. If it returns true, span.Start and span.End are zero-based token positions, with the end excluded. In Visit North Vale., North Vale covers tokens [1, 3), so it includes tokens 1 and 2, not token 3.
How do I convert regex character offsets to token indexes?
Keep two coordinate systems separate. Match.Index is a position in the original string, and Match.Length measures the length of the matched text. .NET strings and OneNLP document offsets both count UTF-16 code units. A token index, by contrast, is a position in the token sequence. The first character of North is at string offset 6 but North is token 1. Run the regex on the exact same string as the document; positions from a shortened or normalized copy may point elsewhere.
Why doesn’t a regex hit line up with token boundaries?
A regex can find a substring inside a word: isit begins after the first character of Visit. It is a valid regex hit but not an aligned token span. A span may cover space between tokens, as North Vale does. It can’t start in a gap before its first token or end in a gap after its last token. An empty match also fails because the end must be after the start.
A regex word boundary (\b) is defined by the regex engine, not by the document’s tokenizer. Even a pattern that checks word boundaries still needs the alignment check if the result will be used as a token span. If your application needs inside-token hits as well, keep those as character ranges rather than treating them as token annotations.