Recursive AI

Match text token patterns in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

Token patterns find sequences of words by their text or attributes, such as a number followed by a unit.

Terminal
dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Matching;
 
var doc = Nlp.Analyze("Order 12 boxes.");
var matcher = new Matcher();
matcher.AddJson("QUANTITY", """
    [{"IS_DIGIT":true},
     {"LOWER":"boxes"}]
    """);
var spans = matcher.MatchSpans(doc);
foreach (var span in spans)
    Console.WriteLine(
        $"{span.Label}: {span}");
Text
QUANTITY: 12 boxes

Unzip the example, then dotnet run.

Variations

Keep the longest repeated sequence. Replace Program.cs with:

C#
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Matching;
 
var doc = Nlp.Analyze("red red red");
var matcher = new Matcher();
matcher.AddJson("RUN", """
    [{"LOWER":"red","OP":"+"}]
    """, GreedyMode.Longest);
var spans = matcher.MatchSpans(doc);
foreach (var span in spans)
    Console.WriteLine(
        $"{span.Label}: {span}");
Text
RUN: red red red

The short answer

Use Matcher from RecursiveAI.OneNlp.Matching to find sequences of tokens. Add a pattern under a key, then call MatchSpans(doc) to get labeled spans. The example finds 12 boxes: one digit token followed by the lowercased word boxes. It needs only the core package, not trained weights.

How do I match text token patterns in C#?

Start with a document and describe one token position per JSON object. Tests in one object must all pass; separate objects describe consecutive positions. The snippet at the top uses IS_DIGIT for the first token and LOWER for the second. The key QUANTITY becomes the returned span’s label. Matching returns results; it does not add entities to the document.

Patterns describe tokens, not arbitrary character positions. A phrase split into three tokens needs three positions unless an operator changes the number consumed. An empty object accepts any one token. A pattern cannot return an empty span, even if all its positions are optional.

The matching documentation defines repetition: omitting OP consumes exactly one token, ? consumes zero or one, + one or more, and * zero or more. Bounded repeats are also supported. An empty OP is invalid; omit the property for the ordinary one-token case.

By default, the matcher reports every reachable range, including overlaps. That matters when counting occurrences. A repeated-word pattern can return shorter subranges as well as the whole run. The variation uses GreedyMode.Longest to prefer longer non-overlapping matches for that key. GreedyMode.First instead prefers the earliest start, breaking ties by length. Filtering is per key: it does not suppress another key’s results.

Malformed JSON patterns raise PatternFormatException when added, before matching.

Which token attributes can I match?

Use ORTH or TEXT for the original spelling, LOWER for invariant-culture lowercase text, and flags such as IS_DIGIT, IS_ALPHA or IS_PUNCT for token properties. LENGTH counts Unicode code points. IS_STOP consults the matcher’s stop-word collection, which starts empty.

Annotation-based tests need the corresponding document columns. POS, LEMMA and DEP do not compute tags, lemmas or dependencies during matching. Without a component that sets them, those columns are empty. Installing a model and annotating the document is a separate step. For a rule based only on spelling or punctuation, leave that dependency out.

Can I use regular expressions in token patterns?

Yes. A string attribute can take a REGEX predicate. It searches within that attribute’s value, rather than running across the whole document. Anchor the expression if the entire token value must match. A regular expression in a single position cannot consume the next token.

OneNLP uses the linear-time regex engine when the expression permits it. Look-around and back-references use the backtracking engine with a documented timeout; a timed-out test counts as no match. Use explicit attribute tests for simple conditions instead of turning every token rule into regex.

This is distinct from C# language pattern matching, which tests values with is and switch. Here the input is natural-language text, and the output is matching token spans.

All onenlp recipes