---
title: "Match text token patterns in C#"
url: https://recursiveai.org/onenlp/cookbook/match-token-patterns/
description: "Find sequences of words with OneNLP token patterns in C#: attribute tests, repetition, greedy matching and labeled spans without a model package."
version: "1.0.0"
---

# Match text token patterns in C#

**OneNLP** is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneNlp) · [GitHub](https://github.com/recursive-ai-oss/onenlp-csharp) · [Docs](https://recursiveai.org/onenlp/docs/) · [Download example (.zip)](https://recursiveai.org/onenlp/cookbook/match-token-patterns/match-token-patterns.zip)

Token patterns find sequences of words by their text or attributes, such as a number followed by a unit.

```shell
dotnet add package RecursiveAI.OneNlp
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Matching;

var doc = Nlp.Analyze("Order 12 boxes.");
var matcher = new Matcher();
matcher.AddJson("QUANTITY", """
    [{"IS_DIGIT":true},
     {"LOWER":"boxes"}]
    """);
var spans = matcher.MatchSpans(doc);
foreach (var span in spans)
    Console.WriteLine(
        $"{span.Label}: {span}");
```

```text
QUANTITY: 12 boxes
```

Unzip [the example](https://recursiveai.org/onenlp/cookbook/match-token-patterns/match-token-patterns.zip), then `dotnet run`.

## Variations

**Keep the longest repeated sequence.** Replace `Program.cs` with:

```csharp
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Matching;

var doc = Nlp.Analyze("red red red");
var matcher = new Matcher();
matcher.AddJson("RUN", """
    [{"LOWER":"red","OP":"+"}]
    """, GreedyMode.Longest);
var spans = matcher.MatchSpans(doc);
foreach (var span in spans)
    Console.WriteLine(
        $"{span.Label}: {span}");
```

```text
RUN: red red red
```

## Related recipes

- [Match phrases in text in C#](https://recursiveai.org/onenlp/cookbook/match-phrases/)
- [Rule-based entity recognition in C#](https://recursiveai.org/onenlp/cookbook/add-rule-based-entities/)
- [Zero-shot entity extraction in C#](https://recursiveai.org/onenlp/cookbook/extract-custom-entity-types/)

---

## The short answer

Use `Matcher` from `RecursiveAI.OneNlp.Matching` to find sequences of tokens. Add a pattern under a key, then call `MatchSpans(doc)` to get labeled spans. The example finds `12 boxes`: one digit token followed by the lowercased word `boxes`. It needs only the core package, not trained weights.

## How do I match text token patterns in C#?

Start with a document and describe one token position per JSON object. Tests in one object must all pass; separate objects describe consecutive positions. The snippet at the top uses `IS_DIGIT` for the first token and `LOWER` for the second. The key `QUANTITY` becomes the returned span’s label. Matching returns results; it does not add entities to the document.

Patterns describe tokens, not arbitrary character positions. A phrase split into three tokens needs three positions unless an operator changes the number consumed. An empty object accepts any one token. A pattern cannot return an empty span, even if all its positions are optional.

The [matching documentation](https://recursiveai.org/onenlp/docs/matching/) defines repetition: omitting `OP` consumes exactly one token, `?` consumes zero or one, `+` one or more, and `*` zero or more. Bounded repeats are also supported. An empty `OP` is invalid; omit the property for the ordinary one-token case.

By default, the matcher reports every reachable range, including overlaps. That matters when counting occurrences. A repeated-word pattern can return shorter subranges as well as the whole run. The variation uses `GreedyMode.Longest` to prefer longer non-overlapping matches for that key. `GreedyMode.First` instead prefers the earliest start, breaking ties by length. Filtering is per key: it does not suppress another key’s results.

Malformed JSON patterns raise `PatternFormatException` when added, before matching.

## Which token attributes can I match?

Use `ORTH` or `TEXT` for the original spelling, `LOWER` for invariant-culture lowercase text, and flags such as `IS_DIGIT`, `IS_ALPHA` or `IS_PUNCT` for token properties. `LENGTH` counts Unicode code points. `IS_STOP` consults the matcher’s stop-word collection, which starts empty.

Annotation-based tests need the corresponding document columns. `POS`, `LEMMA` and `DEP` do not compute tags, lemmas or dependencies during matching. Without a component that sets them, those columns are empty. Installing a model and annotating the document is a separate step. For a rule based only on spelling or punctuation, leave that dependency out.

## Can I use regular expressions in token patterns?

Yes. A string attribute can take a `REGEX` predicate. It searches within that attribute’s value, rather than running across the whole document. Anchor the expression if the entire token value must match. A regular expression in a single position cannot consume the next token.

OneNLP uses the linear-time regex engine when the expression permits it. Look-around and back-references use the backtracking engine with a documented timeout; a timed-out test counts as no match. Use explicit attribute tests for simple conditions instead of turning every token rule into regex.

This is distinct from [C# language pattern matching](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/patterns/pattern-matching), which tests values with `is` and `switch`. Here the input is natural-language text, and the output is matching token spans.