Recursive AI

Match phrases in text in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

Phrase matching finds entries from a phrase dictionary in text, including multiword names.

No model package is needed.

Terminal
dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Matching;
 
var doc = Nlp.Analyze("Visit NEW YORK.");
var matcher = new PhraseMatcher(
    TokenProperty.Lower);
matcher.Add("CITY", ["New York"]);
var spans = matcher.MatchSpans(doc);
foreach (var span in spans)
    Console.WriteLine(
        $"{span.Label}: {span}");
Text
CITY: NEW YORK

Adding an empty phrase raises ArgumentException; a valid dictionary with no occurrences returns no matches.

Unzip the example, then dotnet run.

Variations

Require the original capitalization. Replace Program.cs with:

C#
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Matching;
 
var doc = Nlp.Analyze(
    "New York and NEW YORK");
var matcher = new PhraseMatcher(
    TokenProperty.Orth);
matcher.Add("CITY", ["New York"]);
var spans = matcher.MatchSpans(doc);
foreach (var span in spans)
    Console.WriteLine(
        $"{span.Label}: {span}");
Text
CITY: New York

The short answer

Use PhraseMatcher with a dictionary of phrases and choose the token attribute explicitly. TokenProperty.Lower ignores capitalization; TokenProperty.Orth compares the original spelling. The example matches the dictionary entry New York against NEW YORK while preserving the text found in the document.

How do I match phrases in text in C#?

Create a document, add phrases under a key, then enumerate MatchSpans(doc). The key becomes the label of each returned span. A key can group several phrases, such as a list of cities, products or internal terms. Use separate keys when the caller needs to distinguish groups.

The dictionary is matched as token sequences, not as arbitrary substrings. Both the phrase and document are split into tokens, and the chosen attribute is compared position by position. This differs from searching for a string anywhere inside another string. It also does not infer synonyms, expand abbreviations or resolve a mention to a real-world object.

The phrase matcher documentation says phrases given as text use the matcher’s word tokenizer. Use the same tokenizer that produced the document, particularly if your application customizes segmentation. The constructor can take a pipeline so phrase tokenization follows that pipeline. A mismatch in punctuation or word boundaries can prevent an otherwise familiar-looking phrase from matching.

All occurrences are returned, including nested and overlapping ones. If the dictionary contains a city name and a longer name containing it, both can appear. Results are ordered by end token, then start token, then key insertion order. Do not assume the first result is the earliest-starting or longest phrase. A phrase registered under two keys is reported for each key.

For a task that must choose one entity from competing phrases, use an explicit selection policy or an entity ruler. The phrase matcher itself does not discard those alternatives and does not modify the document’s entity list.

What is the difference between case-sensitive and case-insensitive?

ORTH means the original token spelling, including capitalization. With Orth, New York and NEW YORK are different. The variation matches only the first form in its visible input.

For case-insensitive matching, use this constructor line from the snippet at the top:

C#
var matcher = new PhraseMatcher(TokenProperty.Lower);

LOWER compares each token’s text after converting it to lowercase with .NET’s invariant culture. Both forms match the same entry. The returned span still points to the original document, so printing it preserves the original capitalization. You do not need to lowercase the source string and then try to map positions back.

Case-insensitive does not mean fuzzy. It does not forgive misspellings, remove accents or join words split differently by the tokenizer.

NORM means the document’s normalized token value; if none was set, it uses the lowercase spelling. Shape compares patterns of letters, digits and punctuation: X means an uppercase letter and d means a digit. Lemma, part-of-speech and dependency matching require annotated phrase documents as well as annotated input documents. A plain phrase string cannot supply those annotations; an unannotated phrase token is an error for those attributes.

All onenlp recipes