Recursive AI

Set custom search stop words in C#

OneIndex is a free, open-source search library for .NET from Recursive AI. MIT or Apache-2.0, preview, runs in-process.

Use StopWordSettings to ignore “manual” so “manual pump” can find “pump repair.”

Terminal
dotnet add package RecursiveAI.OneIndex --prerelease

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using OneIndex;
using OneIndex.Analysis;
var words = new StopWordSettings(["manual"]);
AnalyzerRegistry.Register("catalog-en",
    () => new LanguageAnalyzer(
        "en", keepUnstemmed: false,
        stopWords: words));
using var index =
    SearchIndex.InMemory<Entry>();
index.Upsert(new Entry { Text = "pump repair" });
index.Commit();
foreach (var hit
    in index.Search("manual pump"))
    Console.WriteLine(hit.Document.Text);
 
public class Entry
{
    [SearchId]
    public string Id { get; set; } = "pump";
    [FullText(Analyzer = "catalog-en")]
    public string Text { get; set; } = "";
}

Unzip the example, then dotnet run.

Text
pump repair

The catalog-en analyzer processes the field both when you index it and when you search it. The NuGet package includes the source generator that maps Entry from its attributes.

The full program

The download indexes three entries. It compares searches with and without stop words and shows how removing a word affects a quoted phrase.

C#
using OneIndex;
using OneIndex.Analysis;
 
var words = new StopWordSettings(["manual"]);
AnalyzerRegistry.Register("catalog-en", () =>
    new LanguageAnalyzer(
        "en", keepUnstemmed: false,
        stopWords: words));
using var index = SearchIndex.InMemory<Entry>();
index.Upsert([
    new Entry { Id = "A", Text = "pump manual repair" },
    new Entry { Id = "B", Text = "pump repair" },
    new Entry { Id = "C", Text = "valve repair" }
]);
index.Commit();
 
Show("manual pump");
// In Replace mode, "the" is not a stop word, so no entry matches.
Show("the pump");
Show("\"pump repair\"");
Show("manual");
 
void Show(string query)
{
    var ids = index.Search(query).Select(hit => hit.Id).Order();
    Console.WriteLine($"{query}: [{string.Join(", ", ids)}]");
}
 
public class Entry
{
    [SearchId] public string Id { get; set; } = "";
    [FullText(Analyzer = "catalog-en")]
    public string Text { get; set; } = "";
}
Text
manual pump: [A, B]
the pump: []
"pump repair": [A, B]
manual: []

The default Replace mode drops manual, but keeps the because it isn’t in the custom list. If you mistype catalog-en in the field attribute, the index can’t open: the analyzer name must match the one registered before opening the index.

Variations

Replace the words declaration in the full program with one of these. Each variation builds a fresh in-memory index.

Keep the built-in English list too. The the pump search now returns [A, B] because the comes from the built-in list and manual from yours.

C#
var words = new StopWordSettings(["manual"])
{
    Mode = StopWordMode.Extend
};

Keep the space in the index positions. The quoted "pump repair" search now returns only [B]: A still has a position between the two terms even though manual itself isn’t searchable.

C#
var words = new StopWordSettings(["manual"])
{
    PreserveGaps = true
};

The short answer

For custom stop words in C#, create a StopWordSettings list, pass it into a LanguageAnalyzer factory, then register that factory before you open the index. Point [FullText(Analyzer = "catalog-en")] at the registered name. That applies the same list to the document field and to what people type in the search box. Use Replace for just your words or Extend to add them to the built-in language list.

“The” is a built-in English example; “manual” is a custom one in the repair catalog above. Pick words that don’t distinguish one entry from another in the field you’re searching. A technical library might keep “manual” if users need to distinguish a manual from a specification. Stop words aren’t synonyms or ranking hints: the analyzer removes them, so that field’s index can’t require the removed word to be present.

OneIndex folds case before checking the list, so adding "Manual" also removes manual. It checks before stemming (reducing words to a shared form) and accent folding, so adding one form doesn’t automatically remove all its inflections or accented spellings. An empty list in Replace mode turns off stop-word filtering for that analyzer. See the analysis reference for details.

How do stop words affect phrase searches?

By default, StopWordSettings closes the gap when it removes a word. In the sample, “pump manual repair” is indexed with pump and repair next to each other, so it satisfies the quoted search "pump repair". Setting PreserveGaps = true keeps their original spacing in indexed positions. It still removes manual; it does not turn that word into a searchable placeholder. Whether to close gaps depends on what you want a quoted search to mean.

Test stop-only searches separately from empty input. With the typed index’s Search(text) in this example, manual matches nothing once the analyzer removes it. A blank search is different: it returns all documents. The user-text query rules describe that distinction.

Do you need to reindex after changing stop words?

Yes, if you change the list or gap setting on the field analyzer. When you remove a word, the old index may still contain it. If you put a word back, the index may have no entries for it yet. Old positions also don’t change just because a new analyzer now handles queries. Rebuild from your documents with the new settings, and use the same settings when you reopen the rebuilt index. The registered name by itself doesn’t encode the word list.

This is different from the document-level [StopWords] attribute. That attribute only removes words from user queries; it doesn’t remove them from the index. Use the query-only setting if that’s the behavior you need.

All oneindex recipes