Recursive AI

Index Japanese text with bigrams in C#

OneIndex is a free, open-source search library for .NET from Recursive AI. MIT or Apache-2.0, preview, runs in-process.

Bigrams are pairs of adjacent characters.

Terminal
dotnet add package RecursiveAI.OneIndex --prerelease

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using OneIndex;
const string text = "日本語", term = "本語";
using var index =
    SearchIndex.InMemory<Note>();
index.Upsert(new Note
    { Id = "ja", Text = text });
index.Commit();
foreach (var hit in index.Search(term))
    Console.WriteLine(hit.Document.Text);
 
class Note
{
    [SearchId]
    public string Id { get; set; } = "";
    [FullText(Analyzer = "cjk")]
    public string Text { get; set; } = "";
}

Unzip the example, then dotnet run.

Searching for 本語 prints the stored text:

Text
日本語

The full program

The download uses OneIndex 0.4.0-preview.2. It indexes 日本語 (Japanese language) and 英語 (English language) in memory. The NuGet reference in Example.csproj supplies the source generator that maps Note to the index schema. CjkAnalyzer.Analyze lets you inspect the same kind of terms the cjk field uses.

C#
using OneIndex;
using OneIndex.Analysis;
 
using var index = SearchIndex.InMemory<Note>();
index.Upsert([
    new() { Id = "ja", Text = "日本語" },
    new() { Id = "en", Text = "英語" }
]);
index.Commit();
 
var analyzer = new CjkAnalyzer();
var terms = analyzer.Analyze("日本語").Select(token => token.Term);
Console.WriteLine("Terms: " + string.Join(", ", terms));
foreach (var hit in index.Search("本語"))
    Console.WriteLine("Match: " + hit.Document.Text);
 
class Note
{
    [SearchId]
    public string Id { get; set; } = "";
    [FullText(Analyzer = "cjk")]
    public string Text { get; set; } = "";
}

dotnet run prints:

Text
Terms: 日本, 本語
Match: 日本語

Call Commit() after Upsert(): searches only see the last commit. No dictionary package is needed for this example.

Variations

Search for the first pair instead. Insert this before the Note class in the full program. Both 日本 and 本語 appear in the indexed text.

C#
foreach (var hit in index.Search("日本"))
    Console.WriteLine("Match: " + hit.Document.Text);

Added output:

Text
Match: 日本語

Try a one-character query. Insert this before the Note class. The default search finds 日本語; the strict query looks for an exact 本 term and finds nothing.

C#
foreach (var hit in index.Search("本"))
    Console.WriteLine(hit.Document.Text);
Console.WriteLine(index.SearchStrict("Text:本").Count());

Added output:

Text
日本語
0

Full-text search in C# · Search for an exact phrase · Search with synonyms


The short answer

Set [FullText(Analyzer = "cjk")] on a text property, upsert and commit your documents, then call index.Search("本語"). The core OneIndex package indexes 日本語 as 日本 and 本語, so the example finds the document on the second pair without needing a Japanese dictionary. “CJK” here refers to Chinese, Japanese, and Korean scripts handled by the analyzer.

A bigram is a pair of adjacent characters. For Japanese text, the pairs help make pieces of a run searchable even when there are no spaces between words. In 日本語, the first pair is 日本; move the start by one character to get 本語. Both are terms in the index. The returned document still holds the original string 日本語.

This does not identify word boundaries. The same pairing rule applies even if the two characters fall on opposite sides of a word boundary. If your application needs dictionary-based segmentation into Japanese words, use the separate analyzer described in the Asian-language analysis docs; this recipe uses only the built-in analyzer.

OneIndex’s analysis docs define the runs by script: Han, Hiragana, Katakana, or Hangul. Two adjacent characters of different scripts aren’t joined into a bigram. That distinction matters for Japanese input that mixes kanji and kana: the analyzer doesn’t create a token for every possible adjacent pair across the whole string. Inspect representative strings with Analyze before choosing an analyzer for your field.

Can a CJK bigram analyzer match a single character?

A one-character run is indexed as a one-character term. A longer run is indexed as overlapping pairs, not as pairs and all their individual characters. So 日本語 contains the indexed terms 日本 and 本語, but not a standalone 本 term.

In this example, index.Search("本") returns 日本語 because the default search matches prefixes on the last unquoted word, and 本 starts the term 本語. To request that prefix match explicitly, use index.SearchStrict("Text:本*").

A prefix match still isn’t a character-anywhere search: Text:語* finds nothing here, since neither pair starts with 語. For exact single-kanji matches, index a second copy of the text in a [FullText(Analyzer = "standard")] field; that analyzer indexes each Han ideograph on its own.

All oneindex recipes