---
title: "Index Japanese text with bigrams in C#"
url: https://recursiveai.org/oneindex/cookbook/index-japanese-text-with-bigrams/
description: "Index Japanese text with OneIndex’s built-in CJK analyzer in C#. See the overlapping terms for 日本語 and search the index for 本語."
version: "0.4.0-preview.2"
---

# Index Japanese text with bigrams in C#

**OneIndex** is a free, open-source search library for .NET from Recursive AI. MIT or Apache-2.0, preview, runs in-process.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneIndex) · [GitHub](https://github.com/recursive-ai-oss/oneindex-csharp) · [Docs](https://recursiveai.org/oneindex/docs/) · [Download example (.zip)](https://recursiveai.org/oneindex/cookbook/index-japanese-text-with-bigrams/index-japanese-text-with-bigrams.zip)

Bigrams are pairs of adjacent characters.

```shell
dotnet add package RecursiveAI.OneIndex --prerelease
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using OneIndex;
const string text = "日本語", term = "本語";
using var index =
    SearchIndex.InMemory<Note>();
index.Upsert(new Note
    { Id = "ja", Text = text });
index.Commit();
foreach (var hit in index.Search(term))
    Console.WriteLine(hit.Document.Text);

class Note
{
    [SearchId]
    public string Id { get; set; } = "";
    [FullText(Analyzer = "cjk")]
    public string Text { get; set; } = "";
}
```

Unzip [the example](https://recursiveai.org/oneindex/cookbook/index-japanese-text-with-bigrams/index-japanese-text-with-bigrams.zip), then `dotnet run`.

Searching for `本語` prints the stored text:

```text
日本語
```

## The full program

The download uses OneIndex 0.4.0-preview.2. It indexes `日本語` (Japanese language) and `英語` (English language) in memory. The NuGet reference in `Example.csproj` supplies the source generator that maps `Note` to the index schema. `CjkAnalyzer.Analyze` lets you inspect the same kind of terms the `cjk` field uses.

```csharp
using OneIndex;
using OneIndex.Analysis;

using var index = SearchIndex.InMemory<Note>();
index.Upsert([
    new() { Id = "ja", Text = "日本語" },
    new() { Id = "en", Text = "英語" }
]);
index.Commit();

var analyzer = new CjkAnalyzer();
var terms = analyzer.Analyze("日本語").Select(token => token.Term);
Console.WriteLine("Terms: " + string.Join(", ", terms));
foreach (var hit in index.Search("本語"))
    Console.WriteLine("Match: " + hit.Document.Text);

class Note
{
    [SearchId]
    public string Id { get; set; } = "";
    [FullText(Analyzer = "cjk")]
    public string Text { get; set; } = "";
}
```

`dotnet run` prints:

```text
Terms: 日本, 本語
Match: 日本語
```

Call `Commit()` after `Upsert()`: searches only see the last commit. No dictionary package is needed for this example.

## Variations

**Search for the first pair instead.** Insert this before the `Note` class in the full program. Both `日本` and `本語` appear in the indexed text.

```csharp
foreach (var hit in index.Search("日本"))
    Console.WriteLine("Match: " + hit.Document.Text);
```

Added output:

```text
Match: 日本語
```

**Try a one-character query.** Insert this before the `Note` class. The default search finds `日本語`; the strict query looks for an exact `本` term and finds nothing.

```csharp
foreach (var hit in index.Search("本"))
    Console.WriteLine(hit.Document.Text);
Console.WriteLine(index.SearchStrict("Text:本").Count());
```

Added output:

```text
日本語
0
```

## Related recipes

[Full-text search in C#](https://recursiveai.org/oneindex/cookbook/full-text-search-csharp/) · [Search for an exact phrase](https://recursiveai.org/oneindex/cookbook/search-exact-phrases-with-slop/) · [Search with synonyms](https://recursiveai.org/oneindex/cookbook/search-with-synonyms/)

---

## The short answer

Set `[FullText(Analyzer = "cjk")]` on a text property, upsert and commit your documents, then call `index.Search("本語")`. The core OneIndex package indexes `日本語` as `日本` and `本語`, so the example finds the document on the second pair without needing a Japanese dictionary. “CJK” here refers to Chinese, Japanese, and Korean scripts handled by the analyzer.

## What is bigram tokenization in Japanese search?

A bigram is a pair of adjacent characters. For Japanese text, the pairs help make pieces of a run searchable even when there are no spaces between words. In `日本語`, the first pair is `日本`; move the start by one character to get `本語`. Both are terms in the index. The returned document still holds the original string `日本語`.

This does not identify word boundaries. The same pairing rule applies even if the two characters fall on opposite sides of a word boundary. If your application needs dictionary-based segmentation into Japanese words, use the separate analyzer described in the [Asian-language analysis docs](https://recursiveai.org/oneindex/docs/analysis-asian/); this recipe uses only the built-in analyzer.

OneIndex’s [analysis docs](https://recursiveai.org/oneindex/docs/analysis/#cjk-bigrams) define the runs by script: Han, Hiragana, Katakana, or Hangul. Two adjacent characters of different scripts aren’t joined into a bigram. That distinction matters for Japanese input that mixes kanji and kana: the analyzer doesn’t create a token for every possible adjacent pair across the whole string. Inspect representative strings with `Analyze` before choosing an analyzer for your field.

## Can a CJK bigram analyzer match a single character?

A one-character run is indexed as a one-character term. A longer run is indexed as overlapping pairs, not as pairs *and* all their individual characters. So `日本語` contains the indexed terms `日本` and `本語`, but not a standalone `本` term.

In this example, `index.Search("本")` returns `日本語` because [the default search matches prefixes on the last unquoted word](https://recursiveai.org/oneindex/docs/user-text/#typos-as-you-type-and-settings), and `本` starts the term `本語`. To request that prefix match explicitly, use `index.SearchStrict("Text:本*")`.

A prefix match still isn’t a character-anywhere search: `Text:語*` finds nothing here, since neither pair starts with `語`. For exact single-kanji matches, index a second copy of the text in a `[FullText(Analyzer = "standard")]` field; that analyzer indexes each Han ideograph on its own.
