Recursive AI

Split JavaScript text into graphemes in C#

OneEcma is a free, open-source JavaScript interpreter for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

A grapheme is one visible character, even if it takes several Unicode code points.

Terminal
dotnet add package RecursiveAI.OneEcma

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneEcma;
Console.WriteLine(new ScriptEngine()
  .Evaluate("""
const split = new Intl.Segmenter('en');
[...split.segment('👩‍💻e\u0301')]
  .map(part => part.segment).join(' | ');
""").AsString);

Unzip the example, then dotnet run.

Prints:

Text
👩‍💻 | é

The full program

C#
using RecursiveAI.OneEcma;
var engine = new ScriptEngine();
Console.WriteLine(engine.Evaluate("""
const text = 'A👩‍💻e\u0301!';
const segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' });
const segments = segmenter.segment(text);
const lines = [...segments].map(({ index, segment }) => `${index}: ${segment}`);
lines.push(`At offset 3: ${segments.containing(3).segment}`);
lines.push(`UTF-16 length: ${text.length}`);
lines.join('\n');
""").AsString);

dotnet run prints:

Text
0: A
1: 👩‍💻
6: é
8: !
At offset 3: 👩‍💻
UTF-16 length: 9

segments.containing takes a UTF-16 offset, not a grapheme number; segments.containing(3) finds the emoji even though it started at offset 1. An offset outside the string returns undefined.

Variations

Use word and keep isWordLike segments to extract words from this English example. Replace the JavaScript inside Evaluate with the code below. It prints Hi | world.

JAVASCRIPT
const words = new Intl.Segmenter('en', {
  granularity: 'word'
});
[...words.segment('Hi, world!')]
  .filter(part => part.isWordLike)
  .map(part => part.segment).join(' | ');

The short answer

Run Intl.Segmenter inside OneEcma’s ScriptEngine. The default granularity is grapheme, set explicitly in the full program. Iterate the result of .segment(text) to get each grapheme and its start offset. The first snippet returns two graphemes: a woman technologist emoji made of several code points and e followed by an acute accent.

What is a grapheme cluster in JavaScript?

A grapheme cluster is a sequence of one or more code points treated as a unit by the default rules in Unicode Standard Annex #29. For example, the emoji above contains two supplementary code points joined by a zero-width joiner. Iterating code points instead of UTF-16 units still doesn’t keep that sequence together. OneEcma’s segmentation rules include “GB11 emoji ZWJ sequences and GB12/13 regional-indicator pairs,” covering the joiner example and regional-indicator flag pairs.

For a preview that shows only the first few characters, take whole graphemes. Truncating by code units can split a surrogate pair; truncating by code points can still separate an accent from its letter. Segmenting the input doesn’t rewrite or normalize it. The accent in the output is still a separate code point, just part of the same returned segment.

What does Intl.Segmenter return as the index?

The index is the segment’s start in UTF-16 code units. The full program’s first segment A uses one unit. The next segment, 👩‍💻, uses five: two for each supplementary code point and one for the joiner. That’s why the next segment begins at 6, even though it’s the third grapheme. In C#, use these offsets with the same unchanged input string; C# strings are also indexed by UTF-16 units. Intl.Segmenter on MDN describes the iterable interface and its granularity options.

Can Intl.Segmenter split Thai or Japanese text into words?

Not with dictionary boundaries in OneEcma. Grapheme and word boundaries answer different questions: a Thai cluster can be suitable for cursor movement without being a whole word. The text services docs say “Dictionary segmentation (Han, Thai, Lao, Khmer, Burmese, and Japanese words) is not implemented.” Use word with isWordLike for cases like the English variation; don’t assume that requesting a locale makes it a dictionary tokenizer.

All oneecma recipes