A grapheme is one visible character, even if it takes several Unicode code points.
dotnet add package RecursiveAI.OneEcmaRequires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneEcma; Console.WriteLine(new ScriptEngine() .Evaluate(""" const split = new Intl.Segmenter('en'); [...split.segment('👩💻e\u0301')] .map(part => part.segment).join(' | '); """).AsString);
Unzip the example, then dotnet run.
Prints:
👩💻 | éThe full program
using RecursiveAI.OneEcma; var engine = new ScriptEngine(); Console.WriteLine(engine.Evaluate(""" const text = 'A👩💻e\u0301!'; const segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' }); const segments = segmenter.segment(text); const lines = [...segments].map(({ index, segment }) => `${index}: ${segment}`); lines.push(`At offset 3: ${segments.containing(3).segment}`); lines.push(`UTF-16 length: ${text.length}`); lines.join('\n'); """).AsString);
dotnet run prints:
0: A 1: 👩💻 6: é 8: ! At offset 3: 👩💻 UTF-16 length: 9
segments.containing takes a UTF-16 offset, not a grapheme number; segments.containing(3) finds the emoji even though it started at offset 1. An offset outside the string returns undefined.
Variations
Use word and keep isWordLike segments to extract words from this English example. Replace the JavaScript inside Evaluate with the code below. It prints Hi | world.
const words = new Intl.Segmenter('en', { granularity: 'word' }); [...words.segment('Hi, world!')] .filter(part => part.isWordLike) .map(part => part.segment).join(' | ');
Related recipes
- Sort JavaScript strings by locale in C#
- Normalize JavaScript Unicode text in C#
- Format JavaScript dates by locale and time zone in C#
The short answer
Run Intl.Segmenter inside OneEcma’s ScriptEngine. The default granularity is grapheme, set explicitly in the full program. Iterate the result of .segment(text) to get each grapheme and its start offset. The first snippet returns two graphemes: a woman technologist emoji made of several code points and e followed by an acute accent.
What is a grapheme cluster in JavaScript?
A grapheme cluster is a sequence of one or more code points treated as a unit by the default rules in Unicode Standard Annex #29. For example, the emoji above contains two supplementary code points joined by a zero-width joiner. Iterating code points instead of UTF-16 units still doesn’t keep that sequence together. OneEcma’s segmentation rules include “GB11 emoji ZWJ sequences and GB12/13 regional-indicator pairs,” covering the joiner example and regional-indicator flag pairs.
For a preview that shows only the first few characters, take whole graphemes. Truncating by code units can split a surrogate pair; truncating by code points can still separate an accent from its letter. Segmenting the input doesn’t rewrite or normalize it. The accent in the output is still a separate code point, just part of the same returned segment.
What does Intl.Segmenter return as the index?
The index is the segment’s start in UTF-16 code units. The full program’s first segment A uses one unit. The next segment, 👩💻, uses five: two for each supplementary code point and one for the joiner. That’s why the next segment begins at 6, even though it’s the third grapheme. In C#, use these offsets with the same unchanged input string; C# strings are also indexed by UTF-16 units. Intl.Segmenter on MDN describes the iterable interface and its granularity options.
Can Intl.Segmenter split Thai or Japanese text into words?
Not with dictionary boundaries in OneEcma. Grapheme and word boundaries answer different questions: a Thai cluster can be suitable for cursor movement without being a whole word. The text services docs say “Dictionary segmentation (Han, Thai, Lao, Khmer, Burmese, and Japanese words) is not implemented.” Use word with isWordLike for cases like the English variation; don’t assume that requesting a locale makes it a dictionary tokenizer.