Asian-language analysis (RecursiveAI.OneIndex.Analysis.Asian)

Japanese, Chinese, Korean, Thai, Lao, Khmer and Myanmar write words without spaces (Korean with spaces, but with agglutinated endings). The package segments them with packed dictionaries and plugs into the analyzer pipeline of core (docs/analysis.md). It references only core and the BCL.

AsianAnalyzers.Register();                       // once at start-up: makes "ja", "zh", "ko", "th", "lo", "km", "my" known by name
[FullText(Analyzer = "ja")] public string Body { get; set; }

Registration is explicit (AnalyzerRegistry.Register(name, factory), added to core for this package); nothing is discovered by reflection. A schema stores the analyzer name, so an index opens only after the call. The analyzers are Analyzer subclasses (JapaneseAnalyzer, ChineseAnalyzer, KoreanAnalyzer, WordBreakAnalyzer.Thai() ...); each is its tokenizer, NFKC and case folding.

Common to every tokenizer: the text is cut into spans of the target script, which a dictionary segmenter splits; everything else (Latin words, digits, other scripts) goes to the UAX #29 word tokenizer, so "iPhone 15" inside Japanese text stays whole. Tokens keep the offsets of the original text. Function words that are dropped leave a position gap, so phrase queries do not match across them.

Dictionaries and sizes

Dictionaries are data in one container format, OIDICT version 1 (magic, version, sections, CRC-32C footer, DictionaryLimits on every length; malformed files raise DictionaryFormatException, TryLoad beside it). They are built by tools/dictionaries/fetch.sh and build.sh from the sources below and embedded when present at build time. Any dictionary can also be loaded from a file (LatticeLexicon.Load, WordList.Load).

Total about 14.4 MB (under the 50 MB at which a separate package would be proposed). Notices are in THIRD-PARTY-NOTICES.md.

Segmenters

• Lattice (ja, ko): every dictionary word starting at each position is a node, unknown-word nodes come from the character classes of the dictionary, and the path with the lowest sum of word costs and connection costs (left/right context ids) is found by Viterbi search, as in Kudo, Yamamoto and Matsumoto, "Applying Conditional Random Fields to Japanese Morphological Analysis" (EMNLP 2004). The dictionaries' costs are used as published. Japanese search mode adds a length penalty to long Han and Katakana words (after the published description of search-mode segmentation: threshold 2 and penalty 3000 for Han, threshold 7 and penalty 10000 for Katakana; untuned beyond the F1 check below) so compounds are split. Japanese tokens are the base form of inflected words; particles, auxiliary verbs and symbols are dropped (part-of-speech prefixes 助詞, 助動詞, 記号). Korean yields the content morphemes (a verb's stem, not its endings) and a compound noun is the compound followed by its parts at the same position; the classes dropped are the endings, particles, suffixes and symbols (tags beginning with E, J, SC, SE, SF, SP, SSO, SSC, SY, VCP, VCN, XPN, XSA, XSN, XSV).
• Unigram (zh): the segmentation maximising the product of word probabilities from the frequency list (dynamic programming over the word graph); unknown characters stand alone (single characters). Search mode also emits the dictionary parts of long words at the same position.
• Maximal matching (th, lo, km, my): dynamic programming from the end of the text picks the path with the fewest characters left unmatched, then the fewest words, then the longer first word. A boundary never falls before a combining mark or a spacing vowel sign, and unmatched characters are grouped into one token.

Evaluation

Word-span precision, recall and F1 (a predicted word counts when its span equals a gold span) against Universal Dependencies treebanks, which are evaluation data only: fetched on demand, never committed, CC BY-SA licensed (Thai-PUD: CC BY-SA 3.0). Run with oneindex-dictionaries eval <lang> <dictionary> <treebank.conllu>.

Differences to explain:

• Japanese: the treebank follows UniDic short-unit words and IPADIC's units are longer (compound nouns, noun plus suffix), so some of the gap is the two standards disagreeing, not errors.
• Chinese: jieba's list has longer words than the treebank (企业界, 博物馆), so recall is lower than precision. The traditional-character treebank (Chinese-GSD) scores F1 0.5747 because the list is simplified-Chinese; traditional text is not covered by this dictionary.
• Thai: the segmenter predicts more words than the treebank has (precision below recall): the ICU list is a break dictionary with many short entries, so names and compounds the treebank keeps whole are cut. It is not a gold standard of words.
• Korean: the Korean treebank's tokens are eojeol (space-delimited units), not morphemes, so it cannot score a morphological segmenter; no morpheme-level gold with a usable licence was found. Korean is checked by tests on stems and compounds (for example the inflected 갔습니다 gives the stem 가). Lao, Khmer and Myanmar: no gold-segmented set was found. The shared segmenter is unit-tested on a synthetic list (and shared with Thai, which is scored above); the three packed analyzers get a smoke test (no tokens from blank text) and no accuracy figure. Treat them as dictionary breaking, not as evaluated segmentation.

Speed (information)

Segmentation only (the segmenter over text, not the whole analyzer), single thread, one process, warmed up, three runs per language; oneindex-dictionaries speed <lang> <dictionary> <text>, indicative (a shared 16-core AMD Ryzen 9 7950X3D, load about 11 before, during and after; .NET 10.0, x64). Text: UD sentences for ja, ko, zh and th, repeated to about 3 million characters; for lo, km and my random strings of dictionary words (no real text was available), so those three show the cost of matching, not of real prose.

There is no comparator row: the other segmenters were not run (no managed engine's segmenter is in the benchmark set).

Limits and safety

Dictionary readers validate the magic, version, every section length and the checksum before using an offset; sizes and counts are bounded by DictionaryLimits; a hostile file raises DictionaryFormatException. The dictionary and asian fuzz targets cover the readers, the segmenters and the analyzers (docs in fuzz/README.md). Segmenters allocate per call only their lattice buffers, reused by the tokenizer between texts. The package works in the browser: nothing in it depends on threads, native code or reflection.

Not included

Readings (furigana), UniDic's short-unit and long-unit forms, user dictionaries (load another .oidict for that), traditional-Chinese words, Cantonese, Vietnamese (written with spaces) and language detection (the schema names the analyzer).