---
title: "Word frequency file, version 1"
url: https://recursiveai.org/oneocr/docs/word-lm/
description: "A file holds how often words occurred in running text. The recognizer's word segmentation (oneocr-page-v1.md, \"Word segmentation\") uses it to take back a…"
version: "1.0.0"
source: https://github.com/recursive-ai-oss/oneocr-csharp/blob/8db430fb77c4d86d51686f15dfd9b5ccc20c99a7/docs/spec/word-lm-v1.md
---

# Word frequency file, version 1

A file holds how often words occurred in running text. The recognizer's word segmentation (`oneocr-page-v1.md`, "Word segmentation")
uses it to take back a space the network put inside a word. It is produced by `training/wordlm/build_wordlm.py` and read by
`WordFrequencyModel`. All integers are little-endian. Another implementation must reproduce the lookup and the comparison below
exactly: they use integers only.

## What the table holds

Words of Latin letters only, each lowercased by the simple one-to-one lowercase mapping of its code points (a character without one,
and U+0130, stay as they are), as in `char-lm-v1.md`. A word with a digit, an apostrophe, a hyphen, a code symbol or a letter of
another script is not in the table. A word is 1 to 40 UTF-8 bytes. The builder reads the Latin-script books marked `train` in
`training/data/texts.yaml` (the books the character language model is built from; Alice and Pride and Prejudice, the evaluation
page sets' texts, and the held-out books are not among them) and keeps words seen at least `min_count` times (default 3). It can add
counts from the visible text of public-domain US federal web pages (`training/wordlm/README.md`); the file format does not change.

A count is stored as `q = clamp(round(4 * log2(count)), 1, 255)`, a quarter of a bit: the count is about `2^(q/4)`. `q` is never 0.

## Layout

```
magic        4 bytes  "OCWF"
version      u16      1
flags        u16      0 (a reader rejects any other value)
total        u32      the number of word tokens the counts come from
words        u32      the number of words W in the table
q_total      u8       clamp(round(4 * log2(total)), 1, 255)
q_missing    u8       the q taken for a word that is not in the table
blocks       u32      ceil(W / 16)
offsets      u32 x blocks   byte offset of each block in the word data
word data:
  entries, W of them, in ascending byte order of the word (strictly increasing):
    shared   u8       bytes the word shares with the previous entry's word (0 for the first entry of a block)
    suffix   u8       bytes that follow; shared + suffix is 1 to 40
    bytes    suffix bytes of UTF-8
    q        u8       1 to 255
```

Blocks hold 16 entries (the last holds the rest); block `i` starts at `offsets[i]`, which is where block `i - 1` ended (the first
at 0). A reader checks all of this while it parses: lengths stay inside the word and the file, words strictly rise, no bytes
remain after the last entry. The builder sets `q_missing = max(1, q(min_count) - 4)`, a word seen about half as often as the
rarest word in the table.

## Lookup

Binary search over the blocks on the first word of each (the last block whose first word is not above the target), then a scan of
that block, rebuilding each word from the previous one's prefix. Words compare as unsigned bytes, a prefix before its extensions.

## Use

The page reader's word segmentation (`oneocr-page-v1.md`, "Word segmentation", rule 3) takes two neighbouring pieces `a` and `b`
(lowercased, letters only) with joined word `ab`, and compares, in quarter-bits,

    gain = q(ab) + q_total - q(a) - q(b)        (q of a word that is not in the table is q_missing)

which is four times the base-2 log-odds that the letters are one word and not two independent words. All comparisons are on
integers, so every platform decides alike. The bundled table holds 219,000 words from the Latin-script books of 15 languages and modern web text, and is 1.3 MB.
It knows nothing of numbers, codes, names or words with an apostrophe or hyphen, and such words are never joined with it.