---
title: "Models: tokenizer, embedder contract and the default embedder"
url: https://recursiveai.org/oneindex/docs/models/
description: "Package RecursiveAI.OneIndex.Models. It depends on core and on RecursiveAI.OneModel only."
version: "0.4.0-preview.2"
source: https://github.com/recursive-ai-oss/oneindex-csharp/blob/177f49bb3974e9e44df6d1a800342a9d353a0984/docs/models.md
---

# Models: tokenizer, embedder contract and the default embedder

Package `RecursiveAI.OneIndex.Models`. It depends on core and on `RecursiveAI.OneModel` only.

## `IEmbedder` (core, namespace `OneIndex.Vectors`)

- `ModelId`: a stable identity of the model and every setting that changes its output (weights, quantization, prefix, truncation). Two
  embedders with the same id give the same vectors for the same text and role, so the id is safe in an embedding cache key.
- `Dimension`, `IsNormalized` (unit-length vectors: the dot product is cosine similarity).
- `Embed(texts, role, destination, ct)` and `EmbedAsync(...)`: row-major `texts.Count * Dimension` floats written to a caller buffer.
  `EmbeddingRole.Query` or `Document` selects the query instruction some models need. Texts longer than the model's limit are cut,
  never rejected; the empty string is valid; `null` throws; the destination length must match exactly.
- Implementations are thread-safe. The sparse, multi-vector and reranking contracts are siblings of the same shape.

## WordPiece tokenizer (`OneIndex.Models.Tokenization`)

Self-contained: no reference to the engine or to OneModel, only the BCL (and an `ICanonicalDecomposer` for accent stripping, which
`OneIndex.Models.UnicodeDecomposer` implements from OneIndex's own Unicode tables; the runtime's own `Normalize` does nothing for
non-ASCII text when globalization is invariant, so the default fallback is only for hosts that have ICU).

Sources: Devlin et al. 2019 (BERT), Wu et al. 2016 (WordPiece), the BERT README (cleaning, CJK and punctuation rules), the Hugging Face
`tokenizers` documentation. The reference implementation's source was not read; its output was used as a black box to settle open
points (below).

Per stretch of text between special tokens:

1. **Special tokens.** Literal `[PAD]`, `[UNK]`, `[CLS]`, `[SEP]`, `[MASK]` in the input map to their ids before any normalization
   (`SplitOnSpecialTokens`, on by default, as in the reference; turn it off for untrusted text).
2. **Cleaning.** U+0000, U+FFFD and every character of category Cc, Cf, Co or Cs are dropped, except tab, line feed and carriage return.
   (So vertical tab, form feed and U+0085 are dropped, not word breaks. Unassigned characters are kept.) Remaining whitespace
   (`White_Space`) separates words. An unpaired surrogate reads as U+FFFD and is dropped.
3. **CJK.** Each ideograph in U+4E00..9FFF, 3400..4DBF, 20000..2A6DF, 2A700..2B81F, 2B920..2CEAF, F900..FAFF, 2F800..2FA1F is a word of
   its own. The reference leaves U+2B820..2B91F out although the BERT README lists the whole block; that range is reproduced as observed.
4. **Accents and case.** With `StripAccents` (default: follows `LowerCase`), the word is decomposed (NFD) and nonspacing marks (Mn)
   are dropped; then it is lower-cased (simple case mapping, except U+0130, which gives `i` and U+0307).
5. **Punctuation.** Each ASCII non-alphanumeric (33..47, 58..64, 91..96, 123..126) and each character of category P* is a word.
6. **WordPiece.** Greedy longest match from the left; continuation pieces carry `##`. A word of more than 100 characters, or one that
   cannot be cut completely, is one `[UNK]`.

Truncation keeps the first `maxLength - 2` tokens between `[CLS]` and `[SEP]`; a shorter limit gives a prefix of a longer one. Work and
memory per call are bounded by the longest word, and encoding stops when the output is full.

### Conformance

Token ids against the Hugging Face `tokenizers` package (0.23.3, run in a `uv` venv, on the model's own `tokenizer.json`):

- All 1,112,064 Unicode scalar values (surrogates excluded), each as `Ab<c>Cd` and alone: 2,223,033 of 2,224,128 encodings identical.
  The 1,095 other encodings belong to 618 scalar values, listed in `testdata/models/wordpiece-reference-differences.txt`; the test
  requires exactly that set. 614 are characters that the reference does not know (it treats them as ordinary unknown characters,
  while the runtime's Unicode data classifies them as marks, punctuation or format characters); four, U+166D, U+1734, U+111C9 and
  U+1171E, have a different general category in the two data versions (the reference splits U+166D as punctuation and drops
  U+1734 and U+1171E as marks, which the runtime does not).
- 12,547 sampled texts (adversarial strings, random strings over awkward alphabets, every SciFact, ArguAna and SCIDOCS query,
  2,000 passages of each): all identical.

## Default embedder: bge-small-en-v1.5

BAAI/bge-small-en-v1.5, MIT licence, revision `5c38ec7c405ec4b44b94cc5a9bb96e735b38267a`: 12 layers, 384 dimensions, 33M parameters.

- Pooling and normalization as the model card: the first token's hidden state, scaled to unit length (floor 1e-12).
- Queries get `Represent this sentence for searching relevant passages: `; documents get nothing. At most 512 tokens per text.
- Run by OneModel in its default (fast) mode. `ExactArithmetic` selects its exact mode.
- Batching: texts are tokenized, sorted by length and run in batches under a token budget (2,048 padded tokens by default); lengths
  are padded to a multiple of 64 and row counts rounded up to a power of two, so the model sees few distinct shapes (each shape is a
  separate plan inside OneModel and costs memory).

### Weights

`tools/models/prepare_bge_small.py` downloads the published ONNX at the pinned revision (SHA-256 checked), and writes:

- `model.onnx`: fp32 as published (133 MB as OneModel file).
- `model.int8.onnx`: **shipped.** The 72 matrix-multiply weights and the word-embedding table are stored as int8 with one scale per
  output channel (per row for the table); the graph dequantizes with `DequantizeLinear` on constants, which OneModel folds when it plans.
  Activations stay float32, so a text's vector does not depend on the other texts in its batch. 34.9 MB as OneModel file.
- `model.int8dyn.onnx`: comparison only. ONNX Runtime's dynamic quantization (int8 weights and per-tensor int8 activations). Not
  shipped: a text's vector then depends on what else is in the batch (cosine 0.995 between batched and single runs in our test).

`tools/OneIndex.Models.Tool convert` turns each into a OneModel file with `ModelFormat.Convert`; `tools/models/pack-weights.sh` places
the result where `OneIndex.Models.csproj` embeds it. The weights are never committed.

## Measurements

All with the packed model files, query instruction on, exact dot-product retrieval over the whole corpus (document text is the title,
a space, then the text), nDCG@10 under the BEIR convention (a document with the query's own id is skipped), cross-checked with
`pytrec_eval` (`tools/models/crosscheck_run.py`; SciFact 0.7126 on both). Run on the x64 worker (AVX-512 kernels), 8 threads, loaded
machine. The published figures are the `ndcg_at_10` values in the model card's metadata (MTEB results).

| Set | Published | fp32 weights | int8 weights (shipped) | Difference to published |
|---|---|---|---|---|
| SciFact (5,183 passages, 300 queries) | 0.71275 | 0.7127 | 0.7126 | -0.0 |
| ArguAna (8,674, 1,406) | 0.59550 | 0.6035 | 0.6046 | +0.8 / +0.9 points |
| SCIDOCS (25,657, 1,000) | 0.20522 | not run | 0.2041 | -0.1 points |

### NQ (held out from the hybrid tuning)

NQ (BEIR, 2,681,468 passages, 3,452 test queries; data hashes: corpus.jsonl f7e3588d..., queries 72f08f0c..., qrels 6df0cd2c...) cannot be embedded at the CPU rate of the shipped int8 path, so the document encodings come from the **reference fp32 models on a GPU** (RTX 4070, PyTorch, the revisions `.Models` ships; `tools/models/gpu_reference.py`). Everything else (indexing, BM25F, exact kNN, sparse scoring, fusion, nDCG) is our engine. A stand-in proof (below) shows the shipped int8 files agree with those encodings, which is what lets the NQ rows count for the int8 path. GPU numbers are marked as such.

| Set | Published | Ours (GPU fp32 vectors, exact kNN over all passages) | Difference |
|---|---|---|---|
| NQ (2,681,468 passages, 3,452 queries) | 0.50178 | 0.5018 | +0.0 |

`pytrec_eval` on the run file gives 0.5018. Encoding: 1,680 passages/s on the 4070 (about 23 minutes for the corpus), against 15 to 25 passages/s for the int8 path on the CPU.

**Stand-in proof, dense** (random 20,000 passages with the 3,452 queries, int8 OneModel vs the GPU fp32 vectors of the same texts): cosine minimum 0.9988 (passages) and 0.9993 (queries); nDCG@10 on the 20,000-passage sub-corpus 0.9284 with the int8 vectors against 0.9285 with the fp32 ones; top-10 overlap 0.974. So the NQ figure above stands for the shipped weights within 0.1 point; the dense `.Models` gate (within 1 point of published) is met on NQ.

Quantization effect on the vectors themselves, 800 texts (adversarial strings, queries and passages) against ONNX Runtime fp32 run one
text at a time (cosine similarity of unit vectors; documents / queries with the instruction):

| Weights | cosine min | cosine mean | max abs diff | Memory (peak RSS, 4 threads) |
|---|---|---|---|---|
| fp32 | 1.000000 / 1.000000 | 1.000000 / 1.000000 | < 1e-5 | 0.69 GB |
| int8 weights (shipped) | 0.99394 / 0.99855 | 0.99978 / 0.99977 | 0.017 / 0.012 | 1.07 GB |
| int8 dynamic (comparison) | 0.97204 / 0.96137 | 0.99690 / 0.99640 | 0.034 / 0.041 | |

Why int8 weights are the default: retrieval quality is the same as fp32 within noise on both sets (SciFact -0.0001, ArguAna +0.001),
the embedder's weights are 35 MB (the fp32 model file alone is 133 MB; the whole `RecursiveAI.OneIndex.Models` package, which also carries the reranker and the sparse encoder, is 134 MB), and a vector does not depend on its batch. Memory at run time is higher than fp32
because OneModel expands the int8 weights to float32 when it plans each input shape; the shapes are few (see Batching) but each plan
holds its own copy. Throughput is about the same as fp32 (the arithmetic is float32 either way). `BgeSmallEmbedder.Open` takes
any other file of the same model, for example the fp32 one.

Throughput, for information (worker, 8 threads, other jobs running, so not a benchmark): 15 to 17 passages per second on SciFact
(about 300 tokens each), 22 to 25 on ArguAna.

The tokenizer is fuzzed (`fuzz/run.sh wordpiece`): 600 s, 3,767,141 executions, 2,912 coverage features, corpus 503 inputs, no
findings.

## Cross-encoder: ms-marco-MiniLM-L6-v2

The default reranker (`MiniLmReranker`; the contract, the stage and the pair encoding are in `rerank.md`). cross-encoder/ms-marco-MiniLM-L6-v2, Apache-2.0,
revision `233902d25c440f23af6f7d6e94d2946bac0bee0a`, trained on the MS MARCO passage ranking set (we did not train it; the data is not included). Weights
are prepared like the embedder's (`prepare_cross_encoder.py`, `gate.sh setup-reranker`, `pack-weights.sh reranker`): fp32 as published (91 MB as OneModel file),
weight-only int8 shipped (24 MB). They stay loadable from files (`MiniLmReranker.Open`).

### Conformance

- **Pair token ids** against the Hugging Face `tokenizers` package (0.23.3, black box, the model's `tokenizer.json`, truncation `longest_first` at 512), 4,528 pairs:
  27 adversarial pairs (empty texts, special-token text, accents, CJK, emoji, lengths around the truncation boundary on both sides) and 1,500 per BEIR set
  (SciFact, ArguAna, SCIDOCS) of a test query with a judged passage (half) or a random passage. **4,528 matched ids and segment ids; 0 differences**, 647 of them at the
  512 limit. Two differences found on the way and fixed: which text is cut (the shorter keeps up to half the room, the longer the rest) and which counts as longer when
  equal (the second).
- **Scores** against ONNX Runtime 1.31.1 fp32, one pair at a time, graph optimizations off, 600 pairs (the 27 adversarial and every 7th other; reference logits -11.5 to 7.3):

| Weights | max abs diff | mean abs diff | Spearman |
|---|---|---|---|
| fp32 | 0.00001 | 0.00000 | 1.00000 |
| int8 (shipped) | 0.143 | 0.019 | 0.99998 |

  Scoring all 600 pairs batched by query against scoring them one at a time: largest difference 0.00000.

### BEIR reproduction

BM25F top-100 (the relevance gate's first stage: English analyzer, title weight 3, k1 0.9, b 0.4), each hit read as title, a space, text, reranked by the shipped int8 model
through the model tool (`gate.sh rerank-beir`: the same ordering rule as the stage: best score first, ties in first-stage order). nDCG@10 under the BEIR convention (a hit with the
query's own id is skipped), cross-checked with `pytrec_eval` on the run files (equal to four decimals). The published figures are the **BEIR paper's Table 2, BM25+CE column**
(Thakur et al. 2021): a 6-layer, 384-hidden MiniLM cross-encoder reranking the top 100 of Anserini BM25, linked there as cross-encoder/ms-marco-MiniLM-L-6-v2 (the hyphenated
repository name; the model card of the repository used here lists no BEIR figures). The paper's first stage is Anserini BM25, ours is BM25F, and its ArguAna numbers predate the
identical-id convention, so the last column drops no hit.

| Set | Published BM25+CE | BM25F first stage | Reranked (shipped int8) | Difference | Reranked, own-id hit kept |
|---|---|---|---|---|---|
| SciFact | 0.688 | 0.6912 | 0.6890 | +0.1 point | 0.6890 |
| ArguAna | 0.311 | 0.4333 | 0.4164 | +10.5 points under the BEIR convention | 0.3112 (+0.0) |
| SCIDOCS | 0.166 | 0.1531 | 0.1646 | -0.1 point | 0.1646 |

ArguAna's gap is the convention, not the model: its queries are documents of the corpus, so the paper counted the query's own copy as a (never relevant) top hit; the same run scored
that way gives 0.3112 against the published 0.311 (first stage 0.3253 against the paper's BM25 0.315). All three sets are within 1 point of the published figures. NQ (published 0.533)
is below.

**NQ** (3,452 queries, BM25F top-100 from the 2.68-million-passage index, 345,101 pairs scored by the reference fp32 cross-encoder on the GPU in 310 s, 1,113 pairs/s): first stage 0.3419, reranked **0.5404** against the published 0.533 (+0.7 point; `pytrec_eval` 0.5404). Stand-in proof: 60 queries x 100 pairs scored by the shipped int8 model vs the GPU fp32 scores: score difference max 0.1572, mean 0.0231, Spearman 0.99998; nDCG@10 of those 60 queries reranked 0.5993 (int8) vs 0.5971 (fp32), top-10 overlap 0.9917. The rerank gate (within 1 point) is met on NQ for the shipped weights.

Information, SciFact: the default hybrid list (docs/hybrid.md; 0.7257) reranked gives 0.6894. On this set the cross-encoder does not improve on the hybrid first stage.

Throughput, for information: 22 to 25 (query, passage) pairs per second with 8 threads on a shared x64 machine (AVX-512 kernels), 100 passages of 150 to 400 tokens per query.
The model tool's rate on pairs is the limit of the BEIR runs above (about 45 minutes for SciFact, five hours for ArguAna).

## SentencePiece tokenizer (`OneIndex.Models.Tokenization`)

Self-contained like WordPiece (BCL only, no reference to the engine or to OneModel), so it can move to a shared tokenizer library. Unigram
models only: a BPE model is reported as a `SentencePieceFormatException`.

Sources: Kudo, "Subword Regularization" (ACL 2018) for the unigram model and its Viterbi segmentation; Kudo and Richardson, "SentencePiece"
(EMNLP 2018 demo); the `sentencepiece_model.proto` schema as a description of the file format. The reference implementation's source was
not read; its output was used as a black box.

- **Model file:** a protobuf (`ModelProto`) read by a small reader that checks every length against the remaining bytes and
  `SentencePieceLimits` (file size, piece count, piece length); malformed input raises `SentencePieceFormatException`. The `dictionary`
  fuzz target covers it.
- **Normalizer:** `nmt_nfkc`. The model file carries the rule set only as a compiled character map, so the table (4,837 single code
  points and 14,116 sequences, `Tokenization/nmt_nfkc.tsv`) was dumped by running the released normalizer
  (`tools/models/spm_normalizer_dump.py`) over the Unicode scalar values and the sequences its map contains; Hangul jamo composition is
  algorithmic. The table is data observed from the reference's output.
- **Segmentation:** Viterbi search over the piece scores, the first-found path winning ties.

### Conformance

Token ids against `sentencepiece` 0.2.1 (run in a `uv` venv on the model's own `sentencepiece.bpe.model`) over 7,971 texts
(`tools/models/spm_sample.py`, seed 20261103): the Universal Dependencies Japanese, Chinese, Korean and Thai sentences, phrases from
the Snowball vocabularies of 34 languages, edge cases (empty, whitespace, control characters, long runs, emoji, combining sequences)
and random strings over awkward scripts.

- **7,967 identical** (99.95 %); the normalized strings are identical in all 7,971.
- **4 differ, each an exact tie.** All four contain U+2034 (a triple prime), which normalizes to three U+2032; the text then has two
  segmentations with the same total score and the reference picks the other one (we keep the first found; trying "last found" and
  accumulating in double precision gave no better result). Both are valid unigram segmentations.
- Against the model's fast `tokenizer.json` (a different implementation of the same model) 116 of the 7,971 texts differ from the
  `sentencepiece` ids, so "identical to the fast tokenizer" is not a claim we can make; `sentencepiece` is the reference here.

## Multilingual embedder: multilingual-e5-small (`RecursiveAI.OneIndex.Models.Multilingual`)

intfloat/multilingual-e5-small, MIT licence, revision `614241f622f53c4eeff9890bdc4f31cfecc418b3`: 12 layers, 384 dimensions, around 100
languages, with a 250,000-piece embedding table. It is a separate package because the int8 weights are 120 MB (the package is 112 MB; the English default is
35 MB); it depends on `RecursiveAI.OneIndex.Models` and OneModel only.

- Prefixes as the model card says: `query: ` for queries, `passage: ` for documents, both added by role. Mean pooling over the real
  tokens, then scaled to unit length. At most 512 tokens per text.
- Token ids follow the model's numbering: the boundaries are 0 and 2, the padding id is 1, unknown 3, and every other piece id is shifted
  up by one (the fairseq convention of XLM-RoBERTa).
- `MultilingualE5Embedder.Load()` opens the packed int8 weights; `Open(model, sentencepiece, variant)` loads files. Run by OneModel in
  its default mode, batched like the English embedder.
- Weights: `tools/models/prepare_e5_small.py` downloads the published ONNX (SHA-256 checked) and quantizes it (int8 per output channel,
  the table per row); `tools/models/pack-weights.sh e5` places the converted file where the project embeds it. As OneModel files, fp32
  is 470,429,956 bytes and int8 120,038,276.

### Parity with ONNX Runtime

800 texts (with the `passage: ` prefix) from the tokenizer sample: ONNX Runtime 1.31.1 fp32 on the ids of the model's `tokenizer.json`
against our embedder (AVX-512 kernels).

| | cosine min | cosine 1st percentile | cosine median | cosine mean | max abs difference |
|---|---|---|---|---|---|
| fp32 through OneModel | 0.954092 | 0.998780 | 1.000000 | 0.999811 | 0.05686 |
| int8 (shipped) | 0.953278 | 0.998596 | 0.999871 | 0.999666 | 0.05774 |

The low minimums come from the few texts where the fast tokenizer the reference used and `sentencepiece` (which we match) segment
differently, not from the arithmetic: the median is 1.0.

### BEIR reproduction

Dense retrieval over each set's corpus with the shipped int8 weights on CPU (flat vector search, `query: ` and `passage: ` prefixes,
nDCG@10 under the BEIR convention), against the model card's MTEB figures (the `model-index` of its README, nDCG@10 in %):

| Set | Model card | OneIndex int8 | Difference |
|---|---|---|---|
| SciFact | 67.704 | 67.82 | +0.1 point |
| ArguAna | 39.063 | 38.59 | -0.5 point |
| SCIDOCS | 13.895 | 13.75 | -0.1 point |

MIRACL and NQ were not run: MIRACL needs multilingual corpora of millions of passages and NQ is 2.68 million passages, at 15 to 30
passages per second on CPU. ArguAna's queries are documents of the corpus: the BEIR convention (used here) skips a hit whose id equals
the query's, while the model card's MTEB figure may count that copy as a hit, so the two numbers are not computed the same way. We did
not run the other convention for this model, so the split of the 0.5 point between convention, int8 weights and the tokenizer
differences is not measured.