Sentence embeddings

IEmbedder (core) turns text into a fixed-length vector; SentenceEmbedder (RecursiveAI.OneNlp.Neural) runs a sentence-transformers-format model on OneModel. Nlp.Embed is the happy path over the default model in RecursiveAI.OneNlp.Models.Embeddings.

float[] v = Nlp.Embed("how do I reset my password");          // needs RecursiveAI.OneNlp.Models.Embeddings
using SentenceEmbedder e = SentenceEmbedder.Load("my-model-dir");   // any sentence-transformers export
float[][] batch = e.EmbedBatch(texts);                              // sorted by length internally, results in input order
foreach (float[] x in e.EmbedMany(lazyTexts)) { }                   // streaming, in batches of e.BatchSize

Multilingual: RecursiveAI.OneNlp.Models.Embeddings.Multilingual

intfloat/multilingual-e5-small (MIT, about 100 languages, 384 dimensions, mean pooling). Its ONNX file is 470 MB, over the NuGet size limit, so it is cut into three equal byte ranges carried by ...Embeddings.Multilingual.Shard1, .Shard2 and .Shard3; the thin package (tokenizer, configuration, registration, the API below) depends on all three. The ranges are joined in memory by a read-only seekable stream and handed to the loader, with no temporary file and no joined copy on disk (tools/embeddings/make_shards.sh cuts the file and prints the manifest rows).

• Default for Nlp.Embed. The multilingual package registers with a higher priority than the English one, so when it is referenced Nlp.Embed uses it, also when RecursiveAI.OneNlp.Models.Embeddings (bge) is referenced too. To use bge anyway, call EmbeddingsModel.Load() from the English package and keep the embedder.
• Prefixes. e5 models were trained with query: in front of a search query (and of both texts of a similarity task) and passage: in front of a passage. Nlp.Embed and E5Embedder.Embed use query:, which is right for similarity. For retrieval, MultilingualEmbeddingsModel.Load() returns an E5Embedder with EmbedQuery, EmbedPassage and their batch forms EmbedQueries, EmbedPassages. The helpers live on the model, not on Nlp.
• Checks. Parity with sentence-transformers on the joined shards: see the table below. The consumer application (JIT and NativeAOT) works with the multilingual package alone and with both packages.

Model directory

SentenceEmbedder.Load(dir, modelFile = "model.onnx", prefix = "") reads, as data, what sentence-transformers publishes:

All files are bounded (1 MiB each for the JSON files), the pooling folder must lie inside the model directory, and a damaged file throws InvalidDataException. Export a PyTorch model to ONNX with tools/accurate/export_onnx.py (first output = last hidden states). Load(Stream, Stream, EmbeddingConfig) takes weights from memory or an embedded resource.

Behaviour

• Pooling follows Reimers and Gurevych 2019 (Sentence-BERT) §3 over the tokens the attention mask keeps. Normalization scales to unit L2 length (a floor of 1e-12 on the norm) when modules.json has a Normalize module.
• Truncation: inputs are cut to max_seq_length tokens including the special tokens, keeping the closing special token. Longer text is never an error.
• Prefix: prefix is put in front of every input (E5-family models need query: or passage:). The default is none, which is what sentence-transformers does.
• Batching: a batch is sorted longest first and run in groups of BatchSize (default 32); padding never changes a text's vector beyond float rounding. Calls are serialized per embedder.
• int8 (opt-in, never the default): pass an int8-quantized ONNX file of the same model as modelFile. tools/embeddings/quantize_int8.py makes one with ONNX Runtime's dynamic quantization. OneModel runs the integer operators exactly. Deltas are below.

SentencePiece models and white space

For models whose tokenizer has a lone ▁ piece (XLM-R and other SentencePiece Unigram vocabularies) sentence-transformers 6.2.0 (transformers 5.22.1) does not give the same pieces as the bare tokenizer.json pipeline in a few white-space cases. Observed in its output with multilingual-e5-small and reproduced here:

1. Characters the normalizer maps to a space are collapsed to one space (a<U+2003><U+2003>b is a b).
2. White space at the end of the text, or before a special token written in the text (<s>, </s>, <pad>, <unk>), gives no ▁ piece. White space after a special token is kept.
3. U+0085 (next line) acts as a space.

SentenceEmbedder applies these three rules for such tokenizers; WordPiece models are unaffected. Without them 91 of 2,103 parity texts differed in token ids (all texts containing trailing or exotic white space or special-token look-alikes); with them none do.

Parity with sentence-transformers

Reference: sentence-transformers 6.2.0, torch 2.15.0+cpu, CPU, float32. Inputs: 2,103 texts from the multilingual parity corpus (all scripts of the Universal Declaration of Human Rights, Tatoeba sentences, and the synthetic classes: CJK, emoji and ZWJ, combining marks, whitespace runs, special-token look-alikes, URLs and code; 62 to 65 of them longer than 512 tokens). Selection is deterministic (tools/embeddings/parity_reference.py, seed in the script). Token ids come from SentenceTransformer.tokenize.

The vectors match to float tolerance (differences are float32 summation order). int8 against the float reference, for information: bge-small-en-v1.5 max abs diff 0.124, min cosine 0.848, mean cosine 0.966 on this deliberately hostile set.

MTEB

Harness: mteb 2.27.1, run black-box through mteb.evaluate with a small custom model class (tools/embeddings/mteb_run.py); the same class wraps sentence-transformers for the same-harness reference, so the two columns differ only in who computes the vectors. Scores are main scores x 100. "Published" is the model card's model-index (the metric each task type uses as its main score; for the multilingual tasks the mean over the subsets that were run). Prompts: bge-small-en-v1.5 uses its retrieval query instruction on SciFact queries and nothing elsewhere; multilingual-e5-small uses query: for queries and symmetric tasks and passage: for documents. Multilingual runs cover STS22 in en, de, es, fr, it, tr, zh and the cross-lingual pairs among them, and MTOPDomain in en, de, es, fr.

bge-small-en-v1.5

multilingual-e5-small

Reading the tables:

• Ours equals sentence-transformers in the same harness on every task to two decimals (the vectors agree to 1e-7, so the scores cannot differ).
• Published numbers: STS, pair classification, clustering, retrieval and (for bge) reranking reproduce within 0.5. The bold rows do not, and they do not for sentence-transformers either. See the next section for what was tried.
• int8 is smaller (bge 133 MB to 34 MB) and about as fast; quality drops 0.3 to 3.8 points depending on the task (clustering and retrieval most). It is information only; fp32 is what ships.

Card-era harness (mteb 1.x), for the cells that miss

The cells that miss under mteb 2.27.1 were re-run, ours and sentence-transformers 6.2.0 side by side, under two 1.x releases of mteb (each in its own venv; tools/embeddings/mteb1_run.py, mteb1_chunk.sh): 1.1.1 (the first release with these tasks, the closest to the era of the cards) and 1.12.94 (the last 1.12 release), both with datasets 3.2.0 and scikit-learn 1.9.1. The cards state no mteb version, so "the version that produced the card" is not knowable from the card; the datasets' revisions in the cards equal the ones these releases use for Banking77. Per-subset scores are in the result files. Under 1.1.1 STS22 and MTOPDomain would not load (the pinned dataset revisions are loader scripts, which were not run); they were run under 1.12.94 only.

• STS22 now reproduces (66.10 against 66.01): the per-subset scores agree with the card to about 0.1 on most subsets; the 2.x difference came from the harness. Ours equals sentence-transformers on every cell and release (STS22 differs by 7e-5 in the mean, from one subset).
• The other four cells do not move with the mteb release (identical to four decimals across 1.1.1, 1.12.94 and 2.27.1 for the three classification and reranking cells). The harness release is therefore not the cause of the gap, and these cells stay below the card for sentence-transformers itself. MTOPDomain per subset, ours (ST identical): en 88.99, de 86.15, es 85.53, fr 81.50 against the card's 91.07, 87.47, 89.27, 84.23. Not investigated further: the card's numbers cannot be reproduced from the published model with the released harness, whoever computes the vectors.

Speed (indicative)

One run on the shared worker (AMD Ryzen 9 7950X3D, 4 threads for both sides, load average about 35 to 41 during the run, so treat as a rough guide), 204 English sentences of 30 to 250 characters, tokenization included, the same exported model. Reference: ONNX Runtime 1.31.1 with tokenizers and numpy pooling. There is no managed comparator, so this row passes with that reason recorded; it is not a gate.

Reproducing

tools/embeddings/: setup_venv.sh (sentence-transformers, mteb, ONNX Runtime), fetch_model.py and tools/accurate/export_onnx.py (pinned revision to ONNX), parity.sh, make_shards.sh, mteb_chunk.sh, mteb1_chunk.sh (venvs: mteb 1.1.1 or 1.12.94 with pyarrow==18.1.0 datasets==3.2.0), speed_ort.py, quantize_int8.py, and the OneNlp.EmbedTool console project that embeds a text file or serves vectors over a pipe.