# oneindex docs, all pages (OneIndex 0.4.0-preview.2) Every page of https://recursiveai.org/oneindex/docs/ and https://recursiveai.org/oneindex/cookbook/ as markdown, getting started first. Index: https://recursiveai.org/oneindex/llms.txt # Getting started with OneIndex Install OneIndex 0.4.0-preview.2 for .NET 10, index a few objects and search them, in C# with no server and no native code. OneIndex is an embedded search engine: full-text (BM25F), vector (HNSW) and hybrid search in one in-process library. ## Before you use it - **A preview.** In the README’s words: “Everything in the table below works and is tested; APIs and file formats may still change between previews.” Its [overview](https://recursiveai.org/oneindex/docs/overview/) lists what exists and the known gaps. - **Durability, as the project tests it.** A commit that returns survives a crash or `kill -9` of your process, and a damaged file fails the open with an error naming it instead of returning wrong results. The project’s own testing ran about 890,000 fault-injection schedules and more than 7,000 process kills with no failed open and no lost commit. Its limits: “On a power cut the index may roll back to an earlier commit, but it is never corrupt”, and the kill and fault-injection campaigns run on Linux and macOS only; Windows runs the full test suite on every change. Network and FUSE file systems are not supported. [Durability](https://recursiveai.org/oneindex/docs/durability/). - **Speed and relevance** against Lucene.NET are measured in [benchmarks](https://recursiveai.org/oneindex/docs/benchmarks/) and [relevance](https://recursiveai.org/oneindex/docs/relevance/), with their conditions. ## Install The packages target `net10.0`, so you need the .NET 10 SDK, and `dotnet add package` needs `--prerelease` or the exact version. ```shell dotnet new console -o OneIndexStart && cd OneIndexStart dotnet add package RecursiveAI.OneIndex --version 0.4.0-preview.2 ``` Nine packages share one version: | Package | What it is | |---|---| | `RecursiveAI.OneIndex` | the engine and the happy path; no dependencies beyond .NET | | `RecursiveAI.OneIndex.Models` | bundled embedder (bge-small-en-v1.5), reranker and sparse encoder on OneModel; a 134 MB package | | `RecursiveAI.OneIndex.Models.Multilingual` | the multilingual-e5-small embedder, around 100 languages; a 112 MB package | | `RecursiveAI.OneIndex.Analysis.Asian` | Japanese, Chinese, Korean, Thai, Lao, Khmer and Myanmar segmentation | | `RecursiveAI.OneIndex.EntityFrameworkCore` | keeps an index in step with `SaveChanges` | | `RecursiveAI.OneIndex.VectorData` | a vector store for `Microsoft.Extensions.VectorData` | | `RecursiveAI.OneIndex.Files`, `.Files.Ocr` | folder indexing with citations; OCR for scans | | `RecursiveAI.OneIndex.Cli` | the `oneindex` tool: `index`, `query`, `static build` | ## A first program Put this in `Program.cs` and run `dotnet run`: ```csharp using OneIndex; // a directory; SearchIndex.InMemory() keeps it in memory using var index = SearchIndex.Open("data/search"); index.Upsert(new Invoice { Id = "inv-1", Title = "Overdue notice", Body = "Payment from Acme Corp is overdue.", Modified = DateTime.UtcNow, Tags = ["invoice"] }); index.Upsert(new Invoice { Id = "inv-2", Title = "Draft: overdue reminder", Body = "Acme Corp draft, not sent.", Modified = DateTime.UtcNow, Tags = ["invoice", "draft"] }); index.Commit(); // atomic and durable var cutoff = DateTime.UtcNow.AddYears(-1); // what a user types: words, "phrases", -exclusions var results = index.Search("overdue \"acme corp\" -draft") .Where(x => x.Tags.Contains("invoice") && x.Modified > cutoff) .Take(10); foreach (var hit in results) Console.WriteLine($"{hit.Id} {hit.Score:F2} {hit.Document.Title}"); Console.WriteLine($"{index.Count} documents"); public class Invoice { [SearchId] public string Id { get; set; } = ""; [FullText(Boost = 2)] public string Title { get; set; } = ""; [FullText(Language = "en")] public string Body { get; set; } = ""; [Filterable, Sortable] public DateTime Modified { get; set; } [Keyword, Facet] public string[] Tags { get; set; } = []; } ``` ```text inv-1 0.69 Overdue notice 2 documents ``` - A source generator inside the package writes the mapping from the attributes at build time, with no reflection, so it is NativeAOT- and trim-safe. Supported types, diagnostics and highlights: [mapping](https://recursiveai.org/oneindex/docs/mapping/). - `Search(text)` takes what a user types and never throws, whatever is typed: [user text](https://recursiveai.org/oneindex/docs/user-text/). Every word not joined by `OR` must match, typos are forgiven, and the last word also matches as a prefix. - `Where` is translated to filters and never run; anything it cannot translate throws and names the part: [LINQ filters](https://recursiveai.org/oneindex/docs/linq/). - `Commit` writes and syncs every new file, then publishes the commit with one atomic rename. Searches see the last commit. One writer per directory; reader processes are fine. - `TotalHits` is a lower bound unless you add `CountExactly()`. ## Next - [Full-text search in C#](https://recursiveai.org/oneindex/cookbook/full-text-search-csharp/): typos, facets and highlights. - [Vector search in .NET](https://recursiveai.org/oneindex/cookbook/vector-search-dotnet/) and [hybrid search with reranking](https://recursiveai.org/oneindex/cookbook/hybrid-search-reranking-csharp/), with the bundled models. - [Search over EF Core entities](https://recursiveai.org/oneindex/cookbook/search-ef-core-entities/) and [a folder of files with citations](https://recursiveai.org/oneindex/cookbook/search-folder-files-citations-csharp/). - Coming from Lucene.NET: [the migration recipe](https://recursiveai.org/oneindex/cookbook/migrate-from-lucene-net/). - Every topic: [the docs](https://recursiveai.org/oneindex/docs/), starting with [architecture](https://recursiveai.org/oneindex/docs/architecture/). # OneIndex **An embedded search engine for .NET: SQLite for search.** Full-text (BM25F), vector (HNSW) and hybrid search in one in-process library, written in C# with no native code and no server. It runs from a directory, from memory, or in the browser from a static bundle fetched over plain HTTP. OneIndex is unaffiliated with other products named OneIndex, OneSearch or similar. > **Status: preview, on nuget.org as prerelease packages** (`0.4.0-preview.1` and later; add `--prerelease` or a version when installing). Everything in the table below works and is tested; APIs and file formats may still change between previews. Known gaps are listed in the table and in "Known gaps", not hidden. ## What exists | Feature | State | |---|---| | Storage interface (range reads, sync and async); memory, local-file, read-only HTTP and browser backends; fault-injecting test double | Working, tested ([docs/storage.md](https://recursiveai.org/oneindex/docs/storage/)) | | Segments, commits (fsync, then atomic rename), background merges, reader processes on committed snapshots, near-real-time readers | Working. | | Analysis: UAX #29 words and sentences (Unicode 17.0.0), NFC/NFKC, case and accent folding, Snowball stemmers (34 languages) and stop words, CJK bigrams | Working, conformance-tested ([docs/analysis.md](https://recursiveai.org/oneindex/docs/analysis/)) | | Term dictionary, postings, block-max WAND, BM25F; term, boolean, phrase, prefix, fuzzy, regex, range queries | Working, measured ([docs/benchmarks.md](https://recursiveai.org/oneindex/docs/benchmarks/)) | | More-like-this (from a document or a text; LINQ, query JSON) | Working ([docs/more-like-this.md](https://recursiveai.org/oneindex/docs/more-like-this/)) | | Typo tolerance, as-you-type, "did you mean", synonyms, stop words | Working ([docs/typo-tolerance.md](https://recursiveai.org/oneindex/docs/typo-tolerance/)) | | Columns, filters, facets with value search, aggregations, distinct, geo, sort and search-after, function scores, ranking rules, explain | Working | | `[Dynamic]` fields and schema evolution from stored source | Working ([docs/schema-evolution.md](https://recursiveai.org/oneindex/docs/schema-evolution/)) | | Strict query syntax, filter strings, versioned query JSON, `SearchTool.For()`, the LINQ builder (comparisons, `null`, `Contains`, `Any`) | Working, fuzzed ([docs/linq.md](https://recursiveai.org/oneindex/docs/linq/)) | | Vectors: HNSW and exact flat search, int8, int4 and sign-bit codecs with float rerank, filtered traversal, graph merge, learned sparse vectors, rank fusion, reranking, auto-embedding with an embedding cache | Working ([docs/vectors.md](https://recursiveai.org/oneindex/docs/vectors/), [docs/hybrid.md](https://recursiveai.org/oneindex/docs/hybrid/)) | | Multi-vector fields (`[MultiVector]`, encoded at upsert with `From`) with MaxSim as a query stage, a rerank and an exact scan | Working ([docs/multi-vector.md](https://recursiveai.org/oneindex/docs/multi-vector/)) | | `RecursiveAI.OneIndex.Models`: WordPiece tokenizer, bge-small-en-v1.5 embedder (int8), cross-encoder reranker, learned sparse encoder | Working ([docs/models.md](https://recursiveai.org/oneindex/docs/models/)) | | `RecursiveAI.OneIndex.Analysis.Asian` (ja, zh, ko, th, lo, km, my), decompounding (de, nl, sv, da, no, fi, is), `RecursiveAI.OneIndex.Models.Multilingual` (multilingual-e5-small, SentencePiece) | Working | | `RecursiveAI.OneIndex.EntityFrameworkCore`: transactional outbox, applier, pushdown with over-fetch, `ReindexAsync` | Working on SQLite and PostgreSQL ([docs/entity-framework.md](https://recursiveai.org/oneindex/docs/entity-framework/)). | | `RecursiveAI.OneIndex.Files` / `.Files.Ocr`: folder indexing with citations | Working ([docs/files.md](https://recursiveai.org/oneindex/docs/files/)) | | `RecursiveAI.OneIndex.VectorData`: a vector store over an index | Working; the abstraction's conformance suite passes except the listed unsupported cases ([docs/vectordata.md](https://recursiveai.org/oneindex/docs/vectordata/)) | | `oneindex` command line tool | Working ([docs/cli.md](https://recursiveai.org/oneindex/docs/cli/)) | | Browser: memory and OPFS storage, static index over plain HTTP, IVF | Working; the static index's first query is heavy next to Pagefind's ([docs/static-index.md](https://recursiveai.org/oneindex/docs/static-index/), [docs/browser.md](https://recursiveai.org/oneindex/docs/browser/)) | ### Known gaps - **Entity Framework:** `ExecuteUpdate`, `ExecuteDelete` and raw SQL bypass `SaveChanges`, so they are not indexed; repair with `ReindexAsync(query)`. A log row that becomes visible after the applier gave up on its position is applied at the entity's next change or by `ReindexAsync`. SQL Server and MySQL have not been run ([docs/entity-framework.md](https://recursiveai.org/oneindex/docs/entity-framework/)). - **VectorData:** dynamic (dictionary) collections, store-side embedding generation, Manhattan and Hamming distance and whole-array equality filters are not supported, so the conformance tests that need them skip themselves with a reason naming the feature ([docs/vectordata.md](https://recursiveai.org/oneindex/docs/vectordata/)). - **Static index against Pagefind:** a first query costs about 0.65 MB, or 0.74 MB with the manifest (Pagefind: 0.32 MB) and a later text query about 194 KB in 48 small requests (Pagefind: 46 KB in 4); the stored-source blocks and term-dictionary pages set that floor. Static hybrid uses IVF candidates, so its results are close to, not identical with, the server's graph-based hybrid (top-10 overlap 0.89 at the default). The .NET WebAssembly payload is 5.35 MB raw (1.57 MB brotli) against Pagefind's 575 KB. IVF is also a per-field server option (`[Vector(Index = VectorIndexKind.Ivf)]`), 3 to 8 times slower than HNSW at equal recall. ### Relevance (BM25, nDCG@10, BEIR test sets; measured 2026-11) OneIndex at its defaults (user text: BM25F, title weight 3, k1=0.9, b=0.4, proximity bonus 0.1) against the published BM25 baseline (Kamalloo et al., SIGIR 2024, Table 3, multifield) and Lucene.NET 4.8.0-beta00018 on the same runs. Method and all layouts: [docs/relevance.md](https://recursiveai.org/oneindex/docs/relevance/). | Set | OneIndex | Published | Lucene.NET, flat | Lucene.NET, per-field sum | |---|---|---|---|---| | SciFact | 0.6918 | 0.665 | 0.6789 | 0.6647 | | ArguAna | 0.4347 | 0.414 | 0.4118 | 0.4240 | | NQ | 0.3452 | 0.329 | 0.3058 | 0.3252 | | SCIDOCS | 0.1544 | 0.158 | 0.1509 | 0.1587 | Hybrid search (BM25 with bge-small-en-v1.5, rank fusion at its defaults) averages 0.517 over SciFact, ArguAna, SCIDOCS and NQ, against 0.506 for the dense arm alone and 0.405 for BM25, and beats both arms on every set ([docs/hybrid.md](https://recursiveai.org/oneindex/docs/hybrid/)). BM25F scores a document as one weighted stream, so its like-for-like comparison is Lucene.NET's flat layout. On SCIDOCS, summing per-field scores does better (0.1587), but the same layout costs 3.1 points on SciFact and 1.0 on NQ when OneIndex runs it. ### Speed Measured 2026-11 on x64 (AMD Ryzen 9 7950X3D) and arm64 (Apple M2), with peak memory and index size beside each row in [docs/benchmarks.md](https://recursiveai.org/oneindex/docs/benchmarks/): - **Text, against Lucene.NET 4.8.0-beta00018** with matched analysis and commits (302,567 Wikipedia abstracts; 17 rows: indexing, commit, 1-, 2- and 5-term queries, phrase, prefix, as-you-type, fuzzy, typo, filtered, facets, stats, sort, function score). OneIndex's time is 0.62 of Lucene.NET's on x64 and 0.41 on arm64 as a geometric mean. The slowest rows are phrase (1.17x on x64) and function score (1.14x on x64). - **Vectors, against Hnsw.Net 0.1.4** at matched recall@10 of 0.90, 0.95 and 0.99 (0.98 at 384 dimensions, the highest both reach), at 384 and 768 dimensions. OneIndex has 1.19x the queries per second on x64 and 1.15x on arm64 (geometric mean). Build time is 0.94-0.95x on x64 and 0.62-0.75x on arm64. - Native engines (Tantivy, SQLite FTS5, hnswlib, FAISS) are listed for information. Tantivy is 2-3x faster on plain term and phrase queries. ### Durability and the EF Core integration **Durable by design.** A commit that returns survives a crash or `kill -9` of your process. A reader sees one whole commit. Every file is checksummed, and damage fails the open with an error that names the file instead of returning wrong results. We test this with fault injection at every storage operation (about 890,000 schedules on x64 and arm64 for the 2026-11 build) and with more than 7,000 process kills during writes, merges, schema changes and multi-process reading, with no failed open and no lost commit. On a power cut the index may roll back to an earlier commit, but it is never corrupt. See [docs/durability.md](https://recursiveai.org/oneindex/docs/durability/). **EF Core integration.** Changes made through `SaveChanges` reach the index through a transactional outbox log and are applied once. After catch-up the index equals the database, including across kills of the app and of the applier. Verified on SQLite (1,276 kills) and PostgreSQL 16 (1,809 kills, with concurrent writers); SQL Server and MySQL are not tested. A transaction left open longer than the late-row window or age is applied with the entity's next change or `ReindexAsync`. See [docs/entity-framework.md](https://recursiveai.org/oneindex/docs/entity-framework/). ## Getting started (preview) Mark the properties of an ordinary class; a source generator inside the package writes the mapping at build time (no reflection, so it is NativeAOT- and trim-safe): ```csharp using OneIndex; public class Invoice { [SearchId] public string Id { get; set; } = ""; [FullText(Boost = 2)] public string Title { get; set; } = ""; [FullText(Language = "en")] public string Body { get; set; } = ""; [Filterable, Sortable] public DateTime Modified { get; set; } [Keyword, Facet] public string[] Tags { get; set; } = []; } using var index = SearchIndex.Open("data/search"); // or SearchIndex.InMemory() index.Upsert(new Invoice { Id = "inv-1", Title = "Overdue notice", Body = "Payment from Acme Corp is overdue.", Modified = DateTime.UtcNow, Tags = ["invoice"] }); index.Commit(); // atomic and durable; searches see the last commit var cutoff = DateTime.UtcNow.AddYears(-1); var results = index.Search("overdue \"acme corp\" -draft") // what a user types: words, "phrases", -exclusions, OR .Where(x => x.Tags.Contains("invoice") && x.Modified > cutoff) .Take(10); foreach (var hit in results) Console.WriteLine($"{hit.Id} {hit.Score:F2} {hit.Document.Title}"); ``` - User text never throws, whatever is typed: [docs/user-text.md](https://recursiveai.org/oneindex/docs/user-text/). - `Where` is translated to filters and never run; anything it cannot translate throws and names the part: [docs/linq.md](https://recursiveai.org/oneindex/docs/linq/). - Attributes, supported types, compile-time diagnostics, highlights: [docs/mapping.md](https://recursiveai.org/oneindex/docs/mapping/). - `TotalHits` is a lower bound unless you add `CountExactly()`. - Typo tolerance, facets, sorting, geo, vectors and hybrid search: see [docs/](https://github.com/recursive-ai-oss/oneindex-csharp/tree/177f49bb3974e9e44df6d1a800342a9d353a0984/docs) (start with [docs/architecture.md](https://recursiveai.org/oneindex/docs/architecture/)). ## Packages | Package | Contents | |---|---| | `RecursiveAI.OneIndex` | The engine and the happy path (no dependencies beyond .NET) | | `RecursiveAI.OneIndex.Models` | Embedders, reranker and sparse encoder on OneModel | | `RecursiveAI.OneIndex.Analysis.Asian` | Japanese, Chinese, Korean and Southeast Asian segmentation | | `RecursiveAI.OneIndex.Models.Multilingual` | multilingual-e5-small embedder with SentencePiece tokenizer and int8 weights | | `RecursiveAI.OneIndex.EntityFrameworkCore` | Keeps an index in step with `SaveChanges` | | `RecursiveAI.OneIndex.VectorData` | A vector store over an index for Microsoft.Extensions.VectorData | | `RecursiveAI.OneIndex.Files` / `.Files.Ocr` | Folder indexing through OnePDF, onehtml, onearchive and oneocr | | `RecursiveAI.OneIndex.Cli` | The `oneindex` command (`index`, `query`, `static build`) | All share one version. ## Build dotnet build -c Release dotnet test --solution OneIndex.slnx -c Release Build output goes outside the checkout: set `ONEINDEX_ARTIFACTS` to choose where (the default is a folder in your local application data). The .NET SDK in `global.json` is required. ## Layout - `src/OneIndex`: the core library. `Storage`, `Index`, `Analysis` and `Search` are public layers, each built only on the ones before it; the happy path lives in the root namespace. - `src/OneIndex.Generators`: the source generator that ships inside the core package. - `tests/`, `bench/`, `fuzz/`: unit tests, the durability, relevance and differential suites, smoke apps (NativeAOT and browser WebAssembly), benchmarks and fuzz targets. - `docs/`: normative notes on formats and protocols; the index of notes is [docs/README.md](https://github.com/recursive-ai-oss/oneindex-csharp/blob/177f49bb3974e9e44df6d1a800342a9d353a0984/docs/README.md), and the architecture note is [docs/architecture.md](https://recursiveai.org/oneindex/docs/architecture/). ## License MIT OR Apache-2.0, at your option. Copyright (c) Recursive AI, LLC. See [LICENSE-MIT](https://github.com/recursive-ai-oss/oneindex-csharp/tree/177f49bb3974e9e44df6d1a800342a9d353a0984/LICENSE-MIT) and [LICENSE-APACHE](https://github.com/recursive-ai-oss/oneindex-csharp/tree/177f49bb3974e9e44df6d1a800342a9d353a0984/LICENSE-APACHE). Third-party material is listed in [THIRD-PARTY-NOTICES.md](https://github.com/recursive-ai-oss/oneindex-csharp/blob/177f49bb3974e9e44df6d1a800342a9d353a0984/THIRD-PARTY-NOTICES.md). # Architecture OneIndex is one engine in layers. Each layer is public, in its own namespace, and is built only on the layers before it. | Layer | Namespace | Holds | |---|---|---| | Storage | `OneIndex.Storage` | The file interface, backends, the fault injector, the block-transform slot, corruption exceptions, limits. See [storage.md](https://recursiveai.org/oneindex/docs/storage/). | | Index | `OneIndex.Index` | The index file frame (header, footer, CRC-32C), segment and commit descriptions, the commit file and protocol, the codec registry, the schema and untyped documents. See [commit-format.md](https://recursiveai.org/oneindex/docs/commit-format/). | | Analysis | `OneIndex.Analysis` | The token source contract, analyzers and their registry. See [analysis.md](https://recursiveai.org/oneindex/docs/analysis/). | | Search | `OneIndex.Search` | Query objects, the segment-reader interfaces, scorers, collectors and the searcher. See [search-contracts.md](https://recursiveai.org/oneindex/docs/search-contracts/). | | Happy path | `OneIndex` | `SearchIndex`, `SearchTool` and the schema attributes. | Later layers (codecs, aggregations, vectors, query languages, LINQ, the static index) slot between and above these. ## Rules that shape the code - **No assumption of memory mapping or threads.** Readers ask storage for byte ranges into their own buffers. Work that can run in the background (merges, backfills) also runs cooperatively where there are no threads, with identical results. - **Immutable once written.** A segment never changes after it is flushed. A commit is the only way a reader learns about new segments, and it is published by an atomic rename after everything it names is durable. - **Every file is framed and checksummed.** Readers validate magic, version and checksum before they trust an offset, and reject anything malformed with a typed exception that names the file. One `IndexLimits` object travels with every reader. - **Deterministic bytes.** Nothing persisted depends on hash codes, dictionary order or the clock. The same input gives the same file. - **Explicit registries.** Codecs and analyzers are looked up in plain lists in the code, so trimming and ahead-of-time compilation keep exactly what is named. Nothing is discovered by reflection. - **Sync and async twins** on every operation that does I/O, each taking a cancellation token. ## Ownership of the contracts The segment reader (`ISegmentReader`) is deliberately split by part so that independent pieces of the engine do not meet in the same file: the term dictionary and postings implement `ITextFieldReader`, columns implement `IColumnReader`, the point index `IPointIndexReader`, stored source `IStoredFieldsReader`, and delete bitmaps `ILiveDocs`. Queries are plain data; a visitor turns them into scorers, so adding execution for a query kind never edits the query classes. # Mapping documents A document type is an ordinary class whose properties carry attributes. A source generator (it runs at build time inside the `RecursiveAI.OneIndex` package) writes, for each such class, a `SearchMapping`: the schema, and the conversion between the class and an untyped `Document`. Nothing is found by reflection at run time, so mapping is NativeAOT- and trim-safe. The generated mapping registers itself from a module initializer of the assembly that defines the type. ```csharp public class Invoice { [SearchId] public string Id { get; set; } [FullText(Boost = 2)] public string Title { get; set; } [FullText(Language = "en")] public string Body { get; set; } [Filterable, Sortable] public DateTime Modified { get; set; } [Keyword, Facet] public string[] Tags { get; set; } } ``` ## Attributes | Attribute | On | Effect | |---|---|---| | `[SearchId]` | one `string` | The document's identifier: a keyword field, unique. Upsert replaces the document with the same id; `Delete(id)` removes it. Required to open an index. | | `[FullText(Boost, Language, Analyzer)]` | `string`, `string[]`, `List`, `IEnumerable`, ... | Analyzed text searched by user text and scored with BM25F. `Boost` (default 1, finite, above 0) is the field's weight. `Language` (`"en"`, ...) picks the language analyzer; `Analyzer` names a registered analyzer (`"cjk"`, `"en-both"`) and wins over `Language`. Default analyzer: `standard`. | | `[Keyword]` | the same types | The whole value is matched exactly (tags, codes). Filterable with `==` and `Contains`. A collection holds several values. | | `[Filterable]` | numbers, dates, `bool`, strings | Queries can filter on it. On a string it implies `[Keyword]`. | | `[Sortable]` | single-valued members | Kept column-wise for sorting with `OrderBy` and friends ([sorting.md](https://recursiveai.org/oneindex/docs/sorting/)). | | `[Geo]` | one `GeoPoint` (or `GeoPoint?`) | Indexed and kept in a column for radius and bounding-box filters and distance sorting ([geo.md](https://recursiveai.org/oneindex/docs/geo/)). | | `[Facet]` | keyword and value members | Kept for counts of values (`Facets(x => x.Tags)`, [facets.md](https://recursiveai.org/oneindex/docs/facets/)); implies filterable. | | `[Dynamic]` | a `System.Text.Json.JsonElement` (or `JsonElement?`) | A JSON value whose leaf paths are indexed on the fly and can be filtered by path; the whole value is stored and comes back unchanged. See dynamic.md. | | `[Vector(Dimensions, Metric, From)]` | `float[]` | A dense vector searched by `SearchVector`, `Hybrid` and `KnnQuery` ([vectors.md](https://recursiveai.org/oneindex/docs/vectors/)). `From = nameof(Text)` embeds that text member automatically ([auto-embedding.md](https://recursiveai.org/oneindex/docs/auto-embedding/)); then `Dimensions` may be left out and the embedder's dimension is used (an index opened without an embedder throws). An empty array means no vector was supplied. | Supported member types: `GeoPoint` (with `[Geo]`); `string`; `string[]`, `List`, `IEnumerable`, `IReadOnlyList`, `IReadOnlyCollection`, `IList`, `ICollection`; `sbyte`, `byte`, `short`, `ushort`, `int`, `uint`, `long`; `float`, `double`; `bool`; `DateTime`, `DateTimeOffset`; and the nullable forms of the value types (a null is simply absent). A `JsonElement` property is only mapped when it is marked `[Dynamic]`; `[Dynamic]` on any other type is OI0001. Public read/write properties of supported types **without** attributes are stored and returned with the document but cannot be searched or filtered. Read-only properties and properties of other types are not part of the document and come back with their constructor defaults. The type must be a non-generic, non-abstract class, reachable from its assembly (not private), with a public or internal parameterless constructor; the attributed properties need a getter and a plain `set` (not `init`, not `required`). ## Diagnostics | Id | Meaning | |---|---| | OI0001 | An attributed member has a type the attribute cannot map. | | OI0002 | Reserved for an attribute that is declared but not supported; no current attribute raises it. | | OI0003 | Conflicting attributes (`[FullText]` with `[Keyword]`, or with a filtering attribute; `[Sortable]` on a collection; `[Dynamic]` with any other search attribute). | | OI0004 | A bad identifier: several `[SearchId]`, or one that is not a plain string. | | OI0005 | The type cannot be mapped (generic, abstract, private, no usable constructor). | | OI0006 | A member cannot be mapped (a field, no getter or setter, `init`, `required`). | | OI0007 | A bad attribute value (`Boost` not finite or not above 0). | Checked when an index is opened instead, because the generator cannot know them: a type without `[SearchId]` (`InvalidOperationException`), and a `Language` or `Analyzer` that no analyzer is registered under. ## Fields and the schema A field is named like its property. `[FullText]` strings are text fields with positions; keywords and the identifier are keyword fields (column-wise); numbers, dates and booleans are indexed as ordered values when filterable. Opening an existing directory with a type whose schema differs from the one in the index fails: changing a type needs schema evolution, which is not built yet. ## The index `SearchIndex.Open(path)` and `SearchIndex.InMemory()` take no options: everything comes from the attributes. `Upsert`, `Delete` and `Commit` (each with an async twin) go to the single writer; **searches see the last commit only**. A commit is atomic and durable (see [commit-format.md](https://recursiveai.org/oneindex/docs/commit-format/)). Disposing discards uncommitted changes. The index is thread-safe. A search in flight keeps the commit it started on, even if a newer commit is published meanwhile. **Stored source** is on by default (see [stored-source.md](https://recursiveai.org/oneindex/docs/stored-source/)): a hit's `Document` is rebuilt from the compressed source in the segment, so a directory index reopened later returns its documents. ## Highlights `hit.Highlights` maps a property name to fragments of its stored text, computed on first use. A `Fragment` has the plain `Text`, its `Offset` in the field value and the `Matches` (offset and length inside the fragment); `Marked(open, close)` wraps matches (default ``/``, no escaping). - Fields without a match are absent. For a multi-valued field, values are searched in order. - Fragments are cut around the first match not yet shown: about `FragmentLength / 4` characters before it, about `FragmentLength` long in total (default **160**), widened to whole words, never inside a surrogate pair. The first match is always inside its fragment, matches that start inside it are kept whole, and the next fragment starts at the next match beyond. At most `MaxFragments` (default **3**) per field. `Search(...).Highlight(new HighlightOptions { ... })` changes both. - Matching re-analyzes the stored text with the field's analyzer (so stems and case folding match, and CJK bigrams are located) and uses the original offsets, so marks are on the text as written. Excluded operands, range filters and keyword filters mark nothing. # Reads like Each layer of OneIndex is shaped after a .NET API that readers already know. This page puts them side by side. Every OneIndex snippet here is compiled and run by the sample tests (`samples/OneIndex.Samples`: `HappyPath.cs`, `EfCore.cs`, `ReadsLike.cs`, `Migration.cs`). ## The happy path reads like EF Core An EF query and a OneIndex query have the same grammar: a source, `Where` with a lambda, an ordering, a page, then run it. ```csharp // Entity Framework Core var list = await db.Invoices .Where(x => x.CustomerId == 7 && x.Amount > 100) .OrderBy(x => x.Amount) .Take(10) .ToListAsync(); // OneIndex var hits = await index.Search("overdue acme") .Where(x => x.Tags.Contains("invoice") && x.Amount > 100) .OrderBy(x => x.Amount) .Take(10) .ToListAsync(); ``` The lambdas are expression trees that OneIndex walks and never compiles; one it cannot translate fails naming the unsupported sub-expression. Entry points read like `DbContext`: `SearchIndex.Open(path)` and `SearchIndex.InMemory()`; `Upsert` and `Delete` buffer, `Commit()` is `SaveChanges()` (atomic and durable). Settings come from attributes on the class, as EF's data annotations do. With the Entity Framework package the two compose (the index finds, the database filters and loads): ```csharp var list = await db.Invoices.Search("overdue acme") .Where(x => x.CustomerId == 7 && x.Amount > 100) .Take(10) .ToListAsync(); ``` ## Readers read like `Utf8JsonReader` The low-level readers are forward-only and span-based, and do the work the caller asks for and no more. A term dictionary and its postings are walked the way a JSON token stream is: ```csharp // System.Text.Json var reader = new Utf8JsonReader(json); while (reader.Read()) { if (reader.TokenType == JsonTokenType.PropertyName && reader.ValueTextEquals("price"u8)) { reader.Read(); total += reader.GetDouble(); } } // OneIndex: the terms of a field with a prefix, and for each the documents that contain it ITermEnumerator terms = field.GetTerms("head"u8); while (terms.MoveNext()) { PostingsIterator postings = field.GetPostings(terms.Info, PostingsFeatures.Frequencies); while (postings.NextDoc() != DocIdSetIterator.NoMoreDocs) { occurrences += postings.Frequency; // postings.DocId is the document } } ``` `terms.Term` is a `ReadOnlySpan` (the term's UTF-8) valid until the next `MoveNext`, as `reader.ValueSpan` is until the next `Read`. Lookups come in the `Try*` shape the BCL uses: `field.TryGetTerm(term, out TermInfo info)`, `segment.TryFindId(id, out int docId)`, `snapshot.TryFindById(...)`, and `index.TryParse(json, out search, out error)` beside `Parse`. Where an exception is the right answer for damaged input it is typed (`CorruptIndexException`, naming the file). The segment readers are interfaces (`ITextFieldReader`, `IColumnReader`, ...), not `ref struct`s, because each codec supplies its own and a segment's reader must outlive a single stack frame; the *iterators* and the *spans* they return are the parts that behave like `Utf8JsonReader`. ## Vector kernels `System.Numerics.Tensors.TensorPrimitives` is the BCL's public home for vectorized math. OneIndex's distance kernels (`Kernels/VectorKernels.cs`, `QuantizedKernels.cs`) are **internal**: they are fixed-width `Vector128/256/512` loops with a scalar path, shaped for the engine's exact needs (normalized dot, int8/int4/bit codes), and have no public surface to compare. The public vector surface is the query: `KnnQuery`, `SearchVector(x => x.Embedding, vector)`, `Hybrid(...)`. Use `TensorPrimitives` in your own code when you need a distance outside an index. # Migrating from Lucene.NET This page maps the concepts and calls of Lucene.NET 4.8 (the `4.8.0-beta` line on NuGet) to OneIndex. It is written from Lucene.NET's public documentation and from black-box runs of the pinned release in `tools/OneIndex.Harness` (see [relevance.md](https://recursiveai.org/oneindex/docs/relevance/)); neither engine's source was consulted. There is **no compatibility layer**: OneIndex does not mimic Lucene's API, and it cannot open a Lucene index (see [Existing indexes](#existing-indexes)). The mapping tells you which OneIndex call replaces which Lucene.NET call, and where behaviour differs on purpose. Two entry points exist, and most code needs only the first: - **The happy path**: `SearchIndex.Open(path)` over an attributed class. It replaces the writer, reader, searcher, analyzer, query parser and field-by-field document building in one object. - **The low level**: `IndexWriter`, `IndexReader`, `IndexSnapshot`, `IndexSearcher`, query objects and collectors (namespaces `OneIndex.Index` and `OneIndex.Search`). This is the layer to read when you used Lucene.NET's classes directly. ## Concepts | Lucene.NET | OneIndex | Notes | |---|---|---| | `Directory` (`FSDirectory`, `RAMDirectory`) | `IStorage`: `FileSystemStorage`, `MemoryStorage`, `ReadOnlyStorage`, HTTP and OPFS backends | A directory is a range-read interface with sync and async methods. `SearchIndex.Open("path")` and `SearchIndex.InMemory()` build the storage for you. | | `IndexWriterConfig`, `IndexWriter` | `IndexWriterOptions`, `IndexWriter` (low level); `SearchIndex` (`Upsert`, `Delete`, `Commit`) | One writer per index, thread-safe. A second writer on the same directory is refused with `IndexLockedException`. | | `Document`, `Field`, `FieldType` | `Document` and `FieldDefinition` in a `Schema` (low level); attributes on a class (happy path) | Fields are declared once in a schema, not chosen per document. Types: text, keyword, integral, fractional, date, bool, geo point, vector, sparse vector, JSON. | | `TextField` | `[FullText]` / `FieldType.Text` | Analyzed, scored. | | `StringField` | `[Keyword]` / `FieldType.Keyword` | Matched whole; filterable; facetable with `[Facet]`. | | `Int32Field`, `Int64Field`, `DoubleField`, `NumericDocValuesField`, `SortedDocValuesField` | `[Filterable]`, `[Sortable]` on a number, date, bool or string | Indexed (for range filters) and kept as a column (for sort, facets and aggregations) by the same declaration. | | `StoredField` | stored source | The whole document is stored, compressed, unless the schema opts out. There is no per-field stored flag on the happy path. | | `IndexReader`, `DirectoryReader.Open`, `OpenIfChanged` | `IndexReader.Open`, `Refresh()`; `IndexSnapshot` | A snapshot is a committed generation, reference-counted; its files are not deleted while it is open. | | `SearcherManager`, near-real-time reader | `IndexWriter.OpenNearRealTimeReader()` | Opt-in; sees uncommitted changes. | | `IndexSearcher` | `IndexSearcher` (low level), `SearchIndex.Search(...)` | Create with `SnapshotSearch.CreateSearcher(snapshot)`. | | `TopDocs`, `ScoreDoc` | `TopDocs`, `ScoreDoc` (low level); `SearchResults`, `SearchHit` (happy path) | `TopDocs.TotalHitsExact` says whether the total is exact; `TotalHits` is a lower bound unless `CountExactly()` was asked for. | | `Analyzer`, `Tokenizer`, `TokenFilter` | `Analyzer`, `TokenSource` / `Tokenizer`, `TokenFilter`, `CustomAnalyzer` | Registered by name in a static `AnalyzerRegistry` (no discovery by reflection). | | `Similarity` (`DefaultSimilarity`, `BM25Similarity`) | BM25F (`Bm25Parameters` in `QueryEngineOptions`), always | Lucene.NET 4.8's default similarity is TF-IDF; OneIndex scores BM25F with k1 = 0.9, b = 0.4 by default. See [Scoring](#scoring). | | `Query` classes | records in `OneIndex.Search` | See the next table. | | `QueryParser`, `MultiFieldQueryParser` | `Search("text")` (never throws), `SearchStrict(...)` (positioned errors), filter strings, query JSON | See [Queries](#queries). | | `Sort`, `SortField` | `OrderBy`, `ThenBy`; `FieldSort`, `TopFieldCollector` | See [Sorting](#sorting). | | `FacetsCollector`, taxonomy and sorted-set facets | `Facets(x => x.Tags)`; `FacetCollector`; aggregations | No separate taxonomy index or facet configuration. | | `Highlighter`, `QueryScorer`, `SimpleHTMLFormatter` | `hit.Highlights`; `Highlighter`, `HighlightOptions` | Computed from the stored text on first use. | | `IndexSearcher.Explain` | `Explain()`; `IndexSearcher.Explain` | A tree of `Explanation` values. | | `GroupingSearch` | `Distinct(x => x.Field)` | Collapse to the best hit per value. | | `MoreLikeThis` | `MoreLikeThisQuery`, `index.MoreLikeThis(id)` | Selects the most distinctive terms of a text or stored document by tf-idf and searches them ([more-like-this.md](https://recursiveai.org/oneindex/docs/more-like-this/)). | | `TieredMergePolicy`, `ConcurrentMergeScheduler` | `TieredMergePolicy`, `ThreadedMergeScheduler` (`CooperativeMergeScheduler` where there are no threads) | Merges are throttled and results are identical either way. | | `Optimize` / `ForceMerge(n)` | `ForceMerge(n)` | | | `IndexWriter.Commit` | `Commit()` | Writes and fsyncs every new file, then publishes the commit by an atomic rename. A crash leaves the previous commit complete. | ### Query types | Lucene.NET | OneIndex | |---|---| | `TermQuery` | `TermQuery(field, term)` (the term is already analyzed) | | `BooleanQuery` + `Occur.MUST` / `SHOULD` / `MUST_NOT` | `BooleanQuery` + `Occur.Must` / `Should` / `MustNot`, and `Occur.Filter` for a clause that selects without scoring | | `PhraseQuery` (with `Slop`) | `PhraseQuery(field, terms) { Slop = n }` | | `PrefixQuery` | `PrefixQuery` (constant score by default) | | `FuzzyQuery` | `FuzzyQuery { MaxEdits = n }` (0 to 2, transposition counts as one edit) | | `RegexpQuery` | `RegexQuery` | | `WildcardQuery` | none; use `PrefixQuery` or `RegexQuery` | | `NumericRangeQuery`, `TermRangeQuery` | `RangeQuery(field, lower, upper)` with `IncludeLower` / `IncludeUpper` | | `MatchAllDocsQuery` | `MatchAllQuery` | | `ConstantScoreQuery`, filters | a `Filter` clause in a `BooleanQuery`; `Where(...)` | | `FunctionQuery`, `CustomScoreQuery` | `FunctionScoreQuery`, `ScoreBy(...)` (field value factor, decay, weights, delegates) | | `DisjunctionMaxQuery` | `MultiFieldTermQuery` (scores a term across fields as one stream) | | spatial strategies | `[Geo]`, `GeoRadiusQuery`, `GeoBoundingBoxQuery`, `OrderByDistance` | | none in 4.8 | `KnnQuery`, `SparseQuery`, `HybridQuery` (vectors, learned sparse, fusion) | | `SpanQuery` family, block joins, payloads | not in 1.0 (planned later, on evidence) | ### Queries `QueryParser` parses a syntax and throws `ParseException` on bad input. OneIndex has three tiers instead: - `index.Search(text)` takes what a person types: words, `"phrases"`, `-excluded`, `OR`, a typed last word as a prefix. It **never throws**; unbalanced quotes and stray operators are read as plain text ([user-text.md](https://recursiveai.org/oneindex/docs/user-text/)). - `index.SearchStrict(text)` is the opt-in parser with a field syntax (`title:acme`, `price:[10 TO 20]`, `"a b"~2`, `term~1`) and positioned `QueryParseException` errors ([query-syntax.md](https://recursiveai.org/oneindex/docs/query-syntax/)). Its grammar is its own and not Lucene's. - Filters: a LINQ predicate in `Where(x => ...)`, a Meilisearch-style filter string `Tags = invoice AND Price < 100` ([filter-strings.md](https://recursiveai.org/oneindex/docs/filter-strings/)), or versioned query JSON ([query-json.md](https://recursiveai.org/oneindex/docs/query-json/)). A query string is not analyzed by the query object. `TermQuery("title", "headphon")` takes the term as indexed; the happy path and the user-text compiler analyze with the field's analyzer for you. (With Lucene.NET you analyzed the text through the analyzer's token stream or the query parser; do the same with `Analyzer` when you build query objects by hand.) ### Sorting `new Sort(new SortField("price", SortFieldType.DOUBLE))` becomes `.OrderBy(x => x.Price)` (or `new FieldSort("price")` with a `TopFieldCollector`). Add `.ThenBy(...)`. A missing value goes last unless you pass `MissingValuePlacement.First`. Deep paging uses `SearchAfter(cursor)` with a `SortCursor` from the previous page (`results.NextCursor`) rather than `searcher.SearchAfter(lastDoc, ...)`; `Skip(n)` is limited to a 10,000-hit window. ## Analysis equivalences | Lucene.NET | OneIndex name | Difference | |---|---|---| | `StandardAnalyzer` | `standard` | Same word segmentation family (Unicode word boundaries; OneIndex follows UAX #29 revision 47 with a 255-character token limit). **`StandardAnalyzer` also removes the 33-word English stop list; `standard` removes none.** Configure the Lucene side without its stop filter for a like-for-like comparison, or add a `StopFilter`. | | `EnglishAnalyzer` | `en` | Both lower-case, drop possessives, remove English stop words and stem. They are not token-identical: Lucene.NET's `EnglishAnalyzer` uses its own 33-word list and the original Porter stemmer; `en` uses Snowball's English stop list and the Snowball English (Porter2) stemmer. Measured black-box on SCIDOCS texts, the token streams differ in 20 of 6,000 sampled texts when both sides use the same stop list (stems of short words), and 5,286 with each side's own list, almost all stop words ([relevance.md](https://recursiveai.org/oneindex/docs/relevance/)). The effect on nDCG@10 was at most 0.1 point. | | `SimpleAnalyzer`, `WhitespaceAnalyzer` | `whitespace-lowercase` (and `WhitespaceTokenizer`) | | | `KeywordAnalyzer` | `keyword` | | | `CJKAnalyzer` | `cjk` | Bigrams of Han, kana and Hangul runs; NFKC and case folding. | | `*Analyzer` per language | the language code (`de`, `fr`, `es`, ...; 34 Snowball languages) | `-both` indexes the word and its stem at the same position, so exact matches can rank above stem matches. | | `ASCIIFoldingFilter`, `ICUFoldingFilter` | `AsciiFoldFilter`, `AccentFoldFilter`, `CaseFoldFilter` (full Unicode case fold), `NormalizationFilter` (NFC/NFKC) | | | `SynonymFilter` | `[Synonyms]` and `[OneWaySynonyms]` on the document type | Applied at query time with the documented one-way and mutual forms ([synonyms.md](https://recursiveai.org/oneindex/docs/synonyms/)). | | `LengthFilter`, `StopFilter`, `LowerCaseFilter`, `PorterStemFilter` | `LengthFilter`, `StopFilter`, `LowerCaseFilter`, `StemFilter` | | | Japanese, Chinese, Korean analyzers | `ja`, `zh`, `ko` from `RecursiveAI.OneIndex.Analysis.Asian` | Dictionary data packed with notices. | | `CompoundWordTokenFilterBase` | `-compound` analyzers (da, de, fi, is, nl, no, sv) | | Custom analyzers: derive nothing; compose `new CustomAnalyzer(...)` from a tokenizer and filters, and register it with `AnalyzerRegistry.Register(name, factory)` so a schema can name it. ## Scoring - Lucene.NET 4.8's out-of-the-box similarity is classic TF-IDF; `BM25Similarity` is opt-in. OneIndex scores **BM25F** (Robertson and Zaragoza): each field's term frequency is length-normalized, weighted by the field's boost and summed *before* saturation, so a field boost is a weight on evidence, not a multiplier on a score. Scores are therefore not comparable across the two engines, and `Boost` values need re-tuning rather than copying. - BM25 itself matches: black-box `Explain` output of Lucene.NET's `BM25Similarity` shows the same idf `ln(1 + (N - n + 0.5)/(n + 0.5))` and tf saturation as OneIndex's `Explain`. OneIndex's defaults are k1 = 0.9, b = 0.4 (the BEIR lexical baseline); Lucene's `BM25Similarity` defaults are k1 = 1.2, b = 0.75. Set them with `QueryEngineOptions.Bm25` (a `Bm25Parameters`) at the low level if you want Lucene's. - A multi-field query in Lucene is usually a `BooleanQuery` of per-field queries (a sum of per-field scores) or a `DisjunctionMaxQuery`. OneIndex's layout is one weighted stream; the per-field-sum layout stays available through boolean query objects. Relevance on four BEIR sets for both layouts, and what each costs, is in [relevance.md](https://recursiveai.org/oneindex/docs/relevance/). - Scores are `float`s on the same scale only within one query on one index. `TotalHits` can be a lower bound (block-max WAND skips non-competitive blocks); call `CountExactly()` or `Count` when you need the exact number. ## Behaviour that differs - **No schema-less documents.** A field must be declared before it is used (`[Dynamic]` JSON fields are indexed by path for the open-ended part). Changing the schema is a managed evolution: additive changes and analyzer changes rebuild in the background from stored source, the old version serving until the swap ([schema-evolution.md](https://recursiveai.org/oneindex/docs/schema-evolution/)). - **Upsert is first-class.** `Upsert` replaces by id; `UpdateDocument(new Term("id", ...), doc)` is the Lucene equivalent. `AddUnique` refuses a duplicate id. - **Deletes** are by id (`Delete(id)`) or by exact term (`DeleteByTerm(field, value)`), visible after the next commit; deleted documents are skipped through a per-segment bitmap and removed by merges. There is no `DeleteDocuments(Query)`: select with a search and delete the ids. - **Commit** always means durable and atomic, and the same on every storage. There is no separate `Flush` visibility step to reason about; NRT readers are opt-in. - **Typed corruption errors.** A damaged file raises `CorruptIndexException` naming the file; a format from a newer version raises `UnsupportedFormatException`. Every reader runs under one `IndexLimits` object. - **Cancellation and async.** Every I/O entry point takes a `CancellationToken`, and sync and async twins exist at each layer. On local and memory storage the async twin completes synchronously. - **Vectors, hybrid and rerank** have no Lucene.NET 4.8 counterpart; see [vectors.md](https://recursiveai.org/oneindex/docs/vectors/) and [hybrid.md](https://recursiveai.org/oneindex/docs/hybrid/). - **No reflection and no codecs by name from config files.** Analyzers, codecs and query JSON nodes are static registries, so the library is trim- and NativeAOT-safe. ## Existing indexes OneIndex does not read Lucene indexes (the on-disk formats are unrelated), and no importer ships. Migrate by **reindexing from your source of truth**: read the documents from their database or files, and `Upsert` them in batches (`Upsert(IEnumerable)` batches embedding too) followed by one `Commit()`. If the original documents exist only in a Lucene index as stored fields, read them out with Lucene.NET once and write them into OneIndex; a stored-field export is a loop over `searcher.Doc(i)`. ## A worked example The same search in both libraries: products, the words "wireless headphones" in the title or body, only the `audio` category under 100, cheapest first, with the category facet and a highlighted title. The OneIndex half is compiled and tested (`samples/OneIndex.Samples/Migration.cs`); the Lucene.NET half uses its public API and is shown for orientation. ### Lucene.NET 4.8 ```csharp using var dir = FSDirectory.Open("index"); var analyzer = new EnglishAnalyzer(LuceneVersion.LUCENE_48); var config = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer) { Similarity = new BM25Similarity() }; using var writer = new IndexWriter(dir, config); var doc = new Document { new StringField("id", "p1", Field.Store.YES), new TextField("title", "Wireless headphones", Field.Store.YES), new TextField("body", "Over-ear wireless headphones with noise cancelling.", Field.Store.YES), new DoubleField("price", 89, Field.Store.YES), new StringField("category", "audio", Field.Store.YES), new DoubleDocValuesField("price", 89), // doc values, for sorting new SortedSetDocValuesFacetField("category", "audio"), }; writer.UpdateDocument(new Term("id", "p1"), facetsConfig.Build(doc)); writer.Commit(); using var reader = DirectoryReader.Open(dir); var searcher = new IndexSearcher(reader); var parser = new MultiFieldQueryParser(LuceneVersion.LUCENE_48, new[] { "title", "body" }, analyzer); var query = new BooleanQuery { { parser.Parse("wireless headphones"), Occur.MUST }, { new TermQuery(new Term("category", "audio")), Occur.MUST }, { NumericRangeQuery.NewDoubleRange("price", null, 100.0, true, false), Occur.MUST }, }; var sort = new Sort(new SortField("price", SortFieldType.DOUBLE)); var fc = new FacetsCollector(); TopDocs top = FacetsCollector.Search(searcher, query, null, 10, sort, fc); // null: no Filter var facets = new SortedSetDocValuesFacetCounts(new DefaultSortedSetDocValuesReaderState(reader), fc); foreach (ScoreDoc hit in top.ScoreDocs) { Document found = searcher.Doc(hit.Doc); // Highlighter + QueryScorer + SimpleHTMLFormatter for the title fragment } ``` ### OneIndex, happy path ```csharp public class Product { [SearchId] public string Id { get; set; } = ""; [FullText(Boost = 2)] public string Title { get; set; } = ""; [FullText(Language = "en")] public string Body { get; set; } = ""; [Filterable, Sortable] public double Price { get; set; } [Keyword, Facet] public string Category { get; set; } = ""; } using var index = SearchIndex.Open("index"); index.Upsert(products); // replaces by id; batches index.Commit(); var results = index.Search("wireless headphones") .Where(p => p.Category == "audio" && p.Price < 100) .OrderBy(p => p.Price) .Facets(p => p.Category) .Take(10) .Execute(); foreach (var hit in results) Console.WriteLine(hit.Id + " " + hit.Highlights["Title"][0].Marked()); // "Wireless headphones" var counts = results.Facets["Category"].Values; // audio = 2 ``` ### OneIndex, low level ```csharp Schema schema = new SchemaBuilder().Id("id") .Add(new FieldDefinition("title", FieldType.Text) { Analyzer = "en" }) .Keyword("category") .Fractional("price") .Build(); using var writer = IndexWriter.Open(new MemoryStorage(), schema); // or FileSystemStorage writer.Upsert(new Document().Add("id", "p1").Add("title", "Wireless headphones").Add("category", "audio").Add("price", 89.0)); writer.Commit(); using IndexSnapshot snapshot = writer.OpenSnapshot(); SearchContext context = SnapshotSearch.CreateContext(snapshot); var searcher = new IndexSearcher(context, new QueryEngine()); var query = new BooleanQuery( [ new BooleanClause(new TermQuery("title", "wireless"), Occur.Must), new BooleanClause(new TermQuery("title", "headphon"), Occur.Must), // analyzed by the caller: "en" stems it new BooleanClause(new TermQuery("category", "audio"), Occur.Filter), // selects, does not score new BooleanClause(new RangeQuery("price", null, FieldValue.FromFloat64(100)), Occur.Filter), ]); TopDocs top = searcher.Search(query, 10); ``` Differences worth noticing in the example: there is no analyzer object to share between indexing and searching (the schema names the field's analyzer); the price filter and the sort are the same declaration (`[Filterable, Sortable]`); facets and highlights come back with the results rather than from separate collectors; and the replacement for `UpdateDocument(new Term("id", ...), doc)` is `Upsert`. # The `oneindex` tool `oneindex` is a .NET tool (package `RecursiveAI.OneIndex.Cli`) that turns a folder into a searchable passage index, searches it with citations, and builds a static index for browsers. It uses the folder package ([files.md](https://recursiveai.org/oneindex/docs/files/)) and the engine; it adds no behaviour of its own. ``` dotnet tool install --global RecursiveAI.OneIndex.Cli --prerelease oneindex index ~/papers --index ~/papers-index oneindex query ~/papers-index "attention is all you need" --limit 5 oneindex static build ~/papers-index ./site/search ``` The tool package is 151 MB (about 463 MB installed) because it carries the default English models (embedder, reranker and sparse encoder, 134 MB in `RecursiveAI.OneIndex.Models`). `oneindex --version` prints the package version. ## Commands | Command | What it does | |---|---| | `oneindex index [--index ] [--watch] [--embed] [--ocr]` | Reads every file under the folder (text, Markdown, HTML, PDF, archives) and keeps the passage index in `` (default `/.oneindex`). Prints progress. It commits as it goes, so stopping it (Ctrl+C or a kill) and running it again resumes. `--watch` keeps the index current until Ctrl+C. `--embed` also computes passage embeddings with the packaged embedder (slower; needed for `--hybrid`). `--ocr` reads scans: images and PDFs whose pages are pictures. | | `oneindex query "" [--limit N] [--filter ""] [--hybrid] [--json] [--files]` | Prints the best passages with their citation (file, page, character range, heading). `--filter` is the filter-string syntax ([filter-strings.md](https://recursiveai.org/oneindex/docs/filter-strings/)) over the passage fields, named as in `FilePassage` (`Format = pdf AND Page > 2`). `--files` shows one hit per file. `--json` writes one JSON object: `totalHits`, `totalHitsExact` and the hits with their citations, boxes included. It opens the index read-only, so it works while `index --watch` writes. | | `oneindex static build [--chunk-kb N] [--scattered-kb N]` | Writes a static index of the newest commit into `` (default chunks of 16 KiB for vector files and 4 KiB for term, posting and stored files) for plain file servers and the browser ([static-index.md](https://recursiveai.org/oneindex/docs/static-index/)). Refuses an `` that already holds one. | | `oneindex --version`, `--help` | | ## Exit codes and errors 0 done; 1 the command failed (a missing folder, a damaged index, a bad filter); 2 the command line is wrong; 130 cancelled. A user mistake prints one line to standard error, never a stack trace. A damaged index names the file. ## Notes - The tool references only OneIndex packages: the library, `.Files`, `.Files.Ocr`, `.Models` and `.Analysis.Asian`. Install it with `dotnet tool install`; the test `IndexQueryAndStaticBuildFromThePackedTool` packs the tool and its library packages into a throwaway feed, installs it with `--tool-path`, and runs `index`, `query` (text, `--json`, `--filter`, `--files`), a second `index` (resume, nothing changed) and `static build`, then opens the static index it built. - Everything stays local. The index holds the extracted text of the files, so it is as sensitive as they are. - OCR is slow next to reading a text layer (seconds per scanned page); `--ocr` raises the per-file time limit. # User text What a person types into a search box. `index.Search(text)` accepts **any** string and never throws on it: unbalanced quotes, lone operators, control characters, lone surrogates and megabytes of input all give a sensible query. (Running the query can still fail for reasons that have nothing to do with the text: a disposed index, a cancelled token, a damaged file.) This page is normative. The parser is `UserText.Parse`; the compiler to query objects is `UserTextCompiler`. Both are public so that the same text can be inspected or compiled without an index. ## Grammar The input is first **cleaned**: - it is cut to 4,096 UTF-16 code units (a surrogate pair at the cut is dropped whole); - every control character, every Unicode whitespace character and every unpaired surrogate becomes a space. Then it is read left to right as a list of **operands** separated by spaces: | Form | Meaning | |---|---| | `word` | A word: everything up to the next space or double quote. | | `"two words"` | A phrase: the words must appear next to each other, in order. The quote ends at the next `"`; an unclosed quote runs to the end of the input. A phrase with only spaces in it is ignored. | | `-word`, `-"a phrase"` | Exclusion: documents containing it do not match. The `-` must touch the operand. A `-` followed by a space, or alone, is an ordinary character that analysis drops. Only one leading `-` is the operator (`--cat` excludes `-cat`, which analyzes to `cat`). A `-` inside a word is part of the word. | | `a OR b` | Alternatives. `OR` (exactly those capitals) between two operands makes them one group that matches when any member does. `a OR b OR c` is one group. | Rules around `OR`: - Operands that are not joined by `OR` must all match (and). - `OR` with no operand on its left (first in the text) or right (last, or followed by another `OR`) is ignored. `cat OR OR dog` is `cat OR dog`. - An exclusion after `OR` cancels the operator: `cat OR -dog bird` requires `cat` and `bird` and excludes `dog`. - A lowercase `or` is an ordinary word (and is dropped as a stop word by languages that have one). - Nothing else is special: no parentheses, no `AND`/`NOT`, no `field:value`, no wildcards, no escaping. Those characters are ordinary text, and analysis decides what is left of them. At most **32 operands** are read (positive and excluded together); the rest are ignored. ## Compilation Each operand is analyzed with the analyzer of every full-text field. Fields that share an analyzer are searched together as one BM25F term (the fields' boosts are the weights); fields with different analyzers are searched separately with their own terms and their scores add. So a title analyzed as `standard` and a body analyzed as `en` each see the word the way they indexed it. - A word that analyzes to one term is a `MultiFieldTermQuery` (one field: a `TermQuery`), or a `TypoTermQuery` when typo tolerance or as-you-type applies to it. - A word that analyzes to several terms (`state-of-the-art`, `acme.corp`) and every phrase is a `PhraseQuery` per field. Positions that analysis skipped, such as stop words, widen the slop by as many positions, so `"state of the art"` matches the text it came from. - A word that analysis removes entirely (a stop word, punctuation) is dropped. In a group of alternatives the others still count. The query is: each group of alternatives is a `Must` clause; each exclusion is a `MustNot` clause. If only exclusions remain, a `MatchAllQuery` is the `Must` clause, so `-draft` means every document except those with `draft`. | Input | Result | |---|---| | `null`, empty, or only whitespace and control characters | `MatchAllQuery`: every document, score 1 (a *blank search*, like a placeholder search in a search-as-you-type box). | | Words exist but none survives analysis, and nothing is excluded (`?!`, `the` in English) | Matches nothing. | | The schema has no full-text field | Matches nothing for any words. | Scoring is BM25F as in [search-contracts.md](https://recursiveai.org/oneindex/docs/search-contracts/); the filters from `Where` never change the score. ## Typos, as-you-type and settings By default every unquoted word gets Meilisearch-style typo tolerance and the last word also matches words that begin with it; the attributes `[TypoTolerance]`, `[PrefixSearch]`, `[Synonyms]`, `[OneWaySynonyms]` and `[StopWords]` on the document type change that (`docs/typo-tolerance.md`). Quoted phrases and excluded words (`-word`) are matched as written. ## What is not here yet Field-qualified terms arrive with the app-search layer. The strict parser with positioned errors is a separate, opt-in grammar. ## Highlighting A hit's `Highlights` re-analyzes the stored text of each full-text field with the field's analyzer and marks the tokens whose terms the query contains (excluded operands never mark). A phrase marks its words only where they occur in order; overlapping or touching marks, as CJK bigrams are, merge into one run. See [mapping.md](https://recursiveai.org/oneindex/docs/mapping/#highlights). # Strict query syntax The strict syntax is an opt-in query language for people and programs that want exactly what they wrote. It is a separate grammar from the forgiving user text of `index.Search("...")` (see [user-text.md](https://recursiveai.org/oneindex/docs/user-text/)), which accepts anything and never throws. The strict syntax throws `QueryParseException` with the position of the problem, and it checks field names against the schema. It is not compatible with any other engine's query syntax. ```csharp index.SearchStrict("Title:(overdue OR late)^2 AND NOT Tags:draft AND Total:[100 TO *}"); Query q = StrictQuery.Parse(text, index.Schema); // query objects, to compose yourself StrictNode tree = StrictQuery.ParseSyntax(text); // syntax only, no schema string text = StrictQuery.Print(tree); // reads back to an equal tree ``` ## Grammar ``` query := or or := and ( "OR" and )* and := unary ( [ "AND" ] unary )* adjacency is AND unary := ( "NOT" | "-" ) unary | primary primary := group | atom group := [ field ":" ] "(" or ")" suffix atom := [ field ":" ] value suffix value := word | word "*" | word "~" [ digits ] | phrase | regex | range | comparison | "*" phrase := '"' chars '"' [ "~" digits ] regex := "/" chars "/" range := ( "[" | "{" ) bound "TO" bound ( "]" | "}" ) comparison := ( ">" | ">=" | "<" | "<=" ) bound bound := word-without-spaces | '"' chars '"' | "*" "*" is an open end suffix := [ "^" number ] ``` - **Precedence**, tightest first: `NOT` / `-`, `AND` (or adjacency), `OR`. `a b OR c` is `(a AND b) OR c`. This differs from user text, where `OR` joins neighbours. - **Operators are upper case and bare.** `and`, `or`, `not` are words. To search for the word `AND`, write `\AND` or `"AND"`. - **A word** runs up to white space or one of `( ) " ^ ~ : [ ] { }`. A backslash makes the next character literal (`\(`, `\:`, `\*`, `\\`, `\-`, and a backslash before a space). A `*` may only be the last character of a word (a prefix search); anywhere else it is an error, and `\*` is a literal star. - **`-`** at the start of an operand means `NOT`; inside a word it is an ordinary character. A negative number as a term needs a backslash (`\-5`). In a range bound or after `field:>` it needs none. - **Phrase:** inside quotes `\"` and `\\` are the only escapes. `~N` is the slop: how many position moves are allowed (0 to 1000; default 0). - **Fuzzy:** `word~` allows 2 edits; `word~N` allows N (0, 1 or 2). A swap of two neighbouring characters is one edit. - **Regex:** a regular expression between slashes that must match a whole term. `\/` is a slash; every other backslash pair is passed to the expression untouched. The syntax of the expression is in [term-dictionary.md](https://recursiveai.org/oneindex/docs/term-dictionary/). It is checked when the query is compiled, and the error points into the pattern. - **Range:** `[` and `]` include the bound, `{` and `}` exclude it, `*` is no bound. `field:>5`, `>=`, `<`, `<=` are one-sided ranges. `field:*` means "has a value" and is the same as `field:[* TO *]`. A range always needs a field. - **Match all:** a lone `*`. - **Boost:** `^2`, `^0.5`: a non-negative number multiplying the score of the atom or group it follows. Boosts multiply: `(a^2 b)^3` gives `a` a boost of 6. - **Fields:** `field:` applies to the one value after it. Before a group it applies to every unqualified atom inside it: `Title:(a OR Body:b)` searches `a` in Title and `b` in Body. Field names are case-sensitive and may use backslash escapes. - **Limits:** 8,192 characters, nesting depth 64, 1,024 nodes. Longer or deeper input is a `QueryParseException`. ## Compilation against the schema Each atom becomes query objects, using the field's type: | Atom | Full-text field | Keyword field | Number, date, bool field | |---|---|---|---| | word | analyzed with the field's analyzer: one term is a `TermQuery`, several (`acme.corp`) a `PhraseQuery`; nothing left (a stop word) drops the atom | `TermQuery` on the exact text | one-value `RangeQuery` (the value is read like a filter-string value) | | phrase | analyzed; slop is the phrase slop plus positions the analyzer skipped | the whole text is one exact value; slop must be 0 | not allowed | | `word*` | the analyzed word as a `PrefixQuery` | `PrefixQuery` on the exact text | not allowed | | `word~N` | the analyzed word as a `FuzzyQuery` | `FuzzyQuery` on the exact text | not allowed | | regex | `RegexQuery` | `RegexQuery` | not allowed | | range, comparison | not allowed | `RangeQuery` on text | `RangeQuery` (booleans not allowed) | - A prefix or fuzzy word that analyzes to several terms is an error. Stemming analyzers can change the end of a prefix (`runn*` is stemmed like any word), so prefix search on a stemmed field is best done on an unstemmed field. - **Unqualified atoms** search every full-text field the way user text does: fields that share an analyzer are scored together as one BM25F term; prefix, fuzzy and regex atoms are tried on each field and any may match. With no full-text field an unqualified atom matches nothing. - `AND` is a boolean query of `Must` clauses, `NOT x` inside an `AND` a `MustNot` clause (an `AND` of only `NOT`s starts from every document), `OR` a boolean query of `Should` clauses with `MinimumShouldMatch` 1. A lone `NOT x` is every document except `x`. - An operand that analysis removes entirely is dropped from its parent; if nothing is left the query matches no document. - Scoring is BM25F as in [scoring.md](https://recursiveai.org/oneindex/docs/scoring/); range atoms and comparisons score as constants. ## Errors `QueryParseException` carries `Offset` (UTF-16 index), `Length`, `Line` and `Column` (one-based; `\n`, `\r\n` and `\r` end lines), `Expected` (what the parser would have accepted) and `Suggestion`. The message reads `Expected ')' to close the '(' at offset 0, but found the end of the query. (at offset 4, line 1, column 5)`. - Syntax errors point at the first character that cannot continue the query, or at the end of the input. - Unknown fields point at the field name and name the nearest field of the schema (`Did you mean 'Color'?`) and the searchable fields. - Type errors (`Count:abc`) point at the start of the atom. - Regex errors point into the pattern. ## Printing `StrictQuery.Print(tree)` writes a tree as text that `ParseSyntax` reads back to an equal tree (positions are not compared). Composite children are always parenthesized and operators are explicit, so the nesting survives; escapes are added where a character would otherwise be read as syntax. # Filter strings A filter string selects documents by field values, with conditions joined by `AND`, `OR` and `NOT`. The syntax follows Meilisearch's documented filter expressions. A filter never scores. ```csharp Query filter = FilterString.Parse("Tags = invoice AND (Total >= 100 OR Priority = 'high') AND Paid = false", index.Schema); ``` In query JSON a filter string is the value of `"filter"` (see [query-json.md](https://recursiveai.org/oneindex/docs/query-json/)). ## Grammar ``` filter := or or := and ( OR and )* and := unary ( AND unary )* unary := NOT unary | "(" or ")" | condition condition := field "=" value | field "!=" value | field ( ">" | ">=" | "<" | "<=" ) value | field value TO value | field [ NOT ] IN "[" [ value ( "," value )* ] "]" | field [ NOT ] EXISTS | field IS [ NOT ] ( NULL | EMPTY ) | "_geoRadius" "(" [ field "," ] lat "," lng "," meters ")" | "_geoBoundingBox" "(" [ field "," ] "[" lat "," lng "]" "," "[" lat "," lng "]" ")" field := word | quoted value := word | quoted quoted := '...' | "..." inside, \\ \" \' are the escapes ``` - Keywords (`AND OR NOT TO IN EXISTS IS NULL EMPTY`) are case-insensitive. `NOT` binds tighter than `AND`, which binds tighter than `OR`. To use a keyword as a field name or a value, quote it (`Color = "AND"`). - A word runs up to white space or one of `( ) [ ] , = ! < > ' "`. It may hold `:`, `-`, `.`, `+` and the like, so `2024-01-31T10:00:00Z` and `-3.5` need no quotes. Text with spaces needs quotes. - Limits: 8,192 characters, depth 64 (parentheses and nested `NOT`s), 512 conditions, 1,024 values in an `IN` list. ## Fields and values A field must exist in the schema, be indexed (`[Filterable]`, `[Keyword]`, or the identifier), and not be full-text; `Unfiltered = x` and `Body = x` are errors that say why. The value is read by the field's type: | Field | Value | Notes | |---|---|---| | keyword (also multi-valued) | any text | compared exactly: case-sensitive, no analysis (Meilisearch compares strings case-insensitively; this is a documented difference). On a multi-valued field a condition holds when any element matches. `>` `<` `TO` compare in byte order of the UTF-8 text. | | integer | a whole number in the `long` range | a fractional number is rounded inward: `> 2.5` is `>= 3`, `<= 2.5` is `<= 2`, `2.5 TO 7.5` is `3 TO 7`; `= 2.5` matches nothing and `!= 2.5` everything | | number | a finite decimal number, with optional exponent | | | date | `2024-01-31`, `2024-01-31T10:00`, `...T10:00:00`, with fraction and `Z` or an offset | UTC unless an offset is given; stored with millisecond precision | | boolean | `true` or `false` (any case) | only `=` and `!=`, `IN`, `EXISTS` | Comparing with a quoted number or date (`Total > "100"`) is accepted; a value that does not fit the field's type is an error pointing at the value, naming what the field holds. ## Paths in dynamic JSON The name of a `[Dynamic]` property followed by a dot and a path is a field: `Meta.customer.age >= 18`, `Meta.tags = "red"`, `Meta.customer.name EXISTS`. The literal says which kind of value is meant: a quoted string is a string, a bare number is a number, `true` and `false` are booleans, and in an ordering comparison or `TO` range a string that reads as a date (the date formats above) is a date. `Meta.id = 5` does not match the string `"5"`: quote it. See dynamic.md. ## Meaning - `a != v` is `NOT a = v`: documents without a value for the field match it, as `NOT` does in Meilisearch. - `field IN [a, b]` is `field = a OR field = b`; an empty list matches nothing. - `field EXISTS` holds when the document has a value. `IS NULL` and `IS EMPTY` hold when it has none: a document never stores an explicit null, and empty strings and empty arrays are not indexed, so the two tests are the same as `NOT ... EXISTS` (Meilisearch distinguishes null from empty; this index does not). `IS NOT NULL` and `IS NOT EMPTY` are `EXISTS`. - `field a TO b` is `field >= a AND field <= b` (both bounds included). - `_geoRadius(lat, lng, meters)` holds for documents with a point within the distance; `_geoBoundingBox([topRightLat, topRightLng], [bottomLeftLat, bottomLeftLng])` for documents with a point in the rectangle, crossing the antimeridian when the bottom left longitude is greater than the top right one. The geo field is named first when the schema has several (`_geoRadius(Location, 45.47, 9.18, 2000)`); with one searchable geo field it can be left out. The names are case-insensitive; numbers are plain decimals, latitude -90 to 90, longitude -180 to 180, radius not negative, and errors point at the number. Details: [geo.md](https://recursiveai.org/oneindex/docs/geo/). They combine with `AND`, `OR` and `NOT` like any condition. `Print` always writes the field name and numbers in shortest round-trip form. - `CONTAINS`, `STARTS WITH` and array-form filters are not supported yet. ## Query objects `AND` is a boolean query of `Filter` clauses, `OR` of `Should` clauses with `MinimumShouldMatch` 1, `NOT x` is `[Filter MatchAll, MustNot x]`. Conditions are `TermQuery` (keywords), `GeoRadiusQuery` and `GeoBoundingBoxQuery` (the geo functions) and `RangeQuery` (everything else; `=` is a range with equal bounds; `EXISTS` a range with no bounds). A chain `a AND b AND c` is one boolean query with three clauses; parentheses make a nested one. ## Errors and printing `QueryParseException` has the offset, line, column, what was expected, and for unknown fields the nearest field name (`Did you mean 'Color'?`). Syntax errors point at the token that cannot continue the filter; field errors at the field name; value errors at the value; a missing token at the end of the input. `FilterString.Print(query, schema)` writes the query shapes above back as text that parses to an equal query: strings in double quotes, dates as `"yyyy-MM-ddTHH:mm:ss.fffZ"`, a one-valued range as `field = value`, all booleans below the top parenthesized. # LINQ filters `index.Search(text).Where(predicate)` narrows a search with a C# lambda. The lambda is **translated, never run**: the engine walks the expression tree and builds query objects (a `BooleanQuery` of `Filter` clauses over `TermQuery` and `RangeQuery`). It does not compile the tree. Filters never change scores. Several `Where` calls combine with and. The pieces of the lambda that do not mention the document (captured variables, constants, static members, `new DateTime(...)`, `now.AddDays(-7)`, `min - 1`) are read directly from the tree, with `FieldInfo`/`PropertyInfo` reads and method calls, and compared as values. ## What may appear | Form | Fields | Becomes | |---|---|---| | `x.M == v`, `x.M != v` | `[Keyword]` strings; numbers, dates and booleans marked `[Filterable]` (or `[Facet]`) | term / one-value range; `!=` is its complement | | `x.M < v`, `<=`, `>`, `>=` (the constant may be on either side) | numbers and dates | one-sided range | | `x.Flag`, `!x.Flag` | `bool` | `== true`, `== false` | | `x.Tags.Contains(v)` | a collection of strings marked `[Keyword]` | term (any element equals `v`) | | `list.Contains(x.M)` | any filterable member; `list` is captured or constant | any of the list's values; an empty list matches nothing; a `null` in the list matches documents with no value | | `x.M == null`, `x.M != null` (the `null` may be on either side, or a captured variable that is `null`) | any filterable member, including collections | has no value / has a value (`NOT field EXISTS` / `field EXISTS` in a [filter string](https://recursiveai.org/oneindex/docs/filter-strings/), `exists` in [query JSON](https://recursiveai.org/oneindex/docs/query-json/)) | | `x.Tags.Any()` | a collection member | has a value | | `x.Tags.Any(t => t == v)`, `Any(t => t > v)` and the other ordering comparisons, `Any(t => list.Contains(t))`, and `\|\|` of those | a collection member | some element satisfies the test (a term or range on the field) | | `a && b`, `a \|\| b`, `!a` | | boolean clauses | | a constant condition (`true`, a captured `bool`) | | everything or nothing | | `Geo.WithinRadius(x.Location, lat, lng, meters)`, `Geo.WithinBoundingBox(x.Location, north, south, west, east)` | a `[Geo]` `GeoPoint` property (`x.Location.Value` for a nullable one); the numbers may not use the document | geo queries ([geo.md](https://recursiveai.org/oneindex/docs/geo/)) | Numeric widening conversions the compiler inserts are looked through (`x.Count > someLong`). Comparing an integer member with a fractional value rounds the bound inward (`x.Count > 2.5` means `>= 3`). Nullable members compare as C# does for non-null values; a document without the value matches `!=` and `!` and nothing else, as lifted comparisons do. ## Paths in dynamic JSON A `[Dynamic]` property is reached with `Path`: `x.Meta.Path("customer.age") >= 18`, `x.Meta.Path("tags") == "red"`, `x.Meta.Path("customer.name").Exists()`. The comparison is with a string, a number, a `bool` or a date; the member must be on the left. See dynamic.md for what "any value at the path" means and how the path is written. ## What fails Anything else throws `UnsupportedExpressionException` (a `NotSupportedException`) **when `Where` is called**, naming the sub-expression: `The expression '(x.Text == "doc 1")' cannot be used in Where: 'Text' is full-text; match it with the search text, or filter on a separate [Keyword] property.` The usual causes: - a member that is not filterable (`'Note' is not filterable; mark it [Filterable] (or [Keyword] for text)`), or is full-text; - ordering a member against `null` (`x.Count > null`), or comparing with something that uses the document (`x.Count > x.Big`); - string methods (`StartsWith`, `Contains` on a string, `Equals`, `Length`), `ToString()`, other calls on the document's members, nested paths such as `x.Customer.Name`, `All`, conditional expressions; - inside `Any`: `!=` and `&&` (they would need one element to satisfy two tests, which a term index cannot answer), and anything but the forms in the table; - ordering comparisons on keywords or booleans; - a captured value that cannot be read (null reference inside it). ## Null and empty The index stores no explicit nulls, and an empty string or an empty collection is not stored either (an empty keyword is never indexed). So `x.Color == null` means "has no value", which also matches `""`, and `x.Labels == null` also matches an empty array; `x.Labels.Any()` is the exact test for a non-empty collection. Values that are kept in another form are translated for you: a `Guid` is its text (`"D"` format), a `DateOnly` its day number and a `TimeOnly` its ticks, so equality and ordering comparisons on them work for members marked `[Filterable]` (and `[Keyword]` for a `Guid`). ## Precision - Dates are stored as UTC milliseconds. `DateTime` of kind `Local` is converted to UTC, `Unspecified` is read as UTC; reading back gives kind `Utc`. Constants in filters are rounded the same way, so a stored `2024-01-01T00:00:00.0005Z` equals a constant of `...0.000Z`; sub-millisecond differences are not visible to filters. - `float` members are stored as `double`, exactly. - Keyword values are compared exactly (case-sensitive, no analysis). Empty strings are not indexed. ## Ordering `OrderBy`, `OrderByDescending`, `ThenBy`, `ThenByDescending` take a plain property read (`x => x.Modified`) of a keyword, number, date or bool member with a column, and an optional `MissingValuePlacement`; `OrderByDistance`/`ThenByDistance` order by distance from a point, `OrderByRelevance`/`ThenByRelevance` by score. `OrderBy` replaces an earlier ordering, `ThenBy` without one throws. Anything else throws `ArgumentException` naming the expression and what to mark `[Sortable]`. Ties, missing values, multi-valued members and search-after paging are specified in [sorting.md](https://recursiveai.org/oneindex/docs/sorting/). ## Totals and paging `Take(n)` (default 20) and `Skip(n)` page by rank (or by the ordering); `SearchAfter(cursor)` continues a sorted search after the cursor of a hit; `Skip + Take` may not exceed 10,000. `TotalHits` and `Count()` are lower bounds unless `CountExactly()` was called. Enumeration, `ToList()` and `Execute()` run the search; `ExecuteAsync` is the async twin. The differential test (`LinqTests`) compares `Where` against LINQ-to-Objects over seeded random documents (with updates and deletes between commits) and seeded random expression trees, which include `null` comparisons, `Any()` and `Any(t => ...)` (4,000 cases, and the same predicates again as filter strings in `WhereStringTests`). ## Filter strings `Where(string)` takes a [filter string](https://recursiveai.org/oneindex/docs/filter-strings/) (`Where("Rating >= 4 AND Tags = invoice")`). It is checked against the schema at the call and throws `QueryParseException` with the offset, line, column and a suggestion for a misspelt field. It selects without scoring and combines with `Where(predicate)` and other `Where(string)` calls by and. A test runs the same random predicates as the LINQ differential through both forms (the translated query printed as a filter string) and against LINQ-to-Objects. ## Score by functions and explain `ScoreBy(b => ...)` changes the relevance score with functions of column values ([function-scores.md](https://recursiveai.org/oneindex/docs/function-scores/)); `Explain()` attaches the score breakdown to each returned hit (`hit.Explanation`, whose root value equals `hit.Score`). # Query JSON A search as data: a JSON document with `"v": 1`, checked against a schema as it is read. It is meant for programs and language models that build a query without a C# compiler, and it compiles to the same query objects as every other surface. It is not compatible with Elasticsearch's query DSL. ```json { "v": 1, "q": "overdue \"acme corp\" -draft", "filter": "Tags = invoice AND Modified > 2024-01-01", "limit": 10, "highlight": true } ``` ```csharp SearchQuery search = index.Parse(json); // throws QueryParseException with the JSON path QueryRequest request = QueryJson.Read(json, index.Schema); // the same, as data string canonical = QueryJson.Write(request); ``` ## The document | Property | Meaning | |---|---| | `v` | **Required.** The format version, the whole number 1. Any other value is an error. | | `q` | Free text, as typed by a user ([user-text.md](https://recursiveai.org/oneindex/docs/user-text/)). Never an error. | | `query` | A structured query node (below). When both `q` and `query` are given, both must match. | | `filter` | A [filter string](https://recursiveai.org/oneindex/docs/filter-strings/), or a query node. Selects without scoring. | | `limit` | Hits to return, 0 to 10,000. Default 20. | | `offset` | Hits to skip, 0 to 10,000. Default 0. `limit + offset` is at most 10,000. | | `countExactly` | `true` for an exact `TotalHits`. Default false (a lower bound). | | `highlight` | `true` for the default snippets, or `{"fragmentLength": 1..10000, "maxFragments": 1..100}` (defaults 160 and 3). | | `sort` | 1 to 16 strings that order the hits instead of relevance: `Field:asc`, `Field:desc`, `_score`, `_geoPoint(lat, lng):asc`, each optionally ending in `:first` or `:last` for documents without a value. See [sorting.md](https://recursiveai.org/oneindex/docs/sorting/). | | `searchAfter` | A cursor token (`SortCursor.ToToken()`, also `NextCursor` of the previous page) to continue after; needs `sort` made the same way (or `rankingRules`, whose keys the token belongs to), and excludes `offset`. | | `rankingRules` | Ranking rules in order, as strings: `words`, `typo`, `proximity`, `attribute`, `sort`, `exactness`, `field:asc`, `field:desc` (at most 12; see [ranking-rules.md](https://recursiveai.org/oneindex/docs/ranking-rules/)). A property rule must name a keyword, bool, numeric or date column of the schema, or reading fails with a positioned error (`$.rankingRules[1]`) and a suggestion. Absent: the rules the type declares. `[]`: rank by relevance. Added to version 1; documents without it read the same. | With no `q` and no `query`, every document is a candidate. Unknown properties are errors that name the path and the nearest known name. These names are reserved for later versions and rejected with a message of their own: `facets`, `aggregations`, `distinct`, `vector`. `hybrid` is a property: an object that fuses the text search of `q` with a vector search over its embedding, and a node type `hybrid` joins `knn` for callers that bring a vector (both in `hybrid.md`; additive to version 1, so every document that was accepted reads the same). `rerank` is a property too: an object `{"window":50}` that rescores the best hits of the search with the index's reranker; it reads the text of `q` and is described in `rerank.md` (additive to version 1 as well). (`sort` and `searchAfter` were reserved in the first drafts of version 1 and are now properties: documents that used to be rejected are accepted, and every document that was accepted reads the same.) Limits: the document is at most 1 MiB, strings 8,192 characters, query nodes nested 32 deep, 2,048 nodes in all. Invalid UTF-8 and lone surrogates are errors. ## Query nodes A node is an object with a `type`. Every node accepts `"boost"` (a non-negative number; default 1). Fields are checked against the schema: the name must exist and be indexed, and the kind of field decides what the node accepts. | `type` | Properties | Meaning | |---|---|---| | `matchAll` | | every document | | `text` | `text` | user text across the full-text fields (what `q` is) | | `strict` | `text` | a [strict query](https://recursiveai.org/oneindex/docs/query-syntax/); its errors keep their offset, line and column inside the message and `$.….text` as the path | | `filter` | `filter` | a [filter string](https://recursiveai.org/oneindex/docs/filter-strings/) used as a query | | `term` | `field`, `value` | the exact value. A keyword takes a string; a full-text field takes one *analyzed* term (not analyzed again); numbers, dates and booleans take a JSON number, an ISO date string or `true`/`false` | | `phrase` | `field`, `terms` (1 to 256 strings), `slop` (0 to 1000) | analyzed terms that must occur in order | | `prefix` | `field`, `prefix`, `scoreTerms` (default false) | terms that start with the analyzed prefix; every match scores the boost, or with `scoreTerms` the BM25F scores of the terms | | `fuzzy` | `field`, `term`, `maxEdits` (0 to 2, default 2), `prefixLength` (default 0), `transpositions` (default true), `maxExpansions` (1 to 1024, default 50) | the closest terms within the edit distance | | `regex` | `field`, `pattern` | terms that the expression matches in full (checked when read) | | `range` | `field`, at most one of `gt`/`gte`, at most one of `lt`/`lte`, at least one bound | numbers, dates (ISO strings) or keywords (strings); not booleans | | `exists` | `field` | documents that have a value | | `multiFieldTerm` | `fields`, `term` | one analyzed term in any of the full-text or keyword fields, scored as one BM25F term | | `typoTerm` | `fields`, `term`, `maxTypos` (0 to 2, default 2), `prefix` (default false), `exactFields` | one analyzed word matched with typos and optionally as the beginning of longer words; what user text expands to ([typo-tolerance.md](https://recursiveai.org/oneindex/docs/typo-tolerance/)) | | `geoRadius` | `field` (a geo point field), `lat`, `lng`, `meters` | documents with a point within the distance ([geo.md](https://recursiveai.org/oneindex/docs/geo/)) | | `geoBoundingBox` | `field`, `north`, `south`, `west`, `east` | documents with a point in the rectangle; `west` greater than `east` crosses the antimeridian | | `bool` | `must`, `filter`, `should`, `mustNot` (arrays of nodes), `minimumShouldMatch` | all `must` and `filter` clauses, no `mustNot` clause, and enough `should` clauses; `filter` clauses do not score | | `proximity` | `query` (a node), `words` (single-word nodes `term`, `multiFieldTerm` or `typoTerm`, at most 64), `weight` (0 to 4, default 0.1) | the score of `query` plus a term-proximity bonus for the words; see [scoring.md](https://recursiveai.org/oneindex/docs/scoring/). Text queries (`q`, `text`) with two or more words read as this node, and write back as it | | `moreLikeThis` | `text` or `id`, `fields`, `maxQueryTerms` (1 to 1,024, default 25), `minTermFrequency`, `minDocFrequency`, `minimumShouldMatch` (above 0 to 1, default 0.3), `boostTerms`, `includeSource` | documents that resemble a text or an indexed document ([more-like-this.md](https://recursiveai.org/oneindex/docs/more-like-this/)) | | `functionScore` | `query` (a node; default `matchAll`), `functions` (1 to 32 function objects), `scoreMode` (`multiply`, `sum`, `avg`, `first`, `max`, `min`; default `multiply`), `boostMode` (`multiply`, `replace`, `sum`, `avg`, `max`, `min`; default `multiply`), `maxBoost` (above 0), `minScore` (not negative) | the score of `query` changed by functions of column values; see [function-scores.md](https://recursiveai.org/oneindex/docs/function-scores/) | A **function object** has a `type`, an optional `weight` (not negative; default 1) and an optional `filter` (a filter string or a node): | `type` | Properties | |---|---| | `weight` | | | `fieldValueFactor` | `field` (numeric, date or bool column), `factor` (default 1), `modifier` (`none`, `log`, `log1p`, `log2p`, `ln`, `ln1p`, `ln2p`, `square`, `sqrt`, `reciprocal`), `missing` | | `gauss`, `exp`, `linear` | `field` (a numeric, date or geo point column), `origin` (a number; for a date field also an ISO date string; for a geo point `{"lat":…,"lng":…}`), `scale` (above 0; milliseconds for dates, metres for geo points), `offset` (default 0), `decay` (above 0 and below 1; default 0.5) | Function fields need a column, not an index. The custom function of the low-level API has no JSON form. A written `origin` is always a number (a date as milliseconds since 1970). `phrase`, `prefix`, `fuzzy`, `regex` and `multiFieldTerm` carry **already analyzed** terms, as the query objects do: write them lower case for the default analyzer. To search words the way a user would, use `text` or `strict` (`{"type":"strict","text":"Title:\"acme corp\""}`). Numbers on integer fields may be fractional in a `range` (rounded inward, as in filter strings). Dates are ISO strings: `2024-01-31`, `2024-01-31T10:00:00Z`, with an optional fraction and offset. ## Paths in dynamic JSON Wherever a node names a `field`, the name of a `[Dynamic]` property, a dot and a path name a path inside it (`"field":"Meta.customer.age"`). `term`, `range` and `exists` choose the kind of value from the JSON type of the value given (string, number, boolean; a string bound that reads as a date makes a date range); `prefix`, `fuzzy`, `regex` and `phrase` apply to strings. See dynamic.md. ## Errors A `QueryParseException` has `Path` (`$.query.must[1].field`), `Detail`, and `Suggestion` for a misspelt field, property or type. Text that is not JSON has no path; its `Line` and `Column` locate the syntax error. A `filter` or `strict` string with a mistake reports the position inside that string in the message, and the path of the string. ## Canonical form and round trips `QueryJson.Write` is canonical: properties in a fixed order (`v`, `query`, `filter`, `limit`, `offset`, `countExactly`, `highlight`, `sort`, `searchAfter`, `rankingRules`), nodes with `type` first, no white space, defaults left out, boosts as shortest round-trip numbers. - **Object to JSON to object is identity:** `Read(Write(request))` equals `request` for every request whose boolean clauses are grouped as `must`, `filter`, `should`, `mustNot` (the shapes of all our parsers). Interleaved clauses are written grouped, which keeps the order inside each group; their meaning is the same. - **JSON to object to JSON is canonical:** any accepted document, whatever its property order or spacing, and with `q`, `text`, `strict` and `filter` nodes already compiled into query nodes, writes as one canonical document. Writing is deterministic. - `Range(v, v)` with both bounds included and a value that is not text is written as a `term`; `Range` with no bounds as `exists`; a `MatchAll` as `matchAll`. - `Boost` of 1 is not written. Include flags on an open bound are normalized to true. # SearchTool: a search as an LLM tool `SearchTool.For()` describes a search over `T` as a tool a language model can call. The arguments are a [query JSON](https://recursiveai.org/oneindex/docs/query-json/) document; the model's tool call goes straight to `index.Parse`. ```csharp string tool = SearchTool.For(); // {"name":"search_Invoice","description":"...","input_schema":{...}} string schema = SearchTool.SchemaFor(); // only the JSON Schema // register `tool` with the model API; when the model calls it with `arguments` (JSON text): SearchResults hits = index.Parse(arguments).Execute(); ``` `input_schema` is the name most tool APIs use; for an API that wants `parameters`, rename the property. For a runtime schema: `SearchTool.For(schema, "Invoice")` and `SearchTool.SchemaFor(schema)`. ## What the schema says A JSON Schema, draft 2020-12 (`$schema` is set). The root is an object with the query JSON properties (`v`, `q`, `query`, `filter`, `limit`, `offset`, `countExactly`, `highlight`), `required: ["v"]` and `additionalProperties: false`. Query nodes are in `$defs`; `$defs.node` is a `oneOf` over one definition per node kind, each with a fixed `type` (`const`), its properties, and `additionalProperties: false`. The schema is built from the type's own fields, so a model cannot invent a field name and each value has the right JSON type: - `field` is an `enum` of the fields that node can use: `term` over keyword and full-text fields (a string), over numeric fields (a number), date fields (a string matching an ISO date pattern) and boolean fields (a boolean) as separate definitions; `phrase`, `prefix`, `fuzzy`, `regex` and `multiFieldTerm` over full-text and keyword fields; `range` over numeric, date and keyword fields, with bounds typed to match, at least one bound and not both `gt` and `gte` (or `lt` and `lte`); `exists` over every filterable field. - Only fields that are indexed appear (`[FullText]`, `[Keyword]`, `[Filterable]`, the identifier). Members that are only stored or only sortable do not. - Limits are in the schema: string lengths, `limit` and `offset` 0 to 10,000, `slop` 0 to 1,000, `maxEdits` 0 to 2, phrase terms 1 to 256, boosts at least 0. - A `functionScore` node is always offered; its functions are `oneOf` the definitions that the type's columns allow (`weight` always; `fieldValueFactor` over numeric, date and bool columns; `gauss`/`exp`/`linear` over numeric and date columns with a number or date `origin`, and over geo point columns with a `{lat, lng}` origin). `decay` is exclusive of 0 and 1, `scale` above 0. See [function-scores.md](https://recursiveai.org/oneindex/docs/function-scores/). - The tool `description` lists the fields with their kinds. A definition for a node kind is left out when the type has no field it could use. - There is no list of allowed values for keyword fields: the attributes do not declare any. Some of what `Parse` checks cannot be written in a JSON Schema and so is not in it: that an integer is in the `long` range, that a date is a real date, that a regular expression is well formed, that `text`, `strict` and `filter` strings parse, that `limit + offset` is at most 10,000, and nesting and size limits. A document that the schema accepts and `Parse` rejects fails with a `QueryParseException` the application can return to the model as the tool result, so it can correct itself: the message names the path and, for misspelt names, the nearest valid one. ## Verification `tests/OneIndex.Tests` generates random request documents (every node kind, 160 per run) and structurally invalid mutations of them (an unknown property, a wrong version, an unknown type, an unknown field, a negative boost, a missing `v`); `SearchIndex.TryParse` accepts exactly the valid ones. A second test does the same for 250 random function-score documents and their mutations (2,224 documents in all). When `ONEINDEX_JSONSCHEMA_PYTHON` names a Python with the `jsonschema` wheel, the same documents are given to its draft 2020-12 validator through `tools/jsonschema_check.py`, and every verdict must equal `Parse`'s. # Queries: semantics and limits How the query engine (`QueryEngine`, an `IWeightFactory`) runs each query record. Scoring is in [scoring.md](https://recursiveai.org/oneindex/docs/scoring/). A query is data; `QueryEngine.CreateWeight` checks it against the schema, fixes the statistics of the snapshot and returns a `Weight`; a weight makes one `Scorer` per segment. Everything here is deterministic for a given snapshot. ## Fields and typed errors Every query names fields. A field must exist in the schema and be searchable for the query kind, otherwise `QueryException` (an `InvalidOperationException`) says which field and why: - Term-like queries (`TermQuery`, `MultiFieldTermQuery`, `PhraseQuery`, `PrefixQuery`, `FuzzyQuery`, `RegexQuery`): a text field with index options other than `None`, or a keyword field that is indexed. - `PhraseQuery` additionally needs a text field indexed with positions (`Positions` or `Offsets`). A field indexed without positions is an error, not an empty result. - `RangeQuery`: an indexed integer, floating-point, date, boolean or keyword field. A field the schema declares but a segment holds no terms for simply matches nothing in that segment. Other typed errors: a negative or non-finite boost, a negative slop, a fuzzy distance outside 0 to 2, an invalid regular expression (with its position), a bound of the wrong type, a boolean query with more than `MaxClauses` clauses (default 4,096) or nested deeper than `MaxDepth` (default 64), and `TooManyTermsException` (below). Nothing else is thrown for a well-formed query. `Boost` multiplies the score contribution of a query node; 0 is allowed and gives score 0 (the documents still match). ## Term and multi-field term `TermQuery(field, term)` matches documents whose field contains the term exactly as given: the term is already analyzed (a query string is analyzed by the caller with the field's analyzer). `MultiFieldTermQuery(fields, term)` matches documents containing the term in any listed field and scores it as one BM25F term over the fields. A `TermQuery` is the one-field case; both use the schema boost of each field as its weight. ## Boolean Clauses are `Must`, `Should`, `MustNot` and `Filter`. - A document matches when it matches every `Must` and `Filter` clause, no `MustNot` clause, and at least *needed* `Should` clauses, where *needed* is `MinimumShouldMatch` if it is above 0, otherwise 1 when the query has no `Must` or `Filter` clause and 0 when it has one. - `Should` clauses that are not needed still add to the score of documents that match them. - `Filter` matches like `Must` but contributes 0 to the score. - A query with no clauses matches nothing. A query with only `MustNot` clauses matches nothing (a negation alone selects no documents: add a `MatchAllQuery` clause to select "everything but"). A `MinimumShouldMatch` above the number of `Should` clauses matches nothing. - The score is the sum of the scores of the matching `Must` clauses, plus the sum of the matching `Should` clauses, times the boost, each sum in clause order. ## Phrase `PhraseQuery(field, terms)` with `Slop`. The terms must occur in the given order at strictly increasing positions. For each position `p` of the first term the engine takes the nearest in-order chain of the other terms (each at the first position after the previous one); the **gap** of the chain is `last position - p - (terms - 1)`, the number of positions the chain is stretched past an exact phrase. The match counts when `gap <= Slop`, with weight `1 / (1 + gap)`. The phrase frequency of a document is the sum of the weights, and it is scored as the frequency of one pseudo-term (scoring.md). Slop 0 is the exact phrase. Terms of a phrase may repeat. One term is a term query; no terms matches nothing. Terms stacked at one position (stemmed twins) are different positions only if the analyzer gave them different positions: a phrase never matches two terms at the same position. Order is required: `"brown quick"` does not match `quick brown` at any slop (there is no transposition match). ## Prefix, fuzzy and regular expression These **expand** to the terms of the field that match. How the expansion is scored depends on the query. - `PrefixQuery`: terms that start with the prefix (compared as UTF-8 bytes). **Constant score by default**: every document that contains a matching term scores the boost, whatever the term, so no term is weighed and the number of matching terms is not limited. The documents of all matching terms are merged lazily, in increasing id order, which also lets a top-k search stop after its first hits (equal scores go to the lower ids). With `ScoreTerms = true` the prefix scores as a disjunction of the terms' BM25F scores (a boolean of `Should` clauses over the expanded terms, in term byte order, each with its own statistics from the snapshot) and the expansion limit below applies. - `FuzzyQuery`: terms within `MaxEdits` (0 to 2) edits of the term, counting insertion, deletion and substitution of a code point, and with `Transpositions` (default true) the swap of two adjacent code points as one edit (the optimal-string-alignment distance; a character cannot be edited again after being swapped). `PrefixLength` code points at the start must match exactly. `MaxEdits` 0 is a term query. The term itself matches. Candidate terms come from the Levenshtein automaton of the term dictionary. Only the `MaxExpansions` (default 50) closest are kept: fewest edits first, then the most documents, then term bytes; each kept term is scored with BM25F like a term query, and the number of candidates never makes the query fail. Edits are counted as optimal string alignment also when `Transpositions` is false, so ranking treats a swap as one edit there as well. - `RegexQuery`: terms that the whole pattern matches, scored as a disjunction of the terms' BM25F scores; the syntax is the one of the term dictionary (docs/term-dictionary.md): no anchors, no look-around. - `TypoTermQuery`: one analyzed word of user text with typos and optionally as the beginning of longer words (docs/typo-tolerance.md). Every matching word is scored like a term query times its typo and exactness factors. **Expansion limit** (scored prefix and regular expression queries). A query may expand to at most `QueryEngineOptions.MaxExpandedTerms` distinct terms (default 1,024), counted across all segments of the snapshot. Past it the behaviour is chosen by `Overflow`: - `Throw` (default): `TooManyTermsException` naming the field and the limit. Chosen as the default because a result computed from a silently truncated term list is hard to explain. - `KeepMostFrequent`: the `MaxExpandedTerms` terms contained in the most documents (ties by term bytes) are kept and the rest dropped. Never examines more than 1,048,576 distinct matching terms; past that it throws too. The count includes terms that occur only in deleted documents until a merge removes them, so whether a borderline query throws can change at a merge. ## Range `RangeQuery(field, lower, upper)` with `IncludeLower` and `IncludeUpper` (default both true); a `null` bound is open. It matches documents with a value of the field in the range and scores every match as its boost (no length or frequency involved). - Integer, date (milliseconds since the Unix epoch, UTC) and boolean fields take bounds of the same kind (integer and date bounds are interchangeable). A floating-point field takes integer or floating-point bounds. Floating-point values compare numerically; `-0.0` equals `0.0`; NaN is never in a range and is rejected as a bound; an open upper bound includes `+Infinity`. - A keyword field takes text bounds and compares the UTF-8 bytes of its values in ordinal order. - A document with several values matches when any value is in range. A document with no value never matches. **Execution.** A segment that has a point index for the field (`IPointSource`, implemented by the columns and points code) answers through `IPointIndexReader.GetRange(lower, upper)` with inclusive bounds in the ordered 64-bit form of `NumericTerms.Ordered`. Otherwise the engine uses the numeric terms the text codec writes for every indexed number when `TextSegmentCodec.IndexNumericTerms` is on (it is off in the default codec set, because the point index answers ranges and the terms only cost index size; turn it on for an index written without the point codec): 16 hexadecimal digits of the ordered value with the sign bit flipped, so the terms of a range are a contiguous run of the term dictionary, found from the digits the two bounds share; keyword ranges walk the keyword terms in byte order between the bounds. Both collect the matching documents of a segment into a bit set. The fallback cost is the number of distinct values in the walked run. ## Match all `MatchAllQuery` matches every live document and scores it as its boost. ## Results and counting `IndexSearcher.Search(query, count, countExactly)` returns the best `count` documents: higher score first, equal scores by lower global document id (segments in commit order). With `countExactly: false` block-max WAND may skip documents that cannot enter the top `count`; the page is identical to the exhaustive one and `TopDocs.TotalHits` is a lower bound (`TotalHitsExact` is true only when nothing could have been skipped). With `true`, every match is visited and the total is exact. Deleted documents never appear. Only queries that are a disjunction at the top (`Should` clauses, expansions, a term, optionally excluded or boosted) are pruned; conjunctions, phrases and ranges are always evaluated exhaustively. `IndexSearcher.Explain(query, globalDocId)` returns the score breakdown of a document (scoring.md) or `null` when it does not match. ## Statistics scope Document counts, document frequencies and average field lengths are summed over all segments of the snapshot when the weight is made, so a document scores the same whichever segment it is in. Deleted documents still count until a merge rewrites their segment. Scores of a snapshot therefore change slightly across merges that drop deletes, and are identical for two snapshots with identical segments and deletes. ## Other ways to write a query The same query objects are written by four text and data forms: forgiving [user text](https://recursiveai.org/oneindex/docs/user-text/), the opt-in [strict syntax](https://recursiveai.org/oneindex/docs/query-syntax/) with positioned errors, [filter strings](https://recursiveai.org/oneindex/docs/filter-strings/), and versioned [query JSON](https://recursiveai.org/oneindex/docs/query-json/), whose schema for language models is described in [search-tool.md](https://recursiveai.org/oneindex/docs/search-tool/). # More like this Finds documents that resemble a text or an indexed document. `MoreLikeThisQuery` is a query record like the others; the happy path has `index.MoreLikeThis(id, x => x.Body)` and `index.MoreLikeThisText(text)`, query JSON has the `moreLikeThis` node. ## Method The selection is the relevance-feedback term selection of the IR textbooks (a term is a good probe when it is frequent in the source and rare in the collection): 1. The source is the `Text`, or the stored source of the live document with identifier `DocumentId` (a `SourceDocumentNotFoundException`, a `QueryException`, when there is none). 2. For each full-text field (`Fields`, default all indexed ones) the source text is analyzed with the field's own analyzer and the terms are counted. Stacked forms (stemmed and unstemmed) count as terms like any other. 3. A term is a candidate when its count is at least `MinTermFrequency` (default 1) and it occurs in at least `MinDocumentFrequency` indexed documents (default 1; for a document source the source itself is one of them, so a term only the source holds is dropped: it could not find anything else). 4. Its weight is `count * idf`, with the same idf as scoring (scoring.md), taken from the field's statistics over the snapshot. 5. The `MaxQueryTerms` (default 25) best weights are kept; ties break by field then term in ordinal order, so the choice is deterministic. 6. The query is a boolean of `Should` term clauses, one per kept term. With `BoostTerms` (default true) a clause's boost is its weight divided by the best weight. A document needs `ceil(MinimumShouldMatchFraction * kept)` of them (default 0.3, at least one). 7. A document source is excluded from the results with a `MustNot` identifier clause unless `IncludeSource`. The terms are chosen when the weight is made, against the snapshot being searched. A query with no candidate term matches nothing. `Highlights` of the hits are empty for this query, since the words are not known until it runs. Defaults are ours; the parameters follow the names and the idea of the published "more like this" query documentation of Elasticsearch (maximum query terms, minimum term and document frequency, minimum should match) without copying its defaults: this engine's minimum document frequency is 1 because small indexes are common, and a share of terms replaces a percentage string. ## Query JSON `{"type":"moreLikeThis","id":"c1","fields":["Body"],"maxQueryTerms":25,"minTermFrequency":1,"minDocFrequency":1,"minimumShouldMatch":0.3,"boostTerms":true,"includeSource":false}`. Exactly one of `text` and `id`; defaults are not written back. `fields` must be full-text fields. The strict syntax and user text have no form for it: both are for typed words, and the node composes with them through a `bool` node. # Scoring How a document's score is made, what the statistics cover, the defaults and why, and the explain format. Query semantics are in [queries.md](https://recursiveai.org/oneindex/docs/queries/). ## Formula: BM25F Robertson and Zaragoza, *The Probabilistic Relevance Framework: BM25 and Beyond* (2009), section 3.6, equations 3.19 to 3.21, with the length normalization of section 3.4.5. For a term *t* searched in fields *F* (one field for a `TermQuery`, several for a `MultiFieldTermQuery`) and a document *d*: B_f = (1 - b) + b * len_f(d) / avgLen_f length normalization of field f x = sum over f in F of w_f * tf_f(d) / B_f weighted, normalized frequency (the paper's tf~) score = boost * idf(t) * x / (k1 + x) saturation of the total evidence idf = ln(1 + (N - n + 0.5) / (n + 0.5)) - `w_f` is the field's schema boost (`FieldDefinition.Boost`, default 1). `tf_f` is the number of occurrences of the term in the field of the document, `len_f` the number of tokens of the field (stacked tokens such as an added stem count), `avgLen_f` the average length over the documents that have the field. - **Saturation is applied to the sum**, not to each field: a term repeated across title and body does not outscore a term whose combined weighted frequency is the same (paper section 3.6.1). The per-field scores of two `TermQuery`s in a boolean add up after saturation instead; use `MultiFieldTermQuery` for BM25F. - `k1` (default 0.9) and `b` (default 0.4) are common to all fields. `b = 0` switches length normalization off, `k1 = 0` makes the score binary. - `idf` is the Robertson-Sparck Jones weight without relevance information, in the non-negative form (paper equations 3.2 and 3.3). For several fields `N` is the largest document count of the fields and `n` the largest document frequency of the term in any of them. The paper computes the idf "on the entire collection disregarding streams" and says a better definition needs research; the exact union frequency would need a pass over the postings, so the largest per-field frequency stands in for it. **Documented deviation**: for a term that appears in disjoint sets of documents in two fields, `n` is smaller than the union, so its idf is a little larger than the paper's. A try with the rarest field's relative frequency instead scored 2 to 4 nDCG points lower on SciFact and SCIDOCS. - A phrase is scored as one pseudo-term in its field: `tf` is the phrase frequency (queries.md), the `idf` is the sum of the terms' idf values. Expansions (prefix, fuzzy, regex) are the sum of the matching terms' scores. A boolean query adds the scores of its matching `Must` and `Should` clauses in clause order, then multiplies by its boost. Range, match-all and filter clauses score a constant (the boost) or 0. - The query's `Boost` multiplies a node; a term's boost is folded into its `idf` factor. Scores are single-precision floats computed in a fixed order, so the same snapshot gives the same bits whether block-max WAND skipped documents or not. ## Block bounds for WAND For a block of a postings list the postings carry the largest frequency and the shortest field length of its documents (docs/postings.md). `x` grows with the frequency and shrinks with the length, and the score grows with `x`, so the bound of a block is the score of (largest frequency, shortest length), summed over the term's fields. A disjunction keeps these bounds per clause and skips a run of documents when the sum of its clauses' bounds cannot reach the lowest score still in the top k. The bound of a phrase, a conjunction or a boolean is the sum of its parts' bounds over the whole list (no block skipping inside them). Pruning only changes `TotalHits`, never the documents returned or their scores. ## Statistics scope `N`, `n` and `avgLen` are summed over every segment of the snapshot when the weight is made, so every document scores on the same numbers. Documents deleted but not yet merged away still count. Different snapshots of the same data (a merge happened, a document was updated) can therefore score a document slightly differently. ## Parameters: why k1 = 0.9, b = 0.4 The paper gives no rule for the two constants (it reports that 0.5 < b < 0.8 and 1.2 < k1 < 2 are often good). The defaults are the values of the BEIR lexical baseline (Thakur et al. 2021, setup of Anserini BM25 with k1 = 0.9, b = 0.4), chosen so that the gate sets can be compared with the published baselines. Results on the four gate sets for both parameter pairs are in the table below. Lucene's pair (1.2, 0.75) is better on ArguAna and about even elsewhere; it remains a one-line option (`QueryEngineOptions.Bm25`). The title weight of the benchmark schema, 3, was picked from {0.5, 1, 2, 3, 5, 10} on SciFact and SCIDOCS (flat above 3; ArguAna measured at 1, 2 and 3); it is a schema property, not an engine constant. A sweep of 1, 1.5, 2 and 3 on SciFact, ArguAna and SCIDOCS improved every set up to 3 (relevance.md). ## Explain `IndexSearcher.Explain(query, globalDocId)` returns an `Explanation` tree whose root value equals the score returned by the search, bit for bit (a test checks it on every random case). Nodes: - term: `term '': boost * idf * x / (k1 + x), k1 = ...`, with children - `idf = ln(1 + (N - n + 0.5) / (n + 0.5)), n = ..., N = ...` - `x = sum over fields of weight * tf / ((1 - b) + b * length / avgLength)`, with one child per field that holds the term: `field : weight * tf / ((1 - b) + b * length / avgLength ), b = ` - phrase: `phrase in : boost * idf * x / (k1 + x)` with the idf sum and the phrase frequency (`sum of 1 / (1 + gap) over matches, slop `) - boolean: `sum of matching clauses, boost , of should clauses matched (need )`, children are the matching clauses (a filter clause is shown with value 0) - constant matches (range, match all): `, constant score = boost ` - function score: `function score: query score and function score by , boost `, with the inner explanation and the functions' values as children ([function-scores.md](https://recursiveai.org/oneindex/docs/function-scores/)) `null` means the document does not match. On the happy path, `SearchQuery.Explain()` returns this tree on each hit (`SearchHit.Explanation`); its root value equals `SearchHit.Score` exactly, on the plain, sorted, faceted and function-scored paths (a test asserts it on every hit). ## Typo and exactness factors User text with typo tolerance or as-you-type expands a query word to the indexed words it can match (`docs/typo-tolerance.md`). Each expanded word is an ordinary BM25F term (its own idf) with a boost: boost = typo factor * exactness factor typo factor = 1 / (1 + typos) typos 0, 1, 2 -> 1, 0.5, 0.333 exactness factor = 1 whole word, 0.8 a longer word that only begins like the query word A document's score is the sum over the expansions it contains, as for any disjunction. `Explain` lists every matched expansion: query word 'planet' matched 'plnaet' (whole word with typos, 1 typos): typo factor 0.5 = 1 / (1 + typos), exactness factor 1, boost 0.5 with the term's own explanation below it. With typos and prefix matching off the query is a plain term query: no factor, same scores as before. The other Meilisearch-style rules are optional ranking rules ([ranking-rules.md](https://recursiveai.org/oneindex/docs/ranking-rules/)). ## Term proximity User text with two or more words (not quoted phrases) adds a proximity bonus to the BM25F score: documents whose query words occur close together, in any order, score higher. It uses the positions already in the postings; no index format changes. Büttcher, Clarke and Lushman, "Term proximity scoring for ad-hoc retrieval on very large text collections", SIGIR 2006, define it. In a field of the document, take the occurrences of the query words in position order. Every two neighbours in that order that are occurrences of *different* words (no other query word occurrence between them) at distance *d* add `idf(other) / d²` to the accumulator of each word: acc_t = sum over adjacent pairs (t, u) at distance d of idf(u) / d^2 (d >= 1; two words at one position count as 1) K = k1 * (1 - b + b * len_f / avglen_f) the BM25 length normalisation of field f bonus_f = sum over query words t of min(1, idf(t)) * acc_t * (k1 + 1) / (acc_t + K) score = bm25f(d) + weight * max over fields f of bonus_f - `idf(t)` is the word's own idf, as in the BM25F term; `k1` and `b` are the BM25F parameters; `len_f` and `avglen_f` are the field's length in the document and its average. A word with typo or prefix expansions uses the positions of all its matched terms (up to the 32 best terms per word) and the idf of the closest one. - **Deviations from the paper**, each deliberate: the bonus of a document is its best field, because a query word pair split across title and body is not close, and positions are compared inside a field only. The paper has one text field; `weight` is ours (the paper's weight is 1). - **The bound.** `acc * (k1 + 1) / (acc + K) < k1 + 1`, so the bonus is below `weight * (k1 + 1) * sum of min(1, idf)` (at most 32 words). A scorer reports its inner bound plus that, and the inner scorer is told the threshold minus that, so block-max WAND skips only what cannot reach the top k; the returned top k and scores equal exhaustive scoring (a test checks 120 random queries, pruned against exhaustive, bit for bit). - A document in which fewer than two query words occur gets no bonus and its positions are not read. - At most 32 words are compared (the rarest), and a repeated word counts once. - `weight` is `TextSettings.ProximityWeight` / `[Proximity(w)]`, default 0.1, 0 to 4; 0 removes the bonus. In query objects it is `ProximityQuery`; query JSON `{"type":"proximity",...}` (a user-text query compiles to it, so it appears in written query JSON). - Not applied: with ranking rules (they have a `proximity` rule), to one-word queries, to phrases, to typed query objects unless wrapped. The default weight was chosen on the four BEIR gate sets; see below. Cost (302,567 Wikipedia abstracts, one thread, median ms per query over 200 seeded queries, indicative runs on a loaded 32-core x64 machine; Lucene.NET's disjunctive rows on the same run: Terms2 0.373, Terms5 0.648): | Query shape | No bonus | Bonus, weight 0.1 | Notes | |---|---|---|---| | User text, every word required (what the default compiles), 2 words | 0.16 to 0.18 | 0.28 | +0.10 ms; 1.6x its own base, 0.75x Lucene.NET's Terms2 | | Same, 5 words | 0.100 | 0.17 | 1.7x its own base, 0.27x Lucene.NET's Terms5 | | Typed disjunction wrapped by hand, 2 words | 0.25 | 0.44 to 0.59 | 1.8x to 2.4x; against Lucene.NET 1.2x to 1.6x | | Typed disjunction wrapped by hand, 5 words | 0.80 | 1.36 to 1.74 | 1.7x to 2.2x; against Lucene.NET 2.1x to 2.7x | The default never builds a disjunction: user text requires every word, so the bonus is paid only on documents that contain all the words. A hand-wrapped disjunction pays for the positions of every document that matches two of the words and loses most pruning headroom, so wrapping disjunctions is for callers who want it. Typed queries that are not wrapped (and so the benchmark rows Terms1, Terms2, Terms5 and Phrase) are not changed by the bonus. Measured effect of the weight (BEIR gate sets): | Proximity weight | SciFact | ArguAna | SCIDOCS | NQ | |---|---|---|---|---| | 0 (no bonus) | 0.6912 \| 0.6998 | 0.4333 \| 0.5072 | 0.1531 \| 0.1601 | 0.3420 \| 0.3377 | | **0.1 (default)** | 0.6918 \| 0.7033 | 0.4347 \| 0.5067 | 0.1544 \| 0.1615 | 0.3452 \| 0.3412 | | 0.25 | 0.6920 \| 0.7032 | 0.4347 \| 0.5045 | 0.1555 \| 0.1623 | 0.3476 \| 0.3428 | | 0.5 | 0.6903 \| 0.6960 | 0.4343 \| 0.5007 | 0.1556 \| 0.1608 | 0.3451 \| 0.3384 | | 1 | 0.6726 \| 0.6831 | 0.4359 \| 0.4940 | 0.1548 \| 0.1593 | not run | | 2 | 0.6594 \| 0.6540 | 0.4327 \| 0.4810 | 0.1493 \| 0.1501 | not run | | Lucene.NET, summed fields (flat for SCIDOCS 0.1509 at 0.9/0.4) | 0.6647 \| 0.6628 | 0.4240 \| 0.4880 | 0.1587 \| 0.1619 | 0.3252 \| 0.3051 | Each cell is nDCG@10 under the BEIR convention at k1/b = 0.9/0.4 \| 1.2/0.75, BM25F with title weight 3 over the BEIR queries as OR-ed words, the bonus wrapped around them. Weight 0 is the earlier no-bonus score and reproduces the numbers above. Runs of the sweep on the other CPU architecture (x64 and arm64) gave identical values. - **Default 0.1.** It is at or above the no-bonus score on all four sets at both settings except ArguAna at 1.2/0.75 (0.5067 against 0.5072, within noise). Gains at the default setting: SciFact +0.06 points, ArguAna +0.14, SCIDOCS +0.13, NQ +0.32. A weight of 0.25 gains more on NQ (+0.56) and SCIDOCS (+0.24) but gives up 0.27 points on ArguAna at 1.2/0.75; from 0.5 up the bonus costs SciFact and ArguAna, and at 1 and 2 it costs clearly. The sets with long queries (ArguAna) and many near-duplicate documents lose first because an additive bonus reorders near ties. - The first formulation of the bonus (unbounded idf-weighted, no length normalisation, weight 0.5) lost 0.6 points on SciFact and 2.0 on ArguAna at 1.2/0.75; the paper's form (word weight capped at 1, BM25 length normalisation) with a small weight does not. - **Hybrid.** With the bonus in the text arm at the default weight, hybrid (RRF, k 10, ratio 0.75) gives SciFact 0.7258, ArguAna 0.6105, SCIDOCS 0.2061, mean 0.5141 (without it 0.7257, 0.6100, 0.2064, mean 0.5140): still above both arms on every set. ## Relevance results See the results table at the end of [relevance.md](https://recursiveai.org/oneindex/docs/relevance/) for the measured nDCG@10 per set, the baselines and Lucene.NET. # Function scores A function score changes the relevance score of a query with numbers computed from column values: popularity, recency, distance. It does not change which documents match (unless `MinScore` is set); it changes their order. The semantics and formulas follow the published `function_score` documentation of Elasticsearch (names: `field_value_factor`, `gauss`/`exp`/`linear`, `score_mode`, `boost_mode`, `max_boost`, `min_score`); this note is the normative text for OneIndex and lists every difference. ```csharp index.Search("coffee") .ScoreBy(b => { b.FieldValue(x => x.Rating, modifier: FieldValueModifier.Log1p); // popularity b.Decay(x => x.Opened, DecayShape.Gauss, DateTimeOffset.UtcNow, TimeSpan.FromDays(30)); // recency b.Distance(x => x.Location, DecayShape.Exp, 48.85, 2.29, scaleMeters: 2_000); // nearness b.Weight(2f, when: x => x.Open == true); // boost what matches }) .Explain() .Take(10); ``` Low level: `new FunctionScoreQuery(inner, [functions]) { ScoreMode, BoostMode, MaxBoost, MinScore, Boost }` over the query objects; query JSON node `functionScore` ([query-json.md](https://recursiveai.org/oneindex/docs/query-json/)). ## Evaluation For each document the inner query matches: 1. Each function, in order, is tried. A function **applies** when its `Filter` (if any) matches the document and it has a value (below). Its **value** is `raw × weight`, computed in double and rounded to a 32-bit float. 2. The values of the functions that applied combine by `ScoreMode` into the **function score**: `Multiply` the product (default), `Sum`, `Average` (weighted: the sum of the values, which already include their weights, divided by the sum of the weights that applied; 0 when those weights are all 0), `First` (the first function that applied), `Max`, `Min`. 3. The function score is capped: `min(function score, MaxBoost)`. `MaxBoost` defaults to the largest float. 4. It combines with the inner score by `BoostMode`: `Multiply` (default), `Replace` (the function score alone), `Sum`, `Average` (the mean of the two), `Max`, `Min`. 5. The result is multiplied by the query's `Boost`. 6. If no function applied, the inner score is kept (times `Boost`); `BoostMode` plays no part. 7. A document whose score is below `MinScore` is dropped. This makes every match be scored, so such queries do not skip documents. All arithmetic is in 32-bit floats from step 2 on, in the order above and the order of the functions, so the same document always gets the same bits, and the explanation equals the score exactly. A NaN becomes 0 and an overflow the largest float: scores stay finite. ## The functions Every function has `Weight` (finite, not negative; default 1) and an optional `Filter` (a query; its score does not matter). | Function | Raw value | |---|---| | `WeightFunction` | 1 (so its value is its weight; with a filter it boosts what matches) | | `FieldValueFactorFunction(field)` | `modifier(factor × value)` of a numeric, date (milliseconds since 1970) or bool (0, 1) column | | `DecayFunction(shape, field, origin, scale)` | the decay of a numeric or date column (below) | | `GeoDecayFunction(shape, field, lat, lng, scale)` | the decay of the great-circle distance in metres ([geo.md](https://recursiveai.org/oneindex/docs/geo/)) | | `CustomFunction(description, bind)` | your code: `bind` runs once per segment with the segment reader and returns `docId => double?`. Low level only; not in query JSON | **No value.** A function with no value for the document does not apply. For `FieldValueFactorFunction` this means: the column has no value and `Missing` is null, or the result is negative or not finite (the logarithm of 0 or less, the square root of a negative number, 1/0, a negative factor times a positive value). With `Missing` set, that number takes the place of the value, and the factor and modifier still apply to it. For the decay functions a document without a value scores **1** (it applies, as Elasticsearch's documentation says: "If the numeric field is missing in the document, the function will return 1"). A custom function returns `null` for "does not apply"; a negative or non-finite number also means that. **Several values.** The factor reads the first value in stored order. A decay uses the value nearest to the origin (the distance is the minimum), as Elasticsearch's documentation says for its default `multi_value_mode`. Other multi-value modes are not offered. ### Modifiers (`field_value_factor`) With `x = factor × value`: `none` x; `log` log₁₀ x; `log1p` log₁₀(1 + x); `log2p` log₁₀(2 + x); `ln` ln x; `ln1p` ln(1 + x); `ln2p` ln(2 + x); `square` x²; `sqrt` √x; `reciprocal` 1/x. ### Decay With `d = max(0, |value − origin| − offset)` (for geo, the distance in metres in place of `|value − origin|`), `scale > 0`, `offset ≥ 0` (default 0) and `0 < decay < 1` (default 0.5): - `gauss`: `exp(−d² / 2σ²)`, `σ² = −scale² / (2 ln decay)` - `exp`: `exp(λ d)`, `λ = ln(decay) / scale` - `linear`: `max((s − d) / s, 0)`, `s = scale / (1 − decay)` All three equal 1 within `offset` of the origin and `decay` at `scale` beyond it; `linear` reaches 0 at `s` beyond it. **Units.** Numbers: the field's own. Dates: milliseconds since 1970 for `origin`, `scale` and `offset` at the low level and in query JSON (an ISO date string is also accepted for `origin`); the LINQ overload takes a `DateTimeOffset` and `TimeSpan`s. Geo: metres. There is no `now` default for the origin: it is always given, so a query means the same thing whenever it runs. ## Execution A function score has no score bound, so block-max WAND does not prune under it (the inner query's own pruning is not used either); every match of the inner query is scored. Top-k collection is otherwise unchanged. `TotalHits` follows the usual rule of the collector in use. ## Explain `SearchQuery.Explain()` attaches an `Explanation` to each returned hit. For a function score the root reads `function score: query score and function score by , boost `, with two children: the inner query's explanation and `function score, of function(s), capped at `, whose children are one node per function that applied: the function, its raw value, the weight. When no function applied the root says so and has the inner explanation as its only child. The root value equals the hit's score exactly (tests assert it on every hit, on all execution paths). ## Differences from Elasticsearch - No `random_score` and no `script_score` (the custom function is the low-level equivalent; no scripting language). - A function with no value does not apply (Elasticsearch raises an error for a missing field without `missing`, and for a negative score). - No `now` default for date origins; no unit strings (`"2km"`, `"10d"`): numbers, and `TimeSpan` in C#. - Only the nearest value of a multi-valued field decays; no `multi_value_mode`. - `min_score` is `MinScore`; `weight` per function and `filter` per function are kept; `first` is `First`. ## How it is checked `FunctionScoreTests` compares the engine with these formulas computed directly from the documents' own values (not the columns), with the inner score taken from the same search without functions: random corpora over several segments with updates and deletes, random functions (every decay shape on numbers, dates and geo points, every modifier, `Missing` and filters, every score and boost mode, `MaxBoost`, `MinScore`), all matches compared with a relative tolerance of 1e-6 (the measured worst deviation is about 1e-7, 32-bit float rounding), the top 7 against a brute-force rescoring of every match, and the explanation of every hit against its score. # Ranking rules An optional way to rank results: a bucket sort over a list of rules, in the manner of Meilisearch's documented ranking rules. Without rules a search is ranked by the BM25F score ([scoring.md](https://recursiveai.org/oneindex/docs/scoring/)), which already includes a term-proximity bonus. Rules are for applications that want results ordered by explicit, explainable criteria: "documents with more of the query words first, then fewer typos, then closer words, ...". ## Choosing rules - A type declares them with `[RankingRules("words", "typo", "proximity", "attribute", "sort", "exactness")]`; they are recorded in the index (schema section 5) and apply to the next query. At the low level, `TextSettings.RankingRules`. - A search replaces them: `Search(text).RankingRules("words", "Price:asc")`, or `RankByRelevance()` to ignore the declared rules. - Query JSON: `"rankingRules": ["words", "typo"]` (an empty list ranks by relevance). `SearchTool` lists the property. - Meilisearch's default order is `words, typo, proximity, attribute, sort, exactness` (`RankingRule.MeilisearchDefault`). There is no default: with no rules the score ranks. A rule is one of `words`, `typo`, `proximity`, `attribute`, `sort`, `exactness`, or `field:asc` / `field:desc` for a property with a column (`[Sortable]`, `[Filterable]` or `[Facet]`). A built-in rule or a field may be listed once; at most 12 rules. ## How the sort runs The rules, in order, are the keys of one sort ([sorting.md](https://recursiveai.org/oneindex/docs/sorting/)); the relevance score is the last key and the document's identifier breaks what remains, so the order is total and the same on every run. Every rule's value is computed per matching document and a smaller value is better. Because it is a sort over all matches: - `TotalHits` is exact and the search visits every match. - Paging works with `Skip`/`Take` and with search-after cursors (`hit.Cursor`, `SearchAfter(...)`, `"searchAfter"` in JSON); a cursor is valid for the rules it came from. - Facets, aggregations, filters, highlighting, `Explain()` and `ScoreBy` (it shapes the relevance score, the last key) work as usual. `Distinct` cannot be combined with rules. Hybrid searches ignore rules (they rank by fused position). - With the `words` rule the search words are OR-ed; without it every word must match, as without rules. - The rules apply to user text. For a typed query object (`SearchBy`) there are no query words, so `words`, `typo`, `proximity`, `attribute` and `exactness` are all 0 and only `sort`, field rules and relevance order the results. ## The rules "Query words" are the words and quoted phrases of the user text, in text order, after analysis and stop words. A phrase counts as one query word for `words` and takes no part in the other rules (it matches exactly and in order). Synonym readings count towards the word they replace for `words`; the other rules look at the word as typed. | Rule | Value of a document (smaller is better) | Choices where the documentation is silent | |---|---|---| | `words` | Number of query words (or phrases) the document does not match. | Meilisearch drops words from the end of the query only when needed to fill the page; here a document that lacks the last words simply sorts after documents that have them, which gives the same order. | | `typo` | Sum, over matched words, of the typos of the closest matching term (0 for a word found as typed). | Typos are counted as in [typo-tolerance.md](https://recursiveai.org/oneindex/docs/typo-tolerance/) (a first-letter typo is two). | | `proximity` | Sum over consecutive matched words (in query order) of the smallest distance between an occurrence of one and an occurrence of the next, in one field. Adjacent in order is 1; the second before the first costs one more than its distance (so query order wins ties); the distance is capped at 8, which is also the cost of two words that never occur in the same field. | The cap of 8 and the order penalty are ours. A document with fewer than two matched words has 0. | | `attribute` | Per matched word, the rank of the first field that holds it (fields in the type's declaration order, so earlier properties are more important) times 100,000,000 plus the word's first position in that field (capped at 99,999); summed over the words. | The documentation ranks by the order of the searchable attributes and then position; combining both in one number is ours. | | `sort` | The search's `OrderBy`/`ThenBy` keys, in place. With no ordering it changes nothing. | Meilisearch applies `sort` only when the request has a sort; so does this. Without a `sort` rule an `OrderBy` has no effect. | | `exactness` | Number of matched words that were found only with typos or as a longer word (a prefix match). | The documentation also prefers documents whose field equals the query; that part is not implemented. | | `field:asc` / `field:desc` | The column's value; missing values last. | Multi-valued fields sort by their smallest (asc) or largest (desc) value, as in [sorting.md](https://recursiveai.org/oneindex/docs/sorting/). | Matched means the document contains the word (as typed, with typos, or as a beginning of a longer word, per the typo-tolerance settings). When a word has several matching terms the closest one is used. ## Explain With `Explain()` each hit's explanation has the bucket of every key, in order, as a child, plus the text score's explanation as the last child. The root value is the text score, as for any hit (a test asserts it equals `SearchHit.Score`). ``` 0.5731 = ranking rules in order (smaller buckets first), then the text score; the root value is the text score 0 = words: 0 of 2 query words missing 0 = typo: 0 typos in the matched words 1 = proximity: 1 summed distance between matched words (adjacent = 1, at most 8 each) ... ``` ## Checks `tests/OneIndex.Tests/RankingTests.cs` compares the order of generated corpora against a brute-force sort by the rule tuple computed from the raw tokens (48 cases over four rule lists, 80 to 400 documents, several segments), and the order of equal tuples by score. Relevance numbers for the Meilisearch default order, as information, are in [relevance.md](https://recursiveai.org/oneindex/docs/relevance/). ## Relevance of the rules mode (information) `RankingRulesRelevanceReport` (a test that runs when `ONEINDEX_BEIR_DIR` is set) searches the BEIR sets through the typed search with the queries as written (typo tolerance and prefix search off, title weight 3). nDCG@10 under the BEIR convention, test queries: | Set | `words` only (any word, ties by BM25F) | `words, typo, proximity, attribute, exactness` | BM25F with the proximity bonus (docs/scoring.md) | |---|---|---|---| | SciFact (300) | 0.5555 | 0.5018 | 0.6918 | | SCIDOCS (1,000) | 0.1098 | 0.0964 | 0.1544 | Ranking by the number of matched words first is the wrong shape for long natural-language queries (most words are noise, so documents that happen to match one more noise word outrank the relevant ones), and the later rules reorder inside those buckets. This is why BM25F with the proximity bonus is the default and the rules are opt-in, for catalogues and short queries where the buckets are what the user wants. # Typo tolerance and as-you-type What user text does when a word is misspelled or half typed. It applies to forgiving user text (`index.Search("...")`). The strict parser and query JSON keep their explicit `term~edits` fuzzy semantics and do not use these settings. Behaviour and defaults follow Meilisearch's published *Typo tolerance* documentation (settings `enabled`, `minWordSizeForTypos`, `disableOnWords`, `disableOnAttributes`, `disableOnNumbers`, and the description "How typo tolerance works"). Where this page says "counts as" it states what the documentation says; the exact edit model (below) is ours. ## What counts as a typo For a query word `q` and an indexed word `w`, both as code points after analysis: typos(q, w) = osa(q, w) + (first(q) != first(w) ? 1 : 0) - `osa` is the optimal string alignment distance: insertion, deletion, substitution, and the swap of two adjacent characters each cost one (a swapped pair is not edited again). A transposition therefore counts as one typo. - A typo on the first character counts as two: when the first characters differ, one more typo is charged. `caturday` is two typos from `saturday`, so it matches only a word with a budget of two. ## Budget per word | Query word length (code points) | Typos allowed | |---|---| | below `MinWordSizeForOneTypo` (default 5) | 0 | | from `MinWordSizeForOneTypo` | 1 | | from `MinWordSizeForTwoTypos` (default 9) | 2 | Never more than two. Settings are validated: `0 <= one <= two <= 255`. The length is that of the analyzed word (after stemming, if the field's analyzer stems). A word gets no typos (the budget is 0) when typo tolerance is off, when it is in `DisableOnWords` (compared ignoring case, against the analyzed word and the word as typed), or when `DisableOnNumbers` is set and the word is only digits. In a field listed in `DisableOnFields` only matches without typos count. ## As-you-type The last word of the text matches words that **begin** like it (prefix search), with typos allowed in that beginning: a word `w` matches `q` as a beginning when some non-empty beginning `p` of `w` has `typos(q, p)` within the budget. The budget is that of the typed word, so a short word being typed (under 5 characters) matches exact beginnings only. The last word is the one being typed unless: the text ends in a space or a quote, it is quoted, it is excluded (`-word`), or the settings turn prefix search off. Words before the last are matched as whole words with their typo budget. (chosen here: the trailing space rule is not taken from Meilisearch's documentation, which does not state it.) Closer matches score higher and every expanded word is scored as the word it is, so a document that matches the full word does not move down the list as the word is typed out; a test follows one target character by character. ## Ranking BM25F stays the base score. Each indexed word that a query word expands to is scored as a term of its own (its own document frequency and idf) and multiplied by two factors; the products are summed over the expansions a document matches: boost(match) = typo factor * exactness factor typo factor = 1 / (1 + typos) 1, 0.5, 0.333 for 0, 1, 2 typos exactness factor = 1 for a whole word, 0.8 for a longer word that only begins like the query word The constants are choices made here; the order they give (exact word, then longer words, then typos) is what Meilisearch's `typo` and `exactness` ranking rules describe. Proximity is left to the ranking-rules work. With typo tolerance and prefix search off the query is the plain BM25F term query, so scores are unchanged. `Explain` shows each expansion with its typos, the typo factor, the exactness factor and the boost (`docs/scoring.md`). When a word expands to more than `MaxExpandedTerms` (1,024) indexed words, the closest, then whole words before longer ones, then the most frequent are kept; this expansion never throws, because user text must not fail on a short word. ## Matching `TypoTermQuery` (`OneIndex.Search`) holds the query word, its fields, `MaxTypos`, `Prefix` and the fields without typos. Candidates come from the term dictionary with Levenshtein automata (Schulz and Mihov 2002), a superset that each candidate is then costed against with the reference computation above: - budget 1: the first character is fixed (a different first character would cost at least two): one automaton run with that character as the enumeration prefix; - budget 2: the same run with two edits, plus a run with one edit and no fixed first character (a different first character costs one extra); - as-you-type: the automaton is wrapped so that once a non-empty beginning is within the budget every longer word is accepted (prefix Levenshtein); with budget 0 it is a plain prefix scan. ## Settings (attributes on the document type) [TypoTolerance(Enabled = true, MinWordSizeForOneTypo = 5, MinWordSizeForTwoTypos = 9, DisableOnNumbers = false, DisableOnWords = ["shrek"], DisableOnFields = ["Code"])] [PrefixSearch(true)] [Synonyms("tv", "television")] // a group: each member also matches the others [OneWaySynonyms("laptop", "notebook")] // a query for laptop also matches notebook [StopWords("the", "of")] public class Product { ... } All settings are query-time. They are recorded in the schema of each commit (`docs/commit-format.md`, schema version 2), so the index knows the settings it was last written with. Opening an index whose attributes differ only in these settings is not an error: the attributes in the code are used at once, `IndexWriter.SchemaSettingsChanged` reports it, and the next commit records them. - **Synonyms** are analyzed with the field analyzer and matched with the machinery of `docs/synonyms.md`, per operand (a word or a quoted phrase, up to three words), at the synonym boost of 0.8. Synonym words are matched exactly, without typos. Synonyms spanning several operands are not applied. - **Stop words** are dropped from user text before matching (also inside a phrase, which keeps its gap). They are not removed from the index. A text of only stop words matches nothing. ## Checks `tests/OneIndex.Tests/TypoToleranceTests.cs`: the cost against an independent recursion; the documented examples; the automaton matcher against a naive scan of every indexed word, for 0 to 2 typos with and without prefix matching over generated vocabularies (`ONEINDEX_TYPO_CASES` and `ONEINDEX_TYPO_WORDS` scale it); ranking order; explain; schema round trip and reopen; the attributes end to end; ranking while a word is typed character by character. # Synonyms and query-side expansion Synonyms are applied at **query time** over analyzed terms. Nothing is stored in the index for them, so changing the synonym table never needs a reindex. Semantics follow Meilisearch's documented ones (the *Synonyms* page of its relevancy documentation: one-way and mutual associations, multi-word synonyms treated as phrases, normalization with the index settings, the exact query taking precedence over its synonyms, the 1 to 3 word and 50 / 100 word limits). Terms are analyzed by the field's own analyzer, so case, accents, stemming and stop words behave as in the index. ## Design: a token graph, then query objects The analysis layer builds a **token graph** (`OneIndex.Analysis.TokenGraph`), a lattice over the query. The search layer lowers it to the existing query objects (`OneIndex.Search.TokenGraphQueries`, which needs no new query kinds): - Nodes are the boundaries between query words (`n` words give nodes `0..n`). - An **original edge** `i -> i+1` carries the analyzed word (one edge per stacked form when the analyzer stacks a stem beside the word). - A **synonym edge** `i -> j` is added for every rule whose key is the analyzed run of words `i..j-1` (1 to 3 words), one edge per alternative, carrying the alternative's terms (1 or more words). - A **reading** of the query is a path from node 0 to the last node. The original reading comes first. This is the equivalent expansion that keeps positions right: a multi-word synonym is a single edge, so it matches as a phrase, and a phrase query over a word with a synonym is rewritten as one phrase per reading instead of a bag of positions. ### Rules - `SynonymRules.AddOneWay(from, to...)`: `from` also finds each `to`, not the reverse (`phone => iphone`). - `SynonymRules.AddGroup(a, b, c)`: every member reads as every other (`shoe <=> boot <=> slipper`); the same as one-way rules from each member to all the others. `AddAll` takes Meilisearch's settings shape (key to list). - **Not transitive**: with `a => b` and `b => c`, `a` reads as `a` or `b`, never `c` (Meilisearch's mutual association is spelled out per member for the same reason). - `Build(analyzer, limits)` analyzes every key and alternative. A phrase that analyzes to no terms (all stop words) is dropped; an alternative equal to its key is dropped; rules with the same key merge in the order added. Keys longer than `MaxQueryWords` can never fire and are dropped. ### Limits (`SynonymLimits`) | Limit | Default | Behaviour past it | |---|---|---| | `MaxQueryWords` | 3 | A run of more query words is not looked up (Meilisearch: "terms with 4 or more words will not return any synonym matches"). A rule *whose alternative* is longer is fine (`lotr => lord of the rings`). | | `MaxSynonymsPerTerm` | 50 | Later synonyms of the key are dropped, in the order added; counted in `SynonymMap.DroppedSynonyms`. | | `MaxWordsPerTerm` | 100 | The first synonym that would take a key's synonyms past 100 words, and all later ones, are dropped. | | `MaxPaths` | 64 | Readings of one query. Synonym edges are added left to right, shorter spans first, in rule order; an edge that would take the path count past the cap is left out (`TokenGraph.IsTruncated`). The original reading is always kept. | A bag-of-words query does not multiply readings: `CutNodes` finds the nodes every reading passes through, the readings of each stretch between them are alternatives (Should), and the stretches are combined with Must, so the clause count grows with the sum of the stretches' readings, not their product. A phrase query needs the product (one phrase per whole reading), which `MaxPaths` bounds. ### Lowering `TokenGraphQueries.ToWords(graph, field, synonymBoost)`: all words must match, in any position. A reading's original edges are `TermQuery` clauses; a synonym of one word is a `TermQuery`, of several words a `PhraseQuery` (Meilisearch: multi-word synonyms are phrases). A reading is a `Must` group of its parts; readings of a stretch are `Should`. `TokenGraphQueries.ToPhrase(graph, field, slop, synonymBoost)`: one `PhraseQuery` per distinct whole reading (a `TermQuery` for a one-term reading with slop 0), combined with `Should`. **Exact query first.** Parts that come from a synonym carry `synonymBoost` (default **0.8**); the original words carry 1, so a document with the user's own words outscores one with only a synonym. A document with both scores both. A query with no terms (empty, or only stop words) lowers to `MatchAllQuery`. ## Stop words See docs/analysis.md. In short: configured stop words are dropped by the analyzer, so index and query sides agree, and stop words vanish from phrases. Positions: ### Positions and gaps `StopWordSettings.PreserveGaps` (default **false**) decides what a dropped word does to positions. - **Collapsed (default).** The neighbours become adjacent: `king of queens` indexes `king@0 queens@1`. The query phrase `"king of queens"` is `[king, queens]`, exact, and matches text with the stop word and text without it. This is Meilisearch's documented behaviour (stop words are ignored in phrases: `"the great gatsby"` matches what `"great gatsby"` matches). The cost is that proximity cannot tell `king queens` from `king of queens`. - **Preserved.** `king@0 queens@2`. Positions stay truthful, but the query objects have no per-term positions, so a phrase with a removed word becomes a `PhraseQuery` over the remaining terms with `Slop` raised by the number of removed positions between them. That accepts a little extra distance beyond the removed words; it is an over-approximation, documented as such. Synonym rules are analyzed with the same analyzer, so a rule written `the big apple => nyc` fires on the query `big apple`. ## Verification `tests/OneIndex.Tests/SynonymTests.cs`: Meilisearch's documented examples (great/fantastic, phone => iphone, the four-way shoe group, san francisco / sf, lord of the rings / lotr, 50 and 100 limits, the Résumé / CV normalization case, exact-first scoring); overlapping rules; expansion bounds; and 4 seeded runs of 2,400 random (rules, query, document) cases, each lowered query evaluated by a small reference evaluator and compared with an independent recursive enumeration of readings, for both the bag-of-words and the phrase forms (0 disagreements). `StopWordConfigurationTests.cs` covers the stop-word cases. # Did you mean `OneIndex.Suggest.DidYouMean` proposes corrected queries from the terms of a field's dictionary in a snapshot. It is separate from typo-tolerant *matching* (which finds documents for the typo itself); this is the "Did you mean *x*?" line. ## Algorithm 1. **Tokenize** the query with the field's analyzer. Each word keeps its offsets in the query text. Numbers are never corrected. 2. **Known words.** The document frequency of a word is summed over the snapshot's segments (`TryGetTerm`; deleted documents still count, as everywhere in term statistics). In the default `Missing` mode only words with frequency 0 are corrected. In `Popular` mode a known word is also corrected when a close term has at least `PopularFactor` (10) times its frequency. 3. **Edit budget** by the word's length in code points: shorter than `OneEditMinLength` (4): none, the word is left alone; from 4: one edit; from `TwoEditsMinLength` (6): two. Edits are insertion, deletion, substitution and the swap of two adjacent code points, each 1 (optimal string alignment). The first `PrefixLength` (1) code points must be right; people rarely mistype the first letter and fixing it keeps the walk small, and `PrefixLength = 0` lifts it (measured below: it costs about 0.3 to 0.6 points). 4. **Candidates.** For each segment, `ITextFieldReader.Intersect(LevenshteinAutomaton(word, budget), prefix)` walks the term dictionary together with the automaton (Schulz and Mihov 2002), visiting only the neighbourhood of the word. At most `MaxTermsVisited` (20,000) terms are looked at per word and segment, in byte order, so a cut-off is deterministic. Candidates are merged across segments by term, their frequencies summed, and their distance recomputed with `EditDistance` (the same distance the automaton uses, checked against a dynamic-programming matrix on 3,000 random pairs). The best `CandidatesPerWord` (4) are kept: fewest edits, then most documents, then UTF-8 byte order. 5. **Combine.** A multi-word query is a left-to-right beam search (`BeamWidth` 8) over each word's options (its candidates; or itself when it is known or has no candidates). A partial suggestion scores `sum( FrequencyWeight * ln(1 + df) - EditPenalty * edits ) + CooccurrenceWeight * sum( ln(1 + both) )` where `both` is the number of documents that contain a word and the word before it (two adjacent words), counted by leapfrogging their postings lists, at most `MaxCooccurrenceScan` (50,000) steps per pair and segment. Defaults: weights 1, 1, `EditPenalty` 4 (one more edit costs as much as the candidate being ~55 times rarer). 6. **Output.** Combinations that change nothing are dropped; the rest are sorted by score, then by text, and cut to `MaxSuggestions` (3). The suggested text is the query with each corrected word's span replaced by the dictionary term, so the punctuation and untouched words are exactly as typed. The replacement is the indexed (case-folded) form. Everything is deterministic: ties break by term bytes, the beam sorts on (score, choice indexes), and no step depends on hash order. **Cost bounds** per word: one automaton walk per segment (at most `MaxTermsVisited` terms) plus one `TryGetTerm` per segment; per pair of words at most `BeamWidth * CandidatesPerWord` co-occurrence counts, cached within the call. **Stemming.** Dictionary terms of a stemmed field are stems, so suggestions from such a field are stems. Use an analyzer that does not stem for the suggestion field (or a `-both` analyzer's unstemmed terms are indexed beside the stems, but the dictionary cannot tell them apart). The function never throws on query text, including control characters and lone surrogates, and honours the cancellation token between segments and every 256 terms. `SuggestAsync` is the asynchronous twin; readers are synchronous over cached ranges, so it completes synchronously (it works in the browser, where nothing depends on threads). ## Measured accuracy Method: the English word list of the Snowball vocabulary data (28,266 lowercase words of 3+ letters; fetch-only), 20,000 documents of 8 to 15 words drawn with Zipf frequencies (seed 5), 1,000 queries per row. A query is a run of 1 (or 2 to 3) words from a document with one word replaced by a typo of exactly 1 or 2 edits (random substitution, insertion, deletion or swap; the word has 4 or more letters for one edit, 6 or more for two; with prefix 1 the first letter is untouched), where the typo is not itself a word of the corpus. *top-1* is the fraction in which the first suggestion equals the original text; *top-3* any of the three. "Engine = brute force" compares the single-word candidate list with a brute-force scan of the whole vocabulary under the same rule and ordering. | Words | Edits | Prefix | top-1 | top-3 | Engine = brute force | |---|---|---|---|---|---| | 1 | 1 | 1 | 95.2% | 98.1% | 1000 / 1000 | | 1 | 1 | 0 | 94.9% | 97.9% | 1000 / 1000 | | 1 | 2 | 1 | 85.3% | 90.5% | 1000 / 1000 | | 1 | 2 | 0 | 84.8% | 90.1% | 1000 / 1000 | | 2 to 3 | 1 | 1 | 97.2% | 98.5% | | | 2 to 3 | 1 | 0 | 96.5% | 98.1% | | | 2 to 3 | 2 | 1 | 88.3% | 90.8% | | | 2 to 3 | 2 | 0 | 88.3% | 91.1% | | The two-edit threshold was chosen from a run on words of 8 or more letters (typos that delete letters can shorten them below the threshold): two-edit single-word top-1 was 85.5% with the threshold at 8, 92.9% at 7 and 94.1% at 6, with the one-edit rows unchanged. The rows above also include 6- and 7-letter words, the harder set (Meilisearch's own typo threshold is 9 letters; ours is for suggesting a replacement for a word the index does not contain at all, so a lower one is affordable). Per call, one typo, the same corpus (Release, x64, 32-thread host at load ~18, indicative only): one-word query 207 microseconds and 4.2 KB allocated; three-word query 227 microseconds and 31.7 KB allocated. The edit distance kernel allocates nothing (`TestHelpers.AssertNoAllocation`, strings of up to 64 code points on a fixed stack buffer). The call as a whole allocates (strings for candidates, result lists), because the automaton and the term enumerators are created per call. ## Verification `tests/OneIndex.Tests/DidYouMeanTests.cs`: edit distance against a full matrix (3,000 random pairs including supplementary characters), no allocation, unit cases (frequency, co-occurrence tie-break, popular mode, several segments, cancellation, punctuation preserved, nothing suggested for known or short words or numbers), seeded synthetic-vocabulary accuracy with an exact brute-force comparison of candidate lists, and the real-vocabulary report above (skipped without `ONEINDEX_SNOWBALL_DATA`). # Sorting and paging This note is normative for ordering hits by columns and for paging with a cursor. The columns are in [columns.md](https://recursiveai.org/oneindex/docs/columns/); the query surfaces that expose it are [linq.md](https://recursiveai.org/oneindex/docs/linq/) and [query-json.md](https://recursiveai.org/oneindex/docs/query-json/). ```csharp var page = index.Search("overdue") .Where(x => x.Paid == false) .OrderByDescending(x => x.Modified) .ThenBy(x => x.Customer) .Take(20) .Execute(); var next = index.Search("overdue").Where(x => x.Paid == false) .OrderByDescending(x => x.Modified).ThenBy(x => x.Customer) .SearchAfter(page.NextCursor!).Take(20).Execute(); ``` ## The order A sort is one to sixteen **keys**, applied in turn; a later key only separates documents the earlier ones tie. The order is total: 1. Each key compares the documents' values. A key is a column (`FieldSort`), the distance from a point (`GeoDistanceSort`, see [geo.md](https://recursiveai.org/oneindex/docs/geo/)) or the relevance score (`ScoreSort`). 2. Documents with no value for a key go **last**, or **first** when the key says so (`MissingValuePlacement`). The placement does not depend on the direction: a descending key with missing values last puts them after the smallest value. (LINQ-to-Objects would put a missing value first when ascending; this follows Meilisearch and Elasticsearch, which put missing values last, and the placement is a parameter.) 3. The final tie-break is the document identifier in byte order of its UTF-8 text when the schema has an identifier field, else the global document id. The identifier tie-break is what keeps the order the same across merges, which renumber documents. It is byte order, so it agrees with `string.CompareOrdinal` for text without supplementary-plane characters. ### Value order per column type | Column | Order | |---|---| | keyword | byte order of the UTF-8 text | | integral, date | the number; dates are milliseconds since 1970 | | bool | false before true | | fractional | numeric order; `-0.0` and `0.0` tie; NaN after every number | | geo point | not sortable as a column; use `GeoDistanceSort` | | text, bytes, vector, JSON | not sortable | A document with several values for a key is placed by its **smallest** value when the key ascends and its **largest** when it descends (Elasticsearch's `min` and `max` modes). ### Score keys `ScoreSort` orders by the BM25F score, descending by default. A sort without a score key does not compute scores at all (hits report `Score` 0), which is what makes it cheap; use `ThenByRelevance()` to break ties by score. NaN scores sort as the lowest. ## Execution `TopFieldCollector` visits every match and keeps the first *k* in a binary heap whose root is the worst kept document, reading the key values from the segment's columns. The count of matches is therefore exact. Keyword values are copied into a buffer the collector owns, and allocation happens only when a document enters the kept set. Facets, aggregations and exact counts ride the same pass (`MultiCollector`). Sorting by a column is not combined with `Distinct` (which ranks groups by relevance); that combination throws `NotSupportedException`. ## Paging - **`Skip` and `Take`** page within the result window (`Skip + Take` at most 10,000). The cost grows with the depth because the collector keeps `Skip + Take` documents. - **Search-after** (`SearchAfter(cursor)`, Elasticsearch's `search_after`; Meilisearch has only offset and limit paging) continues after a cursor and costs the same on every page. A `SortCursor` is the last hit's sort values plus its identifier (or global document id when the schema has no identifier field). A search-after search keeps only documents that sort **strictly after** the cursor under the same total order; `TotalHits` still counts every match of the query. `Skip` and `SearchAfter` do not combine (`InvalidOperationException`). ### Stability The cursor holds values and an identifier, not a position, so it does not depend on which commit it was made against: - A document that exists unchanged from the first page to the last appears exactly once in a full walk, whatever else is added, changed or removed in between. - A document whose sort values change can appear again (or not at all) if its new place is after (or before) the cursor, which is what the new values mean. - A cursor over a score key is as stable as the scores, which depend on the whole collection. - A cursor made without identifiers (a low-level index whose schema has no identifier field) holds the global document id, which a merge changes: it is valid within one snapshot only. The walk test (`SortingDifferentialTests`) checks every page of every walk against a model that applies the same order to the current documents. ### Token form `SortCursor.ToToken()` writes `version 1`, a CRC-32C of the canonical sort description, the key count and, per key, a kind byte (0 missing, 1 number as 8 bytes little-endian, 2 text as varuint length and bytes), then the identifier (kind 1: varuint length and bytes) or the document id (kind 2: 4 bytes), in unpadded URL-safe base64. A cursor only fits the sort that made it: the checksum is compared and a different sort raises `ArgumentException`. `SortCursor.TryParse` rejects malformed tokens without throwing and never reads past its input (at most 64 keys, 64 Ki characters). ## Query JSON `"sort"` is an array of strings, after Meilisearch's sort parameter: `Field:asc`, `Field:desc`, `_geoPoint(lat, lng):asc` (naming the geo field first, `_geoPoint(Place, lat, lng)`, when the schema has several), `_score` (descending unless `:asc`), each optionally ending in `:first` or `:last` for missing values. `"searchAfter"` is a cursor token. The canonical writer always gives the direction and writes `:first` only when it differs from the default. ## Limits Sixteen keys; windows of 10,000; cursors of 64 Ki characters. The collector's memory is proportional to `Skip + Take`. # Facets, facet value search and distinct This note is normative for `OneIndex.Aggregations.FacetCollector`, `FacetValueSearch`, `OneIndex.Search.DistinctCollector` and the happy-path calls `Facets` and `Distinct` of `SearchQuery`. Aggregations (buckets and metrics beyond facets) are in [aggregations.md](https://recursiveai.org/oneindex/docs/aggregations/). The column layouts they read are in [columns.md](https://recursiveai.org/oneindex/docs/columns/). The vocabulary follows Meilisearch's documented one: *facet distribution* (counts per value), `maxValuesPerFacet`, `sortFacetValuesBy` (`alpha` or `count`), *facet stats* (minimum and maximum of numeric facets), *facet search* (search within a facet's values) and `distinctAttribute`. ## Facets A facet is the count of each value of a field over **all documents the query matches**, not over the page. Fields: keyword, bool, integral, fractional and date fields that keep a column (`[Facet]`, `[Filterable]` or `[Sortable]` on the property; the property may be a collection when it is keyword or numeric). Text, geo, vector, bytes and JSON fields cannot be faceted (`ArgumentException` naming the field). - **Counting.** A document counts once for each distinct value it has. A document listing `["a", "a", "b"]` adds 1 to `a` and 1 to `b`. A document without a value adds nothing. Deleted documents never count. Counts are exact. - **Values as text.** `FacetValue.Value` is the keyword; `true`/`false`; the invariant round-trip text of a number (`12`, `0.5`; `-0` is `0`); an ISO 8601 UTC time with milliseconds (`2024-02-05T00:00:00.000Z`) for dates. `FacetValue.Key` also carries the number. - **`MaxValues`** (default 100, at least 1) keeps the first values in the requested order. `DistinctValues` reports how many distinct values existed before the cut. - **`SortBy`.** `Alpha` (the default, as in Meilisearch): keywords by ordinal comparison of UTF-16 code units, numbers and dates by value, `false` before `true`. `Count`: most documents first, equal counts in `Alpha` order. Both orders are total, so the same snapshot always gives the same list. - **Stats.** For integral and fractional fields `FacetResult.Stats` holds the minimum and maximum over the matching documents' values (null when no matching document has a value). Dates, bools and keywords have none. ### Cost and exactness Counting needs every match, so a facet search cannot use block-max WAND skipping. In the happy path, asking for any facet, aggregation or distinct makes the top-hits collector count exactly (`TotalHits` is exact, as if `CountExactly()` had been called) and the engine visits every match once, feeding all collectors in the same pass (`MultiCollector`). Cost is about one column read per matching document per facet, plus the page's scoring; the top-k collector sees no skip hints. A search that only needs the page and a lower bound stays pruned: leave facets off it. Per segment a keyword facet keeps one `int` counter per dictionary term (and one `int` stamp per term when the column is multi-valued, to count a document once per value). Collecting a document allocates nothing; the keywords are read from the dictionary only once per distinct term per segment at its end. Memory is proportional to the segment's distinct values. ### Low level ```csharp var facets = new FacetCollector(schema, [new FacetRequest("tags") { SortBy = FacetSort.Count, MaxValues = 20 }, new FacetRequest("price")]); searcher.Search(query, facets); IReadOnlyList r = facets.GetResults(); // request order ``` Happy path: `index.Search("shoes").Facets(x => x.Tags).Facets(x => x.Price, sortBy: FacetSort.Count)`; read `results.Facets["Tags"]` (by property name). The property must keep a column; a property that is already faceted is refused. ## Facet value search `FacetValueSearch.Search(searcher, query, field, text, limit)` answers "which values of this facet match what the user typed", counted over the documents `query` matches (so the counts reflect the current filters). `Match(values, text, limit)` applies the same rules to counts already in hand. The result is ordered by value (ordinal; the default, as in Meilisearch) or, with `sortBy: FacetSort.Count`, by count (most documents first, ties by value), then cut at `limit` (default 100). On the happy path the same search is `index.Search(...).Where(...).FacetValues(x => x.Category, "sho")` (and `FacetValuesAsync`): it counts over the documents of that search, ignoring its paging. Normalization of both the value and the input: NFKC, full case folding, accent removal (the same steps as the standard analyzers; `ß` becomes `ss`, `fi` becomes `fi`, final sigma folds to sigma). Then both are split into words (runs of letters, digits and combining marks). A value matches when 1. the input is empty, or 2. the normalized value starts with the normalized input (so punctuation inside the input still works), or 3. every input word matches some word of the value, where an input word `q` matches a value word `w` if `w` starts with `q`, or, within a **typo budget**, if `q` is within that many edits of a prefix of `w` whose length differs from `q`'s by at most the budget. Typo budget by the input word's length in code points, as Meilisearch documents for search: none up to 4, 1 from 5, 2 from 9. An edit is an insertion, deletion, substitution or the swap of two adjacent characters (optimal string alignment, the distance of `Suggest.EditDistance`). Meilisearch's reference says facet search considers only the first word of the query and does not search numeric values. Documented differences from it: every input word is used and may match any word of the value (so `york` finds `New York`, which Meilisearch would not return), numeric and date values are searched as their text, and matching is on whole values by the rules above (words of the value, not the engine's tokenizer, so no language-specific segmentation: a CJK value is one word, so it matches by prefix); there is no separate setting for typo thresholds. ## Distinct `DistinctCollector(field, count)` and `SearchQuery.Distinct(x => x.Property)` keep the best document of each value of a column. - **Groups.** Documents with the same value of the column form a group. A document with several values belongs to the group of its first value (values keep the order the document gave them). A document with no value is a group by itself, as in Meilisearch, where documents without the distinct attribute are not collapsed. - **Ranking.** A group is ranked by its best document: higher score first; equal scores by lower global document id. The best document of a group is chosen the same way. Both are total orders, so a page is a pure function of the snapshot and the query, and `Skip`/`Take` pages tile without overlap or gap. - **Totals.** In the happy path `TotalHits` is the exact number of groups. `DistinctDocs` (low level) reports both the groups and the matching documents. - **Cost.** All matches are visited (a group's best score is known only at the end) and a score is computed for each. Memory is proportional to the number of groups (about 8 bytes plus the dictionary entry for each); it is not bounded by the page. `Skip + Take` stays within the window limit of the happy path (10,000). - Geo points cannot be collapsed by (`ArgumentException`). # Aggregations This note is normative for `OneIndex.Aggregations` (`Aggregation` requests, `AggregationCollector`, the result types and `AggregationBuilder`). Facets, which share the machinery, are in [facets.md](https://recursiveai.org/oneindex/docs/facets/). Names follow Meilisearch where it has a term and Elasticsearch's published aggregation documentation otherwise. An aggregation is computed over **every document a query matches** (deleted documents excluded), from the fields' columns ([columns.md](https://recursiveai.org/oneindex/docs/columns/)). The request classes are plain data (a name, a field, a few settings, optional sub-aggregations) so they can be built in code or read from a description. Results come back in request order in an `AggregationResultSet`, addressable by name. ```csharp var collector = new AggregationCollector(schema, [ new TermsAggregation("by_tag", "tags") { Size = 5, SubAggregations = [new StatsAggregation("price", "price")] }, new DateHistogramAggregation("per_month", "added") { CalendarInterval = DateInterval.Month }, new PercentilesAggregation("p", "price") { Percents = [50, 99] }]); searcher.Search(query, collector); BucketsResult tags = collector.GetResults().Get("by_tag"); ``` Happy path: `index.Search(...).Aggregate(a => a.Terms("by_tag", x => x.Tags, size: 5, sub: s => s.Stats("price", x => x.Price)).DateHistogram("per_month", x => x.Added, DateInterval.Month))`, results in `SearchResults.Aggregations`. The builder exposes `Terms`, `Range`, `Histogram`, `DateHistogram`, `Stats`, `Cardinality` and `Percentiles`; the request classes carry every option. Validation happens when the collector is built (`ArgumentException` naming the aggregation): the field must exist, keep a column, and have a type the aggregation reads; names must be unique among siblings; settings must be in range. ## Counting rules (all bucket aggregations) A document counts **once per bucket**, however many of its values fall in it. Sub-aggregations see the whole document (all its values of the sub-field), not only the values that put it in the bucket. Documents without a value of the bucketed field are in no bucket. Counts are exact. ## Buckets | Request | Fields | Buckets | |---|---|---| | `TermsAggregation` | keyword, bool, integral, fractional, date | one per distinct value. `Size` (default 10) buckets in `Order` (count descending, key ascending or descending; count ties by key ascending), dropping those below `MinDocCount` (default 1). `OtherDocCount` sums the counts of buckets cut by `Size`. Keys: keywords by ordinal UTF-16 order, numbers and dates by value; `-0` and `0` are one key. | | `RangeAggregation` | integral, fractional, date | one per requested range, in request order; `From` inclusive, `To` exclusive, either may be open; ranges may overlap. Values compare as `double` (exact to 2^53, dates are milliseconds). Label `from-to` with `*` for an open end, or the given key. | | `HistogramAggregation` | integral, fractional | fixed width: value `v` goes to bucket `floor((v - Offset) / Interval)`, key `id * Interval + Offset`. With `MinDocCount` 0 (the default, as in Elasticsearch) empty buckets between the first and last used bucket are returned; with 1 or more only buckets that qualify. NaN, infinite or overflowing values are ignored. | | `DateHistogramAggregation` | date | UTC time buckets: a `CalendarInterval` (second, minute, hour, day, week starting Monday, month, quarter, year) or a `FixedInterval` (whole milliseconds, counted from the epoch plus `Offset`). Key `Whole`/`Number` is the bucket start in milliseconds; `Text` is ISO 8601. Empty buckets are filled like the histogram. Month, quarter and year buckets ignore values outside years 1–9999. | A histogram that would return more than `AggregationCollector.MaxBuckets` (65,536) buckets throws `TooManyBucketsException` when results are read. ## Metrics | Request | Result | |---|---| | `StatsAggregation` | `Count` (of **values**: a document with three values adds three), `Min`, `Max`, `Sum`, `Avg`. Integral and date sums are exact (128-bit, rounded once to a double); fractional sums use Neumaier compensated summation. NaN values are skipped. All null/zero for no values. | | `CardinalityAggregation` | the number of distinct values, estimated: see below. | | `PercentilesAggregation` | each requested percent with its estimated value (NaN with no values): see below. | Fractional sums depend on the order values are added, so they can differ in the last bits between an index in one segment and the same documents in several; the differential test allows a relative `1e-9`. Everything else is independent of segment layout, merges and deletes. ## Cardinality: HyperLogLog++ Heule, Nunkesser and Hall, "HyperLogLog in Practice: Algorithmic Engineering of a State of the Art Cardinality Estimation Algorithm", EDBT 2013. Values are hashed to 64 bits (the SplitMix64 finalizer for integers, an 8-bytes-at-a-time multiplicative hash with the same finalizer for keywords; no per-process seed, so results repeat). Precision `p` (default 14, range 4–18) gives `2^p` one-byte registers. - **Sparse form.** While at most `2^p / 8` distinct hashes have been seen the sketch keeps the hashes themselves, so the answer is **exact** (up to 64-bit collisions): 2,048 distinct values at the default. This plays the role of the paper's sparse representation. Difference: the paper packs 25-bit codes; here a whole hash costs 8 bytes and the switch point makes the memory equal to the dense form. - **Dense form.** Registers hold the rank of the first 1 bit among the hash bits below the top `p`. The estimate uses the closed-form **improved estimator of Ertl** ("New cardinality estimation algorithms for HyperLogLog sketches", 2017, Algorithm 6) in place of the paper's empirical bias-correction tables and linear-counting switch. It needs no tables, is accurate from empty to saturated, and was verified against the standard error here. Difference to record: results are not bit-identical to implementations that use the paper's tables. - **Error.** Standard error `1.04 / sqrt(2^p)`: 0.81% at `p = 14`. The tests assert at most 4 standard errors over 42 trials per precision at sizes up to 20·2^p and observed 1.6–2.9 (`p` 8, 12, 14, 16). - **Merging** segment sketches is the union (register-wise maximum, or set union while sparse), so the segment layout does not change the answer. ## Percentiles: t-digest Dunning and Ertl, "Computing Extremely Accurate Quantiles Using t-Digests", 2019 (arXiv:1902.04023): the **merging digest** with the arcsine scale function `k1(q) = δ/(2π)·asin(2q − 1)`. Values are buffered and, when the buffer is full, sorted and merged with the centroids in one pass so that no centroid spans more than one unit of `k`; centroids stay tiny at the extremes and grow towards the middle. `Compression` (δ) defaults to 200. A quantile interpolates linearly between the exact minimum, the centroid centres (half a centroid's weight past the weight before it) and the exact maximum. Chosen over other sketches because it is mergeable, has bounded memory (a few hundred centroids), and is accurate where percentile queries look (p1, p99, p99.9). Error measured here at δ = 200 over 300,000 values (uniform, log-normal, sorted, few distinct): rank error at most 0.035% anywhere and 0.013% in the 0.01% and 99.99% tails; the tests assert 0.5% and 0.05%. For fewer values than centroids the digest holds every value and interpolates between neighbours, within one rank. The result is a pure function of the order values are added; the order follows document order within a segment and segment order across segments, so equal indexes give equal answers, while the same documents in a different segment layout may differ within the error bound. ## Cost Like facets, aggregations need every match: they make a search visit all of them, and WAND skipping does not apply (see facets.md, "Cost and exactness"). Collecting a document allocates nothing once a segment's state exists: counters are `int[]` per dictionary term (keywords) or entries of an open-addressing table (numbers, histograms), one stamp per bucket to count a document once, register and buffer arrays for the sketches. The per-segment state is built when the segment starts (sized by its dictionary) and merged into the result when the next segment starts or results are read. A two-facet query over 2,600 documents allocated about 8.5 KB in total, nearly all of it results. ## Verified against `AggregationDifferentialTests`: seeded random corpora (several segments, merged and not, deleted documents, missing and repeated values, multi-valued keyword and integer fields, -0.0) and random queries; every facet, terms/range/histogram/date-histogram bucket (with sub-aggregations), stat, cardinality (exact range) and distinct page equals a LINQ-to-Objects computation. Percentiles are checked by rank against the sorted values. `AggregationSketchTests` checks the two sketches' error bounds on large inputs. # Geo search This note is normative for geo points: the index, the filters and sorting by distance. The terms follow Meilisearch's documented geosearch (`_geoRadius`, `_geoBoundingBox`, `_geoPoint`); the documents hold one point (or several) per geo field instead of a `_geo` object. ```csharp public class Restaurant { [SearchId] public string Id { get; set; } [Geo] public GeoPoint? Location { get; set; } } index.Search().Where(x => Geo.WithinRadius(x.Location!.Value, 45.4727, 9.1840, 2000)) .OrderByDistance(x => x.Location, 45.4727, 9.1840).Take(10); ``` ## Model - **Coordinates** are degrees: latitude -90 to 90 (north positive), longitude -180 to 180 (east positive). `GeoPoint` rejects anything else and NaN. - **Distance** is the great-circle distance on a sphere of radius **6,371,008.8 m** (the IUGG mean Earth radius), by the haversine formula (Sinnott, "Virtues of the Haversine", *Sky and Telescope* 68(2), 1984): `d = 2R asin(min(1, sqrt(sin²(Δφ/2) + cos φ1 cos φ2 sin²(Δλ/2))))`. Against the WGS84 ellipsoid it is off by up to about 0.5%. - **Radius filter** `_geoRadius(lat, lng, meters)`: points at a distance **less than or equal to** the radius. Radius 0 matches a point exactly at the centre. - **Bounding box filter** `_geoBoundingBox([topRightLat, topRightLng], [bottomLeftLat, bottomLeftLng])`: latitude from the south edge to the north edge, edges included; longitude from the west edge **east** to the east edge. When the west edge is greater than the east edge the box crosses the antimeridian (`[10, -170]` top right and `[-10, 170]` bottom left is the strip from 170° east across 180° to 170° west). A box from -180 to 180 spans every longitude. (Meilisearch's documentation does not say how it treats this case; the rule here is ours.) A box is not a circle: its edges are parallels and meridians, not great circles. - A document with several points matches when **any** point matches; a document without a point matches neither filter and sorts after the others by distance. - **Distance sort** orders by the distance to the document's nearest point (farthest when descending) and reports it as `SearchHit.GeoDistance` in metres; see [sorting.md](https://recursiveai.org/oneindex/docs/sorting/). - A geo field needs `Indexed` and `Column` for filters (the `[Geo]` attribute sets both); a `GeoPoint` property with only `[Sortable]` keeps the column and sorts by distance but cannot be filtered. A geo field cannot be a facet or be collapsed by `Distinct`. ## Index Geo points live in the point index of the segment ([points.md](https://recursiveai.org/oneindex/docs/points/)), as a second kind of tree in the same `points` file (format version 2). - **Key.** Latitude and longitude are each quantized to 31 bits (steps of 180/(2³¹-1) and 360/(2³¹-1) degrees, about 8 mm and 17 mm) and their bits interleaved, latitude in the even bits (a Z-order or Morton key, Morton 1966). Points near each other in the plane are mostly near in the key. - **Leaves.** The pairs (key, document) are sorted and cut into leaves of at most 512, exactly like numeric points. Each leaf record also stores its **bounding cell box**, the top 16 bits of the smallest and largest latitude and longitude quantum of its points, four little-endian `u16` in the record's last eight bytes. A cell is about 0.0027° of latitude (300 m), so the box is a conservative cover. - **Directory.** Each field entry carries a kind byte after its name: 0 for numeric, 1 for geo. - **Query.** The query turns its shape into rectangles of quanta, widened by one quantum on every side (two rectangles when it crosses the antimeridian): for a radius the rectangle that bounds the circle (the circle's latitude band, widened in longitude by asin(sin(r/R)/cos φ), Matuschek, "Finding Points Within a Distance of a Latitude/Longitude Using Bounding Coordinates"; spanning every longitude when the circle reaches a pole), widened by 10⁻⁷ degrees so rounding cannot drop a point that lies on the circle. The resident leaf records are scanned linearly: a leaf whose cell box misses every rectangle is skipped, a leaf wholly inside one contributes all its documents, the others have each point's key checked. Only the leaves that survive are read from storage. - **Exactness.** The index only proposes candidates. The query confirms each candidate against the exact coordinates in the column (haversine for a radius, comparisons for a box), so the result does not depend on the quantization. A segment with no geo values for the field matches nothing. - A linear scan of the leaf records costs one 32-byte record per 512 points (a million points: about 2,000 records); a split tree over the leaves (Procopiuc, Agarwal, Arge and Vitter, "Bkd-tree", SSTD 2003) would replace the scan with a descent and is not needed at this size. Merge copies the leaves of the sources through the document map and rewrites the tree, as for numeric points. ## Query surfaces | Surface | Form | |---|---| | Query objects | `GeoRadiusQuery(field, lat, lng, meters)`, `GeoBoundingBoxQuery(field, north, south, west, east)`; sort keys `GeoDistanceSort(field, lat, lng)` | | LINQ | `Geo.WithinRadius(x.Location, lat, lng, meters)` and `Geo.WithinBoundingBox(x.Location, north, south, west, east)` inside `Where` (they combine with `&&`, `\|\|`, `!`); `OrderByDistance` / `ThenByDistance` | | Filter string | `_geoRadius(lat, lng, meters)`, `_geoBoundingBox([lat, lng], [lat, lng])`; the geo field may be named first (`_geoRadius(Location, lat, lng, meters)`) and must be when the schema has several | | Query JSON | nodes `{"type":"geoRadius","field":..,"lat":..,"lng":..,"meters":..}` and `{"type":"geoBoundingBox","field":..,"north":..,"south":..,"west":..,"east":..}`; sort strings `_geoPoint(lat, lng):asc` | The tool schema (`SearchTool`) lists the geo nodes when the type has a `[Geo]` property, and `sort` and `searchAfter` always. ## Documented deviations from Meilisearch - Geo fields have names (`Location`) instead of the single `_geo` object; the filter functions can omit the name when the schema has one geo field. - The antimeridian rule for boxes (above), and several points per document. - Distances use the haversine sphere with the IUGG mean radius. The radius Meilisearch uses is not stated in its documentation, so a point within a few metres of a circle's edge can be on the other side of it there. A black-box comparison was not made. ## Limits Coordinates are validated at construction (`GeoPoint`), at query construction and at parse. A radius is finite and not negative; any radius larger than half the circumference matches every point. # Vectors: distance kernels, HNSW and exact search This note is normative for the vector part of a segment: what the kernels compute, how closeness becomes a score, how the graph is built and searched, what the two files contain and how a merge treats them. Measured results are at the end. ## Metrics and scores A vector field has a metric, chosen in the schema (`[Vector(Metric = ...)]` or `FieldDefinition.Metric`) and stored with it. | Metric | What is indexed | Internal distance (smaller is nearer) | Score (larger is nearer, never negative) | |---|---|---|---| | `Cosine` (default) | the vector scaled to length 1 | minus the dot product | `(1 + cos) / 2`, clamped to 0..1 | | `DotProduct` | the vector as given | minus the dot product | `1 + dot` if `dot >= 0`, else `1 / (1 - dot)` | | `Euclidean` | the vector as given | squared Euclidean distance | `1 / (1 + d²)` | * The query of a cosine field is scaled the same way. A vector of length 0 stays all zeros (its cosine with anything is taken as 0, score 0.5). The stored source keeps the original vector; the vectors file keeps the scaled one, which is what `TryGetVector` returns. * Documents with a NaN or an infinity in a vector are rejected when added (`ArgumentException`), and so are such queries. * A document may lack a vector. A vector field holds one vector per document (no multi-valued vector fields yet). * Scores are monotone in the distance, so ranking by score is ranking by distance. Ties break towards the smaller document id. ## Distance kernels `OneIndex.Kernels.VectorKernels` (internal, no dependency on the rest of the engine) computes dot product, squared Euclidean distance and sum of squares over `float`, and dot product and squared distance over `sbyte`. * **One result on every machine.** The float order of additions is fixed by the kernel, not by the instruction set: element `i` goes to lane `i mod 16`, in index order, with a separate multiply and add (never a fused multiply-add); the sixteen lanes are folded by halves (`j` with `j+8`, then `j+4`, `j+2`, `j+1`). A 512-bit register is those sixteen lanes, 256-bit code keeps two registers, 128-bit code four, scalar code sixteen variables. A tail shorter than 16 is padded with zeros, which adds exactly zero. All widths, and x64 and arm64, therefore return the same bits, so a graph built on one machine is the graph built on any other. The tests assert bit equality against the scalar path for lengths 0 to 300 and compare with a double-precision sum. * **Special values.** No flushing or clamping: denormals take part at full precision, infinities follow IEEE 754, a NaN input gives a NaN result (payload unspecified). Indexed data and queries are checked finite before they reach a kernel. * **Width.** `Vector512`, `Vector256` or `Vector128` is chosen once from `IsHardwareAccelerated` (false for 512 bits on processors where the runtime judges it slower); `Vector128` is also the WebAssembly SIMD path and the arm64 AdvSimd path. The scalar path is the fallback and the reference. * **Integer kernels** widen to 16 bits, multiply, widen to 32 bits and add; integer sums are exact so every width agrees. Dot product accepts up to 131,071 elements and squared distance up to 32,768 without overflowing `int`; longer inputs throw. * Files with `unsafe` constructs, `Unsafe.*`, `MemoryMarshal` or intrinsics: `Kernels/VectorKernels.cs` (loads through `Vector128/256/512.LoadUnsafe` behind a length check made once per call, and fixed 16-float `stackalloc` tail buffers), `Kernels/QuantizedKernels.cs` (the same loads for the 4-bit dot product, and a byte-to-signed-byte reinterpretation), `Codecs/VectorPartReader.cs` and `Codecs/VectorSegmentCodec.cs` and `Codecs/VectorFiles.cs` (`MemoryMarshal.Cast` between the bytes of a checked block and the floats or ints in it; bounds are those of the block), `Vectors/HnswBuilder.cs` is entirely safe code. ## Exact search `FlatSearch` compares the query with every accepted vector and keeps the best `k` in a bounded heap. It serves segments below the field's flat threshold (default 1,000 vectors; they get no graph), is used when a filter is selective (below), and is the ground truth that the recall measurements and the durability checks use. `FlatSearch.SearchBatch` answers many queries in one pass over the vectors, a tile of 128 vectors at a time against all queries, which is the shape of a matrix product. ## HNSW Implemented from Malkov and Yashunin, "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs", IEEE TPAMI 42(4), 2020 (arXiv:1603.09320): algorithm 1 (INSERT), 2 (SEARCH-LAYER), 4 (SELECT-NEIGHBORS-HEURISTIC, no extended candidates; connections dropped by pruning are not refilled, which is the default of an internal switch) and 5 (K-NN-SEARCH). * Parameters per field: `M` (default 16; layer 0 allows `2M`, the paper's `Mmax0`; see the deviation below), `efConstruction` (default 100), flat threshold (default 1,000). Search breadth `ef` is a query parameter (`KnnQuery.Ef`); 0 uses `max(k, 100)`. * **Layer 0 keeps up to `2M` links chosen at insertion** (the paper chooses `M` and lets reverse links grow a list to `Mmax0`). Measured on the 384-dimension set (150,000 vectors) at the same build time, recall 0.90 is reached at `ef` 220 instead of 270. The upper layers choose `M`. * **Software prefetch was tried and dropped.** Collecting a node's unvisited neighbours and reading the first cache line of each before scoring (which overlaps the memory reads without any intrinsic) changed the throughput at `ef` 220 by -3 to +7 percent across five settings, which is inside the noise of repeated runs on a shared host (medians 1035 to 1141 queries per second, minima 812 to 969), so the plain loop stays. * **Layer assignment** draws the number of times in a row a 64-bit hash of (field seed, node number) is divisible by `M`: the same distribution as the paper's `floor(-ln(uniform) * 1/ln M)` (`P(level >= l) = M^-l`) without a logarithm, whose last bit could differ between platforms. The field seed is a fixed constant xor the CRC-32C of the field name. Levels stop at 24. * **Deterministic build.** One thread, no clock, no random generator: the same vectors in the same order give the same graph on every machine (kernels above). Tests build twice and compare every link. * **Deletes.** A deleted document stays in the graph, because removing it can cut paths. Search walks through it and leaves it out of the results. A segment with many deletes is rebuilt by the merge policy like any other. * **Filters.** A query may carry a filter (a query whose matches are the allowed documents). The segment turns it into a bit set, and the search is described under "Filtered search" below. ### Merge A merge builds the merged segment's graph over the live vectors of the sources in their merged order. When the source with the most vectors has a graph with the same `M` and none of its vectors is deleted, its graph is copied (node numbers moved up by the number of vectors before it) and only the other sources' vectors are inserted, in order (the paper's incremental construction; the starting graph is just a valid state to continue from). Otherwise all vectors are inserted. Both are deterministic functions of the merge inputs; the first differs from flushing the same documents in one segment, which is documented, not a defect. The benchmark runner times both. ## Files Both files have the common header (magic, version) and footer (CRC-32C of the file), and both are read through the storage interface by range. Vectors and level-0 links are read in blocks of about 64 KB holding whole records; each block has its own CRC-32C in a table that is checked when the block is first loaded, so opening a segment does not read its vectors and damage is found by the first read that touches it, raising `CorruptIndexException` with the file name. Loaded blocks stay cached for the life of the reader; reading a cached vector or link neither allocates nor waits. The little-endian layout is read in place, so big-endian platforms are rejected. **`vectors` (magic `OIXV`, version 1)** after the header: `u32` document count of the segment; for each field, in name order: the vectors (`count * dims * 4` bytes, `float` little endian, ordinal order, in blocks), for a segment where not every document has a vector the table of the document of each ordinal (`count * 4` bytes, strictly increasing), and the block CRC table (`4` bytes per block). Then the directory: field count, and per field the name, dimensions, metric, encoding (0: 32-bit floats; later compressed copies add parts of their own and rerank from this file), count, flags (bit 0: sparse), offsets of the three sections, vectors per block and the CRCs of the document table and of the CRC table. Trailer: directory offset (`u64`), CRC-32C of the directory (`u32`). **`hnsw` (magic `OIXH`, version 1)** after the header: `u32` document count; for each field with a graph, in name order: layer 0 as fixed-size records of `2M + 1` `int`s (link count, then the links, unused slots zero), in blocks with a CRC table; then the upper layers as one checked blob: node count, then for each node above layer 0 its number (as the gap from the previous), its level and, for each layer from 1, the link count and the links (varints). Directory per field: name, node count, `M`, top level, entry point, offsets, records per block, CRC of the CRC table, offset, length and CRC of the upper-layer blob. Readers check, before trusting an offset or count: the sections tile the file exactly; counts are within the segment and the limits object; every link is a node other than the one holding it, with at most `2M` links on layer 0 and `M` above; upper-layer links point to nodes that are in that layer; the entry point is in the top layer; the ordinal-to-document table is strictly increasing and inside the segment. A segment smaller than the field's flat threshold has a `vectors` part and no `hnsw` part. Crash behaviour is that of every part: the files are written, fsynced and listed by the commit; a crash before the commit leaves unreferenced files that are reclaimed. A field declared with `VectorIndexKind.Ivf` has no graph: its clustered lists are in part `ivf` (magic `OIXL`, version 1) and the vectors part is the rerank source; see [ivf.md](https://recursiveai.org/oneindex/docs/ivf/). ## Query surfaces * Low level: `KnnQuery(field, vector, k)` with `Ef`, `Filter` and `Exact`. It finds the `k` nearest documents over all segments (each segment's best `k`, then the best `k` overall) and scores them; as a clause of a `BooleanQuery` it combines with text queries. The filter is applied before choosing neighbours; a `Filter` clause beside it in a boolean query would cut the neighbours afterwards. * Happy path: `index.SearchVector(x => x.Embedding, vector, neighbors: 20)` returns a `SearchQuery`; `Where(...)` restricts the candidates (before the nearest are chosen), `Take`/`Skip` page within the `neighbors` found. * Query JSON: `{"type":"knn","field":"Embedding","vector":[...],"k":10,"ef":100,"exact":false,"filter":{...},"boost":1}`; the tool schema adds one such node per indexed vector field with the vector length fixed. * A field with a compressed copy is searched on the copy and reranked exactly (next section). Hybrid fusion and reranking by models are separate. ## Compressed copies A vector field may ask for a compressed copy of its vectors (`[Vector(Quantization = ...)]`, `FieldDefinition.Quantization`). The graph is built on the full-precision vectors, as always; the copy is what a search walks it with. The full-precision vectors stay in the segment and reorder the candidates, so a result's score is always the exact one, whatever the codec. | Codec | Per vector | Distance used while walking | Default oversample | |---|---|---|---| | `None` | the float vector | exact | 1 | | `Scalar8` ("int8") | `d` signed bytes + 16 | integer dot product of 8-bit codes, corrected by the cached interval | 2 | | `Scalar4` ("int4") | `ceil(d / 2)` bytes + 16 | integer dot product of 4-bit codes with the 8-bit query | 3 | | `OneBit` ("sign-bit") | `ceil(d / 64) * 8` bytes | Hamming distance of the sign bits (popcount) | 5 | * **Scalar codecs** (Veasey, "Understanding optimized scalar quantization", Elasticsearch Labs, 2024). Each vector gets its own interval `[a, b]`; component `x` becomes the code `round((2^n - 1)(clamp(x, a, b) - a) / (b - a))`. The interval starts at the vector's mean plus or minus a fixed number of standard deviations (the normal-distribution optimum: 2.514 for 4 bits, 3.611 for 8) clipped to its min and max, and is refined by up to five rounds of coordinate descent on the paper's anisotropic error with lambda 0.1 (an error parallel to the vector counts ten times an error perpendicular to it): for fixed codes the best `(a, b)` solves a 2 by 2 linear system, which is solved in closed form, and a round that does not lower the error is discarded. The record keeps the codes, then `a`, the step `(b - a) / (2^n - 1)`, the sum of the codes and the squared length of the original vector (little-endian `float`, `float`, `int`, `float`). Expanding `(a_x + s_x X)(a_y + s_y Y)` over the `d` components gives the dot product from the integer `X . Y`, the two sums and the intervals; for the Euclidean metric the squared distance is `|x|^2 + |y|^2 - 2 x . y`. The query is compressed to 8 bits with the same procedure whatever the document codec (the paper quantizes the query with at least 4). Int8 codes are stored as `code - 128` so the signed-byte kernel applies (`a` moves by `128 * step`). Int4 codes are packed so that byte `j` holds component `j` in its low nibble and component `ceil(d/2) + j` in its high nibble, which lets the kernel split a register into two halves without shuffling. Vectors are not centred by a corpus mean, a deviation from the paper that keeps each record self-contained (a merge copies nothing). * **Sign bit** keeps one bit per component, set when the component is above zero, and compares by Hamming distance. It needs the cosine or dot-product metric (a Euclidean field with it is a schema error). It is deliberately plain: no per-vector correction, no centering. * **Oversample.** A quantized search walks the graph with `width = max(ef, ceil(oversample * k))` and keeps the best `width` candidates by compressed distance; every candidate is then measured against its full-precision vector and the best `k` are returned (ties towards the smaller ordinal). `oversample` is a field setting (`Oversample`, 1 to 100; 0 means the codec default above) and `ef` is the query's (`KnnQuery.Ef`; 0 means `max(k, 100)`), so at the default `ef` the oversample binds only when `oversample * k` exceeds 100. * **Exact and small cases use floats.** `Exact` searches, segments below the flat threshold (no graph, so no compressed copy is written), selective filters that fall back to scanning, and walks that exceed their visit limit all use the full-precision vectors. * **Files.** Part `vecq` (magic `OIXQ`, version 1) beside `vectors` and `hnsw`, with the same header, footer and 64 KB blocks with per-block CRC-32C. After the header: `u32` document count; for each quantized field with a graph, in name order, its records in blocks and the block CRC table. Directory per field: name, dimensions, codec, record count, record bytes, records per block, data offset, CRC table offset and the CRC of the table. A reader checks that the sections tile the file, that the counts equal the vectors file's, that record and block sizes follow from the codec, and validates each block as it loads (finite correction factors, code sums within range for scalar codecs, zero padding bits after the last dimension for sign bits). Damage raises `CorruptIndexException` naming the file. * **Schema.** Section 4 of the schema record (after section 3): a count, then for each vector field with a codec or an oversample, in ascending field index, the field index, the codec byte and the oversample as the bits of a `float`. A schema without codecs is written byte for byte as before; a reader that does not know section 4 skips it and searches the full-precision vectors, which is correct, only slower. * **Evolution.** Changing the codec or the oversample re-indexes the field from stored source in the background like a metric change. A segment whose compressed copy does not match the schema's codec (written before the change) is searched on full-precision vectors until it is rebuilt. * **Merge** writes the compressed copy of the merged segment from its full-precision vectors; nothing is copied from the sources. * **Determinism.** Interval selection uses IEEE double arithmetic in a fixed order and integer codes; the integer kernels are exact at every width. The records are therefore the same bytes on every machine (a test hashes 100 records of each codec and passes on x64 and arm64). ## Filtered search A `KnnQuery` with a `Filter` (a keyword term, a numeric or date range over a column or point, any query) turns the filter into a bit set of documents per segment and then chooses between two ways of answering. The choice is made per segment, on the number of documents the filter admits. * **Filtered graph walk.** After Patel, Kraft, Guestrin and Zaharia, "ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data", SIGMOD 2024 (section 5, search over the predicate subgraph). The upper layers are walked greedily without the filter, as for an unfiltered query. On layer 0 only documents that pass the filter are scored, queued and kept. Where a neighbour fails the filter, its own neighbours are looked at instead (neighbours of neighbours), so a rejected node costs a list read and no distance, and the walk is not cut off from the accepted nodes behind it. A node stops looking through rejected neighbours once it has collected `2M` accepted neighbours (32 by default), the length of an unfiltered layer-0 list. The walk ends as the unfiltered one does, when the nearest unexpanded candidate is farther than the `ef`-th best accepted result. If it finds fewer than `min(k, accepted)` results (a filter that splits the graph into pieces) the segment scans. * **Exact scan.** Below the switch the segment compares the query with every accepted vector (`FlatSearch`); the result is exact. * **The switch.** The scan is taken when the filter admits at most `20 * max(ef, k)` documents in the segment. The walk's cost depends on `ef` and hardly on the selectivity (measured, below), while the scan's cost is proportional to the number of accepted vectors. On 150,000 vectors of 384 dimensions at `ef` 100 they cost the same at about 1,500 accepted documents (15 times `ef`), and at `ef` 40 at about 650, so the break-even is close to linear in `ef` with a factor of 15. The scan is exact and the walk is not (recall 0.83 to 0.91 at `ef` 100 where the walk alone is used at 1%), so the scan is taken a little past break-even, where it costs up to 1.3 times the walk. At the default `ef` of 100 the scan serves segments with up to 2,000 accepted documents, and a larger `ef` widens that range (a caller asking for more recall gets the exact scan sooner). Compressed copies use the same rule; the scan reads the full-precision vectors. * **Compressed copies.** The filtered walk runs on the int8, int4 or sign-bit codes with the same filter, keeps `max(ef, oversample * k)` accepted candidates, and reranks them with the full-precision vectors, exactly as the unfiltered search does. Deleted documents are rejected nodes in the same way. * **Deviations from the paper.** The graph is the ordinary graph. The paper builds ACORN-gamma with a denser neighbour list (expansion factor gamma) so a predicate subgraph stays connected, and describes ACORN-1 with the ordinary degree, which is the variant used here. The second hop is taken only through rejected neighbours (the paper expands every neighbour). The switch to the exact scan is the paper's own fallback for very selective predicates, with a threshold measured here rather than derived. * **Hybrid, facets and sort.** The filter belongs to the vector arm (`HybridQuery.Vector.Filter`), as text filters belong to the text arm. A filtered knn clause is an ordinary query, so facets, sorts and aggregations see the same K documents. ### Measured Method: `tools/vector_filtered_rows.sh` (bench `--mode filtered`). The embedding sets above, one segment, `ef` 100 (the default) and the `ef` that reaches 0.95 unfiltered (450 at 384 dimensions, 270 at 768, float graph). Recall@10 is against the exact top 10 over the passing documents (brute force). Two filters over the same documents: *uncorrelated*, a hash of the document number (a keyword term, or a range over a number assigned at random, selects the same way), and *correlated*, the lowest slice of the projection of the vector on one fixed random direction (a range over a column that follows the vectors' position in space, so a query's neighbours are mostly inside the filter or mostly outside it). Selectivities 50%, 10%, 1% and 0.1%. Keyword, column and point filters reach the search as the same bit set (`tests/OneIndex.Tests/FilteredVectorTests.cs` runs a keyword term, two numeric ranges and a correlated range through `KnnQuery.Filter`). *Post-filter* is the naive alternative: an unfiltered search for `ceil(40 / selectivity)` neighbours (at most 10,000, with `ef` at least that), keeping those that pass. Timings are indicative: they ran as queued jobs on a shared 32-core host (x64, AMD Ryzen 9 7950X3D) whose load average was above 15 during the runs; an earlier run on a quiet host (load 4) gave 1.5 to 3 times the queries per second shown. Compare rows with each other, not with the unfiltered table. Recall does not depend on load. Recall@10 / queries per second, 384 dimensions, 150,000 vectors, float graph, `ef` 100 (the default): | Filter | Selectivity (accepted) | Filtered walk with switch | Walk alone | Exact scan | Post-filter | |---|---|---|---|---|---| | uncorrelated | 50% (75,002) | 0.866 / 937 | 0.866 / 821 | 1.000 / 56 | 0.843 / 1558 | | uncorrelated | 10% (15,003) | 0.909 / 1439 | 0.909 / 1236 | 1.000 / 202 | 0.930 / 519 | | uncorrelated | 1% (1,505) | 1.000 / 4123 | 0.833 / 2499 | 1.000 / 3729 | 0.993 / 74 | | uncorrelated | 0.1% (152) | 1.000 / 7537 | 0.958 / 5886 | 1.000 / 7481 | 0.210 / 38 | | correlated | 50% (75,000) | 0.880 / 1116 | 0.880 / 989 | 1.000 / 55 | 0.843 / 1403 | | correlated | 10% (15,000) | 0.897 / 1433 | 0.897 / 975 | 1.000 / 220 | 0.869 / 408 | | correlated | 1% (1,500) | 1.000 / 3938 | 0.907 / 2053 | 1.000 / 4362 | 0.801 / 66 | | correlated | 0.1% (150) | 1.000 / 6279 | 0.938 / 5322 | 1.000 / 7073 | 0.164 / 37 | At `ef` 450 (384 dimensions): uncorrelated 50% 0.966 / 304 (post-filter 0.954 / 406), 10% 0.984 / 305 (0.938 / 460), 1% and 0.1% 1.000 (exact scan); correlated 50% 0.969 / 303 (0.950 / 361), 10% 0.980 / 282 (0.881 / 455), 1% and 0.1% 1.000 (exact scan; post-filter 0.801 and 0.164). 768 dimensions, 75,000 vectors, float graph, `ef` 100: uncorrelated 50% 0.905 / 640 (post-filter 0.886 / 1100), 10% 0.938 / 719 (0.948 / 404), 1% 1.000 / 7515 (0.995 / 61), 0.1% 1.000 / 13843 (0.234 / 36); correlated 50% 0.903 / 770 (0.871 / 1100), 10% 0.930 / 971 (0.901 / 309), 1% 1.000 / 5839 (0.859 / 57), 0.1% 1.000 / 12474 (0.158 / 33). At `ef` 270: 50% 0.961 and 0.962, 10% 0.980 and 0.982, 1% and 0.1% 1.000. int4 copy (oversample 3, float rerank), 384 dimensions, `ef` 100: uncorrelated 50% 0.862 / 1029, 10% 0.905 / 1511, 1% 1.000 / 3493, 0.1% 1.000 / 6738; correlated 0.870 / 1196, 0.889 / 2395, 1.000 / 3326, 1.000 / 7060. At `ef` 450 the recalls are 0.962, 0.981, 1.000, 1.000 (uncorrelated) and 0.967, 0.979, 1.000, 1.000 (correlated). 768 dimensions, `ef` 100: uncorrelated 0.907 / 1228, 0.938 / 2330, 1.000 / 6882, 1.000 / 13564; correlated 0.903 / 1200, 0.932 / 1499, 1.000 / 7037, 1.000 / 13618; at `ef` 270: 0.963, 0.983, 1.000, 1.000 and 0.961, 0.980, 1.000, 1.000. What the rows say: * Filtered recall at a given `ef` is about the unfiltered recall at that `ef` (0.86 to 0.91 at 100, 0.96 to 0.98 at 450 or 270) from 50% down to 10%, and exact below the switch. * **Post-filtering is acceptable only for mild filters.** At 50% it is as accurate as the walk at the same `ef` and faster (it asks for 80 neighbours, hardly more than the walk visits). At 10% it is comparable (uncorrelated) or worse (correlated, 0.87 against 0.90 at `ef` 100, 0.88 against 0.98 at 450). Below 10% it fails (correlated 1%: 0.80; 0.1%: 0.16 to 0.21, where the 10,000-neighbour cap stops it) or is 30 to 100 times slower than the scan (uncorrelated 1%: 74 against 4,123 queries per second). Post-filtering does not guarantee a full page of `k`; the walk and the scan return `min(k, accepted)`. * The walk alone loses recall at 1% (0.83 to 0.91 at `ef` 100): few accepted nodes are near a query and the second hop is capped. The switch removes that range. * The second-hop target (`2M`) was chosen on a sweep from 12 to 64 on the 384-dimension set at 50%, 10% and 4%: a smaller target is faster per `ef` and loses recall, and at matched recall the curves are close, so 32 stays (it is the length of an unfiltered list). Information rows (no managed comparator implements a filtered graph search, so there is nothing to gate against): queries per second at the smallest swept `ef` that reaches the recall, filtered walk with switch, float graph, 384 dimensions at 0.90 / 0.95 recall: uncorrelated 50% 808 (`ef` 140) / 418 (300), 10% 860 (100) / 607 (200), 1% and 0.1% exact (2621, 5995); correlated 50% 688 / 376, 10% 1064 / 538, 1% and 0.1% exact (3457, 5893). 768 dimensions: uncorrelated 50% 793 (100) / 438 (200), 10% 1677 (70) / 671 (140); correlated 50% 685 / 461, 10% 1414 / 666; exact below. int4, 384 dimensions: uncorrelated 50% 1328 (140) / 352 (450), 10% 3252 / 1055; correlated 50% 1055 / 629, 10% 2005 / 716. Peak resident memory of the 384-dimension process was 980 MB for the float index and 982 MB for the int4 index (150,000 vectors; the process also holds the data and the truth sets). ## Method of the measurements * Embedding sets: `tools/embeddings.py` (the Wikipedia abstracts of the text benchmarks, BAAI/bge-small-en-v1.5 at 384 and BAAI/bge-base-en-v1.5 at 768 dimensions, text cut to 128 tokens, queries are held-out article titles with the model's query instruction); ground truth is the exact top 100 by dot product computed with numpy. * Recall: `bench/OneIndex.VectorBench` (`--engine oneindex|oneindex-segment|hnswnet|curiosity`), single-threaded build and search, recall@10 against the exact top 10, queries/second as the median of three passes over the query set, `ef` swept over a grid; the reported row for a target is the smallest `ef` that reaches it. Hnsw.Net 0.1.4 and curiosity-ai HNSW 26.9.5494 run through their public API (the two cannot be referenced together, so `bench/OneIndex.VectorBench.Curiosity` is the same runner for the second). * Compressed copies: `tools/vector_codec_table.sh` (the `oneindex-segment` engine with `--quant` and `--oversample`); float rows against the comparators: `tools/vector_bench_group.sh`. * Kernels: `bench/OneIndex.Bench` `KernelBenchmarks`. * Durability: `tests/Durability` carries a vector field (flat threshold 3, so merges of tiny segments build and continue graphs); every recovered snapshot is checked for the vector of every live document and for exact nearest neighbours against brute force. * Filtered search: `tools/vector_filtered_rows.sh`. The durability workload also runs filtered nearest-neighbour queries (a keyword term and a numeric range) after every recovery and requires every hit to pass the filter and the exact result to equal brute force. * Fuzz: `fuzz` target `vectors` (it includes a filtered walk). ## Measured Embedding sets (BAAI/bge-small-en-v1.5 at 384 dimensions, 150,000 base vectors; BAAI/bge-base-en-v1.5 at 768 dimensions, 75,000 base vectors; 1,000 held-out title queries; the first abstracts of the Wikipedia dump; manifests with SHA-256 are written by `tools/embeddings.py`). The sets are generated, never committed. M=16, efConstruction=100, one thread for build and search, recall@10 against exact search, `ef` on a grid (the row for a target is the smallest `ef` that reaches it, so a row can overshoot its target). Queries per second are the median of three passes. The graph builder takes `1.5 x efConstruction` candidates per insertion on layer 0 (see HNSW above). ### Float graph against the managed comparators `tools/vector_bench_group.sh`, one engine after another on an otherwise idle machine. Hnsw.Net 0.1.4 and curiosity-ai HNSW 26.9.5494 run through their public API. | Machine | Set | Engine | Build | Peak RSS | @0.90 (ef, recall, qps) | @0.95 | @0.99 | |---|---|---|---|---|---|---|---| | x64 | 384 | OneIndex | 86.4 s | 510 MB | 180, 0.9065, 1494 | 450, 0.9543, 675 | not reached (0.9839 at 1300) | | x64 | 384 | Hnsw.Net | 88.2 s | 591 MB | 180, 0.9000, 1316 | 640, 0.9604, 445 | not reached (0.9806) | | x64 | 384 | curiosity-ai | 55.2 s | 675 MB | 900, 0.9150, 331 | not reached (0.9333) | not reached | | x64 | 768 | OneIndex | 60.6 s | 504 MB | 140, 0.9152, 1284 | 270, 0.9513, 735 | 1300, 0.9914, 198 | | x64 | 768 | Hnsw.Net | 63.7 s | 577 MB | 140, 0.9073, 1259 | 320, 0.9521, 650 | 1300, 0.9905, 195 | | x64 | 768 | curiosity-ai | 35.4 s | 676 MB | 450, 0.9037, 481 | 1300, 0.9514, 200 | not reached | | arm64 | 384 | OneIndex | 83.3 s | n/a | 180, 0.9065, 1313 | 450, 0.9543, 601 | not reached (0.9839) | | arm64 | 384 | Hnsw.Net | 112.2 s | n/a | 180, 0.9000, 1278 | 640, 0.9604, 437 | not reached (0.9806) | | arm64 | 384 | curiosity-ai | 64.2 s | n/a | 900, 0.9152, 304 | not reached | not reached | | arm64 | 768 | OneIndex | 56.8 s | n/a | 140, 0.9152, 1254 | 270, 0.9513, 707 | 1300, 0.9914, 183 | | arm64 | 768 | Hnsw.Net | 88.3 s | n/a | 140, 0.9073, 1231 | 320, 0.9521, 622 | 1300, 0.9905, 198 | | arm64 | 768 | curiosity-ai | 45.3 s | n/a | 450, 0.9049, 439 | 1300, 0.9527, 180 | not reached | Machines: x64 is a Ryzen 9 7950X3D (32 threads), .NET 10.0.402, load 3.7 before and 2.9 to 6.7 during; arm64 is an Apple M2 (8 cores), .NET 10.0.403, load about 2 to 4 throughout. Index files are 250.6 MB (384) and 240.5 MB (768) for the vectors file plus the graph; Hnsw.Net's saved index is 253.4 MB and 241.9 MB. Throughput at matched recall against Hnsw.Net, the faster managed comparator at every target both reach (curiosity-ai's graph only reaches 0.90 at `ef` 900 on 384 and 450 on 768, and never reaches 0.95 on 384, so it is far behind at matched recall). The ratio is ours over Hnsw.Net, with queries per second linearly interpolated between grid points to the target recall; neither engine reaches 0.99 on 384, so the last 384 row is the best recall both reach (0.98). | Row | x64 | arm64 | |---|---|---| | 384 @0.90 | 1.23x | 1.11x | | 384 @0.95 | 1.34x | 1.24x | | 384 @0.98 (best both reach) | 1.31x | not measured | | 768 @0.90 | 1.12x | 1.11x | | 768 @0.95 | 1.13x | 1.13x | | 768 @0.99 | 1.09x | 1.00x | | Build time, ours over Hnsw.Net (smaller is better) | 0.98x (384), 0.95x (768) | 0.74x, 0.64x | Build time against curiosity-ai is 1.57x (384) and 1.71x (768) on x64: its graph builds faster and is much worse (recall 0.933 at `ef` 1300 on 384). Native libraries, information only (hnswlib 0.8.0, faiss-cpu 1.15.1 through `tools/native/vector_rows.py`, run on a busy host, so their throughput is not comparable to the rows above): 384 hnswlib builds in 127 s (contended) and reaches 0.90 at `ef` 270, 0.95 at 640; faiss builds in 36.5 s and reaches 0.90 at 270, 0.95 at 540. 768: hnswlib 30.9 s, 0.90 at 180, 0.95 at 450, 0.99 at 1800; faiss 24.3 s, 0.90 at 180, 0.95 at 380, 0.99 at 1800. The first version of this graph needed a larger `ef` than Hnsw.Net for the same recall. The changes (layer-0 neighbours chosen up to the paper's `Mmax0`, and a wider candidate list on layer 0 during insertion) follow the paper's parameters and were judged only by recall-against-`ef` curves of the comparator's public API. The curves (150,000 x 384; recall@10 at `ef` 140, 220, 450, 1300) were Hnsw.Net .8825, .9138, .9462, .9806; this the first version of this graph reached 0.90 at `ef` 220 to 270 and 0.95 at 640, and with the changes now in place .8878, .9206, .9543, .9839 (0.90 at 180, 0.95 at 450). On 768 dimensions, factors 1.0, 1.25 and 1.5 reached 0.95 at `ef` 380, 320 and 270, and cost 64.8 s, 94.0 s and 107.2 s to build on a loaded host (Hnsw.Net 135.7 s in the same job); the factor of 1.5 is the one that keeps the 384-dimension build time at or below Hnsw.Net's (0.98x). Topping connections up with pruned candidates (`HnswBuilder.KeepPruned`) raised recall at a fixed small `ef` but lowered throughput at matched recall (50,000 vectors, 0.90: 1540 against 1756 qps; 0.95: 853 against 1048), so it stays off. ### Compressed copies `tools/vector_codec_table.sh`, segment engine (files on disk, rerank from the full-precision file), k = 10. The search width is `max(ef, 10 x oversample)`, so the `ef = 10` column shows each oversample on its own (width 10, 20, 30, 50, 100) and from `ef = 100` the oversample no longer binds. Recall is deterministic; queries per second come from a run on a host at load 10 to 25 and are indicative only (single runs vary by a factor of two between neighbouring grid points). Recall@10 by search width (`ef` for the float and the `bit-o1` rows): | Set | Codec (oversample) | w=10 | 20 | 30 | 50 | 100 | 220 | 450 | 900 | 1300 | Extra bytes per vector | |---|---|---|---|---|---|---|---|---|---|---|---| | 384 | float | .4972 | .6313 | .7076 | .7802 | .8625 | .9214 | .9547 | .9761 | .9837 | 0 | | 384 | int8 (2) | .6341 | .6341 | .7056 | .7805 | .8606 | .9222 | .9550 | .9759 | .9835 | 400 | | 384 | int4 (3) | .6921 | .6921 | .6921 | .7733 | .8538 | .9181 | .9539 | .9755 | .9828 | 208 | | 384 | sign-bit | .2165 | .3500 | .4336 | .5355 | .6649 | .7818 | .8658 | .9264 | .9479 | 48 | | 768 | float | .5486 | .6814 | .7518 | .8248 | .8926 | .9414 | .9719 | .9862 | .9913 | 0 | | 768 | int8 (2) | .6831 | .6831 | .7508 | .8260 | .8920 | .9415 | .9706 | .9874 | .9915 | 784 | | 768 | int4 (3) | .7505 | .7505 | .7505 | .8256 | .8930 | .9422 | .9702 | .9872 | .9918 | 400 | | 768 | sign-bit | .3014 | .4739 | .5640 | .6702 | .7945 | .8888 | .9441 | .9742 | .9840 | 96 | (A quantized row at a width below its oversample times 10 shows the oversample's width: int8 at `ef` 10 is width 20, int4 width 30.) The sign-bit codec by oversample, with the default `ef` of 100 and with `ef` = k (so the oversample alone sets the width): | Set | Oversample | Width at ef=10 | Recall at ef=10 | Recall at default ef=100 | Indicative qps at ef=10 / ef=100 | |---|---|---|---|---|---| | 384 | 1 | 10 | .2165 | .6649 | 10789 / 5682 | | 384 | 2 | 20 | .3500 | .6649 | 6733 / 4457 | | 384 | 3 | 30 | .4336 | .6649 | 6096 / 3414 | | 384 | 5 (default) | 50 | .5355 | .6649 | 5378 / 3822 | | 384 | 10 | 100 | .6649 | .6649 | 3725 / 3845 | | 768 | 1 | 10 | .3014 | .7945 | 8096 / 4953 | | 768 | 2 | 20 | .4739 | .7945 | 7540 / 5877 | | 768 | 3 | 30 | .5640 | .7945 | 5679 / 7201 | | 768 | 5 (default) | 50 | .6702 | .7945 | 14323 / 4611 | | 768 | 10 | 100 | .7945 | .7945 | 4836 / 6347 | What it costs, in words: * **Memory.** The graph and the full-precision vectors are in every row; the copy is extra and is read while walking. Per 150,000 x 384 vectors: float vectors 225,014 KB, graph 19,752 KB, int8 copy 58,597 KB, int4 copy 30,470 KB, sign-bit copy 7,031 KB. For 75,000 x 768: int8 57,425 KB, int4 29,298 KB, sign-bit 7,031 KB. Total index bytes: 384: 250.6 MB float, 310.6 MB int8, 281.8 MB int4, 257.8 MB sign-bit; 768: 240.5, 299.3, 270.5, 247.7 MB. Peak process memory while building was 844 to 918 MB for every codec (the build holds the float vectors). The compressed copy buys speed and a smaller working set for the walk, not a smaller index: the float file stays for the rerank. * **Recall.** int8 and int4 follow the float curve within 0.01 from a width of 100. The sign bit does not: at width 100 it reaches 0.665 (384) and 0.795 (768), and 0.95 needs a width of about 450 on 768 dimensions (0.944) to 640 (0.962 at `ef` 640 in the earlier run), while on 384 dimensions it reaches 0.926 at 900 and 0.948 at 1300 and not 0.95. The cause is not established (the sign of an uncentred component carries little when the vectors share a large common offset is a guess; centring on a corpus mean would test it, and this codec deliberately does not centre). Because the oversample only matters while `10 x oversample` exceeds `ef`, the default of 5 binds only at `ef` below 50; callers who want the high-recall region of this table raise `ef` (or the oversample, to 45 for width 450). * **Speed.** Hamming over 384 bits is 4.5 ns and over 768 bits 8.6 ns per pair against 42 and 74 ns for the float dot product, so a walk is much cheaper per visited node; at equal recall the extra width eats most of it (the sign bit needs a width of 900 for 0.926 on 384 dimensions where the float graph needs 220, so the saving per node is spent on visiting more nodes). int4 at a width of 100 to 450 is about 1.5x to 2.2x the float throughput in the same run. ### Kernels BenchmarkDotNet, in-process short job, mean per call over one pair of vectors, 0 bytes allocated. Width 0 is the scalar path, 1 Vector128, 2 Vector256, 3 Vector512. | Machine | Dims | Kernel | Scalar | Vector128 | Vector256 | Vector512 | |---|---|---|---|---|---|---| | Ryzen 9 7950X3D | 384 | float dot | 177 ns | 42.5 ns | 24.4 ns | 34.2 ns | | Ryzen 9 7950X3D | 384 | int8 dot | 230 ns | 31.9 ns | 16.6 ns | 16.7 ns | | Ryzen 9 7950X3D | 384 | int4 dot | 138 ns | 25.8 ns | 14.4 ns | 16.2 ns | | Ryzen 9 7950X3D | 384 | Hamming | 5.5 ns | 4.5 ns | 4.5 ns | 4.9 ns | | Ryzen 9 7950X3D | 768 | float dot | 340 ns | 74.4 ns | 63.5 ns | 61.5 ns | | Ryzen 9 7950X3D | 768 | int8 dot | 390 ns | 62.7 ns | 31.9 ns | 31.9 ns | | Ryzen 9 7950X3D | 768 | int4 dot | 273 ns | 50.1 ns | 26.6 ns | 26.6 ns | | Ryzen 9 7950X3D | 768 | Hamming | 7.9 ns | 8.6 ns | 8.2 ns | 8.2 ns | | Apple M2 | 384 | float dot | 224 ns | 36.3 ns | n/a | n/a | | Apple M2 | 384 | int8 dot | 133 ns | 26.2 ns | n/a | n/a | | Apple M2 | 384 | int4 dot | 126 ns | 23.0 ns | n/a | n/a | | Apple M2 | 384 | Hamming | 11.8 ns | 12.0 ns | n/a | n/a | | Apple M2 | 768 | float dot | 419 ns | 69.6 ns | n/a | n/a | | Apple M2 | 768 | int8 dot | 310 ns | 59.2 ns | n/a | n/a | | Apple M2 | 768 | int4 dot | 281 ns | 49.8 ns | n/a | n/a | | Apple M2 | 768 | Hamming | 25.8 ns | 26.2 ns | n/a | n/a | The Hamming kernel is population counts over 64-bit words, which the runtime compiles to the hardware instruction on both machines; it has no vector path. All widths give bit-identical results; `VectorKernelTests` and `QuantizedVectorTests` pass on both machines. Code that uses `unsafe`-class APIs (`Unsafe.`, `MemoryMarshal`, `stackalloc`) is limited to `Kernels/VectorKernels.cs` and `Kernels/QuantizedKernels.cs` (the SIMD kernels; the 4-bit dot product at 768 dimensions goes from 273 ns scalar to 26.6 ns with Vector256 on x64 and from 281 ns to 49.8 ns with Vector128 on the M2), `Codecs/VectorPartReader.cs` and `Codecs/VectorFiles.cs` (bulk conversion of file blocks to floats and integers) and `Codecs/VectorSegmentCodec.cs` (the same conversions at merge time); each states its bounds where it is used. ## Schema record and evolution The dimension of a vector field is part of its field record. Its other settings (metric, M, efConstruction, flat threshold) are written in schema section 3 (schema format 2: sections are an ascending id and a length-prefixed payload; a reader that does not know a section skips it). Section 3 is a varint count, then per vector field whose settings differ from the defaults, in ascending field index: the field index, the metric byte, M, efConstruction and the flat threshold as varints. A schema whose vector fields all use the defaults is written as format 1, byte for byte what it was before settings existed. Reading rejects a section that names a field that is not a vector, is out of order, is empty, or holds an unknown metric. On open, a changed metric, M, efConstruction or flat threshold rebuilds the vectors from stored source in the background like an analyzer change (the graph and the scores both change). A changed dimension is a schema evolution error naming the field. ## Embedding set hashes SHA-256 of the files written by `tools/embeddings.py` (the sets are not committed). | Set | File | SHA-256 | |---|---|---| | bge-small-en-v1.5, 384 d, 150,000 vectors (rev 5c38ec7c405ec4b44b94cc5a9bb96e735b38267a) | `base.f32` | `0b5dd91f914327a2f6215f3b1d587c3bf0e86625208e3c82d74c4646a09fce5b` | | | `queries.f32` | `be7b933cd71815d93010c47579432d2fd3ee051a86afc0c4ea6e2cb5255207d7` | | | `truth.i32` | `1f422a4d0304cfdb37c2976a5856c8667ef6274e122be4b2eb74b5f2f9663406` | | bge-base-en-v1.5, 768 d, 75,000 vectors (rev a5beb1e3e68b9ab74eb54cfd186867f64f240e1a) | `base.f32` | `edd186263e2aceee456c19359f3ecf99d575eb5b50842c31f9623b202034c7fc` | | | `queries.f32` | `84c9ec58d1e583172c11dccb0a3a31f079da4cc06c45c173093f679e9f85a04d` | | | `truth.i32` | `b7e271b9963e0aa77e8cb5694fc4bc93a7f1b00e4f6774800ccabd905fec3839` | # Hybrid search: words and meaning in one ranking This note is normative for `HybridQuery`, `SearchQuery.Hybrid(...)` and the `hybrid` forms of query JSON. Automatic embedding is in `auto-embedding.md`; the vector search itself in `vectors.md`. ## What a hybrid query does A hybrid query has two arms and a fusion: * the **text arm** is any query (for plain text, the compiled user text) scored by BM25F as usual; * the **vector arm** is a `KnnQuery` over a vector field: the documents nearest in meaning to the search text, which the index's embedder turns into a vector with the *query* role; * each arm contributes its best `Window` documents (default 100): the text arm by relevance score, the vector arm by similarity (exact below the field's flat threshold, otherwise HNSW as `vectors.md`); * a document in either list matches; its score is the fusion of its position or score in the two lists; hits are ordered by that score, ties towards the earlier segment and smaller document number. **Filters apply to both arms.** `Where(...)`, filter strings and the `filter` of query JSON are applied inside the text query and as the vector arm's filter, so the vector arm picks its neighbours among the documents that pass (a full window whenever enough pass) instead of being filtered afterwards. `ScoreBy` (function scores) wraps the fused score. Sort, facets, aggregations, distinct and paging see the fused matches: `OrderBy` orders them by the column, facets count them. **`TotalHits`** is the number of documents in either window (at most twice `Window`) and is exact for that set. A page deeper than that set is empty; raise `Window` for deeper paging. `Skip`/`Take` page through the fused ranking, which does not depend on them. ## Fusion Notation: for a list `L` (best first) a document at position `i` (1-based) has `rank_L = i`; `r` is `SemanticRatio`. **Reciprocal rank fusion** (default; Cormack, Clarke and Büttcher, "Reciprocal rank fusion outperforms Condorcet and individual rank learning methods", SIGIR 2009): ``` score(d) = 2(1 - r) / (k + rank_text(d)) + 2r / (k + rank_vector(d)) (a missing list contributes 0) ``` `k` is `RankConstant`. The factor 2 makes `r = 0.5` the plain sum of the two reciprocal ranks of the paper. It uses ranks only, so no calibration between BM25 and similarity scores is needed. Without weights (`r = 0.5`), a document in only one list loses to one in both; the weights let one arm lead. **Weighted linear fusion** (opt-in, `FusionMethod.Linear`): each list is min-max normalised over its window, `n(d) = (s(d) - min) / (max - min)` (1 when all scores in the list are equal), and ``` score(d) = (1 - r) * n_text(d) + r * n_vector(d) (a missing list contributes 0) ``` `r = 0` runs the text arm alone and `r = 1` the vector arm alone; the other arm is not even run (no embedding, no search). The query JSON, LINQ and low-level forms validate `r` in `[0, 1]`, `k` in `[1, 1000]` and `Window` in `[1, 10000]`; a violation is a `QueryException` (low level), `ArgumentOutOfRangeException` (LINQ) or a positioned parse error (JSON). ### Defaults and why The Meilisearch `semanticRatio` is the knob (documented range 0 to 1). The defaults here are `SemanticRatio = 0.75`, `RankConstant = 10`, `Window = 100`, RRF (settled by the experiment below). The paper's constant (60) with equal weights does **not** meet the hybrid gate on the mean: the default embedder is much stronger than BM25 on two of the three gate sets, and an unweighted fusion of a strong and a weak list lands between them. A smaller constant weights the top ranks more, and a ratio above one half lets the stronger arm lead. The settings were chosen from a plateau (below), not a single point, on the same sets the gate reports, so they are in-sample; a different embedder or corpus may prefer 0.5 (set it on the query). ## When `Search(text)` is hybrid A search is hybrid automatically when all of these hold: it is built by `Search(text)` with non-blank text (not `SearchBy`, `SearchVector`, `SearchStrict`), the index has an embedder (`SearchIndex.Open(path, embedder)` or `SearchEmbedders.Default`), and `T` has exactly one `[Vector(From = ...)]` member. `Hybrid(0)` turns it off for a query. Explicit `Hybrid(...)` calls always apply (with several vector members, name one: `Hybrid(x => x.Embedding)`); `Hybrid(member, vector)` supplies the query vector itself and needs no embedder. A search with no text (`Search()`) matches everything and is never hybrid. **Excluded words reach every list.** In `Search(text)`, a word or phrase marked `-word` removes documents from the keyword list, the vector list and the sparse list alike; otherwise the vector list would return the very documents the user excluded because their meaning is close. The exclusion is applied as a filter before the nearest neighbours are chosen, so the vector list is still full. ## Query JSON Additive to version 1 (no new version): * a request-level `"hybrid": {"semanticRatio": 0.75, "fusion": "rrf" | "linear", "rankConstant": 10, "window": 100, "field": "Embedding"}` (every key optional; `field` defaults to the only automatically embedded member) applies to the request's `"q"`, which must be present and is then held in the request's `Hybrid.Text` rather than compiled into `Query`. `SearchIndex.Parse` embeds it; `QueryRequest.ToQuery()` leaves it out because turning text into a vector is the index's job. The `filter` applies to both arms. An LLM tool schema offers this form (`SearchTool.For()` adds `hybrid` when the schema has an indexed vector field); * a node `{"type": "hybrid", "text": , "vector": , "fusion": ..., "rankConstant": ..., "semanticRatio": ..., "window": ..., "boost": ...}` for callers that bring their own vector. It is not in the tool schema (vectors are not something a model should write). Defaults are left out when written, so the canonical form round-trips (`QueryJson.Write(QueryJson.Read(x))` is a fixed point). The reserved word `hybrid` no longer produces the "reserved" message. ## Explain `Explain()` on a hybrid hit names the fusion, the ratio and window, and the hit's rank in each list ("text rank 3, vector rank none"). ## Method and results of the gate The gate (`tools/OneIndex.Models.Tool`, command `hybrid`) builds one index per BEIR set with `title` (BM25F weight 3), `text` and the bge-small-en-v1.5 vector of `title + " " + text` computed by the shipped int8 weights (the same vectors as `docs/models.md`), then runs three kinds of query through the engine's own query objects: * **BM25**: the OneIndex default for BEIR (a BM25F multi-field term per analysed query token, OR-ed, English analysis, k1 0.9, b 0.4); * **dense**: `KnnQuery` with `Exact = true`, `K = 100` over the whole corpus (exact, not HNSW, so recall is 1 by construction); the query carries the model's query instruction; * **hybrid**: `HybridQuery(bm25, dense)` with the defaults above. nDCG@10 under the BEIR convention (a document with the query's id is dropped), cross-checked with `pytrec_eval` on the run files (`tools/models/crosscheck_run.py`). The gate: hybrid is at least the better single arm *on the mean over the sets*, and on every set at least the weaker arm. | Set (queries) | BM25 | Dense (exact) | Hybrid, defaults (RRF, k 10, ratio 0.75) | Hybrid, paper setting (RRF, k 60, ratio 0.5) | |---|---|---|---|---| | SciFact (300) | 0.6912 | 0.7126 | **0.7257** | 0.7282 | | ArguAna (1,406) | 0.4333 | 0.6048 | **0.6100** | 0.5616 | | SCIDOCS (1,000) | 0.1531 | 0.2041 | **0.2064** | 0.1919 | | Mean | 0.4259 | 0.5072 | **0.5140** | 0.4939 | * The defaults pass the gate: the mean 0.5140 is above the better arm's mean (dense, 0.5072), and on every set hybrid is above both arms (the weaker arm everywhere is BM25). * The paper setting passes the weaker-arm test everywhere but misses the mean test by 1.3 points, and is below the dense arm on ArguAna and SCIDOCS. * `pytrec_eval` on the run files gives the same four-decimal values for BM25, dense and the default hybrid on all three sets. * NQ was held out of the tuning and is confirmed below. * Other settings on the same index, for information (nDCG@10 SciFact / ArguAna / SCIDOCS): RRF k 10 ratio 0.5 0.7349 / 0.5677 / 0.1970; RRF k 10 ratio 0.3 0.7175 / 0.5140 / 0.1766; linear ratio 0.3 0.7254 / 0.5649 / 0.1805; linear 0.5 0.7377 / 0.6008 / 0.1994 (mean 0.5126); linear 0.7 0.7291 / 0.6055 / 0.2069 (mean 0.5138). * The plateau behind the defaults (mean over the three sets, RRF, window 100): ratio 0.75 with k 3, 5, 10, 20 gives 0.5150, 0.5151, 0.5140, 0.5117; ratio 0.7 gives 0.5141, 0.5149, 0.5115, 0.5093; ratio 0.5 gives 0.5013, 0.5005, 0.5000, 0.4965 for all k. Window 100, 300 and 1,000 differ by under 0.0005. ### NQ: the held-out confirmation The defaults were tuned on the three sets above; NQ (2.68 million passages, 3,452 queries, dense vectors from the reference fp32 model on a GPU, docs/models.md) was run once with them and **not retuned**. Same engine arms, exact kNN: | Set | BM25 | Dense | Hybrid, defaults | Hybrid, paper setting (k 60, ratio 0.5) | |---|---|---|---|---| | SciFact | 0.6912 | 0.7126 | **0.7257** | 0.7282 | | ArguAna | 0.4333 | 0.6048 | **0.6100** | 0.5616 | | SCIDOCS | 0.1531 | 0.2041 | **0.2064** | 0.1919 | | NQ | 0.3420 | 0.5018 | **0.5249** | 0.4862 | | Mean of four | 0.4049 | 0.5058 | **0.5168** | 0.4920 | **Gate verdict, as-is: pass.** The four-set mean 0.5168 is above the better arm's mean (dense, 0.5058), and the hybrid is above both arms on every set. Other NQ settings, for information: RRF k 10 ratio 0.5 0.5008; linear fusion ratio 0.3 / 0.5 / 0.7: 0.4461 / 0.5133 / 0.5292 (min-max per arm). The engine's default hybrid and an offline computation of the same formula from the cached windows agree on the top-10 set for 3,448 of 3,452 queries and on its order for 3,395 (ties and window edges). With the sparse arm (opt-in; docs/sparse.md) NQ text + sparse is 0.4911 and text + dense + sparse 0.5671. The stand-in proof for the vectors is in docs/models.md. ## Related `auto-embedding.md` (how the vectors get there), `vectors.md`, `query-json.md`, `search-tool.md`. # Reranking: `IReranker`, the rerank stage and the default cross-encoder A cross-encoder reads the query and a passage together, which ranks better than comparing a stored vector or counting words, and costs one model pass per pair. So it runs as a **stage** over the best hits of a first-stage search, never over the whole corpus. ## `IReranker` (core, namespace `OneIndex.Vectors`) A sibling of `IEmbedder`: same shape, same rules. - `ModelId`: a stable identity of the model and every setting that changes its scores (weights, quantization, truncation). - `Score(query, passages, scores, ct)` and `ScoreAsync(...)`: one score per passage, written to a caller buffer of exactly `passages.Count` floats. Higher is more relevant. Scores have no fixed range and are comparable only within one query. Pairs longer than the model's limit are cut, never rejected; the empty string is valid; `null` throws; an empty list does nothing. - Thread-safe. The async twin runs on a pooled thread where threads exist and on the caller's turn in the browser; results are identical. - `SearchRerankers.Default` (set once at startup, or by `MiniLmReranker.UseAsDefault()`) is the reranker new indexes open with; `SearchIndex.Open(path, embedder, reranker)` and `InMemory(embedder, reranker)` pass one explicitly. There is no discovery. - A reranker that throws, or returns a NaN or infinity, makes the search fail with `RerankException` naming the model. ## The stage ```csharp var hits = index.Search("overdue acme") .Where(x => x.Tags.Contains("invoice")) .Rerank(window: 100) // the best 100 after filters, rescored, best first .Take(10); ``` 1. The first stage runs as the same search with the window as its page: `max(window, Skip + Take)` hits, best first, by whatever the search does (text, vector, hybrid, filters, function scores). 2. The **passage** of each hit is read from its stored source: the `[FullText(Rerank = true)]` fields, in declaration order, or every full-text field when none is marked, joined by one space (for BEIR documents: title, a space, text). `Rerank(x => x.Body)` names one property. 3. The **query** is the search text as typed (`Rerank(query: "...")` overrides it, and is required when the search was made from query objects, which have no text). 4. One `Score(query, passages)` call rescores the window. Hits are ordered by the score, best first; equal scores keep their first-stage order, so the result is deterministic. 5. `Skip` and `Take` page the reranked list. A page deeper than the window reranks as many hits as the page needs (`Skip 45 + Take 10` with a window of 50 reranks 55). What the hit carries: `Score` is the reranker's score and `FirstStageScore` the score before the stage (`null` when the search was not reranked). `Explain()` wraps the first-stage breakdown in a root node `rerank by ` whose value is the reranker's score (so the root still equals `Score`) and whose single child is the first stage's explanation. What it does not change: `TotalHits` is the first stage's. **Facets and aggregations count all matches, not the window**, because the first stage runs unchanged. `Distinct` ranks its groups by the first stage and the window then holds groups. Highlights are computed from the hit as usual. What it does not combine with: `OrderBy`, `OrderByDistance` and `SearchAfter` (the reranker decides the order): `NotSupportedException` at execution. `Take(0)` is a count and calls no model. A search with no first-stage hits calls no model. ## When rerank turns on by itself Only when all of these hold: the type has a `[FullText(Rerank = true)]` member (the type asks for it), the index has a reranker, the search is plain text (`Search("...")`, not query objects), non-blank, and not ordered. Then `Search(text)` is `Search(text).Rerank()` with the default window of 100. `WithoutRerank()` turns it off for one search. A type without a marked member is never reranked unless the search says `Rerank()`. With hybrid (automatic when the type has a `[Vector(From = ...)]` member and an embedder) the first stage is the fused list, so a type with both is "hybrid, then rerank" with no further code. ## Query JSON `"rerank"` is an additive property of version 1: `{"v":1,"q":"overdue acme","rerank":{"window":50}}`. It reads the text of `q` as the query, so `q` is required (a path-named error otherwise), as with `hybrid`; like hybrid, `QueryRequest.Rerank` holds the text and `ToQuery()` leaves it out while `SearchIndex.Parse` applies it. `window` is 1 to 10,000, default 100 (not written when default); `{}` is valid; unknown properties are errors that name the nearest known one. `SearchTool.For()` lists it. It needs an index with a reranker; without one executing the search throws `InvalidOperationException`. ## Default cross-encoder: `MiniLmReranker` in `RecursiveAI.OneIndex.Models` cross-encoder/ms-marco-MiniLM-L6-v2 (Apache-2.0), revision `233902d25c440f23af6f7d6e94d2946bac0bee0a`: 6 layers, hidden size 384, one output logit. Trained on the MS MARCO passage ranking set (provenance only; the weights are the published ones). - **Pair encoding** (`WordPieceTokenizer.EncodePair`): `[CLS] query [SEP] passage [SEP]`, segment ids 0 for the class token, the query and the first separator, 1 for the rest; at most 512 ids. When the pair is too long, tokens are cut from the end of the longer text, never below half of the room: with `room = 512 - 3`, the shorter text keeps `min(its length, room / 2)` and the longer one the rest; when the two are equally long the second counts as the longer (settled by comparing with the Hugging Face `tokenizers` package as a black box; "longest first" truncation with `max_length = 512`, which is what the sentence-transformers cross-encoder asks for). Texts of more than 8,192 tokens are measured only that far to decide which is longer. - **Score**: the raw logit (the model's own default activation is the identity, `sbert_ce_default_activation_function` in its config). - **Batching**: pairs are sorted by length, padded to a multiple of 32 and run under a token budget (4,096 padded tokens); a pair's score does not depend on its batch (checked: batched against one-at-a-time scores differ by 0.00000). - **Weights**: `tools/models/prepare_cross_encoder.py` downloads the published ONNX at the pinned revision (SHA-256 checked) and writes the weight-only int8 variant (the same scheme as the embedder: 37 matrix-multiply weights and the embedding table as int8 with a scale per channel, activations float32). `gate.sh setup-reranker` converts both to OneModel files and `pack-weights.sh reranker` places the int8 one where `OneIndex.Models.csproj` embeds it with the Apache-2.0 notice. Never committed. ### Conformance and quality Pair token ids (4,528 of 4,528), scores against ONNX Runtime and the BEIR reproduction (all three sets within 1 point of the published BM25+CE figures) are in `models.md` ("Cross-encoder"). NQ: BM25F top-100 reranked 0.5404 against the BEIR paper's BM25+CE 0.533 (+0.7 point), with the GPU-reference scores and the int8 stand-in proof, also in `models.md`. # Automatic embedding and the embedding cache This note is normative for `[Vector(From = ...)]`, the embedder an index uses, and the cache that keeps unchanged text from being embedded again. ## Declaring it ```csharp public class Article { [SearchId] public string Id { get; set; } [FullText] public string Body { get; set; } [Vector(From = nameof(Body), Dimensions = 384)] public float[]? Embedding { get; set; } } ``` `From` names a text or keyword member of the same type (a build-time check in the schema: otherwise `InvalidOperationException` naming the member). `Dimensions` may be left out when `From` is set: the index then takes the embedder's `Dimension` when it is opened or created, and opening it without an embedder throws `InvalidOperationException` naming the member (`SearchTool.For()` needs no embedder). When `Dimensions` is declared, an embedder whose `Dimension` differs is refused when the index opens (`InvalidOperationException` naming both). Either way the length is recorded in the index's schema, so an index made with one length refuses to open with another. A non-nullable `float[]` property needs an initial value; an empty array means no vector was supplied. A multi-valued source is joined with a line feed. ## Which embedder There is no options object on `SearchIndex.Open`. The embedder comes from, in this order: 1. the argument: `SearchIndex.Open(path, embedder)` and `SearchIndex.InMemory(embedder)` (`null` = none); 2. `SearchEmbedders.Default`, set once at startup by the application, or by `BgeSmallEmbedder.UseAsDefault()` in `RecursiveAI.OneIndex.Models`, which loads the packed model and sets it. Nothing registers itself: no reflection, no module initializer, so referencing `.Models` alone changes nothing and the behaviour is the same under NativeAOT and trimming. `SearchIndex.Embedder` returns the one in use. An index without an embedder accepts documents without vectors and searches stay text-only; a vector the application sets on the document is always kept and never overwritten. ## When text is embedded * **Upsert** (`Upsert`, `Upsert(IEnumerable)`, `UpsertAsync`): before the document reaches the writer, with the document role. `Upsert(IEnumerable)` embeds in batches of at most 32 distinct texts (the documents are taken 64 at a time), `UpsertAsync` uses `IEmbedder.EmbedAsync`. Nothing is embedded at commit, flush or merge. * **Search**: the search text is embedded with the *query* role when a search is hybrid (`hybrid.md`), and cached in memory (the last 256 texts), so paging or repeating a search embeds once. **Failure.** If the embedder throws, returns a NaN or an infinity, or is cancelled, `Upsert` throws (`EmbeddingException` wrapping the cause and naming the model; `OperationCanceledException` for cancellation) **before** anything of the call is written: a batch call writes none of its documents' batch in that group of 64, and no half-embedded document ever exists, because the document is complete before the writer sees it. Documents of earlier groups in the same call stay buffered, as they would after any other exception. The index stays usable. ## The cache A cache entry is the SHA-256 of the model identity (`IEmbedder.ModelId`), a zero byte, and the UTF-8 source text. A document stores its entry in a hidden bytes field `#embedding` of the stored source, next to the vector (the stored source keeps the original, un-normalised vector). Writing it into the schema makes it part of the type's field list; an index created before the member had `From` gains the field through the normal additive schema evolution, and its documents count as stale until re-embedded. An upsert looks the text up in this order and embeds only on a miss: 1. an in-memory map of recent entries (up to 8,192 vectors; cleared when full), which serves identical texts of different documents and documents not yet committed; 2. the stored source of the committed document with the same identifier: same entry, so the stored vector is reused. Hence: re-upserting the same documents (a reindex from a database), reopening the index, merges, and schema-evolution rebuilds embed nothing. Merges and rebuilds move stored documents, vectors included, and never reference the embedder. Changing the text changes the entry, so only that document is embedded. Changing the model identity changes every entry: * new upserts embed again under the new identity; * `SearchIndex.ReembedStale()` (and `ReembedStaleAsync`) embeds again every committed document whose entry is missing or differs, buffering the replacements for the next commit, and returns how many; unchanged documents cost nothing. Call it after switching the embedder. A model change is not a schema difference, so it does not start the background schema rebuild by itself; `ReembedStale` is the explicit step. A model identity must change whenever its output changes (`IEmbedder.ModelId` documents this), and it also covers a changed query instruction or quantization. ## Durability Embedding happens before the write, so the commit protocol is unchanged: a document is either in a commit with its vector and entry or not in it. The kill campaign `tests/Durability` (`embed-kill`) runs a writer child that upserts through the index with a slow deterministic embedder, single and batched, commits every few operations and is killed at random moments; after each kill the directory must reopen, hold exactly the last acknowledged commit or the one in flight, every document must carry the vector its text determines, and a vector search for it must find it. ## Limits One vector field per `From` member; the source member must fit the model's input (the model cuts, it does not refuse). Texts are embedded in the caller's thread; a slow embedder slows `Upsert`. ## Sparse and multi-vector members `[Sparse(From = ...)]` (docs/sparse.md) and `[MultiVector(From = ...)]` (docs/multi-vector.md) use the same scheme with their own encoders (`ISparseEncoder`, `IMultiVectorEncoder`; defaults `SearchSparseEncoders.Default` and `SearchMultiVectorEncoders.Default`) and their own hidden cache fields (`#sparse`, `#multivector`). `ReembedStale` handles all three kinds in one pass. # Models: tokenizer, embedder contract and the default embedder Package `RecursiveAI.OneIndex.Models`. It depends on core and on `RecursiveAI.OneModel` only. ## `IEmbedder` (core, namespace `OneIndex.Vectors`) - `ModelId`: a stable identity of the model and every setting that changes its output (weights, quantization, prefix, truncation). Two embedders with the same id give the same vectors for the same text and role, so the id is safe in an embedding cache key. - `Dimension`, `IsNormalized` (unit-length vectors: the dot product is cosine similarity). - `Embed(texts, role, destination, ct)` and `EmbedAsync(...)`: row-major `texts.Count * Dimension` floats written to a caller buffer. `EmbeddingRole.Query` or `Document` selects the query instruction some models need. Texts longer than the model's limit are cut, never rejected; the empty string is valid; `null` throws; the destination length must match exactly. - Implementations are thread-safe. The sparse, multi-vector and reranking contracts are siblings of the same shape. ## WordPiece tokenizer (`OneIndex.Models.Tokenization`) Self-contained: no reference to the engine or to OneModel, only the BCL (and an `ICanonicalDecomposer` for accent stripping, which `OneIndex.Models.UnicodeDecomposer` implements from OneIndex's own Unicode tables; the runtime's own `Normalize` does nothing for non-ASCII text when globalization is invariant, so the default fallback is only for hosts that have ICU). Sources: Devlin et al. 2019 (BERT), Wu et al. 2016 (WordPiece), the BERT README (cleaning, CJK and punctuation rules), the Hugging Face `tokenizers` documentation. The reference implementation's source was not read; its output was used as a black box to settle open points (below). Per stretch of text between special tokens: 1. **Special tokens.** Literal `[PAD]`, `[UNK]`, `[CLS]`, `[SEP]`, `[MASK]` in the input map to their ids before any normalization (`SplitOnSpecialTokens`, on by default, as in the reference; turn it off for untrusted text). 2. **Cleaning.** U+0000, U+FFFD and every character of category Cc, Cf, Co or Cs are dropped, except tab, line feed and carriage return. (So vertical tab, form feed and U+0085 are dropped, not word breaks. Unassigned characters are kept.) Remaining whitespace (`White_Space`) separates words. An unpaired surrogate reads as U+FFFD and is dropped. 3. **CJK.** Each ideograph in U+4E00..9FFF, 3400..4DBF, 20000..2A6DF, 2A700..2B81F, 2B920..2CEAF, F900..FAFF, 2F800..2FA1F is a word of its own. The reference leaves U+2B820..2B91F out although the BERT README lists the whole block; that range is reproduced as observed. 4. **Accents and case.** With `StripAccents` (default: follows `LowerCase`), the word is decomposed (NFD) and nonspacing marks (Mn) are dropped; then it is lower-cased (simple case mapping, except U+0130, which gives `i` and U+0307). 5. **Punctuation.** Each ASCII non-alphanumeric (33..47, 58..64, 91..96, 123..126) and each character of category P* is a word. 6. **WordPiece.** Greedy longest match from the left; continuation pieces carry `##`. A word of more than 100 characters, or one that cannot be cut completely, is one `[UNK]`. Truncation keeps the first `maxLength - 2` tokens between `[CLS]` and `[SEP]`; a shorter limit gives a prefix of a longer one. Work and memory per call are bounded by the longest word, and encoding stops when the output is full. ### Conformance Token ids against the Hugging Face `tokenizers` package (0.23.3, run in a `uv` venv, on the model's own `tokenizer.json`): - All 1,112,064 Unicode scalar values (surrogates excluded), each as `AbCd` and alone: 2,223,033 of 2,224,128 encodings identical. The 1,095 other encodings belong to 618 scalar values, listed in `testdata/models/wordpiece-reference-differences.txt`; the test requires exactly that set. 614 are characters that the reference does not know (it treats them as ordinary unknown characters, while the runtime's Unicode data classifies them as marks, punctuation or format characters); four, U+166D, U+1734, U+111C9 and U+1171E, have a different general category in the two data versions (the reference splits U+166D as punctuation and drops U+1734 and U+1171E as marks, which the runtime does not). - 12,547 sampled texts (adversarial strings, random strings over awkward alphabets, every SciFact, ArguAna and SCIDOCS query, 2,000 passages of each): all identical. ## Default embedder: bge-small-en-v1.5 BAAI/bge-small-en-v1.5, MIT licence, revision `5c38ec7c405ec4b44b94cc5a9bb96e735b38267a`: 12 layers, 384 dimensions, 33M parameters. - Pooling and normalization as the model card: the first token's hidden state, scaled to unit length (floor 1e-12). - Queries get `Represent this sentence for searching relevant passages: `; documents get nothing. At most 512 tokens per text. - Run by OneModel in its default (fast) mode. `ExactArithmetic` selects its exact mode. - Batching: texts are tokenized, sorted by length and run in batches under a token budget (2,048 padded tokens by default); lengths are padded to a multiple of 64 and row counts rounded up to a power of two, so the model sees few distinct shapes (each shape is a separate plan inside OneModel and costs memory). ### Weights `tools/models/prepare_bge_small.py` downloads the published ONNX at the pinned revision (SHA-256 checked), and writes: - `model.onnx`: fp32 as published (133 MB as OneModel file). - `model.int8.onnx`: **shipped.** The 72 matrix-multiply weights and the word-embedding table are stored as int8 with one scale per output channel (per row for the table); the graph dequantizes with `DequantizeLinear` on constants, which OneModel folds when it plans. Activations stay float32, so a text's vector does not depend on the other texts in its batch. 34.9 MB as OneModel file. - `model.int8dyn.onnx`: comparison only. ONNX Runtime's dynamic quantization (int8 weights and per-tensor int8 activations). Not shipped: a text's vector then depends on what else is in the batch (cosine 0.995 between batched and single runs in our test). `tools/OneIndex.Models.Tool convert` turns each into a OneModel file with `ModelFormat.Convert`; `tools/models/pack-weights.sh` places the result where `OneIndex.Models.csproj` embeds it. The weights are never committed. ## Measurements All with the packed model files, query instruction on, exact dot-product retrieval over the whole corpus (document text is the title, a space, then the text), nDCG@10 under the BEIR convention (a document with the query's own id is skipped), cross-checked with `pytrec_eval` (`tools/models/crosscheck_run.py`; SciFact 0.7126 on both). Run on the x64 worker (AVX-512 kernels), 8 threads, loaded machine. The published figures are the `ndcg_at_10` values in the model card's metadata (MTEB results). | Set | Published | fp32 weights | int8 weights (shipped) | Difference to published | |---|---|---|---|---| | SciFact (5,183 passages, 300 queries) | 0.71275 | 0.7127 | 0.7126 | -0.0 | | ArguAna (8,674, 1,406) | 0.59550 | 0.6035 | 0.6046 | +0.8 / +0.9 points | | SCIDOCS (25,657, 1,000) | 0.20522 | not run | 0.2041 | -0.1 points | ### NQ (held out from the hybrid tuning) NQ (BEIR, 2,681,468 passages, 3,452 test queries; data hashes: corpus.jsonl f7e3588d..., queries 72f08f0c..., qrels 6df0cd2c...) cannot be embedded at the CPU rate of the shipped int8 path, so the document encodings come from the **reference fp32 models on a GPU** (RTX 4070, PyTorch, the revisions `.Models` ships; `tools/models/gpu_reference.py`). Everything else (indexing, BM25F, exact kNN, sparse scoring, fusion, nDCG) is our engine. A stand-in proof (below) shows the shipped int8 files agree with those encodings, which is what lets the NQ rows count for the int8 path. GPU numbers are marked as such. | Set | Published | Ours (GPU fp32 vectors, exact kNN over all passages) | Difference | |---|---|---|---| | NQ (2,681,468 passages, 3,452 queries) | 0.50178 | 0.5018 | +0.0 | `pytrec_eval` on the run file gives 0.5018. Encoding: 1,680 passages/s on the 4070 (about 23 minutes for the corpus), against 15 to 25 passages/s for the int8 path on the CPU. **Stand-in proof, dense** (random 20,000 passages with the 3,452 queries, int8 OneModel vs the GPU fp32 vectors of the same texts): cosine minimum 0.9988 (passages) and 0.9993 (queries); nDCG@10 on the 20,000-passage sub-corpus 0.9284 with the int8 vectors against 0.9285 with the fp32 ones; top-10 overlap 0.974. So the NQ figure above stands for the shipped weights within 0.1 point; the dense `.Models` gate (within 1 point of published) is met on NQ. Quantization effect on the vectors themselves, 800 texts (adversarial strings, queries and passages) against ONNX Runtime fp32 run one text at a time (cosine similarity of unit vectors; documents / queries with the instruction): | Weights | cosine min | cosine mean | max abs diff | Memory (peak RSS, 4 threads) | |---|---|---|---|---| | fp32 | 1.000000 / 1.000000 | 1.000000 / 1.000000 | < 1e-5 | 0.69 GB | | int8 weights (shipped) | 0.99394 / 0.99855 | 0.99978 / 0.99977 | 0.017 / 0.012 | 1.07 GB | | int8 dynamic (comparison) | 0.97204 / 0.96137 | 0.99690 / 0.99640 | 0.034 / 0.041 | | Why int8 weights are the default: retrieval quality is the same as fp32 within noise on both sets (SciFact -0.0001, ArguAna +0.001), the embedder's weights are 35 MB (the fp32 model file alone is 133 MB; the whole `RecursiveAI.OneIndex.Models` package, which also carries the reranker and the sparse encoder, is 134 MB), and a vector does not depend on its batch. Memory at run time is higher than fp32 because OneModel expands the int8 weights to float32 when it plans each input shape; the shapes are few (see Batching) but each plan holds its own copy. Throughput is about the same as fp32 (the arithmetic is float32 either way). `BgeSmallEmbedder.Open` takes any other file of the same model, for example the fp32 one. Throughput, for information (worker, 8 threads, other jobs running, so not a benchmark): 15 to 17 passages per second on SciFact (about 300 tokens each), 22 to 25 on ArguAna. The tokenizer is fuzzed (`fuzz/run.sh wordpiece`): 600 s, 3,767,141 executions, 2,912 coverage features, corpus 503 inputs, no findings. ## Cross-encoder: ms-marco-MiniLM-L6-v2 The default reranker (`MiniLmReranker`; the contract, the stage and the pair encoding are in `rerank.md`). cross-encoder/ms-marco-MiniLM-L6-v2, Apache-2.0, revision `233902d25c440f23af6f7d6e94d2946bac0bee0a`, trained on the MS MARCO passage ranking set (we did not train it; the data is not included). Weights are prepared like the embedder's (`prepare_cross_encoder.py`, `gate.sh setup-reranker`, `pack-weights.sh reranker`): fp32 as published (91 MB as OneModel file), weight-only int8 shipped (24 MB). They stay loadable from files (`MiniLmReranker.Open`). ### Conformance - **Pair token ids** against the Hugging Face `tokenizers` package (0.23.3, black box, the model's `tokenizer.json`, truncation `longest_first` at 512), 4,528 pairs: 27 adversarial pairs (empty texts, special-token text, accents, CJK, emoji, lengths around the truncation boundary on both sides) and 1,500 per BEIR set (SciFact, ArguAna, SCIDOCS) of a test query with a judged passage (half) or a random passage. **4,528 matched ids and segment ids; 0 differences**, 647 of them at the 512 limit. Two differences found on the way and fixed: which text is cut (the shorter keeps up to half the room, the longer the rest) and which counts as longer when equal (the second). - **Scores** against ONNX Runtime 1.31.1 fp32, one pair at a time, graph optimizations off, 600 pairs (the 27 adversarial and every 7th other; reference logits -11.5 to 7.3): | Weights | max abs diff | mean abs diff | Spearman | |---|---|---|---| | fp32 | 0.00001 | 0.00000 | 1.00000 | | int8 (shipped) | 0.143 | 0.019 | 0.99998 | Scoring all 600 pairs batched by query against scoring them one at a time: largest difference 0.00000. ### BEIR reproduction BM25F top-100 (the relevance gate's first stage: English analyzer, title weight 3, k1 0.9, b 0.4), each hit read as title, a space, text, reranked by the shipped int8 model through the model tool (`gate.sh rerank-beir`: the same ordering rule as the stage: best score first, ties in first-stage order). nDCG@10 under the BEIR convention (a hit with the query's own id is skipped), cross-checked with `pytrec_eval` on the run files (equal to four decimals). The published figures are the **BEIR paper's Table 2, BM25+CE column** (Thakur et al. 2021): a 6-layer, 384-hidden MiniLM cross-encoder reranking the top 100 of Anserini BM25, linked there as cross-encoder/ms-marco-MiniLM-L-6-v2 (the hyphenated repository name; the model card of the repository used here lists no BEIR figures). The paper's first stage is Anserini BM25, ours is BM25F, and its ArguAna numbers predate the identical-id convention, so the last column drops no hit. | Set | Published BM25+CE | BM25F first stage | Reranked (shipped int8) | Difference | Reranked, own-id hit kept | |---|---|---|---|---|---| | SciFact | 0.688 | 0.6912 | 0.6890 | +0.1 point | 0.6890 | | ArguAna | 0.311 | 0.4333 | 0.4164 | +10.5 points under the BEIR convention | 0.3112 (+0.0) | | SCIDOCS | 0.166 | 0.1531 | 0.1646 | -0.1 point | 0.1646 | ArguAna's gap is the convention, not the model: its queries are documents of the corpus, so the paper counted the query's own copy as a (never relevant) top hit; the same run scored that way gives 0.3112 against the published 0.311 (first stage 0.3253 against the paper's BM25 0.315). All three sets are within 1 point of the published figures. NQ (published 0.533) is below. **NQ** (3,452 queries, BM25F top-100 from the 2.68-million-passage index, 345,101 pairs scored by the reference fp32 cross-encoder on the GPU in 310 s, 1,113 pairs/s): first stage 0.3419, reranked **0.5404** against the published 0.533 (+0.7 point; `pytrec_eval` 0.5404). Stand-in proof: 60 queries x 100 pairs scored by the shipped int8 model vs the GPU fp32 scores: score difference max 0.1572, mean 0.0231, Spearman 0.99998; nDCG@10 of those 60 queries reranked 0.5993 (int8) vs 0.5971 (fp32), top-10 overlap 0.9917. The rerank gate (within 1 point) is met on NQ for the shipped weights. Information, SciFact: the default hybrid list (docs/hybrid.md; 0.7257) reranked gives 0.6894. On this set the cross-encoder does not improve on the hybrid first stage. Throughput, for information: 22 to 25 (query, passage) pairs per second with 8 threads on a shared x64 machine (AVX-512 kernels), 100 passages of 150 to 400 tokens per query. The model tool's rate on pairs is the limit of the BEIR runs above (about 45 minutes for SciFact, five hours for ArguAna). ## SentencePiece tokenizer (`OneIndex.Models.Tokenization`) Self-contained like WordPiece (BCL only, no reference to the engine or to OneModel), so it can move to a shared tokenizer library. Unigram models only: a BPE model is reported as a `SentencePieceFormatException`. Sources: Kudo, "Subword Regularization" (ACL 2018) for the unigram model and its Viterbi segmentation; Kudo and Richardson, "SentencePiece" (EMNLP 2018 demo); the `sentencepiece_model.proto` schema as a description of the file format. The reference implementation's source was not read; its output was used as a black box. - **Model file:** a protobuf (`ModelProto`) read by a small reader that checks every length against the remaining bytes and `SentencePieceLimits` (file size, piece count, piece length); malformed input raises `SentencePieceFormatException`. The `dictionary` fuzz target covers it. - **Normalizer:** `nmt_nfkc`. The model file carries the rule set only as a compiled character map, so the table (4,837 single code points and 14,116 sequences, `Tokenization/nmt_nfkc.tsv`) was dumped by running the released normalizer (`tools/models/spm_normalizer_dump.py`) over the Unicode scalar values and the sequences its map contains; Hangul jamo composition is algorithmic. The table is data observed from the reference's output. - **Segmentation:** Viterbi search over the piece scores, the first-found path winning ties. ### Conformance Token ids against `sentencepiece` 0.2.1 (run in a `uv` venv on the model's own `sentencepiece.bpe.model`) over 7,971 texts (`tools/models/spm_sample.py`, seed 20261103): the Universal Dependencies Japanese, Chinese, Korean and Thai sentences, phrases from the Snowball vocabularies of 34 languages, edge cases (empty, whitespace, control characters, long runs, emoji, combining sequences) and random strings over awkward scripts. - **7,967 identical** (99.95 %); the normalized strings are identical in all 7,971. - **4 differ, each an exact tie.** All four contain U+2034 (a triple prime), which normalizes to three U+2032; the text then has two segmentations with the same total score and the reference picks the other one (we keep the first found; trying "last found" and accumulating in double precision gave no better result). Both are valid unigram segmentations. - Against the model's fast `tokenizer.json` (a different implementation of the same model) 116 of the 7,971 texts differ from the `sentencepiece` ids, so "identical to the fast tokenizer" is not a claim we can make; `sentencepiece` is the reference here. ## Multilingual embedder: multilingual-e5-small (`RecursiveAI.OneIndex.Models.Multilingual`) intfloat/multilingual-e5-small, MIT licence, revision `614241f622f53c4eeff9890bdc4f31cfecc418b3`: 12 layers, 384 dimensions, around 100 languages, with a 250,000-piece embedding table. It is a separate package because the int8 weights are 120 MB (the package is 112 MB; the English default is 35 MB); it depends on `RecursiveAI.OneIndex.Models` and OneModel only. - Prefixes as the model card says: `query: ` for queries, `passage: ` for documents, both added by role. Mean pooling over the real tokens, then scaled to unit length. At most 512 tokens per text. - Token ids follow the model's numbering: the boundaries are 0 and 2, the padding id is 1, unknown 3, and every other piece id is shifted up by one (the fairseq convention of XLM-RoBERTa). - `MultilingualE5Embedder.Load()` opens the packed int8 weights; `Open(model, sentencepiece, variant)` loads files. Run by OneModel in its default mode, batched like the English embedder. - Weights: `tools/models/prepare_e5_small.py` downloads the published ONNX (SHA-256 checked) and quantizes it (int8 per output channel, the table per row); `tools/models/pack-weights.sh e5` places the converted file where the project embeds it. As OneModel files, fp32 is 470,429,956 bytes and int8 120,038,276. ### Parity with ONNX Runtime 800 texts (with the `passage: ` prefix) from the tokenizer sample: ONNX Runtime 1.31.1 fp32 on the ids of the model's `tokenizer.json` against our embedder (AVX-512 kernels). | | cosine min | cosine 1st percentile | cosine median | cosine mean | max abs difference | |---|---|---|---|---|---| | fp32 through OneModel | 0.954092 | 0.998780 | 1.000000 | 0.999811 | 0.05686 | | int8 (shipped) | 0.953278 | 0.998596 | 0.999871 | 0.999666 | 0.05774 | The low minimums come from the few texts where the fast tokenizer the reference used and `sentencepiece` (which we match) segment differently, not from the arithmetic: the median is 1.0. ### BEIR reproduction Dense retrieval over each set's corpus with the shipped int8 weights on CPU (flat vector search, `query: ` and `passage: ` prefixes, nDCG@10 under the BEIR convention), against the model card's MTEB figures (the `model-index` of its README, nDCG@10 in %): | Set | Model card | OneIndex int8 | Difference | |---|---|---|---| | SciFact | 67.704 | 67.82 | +0.1 point | | ArguAna | 39.063 | 38.59 | -0.5 point | | SCIDOCS | 13.895 | 13.75 | -0.1 point | MIRACL and NQ were not run: MIRACL needs multilingual corpora of millions of passages and NQ is 2.68 million passages, at 15 to 30 passages per second on CPU. ArguAna's queries are documents of the corpus: the BEIR convention (used here) skips a hit whose id equals the query's, while the model card's MTEB figure may count that copy as a hit, so the two numbers are not computed the same way. We did not run the other convention for this model, so the split of the 0.5 point between convention, int8 weights and the tokenizer differences is not measured. # Learned sparse vectors A sparse vector field holds a weight per token (a SPLADE-style expansion, Formal et al., SIGIR 2021 and arXiv:2109.10086, or the output of an inference-free encoder). Documents are indexed on the existing postings; a query scores by dot product. ## Field and value - `FieldType.Sparse` (schema type byte 11), `FieldValueKind.Sparse` (10), `SchemaBuilder.Sparse(name)`, `Document.Add(name, SparseVector)`. - `SparseVector`: unique tokens in ordinal order, finite non-negative weights, zero weights dropped, at most 65,536 entries. A repeated token keeps its larger weight. - A sparse field is single-valued, stored (the vector is returned with the document and survives merges and schema rebuilds from the stored source), and indexed unless `Indexed = false`. ## Postings layout (codec `postings` version 1, unchanged) Each token is a term of the field; a document's posting carries the **quantized weight as its frequency**: `q = round(weight * 128)` (`SparseVector.Scale`), tokens with `q = 0` are not indexed. No positions, no offsets. The field length (unused by scoring) is the number of indexed tokens. Because nothing changes in the files, merges, deletes, checksums, the fault injector's guarantees and the term-dictionary and postings readers apply as they are, and block impacts (largest frequency in a run of documents) are largest weights, which is what block-max WAND needs (Ding and Suel, SIGIR 2011). Error bound: a stored weight differs from the original by at most `0.5 / 128`, so a document's score differs from the exact dot product by at most `0.5 / 128 * sum(query weights)`. Top-k is exhaustive-equal over the quantized weights (test below). ## Query `SparseQuery(field, vector)` (boost multiplies): for each query token present in a segment, a `PostingsScorer` with score `frequency * queryWeight * boost / 128` (non-decreasing in the frequency, as the bound requires), combined by `BlockMaxWandScorer`. A document matches when it shares a token with the query; its score is the sum over shared tokens. `Explain` lists the shared tokens. Query JSON: `{"type":"sparse","field":"Expansion","weights":{"token":1.5}}`, additive to version 1. ## Encoders `ISparseEncoder` (core, `OneIndex.Vectors`): `ModelId`, `Encode`/`EncodeAsync(texts, role, destination)`; same rules as `IEmbedder`. `NeuralSparseEncoder` (`.Models`) is the inference-free encoder for models of the OpenSearch neural sparse "doc" family; files are supplied by the caller (`OpenDirectory`, `Open(paths)`, `Open(streams)`): a OneModel/ONNX graph with inputs `input_ids`, `attention_mask` and output `sparse` (float, batch x vocabulary), `vocab.txt`, and the query weight table `idf.json`. - Document side: WordPiece with `[CLS]`/`[SEP]`, the masked language model's logits, `log(1 + relu(x))` max-pooled over the tokens (padding masked) - built into the exported graph by `tools/models/prepare_neural_sparse.py`. - Query side: no model run. Tokenize without special tokens; each distinct token takes its weight from the table; tokens with weight 0 are dropped. A special token written in the text keeps its table weight (observed in the reference). - **Observed deviation resolved by experiment:** the reference sentence-transformers pipeline (5.x, run only) gives document weights equal to `log(1 + w)` of the single-activation pooled weights (checked on 200 texts: with one extra `log1p`, fp32 cosine 1.000000, max abs difference 0.0000 at identical support on 199 of 200 texts). Without it SciFact is expected to fall outside the card's number; see the results. `OpenSearchSparseEncoder` sets `ExtraLog1p = 1`. `OpenSearchSparseEncoder` packs opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill (Apache-2.0; provenance in THIRD-PARTY-NOTICES.md) and opens it with `Load()`, or from a directory with `Open(directory, variant)`. ## Happy path ```csharp public class Note { [SearchId] public string Id { get; set; } [FullText] public string Body { get; set; } [Sparse(From = nameof(Body))] public SparseVector? Expansion { get; set; } // encoded when an ISparseEncoder is configured } OpenSearchSparseEncoder.UseAsDefault(); // or SearchIndex.Open(path, embedder, encoder) var hits = index.Search("red apple").Sparse().Take(10); // text + sparse, fused ``` - `[Sparse]` goes on an `OneIndex.Index.SparseVector` property. With `From`, the index encodes the named text member (document role) before the document reaches the writer. The cache key `#sparse` = SHA-256(model id, 0, text) is stored in the stored source beside the vector, so reupserting unchanged text, reopening, committing, merging and rebuilding for schema evolution never call the encoder; a new model id is stale, and `ReembedStale()` encodes (and embeds) exactly the stale documents. A vector the application sets is kept as it is. The encoder is supplied like the embedder: `SearchSparseEncoders.Default`, or the three-argument `Open`/`InMemory`. - `Search(text).Sparse(weight?)` adds the sparse list (the text encoded with the query role). Combinations: text + sparse, and text + vector + sparse when the type also has an automatically embedded vector (the default hybrid applies; `Hybrid(0)` drops the vector list). Sparse search is opt-in: plain text search does not use it, because the default hybrid was tuned without it. - `HybridQuery` gains `Sparse` (a `SparseQuery` or a boolean around it that filters it) and `SparseWeight`; `Vector` may be null (text + sparse). Fusion: each list contributes `weight / (RankConstant + rank)` (rank fusion) or `weight * normalized score` (linear); the text and vector weights are as before (`2(1-r)`, `2r` for rank fusion, `1-r`, `r` for linear) and 1 for the text list when there is no vector list; the sparse weight defaults to 1 (rank fusion) or 0.5 (linear). Explain names the rank in each list. - Query JSON: a `hybrid` node takes `sparse` (a node) and `sparseWeight`, and `vector` becomes optional (one of the two is required); a request `{"q":..., "hybrid":{"sparse":true,"sparseWeight":..}}` adds the sparse list. `SearchTool` describes a `sparse` node per sparse field and the `sparse`/`sparseWeight` request properties. ## Results (information) int8 weights, `ExtraLog1p = 1`, our engine's `SparseQuery` over a force-merged index, nDCG@10 under the BEIR convention (a hit with the query's own id is dropped): | Set | Docs | Ours | Model card | Entries per doc | Sparse-only index | Sparse query (on disk, warm) | |---|---|---|---|---|---|---| | SciFact | 5,183 | 0.7056 | 0.708 | - | 13.1 MB | 1.4 ms | | ArguAna | 8,674 | 0.5194 | 0.520 | 248.2 | 23.7 MB | 67 ms (queries are whole passages) | | SCIDOCS | 25,657 | 0.1628 | 0.163 | 224.5 | 70.2 MB | 3.4-4.0 ms | All three are within 1 point of the card. SciFact is one of the model's training sets (its card lists scifact), so it is not an out-of-domain figure. One thread, one machine, indicative only. Hybrid, same sets, on one on-disk index holding BM25 (title and text), the bge-small dense vectors and the sparse vectors (default rank constant 10, semantic ratio 0.75, window 100; sparse weight 1 under rank fusion): | Configuration | SciFact | ArguAna | SCIDOCS | Mean | |---|---|---|---|---| | BM25 | 0.6912 | 0.4333 | 0.1531 | 0.4259 | | dense | 0.7126 | 0.6048 | 0.2041 | 0.5072 | | sparse | 0.7056 | 0.5194 | 0.1628 | 0.4626 | | text + dense (the default hybrid) | 0.7257 | 0.6100 | 0.2064 | 0.5140 | | text + sparse | 0.7209 | 0.5020 | 0.1664 | 0.4631 | | text + dense + sparse, rank fusion | 0.7374 | 0.6049 | 0.2063 | 0.5162 | | text + dense + sparse, linear fusion | 0.7439 | 0.6149 | 0.2066 | 0.5218 | NQ (held out; 2,681,468 passages, 3,452 queries; document vectors from the reference fp32 model on a GPU, 550 to 616 passages/s; queries encoded by the shipped int8 path, `ExtraLog1p = 1`; full-corpus index of 10.4 GB with text, dense and sparse parts; `pytrec_eval` equal to four decimals): | Configuration | NQ | Four-set mean | |---|---|---| | sparse | 0.5432 (card 0.544, -0.1 point) | 0.4828 | | text + sparse | 0.4911 | 0.4701 | | text + dense + sparse, rank fusion | 0.5671 | 0.5289 | Stand-in proof (8,000 random passages, shipped int8 files vs the GPU vectors): cosine minimum 0.9981, nDCG@10 on the sub-corpus 0.9480 (int8) vs 0.9475 (fp32), top-10 overlap 0.984. The sparse gate (within 1 point of the card) is met on NQ. With NQ in, three-way rank fusion has the highest four-set mean among the rank-fusion configurations (0.5289 vs the default hybrid 0.5168), still below the default on ArguAna, so it stays opt-in. Sparse-only query time over the full corpus: 103 s for 3,452 queries on 6 threads (about 30 ms per query, indicative, shared machine). The sparse arm is opt-in (`Sparse()`); it helps SciFact and is neutral on the other two sets with rank fusion, and it is not part of the default hybrid. - Reference parity (200 texts, fp32): identical to the reference runtime. int8 weights: document cosine median 0.9991, min 0.9818. # Multi-vector search: MaxSim Late interaction (ColBERT; Khattab and Zaharia, SIGIR 2020) represents a text as one vector per token and scores a query against a document by **MaxSim**: for each query token take the best similarity among the document's tokens, then sum (section 3.5, equation 3). OneIndex defines the interface and the scoring; the application supplies the model (no multi-vector model is packed with OneIndex). ```csharp public interface IMultiVectorEncoder { string ModelId { get; } int Dimension { get; } float[][] Encode(string text, EmbeddingRole role, CancellationToken ct = default); } doc.Tokens = MultiVector.Pack(encoder.Encode(text, EmbeddingRole.Document)); // a Bytes property of the document type var first = index.Search("late interaction").Take(100).Execute(); // any first stage var best = MaxSim.Rerank(first, d => d.Tokens, encoder.Encode(query, EmbeddingRole.Query), take: 10); var scan = MaxSim.Search(index.Search().Where(d => d.Lang == "en"), d => d.Tokens, queryTokens, k: 10); // MaxSim alone ``` - `MultiVector.Pack` / `Unpack`: a little-endian `u32` token count, a `u32` dimension, then the vectors as little-endian floats. Unpack validates the lengths and raises `FormatException`. Packing refuses unequal lengths and NaN or infinity. - `MaxSim.Score(query, document, dimension)`: the vectors are scaled to length 1 first, so the similarity is cosine; either side with no tokens scores 0. - `MaxSim.Rerank` re-orders first-stage hits (the rerank stage); documents without a multi-vector score 0 and sort last, ties keep the first-stage order. - `MaxSim.Search` is an exact scan of the documents that pass the optional filter, limited to `SearchQuery.MaxResultWindow` (10,000) candidates. For larger collections narrow with a filter, or rerank a first stage. There is no approximate multi-vector index. ## The `[MultiVector]` field ```csharp public class Passage { [SearchId] public string Id { get; set; } = ""; [FullText] public string Text { get; set; } = ""; [Keyword] public string Lang { get; set; } = ""; [MultiVector(Dimensions = 128)] public float[][]? Tokens { get; set; } // one row per token } var first = index.Search("late interaction").Take(100).Execute(); var best = MaxSim.Rerank(first, p => p.Tokens, encoder.Encode(query, EmbeddingRole.Query), take: 10); var scan = MaxSim.Search(index.Search().Where(p => p.Lang == "en"), p => p.Tokens, queryTokens, k: 10); ``` The generated mapping packs the rows with `MultiVector.Pack(rows, Dimensions)` into a stored `Bytes` field and unpacks them on read (AOT-clean, no reflection). A row of the wrong length, a NaN or an infinity is refused at `Upsert` with an `ArgumentException`. `MaxSim.Rerank` and `MaxSim.Search` have overloads taking `Func` beside the packed-bytes ones. A multi-vector cannot also be `[FullText]`, `[Keyword]`, `[Filterable]`, `[Sortable]`, `[Facet]`, `[Geo]`, `[SearchId]`, `[Vector]` or `[Sparse]` (a compile-time diagnostic). ## Encoding at upsert: `[MultiVector(From = ...)]` ```csharp public class Passage { [SearchId] public string Id { get; set; } = ""; [FullText] public string Text { get; set; } = ""; [MultiVector(From = nameof(Text), Dimensions = 128)] public float[][]? Tokens { get; set; } } using var index = SearchIndex.Open("data/passages", embedder: null, sparseEncoder: null, reranker: null, multiVectorEncoder: encoder); index.Upsert(new Passage { Id = "p1", Text = "..." }); // Tokens is encoded from Text var best = index.Search("late interaction").MaxSim(window: 100).Take(10).Execute(); // the query is encoded too ``` The encoder is an `IMultiVectorEncoder`, supplied as for the dense embedder: an argument of `SearchIndex.Open` / `InMemory` (the five-argument `Open` and four-argument `InMemory` use exactly what they are given, `null` = none) or `SearchMultiVectorEncoders.Default`, set once by the application. Nothing registers itself. `SearchIndex.MultiVectorEncoder` returns the one in use. * **Declaring.** `From` names a text or keyword member of the same type. `Dimensions` is required and must equal the encoder's `Dimension`; a mismatch is refused when the index opens (`InvalidOperationException` naming both). The generator refuses (compile error `OI0007`) an empty `From`, one that names the member itself and one that names no member of the type. Whether the named member is a text or keyword field is checked when the mapping is built, as for `[Vector]`. * **When.** `Upsert`, `Upsert(IEnumerable)` and the async twins encode a document whose `[MultiVector]` member is null, before it reaches the writer, in batches of at most 32 distinct texts with the document role (`IMultiVectorEncoder.Encode(IReadOnlyList, ...)`, whose default calls the single-text method). Token vectors the application sets are kept and never overwritten. An encoder that throws, returns null, a row of the wrong length or a NaN or infinity makes the call throw `EmbeddingException` before anything is written. * **The cache** is the one of dense embedding (docs/auto-embedding.md): the SHA-256 of the model identity, a zero byte and the UTF-8 text, in a hidden bytes field `#multivector` of the stored source next to the packed token vectors. An unchanged text under an unchanged `ModelId` is never encoded again: re-upserting, reopening, merges and schema rebuilds reuse the stored vectors. `SearchIndex.ReembedStale()` (it covers dense, sparse and multi-vector members) encodes again every committed document whose entry is missing or differs, buffering the replacements for the next commit. A document whose tokens were supplied by the application has no entry and counts as stale. An index written before `From` was added opens unchanged; the hidden field arrives through additive schema evolution and no stored format changes. * **The query stage.** `SearchQuery.MaxSim(window = 100, encoder = null, query = null)` and `MaxSim(x => x.Tokens, ...)` (for types with several `[MultiVector]` members) re-order the best `window` hits of the search by MaxSim against the query encoded with the query role (cached for the last 256 texts). It sits beside `Rerank`: hits carry the MaxSim score as `Score` and the earlier one as `FirstStageScore`; documents without token vectors score 0 and sort last; ties keep the first-stage order; `Explain` adds a line; facets and aggregations still count all matches. With no search text the first stage is every document passing the filters, so `index.Search().Where(...).MaxSim(window: 1000, query: "text")` is an exact scan limited to `window` candidates (at most 10,000). It cannot be combined with `Rerank`, `OrderBy`, `SearchAfter` or `OrderByDistance` (`NotSupportedException`), has no query JSON or LINQ-keyword form, and needs an encoder (`InvalidOperationException` otherwise). `MaxSim.Rerank` and `MaxSim.Search` above stay for vectors you encoded yourself. ## Not done The token vectors are stored with the document, not indexed: a MaxSim query is a rerank or a bounded scan, not an index lookup, and there is no approximate multi-vector index. ## Checks `MaxSimTests.MatchesBruteForceOnGeneratedData`: 200 seeded trials (dimension 1 to 12, 1 to 6 query tokens, 1 to 8 documents of 0 to 8 tokens each; more than 500 query/document scores), each equal to a brute-force sum of maxima of cosines to 4 decimals, and each document's pack/unpack round trip equal to the input. Damaged packed bytes (truncated, wrong count) raise `FormatException`, and the fuzz target `static` feeds arbitrary bytes to `Unpack`. `MultiVectorFieldTests`: a generated-mapping document type with a `[MultiVector(Dimensions = 5)]` property; 150 documents (some with no tokens, some with none stored), 20 seeded queries of 1 to 4 tokens; `MaxSim.Search` over a filter and `MaxSim.Rerank` over a 60-hit first stage each equal the brute-force sum of cosine maxima to 4 decimals, and the property round-trips; a wrong row length is refused at `Upsert`. # IVF: a clustered vector index An inverted-file (IVF) index groups vectors by cluster and answers a query by scanning only the few clusters whose centres are nearest to it (Jegou, Douze and Schmid, "Product quantization for nearest neighbor search", TPAMI 2011, section 2; the clustered lists follow Sivic and Zisserman, ICCV 2003). Unlike a graph, a search reads a handful of contiguous ranges, which is what a static index needs: one range read per probed list. ```csharp byte[] image = IvfBuilder.Build(vectors, dimensions, new IvfOptions { Storage = IvfStorage.EightBit }); // or IvfBuilder.Write(storage, name, ...) using IvfIndex index = IvfIndex.Open(storage.OpenRead("v.ivf")); IvfHit[] hits = index.Search(query, k: 10, nprobe: 16); // Id is the vector's number; Score larger is nearer ``` IVF is the vector layout of the static index ([static-index.md](https://recursiveai.org/oneindex/docs/static-index/)). On the server a vector field uses HNSW by default; `[Vector(Index = VectorIndexKind.Ivf)]` (or `FieldDefinition.VectorIndex`) makes it an IVF field, described in "IVF as a per-field index" below. ## Building `IvfBuilder` runs k-means on a sample (64 training vectors per list by default, 12 Lloyd iterations, seeded; spherical k-means for cosine, so the centres are unit vectors). The number of lists defaults to about the square root of the vector count, from 1 to 4096. The same input and options give the same bytes. Every vector goes to its nearest centre. `IvfOptions.Storage`: - `FullPrecision`: lists hold float vectors; probed lists are scored exactly. - `EightBit` (the static default): lists hold the vector's 8-bit codes with one scale per vector; probed lists are scored on the codes, the best `oversample × k` candidates (default 4) are then re-scored from a full-precision region at the end of the file, so reads of full rows are limited to the candidates. Cosine, dot product and Euclidean metrics follow [vectors.md](https://recursiveai.org/oneindex/docs/vectors/). A filter callback on the vector number is applied while scanning. ## File format `OIVF` version 1 All integers little-endian. | Offset | Content | |---|---| | 0 | magic `OIVF` (`0x4656494F`), `u32` version 1, `u32` dimensions, `u32` metric, `u32` list count, `u32` vector count, `u32` storage (0 full precision, 1 eight bit), 4 bytes zero | | 32 | `i64` offset of the full-precision region (0 when storage is full precision), 8 bytes zero. The header is 48 bytes. | | 48 | centroids: list count × dimensions `f32` | | after | directory: per list `i64` file offset of the list, `u32` count, `u32` first position | | lists | per list: its vector numbers (`u32` each), then its vectors (`f32 × dimensions`) or codes (`f32` scale, then `i8 × dimensions`) | | rerank | eight-bit only: the full-precision vectors in list order (`f32 × dimensions` each), reached by position | | end | CRC-32C of everything before it (`u32`), magic `OIVF` again (`u32`) | Readers check the magic, version, sizes against the file length, each directory entry against the list region, each vector number against the count, and the CRC (on by default; `verifyChecksum: false` for a reader that has checked the bytes another way, such as a static index's per-chunk CRCs). Damage raises `CorruptIndexException`. The file is at most 2 GiB. ## IVF as a per-field index `FieldDefinition.VectorIndex` (`Hnsw` default, or `Ivf`) and `IvfLists` (0 = about the square root of the segment's vector count) are stored in schema section 6; a reader that does not know the section skips it and searches the full-precision vectors. Changing the option re-indexes the field from stored source in the background, like a metric change. - **Part `ivf`** (magic `OIXL`, codec version 1), beside `vectors`: `u32` document count, then for each IVF field a complete `OIVF` image, then a directory (field count; per field its name, dimensions, vector count, image offset and length) and the checksummed footer. The images use `IvfStorage.EightBitCodes` (storage value 2: the codes with a scale per vector and no float rerank region; version 1 of `OIVF` allows storage 0 to 2). The full-precision vectors stay in the `vectors` part and are what the rerank reads. - **When it is written.** On flush and on merge, for segments with at least `FlatThreshold` vectors (smaller segments are scanned exactly); a merge rebuilds the lists from the merged vectors (no graph is written for the field). The part is skipped, and the field is scanned exactly, if its image would pass 2 GiB. - **Search.** `KnnQuery.Ef` means the number of probed lists for an IVF field (0 = max(8, lists / 8)). The probed lists are scored on the codes, the best 3 x k candidates (the oversample) are re-scored from `vectors`, and a filter and the deletes apply while scanning. Sync and async twins as for every vector query. - **Damage.** The reader validates the directory against the file and every image through `IvfIndex.Open`; a bad file raises `CorruptIndexException` naming the part. The `vectors` fuzz target covers the part. Measured (indicative: the worker, one thread per query, load average 7-18 from other jobs, the same 150,000 x 384 and 75,000 x 768 embedding sets as below, M = 16 and efConstruction = 100 for HNSW, `tools/vector_ivf_rows.sh`; the sizes are the whole segment, `vectors` included): | | recall@10 | QPS | build | index bytes | peak RSS | |---|---:|---:|---:|---:|---:| | 384 d HNSW ef 100 / 200 / 400 | 0.863 / 0.915 / 0.951 | 2,174 / 1,237 / 663 | 89.9 s | 250.6 MB | 945 MB | | 384 d IVF nprobe 16 / 48 / 96 | 0.734 / 0.879 / 0.938 | 434 / 160 / 82 | 9.2 s | 289.8 MB | 1,414 MB | | 768 d HNSW ef 100 / 200 / 400 | 0.893 / 0.937 / 0.968 | 1,541 / 840 / 423 | 57.6 s | 240.5 MB | 856 MB | | 768 d IVF nprobe 16 / 48 / 96 | 0.757 / 0.891 / 0.953 | 289 / 100 / 45 | 10.3 s | 289.5 MB | 1,360 MB | On text embeddings HNSW is 3 to 8 times faster at equal recall; IVF builds about 6-10 times faster and its file is a plain list layout (no pointer walk), which is why it is the static layout and an option for fields that are rebuilt often or read from slow storage. Its recall includes the float rerank. Real text embeddings have little cluster structure, so recall 0.95 needs about a quarter of the lists. ## Recall and speed Measured on the worker (AMD Ryzen 9 7950X3D, 32 threads; one thread per query; Linux; the file was read through `FileSystemStorage`, in the page cache; the machine was shared with other jobs, so the QPS are indicative: load average 27 at the start of the run and 3 at its end). Sets: 150,000 Wikipedia abstracts embedded with bge-small-en-v1.5 (384 d) and 75,000 with bge-base-en-v1.5 (768 d); 1,000 queries; recall@10 against the exact top 10; cosine; 387 and 274 lists; `oversample` 4; median of 3 passes. Tool: `bench/OneIndex.VectorBench --mode ivf`. There is no managed IVF comparator, so these rows are reported, not gated. **384 d, 150,000 vectors, 387 lists** | nprobe | lists scanned | recall@10 | QPS full precision | QPS eight bit | |---:|---:|---:|---:|---:| | 1 | 0.3% | 0.309 | 2,550 | 3,638 | | 4 | 1.0% | 0.536 | 834 | 1,423 | | 16 | 4.1% | 0.749 | 209 | 413 | | 48 | 12.4% | 0.880 | 70 | 137 | | 96 | 24.8% | 0.944 | 35 | 69 | | 192 | 49.6% | 0.983 | 18 | 35 | **768 d, 75,000 vectors, 274 lists** | nprobe | lists scanned | recall@10 | QPS full precision | QPS eight bit | |---:|---:|---:|---:|---:| | 1 | 0.4% | 0.351 | 2,028 | 2,740 | | 4 | 1.5% | 0.567 | 581 | 1,066 | | 16 | 5.8% | 0.766 | 148 | 294 | | 48 | 17.5% | 0.894 | 49 | 97 | | 96 | 35.0% | 0.954 | 25 | 49 | | 192 | 70.1% | 0.991 | 13 | 25 | The eight-bit layout gives the same recall as full precision at every probe count (the float re-score repairs the code error) and about twice the speed. Real text embeddings have little cluster structure, so reaching recall 0.95 takes a quarter to a third of the lists; this is the nature of an IVF layout without product quantization, and the reason the static index also keeps the HNSW graph for hybrid queries made through the engine. Sizes: full precision 226,172 KiB (1.01 × the float vectors) for 384 d and 226,119 KiB (1.00×) for 768 d; eight bit 283,008 KiB and 282,662 KiB (1.26×, the codes plus the float rerank region). Build: 6 to 9 s for either set on one thread. Peak working set of the benchmark process: 1.1 to 2.3 GB, which includes the input vectors (230 MB) and the build's working copies. # Entity Framework Core integration `RecursiveAI.OneIndex.EntityFrameworkCore` keeps OneIndex indexes in step with an EF Core database and lets a search compose with an EF query. It is **not an EF provider**: the database stays the source of truth, and the index is a derived copy that can always be rebuilt from it. This document is normative for the package's behaviour. Its guarantees are those tested below; [durability.md](https://recursiveai.org/oneindex/docs/durability/) states them with their conditions. ## The sample ```csharp public class Invoice // an EF entity and a OneIndex document at once { [SearchId] public string Id { get; set; } = ""; // a string key [FullText(Boost = 2)] public string Title { get; set; } = ""; [Keyword, Facet] public string Status { get; set; } = ""; [Filterable, Sortable] public int Amount { get; set; } public int CustomerId { get; set; } // not indexed: only the database can filter on it [Dynamic] public JsonElement Meta { get; set; } } public sealed class AppDb(DbContextOptions options) : DbContext(options) { public DbSet Invoices => Set(); protected override void OnModelCreating(ModelBuilder modelBuilder) => modelBuilder.UseOneIndexOutbox(); // the log tables } var options = new DbContextOptionsBuilder() .UseSqlite("Data Source=app.db") .UseOneIndex("data/search", o => { o.Index(); o.ApplyInline = true; }) // console app: apply before SaveChanges returns .Options; using var db = new AppDb(options); db.Invoices.Add(inv); await db.SaveChangesAsync(); // the outbox row is written in the same transaction var list = await db.Invoices.Search("overdue acme").Where(x => x.CustomerId == 7 && x.Amount > 100).Take(10).ToListAsync(); ``` `tests/OneIndex.EntityFrameworkCore.Tests` runs this model (`tests/EfDurability/Model.cs`). With a host, leave `ApplyInline` off and run the applier loop from a hosted service (the package has no dependency on the hosting abstractions, so it does not ship the `BackgroundService`): ```csharp public sealed class IndexApplier(IDbContextFactory factory) : BackgroundService { protected override Task ExecuteAsync(CancellationToken stoppingToken) { using var db = factory.CreateDbContext(); return OneIndexRuntime.For(db).RunAsync(stoppingToken); // set o.ContextFactory = () => factory.CreateDbContext() in UseOneIndex } } ``` ## How it works 1. **The log.** `UseOneIndexOutbox()` maps two tables: the outbox (`Seq`, `EntityType`, `EntityId`, `Operation`) and a one-row state table (`LogId`, `PrunedThrough`). Names and schema are parameters. They are part of the model, so `dotnet ef migrations add` creates them like any other table; apps without migrations get them from `EnsureCreated`. 2. **The interceptor** runs in `SavingChanges`/`SavingChangesAsync`. It detects changes, and for each added, modified or deleted entity of a registered type it adds one `OutboxEntry` (one per entity per save) to the same context, so the row is written by the same `SaveChanges` and the same transaction: both commit or neither does. A change to an owned entity or JSON column of a registered entity logs the owner. A failed save detaches its rows so a retry does not duplicate them. 3. **The applier** (one per index, all driven by `OneIndexRuntime`) reads the log rows after the index's position for its entity type, in `Seq` order and in batches (default 500), loads the **current database state** of the changed ids, upserts the rows that exist, deletes the ids that do not, and commits the index with the new position in the same commit (`SearchIndex.Commit(position)`). The log says "this entity changed"; the database says how. Applying a row twice, or a stale row, therefore gives the same index. 4. **The position** is stored as the commit's applied position (at most 1,024 bytes: a version byte, the log id, the high-water mark, the rebuild base, the applied-row count, and the list of gaps below the mark). Documents and position become durable together or not at all, so after any crash the index is exactly at some position and replays from there. 5. **Out-of-order visibility (gaps).** A database hands out a log position when a transaction inserts its row but shows the row only when the transaction commits, so with concurrent writers position 7 can be visible before position 6 (SQL Server, PostgreSQL, MySQL; SQLite serializes writers and never does). Reading "everything after the highest position seen" would skip 6 for good. So the applier works with a *high-water mark* and a list of *gaps*: positions at or below the mark that had no visible row when it passed them. Each pass reads the window after the mark and also looks the gaps up again; a row that has appeared is applied then and the gap is removed. Because applying reads the entity's current state, a row applied early, late or twice leaves the same index. Gaps live in the index commit with the mark, so they survive a crash. 6. **Giving up on a gap.** A rolled-back insert, or a process that died mid-transaction, leaves a position that never fills. Two bounded rules release it, both configurable: the gap is `LateRowWindow` positions behind the newest row (default 10,000), or this process has waited `LateRowAge` since it first noticed it (default 30 seconds; it must exceed the longest transaction that writes indexed entities; the clock restarts when a process restarts, which only waits longer). Giving up is the only way a late row can be missed: a row that appears at a given-up position is applied by the next change to that entity or by `ReindexAsync`. A position left by a failed save is common on PostgreSQL (sequences are not rolled back), so gaps are normal there and cost a byte or two each. 7. **When the gaps would not fit.** The position is capped at the engine's 1,024-byte limit for the marker (about a thousand gaps when they are close together, fewer when far apart). If recording one more gap would exceed it, the mark **stops before that gap**: it does not advance past it and nothing is released early, but the rows beyond are still applied in that pass (and again in the next, which is harmless), so the index stays equal to the database. The mark moves on as gaps fill or are given up. A rebuild uses the same rule for the positions it saw missing. 8. **Startup reconcile and retention.** At each catch-up the applier compares **positions, not data**. An index with no position (new), a position from another log (restored or replaced database), a position, or an unresolved gap, at or below `PrunedThrough` (retention removed rows it has not applied) or above the log's highest position is **rebuilt from the tables**: the log position is read first, every row is upserted page by page, documents without a row are deleted, and one commit publishes documents and position. Changes after that position are replayed (harmlessly, if the tables already show them). Retention keeps the newest `RetainRows` positions (default 100,000; the applier loop never prunes into the newest `LateRowWindow` positions, so a smaller `RetainRows` acts as that window, because a transaction still open there may commit its row later and positions below the pruning mark are not looked at again; a direct `Prune` call keeps exactly what it is told to and a transaction open across it can lose its row; `Prune`/`PruneAsync` for manual control); the mark moves forward only, and an applier re-checks it after reading rows, so a prune racing a read cannot leave a hole. 9. **One index per process.** Each registered type has its own index directory `/` with the engine's single writer ([processes.md](https://recursiveai.org/oneindex/docs/processes/)); two app instances sharing a database each have their own index directory and apply the same log independently. ### Guarantees - **Exactly-once per index, on any provider**: after catch-up, every index equals the database, including when transactions commit out of position order. Every log row is reflected in each index's commit, and the position says which; it also carries `Count`, the number of rows applied after the rebuild position, which equals the number of such rows in the log while retention has not removed any of them, so a row applied twice or skipped shows. The bounded exception is a position given up on (item 6 above): a transaction that stays open longer than `LateRowAge` and then commits is applied only with its entity's next change or a `ReindexAsync`. - **No lost commits**: a position acknowledged by a returned commit is present after any kill; the campaign records it in a side file and compares. - **Eventual consistency by default**: a search sees the index's last commit; the entities it returns are the database's current rows. - **Cooperative and threaded give the same index**: `CatchUp()` (inline), `CatchUpAsync()` and the `RunAsync` loop use the same code; a test runs one seeded workload through each and compares the three indexes. ### Limits and the documented gap - **Providers run.** SQLite and PostgreSQL are exercised (above). SQL Server and MySQL are not; the gap handling assumes nothing provider-specific beyond a position that grows and becomes visible at commit. - **Bulk statements.** `ExecuteUpdate`, `ExecuteDelete` and raw SQL do not go through `SaveChanges`, so they log nothing. `OneIndexRuntime.ReindexAsync(query)` re-reads the rows a query returns and upserts them (it does not delete); `ReindexAsync()` rebuilds the whole index, including deletions. - **Owned types and JSON columns** reach the index through `[Dynamic]` on a property whose type is a class of plain properties (docs/dynamic.md): EF maps it as an owned type or a JSON column, the generated mapping writes it as a dynamic JSON field. A change to the owned part logs the owner, so the owner's document is rebuilt. - **A failed index commit does not stop the applier for good.** An index writer whose commit failed refuses further work (it cannot know whether the commit became durable). The applier closes that index and opens it again at the next catch-up, which finds the last commit that stands and applies the same log rows; the error is kept in `LastApplyError`. - **A directory is opened by one process at a time.** Inline apply and the loop of the same process share one runtime per directory. - **Inline apply is skipped inside an explicit transaction** (the rows are not committed yet); the loop or the next catch-up covers it. An inline failure is kept in `LastApplyError`, never thrown from `SaveChanges`, because the database change has committed. - **Inline apply and hosted loops need a way to create a context.** The runtime builds its own context for reading, by `OneIndexOptions.ContextFactory` or else by the public constructor of the context type that takes its `DbContextOptions`, learned from the first context it sees (the first `SaveChanges` is enough). If neither is possible, `SaveChanges` with `ApplyInline` and `RunAsync` throw `InvalidOperationException` naming `ContextFactory` before anything is written, rather than leave the index silently behind. Earlier previews learned the constructor only from `OneIndexRuntime.For(db)`, so the first `SaveChanges` of the sample above was not indexed (see the changelog). - **Package shape.** The package references `Microsoft.EntityFrameworkCore` and its relational layer `Microsoft.EntityFrameworkCore.Relational` (table mapping), nothing else. It uses reflection in one place (building a context when none is configured) and EF Core itself is not trim- or AOT-clean, so the package is not marked AOT-compatible. The core package is unchanged in that respect; the AOT smoke app still publishes warning-free and runs. ## Searching `db.Invoices.Search(text)` returns an `EfSearchQuery`: - `Where(predicate)`: the predicate is split on `&&`; each part OneIndex can translate ([linq.md](https://recursiveai.org/oneindex/docs/linq/)) runs **in the index**, every other part runs **in the database**, over the candidates the index returns. A part that neither side can translate fails with EF's message naming the expression; an index-side failure names the sub-expression ([linq.md](https://recursiveai.org/oneindex/docs/linq/)). - `Take(n)` (default 20), `Skip(n)`: the page is **full** unless fewer entities match. Candidates are read from the index in windows (starting at twice the page, doubling to 1,000) and checked against the database until `Skip + Take` survive or the index is exhausted (at most 10,000 candidates). Results are in the index's order; entities deleted since the last apply are dropped; the rows returned are the database's current rows. - `Configure(q => q.OrderBy(...))` adds index-side stages (sort, facets, ranking, highlight); `AsTracking()`; `AfterPendingChanges()`; `ToList()`/`ToListAsync()`. - **Read-your-writes is opt-in.** `await db.SaveChangesAndIndexAsync()` saves and then catches the indexes up; `.AfterPendingChanges()` catches up before one search. By default a search may lag the database by the polling interval (250 ms) or by the time to the next `RunAsync` round. ## Tests and measurements - `tests/OneIndex.EntityFrameworkCore.Tests` (35 tests): outbox atomic with the save and rolled back with it, mirrors of insert/update/delete, position durable across reopen with no re-application, inline apply (the sample above as written, first save, synchronous and asynchronous; an apply that fails once and is caught up by the next; a context the runtime cannot rebuild refused before the save), opt-in read-your-writes, cooperative vs inline vs threaded identical to each other and to the database (150 seeded saves), retention rebuild and in-window continuation, `ReindexAsync(query)` and `ReindexAsync()`, a context factory, full pages under a 1-in-20 database-only filter, conjunct splitting with a `[Dynamic]` path, rows deleted behind the index, paging equals the unpaged list, a differential of 300 random searches (text, 0-3 predicates from seven shapes mixing indexed and database-only, random `Skip`/`Take`) against the index's own hit list filtered by LINQ-to-Objects over the database rows (600 rows; every page equal), and the out-of-order cases (`GapTests`): a row visible below the mark is applied; five seeded workloads of 160 transactions made visible in shuffled order with catch-ups between random batches end equal to the database with no gaps and `Count` 160; give-up by age (not before 30 s) and by distance; 9,751 gaps that do not fit stall the mark but all 49 rows are applied, the position stays within 1,024 bytes and the mark reaches the newest row once gaps are released; a rebuild records the positions it saw missing; a gap that retention has passed forces a rebuild; the pruning mark never moves backwards (the interleaving that once lowered it); and a change to an owned JSON column reaches the index through its owner. Out-of-order visibility is simulated on SQLite (which serializes writers) by committing the entity change and a log row with an explicit position together, in a chosen order. - `tests/EfDurability` (`kill`): per cycle two application processes (instances `a` and `b`, each with its own index directory) write seeded inserts, updates and deletes of a 400-id space to **one database** (SQLite in write-ahead mode, or PostgreSQL) while applying (threaded loop or inline) with a small retention window; each process runs one or more concurrent writers, and with `--hold` a writer sometimes keeps its transaction open for up to 40 ms after `SaveChanges`, so later positions commit first. Each process is killed with SIGKILL at a random time or at a named stage reached by counting probe calls (`save.outbox-added`, `apply.read`, `apply.before-commit`, `apply.after-commit`, `reindex.start`, `reindex.page`, `reindex.before-commit`, `reindex.after-commit`). In about 40% of cycles the harness then writes more rows and prunes the log past the indexes so the next start must rebuild. After kills (every fourth cycle, at the end and at random) each index is reopened, its position compared with the positions its process acknowledged (lost commit), caught up, and compared with the database: same ids and count, every row equal in version, title, status, amount and the owned JSON column, and the applied count equal to the log rows after the rebuild position. A failed open or any difference is a failure. `tools/ef_kill_campaign.sh` runs lanes in parallel (`EF_PG` selects PostgreSQL, `EF_WRITERS`, `EF_HOLD`, `EF_HOOK` shape the load). - **SQLite**, four lanes of 150 cycles: 1,276 kills (400 at named stages of the applier, 876 at random times; seeds 81000 to 81003), 472 index checks, 209 rebuilds after retention, 0 failed opens, 0 lost commits, 0 differences. - **PostgreSQL 16.2** (a user-local server from the `pgserver` wheel, unix socket, no container), four lanes of 150 cycles, two application processes per lane with two concurrent writers each (four concurrent writers per database): 1,809 kills (two settings: 926 and 883; seeds 82000 to 82003 and 83000 to 83003), 654 index checks, 331 rebuilds, 3,433 commits made while positions were still missing, 372 rows applied because they became visible below the mark, 0 failed opens, 0 lost commits, 0 differences. - The campaign found two bugs in this area: a retention pass that read an old pruning mark could lower a newer pass's mark and leave an index silently behind the log (fixed; the interleaving is a test), and a count that included rows applied at positions the rebuild had recorded as gaps (the count now covers rows above the rebuild position only). - `tests/EfDurability` (`bench`): SaveChanges overhead and applier throughput (indicative, 2,000 rows, three runs on an x64 Linux host with 32 cores under other load). One insert per `SaveChanges`: 22.5 / 29.9 / 25.2 ms without the interceptor, 22.1 / 27.5 / 30.0 ms with the outbox row, 28.9 / 29.8 / 24.4 ms with inline apply: the SQLite commit dominates and the outbox row is within the noise. Applier: a rebuild of 2,000 rows 6,300-6,700 rows/s; replay of 2,000 logged updates 4,760-5,510 rows/s. # Folder indexing `RecursiveAI.OneIndex.Files` turns a folder into searchable, cited passages. `RecursiveAI.OneIndex.Files.Ocr` adds scans and photos of text. A passage index is an ordinary OneIndex index of one document type, `FilePassage`, so hybrid search, the embedder, reranking, highlighting and the LINQ builder work on it unchanged. **Everything stays local.** Nothing in either package uses the network. The index holds the extracted text of your files, which is as sensitive as the files themselves: protect the index directory like the folder it was built from, and delete it when you stop needing it. ```csharp using OneIndex; using OneIndex.Files; using SearchIndex index = FolderIndexer.OpenIndex("data/passages"); // add an IEmbedder here for hybrid search FolderIndexResult run = FolderIndexer.Run(index, "docs", progress: new Progress(p => Console.WriteLine($"{p.FilesIndexed}/{p.FilesToIndex} files"))); Console.WriteLine($"{run.Added} added, {run.Updated} updated, {run.Renamed} renamed, {run.Removed} removed, {run.Unchanged} unchanged"); foreach (SearchHit hit in index.Passages("overdue \"acme corp\"").Distinct(p => p.FileId).Take(5)) { Citation c = hit.Cite(); // file, archive entry, page, character range, boxes Console.WriteLine($"{c.File}{(c.Entry is null ? "" : " > " + c.Entry)} page {c.Page} [{c.Start}..{c.End}]: {c.Text}"); } ``` This sample is compiled and run by `tests/OneIndex.Files.Tests` (`DocsSampleRuns`). Add the OCR extractor to read scans: ```csharp var options = new FolderIndexOptions { Extractors = [new TextExtractor(), new HtmlExtractor(), new PdfExtractor(), new OcrImageExtractor()] }; ``` ## What is read | Files | Read by | Units | Geometry | |---|---|---|---| | `.txt`, logs, CSV, source code and other text (a file without a known extension is read as text if it looks like text) | `TextExtractor` | the whole file | no | | `.md`, `.markdown` | `TextExtractor` | the whole file; headings, lists, tables and fenced code are blocks | no | | `.html`, `.htm`, `.xhtml` | `HtmlExtractor` (onehtml) | the page's visible text | no | | `.pdf` | `PdfExtractor` (OnePDF) | one unit per page with text | boxes in points | | `.png`, `.jpg`, `.tif`, `.webp`, `.bmp`, `.gif` | `OcrImageExtractor` (oneocr, package `.Files.Ocr`) | one unit per frame | boxes in pixels | | `.zip`, `.tar`, `.tgz`, `.7z`, `.gz`, `.bz2`, `.xz`, `.zst`, `.cpio`, `.iso`, `.cab`, `.rar` | the indexer, entry by entry (onearchive) | each entry is a file of its own | as its format | | anything else | not read | a record with format `binary` | no | Archives nest: `/backup.zip!/reports/annual.zip!/summary.pdf` is a PDF inside a zip inside a zip, down to `MaxArchiveDepth` (default 4). Entry sizes, the entry count and the total size are limited (`FolderIndexOptions`); an entry over its limit, a path that climbs out with `..` and a damaged archive never stop the run. A scanned PDF without a text layer yields no passages; its page images are not read (see "Not done"). Ignore rules follow `.gitignore`: `.gitignore` and `.oneindexignore` files in any folder, `FolderIndexOptions.IgnorePatterns`, and by default version-control folders, `node_modules`, `__pycache__`, editor and system litter (`IgnoreRules.Defaults`). Links are not followed. ## Identifiers A file's identifier is its path below the root with `/` separators and a leading `/` (`/docs/a.md`); an archive entry is `/archive.zip!/inner/path.txt`. A passage is `FileId#ordinal`; a file's record is `file:FileId`. The three never collide, so ids are deterministic and an interrupted and a clean run give the same ids. Search results come from passages only (`index.Passages(text)`); `index.Errors()` lists the error documents. `Distinct(p => p.FileId)` groups hits per file. `Where(p => p.Format == "pdf" && p.Page == 3)` and the rest of the LINQ surface work on `FilePassage`'s members. ## Passages The chunker (`Chunker.Cut`) follows the structure of each unit: - A heading starts a passage; a heading followed at once by a deeper one stays with it. The headings above a passage, outermost first, are in `Heading` (searched at weight 2). - Blocks (paragraphs, list items, tables, code) fill a passage to the **target** of 256 estimated tokens. A passage under 48 tokens takes the next block even past the target, up to the **maximum** of 384. - A block over the maximum is cut at UAX #29 sentence boundaries (at lines for tables and code); a sentence over the maximum is cut at word ends, then anywhere. - A passage never spans units (pages), and is always an exact substring of its unit's text, so `unit.Text[Start..End]` is the passage. - Tokens are estimated, not counted: four characters of Latin text are a token, two of other alphabetic scripts, one CJK character or kana. `ChunkOptions` sets the three sizes. The unit text depends on the format. For text and Markdown it is the decoded file (UTF-8 with or without a byte-order mark, UTF-16 with a mark, otherwise Latin-1). For HTML it is the extractor's rendering of the page; scripts, styles, templates and SVG are dropped, whitespace is collapsed, blocks are separated by a blank line, table rows by a line feed and cells by a tab. For PDF and scans it is the blocks of the page in reading order, a block's lines joined by a line feed and blocks by a blank line; headers and footers that repeat across a PDF's pages are left out. ## Citations `hit.Cite()` returns a `Citation`: `File` (path below the root), `Entry` (path inside the archive, or null), `Page` (from 1; 0 without pages), `Start` and `End` (UTF-16 code units in the page's extracted text), `Boxes`, `BoxUnits`, `PageWidth`, `PageHeight`, `Heading`, `Text`. - **The range is exact.** `pageText[Start..End] == Text`. `citation.ReadSourceText(root, options)` extracts the page again from the file and returns that substring: use it to show the source, or to verify a citation. - **Boxes** (PDF and scans): one box per line of text the passage touches, the union of its words' boxes on that line, with the origin at the top left and y growing downwards, in points (PDF) or pixels (scans), inside the page of `PageWidth` by `PageHeight`. They exist for every passage of a PDF or scan. - **HTML** ranges count characters of the extracted text, not of the markup. - A file that changed since it was indexed can no longer be re-read to the same text; `ReadSourceText` then returns different text or throws `InvalidDataException`. Run the indexer again. ## Keeping the index current `FolderIndexer.Run` is one pass and can be repeated at will: 1. Walk the folder in path order and compare each file with its record: **size and modification time**. Equal means unchanged and the file is not opened. 2. A file whose size or time differ is hashed (SHA-256); the same bytes only update the record (a touch). Other bytes are read again, and the file's old passages that the new version does not replace are deleted. 3. A new file whose size and hash equal a vanished file's is a **rename**: its passages are copied under the new identifiers without extraction (and without embedding again). 4. Files that vanished are removed with all their passages and, for an archive, its entries. 5. A file indexed with other extractors or other chunk sizes (the record's `Fingerprint`) is read again. Everything for one file is written between two commits, and the indexer commits every `CommitEveryFiles` (200) files or `CommitEveryBytes` (256 MiB) read. After a crash or a kill the index holds whole files only; running again picks up where the last commit left off and ends with the same passage identifiers and text as an uninterrupted run (the kill campaign below tests exactly that). Cancelling a run commits what is complete first. After any other exception from `Run`, dispose the index: the uncommitted tail is discarded. **Extraction failures** become documents of kind `error`: the record carries the exception's type and message in `Error`, searchable with `index.Errors()`. They are not retried while the file stays as it was. A per-file time limit (`FileTimeout`, default 5 minutes) covers reading, extraction and chunking, archives included; `MaxFileBytes` (1 GiB) turns an oversized file into an error document. **Threads.** The default (`MaxDegreeOfParallelism = 1`) does everything on the calling thread. A larger value extracts that many files at once on pool threads and still writes them in path order, so the index is the same for every setting. `RunAsync` is `Run` on a pool thread. **Watching.** `FolderWatcher` indexes once, then reruns the indexer a debounce (default 2 s) after file-system events and every 10 minutes regardless (watchers miss events: queue overflow, network shares, platform limits). It runs on one low-priority background thread and is the index's only writer while alive. A rerun is a directory walk plus the files that changed. Tested on Linux (inotify) and macOS (FSEvents); on Windows it uses the platform watcher and the same rerun. ## Embeddings `FilePassage.Embedding` is filled from `Text` when the index has an embedder (`FolderIndexer.OpenIndex(path, embedder)`). The vector takes the embedder's dimension (384 for the packaged default, 768 for a base-size model) when the index is created; the size is part of the index, and opening it with an embedder of another size is refused. `tests/OneIndex.Files.Tests/EmbeddingSizeTests` runs a 768-dimension embedder. Hybrid search then works as described in [hybrid.md](https://recursiveai.org/oneindex/docs/hybrid/). Without an embedder the index is text-only. Embedding dominates the cost of indexing a folder (see [models.md](https://recursiveai.org/oneindex/docs/models/) for throughput); the measurements below are text-only. ## Checks All numbers are from the generated corpus in `tools/OneIndex.Files.Tool` (seed 1: `gen --files 6000 --mib 1100 --scans 12`): 6,191 files, 1.1 GiB, in 10 folders — 1,680 text files, 1,368 short notes, 1,200 Markdown, 900 HTML, 480 PDF (3 to 28 pages each, generated with OnePDF, with a repeating header and footer), 300 zip archives (every third with an archive inside), 12 scanned pages (rendered text in PNG, read with oneocr), 62 corrupt files (random bytes named `.pdf`, `.zip`, `.png`, deeply nested HTML, truncated PDF and zip), 10 encoding and ignore-rule cases and 179 filler binaries of 3 to 5 MB (696 MiB). Entries of the archives count as files in the index: 12,843 file records and 581,632 passages in all. - **Kill campaign** (`oneindex-files-tool campaign`, the corpus above on x64 Linux, Release, one extraction thread, commit every 10 files): a clean run, then 100 runs of the indexing child process each killed with SIGKILL after a seeded random 0.3 to 2.5 s, the index reopened after every kill, then a final run to completion. Result: 100 of 100 kills delivered, 100 of 100 reopens succeeded, the file-record count never fell between kills (no lost commit), 0 differences between the resumed and the clean index (identifier, kind, format, page, range, record counts, hashes of text, boxes and headings: 12,844 file records, 61 error documents, 581,678 passages), 0 duplicate file records, 0 files whose passage count differed from their record, 0 orphan passages, 61 error documents for 60 corrupt-on-purpose files plus the truncated zip. An earlier campaign with four extraction threads and seed 1 delivered 24 kills before the folder was done and gave the same result. - **Citations**: 1,500 passages sampled with seed 3 across formats (HTML 362, images 12, Markdown 408, PDF 311, text 407; archive entries among them): the page re-extracted from the file gave exactly the passage text for 1,500 of 1,500, and every PDF (311) and scan (12) passage had boxes inside the page. - **Change detection**: `FolderIndexerTests.ARescanAppliesExactlyTheChanges`: modify, rename, delete, add and touch files; a rescan reports exactly 1 updated, 1 renamed, 1 removed, 1 added and 1 unchanged (touch), reads no file of an unchanged folder, and the resulting index equals one built from scratch. - **Throughput** (indicative, not a benchmark; a 32-core x64 Linux machine shared with other jobs, load average 1 to 19 during the runs; first index of each folder into a fresh index, text only, OCR for the scans): | Folder | Files | MiB | Passages | 8 threads: seconds, files/s, MiB/s | 1 thread: seconds, files/s, MiB/s | |---|---:|---:|---:|---|---| | text | 1,680 | 163 | 209,453 | 27.3 s, 62, 6.0 | 28.1 s, 60, 5.8 | | md | 1,200 | 93 | 137,822 | 14.3 s, 84, 6.5 | 14.4 s, 83, 6.5 | | web (HTML) | 900 | 78 | 107,153 | 11.9 s, 76, 6.6 | 10.8 s, 83, 7.2 | | archives | 300 | 36 | 100,273 | 12.3 s, 24, 2.9 | 12.4 s, 24, 2.9 | | pdf | 480 | 29 | 21,041 | 3.4 s, 143, 8.7 | 4.9 s, 98, 6.0 | | scans (OCR) | 12 | 1 | 12 | 0.7 s, 18, 1.4 | 1.3 s, 9, 0.7 | | notes | 1,368 | 4 | 5,783 | 0.9 s, 1,562, 4.8 | 0.7 s, 1,961, 6.1 | | blobs (binary, hashed only) | 179 | 695 | 0 | 0.2 s | 0.4 s | Peak working set 1.3 GiB (8 threads) and 1.1 GiB (1 thread) for the whole process. Writing passages into the index is the limit: more extraction threads help only PDF and OCR. The text index builds about 7,500 passages per second here. ## Scanned PDFs `OcrPdfExtractor` (package `.Files.Ocr`, on oneocr's PDF package) reads PDFs whose pages may be pictures: each page is its text layer where it has one plus the recognised text of the images on it, with a box in PDF points (origin top left) for every word. Put it before the default PDF extractor in `FolderIndexOptions.Extractors`; it replaces that extractor for PDFs (`oneindex index --ocr` does this). Recognition takes seconds per page, so give it a generous `FileTimeout`. `OcrTests.AScannedPdfPageBecomesCitedPassagesWithBoxesInPoints` draws a scan, wraps it in a one-page PDF with OnePDF, indexes it, and checks that every passage has page 1, boxes in points inside the page. ## Not done - Office documents, mail and media wait for their libraries. - A renamed archive is read again (its entries' identifiers change with its name). - A file and an archive entry can only share an identifier if a folder is named `name!` next to an archive `name`. # The VectorData adapter `RecursiveAI.OneIndex.VectorData` is a vector-store collection over OneIndex for `Microsoft.Extensions.VectorData` (version 10.11.0 of the abstractions, the one dependency). It covers what the abstraction defines: collections, upsert, get, delete, vector search with filters, and keyword plus vector hybrid search. ```csharp var store = new OneIndexVectorStore("data/vectors"); // a directory; new OneIndexVectorStore() keeps collections in memory var hotels = store.GetCollection("hotels"); await hotels.EnsureCollectionExistsAsync(); await hotels.UpsertAsync(hotel); // every write is committed before it returns await foreach (var r in hotels.SearchAsync(queryVector, top: 5, new() { Filter = h => h.Rating >= 4 })) Console.WriteLine($"{r.Record.Name} {r.Score}"); ``` ## What is supported - Keys `string`, `Guid`, `int`, `long`. A `Guid` key is generated when the record carries `Guid.Empty` and the key is auto-generated (the default for `Guid`). Asking for a collection with a key type other than the record's key property throws `InvalidOperationException`. - Data properties: strings (full-text indexed when `IsFullTextIndexed`), integers, floating point numbers, `bool`, `DateTime`, `DateTimeOffset`, string collections. `IsIndexed` makes a property filterable. - Vector properties `ReadOnlyMemory` or `float[]`, with the distances cosine similarity and distance, dot product, negative dot product, and Euclidean (squared) distance. Scores come back in the unit of the chosen distance function. The vector index is the engine's: flat below the segment threshold, HNSW above. - Filters are the abstraction's LINQ predicates; they are applied while the neighbours are chosen, so a filtered search returns a full page when enough records pass. `Skip`, `ScoreThreshold`, `IncludeVectors` work. - Hybrid search (`IKeywordHybridSearchable`): the keywords search one full-text property (the one named by `AdditionalProperty`, or the only one; none or several without a choice throws), fused with the vector by reciprocal rank fusion ([hybrid.md](https://recursiveai.org/oneindex/docs/hybrid/)). - Records are described by the abstraction's attributes or by a `VectorStoreCollectionDefinition`. ## What is not - **Dynamic collections** (`GetDynamicCollection`, dictionary records) and **embedding generation** by the store: `NotSupported`. Pass vectors yourself. - Distance functions Manhattan and Hamming. - **Data-type suite.** Each of its tests also creates a dynamic collection, so all 16 are excluded; the typed path (`Guid`, `DateOnly`, `TimeOnly`, nullable values, string collections) is covered by `ContractTests`. Filters comparing a whole array (`r.StringArray == array`) are not translatable. - **Registration extensions.** The package ships no `AddOneIndex...` dependency-injection extension (it would add a dependency beyond the abstraction); applications register `OneIndexVectorStore` and their collections as services, as the tests do. Embedding generators are not resolved from the container (`EmbeddingGeneratorIsResolved` is excluded). - A caveat on `Guid` keys: the abstraction's key-type test also exercises a dynamic collection, which is not supported, so that one test is excluded. ## Reflection, trimming and AOT The adapter reads and writes user record types through reflection, as the abstraction itself does. `OneIndexVectorStore.GetCollection` and `OneIndexRecords` carry `RequiresUnreferencedCode` and `RequiresDynamicCode`, so a trimmed or NativeAOT application gets warnings at the call, not silent breakage. The package is therefore **not** marked `IsAotCompatible` or `IsTrimmable`. The core package stays reflection-free; an application that needs AOT uses `SearchIndex` with the generated mapping directly. ## Conformance The tests run the abstraction's own conformance suite (`Microsoft.Extensions.VectorData.ConformanceTests` 10.11.0) against a store over a temporary directory, plus 8 contract tests of our own. | | Tests | |---|---:| | Contract tests, wired conformance suites, dependency-injection suite (`dotnet test`) | 168 passed, 0 failed, the unsupported cases below reported as skipped | | Skipped with a reason (`tests/OneIndex.VectorData.Tests/Unsupported.cs`) | Manhattan and Hamming distance (2); the `Guid` key test, which also needs a dynamic collection (1); `EmbeddingGeneratorIsResolved` in 9 variants (lifetime and service key); the 16 data-type tests, which need a dynamic collection | Filters comparing with `null` and `Contains` / `Any` over a collection property, excluded before, now pass (10 tests). Wired suites: collection management, distance functions, index kinds, filters (typed), basic, no-data, no-vector and multi-vector models, key types, hybrid search, dependency injection, data types (all excluded). Not wired: dynamic-model, embedding-generation and embedding-type suites (no dynamic collections, no generated embeddings). A plain `dotnet test` runs them all; the unsupported cases skip themselves, each naming the feature. # OneIndex in the browser Everything in the core package runs in browser WebAssembly (`browser-wasm`, no threads). Storage there is one of three things: | Backend | Use | Notes | |---|---|---| | `MemoryStorage` | an index that lives for the page | read and write; the same code as on the server | | OPFS, through `OpfsPersistence` | an index kept between page loads | the engine writes synchronously and the main thread cannot block on file writes, so the index lives in a `MemoryStorage` while the page runs and is saved to, or loaded from, OPFS whole, file by file, through the application's `IOpfsHost` (four async methods over `navigator.storage.getDirectory()`). OneIndex has no JavaScript interop itself, so the host is the application's `[JSImport]` code. A save writes the commit file last and skips files no commit lists, so an interrupted save leaves the previous commit or the new one. | | `ChunkedStaticStorage` over `HttpStaticSource` | a read-only static index from a file server or CDN | [static-index.md](https://recursiveai.org/oneindex/docs/static-index/) | The static index is chunked whole-object GETs rather than HTTP range requests: it needs nothing from the host and every chunk is cacheable. Reads through it throw `ChunkNotLoadedException` for a chunk not yet fetched, and the query helper fetches and re-runs. ## Browser suite `tests/BrowserSuite` publishes `BrowserApp` (browser-wasm, trimmed Release) and drives it in headless Chromium with `playwright-core` (the system Chrome through `executablePath`; Playwright's files are not read). `run.mjs` serves the app and a static index over HTTP, counts the bytes it sends, and compares what the browser prints for each of 40 queries (12 text, 8 filter, 8 hybrid, 8 vector, 4 highlight; ids and scores) with what the same static index returns on the server (`tools/BrowserFixture`). Three modes: **http** (the static index), **memory** (the index copied into `MemoryStorage`) and **opfs** (saved to and reloaded from the real OPFS, then queried). `run-worker.sh` does the whole run for a given abstracts file. Result on 30,000 Wikipedia abstracts (24 files, 31.0 MB, 64 KiB chunks), Chromium 155.0.8059.84 headless on the worker, Node 22.22.1, playwright-core 1.49.1: | Mode | Queries compared | Matched | Median ms | p95 ms | |---|---:|---:|---:|---:| | http | 40 | 40 | 98 | 697 | | memory | 32 | 32 | 10 | 42 | | opfs | 32 | 32 | 10 | 31 | (Memory and OPFS have no IVF file, so they skip the 8 vector queries.) The 40 http queries made 455 requests and 215 re-runs. Cache-warm session, queries in sequence; the median over all kinds is 98 ms, the p95 is a hybrid query. Over http, per query kind (the first query excluded; it opens the index): | Kind | Queries | Median ms | Bytes fetched (avg) | Requests (avg) | |---|---:|---:|---:|---:| | text | 11 | 98 | 536 KB | 8.2 | | filter | 8 | 72 | 139 KB | 2.1 | | hybrid | 8 | 387 | 1,294 KB | 19.8 | | vector (IVF) | 8 | 97 | 718 KB | 11.1 | | highlight | 4 | 55 | 115 KB | 1.8 | | first query | 1 | | 5,291 KB | | **Payload** of the .NET WebAssembly app (runtime, the engine, the test app): 5,247,641 bytes raw, 1,933,623 gzip, 1,539,151 brotli (the app bundle, symbols and maps excluded). **Re-run with paged reads, 4 KiB chunks for scattered files and hybrid through IVF** (the current default; the same 30,000 abstracts, Google Chrome for Testing 156.0.8078.47 headless, Node 22.22.1, playwright-core 1.49.1, the worker, x64): http 40 of 40 equal to the server's static index, memory 24 of 24, OPFS 24 of 24 (memory and OPFS open the files as an ordinary index, whose hybrid uses the HNSW graph, so their 8 hybrid queries are not counted: they legitimately differ from the IVF-based static hybrid, which `static-index.md` compares by overlap). Http: median 155 ms, p95 941 ms over all 40; first query 646 KB (plus the manifest, about 737 KB); per kind (median ms, average KB, average requests): text 74 / 198 / 48.5, filter 107 / 61 / 14.9, hybrid 549 / 371 / 65.5, vector (IVF) 145 / 462 / 72, highlight 208 / 186 / 45.5. Compared with the 16 KiB run before it (first query 2,686 KB; text 60 ms / 214 KB / 13 requests; hybrid 638 ms / 1,341 KB / 82 requests): bytes fall, requests and latency rise for text because a 4 KiB chunk is one more round trip (the engine re-runs a query once per missing chunk, serially). App payload 5,351,162 bytes raw (gzip 1,975,115, brotli 1,573,139). Pagefind 1.6.0, same run: first query 323,199 bytes, median 10.9 ms, p95 102.7 ms, 69,501 bytes per query over all 12 (46,437 over the 11 after the first). Details: [static-index.md](https://recursiveai.org/oneindex/docs/static-index/). ## Beside Pagefind Pagefind 1.6.0 (MIT, the npm package, run as a black box) indexes the same 30,000 abstracts as HTML pages (one per abstract; `tests/BrowserSuite/pagefind.mjs`), and the same Chromium build measures the 12 text queries of the suite: `search()` plus loading the first ten results' data. | | OneIndex static index | Pagefind 1.6.0 | |---|---:|---:| | Runtime payload before the first query | 5.25 MB raw, 1.93 MB gzip, 1.54 MB brotli | 575 KB raw, 226 KB gzip, 183 KB brotli (JS and wasm, plus 90 KB of index metadata fetched at init) | | Index on the server | 31.1 MB (the static directory; vector files are about 13 MB of it) | 15.3 MB in 30,182 files | | First query, bytes | 646 KB (737 KB with the manifest) | 323 KB | | Text query after the first, bytes (avg) | 198 KB in 48.5 requests | 46 KB in 4.4 requests (the 11 after the first; all 12 average 70 KB) | | Text query, median latency | 74 ms | 10.9 ms (p95 102.7 ms) | Pagefind is smaller and faster on this row, by an order of magnitude in payload and bytes. It is a different thing: a pure keyword search over pre-split index files with no filters, vectors, hybrid or exact engine results. The causes of our numbers are in [static-index.md](https://recursiveai.org/oneindex/docs/static-index/): a .NET runtime payload; stored-source blocks for each hit (about 60 KB of the 198 KB); term-dictionary pages (about 94 KB). These rows are reported, not gated. ## Other platforms The WebAssembly smoke app (`tests/WasmSmoke`, trimmed Release) runs the shared scenarios (index, analysis, queries, ranking, vectors, a writer with merges, file-system storage in the runtime's in-memory file system, and the IVF index with MaxSim) in Node and prints OK. The NativeAOT smoke app (`tests/AotSmoke`) adds the static index build and read; it publishes with no warnings (warnings are errors) and runs. A directory index takes a writer lock, and the browser runtime has no file locks, so there is no directory-backed writer in the browser: use `MemoryStorage` and OPFS persistence. # Static index A static index is an ordinary OneIndex index cut into small immutable files, so a browser (or any HTTP client) can search it from a plain file server or CDN with no server process. `oneindex static build` writes one; `StaticIndex` reads one. ```csharp StaticManifest manifest = StaticIndexBuilder.Build("data/index", "site/search"); // build time StaticIndex index = await StaticIndex.OpenAsync(new HttpStaticSource(new HttpClient(), new Uri("https://example.org/search/"))); var hits = await index.QueryAsync(i => i.Search("fox").Where(d => d.Year > 2000).Take(10).Execute()); var near = await index.SearchVectorAsync("Embedding", vector, k: 10, nprobe: 8); ``` ## Layout The directory holds `static.json` and the files of **one commit** of the source index (the newest), each cut into chunks. Files read in runs (the IVF file and the vectors) use `ChunkBytes` (default 16 KiB, 1 KiB to 64 MiB); files probed at scattered places (term dictionaries, lengths, identifier tables, columns, points, packs, postings, stored source) use `ScatteredChunkBytes` (default 4 KiB), so a lookup fetches a few small chunks instead of a few large ones. A file of one chunk or less keeps its name; a longer file is stored as `name.c00000`, `name.c00001`, and so on. For every vector field the build adds `ivf-.ivf` ([ivf.md](https://recursiveai.org/oneindex/docs/ivf/)) and `ivf-.ids` (the table from IVF numbers to document identifiers), also chunked. The source index's locks, leases, temporary files and older commit files are left out; unreferenced segment files from an unmerged index come along, so build from a freshly committed or merged index. `static.json` (UTF-8 JSON, version 1): ```json { "v": 1, "chunkBytes": 16384, "commit": "commit-000000000000000f.oic", "files": [ { "name": "s0000000000000014.stored", "size": 7853755, "chunk": 4096, "crc": [3735928559, ...] }, ... ] } ``` `crc` holds the CRC-32C of each chunk in order. `chunk` (optional) is the file's own chunk size; without it the manifest's `chunkBytes` applies, so manifests written before it existed still read. The parser is strict: unknown versions, missing or mistyped fields, sizes or counts past the limits, and crc lists of the wrong length raise `CorruptIndexException`. ## Reading `ChunkedStaticStorage` is a read-only `IStorage` over an `IStaticSource` (`HttpStaticSource` over `HttpClient`, `DirectoryStaticSource` for tests). Every chunk fetched is checked against the manifest's CRC before it is used; a mismatch is a `CorruptIndexException` naming the file. Chunks are plain whole-object GETs, so any static host works: no range support, no server code, and each chunk caches in the browser and on a CDN. (The storage interface also admits true range backends; chunking was chosen because it needs nothing from the host.) The engine reads synchronously, but a browser fetch cannot block. So `QueryAsync` runs the ordinary engine query against the chunks that are cached; when the engine touches a chunk that has not been fetched, the read throws `ChunkNotLoadedException`, the query stops, the missing chunk and, for the IVF and vector files only, `ReadAhead` (default 1) chunks after it are fetched asynchronously, and the query runs again. The final run is the normal query, so **results equal the same query on the source index** (ids and scores), by construction. The cache holds up to 128 MiB of chunks by default. **Paged reads.** On static storage a segment's term dictionary and field lengths are not read whole at open (`PagedBytes`: 4 KiB pages filled on first touch, each bounded by the file size and the limits), and the whole-file checksum read is skipped, because every chunk is already CRC-checked against the manifest; only the header and footer are validated. Local storage keeps its eager open and whole-file checks. The paged reader answers exactly like the resident one (a test compares them over a random corpus, and the `paged` fuzz target feeds it hostile bytes). **Parts open on first use.** On a static index the column, point and vector parts of a segment are not opened with the segment: they are opened by the first query that asks for a column, a point field or a vector field (`IDeferredOpenStorage`; the same code on local storage opens everything eagerly, so corruption there is still found at open). A text query therefore does not fetch the columns, the points or the HNSW graph; it reads the directory (last chunk) of the vector files at most. A failure while opening a deferred part is raised by the query that needs it and is not remembered. `Retries` counts the re-runs. The query delegate must be repeatable and must materialise its result inside the delegate. Vector search through `SearchVectorAsync` uses the IVF file: it reads the centroids once, then only the probed lists and the candidates' full-precision rows. **Hybrid through IVF.** A `Hybrid` query run through `QueryAsync` takes its vector arm from the IVF file (`HybridNprobe`, default 16; the approximate code scores, no float rows) and fuses it with the text arm by the usual fusion, so it never walks the HNSW graph. The filter and deletes apply to the IVF candidates; when fewer than K survive, the arm asks again for more (4x, up to the maximum). The build writes `ivf-.docs` (document id per vector number) when that is not the identity mapping. Because the candidate set differs from the server's graph search, static hybrid results are close to, not identical with, the server's hybrid (measured below); the fusion itself is the same code, and a test checks it equals the server's fusion over the same candidates. ## Limits and failures A chunk that fails its CRC, a manifest that fails to parse, a file shorter than the manifest says and a source that cannot return a chunk all raise typed exceptions; nothing is trusted before its checksum. The reader has a fuzz target (`static`: manifest, IVF file and packed multi-vector bytes, including `MultiVector.ToRows`; 600 s on one core: 31,659,431 executions, 0 findings in the final run; the first run found an `OverflowException` on a list entry above `int.MaxValue`, now `CorruptIndexException`, with its repro in `testdata/fuzz/static-ivf-list-id-overflow.bin`). ## Measured The browser suite ([browser.md](https://recursiveai.org/oneindex/docs/browser/)) builds a static index of 30,000 Wikipedia abstracts and runs 40 queries in headless Chromium (12 text, 8 filter, 8 hybrid, 8 vector, 4 highlight). `tools/StaticBytes` runs the same queries through a counting source and prints bytes and requests per kind and compares every text, filter and highlight result with the same query on the ordinary index (24 of 24 equal) and the hybrid results by overlap. Machine: the LAN worker (x64), .NET 10; the numbers are bytes, so load does not matter. | KB per query after the first (open's manifest read in the last column) | first query | text | filter | hybrid | vector (IVF) | highlight | manifest | |---|---:|---:|---:|---:|---:|---:|---:| | 64 KiB chunks, whole parts at open | 5,167 | 524 | 136 | 1,264 | 701 | 112 | | | 16 KiB, parts on demand | 2,623 | 210 | 108 | 1,310 | 565 | 188 | 40 | | + paged reads (16 KiB chunks everywhere) | 1,407 | 298 | 110 | 1,310 | 579 | 192 | 40 | | + 4 KiB chunks for terms, lengths, ids, columns, points, packs | 867 | 307 | 110 | 1,308 | 589 | 192 | 55 | | **default**: 4 KiB for postings and stored too, hybrid through IVF (nprobe 16) | **631** | **194** | **60** | **338** | 463 | 187 | 106 | The first query is about 646 KB, or 737 KB with the manifest (it was 2.6 MB), against Pagefind 1.6.0's 280-323 KB; a later text query is 194 KB against Pagefind's 46-70 KB. **The text query stays above the 120 KB aim:** per text query about 94 KB is term-dictionary pages, 60 KB stored source (ten hits, each a 32 KiB raw block that the stored codec compresses in whole blocks, so a hit costs at least its compressed block; only a smaller stored block size, a writer-side option with a cost in compression, would lower it), 21.5 KB postings and 14.5 KB packs. We accept 194 KB for now. Costs: the manifest grew from 40 to 106 KB because every chunk has a CRC entry (a compact encoding would about halve it); a 4 KiB chunk raises the requests of a text query from 13 to about 48 (Chromium: median 74 ms per text query, see [browser.md](https://recursiveai.org/oneindex/docs/browser/)). **Static hybrid against the server's hybrid** (the server's vector arm is the HNSW graph; 8 queries, 30,000 abstracts, overlap of the top-10 ids and the number of identical lists): nprobe 4: 0.500, 0 of 8; 8: 0.713, 0; **16 (default): 0.887, 1**; 32: 0.950, 2. Hybrid bytes fall from 1,310 KB to 338 KB at nprobe 16. Raise `HybridNprobe` for closer agreement at more bytes. # Durability What OneIndex promises about your data when a process is killed, a disk call fails or the power goes out, under which conditions, what it does not promise, and how each promise is tested. File layouts are in [commit-format.md](https://recursiveai.org/oneindex/docs/commit-format/), the storage contract in [storage.md](https://recursiveai.org/oneindex/docs/storage/), the multi-process rules in [processes.md](https://recursiveai.org/oneindex/docs/processes/) and the Entity Framework Core integration in [entity-framework.md](https://recursiveai.org/oneindex/docs/entity-framework/). ## The promises 1. **A commit that returned is not lost.** `Commit` returns only after every new file is written and fsynced, the commit file is written and fsynced, and it has been renamed into place. After any later kill of the process, `kill -9`, crash or reboot of the application, the next open sees that commit or a newer one. 2. **A reader sees one whole commit.** The commit file is published by one atomic rename, so a reader (in this process or another) sees a complete earlier commit or the complete new one, never a mixture. Files a commit lists are never changed after the rename; a commit and a snapshot keep their files alive until the last reader lets go, in any process. 3. **A crash at any instant leaves an openable index.** The last commit stands. The files a crashed writer left behind are unreferenced and are removed by the next writer's `Open`. Opening never falls back silently to an older commit: if the newest commit or a file it lists is damaged, the open fails with `CorruptIndexException` naming the file. 4. **Damage is detected, not served.** Every index file has a magic, a format version and a CRC-32C footer; the commit records each file's length and footer checksum; readers check lengths and counts against limits before trusting an offset. Torn, lost, reordered or bit-flipped bytes become `CorruptIndexException` (`UnsupportedFormatException` for an intact file from a newer writer), or a failed open of the damaged commit, never a silently wrong result. 5. **One writer.** A second writer on the same directory, from any process on the same machine, is refused at once with `IndexLockedException`; the lock is released when its process dies. 6. **A failed disk call does not corrupt.** If a storage call fails (full disk, I/O error, cancellation), the commit in progress is not published, the previous commit stands, and an index reopened afterwards holds exactly the commits that returned. A writer whose commit call failed refuses further work, because it cannot know whether the commit became durable; open the index again. (The Entity Framework applier does this for you.) 7. **Near-real-time views are not durable.** A view from `OpenNearRealTimeReader` shows uncommitted work and says so (`IsDurable` is false). It disappears with the process. ## Conditions - A local file system on one machine that honours `fsync` and atomic `rename`: ext4, XFS, btrfs, APFS, NTFS. Network and FUSE file systems are not supported: they may not implement the exclusive lock, or may lose a rename they acknowledged. - The index directory is changed only by OneIndex. Files edited or truncated by something else are detected by checksum when read (promise 4), but the index is then damaged. - Hardware that does not lie about flushing. A disk that acknowledges a flush it did not perform breaks every database's promises, this one's included. - Linux (x64, arm64), macOS (arm64) and Windows (x64, NTFS) run the full test suite on every change; the kill and fault-injection campaigns run on Linux and macOS only. See [processes.md](https://recursiveai.org/oneindex/docs/processes/). ## What is not promised - **Power loss is weaker than a process kill.** .NET cannot open a directory to fsync it, so `SyncDirectory` fsyncs each file created or renamed since the last call instead of the directory. On Linux file systems with ordered metadata journals this makes the rename durable in practice, but that is a property of those file systems, not of POSIX, and it has not been tested by cutting power. The worst case is that the newest commit (or several, if a file system does not keep order) is rolled back to an earlier complete one. It is never a corrupt index: a commit lists only files that were synced and checked. The fault injector models the strict case (a rename survives only after the directory is synced) and the engine passes it. See [storage.md](https://recursiveai.org/oneindex/docs/storage/). - **After a power cut the open index may be an earlier acknowledged commit.** An application that cannot tolerate that should run a directory fsync of its own (for example from a supervising process). - **An index is a derived structure for the Entity Framework integration.** The database is the source of truth; the guarantee is that the index equals the database after catch-up, not that it is a second durable copy (see below). - **Static indexes are built, not committed.** `oneindex static build` writes chunk files and then a manifest, without fsync. A killed build leaves no manifest (the manifest is written last, under a temporary name, and renamed), so nothing serves a half-built index, and the build can be rerun into the same directory. After a power cut during or just after a build, rebuild before uploading. Readers check every chunk against the manifest's CRC-32C and fail with `CorruptIndexException` on a chunk that does not match; a chunk that is missing fails the read. - **Browser persistence needs an atomic host write.** `OpfsPersistence.SaveAsync` writes segment files first and commit files last; an interrupted save leaves the previous commit or the new one, provided the host's `WriteAsync` replaces a file atomically (the File System Standard expects a writable stream's close to do so). A host that can leave a half-written file can make the newest commit unreadable, which fails to open with `CorruptIndexException`. - **Entity Framework: a log row can be lost if a transaction stays open longer than the late-row window or is pruned past.** See the next section. ## Entity Framework Core integration The guarantee, as tested: **after catch-up, each index equals the database** (same identifiers and the same indexed fields for every row), and a position the index acknowledged is never lost, including across kills of the application and of the applier, with two application instances (each with its own index directory) on one database. Conditions and limits: - **Providers verified: SQLite and PostgreSQL 16.** SQL Server and MySQL have not been run. The applier assumes only that log positions grow and that a row becomes visible when its transaction commits, which those databases satisfy, but nothing here proves it for them. - **Late rows.** A transaction that stays open while more than `LateRowWindow` later positions are logged (default 10,000), or longer than `LateRowAge` (default 30 s), is given up on. If it then commits, its change reaches the index with that entity's next change or `ReindexAsync`, not before. `LateRowAge` must exceed your longest transaction that writes indexed entities. - **Retention.** The applier loop prunes the log to `RetainRows` positions (default 100,000) but never into the newest `LateRowWindow` positions. A direct call to `Prune` or `PruneAsync` keeps exactly what it is told to, and a transaction open across such a call can lose its row; run manual pruning when writers are quiet. - **Bulk statements** (`ExecuteUpdate`, `ExecuteDelete`, raw SQL) bypass the interceptor and are repaired by `ReindexAsync`. - **Failures.** If the index cannot commit (disk full, permission), the applier records the error in `LastApplyError`, closes the index and opens it again at the next round, and applies the same log rows then. Before this was done the applier stayed stopped until the process restarted. - **One process per index directory.** A second application process that opens the same index directory fails with `IndexLockedException`; give each instance its own directory (they apply the same log independently). ## How it is tested All numbers below are for the code in this repository at the time of writing; each is re-run before a release. Seeds and commands are in the harness (`tests/Durability`, `tests/EfDurability`, `tools/ef_kill_campaign.sh`). | Check | What it does | Result | |---|---|---| | Fault sweep (`Durability sweep`) | A seeded workload (upserts, deletes, merges, commits, held snapshots, vectors) under the fault injector, with a power cut at every storage operation of every run, each of three crash policies (lose everything unsynced, tear, reorder) and several crash seeds, plus a failed operation at every index with and without a power cut. After each cut the storage must open, hold exactly an acknowledged commit state (never fewer commits than acknowledged), verify every file, give identical content and query results, accept a writer again and leave no stray files. | x64: 367,458 schedules, 0 failures. arm64: 198,030 schedules, 0 failures. The workload includes a clustered (IVF) vector field beside the HNSW one. | | Evolution sweep (`Durability evolve-sweep`) | The same under a schema change that rebuilds every segment in the background, cut at every operation of the rebuild and the swap. | x64: 208,978 schedules, 0 failures. arm64: 116,809, 0 failures. | | Kill harness (`Durability kill`) | A child process writes, merges and commits; the parent kills it with SIGKILL at random moments (including inside a commit call), reopens the directory and compares it with the commits the child acknowledged on a side channel: it must open, show every acknowledged commit, have the exact content of one of them, and pass a query check. | x64: 4,000 kills in four runs (seeds 31100 to 34100), 3,195 commits acknowledged, 2,362 kills inside a commit call, 0 failed opens, 0 lost commits, 0 wrong results. arm64: 2,100 kills in three runs, 3,581 commits, 1,029 inside a commit call, 0 failures. | | Kill harness, schema evolution (`kill --evolve true`) | The same while the child rebuilds segments after a schema change. | x64: 500 kills (210 commits, 151 inside a commit), 0 failures. arm64: 500 kills (759 commits, 172 inside), 0 failures. | | Reader processes (`Durability procs`) | One writer process and four reader processes; readers hold snapshots across the writer's commits and merges and verify them; the parent kills writers and readers at random, restarts them, starts second writers that must be refused, and checks the directory against acknowledged commits. | x64: 500 kills, 2,026 commits, 54,820 snapshot verifications, 100 second writers refused, 0 problems. arm64: 500 kills, 2,566 commits, 33,420 verifications, 100 refused, 0 problems. | | Sparse-vector kills | Kills during sparse-vector writes. | x64: 100 kills, 231 commits, 14,931 documents checked, 0 failures. arm64: 100 kills, 501 commits, 16,050 documents, 0 failures. | | Embedding kills | Kills during writes that embed text at upsert (the embedding cache is part of the stored source). | x64: 100 kills, 130 commits, 14,435 documents, 0 failures. arm64: 100 kills, 231 commits, 14,832 documents, 0 failures. | | Large flush, every cut (`PackSpillFaultTests`) | A flush whose parts exceed the 4 MiB pack limit (the spill path the sweeps' small segments never reach), cut at every storage operation under every crash policy. | 0 failures. | | Failed call, any point (`TransientFailureTests`) | One failed storage call at every operation of two commits: the writer either retries successfully with every document or refuses; an index reopened holds every acknowledged commit and no stray file. | 0 failures. | | EF kill campaign, SQLite (`tools/ef_kill_campaign.sh`) | Per cycle two application processes (each with its own index directory) apply inserts, updates and deletes to one database while an applier (inline or threaded) runs; each is killed at a random time or at a named stage of the applier (read, before and after commit, rebuild steps); sometimes the log is pruned past the indexes. Each index is then reopened, caught up and compared with the database row by row, and its position is compared with the positions it acknowledged. | 1,276 kills (400 at named stages of the applier, 876 at random times; four lanes of 150 cycles, seeds 81000 to 81003), 472 index checks, 209 rebuilds after retention, 0 first-pass mismatches, 0 failed opens, 0 lost commits, 0 differences from the database. | | EF kill campaign, PostgreSQL 16.2 | The same with two concurrent writers per process and transactions held open up to 40 ms, so later positions become visible first (a user-local server from the `pgserver` package, no container). Two settings: retention below the late-row window, and retention and window both 1,000. | 1,809 kills (the two settings: 926 and 883; seeds 82000 to 82003 and 83000 to 83003), 654 index checks, 331 rebuilds, 3,433 commits made while positions were still missing, 372 rows applied because they became visible below the mark, 0 first-pass mismatches, 0 failed opens, 0 lost commits, 0 differences. An earlier run of the same campaign, before it waited for a killed client's last transaction to end before comparing, reported one mismatch of three rows (one save) in about 8,000 kills (seed 9203, cycle 23); 6,900 further kills of that version did not repeat it. The mismatch is consistent with the comparison racing a killed client's final commit, but that was not proven. | | `.Files` resume campaign (`oneindex-files-tool campaign`) | A generated corpus of 6,191 files (text, Markdown, HTML, PDF, archives, corrupt files); the indexer process is killed 100 times at random moments and resumed; the final index is compared with an uninterrupted run. | 100 kills delivered (0.3 to 2.5 s after each start), every reopen succeeded, the file-record count never fell, 0 differences between the resumed and the clean index (12,844 file records, 61 error documents for the corrupt files, 581,678 passages), 0 duplicate records, 0 orphan passages. | The sweeps run on a volatile in-memory disk behind the fault injector, so they test the engine's protocol, not the host file system; the kill harness, the reader-process campaign and the EF and Files campaigns use real files on real disks (x64 Linux, arm64 macOS APFS). Kills are process kills, which are covered by the guarantee in full; power cuts are not exercised on real hardware. The Entity Framework and `.Files` campaigns ran on x64 Linux only. ## Reporting a problem A `CorruptIndexException` names the file. Keep the directory, the exception text and the process history (was a process killed, was a disk full) and open an issue; do not delete files by hand. # One writer, many reader processes An index directory can be open in one writer and in any number of readers at once, in different processes. This note says how, what each operating system gives, and what is promised. ## The promise - **One writer.** A second `IndexWriter.Open` on the same index, from this process or another, fails at once with `IndexLockedException` (an `IOException`; `LockName` is `write.lock`). It does not wait. When the first writer is disposed, or its process exits or is killed, the next `Open` succeeds immediately. - **Readers see commits only.** `IndexSnapshot.Open(storage)` and `IndexReader.Open(storage)` show one complete commit: the newest one when they looked. Nothing a writer has not committed is visible, and a commit is visible whole or not at all (the atomic rename of docs/commit-format.md). - **A reader keeps its commit.** While a snapshot is open, in any process, the writer does not delete a file of that commit however many commits and merges it makes. When the snapshot is disposed, or the reader's process dies, the files become garbage and are deleted by whoever looks next. - **Refresh is cheap.** `IndexReader.Refresh()` lists the directory (one call). Only if a newer commit exists does it open it, pinning the new commit before letting go of the old one. - **Read-only storage** (HTTP range, `ReadOnlyStorage`, `new FileSystemStorage(dir, readOnly: true)`) cannot lock, so it cannot pin. It still reads, and needs no lock because it cannot write; opening a commit while a writer deletes files can fail with `CorruptIndexException` naming a vanished file, and the caller opens again. ## Files that coordinate | File | Meaning | |---|---| | `write.lock` | Empty. Held with an exclusive lock for the life of a writer. Stays on disk. | | `reader--.lease` | Empty. Held with an exclusive lock for the life of one open snapshot of that generation. Deleted when the snapshot is disposed. | | `viewer--.lease` | Empty. Held by the writer for each open near-real-time view. Pins that generation (1 before the first commit) and every later commit, because the view may read a segment that only a later commit lists and a merge may drop it from the one after. | All are created through `IStorage.TryLock(name)`, so the fault-injecting wrapper sees them and a power cut can lose or keep them like any file. They carry no data, so there is nothing to corrupt; a stale one is only a name. `TryLock` creates the file if needed and takes an exclusive, non-blocking lock tied to the open handle. It returns `null` if somebody else holds it. The lock goes away when the handle is disposed or the process ends. ## Pinning protocol Reader, to pin the newest commit: 1. List the directory; let *G* be the newest generation. 2. Create and lock a lease for *G*. If the lock is refused, start over (a new random name). 3. List again. If *G* is still the newest and the lease file is still listed, the commit is pinned. Otherwise drop the lease and start over with the newer generation. 4. Open commit *G* by name and verify its files. Writer, before it deletes any file that no commit and no in-process snapshot needs any more (after a commit, when a merge's sources are released, at `Open`, at `Dispose`, and at `ReclaimFiles()`): 1. List the leases. For each, try to take its lock. If that works, the holder is gone: delete the lease. If it is refused, the holder is alive. 2. Parse the commit of every live lease's generation; its commit file and every file it lists are not deleted now but remembered, and deleted by a later pass when no lease pins them. Why this is safe: the writer publishes commit *G*+1 (rename) before it scans leases for the first time after that. A reader whose step 3 saw *G* still newest ran before the rename, so its lease existed before the scan and the scan sees it. A reader whose step 3 ran after the rename sees *G*+1 and starts over. The lease is locked before it is checked, so there is no moment where a commit is relied on without a visible lease. A reader that holds nothing yet can only ever be handed files of the newest commit, which the writer never deletes. One race needs care on Unix: a reader that created a lease file but has not locked it yet could be found "dead" by a scanner. The scanner takes the lock itself, deletes the file while still holding it, and only then lets go; the reader's own lock attempt then either fails (it starts over) or finds its file gone in step 3 (it starts over). On Windows an open file cannot be deleted, but creation and lock are one atomic call there. ## Who reclaims - The **writer** reclaims at every commit, at `Dispose` and on `ReclaimFiles()`, and `Open` removes everything no commit lists except what a live lease pins. - A **reader** that disposes its last snapshot runs `IndexMaintenance.CollectGarbage`, which deletes the files of superseded commits that no live lease pins. It never touches a file that no commit lists, because those may be a running writer's work in progress. So a directory whose writer has gone does not stay full of old commits after its readers leave; files an interrupted writer left behind go at the next writer's `Open`. - A **killed reader** leaves a lease nobody holds. The next scan, by any writer or reader, deletes it. ## What each operating system gives `FileSystemStorage.TryLock` opens the file with `FileShare.None`. What .NET does with that: | OS | Mechanism | Observed | |---|---|---| | Linux | The runtime takes `flock(LOCK_EX \| LOCK_NB)` on the open file. Advisory, owned by the open file description, so a second open in the same process conflicts too. Released by the kernel when the descriptor closes or the process dies. A refusal arrives as an `IOException`. | Verified here: same-process and cross-process refusal, release on `kill -9`. Switching it off with `DOTNET_SYSTEM_IO_DISABLEFILELOCKING` makes it a silent no-op; `TryLock` detects that (a second exclusive open succeeds) and throws `NotSupportedException` rather than let two writers think they are alone. | | macOS | The same `flock` path in the runtime. | Verified by the multi-process campaign on an arm64 Mac (see Tests). | | Windows | `FileShare.None` is a kernel share mode: the second open fails with a sharing violation, and the handle is closed when the process ends. Mandatory rather than advisory. Every handle the storage opens for a data file allows sharing delete, so the writer can delete a file a reader still has open; the name vanishes at once, so later opens of it fail, which is why the lease is needed there too. | Run on every change in CI (Windows Server, NTFS): the storage, lock, lease and sample tests pass. The multi-process and kill campaigns have not been run on Windows. | Limits worth knowing: - Network and FUSE file systems may not implement `flock` (or implement it per machine); two machines sharing one directory are not supported. The lock is a same-machine mechanism. - The lock is cooperative: a program that ignores `TryLock` can still write the files. - A process that is frozen but alive keeps its lease; the files it pins stay until it exits. - Leases are plain file creates and deletes; on a read-only file system a pinned open cannot be made (use a read-only storage). ## Near-real-time reader `IndexWriter.OpenNearRealTimeReader()` returns an `IndexReader` over the writer's own state: flushed segments plus its deletes, committed or not. `Refresh()` flushes the buffer into a new segment and shows it; it reopens only if something changed. Rules: - It is **not durable**. `IsDurable` is `false` on the reader and on every snapshot it hands out, and `Generation` is the last commit the view extends (0 before the first). A crash, or `Dispose` without `Commit`, discards everything it showed. - It exists only in the writer's process. Other processes read commits. - Its segment files are pinned by the writer's file counts and by a viewer lease, so neither a merge installed by a commit nor a reader process tidying up superseded commits deletes files a view still reads. Old commits therefore stay on disk until the view is refreshed or disposed. - It is a copy of the delete state at the moment of the refresh: later deletes do not change a view already handed out. - Each refresh that finds buffered documents writes a small segment; the merge policy folds them at the next commit. Refresh at the rate freshness needs, not more. - Dispose NRT readers before the writer. ## Tests - `tests/OneIndex.Tests/ProcessCoordinationTests.cs`: second writer refused (directory and memory), lock released on dispose and on a failed open, lease keeps files until released, stale lease removed, refresh, read-only open, NRT against committed reads and isolation, NRT not durable. - `tests/Durability`, command `procs`: one writer process and N reader processes. Readers open, hold, refresh and verify snapshots, comparing each generation's content with the hash the writer recorded before committing, and re-verify every file after the writer committed past them. The parent kills writers and readers at random, restarts them, checks each dead writer's directory against what it acknowledged, periodically starts a second writer that must be refused, and at the end checks that a new writer opens and that no file is left over. `sweep` also opens a writer snapshot and a reader snapshot after every commit, so power cuts and failed operations land on lease files too. # Storage interface `IStorage` is a flat set of named files. It is the only way the engine touches bytes, so one implementation can be local files, memory, read-only HTTP ranges or browser storage, and a wrapper can see (and break) every write. ## Names A name is 1 to 255 characters from `A-Z a-z 0-9 . _ -` and does not start with a dot. There are no directories. A name can therefore never leave the storage root, and it is safe in a URL. ## Reading `OpenRead(name)` returns an `IRangeReader` with a fixed `Length`. `Read(offset, buffer)` fills the caller's buffer and returns the bytes read; it returns fewer than requested only when the range reaches the end of the file, and 0 at or past the end. `ReadExactly` turns a short read into `CorruptIndexException`. A reader is safe to use from several threads. The local-file backend serves files of 64 KiB or more on Unix from a read-only memory mapping, copying the requested range into the caller's buffer (no system call on a cached range); smaller files, Windows (where a mapped file could not be deleted by a merge), the browser, and any file that cannot be mapped use positional reads. On Windows readers use ordinary, not overlapped, handles: a synchronous read on an overlapped handle allocates about 176 bytes per call there (measured), which would break the allocation-free hot path, so the async read completes inline. Writers allow sharing delete too, so a file a crashed or abandoned writer left open can still be removed. Every read still goes through `IRangeReader`, so the fault injector and the transform wrappers see the same calls, and the engine never holds a pointer into a mapping. Index files are immutable, so a mapping never sees a file change; a range past the length seen at open reads short like any other. The hot path (a cached range) neither awaits nor allocates; the async form exists for backends that really wait (HTTP). ## Writing `Create(name)` makes a new empty file (it fails if the name exists) and returns an append-only `IStorageWriter`. `Sync()` flushes and fsyncs the file; bytes are durable only after it returns. Disposing a writer without syncing hands bytes to the operating system but promises nothing about a crash. A storage that reports `SupportsConcurrentSync` (the local-file backend, except in the browser) accepts `Sync()` calls on writers of different files from different threads at the same time; the index writer uses this to flush all files of a new segment together, because on a disk with an expensive cache flush the concurrent fsyncs complete in far less than the sum of their times. Storages that do not report it are synced one file after the other, with the same result. `Rename(source, destination)` atomically replaces the destination, also when a reader has the destination open (on Windows through `ReplaceFile`, which swaps the names when the other handle allows sharing delete, as every handle of this storage does). `Delete(name)` ignores a missing file. `List()` returns names in ordinal order. `SyncDirectory()` makes creations, renames and deletions durable. `TryLock(name)` (where `SupportsLocks` is true: local files and memory) creates the named file if needed and takes an exclusive lock on it, or returns `null` if another holder has it. The engine uses it for the writer lock and reader leases; see docs/processes.md for the per-OS behaviour. The fault injector counts a lock on a file that did not exist as a `Create`, and a power cut releases every lock it handed out. A read-only backend reports `IsReadOnly` and throws `NotSupportedException` from every mutating member. ## What durable means, by platform The commit protocol needs two promises: a synced file's bytes survive a crash, and, after `SyncDirectory`, so do the names created or renamed before it. | Platform | File data (`Sync`) | Names (`SyncDirectory`) | |---|---|---| | Linux (ext4, XFS, btrfs) | `fsync` on the file | .NET cannot open a directory to fsync it, so each file renamed into place, or created and not yet synced, since the last call is fsynced. A file that was synced after it was created is not synced a second time: that sync already committed its directory entry. In practice these file systems commit the directory entry in the same journal transaction, which makes the entry durable (not a POSIX guarantee; see below). Deletions are not forced, which is harmless: a deleted file that returns after a crash is garbage no commit names. | | macOS | full sync of the file (`F_FULLFSYNC`, as .NET does for `FlushToDisk`) | the same file flushes | | Windows (NTFS) | `FlushFileBuffers` | the same file flushes; NTFS journals metadata | | Memory | nothing is ever durable beyond the process | no effect | ### Why `SyncDirectory` does not fsync the directory .NET has no managed way to open a directory for syncing. On Linux (.NET 10.0.11) every attempt throws `UnauthorizedAccessException`: `File.OpenHandle` and `FileStream` on the directory path, on `/.`, and on `/proc/self/cwd`, read-only or read-write. A `SafeFileHandle` can only wrap a descriptor that something else opened, and getting one needs `open(2)`, which means native interop; spawning `sync` as a child process is possible but absent in sandboxes and in the browser, so the engine does not do it. (Opening a directory is not possible with `FileStream` on Windows either; this was not tested here.) The BCL offers no other durable-directory call, so the engine does not claim one. What the engine does instead, and what that guarantees: - **Process kill, crash of the application, `kill -9`:** fully guaranteed on every platform. The operating system has every completed write and rename, whatever the disk does later. The kill harness exercises exactly this. - **Power loss or kernel crash, Linux:** POSIX asks for a directory fsync to make a created or renamed name durable. Without one, durability rests on how the file system journals: on ext4, XFS and btrfs, fsync of a file also commits the journal or log entry that created or renamed it, so syncing each new file, then the renamed commit file, after the rename makes the name durable in practice. That is a property of those implementations, not of POSIX; it was not tested by pulling power, and a file system that behaves differently (some network and FUSE file systems) can lose a rename it acknowledged. The loss is a rollback to the previous commit, never a corrupt index: the commit protocol only ever lists files that were synced and verified, and a reader checks every listed file's length and checksum. - **Power loss, macOS and Windows:** `FlushToDisk` (`F_FULLFSYNC` / `FlushFileBuffers`) of the file, and the same reliance on the file system's metadata journal. Nothing here was tested with a power cut; the worst case is the same rollback. - **Memory:** nothing outlives the process. The fault injector models the strict case (a rename survives only after `SyncDirectory`), and the engine passes it, so an application that can run a directory fsync of its own (for instance from a supervising process) can add it without changing the engine. ## The fault-injecting wrapper `FaultInjectingStorage` wraps a writable storage and models a disk with a volatile cache. It sees every `Create`, `Write`, `Sync`, `Rename`, `Delete` and `SyncDirectory` and numbers them. Its model: - A file's bytes become durable at that file's `Sync`. - A name (creation, rename, deletion) becomes durable at `SyncDirectory`. - A rename is atomic: it survives whole or not at all. A seeded schedule can fail chosen operations (`IOException`, no effect) and can cut the power at a chosen operation (`SimulatedCrashException`; a write that is cut is recorded first, so it may survive torn). `Crash(policy)` then rewrites the wrapped storage to what a real disk may hold: | Policy | Unsynced writes | Unsynced name changes | |---|---|---| | `LoseUnsynced` | all lost | all lost | | `Tear` | an ordered prefix survives, the write at the cut is torn | an ordered prefix survives | | `Reorder` | each survives whole, torn or not at all, so a later write can persist without an earlier one (a zero-filled gap) | each survives or not independently | The same seed gives the same result. After `Crash` the wrapper is usable again with its counter reset. ## Block transform `TransformingStorage` applies an `IBlockTransform` to every byte by file position: `Encode(offset, data)` on write and `Decode(offset, data)` on read, in place and length-preserving. The identity transform changes nothing. The slot is where encryption at rest will go; the transform must not depend on a file's name because files are renamed into place. ## Read-only opens, static chunks and OPFS `SearchIndex.OpenReadOnly(path)` opens the newest commit of an index directory without the writer lock: it works while another process writes, on read-only storage, and `Upsert`, `Delete` and `Commit` throw `NotSupportedException`. A read-only storage reports `IsReadOnly`. `ChunkedStaticStorage` ([static-index.md](https://recursiveai.org/oneindex/docs/static-index/)) is a read-only `IStorage` over fetched chunks. In the browser, `OpfsPersistence` saves and loads an index held in `MemoryStorage` whole through an application-supplied `IOpfsHost` ([browser.md](https://recursiveai.org/oneindex/docs/browser/)); the fault-injecting wrapper sits in front of the same storage interface, and the OPFS save was checked by crashing a fake host at every change index until the save completed (each state loaded and opened as the old or the new commit). ## Limits and exceptions `IndexLimits` bounds every count and length a reader accepts (commit size, segments, files per segment, string length, schema size, fields, vector dimensions). `CorruptIndexException` names the damaged file; `UnsupportedFormatException` (a subclass) means the file is intact but newer than this build understands. # Index files, commit file and commit protocol All multi-byte integers are little-endian. A *varint* is an unsigned LEB128 integer in its shortest form (a reader rejects a trailing zero group and values past 64 bits). A *string* is a varint byte length and that many bytes of UTF-8 (invalid UTF-8 is rejected). *Bytes* are a varint length and the bytes. ## The index file frame Every index file has the same frame: offset 0 u32 magic file-type tag, four ASCII characters (commit file: "OIXC") offset 4 u32 version format version of this file type, at least 1 offset 8 ... body offset length-12 u32 footer magic 0x4658494F ("OIXF") offset length-8 u32 algorithm 1 = CRC-32C offset length-4 u32 checksum CRC-32C of every byte before this field CRC-32C is the Castagnoli polynomial (reflected 0x82F63B78), initial value and final xor all ones; the check value of "123456789" is `E3069283`. A reader checks, in this order: the file is at least 20 bytes, the magic, the footer magic, the algorithm, the checksum, then the version (so a file that is corrupt is reported as corrupt, and only an intact file from a newer writer as unsupported). The footer checksum doubles as the file's identity: a commit records it for each file it lists, so a file replaced by different bytes of the same length is noticed from 12 bytes at the end. ## Commit file, versions 1, 2 and 3 A commit is written as version 1 unless the index has been through a schema change that needed its segments rebuilt (see schema-evolution.md), in which case it is written as version 2, which adds the epoch fields and the pending-rebuild section marked below. A commit that lists a part stored inside a pack file (below) is written as version 3: version 2's layout, always with the epoch fields and the pending-rebuild flag, plus one varint after each file's checksum. All three versions are read. A version 2 commit whose epochs are all 0 and that has no pending rebuild is refused: version 1 is its canonical form. A version 3 commit that lists no file at a nonzero offset is refused for the same reason. File name `commit-.oic`. Body: u64 generation at least 1; must equal the generation in the file name u64 next segment id the number the writer names the next segment with bytes applied position opaque, at most 1,024 bytes; the application's marker (for example how far an external change log has been applied) bytes schema the serialized schema (below), at most 4 MiB; empty if none varint user data count at most 1,024 string key, string value keys strictly ascending by ordinal byte order varint schema epoch version 2 only: 0 until a schema change needed segments rebuilt, then one more per such change varint segment count at most 100,000 string segment name varint document count including deleted documents varint deleted count at most the document count varint delete generation 0 when the segment has no delete bitmap varint file count at most 1,024 string part "terms", "postings", "lengths", "columns", "points", "stored", "deletes", "ids" string file name a valid storage name; no file is listed twice in a commit (a pack file is listed once per part it holds, each at its own offset, and never also as a plain file) string codec name varint codec version varint file length u32 file checksum the footer checksum of that file varint offset version 3 only: 0 for a file that holds the part alone; otherwise the part is inside the pack file named above, and its framed bytes (header to footer, `file length` of them) start at this offset, which is at least 8 varint schema epoch version 2 only: the epoch the segment was written under; at most the commit's Version 2 then ends with the pending-rebuild section (schema-evolution.md): varint pending flag 0 or 1 (when 1) bytes pending schema the schema served after the swap varint rebuilt count may be 0 segment record as above, with an epoch of exactly the commit's plus one string source the name of a segment listed above that it was rebuilt from; each source at most once varint source deleted the source's deleted count when it was rebuilt The segments listed first are the served ones; rebuilt segments are never read until a later commit makes them the served set. Segment names are unique within a commit (rebuilt segments and all file names included). A parsed commit that is serialized again gives the same bytes: the format is canonical, which the fuzz target checks. ### Pack files Writing a small segment as seven files costs seven creates and seven fsyncs, which on some file systems (macOS most of all) dominates a small commit. A flush therefore collects the text, column, point, stored and identifier parts in memory and writes them as one *pack file* named `.pack`: the frame of this document (magic `OIXK`, version 1, footer checksum over the whole file) around the parts' complete files laid end to end. Each part keeps its own header, footer and checksum, so everything that checks a part (the commit's recorded length and checksum, `Verify`) checks it in place through a window over the pack; a pack that ends before a listed part does is corrupt and named. Vector parts and delete files are never packed. If the parts held back exceed 4 MiB, the part that crosses it is written to a file of its own as before, and a flush with a single part left writes it as a plain file. Merges write plain files. Packs are immutable like every segment file and are deleted with their segment. ### Codec registry The codecs this build reads are a fixed list in `CodecRegistry`, each with the newest version it understands. A commit that names an unknown codec, or a newer version than listed, is refused as `UnsupportedFormatException`. The codec's owner raises the number when its on-disk format changes; reading older versions stays supported until a release says otherwise. ## Schema, versions 1 and 2 4 bytes "OIXS" u32 version (1; 2 when the schema holds fields derived from JSON paths or text settings) varint field count at most 4,096 string name u8 type 1 text, 2 keyword, 3 integral, 4 fractional, 5 date, 6 bool, 7 geo point, 8 bytes, 9 vector, 10 json u8 flags 1 indexed, 2 stored, 4 column, 8 facet, 16 multi-valued u32 boost IEEE 754 single bits string analyzer empty when none u8 index options 0 none, 1 docs, 2 freqs, 3 positions, 4 offsets varint dimensions vector fields only string identifier field name, empty when none version 2 only: varint section count varint section id strictly ascending; 1 = derived fields, 2 = text settings bytes section body Sections with ids this build does not define are skipped on reading (their length is checked and they are dropped when the schema is written again). Section 1 lists the fields derived from the paths of JSON fields (dynamic.md): a varint count of at least 1, then for each, in ascending order, the varint field id of the derived field and the varint field id of the JSON field it belongs to (always an earlier field). Section 2 holds the text settings (query-time; they never change what is indexed): u8 flags 1 typo tolerance off, 2 no typos on numbers, 4 prefix search off varint word size for one typo, varint word size for two typos 0 to 255, one <= two list words that never get typos list = varint count, then strings list fields without typos list stop words varint synonym rule count list from (a group's members, or the one-way word) list to (empty for a group) A schema whose text settings are all defaults and that has no derived fields is written as version 1, byte for byte as before. Opening a writer with a schema that has the committed fields but different text settings is allowed: the new settings are used at once and the next commit records them; they never trigger a rebuild. Fields keep declaration order; a field's position is its field id. The reader runs the same validation as the schema builder (unique names, types and options that make sense together, a stored single-valued keyword for the identifier). ## Commit protocol A commit makes a set of already-written segment files visible: 1. The caller writes each new segment file with `IndexFileWriter`, which appends the footer and either fsyncs the file or leaves it open to be synced together with the rest of the segment, and records its length and checksum. 2. `CommitProtocol.Commit` serializes the commit, then checks that the generation is above the newest commit in storage and that every file the commit lists exists with the recorded length and footer checksum. 3. It verifies the files that no earlier commit listed (segment files never change, so files an earlier commit verified are not opened again), then calls `SyncDirectory` so the names of the new files are durable before any commit refers to them. 4. It deletes any leftover temporary commit file, writes the commit to `commit-.tmp`, and fsyncs it. 5. It renames the temporary file to `commit-.oic`, atomically. 6. It calls `SyncDirectory` again. Only now does `Commit` return: a commit that was acknowledged is durable. A crash before step 5 leaves the previous commit as the newest. A crash after it leaves the new one (or, if the rename was not yet durable, the previous one; the caller had not been told yet). No instant in between is observable by a reader, because a reader only ever lists `*.oic` files and the rename is atomic. `OpenLatest` finds the highest generation among `commit-*.oic` names, parses it, checks that the name and the recorded generation agree, and checks that every listed file exists with the recorded length and footer checksum. It returns `null` for storage that holds no commit. If the newest commit or a file it lists is damaged it throws `CorruptIndexException` naming that file; it never falls back to an older commit silently, because that would drop acknowledged data without a word. `TryOpenLatest` reports the same as a return value. `VerifyFiles` additionally reads every byte of every listed file and checks the checksums. Older commit files and the segment files only they list are removed by the writer, after no reader snapshot refers to them (reference counts per file; see `segments.md`). The `ids` and `deletes` parts and the naming of segment files are defined there. Leftover `*.tmp` files are ignored by readers and deleted by the next commit of the same generation or by cleanup. ## How this is tested A four-commit workload runs under the fault injector with a power cut at every storage operation (77 positions, from before the first operation to after the last), each of the three crash policies and eight seeds: 1,848 schedules. After each, the wrapped storage must open without error, must hold every acknowledged commit or a newer one, and must pass `VerifyFiles`. The same suite run against a protocol with the syncs removed finds violations, which shows the schedules are able to catch a broken protocol. # Segments, deletes, snapshots and merges This note is normative for the index lifecycle. File frames, the commit file and the commit protocol are in `commit-format.md`; the storage interface is in `storage.md`. ## The segment A segment is an immutable set of files that holds a range of documents numbered `0 … docCount-1`. Its name is `s` plus 16 lowercase hex digits (`s0000000000000007`); names are allocated from the commit's *next segment id* counter and only ever grow, so name order is creation order. A file is named `.` (`s0000000000000007.ids`); a delete file is `.del`. Parts: | Part | Owner | Contents | |---|---|---| | `ids` | the index | document identifier to document id (below). Present when the schema has an identifier field | | `deletes` | the index | the deleted documents of one generation (below) | | `terms`, `postings`, `lengths`, `columns`, `points`, `stored` | codecs | defined by the codec | A small flush writes most of these parts into one `.pack` file instead (see "Pack files" in `commit-format.md`); which parts share it is recorded in the commit, and readers see no difference. Every file uses the frame of `commit-format.md` (magic, version, checksummed footer). The commit records each file's length, footer checksum, part and codec version, so a file replaced after the commit is noticed when the index opens. Document ids are dense and local to a segment. Deleting a document does not renumber anything; a merge renumbers. ## The codec interface A codec is an `ISegmentCodec` registered explicitly in `IndexWriterOptions.Codecs`. It owns one or more *parts*, each listed in `CodecRegistry` (the registry number is the format version a file of that part is written with, and a codec owner raises it when the format changes). A part is owned by exactly one codec; `ids` and `deletes` belong to the index. ```csharp public interface ISegmentCodec { string Name { get; } IReadOnlyList Parts { get; } void Flush(SegmentFlushContext context, SegmentFileSink sink, CancellationToken ct); void Merge(SegmentMergeContext context, SegmentFileSink sink, CancellationToken ct); ISegmentPartReader? OpenReader(SegmentReaderContext context, CancellationToken ct); } ``` **Flush.** `SegmentFlushContext` carries the schema, the segment name and `Documents`: the live buffered documents in order. Document `i` of the list becomes document id `i`. Documents that were replaced or deleted while still buffered are already removed, so a new segment has no deletes. The codec writes the files for the fields it handles (it reads the field definitions from `Schema`) through the `SegmentFileSink`: `sink.Create(part, magicTag, version)` returns a `SegmentFileBuilder` with `Write(span)` and `Complete()`. `Complete` writes the footer, fsyncs the file and records it for the commit. A codec with nothing to write for the schema writes nothing. If `Flush` throws, the writer deletes every file the sink created. **Merge.** `SegmentMergeContext` carries the schema, the new segment name, the total live document count and `Sources`, one `MergeSource` per source segment in merge order. A source has `Info`, `MaxDoc`, this codec's own `Reader` for that segment (an `ISegmentPartReader`, or null if the segment had none of the codec's parts), `Live` (the source's deleted set as of the merge start) and `DocMap`: `DocMap[old]` is the new document id or `-1` for a deleted document. Live documents are numbered consecutively in source order, `DocMap.Base` being the first new id of a source and `DocMap.LiveCount` the number it contributes. The codec writes the same parts it would write on flush, covering exactly the live documents. Documents deleted after the merge began are still copied; the writer deletes them again in the result. **Read.** `OpenReader` receives a `SegmentReaderContext` (storage, schema, `SegmentInfo`, limits; `OpenPart(part, out file)` opens a part's file after checking its length against the commit). It returns the codec's view of the segment or null. A part reader may implement any of the reader contracts of the layers above (`ITextFieldReader`, `IColumnReader`, `IStoredFieldsReader` …) and `SegmentReader.GetPart()` finds it by type. Readers are used from several threads and must be thread-safe. A codec holds no per-segment state; flush and merge may run on a background thread. **Keyword lookup.** A part reader may implement `ITermDocLookup` so that `IndexWriter.DeleteByTerm(field, value)` reaches committed segments. Deleting by the identifier field never needs it (the `ids` part serves it). **Rules for codecs.** All bytes written must be deterministic for the same input (no hash-order iteration, no timestamps). Readers validate every offset and length before trusting it and raise `CorruptIndexException` naming the file; they honour the `IndexLimits` in the context. The sink calls are synchronous. ## The `ids` part, `OIXI` version 1 A sorted map from identifier to document id, so the index can replace, delete and find a document by identifier with no other codec: `u32 count`, a table of `count` entries `(u32 dataOffset, u32 docId)` sorted by the identifier's UTF-8 bytes (ordinal, strictly ascending), then the identifiers as varint length plus bytes, back to back in table order. A lookup is a binary search with range reads; nothing is held in memory and a lookup of an identifier up to 256 UTF-8 bytes allocates nothing. Identifiers are unique among the live documents of the index (`Upsert` replaces); a segment never holds two entries for one identifier, because a document replaced while still buffered is dropped before the flush. ## The delete file, `OIXD` version 1 Body: `u32 maxDoc`, `u32 deletedCount`, `u8 encoding`, then encoding 0, a bitmap of `ceil(maxDoc/64)` little-endian `u64` words, or encoding 1, the sorted deleted ids as varints (the first id, then each gap minus one). The writer picks the smaller, the bitmap on a tie. A reader rejects: a count that disagrees with the ids or with the commit, bits or ids at or past `maxDoc`, ids out of order, trailing bytes, an unknown encoding. A sparse delete file is held as a sorted array, so a tiny file can never force a large allocation. A segment's deletes are versioned by the *delete generation* in the commit: generation 0 means no delete file; a commit that adds deletes writes `.del` and lists it in place of generation `g`, whose file stays until the previous commit is unreferenced. A delete file is never modified. ## Writer lifecycle `IndexWriter` is the single writer of a storage. Calls take turns (a lock), so it is safe from several threads. 1. **Buffer.** `Upsert(document)` validates against the schema, deletes the live document with the same identifier (in the buffer or in a segment) and buffers the new one. `Add` is the same for schemas without an identifier. At `MaxBufferedDocuments` the buffer flushes. 2. **Flush.** The buffer becomes a new segment: every codec's `Flush`, then `ids`. The files of the segment are completed without a sync and then fsynced together (concurrently where the storage allows it) before the flush returns; a merge does the same. Nothing refers to them yet. 3. **Delete.** `DeleteById` and `DeleteByTerm` mark documents in the buffer or set bits in the segment's in-memory deleted set. Delete files are written at commit. 4. **Merge.** `MaybeMerge` (run after every commit by default) asks the `TieredMergePolicy` for groups and queues them on the `IMergeScheduler`. A merge reads its sources and writes a new segment; it changes nothing visible. A finished merge is **installed by the next commit**: the sources are replaced by the result, and deletes made to a source after the merge began are re-applied to the result through the document map. 5. **Commit.** Install finished merges, flush, drop segments whose documents are all deleted, write delete files for changed segments, build the commit (generation + 1) and run the commit protocol of `commit-format.md`. Then the commit's files are pinned, the previous commit's pin and the writer's own pending pins are released; files whose last reference went are deleted. A commit with no change writes nothing. If the commit protocol throws, the writer cannot know whether the commit became durable and refuses further work; opening the index again tells. ### Tiered merge policy Segments are grouped into tiers by live document count: tier 0 holds segments below `FloorSegmentDocs × SegmentsPerTier` (1,000 × 10), and each next tier's limit is `SegmentsPerTier` times larger. A tier with at least `SegmentsPerTier` (10) segments merges its oldest `MaxMergeAtOnce` (10) segments, never producing more than `MaxMergedSegmentDocs` (5,000,000) live documents. A segment more than `DeletesPctAllowed` (33) percent deleted is rewritten alone. The plan is a pure function of the segment list. **Size ratio.** The segments the tiers leave alone are also consolidated from the small end: sort them by live documents (name order breaks ties), count sizes below `FloorSegmentDocs` as the floor, and merge the largest group of the `k` smallest (2 to `MaxMergeAtOnce`) whose `k`-th member is at most `SizeRatio` (2) times the combined size of the smaller ones, within `MaxMergedSegmentDocs`. Segment sizes then fall off geometrically, so an index of any size keeps a handful of segments. Why: every query pays a fixed cost per segment, and fuzzy, typo and prefix expansion walk the top of each segment's term dictionary again, so on 150,000 abstracts six segments answered a two-edit fuzzy query in 2.2 ms and one segment in 1.4 ms. The price is rewriting: with flushes of 20,000 documents, 302,567 documents are written about 3.8 times in all (1.7 with `SizeRatio = 0`, the plain tiered policy), growing by about one write per doubling. On a bulk load the extra merges run on the merge scheduler, not on the thread that adds documents. ### Scheduler seam and throttle `IMergeScheduler` runs queued work. `CooperativeMergeScheduler` (the default; no threads) runs work on the caller during `Poll` (the start of every commit) and `WaitForIdle`; `ThreadedMergeScheduler` runs it on one background thread. Merge output is a function of the sources, the codecs and the new segment name only, and the name is allocated when the merge is queued, so for the same sequence of operations the segment bytes are identical under both schedulers when the caller waits for merges at the same points (`WaitForMerges`). Without waiting, a threaded merge may finish before or after a given commit, so the layout then depends on timing, and the contents never do. `MaxMergeBytesPerSecond` makes a threaded merge pause (through the scheduler) so that its output does not exceed the rate; a cooperative scheduler cannot pause and ignores it. ## Snapshots and file lifetime Every file has a reference count. References are held by: the newest commit (its commit file and every file it lists), each open `IndexSnapshot`, each running merge (its sources) and the writer for files written since the last commit. A file is deleted when its count reaches zero, never before. So a snapshot of commit *N* keeps reading correctly while the writer commits and merges past it, and the files are released when the snapshot is disposed. `IndexSnapshot.Open(storage)` opens the newest commit from any process, with or without a writer. Where the storage can lock, it also takes a reader lease that tells a writer in another process to keep that commit's files (docs/processes.md); the writer defers deleting any file a live lease pins and deletes it once the lease is gone, including a lease whose process was killed. `IndexReader` keeps a pinned snapshot and refreshes it to the newest commit. ## Recovery `IndexWriter.Open` reads the newest commit with `CommitProtocol.OpenLatest` (raising `CorruptIndexException` for a damaged commit; there is no fallback to an older one) and deletes every file that matches the engine's names (`commit-*.oic`, `commit-*.tmp`, `s<16 hex>.*`) and is not listed by the newest commit, except files that a live reader lease in another process still pins. Files with other names are left alone. `Open` first takes `write.lock`; a second writer fails with `IndexLockedException`. After a crash at any instant the last acknowledged commit, or a commit the caller had not yet been told about, opens and is complete. ## What is tested - `tests/Durability` runs the writer under the fault injector with a power cut at every storage operation of a workload (flushes, commits, delete-generation writes, merges), for each crash policy and several seeds, and checks that the reopened index equals an acknowledged commit exactly. It also fails every operation in turn. - The multi-process campaign (`Durability procs`) runs a writer process and several reader processes against one directory under random kills of both (docs/processes.md). - The kill harness runs a writer in a child process that is killed at random points; the parent reopens the directory and compares it with the commits the child acknowledged on a side channel. # Schema evolution An index is opened with the schema the application declares (for the happy path, the attributes of the document type). When the index already holds a schema, the two are compared at open and the index follows the application: no migration step, no separate tool. ## What happens at open `IndexWriter.Open` (and so `SearchIndex.Open`) compares the declared schema with the committed one, field by field: | Change | Result | |---|---| | A new field | Takes effect at once. Existing documents have no value for it (their stored source never held one), so there is nothing to rebuild. | | A removed field | The field stops being used. Its data stays in old segments until they are merged or rebuilt away. | | `Indexed`, `Column` or `Facet` turned on for a field; a different analyzer or index options for a text field; `Multi` turned on; keyword to text or back; integer to floating point | The index is **rebuilt in the background** from stored source (below); the new schema becomes visible in one commit when the rebuild is complete. | | Boost, `Stored` | Take effect at once, for new documents. No rebuild. | | An unchanged field | Nothing. | | A change to a type the stored values cannot be converted to (anything but the three conversions above), `Multi` turned off, a different vector dimension, a different identifier field, an analyzer that is not registered | `SchemaEvolutionException` at open, naming the field (`Field`). The index is left as it was. | | A rebuild is needed but a field that would have to be rebuilt, or any other searchable field, is not stored | `SchemaEvolutionException` naming that field: the index cannot be recovered from stored source that was never kept. | ## The rebuild: one version serves, one swap The commit file holds two things while a rebuild runs: the **served** schema and segments (the old ones, unchanged) and a **pending** section: the schema to be served next and the segments rebuilt for it so far, each with the name of the segment it was rebuilt from and how many of that source's documents were deleted at the time (commit-format.md). - **One version at a time.** Queries, snapshots, reader processes and near-real-time views only ever see the served schema and segments. A rebuilt segment is written, listed in the pending section and fsynced with the commit, but never read. Any commit at any kill point therefore serves exactly one schema version with every document present. - **The swap is one commit.** When every served segment has a current rebuilt twin and nothing is in flight, the next commit lists the rebuilt segments as the served ones, the pending schema as the schema and raises the schema epoch (each segment records the epoch it was written under; the commit records the served one). Readers move from the old version to the new in that atomic rename. - **Resumable.** Each commit starts at most four rebuilds and lists the finished ones, so a process that dies loses at most the segments in flight. Opening again with the same declared schema keeps the listed rebuilt segments and continues. Opening with a different change discards them and rebuilds for the new one; opening with the schema being served abandons the rebuild. - **Rebuilt twins are only good for the source they saw.** A delete (including an upsert's) on a source after it was rebuilt makes the twin stale: it is dropped and rebuilt again. This keeps the swap trivial, at the price of rebuilding a segment more than once under a steady stream of deletes. - **Writes finish the rebuild.** A write of a new document completes the rebuild first and makes the swap in memory (the commit that follows publishes it, so nothing becomes visible or durable earlier than the application asked), so documents are always written under one schema. Deletes are allowed while the rebuild runs and are served at once. Merges wait until the swap, and `ForceMerge` completes the rebuild. - **Pace.** With the default cooperative scheduler the rebuilds run during the next commit on the caller's thread; with a threaded scheduler on its thread. `CompleteSchemaEvolution` (on `IndexWriter` and `SearchIndex`) runs the rest to the end, committing as it goes. `SchemaEvolution` says how many segments still wait. - **Threads are optional.** Rebuilds run through the same scheduler seam as merges. The cooperative and the threaded run produce byte-identical files (tested), because segment names are taken in a fixed order when the rebuilds are queued, results are listed in name order, and the derived fields of dynamic JSON are ordered by name. - **Memory.** A rebuild holds the live documents of one segment in memory. Segments are bounded by the merge policy (5,000,000 live documents by default); lower `MaxMergedSegmentDocs` where documents are large. - **Storage.** All reads and writes go through the storage interface; the fault injector sees them. A failed rebuild (for example a stored value that no longer fits the new type) leaves the index as it was and is reported by `WaitForMerges` / `CompleteSchemaEvolution` as an `IOException` whose inner exception says why. ## Reader processes A reader process keeps working across the swap. It reads the commit it opened, whose files stay until it lets go (leases, processes.md); `Refresh` moves it to a newer commit, with the schema that commit serves. ## Verified by Tests: analyzer, column, indexing, removal and addition; refusals naming the field; reopening halfway (the served schema and segments unchanged, rebuilt twins listed); deletes and upserts during the rebuild; cooperative against threaded byte identity; a typed document type that changes between runs; a reader process across the swap; the pending section round trip. Durability: the kill campaign with schema changes between runs (writer processes killed at random moments of writes, commits, merges and rebuilds, then reopened and checked against the acknowledged commits, every reopen serving segments of a single epoch, the rebuild run to the end every 25 cycles) and the fault sweep with a power cut or failed operation at every storage operation of a run that changes schema halfway (durability method in storage.md). # Analysis An `Analyzer` turns text into terms. It is a tokenizer followed by filters, built once and reused. This page is normative for the contract, the registered analyzers, the Unicode and Snowball versions and the folding rules; two indexes built by different builds agree on terms only if the analyzer name and these rules agree. ## Token `TokenSource` is a pull source: `Reset(text)`, then `MoveNext()` until it returns false. One mutable `Token` object is shared by the whole chain; a tokenizer fills it, each filter edits it in place or drops it. Once the term buffer has grown to the longest term, a chain allocates nothing per token (the tests measure this for `standard`, `en`, `en-both` and `cjk`). - **Term text** is UTF-16. .NET strings are UTF-16, so offsets index the original string directly (highlighting, snippets), case and accent folding edit characters in place, and the single conversion to UTF-8 happens where a term enters the term dictionary. - **Offsets** are `[StartOffset, EndOffset)` in characters of the original text. A term made by a filter (a stem, a folded form) keeps the offsets of the word it came from; a CJK bigram covers both characters. - **Positions**: the first token's position is its increment minus 1, so an increment of 1 gives position 0. An increment of 0 stacks a token on the previous position (synonyms, a stemmed twin). A filter that removes a token adds its increment to the next token's, so a phrase across a removed stop word keeps its gap. - **Type** classifies a token: `Word`, `Number` (digits only), `Ideographic` (Han, Hiragana, Katakana or Hangul text), `Synonym` (added by a filter at another token's position), `Other`. - **Marks** are flags: `Keyword` (a filter must not rewrite the term; the stemmer honours it) and `Stemmed` (the term is a stem a stemming filter added beside its word). ### Stemmed and unstemmed forms `StemFilter` has two outputs. `Replace` swaps the word for its stem. `Both` keeps the word and, when the stem differs, adds the stem right after it as a second token at the same position: increment 0, type `Synonym`, marks `Stemmed`, same offsets. A stem is never empty; a word the algorithm would delete is kept as it is. An indexer that wants exact matches to rank above stem matches reads the mark: tokens without `Stemmed` are the words as written (after case folding), tokens with it are stems, and the indexer can store them in separate term spaces or fields. A query analyzed with the same `-both` analyzer produces the same pairs; one analyzed with the plain language analyzer produces only stems. The registry names the two forms `en` (stems only) and `en-both`. ## Analyzer and registry An analyzer instance is not thread-safe; each thread asks `AnalyzerRegistry` for its own. The name is what a schema stores. The registry is a plain `switch` and a fixed table in code, not discovery by reflection, so trimming and NativeAOT keep what is named and nothing else. | Name | Chain | |---|---| | `standard` | `StandardTokenizer` → `CaseFoldFilter`. The default for text fields. No stop words, no stemming, no accent folding. | | `cjk` | `StandardTokenizer` with CJK bigrams → `NormalizationFilter` (NFKC) → `CaseFoldFilter`. | | `` | `StandardTokenizer` → [`TurkishCaseFilter` for `tr`] → `CaseFoldFilter` → `ApostropheFilter` → [`ElisionFilter` for `fr`, `it`, `ca`] → [`StopFilter` where Snowball publishes a list] → `StemFilter(Replace)` → [`AsciiFoldFilter` for `en`; `AccentFoldFilter` for `fr`, `es`, `it`, `pt`, `ca`, `nl`, `ro`]. | | `-both` | The same with `StemFilter(Both)`. | | `-compound`, `-compound-both` | For `da`, `de`, `fi`, `is`, `nl`, `no`, `sv`: the language chain with a `DecompoundFilter` after case folding (docs/decompounding.md). | | `ja`, `zh`, `ko`, `th`, `lo`, `km`, `my` | Registered by `AsianAnalyzers.Register()` from the Asian package (`AnalyzerRegistry.Register`); docs/analysis-asian.md. | | `keyword` | The whole text as one term. | | `whitespace-lowercase` | Split at white space, invariant lower case; a reference for tests. | Accent folding runs after stemming on purpose: the stemming algorithms read accented letters. It is applied to the Latin-script languages whose readers routinely drop accents when typing; it is not applied to languages where the accented letters are distinct letters (Danish, Norwegian, Swedish, Finnish, Turkish, Polish, Czech and the others). Language codes and the Snowball algorithm each uses: `ar` arabic, `ca` catalan, `cs` czech, `da` danish, `de` german, `el` greek, `en` english (Porter2), `eo` esperanto, `es` spanish, `et` estonian, `eu` basque, `fa` persian, `fi` finnish, `fr` french, `ga` irish, `hi` hindi, `hu` hungarian, `hy` armenian, `id` indonesian, `it` italian, `lt` lithuanian, `ne` nepali, `nl` dutch, `no` norwegian, `pl` polish, `pt` portuguese, `ro` romanian, `ru` russian, `sr` serbian, `st` sesotho, `sv` swedish, `ta` tamil, `tr` turkish, `yi` yiddish. `standard` is the analyzer to compare with other engines' "standard tokenization, lower-case, no stemming": Lucene's own `StandardAnalyzer` also removes English stop words, so a comparison configures that side without its stop filter. ## Word and sentence boundaries `StandardTokenizer` splits at the word boundaries of Unicode Standard Annex #29 revision 47 (Unicode 17.0.0), rules WB1 to WB999, implemented in `Analysis/Unicode/WordBreaker.cs`. A segment becomes a token when it contains a letter, a decimal digit or a letter number (general category L\*, Nd, Nl); white space, punctuation, symbols and emoji are dropped. `SentenceBreaker.cs` implements SB1 to SB998 for the file chunker. Both walk the text once with a few facts about the segment they are building, so cost is linear even for a million combining marks, a million flag symbols or a million closing brackets (a test runs those). They read UTF-16 and treat a lone surrogate as a character with no properties. Conformance: 100% of `WordBreakTest.txt` (1,944 cases) and `SentenceBreakTest.txt` (512 cases) from the UCD 17.0.0 pass (`tests/OneIndex.Tests/Uax29ConformanceTests.cs`; the two files are in `testdata/ucd/` with a manifest). Choices the Annex leaves open: - **Token length.** A word longer than 255 characters is cut into pieces of 255 (never inside a surrogate pair), so no input makes an unbounded term. `StandardTokenizer(maxTokenLength)` changes the limit. - **What is a token.** Letters and digits as the UCD classifies them for Unicode 17.0.0 (generated, not taken from the platform's older tables). - **CJK.** Ideographs, kana and Hangul are letters, so each Han ideograph and each kana is its own token (UAX #29 breaks around them), Katakana and Hangul words stay whole, and `cjk` makes bigrams from them. ### CJK bigrams With `cjkBigrams`, a run of characters of one script (Han, Hiragana, Katakana or Hangul, from `Scripts.txt`; U+30FC and U+FF70 count as Katakana) becomes overlapping two-character terms: 日本語 gives 日本 and 本語, and a lone character stays a one-character term. Combining marks and format characters stay with their base character. Runs of different scripts do not bigram together. Each term has increment 1 and type `Ideographic`. Half-width Katakana voicing marks (U+FF9E, U+FF9F) are Common script and end a run before NFKC joins them; dictionary segmentation (Japanese, Chinese, Korean) is the job of a separate package. ## Normalization and folding - **Normalization (NFC, NFD, NFKC, NFKD)** is OneIndex's own implementation of UAX #15 over tables generated from the UCD (`Analysis/Unicode/Normalizer.cs`), not the platform's. Finding that decided it: with invariant globalization (the default for browser WebAssembly builds, and set for this repository's test projects), `string.Normalize` and `IsNormalized` return non-ASCII text unchanged without raising anything, so an index written in such a process would hold different terms from one written with ICU. The implementation passes all of `NormalizationTest.txt` (UCD 17.0.0, about 19,000 lines, all five columns and all four forms). The test reads the file from a cache because it is 2.5 MB; set `ONEINDEX_UCD_DIR`. - **Case folding** is full folding from `CaseFolding.txt` (statuses C and F): `ß` → `ss`, `ſ` → `s`, final sigma → sigma, `İ` → `i` plus a combining dot. It does not depend on the culture. The Turkic mappings (status T) are not used; the `tr` analyzer maps `I` → `ı` and `İ` → `i` first. - **Accent folding** (`AccentFoldFilter`): decompose (NFD), drop the combining marks of the diacritical blocks (U+0300–036F, 1AB0–1AFF, 1DC0–1DFF, 20D0–20FF, FE20–FE2F), compose (NFC). Hangul and kana come back unchanged. - **ASCII folding** (`AsciiFoldFilter`): compatibility decomposition (NFKD) so ligatures, full-width forms and superscripts become plain, accents dropped, then a short table for Latin letters that do not decompose: Æ→AE, Œ→OE, ß→ss, Ø→O, Đ→D, Ð→D, Ł→L, Ħ→H, ı→i, Ŧ→T, Þ→TH, ĸ→q, Ŋ→N and the same lower case, plus typographic quotes and dashes. Characters with no ASCII form stay. - **Apostrophes** (`ApostropheFilter`): U+2018, U+2019, U+02BC and U+FF07 become `'`, so stop words and elision match. - **Elision** (`ElisionFilter`): French `l' m' t' qu' n' s' j' d' c' jusqu' quoiqu' lorsqu' puisqu'`, Italian `c' l' all' dall' dell' nell' sull' coll' pell' gl' agl' dagl' degl' negl' sugl' un' m' t' s' v' d'`, Catalan `d' l' m' n' s' t'`. ## Stemmers Thirty-four stemmers, one per language above, generated from the Snowball algorithm scripts by `tools/SnowballGen`, a compiler written for this repository from the Snowball manual (). Only the `.sbl` scripts and the manual were read; the Snowball project's own compiler, runtime and generated stemmers were not. - **Scripts:** `algorithms/.sbl` of Snowball release v3.1.1 (commit cd195b51e948a902a4312f023f4a14392516a543), copied unchanged into `tools/SnowballGen/algorithms/`. BSD-3-Clause, see `THIRD-PARTY-NOTICES.md`. - **Runtime:** `Analysis/Snowball/SnowballProgram.cs` is the string machine the manual describes (cursor, limits, slice, `among` with longest match and routine conditions, forward and backward modes). Strings are UTF-16 code units, which is exact for every supported script (all in the Basic Multilingual Plane). - **Choices where the manual is silent:** after a replacement, a cursor at or past the replaced text moves with it and a cursor inside it moves to its start; a routine named in an `among` entry runs with the cursor just past the matched text; backward `insert` leaves the cursor on the left of the new text and `attach` on the right (forward is the mirror image); saved cursors in backward code are kept as the distance from the end of the text so deletions to the left do not disturb them. The published vocabularies confirm each. - **Conformance:** every stemmer reproduces its published vocabulary exactly, 100% of 11,481,578 words across the 34 languages (table below). The test needs the snowball-data repository (, main, archive SHA-256 `5bacb3c4f822d167707d87e08e807f65c27ac517543ef57fe4a2bfcf5bfcb7ed`) in a cache; set `ONEINDEX_SNOWBALL_DATA`. The data is fetch-only. Its licences are per directory (BSD-3-Clause for the original lists; Wikipedia-derived lists CC BY-SA; the Arabic list GPL-3.0), which is acceptable for a test oracle that is never redistributed. | Language | Words | Language | Words | Language | Words | |---|---|---|---|---|---| | arabic | 9,196,214 | hindi | 65,118 | polish | 62,576 | | armenian | 101,780 | hungarian | 29,881 | portuguese | 32,016 | | basque | 21,386 | indonesian | 64,586 | romanian | 87,642 | | catalan | 48,897 | irish | 186,280 | russian | 49,785 | | czech | 58,137 | italian | 37,426 | serbian | 29,996 | | danish | 23,868 | lithuanian | 86,109 | sesotho | 9,848 | | dutch | 45,670 | nepali | 4,000 | spanish | 28,378 | | english | 42,649 | norwegian | 20,913 | swedish | 30,738 | | esperanto | 84,456 | persian | 104,268 | tamil | 443,271 | | estonian | 57,537 | | | turkish | 96,325 | | finnish | 84,399 | | | yiddish | 99,996 | | french | 21,653 | | | | | | german | 35,053 | | | | | | greek | 90,727 | | | | | Every row matched in full (matched = words). ## Stop words The lists Snowball publishes at `https://snowballstem.org/algorithms//stop.txt`, for Danish, Dutch, English, Finnish, French, German, Hungarian, Indonesian, Irish, Italian, Norwegian, Portuguese, Russian, Sesotho, Spanish and Swedish (`tools/generate-stopwords.py` writes `StopWordData.g.cs`, recording each list's SHA-256). Other languages have no list and keep their stop words. Stop words are matched after case folding and before stemming, exactly as written (so accented forms are listed with accents); `StopWords` is a sorted array searched without allocating. ### Configured stop words `StopWordSettings` (words, `Mode`, `PreserveGaps`) configures stop words per index and is passed to `LanguageAnalyzer` (or to a custom chain through `CreateFilter`). `Mode.Replace` (default) uses only the configured words, so an empty list turns stop words off; `Mode.Extend` adds them to the language's list (for a language with no list, the configured words alone). Configured words are folded as the standard analyzer folds text (`The` is `the`) and are matched, like the defaults, after case folding and before stemming and accent folding. Stop words are removed by the analyzer, so they are handled at **both ends** by the same code: a stop word is never indexed, and it is dropped from the query before any query object is built, also inside quoted phrases (`"the great gatsby"` is the phrase `great gatsby`, matching the text with and without the stop word). Positions: with `PreserveGaps` false (the default for configured lists) the neighbours of a removed word become adjacent; with true the gap stays in the positions (the unconfigured language lists keep their gaps, as before). A query made only of stop words has no terms and lowers to match-all, as an empty search returns every document. The analyzer name does not change with the settings: the settings belong to the index and must be the same when it is built and when it is searched. Changing them needs a reindex. See docs/synonyms.md, "Positions and gaps". ## Regenerating tools/UcdTables dotnet run --project tools/UcdTables -- src/OneIndex/Analysis/Unicode tools/SnowballGen dotnet run --project tools/SnowballGen -- tools/SnowballGen/algorithms src/OneIndex/Analysis/Snowball/Languages "Snowball v3.1.1" tools/generate-stopwords.py .txt> src/OneIndex/Analysis/StopWordData.g.cs The generated files record their inputs. The UCD files used are `WordBreakProperty.txt`, `SentenceBreakProperty.txt`, `emoji-data.txt`, `CaseFolding.txt`, `UnicodeData.txt`, `Scripts.txt` and `DerivedNormalizationProps.txt`; `tools/fetch-analysis-data.sh` fetches and checks them. ## Limits and safety Analysis never throws on text: lone surrogates, control characters and megabyte runs of one character are valid input (an analysis fuzz target covers them). Term length is capped by the tokenizer (255 by default); the keyword analyzer is the one exception and emits the text as given, so a field that uses it is limited by the index's term-length limit. Working buffers grow to the longest term seen and stay. # Asian-language analysis (`RecursiveAI.OneIndex.Analysis.Asian`) Japanese, Chinese, Korean, Thai, Lao, Khmer and Myanmar write words without spaces (Korean with spaces, but with agglutinated endings). The package segments them with packed dictionaries and plugs into the analyzer pipeline of core (`docs/analysis.md`). It references only core and the BCL. ```csharp AsianAnalyzers.Register(); // once at start-up: makes "ja", "zh", "ko", "th", "lo", "km", "my" known by name [FullText(Analyzer = "ja")] public string Body { get; set; } ``` Registration is explicit (`AnalyzerRegistry.Register(name, factory)`, added to core for this package); nothing is discovered by reflection. A schema stores the analyzer name, so an index opens only after the call. The analyzers are `Analyzer` subclasses (`JapaneseAnalyzer`, `ChineseAnalyzer`, `KoreanAnalyzer`, `WordBreakAnalyzer.Thai()` ...); each is its tokenizer, NFKC and case folding. Common to every tokenizer: the text is cut into spans of the target script, which a dictionary segmenter splits; everything else (Latin words, digits, other scripts) goes to the UAX #29 word tokenizer, so "iPhone 15" inside Japanese text stays whole. Tokens keep the offsets of the original text. Function words that are dropped leave a position gap, so phrase queries do not match across them. ## Dictionaries and sizes Dictionaries are data in one container format, `OIDICT` version 1 (magic, version, sections, CRC-32C footer, `DictionaryLimits` on every length; malformed files raise `DictionaryFormatException`, `TryLoad` beside it). They are built by `tools/dictionaries/fetch.sh` and `build.sh` from the sources below and embedded when present at build time. Any dictionary can also be loaded from a file (`LatticeLexicon.Load`, `WordList.Load`). | Language | Segmenter | Dictionary | Licence | Packed size | |---|---|---|---|---| | ja | lattice search (`LatticeSegmenter`), search mode | IPADIC 2.7.0-20070801 (392,126 words, connection matrix 1316 x 1316) | NAIST/ICOT permissive custom licence, notice packed | 4,186,442 B | | ko | lattice search, normal mode | mecab-ko-dic 2.1.1-20180720 (501,293 words after dropping the person, actor and address name classes; matrix 3822 x 2693) | Apache-2.0 | 7,242,507 B | | zh | most probable path (`UnigramSegmenter`), search mode | jieba `dict.txt` word frequencies | MIT | 2,023,493 B | | th | maximal matching (`MaximalMatchSegmenter`) | ICU `thaidict.txt` | Unicode License v3 | 133,499 B | | lo | maximal matching | ICU `laodict.txt` | Unicode License v3, with BSD-2 text | 162,031 B | | km | maximal matching | ICU `khmerdict.txt` | Unicode License v3 | 449,281 B | | my | maximal matching | ICU `burmesedict.txt` | Unicode License v3, with BSD-2 text | 227,505 B | Total about 14.4 MB (under the 50 MB at which a separate package would be proposed). Notices are in `THIRD-PARTY-NOTICES.md`. ## Segmenters - **Lattice (ja, ko):** every dictionary word starting at each position is a node, unknown-word nodes come from the character classes of the dictionary, and the path with the lowest sum of word costs and connection costs (left/right context ids) is found by Viterbi search, as in Kudo, Yamamoto and Matsumoto, "Applying Conditional Random Fields to Japanese Morphological Analysis" (EMNLP 2004). The dictionaries' costs are used as published. Japanese search mode adds a length penalty to long Han and Katakana words (after the published description of search-mode segmentation: threshold 2 and penalty 3000 for Han, threshold 7 and penalty 10000 for Katakana; untuned beyond the F1 check below) so compounds are split. Japanese tokens are the base form of inflected words; particles, auxiliary verbs and symbols are dropped (part-of-speech prefixes 助詞, 助動詞, 記号). Korean yields the content morphemes (a verb's stem, not its endings) and a compound noun is the compound followed by its parts at the same position; the classes dropped are the endings, particles, suffixes and symbols (tags beginning with E, J, SC, SE, SF, SP, SSO, SSC, SY, VCP, VCN, XPN, XSA, XSN, XSV). - **Unigram (zh):** the segmentation maximising the product of word probabilities from the frequency list (dynamic programming over the word graph); unknown characters stand alone (single characters). Search mode also emits the dictionary parts of long words at the same position. - **Maximal matching (th, lo, km, my):** dynamic programming from the end of the text picks the path with the fewest characters left unmatched, then the fewest words, then the longer first word. A boundary never falls before a combining mark or a spacing vowel sign, and unmatched characters are grouped into one token. ## Evaluation Word-span precision, recall and F1 (a predicted word counts when its span equals a gold span) against Universal Dependencies treebanks, which are evaluation data only: fetched on demand, never committed, CC BY-SA licensed (Thai-PUD: CC BY-SA 3.0). Run with `oneindex-dictionaries eval `. | Language | Treebank (test set) | Sentences / gold words | Precision | Recall | F1 | |---|---|---|---|---|---| | ja, normal mode | UD Japanese-GSD | 543 / 13,034 | 0.9375 | 0.9071 | 0.9221 | | ja, search mode | UD Japanese-GSD | 543 / 13,034 | 0.9387 | 0.9138 | 0.9261 | | zh | UD Chinese-GSDSimp | 500 / 12,010 | 0.8120 | 0.7722 | 0.7916 | | th | UD Thai-PUD | 1,000 / 22,330 | 0.7097 | 0.8175 | 0.7598 | | ko, lo, km, my | none admissible | | | | | Differences to explain: - **Japanese:** the treebank follows UniDic short-unit words and IPADIC's units are longer (compound nouns, noun plus suffix), so some of the gap is the two standards disagreeing, not errors. - **Chinese:** jieba's list has longer words than the treebank (企业界, 博物馆), so recall is lower than precision. The traditional-character treebank (Chinese-GSD) scores F1 0.5747 because the list is simplified-Chinese; traditional text is not covered by this dictionary. - **Thai:** the segmenter predicts more words than the treebank has (precision below recall): the ICU list is a break dictionary with many short entries, so names and compounds the treebank keeps whole are cut. It is not a gold standard of words. - **Korean:** the Korean treebank's tokens are eojeol (space-delimited units), not morphemes, so it cannot score a morphological segmenter; no morpheme-level gold with a usable licence was found. Korean is checked by tests on stems and compounds (for example the inflected 갔습니다 gives the stem 가). **Lao, Khmer and Myanmar:** no gold-segmented set was found. The shared segmenter is unit-tested on a synthetic list (and shared with Thai, which is scored above); the three packed analyzers get a smoke test (no tokens from blank text) and no accuracy figure. Treat them as dictionary breaking, not as evaluated segmentation. ## Speed (information) Segmentation only (the segmenter over text, not the whole analyzer), single thread, one process, warmed up, three runs per language; `oneindex-dictionaries speed `, indicative (a shared 16-core AMD Ryzen 9 7950X3D, load about 11 before, during and after; .NET 10.0, x64). Text: UD sentences for ja, ko, zh and th, repeated to about 3 million characters; for lo, km and my random strings of dictionary words (no real text was available), so those three show the cost of matching, not of real prose. | Language | Method | Million characters per second (3 runs) | MB per second of UTF-16 | |---|---|---|---| | ja | lattice, search mode | 1.75 to 1.91 | 3.5 to 3.8 | | ko | lattice | 1.20 to 1.33 | 2.4 to 2.7 | | zh | unigram | 4.45 to 4.56 | 8.9 to 9.1 | | th | maximal matching | 5.80 to 6.09 | 11.6 to 12.2 | | lo | maximal matching | 4.94 to 6.10 | 9.9 to 12.2 | | km | maximal matching | 4.97 to 5.18 | 9.9 to 10.4 | | my | maximal matching | 6.03 to 6.13 | 12.1 to 12.3 | There is no comparator row: the other segmenters were not run (no managed engine's segmenter is in the benchmark set). ## Limits and safety Dictionary readers validate the magic, version, every section length and the checksum before using an offset; sizes and counts are bounded by `DictionaryLimits`; a hostile file raises `DictionaryFormatException`. The `dictionary` and `asian` fuzz targets cover the readers, the segmenters and the analyzers (docs in `fuzz/README.md`). Segmenters allocate per call only their lattice buffers, reused by the tokenizer between texts. The package works in the browser: nothing in it depends on threads, native code or reflection. ## Not included Readings (furigana), UniDic's short-unit and long-unit forms, user dictionaries (load another `.oidict` for that), traditional-Chinese words, Cantonese, Vietnamese (written with spaces) and language detection (the schema names the analyzer). # Decompounding German, Dutch, the Scandinavian languages, Finnish and Icelandic write compounds as one word ("Haustür", "kirjakauppa"). A search for "Tür" should find "Haustür", and one for "Haustür" should find "Tür". `CompoundSplitter` cuts a compound into the words it is made of, and `DecompoundFilter` indexes the compound and its parts at one position, so either form matches. ## Method Corpus-statistics splitting after Koehn and Knight, "Empirical Methods for Compound Splitting" (EACL 2003), section 3: 1. A word list with counts (`CompoundVocabulary`). A plain list is accepted too (every count 1). 2. Every way to cover the word with list words is a candidate. Parts may be joined by a linking element of the language ("Liebe-s-brief", `CompoundOptions.LinkingElements`); the linking element belongs to neither part. 3. Among candidates with the fewest parts (at most `MaxParts`, default 4), the one with the best geometric mean of the parts' counts wins. Preferring fewer parts is a deviation from the paper (which maximises the geometric mean over all part counts): on our samples the plain geometric mean cuts words into short, very frequent fragments ("mittagessen" into "mit", "tag", "essen"; "sommerferie" into "som", "mer", "ferie"). `PreferFewerParts = false` restores the paper's rule. 4. With counts, a word is not split when its own count is at least the geometric mean of the parts' counts (the paper's rule: a frequent compound is a word in its own right). This also means that frequent compounds ("Kühlschrank") are not split. 5. Parts have at least 3 characters (`MinPartLength`); words shorter than 6 are not tried (`MinWordLength`). The splitter does not repair stems (it does not know that "skol-" is "skola" or that Icelandic "bóka-" is "bók"); a compound whose first part is such a form is left whole. The filter never drops the compound: a wrong split adds a few extra terms and does not lose the original. ## Use ```csharp CompoundSplitter splitter = CompoundSplitter.ForLanguage("de"); // packed list and the language's linking elements var parts = new List<(int Start, int Length)>(); bool split = splitter.TrySplit("Haustür", parts); // parts: "Haus", "tür" var analyzer = new LanguageAnalyzer("de", keepUnstemmed: false, decompound: true); // registered as "de-compound" ``` The registered names are `-compound` and `-compound-both` for `de`, `nl`, `sv`, `da`, `no`, `fi` and `is`. The chain is the language analyzer's with the decompounder after case folding, so each part is stop-filtered and stemmed like any other word. Lists of your own load with `CompoundVocabulary.Read` (`word` or `word TAB count` per line). ## Packed word lists Built by `tools/decompounding/fetch.sh` and `tools/decompounding/build_lists.py`, embedded in `RecursiveAI.OneIndex` as gzip text (the build without them still compiles; `CompoundVocabulary.ForLanguage` then reports the missing list). | Language | Source | Licence | Words | Packed size | |---|---|---|---|---| | de | Leipzig Corpora Collection, German news 2024, 300K sentences | CC BY 4.0 | 69,346 | 326,516 B | | sv | Leipzig, Swedish news 2023 | CC BY 4.0 | 62,652 | 291,204 B | | da | Leipzig, Danish news 2022 | CC BY 4.0 | 55,227 | 259,160 B | | no | Leipzig, Norwegian news 2023 | CC BY 4.0 | 57,288 | 263,959 B | | fi | Leipzig, Finnish news 2022 | CC BY 4.0 | 97,120 | 437,407 B | | is | Leipzig, Icelandic news crawl 2019 | CC BY 4.0 | 74,178 | 348,348 B | | nl | OpenTaal word list 2.20.23 with counts from the Leipzig Dutch news 2023 corpus | Revised BSD (OpenTaal's choice of Revised BSD and/or CC BY 3.0); CC BY 4.0 | 391,648 | 1,413,732 B | Total 3.3 MB. Leipzig lists keep words of 3 to 40 letters seen at least 3 times in the 300,000 sentences, case folded (the lists are frequency lists of running text, so they contain names, some typos and inflected forms; the counts are what lets the splitter prefer real words). The licence of the downloadable Leipzig corpora is stated on https://wortschatz-leipzig.de/en/usage ("The text corpora offered for download are made available under the Creative Commons licence CC BY"); attribution is in `THIRD-PARTY-NOTICES.md` and the package readme. Wikipedia-derived corpora (CC BY-SA) are not used. ## Evaluation No gold-standard splitting set with a licence that allows this use was found (GermaNet's splits and similar sets are restricted), so the evaluation uses small hand-written samples committed in `testdata/decompounding/.tsv`: 26 to 40 compounds with accepted splits and 12 to 20 plain words that must stay whole per language. They were written for this purpose, are not tuned on, and are small: read the numbers as a smoke measurement, not a benchmark. Numbers are in `DecompoundingTests.PackedListsSplitTheSample`, which fails below floors set under them. | Language | Compounds | Split | Exact (accepted split) | Plain words | Falsely split | |---|---|---|---|---|---| | de | 40 | 35 (0.875) | 33 (0.825) | 20 | 0 | | nl | 29 | 28 (0.966) | 27 (0.931) | 15 | 3 (0.200) | | sv | 26 | 22 (0.846) | 22 (0.846) | 14 | 0 | | da | 26 | 23 (0.885) | 23 (0.885) | 13 | 1 (0.077) | | no | 26 | 24 (0.923) | 22 (0.846) | 12 | 1 (0.083) | | fi | 27 | 24 (0.889) | 24 (0.889) | 14 | 0 | | is | 26 | 20 (0.769) | 16 (0.615) | 12 | 1 (0.083) | Misses fall into three groups: frequent compounds kept whole by the frequency rule (Kühlschrank, Flughafen, Wohnzimmer, Lufthavn), first parts that are not words as written (Swedish skol-bok, Icelandic bóka-búð, Finnish consonant gradation), and wrong cuts into short fragments (eisenbahn into "eis" and "bahn"; the Dutch plain words vriendelijk, aardappel and venster, and the Danish and Norwegian "frihed"/"frihet" are split because the suffix or a short piece is itself a frequent word). ### Settling the defaults On the seven samples together (mean over languages; lists with the counts of the 300K-sentence corpora): | MinPartLength | Fewest parts | Frequency rule | Count floor | Split | Exact | False split | |---|---|---|---|---|---|---| | 3 | no | off | 1 | 0.918 | 0.820 | 0.167 | | 3 | no | on | 1 | 0.884 | 0.794 | 0.063 | | **3** | **yes** | **on** | **1** | **0.879** | **0.834** | **0.063** | | 4 | yes | on | 1 | 0.627 | 0.602 | 0.019 | | 3 | yes | on | 8 | 0.794 | 0.758 | 0.051 | Part length 4 loses a third of the compounds (tür, hof, bus, rad ...). A count floor of 8 (the lists already hold words seen at least 3 times) loses coverage and gains little. Requiring more occurrences for 3-letter parts trades recall for false splits one for one. The bold row is the default (the packed lists keep words seen at least 3 times). ## Not included Hunspell dictionaries (mostly GPL, LGPL or MPL) are not used. A trained splitter, truncation or umlaut repair, and per-language part-of-speech checks are not implemented. The other languages with compounding (Afrikaans, Hungarian, Estonian ...) have no packed list; load one with `CompoundVocabulary.Read` and use `CompoundOptions` directly. # Term dictionary Format of the **terms** part of a segment (codec `terms`, versions 1 and 2) and the term-lookup operations built on it. Version 2 is written by this build; version 1 files stay readable and a merge rewrites them as version 2. The two versions differ only in how a node of the automaton is stored (below): the file layout, the statistics table, the ordinals and every query answer are the same. Integers, varints and the file frame are as in `commit-format.md`, except that varints in this file are read leniently for length (a reader accepts a non-shortest form but rejects one past 63 bits); a writer always emits the shortest form, so the bytes are canonical. ## What it holds For each text field of a segment that has at least one term: an acyclic automaton that maps every term to its **ordinal** (its rank in byte order), a table of per-term statistics indexed by ordinal, and field counts. The postings themselves are in the postings part (`postings.md`); a term's statistics carry the offset of its postings. Terms are byte strings (UTF-8 for text), compared as unsigned bytes. UTF-8 byte order equals code-point order, which differs from .NET's UTF-16 ordinal order only for characters outside the BMP. A term is at most `IndexLimits.MaxTermBytes` bytes (default 32,766). ## File layout frame header magic "OIXT", version 1 or 2 varint maxDoc documents in the segment; equals the lengths file's repeat per field, in increasing ordinal order of the field name: u8 1 field marker string name u8 flags bit 0 positions stored, bit 1 offsets stored (offsets imply positions) varint docCount documents with a non-zero field length varint sumTotalTermFreq occurrences of all terms varint sumDocFreq sum of document frequencies varint termCount at least 1 varint totalTermBytes sum of the terms' byte lengths varint maxTermBytes length of the longest term varint fstLength, varint fstRoot fstLength bytes the automaton varint tableLength, tableLength bytes the statistics table varint sampleCount ceil(termCount / 32) sampleCount x (u32 tableOffset, u64 postingsOffset) u8 0 end marker frame footer A field with no terms (for example after a merge dropped all its documents) is omitted. ## The automaton The automaton is the minimal acyclic deterministic automaton of the sorted term list, built incrementally from sorted input (Daciuk, Mihov, Watson and Watson 2000, section 4): only the path of the last term stays open; when the next term diverges, the nodes below the divergence are *frozen* and looked up in a register so equal sub-automata are stored once. The result is the unique minimal automaton, so the bytes depend only on the term set. Nodes are written children first, so an arc always points to a lower address and cycles cannot exist. The root is the last node; `fstRoot` is its address. **Node, version 1.** Arcs are read in sequence: varint (arcCount << 1) | isFinal arcCount at most 256 per arc, labels strictly increasing: u8 label varint words number of terms reachable through the arc varint distance node address minus child address, at least 1 **Node, version 2.** The same information in three tables of fixed-width numbers, so an arc is found by its label, and the terms before it counted, without decoding the arcs in front of it: u8 head bit 0 isFinal; bits 1-2 cumulative-count width minus 1; bits 3-4 distance width minus 1; bits 5-7 arcCount if below 7, else 7 [varint arcCount - 7] only when bits 5-7 are 7; arcCount at most 256 u8 label[arcCount] strictly increasing cumulative[arcCount] little endian, width 1 to 4 bytes: words through arc i, that is the sum of the words of arcs 0 to i distance[arcCount] little endian, width 1 to 4 bytes: node address minus child address, at least 1 A writer chooses the narrowest widths that hold the node's largest values, so the bytes are still canonical. The words of arc *i* are `cumulative[i] - cumulative[i-1]`, and the terms before it `cumulative[i-1]` (0 for the first). A leaf has a head byte only. Nodes are shared exactly as in version 1 (the register compares sub-automata, not encodings). On a dictionary of 407,437 distinct lowercased word tokens of 302,567 Wikipedia abstracts the automaton is 2,764,464 bytes in version 2 against 2,535,846 in version 1 (+9.0%). The **words** counts number the terms: the ordinal of a term is the count of terms that sort before it, computed while walking the term: one for each final node passed (a node's own term sorts before its extensions), plus the `words` of every arc passed over to the left of the arc taken (Lucchesi and Kowaltowski 1993: perfect hashing from a finite automaton). This gives exact lookup in time proportional to the term length, ordered enumeration with ordinals, and the ordinal range of a whole subtree without visiting it. ### Statistics table One entry per term in ordinal order, each three varints: `docFreq` (at least 1), `totalTermFreq - docFreq`, and the postings offset as a difference from the previous entry's offset (at least 0). The first entry of every group of 32 omits the offset: it is the group's sample. A sample is `(tableOffset, postingsOffset)` giving the table position and absolute postings offset of the group's first entry. Looking up ordinal *n* decodes at most 31 entries from the sample of `n / 32`; enumeration reads entries sequentially. ## Operations - **Exact lookup**, **ordered iteration** and **prefix enumeration** (walk the prefix bytes, then enumerate the subtree) work on the automaton as described. - **Intersection with a code-point automaton** (`TermAutomaton`) walks the dictionary and the automaton together, depth first in byte order, and skips a subtree (adding its `words` to the running ordinal) as soon as the automaton is dead. UTF-8 is decoded on the fly; the automaton is advanced when a code point completes, and malformed UTF-8 in a term never matches. An optional exact prefix is applied first (for example a typo-free first letter), then the automaton continues from its state. In a version 2 file, when the automaton says that only a few first bytes can keep a state alive (a Levenshtein state with no edits left reads only the letters of the word near its position), the walk looks those labels up directly in the node's label table instead of visiting every arc, and adds the skipped arcs' words from the cumulative table. The answers are identical to the sequential walk; only the arcs read differ. - **Fuzzy** (`LevenshteinAutomaton`): terms within *k* edits (0 to 3) of a word, counting substitution, insertion and deletion of a code point, and optionally a swap of two adjacent code points, as one edit each. A state of the deterministic automaton is a set of (word position, edits, swap pending) triples of the non-deterministic automaton of Schulz and Mihov (2002, section 2). Following their section 4, the set is stored relative to its smallest position (a *shape*), and the effect of a code point depends only on the shape and on which of the next 2*k*+2 positions of the word hold that code point (the characteristic vector), plus how many of those positions lie inside the word. The transitions of each shape are computed the first time they are needed and shared by every automaton with the same edit budget and swap setting, on all threads; a step then costs a comparison of the code point with up to 2*k*+2 letters and two table reads. A state is two integers (shape, word position). With swaps the distance is the optimal string alignment distance (a swapped pair is not edited again); this is checked against dynamic programming over code points. - **Regular expressions** (`RegexAutomaton`): compiled to a Thompson NFA (Thompson 1968) and advanced as a set of states per code point. Always matches the whole term. ### Regular-expression syntax | Syntax | Meaning | |---|---| | any other character | itself (a code point; surrogate pairs are one character) | | `.` | any code point (including line breaks) | | `[abc]` `[a-z]` `[^...]` | class, range, negated class; a `]` first in a class is literal; escapes work inside | | `(...)` | group (not capturing) | | `a\|b` | alternation; an empty branch matches the empty string | | `*` `+` `?` | zero or more, one or more, zero or one | | `{n}` `{n,}` `{n,m}` | repeat counts, each at most 1,000 | | `\d` `\D` | ASCII digits `0-9`, and everything else | | `\w` `\W` | `A-Z a-z 0-9 _`, and everything else | | `\s` `\S` | space, tab, line feed, carriage return (`\t`-`\r`), and everything else | | `\n \r \t \f \v` | the control characters | | `\uXXXX`, `\U{X...}` | a code point in hex (four digits; or any number of digits in braces, at most 10FFFF) | | `\` + punctuation | that character literally | Not supported, and rejected with `RegexSyntaxException` (with the position): `^` and `$` (the match is always anchored), look-around, back-references, named groups, `\b`, Unicode property classes, unknown letter escapes, unbalanced brackets, empty repeats (`*a`), backward ranges, a repeat bound above 1,000, nesting deeper than 200 groups, or a pattern needing more than 10,000 automaton states. Lazy and possessive quantifiers are not special: `a*?` is "(a*) optionally". ## Reading and corruption The terms file is read into memory once (limit `IndexLimits.MaxResidentFileBytes`, default 1 GiB), its checksum verified, and its structure validated: marker bytes, field order, counts consistent with each other, section lengths inside the file, sample count equal to `ceil(termCount/32)`. During lookups every address, arc count and number is bounds-checked: arc labels must increase, a child address must be lower than its node, `words` cannot exceed `termCount`, an ordinal must be below `termCount`, enumeration depth cannot exceed `maxTermBytes`, and the number of nodes visited in one walk cannot exceed `totalTermBytes + 2` (a trie has at most that many), which bounds work on a crafted shared-node structure. Any violation raises `CorruptIndexException` naming the file; a newer format version raises `UnsupportedFormatException`. Lookup and enumeration do not allocate besides the enumerator object itself. ## Deviations and choices - The automaton and statistics are resident rather than range-read per node: a term dictionary is a few bytes per term, and the static (browser) index will use its own chunked layout. - Version 1 scans arcs linearly within a node (at most 256 per node); a node's arcs are sorted so a miss stops early. Version 2 looks an arc up in the label table (a vectorised byte search over at most 256 labels). - Version 2 does not check at read time that a node's labels are sorted when it looks an arc up directly; a damaged label table can at worst make a lookup miss or find a wrong child, and the file checksum, the child-address rule, the count bounds and the walk limit above still hold. The sequential walk checks that labels increase. - Size and effect on the 302,567-abstract corpus: the automaton grows by 9.0% (2,535,846 to 2,764,464 bytes) and the whole index by 0.8%; the Fuzzy2 query falls from 2.57 ms (version 1 dictionary, same code otherwise) to 1.23 ms on x64 with the other changes of the same build (`benchmarks.md`). - Reading a version 1 file keeps the version 1 node decoder; nothing converts a file in place. Segments get version 2 when they are written (flush or merge), so an index moves to version 2 as its old segments merge away. # Postings, field lengths and block-max WAND Formats of the **postings** part (codec `postings`, version 1) and the **lengths** part (codec `lengths`, version 1), and the top-k disjunction that uses their block impacts. File frame, integers and varints as in `commit-format.md`. ## Postings file frame header magic "OIXP", version 1 postings blobs, one per term, back to back, in the order the terms were written (fields in ordinal order, terms in byte order within a field) frame footer A term's blob is found from the offset in its statistics (`term-dictionary.md`); the blob does not say how long it is, and a reader never needs to know. Whether positions and offsets are stored is a property of the field (the flags in the terms file), not of the blob. ### Blob A term with `df` documents has `blocks = ceil(df / 128)` blocks; all are full (128 documents) but the last. Within a blob document ids are absolute and strictly increasing. blocks = 1: varint lastDoc, varint maxFreq, varint minLen, then the block blocks > 1: varint maxFreq, varint minLen term-level impact blocks x skip entry (16 bytes: u32 lastDoc, u32 dataOffset, u32 maxFreq, u32 minLen) the blocks `dataOffset` is the offset of a block from the start of the first block. **Impact** of a run of documents = (largest term frequency in the run, shortest field length of any document in it); the term-level impact covers every document, a skip entry covers its block (Ding and Suel 2011: the per-block maximum is what lets a query skip blocks). ### Block u8 docWidth, u8 freqWidth bits per value, 0 to 32 packed gaps n values of docWidth bits gap = doc - previousDoc - 1; the first previousDoc is the previous block's last document (-1 for the first block) packed freqs n values of freqWidth bits frequency - 1 positions only if the field stores positions, then per document, per occurrence: varint position - previous position of the same document (the first is absolute) with offsets also: varint zigzag(startOffset - previous start), varint endOffset - startOffset Values are bit-packed least significant bit first into `ceil(n * width / 8)` bytes. The width is the narrowest that fits the largest value in the block (frame of reference with delta coding: Zukowski et al. 2006; Lemire and Boytsov 2015), so a dense run of documents costs 0 bits per gap and a list of frequencies of 1 costs 0 bits per frequency. Positions are varints, not bit-packed; a reader skips the positions of documents it passes without decoding them into values. Choice of 128: 128 gaps decode in one tight loop, a block's doc and frequency bytes fit one small read (at most 1,026 bytes), and the skip table costs 16 bytes per 128 documents (about one bit per document). ### Reading `TermPostings` reads a term's postings straight from storage ranges: the blob header, the 16-byte skip entries it needs, one block at a time (one read for the widths, gaps and frequencies), and a 512-byte window over the positions. Advance finds the target's block by galloping and then binary-searching the skip entries (reading each entry from storage), loads that block and searches inside it. After `Reset` there are no allocations while iterating, advancing or reading positions; one instance can be pointed at another term with `Reset`. Checks, each raising `CorruptIndexException` naming the file: the term's statistics (document frequency in 1 to `maxDoc`, total frequency at least document frequency, offset inside the data area); header numbers in range; every skip entry's last document below `maxDoc`; a block inside the file; widths at most 32; documents strictly increasing, ending exactly at the block's last document, and at least as far apart as the block needs; frequencies below 2^31; position numbers within 32 bits and positions and offsets in range. Reads outside the file fail before they allocate. ## Lengths file The number of terms each document has in each field (the BM25 field length): the count of tokens added, stacked synonyms included, so a field's lengths sum to its `sumTotalTermFreq`. frame header magic "OIXL", version 1 varint maxDoc repeat per field, ordinal order: u8 1, string name, u8 width (1 to 4 bytes), maxDoc x width-byte little-endian lengths u8 0 frame footer Read into memory at open (limit `IndexLimits.MaxResidentFileBytes`); document count must equal the terms file's. ## Writing `InvertedBuffer` collects tokens per field (terms in a hash table, each term's documents, frequencies and positions in one growing integer array) and `WriteTo` feeds a `TextSegmentWriter` in term order. `TextSegmentMerger` feeds the same writer from several segments: per field a k-way merge of the term enumerators by byte order, postings copied through a document map (deleted documents map to -1; a term left with no documents is dropped). The writer is the only code that produces these bytes, and the format has exactly one encoding for a given content, so a merged segment's three files are byte-identical to files built directly from the surviving documents (tested). Terms longer than the limit are skipped by the buffer and refused by the writer. The writer's calls are synchronous (it writes through the storage interface as terms finish); `Finish` and `FinishAsync` fsync the three files. A crash before the commit that lists the files leaves unreferenced files and nothing else. ## Block-max WAND `BlockMaxWandScorer` is a disjunction of `Scorer`s that returns documents in increasing order and uses the collector's threshold (`SetMinCompetitiveScore`) to skip (Broder et al. 2003; Ding and Suel 2011). Each round: clauses are ordered by current document; the **pivot** is the first clause at which the sum of the clauses' `MaxScore` bounds reaches the threshold; clauses before the pivot are advanced to the pivot document; then, if the clauses up to the pivot are all at the pivot document, the bounds of their *blocks* containing it (`AdvanceShallow`, `GetMaxScore`) are summed, and if that sum is below the threshold every document up to the nearest block end is skipped (to the next clause's document if that is closer); otherwise the document is returned for scoring. A bound is skipped past only when it is below the threshold by a relative margin of 1e-5, which covers float rounding from adding bounds in a different order than scores. A document's score is the sum of its clauses' scores in clause order, so it equals exhaustive evaluation bit for bit, and with no threshold the scorer is a plain disjunction. `PostingsScorer` connects a `TermPostings` to the bound: its score function takes (frequency, field length) and must be non-negative, non-decreasing in frequency and non-increasing in length; the block bound is the function at (max frequency, min length). BM25F supplies its own function. ### Measured on seeded corpora (test run) Top 10 (and 3, 100) over 240 queries per corpus of 1, 2 and 5 terms drawn from common and rare terms; the result equals the exhaustive top-k (documents, order, scores) in every query. Documents handed to the collector, as a share of all matching documents: 24% to 29% for k = 10 on 6,000 and 12,000 documents, 25% for k = 3, 61% for k = 100 on 3,000 documents. Block bounds never returned more documents than whole-list bounds alone (about 5% fewer in these runs). # Columns This note is normative for the `columns` part of a segment: per-document values of the fields that filtering, sorting, faceting and aggregation read. File frames and the codec interface are in `commit-format.md` and `segments.md`. ## What gets a column A field has a column when the schema marks it `Column` or `Facet` and its type is keyword, integral, fractional, date, bool or geo point. A segment holds one `columns` file with the columns that have at least one value in the segment; a field no document of the segment has a value for has no column, and `GetColumn` returns `null`. The file is absent when no field qualifies. ## Value model A column is a sequence of *values*, in document order, plus a map from documents to runs of values. A document has any number of values (0 for a field it lacks; 1 for single-valued fields; any number for `Multi` fields). Values of one document keep the order the document gave them; they are not sorted or de-duplicated, so every value reads back exactly as written. Every value is held as a signed 64-bit integer (its *raw* form): | Type | Raw form | |---|---| | integral, date | the integer; a date is milliseconds since 1970-01-01T00:00:00Z | | bool | 0 or 1 | | fractional | `SortableBits.Encode(value)`: an order-preserving bijection of the IEEE 754 bits, so NaN payloads and the sign of zero are kept exactly and the minimum and maximum of a column are meaningful | | geo point | two sequences: latitude and longitude, each `SortableBits.Encode` of the double | | keyword | the ordinal of the value in the segment's sorted dictionary | ## File, `OIXK` version 1 ``` header (8 bytes) magic "OIXK", version 1 u32 maxDoc documents in the segment; must equal the commit's count sections one per column, back to back directory see below u64 directory offset absolute offset of the directory footer (12 bytes) checksum ``` Directory: `varuint count`, then per column, ordered by field name (ordinal comparison of the UTF-16 name, strictly increasing): `varuint nameLength, name (UTF-8)`, `u8 type` (the `FieldType` value), `u8 layout`, `varuint documentsWithValue`, `varuint valueCount`, `i64 minRaw`, `i64 maxRaw` (the smallest and largest raw value; a column with no values is not written), `varuint sectionOffset`, `varuint sectionLength`. Sections tile the space between the `maxDoc` field and the directory exactly. ### Layouts The layout is the cheapest map from a document to its values: | Layout | When | Section | |---|---|---| | 0 dense | every document has exactly one value | the value sequence; the value index is the document id | | 1 optional | no document has more than one value, some have none | a presence bitmap (`ceil(maxDoc/64)` little-endian `u64`), then the value sequence of the documents that have one; the value index is the number of set bits before the document (the reader keeps a rank per 64 documents) | | 2 multi | some document has several values | an offsets sequence of `maxDoc + 1` integers (value index where each document's run starts; the last is the value count), then the value sequence | A keyword column's section starts with its dictionary: `u32 termCount`, `u32 totalBytes`, `termCount + 1` `u32` offsets into the bytes, then the UTF-8 terms back to back. Terms are distinct and strictly increasing in byte order, so an ordinal order is the keyword order and a keyword is found by binary search. A geo column has a second value sequence (longitude) after the first. ### Integer sequences Offsets and values use one encoding, chosen for the access pattern: a read of one value must not decode a block, and a scan must decode whole blocks fast. ``` u32 count; u32 blockCount; u32 dataLength (includes 8 zero bytes of padding at the end) blockCount records of 24 bytes: u8 kind, u8 width, u16 0, u32 dataOffset, i64 base, i64 aux data ``` Blocks hold 1,024 values (the last holds the rest). Each block is stored in the smallest of three forms: 1. **Constant** (kind 0): every value equals `base`; no data. 2. **Packed** (kind 1): frame of reference with a common divisor. `aux` is the greatest common divisor `g` of the differences from the block minimum (`base`); value = `base + g × u`, with `u` packed in `width` bits (1 to 64), little-endian bit fields. The divisor catches timestamps in whole seconds and prices in cents; the base catches clustered values such as sorted or recent timestamps and the monotone offsets of multi-valued columns. 3. **Table** (kind 2): used when the block has 2 to 256 distinct values and the range is wide (more than 8 bits) and a table is smaller. `aux` is the number of distinct values `n`; data is `n` sorted `i64` followed by the values' indexes in `width = bits(n-1)` bits each. This catches status codes, enum-like values and a few repeated large numbers. Why not delta chaining (each value minus the previous) or entropy coding: both make a random read cost a prefix scan or a variable-length decode. Frame of reference per 1,024 values costs a few bits more and lets `Get(i)` be one record read plus one unaligned 8-byte load, with no allocation. ## Reading The file is read into memory when the segment is opened (bounded by `IndexLimits.MaxResidentFileBytes`) and its checksum is verified. Opening checks the whole structure: the directory tiles the file; section lengths agree; each integer sequence's block records are valid and stay inside their data (kind, width, divisor, table size); a dense column has one value per document; a presence bitmap's population equals the value count; offsets start at 0, never decrease and end at the value count; dictionary offsets span the bytes and terms strictly increase. Anything else raises `CorruptIndexException` naming the file. Keyword ordinals and table indexes are range-checked when read, since checking every value at open would cost a full scan. After open, `HasValue`, `ValueCount`, `GetInt64`, `GetFloat64`, `GetKeyword`, `GetOrdinal`, `GetGeoPoint` and `ReadValues` neither allocate nor await. `ReadValues(firstValue, destination)` decodes a run of raw values in bulk (a dense column's value index is the document id). `GetKeyword` returns a span into the resident file, valid for the life of the reader. Readers are immutable and safe for concurrent use. ## Merge A merge reads each live document's values from the sources through the document map and writes the result with the same code a flush uses. A keyword dictionary is rebuilt from the keywords the surviving documents use (so a deleted document's keywords disappear) and ordinals are remapped. The merged file is byte-for-byte the file a flush of the same documents in the same order produces; a test asserts it. ## Limits Value counts and document counts are 32-bit. A dictionary holds at most 1 GiB of keyword bytes. A column file is bounded by `MaxResidentFileBytes` (1 GiB by default). # Point index This note is normative for the `points` part of a segment: range lookups over numeric, date and bool fields without scanning the documents, and rectangle lookups over geo points ([geo.md](https://recursiveai.org/oneindex/docs/geo/)). Frames and the codec interface are in `commit-format.md` and `segments.md`. ## What is indexed Every field that is indexed and numeric (integral, fractional, date, bool) gets a tree of (value, document) pairs, one pair per value of each document (a multi-valued field contributes several). The value is the same ordered 64-bit integer the query layer uses for range bounds (`NumericTerms.Ordered`): the integer itself for integral and date, 0 or 1 for bool, and `SortableBits.Encode(value + 0.0)` for fractional, so that -0.0 and +0.0 are one value and NaN sorts above every number (a range with an infinite bound leaves NaN out). ## Structure The tree is one-dimensional, so it is the degenerate case of a block KD-tree: the pairs are sorted by (value, document) and cut into **leaves of at most 512 pairs**; the "inner nodes" are the sorted array of leaf minima and maxima, searched by binary search. A simpler structure would not be faster in one dimension, and the leaves are what a second dimension (geo) would keep: a later format version adds a split tree above leaves of the same shape (Procopiuc, Agarwal, Arge and Vitter, "Bkd-tree: a dynamic scalable kd-tree", SSTD 2003, describe the leaf blocks and bulk-built tree this follows). Query cost: a binary search over the directory, then the leaves the range touches. A leaf wholly inside the range contributes its documents without looking at values; only the (at most two) boundary leaves are filtered. A leaf is read from storage when a query needs it; only the directory is resident. ## File, `OIXR` versions 1 and 2 ``` header (8 bytes) magic "OIXR", version 2 (version 1 files, without the kind byte, are still read) u32 maxDoc trees one per field, back to back directory u64 directory offset footer (12 bytes) ``` Directory: `varuint count`, then per field, ordered by name (strictly increasing): `varuint nameLength, name`, `u8 kind` (version 2 only; 0 numeric, 1 geo, others are corrupt), `varuint pairCount`, `i64 minValue`, `i64 maxValue`, `varuint treeOffset`, `varuint treeLength`. Trees tile the file between the `maxDoc` field and the directory. Tree: `u32 leafCount`, `u32 dataLength`, `leafCount` records of 32 bytes, then the leaf data. Leaf record: `i64 min`, `i64 max`, `u32 dataOffset` (from the start of the leaf data; leaves are contiguous), `u16 count` (1 to 512), `u8 valueWidth`, `u8 docWidth`, 8 bytes that are zero in a numeric tree and, in a geo tree, the leaf's bounding cells (four little-endian `u16`: latitude low, latitude high, longitude low, longitude high; the top 16 bits of the quanta, see [geo.md](https://recursiveai.org/oneindex/docs/geo/)). Leaf data: the `count` values minus `min` packed in `valueWidth` bits (the exact number of bits `max - min` needs), then the `count` document ids packed in `docWidth` bits (at most 31). Writers sort by (value, document), so equal inputs give equal bytes however the pairs were gathered; a merge reads the sources' leaves, drops deleted documents, maps the others through the document map and sorts again, giving the bytes a flush of the same documents would. ## Reading `GetRange(lower, upper)` takes inclusive ordered integers (the query layer turns exclusive and open bounds into inclusive ones) and returns a `DocIdSetIterator` over the matching documents in increasing order, each once however many of its values match. The iterator is built when `GetRange` returns: matches are gathered into a sorted array when few (an upper bound is known from the leaf counts) and into a bitmap when many (`bound × 32 ≥ maxDoc`: a bitmap costs `maxDoc/8` bytes, an array 4 bytes per match). Iterating, `NextDoc` and `Advance` allocate nothing; building allocates the array or bitmap and a pooled read buffer. A geo tree holds the Z-order key of a point in the value slot (non-negative, below 2⁶²) and is read with `GetCandidates(south, north, west, east)`, which returns the documents whose points may lie in the rectangle (see geo.md); `GetRange` on it is not meaningful. Opening reads the header, the trailer, the directory and each tree's leaf records, and checks: the footer; the directory tiles the file; leaves are in non-decreasing value order and each leaf's value width is exactly what its range needs; the leaf data lengths add up to the tree length; numeric leaves have zero reserved bytes, geo leaves a valid box (low not above high) and keys below 2⁶²; the pair counts agree with the directory. The file's full checksum is the commit's to check (as for postings); a leaf's document ids are checked against `maxDoc` as they are used. All failures raise `CorruptIndexException` naming the file. ## Limits Document ids and leaf counts are 32-bit. A leaf is 512 pairs, at most about 6 KB. Only the leaf records are resident (32 bytes per 512 pairs). # Stored source This note is normative for the `stored` part of a segment: the original values of the fields the schema stores, compressed in blocks, with random access by document id. Frames and the codec interface are in `commit-format.md` and `segments.md`. ## What is stored Every field whose `Stored` flag is set (the default) is kept, with the value exactly as given: text, keyword, integer, float (every bit), bool, date, geo point, bytes, vector and JSON. Values keep the order the document gave them, so multi-valued fields read back in order. A schema in which no field is stored has no `stored` part; a type opts out of stored source by marking its fields not stored, and a field marked not stored is simply absent from the documents read back. The stored form is also the *source* of a document: `TryGetSource` returns its bytes (the encoding below), and `GetDocument` decodes them into a `Document`. The schema evolution machinery reads documents back through the same reader to rebuild other parts. ## File, `OIXQ` version 1 ``` header (8 bytes) magic "OIXQ", version 1 blocks back to back, each compressed or raw table of contents u64 table offset footer (12 bytes) ``` Table of contents: `u32 documentCount`, `u32 blockCount`, `u8 method` (1 = OneIndex LZ), `varuint fieldCount` and the stored field names (`varuint length`, UTF-8) in schema order, then `blockCount` entries of 20 bytes: `u32 firstDocument`, `u64 fileOffset`, `u32 storedLength`, `u32 rawLength`. A block whose `storedLength` equals `rawLength` is stored raw (the compressor did not shrink it); otherwise it is compressed. The field table is the file's own: records name fields by their index in it, not by schema position, so a file stays readable when the schema later changes. A merge into a different table rewrites the records (below). ### Blocks Documents are appended to a block until its records reach 32 KB (documents are never split, so a larger document is a block of its own). A block holds `n × u32` cumulative record end offsets, then the records. Finding document *d*: binary search the table for its block, read and decompress the block, slice the record. A record is `varuint sourceLength`, the source, `varuint auxLength`, the auxiliary bytes. The auxiliary section is empty in version 1; it is reserved for per-document data kept beside the source, such as cached embeddings, so adding it needs no format change. ### Source encoding `varuint count`, then per value: `varuint fieldIndex`, `u8 kind` (the `FieldValueKind` number), payload: | Kind | Payload | |---|---| | text, JSON | `varuint length`, UTF-8 (JSON as written by `JsonElement.GetRawText`) | | integral, date | zig-zag varint (a date is milliseconds since the epoch) | | fractional | 8 bytes, the IEEE 754 bits, little-endian | | bool | 1 byte | | geo point | 8 + 8 bytes, latitude then longitude bits | | bytes | `varuint length`, the bytes | | vector | `varuint dimensions`, then the 32-bit float bits | ## Compression Blocks use **OneIndex LZ**, a byte-oriented LZ77 written for this format: a token byte (literal length in the high nibble, match length minus 4 in the low nibble, 15 meaning "add the following bytes, each 255 meaning continue"), the literals, a 16-bit match offset (1 to 65,535) and the match length extension; the last token has literals only. Ziv and Lempel (1977) is the idea; the compressor searches a hash chain of up to 16 candidates per position and takes the longest match. Why not the BCL's compression (measured with `bench/OneIndex.Bench`, `StoredCompressionBenchmarks`, and a browser-wasm check): - **Determinism.** Persisted bytes must be the same on every platform. `DeflateStream` and `ZLibStream` output depends on the zlib build beneath the runtime, which differs between runtime versions and platforms. - **Browser.** In a browser-wasm build (checked in headless Chromium and in Node, .NET 10.0), `DeflateStream`, `ZLibStream` and `GZipStream` round-trip, and gave the same compressed bytes as desktop .NET in that check; `BrotliEncoder` and `BrotliDecoder` throw `PlatformNotSupportedException`. So Brotli cannot be used, and deflate's bytes are not guaranteed to match across builds. - **Speed.** A document fetch decompresses a whole block. The numbers are in the section below. Measured on the first 60,000 Wikipedia abstracts (655 blocks of about 32 KB, 21.9 MB of encoded documents; BenchmarkDotNet short runs, in process, on an x64 desktop, indicative only): | Codec | Ratio | Compress all blocks | Decompress all blocks | |---|---|---|---| | OneIndex LZ (16 candidates) | 1.731 | 215 ms | 24 ms | | deflate, fastest | 1.679 | 74 ms | 42 ms | | deflate, optimal | 2.245 | 228 ms | 46 ms | | zlib, optimal | not recorded | 233 ms | 40 ms | | Brotli quality 4 | not recorded | 255 ms | 52 ms | Limiting the search to 1, 4 or 64 candidates gives ratios of 1.606, 1.696 and 1.742. OneIndex LZ decodes about twice as fast as deflate and compresses to within 3% of the fastest deflate level; optimal deflate is 30% smaller but costs the determinism and browser guarantees above. The file records its compression method, so a later method with entropy coding can be added without changing the layout. ## Reading Opening reads the header, trailer and table of contents and validates them: the footer; the document count against the commit; block entries tile the file in order and cover the documents; `storedLength ≤ rawLength ≤` the resident file limit; a compressed block cannot expand beyond 255× (so a small entry cannot claim a huge allocation). A fetch reads the block's range from storage into a pooled buffer, decompresses into another, validates the record table (offsets rise and fill the block) and decodes the record. Decompression refuses anything that is not exactly a valid stream for the recorded size. Every failure raises `CorruptIndexException` naming the file; text that is not UTF-8, JSON that does not parse, and out-of-range dates or coordinates are corruption too. Readers are immutable and safe for concurrent use. A fetch does not cache blocks; merges read sources through a cursor that decompresses each block once. ## Merge Live documents are copied in order. When the source's field table equals the target's, records are copied byte for byte (source and auxiliary section). Otherwise the source part of each record is rewritten to the target's field indexes, values of fields the target does not store are dropped and the auxiliary section is copied unchanged. A merged file equals the file a flush of the same documents produces. ## Limits Block size is 32 KB of records, or one document. A block's raw length is bounded by `IndexLimits.MaxResidentFileBytes`. The table of contents is resident (20 bytes per block). # Dynamic JSON fields A `[Dynamic]` property holds a JSON value whose shape is not known in advance. The whole value is stored with the document and comes back unchanged. Its leaf values are also indexed by path, so documents can be filtered by what is inside the JSON without declaring each key. ```csharp public class Ticket { [SearchId] public string Id { get; set; } = ""; [Dynamic] public JsonElement Fields { get; set; } } index.Search().Where(t => t.Fields.Path("customer.age") >= 18 && t.Fields.Path("tags") == "red"); ``` A `[Dynamic]` property may also be a class of plain properties instead of a `JsonElement`; the generated mapping writes it to JSON and reads it back, as ordinary C# (no reflection, AOT-safe): ```csharp public class Customer { public string Name { get; set; } = ""; public Address? Home { get; set; } public List
Branches { get; set; } = []; } public class Ticket { [SearchId] public string Id { get; set; } = ""; [Dynamic] public Customer Customer { get; set; } = new(); } index.Search().Where("Customer.Home.City = Paris"); ``` The JSON keys are the property names. Supported property types: strings, booleans, integers, floating point (a non-finite value is written as null), `decimal`, `DateTime`, `DateTimeOffset`, `Guid`, nullable versions of those value types, nested classes of plain properties (a type may refer to itself), and arrays or `List`/`IEnumerable`/`IReadOnlyList`/`IList` of those (not of arrays or of nullable values). Only public properties with a public getter and a plain setter take part; the class needs a parameterless constructor. Anything else is a build error (OI0001) naming the property. This is also how an EF Core owned type or JSON column reaches the index. At the low level the field is `FieldType.Json` (`SchemaBuilder.Json(name)`), the value `FieldValue.FromJson`. A JSON field is indexed by default; with `Indexed = false` it is only stored (and can be switched on later, which rebuilds the index from stored source: schema-evolution.md). ## Paths A path is the object keys from the root down, joined by dots: `customer.address.city`. Arrays take no part in the path: the elements of an array count as several values at the same path, at any depth, so `{"tags":["a","b"]}` has two values at `tags` and `{"items":[{"sku":"x"},{"sku":"y"}]}` has two at `items.sku`. A dot inside a key is written `\.` and a backslash `\\`. An empty key and an empty string are not indexed (the keyword index holds no empty term), and neither is a scalar (or an array of scalars) at the root, which has no path; it stays in the stored source. The logical name of a path is the property name, a dot and the path: `Fields.customer.age`. That is the name the filter-string grammar and query JSON use; LINQ takes the property and the path separately (`Fields.Path("customer.age")`). ## What is indexed Each leaf becomes a value of a derived field, named after its path and a letter for the kind: | JSON leaf | Derived field | Notes | |---|---|---| | string | `path#k` keyword | exact match, prefix, fuzzy, regex, phrase of one term, ordering by UTF-8 bytes. Not analyzed: use a declared `[FullText]` property for text search. Strings longer than 1,024 bytes are not indexed. | | string that reads as a date | also `path#d` date | formats `yyyy-MM-dd`, `yyyy-MM-ddTHH:mm[:ss[.fffffff]]` with an optional zone; no zone means UTC | | number | `path#n` fractional | every number is a 64-bit float, so integers above 2^53 lose precision in the index (the stored JSON keeps them). Numbers that are not finite are not indexed. | | `true`, `false` | `path#b` bool | | | `null`, empty arrays and objects | nothing | a null is a missing value | Mixed kinds at one path are fine: a path that holds numbers in some documents and strings in others has a `#n` and a `#k` field, and a query reaches the one that matches the kind of its value. Every derived field is multi-valued and kept in a column (so facets and the filters work), is never stored on its own, and holds each distinct value of a document once. The derived fields are appended to the schema as paths first appear in a flushed segment, ordered by name after the declared fields so that the schema depends on which paths exist, not on the order they arrived in. A segment written before a path appeared has no values for it. Merges handle that like any other absent field. ## Queries A comparison on a path matches a document when **any** value at the path of the kind compared satisfies it: - `== "x"` any string equals x; `== 5` any number equals 5; `== true` any boolean is true; `== date` any string that reads as a date is that instant. - `>`, `>=`, `<`, `<=` the same for numbers, strings (byte order) and dates. Booleans cannot be ordered. - `!=` and `NOT` are the complement of `==`: they also match documents without the path or with only other kinds of value there. - `Exists()` (`EXISTS`, `exists`) the path holds a string, number or boolean. - A path nothing has ever produced matches nothing. `DynamicValue` evaluates the same comparisons on a `JsonElement` directly, so a predicate can be checked against plain objects. The differential test does exactly that: generated documents with mixed kinds per path, arrays and missing paths, every predicate run both ways. Filter strings and query JSON choose the kind from the literal (filter-strings.md, query-json.md); the strict query syntax does not reach paths. ## Limits One limits object (`IndexLimits`) bounds what hostile JSON can do. Leaves over a limit are not indexed; they stay in the stored value and nothing fails. | Limit | Default | Meaning | |---|---|---| | `MaxDynamicDepth` | 16 | nesting of objects and arrays down to a leaf | | `MaxDynamicPaths` | 1,000 | derived fields per JSON field in the index; when full, further paths are not indexed. Candidates in one flush are admitted in name order | | `MaxDynamicPathBytes` | 256 | length of a path | | `MaxDynamicValues` | 10,000 | leaf values indexed per document per JSON field | | `MaxDynamicKeywordBytes` | 1,024 | length of a string indexed as a keyword | | `MaxFields` | 4,096 | the schema as a whole, derived fields included | When the path limit is reached while documents are written in a different order (a threaded rebuild against a cooperative one), which paths made it in can differ; below the limit the result never depends on order. ## Derived fields in the schema The serialized schema lists the derived fields in a section of its own (commit-format.md). Opening an index with the declared schema compares the declared fields only: the derived fields of the committed schema whose JSON field is still declared and indexed are kept, the others are dropped. Switching `Indexed` off on a JSON field removes its derived fields from the schema; the stored value remains. # Relevance: method How retrieval quality is measured on the BEIR sets, with the comparator side first. Speed is in [benchmarks.md](https://recursiveai.org/oneindex/docs/benchmarks/). ## Datasets BEIR test splits of **SciFact**, **ArguAna**, **NQ** and **SCIDOCS**, fetched and hash-checked by `tools/fetch.sh` (BEIR zips; the MD5s are BEIR's own, the SHA-256s are ours; both are in `tools/pins.tsv`). Datasets are never committed. Queries are the test queries that have judgements; judgements are the `qrels/test.tsv` grades. Other BEIR sets are not used. ## Runner `oneindex-harness beir` (`tools/run_relevance.sh`) drives any `ITextEngine`: it indexes the corpus (title and text), retrieves the top 1,000 documents for every test query under each BM25 setting, writes a TREC run file per setting, and scores it. - **Settings.** Three field layouts times two parameter sets, so six runs per dataset: - layouts: `Flat` (title and text concatenated into one field, Kamalloo et al.'s "flat"); `MultiFieldSum` (title and text as two fields, scores summed with equal weights, Kamalloo et al.'s "multifield" and the Anserini configuration the BEIR paper describes); `MultiFieldMax` (the better of the two fields' scores, Elasticsearch's `best_fields`, the BEIR leaderboard's original Elasticsearch run); - parameters: k1=0.9, b=0.4 (Anserini's defaults, the ones the BEIR paper states) and k1=1.2, b=0.75 (Lucene's defaults). - **Analysis.** The engine's English chain: lower-casing, possessive removal, English stop words and Porter-family stemming. Queries are analysed with the same chain and the terms OR-ed within each field. - **nDCG@10.** Our implementation (`TrecEval.NdcgAt`) follows trec_eval: gain is the judged grade, discount 1/log2(rank+1), ranking by score descending with ties broken by document id descending, the mean taken over queries present in both run and judgements, ideal DCG from all positively graded judgements. Two numbers are reported per run: the raw one, and the **BEIR-convention** one, which drops a hit whose document id equals the query id before scoring (BEIR's default; it matters for ArguAna, whose queries are documents of the corpus). - **Cross-check.** `tools/crosscheck_ndcg.py` evaluates the same run files with `pytrec_eval` (`ndcg_cut_10`), the evaluator BEIR uses, and fails if any run differs from ours by more than 5e-5 (agreement to four decimals). ## Published baselines Quoted with their source; "ours" are compared with both. - **BEIR paper:** Thakur, Reimers, Rücklé, Srivastava, Gurevych, *BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models*, NeurIPS 2021 Datasets and Benchmarks, arXiv:2104.08663 (v4), Table 2, BM25 column. The text describes the lexical baseline as Anserini with k1=0.9 and b=0.4, title and passage indexed as separate fields. - **Kamalloo et al.:** *Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard*, arXiv:2306.07471 (SIGIR 2024), Table 3, BM25 columns "multifield" (the BEIR paper's configuration, equal weights over title and contents), "flat" (title and text in one field), both using Lucene's default English analyzer. | Dataset | BEIR paper Table 2 | Kamalloo Table 3 multifield | Kamalloo Table 3 flat | |---|---|---|---| | SciFact | 0.665 | 0.665 | 0.679 | | ArguAna | 0.315 | 0.414 | 0.397 | | NQ | 0.329 | 0.329 | 0.306 | | SCIDOCS | 0.158 | 0.158 | 0.149 | The ArguAna figures differ between the two papers as printed (0.315 and 0.414); we report against both and do not reconcile them. ## Gate OneIndex BM25 nDCG@10 on each set is at most one point below the published baseline (above it is fine) and not below Lucene.NET at matched settings: OneIndex's default k1 and b against Lucene.NET at the same layout, k1 and b. ## Results: Lucene.NET 4.8.0-beta00018 (the comparator side) Engine: Lucene.NET `4.8.0-beta00018` (assembly build `commit:[df99b8abd9]`; packages Lucene.Net, Lucene.Net.Analysis.Common, Lucene.Net.Facet) on .NET 10.0.13 (SDK 10.0.402), Linux x64. Analysis: `EnglishAnalyzer` (lower-case, possessive removal, English stop words, Porter stemming) for documents and queries. Top 1,000 per query, test split. Worker: 32 logical cores (AMD Ryzen 9 7950X3D), NVMe ext4, 8 indexing and search threads. One run covered all four sets and the pytrec_eval cross-check; an earlier attempt produced identical nDCG values. Code at commit `1956a0d`. Datasets (SHA-256 of the zip, MD5 matches BEIR): SciFact `536e1444…0165` (5,183 documents, 300 queries), ArguAna `cfdf79ad…557b` (8,674; 1,406), SCIDOCS `96640201…17ef1` (25,657; 1,000), NQ `2553bf7b…e752` (2,681,468; 3,452). Full hashes: `tools/pins.tsv`. nDCG@10 (BEIR convention in parentheses where it differs; it drops a hit whose id equals the query id): | Run (layout, k1/b) | SciFact | ArguAna | SCIDOCS | NQ | |---|---|---|---|---| | Flat, 0.9/0.4 | 0.6789 | 0.3111 (0.4118) | 0.1509 | 0.3058 | | Flat, 1.2/0.75 | 0.6848 | 0.3642 (0.4882) | 0.1566 | 0.2775 | | MultiFieldSum, 0.9/0.4 | 0.6647 | 0.3217 (0.4240) | 0.1587 | 0.3252 | | MultiFieldSum, 1.2/0.75 | 0.6628 | 0.3632 (0.4880) | 0.1619 | 0.3051 | | MultiFieldMax, 0.9/0.4 | 0.6685 | 0.2877 (0.3784) | 0.1461 | 0.2437 | | MultiFieldMax, 1.2/0.75 | 0.6684 | 0.3414 (0.4584) | 0.1480 | 0.2309 | | **Published: BEIR paper Table 2** | 0.665 | 0.315 | 0.158 | 0.329 | | **Published: Kamalloo Table 3 multifield** | 0.665 | 0.414 | 0.158 | 0.329 | | **Published: Kamalloo Table 3 flat** | 0.679 | 0.397 | 0.149 | 0.306 | Reading: with k1=0.9, b=0.4, Lucene.NET reproduces the Kamalloo flat column on SciFact (0.6789 vs 0.679) and NQ (0.3058 vs 0.306) to three decimals and its multifield column on SciFact (0.6647 vs 0.665), SCIDOCS (0.1587 vs 0.158) and, with the BEIR convention, ArguAna (0.4240 vs 0.414, 1.0 point high). NQ MultiFieldSum is 0.3252 against the published 0.329 (0.4 point low). The BEIR paper's ArguAna value (0.315) is close to the raw, convention-free figure (0.3217), the Kamalloo values to the BEIR-convention figures; the two papers' ArguAna numbers do not agree with each other, so the gate on ArguAna should be read against Kamalloo with the convention applied. `MultiFieldMax` (Elasticsearch best_fields over the two fields) is clearly worst on NQ, where the title field alone matches many irrelevant documents; it is reported for completeness, not as a target. Lucene defaults (1.2/0.75) beat 0.9/0.4 on ArguAna and SCIDOCS and SciFact-flat, 0.9/0.4 on NQ. pytrec_eval 0.5.11 (`ndcg_cut_10`) on the same run files agrees with our nDCG to better than 2e-15 on all 48 run files (raw and BEIR-convention variants of 24 runs): four decimals and far beyond. Indexing time (8 threads, one commit): SciFact 1.8 s, ArguAna 1.0 s, SCIDOCS 1.6 s, NQ 26.8 s (index 1.1 GiB). All six settings for NQ answered 3,452 queries in 15 to 22 s each. ## Results: OneIndex against Lucene.NET and the baselines Run on the worker (32 logical cores, AMD Ryzen 9 7950X3D, NVMe ext4), .NET 10 (SDK 10.0.4xx), 8 threads, same datasets and hashes as above. OneIndex defaults: `Bm25F` layout (one score per term over `title` and `text`, title weight 3), k1 = 0.9, b = 0.4, English analysis (lower-case, possessive removal, Snowball English stop words, Snowball English stemmer). NQ was indexed and searched in separate runs (index phase, then a search phase on the reopened index). Every run file was cross-checked with pytrec_eval (`ndcg_cut_10`); the largest difference to our nDCG@10 is 1.8e-15. nDCG@10, BEIR convention (a hit whose id equals the query id is dropped): | Set | OneIndex `Bm25F` 0.9/0.4 (default) | OneIndex `Bm25F` 1.2/0.75 | Lucene.NET `MultiFieldSum` 0.9/0.4 | Lucene.NET `MultiFieldSum` 1.2/0.75 | Lucene.NET `Flat` 0.9/0.4 | Published baseline (multifield) | |---|---|---|---|---|---|---| | SciFact | **0.6912** | 0.6998 | 0.6647 | 0.6628 | 0.6789 | 0.665 | | ArguAna | **0.4333** | 0.5072 | 0.4240 | 0.4880 | 0.4118 | 0.414 | | SCIDOCS | **0.1531** | 0.1601 | 0.1587 | 0.1619 | 0.1509 | 0.158 | | NQ | **0.3420** | 0.3377 | 0.3252 | 0.3051 | 0.3058 | 0.329 | SciFact (2.6 points) and ArguAna (1.9) are more than one point *above* the baseline, NQ is 1.3 above; SCIDOCS is 0.2 below it. The Lucene.NET figures are those of the section above (re-run once more in the equal-weight job; identical to four decimals). OneIndex with the same layout as Lucene.NET's `MultiFieldSum` (two separate field scores added, title weight 1, so no BM25F): | Set | OneIndex `MultiFieldSum`, title weight 1, 0.9/0.4 | Lucene.NET `MultiFieldSum` 0.9/0.4 | OneIndex, 1.2/0.75 | Lucene.NET, 1.2/0.75 | |---|---|---|---|---| | SciFact | 0.6598 | 0.6647 | 0.6653 | 0.6628 | | ArguAna | 0.4270 | 0.4240 | 0.4861 | 0.4880 | | SCIDOCS | 0.1587 | 0.1587 | 0.1641 | 0.1619 | | NQ | 0.3323 | 0.3252 | 0.3114 | 0.3051 | At matched layout and parameters OneIndex's BM25 scores equal Lucene.NET's within noise (identical on SCIDOCS at 0.9/0.4, above on ArguAna and NQ, 0.5 point below on SciFact). ### SCIDOCS against Lucene.NET `MultiFieldSum` `Bm25F` (0.1531) is 0.56 points below Lucene.NET's `MultiFieldSum` (0.1587) on SCIDOCS, and above its `Flat` layout (0.1509). Measured causes: - **It is the layout, not the scoring or the analysis.** The equal-weight sum in OneIndex reproduces Lucene.NET to four decimals on SCIDOCS (0.1587 / 0.1587). Lucene.NET's `Explain` output (public API) shows the same idf and tf formula as OneIndex's: idf `ln(1 + (N - n + 0.5)/(n + 0.5))`, `tf (k1+1) / (tf + k1 (1 - b + b len/avgLen))`. - **Analysis is not it.** Experimental builds (not kept) with Lucene's 33-word stop list and the original Porter stemmer moved the sum layout by at most 0.1 point either way (0.1552 default, 0.1548 Lucene stop list, 0.1551 Porter, 0.1543 both), while the title weight moved it 0.35 (the harness had given the sum layout OneIndex's title weight 3 until this run; at weight 1 it is 0.1587). With the same stop list the two analyzers' token streams differ in 20 of 6,000 sampled texts (stems of short words); with each side's own list they differ in 5,286, almost all by stop words. - **Collection size and average length are not it.** Counting empty-field documents in N and average length (two variants, run on all three small sets) changed SCIDOCS `Bm25F` by at most 0.04 points. - **What decides it:** SCIDOCS (queries are paper titles, relevant documents are cited papers) rewards a document whose title *and* abstract both match, which adding two saturating field scores does and BM25F (one saturation over the weighted sum) does not. Title weight sweep, `Bm25F` 0.9/0.4: | Title weight | SciFact | ArguAna | SCIDOCS | |---|---|---|---| | 1 | 0.6838 | 0.4188 | 0.1502 | | 1.5 | 0.6878 | 0.4248 | 0.1517 | | 2 | 0.6872 | n/a | 0.1520 | | 3 (default) | 0.6912 | 0.4333 | 0.1531 | No weight reaches 0.1587 on SCIDOCS while all four sets keep their other results; the sum layout wins on SCIDOCS and loses 3 points on SciFact and 1 on NQ against `Bm25F`. BM25F is the engine's documented base score, so it stays the default; the sum layout is a query composition (`BooleanQuery` of per-field `TermQuery`) any caller can write. ### Operational notes - Loading NQ takes 173 s at 8 threads (index 746 MiB; Lucene.NET: 26.8 s, 1.1 GiB), commit and merges included; `IndexWriter.AddUnique` skips the per-document identifier lookup that made the first attempt time out at 30 minutes. Reopening the finished index and reading 2.68 M identifiers takes 36 s; the 3,452 queries take 12 to 25 s per setting. - The runner takes `ONEINDEX_PHASE=index|search` (build and commit in one job, search in another), `ONEINDEX_LAYOUTS`, `ONEINDEX_WORK` and the OneIndex adapter `ONEINDEX_TITLE_WEIGHT`. ## Typo tolerance on (information) `ONEINDEX_TYPO=on` makes the OneIndex adapter match the Bm25F layout with `TypoTermQuery` (default typo budgets on the analyzed word, the typo and exactness factors, no prefix matching). It is information only; the gated runs leave it off, where queries are exactly the plain BM25F term queries. ## Term proximity bonus (default weight 0.1) The bonus of docs/scoring.md ("Term proximity") on the gate sets, by weight. The table and its reading are in that section; the default keeps every set at or above the no-bonus numbers of the table above (within noise on ArguAna at 1.2/0.75) and adds 0.1 to 0.3 points at the default BM25 setting. ## Milestone re-run The gate sets re-run on the code of the first public version. nDCG@10 under the BEIR convention (a hit whose id equals the query id is dropped); our runner, cross-checked with `pytrec_eval` (largest difference 2.2e-15). Lucene.NET is 4.8.0-beta00018 with matched analysis. The BM25 numbers equal the recorded ones except the proximity default (weight 0.1 on user text, described above and in docs/scoring.md). ### BM25 at k1 = 0.9, b = 0.4 | Set | OneIndex default (BM25F + proximity 0.1) | OneIndex without the bonus | Lucene.NET Flat | Lucene.NET MultiFieldSum | Baseline (multifield) | Default minus baseline | |---|---|---|---|---|---|---| | SciFact | 0.6918 | 0.6912 | 0.6789 | 0.6647 | 0.665 | +2.7 | | ArguAna | 0.4347 | 0.4333 | 0.4118 | 0.4240 | 0.414 | +2.1 | | SCIDOCS | 0.1544 | 0.1531 | 0.1509 | 0.1587 | 0.158 | -0.4 | | NQ | 0.3452 | 0.3420 | 0.3058 | 0.3252 | 0.329 | +1.6 | ArguAna without the BEIR own-hit convention: 0.3260 (OneIndex) against 0.3111 (Lucene.NET Flat). Every set is within one point of the baseline or above it, and at or above Lucene.NET Flat. SCIDOCS is read against Flat, as ruled (BM25F is a single-stream model); it sits 0.4 below the per-field-sum layout, which loses on SciFact and NQ when OneIndex runs it (below). At k1 = 1.2, b = 0.75 (SciFact / ArguAna / SCIDOCS / NQ): | Run | SciFact | ArguAna | SCIDOCS | NQ | |---|---|---|---|---| | OneIndex default | 0.7033 | 0.5067 | 0.1615 | 0.3412 | | OneIndex without the bonus | 0.6998 | 0.5072 | 0.1601 | 0.3377 | | Lucene.NET Flat | 0.6848 | 0.4882 | 0.1566 | 0.2775 | | Lucene.NET MultiFieldSum | 0.6628 | 0.4880 | 0.1619 | 0.3051 | Other layouts, 0.9/0.4 then 1.2/0.75 (SciFact / ArguAna / SCIDOCS / NQ): OneIndex Flat 0.6822 / 0.4181 / 0.1513 / 0.3148 then 0.6891 / 0.4930 / 0.1568 / 0.2891; OneIndex MultiFieldSum 0.6452 / 0.4272 / 0.1552 / 0.3228 then 0.6456 / 0.4818 / 0.1592 / 0.3042; Lucene.NET MultiFieldMax 0.6685 / 0.3784 / 0.1461 / 0.2437 then 0.6684 / 0.4584 / 0.1480 / 0.2309. ### Dense and hybrid Typed BM25 arm, bge-small dense arm, default hybrid (weighted RRF, rank constant 10, semantic ratio 0.75). | Set | BM25 | Dense | Hybrid | |---|---|---|---| | SciFact | 0.6912 | 0.7126 | 0.7257 | | ArguAna | 0.4333 | 0.6048 | 0.6100 | | SCIDOCS | 0.1531 | 0.2041 | 0.2064 | | NQ (recorded, not re-run) | 0.3420 | 0.5018 | 0.5249 | | Mean | 0.4049 | 0.5058 | 0.5168 | Hybrid is above both arms on every set and on the mean. NQ needs GPU-embedded vectors, which this run did not use, so its row is the recorded one (held-out confirmation). Other fusion settings reproduce the recorded numbers to four decimals on the three CPU sets (for example SciFact RRF k 60 ratio 0.5 gives 0.7282 and linear 0.5 gives 0.7377). ### `.Models` rows | Model | SciFact | ArguAna | SCIDOCS | NQ | State | |---|---|---|---|---|---| | bge-small dense (int8) | 0.7126 | 0.6048 | 0.2041 | 0.5018 | three sets re-embedded on CPU this run; NQ recorded | | multilingual-e5-small (int8) | 0.6782 | 0.3859 | 0.1375 | | re-embedded on CPU this run; card 0.67704 / 0.39063 / 0.13895 | | cross-encoder rerank of BM25 | 0.6890 | 0.4164 (0.3112 with own hits kept) | 0.1646 | 0.5404 | scores reused from the cache (the run scored 0 pairs); NQ recorded | | OpenSearch sparse | | | | 0.5432 | recorded, not re-run | What stands as recorded: all NQ model rows, the sparse encoder, and the cross-encoder scores (the cache was valid; the pair scores were not recomputed). # Benchmarks: method How the text rows are measured against Lucene.NET and the native engines. The relevance method is in [relevance.md](https://recursiveai.org/oneindex/docs/relevance/). The published rows are in the milestone section below, from exclusive benchmark runs on the machines listed in the machine record, and every number carries that record. Other sections, including "Earlier results" at the end, record how the method developed and are history, not published results. ## What is measured All rows run over the **Wikipedia abstracts**: the first paragraph of each article of the English Wikipedia 20231101 dump, shards 00000 and 00001 of the Hugging Face `wikimedia/wikipedia` dataset (CC BY-SA 4.0 / GFDL), built by `tools/wikipedia_abstracts.py` into `abstracts-20231101.en-00000-00001.jsonl`. Each line also carries two derived columns: `len` (abstract length in characters, numeric) and `cat` (first letter of the title, keyword, naturally skewed). A run indexes the first N abstracts (50,000 for an indicative pass). **Matched analysis.** Standard word tokenization, lower-casing, no stop words, no stemming, on every engine. Lucene.NET uses `StandardAnalyzer` with an empty stop set; tantivy its `default` tokenizer; FTS5 `unicode61`. Only the body text is indexed (title is not searched). **Matched commits.** One bulk commit after the first N-1000 documents, then five commits each after 200 further documents. The write buffer is 256 MiB; merge policy and scheduler are the engine's defaults. Documents are added by one thread unless a run says otherwise. Query rows run after the last commit, single-threaded, with the index opened once. **Rows.** | Row | Query | Engines | |---|---|---| | index | documents per second including the bulk commit, allocated bytes per document | all | | commit | median and maximum latency of the five small commits | all | | Terms1 / Terms2 / Terms5 | BM25 top 10 over a disjunction of 1, 2 or 5 terms | all | | Phrase | two adjacent terms as an exact phrase | all | | Prefix | a three-letter prefix | all | | AsYouType | one full term plus a partial last word | all | | Fuzzy1 / Fuzzy2 | one term with one / two edits, edit distance 1 / 2 | Lucene.NET, tantivy | | Typo | one word of 6+ characters with one substituted letter (not the first), default typo budgets | Lucene.NET (FuzzyQuery, first character fixed) | | TypoAsYouType | one full term plus a typed 6-letter beginning with a substituted letter, matched as a beginning with typos | Lucene.NET (prefix OR fuzzy: Lucene has no Levenshtein over word beginnings, so its side matches less) | | Filtered | two terms, numeric range on `len` and three categories | Lucene.NET, tantivy | | Facets | two terms, top ten `cat` counts over all matches | Lucene.NET | | Stats | two terms, count/min/max/sum of `len` over all matches | Lucene.NET | | Sort | two terms ordered by `len` descending | Lucene.NET | | FunctionScore | two terms, BM25 times a Gaussian decay of `len` (origin the median, scale the median, 0.5 at the scale), top 10 by the product | Lucene.NET, through a collector over the `len` doc values (its public API's way to a function score; every match is visited in both engines) | **Queries** come from `QueryGenerator`, seeded (seed 1 by default) and drawn from the corpus itself: terms occur in at least 5 and at most a fifth of the documents, phrases are adjacent words of one abstract, fuzzy terms are corpus terms with substitutions or a deletion. The same seed gives byte-identical query sets for every engine, and `oneindex-harness queries` writes them as JSON for the native runners. Each row has 200 queries. `matchChecksum` (the sum of match counts, hit counts and facet counts) lets two engines with matched analysis be compared for agreement. ## Procedure 1. **Corpus driver** (`oneindex-harness driver`, `tools/run_bench_sample.sh`): builds the index, then times each row: one warm-up pass, then three timed passes of the whole query set; the **median** pass gives milliseconds per query. It records, per row, the allocated bytes per query (managed allocations of the process), the **peak resident set** (Linux: the kernel's high-water mark, reset before each row through `/proc/self/clear_refs`; elsewhere the process peak since start) and the **index size** (sum of file lengths). Process-wide peak memory includes the loaded corpus and query sets, which is the same for every engine. 2. **BenchmarkDotNet suite** (`bench/OneIndex.Bench`, `TextRows`, `[MemoryDiagnoser]`): one benchmark per query row, one invocation answers the full query set (`OperationsPerInvoke`). Parameterised by the engines that can run. Environment: `ONEINDEX_BENCH_CORPUS`, `ONEINDEX_BENCH_DOCS`, `ONEINDEX_BENCH_WORK`. Use `--filter '*TextRows*' --inProcess` to avoid the auto-generated build project; drop `--job short` for publishable runs. 3. **Native, information only:** `tools/native/tantivy_rows.py` (tantivy-py in the oracle venv) and `tools/native/fts5_rows.sh` (the pinned `sqlite3` command-line tool) run the same queries. The FTS5 per-query time includes a small process start-up share because each row is one `sqlite3` invocation. Neither is a gate; the target is within 1.5x of them. OneIndex is driven through the same `ITextEngine` adapter (`OneIndexEngine`). On the benchmark profile (body text with standard analysis, the `len` integral column and the `cat` keyword facet column) it answers the term, Facets and Stats rows; the others are reported as skipped ("unsupported by engine") until their query types exist. Facets: the top ten `cat` counts by frequency over all matches together with the top page, in one pass; Stats: count, minimum, maximum and sum of `len` over all matches. The Facets and Stats `matchChecksum` values agree with Lucene.NET's, which is how the two are checked to compute the same thing. A Terms row's checksum differs because OneIndex prunes with block-max WAND and reports a lower-bound total. ## The machine record Every result file carries (in `machine`, `loadBefore`, `loadDuring`, `loadAfter`, `pageCache`): - CPU model, logical core count, instruction sets the runtime reports (SSE4.2, AVX, AVX2, FMA, BMI2, AVX-512, NEON, CRC32, dot product), operating system, total memory; - storage: file system type, mount, size and free space of the index directory's volume (and whether it is a local disk); - .NET SDK and runtime versions, GC mode; - comparator versions (engine name and version in the result, the Python and tool versions of the native runs in their output); - dataset: the abstracts file's SHA-256 (from `tools/fetch.sh`) and the number of documents; - load averages (1, 5, 15 minutes) before the run, after each row and after the run; - page cache: the index is written by the same process, so reads are served from the OS cache ("warm"). A cold-cache run states how the cache was dropped. Publishable runs are exclusive: nothing else runs on the machine, which the record's load figures must show. Timings taken on a shared or loaded host, or on a development machine, are indicative only and are labelled so. ## Gates (for reference) At most 1.0x Lucene.NET's time as a geometric mean over the rows, no row worse than 1.25x; memory and index size are reported next to every row. Rows with no managed comparator pass, with the reason recorded. ## Milestone rows (final code of the first public version) The numbers a release quotes. Method: [Procedure](#procedure) with the scripts `tools/bench_text_run.sh` (text rows), `tools/vector_bench_group.sh` and `tools/vector_ratios.py` (vector rows) and `tools/bench_summary.py` (the tables below); every group ran as an exclusive benchmark-class job on its machine (the queue starts it only when nothing else runs and waits for the CPU to settle), at most 20 minutes each, one engine after another. **Machines.** x64: AMD Ryzen 9 7950X3D, 16 cores / 32 threads (SSE4.2, AVX, AVX2, FMA, BMI2, AVX-512 F/BW), NVMe, Linux, .NET SDK 10.0.402 / runtime 10.0.13, workstation GC. arm64: Apple M2, 8 cores, APFS, macOS, .NET SDK 10.0.403 / runtime 10.0.14. Pins: Lucene.NET 4.8.0-beta00018, Hnsw.Net 0.1.4, curiosity-ai HNSW 26.9.5494; corpus SHA-256 `1759e651...4893` (302,567 abstracts), seed 1, 200 queries per row. Indexes were written by the running process, so reads came from the page cache (warm). **Load.** One-minute load average before / after the runs (the queue waits for the CPU to be idle, so these are the decay of earlier work): x64 text repetitions 8.3 / 5.8, 3.0 / 2.4 and 2.7 / 2.5 for OneIndex, 5.8 / 4.1, 2.4 / 2.3 and 2.5 / 2.1 for Lucene.NET; vector groups 1.9 / 1.5 (384) and 1.4 / 1.6 (768). arm64 text repetitions 11.4 / 8.5, 4.4 / 3.6 and 2.7 / 2.6 for OneIndex, 8.5 / 6.5, 3.6 / 3.0 and 2.6 / 2.4 for Lucene.NET (the first repetition started right after an unrelated job, and its figures are not different from the other two beyond noise); vector groups 2.9 / 2.5 and 2.1 / 1.8. A Mac's baseline load is 2 to 4. ### Text rows: OneIndex against Lucene.NET Three repetitions of the corpus driver per engine, interleaved (OneIndex, Lucene.NET, the matched as-you-type typo run), each driver run a warm-up plus three timed passes; the table gives the median of the three medians. Ratios are ours over theirs in time (index: theirs over ours in throughput), so below 1.0 is faster than Lucene.NET. | Row | OneIndex x64 | Lucene.NET x64 | ratio | OneIndex arm64 | Lucene.NET arm64 | ratio | |---|---|---|---|---|---|---| | index (docs/s, incl. bulk commit) | 60,590 | 37,232 | 0.61 | 64,544 | 38,080 | 0.59 | | commit (median ms, 5 x 200 docs) | 30.7 | 50.3 | 0.61 | 13.0 | 15.5 | 0.84 | | Terms1 (ms/query) | 0.131 | 0.246 | 0.53 | 0.183 | 0.320 | 0.57 | | Terms2 | 0.196 | 0.450 | 0.43 | 0.221 | 0.503 | 0.44 | | Terms5 | 0.525 | 0.636 | 0.82 | 0.631 | 0.690 | 0.91 | | Phrase | 0.190 | 0.162 | 1.17 | 0.245 | 0.341 | 0.72 | | Prefix | 0.111 | 0.330 | 0.34 | 0.121 | 0.387 | 0.31 | | AsYouType | 0.158 | 0.203 | 0.78 | 0.222 | 0.241 | 0.92 | | Fuzzy1 | 0.255 | 0.553 | 0.46 | 0.284 | 0.873 | 0.33 | | Fuzzy2 | 1.298 | 2.138 | 0.61 | 1.433 | 3.909 | 0.37 | | Typo | 0.220 | 0.506 | 0.43 | 0.238 | 0.665 | 0.36 | | TypoAsYouType (matched semantics; gates) | 0.084 | 0.184 | 0.46 | 0.087 | 0.257 | 0.34 | | TypoAsYouType (documented semantics; information) | 0.256 | 0.184 | 1.39 | 0.294 | 0.257 | 1.14 | | Filtered | 1.626 | 1.914 | 0.85 | 1.892 | 2.297 | 0.82 | | Facets | 0.458 | 0.602 | 0.76 | 0.507 | 0.701 | 0.72 | | Stats | 0.288 | 0.543 | 0.53 | 0.319 | 9.731 | 0.03 | | Sort | 0.379 | 0.673 | 0.56 | 0.438 | 11.847 | 0.04 | | FunctionScore | 0.961 | 0.841 | 1.14 | 1.157 | 1.114 | 1.04 | **Geometric mean** of the 17 gating rows: x64 **0.62**, arm64 **0.41** (0.57 without Stats and Sort, where Lucene.NET's arm64 times are outliers). The bar (at most 1.0) is met on both machines. **No row is above 1.25**; the highest are Phrase on x64 (1.17) and FunctionScore on x64 (1.14) and arm64 (1.04). Phrase is a per-segment cost (the settled index has six segments; see "Phrase on x64 and the segment count" below). FunctionScore visits every match in both engines; OneIndex evaluates the decay through a delegate over the column. The documented-semantics as-you-type typo row matches the beginning of a word with typos (prefix Levenshtein); Lucene.NET has no counterpart, so its query matches fewer documents and the comparison is information (1.39x on x64, 1.14x on arm64; the matched row gates). | Memory and size (x64) | OneIndex | Lucene.NET | |---|---|---| | index size after the settled merges (bytes) | 60,589,958 | 45,426,027 | | index size after the bulk commit (bytes) | 70,161,632 | 45,094,841 | | peak resident set, index row / query rows (MiB; includes the 120 MB corpus held by the driver) | 1,394 / 500 to 1,153 | 1,027 / 434 to 603 | | allocated bytes per document, index row | 5,461 | 18,120 | | allocated bytes per query: Terms1 / Terms2 / Terms5 | 4,918 / 9,185 / 21,054 | 59,203 / 805,364 / 970,469 | | Phrase / Prefix / AsYouType / Fuzzy1 / Fuzzy2 / Typo | 9,377 / 62,868 / 57,659 / 19,456 / 120,550 / 19,311 | 210,850 / 153,187 / 189,797 / 703,984 / 2,281,455 / 1,244,174 | | Filtered / Facets / Stats / Sort / FunctionScore | 61,684 / 23,414 / 8,990 / 12,743 / 9,318 | 650,116 / 856,043 / 799,404 / 806,659 / 604,315 | The index is larger because OneIndex stores each document's source and keeps an identifier table and a point index for the numeric column. Peak resident set was not captured on arm64 (the file-backed index rows are the same sizes there). Match checksums: Phrase, AsYouType, TypoAsYouType (matched) and Filtered agree exactly; Terms1, Facets, Stats, Sort and FunctionScore agree to within 2 to 9 documents (the U+202F tokenization difference below); the other rows differ by design (block-max WAND reports a lower bound, prefix queries are constant-score, fuzzy expansion is limited). The FunctionScore checksum is 3,044,177 against 3,044,186 on both machines (the same on x64 and arm64). The checksum is the sum of match counts, not of scores or ranks, and the row visits every match in both engines, so the nine-document gap is a match-count difference of the kind the other rows show (Terms1 2, Facets 3, Stats 3, Sort 2): the comparator counts documents whose query term sits next to a U+202F in its older tables. It is not a float tie at the top-k cutoff or a decay rounding difference: those cannot change a match count. This was read from the existing summaries, with no new run; the top-10 lists were not compared. ### Vector rows: OneIndex against the faster managed comparator at matched recall `tools/vector_bench_group.sh`, 150,000 x 384 (bge-small) and 75,000 x 768 (bge-base) vectors of the abstracts, M=16, efConstruction=100, one thread, recall@10 against exact search, ef on a grid, queries per second the median of three passes, queries per second linearly interpolated to the target recall (`tools/vector_ratios.py`). Ratio is ours over the faster comparator's throughput: above 1.0 is faster. Hnsw.Net is the faster comparator at every target both reach. | Row | x64 ours / Hnsw.Net (qps) | ratio | arm64 ours / Hnsw.Net (qps) | ratio | |---|---|---|---|---| | 384 @0.90 | 1586 / 1412 | 1.12 | 1433 / 1239 | 1.16 | | 384 @0.95 | 745 / 597 | 1.25 | 678 / 549 | 1.23 | | 384 @0.99 | not reached by either (ours 0.9839, Hnsw.Net 0.9806 at ef 1300) | | same | | | 384 best both reach (0.9806) | 311 / 260 | 1.20 | 284 / 242 | 1.17 | | 768 @0.90 | 1701 / 1455 | 1.17 | 1532 / 1361 | 1.13 | | 768 @0.95 | 831 / 681 | 1.22 | 748 / 647 | 1.16 | | 768 @0.99 | 241 / 207 | 1.16 | 216 / 207 | 1.04 | | build 384 (s; ours / Hnsw.Net) | 76.7 / 81.9 | 0.94 | 82.4 / 109.4 | 0.75 | | build 768 | 57.3 / 60.3 | 0.95 | 53.5 / 86.5 | 0.62 | curiosity-ai HNSW reaches 0.90 only at ef 900 (384: 377 qps x64, 307 arm64) and 450 (768: 518 qps on x64; on arm64 ef 640, 331 qps), and never reaches 0.95 on 384 (best 0.9318 x64, 0.9269 arm64) nor 0.99 on either; at 768 it reaches 0.95 only at ef 1300 on x64 (213 qps against ours 831). It builds faster than ours (ours / curiosity: 1.57 and 1.79 on x64, 1.32 and 1.18 on arm64) but is far slower at matched recall, so Hnsw.Net is the comparator that gates and the build ratio against curiosity is information. Peak resident set while building (x64): ours 512 MB (384) and 505 MB (768), Hnsw.Net 592 and 578 MB, curiosity 678 and 679 MB; the index files are 250.6 MB (384) and 240.5 MB (768) for ours, 253.4 and 241.9 MB for Hnsw.Net. Everything is at or above 1.0 in throughput and at or below 1.0 in build time against Hnsw.Net. Geometric mean of the throughput ratios (the six reached rows, with the 384 best-both-reach row): x64 1.19, arm64 1.15. ### Native engines (information only) Run as ordinary shared-queue jobs on a busy x64 machine (one-minute load 13 to 26 before and after the runs) and labelled indicative: they are the native libraries run through their Python bindings or command-line tool, not gates, and their absolute numbers move with the load. tantivy-py 0.27.0 (Python binding overhead included), SQLite 3.54.1 FTS5 (each row one `sqlite3` invocation, so a process start-up share is in every figure), hnswlib 0.8.0 and faiss-cpu 1.15.1. | Row (ms/query unless noted) | OneIndex (exclusive run) | tantivy-py | SQLite FTS5 | |---|---|---|---| | index (docs/s) | 60,590 | 95,223 | 73,914 (4.09 s) | | Terms1 / Terms2 / Terms5 | 0.131 / 0.196 / 0.525 | 0.053 / 0.112 / 0.265 | 4.57 / 10.5 / 28.9 | | Phrase | 0.190 | 0.060 | 0.72 | | Prefix / AsYouType | 0.111 / 0.158 | 0.303 / 0.117 | 18.4 / 2.70 | | Fuzzy1 / Fuzzy2 | 0.255 / 1.298 | 0.887 / 2.905 | n/a | | Filtered | 1.626 | 0.459 | n/a | | index size (bytes) | 60.6 M settled | 49.0 M | 156.7 M | Against tantivy-py OneIndex is 2.0 to 3.2 times slower on Terms and Phrase and faster on Prefix, Fuzzy and Filtered here (the 1.5x target is for information and is missed on Terms1 to Terms5 and Phrase; the loaded host and the Python overhead make the comparison coarse). Vectors, hnswlib / faiss (recall at ef 270 / 640 and best throughput near 0.90 / 0.95): 384: hnswlib 0.9069 / 0.9534 (765 / 498 qps), faiss 0.9120 at ef 270; 768: hnswlib 0.9047 at ef 180 (1105 qps), 0.9579 at ef 450 (505), 0.9910 at ef 1800 (201); faiss 0.9072 at ef 180 (1279), 0.9505 at ef 380 (740), 0.9902 at ef 1800 (199). Build: hnswlib 46.5 s (384), faiss 48.2 s (384) and 26.2 s (768), OneIndex 76.7 s and 57.3 s on an idle machine (both libraries are native, with the Python binding's own overhead around the calls, and ran loaded, so none of this is a gate). ## Earlier results (superseded by the milestone rows above) ### Text rows, 302,567 abstracts All Wikipedia abstracts of the corpus above, one run per machine of the corpus driver (`tools/run_bench_sample.sh`, rows `Terms1,Terms2,Terms5,Phrase,Filtered,Facets,Stats`, 200 queries per row, warm-up plus three timed passes, median), run alone on the machine as exclusive benchmark jobs; OneIndex first, then Lucene.NET 4.8.0-beta00018, same corpus file (sha256 1759e651...4893) and seed 1. The multi-term rows (Prefix, AsYouType, Fuzzy1, Fuzzy2) are measured with the typo-tolerance work and are not in this table. `bench/OneIndex.Bench` also has `TermDictionaryRows`: term expansion over a dictionary alone (set `ONEINDEX_BENCH_TERMS` to a file of `termdocument frequency` lines in byte order; `ONEINDEX_TERMS_QUICK=1` prints a stopwatch pass with a checksum of what matched, `ONEINDEX_TERMS_VERSION=1` builds the older node layout for comparison). It isolates the dictionary walk from postings and scoring. The write buffer is 256 MiB for Lucene.NET and 20,000 documents for OneIndex; both add documents from one thread with the default merge policy. OneIndex adds with the bulk `AddUnique` path (identifiers are unique), as Lucene.NET's `AddDocument` does not look for a previous version either. The harness runs OneIndex's merges on a background thread (`ThreadedMergeScheduler`), as Lucene.NET's default scheduler does, so a commit does not wait for the merges it starts. A finished merge is installed by the next commit, so before the query rows the harness commits until no merge installs (the settled index has the 200,000-document segment, five of 20,000, and the small segments of the commit row). The library's own default scheduler is cooperative (it needs no threads): merges then run inside the commit that polls them, which keeps results identical in the browser; applications that want Lucene.NET's behaviour pass a `ThreadedMergeScheduler`. | Row | OneIndex x64 | Lucene.NET x64 | ratio | OneIndex arm64 | Lucene.NET arm64 | ratio | |---|---|---|---|---|---|---| | index (docs/s, incl. bulk commit) | 61,135 | 35,923 | 0.59 | 63,859 | 37,849 | 0.59 | | commit (median ms, 5 x 200 docs) | 28.3 | 48.1 | 0.59 | 21.7 | 14.4 | **1.50** | | commit (max ms) | 35.1 | 54.6 | 0.64 | 25.2 | 14.6 | 1.73 | | Terms1 (ms/query) | 0.142 | 0.323 | 0.44 | 0.182 | 0.351 | 0.52 | | Terms2 | 0.227 | 0.463 | 0.49 | 0.262 | 0.405 | 0.65 | | Terms5 | 0.658 | 0.642 | 1.03 | 0.771 | 0.696 | 1.11 | | Phrase | 0.208 | 0.168 | 1.24 | 0.248 | 0.414 | 0.60 | | Filtered | 1.636 | 1.854 | 0.88 | 1.782 | 2.185 | 0.82 | | Facets | 0.611 | 0.677 | 0.90 | 0.723 | 0.755 | 0.96 | | Stats | 0.369 | 0.612 | 0.60 | 0.405 | 10.622 | 0.04 | Geometric mean of the time ratios (index inverted): x64 0.71; arm64 0.57, or 0.79 without Stats, where Lucene.NET's arm64 figure is an outlier. The geometric-mean bar (at most 1.0) is met on both machines. **Rows above 1.25:** the commit row on arm64 (1.50). Phrase on x64 is 1.24, just inside; earlier runs of the same code on a busier host read 1.2 to 1.4, so it sits on the line. | Memory and size | OneIndex | Lucene.NET | |---|---|---| | index size after the settled merges (bytes) | 64,321,449 | 45,426,027 | | peak resident set, index row, x64 (MiB; includes the 120 MB corpus held by the driver) | 1,253 | 1,023 | | allocated bytes per document, index row | 4,831 | 18,117 | | allocated bytes per query: Terms1 / Terms2 / Terms5 | 41,554 / 81,259 / 200,141 | 59,203 / 805,364 / 970,469 | | allocated bytes per query: Phrase / Filtered / Facets / Stats | 72,174 / 265,172 / 104,644 / 82,503 | 210,813 / 649,518 / 856,765 / 801,275 | The index is larger because OneIndex also stores each document's source and keeps a per-document identifier table and a point index for the numeric column; Lucene.NET stores only the identifier. Phrase and Filtered match checksums agree exactly. Facets and Stats agree to within 2 and 4 documents for the tokenization reason in the next section. Terms rows differ by design: OneIndex prunes with block-max WAND and reports a lower bound. **Commit on arm64.** A small commit used to write the buffered documents as a new segment of seven files (text, column, point, stored, identifier and length parts) and fsync them together with the commit file: 8 new files and 8 fsyncs per commit against 4 files for Lucene.NET (a compound `.cfs`, `.cfe`, `.si` and the segments file). On an M2 under APFS one flush-to-disk of a 30 KB file measured about 2.9 ms, eight in sequence 22 ms (7 ms in parallel), so the cost of a commit there is the number of files times the cost of making each durable, and the x64 NVMe disk hides it. A flush now writes its text, column, point, stored and identifier parts into one pack file (commit format 3, docs/commit-format.md), so a small commit adds two files: the pack and the commit file. Durability is unchanged: the same files are fsynced before the commit file is renamed, and the pack is checksummed like any segment file. Fsyncing the files one after another instead of together did not help (23.7 to 31.2 ms against 30.0 to 31.8 ms, interleaved, noisy). Final rows for that change, from the corpus driver on 50,000 abstracts (one bulk commit of 49,000, five commits of 200), three runs per engine interleaved on each machine, the median of the three medians, run alone under the benchmark queue. "Before" is the same harness on the code before the pack file. | Row | machine | OneIndex now | OneIndex before | Lucene.NET | ratio now | ratio before | |---|---|---|---|---|---|---| | commit (median ms) | x64 | 23.5 | 30.3 | 40.7 | 0.58 | 0.74 | | commit (median ms) | arm64 | 19.5 | 23.7 | 16.5 | **1.18** | 1.43 | | index (docs/s, incl. bulk commit) | x64 | 38,360 | 39,490 | 17,960 | 0.47 (inverted) | 0.45 | | index (docs/s, incl. bulk commit) | arm64 | 39,680 | 38,670 | 16,490 | 0.42 (inverted) | 0.43 | | Terms1 (ms/query) | x64 | 0.155 | 0.157 | 0.193 | 0.80 | 0.81 | | Terms1 (ms/query) | arm64 | 0.159 | 0.160 | 0.157 | 1.01 | 1.02 | | files added per small commit | both | 2 | 8 | 4 | | | | allocated bytes per document, index row | both | 5,694 | 5,091 | 21,040 | | | | peak resident set, x64 (MiB) | x64 | 238 | 237 | 218 | | | | index size, index row (bytes) | both | 11.64 M | 11.64 M | 8.16 M | | | Where a small commit spent its time on the M2 before the change (milliseconds, indicative, a stopwatch around each step on a loaded host): the flush 13 to 30 (text parts 6 to 11, stored 3 to 4, columns 0.3 to 11, and 7 to 13 of syncing the parts), the commit protocol about 4 (temporary file sync 1.3 to 3, final sync 1.1 to 3). The syncs of seven part files were the largest single share. With one pack file the whole five-commit sequence measured 21.7 ms against Lucene.NET's 22.0 on that host, and 19.5 against 16.5 in the final rows above; the steps were not timed again after the change. The arm64 commit row is inside the 1.25 bar and not yet at 1.0: what remains is one pack plus one commit file (two fsyncs and a rename) against Lucene.NET's one compound file plus its small files, and the segment's own write. Index throughput on x64 is within run-to-run spread of before (38.4k against 39.5k; the three runs span 36.6k to 38.5k and 38.7k to 39.6k), and the 12% rise in allocation per document is the pack buffer. Peak resident set was not captured on arm64. Load average before / after: x64 3.05 / 1.96 (one-minute), arm64 1.81 / 1.68, flat during the runs. **Phrase on x64 and the segment count.** The same index merged to one segment answers Phrase in 0.169 ms (Lucene.NET 0.165 to 0.168); with the 12 segments of the settled index it is 0.20, so about 3 microseconds per segment per query is the per-segment cost of looking up two terms and positioning two iterators. A skip over position varints with 16-byte vector steps was built and timed (interleaved with the scalar version, same host): 0.249 and 0.240 ms against 0.241 and 0.261 ms, no difference, so it was removed. The phrase scorer already leads with the rarest term and advances the others only to its documents. Machines: x64 AMD Ryzen 9 7950X3D, 16 cores / 32 threads (SSE4.2, AVX2, FMA, BMI2, AVX-512), NVMe ext4, Linux Mint 22.3, .NET SDK 10.0.402 / runtime 10.0.13, workstation GC; 1-minute load average 2.6 before the OneIndex run, 2.6 to 2.8 during and 2.7 after the Lucene.NET run (the 15-minute figure, 8.4, still carried an earlier job); arm64 Apple M2, 8 cores, 8 GiB, APFS, macOS, .NET SDK 10.0.403 / runtime 10.0.14, load 4.8 before, 3.9 after the OneIndex run and 3.3 after the Lucene.NET run (the Mac was not idle: a load of 3 to 4 is its baseline). Both indexes were warm in the page cache. Peak resident set was not captured on arm64. Native engines, information only (x64, same queries, tantivy-py 0.27.0 with its Python binding overhead, SQLite 3.54.1 command-line with the process start-up share in each row), one run of the corpus driver's native runners on the same machine: | Row | tantivy-py (ms/query) | SQLite FTS5 (ms/query) | OneIndex (ms/query) | |---|---|---|---| | index (docs/s) | 129,446 | 91,774 (3.30 s for 302,567) | 61,135 | | Terms1 / Terms2 / Terms5 | 0.046 / 0.104 / 0.224 | 3.75 / 7.53 / 18.1 | 0.142 / 0.227 / 0.658 | | Phrase | 0.056 | 0.66 | 0.208 | | Filtered | 0.415 | n/a | 1.636 | OneIndex is 2 to 4 times tantivy-py on query rows here, outside the 1.5x target, which is not a gate. Tantivy's index is 49.0 MB, OneIndex's 64.3 MB, Lucene.NET's 45.4 MB. ### Counting differences in the Facets and Stats rows At 302,567 documents the match checksums of the Facets and Stats rows differ by 2 and 4 between the engines. The cause is tokenization, not a collection bug: six abstracts contain U+202F NARROW NO-BREAK SPACE, which the Unicode 17 word-break property file lists as ExtendNumLet (UAX #29 rule WB13b), so OneIndex keeps "they\u202fare" as one token, as the standard prescribes, while the comparator's older tables break there. Four of the six documents hold a query term next to such a space; the single-term counts (`oneindex-harness count-diff`) differ for exactly those terms, each by one document that only the comparator counts. ### Fuzzy, prefix and as-you-type rows, term dictionary version 2 Same corpus, driver, settings and Lucene.NET build as above, after the version 2 term dictionary, the single-load number reads, the merge policy size ratio of 2 and the block-max WAND change that keeps clause order between rounds. Each engine ran as its own exclusive benchmark-class job, Lucene.NET right after OneIndex on the same machine (x64), three repetitions, medians. Match checksums of the Fuzzy, Typo and Prefix rows differ by design (OneIndex prunes with WAND and prefix queries are constant-score; Lucene.NET expands and counts all matches); the as-you-type typo row has a matched variant (`ONEINDEX_TYPO_MATCHED=1`, same query shape as the comparator's prefix-OR-fuzzy) whose checksums agree (2,925). | Row (ms/query) | OneIndex x64 | Lucene.NET x64 | ratio | OneIndex arm64 | Lucene.NET arm64 | ratio | |---|---|---|---|---|---|---| | index (docs/s) | 62,540 | 37,947 | 0.61 | 62,869 | 37,849 | 0.60 | | commit (median ms, 5 x 200 docs) | 28.3 | 52.4 | 0.54 | 20.7 | 14.5 | 1.43 | | Terms1 | 0.123 | 0.214 | 0.58 | 0.179 | 0.318 | 0.56 | | Terms2 | 0.245 | 0.436 | 0.56 | n/r | n/r | | | Terms5 | 0.487 | 0.635 | 0.77 | 0.620 | 0.752 | 0.82 | | Phrase | 0.179 | 0.157 | 1.14 | 0.250 | 0.389 | 0.64 | | Prefix | 0.105 | 0.317 | 0.33 | 0.124 | 0.410 | 0.30 | | AsYouType | 0.151 | 0.198 | 0.76 | 0.234 | 0.223 | 1.05 | | Fuzzy1 | 0.232 | 0.530 | 0.44 | 0.277 | 0.945 | 0.29 | | Fuzzy2 | 1.227 | 2.083 | 0.59 | 1.386 | 4.072 | 0.34 | | Typo | 0.211 | 0.484 | 0.43 | 0.228 | 0.699 | 0.33 | | TypoAsYouType (matched variant) | 0.081 | 0.173 | 0.47 | 0.087 | n/r | | | TypoAsYouType (documented semantics, information) | 0.242 | 0.173 | 1.40 | n/r | | | | Filtered | 1.540 | 1.888 | 0.82 | n/r | | | | Facets / Stats / Sort | 0.427 / 0.276 / 0.370 | 0.601 / 0.514 / 0.668 | 0.71 / 0.54 / 0.55 | n/r | | | n/r: not re-run for this change. The arm64 Lucene.NET column and the arm64 Terms1, Phrase, Prefix, AsYouType and TypoAsYouType figures come from the earlier benchmark-class run of the same Lucene.NET build and an earlier build of the dictionary code; Terms5, Fuzzy1, Fuzzy2, Typo and commit on arm64 were re-run on the final code (they did not move with the clause-order change: Fuzzy2 1.395 to 1.386 ms). Commit on arm64 is the one row above 1.25; it is governed by the number of files fsynced per commit, not by the dictionary. Allocated bytes per query, OneIndex against Lucene.NET (x64): Terms1 4,982 / 59,203; Terms5 21,054 / 970,469; Prefix 62,868 / 153,187; Fuzzy1 19,456 / 703,984; Fuzzy2 120,550 / 2,281,455; Typo 19,311 / 1,244,174; AsYouType 57,659 / 189,760. Size and memory (x64): settled index 60,589,958 bytes (62 MB before the merge policy change was 64.3 MB; Lucene.NET 45,426,027); peak resident set during the index row 1,269 MiB against 1,021 MiB (includes the 120 MB corpus held by the driver), during the query rows 443 to 1,171 MiB against 434 to 603 MiB. Machine records: x64 as above (Ryzen 9 7950X3D, 32 logical cores, ext4 NVMe, .NET 10.0.13, warm page cache). 1-minute load average before / during / after: OneIndex run 3.3 / 3.0 / 2.5, Lucene.NET run 7.3 / 5.9 / 5.2 (other users' light jobs on the host), matched typo run 1.3 / 1.2 / 1.2. arm64: Apple M2, 8 cores, .NET 10.0.14, load 3.0 / 2.9 / 2.9. ### Commit and Phrase with packed flush parts (commit format 3) and term dictionary version 2 together Both engines in one exclusive benchmark-class job per machine, filtered to the Phrase row (the index and commit rows always run), same settings as above. | Row | OneIndex x64 | Lucene.NET x64 | ratio | OneIndex arm64 | Lucene.NET arm64 | ratio | |---|---|---|---|---|---|---| | index (docs/s) | 60,363 | 37,169 | 0.62 | 65,505 | 37,308 | 0.57 | | commit (median ms, 5 x 200 docs) | 22.3 | 36.0 | 0.62 | 14.1 | 14.6 | 0.96 | | commit (max ms) | n/r | n/r | | 90.5 | 16.0 | 5.7 | | Phrase (ms/query) | 0.165 | 0.241 | 0.68 | 0.239 | 0.641 | 0.37 | The Phrase figure for Lucene.NET moves a lot between passes (0.157 to 0.241 ms on x64, 0.389 to 0.641 ms on arm64 across the runs in this document, with the same index layout), so its ratio is noisy; OneIndex's own figure is stable (0.165 to 0.179 ms x64). Against the best Lucene.NET figure seen on x64 (0.157) Phrase is 1.05. The arm64 commit maximum is one slow commit (90 ms) among five. Index size 70,161,632 bytes after the bulk commit and 60,589,958 settled. Load (1 minute, before / after): x64 2.6 / 2.0 (OneIndex), 1.9 / 1.7 (Lucene.NET); arm64 3.4 / 3.0 and 2.9 / 2.8. ## Earlier indicative pass (not publishable) A first run of the driver on 50,000 abstracts, to prove the rows run end to end and to capture memory and size. It ran as a shared-queue job on the worker beside other jobs (load average 11 to 12 on 32 logical cores before and after; the machine was **not** quiet), so the absolute numbers are indicative only and no ratio may be drawn from them. Code at commit `1956a0d`. Machine: AMD Ryzen 9 7950X3D (32 logical cores; SSE4.2, AVX, AVX2, FMA, BMI2, AVX-512 F/BW), 124 GiB, NVMe ext4, Linux Mint 22.3, .NET SDK 10.0.402, runtime 10.0.13, workstation GC; Lucene.NET 4.8.0-beta00018; tantivy-py 0.27.0; SQLite 3.54.1. Dataset: abstracts SHA-256 `1759e651…4893` (the generator was run twice on the worker with identical output), first 50,000 lines, seed 1, 200 queries per row, median of 3 passes, single indexing thread, index freshly written (warm page cache). Lucene.NET (index 49,000 documents in one bulk commit: 13,188 documents/s including the 1.25 s commit; 20.7 KB allocated per document; peak resident 288 MiB; five 200-document commits: median 66 ms, maximum 75 ms; index 8.5 MB): | Row | ms/query | allocated KB/query | peak resident MiB | |---|---|---|---| | Terms1 | 0.254 | 57 | 233 | | Terms2 | 0.554 | 787 | 174 | | Terms5 | 0.462 | 944 | 172 | | Phrase | 0.145 | 211 | 174 | | Prefix | 0.218 | 106 | 174 | | AsYouType | 0.131 | 143 | 170 | | Fuzzy1 | 0.388 | 586 | 175 | | Fuzzy2 | 1.620 | 2,116 | 177 | | Filtered | 0.517 | 605 | 179 | | Facets | 0.408 | 804 | 179 | | Stats | 0.285 | 780 | 183 | | Sort | 0.302 | 786 | 183 | Native, information only (same queries): tantivy-py indexed 50,000 documents at about 70,600 documents/s (peak resident 74 MiB; a single-threaded writer with a 256 MB heap) and answered Terms1 to Filtered in 0.012 (Terms1), 0.037, 0.070, 0.022 (Phrase), 0.086 (Prefix), 0.026 (AsYouType), 0.219 (Fuzzy1), 1.125 (Fuzzy2) and 0.081 (Filtered) ms/query; SQLite FTS5 loaded the same documents in 0.53 s of wall time per 50,000 insert statements and answered Terms1, Terms2, Terms5, Phrase, Prefix and AsYouType in 0.58, 1.38, 3.13, 0.23, 2.79 and 0.38 ms/query including its process start-up share. tantivy-py figures include the Python binding overhead; they are for orientation, not comparison. These native numbers come from the same noisy run. The full JSON (all rows, load figures, machine record) is produced by the scripts above; stage-close runs replace this section. # Search contracts These are the shapes the query engine is built against. - **Queries are data** (`TermQuery`, `BooleanQuery` with Must/Should/MustNot/Filter clauses, `PhraseQuery` with slop, `PrefixQuery`, `FuzzyQuery`, `RangeQuery`, `MatchAllQuery`), with value equality and a `Boost`. A `QueryVisitor` turns them into other things; execution is a visitor that produces a `Weight`, so execution code never edits the query classes. - **Segment reader.** `ISegmentReader` gives a segment's parts: `ITextFieldReader` (term lookup, prefix enumeration, postings, per-document field length, field statistics), `IColumnReader`, `IPointIndexReader`, `IStoredFieldsReader`, `ILiveDocs`. Document ids are dense and local to a segment; the search context adds each segment's offset to make global ids. - **Iterators and scorers.** `DocIdSetIterator` (`NextDoc`, `Advance`, `Cost`). `Scorer` adds `Score()` and three hooks for block-max WAND: `AdvanceShallow(target)` and `GetMaxScore(upTo)` bound the score of a run of documents without decoding them, and `SetMinCompetitiveScore` lets a collector tell the scorer which scores no longer matter. `PostingsIterator.ShallowAdvance` returns a `BlockImpact` (last document, maximum frequency, minimum field length): because BM25 grows with frequency and shrinks with length, those two values bound a block's best score. - **Collectors.** `TopScoreCollector` keeps the best k by score with a total, deterministic order: higher score first, then lower global document id. NaN scores rank last. It allocates nothing per document and raises the scorer's threshold as the heap fills. `CountCollector` needs no scores. - **Searcher.** `IndexSearcher` asks an `IWeightFactory` for a weight, walks each segment's scorer, skips deleted documents and feeds the collector, checking cancellation every 1,024 documents. - **Totals.** `TopDocs.TotalHits` is exact when every match is visited (`ScoreMode.Complete`) and a lower bound under `ScoreMode.TopScores`. - **Doubles in points.** `SortableBits.Encode` maps a double to a long with the same order, so a single integer point index serves both numeric kinds. # Changelog All notable changes are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project follows [Semantic Versioning](https://semver.org/). ## [Unreleased] ## [0.4.0-preview.2] - 2026-11-03 - **Fixed: Entity Framework Core inline apply skipped the first `SaveChanges`.** With `ApplyInline = true` and no `ContextFactory` (the configuration of the documented sample), the runtime had not yet learned how to create its own context when the first `SaveChanges` finished, so that save was not applied to the index. The error was kept only in `OneIndexRuntime.LastApplyError` and searches returned nothing for it. The rows were safely in the outbox. Anyone using `RecursiveAI.OneIndex.EntityFrameworkCore` 0.4.0-preview.1 that way should upgrade; after the upgrade the first apply (the next `SaveChanges`, or `OneIndexRuntime.For(db).CatchUp()`) catches the index up from the outbox, so nothing needs re-entering. On 0.4.0-preview.1 itself, calling `OneIndexRuntime.For(db)` once (any search through the context does) and then `CatchUp()` or `ReindexAsync` does the same. The context is now learned before the first change is saved. A configuration that can never apply (no `ContextFactory` and no public constructor taking the context's `DbContextOptions`) now throws `InvalidOperationException` from `SaveChanges` (with `ApplyInline`) or `RunAsync`, before anything is written, instead of failing silently. - The `oneindex` tool's `--version` prints the package version (for example `0.4.0-preview.2`), not the four-number assembly version. - Documentation: the install line for the tool needs `--prerelease`; package sizes (`.Models` 134 MB, `.Models.Multilingual` 112 MB, the tool 151 MB); the README says the previews are on nuget.org; the EF and `.Files` kill-campaign figures and the static index's first-query bytes (about 0.65 MB for the query, 0.74 MB with the manifest) agree across documents; internal markers removed from the synonyms and did-you-mean pages; the benchmark method page says where the published rows are. ## [0.4.0-preview.1] - 2026-11-03 The first public preview. - Static index: term dictionaries and lengths are read in 4 KiB pages on first use and scattered files use 4 KiB chunks (`StaticBuildOptions.ScatteredChunkBytes`, `oneindex static build --scattered-kb`), so a first query fetches about 0.63 MB instead of 2.6 MB. Static hybrid search takes its vector arm from IVF (`StaticIndex.HybridNprobe`). - IVF as a per-field vector index on the server (`[Vector(Index = VectorIndexKind.Ivf, IvfLists = n)]`), stored in a new segment part `ivf`. - Query JSON: a ranking rule naming a field that cannot be sorted is a positioned parse error instead of an exception. - Multi-vector auto-encoding: `[MultiVector(From = ...)]` encodes the source text at upsert through an `IMultiVectorEncoder` (`SearchIndex.Open(path, embedder, sparse, reranker, multiVectorEncoder)` or `SearchMultiVectorEncoders.Default`), caches by model identity and text in stored source, re-encodes stale entries in `ReembedStale()`, and `SearchQuery.MaxSim(...)` is the late-interaction query stage beside `Rerank`. `IMultiVectorEncoder` gains batch `Encode`/`EncodeAsync` with default implementations. - Windows: file handles allow sharing delete, rename replaces a destination that a reader has open, and readers use ordinary handles so cached lookups allocate nothing; the full test suite now runs on Windows in CI. The browser smoke app builds on its own CI job (it needs the wasm-tools workload) instead of with the solution. - Durability harness: the campaigns start their child processes correctly whether run through `dotnet run`, `dotnet Durability.dll` or the application host. - `RecursiveAI.OneIndex.Analysis.Asian`: Japanese and Korean lattice segmentation (IPADIC, mecab-ko-dic), Chinese most-probable-path segmentation (jieba word list) and Thai, Lao, Khmer and Myanmar maximal matching (ICU word lists), as the analyzers `ja`, `ko`, `zh`, `th`, `lo`, `km` and `my` (`AsianAnalyzers.Register()`; core gains `AnalyzerRegistry.Register`). Dictionaries are a checksummed container format (`OIDICT` 1) with typed errors. - Decompounding in core: `CompoundSplitter` (Koehn and Knight splitting, fewest parts first), `DecompoundFilter`, packed word lists for de, nl, sv, da, no, fi and is, and the analyzers `-compound` and `-compound-both`. - `RecursiveAI.OneIndex.Models`: SentencePiece unigram tokenizer (`SentencePieceModel`, `SentencePieceTokenizer`) with the nmt_nfkc normalizer. - New package `RecursiveAI.OneIndex.Models.Multilingual`: `MultilingualE5Embedder` (multilingual-e5-small, int8 weights packed). - More-like-this: `MoreLikeThisQuery` picks the most distinctive terms of a text or a stored document by term frequency times idf and searches them; `index.MoreLikeThis(id)`, `index.MoreLikeThisText(text)` and the query JSON node `moreLikeThis` (docs/more-like-this.md). A missing source document raises `SourceDocumentNotFoundException`. - Facet value search on the happy path: `search.FacetValues(x => x.Category, "sho")` counts over the documents of that search. - Release builds fetch the packed model weights from a release asset checked against a pinned SHA-256 and fail without them (`tools/models/fetch-release-weights.sh`). ## [0.3.0-preview.1] - 2026-11-03 - Commit format 3: a flushed segment writes its text, column, point, stored and identifier parts into one pack file, so a small commit adds two files instead of eight (fewer flushes to disk per commit). Indexes written by earlier formats open, search and merge unchanged. - Term proximity: user-text searches add a proximity bonus to the BM25F score (after Büttcher, Clarke and Lushman, SIGIR 2006), weight 0.1 by default; typed queries are unchanged. - Ranking rules (opt-in): a bucket-sort order of `words`, `typo`, `proximity`, `attribute`, `sort` and `exactness` rules, declared with `[RankingRules(...)]`, set per search with `.RankingRules(...)`, or given as `rankingRules` in query JSON. - Query JSON: a string holding an unpaired surrogate escape is a positioned parse error instead of an exception. - Term dictionary format 2: nodes keep their labels, cumulative term counts and child distances in fixed-width tables, so an arc is found by label and fuzzy, typo and as-you-type expansion can look up only the arcs a Levenshtein state can still use. Fuzzy automata share precomputed shape transitions across queries. Version 1 segments stay readable and are rewritten as version 2 when merged. - Block-max WAND re-sorts its clauses from the previous round's order instead of from scratch, which makes disjunctions of dozens of terms (fuzzy and typo expansions) about twice as fast. - Merge policy: `TieredMergePolicy.SizeRatio` (default 2) also merges the smallest segments when the largest of a group is at most that many times the others combined, so a settled index keeps a handful of segments and fuzzy, typo and prefix queries walk fewer term dictionaries; set it to 0 for the plain tiered policy. Bulk loads rewrite documents more often (about 3.8 times at 15 flushes against 1.7). - Filtered nearest-neighbour search: a filter now steers the HNSW walk (only passing documents are scored, with neighbours-of-neighbours expansion through rejected nodes, after the ACORN paper) and a selective filter switches to an exact scan of the passing vectors; both work on the compressed copies. ## [0.1.0-preview.1] - 2026-11-01 - Automatic embedding: `[Vector(From = ...)]` embeds the source text at upsert through the index's embedder (`SearchIndex.Open(path, embedder)` or `SearchEmbedders.Default`; `BgeSmallEmbedder.UseAsDefault()` in the models package), keeps the text and model identity digest beside the vector in stored source so reindexing, reopening, merges and schema rebuilds never embed unchanged text, and `ReembedStale` after a model change. - Reranking: `IReranker` (a sibling of `IEmbedder`), `SearchQuery.Rerank(...)` as a stage over the best hits of any search (text, vector or hybrid) with the reranker's score and the first-stage score on each hit, `Explain` for the stage, `[FullText(Rerank = true)]` to rerank plain text searches by itself, the `rerank` property of query JSON, and `MiniLmReranker` in the models package (cross-encoder/ms-marco-MiniLM-L6-v2, Apache-2.0) with `WordPieceTokenizer.EncodePair`. - Hybrid search: `HybridQuery`, `SearchQuery.Hybrid(...)` and the `hybrid` forms of query JSON fuse a text arm and a vector arm by reciprocal rank fusion (default) or weighted linear fusion; filters apply to both arms. `Search(text)` is hybrid automatically when the type has one auto-embedded vector member and the index has an embedder. - Repository foundation: solution, shared build properties, packaging metadata, CI and release workflows. - Storage layer: a range-read interface with synchronous and asynchronous members, memory and local-file backends, a fault-injecting wrapper (torn, lost and reordered writes), and a block-transform slot. - Commit file format with a checksummed frame, and a crash-safe commit protocol, verified under seeded fault schedules. - Schema model with deterministic serialization, untyped documents, the analyzer pipeline contract with a reference analyzer, and the query, scorer, collector and segment-reader contracts. - Attribute types, the schema descriptor source generator, and shapes for `SearchIndex` and `SearchTool` (not implemented). - Dynamic JSON fields: a `[Dynamic]` property is stored whole and its leaf paths are indexed by kind (keyword, number, bool, date), filterable by path from LINQ (`Path`), filter strings and query JSON, within limits on depth, paths and lengths. - Schema evolution: opening an index with a changed schema rebuilds the segments that need it in the background from stored source, resumably, while the old schema keeps serving every segment; one commit makes the new schema visible on all of them; changes that cannot be converted fail at open naming the field. Commit format version 2 records the schema epoch and the rebuilt segments written ahead. # Full-text search in C# Mark a class with `[FullText]`, `[Keyword]` and `[Filterable]` attributes, `Upsert` your objects, `Commit`, and call `Search(text)` with whatever the user typed. OneIndex ranks with BM25F, forgives typos, and returns highlights and facet counts with the hits. It runs in your process: no server, no native library. ```shell dotnet new console -o FullTextSearch && cd FullTextSearch dotnet add package RecursiveAI.OneIndex --version 0.4.0-preview.2 ``` Put this in `Program.cs` and run `dotnet run`: ```csharp using OneIndex; using var index = SearchIndex.Open
("search-data"); // a directory index.Upsert( [ new() { Id = "1", Section = "ops", Year = 2024, Title = "Getting started with Kubernetes", Body = "Deploy your first container and scale it." }, new() { Id = "2", Section = "security", Year = 2025, Title = "Container security checklist", Body = "Scan images, drop root and pin base images." }, new() { Id = "3", Section = "data", Year = 2023, Title = "Scaling PostgreSQL reads", Body = "Read replicas and connection pooling for busy apps." }, new() { Id = "4", Section = "ops", Year = 2025, Title = "Kubernetes networking explained", Body = "Services, ingress and network policies for containers." }, ]); index.Commit(); // durable; searches see the last commit // What a user types: words, "phrases", -exclusions; a typo is forgiven. foreach (var hit in index.Search("kubernets containers -security").Take(5)) Console.WriteLine($"{hit.Id} {hit.Score:F2} {hit.Document.Title}"); Console.WriteLine(); var results = index.Search("container") .Where(a => a.Year >= 2024) // a filter, translated, never run .Facets(a => a.Section) .Execute(); Console.WriteLine($"{results.TotalHits} hits since 2024"); foreach (var hit in results) // highlights of the fields that matched foreach (var (field, fragments) in hit.Highlights) Console.WriteLine($"{hit.Id} {field}: {fragments[0].Marked("[", "]")}"); foreach (var v in results.Facets["Section"].Values) Console.WriteLine($" {v.Value}: {v.Count}"); public class Article { [SearchId] public string Id { get; set; } = ""; [FullText(Boost = 2)] public string Title { get; set; } = ""; [FullText(Language = "en")] public string Body { get; set; } = ""; [Keyword, Facet] public string Section { get; set; } = ""; [Filterable, Sortable] public int Year { get; set; } } ``` ```text 1 0.43 Getting started with Kubernetes 4 0.41 Kubernetes networking explained 3 hits since 2024 2 Title: [Container] security checklist 1 Body: Deploy your first [container] and scale it. 4 Body: Services, ingress and network policies for [containers]. ops: 2 security: 1 ``` - `SearchIndex.Open("search-data")` creates or opens an index in that directory. A source generator in the package writes the mapping from the attributes at build time, with no reflection, so it works under trimming and NativeAOT. `SearchIndex.InMemory()` keeps it in memory instead. - `Search(text)` takes user text as typed and never throws on it: words, `"quoted phrases"`, `-exclusions` and `OR`. Typo tolerance is on for plain user text, so `kubernets` matches `Kubernetes`. - `[FullText(Boost = 2)]` makes a title match count twice a body match; `Language = "en"` stems English (`containers` matches `container`). - `Where` is translated into index filters and never run as C#; a predicate it can’t translate throws and names the part. - `Commit` is atomic: a search sees the last commit, and a commit that returned survives a crash or kill of your process. ## Limits - A preview (0.4.0-preview.2): APIs and file formats may still change between previews. - `TotalHits` is a lower bound unless you add `CountExactly()`. - One writer per index directory: a second writer, even from another process, gets `IndexLockedException`. Readers in other processes are fine. - A field must be declared before you search it; `[Dynamic]` JSON fields cover open-ended data. - Run it once per directory as written: a second run upserts the same ids again, which replaces them, so the output is the same. [Getting started](https://recursiveai.org/oneindex/docs/getting-started/) · [User text](https://recursiveai.org/oneindex/docs/user-text/) · [Mapping](https://recursiveai.org/oneindex/docs/mapping/) · [All recipes](https://recursiveai.org/oneindex/cookbook/) # Vector search in .NET Mark a `float[]` property `[Vector(From = nameof(Text))]` and OneIndex embeds the text when you `Upsert`, with a bundled English model that runs in your process. Then embed the question with the same model and call `SearchVector`: you get the documents nearest in meaning, even when they share no words with it. ```shell dotnet new console -o VectorSearch && cd VectorSearch dotnet add package RecursiveAI.OneIndex --version 0.4.0-preview.2 dotnet add package RecursiveAI.OneIndex.Models --version 0.4.0-preview.2 ``` `RecursiveAI.OneIndex.Models` carries the model weights inside the package (a 134 MB download from NuGet, once). Nothing is fetched at run time. Put this in `Program.cs` and run `dotnet run`: ```csharp using OneIndex; using OneIndex.Models; using OneIndex.Vectors; var embedder = BgeSmallEmbedder.UseAsDefault(); // bge-small-en-v1.5, 384 dimensions using var index = SearchIndex.Open("notes-index"); index.Upsert( [ new() { Id = "a", Topic = "hardware", Text = "Overheating notebook? Clean the fan and vents." }, new() { Id = "b", Topic = "hardware", Text = "A swollen battery must be replaced at once." }, new() { Id = "c", Topic = "cooking", Text = "Let bread dough rest overnight in the fridge." }, new() { Id = "d", Topic = "software", Text = "Disable startup programs so Windows boots faster." }, new() { Id = "e", Topic = "software", Text = "Update graphics drivers to fix screen flicker." }, new() { Id = "f", Topic = "travel", Text = "Pack a universal plug adapter for trips abroad." }, ]); // each Text is embedded here index.Commit(); float[] question = embedder.Embed("my laptop gets too hot", EmbeddingRole.Query); foreach (var hit in index.SearchVector(n => n.Embedding, question, neighbors: 2)) Console.WriteLine($"{hit.Id} {hit.Score:F3} {hit.Document.Text}"); Console.WriteLine(); question = embedder.Embed("PC takes ages to turn on", EmbeddingRole.Query); var search = index.SearchVector(n => n.Embedding, question, neighbors: 2) .Where(n => n.Topic == "software"); // filter before neighbors foreach (var hit in search) Console.WriteLine($"{hit.Id} {hit.Score:F3} {hit.Document.Text}"); public class Note { [SearchId] public string Id { get; set; } = ""; [FullText] public string Text { get; set; } = ""; [Keyword, Filterable] public string Topic { get; set; } = ""; [Vector(From = nameof(Text))] public float[]? Embedding { get; set; } } ``` ```text a 0.873 Overheating notebook? Clean the fan and vents. e 0.788 Update graphics drivers to fix screen flicker. d 0.835 Disable startup programs so Windows boots faster. e 0.809 Update graphics drivers to fix screen flicker. ``` - `BgeSmallEmbedder.UseAsDefault()` loads the bundled model and makes it the embedder new indexes open with. Nothing registers itself: without this line (or an embedder passed to `SearchIndex.Open(path, embedder)`), no text is embedded. - `[Vector(From = nameof(Text))]` embeds `Text` at `Upsert` with the document role. The length (384) comes from the model. - The question is embedded with `EmbeddingRole.Query`, which adds the instruction the model expects for queries. - `SearchVector(member, vector, neighbors)` returns the nearest documents by cosine similarity, highest score first. `Where(...)` picks the neighbors among documents that pass, so a filtered search still fills its page. - Each document stores its embedding beside a hash of the model and the text, so re-upserting unchanged text, reopening and merging never embed again. ## Limits - A preview (0.4.0-preview.2): APIs and file formats may still change between previews. - The bundled embedder is English. `RecursiveAI.OneIndex.Models.Multilingual` has multilingual-e5-small (around 100 languages; its int8 weights are 120 MB). - Embedding runs on the CPU in the thread that calls `Upsert`, so a large first load takes time. The model reads at most its input limit and cuts longer text; split long documents into passages. - Small collections are searched exactly; larger ones use an HNSW graph, which is approximate (recall below 1.0 at a given setting). - Model licenses: bge-small-en-v1.5 is MIT. The weights are the model authors’ and are distributed under that license; the training data’s terms are not conferred on them. [Getting started](https://recursiveai.org/oneindex/docs/getting-started/) · [Vectors](https://recursiveai.org/oneindex/docs/vectors/) · [Automatic embedding](https://recursiveai.org/oneindex/docs/auto-embedding/) · [Models](https://recursiveai.org/oneindex/docs/models/) · [All recipes](https://recursiveai.org/oneindex/cookbook/) # Hybrid search with reranking in C# Give a class one `[Vector(From = ...)]` property and an embedder, and `Search(text)` becomes hybrid: a BM25 keyword list and a vector list, fused by reciprocal rank fusion. Add `.Rerank()` and a cross-encoder reads the query beside each of the best hits and orders them again. Both models are bundled and run in your process. ```shell dotnet new console -o HybridSearch && cd HybridSearch dotnet add package RecursiveAI.OneIndex --version 0.4.0-preview.2 dotnet add package RecursiveAI.OneIndex.Models --version 0.4.0-preview.2 ``` `RecursiveAI.OneIndex.Models` carries the embedder and reranker weights inside the package (a 134 MB download from NuGet, once). Put this in `Program.cs` and run `dotnet run`: ```csharp using OneIndex; using OneIndex.Models; BgeSmallEmbedder.UseAsDefault(); // embeds documents and queries MiniLmReranker.UseAsDefault(); // cross-encoder for .Rerank() using var index = SearchIndex.Open("kb-index"); index.Upsert( [ new() { Id = "1", Text = "error CS0246: the type or namespace name could not be found" }, new() { Id = "2", Text = "Add a package reference so the compiler can resolve a type." }, new() { Id = "3", Text = "Namespaces group related types and avoid name clashes." }, new() { Id = "4", Text = "A missing using directive leaves a class name unresolved." }, new() { Id = "5", Text = "The project file lists the NuGet packages an app uses." }, new() { Id = "6", Text = "Fix a build that cannot find a type: check usings and packages." }, ]); index.Commit(); string q = "compiler says type not found"; Show("Keywords only (BM25)", index.Search(q).Hybrid(0)); Show("Hybrid (default)", index.Search(q)); Show("Hybrid, reranked", index.Search(q).Rerank(window: 20)); void Show(string title, SearchQuery search) { Console.WriteLine(title); var hits = search.Take(3).ToList(); if (hits.Count == 0) Console.WriteLine(" (no hits)"); foreach (var hit in hits) Console.WriteLine($" {hit.Id} {hit.Score,7:F3} {hit.Document.Text}"); } public class Doc { [SearchId] public string Id { get; set; } = ""; [FullText(Language = "en")] public string Text { get; set; } = ""; [Vector(From = nameof(Text))] public float[]? Embedding { get; set; } } ``` ```text Keywords only (BM25) (no hits) Hybrid (default) 6 0.136 Fix a build that cannot find a type: check usings and packages. 1 0.125 error CS0246: the type or namespace name could not be found 2 0.115 Add a package reference so the compiler can resolve a type. Hybrid, reranked 1 0.652 error CS0246: the type or namespace name could not be found 2 -2.409 Add a package reference so the compiler can resolve a type. 6 -3.335 Fix a build that cannot find a type: check usings and packages. ``` - The keyword list is empty: in user text every word not joined by `OR` must match, and no document contains “says”. The vector list still finds the documents that mean the same thing, so the hybrid search answers. - `Search(text)` is hybrid automatically when the index has an embedder and the class has exactly one `[Vector(From = ...)]` property. `Hybrid(0)` turns it off for one query; `Hybrid(0.5)` weighs the arms equally. - The defaults are reciprocal rank fusion with `SemanticRatio = 0.75` (the vector list leads), `RankConstant = 10` and a window of 100 hits per list. Filters from `Where(...)` apply inside both lists, and a `-word` in the user’s text removes documents from both. - Scores are on different scales: BM25, then the fused rank score, then the cross-encoder’s score, which has no fixed range. Compare positions, not numbers, across the three. - `Rerank(window: 20)` rescores the best 20 hits of the first stage with the cross-encoder (cross-encoder/ms-marco-MiniLM-L6-v2) and orders them by its score. Each hit keeps the first-stage score in `FirstStageScore`. ## How much it helps From the project’s own [relevance measurements](https://recursiveai.org/oneindex/docs/hybrid/) (nDCG@10 on BEIR test sets, the bundled int8 bge-small-en-v1.5): hybrid at its defaults averages 0.5168 over SciFact, ArguAna, SCIDOCS and NQ, against 0.5058 for the vector arm alone and 0.4049 for BM25, and is above both arms on every set. The defaults were tuned on SciFact, ArguAna and SCIDOCS, and NQ was held out. Your data may prefer another ratio: set it per query. ## Limits - A preview (0.4.0-preview.2): APIs and file formats may still change between previews. - The reranker costs one model pass per hit in the window, on the CPU: keep the window small (tens, not thousands). - `TotalHits` of a hybrid search counts the documents in either list (at most twice the window); a page deeper than that is empty unless you raise the window. - The bundled models are English. Licenses: bge-small-en-v1.5 is MIT; cross-encoder/ms-marco-MiniLM-L6-v2 is Apache-2.0 and was trained on MS MARCO, whose terms are the data owner’s and apply to that data, which is not included. [Hybrid search](https://recursiveai.org/oneindex/docs/hybrid/) · [Reranking](https://recursiveai.org/oneindex/docs/rerank/) · [Models](https://recursiveai.org/oneindex/docs/models/) · [All recipes](https://recursiveai.org/oneindex/cookbook/) # Full-text search over EF Core entities Put OneIndex attributes on an EF Core entity, add `UseOneIndex(...)` to the context options and `UseOneIndexOutbox()` to the model. Every `SaveChanges` then logs the change in the same transaction, the index is updated from the database, and `db.Set.Search(text)` returns tracked entities with a `Where` that runs partly in the index and partly in the database. ```shell dotnet new console -o EfSearch && cd EfSearch dotnet add package RecursiveAI.OneIndex.EntityFrameworkCore --version 0.4.0-preview.2 dotnet add package Microsoft.EntityFrameworkCore.Sqlite --version 10.0.14 ``` Put this in `Program.cs` and run `dotnet run`: ```csharp using Microsoft.EntityFrameworkCore; using OneIndex; using OneIndex.EntityFrameworkCore; var options = new DbContextOptionsBuilder() .UseSqlite("Data Source=app.db") .UseOneIndex("search-data", o => o.Index()) .Options; using var db = new AppDb(options); db.Database.EnsureCreated(); // creates Tickets and the outbox tables db.Tickets.AddRange( new Ticket { Id = "t1", Status = "open", CustomerId = 7, Title = "Login page times out" }, new Ticket { Id = "t2", Status = "open", CustomerId = 7, Title = "Password reset email never arrives" }, new Ticket { Id = "t3", Status = "closed", CustomerId = 9, Title = "Login works after reset" }); await db.SaveChangesAndIndexAsync(); // rows + outbox in one transaction, then index // Title and Status are filtered in the index; CustomerId only in the database. var found = await db.Tickets.Search("login") .Where(t => t.Status == "open" && t.CustomerId == 7) .Take(10).ToListAsync(); foreach (var t in found) Console.WriteLine($"{t.Id} {t.Title}"); var t2 = await db.Tickets.FindAsync("t2"); t2!.Title = "Login blocked: password reset email never arrives"; await db.SaveChangesAndIndexAsync(); // the change reaches the index too Console.WriteLine(); foreach (var t in await db.Tickets.Search("login").Take(10).ToListAsync()) Console.WriteLine($"{t.Id} {t.Title}"); public class Ticket // an EF entity and a search document at once { [SearchId] public string Id { get; set; } = ""; [FullText] public string Title { get; set; } = ""; [Keyword, Facet] public string Status { get; set; } = ""; public int CustomerId { get; set; } // not indexed: the database filters on it } public sealed class AppDb(DbContextOptions options) : DbContext(options) { public DbSet Tickets => Set(); protected override void OnModelCreating(ModelBuilder b) => b.UseOneIndexOutbox(); } ``` ```text t1 Login page times out t1 Login page times out t3 Login works after reset t2 Login blocked: password reset email never arrives ``` - The index is not an EF provider and not a second copy of your data: the database stays the source of truth. The outbox row says “this entity changed”; the applier reads the entity’s current row and upserts or deletes it in the index, so applying a row twice gives the same index. - `SaveChangesAndIndexAsync()` saves, then catches the index up, so the next search sees the change. A plain `SaveChanges` writes the outbox row too; in a web app run `OneIndexRuntime.For(db).RunAsync(token)` from a hosted service to apply it, and a search may lag the database by about 250 ms unless it asks for `.AfterPendingChanges()`. - `Where` is split on `&&`: each part OneIndex can translate runs in the index, every other part runs in the database over the candidates the index returns. The page is filled unless fewer entities match. The entities returned are the database’s current rows. - The outbox tables are part of your model, so `dotnet ef migrations add` creates them like any other table. ## Limits - A preview (0.4.0-preview.2): APIs and file formats may still change between previews. - A configuration that can never apply now fails loudly. From the 0.4.0-preview.2 changelog: “A configuration that can never apply (no `ContextFactory` and no public constructor taking the context’s `DbContextOptions`) now throws `InvalidOperationException` from `SaveChanges` (with `ApplyInline`) or `RunAsync`, before anything is written, instead of failing silently.” - **Bulk writes are not indexed.** `ExecuteUpdate`, `ExecuteDelete` and raw SQL bypass `SaveChanges`, so they log nothing. Repair with `OneIndexRuntime.ReindexAsync(query)` (re-reads those rows) or `ReindexAsync()` (rebuilds the whole index, deletions included). - **SQL Server and MySQL have not been run.** The project’s kill tests covered SQLite and PostgreSQL 16; in its own words, “SQL Server and MySQL are not tested.” - A transaction that stays open longer than `LateRowAge` (default 30 seconds) or past `LateRowWindow` (10,000 later log positions) and then commits reaches the index only with that entity’s next change or `ReindexAsync`. Set `LateRowAge` above your longest transaction. - One process per index directory: two app instances on one database each need their own index directory. - Run it once as written: a second run adds the same keys again and EF Core refuses them. Delete `app.db` and `search-data` to start over. [EF Core integration](https://recursiveai.org/oneindex/docs/entity-framework/) · [Durability](https://recursiveai.org/oneindex/docs/durability/) · [All recipes](https://recursiveai.org/oneindex/cookbook/) # Search a folder of PDFs and documents in C# with citations `FolderIndexer.Run(index, folder)` reads every file it understands (PDF, Markdown, HTML, text, and the entries of zip, tar, 7z and other archives), cuts each into passages, and indexes them. `index.Passages(text)` searches them, and `hit.Cite()` tells you where each answer came from: the file, the page and the exact character range. ```shell dotnet new console -o FolderSearch && cd FolderSearch dotnet add package RecursiveAI.OneIndex.Files --version 0.4.0-preview.2 mkdir -p docs && cd docs curl -sLo aes-fips-197.pdf \ https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.197-upd1.pdf printf '# Key rotation\n\nRotate AES keys every 90 days.\n' > notes.md && cd .. ``` Put this in `Program.cs` and run `dotnet run`: ```csharp using OneIndex; using OneIndex.Files; using SearchIndex index = FolderIndexer.OpenIndex("passages-index"); FolderIndexResult run = FolderIndexer.Run(index, "docs"); Console.WriteLine($"{run.Added} added, {run.Updated} updated, {run.Removed} removed, " + $"{run.Unchanged} unchanged"); foreach (var q in new[] { "number of rounds", "rotate keys" }) { Console.WriteLine($"\n\"{q}\""); foreach (var hit in index.Passages(q).Distinct(p => p.FileId).Take(3)) { Citation c = hit.Cite(); string text = c.Text.ReplaceLineEndings(" "); Console.WriteLine($" {c.File} page {c.Page} [{c.Start}..{c.End}]"); Console.WriteLine($" {(text.Length > 80 ? text[..80] + "…" : text)}"); } } FolderIndexResult again = FolderIndexer.Run(index, "docs"); // only changes are read Console.WriteLine($"\nSecond run: {again.Unchanged} unchanged, {again.Added} added"); ``` ```text 2 added, 0 updated, 0 removed, 0 unchanged "number of rounds" aes-fips-197.pdf page 19 [751..1638] The block ciphers AES-128, AES-192, and AES-256 differ in three respects: 1) the… "rotate keys" notes.md page 0 [0..46] # Key rotation Rotate AES keys every 90 days. Second run: 2 unchanged, 0 added ``` - `FolderIndexer.OpenIndex(path)` opens an ordinary OneIndex index of `FilePassage` documents, so filters (`Where(p => p.Format == "pdf")`), highlights, hybrid search and reranking all work on it. - Passages follow the document’s structure: a heading starts one, blocks fill it to about 256 estimated tokens, and a passage never spans two pages. Each is an exact substring of its page’s extracted text. - `Cite()` returns `File`, `Entry` (the path inside an archive), `Page` (from 1; 0 for files without pages), `Start` and `End` in the page’s extracted text, `Text`, and for PDFs the line boxes in points. `citation.ReadSourceText(root, options)` extracts the page again to show or verify the source. - `Distinct(p => p.FileId)` keeps the best passage per file. - `Run` can be repeated. A file with the same size and modification time is not opened; a changed one is hashed and read again only if its bytes differ; a renamed file (same size and hash) keeps its passages without being read. After a crash or kill the index holds whole files only, and the next run picks up where the last commit left off. - Ignore rules follow `.gitignore` (`.gitignore` and `.oneindexignore` in any folder), and `node_modules` and version-control folders are skipped by default. ## From the command line The `oneindex` tool does the same without writing code. It is a preview, so install it with `--prerelease`. The tool package is 151 MB, because it carries the default English models: ```console dotnet tool install --global RecursiveAI.OneIndex.Cli --prerelease oneindex index docs --index passages-index oneindex query passages-index "number of rounds" --limit 2 ``` `oneindex index` prints a summary (here “2 files: 2 added … 96 passages in 0.5 s”), and the query prints: ```text 1. aes-fips-197.pdf page 19 [751-1638] score 5.142 The block ciphers AES-128, AES-192, and AES-256 differ in three respects: 1) the length of the key; 2) the number of rounds, which determines the size of the required key schedule; and 3) the speci... 2. aes-fips-197.pdf page 19 [1640-2278] score 3.564 The three inputs to CIPHER() are: 1) the data input in, which is a block represented as a linear array of 16 bytes; 2) the number of rounds Nr for the instance; and 3) the round keys. Thus, AES-128... ``` `--watch` keeps the index current, `--ocr` reads scans, `--embed` adds vectors for `--hybrid` queries, and `oneindex static build` writes an index a browser can query from a plain file server. ## Limits - A preview (0.4.0-preview.2): APIs and file formats may still change between previews. - The index holds the extracted text of your files: protect its directory like the folder itself. Nothing is sent over the network. - A scanned PDF without a text layer yields no passages. `RecursiveAI.OneIndex.Files.Ocr` adds OCR for scans and photos (`OcrPdfExtractor`, `OcrImageExtractor`), at seconds per page. - Office documents, mail and media files are not read yet; they get a record with format `binary`. - A renamed archive is read again, and its entries get new identifiers. [Folder indexing](https://recursiveai.org/oneindex/docs/files/) · [The oneindex tool](https://recursiveai.org/oneindex/docs/cli/) · [All recipes](https://recursiveai.org/oneindex/cookbook/) # Migrate from Lucene.NET to OneIndex The Lucene.NET way is a writer, a reader, a searcher, an analyzer, a query parser, a sort, a facets collector and a highlighter. In OneIndex it is one attributed class and one `SearchIndex`. Here is the docs’ worked example (products: “wireless headphones” in the `audio` category under 100, cheapest first, with category counts and a highlighted title) as a program you can run. ```shell dotnet new console -o FromLucene && cd FromLucene dotnet add package RecursiveAI.OneIndex --version 0.4.0-preview.2 ``` Put this in `Program.cs` and run `dotnet run`: ```csharp using OneIndex; // FSDirectory + IndexWriter + IndexSearcher in one object using var index = SearchIndex.Open("products-index"); index.Upsert( // UpdateDocument(new Term("id", ...), doc) [ new() { Id = "p1", Title = "Wireless headphones", Price = 89, Category = "audio", Body = "Over-ear wireless headphones with noise cancelling." }, new() { Id = "p2", Title = "Wireless earbuds", Price = 59, Category = "audio", Body = "In-ear headphones, wireless, with a charging case." }, new() { Id = "p3", Title = "Studio headphones", Price = 149, Category = "audio", Body = "Wired headphones for mixing." }, new() { Id = "p4", Title = "Wireless mouse", Price = 25, Category = "computers", Body = "A wireless mouse with a USB receiver." }, ]); index.Commit(); // writer.Commit(): durable and atomic var results = index.Search("wireless headphones") // MultiFieldQueryParser .Where(p => p.Category == "audio" && p.Price < 100) // TermQuery + range query .OrderBy(p => p.Price) // Sort(SortField) .Facets(p => p.Category) // FacetsCollector .Take(10) .Execute(); foreach (var hit in results) // Highlighter { var title = hit.Highlights.TryGetValue("Title", out var f) ? f[0].Marked() : hit.Document.Title; Console.WriteLine($"{hit.Id} {hit.Document.Price,4} {title}"); } foreach (var v in results.Facets["Category"].Values) Console.WriteLine($"{v.Value} = {v.Count}"); public class Product // replaces Document + Field per document { [SearchId] // StringField "id" public string Id { get; set; } = ""; [FullText(Boost = 2)] // TextField public string Title { get; set; } = ""; [FullText(Language = "en")] // TextField with EnglishAnalyzer public string Body { get; set; } = ""; [Filterable, Sortable] // DoubleField + doc values public double Price { get; set; } [Keyword, Facet] // StringField + facet field public string Category { get; set; } = ""; } ``` ```text p2 59 Wireless earbuds p1 89 Wireless headphones audio = 2 ``` ## Which call replaces which | Lucene.NET 4.8 | OneIndex | |---|---| | `FSDirectory`, `IndexWriter`, `IndexWriterConfig` | `SearchIndex.Open(path)`; `Upsert`, `Delete`, `Commit` | | `Document`, `TextField`, `StringField`, numeric fields, doc values | attributes on a class: `[FullText]`, `[Keyword]`, `[Filterable, Sortable]` | | `QueryParser`, `MultiFieldQueryParser` | `Search(text)` (never throws); `SearchStrict(text)` for a field syntax with positioned errors | | `BooleanQuery` filters, `NumericRangeQuery` | `Where(p => …)`, or a filter string `Category = audio AND Price < 100` | | `Sort`, `SortField` | `OrderBy`, `ThenBy`; `SearchAfter(cursor)` for deep paging | | `FacetsCollector`, `SortedSetDocValuesFacetField` | `[Facet]` and `Facets(p => p.Category)`: no facet config or taxonomy index | | `Highlighter`, `QueryScorer` | `hit.Highlights` | | `GroupingSearch` | `Distinct(p => p.Field)` | | `IndexWriter`, `IndexReader`, `IndexSearcher`, query classes | the same names at the low level (`OneIndex.Index`, `OneIndex.Search`) | ## What behaves differently - **Scoring.** Lucene.NET 4.8’s default similarity is TF-IDF and `BM25Similarity` is opt-in. OneIndex always scores BM25F (k1 = 0.9, b = 0.4), with title and body weighed as one stream. Scores aren’t comparable between the two engines. - **Analysis.** `en` uses Snowball’s English stop list and the Porter2 stemmer; Lucene.NET’s `EnglishAnalyzer` uses its own 33-word list and the original Porter stemmer, so a few token streams differ. - **Every word must match** in `Search(text)` unless joined by `OR`; a typed last word also matches as a prefix, and typos are forgiven. - **No schema-less documents.** Fields are declared once; changing the schema rebuilds in the background from the stored documents. - **Commit** is always durable and atomic. There is no separate flush step; near-real-time readers are opt-in. - **Vectors, hybrid search and reranking** have no Lucene.NET 4.8 counterpart. ## Moving an existing index OneIndex cannot open a Lucene index: the formats are unrelated and no importer ships. Reindex from your source of truth (database or files) with `Upsert` in batches and one `Commit()`. If the documents exist only as stored fields in Lucene, read them out once with Lucene.NET (a loop over `searcher.Doc(i)`) and upsert them. ## Speed and relevance, as the project measured them - **Speed.** On 302,567 Wikipedia abstracts with matched analysis and commits, 17 rows (indexing, commit, term, phrase, prefix, as-you-type, fuzzy, typo, filtered, facets, stats, sort and function-score queries), OneIndex’s time divided by Lucene.NET 4.8.0-beta00018’s, so below 1 is faster: geometric mean 0.62 on x64 (AMD Ryzen 9 7950X3D, Linux) and 0.41 on arm64 (Apple M2, macOS; 0.57 without the Stats and Sort rows, where Lucene.NET’s arm64 times are outliers). The slowest rows are Phrase at 1.17 on x64 and FunctionScore at 1.14 on x64. The index is larger (60.6 MB settled against 45.4 MB), because OneIndex stores each document’s source. [Every row](https://recursiveai.org/oneindex/docs/benchmarks/). - **Relevance.** nDCG@10 with BM25 at the defaults, against Lucene.NET’s flat layout on the same runs: SciFact 0.6918 against 0.6789, ArguAna 0.4347 against 0.4118, NQ 0.3452 against 0.3058, SCIDOCS 0.1544 against 0.1509. Summing per-field scores in Lucene.NET does better on SCIDOCS (0.1587). [Method](https://recursiveai.org/oneindex/docs/relevance/). ## Limits - A preview (0.4.0-preview.2): APIs and file formats may still change between previews. - Not in OneIndex yet: the `SpanQuery` family, block joins, payloads and `WildcardQuery` (use a prefix or regex query). - `TotalHits` is a lower bound unless you ask for `CountExactly()`. [Migrating from Lucene.NET, in full](https://recursiveai.org/oneindex/docs/migrating-from-lucene-net/) · [Benchmarks](https://recursiveai.org/oneindex/docs/benchmarks/) · [All recipes](https://recursiveai.org/oneindex/cookbook/)