oneindexDocs
oneindex docs
Everything about OneIndex: getting started, mapping a class, user text, filters, facets and sorting, vectors, hybrid search and reranking with bundled models, the EF Core, folder and VectorData packages, the browser and static index, durability and how it is tested, relevance and benchmarks against Lucene.NET, and the file formats. OneIndex is a preview, 0.4.0-preview.2.
Start
Getting startedInstall OneIndex 0.4.0-preview.2 for .NET 10, index a few objects and search them, in C# with no server and no native code. OneIndex is an embedded search engine: full-text (BM25F), vector (HNSW) and hybrid search in one in-process library.OverviewOneIndex is unaffiliated with other products named OneIndex, OneSearch or similar.ArchitectureOneIndex is one engine in layers. Each layer is public, in its own namespace, and is built only on the layers before it.Mapping documentsA document type is an ordinary class whose properties carry attributes. A source generator (it runs at build time inside the RecursiveAI.OneIndex package) writes, for each such class, a SearchMapping: the schema, and the conversion between the class and an untyped Document.…Reads like .NETEach layer of OneIndex is shaped after a .NET API that readers already know. This page puts them side by side. Every OneIndex snippet here is compiled and run by the sample tests (samples/OneIndex.Samples: HappyPath.cs, EfCore.cs, ReadsLike.cs, Migration.cs).Migrating from Lucene.NETThis page maps the concepts and calls of Lucene.NET 4.8 (the 4.8.0-beta line on NuGet) to OneIndex. It is written from Lucene.NET's public documentation and from black-box runs of the pinned release in tools/OneIndex.Harness (see relevance.md); neither engine's source was…The oneindex tooloneindex is a .NET tool (package RecursiveAI.OneIndex.Cli) that turns a folder into a searchable passage index, searches it with citations, and builds a static index for browsers. It uses the folder package (files.md) and the engine; it adds no behaviour of its own.
Searching
User textWhat a person types into a search box. index.Search(text) accepts any string and never throws on it: unbalanced quotes, lone operators, control characters, lone surrogates and megabytes of input all give a sensible query. (Running the query can still fail for reasons that…Strict query syntaxThe strict syntax is an opt-in query language for people and programs that want exactly what they wrote. It is a separate grammar from the forgiving user text of index.Search("...") (see user-text.md), which accepts anything and never throws. The strict syntax throws…Filter stringsA filter string selects documents by field values, with conditions joined by AND, OR and NOT. The syntax follows Meilisearch's documented filter expressions. A filter never scores.LINQ filtersindex.Search(text).Where(predicate) narrows a search with a C# lambda. The lambda is translated, never run: the engine walks the expression tree and builds query objects (a BooleanQuery of Filter clauses over TermQuery and RangeQuery). It does not compile the tree. Filters…Query JSONA search as data: a JSON document with "v": 1, checked against a schema as it is read. It is meant for programs and language models that build a query without a C# compiler, and it compiles to the same query objects as every other surface. It is not compatible with…Search as an LLM toolSearchTool.For() describes a search over T as a tool a language model can call. The arguments are a query JSON document; the model's tool call goes straight to index.Parse.Queries and limitsHow the query engine (QueryEngine, an IWeightFactory) runs each query record. Scoring is in scoring.md. A query is data; QueryEngine.CreateWeight checks it against the schema, fixes the statistics of the snapshot and returns a Weight; a weight makes one Scorer per segment.…More like thisFinds documents that resemble a text or an indexed document. MoreLikeThisQuery is a query record like the others; the happy path has index.MoreLikeThis(id, x => x.Body) and index.MoreLikeThisText(text), query JSON has the moreLikeThis node.
Ranking and refining
ScoringHow a document's score is made, what the statistics cover, the defaults and why, and the explain format. Query semantics are in queries.md.Function scoresA function score changes the relevance score of a query with numbers computed from column values: popularity, recency, distance. It does not change which documents match (unless MinScore is set); it changes their order. The semantics and formulas follow the published…Ranking rulesAn optional way to rank results: a bucket sort over a list of rules, in the manner of Meilisearch's documented ranking rules. Without rules a search is ranked by the BM25F score (scoring.md), which already includes a term-proximity bonus. Rules are for applications that want…Typo tolerance and as-you-typeWhat user text does when a word is misspelled or half typed. It applies to forgiving user text (index.Search("...")). The strict parser and query JSON keep their explicit term~edits fuzzy semantics and do not use these settings.SynonymsSynonyms are applied at query time over analyzed terms. Nothing is stored in the index for them, so changing the synonym table never needs a reindex. Semantics follow Meilisearch's documented ones (the Synonyms page of its relevancy documentation: one-way and mutual…Did you meanOneIndex.Suggest.DidYouMean proposes corrected queries from the terms of a field's dictionary in a snapshot. It is separate from typo-tolerant matching (which finds documents for the typo itself); this is the "Did you mean x?" line.Sorting and pagingThis note is normative for ordering hits by columns and for paging with a cursor. The columns are in columns.md; the query surfaces that expose it are linq.md and query-json.md.Facets and distinctThis note is normative for OneIndex.Aggregations.FacetCollector, FacetValueSearch, OneIndex.Search.DistinctCollector and the happy-path calls Facets and Distinct of SearchQuery. Aggregations (buckets and metrics beyond facets) are in aggregations.md. The column layouts they…AggregationsThis note is normative for OneIndex.Aggregations (Aggregation requests, AggregationCollector, the result types and AggregationBuilder). Facets, which share the machinery, are in facets.md. Names follow Meilisearch where it has a term and Elasticsearch's published aggregation…Geo searchThis note is normative for geo points: the index, the filters and sorting by distance. The terms follow Meilisearch's documented geosearch (_geoRadius, _geoBoundingBox, _geoPoint); the documents hold one point (or several) per geo field instead of a _geo object.
Vectors and models
Vectors and HNSWThis note is normative for the vector part of a segment: what the kernels compute, how closeness becomes a score, how the graph is built and searched, what the two files contain and how a merge treats them. Measured results are at the end.Hybrid searchThis note is normative for HybridQuery, SearchQuery.Hybrid(...) and the hybrid forms of query JSON. Automatic embedding is in auto-embedding.md; the vector search itself in vectors.md.RerankingA cross-encoder reads the query and a passage together, which ranks better than comparing a stored vector or counting words, and costs one model pass per pair. So it runs as a stage over the best hits of a first-stage search, never over the whole corpus.Automatic embeddingThis note is normative for [Vector(From = ...)], the embedder an index uses, and the cache that keeps unchanged text from being embedded again.ModelsPackage RecursiveAI.OneIndex.Models. It depends on core and on RecursiveAI.OneModel only.Learned sparse vectorsA sparse vector field holds a weight per token (a SPLADE-style expansion, Formal et al., SIGIR 2021 and arXiv:2109.10086, or the output of an inference-free encoder). Documents are indexed on the existing postings; a query scores by dot product.Multi-vector: MaxSimLate interaction (ColBERT; Khattab and Zaharia, SIGIR 2020) represents a text as one vector per token and scores a query against a document by MaxSim: for each query token take the best similarity among the document's tokens, then sum (section 3.5, equation 3). OneIndex…IVF vector indexAn inverted-file (IVF) index groups vectors by cluster and answers a query by scanning only the few clusters whose centres are nearest to it (Jegou, Douze and Schmid, "Product quantization for nearest neighbor search", TPAMI 2011, section 2; the clustered lists follow Sivic…
Packages and platforms
EF Core integrationRecursiveAI.OneIndex.EntityFrameworkCore keeps OneIndex indexes in step with an EF Core database and lets a search compose with an EF query. It is not an EF provider: the database stays the source of truth, and the index is a derived copy that can always be rebuilt from it.Folder indexingRecursiveAI.OneIndex.Files turns a folder into searchable, cited passages. RecursiveAI.OneIndex.Files.Ocr adds scans and photos of text. A passage index is an ordinary OneIndex index of one document type, FilePassage, so hybrid search, the embedder, reranking, highlighting…VectorData adapterRecursiveAI.OneIndex.VectorData is a vector-store collection over OneIndex for Microsoft.Extensions.VectorData (version 10.11.0 of the abstractions, the one dependency). It covers what the abstraction defines: collections, upsert, get, delete, vector search with filters, and…In the browserEverything in the core package runs in browser WebAssembly (browser-wasm, no threads). Storage there is one of three things:Static indexA static index is an ordinary OneIndex index cut into small immutable files, so a browser (or any HTTP client) can search it from a plain file server or CDN with no server process. oneindex static build writes one; StaticIndex reads one.
Durability and storage
DurabilityWhat OneIndex promises about your data when a process is killed, a disk call fails or the power goes out, under which conditions, what it does not promise, and how each promise is tested. File layouts are in commit-format.md, the storage contract in storage.md, the…Writer and reader processesAn index directory can be open in one writer and in any number of readers at once, in different processes. This note says how, what each operating system gives, and what is promised.Storage interfaceIStorage is a flat set of named files. It is the only way the engine touches bytes, so one implementation can be local files, memory, read-only HTTP ranges or browser storage, and a wrapper can see (and break) every write.Commit formatAll multi-byte integers are little-endian. A varint is an unsigned LEB128 integer in its shortest form (a reader rejects a trailing zero group and values past 64 bits). A string is a varint byte length and that many bytes of UTF-8 (invalid UTF-8 is rejected). Bytes are a…Segments and mergesThis note is normative for the index lifecycle. File frames, the commit file and the commit protocol are in commit-format.md; the storage interface is in storage.md.Schema evolutionAn index is opened with the schema the application declares (for the happy path, the attributes of the document type). When the index already holds a schema, the two are compared at open and the index follows the application: no migration step, no separate tool.
Text index
AnalysisAn Analyzer turns text into terms. It is a tokenizer followed by filters, built once and reused. This page is normative for the contract, the registered analyzers, the Unicode and Snowball versions and the folding rules; two indexes built by different builds agree on terms…Asian-language analysisJapanese, Chinese, Korean, Thai, Lao, Khmer and Myanmar write words without spaces (Korean with spaces, but with agglutinated endings). The package segments them with packed dictionaries and plugs into the analyzer pipeline of core (docs/analysis.md). It references only core…DecompoundingGerman, Dutch, the Scandinavian languages, Finnish and Icelandic write compounds as one word ("Haustür", "kirjakauppa"). A search for "Tür" should find "Haustür", and one for "Haustür" should find "Tür". CompoundSplitter cuts a compound into the words it is made of, and…Term dictionaryFormat of the terms part of a segment (codec terms, versions 1 and 2) and the term-lookup operations built on it. Version 2 is written by this build; version 1 files stay readable and a merge rewrites them as version 2. The two versions differ only in how a node of the…Postings and block-max WANDFormats of the postings part (codec postings, version 1) and the lengths part (codec lengths, version 1), and the top-k disjunction that uses their block impacts. File frame, integers and varints as in commit-format.md.ColumnsThis note is normative for the columns part of a segment: per-document values of the fields that filtering, sorting, faceting and aggregation read. File frames and the codec interface are in commit-format.md and segments.md.Point indexThis note is normative for the points part of a segment: range lookups over numeric, date and bool fields without scanning the documents, and rectangle lookups over geo points (geo.md). Frames and the codec interface are in commit-format.md and segments.md.Stored sourceThis note is normative for the stored part of a segment: the original values of the fields the schema stores, compressed in blocks, with random access by document id. Frames and the codec interface are in commit-format.md and segments.md.Dynamic JSON fieldsA [Dynamic] property holds a JSON value whose shape is not known in advance. The whole value is stored with the document and comes back unchanged. Its leaf values are also indexed by path, so documents can be filtered by what is inside the JSON without declaring each key.
Measurement
RelevanceHow retrieval quality is measured on the BEIR sets, with the comparator side first. Speed is in benchmarks.md.BenchmarksHow the text rows are measured against Lucene.NET and the native engines. The relevance method is in relevance.md. The published rows are in the milestone section below, from exclusive benchmark runs on the machines listed in the machine record, and every number carries that…