Migrate from Lucene.NET to OneIndex

The Lucene.NET way is a writer, a reader, a searcher, an analyzer, a query parser, a sort, a facets collector and a highlighter. In OneIndex it is one attributed class and one SearchIndex<T>. Here is the docs’ worked example (products: “wireless headphones” in the audio category under 100, cheapest first, with category counts and a highlighted title) as a program you can run.

dotnet new console -o FromLucene && cd FromLucene
dotnet add package RecursiveAI.OneIndex --version 0.4.0-preview.2

Put this in Program.cs and run dotnet run:

using OneIndex;
 
// FSDirectory + IndexWriter + IndexSearcher in one object
using var index = SearchIndex.Open<Product>("products-index");
index.Upsert(                     // UpdateDocument(new Term("id", ...), doc)
[
    new() { Id = "p1", Title = "Wireless headphones", Price = 89, Category = "audio",
            Body = "Over-ear wireless headphones with noise cancelling." },
    new() { Id = "p2", Title = "Wireless earbuds", Price = 59, Category = "audio",
            Body = "In-ear headphones, wireless, with a charging case." },
    new() { Id = "p3", Title = "Studio headphones", Price = 149, Category = "audio",
            Body = "Wired headphones for mixing." },
    new() { Id = "p4", Title = "Wireless mouse", Price = 25, Category = "computers",
            Body = "A wireless mouse with a USB receiver." },
]);
index.Commit();                   // writer.Commit(): durable and atomic
 
var results = index.Search("wireless headphones")      // MultiFieldQueryParser
    .Where(p => p.Category == "audio" && p.Price < 100)  // TermQuery + range query
    .OrderBy(p => p.Price)                               // Sort(SortField)
    .Facets(p => p.Category)                             // FacetsCollector
    .Take(10)
    .Execute();
foreach (var hit in results)                             // Highlighter
{
    var title = hit.Highlights.TryGetValue("Title", out var f)
        ? f[0].Marked() : hit.Document.Title;
    Console.WriteLine($"{hit.Id}  {hit.Document.Price,4}  {title}");
}
foreach (var v in results.Facets["Category"].Values)
    Console.WriteLine($"{v.Value} = {v.Count}");
 
public class Product              // replaces Document + Field per document
{
    [SearchId]                    // StringField "id"
    public string Id { get; set; } = "";
    [FullText(Boost = 2)]         // TextField
    public string Title { get; set; } = "";
    [FullText(Language = "en")]   // TextField with EnglishAnalyzer
    public string Body { get; set; } = "";
    [Filterable, Sortable]        // DoubleField + doc values
    public double Price { get; set; }
    [Keyword, Facet]              // StringField + facet field
    public string Category { get; set; } = "";
}

p2    59  <em>Wireless</em> earbuds
p1    89  <em>Wireless</em> <em>headphones</em>
audio = 2

Which call replaces which

What behaves differently

• Scoring. Lucene.NET 4.8’s default similarity is TF-IDF and BM25Similarity is opt-in. OneIndex always scores BM25F (k1 = 0.9, b = 0.4), with title and body weighed as one stream. Scores aren’t comparable between the two engines.
• Analysis. en uses Snowball’s English stop list and the Porter2 stemmer; Lucene.NET’s EnglishAnalyzer uses its own 33-word list and the original Porter stemmer, so a few token streams differ.
• Every word must match in Search(text) unless joined by OR; a typed last word also matches as a prefix, and typos are forgiven.
• No schema-less documents. Fields are declared once; changing the schema rebuilds in the background from the stored documents.
• Commit is always durable and atomic. There is no separate flush step; near-real-time readers are opt-in.
• Vectors, hybrid search and reranking have no Lucene.NET 4.8 counterpart.

Moving an existing index

OneIndex cannot open a Lucene index: the formats are unrelated and no importer ships. Reindex from your source of truth (database or files) with Upsert in batches and one Commit(). If the documents exist only as stored fields in Lucene, read them out once with Lucene.NET (a loop over searcher.Doc(i)) and upsert them.

Speed and relevance, as the project measured them

• Speed. On 302,567 Wikipedia abstracts with matched analysis and commits, 17 rows (indexing, commit, term, phrase, prefix, as-you-type, fuzzy, typo, filtered, facets, stats, sort and function-score queries), OneIndex’s time divided by Lucene.NET 4.8.0-beta00018’s, so below 1 is faster: geometric mean 0.62 on x64 (AMD Ryzen 9 7950X3D, Linux) and 0.41 on arm64 (Apple M2, macOS; 0.57 without the Stats and Sort rows, where Lucene.NET’s arm64 times are outliers). The slowest rows are Phrase at 1.17 on x64 and FunctionScore at 1.14 on x64. The index is larger (60.6 MB settled against 45.4 MB), because OneIndex stores each document’s source. Every row.
• Relevance. nDCG@10 with BM25 at the defaults, against Lucene.NET’s flat layout on the same runs: SciFact 0.6918 against 0.6789, ArguAna 0.4347 against 0.4118, NQ 0.3452 against 0.3058, SCIDOCS 0.1544 against 0.1509. Summing per-field scores in Lucene.NET does better on SCIDOCS (0.1587). Method.

Limits

• A preview (0.4.0-preview.2): APIs and file formats may still change between previews.
• Not in OneIndex yet: the SpanQuery family, block joins, payloads and WildcardQuery (use a prefix or regex query).
• TotalHits is a lower bound unless you ask for CountExactly().

Migrating from Lucene.NET, in full · Benchmarks · All recipes