Relevance: method

How retrieval quality is measured on the BEIR sets, with the comparator side first. Speed is in benchmarks.md.

Datasets

BEIR test splits of SciFact, ArguAna, NQ and SCIDOCS, fetched and hash-checked by tools/fetch.sh (BEIR zips; the MD5s are BEIR's own, the SHA-256s are ours; both are in tools/pins.tsv). Datasets are never committed. Queries are the test queries that have judgements; judgements are the qrels/test.tsv grades. Other BEIR sets are not used.

Runner

oneindex-harness beir (tools/run_relevance.sh) drives any ITextEngine: it indexes the corpus (title and text), retrieves the top 1,000 documents for every test query under each BM25 setting, writes a TREC run file per setting, and scores it.

• Settings. Three field layouts times two parameter sets, so six runs per dataset:
  • layouts: Flat (title and text concatenated into one field, Kamalloo et al.'s "flat"); MultiFieldSum (title and text as two fields, scores summed with equal weights, Kamalloo et al.'s "multifield" and the Anserini configuration the BEIR paper describes); MultiFieldMax (the better of the two fields' scores, Elasticsearch's best_fields, the BEIR leaderboard's original Elasticsearch run);
  • parameters: k1=0.9, b=0.4 (Anserini's defaults, the ones the BEIR paper states) and k1=1.2, b=0.75 (Lucene's defaults).
• Analysis. The engine's English chain: lower-casing, possessive removal, English stop words and Porter-family stemming. Queries are analysed with the same chain and the terms OR-ed within each field.
• nDCG@10. Our implementation (TrecEval.NdcgAt) follows trec_eval: gain is the judged grade, discount 1/log2(rank+1), ranking by score descending with ties broken by document id descending, the mean taken over queries present in both run and judgements, ideal DCG from all positively graded judgements. Two numbers are reported per run: the raw one, and the BEIR-convention one, which drops a hit whose document id equals the query id before scoring (BEIR's default; it matters for ArguAna, whose queries are documents of the corpus).
• Cross-check. tools/crosscheck_ndcg.py evaluates the same run files with pytrec_eval (ndcg_cut_10), the evaluator BEIR uses, and fails if any run differs from ours by more than 5e-5 (agreement to four decimals).

Published baselines

Quoted with their source; "ours" are compared with both.

• BEIR paper: Thakur, Reimers, Rücklé, Srivastava, Gurevych, BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS 2021 Datasets and Benchmarks, arXiv:2104.08663 (v4), Table 2, BM25 column. The text describes the lexical baseline as Anserini with k1=0.9 and b=0.4, title and passage indexed as separate fields.
• Kamalloo et al.: Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard, arXiv:2306.07471 (SIGIR 2024), Table 3, BM25 columns "multifield" (the BEIR paper's configuration, equal weights over title and contents), "flat" (title and text in one field), both using Lucene's default English analyzer.

The ArguAna figures differ between the two papers as printed (0.315 and 0.414); we report against both and do not reconcile them.

Gate

OneIndex BM25 nDCG@10 on each set is at most one point below the published baseline (above it is fine) and not below Lucene.NET at matched settings: OneIndex's default k1 and b against Lucene.NET at the same layout, k1 and b.

Results: Lucene.NET 4.8.0-beta00018 (the comparator side)

Engine: Lucene.NET 4.8.0-beta00018 (assembly build commit:[df99b8abd9]; packages Lucene.Net, Lucene.Net.Analysis.Common, Lucene.Net.Facet) on .NET 10.0.13 (SDK 10.0.402), Linux x64. Analysis: EnglishAnalyzer (lower-case, possessive removal, English stop words, Porter stemming) for documents and queries. Top 1,000 per query, test split. Worker: 32 logical cores (AMD Ryzen 9 7950X3D), NVMe ext4, 8 indexing and search threads. One run covered all four sets and the pytrec_eval cross-check; an earlier attempt produced identical nDCG values. Code at commit 1956a0d.

Datasets (SHA-256 of the zip, MD5 matches BEIR): SciFact 536e1444…0165 (5,183 documents, 300 queries), ArguAna cfdf79ad…557b (8,674; 1,406), SCIDOCS 96640201…17ef1 (25,657; 1,000), NQ 2553bf7b…e752 (2,681,468; 3,452). Full hashes: tools/pins.tsv.

nDCG@10 (BEIR convention in parentheses where it differs; it drops a hit whose id equals the query id):

Reading: with k1=0.9, b=0.4, Lucene.NET reproduces the Kamalloo flat column on SciFact (0.6789 vs 0.679) and NQ (0.3058 vs 0.306) to three decimals and its multifield column on SciFact (0.6647 vs 0.665), SCIDOCS (0.1587 vs 0.158) and, with the BEIR convention, ArguAna (0.4240 vs 0.414, 1.0 point high). NQ MultiFieldSum is 0.3252 against the published 0.329 (0.4 point low). The BEIR paper's ArguAna value (0.315) is close to the raw, convention-free figure (0.3217), the Kamalloo values to the BEIR-convention figures; the two papers' ArguAna numbers do not agree with each other, so the gate on ArguAna should be read against Kamalloo with the convention applied. MultiFieldMax (Elasticsearch best_fields over the two fields) is clearly worst on NQ, where the title field alone matches many irrelevant documents; it is reported for completeness, not as a target. Lucene defaults (1.2/0.75) beat 0.9/0.4 on ArguAna and SCIDOCS and SciFact-flat, 0.9/0.4 on NQ.

pytrec_eval 0.5.11 (ndcg_cut_10) on the same run files agrees with our nDCG to better than 2e-15 on all 48 run files (raw and BEIR-convention variants of 24 runs): four decimals and far beyond.

Indexing time (8 threads, one commit): SciFact 1.8 s, ArguAna 1.0 s, SCIDOCS 1.6 s, NQ 26.8 s (index 1.1 GiB). All six settings for NQ answered 3,452 queries in 15 to 22 s each.

Results: OneIndex against Lucene.NET and the baselines

Run on the worker (32 logical cores, AMD Ryzen 9 7950X3D, NVMe ext4), .NET 10 (SDK 10.0.4xx), 8 threads, same datasets and hashes as above. OneIndex defaults: Bm25F layout (one score per term over title and text, title weight 3), k1 = 0.9, b = 0.4, English analysis (lower-case, possessive removal, Snowball English stop words, Snowball English stemmer). NQ was indexed and searched in separate runs (index phase, then a search phase on the reopened index). Every run file was cross-checked with pytrec_eval (ndcg_cut_10); the largest difference to our nDCG@10 is 1.8e-15.

nDCG@10, BEIR convention (a hit whose id equals the query id is dropped):

SciFact (2.6 points) and ArguAna (1.9) are more than one point above the baseline, NQ is 1.3 above; SCIDOCS is 0.2 below it. The Lucene.NET figures are those of the section above (re-run once more in the equal-weight job; identical to four decimals).

OneIndex with the same layout as Lucene.NET's MultiFieldSum (two separate field scores added, title weight 1, so no BM25F):

At matched layout and parameters OneIndex's BM25 scores equal Lucene.NET's within noise (identical on SCIDOCS at 0.9/0.4, above on ArguAna and NQ, 0.5 point below on SciFact).

SCIDOCS against Lucene.NET MultiFieldSum

Bm25F (0.1531) is 0.56 points below Lucene.NET's MultiFieldSum (0.1587) on SCIDOCS, and above its Flat layout (0.1509). Measured causes:

• It is the layout, not the scoring or the analysis. The equal-weight sum in OneIndex reproduces Lucene.NET to four decimals on SCIDOCS (0.1587 / 0.1587). Lucene.NET's Explain output (public API) shows the same idf and tf formula as OneIndex's: idf ln(1 + (N - n + 0.5)/(n + 0.5)), tf (k1+1) / (tf + k1 (1 - b + b len/avgLen)).
• Analysis is not it. Experimental builds (not kept) with Lucene's 33-word stop list and the original Porter stemmer moved the sum layout by at most 0.1 point either way (0.1552 default, 0.1548 Lucene stop list, 0.1551 Porter, 0.1543 both), while the title weight moved it 0.35 (the harness had given the sum layout OneIndex's title weight 3 until this run; at weight 1 it is 0.1587). With the same stop list the two analyzers' token streams differ in 20 of 6,000 sampled texts (stems of short words); with each side's own list they differ in 5,286, almost all by stop words.
• Collection size and average length are not it. Counting empty-field documents in N and average length (two variants, run on all three small sets) changed SCIDOCS Bm25F by at most 0.04 points.
• What decides it: SCIDOCS (queries are paper titles, relevant documents are cited papers) rewards a document whose title and abstract both match, which adding two saturating field scores does and BM25F (one saturation over the weighted sum) does not. Title weight sweep, Bm25F 0.9/0.4:

No weight reaches 0.1587 on SCIDOCS while all four sets keep their other results; the sum layout wins on SCIDOCS and loses 3 points on SciFact and 1 on NQ against Bm25F. BM25F is the engine's documented base score, so it stays the default; the sum layout is a query composition (BooleanQuery of per-field TermQuery) any caller can write.

Operational notes

• Loading NQ takes 173 s at 8 threads (index 746 MiB; Lucene.NET: 26.8 s, 1.1 GiB), commit and merges included; IndexWriter.AddUnique skips the per-document identifier lookup that made the first attempt time out at 30 minutes. Reopening the finished index and reading 2.68 M identifiers takes 36 s; the 3,452 queries take 12 to 25 s per setting.
• The runner takes ONEINDEX_PHASE=index|search (build and commit in one job, search in another), ONEINDEX_LAYOUTS, ONEINDEX_WORK and the OneIndex adapter ONEINDEX_TITLE_WEIGHT.

Typo tolerance on (information)

ONEINDEX_TYPO=on makes the OneIndex adapter match the Bm25F layout with TypoTermQuery (default typo budgets on the analyzed word, the typo and exactness factors, no prefix matching). It is information only; the gated runs leave it off, where queries are exactly the plain BM25F term queries.

Term proximity bonus (default weight 0.1)

The bonus of docs/scoring.md ("Term proximity") on the gate sets, by weight. The table and its reading are in that section; the default keeps every set at or above the no-bonus numbers of the table above (within noise on ArguAna at 1.2/0.75) and adds 0.1 to 0.3 points at the default BM25 setting.

Milestone re-run

The gate sets re-run on the code of the first public version. nDCG@10 under the BEIR convention (a hit whose id equals the query id is dropped); our runner, cross-checked with pytrec_eval (largest difference 2.2e-15). Lucene.NET is 4.8.0-beta00018 with matched analysis. The BM25 numbers equal the recorded ones except the proximity default (weight 0.1 on user text, described above and in docs/scoring.md).

BM25 at k1 = 0.9, b = 0.4

ArguAna without the BEIR own-hit convention: 0.3260 (OneIndex) against 0.3111 (Lucene.NET Flat). Every set is within one point of the baseline or above it, and at or above Lucene.NET Flat. SCIDOCS is read against Flat, as ruled (BM25F is a single-stream model); it sits 0.4 below the per-field-sum layout, which loses on SciFact and NQ when OneIndex runs it (below).

At k1 = 1.2, b = 0.75 (SciFact / ArguAna / SCIDOCS / NQ):

Other layouts, 0.9/0.4 then 1.2/0.75 (SciFact / ArguAna / SCIDOCS / NQ): OneIndex Flat 0.6822 / 0.4181 / 0.1513 / 0.3148 then 0.6891 / 0.4930 / 0.1568 / 0.2891; OneIndex MultiFieldSum 0.6452 / 0.4272 / 0.1552 / 0.3228 then 0.6456 / 0.4818 / 0.1592 / 0.3042; Lucene.NET MultiFieldMax 0.6685 / 0.3784 / 0.1461 / 0.2437 then 0.6684 / 0.4584 / 0.1480 / 0.2309.

Dense and hybrid

Typed BM25 arm, bge-small dense arm, default hybrid (weighted RRF, rank constant 10, semantic ratio 0.75).

Hybrid is above both arms on every set and on the mean. NQ needs GPU-embedded vectors, which this run did not use, so its row is the recorded one (held-out confirmation). Other fusion settings reproduce the recorded numbers to four decimals on the three CPU sets (for example SciFact RRF k 60 ratio 0.5 gives 0.7282 and linear 0.5 gives 0.7377).

.Models rows

What stands as recorded: all NQ model rows, the sparse encoder, and the cross-encoder scores (the cache was valid; the pair scores were not recomputed).