Search a folder of PDFs and documents in C# with citations

FolderIndexer.Run(index, folder) reads every file it understands (PDF, Markdown, HTML, text, and the entries of zip, tar, 7z and other archives), cuts each into passages, and indexes them. index.Passages(text) searches them, and hit.Cite() tells you where each answer came from: the file, the page and the exact character range.

dotnet new console -o FolderSearch && cd FolderSearch
dotnet add package RecursiveAI.OneIndex.Files --version 0.4.0-preview.2
mkdir -p docs && cd docs
curl -sLo aes-fips-197.pdf \
  https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.197-upd1.pdf
printf '# Key rotation\n\nRotate AES keys every 90 days.\n' > notes.md && cd ..

Put this in Program.cs and run dotnet run:

using OneIndex;
using OneIndex.Files;
 
using SearchIndex<FilePassage> index = FolderIndexer.OpenIndex("passages-index");
 
FolderIndexResult run = FolderIndexer.Run(index, "docs");
Console.WriteLine($"{run.Added} added, {run.Updated} updated, {run.Removed} removed, "
                + $"{run.Unchanged} unchanged");
 
foreach (var q in new[] { "number of rounds", "rotate keys" })
{
    Console.WriteLine($"\n\"{q}\"");
    foreach (var hit in index.Passages(q).Distinct(p => p.FileId).Take(3))
    {
        Citation c = hit.Cite();
        string text = c.Text.ReplaceLineEndings(" ");
        Console.WriteLine($"  {c.File} page {c.Page} [{c.Start}..{c.End}]");
        Console.WriteLine($"    {(text.Length > 80 ? text[..80] + "…" : text)}");
    }
}
 
FolderIndexResult again = FolderIndexer.Run(index, "docs");   // only changes are read
Console.WriteLine($"\nSecond run: {again.Unchanged} unchanged, {again.Added} added");

2 added, 0 updated, 0 removed, 0 unchanged
 
"number of rounds"
  aes-fips-197.pdf page 19 [751..1638]
    The block ciphers AES-128, AES-192, and AES-256 differ in three respects: 1) the…
 
"rotate keys"
  notes.md page 0 [0..46]
    # Key rotation  Rotate AES keys every 90 days.
 
Second run: 2 unchanged, 0 added

• FolderIndexer.OpenIndex(path) opens an ordinary OneIndex index of FilePassage documents, so filters (Where(p => p.Format == "pdf")), highlights, hybrid search and reranking all work on it.
• Passages follow the document’s structure: a heading starts one, blocks fill it to about 256 estimated tokens, and a passage never spans two pages. Each is an exact substring of its page’s extracted text.
• Cite() returns File, Entry (the path inside an archive), Page (from 1; 0 for files without pages), Start and End in the page’s extracted text, Text, and for PDFs the line boxes in points. citation.ReadSourceText(root, options) extracts the page again to show or verify the source.
• Distinct(p => p.FileId) keeps the best passage per file.
• Run can be repeated. A file with the same size and modification time is not opened; a changed one is hashed and read again only if its bytes differ; a renamed file (same size and hash) keeps its passages without being read. After a crash or kill the index holds whole files only, and the next run picks up where the last commit left off.
• Ignore rules follow .gitignore (.gitignore and .oneindexignore in any folder), and node_modules and version-control folders are skipped by default.

From the command line

The oneindex tool does the same without writing code. It is a preview, so install it with --prerelease. The tool package is 151 MB, because it carries the default English models:

dotnet tool install --global RecursiveAI.OneIndex.Cli --prerelease
oneindex index docs --index passages-index
oneindex query passages-index "number of rounds" --limit 2

oneindex index prints a summary (here “2 files: 2 added … 96 passages in 0.5 s”), and the query prints:

1. aes-fips-197.pdf page 19 [751-1638]  score 5.142
   The block ciphers AES-128, AES-192, and AES-256 differ in three respects: 1) the length of the key; 2) the number of rounds, which determines the size of the required key schedule; and 3) the speci...
2. aes-fips-197.pdf page 19 [1640-2278]  score 3.564
   The three inputs to CIPHER() are: 1) the data input in, which is a block represented as a linear array of 16 bytes; 2) the number of rounds Nr for the instance; and 3) the round keys. Thus, AES-128...

--watch keeps the index current, --ocr reads scans, --embed adds vectors for --hybrid queries, and oneindex static build writes an index a browser can query from a plain file server.

Limits

• A preview (0.4.0-preview.2): APIs and file formats may still change between previews.
• The index holds the extracted text of your files: protect its directory like the folder itself. Nothing is sent over the network.
• A scanned PDF without a text layer yields no passages. RecursiveAI.OneIndex.Files.Ocr adds OCR for scans and photos (OcrPdfExtractor, OcrImageExtractor), at seconds per page.
• Office documents, mail and media files are not read yet; they get a record with format binary.
• A renamed archive is read again, and its entries get new identifiers.

Folder indexing · The oneindex tool · All recipes