---
title: "Durability"
url: https://recursiveai.org/oneindex/docs/durability/
description: "What OneIndex promises about your data when a process is killed, a disk call fails or the power goes out, under which conditions, what it does not…"
version: "0.4.0-preview.2"
source: https://github.com/recursive-ai-oss/oneindex-csharp/blob/177f49bb3974e9e44df6d1a800342a9d353a0984/docs/durability.md
---

# Durability

What OneIndex promises about your data when a process is killed, a disk call fails or the power goes out, under which conditions, what it does not promise, and how each promise is tested. File layouts are in [commit-format.md](https://recursiveai.org/oneindex/docs/commit-format/), the storage contract in [storage.md](https://recursiveai.org/oneindex/docs/storage/), the multi-process rules in [processes.md](https://recursiveai.org/oneindex/docs/processes/) and the Entity Framework Core integration in [entity-framework.md](https://recursiveai.org/oneindex/docs/entity-framework/).

## The promises

1. **A commit that returned is not lost.** `Commit` returns only after every new file is written and fsynced, the commit file is written and fsynced, and it has been renamed into place. After any later kill of the process, `kill -9`, crash or reboot of the application, the next open sees that commit or a newer one.
2. **A reader sees one whole commit.** The commit file is published by one atomic rename, so a reader (in this process or another) sees a complete earlier commit or the complete new one, never a mixture. Files a commit lists are never changed after the rename; a commit and a snapshot keep their files alive until the last reader lets go, in any process.
3. **A crash at any instant leaves an openable index.** The last commit stands. The files a crashed writer left behind are unreferenced and are removed by the next writer's `Open`. Opening never falls back silently to an older commit: if the newest commit or a file it lists is damaged, the open fails with `CorruptIndexException` naming the file.
4. **Damage is detected, not served.** Every index file has a magic, a format version and a CRC-32C footer; the commit records each file's length and footer checksum; readers check lengths and counts against limits before trusting an offset. Torn, lost, reordered or bit-flipped bytes become `CorruptIndexException` (`UnsupportedFormatException` for an intact file from a newer writer), or a failed open of the damaged commit, never a silently wrong result.
5. **One writer.** A second writer on the same directory, from any process on the same machine, is refused at once with `IndexLockedException`; the lock is released when its process dies.
6. **A failed disk call does not corrupt.** If a storage call fails (full disk, I/O error, cancellation), the commit in progress is not published, the previous commit stands, and an index reopened afterwards holds exactly the commits that returned. A writer whose commit call failed refuses further work, because it cannot know whether the commit became durable; open the index again. (The Entity Framework applier does this for you.)
7. **Near-real-time views are not durable.** A view from `OpenNearRealTimeReader` shows uncommitted work and says so (`IsDurable` is false). It disappears with the process.

## Conditions

- A local file system on one machine that honours `fsync` and atomic `rename`: ext4, XFS, btrfs, APFS, NTFS. Network and FUSE file systems are not supported: they may not implement the exclusive lock, or may lose a rename they acknowledged.
- The index directory is changed only by OneIndex. Files edited or truncated by something else are detected by checksum when read (promise 4), but the index is then damaged.
- Hardware that does not lie about flushing. A disk that acknowledges a flush it did not perform breaks every database's promises, this one's included.
- Linux (x64, arm64), macOS (arm64) and Windows (x64, NTFS) run the full test suite on every change; the kill and fault-injection campaigns run on Linux and macOS only. See [processes.md](https://recursiveai.org/oneindex/docs/processes/).

## What is not promised

- **Power loss is weaker than a process kill.** .NET cannot open a directory to fsync it, so `SyncDirectory` fsyncs each file created or renamed since the last call instead of the directory. On Linux file systems with ordered metadata journals this makes the rename durable in practice, but that is a property of those file systems, not of POSIX, and it has not been tested by cutting power. The worst case is that the newest commit (or several, if a file system does not keep order) is rolled back to an earlier complete one. It is never a corrupt index: a commit lists only files that were synced and checked. The fault injector models the strict case (a rename survives only after the directory is synced) and the engine passes it. See [storage.md](https://recursiveai.org/oneindex/docs/storage/).
- **After a power cut the open index may be an earlier acknowledged commit.** An application that cannot tolerate that should run a directory fsync of its own (for example from a supervising process).
- **An index is a derived structure for the Entity Framework integration.** The database is the source of truth; the guarantee is that the index equals the database after catch-up, not that it is a second durable copy (see below).
- **Static indexes are built, not committed.** `oneindex static build` writes chunk files and then a manifest, without fsync. A killed build leaves no manifest (the manifest is written last, under a temporary name, and renamed), so nothing serves a half-built index, and the build can be rerun into the same directory. After a power cut during or just after a build, rebuild before uploading. Readers check every chunk against the manifest's CRC-32C and fail with `CorruptIndexException` on a chunk that does not match; a chunk that is missing fails the read.
- **Browser persistence needs an atomic host write.** `OpfsPersistence.SaveAsync` writes segment files first and commit files last; an interrupted save leaves the previous commit or the new one, provided the host's `WriteAsync` replaces a file atomically (the File System Standard expects a writable stream's close to do so). A host that can leave a half-written file can make the newest commit unreadable, which fails to open with `CorruptIndexException`.
- **Entity Framework: a log row can be lost if a transaction stays open longer than the late-row window or is pruned past.** See the next section.

## Entity Framework Core integration

The guarantee, as tested: **after catch-up, each index equals the database** (same identifiers and the same indexed fields for every row), and a position the index acknowledged is never lost, including across kills of the application and of the applier, with two application instances (each with its own index directory) on one database.

Conditions and limits:

- **Providers verified: SQLite and PostgreSQL 16.** SQL Server and MySQL have not been run. The applier assumes only that log positions grow and that a row becomes visible when its transaction commits, which those databases satisfy, but nothing here proves it for them.
- **Late rows.** A transaction that stays open while more than `LateRowWindow` later positions are logged (default 10,000), or longer than `LateRowAge` (default 30 s), is given up on. If it then commits, its change reaches the index with that entity's next change or `ReindexAsync`, not before. `LateRowAge` must exceed your longest transaction that writes indexed entities.
- **Retention.** The applier loop prunes the log to `RetainRows` positions (default 100,000) but never into the newest `LateRowWindow` positions. A direct call to `Prune` or `PruneAsync` keeps exactly what it is told to, and a transaction open across such a call can lose its row; run manual pruning when writers are quiet.
- **Bulk statements** (`ExecuteUpdate`, `ExecuteDelete`, raw SQL) bypass the interceptor and are repaired by `ReindexAsync`.
- **Failures.** If the index cannot commit (disk full, permission), the applier records the error in `LastApplyError`, closes the index and opens it again at the next round, and applies the same log rows then. Before this was done the applier stayed stopped until the process restarted.
- **One process per index directory.** A second application process that opens the same index directory fails with `IndexLockedException`; give each instance its own directory (they apply the same log independently).

## How it is tested

All numbers below are for the code in this repository at the time of writing; each is re-run before a release. Seeds and commands are in the harness (`tests/Durability`, `tests/EfDurability`, `tools/ef_kill_campaign.sh`).

| Check | What it does | Result |
|---|---|---|
| Fault sweep (`Durability sweep`) | A seeded workload (upserts, deletes, merges, commits, held snapshots, vectors) under the fault injector, with a power cut at every storage operation of every run, each of three crash policies (lose everything unsynced, tear, reorder) and several crash seeds, plus a failed operation at every index with and without a power cut. After each cut the storage must open, hold exactly an acknowledged commit state (never fewer commits than acknowledged), verify every file, give identical content and query results, accept a writer again and leave no stray files. | x64: 367,458 schedules, 0 failures. arm64: 198,030 schedules, 0 failures. The workload includes a clustered (IVF) vector field beside the HNSW one. |
| Evolution sweep (`Durability evolve-sweep`) | The same under a schema change that rebuilds every segment in the background, cut at every operation of the rebuild and the swap. | x64: 208,978 schedules, 0 failures. arm64: 116,809, 0 failures. |
| Kill harness (`Durability kill`) | A child process writes, merges and commits; the parent kills it with SIGKILL at random moments (including inside a commit call), reopens the directory and compares it with the commits the child acknowledged on a side channel: it must open, show every acknowledged commit, have the exact content of one of them, and pass a query check. | x64: 4,000 kills in four runs (seeds 31100 to 34100), 3,195 commits acknowledged, 2,362 kills inside a commit call, 0 failed opens, 0 lost commits, 0 wrong results. arm64: 2,100 kills in three runs, 3,581 commits, 1,029 inside a commit call, 0 failures. |
| Kill harness, schema evolution (`kill --evolve true`) | The same while the child rebuilds segments after a schema change. | x64: 500 kills (210 commits, 151 inside a commit), 0 failures. arm64: 500 kills (759 commits, 172 inside), 0 failures. |
| Reader processes (`Durability procs`) | One writer process and four reader processes; readers hold snapshots across the writer's commits and merges and verify them; the parent kills writers and readers at random, restarts them, starts second writers that must be refused, and checks the directory against acknowledged commits. | x64: 500 kills, 2,026 commits, 54,820 snapshot verifications, 100 second writers refused, 0 problems. arm64: 500 kills, 2,566 commits, 33,420 verifications, 100 refused, 0 problems. |
| Sparse-vector kills | Kills during sparse-vector writes. | x64: 100 kills, 231 commits, 14,931 documents checked, 0 failures. arm64: 100 kills, 501 commits, 16,050 documents, 0 failures. |
| Embedding kills | Kills during writes that embed text at upsert (the embedding cache is part of the stored source). | x64: 100 kills, 130 commits, 14,435 documents, 0 failures. arm64: 100 kills, 231 commits, 14,832 documents, 0 failures. |
| Large flush, every cut (`PackSpillFaultTests`) | A flush whose parts exceed the 4 MiB pack limit (the spill path the sweeps' small segments never reach), cut at every storage operation under every crash policy. | 0 failures. |
| Failed call, any point (`TransientFailureTests`) | One failed storage call at every operation of two commits: the writer either retries successfully with every document or refuses; an index reopened holds every acknowledged commit and no stray file. | 0 failures. |
| EF kill campaign, SQLite (`tools/ef_kill_campaign.sh`) | Per cycle two application processes (each with its own index directory) apply inserts, updates and deletes to one database while an applier (inline or threaded) runs; each is killed at a random time or at a named stage of the applier (read, before and after commit, rebuild steps); sometimes the log is pruned past the indexes. Each index is then reopened, caught up and compared with the database row by row, and its position is compared with the positions it acknowledged. | 1,276 kills (400 at named stages of the applier, 876 at random times; four lanes of 150 cycles, seeds 81000 to 81003), 472 index checks, 209 rebuilds after retention, 0 first-pass mismatches, 0 failed opens, 0 lost commits, 0 differences from the database. |
| EF kill campaign, PostgreSQL 16.2 | The same with two concurrent writers per process and transactions held open up to 40 ms, so later positions become visible first (a user-local server from the `pgserver` package, no container). Two settings: retention below the late-row window, and retention and window both 1,000. | 1,809 kills (the two settings: 926 and 883; seeds 82000 to 82003 and 83000 to 83003), 654 index checks, 331 rebuilds, 3,433 commits made while positions were still missing, 372 rows applied because they became visible below the mark, 0 first-pass mismatches, 0 failed opens, 0 lost commits, 0 differences. An earlier run of the same campaign, before it waited for a killed client's last transaction to end before comparing, reported one mismatch of three rows (one save) in about 8,000 kills (seed 9203, cycle 23); 6,900 further kills of that version did not repeat it. The mismatch is consistent with the comparison racing a killed client's final commit, but that was not proven. |
| `.Files` resume campaign (`oneindex-files-tool campaign`) | A generated corpus of 6,191 files (text, Markdown, HTML, PDF, archives, corrupt files); the indexer process is killed 100 times at random moments and resumed; the final index is compared with an uninterrupted run. | 100 kills delivered (0.3 to 2.5 s after each start), every reopen succeeded, the file-record count never fell, 0 differences between the resumed and the clean index (12,844 file records, 61 error documents for the corrupt files, 581,678 passages), 0 duplicate records, 0 orphan passages. |

The sweeps run on a volatile in-memory disk behind the fault injector, so they test the engine's protocol, not the host file system; the kill harness, the reader-process campaign and the EF and Files campaigns use real files on real disks (x64 Linux, arm64 macOS APFS). Kills are process kills, which are covered by the guarantee in full; power cuts are not exercised on real hardware. The Entity Framework and `.Files` campaigns ran on x64 Linux only.

## Reporting a problem

A `CorruptIndexException` names the file. Keep the directory, the exception text and the process history (was a process killed, was a disk full) and open an issue; do not delete files by hand.