IStorage is a flat set of named files. It is the only way the engine touches bytes, so one implementation can be local files, memory, read-only HTTP ranges or browser storage, and a wrapper can see (and break) every write.
Names
A name is 1 to 255 characters from A-Z a-z 0-9 . _ - and does not start with a dot. There are no directories. A name can therefore never leave the storage root, and it is safe in a URL.
Reading
OpenRead(name) returns an IRangeReader with a fixed Length. Read(offset, buffer) fills the caller's buffer and returns the bytes read; it returns fewer than requested only when the range reaches the end of the file, and 0 at or past the end. ReadExactly turns a short read into CorruptIndexException. A reader is safe to use from several threads. The local-file backend serves files of 64 KiB or more on Unix from a read-only memory mapping, copying the requested range into the caller's buffer (no system call on a cached range); smaller files, Windows (where a mapped file could not be deleted by a merge), the browser, and any file that cannot be mapped use positional reads. On Windows readers use ordinary, not overlapped, handles: a synchronous read on an overlapped handle allocates about 176 bytes per call there (measured), which would break the allocation-free hot path, so the async read completes inline. Writers allow sharing delete too, so a file a crashed or abandoned writer left open can still be removed. Every read still goes through IRangeReader, so the fault injector and the transform wrappers see the same calls, and the engine never holds a pointer into a mapping. Index files are immutable, so a mapping never sees a file change; a range past the length seen at open reads short like any other. The hot path (a cached range) neither awaits nor allocates; the async form exists for backends that really wait (HTTP).
Writing
Create(name) makes a new empty file (it fails if the name exists) and returns an append-only IStorageWriter. Sync() flushes and fsyncs the file; bytes are durable only after it returns. Disposing a writer without syncing hands bytes to the operating system but promises nothing about a crash. A storage that reports SupportsConcurrentSync (the local-file backend, except in the browser) accepts Sync() calls on writers of different files from different threads at the same time; the index writer uses this to flush all files of a new segment together, because on a disk with an expensive cache flush the concurrent fsyncs complete in far less than the sum of their times. Storages that do not report it are synced one file after the other, with the same result.
Rename(source, destination) atomically replaces the destination, also when a reader has the destination open (on Windows through ReplaceFile, which swaps the names when the other handle allows sharing delete, as every handle of this storage does). Delete(name) ignores a missing file. List() returns names in ordinal order. SyncDirectory() makes creations, renames and deletions durable.
TryLock(name) (where SupportsLocks is true: local files and memory) creates the named file if needed and takes an exclusive lock on it, or returns null if another holder has it. The engine uses it for the writer lock and reader leases; see docs/processes.md for the per-OS behaviour. The fault injector counts a lock on a file that did not exist as a Create, and a power cut releases every lock it handed out.
A read-only backend reports IsReadOnly and throws NotSupportedException from every mutating member.
What durable means, by platform
The commit protocol needs two promises: a synced file's bytes survive a crash, and, after SyncDirectory, so do the names created or renamed before it.
| Platform | File data (Sync) | Names (SyncDirectory) |
|---|---|---|
| Linux (ext4, XFS, btrfs) | fsync on the file | .NET cannot open a directory to fsync it, so each file renamed into place, or created and not yet synced, since the last call is fsynced. A file that was synced after it was created is not synced a second time: that sync already committed its directory entry. In practice these file systems commit the directory entry in the same journal transaction, which makes the entry durable (not a POSIX guarantee; see below). Deletions are not forced, which is harmless: a deleted file that returns after a crash is garbage no commit names. |
| macOS | full sync of the file (F_FULLFSYNC, as .NET does for FlushToDisk) | the same file flushes |
| Windows (NTFS) | FlushFileBuffers | the same file flushes; NTFS journals metadata |
| Memory | nothing is ever durable beyond the process | no effect |
Why SyncDirectory does not fsync the directory
.NET has no managed way to open a directory for syncing. On Linux (.NET 10.0.11) every attempt throws UnauthorizedAccessException: File.OpenHandle and FileStream on the directory path, on <dir>/., and on /proc/self/cwd, read-only or read-write. A SafeFileHandle can only wrap a descriptor that something else opened, and getting one needs open(2), which means native interop; spawning sync as a child process is possible but absent in sandboxes and in the browser, so the engine does not do it. (Opening a directory is not possible with FileStream on Windows either; this was not tested here.) The BCL offers no other durable-directory call, so the engine does not claim one.
What the engine does instead, and what that guarantees:
- Process kill, crash of the application,
kill -9: fully guaranteed on every platform. The operating system has every completed write and rename, whatever the disk does later. The kill harness exercises exactly this. - Power loss or kernel crash, Linux: POSIX asks for a directory fsync to make a created or renamed name durable. Without one, durability rests on how the file system journals: on ext4, XFS and btrfs, fsync of a file also commits the journal or log entry that created or renamed it, so syncing each new file, then the renamed commit file, after the rename makes the name durable in practice. That is a property of those implementations, not of POSIX; it was not tested by pulling power, and a file system that behaves differently (some network and FUSE file systems) can lose a rename it acknowledged. The loss is a rollback to the previous commit, never a corrupt index: the commit protocol only ever lists files that were synced and verified, and a reader checks every listed file's length and checksum.
- Power loss, macOS and Windows:
FlushToDisk(F_FULLFSYNC/FlushFileBuffers) of the file, and the same reliance on the file system's metadata journal. Nothing here was tested with a power cut; the worst case is the same rollback. - Memory: nothing outlives the process.
The fault injector models the strict case (a rename survives only after SyncDirectory), and the engine passes it, so an application that can run a directory fsync of its own (for instance from a supervising process) can add it without changing the engine.
The fault-injecting wrapper
FaultInjectingStorage wraps a writable storage and models a disk with a volatile cache. It sees every Create, Write, Sync, Rename, Delete and SyncDirectory and numbers them. Its model:
- A file's bytes become durable at that file's
Sync. - A name (creation, rename, deletion) becomes durable at
SyncDirectory. - A rename is atomic: it survives whole or not at all.
A seeded schedule can fail chosen operations (IOException, no effect) and can cut the power at a chosen operation (SimulatedCrashException; a write that is cut is recorded first, so it may survive torn). Crash(policy) then rewrites the wrapped storage to what a real disk may hold:
| Policy | Unsynced writes | Unsynced name changes |
|---|---|---|
LoseUnsynced | all lost | all lost |
Tear | an ordered prefix survives, the write at the cut is torn | an ordered prefix survives |
Reorder | each survives whole, torn or not at all, so a later write can persist without an earlier one (a zero-filled gap) | each survives or not independently |
The same seed gives the same result. After Crash the wrapper is usable again with its counter reset.
Block transform
TransformingStorage applies an IBlockTransform to every byte by file position: Encode(offset, data) on write and Decode(offset, data) on read, in place and length-preserving. The identity transform changes nothing. The slot is where encryption at rest will go; the transform must not depend on a file's name because files are renamed into place.
Read-only opens, static chunks and OPFS
SearchIndex.OpenReadOnly<T>(path) opens the newest commit of an index directory without the writer lock: it works while another process writes, on read-only storage, and Upsert, Delete and Commit throw NotSupportedException. A read-only storage reports IsReadOnly. ChunkedStaticStorage (static-index.md) is a read-only IStorage over fetched chunks. In the browser, OpfsPersistence saves and loads an index held in MemoryStorage whole through an application-supplied IOpfsHost (browser.md); the fault-injecting wrapper sits in front of the same storage interface, and the OPFS save was checked by crashing a fake host at every change index until the save completed (each state loaded and opened as the old or the new commit).
Limits and exceptions
IndexLimits bounds every count and length a reader accepts (commit size, segments, files per segment, string length, schema size, fields, vector dimensions). CorruptIndexException names the damaged file; UnsupportedFormatException (a subclass) means the file is intact but newer than this build understands.