A document bin stores annotated text with a shared string table.
dotnet add package RecursiveAI.OneNlpRequires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
// annotated.conllu: zip input shown below. using RecursiveAI.OneNlp; using RecursiveAI.OneNlp.Data; using RecursiveAI.OneNlp.Serialization; var bin = new DocBin(); foreach (var sentence in ConllUReader.Read("annotated.conllu")) { sentence.Doc.SetCats([new("ANIMAL", 1f)]); bin.Add(sentence.Doc); } File.WriteAllBytes("docs.bin", bin.ToBytes()); var restored = DocBin.FromBytes( File.ReadAllBytes("docs.bin")); Console.WriteLine($"Restored {restored.Count} documents");
Restored 2 documentsUnzip the example, then dotnet run.
Full program
The full program adds an ANIMAL category to each document, writes docs.bin and checks the restored documents against the original JSON. It then tries a truncated copy. The offsets printed below are half-open UTF-16 ranges: the start is included and the end is excluded.
annotated.conllu (tab-separated):
1 Cats cats NOUN NNS _ 2 nsubj _ _ 2 chase chase VERB VBP _ 0 root _ _ 3 birds birds NOUN NNS _ 2 obj _ _ 1 Dogs dogs NOUN NNS _ 2 nsubj _ _ 2 chase chase VERB VBP _ 0 root _ _ 3 cats cats NOUN NNS _ 2 obj _ _
using RecursiveAI.OneNlp; using RecursiveAI.OneNlp.Data; using RecursiveAI.OneNlp.Serialization; var bin = new DocBin(); foreach (var sentence in ConllUReader.Read("annotated.conllu")) { sentence.Doc.SetCats([new("ANIMAL", 1f)]); bin.Add(sentence.Doc); } File.WriteAllBytes("docs.bin", bin.ToBytes()); var restored = DocBin.FromBytes( File.ReadAllBytes("docs.bin")); for (int i = 0; i < restored.Count; i++) { var doc = restored.Docs[i]; if (doc.ToJson() != bin.Docs[i].ToJson()) throw new InvalidDataException("Round-trip changed annotations."); Console.WriteLine($"{doc.Text} | ANIMAL={doc.Cats["ANIMAL"]}"); foreach (var token in doc) Console.WriteLine($" {token.Text} [{token.Idx},{token.EndIdx}) head={token.HeadIndex} {token.Dep}"); } Console.WriteLine("All document JSON matches after binary round-trip"); byte[] bytes = File.ReadAllBytes("docs.bin"); try { DocBin.FromBytes(bytes.AsSpan(0, bytes.Length - 1)); throw new Exception("Truncated bin was accepted."); } catch (InvalidDataException) { Console.WriteLine("Truncated bin: InvalidDataException"); }
dotnet run prints:
Cats chase birds | ANIMAL=1 Cats [0,4) head=1 nsubj chase [5,10) head=1 root birds [11,16) head=1 obj Dogs chase cats | ANIMAL=1 Dogs [0,4) head=1 nsubj chase [5,10) head=1 root cats [11,15) head=1 obj All document JSON matches after binary round-trip Truncated bin: InvalidDataException
Truncated or invalid binary input throws InvalidDataException; a bin reader checks the checksum after the last document.
Variations
Read documents one at a time
After writing docs.bin, enumerate its records instead of materializing a second whole bin:
using var stream = File.OpenRead("docs.bin"); foreach (var doc in new DocBinReader(stream)) Console.WriteLine(doc.Text);
Cats chase birds Dogs chase cats
Related recipes
- Use pretrained part-of-speech tagging
- Use a pretrained dependency parser
- Read CoNLL-U training data
- Save NLP results as JSON
The short answer
Use OneNLP’s DocBin to serialize several annotated documents into one binary container. Add each Doc, write ToBytes() to a file, then reconstruct it with DocBin.FromBytes. Use DocBinReader when you want to consume records one at a time. This is a document format, not general-purpose object serialization or a trained-model checkpoint.
How do you serialize an NLP document to binary in C#?
Create or load annotated documents first, then add them to a bin. A bin keeps references rather than copies, so changes made to documents before serialization affect what is written. Documents may come from different StringStore instances; the writer constructs the shared table for the resulting container. The binary format names the many-document container ONDB and the single-document container ONDC.
The example starts from CoNLL-U so that the inputs visibly contain token boundaries, POS labels, heads and dependency relations. It adds a document category in code. The full program compares the original and restored JSON representations, then prints text, offsets, head indices and the category score. Those checks exercise this particular fixture; they are not a substitute for checking whichever annotations your own application relies on.
The shared string table stores repeated annotation values once across the bin. It is not a table of trained weights. If you need to keep an encoder and its prediction heads, save a pipeline file instead. If you need human-readable annotated text for inspection or interchange, document JSON is the corresponding alternative.
How do you deserialize binary documents in C#?
DocBin.FromBytes materializes the complete bin. DocBinReader takes a stream and yields records in order, allowing you to process each document without holding the full restored collection. Its constructor reads the header and string table; each read then decodes a record. The checksum is verified after the last record. Finish enumeration if you need that final checksum check.
The stream variation opens the file with using, which disposes the file handle after enumeration. Streaming the read does not make the write incremental: WriteBinary uses the same bytes as ToBytes, and the implementation builds them before writing. Plan memory use around the actual API rather than assuming that every stream overload avoids buffering.
How does binary serialization differ from JSON serialization?
Both can preserve document annotations, but their representations serve different needs. JSON is readable text; the binary containers use section lengths, table indices, token offsets and an optional CRC-32 trailer. The binary specification is language-neutral and versioned. Unknown sections can be skipped by length, so readers can retain the parts they understand.
Keep readers and writers aligned with the annotations you need. The pinned specification notes that categories and span groups were added to version 1 after its first release, and older readers skip those sections. The released package used here round-trips the category in the example bin; the JSON equality check makes that visible rather than assuming every historical reader supports it.
Readers reject invalid input with InvalidDataException, including bad magic or version, truncation, invalid indices and failed checksums. The documented token-count cap is 33,554,432. Stream readers accept at most a declared length of 1 GiB and read it in 1 MiB steps, allocating no more than the stream supplies. These are format-reader limits, not a recommended document size or a guarantee that a process has enough memory for the maximum. The truncated-input case in the program shows how to catch the same exception your application should handle when a stored file is incomplete.