Recursive AI

Serialize annotated NLP documents to binary in C#

OneNLP is a free, open-source natural language processing library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

A document bin stores annotated text with a shared string table.

Terminal
dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
// annotated.conllu: zip input shown below.
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Data;
using RecursiveAI.OneNlp.Serialization;
 
var bin = new DocBin();
foreach (var sentence in ConllUReader.Read("annotated.conllu"))
{
    sentence.Doc.SetCats([new("ANIMAL", 1f)]);
    bin.Add(sentence.Doc);
}
File.WriteAllBytes("docs.bin", bin.ToBytes());
var restored = DocBin.FromBytes(
    File.ReadAllBytes("docs.bin"));
Console.WriteLine($"Restored {restored.Count} documents");
Text
Restored 2 documents

Unzip the example, then dotnet run.

Full program

The full program adds an ANIMAL category to each document, writes docs.bin and checks the restored documents against the original JSON. It then tries a truncated copy. The offsets printed below are half-open UTF-16 ranges: the start is included and the end is excluded.

annotated.conllu (tab-separated):

Text
1	Cats	cats	NOUN	NNS	_	2	nsubj	_	_
2	chase	chase	VERB	VBP	_	0	root	_	_
3	birds	birds	NOUN	NNS	_	2	obj	_	_
 
1	Dogs	dogs	NOUN	NNS	_	2	nsubj	_	_
2	chase	chase	VERB	VBP	_	0	root	_	_
3	cats	cats	NOUN	NNS	_	2	obj	_	_
C#
using RecursiveAI.OneNlp;
using RecursiveAI.OneNlp.Data;
using RecursiveAI.OneNlp.Serialization;
 
var bin = new DocBin();
foreach (var sentence in ConllUReader.Read("annotated.conllu"))
{
    sentence.Doc.SetCats([new("ANIMAL", 1f)]);
    bin.Add(sentence.Doc);
}
File.WriteAllBytes("docs.bin", bin.ToBytes());
var restored = DocBin.FromBytes(
    File.ReadAllBytes("docs.bin"));
for (int i = 0; i < restored.Count; i++)
{
    var doc = restored.Docs[i];
    if (doc.ToJson() != bin.Docs[i].ToJson())
        throw new InvalidDataException("Round-trip changed annotations.");
    Console.WriteLine($"{doc.Text} | ANIMAL={doc.Cats["ANIMAL"]}");
    foreach (var token in doc)
        Console.WriteLine($"  {token.Text} [{token.Idx},{token.EndIdx}) head={token.HeadIndex} {token.Dep}");
}
Console.WriteLine("All document JSON matches after binary round-trip");
byte[] bytes = File.ReadAllBytes("docs.bin");
try
{
    DocBin.FromBytes(bytes.AsSpan(0, bytes.Length - 1));
    throw new Exception("Truncated bin was accepted.");
}
catch (InvalidDataException)
{
    Console.WriteLine("Truncated bin: InvalidDataException");
}

dotnet run prints:

Text
Cats chase birds | ANIMAL=1
  Cats [0,4) head=1 nsubj
  chase [5,10) head=1 root
  birds [11,16) head=1 obj
Dogs chase cats | ANIMAL=1
  Dogs [0,4) head=1 nsubj
  chase [5,10) head=1 root
  cats [11,15) head=1 obj
All document JSON matches after binary round-trip
Truncated bin: InvalidDataException

Truncated or invalid binary input throws InvalidDataException; a bin reader checks the checksum after the last document.

Variations

Read documents one at a time

After writing docs.bin, enumerate its records instead of materializing a second whole bin:

C#
using var stream = File.OpenRead("docs.bin");
foreach (var doc in new DocBinReader(stream))
    Console.WriteLine(doc.Text);
Text
Cats chase birds
Dogs chase cats

The short answer

Use OneNLP’s DocBin to serialize several annotated documents into one binary container. Add each Doc, write ToBytes() to a file, then reconstruct it with DocBin.FromBytes. Use DocBinReader when you want to consume records one at a time. This is a document format, not general-purpose object serialization or a trained-model checkpoint.

How do you serialize an NLP document to binary in C#?

Create or load annotated documents first, then add them to a bin. A bin keeps references rather than copies, so changes made to documents before serialization affect what is written. Documents may come from different StringStore instances; the writer constructs the shared table for the resulting container. The binary format names the many-document container ONDB and the single-document container ONDC.

The example starts from CoNLL-U so that the inputs visibly contain token boundaries, POS labels, heads and dependency relations. It adds a document category in code. The full program compares the original and restored JSON representations, then prints text, offsets, head indices and the category score. Those checks exercise this particular fixture; they are not a substitute for checking whichever annotations your own application relies on.

The shared string table stores repeated annotation values once across the bin. It is not a table of trained weights. If you need to keep an encoder and its prediction heads, save a pipeline file instead. If you need human-readable annotated text for inspection or interchange, document JSON is the corresponding alternative.

How do you deserialize binary documents in C#?

DocBin.FromBytes materializes the complete bin. DocBinReader takes a stream and yields records in order, allowing you to process each document without holding the full restored collection. Its constructor reads the header and string table; each read then decodes a record. The checksum is verified after the last record. Finish enumeration if you need that final checksum check.

The stream variation opens the file with using, which disposes the file handle after enumeration. Streaming the read does not make the write incremental: WriteBinary uses the same bytes as ToBytes, and the implementation builds them before writing. Plan memory use around the actual API rather than assuming that every stream overload avoids buffering.

How does binary serialization differ from JSON serialization?

Both can preserve document annotations, but their representations serve different needs. JSON is readable text; the binary containers use section lengths, table indices, token offsets and an optional CRC-32 trailer. The binary specification is language-neutral and versioned. Unknown sections can be skipped by length, so readers can retain the parts they understand.

Keep readers and writers aligned with the annotations you need. The pinned specification notes that categories and span groups were added to version 1 after its first release, and older readers skip those sections. The released package used here round-trips the category in the example bin; the JSON equality check makes that visible rather than assuming every historical reader supports it.

Readers reject invalid input with InvalidDataException, including bad magic or version, truncation, invalid indices and failed checksums. The documented token-count cap is 33,554,432. Stream readers accept at most a declared length of 1 GiB and read it in 1 MiB steps, allocating no more than the stream supplies. These are format-reader limits, not a recommended document size or a guarantee that a process has enough memory for the maximum. The truncated-input case in the program shows how to catch the same exception your application should handle when a stored file is incomplete.

All onenlp recipes