Save NLP results as JSON in C#

Save a tokenized document and its annotations as JSON, then restore it as a OneNLP document.

dotnet add package RecursiveAI.OneNlp

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

using System.Text.Json;
using RecursiveAI.OneNlp;
 
using var nlp = Pipeline.Blank(Language.English);
Doc doc = nlp.Process("Ada ordered tea.");
string json = JsonSerializer.Serialize(doc);
File.WriteAllText("result.json", json);
Doc restored = JsonSerializer.Deserialize<Doc>(
    File.ReadAllText("result.json"))!;
Console.WriteLine(restored.Text);
Console.WriteLine($"Tokens: {restored.Length}");

Ada ordered tea.
Tokens: 4

The input is “Ada ordered tea.”. A blank pipeline tokenizes it without adding model annotations. The program saves result.json, reads it back and prints the restored text and token count.

Invalid document JSON passed through JsonSerializer.Deserialize<Doc> throws JsonException.

result.json contains:

{"text":"Ada ordered tea.","tokens":[{"id":0,"start":0,"end":3},{"id":1,"start":4,"end":11},{"id":2,"start":12,"end":15},{"id":3,"start":15,"end":16}]}

Unzip the example, then dotnet run.

Variations

Use the direct document API. Run this after creating doc in the full program.

string direct = doc.ToJson();
Doc copy = Doc.FromJson(direct);
Console.WriteLine(copy.Text);

Ada ordered tea.

Related recipes

• Tokenize text
• Named entity recognition

The short answer

Serialize a OneNLP Doc with JsonSerializer.Serialize, save the resulting string, and read it back with JsonSerializer.Deserialize<Doc>. OneNLP supplies the document converter through an attribute on Doc; no custom converter registration is needed. The example saves tokenized text as result.json, restores it, and prints its text and token count.

A blank pipeline only tokenizes this sentence. It does not add entity labels, part-of-speech tags or dependencies. If your pipeline has produced those annotations, the same document serializer carries the supported fields. The JSON format documentation specifies which fields are written and how missing values are treated.

How to serialize a C# object to JSON?

For a Doc, use the standard System.Text.Json calls shown above. The document’s converter writes the same format as Doc.ToJson(). The variation shows the direct document API when you do not need to embed the result in another JSON object.

The output is a document representation, not a serialized trained model. It contains the original text, token boundaries and annotations that were set. Saving it does not save your pipeline’s encoder weights or label heads. To load trained behavior for analyzing new text, save a pipeline file instead.

The writer omits annotation fields nobody set. This matters when reading stored results: an absent entity field is not the same as an explicitly empty list of entities. The latter means entities were set and none were found. Categories are written as a label-to-score object, and named span groups are written as lists of ranges and labels. Overlapping spans remain overlapping.

Which NLP annotations and offsets does document JSON preserve?

Process text with the pipeline you need, then serialize the resulting Doc, rather than serializing only its text or projecting tokens into an unrelated object. Keeping the document format preserves the connection between the text and its offsets. The example’s result.json is shown above so you can inspect exactly what was saved.

JSON offsets count Unicode code points. In-memory Doc offsets count UTF-16 code units. Those units differ after a character outside the Basic Multilingual Plane, such as many emoji. The writer and reader convert between them; do not manually reuse a JSON offset as a C# string index after such a character. End positions are exclusive, and entity and sentence ranges must align with token boundaries.

Not every possible in-memory value can be represented in this format. Unpaired surrogates and token boundaries inside surrogate pairs cause InvalidOperationException on writing. Unknown fields and user_data are skipped on reading because Doc does not store them. JSON round-tripping is therefore not a promise to preserve arbitrary application state attached elsewhere.

How to deserialize JSON in C#?

For this format, deserialize to Doc, not to a standalone Token or Span. Those two types can be written individually but cannot be read on their own. A document provides the text and context their ranges refer to.

An invalid document supplied through JsonSerializer throws JsonException. The reader checks field types, token indexes, heads, confidence ranges and offsets instead of accepting arbitrary JSON with a matching property name. For NativeAOT applications that wrap a Doc inside a DTO, list that DTO in a source-generated JsonSerializerContext as usual. The Doc converter itself does not rely on reflection.