Recursive AI

Measure OCR character and word error rates in C#

OneOcr is a free, open-source OCR library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Score an OCR model on lines it did not train on by counting character and word edits. The zip’s program exports recognizer.onm from the bundled models and runs oneocr eval; the snippet scores the same file directly in C#.

Terminal
dotnet add package RecursiveAI.OneOcr.Training

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using OneOcr.Training;
 
const string model = "recognizer.onm";
var report = OcrTrainer.Evaluate(model, new() {
    ModelPath = model,
    WorkDirectory = "eval-data"
});
Console.WriteLine(OcrTrainer.ToJson(report));

Unzip the example, then dotnet run.

Prints character and word error rates as JSON. The zip includes eval-data/ with the line crops for Rack 24, Bin 51, and Shelf 82, plus their hand-checked references. Each labels/*.json stores its line’s cropWidth and base64 crop bytes; reviewed.json stores the verified text. One review entry:

JSON
{ "id": "4e1c39d61a01518a-0", "text": "Rack 24" }

The part before -0 is the label file’s id. The default held-out split selects one of the three fixture documents; each has one line, so only one line is scored. The snippet prints:

JSON
{
  "rates": {
    "cer": 0,
    "wer": 0,
    "lines": 1,
    "exactLines": 1
  },
  "suggestedLocalFirstThreshold": null,
  "thresholdSupport": 0
}

suggestedLocalFirstThreshold is a suggested confidence cutoff for accepting a local reading, and thresholdSupport counts the scored lines at or above it; one held-out line isn’t enough to suggest a cutoff.

Full program

dotnet oneocr eval runs the tool pinned in the zip’s dotnet-tools.json, so dotnet run does not need a global tool installation. To build the runner in your own console project, also install the package it uses to get the bundled recognizer:

Terminal
dotnet add package RecursiveAI.OneOcr.Models
C#
using System.Diagnostics;
using OneOcr.Models;
 
File.WriteAllBytes("recognizer.onm", BundledModels.ReadRecognizer());
Run("tool", "restore");
Run("oneocr", "eval", "--model", "recognizer.onm", "--work", "eval-data");
 
static void Run(params string[] arguments)
{
    var command = new ProcessStartInfo("dotnet");
    foreach (string argument in arguments)
        command.ArgumentList.Add(argument);
    using var process = Process.Start(command)!;
    process.WaitForExit();
    if (process.ExitCode != 0)
        throw new InvalidOperationException($"dotnet {arguments[0]} failed.");
}

After restoring the tool, the program prints the same JSON report as the C# snippet. Restore can fetch packages from NuGet; evaluation itself does not use the network. If no held-out line has reference text, eval fails instead of printing a zero-error report.

Variations

In the unzipped example, after dotnet run, evaluate two of the three fixture documents by setting --held-out:

Terminal
dotnet oneocr eval --model recognizer.onm --work eval-data --held-out 0.67

The report then shows "lines": 2 and "exactLines": 2 for these crops.

The short answer

Call OcrTrainer.Evaluate in C#, or run oneocr eval, with a recognizer file and a local work directory of transcribed line crops. Either compares the recognizer’s readings with the references in the held-out split. Read rates.cer and rates.wer as ratios, and check rates.lines to see how many lines the report actually covers. This measures line recognition, not whether a full-page pipeline finds all the lines or puts them in the right order.

To build the files from your own line images without a provider, use the crop and transcription fields in the work-directory spec; the CLI labeling and review flow writes those files from page images if you configure a labeling provider.

How do you calculate character error rate and word error rate?

Sum the minimum number of edits needed for each line: a substitution, deletion, or insertion counts as one. Then divide by the total number of reference units across the scored lines. For CER the units are Unicode code points, including normalized spaces; for WER they are space-separated words. OneOcr normalizes both sides to NFC, trims and collapses whitespace, and keeps case and punctuation. See the scoring implementation.

For example, reading Rack 24 as Rack 2A costs one character edit out of seven, so CER is about 14.3%. It also costs one word edit out of two, so WER is 50%. The CLI prints ratios and line counts, not separate counts for insertions, substitutions, and deletions. Enough insertions can make the ratio greater than 1, since the denominator is the reference length, not the length of the OCR output.

What does a zero OCR error rate tell you?

It tells you that the scored text matched the supplied references after normalization, not that every new page will be read correctly. Here it covers only one synthetic line. Keep verified references for documents not used for training, including the kinds of print and scan defects you expect. When comparing two recognizer files against the same work directory, use the same --seed and --held-out for both. Otherwise the models may be scored on different documents.

The threshold in the report is separate from the error rates. OneOcr only suggests a confidence cutoff if at least five held-out lines support it and the requested share of lines at or above it are exact matches. This fixture is too small to suggest one, even with zero measured errors.

All oneocr recipes