Evaluation

Speech recognition for .NET. These tools score every system with the same reference text and normalizer. They are not part of the runtime package. The project is unaffiliated with other products or companies named OneSpeech.

Normalizer version 1

en is our English scoring normalizer, written from the descriptions in Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, arXiv:2212.04356, Appendix C and §4.4, and The Open ASR Leaderboard, arXiv:2510.06961, §2.3. It is not a claim of byte-exact equivalence to either project's software. Report en/1 in every score row; changing these rules requires a new version and rescoring both systems.

Repetitions and repairs are preserved. The leaderboard §2.2 discusses dataset transcriptions with those disfluencies removed; it does not specify an executable semantic repair algorithm. Deleting repeated words could conceal actual decoder errors. All finite tables above are project choices where the papers leave details unspecified, not transcriptions of a reference implementation's tables.

basic/1: NFKC, Unicode 17.0.0 default full case folding (CaseFolding.txt C and F mappings, no Turkic tailoring), replace Unicode M/S/P categories with spaces, collapse whitespace; retains precomposed accents. The generated case-fold table comes from the Unicode Character Database under Unicode License V3 (notice in eval/OneSpeech.Eval/Unicode/LICENSE.txt). CER counts Unicode scalar values after normalization, omitting whitespace; no grapheme segmentation. Use CER, rather than whitespace WER, for text without word separators. The sources describe basic multilingual normalization but do not supply language-specific linguistic rules.

Metrics

WER = (substitutions + deletions + insertions) / reference word count. Corpus WER is the ratio of summed counts, not the mean of utterance rates. CER uses the same formula over scalar values. Levenshtein cost is one per edit, zero for an exact match. Ties prefer diagonal, then deletion, then insertion. Counts use linear-row memory; optional alignment traces are bounded to 16 million cells. An empty reference has undefined rate (null), but its insertion count still contributes to a corpus with a nonempty reference. Extra and missing IDs are errors; a failed decode must be an explicit empty hypothesis.

Bootstrap: sort utterances by ordinal ID, draw N utterances with replacement for each replicate, summing counts before dividing. Two systems use the same draw. Difference is first WER minus second WER, in rate units (multiply by 100 for percentage points). Default 1000 resamples, seed 1729; fewer than 1000 are refused. SplitMix64 selects an index by unsigned remainder modulo N. Sort replicates and report percentile bounds at floor(0.025×(B−1)), ceil(0.975×(B−1)). These are descriptive percentile intervals, not hypothesis-test p-values. Fewer than two utterances or any zero-length normalized reference makes CIs unavailable (avoids undefined zero-denominator resamples); the corpus score is still reported.

Verdict: within-noise iff the paired difference CI includes zero. Only if a CI is unavailable, absolute difference ≤0.005 (0.5 percentage points) gives within-fallback. Otherwise different; undefined corpus rates or no second system give not-comparable. A small difference with a computable CI excluding zero does not use the fallback.

Timing: align word objects by their normalized text with the same Levenshtein tie policy. Compare only exactly matching diagonal pairs; do not interpolate unmatched tokens or split multi-token expansions into invented timed words. Report matched, timed, missing counts; mean and median absolute start/end error in milliseconds and separate start/end fractions within inclusive 50/100 ms. Missing intervals are counted, not assigned a zero error. Timing summaries appear per utterance in JSON; no identity or temporal nearest-neighbor matching is used.

Files

UTF-8 JSON lines, one object per utterance:

{"id":"1089-134686-0000","text":"Hello world","words":[{"text":"Hello","start":0.25,"end":0.65}]}

IDs must be nonblank and unique. Words are optional. Times are finite nonnegative seconds on the original file clock; both endpoints must be null or both numbers, end ≥ start. JSONL lines are limited to 2 Mi characters and JSON nesting to 32. An optional first record with "type":"header" holds reproducibility metadata and is ignored by the scorer. Records with missing text, duplicate IDs, malformed JSON or invalid word intervals fail scoring; no records are silently dropped. Input parsers are for local evaluation artifacts, not internet endpoints.

A single version-1 Transcript JSON document is also accepted through the core's validated reader, with the filename stem as utterance ID. See Transcript JSON. It cannot stand in for multiple utterances without an explicit segmentation manifest.

Audio manifest for oracle drivers (not scoring inputs): JSONL {"id":"...","audio":"/caller/selected/file.flac"} in fixed order. Ranges are zero-based, start-inclusive/end-exclusive. Relative audio paths resolve from the process's working directory. Save the manifest and its hash with the row.

Running a row

bash tools/scripts/fetch-librispeech.sh "$DATA/librispeech"
bash tools/scripts/fetch-earnings22.sh "$DATA/earnings22"
bash tools/scripts/install-oracles.sh "$ORACLES"
dotnet run --project eval/OneSpeech.Eval -- references --dataset librispeech --input "$DATA/librispeech/LibriSpeech/test-clean" --out refs.jsonl
dotnet run --project eval/OneSpeech.Eval -- references --dataset earnings22 --input "$DATA/earnings22/transcripts/nlp_references" --out earnings-refs.jsonl

LibriSpeech reads the text after the ID in each *.trans.txt. Earnings-22 reads column 1 of each pipe-separated .nlp row (eight columns, header required); punctuation/case metadata is not needed for normalized scoring. It also preserves the optional timing columns. Use original nlp_references, not force-aligned copies, so files do not duplicate IDs.

The fetch scripts verify existing archives/media without downloading them again. Text files are verified against Git blob IDs at pinned revision c05ab6fd8b4b627d123c922a22a39e993dd37635; SHA-256 and URL/licence records are written beside the cache. Existing hash mismatches fail instead of being overwritten. LibriSpeech is CC BY 4.0; Earnings-22 CC BY-SA 4.0, evaluation only, never committed or redistributed by these scripts. Extraction is from hash-verified archives. Delete the extraction marker to re-extract.

Oracle environments are separate uv projects with lockfiles and CPU torch. run.py uses only public APIs. Every system receives the same soundfile float32 decode, arithmetic channel mean, and librosa soxr_hq resampling to 16 kHz, with no amplitude normalization or ffmpeg. CPU/fp32, language English, task transcription, greedy decoding, no previous-text conditioning, and no stochastic fallback are explicit. No GPU support is offered. The output header includes the public transcribe signature defaults and DecodingOptions for OpenAI, and the model and explicitly supplied effective generation configurations for Transformers. Signature metadata is inspected without reading source or bytecode. These configurations are not a promise that long-form pipelines are equivalent. Transformers uses 30-second pipeline chunks; OpenAI uses its own long-form transcribe procedure. Consequently this driver is a reference sanity check, not an assertion of token identity.

The OpenAI driver calls whisper.load_model("tiny", download_root=...) when --model tiny is specified without --checkpoint. The released checkpoint is downloaded and verified by that public API; the driver records its SHA-256. An explicit --checkpoint can instead select an already downloaded release artifact. Use the HF tiny revision below for Transformers; do not substitute tiny.en.

wq run --class heavy --max 30m --cores 8 --label "onespeech reference tiny shard" -- \
  "$ORACLES/openai-whisper/bin/python" eval/drivers/run.py --oracle openai-whisper \
  --manifest manifest.jsonl --audio-root "$DATA/librispeech" --output tiny-0000.jsonl --model tiny --revision released-20250625 \
  --cache "$MODELS" --start 0 --end 100 --threads 8
wq run --class heavy --max 30m --cores 8 --label "onespeech reference transformers shard" -- \
  "$ORACLES/transformers/bin/python" eval/drivers/run.py --oracle transformers \
  --manifest manifest.jsonl --audio-root "$DATA/librispeech" --output hf-0000.jsonl --model openai/whisper-tiny \
  --revision 169d4a4341b33bc18d8881c4b69c2e104e1cc0af --cache "$MODELS" --start 0 --end 100 --threads 8
dotnet run --project eval/OneSpeech.Eval -- score --ref refs.jsonl --hyp tiny.jsonl --hyp2 hf.jsonl --normalizer en --resamples 1000 --seed 1729 --out scores.json

Repeat disjoint ranges through 2620 for the complete test-clean set; concatenate records (headers are ignored by the scorer). Choose smaller shards if needed to remain under the job bound. Each output header records versions, settings, range and manifest hash; completed utterances are flushed and fsynced. Resume checks the exact header and IDs. A truncated final JSON line fails rather than silently skipping it: remove only that incomplete line before resuming. Keep all shard headers with the row. The faster-whisper variant uses --oracle faster-whisper and a pinned compatible publisher checkpoint, never assumes HF safetensors are CTranslate2 weights. Fetch scripts do not download models.

Output JSON includes per-utterance WER/CER counts, timing summaries, corpus counts, bootstrap bounds and verdict; rates are fractions, console output is percent. Keep model weight hashes, dataset hashes, oracle locks, machine CPU/ISA/core count, load, thread count, SDK/runtime and full command/job IDs with measured rows. These tools alone do not constitute a measured quality claim.

Fixed targeted slice

eval/manifests/librispeech-test-clean-first100.jsonl pins the first 100 test-clean utterance IDs in ordinal order, their relative audio paths and each FLAC's SHA-256. It contains metadata only, not audio or reference text. Regenerate it from the verified cache with eval/drivers/prepare_slice.py; that script also writes a caller-selected reference JSONL outside the repository. Pass the dataset cache as --audio-root and use --start 0 --end 100. The driver checks manifest audio hashes before decoding. This small sanity check does not replace full-corpus evaluation.

The fixed slice was checked with openai-whisper 20250625 (tiny, original MIT release artifact) and Transformers 5.22.2 (openai/whisper-tiny, revision 169d4a4341b33bc18d8881c4b69c2e104e1cc0af). Both used fp32 CPU, greedy decoding, temperature 0 and English. Original tiny.pt SHA-256: 65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9; it is a different artifact from the HF safetensors. With en/1, 1000 paired bootstrap resamples and seed 1729:

Paired difference (OpenAI minus HF): 0.085 percentage points, 95% CI [−0.626, 1.077], verdict within-noise. This is a small correctness check, not a full-corpus quality claim or a performance comparison. The run used Linux x64 on an AMD Ryzen 9 7950X3D (16 cores / 32 logical CPUs), eight oracle threads, torch 2.15.0+cpu, numpy 2.5.4, soundfile 0.13.1 and librosa 0.11.0; no GPU. Full per-utterance hypotheses and resolved decoding headers are evaluation artifacts, not embedded library data.

Diarization scoring

onespeech-eval der --ref reference.rttm --hyp hypothesis.rttm --out der.json

The der command implements the definitions in the Third DIHARD Challenge Evaluation Plan, version 1.2 (2020-10-30), §4 and Appendix C. This is a scorer specification reference only: no DIHARD data is fetched. Rates in JSON are fractions, not percentages. Output includes reference speaker-seconds, miss, false alarm, confusion, reference speaker count, DER and JER. No collar is applied and all overlap is scored. Scoring rejects endpoints beyond TimeSpan.MaxValue.TotalSeconds, even if a syntactically valid RTTM contains such numbers; this avoids overflowing assignment arithmetic on impossible audio clocks. These definitions are implemented independently, with hand-computed tests; an external released-scorer cross-check is still pending.

Each recording/channel is scored separately with independent, optimal one-to-one speaker mappings; speaker names have no significance across recordings. Corpus DER sums error counts before dividing by total reference speaker-seconds. Duplicate or overlapping segments of the same speaker are unioned. For each constant-activity interval of duration d, with R reference speakers, H hypothesis speakers and C correctly mapped concurrent speakers:

• miss = d × max(0, R−H);
• false alarm = d × max(0, H−R);
• confusion = d × (min(R,H)−C).

The DER mapping maximizes total pairwise intersection duration via Hungarian assignment. Reference speaker-time includes multiplicity in overlap. DER can exceed 100%. Empty reference speech has undefined DER (null), but false-alarm seconds are retained.

JER uses a separate Hungarian assignment maximizing the sum of per-reference-speaker intersection-over-union. An unmatched reference speaker contributes error 1. JER is the reference-speaker-macro mean, including across recordings; unmatched hypothesis speakers do not add terms of their own (but can affect the pairing/union). No reference speakers means undefined JER (null). Maximum 512 distinct speakers per recording/channel; more is refused rather than allocating an unbounded assignment matrix.

The scored domain is the union of all reference and hypothesis intervals, including hypothesis-only files/channels, which count as false alarms. There is currently no UEM support: do not use this command for a benchmark with excluded or restricted scoring regions without first clipping both sides to those regions. Missing hypothesis files are treated as empty, never dropped from corpus denominators. Pure silence outside all listed turns contributes zero.

RTTM input is strict UTF-8 and accepts blank lines, # comment lines and ten-field SPEAKER records. Fields are type, recording ID, positive integer channel, onset seconds, duration seconds, orthography, speaker type, speaker ID, confidence and lookahead. Onset must be finite and nonnegative; duration positive, finite and distinguishable from zero when added to onset. Unsupported record types, malformed fields, invalid UTF-8, lines exceeding 16,384 UTF-16 code units or more than one million segments are rejected. The confidence/lookahead/orthography/type placeholders are not used by scoring. Segments of the same speaker are treated as a set even when duplicated.

Either input may instead be a version-1 Transcript JSON file (extension .json). Assigned turns become RTTM-style channel-1 segments, ignoring turn text and words; Unassigned turns are omitted. Set --recording meeting-id to identify the transcript when comparing JSON to RTTM. Default ID is recording; JSON-vs-JSON inputs therefore share a clock by default. A transcript has no channel field, so a multi-channel RTTM must be converted explicitly rather than silently conflated.

fuzz/run-rttm-smoke.sh <absolute-scratch-directory> [seconds] exercises the bounded RTTM reader through SharpFuzz/libFuzzer, with the same pinned development tools as the other reader smoke scripts; maximum duration 300 seconds.

CAM++ diarization development and floor runner

onespeech-eval diarize runs the fixed meeting slices with a SHA-verified converted CAM++ English graph. Paths below are caller-selected caches/result directories outside git. Fetch/prepare datasets first with the scripts in tools/scripts/; fetch-ami-dev-channels.sh adds only the four development headset files for channel-layout calibration.

onespeech-eval diarize dev GRAPH AMI_ROOT ES2011a RESULTS
onespeech-eval diarize dev GRAPH AMI_ROOT ES2011b RESULTS
onespeech-eval diarize select RESULTS
onespeech-eval diarize channels AMI_ROOT RESULTS
onespeech-eval diarize holdout GRAPH AMI_ROOT ES2004a RESULTS S128 A64-0.15
onespeech-eval diarize holdout GRAPH AMI_ROOT ES2004b RESULTS S128 A64-0.15
onespeech-eval diarize holdout GRAPH ALI_ROOT R8001_M8004_MS801 RESULTS S128 A64-0.15
onespeech-eval diarize holdout GRAPH ALI_ROOT R8003_M8001_MS801 RESULTS S128 A64-0.15
onespeech-eval diarize summary RESULTS
onespeech-eval diarize excerpt GRAPH AMI_ROOT ES2004a RESULTS/excerpt S128

Development commands refuse other IDs or non-development manifest entries. Selection requires the complete 57-profile grid on both development meetings, minimizing pooled DER, then speaker-macro JER, then ordinal profile ID. Freeze selection.json and channel-selection.json by SHA-256 before running holdouts; do not choose parameters from holdout scores. The example profile IDs are the measured selection in diarization, not an instruction to select them using a new holdout set. The holdout command accepts only the four named held-out IDs and explicitly supplied profile names. Completed meeting score files are not silently overwritten; a new experiment uses a new result root. Per-meeting completion makes runs resumable without combining partial meetings.

Only the first profile for a meeting runs embedding/VAD inference; subsequent clustering profiles replay in-RAM outputs after verifying identical PCM hashes and sequence lengths. Model weights are never fitted, caches are discarded at the meeting boundary, and no embeddings are written. All rows record input/reference/manifest SHA-256, checkpoint revision, converted SHA, precision, threads, VAD settings, clustering settings, reference speaker-seconds, miss/FA/confusion, DER/JER, speaker/window counts and hypothesis RTTM path. No collar; overlaps scored; no text normalizer. summary pools error counts rather than averaging meeting DER and reports reference-speaker-macro JER. Its descriptive 95% intervals resample the two meeting rows 1,000 times with seeded .NET Random(1729), with floor/ceiling percentile indices as in the scoring method; they are not intervals over a new independent speaker population. Individual-meeting CIs are unavailable.

Channel calibration uses ten correlated layout proxies from AMI dev headset 0/1: physical separated microphones and 1% statistical leakage are positives, 25%/50% leakage and dual mono negatives. For original per-frame energies X,Y and cross-energy C, symmetric leakage alpha gives energies (X+2*alpha*C+alpha²*Y)/(1+alpha)² and its channel-swapped counterpart, and cross-energy (alpha*(X+Y)+(1+alpha²)*C)/(1+alpha)². No modified speech waveform is generated. Search 10/15/20/25/30 dB and normalized-dot limits 0.2/0.3/0.4/0.5/0.6; minimize balanced error, then distance from the initial 20 dB/0.3, then higher dB/lower dot limit. Three dominant frames and energy floor1e-8 remain fixed. This proxy fit is not a telephony accuracy estimate.

The excerpt command uses the real diarizer but a generated word-timing fixture, because ASR integration is separate. It saves a ≤300-second derived WAV in the dataset cache and emits a Transcript JSON plus an explicit fixture summary, including assigned/unassigned counts and JSON timing quantization. It is not an ASR hypothesis, WER or word-alignment evaluation. See diarization for measured floors and limitations.

tools/oracles/speaker-modelscope/diarize.py probes the released ModelScope public pipeline using only the approved English checkpoint already present locally, on CPU, with network disabled. It never opens installed implementation files or prints dependency exception frames. A registry failure or need for unchecked auxiliary models is reported as unavailable, not as a scored reference. The existing locked oracle environment supplies dependencies; no repository recipe is executed.