Measured results and limits

The release revalidation table distinguishes full-split measurements from the earlier fixed-slice evidence below. The tables disclose incomplete datasets, qualified references and remaining gaps; these results are not a cross-hardware guarantee. Lower is better for all errors below. Scores are percentages unless otherwise stated. Unless noted, recognition quality rows use float32. Models and publisher revisions are pinned in the catalog; see notices for attribution. No speed comparison with a native implementation is made here.

CI convention: the release table labels absolute-score and paired-difference 95% intervals separately. The older slice tables give only intervals for our score minus the reference score, in percentage points. Recognition uses 1,000 utterance resamples, seed 1729, and the same en/1 scoring normalizer on both sides. “Within noise” means the difference interval includes zero. With no computable CI, the comparison uses a 0.5-point bound. Slices must not be interpreted as full-test results.

Release quality revalidation (2026-11-08–11)

The inference measurements started from core source dd63517 through the public recognizer, live-session and embedding APIs. Later reader hardening changes invalid-input handling, not valid-model inference. Moonshine uses its default greedy decoding, 6.5 generated tokens per audio second, no biasing, and two CPU threads. The reference uses the pinned publisher weights, Transformers 5.22.2, float32, eager attention and deterministic generation. Speaker extraction uses eight threads on each side. Whisper int8 rows use temperature-zero greedy decoding without word timestamps on both sides; the reference uses beam/best-of one, no VAD filter and previous-text conditioning. Native CPU reservations changed from eight to two threads during the resumable run; precision and decoding controls did not change. No previously saved hypotheses or embeddings were substituted for the new inference runs.

Full-split Moonshine token/text identity is 5,558/5,559 sequences. The differing test-other recording is 6070-86744-0018 (33.645 seconds); it contains punctuation and word differences, not merely a different special-token representation. The within-noise WER result does not imply token-identical decoding on every recording.

Live finalisation latency means audio-time Final availability minus an edit-matched word's estimated end, not CPU inference latency. There are 4,427 matched tokens out of 4,685 reference tokens on clean, and 2,524 out of 3,086 on other. The calculation retains 35/32 negative estimates rather than clamping them. Latency intervals resample recordings 1,000 times with seed 1729. The 200-recording floors and the earlier 20-recording floors below have different populations and must not be compared as a regression test. Scoring the retained first-20 subset separately reproduces the earlier rounded floors: WER 1.70/8.33% and mean latency 3057/1617 ms.

Speaker trials contain 10,000 same-speaker and 10,000 different-speaker pairs, 73 speakers and 1,460 distinct trial utterances. Extraction covered 1,490 utterances, including additional parity probes. Paired bootstrap resamples trials within same/different strata, conditional on this fixed population; minDCF uses prior 0.01, unit costs and normalization. These rows do not replace the unavailable VoxCeleb1-O evaluation. Model revisions are Moonshine f8e9dfd8c562c257c151a907b7b7f2fe8ff8511a, CAM++ English 032b8131a7ad812f87061955ca974c99060c5a03, and ECAPA 0f99f2d0ebe89ac095bcc5903c4dd8f72b367286.

Environment: CPU-only x64, AMD Ryzen 9 7950X3D (16 physical / 32 logical CPUs; FMA, AVX2 and AVX-512 available), .NET SDK 10.0.403 / runtime 10.0.14, OneModel 1.2.0 and OneAudio 1.7.2. Language ID uses eight threads. Runs used a shared CPU queue; these are correctness measurements, not publishable speed rows.

Coverage: full Whisper turbo/small fp32 and int8, and full Moonshine fp32/int8 rows are complete. Each LibriSpeech comparison contains all 2,620 clean or 2,939 other utterances. Parakeet full-set, Qwen first-500 CPU-limited slices, five-recording long-form comparisons and timing revalidation are in the additional ASR section below. Full-set diarization and Nemotron streaming are reported in their separate section below. The fixed shortest-duration Earnings-22 selection is 4474955, 4483046, 4468919, 4479524, 4481221 (875.020, 1193.180, 1214.487, 1226.213, 1236.904 seconds). Its measured long-form rows and both temperature policies are reported below. Earlier misses below remain visible and are not superseded by unmeasured rows.

Reader mutation campaigns

Each listed target completed a 600-second mutation budget (libFuzzer reported 601 seconds), with one fuzzer running at a time and four CPU cores reserved for publishing and mutation. SharpFuzz 2.2.0 instruments a disposable copy of the assembly containing the reader; libfuzzer-dotnet is release v2025.05.02.0904. Mutation input limits are 64 KiB, except SentencePiece (128 KiB) and Parakeet vocabulary (256 KiB). These bounded campaigns are not an exhaustive proof for all input sizes or dependency readers.

The checkpoint campaigns instrument the core assembly where the reader now resides, not the converter assembly. General audio decoding is exercised without the WAV-only prefilter; OneAudio itself is not instrumented. RTTM is an evaluation-tool reader, not a shipped core reader. No mutated-input crash was found. New hostile-input tests also cover malformed byte-BPE/configuration shapes, empty special tokens and invalid generation IDs/heads. The complete reader inventory distinguishes owned readers from dependency parsers. One new harness startup failed because it touched an instrumented static initializer before fuzzer initialization; setup was corrected before the clean campaigns, with no input crash to minimize.

Offline recognition

LibriSpeech rows are the first 200 sorted utterance IDs of each named split. Each row is separate; results are not averaged across splits or models.

The omniASR row tests the aligner's acoustic decoding, not general-purpose ASR coverage. Moonshine produced 1,600/1,600 identical token sequences across its eight rows; Parakeet produced identical tokens and frames across the two slices (14,365 tokens).

Timing

Boundary errors are mean absolute start/end error in milliseconds. Confidence intervals were not supplied for these rows. A timing floor is a first measurement against independent truth, not a claim of equivalence to that truth.

MFA truth uses English MFA acoustic model/dictionary v3.0.0 with speaker adaptation disabled. The CTC aligner left 0/4,634 clean words and 9/1,562 other words untimed. Common-word comparisons exclude words not timed by every compared method; they do not erase these failures.

Diarization

DER uses no collar and scores overlap. Values below are DER / JER percentages. Single-recording rows have no bootstrap CI. AMI uses Mix-Headset test recordings ES2004a/b; AliMeeting uses Eval far-field channel 1. The tier-1 method is Silero + CAM++ English (AMI) or Chinese (AliMeeting), spectral clustering and cosine resegmentation; its settings were selected on AMI development ES2011a/b, not these test recordings.

Tier-1 overlap detection is absent. These misses are real limitations, not rounded-away successes. Short recordings (at most 60 seconds) now use cosine AHC: a separate seeded LibriSpeech short-recording check improved speaker-count accuracy from 9/24 to 23/24 (singles 12/12, pairs 11/12), with DER 5.85%. It is not an AMI/VoxConverse population estimate.

Online clustering floors with 100 ms pushes: ES2004a 47.08/54.66, ES2004b 26.88/27.71, R8001 56.39/53.94, R8003 39.41/42.64 DER/JER. Turn availability delay averaged 1.32–1.71 seconds. No equivalent reference-policy CI is available; live and whole-recording clustering are not interchangeable.

Speaker embeddings

LibriSpeech test-clean + test-other, 20,000 balanced trials, 73 speakers, seed 1729. These are identity-only trials on this fixed population, not open-set VoxCeleb1-O results. The paired-trial CI is conditional on these speakers. minDCF uses prior 0.01, miss/false-alarm costs 1, normalized; scores use raw cosine.

Authorized local segmentation

The local segmentation-3.0 adapter uses the pinned publisher checkpoint with float32 OneModel 1.2.0. On the first five ordinal LibriSpeech test-clean clips (1089-134686-0000 through 0004), padded/truncated to 10 seconds, probability-space max absolute error against pyannote.audio 4.0.7 / CPU Torch 2.11.0 is 9.19e-6; the largest per-clip mean error is 2.64e-7. All 2,945 powerset frame decisions match. This probe row uses four threads on x64 (AMD Ryzen 9 7950X3D); it does not establish a bitwise or all-ISA guarantee. Raw log-probability errors and crafted low-variance cases are described in the segmentation contract; this is not bitwise parity.

The following are floors for our segmentation-plus-CAM++ clustering policy, not community-1 equivalence. Both sides use fp32, no collar, overlap scored, and the same scorer/truth. AMI v1.6.2 uses Mix-Headset audio; AliMeeting Eval uses far-field channel 1. Each row is one recording, without a bootstrap CI. Our side uses four CPU threads, independent 10 s chunks and the existing spectral/AHC defaults; CAM++ English on AMI and Chinese on AliMeeting. Community-1 is the publisher's complete pipeline (different embedder and PLDA/VBx), run offline through pyannote.audio 4.0.7, for information only. Its ES2004a reference uses four CPU threads; the remaining reference shards use eight to fit the per-recording run bound, with unchanged model/pipeline parameters.

Community-1's native WeSpeaker/PLDA/VBx adapter is not implemented. Its published PLDA does not require fitting, but its exact transform/dataflow contract has not yet been established for an independent implementation. Its PLDA is never silently substituted for a CAM++ backend. The tier-1 default is unchanged.

Additional ASR, long-form and timing results

All WER rows: fp32, en/1 normalizer, paired utterance (or file) bootstrap, 1000 resamples, seed 1729, scored by onespeech-eval score. "Delta" is ours minus reference; CIs are of the paired difference. Machine: AMD Ryzen 9 7950X3D (16C/32T, AVX2/FMA/AVX-512), shared worker at load 22-25, CPU only. .NET SDK 10.0.403, OneModel 1.2.0, OneAudio 1.7.2. Jobs ran as queue shards of 4-6 cores; no timing claim is made from them.

Parakeet: all 99,653 (clean) and 97,717 (other) reference tokens are identical to NeMo's, and so are all emission frames (2620/2620 and 2939/2939 utterances). The raw text strings differ in 6 and 5 utterances only by spacing; the normalized scores are identical.

Qwen3-ASR-1.7B: the first 500 sorted IDs per split, not the full splits. The CPU reference costs about 115 utterances per hour on a shared host, so the full 5559 utterances did not fit. Raw text differs in 2 of 500 (clean) and 1 of 500 (other) utterances.

Long-form, per file (ours; reference at temperature 0; reference with default fallback), WER %:

Why the reference rows are large: at temperature 0, transcribe() with its default previous-text conditioning falls into repetition loops on these long recordings (reference output for 4479524 has 4000 words against 2122 reference words; for 4474955 one segment text repeats 346 times, for 4481221 1013 times). Our pipeline chunks at speech boundaries and does not condition across chunks, so it does not loop. Ours is identical with and without fallback on this set. These rows are better than the reference pipeline, not an equality or within-noise claim; the comparison is against the reference's own pipeline. The 4483046 default-fallback row is the one where the reference is better than ours-at-T=0 in per-file terms (20.711 vs 21.119, paired, no CI at n=1).

Timing revalidation on current code

Mean absolute boundary error (ms) and share within 50/100 ms, pooled over timed words, against MFA 3.4.3 (English v3.0.0, speaker adaptation off) alignments of the same LibriSpeech slices (test-clean first 200, test-other first 100). Floors are the values recorded earlier; no row regressed.

Qwen3-ForcedAligner vs the toolkit, clean 200 (4634 words): start/end mean 28.73/18.90 ms; start within 50/100 ms 90.80/94.37 %, end 98.07/98.10 %; 67 words untimed by ours, 0 by the toolkit. Identical to the recorded row (post-processing parity remains a documented gap).

Numbers are word-pooled and cover the matched words of each method's own transcript, so the turbo row (its own recognized words) and the CTC row (reference text) have different word sets, as in the earlier floors.

Diarization and cache-aware streaming — full-set evaluation

Float32 CPU evaluation on an AMD Ryzen 9 7950X3D (x64; AVX2/FMA/AVX512 available), .NET SDK 10.0.403/runtime 10.0.14, OneModel 1.2.0 and OneAudio 1.7.2. These are quality and audio-clock availability measurements, not throughput benchmarks.

Data and scoring

• AMI: all 16 Full-corpus-ASR test meetings: EN2002a–d, ES2004a–d, IS1009a–d and TS3003a–d. Original Mix-Headset audio, 16 kHz mono, word annotations v1.6.2; positive silence gaps retained. CC-BY-4.0.
• AliMeeting: all eight Eval far-field recordings, first input channel, 16 kHz mono, nonempty TextGrid intervals. CC-BY-SA-4.0, evaluation only.
• DER has no collar and scores overlap. Dataset DER pools reference speaker-seconds; JER averages reference-speaker errors. Paired 95% intervals use 1,000 meeting resamples, seed 1729. Meetings within an AMI series share participants; these are meeting, not independent-speaker, resamples.
• default: the unchanged file-helper policy, Silero VAD + CAM++ English for AMI/Chinese for AliMeeting + spectral/AHC clustering and cosine resegmentation. It does not detect overlap. Reference: released ModelScope 1.40.2 CAM++ bundle, pinned local auxiliaries, float32 CPU, four threads. No test-set fitting.
• nemotron: Nemotron 3 Diarization, explicitly selected tier 2, compared with Transformers 5.22.2 at the same publisher pin and precision; four threads on both sides.
• pyannote: the segmentation-3.0/CAM++ slot, four threads, a floor, not a community-1 reimplementation. The pyannote.audio 4.0.7 community-1 pipeline is information only: its embedding, stitching and PLDA/VBx backend differ. Authorized local snapshots, offline; eight CPU threads, or sixteen on five recordings that exceeded the bounded eight-thread reservation. Neither tier-2 path is a default.

Raw and qualified reference results

The ModelScope public output for EN2002c contains one reversed interval, [2043.28, 2043.10, speaker 1], among 513 returned turns. The raw output and strict error are preserved. A negative-duration interval has no valid RTTM representation; a numeric raw reference score is therefore unavailable. An explicit qualified projection omits only this interval and retains the other 512 turns. It never swaps endpoints or invents timing.

All other ModelScope recordings were valid. Dataset-level tier-1 differences include zero in their paired intervals, but that does not hide substantial individual-meeting differences: for example EN2002b is +6.63 DER/+15.59 JER points. Every meeting is listed below. Nemotron is within paired noise on both complete sets. The pyannote slot remains below community-1 quality: AMI differences are +3.68 DER/+8.12 JER points and AliMeeting +4.65/+5.67 points, with positive paired intervals; those are disclosed information-baseline gaps, not equivalence passes.

Nemotron 3.5 ASR streaming 0.6B

First 200 ordinal LibriSpeech test-clean/test-other utterances, float32 greedy English decoding. Reference: Transformers 5.22.2 CPU, two threads; OneSpeech four threads, actual published live-event policy. Known utterance boundaries isolate recognition from endpoint-detector differences. Normalizer en/1, 1,000 paired utterance resamples, seed 1729.

All 800 native/reference utterance/chunk pairs have identical lexical token sequences. Both sides were freshly generated. Paired WER differences and their 95% intervals are [0,0] on every row.

Stable-word availability closes a word at the next word boundary or EOF. Both engines use the same saved offline 80-ms RNN-T acoustic endpoints as a timing proxy, not forced-alignment truth. Signed negative delays are retained. Latency means are over matched words; unmatched counts are shown. This is audio-time availability, not CPU speed or a learned endpointing-policy comparison. Publication remains approximately 55–58 ms later in OneSpeech, so latency parity is not claimed.

Actual EOF Final availability relative to the last common offline word end is identical for both engines and chunk sizes: test-clean mean/median 33.875/0 ms, test-other 12.925/0 ms. Utterance EOF is known in this experiment. These descriptive latency statistics have no bootstrap interval.

Diarized transcript sanity check

Whisper small through Speech.Transcribe, English, AMI ES2004a's first 300 seconds: en/1 WER 16.0584% on 411 reference words. After optimal one-to-one word-count speaker mapping, 287/344 = 83.4302% of all Word objects have the correct maximum-overlap reference speaker, or 287/321 = 89.4081% among eligible timed/reference-covered words. There are 29 Unassigned, zero untimed words and zero overlap flags. This speaker measure is temporal attribution, not lexical correctness.

Timing provenance is 333 ForcedAligner, five CtcAligner and six Dtw words. The earlier all-word sanity figure was 84.3%; the current value is 83.43%, while WER remains 16.06%. The timing chain has changed since the earlier DTW-only measurement; no improvement claim is made.

Complete-set and per-meeting diarization rows

All rates are percentages. Difference intervals and single-meeting differences are percentage points. The single-meeting 0.5-point diagnostic is descriptive; complete-set verdicts use paired meeting intervals. Pyannote/community-1 rows are information, not native-equivalence bars.

ami — complete-set comparisons

ami — every meeting

alimeeting — complete-set comparisons

alimeeting — every meeting

Speed

Measured 2026-11-08 using the structural harness, measurement version 5, one repetition per row. These are our own CPU measurements, not comparisons with another implementation. All models use fp32 at the publisher revisions pinned in the catalog.

• x64: AMD Ryzen 9 7950X3D, 16 physical / 32 logical CPUs, AVX2/AVX512F/FMA, Linux Mint 22.3.
• arm64: Apple M2, 8 logical CPUs, 8 GiB RAM, AdvSimd, macOS 27.0.1.
• Both use .NET SDK 10.0.403, runtime 10.0.14, OneModel 1.2.0 and OneAudio 1.7.2; no GPU. The process CPU-count limit equals the row's thread count. Silero retains its production one-thread setting.

Each filtered row ran in an exclusive bench queue job bounded to 20 minutes. Models were loaded and warmed on a different ten-second clip before timing. Load time is separate; download, conversion and verification are excluded. End-to-end is encoded audio to Transcript using retained model instances and the production recognition, deferred alignment, clustering and merge algorithms. This is not cold Speech.Transcribe latency: that helper releases sessions between stages, while this warm-throughput harness retains them and can use more memory.

Inputs are deterministic LibriSpeech test-clean concatenations of 60, 600 and 3600 seconds, without inserted silence; clustering uses the 1049.355-second AMI ES2004a Mix-Headset recording. These constructed inputs make no quality claim. align is omniASR-CTC forced alignment; diarize is Silero + CAM++ English clustering; live is Moonshine streaming-tiny with 100 ms chunks; long is the full file pipeline with Whisper tiny on 3600 seconds.

The structural bar is end-to-end / summed graph wall time ≤1.15. RTF is elapsed seconds / audio seconds: ≤1 means at least real time. RSS is decimal GB, sampled every five seconds; “steady” discards the first 30 seconds but includes phase changes, so its range is not evidence of a constant-memory plateau. Short rows without steady samples fall back to the sampled peak. Live working B/chunk excludes graph-runtime allocations and immutable emitted payloads (500 measured chunks after 100 warm chunks); it is not total process allocation.

The load-average table below reports the one-minute component before / during / after each timed call. The during interval includes the workload itself. Queue startup CPU snapshots showed no other CPU-bound user workload; desktop and OS services remained active. Large initial load averages can reflect decaying earlier activity, not concurrent CPU utilization. Full 1/5/15-minute samples and queue logs were retained with the measurement records.

Results: every measured structural ratio passes (maximum 1.081 on x64, 1.056 on arm64). The tier-1 live default meets real time on one core on both machines, with 0 B/chunk of our working allocation. This does not imply that all file models run in real time on one core: turbo's measured RTF is 1.712 on 60 seconds and 1.332 on 600 seconds.

Working-set qualification: the one-hour run is not literally flat on x64. After discarding the first minute of each phase, recognition's first/last fifth means are 8.001/8.013 GB and diarization's are 9.356/9.376 GB, but alignment rises from 8.837 to 9.772 GB (+10.6%, fitted slope +6.47 MB/s). The whole-run first/last fifth means are 8.003/9.365 GB. This row therefore does not establish a strict flat-RSS bar; it also cannot distinguish bounded buffer-pool residency from input-length growth. On arm64, the corresponding recognition means are 0.841/0.865 GB, alignment 4.352/2.026 GB, and diarization 0.610/0.525 GB. No tuning or forced collection was used to alter these rows. These observations should not be turned into a universal constant-memory guarantee.

Missing row: Qwen3-ASR-1.7B was not measured. The approximately 62 GiB shared model/data tree plus its upstream and converted files would exceed the run's approximately 70 GiB storage budget. No Qwen speed claim or cross-model default re-selection is inferred from this omission. The one-hour memory rows use tiny; turbo is covered only through 600 seconds. No int8 speed row was measured here.

Model choices and remaining evidence

• Files: Whisper large-v3-turbo remains the default. Its test-other WER is 5.758% in the file-model comparison; Qwen3-ASR 1.7B is 4.131%, a turbo-minus-Qwen difference of +1.63 points (paired CI [1.02, 2.31]). Qwen 1.7B is available by name for accuracy. The fixed-slice CPU comparison favoured turbo's resource requirements; the completed structural measurements and their limits are reported in the Speed section.
• Live: Moonshine streaming-tiny is the default. On 20-file clean/other streaming slices, its WER floor is 1.70/8.33%, with mean finalisation latency 3057/1617 ms. Streaming-small is 1.95/3.74%, 3073/1622 ms. Latency is relative to Whisper-tiny DTW word ends, not independent boundary truth. There is no vendor streaming-policy reference for these floors.
• Language identification: Whisper and its reference each identified 15/15 FLEURS probes (five each English/French/German). This is information only, not multilingual coverage evidence.
• Publisher availability: VoxCeleb1 and VoxConverse audio are no longer available from the publisher. Their intended rows are unavailable; no unofficial re-upload is substituted.
• Gated segmentation: authorized segmentation parity and complete-set slot measurements are reported above. The segmentation/CAM++ slot is not a community-1 reimplementation; the community-1 comparison remains information, with its quality gaps disclosed.
• Long-form: full long-form rows are reserved for the 1.0 evaluation. Existing single-file observations do not establish equivalence. In particular, an Earnings-22 4474955 turbo temperature-zero comparison was 55.159% versus 58.386% reference WER: outside the 0.5-point bound, in our favour because the reference appended repeated tail text. Other decoding/pipeline settings produced different scores; they must not be combined into that row. Whole-recording Qwen reference runs exceeded the CPU run bounds.
• Quantization and speed: full Moonshine int8 quality is reported above; full Whisper fp32 and int8 comparisons are complete and within paired noise. The Speed section reports the measured structural-overhead and live-allocation rows, including the Nemotron allocation residual and unmeasured Qwen row; it does not establish a blanket throughput guarantee.

These results do not yet demonstrate every release target. See evaluation conventions for scoring and word timing for failure semantics.