Label speakers in live audio in C#

Label two speakers in the zip’s sample.flac.

dotnet add package RecursiveAI.OneSpeech

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

using OneSpeech;
using OneSpeech.Audio;
await foreach (var update in Speech.Listen(
    Pcm(), language: "en"))
    if (update is SpeakerTurnEvent speaker)
        Console.WriteLine(
            speaker.Turn.Speaker.Id);
 
static async IAsyncEnumerable<AudioBuffer> Pcm()
{
    using var input = AudioFileSource.Open(
        "sample.flac", CancellationToken.None);
    var buffer = new float[1600]; // 100 ms mono.
    int count;
    while ((count = input.Read(
        buffer, CancellationToken.None)) != 0)
    {
        yield return new AudioBuffer(
            buffer.AsMemory(0, count), 16000);
        await Task.Yield();
    }
}

First run downloads Moonshine streaming tiny weights (about 176 MB) and CAM++ English speaker-model weights (about 29 MB). Change the model cache.

Speaker embeddings are biometric data. Obtain informed consent, limit retention and access, and check applicable law, including Illinois BIPA and GDPR Article 9, before enrollment, identification or verification. The library does not persist embeddings; applications remain responsible for audio, transcripts, logs and any copies they retain. This is not legal advice.

Unzip the example, then dotnet run.

Prints:

speaker-1
speaker-2

The full program

The zip’s sample.flac joins two LibriSpeech speakers: recording 1089-134686-0001 (“stuff it into you his belly counselled him”) and the first four seconds of 1272-128104-0000 (“Mister Quilter is the apostle…”), with half a second of silence after each. Credit: Vassil Panayotov, Guoguo Chen, Daniel Povey and Sanjeev Khudanpur (LibriSpeech), CC BY 4.0. The second recording comes from the public LibriSpeech subset.

using OneSpeech;
using OneSpeech.Audio;
 
await foreach (var update in Speech.Listen(Pcm(), language: "en"))
    if (update is SpeakerTurnEvent speaker)
        Console.WriteLine($"{speaker.Turn.Speaker.Id}: {speaker.Turn.Start}–{speaker.Turn.End}");
 
static async IAsyncEnumerable<AudioBuffer> Pcm()
{
    using var input = AudioFileSource.Open("sample.flac", CancellationToken.None);
    var buffer = new float[1600]; // Mono sample: 100 ms.
    int count;
    while ((count = input.Read(buffer, CancellationToken.None)) != 0)
    {
        yield return new AudioBuffer(buffer.AsMemory(0, count), 16000);
        await Task.Yield();
    }
}

dotnet run prints:

speaker-1: 00:00:00.5460000–00:00:02.9100000
speaker-2: 00:00:04.2900000–00:00:07.8060000

The input has two speakers, and the output assigns two session-local IDs. Pcm reads mono 16 kHz samples in 100 ms blocks. Speech.Listen adds online speaker clustering automatically; it does not capture a microphone. For application-owned microphone input, use the real-time speech recognition recipe. The buffer may be reused after enumeration advances.

Missing models download on first use. The live default is moonshine-ai/moonshine-streaming-tiny, with CAM++ English for the explicit en hint. Use the model download guide to prepare their files before an offline deployment.

Input in another PCM format, or a block longer than ten seconds, is rejected with ArgumentException; a processing failure faults the session, so start a new one rather than retrying that block.

Variations

Read labels from a final instead of handling turn events. In the full program, replace the event handler with:

    if (update is FinalEvent final)
    {
        Console.WriteLine(final.Segment.Text);
        Console.WriteLine($"Speaker turns: {final.SpeakerTurns.Length}");
        Console.WriteLine($"Untimed words: {final.Segment.Words.Count(w => w.Start is null)}");
    }

Here update is the current item from the Speech.Listen loop. With the same sample, this prints:

Stuff it into you, his belly counseled him.
Speaker turns: 1
Untimed words: 8
Mr. Quilter is the apostle of the middle classes, and we are glad.
Speaker turns: 1
Untimed words: 13

The final has speaker activity even though its words have no timestamps; do not assign those words to individual turns by guessing.

Related recipes

Display partial speech recognition results in C# · Limit live speech utterance duration in C# · Transcribe audio from a stream in C#

The short answer

Use Speech.Listen with mono 16 kHz PCM and consume SpeakerTurnEvent, or read the immutable SpeakerTurns on a FinalEvent. OneSpeech performs online clustering at recognition endpoints. A speaker ID means a cluster within this session, not a person’s name, an enrolled identity or a durable account identifier.

How can I label speakers in a transcription?

For a live source, feed small PCM blocks in order and let the recognizer close each utterance. The speaker-labeling session keeps that stretch of audio, compares voices over nonoverlapping windows and assigns each window to a speaker group when it matches closely enough. Adjacent windows with the same speaker merge into a turn within the utterance. Already-emitted turns do not merge across utterances.

Turn events arrive before the corresponding final. You can show them as activity intervals alongside the text, or use the final’s speaker snapshot when a UI wants one committed update. Do not count both as separate activity: they describe the same labeling. The turn text is empty because a turn is speaker activity, not a second transcription.

Word labels require word times. A timed word receives the speaker with the greatest positive interval overlap; untimed or uncovered words remain Unassigned. The sample demonstrates why these outputs are separate: its final includes speaker activity while its words remain untimed. The library does not manufacture word boundaries from VAD intervals. See the pinned live speaker-label documentation for the event contract.

What is the correct way to write speaker labels?

Keep the generated ID with the session and display it consistently, for example speaker-1. Do not relabel earlier finals after a later chunk makes a speaker seem familiar. IDs are never retrospectively merged, renumbered or reused. Two early clusters can remain separate even if later audio suggests one person.

Explicit language: "en" selects CAM++ English. Chinese hints select the Chinese embedder. Automatic detection loads both; the two embedding spaces use different prefixes, so a language switch can split one person into two IDs. These IDs are therefore unsuitable as cross-recording identity keys. If your application associates an ID with a display name, treat that association as application data, not a recognition guarantee.

Can I get live speech speaker labels for free in C#?

Yes. The downloadable example uses the free, open-source OneSpeech package and freely licensed models. Missing model files are acquired and verified on first use; verified cached files can be reused without a network request. Plan model storage before opening a live source rather than letting first-use preparation interrupt capture.

“Live” does not mean instant speaker-change detection. Labels wait until the recognizer closes the utterance. Large pushes also delay when updates become available to the caller. Feed blocks small enough for your application’s responsiveness and choose an utterance maximum appropriate to it. Clustering adds endpoint computation without looking into future utterances.

The online method does not detect overlap, split simultaneous speakers or repair early decisions using later audio. Fixed windows can straddle a change, and short interjections can be missed. Use offline speaker diarization when recording-wide consistency matters more than immutable live updates; it is a different workflow, not a hidden second pass in this recipe.