Extract the audio from a video to WAV in C#

OneMedia reads the audio track’s packets out of the video file, and the RecursiveAI.OneMedia.Audio adapter hands them to OneAudio, which decodes them. media.Decode() gives you OneAudio’s decoder for the file’s audio, and OneAudio’s WavEncoder writes the samples to a WAV file. OneMedia itself decodes nothing. The input is MDN’s CC0 flower clip, H.264 video with AAC audio.

dotnet new console -o VideoToWav && cd VideoToWav
dotnet add package RecursiveAI.OneMedia.Audio --version 0.1.0-preview.1
curl -L -o flower.mp4 https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.mp4

RecursiveAI.OneMedia.Audio brings in RecursiveAI.OneMedia and RecursiveAI.OneAudio 1.4.0. Put this in Program.cs and run dotnet run:

using OneAudio;
using RecursiveAI.OneMedia;
using RecursiveAI.OneMedia.Audio;
 
using MediaFile media = Media.Open("flower.mp4");
Console.WriteLine($"Audio track: {media.Audio?.Codec}, {media.Audio?.Audio?.SampleRate} Hz");
 
// The default audio track as a OneAudio decoder (AAC here), float samples out.
using IAudioDecoder pcm = media.Decode();
AudioFormat format = pcm.Format;
Console.WriteLine($"Decoder: {pcm.GetType().Name}, {format.SampleRate} Hz," +
                  $" {format.Channels} ch, {pcm.LengthFrames} frames");
 
// Write 16-bit PCM WAV block by block, so memory stays flat for long files.
using (FileStream file = File.Create("flower.wav"))
{
    var wav = new WavEncoder(file, format, new WavEncoderSettings(), leaveOpen: true);
    float[] block = new float[4096 * format.Channels];
    long frames = 0;
    int n;
    while ((n = pcm.Read(block)) > 0)
    {
        wav.Write(block.AsSpan(0, n * format.Channels));
        frames += n;
    }
    wav.Finish();
    Console.WriteLine($"flower.wav: {frames} frames," +
                      $" {frames / (double)format.SampleRate:0.000} s, {file.Length:N0} bytes");
}

Audio track: aac, 48000 Hz
Decoder: AacDecoder, 48000 Hz, 2 ch, 242640 frames
flower.wav: 242640 frames, 5.055 s, 970,640 bytes

The decoder starts and stops where the file says the audio does: the MP4 edit list cuts priming at the start (none in this file) and padding at the end (48 frames here), so the WAV is 242,640 frames, the track’s edited length. Decoded with FFmpeg 8.0.1 to 16-bit PCM, the same track gives the same samples over those 242,640 frames; FFmpeg’s output is 48 frames longer because it keeps the end the edit list cuts.

Another track

media.Decode(track) decodes a track you choose, for example the second audio track of a Matroska file (see List the tracks of a video):

using RecursiveAI.OneMedia;
using RecursiveAI.OneMedia.Audio;
 
using MediaFile media = Media.Open("movie.mkv");
MediaTrack commentary = media.Tracks.First(t => t.Name == "Commentary");
using OneAudio.IAudioDecoder pcm = media.Decode(commentary);

Without a track, Decode() takes the first audio track marked default, else the first one OneAudio has a decoder for.

Limits

• Codecs. OneAudio decodes AAC-LC, Opus, Vorbis, FLAC, MP3 and AMR.
• HE-AAC decodes as its AAC-LC core, at half the rate, without an error: the commentary track of test5.mkv, 44,100 Hz in the container, comes out at 22,050 Hz. The adapter docs list HE-AAC with SBR and PS as a gap. AC-3, E-AC-3, PCM, ALAC and protected tracks throw UnsupportedAudioFormatException naming the codec. See OneAudio adapter.
• No video. The video track is not decoded; there is no way to get frames from OneMedia.
• Matroska lengths. For Matroska, LengthFrames is -1 (unknown) because the container states only a rounded duration; decode to the end to count.
• End padding. Opus end trimming is applied; for Vorbis in Matroska the adapter can return up to 16 more frames than an Ogg copy, and some discard padding can’t be passed to OneAudio (AudioSeamLimitations on the packet source reports it).

For audio files rather than videos, use OneAudio directly: Decode AAC and M4A.