Recursive AI

Extract closed captions from video in C#

OneMedia is a free, open-source media library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

NuGet GitHub Docs Download example (.zip)

Embedded CEA-608/708 captions are text in video packets, not speech or subtitles burned into the picture.

Terminal
dotnet add package RecursiveAI.OneMedia

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

captions.ts is the generated black-video sample in the zip; its CC1 caption says HI.

C#
using RecursiveAI.OneMedia;
using RecursiveAI.OneMedia.Captions;
 
using var file = Media.Open("captions.ts");
foreach (var track in file.Tracks)
{
    var captions = EmbeddedCaptions.Read(file, track);
    foreach (var caption in captions.Cea608)
        Console.WriteLine(
            $"608 {caption.Channel}: " +
            caption.Text);
    foreach (var caption in captions.Cea708)
        Console.WriteLine(
            $"708 service {caption.Service}: " +
            caption.Text);
}
Text
608 CC1: HI

The sample contains MPEG-2 video in a transport stream. The program extracts caption data from its video packets and decodes the caption commands in presentation order. It prints 608 CC1: HI, as shown above. It does not decode pictures or run speech recognition.

If a track has no supported caption carriage, EmbeddedCaptions.Read returns no captions; an empty result does not prove the file has no other subtitle track or visible subtitles.

Unzip the example, then dotnet run.

Variations

Save CC1 captions as SRT. This standalone example uses the same captions.ts:

C#
using RecursiveAI.OneMedia;
using RecursiveAI.OneMedia.Captions;
using RecursiveAI.OneMedia.Captions.Cea608;
 
using var file = Media.Open("captions.ts");
var captions = EmbeddedCaptions.Read(file, file.Tracks[0]);
var result = CaptionCues.FromCea608(
    captions.Cea608, Cea608Channel.CC1);
var report = Subtitles.Write(result.Document, "captions.srt");
foreach (var loss in result.Losses)
    Console.WriteLine(loss);
foreach (var loss in report.Losses)
    Console.WriteLine($"{loss.Kind}: {loss.Feature}");
Console.WriteLine($"Wrote {result.Document.Cues.Count} cue(s)");
Text
Dropped: Position
Dropped: CaptionMode
Dropped: Extras
Wrote 1 cue(s)

It writes this captions.srt:

Text
1
00:00:01,560 --> 00:00:02,640
HI

SRT keeps the text and timing, but not the caption’s grid position, display mode or format-specific extras. Check both conversion stages’ loss lists, as the snippet does. The writer refuses an existing captions.srt with IOException; choose a new output name before rerunning.

The short answer

Open the file with Media.Open, pass a track to EmbeddedCaptions.Read, and inspect Cea608 and Cea708 separately. Each decoded caption contains text and a start and end time. CEA-608 captions identify a channel, such as CC1; CEA-708 captions identify a service. The snippet at the top handles both collections even though the sample contains only CEA-608.

This page uses the released RecursiveAI.OneMedia 1.0.2 package for .NET 10. Its caption documentation covers the supported video carriage and decoding behavior. “Extract” here means reading existing caption bytes: it does not generate new captions from audio, recognize lettering in images or fetch a transcript from a website.

How can I extract closed captions from a video?

First identify the kind of caption you have. Embedded CEA captions travel in video packet data rather than as an ordinary text-subtitle track. OneMedia can read the ATSC caption messages in H.264 SEI, HEVC prefix or suffix SEI, and MPEG-2 video user data. It also reads QuickTime c608 closed-caption tracks. The documented ISO c708 sample-entry layout is not read; that is distinct from CEA-708 carried inside supported video messages.

The high-level EmbeddedCaptions.Read call joins extraction and decoding. Underneath it, CaptionExtractor.ReadFrames returns caption frames in presentation order using a stable sort on packet presentation time. This matters because the order compressed pictures are decoded is not necessarily the order a viewer sees them. Caption decoders need the display sequence to reconstruct the right screens. Within a picture, pairs of the same type are spread evenly across that picture’s duration.

CEA-608 data is not simply a string per packet. It includes commands to load an off-screen caption, display it, roll lines or erase the screen. The decoder produces displayed screens with timing, rows, grid positions and styles. The sample loads HI, displays it, then erases it. Its text is printed only after those commands have been interpreted, rather than by treating every pair of bytes as letters.

Use EmbeddedCaptions.IsEmpty when you need to distinguish a caption result with no decoded screens. There may be no supported data, only padding, another subtitle format or a different kind of visible text. The library’s default CEA-608 decoder drops byte pairs without odd parity; it also ignores null pairs and collapses an immediately repeated control pair. These decoding rules are documented in the CEA-608 section.

How can I extract subtitles from an MP4 video?

The container extension alone does not tell you which extraction path to use. For captions embedded in H.264 or HEVC video packets, use the same EmbeddedCaptions.Read call shown above. The extractor walks both length-prefixed samples and Annex B streams, reading the caption-bearing messages without decoding the picture.

A text subtitle track is different. Inspect the file’s subtitle tracks and use their cue documents rather than expecting their text in Cea608 or Cea708. The text-track documentation explains those container tracks. Burned-in subtitles are different again: because this API does not decode pictures, letters visible only in the image are not an input it can extract.

When exporting embedded captions, choose the channel or service deliberately. CaptionCues.FromCea608 selects one channel; the variation chooses CC1. Mixing every 608 channel into one document would combine unrelated screens. The mapping retains caption details in the cue model where possible, and reports details it cannot represent. SRT is a simpler destination: the sample’s writer reports lost position, mode and extras rather than pretending the result is a complete visual copy.

How do I get captions off a video?

If you mean a separate text file, extract the captions, map them to cues and write those cues to SRT as the variation does. Keep the original video if you need its full caption presentation. CEA-708 includes windows and styling beyond plain text; the documentation lists parsed-and-dropped attributes, and CaptionCues.FromCea708 reports the lost detail when mapping to cues.

If you mean removing captions from the video itself, writing an SRT does not do that. It creates a sidecar file and leaves the source unchanged. A remux also carries video samples unchanged, so caption messages inside those samples ride along. Extraction and removal are separate operations; this recipe only reads caption data and optionally writes a subtitle file.

All onemedia recipes