Get video keyframe timestamps in C#

media.Packets(track) reads a track’s packets in file order, each with its timestamps, keyframe flag, size and byte position in the file, without decoding anything. Keyframes are where a player can start decoding, so they are where a cut or a seek can land without re-encoding. The input is the Sintel trailer, 4.4 MB, from the Blender Foundation (CC BY 3.0).

dotnet new console -o Keyframes && cd Keyframes
dotnet add package RecursiveAI.OneMedia --version 0.1.0-preview.1
curl -L -o sintel.mp4 https://download.blender.org/durian/trailer/sintel_trailer-480p.mp4

Put this in Program.cs and run dotnet run:

using RecursiveAI.OneMedia;
 
using MediaFile media = Media.Open("sintel.mp4");
MediaTrack video = media.Video!;
 
// Packet times are on the track's own media timeline. Shift them by the edit list
// to get the time a player shows.
TrackEditInfo edit = video.Edit;
int packets = 0, keyframes = 0;
foreach (OwnedPacket packet in media.Packets(video))
{
    using (packet)
    {
        packets++;
        if (!packet.Info.IsKeyframe) continue;
        keyframes++;
        MediaTime shown = packet.Info.Pts - edit.Priming + edit.StartTime;
        Console.WriteLine($"{shown.ToTimeSpan():mm\\:ss\\.fff}  byte {packet.Info.SourceRange.Offset,8}" +
                          $"  {packet.Info.Size,6} bytes");
    }
}
Console.WriteLine($"{keyframes} keyframes in {packets} video packets");

00:00.000  byte       48     766 bytes
00:09.333  byte   767856    1942 bytes
00:13.916  byte  1064100   42465 bytes
00:16.500  byte  1213115   14998 bytes
00:21.916  byte  1793351    6636 bytes
00:23.791  byte  2103099   28047 bytes
00:25.875  byte  2454669   11924 bytes
00:27.416  byte  2683094   10664 bytes
00:32.166  byte  3365431    5389 bytes
00:42.583  byte  4107389    4951 bytes
00:47.500  byte  4235471    1648 bytes
11 keyframes in 1253 video packets

Pts and Dts are exact rational times (MediaTime) in the track’s timescale, so nothing is rounded until you call ToTimeSpan(). Packets(track) switches the other tracks off while it runs, so their bytes are not read. Each packet’s Data is its bytes, rented from a pool: dispose the packet when you are done with it. SourceRange is where those bytes sit in the file.

Why the edit list matters

OneMedia doesn’t apply edit lists to packet times: they stay on the track’s media timeline, as the sample tables state them. This file’s edit list starts the picture 2/24 s into the media (to make room for B-frame reordering), so the raw first Pts is 00:00.083 and the shifted one is 00:00.000. ffprobe applies the edit list, and the shifted times match its keyframe times (the C# format string above cuts to milliseconds, so 13.916667 prints as 13.916):

ffprobe -v error -select_streams v -skip_frame nokey -show_entries frame=pts_time -of csv=p=0 sintel.mp4

0.000000
9.333333
13.916667
16.500000
21.916667
23.791667
25.875000
27.416667
32.166667
42.583333
47.500000

Pts - Priming + StartTime is right for a track with one edit, which is what nearly every file has. A track with several edits maps pieces of media onto the timeline separately; edit.Edits lists them as stored. See MP4, MOV and 3GP.

Limits

• No frames. A keyframe here is a packet the container flags as a random-access point. OneMedia can’t decode it into a picture or make a thumbnail.
• No cutting. Trimming or splitting a video at keyframes needs a writer; writing and remuxing are not in 0.1.
• Matroska packets come from Blocks and SimpleBlocks; a keyframe is a SimpleBlock with the keyframe bit or a Block without references. Matroska has no edit lists, so Pts is already the shown time (a track’s CodecDelay is already subtracted).
• Whole file read. Reading every packet reads the track’s bytes. To read only the index, open the file and look at media.Tracks; the packet count of an MP4 track is track.PacketCount.

To read packets of a file on a web server without downloading it all, see Read video metadata from a URL.