ISO base media files (MP4, MOV, 3GP, fragmented MP4, CMAF)

How RecursiveAI.OneMedia.IsoBmff reads these files. Specifications: ISO/IEC 14496-12 (clause numbers in the code comments), the QuickTime File Format specification for the atom differences, 3GPP TS 26.244 for 3GP. Codec configuration boxes (avcC, hvcC, esds, …) are carried as opaque bytes; they are parsed by the codec layer.

Layers

Box rules

• 32-bit size, 64-bit largesize (size field 1), size 0 ("to the end of the enclosing range"), and uuid boxes with their 16-byte user type are read (§4.2). Unknown and vendor boxes are kept as ranges.
• Containers: moov trak mdia minf stbl edts dinf mvex moof traf mfra udta tref sinf schi meta ilst stsd dref and the item boxes inside ilst. stsd and dref start with version, flags and an entry count (8 bytes) before their children. meta is a full box in ISO files and a plain container in QuickTime files; the reader tells them apart by looking for hdlr right after the header.
• A QuickTime list may end with a 32-bit zero, and writers pad with zero bytes: reported as an Info defect (UnaccountedBytes), nothing more.

Defects

All carry the byte offset of the box (or entry) they concern. Reading continues where it safely can.

Limits used: MaxStructureDepth, MaxElementCount, MaxTracks, MaxSampleTableEntries, MaxAllocationBytes (the movie box is only read whole for ranged and forward-only sources; each movie fragment box is read whole), MaxPacketBytes, MaxTagBytes (the gapless tag), MaxDefects. Every table's declared entry count is checked against the box size and the limit before anything is read.

Time

Packet times are on the track's media timeline in the track's timescale, exact: Dts is the running sum of the decoding-time-to-sample deltas (stts) or the fragment's base decode time plus the running sum of run durations; Pts = Dts + composition offset. Composition offsets (ctts version 0 and 1, trun version 0 and 1) are read as signed 32-bit values; a version-0 value above 2^31 is not credible for a real file and old QuickTime writers store negative offsets that way. Duration is the table's duration for the sample.

Edit lists are not applied to packet times. They are reported in MediaTrack.Edit:

• Edits: every entry (MediaStart unknown for an empty edit, Duration in the movie timescale);
• StartTime: the total of leading empty edits (movie timescale);
• Priming: the media time at which the first non-empty edit starts, in the track timescale (§8.6.6). For AAC this is the encoder delay the player skips. When the movie carries an iTunes iTunSMPB tag that parses (three hexadecimal fields: delay, padding, valid frames), it replaces Priming and EndPadding for the first audio track, because the tag states encoder delay and padding exactly where the edit list rounds;
• EndPadding: for a single non-empty edit, the media after the end of the edit.

Presentation time on the edited timeline of a packet is Pts - Priming + StartTime (rescaled to a common scale). A packet that ends at or before the first edit's start (Pts + Duration <= media_time) carries PacketAttributes.Discard: it is decoded but never shown.

Sample tables without loading them

SampleCursor walks stts, ctts, stss, stsz/stz2, stsc, stco/co64, and sdtp together in sample order through 4 KiB windows. Memory is constant whatever the table length, and moving to the next sample costs no allocation. A seek positions each table directly (binary search in stss, run walk in stts/ctts/stsc, direct indexing in the fixed-size tables). Keyframe seeking: the sync sample at or before (or after) the target; the other tracks land on their last sync sample not after the primary track's landing time. The primary track is the first enabled video track, else the first enabled track.

Keyframes: samples listed in stss; every sample when stss is absent. Repeated stss entries are tolerated. Disposable: sdtp sample_is_depended_on == 2. Sample group boxes (sgpd, sbgp) are kept as ranges and do not change packet flags.

Packet order

Across the enabled tracks, the packet with the lowest source offset comes next. For interleaved files this is the order the data is stored in, so a plain read is sequential; for a file whose tracks are stored one after the other (all video, then all audio), it is the same storage order. Within a track packets are in decode order. A disabled track costs nothing: its cursor is never advanced and its payload is never read.

Payload reads go through a window that starts at 4 KiB and doubles (to 64 KiB) while reads continue where the last one ended. After a jump (over the chunks of a switched-off track, say) the first window is as long as the run that just ended, rounded up by an eighth and to 4 KiB, because the next chunk of an interleaved track is about as long as the last: a chunk then costs one read instead of a climb from 4 KiB. Samples over 32 KiB bypass the window.

Fragmented files

mvex/trex mark a file as fragmented. Packets of the movie box's own tables come first (usually none: empty_moov), then each moof in file order. A fragment is held in memory only while it is read. Defaults inherit per §8.8.7 and §8.8.8: tfhd, then trex; first_sample_flags override the first sample; the base data offset is the explicit base_data_offset, else the start of the moof (default-base-is-moof and the common case), else the end of the previous track fragment's data; a trun without data_offset continues after the previous run. A missing tfdt continues the track's previous decode time; for the first fragment that is the end of the samples the movie box itself describes (a hybrid file, zero for empty_moov). Non-sync samples have bit 16 of the sample flags set.

Seeking in a fragmented file whose movie box holds no samples picks the first enabled video track (else the first enabled track) and scans movie fragment headers (never media data) until the target is passed, then reloads the landing fragment. The scan starts from an index when there is one, so the cost is a few header reads instead of one per fragment:

1. the movie fragment random access box (mfra, §8.8.9), found through the mfro box that closes the file: two reads at the first seek (16 bytes, then the box), used when its tfra for the track has an entry at or before the target;
2. else the segment index boxes (sidx, §8.16.3) between moov and the first fragment, read at open, when they are not hierarchical.

The index only names where to start: the scan begins one entry before the best one, and an index is ignored (scan from the first fragment) if its offset is not a moof or that fragment does not state a tfdt for every track fragment, because the running decode times are unknown without the earlier fragments. A file with neither index, or only per-segment sidx boxes (CMAF/DASH live segments), pays the full header scan.

What a probe reads

• Seekable file: the registry's first look at the file (one read of 4 KiB, to recognise it) is handed to the reader, which starts from those bytes instead of reading them again. After that the read-ahead window (32 KiB) serves the start of each top-level box on the path to moov (ftyp, then mdat is skipped by size, then moov), and the boxes inside moov except the table entries. With moov at the end this is one seek, not a scan of mdat.
• A movie box of at most 128 KiB is then read once, whole, into a pooled buffer; it is parsed from there and the sample-table windows below read from it too, so a small file opens with a read for the head, one for the box header and one for the box. A larger movie box is read through the windows as before, so the buffer never grows with the file. Reads outside the box go to the file.
• Table windows: one 4 KiB window per table when a packet cursor is created (at open); from the buffered movie box when there is one.
• Fragmented file: moov, then the box headers up to the first moof or mdat, to find sidx (served by the same read-ahead window as moov). The total duration is mvhd, else mehd, else the sum of sidx durations; when none states it, Duration is unknown. Knowing it exactly would take one read per fragment (what ffprobe does), so the probe does not. The mfra box at the end is read only on the first seek.
• Ranged sources (IsoBmffAsyncDemuxer): moov is fetched in one request and parsed from memory; each packet is one request, each fragment one request for its header box.

Measured numbers are in the unit report.

Forward-only sources

IsoBmffDemuxer.Open(IForwardMediaSource) reads boxes in order: moov must come before the media data (fragmented streams, faststart files); otherwise UnsupportedMediaException. After each moof the following mdat header is read so its end is known; samples are read by skipping to their offsets, which must not go backwards (isobmff.forward-order).

Sample entries

SampleEntry keeps the fixed fields that matter (dimensions, channels, bits, sample rate, QuickTime sound version 0–2) and every box after them in Extensions (QuickTime wave children are surfaced too). Configuration is the payload of the codec's configuration box, tagged with the box type: avcC, hvcC, vvcC, av1C, vpcC, esds (also for mp4v), dOps, dfLa, dac3, dec3, damr, alac; for text and metadata entries it is the bytes after the eight common entry bytes. Protected entries (encv, enca, …) report the clear format from sinf/frma as OriginalFormat and set IsProtected.

Codec ids: avc1/avc3 → h264, hvc1/hev1/dvh1/dvhe → hevc, vvc1/vvi1 → vvc, av01 → av1, vp08/vp09 → vp8/vp9, jpeg/mjpa/mjpb → mjpeg, mp4a and mp4v by the esds object type (0x40/0x66–0x68 → aac, 0x69/0x6B → mp3, 0x20 → mpeg4 (MPEG-4 part 2), 0x60–0x65 → mpeg2video, 0x6A → mpeg1video, 0xA5/0xA6 → ac3/eac3, 0xDD → vorbis), .mp3 and ms\0U → mp3, Opus, fLaC, ac-3, ec-3, samr, sawb, alac, the QuickTime PCM codes → pcm, tx3g, wvtt, stpp → ttml. Anything else maps to Unknown and keeps the code in MediaTrack.CodecTag.

Language: the media header's 15-bit packed ISO-639-2/T code, or, for values below 0x400, the QuickTime Macintosh language code mapped to ISO-639-2/T (0 = eng).

AMR-WB (sawb) carries the same damr setup box as AMR-NB (3GPP TS 26.244). Files whose sawb entry has no such box (observed in files written by ffmpeg, which MP4Box calls invalid) get an empty configuration.

Disposition: Default is set on the first enabled track of each kind, within its alternate group when it has one; Disabled marks tracks whose tkhd enabled flag is clear.

PCM audio (sowt, twos, in24, fl32, raw, ...) is stored one sample frame per table sample. The demuxer joins frames that follow each other in the file into packets of up to 1024 frames (the packet size ffprobe uses), so a minute of 48 kHz audio is 2,813 packets rather than 2.9 million. The first packet after a seek starts at the sample the seek landed on. MediaTrack.PacketCount is unknown (-1) for these tracks. Other codecs are never joined: QuickTime tracks that store a framed codec as single bytes (observed with MP3 in multitrack.mov, 17.6 million one-byte samples) are reported per table sample, because finding the frames is codec work.

A track may have several sample descriptions; IsoBmffDemuxer.SampleDescriptionIndex is the 1-based index for the packet last returned.

Differences from ffprobe -show_packets (7.0.2 and 8.0.1), each checked against the files

The comparison tool applies ffprobe's edit-list handling before comparing (it subtracts the first non-empty edit's start from every timestamp and adds leading empty edits) and lists the following as explained, never as agreement:

Everything else (size, byte offset, key flag, durations, timestamps) matches on every packet of every compared file.