Muxing

How packets become a file. The neutral contract is RecursiveAI.OneMedia.IMuxer (src/OneMedia/Muxing.cs); the ISO base media muxer is RecursiveAI.OneMedia.IsoBmff.IsoMuxer (src/OneMedia/IsoBmff/Mux/). This page is normative for the contract and for the ISO muxer; other muxers follow the contract and add their own page section.

The contract

1. AddTrack(MediaTrack, options?) declares a track from the reader's MediaTrack (codec id, CodecConfiguration, time base, edit/priming info, language, disposition). It returns the output track index. All tracks are added before the first packet.
2. WritePacket(PacketInfo, ReadOnlySpan<byte>) takes packets in decode order, all tracks interleaved in the order the caller wants them in the file. The payload is copied unchanged to the output. Nothing is repackaged by the muxer: Annex B → avcC and ADTS → raw are the codec layer's repackagers, called before the muxer.
3. Finish() finalises the file and returns a MuxReport (bytes written, packets and payload bytes per track, notes about anything the muxer rounded, guessed or synthesised). Dispose without Finish leaves the output incomplete.
4. Every call that can touch I/O takes a CancellationToken.

Refusals are MuxRefusedException with a MuxRefusalReason, the output track index and the codec: UnsupportedCodec, MissingConfiguration, InvalidConfiguration, NonMonotonicDts, MissingTimestamp, InvalidTrack, InvalidState, LimitExceeded, OutputNotSuitable. A refused AddTrack or WritePacket leaves the muxer usable. Each reason has a test (IsoMuxerTests).

Time is MediaTime. A packet's time base may differ from the track's; the muxer rescales exactly with round-to-nearest and reports a note when it rounded.

ISO base media muxer

Flavours and brands

CompatibleBrands and MinorVersion override the defaults.

Layouts

A sidx (WriteSegmentIndex) needs the fragments' sizes before it can be written in front of them, so fragments go through the spool in that mode. mfra (WriteRandomAccess) is written at the end.

What is stored per packet (constant working set)

Sample tables are paged lists, so memory grows with the packet count and never with the payload:

A 4.38 GB, 64,576-packet file is copy-muxed moov-first with 64 MB peak RSS (below). CompactSampleSizes writes stz2 when every size fits 4, 8 or 16 bits. stco becomes co64 per track when the last chunk offset passes 4 GiB; the movie box is rebuilt once its own size is known, since the offsets in front of the media data depend on it.

Chunks are runs of adjacent packets of one track. A source that interleaves finely produces many small chunks (8 bytes each); optional chunk buffering is a follow-up.

Time, edits and priming

• ISO/IEC 14496-12 §8.6.1.3: the first decode time of a track is zero. A track whose first DTS is not zero is shifted to zero and the difference becomes a leading empty edit (media_time −1) in the elst, so presentation time is preserved.
• The ctts is version 1 (signed) when any composition offset is negative.
• Durations are the gaps between decode times; the last packet's duration is the declared one (else the previous gap).
• The edit list is copied from MediaTrack.Edit.Edits when the reader gave one. It is read when Finish is called, so an operation that learns the list while packets stream (a trim knows the last frame only after seeing it) passes a list it fills before Finish. Otherwise it is built from Priming (a media_time after the priming frames), EndPadding (shorter segment) and the shift above. The movie timescale is the first edit's timescale, else the first track that needs a synthesised edit's timescale, else 1000 (MovieTimescale overrides).
• Creation and modification times are zero unless CreationTime is set, so output is a pure function of the input (tested: OutputIsDeterministic).
• A packet flagged Disposable makes the track write sdtp (is_depended_on = 2); fragments set the same bits in the sample flags.
• Opus gets a roll sample group (sgpd/sbgp, −80 ms of roll) and the iso2 brand.

Sample entries

Entries come from CodecConfiguration; a configuration in a different layout than the target's is refused as InvalidConfiguration naming the expected box.

IsoTrackOptions adds a track id, a handler type, extra sample-entry boxes (colr, pasp, clap…) and track-level boxes after mdia (tref, udta), written as given. One sample description per track.

Opus-in-ISOBMFF and FLAC-in-ISOBMFF come from Xiph's published encapsulation documents (see the sources list).

Fragmented files and CMAF

moov carries mvex/trex; each fragment is moof(mfhd, traf(tfhd, tfdt v1, trun)) + mdat, with default-base-is-moof and data offsets from the moof. A constant sample duration goes in tfhd; trun has v1 when a composition offset is negative. Cmaf = one track, brands iso6 + cmfc, fragments start at a keyframe (or any sample for audio). Per-fragment bookkeeping (sidx references, mfra entries) exists only when those boxes are asked for; a plain fragmented file keeps nothing per fragment. Incremental output: a fragment is on the destination as soon as the next fragment starts, so live output to a non-seekable destination holds one fragment.

Refusals

Non-monotonic DTS (NonMonotonicDts), a packet without a time (MissingTimestamp), unmappable codecs (UnsupportedCodec), missing or broken configuration, a track after the first packet or a packet after Finish (InvalidState), exceeded MediaLimits (LimitExceeded), a sidx with more than 65,535 references, CMAF with more than one track, and a non-seekable destination for a layout that needs to go back (OutputNotSuitable).

Not stored

Edit lists beyond what the reader reports, sparse tmcd and hint tracks (carry them through TrackBoxes), sgpd/sbgp other than Opus roll, subs, saiz/saio, any encryption box. Annex B → avcC repackaging belongs to remux, not to the muxer.

Differential method

tools/OneMedia.MuxCheck copy demuxes a file with our reader and muxes it (kind from the source's major brand), writing <out>.map (output track ← input track) and <out>.pkts (the reader's view of every packet, with SHA-256). tools/mux-compare.py runs ffprobe -show_packets -show_data_hash on input and output and compares per stream, in order: pts, dts, duration, size, flags and the data hash. tools/run-mux-diff.sh adds MP4Box -info and ffmpeg -v error -f null - on both files and counts messages that appear only for the output. tools/run-mux-sweep.sh runs the layouts: moov-first, moov-last, moov-last on a non-seekable output, fragmented, fragmented CMAF with sidx and mfra.

A difference is explained only when it falls in one of these classes (counted separately in the summary):

Deviations seen in the corpus sweep

• in-reader also covers: fragmented sources whose tfdt repeats (ffprobe collapses the timeline; the reader follows tfdt), QuickTime files with one-byte MP3 samples, and MP3 samples that ffprobe splits in progressive files only.
• A segment index (WriteSegmentIndex) makes ffprobe 7.0.2 shift the presentation times of the first fragment by an amount of its own when composition offsets are negative. The same bytes with the sidx renamed to free read back with one constant shift. Decode times are unaffected.
• A source that holds two samples at one decode time (generated/fmp4/every-frame.mp4) gives a progressive output with zero durations in stts; MP4Box refuses to open that file ("not enough bytes in box stts"; with the zero deltas patched it opens). The muxer notes it in the report and copies the source's timeline.
• MP4Box -info prints "Error fetching sample: Bad Parameter" for fragmented MP3 tracks, ours and ffmpeg's alike.
• ffprobe prints N/A as the duration of fragmented samples in places where the input, a progressive file, had one.
• ffmpeg's null muxer repeats "non monotonically increasing dts" for outputs of sources that already trigger it, with other numbers.