Audio elementary streams and their tags

MP3 (and MP2/MP1), ADTS AAC, AC-3 and E-AC-3, and AMR-NB/AMR-WB files, plus the tag systems that travel with them: ID3v1/1.1, ID3v2.2–2.4, APEv2 and Lyrics3. RecursiveAI.OneMedia.AudioStreams holds the readers, RecursiveAI.OneMedia.Tags.Id3, .Ape and .Lyrics3 the tag layers. Reading only; writing tags is a later feature, and the models keep byte ranges so it can round-trip them.

What a reader gives

One track and one packet per frame, with the frame's bytes exactly as stored (an ADTS packet keeps its header). Packets are keyframes. Times are exact in the stream's sample rate (MediaTime ticks of 1/sample-rate): packet n of an MPEG-1 Layer III stream starts at n × 1152 ticks.

All readers work over seekable, forward-only and ranged sources (Open, Open(IForwardMediaSource), OpenAsync). A leading ID3v2 tag is skipped by every one of them. A forward-only source cannot look behind a tag of more than about 200 KiB and cannot see the tail.

Duration and timing

• MPEG audio with a Xing, Info or VBRI frame (the first frame, no audio): the frame is not a packet. Its frame count gives the duration exactly and costs no scan. The LAME extension carries the encoder delay d and padding p (12 bits each). The track states Edit.Priming = d + 529 and Edit.EndPadding = max(0, p − 529) samples, and Duration is the presented length: frames × samples per frame − priming − padding. 529 is the reference decoder's own delay (528 + 1 samples), which LAME documents as part of the gapless arithmetic. The first frame is kept as Configuration (format mpeg-audio-info-frame) so the audio adapter can hand it to OneAudio's MP3 reader.
• Without a count (a plain CBR file, an ADTS or AC-3 file, AMR): Duration is an estimate, IsDurationEstimated is true. It is the audio's byte length times the samples-per-byte of the first (up to 128) units found in the first 16 KiB. Exact for constant bit rate, approximate for VBR. No scan, no extra reads. The audio's length excludes leading and trailing tags.
• AC-3/E-AC-3 times are exact in the stream's rate. ffprobe reports a 1/90000 base, which cannot hold 1536/44100 s, so its times round; the differential allows one tick there.

Resynchronisation and defects

A frame is accepted when its header is valid and either another frame of the same stream (the same stream signature: version, layer and rate for MPEG audio; the fixed header fields for ADTS; sample rate and bitstream class for AC-3/E-AC-3) starts right after it, or the data ends there. Bytes between accepted frames are skipped and reported (audio.junk, a warning at the offset of the first skipped byte, one per run of junk). A change of stream parameters in the middle is treated as junk, not as a new stream. A last frame that is cut off is delivered with PacketAttributes.Corrupt and reported once (audio.truncated-frame).

Deviation from ffprobe (junk). ffprobe folds the skipped bytes into the next packet, so its packet is longer than the frame and starts earlier; ours is the frame and the junk is reported. Packet counts agree; sizes and positions differ for the packets after junk. Damaged files live in the corpus' _defects folder, which sweeps skip.

Free format (MPEG audio with bit-rate index 0, ISO/IEC 11172-3 §2.4.2.3): the frame length is the distance to the next header with the same stable fields, confirmed by the header after that; the length found is tried first for the following frames, so each costs one comparison. ffprobe 7 and 8 cannot open such files ("Failed to find two consecutive MPEG audio frames"), so they have no differential; the reader is checked against the constant-bit-rate file the free-format file was made from (same frames, headers rewritten), see the results.

AMR is strict: a table-of-contents byte must have its padding bits (0x83) clear and an FT that is a speech mode or NO_DATA; anything else is not a frame. The #!AMR\n, #!AMR-WB\n magic and the multichannel forms (#!AMR_MC1.0\n, #!AMR-WB_MC1.0\n with a channel-description word) are read; a multichannel file's frames hold one TOC per channel and are delivered whole.

AC-3 / E-AC-3 syncframes are found by 0x0B77 plus a valid header (fscod, frmsizecod or frmsiz, bsid ≤ 10 for AC-3, 11–16 for E-AC-3). A dependent frame (strmtyp 1) or a frame of a further substream belongs to the access unit that precedes it (ETSI TS 102 366, Annex E).

Tags at the ends

At open, a seekable source is asked for the last 128 bytes (ID3v1), the 32 bytes before that (an APEv2 footer), the 15 before that (Lyrics3), and a 3DI footer, up to six rounds, so the usual orders (audio, Lyrics3, APE, ID3v1 and audio, APE, ID3v1) are found with a handful of small reads. Those blocks are removed from the audio (AudioEnd is where they start). An ID3v2 tag in the middle of a stream (a spliced file) is skipped and reported as audio.inner-id3 (info). An ID3v2 tag appended after the audio is only found when it has a footer.

Seeking

There is no index in these formats. Reading remembers the offset and time of every 256th unit; Seek jumps to the last remembered unit at or before the target, then walks headers (no payload decoding, only header parses over the buffered bytes) to the target. The cost is at most 256 units from a place already read, or the whole distance the first time a position beyond what was read is asked for. Seek lands on a frame boundary: PreviousKeyframe on the frame containing the target, NextKeyframe on the first frame starting at or after it. Forward-only sources cannot seek.

ID3v2

• Header (ID3v2.4 §3.1): ID3, version and revision (never 0xFF), flags, 28-bit synchsafe size. A footer (3DI) is accounted for. The tag extends over header + body (+ footer); Padding is the run of zero bytes after the last frame.
• Versions. 2.2 has 3-character frame ids and 24-bit sizes; 2.3 has 4-character ids and plain big-endian sizes; 2.4 has synchsafe sizes. Frame ids are kept as stored (a 2.2 tag keeps TT2); the common-view mapping knows all three spellings.
• Unsynchronisation. A tag-wide flag (2.2/2.3: the whole body; 2.4: set only when every frame is) is undone by reading the tag whole (bounded by MaxTagBytes) and serving values from memory. A 2.4 frame-level flag is undone for that frame. Byte ranges still point into the file.
• Extended header (2.3: size, flags, padding size, optional CRC; 2.4: synchsafe size, flag bytes, optional CRC/restrictions) is skipped and its range kept.
• Frame flags. Compression (2.3: zlib with a 4-byte decompressed size; 2.4: via the data length indicator) is inflated within MaxTagBytes; a zlib failure is id3.compressed-data and the frame stays opaque. Encrypted frames and frames with a group id are exposed, never decrypted: the frame has Flags, EncryptionMethod, GroupId and its DataRange, and a binary NativeTag.
• Text frames decode all four encodings (ISO-8859-1, UTF-16 with BOM, UTF-16BE, UTF-8). 2.2/2.3 tags that name encoding 3 are read as UTF-8 and reported (id3.text-encoding). Multiple values (2.4 nul-separated) are kept in Id3Frame.Values; NativeTag.Text joins them with "; ".
• Frames understood beyond text: TXXX, WXXX, COMM, USLT, SYLT, APIC/PIC (cover art, MIME, picture type, lazy value), POPM, UFID, PRIV, GEOB, CHAP/CTOC (chapters with sub-frames, Id3Tag.Chapters()). ReplayGain is read from TXXX frames. Any other frame is a binary NativeTag with its range, so nothing is lost. Native keys are the stored frame id, plus :description for the frames that have one (TXXX:REPLAYGAIN_TRACK_GAIN, COMM:desc when the description is non-empty).
• Dates. TDRC/TYER+TDAT+TIME give the date string; TDRC is also a capture-time candidate (timezone Unknown).
• Entry points for other containers. Id3Tag.Read(IMediaSource, offset, …) reads a tag that starts at ID3 anywhere in a source (a WAV id3 chunk, an AIFF ID3 chunk, a FLAC prefix); Id3Tag.Read(ReadOnlyMemory<byte>, baseOffset, …) reads one from memory (an MP4 ID32 payload after its 6-byte header). Id3Tag.TryRead returns false instead of throwing for a non-tag.

ID3v1, APEv2, Lyrics3

• ID3v1 / 1.1: TAG + 30/30/30/4/30 bytes + genre; a zero byte at offset 125 with a non-zero track byte makes it 1.1. The enhanced TAG+ block before it is kept as Extended. Genre numbers 0–191 (ID3v1 and Winamp extensions) resolve through Id3Genre. Text is ISO-8859-1.
• APEv2. 32-byte header and footer (APETAGEX, version 1000/2000, tag size including the footer and items but not the header, item count, flags). Item: value size, flags, key (2–255 characters of 0x20–0x7E, zero-terminated), value. Item flag bits 1–2: 0 UTF-8 text, 1 binary, 2 locator, 3 reserved; bit 0 read-only. Global flags: bit 31 header present, bit 30 the footer flag, bit 29 "this is the header". A tag is found from its footer at the end (header optional) or from a header at the start. Text values with several entries are zero-separated and are joined with "; ". Cover art (Cover Art (Front), …) is a binary item: file name, zero byte, image; it becomes an image tag with MIME sniffed from the image. Keys are compared case-insensitively by the common view; the file keeps their case. Sources and the one conflict. Two public descriptions of the APEv2 layout were compared item by item (the mutagen-specs page and the Hydrogenaudio wiki pages, archived copies): layout, sizes, item format, key alphabet and item-flag bits agree. They disagree on bit 30 (mutagen-specs: 1 = no footer; Hydrogenaudio: 1 = footer). Real files settle it. ExifTool 13.60 cannot write APE tags ("isn't writable"), so the writers observed are mp3gain 1.6.2 (MP3, mp3gain -s a) and WavPack 5.9.0 (wvtag, a .wv file): both write a header with flags 0xA0000000 and a footer with 0x80000000. With a footer present, bit 30 is clear, which is the mutagen-specs reading ("1 = no footer"); the Hydrogenaudio reading would have set it. ExifTool reads the mp3gain file (Replaygain Track Gain, Mp 3gain Minmax, …) and so do we (fixture testdata/audio-es/ape-mp3gain.mp3, a test). We follow mutagen-specs; the bit is used only to decide whether a tag read from its header has a footer. A key with characters outside 0x20–0x7E is kept and reported (ape.key-chars); a key shorter than two characters or unterminated stops the tag (ape.key).
• Lyrics3. v2.00 (LYRICSBEGIN … 6-digit size … LYRICS200; the size counts LYRICSBEGIN and the fields) is split into its three-letter fields with five-digit sizes; v1 (LYRICSBEGIN … LYRICSEND, ≤ 5100 bytes) is one Latin-1 text item LYRICS. Both are native tags in system lyrics3 and are kept out of the audio. The common view takes ETT/EAR/EAL as title/artist/album and LYR (or the v1 block) as lyrics, after ID3v2 and APE.

Defect catalogue

A typed exception (MediaFormatException) is raised only when no frame can be found at all (MissingRequired) or the AMR magic is wrong. Limits that apply: MaxTagBytes (tag size, decoded and inflated frames), MaxElementCount (frames and items), MaxAttachmentBytes (pictures when read), MaxPacketBytes.

Common view

TagMappingTable has rows for id3v2 (all three spellings), id3v1, ape and lyrics3; the generated table is in tag-mapping.md. Title, artist, album, album artist, composer, genre (ID3 TCON references and ID3v1 numbers resolve to names), date, comment, lyrics, track and disc number/total (3/12 splits), ReplayGain track/album gain and peak, encoder, and cover art. The first system with a value wins in the order ID3v2, APE, ID3v1.

Ranged sources and I/O

OpenAsync fetches the first 256 KiB and the last 16 KiB of the source, opens the same synchronous reader over those bytes and, when a read falls outside what was fetched, fetches the missing window (256 KiB at least) and retries. Memory stays at five windows however large the source is. A normal file opens in three requests (the format sniff, the head, the tail); packets are pulled in 256 KiB steps. Large tags (cover art) are fetched when a value is read. The measured bytes and range counts, next to ffprobe, MediaInfo and ExifTool, are in the results below.

The audio adapter (RecursiveAI.OneMedia.Audio)

media.Decode() works for all of them: AAC from ADTS gets the AudioSpecificConfig and the packets without their ADTS headers; AMR gets a damr configuration; MP3 goes to OneAudio's MP3 reader as a byte stream that starts with the Xing/Info frame, so OneAudio's own gapless handling applies and the decoded length of a LAME file equals ffmpeg's. The adapter does not rebuild gapless data.

Differential method

tools/audio-es/run-all.sh runs on the worker: generate.sh writes the corpus (ffmpeg 7.0.2 static build for the MP3/MP2/AAC/AC-3/E-AC-3/AMR files; make_tags.py for ID3v2.2/2.3/2.4/APE/Lyrics3 layouts that no encoder writes; mp3gain and id3lib's id3v2 as independent writers of APEv2 and ID3v1/ID3v2 (ExifTool cannot write MP3 files)), then compares packets (diff), stream facts (infocheck), decoded length (audiodecode), tags (tagcompare in the bench project) and I/O (io). Documented differences from ffprobe, each with its rule in tools/OneMedia.Oracles/exceptions.tsv:

Results (worker, Linux x64; oracles ffprobe/ffmpeg 7.0.2, ExifTool 13.60, MediaInfo 26.10)

• Packets vs ffprobe 7.0.2 (diff): 132 generated files (MP3/MP2 including MPEG-2.5, CBR/ABR/VBR, with and without Xing/Info/LAME and with ID3v1/v2.2/2.3/2.4, APEv1/v2 and Lyrics3 tags, 2 free-format; ADTS LC and Main in 8–96 kHz and up to 6 channels; AC-3 and E-AC-3; AMR-NB and -WB at every mode). 130 equal (size, hash, pts, dts, duration, key flag); the 2 others are the free-format files, which ffprobe cannot open. The free-format files give the same packets as their constant-bit-rate twins (116 packets / 48,483 bytes and 117 / 24,450). Every difference seen on the way is a rule in exceptions.tsv (table above).
• Stream facts vs ffprobe (infocheck): 129 files, 0 differences (free-format files skipped).
• Decoded length (media.Decode() through the adapter, against ffmpeg -f s16le): 107 files with a decoder in OneAudio (MP3 with and without LAME gapless data, ADTS LC, AMR-NB/-WB), all equal to the frame. OneAudio's own reader on the whole file differs where the file has no Info frame (it drops 529 frames of decoder delay that ffmpeg keeps) and where a block that is not ID3v1 trails the audio (APEv1, footer-only APEv2, Lyrics3: it decodes the tag bytes); through the adapter neither happens. AAC with more than two channels and AC-3/E-AC-3 are not decoded by OneAudio.
• Tags (tagcompare, 69 files, 497 field readings, ExifTool 13.60, ATL.NET 7.19.0, TagLib# 2.3.0): ours agree with all three on 404 readings. The rest, each explained: ATL.NET puts the file name in Title when there is no tag (41); TagLib# returns nothing for frames with a data-length indicator (ExifTool and ours read them); ATL.NET shows compressed frame bytes and does not undo v2.3 unsynchronisation inside frames; ID3v2.4 TDRC is kept as the ISO 8601 string where ExifTool rewrites it as 2019:05:17 10:30:00 and ATL.NET/TagLib# keep the year; with an ID3v1 and an APE tag present the APE value wins (TagLib#'s order; ID3v1 truncates at 30 characters). Writers behind the fixtures: ffmpeg 7.0.2 (ID3v2.3/2.4, Info), our generator for layouts no encoder writes, mp3gain 1.6.2 (APEv2), id3lib's id3v2 0.1.12 (ID3v2.3, ID3v1). ExifTool cannot write MP3 files.
• I/O, probe plus metadata (io; bytes / read calls; HTTP = ranged requests to the local range server, strace = read/pread64): the 2.2 GB MP3 with and without Xing frame: ours 282,624 B / 4 requests over HTTP and 151,783 B / 12 reads locally; ffprobe 360,576 B / 3 and 163,968 B / 5; MediaInfo 180,384–196,768 B / 3 and 147,616 B / 4; ExifTool 96,159 B / 16. Without a Xing frame the duration is the documented estimate and nothing is scanned. Short files are read whole by every tool (the clips are 3–60 s); the tables are in docs/io-results.txt style in the worker's results directory.
• Allocation, pull path (Release, a second full pass after Seek to the start, the 2.2 GB file included): 0 bytes over 5,263,848 packets; MP3, AAC, AC-3 and AMR 0 B per packet. The first pass allocates the seek index (two lists, one entry per 256 packets, 512 KiB at 2.2 GB). Peak resident memory of the whole 2.2 GB pull: 35 MB.
• Fuzz (SharpFuzz under libFuzzer, 601 s each, one process, 0 crashes): id3 4,013,702 runs (one crash found earlier, a zlib failure inside a compressed frame, fixed and kept as testdata/fuzz/id3/), ape 12,973,686, mpa-es 4,049,038, adts-ac3-amr-es 5,742,026.
• Release suite (worker, dotnet test -c Release): 403 tests pass, none skipped. NativeAOT publish of the smoke app (linux-x64, warnings as errors): 0 trim or AOT warnings, the app runs and its audio-stream checks pass. WebAssembly smoke app under node: 0 failures.
• Tag-read speed, indicative (worker, 7 files, --reps 3, run as a queued job so other work shared the machine; not a publishable row): geomean 0.71× of the fastest managed comparator (TagLib#; ATL.NET is slower on every file), worst 0.94×, allocations 89–313 KiB per open against TagLib#'s 15–47 KiB (the read buffer is rented from the shared pool; the first run, which allocated it, measured 1.03× geomean and 1.69× worst).