Device corpus

Real recordings from freely licensed sources, used to check demuxing, facts, telemetry, strip and repair against ffprobe, ExifTool and MediaInfo. The files are never committed: tools/corpus/devices-manifest.tsv lists each one (URL, SHA-256, size, licence, credit, device as the file states it, all fetch-only) and tools/corpus/devices-fetch.sh downloads them. devices-discover*.py and devices-probe.py found them (Internet Archive items and Zenodo records with a stated CC0, public-domain-mark, CC BY or CC BY-SA licence; a file is kept when it names a camera or phone, or is a screen recording). This page names devices and formats only.

Oracles: ffprobe 7.0.2 (8.1.3 for transport streams, where 7.0.2 crashes), ExifTool 13.60, MediaInfo 26.10, gpmfdemo from GoPro's gpmf-parser 2.2.1 release. Driver scripts: tools/run-device-corpus.sh (packets, info, facts), tools/run-device-triage.sh (cause per track), tools/device-repair-check.py (repair).

What is in it (193 files, 6.1 GB)

Demux against ffprobe

Every track of every file, packet by packet (size, payload hash, position, DTS, PTS, duration, key flag). 317,942 packets in ffprobe's listings (ISO/QuickTime/AVI files 210,958 in 130 files; 106,984 in the 63 transport streams); ours 210,422 and 106,984. Payload bytes, positions and hashes agree for every packet both list. The difference is listing and timing choices, sorted per track by tools/run-device-triage.sh (counts are tracks):

Causes (each is a documented choice in docs/isobmff.md and docs/mpeg.md, or recorded here):

• last-duration (61 tracks): the last sample's duration; ffprobe takes it from the edit or movie end, ours from the sample table.
• pos-key (54 tracks, TS): ours marks an intra picture that opens with a parameter set as a key frame; ffprobe 7.0.2 and 8.1.3 want a recovery point SEI (1,203 pictures in 45 camcorder files; where the SEI is present both agree).
• edit-drop (18): samples after the end of the last edit; ffprobe leaves them out, we list them.
• dts-shift (9): ffprobe moves DTS earlier for negative composition offsets.
• edit-repeat (4): an edit list that plays media twice.
• timing-other (23 tracks): all in the edit-start family: the first 3 or 4 packets differ in DTS or duration where ffprobe 7.0.2 applies the edit start to video DTS and to timed-metadata tracks in its own way; sample counts and bytes are equal. Not a bug found; no further cause isolated.
• key-superset (5 after the fix, 5 H.264 tracks): ffprobe marks more key frames than the sample table's sync list (6 vs 2, 21 vs 11, 25 vs 13, 27 vs 7, 28 vs 14). The extra ones are open-GOP I pictures that the stss box does not list; ffprobe marks them from the stream. Cause inferred from the counts; not checked picture by picture. We keep the container's statement.
• regroup (1, PCM): ours uses the sample table's chunk as the packet (22,050 bytes), ffprobe cuts 2,048-byte packets; bytes equal.
• extra-ffprobe-track (1): ffprobe lists one track more than ours in one file.

Fixed during this pass: an AVI whose index leaves the flag clear on every sound chunk (37 PCM packets): ffprobe marks all audio as key frames, ours took the index flag (marks 0). Audio chunks are key frames now (test AudioChunksAreKeyFramesEvenWhenTheIndexClearsTheFlag; the file re-checked, 1,147 packets equal).

Stream facts (tools/run-device-corpus.sh, infocheck)

130 non-TS files: 9 differ from ffprobe in a track or container fact; 63 transport streams: 0 (1 excused, listed in exceptions.tsv). Remaining differences:

Facts against ExifTool 13.60 and MediaInfo 26.10

metadata-compare over all 193 files (native keys, common fields, facts). Rows by field, ours against ExifTool and MediaInfo:

Native layer: 672 rows differ in form (ExifTool names and prints the same value differently); 418 match exactly, 201 match one of the two tools, 5 are not reported by the tools and 4 are not parsed by us.

Time zones: AVCHD clips carry the camera clock zone in the MDPM flags byte (63 files, read against ExifTool's printed zone); iPhone files carry creationdate with an offset.

Telemetry

• GoPro HERO10 GPMF: every stream equals ExifTool and gpmfdemo (counts and values).
• DJI Osmo Pocket 3 djmd: 40 of 40 samples equal ExifTool's reading.
• iPhone mebx: key and sample counts equal ExifTool on every file.

Strip

10 files carrying location or device ids, from several classes: 9 come out clean (ExifTool and MediaInfo show no location and no device id). The DJI Osmo Pocket 3 djmd samples still hold the camera serial number and position: the samples name protocol PP-101, which is not in the table of known layouts, so the strip leaves them unchanged and the report lists the track as not removed (strip.md).

Repair (cut at 60 %, tools/device-repair-check.py)

Twelve device models; "none" = no reference, "sibling" = another recording of the same model, "self" = the healthy file. Packets compared with the original's (bytes / key flags / times). All counts are samples of the recovered range.

What the table says: bytes and key flags equal for every recovered sample but one cause below; the one failure with a sibling reference is unexplained (Pixel AAC with a sibling: 7 of 343). Without a reference only files with in-band parameter sets recover (phones, some at Low); compact cameras and Sony MP4 need a reference. Time differences are limited to variable frame rate phones (the real durations are in the lost sample table) and to edit-list hiding of leading pictures.

Fixes made on these files (tests: ReconstructionTests, 30; VendorCameraTests, 9):

• Nikon L340 / D5300: the camera does not escape slice data (358 of 570 pictures hold 00 00 00 inside a NAL); the reference now switches the escape check off, and PCM runs end at a confirmed picture at any frame position. L340 self-reference was 1 of 570 frames; now 337 (video) and 564 PCM packets, all equal.
• PCM packets now equal the original's (2,048 bytes, short last packet of a chunk), compared byte for byte; only the final partial packet at the cut differs.
• Pixel AAC: a mett metadata track (206 to 232 byte samples, sometimes runs of equal sizes) sits between samples; gaps over 32 bytes broke the chain and false AAC starts inside it won. The sizes of the reference's unframed tracks now set the hop and the allowed gap. Self reference: AAC 348 of 348 equal (was 7). Xiaomi (runs of 452-byte samples): AAC 991 equal.
• Make, model and software from vendor atoms (Nikon NCDT/NCTG and TAGS, Olympus TAGS, Panasonic PANA, Canon CNMN, DJI ©mdl, Nikon AVI): all Make and Model rows on the corpus match ExifTool except the two listed above.

Planned for 1.0.x

Privacy first:

1. Strip leaves the DJI Osmo Pocket 3 serial number and location in its djmd telemetry samples: its protocol (PP-101) is not in the table of known layouts, so those samples are left unchanged. StripDeviceIds reports the field as not removed, so the leftover is visible in the report; until it is fixed, do not rely on strip to remove device identifiers from that camera's files.
2. TracksDuration takes telemetry tracks into account (DJI FC6310R, above).
3. AAC channel count: prefer the AudioSpecificConfig over the sample entry when they disagree.
4. Pixel AAC reconstructed with a sibling recording as reference (7 of 343 packets equal).
5. Panasonic "Ver.x" software, Fujifilm AVI make and model, Olympus MOV1/MOV2 layouts (no sample).
6. Leading pictures in the edit list of reconstructed compact-camera files.
7. The coverage gaps below: each is a device class to add and check when freely licensed files are found.

Coverage gaps

No Samsung phone; no iPhone with Dolby Vision, spatial video or slow motion; no XAVC; no real dashcam originals (the two found are re-encoded); no OBS file with its own metadata; no Olympus MOV1/MOV2, no Leica. One HERO10 for GPMF, one DJI Osmo Pocket 3 for djmd, one iPhone family for mebx. Transport streams come from AVCHD camcorders only; M2TS has no oracle sweep beyond ffprobe 8.1.3. Real recordings cut by a power loss (as opposed to cut by our generator) are absent.