feat: Dolby Vision IPT reshaping, HDR colour tagging and host-stated audio width

[bilipp]

Summary

Correct picture and sound for Dolby Vision / HDR / object-audio content. Engine side of bilipp/Lume#207; pairs with bilipp/Lume#220.

What a user saw, and what fixes it:

Symptom	Cause	Fix
DV profile 5 / 20 / 10.0 renders pink/green	The base layer is IPT-PQ-c2, not YCbCr (bl_signal_compatibility_id == 0); every path displayed it as BT.2020	GPU reshaping from the per-frame RPU → HDR10
Interlaced HDR broadcast renders washed out	CPU-produced buffers (software decode, and the deinterlacer, which downloads HW frames) carried no colour metadata	Colour tags on every PixelBufferFactory buffer
5.1 E-AC-3 JOC could play as stereo	The audio lane read the session width before the app had widened the route	Host-stated width via PlayerConfiguration.maxOutputChannels

Dolby Vision IPT reshaping (P5 / P20 / P10.0)

• Apple does not do this for us. AVAssetImageGenerator renders DVprofile20.mp4 exactly as tinted on macOS, so forwarding dvh1 signalling to VideoToolbox would not fix it either.

• DolbyVisionReshaper is a Metal compute kernel. It reads AV_FRAME_DATA_DOVI_METADATA and applies libplacebo's algorithm in order:

  1. polynomial / MMR reshaping curves
  2. the RPU's nonlinear ycc_to_rgb matrix and offset
  3. PQ EOTF
  4. inverse rgb_to_lms crosstalk × BT.2020 HPE LMS→RGB
  5. PQ OETF

  The output is a pooled x420 buffer tagged BT.2020 / PQ / BT.2020, so it's ordinary HDR10 from there on.

• Scope: runs only for frames whose colour space is AVCOL_SPC_IPT_C2. P8.1 (HDR10 base layer) and all other content never touch Metal.

• Both decode paths: reads VideoToolbox surfaces in place through a Metal texture cache, and handles software / PixelBufferFactory buffers the same way. The kernel is compiled from source on the first IPT frame, because SwiftPM compiles .metal resources only under Xcode.

• Degrades, never dies:

  • a frame without an RPU reuses the last one
  • with no Metal device or a GPU error, the frame is delivered untouched
  • the outcome is logged once on engine.lume/diagnostics as dv-reshape applied|skipped|failed

• Not covered:

  • the enhancement layer of dual-layer profiles (4 / 7)
  • HDR10 mastering metadata derived from DV L6 (only static metadata the source already carries is copied)

Colour metadata on CPU-produced buffers

• PixelBufferFactory attaches primaries, transfer function and YCbCr matrix, plus MDCV and CLL when the frame carries them.
• AVCOL_*_UNSPECIFIED maps to no attachment at all, never a guessed default: SD/IPTV emits UNSPECIFIED constantly, and a wrong tag would regress ordinary channels. Every key is set or removed, never left alone, because pool buffers are recycled.
• The VideoToolbox zero-copy path is left alone for non-IPT content. On an Apple TV 4K with a DV P8.1 title it already carries all three keys correctly and the picture is clean.

Detection, audio width, diagnostics

• DV detection: TrackInfo.Video.dolbyVision is read from AV_PKT_DATA_DOVI_CONF.
• Object audio: TrackInfo.Audio gains profile, isObjectAudio and channelLayoutName. The Atmos check is gated on codec id: the constant is 30 for both E-AC-3 and TrueHD, so an ungated check false-positives on any codec whose profile happens to be 30. Detection only; Atmos is still decoded to its channel bed.
• Audio width: PlayerConfiguration.maxOutputChannels lets the host state the negotiated width. nil keeps the old behaviour (read the session directly).
• Once-per-stream diagnostics: stream-open, video-format (names the delivery path), audio-width, dv-reshape. Never per frame.

Compressed-audio passthrough stays out of scope, documented in PLAN.md §7. That row for DV P5 now reads implemented.

Type of change

• feat — new capability

Invariants

• No blocking FFmpeg call was added to an actor, and no stage bypasses Channel. Reshaping runs synchronously on the video decode thread, which is the data plane.
• No rebuild-in-place. New knobs live on PlayerConfiguration; reshaper state is per VideoDecoder.
• Raw container timestamps still stop at TimestampUnwrapper.
• A/V sync is still owned by AVSampleBufferRenderSynchronizer.
• No force unwraps, as!, fatalError or assertionFailure in new engine code; failures are typed EngineErrors, and the caller degrades.
• No new global mutable state.
• No control flow derived from av_log strings. DV is read from packet/frame side data.

Verification

• swift build is warning-free for the new code.
• swift test is green:

Test run with 118 tests in 18 suites passed

• Tests added:
  • DolbyVisionReshapeTests:
    • GPU kernel vs CPU reference to within ±1 code
    • neutral IPT → neutral RGB
    • HDR10 tags on the output
    • on the real P20 sample: hardware and software paths both reshape and agree; the reported frame's water comes out blue-green (Cb above neutral) rather than green. Skipped when the gitignored TestStreams/DolbyVision/DVprofile20.mp4 is absent.
  • ColorTaggingTests: the AVCOL→CoreVideo table, exhaustively.
  • TrackDolbyDetectionTests: includes the profile-30 negative case.
  • StreamOpenDiagnosticsTests.
• Visual check. Frame at 10 s of DVprofile20.mp4: a plain decode and Apple's decoder both show a purple hippo in green water. The engine output shows a grey hippo in blue-teal sea, matching an independent numpy implementation of the same RPU math.
• Lume builds against this branch for the iOS and tvOS Simulators.

Platforms verified

• macOS: tests, including the real-sample hardware and software paths
• iOS / iPadOS: builds
• tvOS: builds
• visionOS

Reshaping has been verified on macOS only. It still needs a run on Apple TV / iPhone hardware: look for dv-reshape applied input='xf20' output='x420'. visionOS is not verified; no simulator runtime is installed here.

Real-world content

• DV P20: DVprofile20.mp4 — HEVC 3840×2160 @ 24 fps, 10-bit, full range, DV profile 20 level 6 (bl_signal_compatibility_id=0, RPU present, polynomial curves).
• DV P8.1: Matroska VOD on an Apple TV 4K — HEVC 2160p, BT.2020 / PQ, DV P8.1/L6, E-AC-3 JOC 5.1. Clean on the VideoToolbox path.

No URLs or credentials.