From 622ac6d7c4a5b46d16f5094d396f40151ad844ab Mon Sep 17 00:00:00 2001 From: Philipp Bischoff Date: Tue, 29 Sep 2026 18:07:12 +0200 Subject: [PATCH 1/2] feat(decode): convert IPT-base-layer Dolby Vision to HDR10 on the GPU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dolby Vision profiles 5, 10.0 and 20 (base-layer signal compatibility ID 0) carry IPTPQc2, not YCbCr. The renderer showed it as YCbCr, so DVprofile20.mp4 played with a pink/green cast. No colour tag fixes that: FFmpeg drops the matrix attachment (CoreVideo has none for IPT on decoded buffers), and the RPU reshaping changes per scene. Apple's own decoders don't apply it either — AVAssetReader and AVAssetImageGenerator both return raw IPT from a correctly signalled dvh1 track. VideoDecoder.deliver now hands such frames to DolbyVisionConverter, a Metal compute pass that applies the frame's RPU (FFmpeg's AV_FRAME_DATA_DOVI_METADATA): polynomial and MMR reshaping, IPT→L'M'S', PQ decode, LMS→BT.2020 through the RPU matrix and a crosstalk-free Hunt-Pointer-Estevez transform (derived from BT.2100's ICtCp matrix, matching libplacebo's published constant), then HDR10 Y'CbCr into a tagged 10-bit video-range buffer. The renderer tone-maps it like any HDR10 stream. Hardware frames stay on the GPU via the texture cache. Detection is the stream's DV configuration record or an IPT_C2 frame — many P5 encodes leave the VUI matrix unspecified. HDR10-compatible profiles (8.1 etc.) stay on the zero-copy path. A GPU refusal skips conversion for that frame only (iOS refuses a backgrounded app); only a converter that cannot exist gives up for the session, degrading to the old tinted picture rather than stopping. Dynamic DV display management (L1/L2/L8 trims) is not carried over; the output is static HDR10. Tests: metadata extraction from a synthetic AVDOVIMetadata, GPU output against an independent Double reference (poly + MMR, odd sizes, two chroma sitings; within a code), and real-stream cases on the local, gitignored DV samples (skipped where absent, CI included). The test target now depends on CFFmpeg directly to build FFmpeg structs. --- AGENTS.md | 3 +- PLAN.md | 4 +- Package.swift | 4 +- .../Decode/DolbyVisionConverter.swift | 452 ++++++++++++++++ .../Decode/DolbyVisionMapping.swift | 201 ++++++++ .../LumeEngineCore/Decode/VideoDecoder.swift | 80 ++- Sources/LumeEngineCore/FFI/Frame.swift | 5 +- .../DolbyVisionTests.swift | 484 ++++++++++++++++++ 8 files changed, 1225 insertions(+), 8 deletions(-) create mode 100644 Sources/LumeEngineCore/Decode/DolbyVisionConverter.swift create mode 100644 Sources/LumeEngineCore/Decode/DolbyVisionMapping.swift create mode 100644 Tests/LumeEngineCoreTests/DolbyVisionTests.swift diff --git a/AGENTS.md b/AGENTS.md index 8c83fc3..76a71a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,7 @@ This split is the core invariant of the codebase: - **RAII FFI wrappers:** `FFI/Packet`, `FFI/Frame` etc. wrap `AVPacket*`/`AVFrame*` with deinit-based free. `@unchecked Sendable` is allowed **only** on these audited immutable-after-construction wrappers and on `Channel` — never on state-bearing types. No `Unmanaged` outside FFI. - **A/V sync is Apple's job:** decoded frames are enqueued into `AVSampleBufferRenderSynchronizer` (`Render/SystemRenderer`). Do not add drop-frame heuristics or hand-rolled clocks. - **Deinterlacing never costs the hardware decoder.** `Decode/VideoFilterGraph` (bwdif, falling back to yadif) sits between `avcodec_receive_frame` and delivery; VideoToolbox frames are downloaded with `av_hwframe_transfer_data` and filtered on the CPU. Do *not* "simplify" this into switching the codec to software when a stream is interlaced: that costs a mid-stream codec rebuild and keyframe resync at every interlaced/progressive splice, which live broadcasts (ad breaks in a 1080i match) are full of. Engagement is decided per frame from `AV_FRAME_FLAG_INTERLACED` (PLAN.md §3, failure 6) and, once engaged, everything stays on one path — `deint=interlaced` passes progressive frames through. Filter output carries the sink's time base, which a field-doubling filter *halves*; PTS and durations are rescaled back to engine µs in `VideoDecoder`. +- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. - **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs. - **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes. - **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`. @@ -81,4 +82,4 @@ This split is the core invariant of the codebase: ### Tests -`Tests/LumeEngineCoreTests/` covers the historically crash-prone areas by design: timestamp wraparound (against a real wrap-seam fixture `wrap.ts`), channel backpressure/teardown, fixture-based decode, deinterlacing (against a field-coded `interlaced.ts`), playback, and resilience via `Support/TortureHTTPServer.swift` (a fault-injecting local HTTP server — mid-stream errors, stalls, disconnects). New work in demux/decode/time/concurrency should come with tests here; add new media fixtures to `generate-fixtures.sh`, not as committed binaries. +`Tests/LumeEngineCoreTests/` covers the historically crash-prone areas by design: timestamp wraparound (against a real wrap-seam fixture `wrap.ts`), channel backpressure/teardown, fixture-based decode, deinterlacing (against a field-coded `interlaced.ts`), Dolby Vision conversion (GPU output against an independent CPU reference; the real-stream cases use the non-redistributable samples in the gitignored `TestStreams/DolbyVision/` and skip without them, CI included), playback, and resilience via `Support/TortureHTTPServer.swift` (a fault-injecting local HTTP server — mid-stream errors, stalls, disconnects). New work in demux/decode/time/concurrency should come with tests here; add new media fixtures to `generate-fixtures.sh`, not as committed binaries. diff --git a/PLAN.md b/PLAN.md index f6fd498..13c1b8e 100644 --- a/PLAN.md +++ b/PLAN.md @@ -196,7 +196,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the - **Zero-delay switching:** `prepare(next:)` opens a second session through first-frame-decoded, then atomically swaps renderer attachment; old session torn down async. (Also powers Lume's next-episode auto-advance and channel zapping.) - **Precache:** same AVIO disk-cache layer, `precache(url:leadingSeconds:)` background task with LRU budget. - **Upscaling:** MetalFX spatial scaler stage (D5), quality/perf switch, auto-disable on thermal pressure (`ProcessInfo.thermalState`). -- **HDR:** HDR10/HLG passthrough via sample-buffer attachments; DV P5/P8: preserve `dvcC` config + RPU side data into CMSampleBuffer format description extensions (VideoToolbox tone-maps); HDR10+ dynamic metadata via per-frame attachments. EDR headroom handling on macOS/visionOS Metal path. +- **HDR:** HDR10/HLG passthrough via sample-buffer attachments. DV with an HDR10-compatible base layer (P8.1, P10.1, …) plays as HDR10. DV with an IPTPQc2 base layer (P5, P10.0, P20 — compatibility ID 0) is converted to HDR10 before enqueue by a Metal compute pass (`Decode/DolbyVisionConverter`) that applies each frame's RPU reshaping (polynomial + MMR) and the IPT→LMS→BT.2020 matrices FFmpeg exports as `AV_FRAME_DATA_DOVI_METADATA`; the renderer then tone-maps it like any HDR10 stream. Dynamic DV display management (L1/L2/L8 trims) is not carried over. HDR10+ dynamic metadata via per-frame attachments. EDR headroom handling on macOS/visionOS Metal path. - **360°/VR:** MetalRenderer projection (equirect sphere / fisheye), CoreMotion + gesture camera; visionOS immersive presentation later. - **Blu-ray ISO/DVD:** libbluray/libdvdnav behind the `AVIOProtocol` seam; menu-less title playback first (longest-title heuristic), chapters mapped to the chapter API. - **AI subtitles:** whisper.cpp (Metal) tapping the decoded-audio channel; streaming word-level segments feed the subtitle model as a synthetic track; on-device translation via Translation.framework. @@ -208,7 +208,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the | Risk | Assessment | Mitigation | |---|---|---| -| Dolby Vision P5 (proprietary IPT-PQ-C2) | Hard. VideoToolbox handles DV streams only when correctly signaled; P5 has no BL fallback. | Phase-gated spike: test `dvh1/dvhe` signaling through `AVSampleBufferVideoRenderer` on-device early (Phase 3). If VT won't tone-map P5, fall back to libplacebo-style Metal tone mapping (stretch) or document as limitation. | +| Dolby Vision P5/P20 (proprietary IPT-PQ-C2) | Resolved for decoded-frame enqueue: VideoToolbox does *not* reshape — `AVAssetReader` and `AVAssetImageGenerator` both return raw IPT even from a correctly signaled `dvh1` track, and tagging a buffer `IPT_C2` changes nothing. | Metal DV→HDR10 conversion (§6 HDR). Still open: compressed `dvh1` enqueue into `AVSampleBufferVideoRenderer`, the only route to true DV output (TV in DV mode) — needs an on-device spike, and would be a second decode path. | | Dolby AC-4 | No FFmpeg decoder (patent-encumbered); Apple decodes AC-4 only via its own frameworks. | Investigation ticket: attempt `AVSampleBufferAudioRenderer` passthrough of AC-4 `CMSampleBuffer`s (compressed enqueue). If rejected → out of scope, documented. | | Audio passthrough (compressed Dolby out) | Apple exposes no raw bitstream-out API; `AVSampleBufferAudioRenderer` *can* accept some compressed formats. | Same compressed-enqueue spike (EAC-3/Atmos-in-EAC3 first — that's the one that matters for AVRs). | | `AVSampleBufferRenderSynchronizer` edge cases (rate ≠ 1 with filters, very high fps) | Medium. | MetalRenderer is the escape hatch; sync-policy abstraction keeps both backends behind one interface. | diff --git a/Package.swift b/Package.swift index e8bfcff..2636e3f 100644 --- a/Package.swift +++ b/Package.swift @@ -118,7 +118,9 @@ let package = Package( .testTarget( name: "LumeEngineCoreTests", - dependencies: ["LumeEngineCore", "LumeEngine"], + // CFFmpeg directly: some tests build FFmpeg structs (e.g. Dolby + // Vision metadata) to exercise the engine's readers of them. + dependencies: ["LumeEngineCore", "LumeEngine", "CFFmpeg"], exclude: ["Fixtures/generate-fixtures.sh"] ), ], diff --git a/Sources/LumeEngineCore/Decode/DolbyVisionConverter.swift b/Sources/LumeEngineCore/Decode/DolbyVisionConverter.swift new file mode 100644 index 0000000..878dbf7 --- /dev/null +++ b/Sources/LumeEngineCore/Decode/DolbyVisionConverter.swift @@ -0,0 +1,452 @@ +import CoreVideo +import Metal + +/// Converts Dolby Vision frames whose base layer is IPTPQc2 (profiles 5, 10.0, +/// 20) into HDR10 — BT.2020 PQ YCbCr in a 10-bit video-range buffer — on the +/// GPU, CVPixelBuffer to CVPixelBuffer, before enqueue (PLAN.md §4, §6). +/// +/// Per pixel, in one compute pass: reshape the base layer with the frame's +/// RPU curves, IPT→L'M'S' with the RPU matrix, PQ-decode, LMS→BT.2020, then +/// re-encode as PQ BT.2020 YCbCr. The renderer takes it from there exactly as +/// it takes any HDR10 stream, tone mapping included. What this does *not* +/// carry over is Dolby's dynamic display management (L1/L2/L8 trims): the +/// output is static HDR10, and Apple's HDR10 path does the tone mapping. +/// +/// Decode-thread-only, like `PixelBufferFactory`. Hardware frames never leave +/// the GPU: VideoToolbox surfaces are bound as Metal textures through the +/// texture cache, zero-copy. +final class DolbyVisionConverter { + private let device: MTLDevice + private let queue: MTLCommandQueue + private let pipeline: MTLComputePipelineState + private let textureCache: CVMetalTextureCache + + private var pool: CVPixelBufferPool? + private var poolWidth = 0 + private var poolHeight = 0 + + init() throws { + guard let device = MTLCreateSystemDefaultDevice(), + let queue = device.makeCommandQueue() + else { + throw EngineError(code: .unsupported, message: "Dolby Vision conversion needs Metal; no GPU available") + } + do { + // Compiled from source at runtime: SwiftPM's command-line build does + // not compile .metal files, and the kernel is small. Once per session + // that actually carries Dolby Vision. + let options = MTLCompileOptions() + // Fast math halves the kernel's cost (it is mostly `pow`), which + // matters on an Apple TV: the decode thread waits for every frame. + // Safe here — every `pow` input is clamped non-negative and no + // divisor can reach zero — and it stays within a code of the + // Double reference (`DolbyVisionTests`). + options.mathMode = .fast + let library = try device.makeLibrary(source: DolbyVisionShader.source, options: options) + guard let function = library.makeFunction(name: DolbyVisionShader.kernelName) else { + throw EngineError(code: .internalError, message: "Dolby Vision kernel missing from library") + } + pipeline = try device.makeComputePipelineState(function: function) + } catch let error as EngineError { + throw error + } catch { + throw EngineError(code: .internalError, message: "Dolby Vision kernel failed to compile: \(error)") + } + + var cache: CVMetalTextureCache? + guard CVMetalTextureCacheCreate(nil, nil, device, nil, &cache) == kCVReturnSuccess, let cache else { + throw EngineError(code: .internalError, message: "CVMetalTextureCacheCreate failed") + } + self.device = device + self.queue = queue + self.textureCache = cache + } + + /// Where the source's chroma samples sit relative to luma, in luma pixels. + /// FFmpeg reports it per frame (`AVFrame.chroma_location`). + struct ChromaSiting: Equatable { + var horizontal: Float + var vertical: Float + + /// Both co-sited with the top-left luma sample. + static let topLeft = ChromaSiting(horizontal: 0, vertical: 0) + /// Horizontally co-sited, vertically between rows (MPEG-2, HEVC default). + static let left = ChromaSiting(horizontal: 0, vertical: 0.5) + static let center = ChromaSiting(horizontal: 0.5, vertical: 0.5) + static let top = ChromaSiting(horizontal: 0.5, vertical: 0) + } + + /// Converts one frame. Blocks until the GPU is done — this runs on the + /// decode thread, which is allowed to block (PLAN.md D1). + /// + /// Throws `.renderFailed` when the GPU failed or refused the work, which + /// is transient (iOS refuses GPU work from a backgrounded app); any other + /// error means this source cannot be converted at all. + func convert( + _ source: CVPixelBuffer, + mapping: DolbyVisionMapping, + chromaSiting: ChromaSiting + ) throws -> CVPixelBuffer { + let width = CVPixelBufferGetWidth(source) + let height = CVPixelBufferGetHeight(source) + guard let input = InputFormat(CVPixelBufferGetPixelFormatType(source)), + CVPixelBufferGetPlaneCount(source) == 2 + else { + throw EngineError( + code: .unsupported, + message: "Dolby Vision conversion: unsupported source pixel format \(CVPixelBufferGetPixelFormatType(source))" + ) + } + + let output = try dequeueBuffer(width: width, height: height) + + let sourceLuma = try texture(source, plane: 0, format: input.lumaFormat) + let sourceChroma = try texture(source, plane: 1, format: input.chromaFormat) + let outputLuma = try texture(output, plane: 0, format: .r16Unorm) + let outputChroma = try texture(output, plane: 1, format: .rg16Unorm) + defer { CVMetalTextureCacheFlush(textureCache, 0) } + + guard let sourceLumaTexture = CVMetalTextureGetTexture(sourceLuma), + let sourceChromaTexture = CVMetalTextureGetTexture(sourceChroma), + let outputLumaTexture = CVMetalTextureGetTexture(outputLuma), + let outputChromaTexture = CVMetalTextureGetTexture(outputChroma), + let commandBuffer = queue.makeCommandBuffer(), + let encoder = commandBuffer.makeComputeCommandEncoder() + else { + throw EngineError(code: .internalError, message: "Dolby Vision conversion: Metal setup failed") + } + + let parameters = DolbyVisionShader.parameters( + for: mapping, + inputScale: input.normalizationScale(baseLayerBitDepth: mapping.baseLayerBitDepth), + chromaSiting: chromaSiting + ) + + encoder.setComputePipelineState(pipeline) + encoder.setTexture(sourceLumaTexture, index: 0) + encoder.setTexture(sourceChromaTexture, index: 1) + encoder.setTexture(outputLumaTexture, index: 2) + encoder.setTexture(outputChromaTexture, index: 3) + parameters.withUnsafeBytes { bytes in + if let base = bytes.baseAddress { + encoder.setBytes(base, length: bytes.count, index: 0) + } + } + + // One thread per output chroma sample, i.e. per 2×2 block of luma. + let blocks = MTLSize(width: outputChromaTexture.width, height: outputChromaTexture.height, depth: 1) + let groupWidth = pipeline.threadExecutionWidth + let groupHeight = max(1, pipeline.maxTotalThreadsPerThreadgroup / groupWidth / 4) + let group = MTLSize(width: groupWidth, height: groupHeight, depth: 1) + let grid = MTLSize( + width: (blocks.width + groupWidth - 1) / groupWidth, + height: (blocks.height + groupHeight - 1) / groupHeight, + depth: 1 + ) + encoder.dispatchThreadgroups(grid, threadsPerThreadgroup: group) + encoder.endEncoding() + commandBuffer.commit() + commandBuffer.waitUntilCompleted() + + guard commandBuffer.status == .completed else { + let detail = commandBuffer.error.map { " — \($0.localizedDescription)" } ?? "" + throw EngineError(code: .renderFailed, message: "Dolby Vision conversion failed on the GPU\(detail)") + } + // Textures (and with them the surfaces) are held until here by the + // CVMetalTexture locals above. + withExtendedLifetime((sourceLuma, sourceChroma, outputLuma, outputChroma)) {} + + tagAsHDR10(output) + return output + } + + // MARK: Buffers + + private func dequeueBuffer(width: Int, height: Int) throws -> CVPixelBuffer { + if pool == nil || poolWidth != width || poolHeight != height { + let attributes: [CFString: Any] = [ + kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + kCVPixelBufferWidthKey: width, + kCVPixelBufferHeightKey: height, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary, + kCVPixelBufferMetalCompatibilityKey: true, + ] + var newPool: CVPixelBufferPool? + let status = CVPixelBufferPoolCreate(nil, nil, attributes as CFDictionary, &newPool) + guard status == kCVReturnSuccess, let newPool else { + throw EngineError(code: .decodeFailed, message: "CVPixelBufferPoolCreate failed (\(status))") + } + pool = newPool + poolWidth = width + poolHeight = height + } + + var buffer: CVPixelBuffer? + guard let pool else { + throw EngineError(code: .internalError, message: "pixel buffer pool missing") + } + let status = CVPixelBufferPoolCreatePixelBuffer(nil, pool, &buffer) + guard status == kCVReturnSuccess, let buffer else { + throw EngineError(code: .decodeFailed, message: "pixel buffer allocation failed (\(status))") + } + return buffer + } + + private func texture(_ buffer: CVPixelBuffer, plane: Int, format: MTLPixelFormat) throws -> CVMetalTexture { + var texture: CVMetalTexture? + let status = CVMetalTextureCacheCreateTextureFromImage( + nil, textureCache, buffer, nil, format, + CVPixelBufferGetWidthOfPlane(buffer, plane), + CVPixelBufferGetHeightOfPlane(buffer, plane), + plane, &texture + ) + guard status == kCVReturnSuccess, let texture else { + throw EngineError(code: .internalError, message: "Metal texture for plane \(plane) failed (\(status))") + } + return texture + } + + /// What the kernel wrote, spelled out for CoreMedia. The format description + /// is built from these attachments (`SampleBufferBuilder`), which is what + /// makes the renderer treat the stream as HDR10. + private func tagAsHDR10(_ buffer: CVPixelBuffer) { + let tags: [CFString: CFString] = [ + kCVImageBufferYCbCrMatrixKey: kCVImageBufferYCbCrMatrix_ITU_R_2020, + kCVImageBufferColorPrimariesKey: kCVImageBufferColorPrimaries_ITU_R_2020, + kCVImageBufferTransferFunctionKey: kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ, + // The kernel averages each 2×2 block into one chroma sample. + kCVImageBufferChromaLocationTopFieldKey: kCVImageBufferChromaLocation_Center, + kCVImageBufferChromaLocationBottomFieldKey: kCVImageBufferChromaLocation_Center, + ] + for (key, value) in tags { + CVBufferSetAttachment(buffer, key, value, .shouldPropagate) + } + } + + /// Source layouts the kernel reads: 4:2:0 bi-planar, 8 or 10 bit. Both + /// the VideoToolbox and the software path deliver one of these. + private struct InputFormat { + let lumaFormat: MTLPixelFormat + let chromaFormat: MTLPixelFormat + /// Bits of real code value per sample (10-bit formats keep them in the + /// top of 16). + let significantBits: Int + + init?(_ pixelFormat: OSType) { + switch pixelFormat { + case kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + kCVPixelFormatType_420YpCbCr10BiPlanarFullRange: + lumaFormat = .r16Unorm + chromaFormat = .rg16Unorm + significantBits = 10 + case kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, + kCVPixelFormatType_420YpCbCr8BiPlanarFullRange: + lumaFormat = .r8Unorm + chromaFormat = .rg8Unorm + significantBits = 8 + default: + return nil + } + } + + /// Factor from a sampled unorm value to the base layer's code value as + /// a fraction of its range — what the RPU's pivots are expressed in. + /// The range flag of the pixel format plays no part: it labels the + /// codes, the codes themselves are what the curves were built for. + func normalizationScale(baseLayerBitDepth: Int) -> Float { + let containerBits = lumaFormat == .r16Unorm ? 16 : 8 + // unorm · (2^container − 1) is the raw container value; the code + // sits in its top `significantBits`. + let code = Double((1 << containerBits) - 1) / Double(1 << (containerBits - significantBits)) + let rescaled = code * Double(1 << baseLayerBitDepth) / Double(1 << significantBits) + return Float(rescaled / Double((1 << baseLayerBitDepth) - 1)) + } + } +} + +// MARK: - Kernel + +/// The kernel and the layout of its parameter block. The layout constants are +/// interpolated into the Metal source, so Swift and the shader cannot drift. +enum DolbyVisionShader { + static let kernelName = "lume_dovi_to_hdr10" + + // Parameter block, in floats. + static let inputScaleIndex = 0 + static let chromaOffsetIndex = 1 // x, y + static let yccToRGBIndex = 4 // 9, row-major + static let yccOffsetIndex = 13 // 3 + static let lmsToRGBIndex = 16 // 9, row-major + static let curvesIndex = 28 + + // One reshaping curve: piece count, pivots, pieces. + static let maxPieces = 8 + static let curvePivotsIndex = 1 // maxPieces + 1 + static let curvePiecesIndex = 10 + // One piece: method, polynomial (3), MMR order, MMR constant, MMR (3 × 7). + static let pieceStride = 27 + static let curveStride = curvePiecesIndex + maxPieces * pieceStride + static let parameterCount = curvesIndex + 3 * curveStride + + static func parameters( + for mapping: DolbyVisionMapping, + inputScale: Float, + chromaSiting: DolbyVisionConverter.ChromaSiting + ) -> [Float] { + var block = [Float](repeating: 0, count: parameterCount) + block[inputScaleIndex] = inputScale + block[chromaOffsetIndex] = chromaSiting.horizontal + block[chromaOffsetIndex + 1] = chromaSiting.vertical + for (offset, value) in mapping.yccToRGB.enumerated() { block[yccToRGBIndex + offset] = Float(value) } + for (offset, value) in mapping.yccToRGBOffset.enumerated() { block[yccOffsetIndex + offset] = Float(value) } + for (offset, value) in mapping.linearLMSToRGB.enumerated() { block[lmsToRGBIndex + offset] = Float(value) } + + for (component, curve) in mapping.curves.prefix(3).enumerated() { + let base = curvesIndex + component * curveStride + let pieces = curve.pieces.prefix(maxPieces) + block[base] = Float(pieces.count) + for index in 0...maxPieces { + // Pad with the last pivot so a stray read stays in range. + let pivot = curve.pivots[min(index, curve.pivots.count - 1)] + block[base + curvePivotsIndex + index] = Float(pivot) + } + for (index, piece) in pieces.enumerated() { + let start = base + curvePiecesIndex + index * pieceStride + switch piece { + case .polynomial(let coefficients): + block[start] = 0 + for (term, value) in coefficients.prefix(3).enumerated() { block[start + 1 + term] = Float(value) } + case .mmr(let constant, let coefficients): + block[start] = 1 + block[start + 4] = Float(min(coefficients.count, 3)) + block[start + 5] = Float(constant) + for (order, row) in coefficients.prefix(3).enumerated() { + for (term, value) in row.prefix(7).enumerated() { + block[start + 6 + order * 7 + term] = Float(value) + } + } + } + } + } + return block + } + + static let source = """ + #include + using namespace metal; + + constant int kInputScale = \(inputScaleIndex); + constant int kChromaOffset = \(chromaOffsetIndex); + constant int kYccToRGB = \(yccToRGBIndex); + constant int kYccOffset = \(yccOffsetIndex); + constant int kLMSToRGB = \(lmsToRGBIndex); + constant int kCurves = \(curvesIndex); + constant int kCurveStride = \(curveStride); + constant int kCurvePivots = \(curvePivotsIndex); + constant int kCurvePieces = \(curvePiecesIndex); + constant int kPieceStride = \(pieceStride); + + // SMPTE ST 2084. Linear values are relative to 10 000 cd/m². + constant float kM1 = 2610.0 / 16384.0; + constant float kM2 = 2523.0 / 4096.0 * 128.0; + constant float kC1 = 3424.0 / 4096.0; + constant float kC2 = 2413.0 / 4096.0 * 32.0; + constant float kC3 = 2392.0 / 4096.0 * 32.0; + + static float3 pq_eotf(float3 e) { + float3 p = pow(max(e, 0.0), 1.0 / kM2); + return pow(max(p - kC1, 0.0) / (kC2 - kC3 * p), 1.0 / kM1); + } + + static float3 pq_oetf(float3 y) { + float3 p = pow(clamp(y, 0.0, 1.0), kM1); + return pow((kC1 + kC2 * p) / (1.0 + kC3 * p), kM2); + } + + static float3 mul3(constant float *m, float3 v) { + return float3(dot(float3(m[0], m[1], m[2]), v), + dot(float3(m[3], m[4], m[5]), v), + dot(float3(m[6], m[7], m[8]), v)); + } + + // One component of the RPU reshaping. The piece is chosen by the + // component's own input; MMR pieces regress on all three. + static float reshape(constant float *curve, float3 sig, int component) { + int pieces = int(curve[0]); + constant float *pivots = curve + kCurvePivots; + float s = clamp(sig[component], pivots[0], pivots[pieces]); + int piece = 0; + for (int i = 1; i < pieces; i++) { + if (s >= pivots[i]) piece = i; + } + constant float *p = curve + kCurvePieces + piece * kPieceStride; + float out; + if (p[0] < 0.5) { + out = p[1] + s * (p[2] + s * p[3]); + } else { + float4 cross = float4(sig.x * sig.y, sig.x * sig.z, sig.y * sig.z, sig.x * sig.y * sig.z); + float3 power = sig; + float4 crossPower = cross; + out = p[5]; + constant float *c = p + 6; + int order = int(p[4]); + for (int k = 0; k < order; k++) { + out += dot(float3(c[0], c[1], c[2]), power) + + dot(float4(c[3], c[4], c[5], c[6]), crossPower); + c += 7; + power *= sig; + crossPower *= cross; + } + } + return clamp(out, 0.0, 1.0); + } + + // Base-layer sample → HDR10 Y'CbCr (chroma centred on 0). + static float3 convert(constant float *params, float3 sig) { + float3 reshaped = float3(reshape(params + kCurves, sig, 0), + reshape(params + kCurves + kCurveStride, sig, 1), + reshape(params + kCurves + 2 * kCurveStride, sig, 2)); + float3 offset = float3(params[kYccOffset], params[kYccOffset + 1], params[kYccOffset + 2]); + float3 lms = pq_eotf(mul3(params + kYccToRGB, reshaped - offset)); + float3 rgb = pq_oetf(max(mul3(params + kLMSToRGB, lms), 0.0)); + float y = dot(rgb, float3(0.2627, 0.6780, 0.0593)); + return float3(y, (rgb.b - y) / 1.8814, (rgb.r - y) / 1.4746); + } + + // One thread per 2×2 luma block: four luma samples, one chroma sample. + kernel void \(kernelName)(texture2d srcLuma [[texture(0)]], + texture2d srcChroma [[texture(1)]], + texture2d dstLuma [[texture(2)]], + texture2d dstChroma [[texture(3)]], + constant float *params [[buffer(0)]], + uint2 block [[thread_position_in_grid]]) { + if (block.x >= dstChroma.get_width() || block.y >= dstChroma.get_height()) return; + constexpr sampler bilinear(coord::pixel, address::clamp_to_edge, filter::linear); + + uint width = srcLuma.get_width(); + uint height = srcLuma.get_height(); + float scale = params[kInputScale]; + float2 siting = float2(params[kChromaOffset], params[kChromaOffset + 1]); + + float2 chroma = 0.0; + float count = 0.0; + for (uint dy = 0; dy < 2; dy++) { + for (uint dx = 0; dx < 2; dx++) { + uint2 pixel = block * 2 + uint2(dx, dy); + if (pixel.x >= width || pixel.y >= height) continue; + // Chroma sample i sits at luma position 2i + siting; +0.5 is + // the texel centre in pixel coordinates. + float2 at = (float2(pixel) - siting) * 0.5 + 0.5; + float3 sig = float3(srcLuma.read(pixel).r, srcChroma.sample(bilinear, at).rg) * scale; + float3 ycc = convert(params, sig); + // 10-bit video range, stored in the top of 16 bits. + dstLuma.write(float4(round(64.0 + 876.0 * ycc.x) * 64.0 / 65535.0), pixel); + chroma += ycc.yz; + count += 1.0; + } + } + float2 codes = round(512.0 + 896.0 * clamp(chroma / count, -0.5, 0.5)); + dstChroma.write(float4(codes * 64.0 / 65535.0, 0.0, 0.0), block); + } + """ +} diff --git a/Sources/LumeEngineCore/Decode/DolbyVisionMapping.swift b/Sources/LumeEngineCore/Decode/DolbyVisionMapping.swift new file mode 100644 index 0000000..e532103 --- /dev/null +++ b/Sources/LumeEngineCore/Decode/DolbyVisionMapping.swift @@ -0,0 +1,201 @@ +internal import CFFmpeg + +/// The per-frame Dolby Vision metadata needed to turn an IPTPQc2 base layer +/// into displayable pixels: the reshaping curves from the RPU, plus the +/// colour matrices that take the reshaped signal to linear BT.2020. +/// +/// Dolby Vision profiles 5, 10.0 and 20 carry a base layer with signal +/// compatibility ID 0 — it is *not* YCbCr. Shown as if it were, it comes out +/// with a pink/green cast, and no colour tag can fix that: the reshaping +/// changes per scene and CoreVideo has no IPT matrix for decoded buffers. +/// `DolbyVisionConverter` applies this metadata on the GPU and hands the +/// renderer HDR10 (PLAN.md §6). +/// +/// Everything is normalized here, once per frame, so the shader only does +/// arithmetic: pivots are fractions of the base layer's code range and +/// coefficients are plain values (FFmpeg delivers them as fixed point over +/// `2^coef_log2_denom`, whatever the bitstream used). +struct DolbyVisionMapping: Sendable, Equatable { + enum Piece: Sendable, Equatable { + /// `c0 + c1·x + c2·x²` of the component's own input. + case polynomial([Double]) + /// Multivariate multiple regression over all three inputs `(y, u, v)`: + /// `constant + Σₖ (a·yᵏ + b·uᵏ + c·vᵏ + d·(yu)ᵏ + e·(yv)ᵏ + f·(uv)ᵏ + g·(yuv)ᵏ)` + /// for `k` in `1...order`, one row of seven coefficients per order. + case mmr(constant: Double, coefficients: [[Double]]) + } + + struct Curve: Sendable, Equatable { + /// Ascending piece boundaries, normalized to `0...1`. One more than + /// `pieces`. + var pivots: [Double] + var pieces: [Piece] + } + + /// Bit depth of the base layer the curves' pivots refer to. + var baseLayerBitDepth: Int + /// Reshaping curves for the three base-layer components (I, P, T). + var curves: [Curve] + /// Row-major 3×3: reshaped signal (minus `yccToRGBOffset`) to PQ-coded LMS. + var yccToRGB: [Double] + var yccToRGBOffset: [Double] + /// Row-major 3×3 from the RPU, applied to linear LMS. FFmpeg documents its + /// output as feeding a Hunt-Pointer-Estevez LMS→RGB matrix *without* + /// crosstalk — `linearLMSToRGB` composes the two. + var rgbToLMS: [Double] + + /// Row-major 3×3: linear LMS (after PQ decoding) to linear BT.2020 RGB. + var linearLMSToRGB: [Double] { + Matrix3.multiply(Matrix3.hpeLMSToBT2020, rgbToLMS) + } +} + +// MARK: - Extraction from FFmpeg + +extension DolbyVisionMapping { + /// Reads the mapping FFmpeg attached to a decoded frame + /// (`AV_FRAME_DATA_DOVI_METADATA`). Nil when the frame carries none. + init?(frame: UnsafePointer) { + guard let sideData = av_frame_get_side_data(frame, AV_FRAME_DATA_DOVI_METADATA), + let bytes = sideData.pointee.data + else { return nil } + self.init(metadata: UnsafeRawPointer(bytes).assumingMemoryBound(to: AVDOVIMetadata.self)) + } + + /// Reads a mapping from FFmpeg's metadata struct. Nil when it is internally + /// inconsistent (FFmpeg validates the bitstream, so this is a backstop, not + /// an expected path). + init?(metadata: UnsafePointer) { + guard let header = av_dovi_get_header(metadata)?.pointee, + let mapping = av_dovi_get_mapping(metadata), + let color = av_dovi_get_color(metadata)?.pointee + else { return nil } + + let bitDepth = Int(header.bl_bit_depth) + guard (8...16).contains(bitDepth) else { return nil } + let codeMax = Double((1 << bitDepth) - 1) + let scale = Double(sign: .plus, exponent: -Int(header.coef_log2_denom), significand: 1) + + var curves: [Curve] = [] + for component in 0..<3 { + let curve = Self.curve(mapping, component) + let pivotCount = Int(curve.num_pivots) + guard (2...Int(AV_DOVI_MAX_PIECES) + 1).contains(pivotCount) else { return nil } + + let pieceCount = pivotCount - 1 + let rawPivots: [UInt16] = Self.array(curve.pivots, count: pivotCount) + let methods: [AVDOVIMappingMethod] = Self.array(curve.mapping_idc, count: pieceCount) + let mmrOrders: [UInt8] = Self.array(curve.mmr_order, count: pieceCount) + let mmrConstants: [Int64] = Self.array(curve.mmr_constant, count: pieceCount) + // Flattened [piece][3] and [piece][order][7]. + let polyCoefficients: [Int64] = Self.array(curve.poly_coef, count: pieceCount * 3) + let mmrCoefficients: [Int64] = Self.array(curve.mmr_coef, count: pieceCount * 3 * 7) + + var pieces: [Piece] = [] + for index in 0..) -> Bool { + guard let sideData = av_packet_side_data_get( + parameters.pointee.coded_side_data, + parameters.pointee.nb_coded_side_data, + AV_PKT_DATA_DOVI_CONF + ), let bytes = sideData.pointee.data, + sideData.pointee.size >= MemoryLayout.size + else { return false } + let record = UnsafeRawPointer(bytes).loadUnaligned(as: AVDOVIDecoderConfigurationRecord.self) + // Profile 4 also reports ID 0, but its base layer is SDR plus an + // enhancement layer this path does not decode. + return record.dv_bl_signal_compatibility_id == 0 && [5, 10, 20].contains(record.dv_profile) + } + + private static func value(_ rational: AVRational) -> Double { + rational.den == 0 ? 0 : Double(rational.num) / Double(rational.den) + } + + // The C arrays import as tuples; these copy them out by reinterpreting the + // tuple's storage, which is laid out exactly like the C array. + + private static func curve( + _ mapping: UnsafePointer, _ component: Int + ) -> AVDOVIReshapingCurve { + withUnsafeBytes(of: mapping.pointee.curves) { + $0.load(fromByteOffset: component * MemoryLayout.stride, as: AVDOVIReshapingCurve.self) + } + } + + /// First `count` elements of a C array imported as a (possibly nested) + /// tuple. Callers name `Element`, and the count never exceeds the array. + private static func array(_ tuple: Tuple, count: Int) -> [Element] { + withUnsafeBytes(of: tuple) { bytes in + (0...stride, as: Element.self) } + } + } +} + +// MARK: - Fixed colour science + +/// Row-major 3×3 helpers. Tiny and allocation-happy by design: they run once +/// per frame on nine numbers. +enum Matrix3 { + static func multiply(_ a: [Double], _ b: [Double]) -> [Double] { + (0..<9).map { index in + let row = index / 3, column = index % 3 + return (0..<3).reduce(0) { $0 + a[row * 3 + $1] * b[$1 * 3 + column] } + } + } + + static func inverse(_ m: [Double]) -> [Double] { + let cofactors = [ + m[4] * m[8] - m[5] * m[7], m[2] * m[7] - m[1] * m[8], m[1] * m[5] - m[2] * m[4], + m[5] * m[6] - m[3] * m[8], m[0] * m[8] - m[2] * m[6], m[2] * m[3] - m[0] * m[5], + m[3] * m[7] - m[4] * m[6], m[1] * m[6] - m[0] * m[7], m[0] * m[4] - m[1] * m[3], + ] + let determinant = m[0] * cofactors[0] + m[1] * cofactors[3] + m[2] * cofactors[6] + return cofactors.map { $0 / determinant } + } + + /// Linear LMS (Hunt-Pointer-Estevez, no crosstalk) to linear BT.2020 RGB. + /// + /// Derived rather than transcribed: BT.2100 defines ICtCp's RGB→LMS matrix + /// as this HPE transform with 4 % crosstalk already folded in, + /// `M = X(0.04) · HPE`, where `X(c)` has `1 − 2c` on the diagonal and `c` + /// elsewhere. So `HPE⁻¹ = M⁻¹ · X(0.04)`. + static let hpeLMSToBT2020: [Double] = { + let ictcp = [1688.0, 2146, 262, 683, 2951, 462, 99, 309, 3688].map { $0 / 4096 } + let crosstalk = [0.92, 0.04, 0.04, 0.04, 0.92, 0.04, 0.04, 0.04, 0.92] + return multiply(inverse(ictcp), crosstalk) + }() +} diff --git a/Sources/LumeEngineCore/Decode/VideoDecoder.swift b/Sources/LumeEngineCore/Decode/VideoDecoder.swift index 89cdba5..69873d1 100644 --- a/Sources/LumeEngineCore/Decode/VideoDecoder.swift +++ b/Sources/LumeEngineCore/Decode/VideoDecoder.swift @@ -83,6 +83,7 @@ public final class VideoDecoder: @unchecked Sendable { private var stopRequested = false private var drainRequested = false private var deinterlaceActive = false + private var dolbyVisionActive = false // Decode-thread-only state. private var codecContext: UnsafeMutablePointer? @@ -107,6 +108,24 @@ public final class VideoDecoder: @unchecked Sendable { private var downloadedFrame: UnsafeMutablePointer? private var filteredFrame: UnsafeMutablePointer? + // Dolby Vision state, decode-thread-only. + /// The stream's Dolby Vision configuration says its base layer is IPTPQc2 + /// (profiles 5, 10.0, 20), which no renderer can show as YCbCr. + private let dolbyVisionBaseLayerIsIPT: Bool + /// Built on the first frame that needs it, so sessions without Dolby + /// Vision never touch Metal. + private var dolbyVisionConverter: DolbyVisionConverter? { + didSet { + lock.lock() + dolbyVisionActive = dolbyVisionConverter != nil + lock.unlock() + } + } + /// Set when the GPU path could not be built or failed mid-stream. The rest + /// of the session plays unconverted, tinted rather than stopped (PLAN.md + /// §3.3 — degrade, never crash), exactly like a failed deinterlacer. + private var dolbyVisionGivenUp = false + private let maxConsecutiveErrors = 100 public init( @@ -121,6 +140,7 @@ public final class VideoDecoder: @unchecked Sendable { self.output = output self.policy = policy self.deinterlacing = deinterlacing + self.dolbyVisionBaseLayerIsIPT = DolbyVisionMapping.baseLayerIsIPT(parameters.raw) var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -145,6 +165,14 @@ public final class VideoDecoder: @unchecked Sendable { return deinterlaceActive } + /// True while Dolby Vision frames with an IPT base layer are being + /// converted to HDR10 on the GPU. + public var isConvertingDolbyVision: Bool { + lock.lock() + defer { lock.unlock() } + return dolbyVisionActive + } + // MARK: Control (any thread) public func start() { @@ -228,6 +256,7 @@ public final class VideoDecoder: @unchecked Sendable { } defer { filterGraph = nil + dolbyVisionConverter = nil av_frame_free(&downloadedFrame) av_frame_free(&filteredFrame) } @@ -444,8 +473,8 @@ public final class VideoDecoder: @unchecked Sendable { let duration = max(duration, 0) let serial = currentSerial ?? 0 - let pixelBuffer: CVPixelBuffer - let hardware = frame.pointee.format == AV_PIX_FMT_VIDEOTOOLBOX.rawValue + var pixelBuffer: CVPixelBuffer + var hardware = frame.pointee.format == AV_PIX_FMT_VIDEOTOOLBOX.rawValue if hardware { guard let opaque = frame.pointee.data.3 else { return } // +0 borrow; storing into VideoFrame retains it before av_frame_unref. @@ -459,6 +488,11 @@ public final class VideoDecoder: @unchecked Sendable { } } + if let converted = convertDolbyVision(frame, pixelBuffer) { + pixelBuffer = converted + hardware = false // an engine-written surface, not VideoToolbox's + } + let videoFrame = VideoFrame( pixelBuffer: pixelBuffer, pts: pts, @@ -470,6 +504,48 @@ public final class VideoDecoder: @unchecked Sendable { try? output.send(videoFrame) } + // MARK: Dolby Vision (decode thread only) + + /// Converts a Dolby Vision frame whose base layer is IPTPQc2 to HDR10; + /// nil for every other frame, which is delivered as decoded. Profile 8 + /// and the like carry the same metadata, but their base layer is already + /// HDR10, so they stay on the zero-copy path. + private func convertDolbyVision( + _ frame: UnsafeMutablePointer, + _ pixelBuffer: CVPixelBuffer + ) -> CVPixelBuffer? { + guard !dolbyVisionGivenUp, + dolbyVisionBaseLayerIsIPT || frame.pointee.colorspace == AVCOL_SPC_IPT_C2, + let mapping = DolbyVisionMapping(frame: frame) + else { return nil } + + do { + let converter = try dolbyVisionConverter ?? DolbyVisionConverter() + dolbyVisionConverter = converter + return try converter.convert(pixelBuffer, mapping: mapping, chromaSiting: Self.chromaSiting(of: frame)) + } catch let error as EngineError where error.code == .renderFailed { + // The GPU refused this frame — iOS does that to a backgrounded app — + // so this one goes out as decoded and the next one tries again. + // Giving up here would leave the picture tinted after the app comes + // back to the foreground. + return nil + } catch { + dolbyVisionGivenUp = true + dolbyVisionConverter = nil + return nil + } + } + + private static func chromaSiting(of frame: UnsafeMutablePointer) -> DolbyVisionConverter.ChromaSiting { + switch frame.pointee.chroma_location { + case AVCHROMA_LOC_TOPLEFT: .topLeft + case AVCHROMA_LOC_CENTER: .center + case AVCHROMA_LOC_TOP: .top + // Unspecified reads as the HEVC (and MPEG-2) default. + default: .left + } + } + private func drainCodec(into frame: UnsafeMutablePointer, emitEOFSerial serial: UInt64?) { guard let context = codecContext else { return } avcodec_send_packet(context, nil) diff --git a/Sources/LumeEngineCore/FFI/Frame.swift b/Sources/LumeEngineCore/FFI/Frame.swift index 24d3996..22c0ff2 100644 --- a/Sources/LumeEngineCore/FFI/Frame.swift +++ b/Sources/LumeEngineCore/FFI/Frame.swift @@ -13,8 +13,9 @@ public final class VideoFrame: @unchecked Sendable { public let width: Int public let height: Int /// True when the pixel buffer is a VideoToolbox surface handed over - /// zero-copy. False for anything the CPU had to write, including frames - /// that were hardware-decoded and then deinterlaced. + /// zero-copy. False for anything the engine had to write, including frames + /// that were hardware-decoded and then deinterlaced (CPU) or converted + /// from Dolby Vision to HDR10 (GPU). public let isHardwareDecoded: Bool init( diff --git a/Tests/LumeEngineCoreTests/DolbyVisionTests.swift b/Tests/LumeEngineCoreTests/DolbyVisionTests.swift new file mode 100644 index 0000000..5a2d2f4 --- /dev/null +++ b/Tests/LumeEngineCoreTests/DolbyVisionTests.swift @@ -0,0 +1,484 @@ +import CFFmpeg +import CoreVideo +import Foundation +import Metal +import Testing +@testable import LumeEngineCore + +/// Dolby Vision with an IPTPQc2 base layer (profiles 5, 10.0, 20). Shown as +/// YCbCr it comes out pink/green; the engine reshapes it with the frame's RPU +/// and hands the renderer HDR10 instead. +@Suite("Dolby Vision", .serialized) +struct DolbyVisionTests { + // MARK: Colour science + + @Test("HPE LMS→RGB matches the published matrix and preserves white") + func hpeMatrix() { + // libplacebo's pl_ipt_lms2rgb, derived from the unquantized HPE + // transform. The engine derives it from BT.2100's 1/4096-quantized + // ICtCp matrix instead, so they agree to that quantization. + let published = [ + 3.06441879, -2.16597676, 0.10155818, + -0.65612108, 1.78554118, -0.12943749, + 0.01736321, -0.04725154, 1.03004253, + ] + let derived = Matrix3.hpeLMSToBT2020 + for (lhs, rhs) in zip(derived, published) { + #expect(abs(lhs - rhs) < 5e-4, "\(derived) vs \(published)") + } + for row in 0..<3 { + let sum = derived[row * 3] + derived[row * 3 + 1] + derived[row * 3 + 2] + #expect(abs(sum - 1) < 1e-9, "row \(row) sums to \(sum): white would shift") + } + } + + // MARK: Metadata extraction + + @Test("reads FFmpeg's metadata: pivots normalized, fixed point unscaled, MMR rows intact") + func extraction() throws { + var size = 0 + let metadata = try #require(av_dovi_metadata_alloc(&size)) + defer { av_free(metadata) } + let one: Int64 = 1 << 23 + + let header = try #require(av_dovi_get_header(metadata)) + header.pointee.bl_bit_depth = 10 + header.pointee.coef_log2_denom = 23 + + let mapping = try #require(av_dovi_get_mapping(metadata)) + // I: two polynomial pieces. + mapping.pointee.curves.0.num_pivots = 3 + mapping.pointee.curves.0.pivots.0 = 0 + mapping.pointee.curves.0.pivots.1 = 400 + mapping.pointee.curves.0.pivots.2 = 1023 + mapping.pointee.curves.0.mapping_idc.0 = AV_DOVI_MAPPING_POLYNOMIAL + mapping.pointee.curves.0.mapping_idc.1 = AV_DOVI_MAPPING_POLYNOMIAL + mapping.pointee.curves.0.poly_coef.0 = (-612_867, 9_803_467, 0) + mapping.pointee.curves.0.poly_coef.1 = (one, 0, -one / 4) + // P: one second-order MMR piece. + mapping.pointee.curves.1.num_pivots = 2 + mapping.pointee.curves.1.pivots.0 = 0 + mapping.pointee.curves.1.pivots.1 = 1023 + mapping.pointee.curves.1.mapping_idc.0 = AV_DOVI_MAPPING_MMR + mapping.pointee.curves.1.mmr_order.0 = 2 + mapping.pointee.curves.1.mmr_constant.0 = -one / 8 + mapping.pointee.curves.1.mmr_coef.0 = ( + (one, 2 * one, 3 * one, 4 * one, 5 * one, 6 * one, 7 * one), + (-one, -2 * one, -3 * one, -4 * one, -5 * one, -6 * one, -7 * one), + (99 * one, 99 * one, 99 * one, 99 * one, 99 * one, 99 * one, 99 * one) // beyond order: ignored + ) + // T: identity. + mapping.pointee.curves.2.num_pivots = 2 + mapping.pointee.curves.2.pivots.0 = 0 + mapping.pointee.curves.2.pivots.1 = 1023 + mapping.pointee.curves.2.poly_coef.0 = (0, one, 0) + + let color = try #require(av_dovi_get_color(metadata)) + color.pointee.ycc_to_rgb_matrix.0 = AVRational(num: 8192, den: 8192) + color.pointee.ycc_to_rgb_matrix.1 = AVRational(num: 799, den: 8192) + color.pointee.ycc_to_rgb_offset.1 = AVRational(num: 1 << 27, den: 1 << 28) + color.pointee.rgb_to_lms_matrix.0 = AVRational(num: 17081, den: 16384) + color.pointee.rgb_to_lms_matrix.4 = AVRational(num: 16384, den: 16384) + + let parsed = try #require(DolbyVisionMapping(metadata: metadata)) + #expect(parsed.baseLayerBitDepth == 10) + #expect(parsed.curves.count == 3) + + #expect(parsed.curves[0].pivots == [0, 400.0 / 1023, 1]) + #expect(parsed.curves[0].pieces == [ + .polynomial([-612_867.0 / 8_388_608, 9_803_467.0 / 8_388_608, 0]), + .polynomial([1, 0, -0.25]), + ]) + #expect(parsed.curves[1].pieces == [ + .mmr(constant: -0.125, coefficients: [[1, 2, 3, 4, 5, 6, 7], [-1, -2, -3, -4, -5, -6, -7]]), + ]) + #expect(parsed.curves[2].pieces == [.polynomial([0, 1, 0])]) + + #expect(parsed.yccToRGB[0] == 1) + #expect(parsed.yccToRGB[1] == 799.0 / 8192) + #expect(parsed.yccToRGB[2] == 0, "zero-initialized denominators must read as 0, not NaN") + #expect(parsed.yccToRGBOffset == [0, 0.5, 0]) + #expect(parsed.rgbToLMS[0] == 17081.0 / 16384) + #expect(parsed.rgbToLMS[4] == 1) + } + + // MARK: GPU conversion + + @Test( + "GPU output matches an independent CPU reference", + .enabled(if: MTLCreateSystemDefaultDevice() != nil, "needs a Metal device"), + arguments: [DolbyVisionConverter.ChromaSiting.topLeft, .left] + ) + func gpuMatchesReference(siting: DolbyVisionConverter.ChromaSiting) throws { + // Odd on both axes: the last column and row of blocks are partial. + let width = 33, height = 17 + let chromaWidth = (width + 1) / 2, chromaHeight = (height + 1) / 2 + let luma = (0..<(width * height)).map { index -> Int in + let x = index % width, y = index / width + return 64 + (x * 29 + y * 13) % 900 + } + let chroma = (0..<(chromaWidth * chromaHeight)).map { index -> (Int, Int) in + let x = index % chromaWidth, y = index / chromaWidth + return (350 + (x * 37 + y * 11) % 320, 360 + (x * 23 + y * 41) % 300) + } + let source = try Self.makeSource(width: width, height: height, luma: luma, chroma: chroma) + + let mapping = Self.syntheticMapping + let converter = try DolbyVisionConverter() + let output = try converter.convert(source, mapping: mapping, chromaSiting: siting) + + #expect(CVPixelBufferGetPixelFormatType(output) == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange) + #expect(Self.attachment(output, kCVImageBufferYCbCrMatrixKey) == kCVImageBufferYCbCrMatrix_ITU_R_2020 as String) + #expect(Self.attachment(output, kCVImageBufferColorPrimariesKey) == kCVImageBufferColorPrimaries_ITU_R_2020 as String) + #expect(Self.attachment(output, kCVImageBufferTransferFunctionKey) == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String) + + let reference = Reference(mapping: mapping, siting: siting, width: width, height: height, luma: luma, chroma: chroma) + let (outLuma, outChroma) = Self.readCodes(output) + + var worstLuma = 0, worstChroma = 0 + for y in 0.. URL { + Fixtures.repoRoot.appendingPathComponent("TestStreams/DolbyVision/\(name)") + } + + private static func hasSample(_ name: String) -> Bool { + FileManager.default.fileExists(atPath: sample(name).path) + } + + @Test( + "profile 20 (IPT base layer) is converted to HDR10 with natural colour", + .enabled(if: hasSample("DVprofile20.mp4"), "local Dolby Vision sample missing"), + .enabled(if: MTLCreateSystemDefaultDevice() != nil, "needs a Metal device"), + .timeLimit(.minutes(2)), + arguments: [VideoDecoder.HardwarePolicy.videoToolbox, .software] + ) + func profile20(policy: VideoDecoder.HardwarePolicy) async throws { + let frames = try await decodeFirstFrames(Self.sample("DVprofile20.mp4"), count: 6, policy: policy) + #expect(frames.count == 6) + for frame in frames { + #expect(frame.converting, "the decoder must report the conversion engaged") + #expect(!frame.zeroCopy, "a converted frame is an engine-written surface") + #expect(frame.format == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange) + #expect(frame.matrix == kCVImageBufferYCbCrMatrix_ITU_R_2020 as String) + #expect(frame.transfer == kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ as String) + + // The opening shot is sea water. Read as YCbCr it is saturated + // green (G' far above B'); reshaped it is turquoise — blue and + // green close together, both well above red. + let (r, g, b) = frame.meanRGB + #expect(b > r + 0.05 && g > r + 0.05, "water must not turn pink: R'G'B' = \(r), \(g), \(b)") + #expect(abs(g - b) < 0.08, "water must not turn green: R'G'B' = \(r), \(g), \(b)") + } + } + + @Test( + "profile 8 (HDR10 base layer) is left alone", + .enabled(if: hasSample("DV8_TEST.mp4"), "local Dolby Vision sample missing"), + .timeLimit(.minutes(2)) + ) + func profile8Untouched() async throws { + let frames = try await decodeFirstFrames(Self.sample("DV8_TEST.mp4"), count: 6, policy: .videoToolbox) + #expect(frames.count == 6) + for frame in frames { + #expect(!frame.converting, "profile 8's base layer is HDR10 already") + #expect(frame.matrix == kCVImageBufferYCbCrMatrix_ITU_R_2020 as String) + } + } + + // MARK: Support + + private struct DecodedFrame: Sendable { + let format: OSType + let zeroCopy: Bool + let converting: Bool + let matrix: String? + let transfer: String? + /// Frame-average non-linear R'G'B', from a sparse grid of samples. + let meanRGB: (Double, Double, Double) + } + + /// Decodes the first `count` frames of a stream, then stops — the samples + /// are long 4K files and six frames say everything these tests ask. + private func decodeFirstFrames( + _ url: URL, + count: Int, + policy: VideoDecoder.HardwarePolicy + ) async throws -> [DecodedFrame] { + let demuxer = Demuxer(url: url.path) + defer { demuxer.shutdown() } + var demuxEvents = demuxer.events.makeAsyncIterator() + demuxer.start() + guard case .opened(let info)? = await demuxEvents.next() else { + throw EngineError(code: .openFailed, message: "\(url.lastPathComponent) failed to open") + } + + let track = try #require(info.videoTracks.first) + let parameters = try #require(demuxer.codecParameters(forStream: track.index)) + let packets = Channel(capacity: 64) + let frames = Channel(capacity: 4, measure: { $0.duration }) + demuxer.attach(channel: packets, toStream: track.index) + + let decoder = VideoDecoder(parameters: parameters, input: packets, output: frames, policy: policy) + decoder.start() + demuxer.resume() + + // Stopping the decoder from the drain closes the frame channel, which + // is what ends the drain (see `ChannelDrain`). + let collector = ChannelDrain(frames, into: [DecodedFrame]()) { collected, frame in + guard collected.count < count else { return } + collected.append(DecodedFrame( + format: CVPixelBufferGetPixelFormatType(frame.pixelBuffer), + zeroCopy: frame.isHardwareDecoded, + converting: decoder.isConvertingDolbyVision, + matrix: Self.attachment(frame.pixelBuffer, kCVImageBufferYCbCrMatrixKey), + transfer: Self.attachment(frame.pixelBuffer, kCVImageBufferTransferFunctionKey), + meanRGB: Self.meanRGB(frame.pixelBuffer) + )) + if collected.count == count { decoder.shutdown() } + } + let result = await collector.value + decoder.shutdown() + return result + } + + private static func attachment(_ buffer: CVPixelBuffer, _ key: CFString) -> String? { + CVBufferCopyAttachment(buffer, key, nil) as? String + } + + /// Average R'G'B' of a 10-bit video-range BT.2020 buffer, from every 16th + /// pixel on both axes — read in place, since copying a 4K plane per frame + /// is what a debug build is slowest at. Anything else reads as black, + /// which the colour assertions then reject. + private static func meanRGB(_ buffer: CVPixelBuffer) -> (Double, Double, Double) { + guard CVPixelBufferGetPixelFormatType(buffer) == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange else { + return (0, 0, 0) + } + CVPixelBufferLockBaseAddress(buffer, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) } + guard let lumaBase = CVPixelBufferGetBaseAddressOfPlane(buffer, 0), + let chromaBase = CVPixelBufferGetBaseAddressOfPlane(buffer, 1) + else { return (0, 0, 0) } + let lumaRow = CVPixelBufferGetBytesPerRowOfPlane(buffer, 0) + let chromaRow = CVPixelBufferGetBytesPerRowOfPlane(buffer, 1) + func code(_ base: UnsafeMutableRawPointer, _ rowBytes: Int, _ row: Int, _ index: Int) -> Double { + Double((base + row * rowBytes).assumingMemoryBound(to: UInt16.self)[index] >> 6) + } + + var sum = (0.0, 0.0, 0.0), samples = 0.0 + for y in stride(from: 0, to: CVPixelBufferGetHeight(buffer), by: 16) { + for x in stride(from: 0, to: CVPixelBufferGetWidth(buffer), by: 16) { + let luma = (code(lumaBase, lumaRow, y, x) - 64) / 876 + let cb = (code(chromaBase, chromaRow, y / 2, x / 2 * 2) - 512) / 896 + let cr = (code(chromaBase, chromaRow, y / 2, x / 2 * 2 + 1) - 512) / 896 + let r = luma + 1.4746 * cr, b = luma + 1.8814 * cb + let g = (luma - 0.2627 * r - 0.0593 * b) / 0.6780 + sum = (sum.0 + r, sum.1 + g, sum.2 + b) + samples += 1 + } + } + return (sum.0 / samples, sum.1 / samples, sum.2 / samples) + } + + /// 10-bit codes of a bi-planar MSB-aligned buffer, planes packed tightly. + private static func readCodes(_ buffer: CVPixelBuffer) -> ([Int], [(Int, Int)]) { + CVPixelBufferLockBaseAddress(buffer, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) } + func plane(_ index: Int, components: Int) -> [Int] { + guard let base = CVPixelBufferGetBaseAddressOfPlane(buffer, index) else { return [] } + let rowBytes = CVPixelBufferGetBytesPerRowOfPlane(buffer, index) + let width = CVPixelBufferGetWidthOfPlane(buffer, index) + var values: [Int] = [] + for row in 0..> 6)) } + } + return values + } + let chroma = plane(1, components: 2) + return (plane(0, components: 1), stride(from: 0, to: chroma.count, by: 2).map { (chroma[$0], chroma[$0 + 1]) }) + } + + /// A full-range 10-bit IPT base layer, as VideoToolbox hands it over. + private static func makeSource(width: Int, height: Int, luma: [Int], chroma: [(Int, Int)]) throws -> CVPixelBuffer { + var buffer: CVPixelBuffer? + let attributes: [CFString: Any] = [ + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary, + kCVPixelBufferMetalCompatibilityKey: true, + ] + let status = CVPixelBufferCreate( + nil, width, height, kCVPixelFormatType_420YpCbCr10BiPlanarFullRange, + attributes as CFDictionary, &buffer + ) + let source = try #require(status == kCVReturnSuccess ? buffer : nil) + CVPixelBufferLockBaseAddress(source, []) + defer { CVPixelBufferUnlockBaseAddress(source, []) } + let lumaBase = try #require(CVPixelBufferGetBaseAddressOfPlane(source, 0)) + let lumaRow = CVPixelBufferGetBytesPerRowOfPlane(source, 0) + for y in 0.. Int { Int((64 + 876 * y).rounded()) } + static func chromaCode(_ c: Double) -> Int { Int((512 + 896 * min(max(c, -0.5), 0.5)).rounded()) } + + /// Output Y'CbCr of one pixel. + func pixel(_ x: Int, _ y: Int) -> (y: Double, cb: Double, cr: Double) { + let (p, t) = sampleChroma(x, y) + return convert([Double(luma[y * width + x]) / 1023, p, t]) + } + + /// Output chroma codes of one block: the average over its pixels. + func block(_ i: Int, _ j: Int) -> (Int, Int) { + var cb = 0.0, cr = 0.0, count = 0.0 + for y in (2 * j).. (Double, Double) { + let u = (Double(x) - Double(siting.horizontal)) / 2 + let v = (Double(y) - Double(siting.vertical)) / 2 + let i0 = Int(u.rounded(.down)), j0 = Int(v.rounded(.down)) + let fu = u - Double(i0), fv = v - Double(j0) + func at(_ i: Int, _ j: Int) -> (Double, Double) { + let value = chroma[min(max(j, 0), chromaHeight - 1) * chromaWidth + min(max(i, 0), chromaWidth - 1)] + return (Double(value.0) / 1023, Double(value.1) / 1023) + } + let a = at(i0, j0), b = at(i0 + 1, j0), c = at(i0, j0 + 1), d = at(i0 + 1, j0 + 1) + func mix(_ a: Double, _ b: Double, _ c: Double, _ d: Double) -> Double { + (a * (1 - fu) + b * fu) * (1 - fv) + (c * (1 - fu) + d * fu) * fv + } + return (mix(a.0, b.0, c.0, d.0), mix(a.1, b.1, c.1, d.1)) + } + + private func reshape(_ curve: DolbyVisionMapping.Curve, _ sig: [Double], _ component: Int) -> Double { + let s = min(max(sig[component], curve.pivots.first ?? 0), curve.pivots.last ?? 1) + let index = (1..= curve.pivots[$0] } ?? 0 + let out: Double + switch curve.pieces[index] { + case .polynomial(let c): + out = c[0] + c[1] * s + c[2] * s * s + case .mmr(let constant, let rows): + let (y, u, v) = (sig[0], sig[1], sig[2]) + let terms = [y, u, v, y * u, y * v, u * v, y * u * v] + out = rows.enumerated().reduce(constant) { total, row in + let order = Double(row.offset + 1) + return total + zip(row.element, terms).reduce(0) { $0 + $1.0 * pow($1.1, order) } + } + } + return min(max(out, 0), 1) + } + + private func convert(_ sig: [Double]) -> (y: Double, cb: Double, cr: Double) { + let reshaped = (0..<3).map { reshape(mapping.curves[$0], sig, $0) } + let centred = zip(reshaped, mapping.yccToRGBOffset).map { $0 - $1 } + let lmsPQ = Self.apply(mapping.yccToRGB, centred) + let lms = lmsPQ.map(Self.pqEOTF) + let lmsToRGB = Self.multiply(Self.publishedHPEToBT2020, mapping.rgbToLMS) + let rgb = Self.apply(lmsToRGB, lms).map { Self.pqOETF(max($0, 0)) } + let y = 0.2627 * rgb[0] + 0.6780 * rgb[1] + 0.0593 * rgb[2] + return (y, (rgb[2] - y) / 1.8814, (rgb[0] - y) / 1.4746) + } + + private static let publishedHPEToBT2020 = [ + 3.06441879, -2.16597676, 0.10155818, + -0.65612108, 1.78554118, -0.12943749, + 0.01736321, -0.04725154, 1.03004253, + ] + + private static func apply(_ m: [Double], _ v: [Double]) -> [Double] { + (0..<3).map { row in (0..<3).reduce(0) { $0 + m[row * 3 + $1] * v[$1] } } + } + + private static func multiply(_ a: [Double], _ b: [Double]) -> [Double] { + (0..<9).map { index in (0..<3).reduce(0) { $0 + a[index / 3 * 3 + $1] * b[$1 * 3 + index % 3] } } + } + + private static let m1 = 2610.0 / 16384, m2 = 2523.0 / 4096 * 128 + private static let c1 = 3424.0 / 4096, c2 = 2413.0 / 4096 * 32, c3 = 2392.0 / 4096 * 32 + + static func pqEOTF(_ e: Double) -> Double { + let p = pow(max(e, 0), 1 / m2) + return pow(max(p - c1, 0) / (c2 - c3 * p), 1 / m1) + } + + static func pqOETF(_ y: Double) -> Double { + let p = pow(min(max(y, 0), 1), m1) + return pow((c1 + c2 * p) / (1 + c3 * p), m2) + } +} From b267e7a30b53ee86ae59eb115592652ddc01421e Mon Sep 17 00:00:00 2001 From: Philipp Bischoff Date: Tue, 29 Sep 2026 18:08:18 +0200 Subject: [PATCH 2/2] fix(decode): keep MV-HEVC's second view away from VideoToolbox MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DVprofile20.mp4 stuttered badly on Apple TV. It was never on the hardware decoder. The stream is MV-HEVC, and FFmpeg's HEVC decoder files the second view's SPS/PPS (nuh_layer_id 1) alongside the base layer's; its VideoToolbox glue then packs every parameter set into one hvcC, where Apple expects the second layer's in a separate lhvC. VTDecompressionSessionCreate fails ("VideoToolbox malfunction") and FFmpeg falls back to software without returning an error — 4K 10-bit HEVC on the CPU, ~20 fps on an M1 Pro. Stripping layer 1 from a 5 s extract took ffmpeg's hwaccel run from 8.5 s to 0.15 s of CPU time. HEVCBaseLayerFilter drops every nuh_layer_id > 0 NAL unit before the codec sees it — from packets (length-prefixed or Annex B) and from the extradata, into which FFmpeg's MP4 demuxer merges lhvC. The engine only presents the base view, so nothing visible is lost; packets without enhancement data pass through uncopied. Pipeline throughput on the profile 20 sample: 19.6 → ~101 fps, on VideoToolbox. What hid it: isHardwareActive reported "a VideoToolbox device is attached", not where frames came from. It now reads the frames' pixel format, and a silent fallback emits .downgradedToSoftware like any other downgrade. That also surfaces an existing one: VideoToolbox refuses the field-coded H.264 fixture here, so interlaced.ts was always decoded in software — the deinterlace tests now wait past that informational event instead of expecting .endOfStream first. Verified on an Apple TV 4K (A15): smooth. --- AGENTS.md | 1 + .../Decode/HEVCBaseLayerFilter.swift | 182 ++++++++++++++++++ .../LumeEngineCore/Decode/VideoDecoder.swift | 112 ++++++++++- .../DeinterlaceTests.swift | 10 +- .../DolbyVisionTests.swift | 11 ++ .../HEVCBaseLayerFilterTests.swift | 111 +++++++++++ 6 files changed, 419 insertions(+), 8 deletions(-) create mode 100644 Sources/LumeEngineCore/Decode/HEVCBaseLayerFilter.swift create mode 100644 Tests/LumeEngineCoreTests/HEVCBaseLayerFilterTests.swift diff --git a/AGENTS.md b/AGENTS.md index 76a71a5..9883ad1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,7 @@ This split is the core invariant of the codebase: - **RAII FFI wrappers:** `FFI/Packet`, `FFI/Frame` etc. wrap `AVPacket*`/`AVFrame*` with deinit-based free. `@unchecked Sendable` is allowed **only** on these audited immutable-after-construction wrappers and on `Channel` — never on state-bearing types. No `Unmanaged` outside FFI. - **A/V sync is Apple's job:** decoded frames are enqueued into `AVSampleBufferRenderSynchronizer` (`Render/SystemRenderer`). Do not add drop-frame heuristics or hand-rolled clocks. - **Deinterlacing never costs the hardware decoder.** `Decode/VideoFilterGraph` (bwdif, falling back to yadif) sits between `avcodec_receive_frame` and delivery; VideoToolbox frames are downloaded with `av_hwframe_transfer_data` and filtered on the CPU. Do *not* "simplify" this into switching the codec to software when a stream is interlaced: that costs a mid-stream codec rebuild and keyframe resync at every interlaced/progressive splice, which live broadcasts (ad breaks in a 1080i match) are full of. Engagement is decided per frame from `AV_FRAME_FLAG_INTERLACED` (PLAN.md §3, failure 6) and, once engaged, everything stays on one path — `deint=interlaced` passes progressive frames through. Filter output carries the sink's time base, which a field-doubling filter *halves*; PTS and durations are rescaled back to engine µs in `VideoDecoder`. +- **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again. - **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session. - **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs. - **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes. diff --git a/Sources/LumeEngineCore/Decode/HEVCBaseLayerFilter.swift b/Sources/LumeEngineCore/Decode/HEVCBaseLayerFilter.swift new file mode 100644 index 0000000..53b8794 --- /dev/null +++ b/Sources/LumeEngineCore/Decode/HEVCBaseLayerFilter.swift @@ -0,0 +1,182 @@ +/// Reduces a multi-layer HEVC stream (MV-HEVC stereo — Dolby Vision profile +/// 20, Apple spatial video — as well as SHVC and alpha layers) to its base +/// layer by dropping every NAL unit with `nuh_layer_id > 0`. +/// +/// The engine only ever presents the base view, so nothing is lost. What is +/// gained is hardware decoding: FFmpeg's decoder files the enhancement layer's +/// SPS/PPS alongside the base layer's, and its VideoToolbox glue hands all of +/// them to VideoToolbox in one `hvcC` — where Apple expects the second layer's +/// in a separate `lhvC`. VideoToolbox rejects that session outright and FFmpeg +/// falls back to software decoding without an error, which for 4K 10-bit HEVC +/// is too slow to play on an Apple TV. Stripped, the same stream decodes on +/// VideoToolbox. (FFmpeg's MP4 demuxer even merges `lhvC` into the extradata, +/// so both the extradata and the in-band parameter sets need filtering.) +/// +/// Pure byte work, no FFmpeg types: the decoder applies it to packets and to +/// its codec context's extradata. Malformed input is passed through untouched +/// — the decoder is better at judging broken bitstreams than this filter. +struct HEVCBaseLayerFilter: Sendable, Equatable { + enum Framing: Sendable, Equatable { + /// Each NAL unit preceded by its big-endian length in this many bytes + /// (MP4/MKV, signalled by an `hvcC` extradata). + case lengthPrefixed(Int) + /// Start-code delimited (MPEG-TS, raw `.hevc`). + case annexB + } + + let framing: Framing + + /// Picks the framing FFmpeg's decoder will assume, the same way it does: an + /// `hvcC` (configuration version 1) means length-prefixed packets. + init(extradata: UnsafeRawBufferPointer) { + if extradata.count >= 23, extradata[0] == 1 { + framing = .lengthPrefixed(Int(extradata[21] & 0x3) + 1) + } else { + framing = .annexB + } + } + + init(framing: Framing) { + self.framing = framing + } + + /// `nuh_layer_id` from a NAL unit's two header bytes. + static func layerID(_ first: UInt8, _ second: UInt8) -> Int { + Int(first & 0x1) << 5 | Int(second >> 3) + } + + // MARK: Packets + + /// Byte ranges of `data` that belong to the base layer, in order. Nil when + /// nothing needs dropping — by far the common case, which then costs no + /// copy. An empty result means the packet held no base-layer data at all. + func baseLayerRanges(of data: UnsafeRawBufferPointer) -> [Range]? { + guard let units = nalUnits(of: data) else { return nil } + var kept: [Range] = [] + var dropped = false + for unit in units { + if let layer = unit.layer, layer > 0 { + dropped = true + } else if let last = kept.last, last.upperBound == unit.range.lowerBound { + kept[kept.count - 1] = last.lowerBound.. + /// Nil when the unit is too short to carry a header; such units are + /// kept. + let layer: Int? + } + + private func nalUnits(of data: UnsafeRawBufferPointer) -> [Unit]? { + switch framing { + case .lengthPrefixed(let size): + return lengthPrefixedUnits(of: data, lengthSize: size) + case .annexB: + return annexBUnits(of: data) + } + } + + private func lengthPrefixedUnits(of data: UnsafeRawBufferPointer, lengthSize: Int) -> [Unit]? { + var units: [Unit] = [] + var offset = 0 + while offset < data.count { + guard offset + lengthSize <= data.count else { return nil } + var length = 0 + for index in 0..= 2 ? Self.layerID(data[payload], data[payload + 1]) : nil + units.append(Unit(range: offset..<(payload + length), layer: layer)) + offset = payload + length + } + return units + } + + private func annexBUnits(of data: UnsafeRawBufferPointer) -> [Unit]? { + // Offsets of every 00 00 01; a four-byte start code's leading zero + // stays with the unit before it, which is harmless either way. + var starts: [Int] = [] + var index = 0 + while index + 2 < data.count { + if data[index + 2] > 1 { + index += 3 + } else if data[index] == 0, data[index + 1] == 0, data[index + 2] == 1 { + starts.append(index) + index += 3 + } else { + index += 1 + } + } + guard let first = starts.first else { return nil } + + var units: [Unit] = [] + if first > 0 { units.append(Unit(range: 0.. [UInt8]? { + switch framing { + case .annexB: + guard let ranges = baseLayerRanges(of: extradata) else { return nil } + return ranges.flatMap { Array(extradata[$0]) } + case .lengthPrefixed: + return Self.baseLayerHVCC(extradata) + } + } + + /// Rewrites an `hvcC` record (ISO/IEC 14496-15 §8.3.3), dropping + /// enhancement-layer NAL units and any array they leave empty. + static func baseLayerHVCC(_ hvcC: UnsafeRawBufferPointer) -> [UInt8]? { + guard hvcC.count >= 23 else { return nil } + var arrays: [[UInt8]] = [] + var dropped = false + var offset = 23 + for _ in 0..= 2, layerID(hvcC[offset + 2], hvcC[offset + 3]) > 0 { + dropped = true + } else { + nalUnits.append(Array(hvcC[offset..> 8), UInt8(nalUnits.count & 0xff)] + nalUnits.joined()) + } + guard dropped else { return nil } + + var result = Array(hvcC[0..<22]) + result.append(UInt8(arrays.count)) + for array in arrays { result += array } + // Anything after the arrays is not ours to interpret; keep it. + if offset < hvcC.count { result += hvcC[offset...] } + return result + } +} diff --git a/Sources/LumeEngineCore/Decode/VideoDecoder.swift b/Sources/LumeEngineCore/Decode/VideoDecoder.swift index 69873d1..66ced85 100644 --- a/Sources/LumeEngineCore/Decode/VideoDecoder.swift +++ b/Sources/LumeEngineCore/Decode/VideoDecoder.swift @@ -4,8 +4,10 @@ import Foundation /// Events emitted by decoders on their own threads. public enum DecodeEvent: Sendable { - /// Hardware decoding failed mid-stream; the decoder rebuilt itself in - /// software and continues from the next keyframe. Informational. + /// Frames now come from software decoding: either hardware decoding failed + /// mid-stream and the decoder rebuilt itself in software (continuing from + /// the next keyframe), or VideoToolbox refused the stream and FFmpeg fell + /// back on its own. Informational. case downgradedToSoftware(EngineError) /// The codec has been fully drained for the given serial. case endOfStream(serial: UInt64) @@ -84,9 +86,20 @@ public final class VideoDecoder: @unchecked Sendable { private var drainRequested = false private var deinterlaceActive = false private var dolbyVisionActive = false + /// Whether the frames actually coming out are VideoToolbox surfaces; nil + /// until the current codec has produced one. Not the same as + /// `usingHardware`: FFmpeg falls back to software on its own, without an + /// error, when VideoToolbox refuses a stream. + private var framesFromHardware: Bool? // Decode-thread-only state. private var codecContext: UnsafeMutablePointer? + /// HEVC only: strips enhancement layers (MV-HEVC's second view) before + /// the codec sees them — see `HEVCBaseLayerFilter` for why that decides + /// between hardware and software decoding. + private let baseLayerFilter: HEVCBaseLayerFilter? + /// Scratch packet for the base-layer copy of a multi-layer packet. + private var baseLayerPacket: UnsafeMutablePointer? private var usingHardware = false private var currentSerial: UInt64? private var waitingForKeyframe = false @@ -141,6 +154,12 @@ public final class VideoDecoder: @unchecked Sendable { self.policy = policy self.deinterlacing = deinterlacing self.dolbyVisionBaseLayerIsIPT = DolbyVisionMapping.baseLayerIsIPT(parameters.raw) + self.baseLayerFilter = parameters.raw.pointee.codec_id == AV_CODEC_ID_HEVC + ? HEVCBaseLayerFilter(extradata: UnsafeRawBufferPointer( + start: parameters.raw.pointee.extradata, + count: Int(max(parameters.raw.pointee.extradata_size, 0)) + )) + : nil var continuation: AsyncStream.Continuation! events = AsyncStream(bufferingPolicy: .unbounded) { continuation = $0 } eventSink = continuation @@ -150,11 +169,13 @@ public final class VideoDecoder: @unchecked Sendable { eventSink.finish() } - /// True when frames are currently produced by VideoToolbox. + /// True when frames are currently produced by VideoToolbox — judged from + /// the frames themselves once there are any, so a silent FFmpeg fallback + /// to software reads as software. public var isHardwareActive: Bool { lock.lock() defer { lock.unlock() } - return usingHardware + return framesFromHardware ?? usingHardware } /// True while frames are being routed through the deinterlacer — i.e. the @@ -261,6 +282,9 @@ public final class VideoDecoder: @unchecked Sendable { av_frame_free(&filteredFrame) } + if baseLayerFilter != nil { baseLayerPacket = av_packet_alloc() } + defer { av_packet_free(&baseLayerPacket) } + while true { lock.lock() let stop = stopRequested @@ -306,16 +330,18 @@ public final class VideoDecoder: @unchecked Sendable { } private func decode(packet: Packet, into frame: UnsafeMutablePointer) { - guard let context = codecContext else { return } + guard let context = codecContext, + let input = baseLayer(of: packet) + else { return } - var sendResult = avcodec_send_packet(context, packet.raw) + var sendResult = avcodec_send_packet(context, input) if lume_is_eagain(sendResult) != 0 { // Decoder is full: pull frames, then retry once. Draining can // rebuild the codec underneath us (a delivery failure downgrades to // software), so the context is re-read rather than reused. receiveFrames(into: frame) guard let retryContext = codecContext else { return } - sendResult = avcodec_send_packet(retryContext, packet.raw) + sendResult = avcodec_send_packet(retryContext, input) } if sendResult < 0 && lume_is_eagain(sendResult) == 0 && lume_is_eof(sendResult) == 0 { handleDecodeError(sendResult) @@ -336,11 +362,81 @@ public final class VideoDecoder: @unchecked Sendable { return } consecutiveErrors = 0 + noteFrameSource(frame) emit(frame: frame) av_frame_unref(frame) } } + /// The packet the codec should see: the demuxer's own, or a base-layer copy + /// in the scratch packet when it carries enhancement layers. Nil when + /// nothing of the base layer is left — never send an empty packet, which + /// FFmpeg reads as end of stream. + private func baseLayer(of packet: Packet) -> UnsafeMutablePointer? { + guard let filter = baseLayerFilter, + let scratch = baseLayerPacket, + let data = packet.raw.pointee.data, + packet.raw.pointee.size > 0 + else { return packet.raw } + + let source = UnsafeRawBufferPointer(start: data, count: Int(packet.raw.pointee.size)) + guard let ranges = filter.baseLayerRanges(of: source) else { return packet.raw } + let size = ranges.reduce(0) { $0 + $1.count } + guard size > 0 else { return nil } + + av_packet_unref(scratch) + guard av_new_packet(scratch, Int32(size)) >= 0, + av_packet_copy_props(scratch, packet.raw) >= 0, + let destination = scratch.pointee.data + else { + // Out of memory: the full packet still decodes, just in software. + return packet.raw + } + var offset = 0 + for range in ranges { + (destination + offset).update(from: data + range.lowerBound, count: range.count) + offset += range.count + } + return scratch + } + + /// Replaces the context's extradata with its base-layer version (FFmpeg's + /// MP4 demuxer folds `lhvC` into it). Called before `avcodec_open2`. + private func stripEnhancementLayers(fromExtradataOf context: UnsafeMutablePointer) { + guard let filter = baseLayerFilter, + let extradata = context.pointee.extradata, + context.pointee.extradata_size > 0, + let stripped = filter.baseLayerExtradata(UnsafeRawBufferPointer( + start: extradata, count: Int(context.pointee.extradata_size) + )), + let replacement = av_mallocz(stripped.count + Int(AV_INPUT_BUFFER_PADDING_SIZE)) + else { return } + replacement.copyMemory(from: stripped, byteCount: stripped.count) + av_freep(&context.pointee.extradata) + context.pointee.extradata = replacement.assumingMemoryBound(to: UInt8.self) + context.pointee.extradata_size = Int32(stripped.count) + } + + /// Tracks where frames really come from. A hardware context whose frames + /// arrive in software means FFmpeg gave up on VideoToolbox by itself — + /// reported like any other downgrade, because silently decoding 4K in + /// software is exactly what makes a stream stutter. + private func noteFrameSource(_ frame: UnsafeMutablePointer) { + let hardware = frame.pointee.format == AV_PIX_FMT_VIDEOTOOLBOX.rawValue + lock.lock() + let previous = framesFromHardware + framesFromHardware = hardware + let attached = usingHardware + lock.unlock() + + if attached, !hardware, previous != false { + eventSink.yield(.downgradedToSoftware(EngineError( + code: .decoderInitFailed, + message: "VideoToolbox refused \(parameters.codecName); FFmpeg is decoding in software" + ))) + } + } + // MARK: Deinterlace (decode thread only) /// Routes one decoded frame to the renderer, through the deinterlacer when @@ -618,6 +714,7 @@ public final class VideoDecoder: @unchecked Sendable { avcodec_free_context(&pointer) return false } + stripEnhancementLayers(fromExtradataOf: context) // Demux boundary rewrote packet timestamps to engine µs. context.pointee.pkt_timebase = lume_av_time_base_q() @@ -646,6 +743,7 @@ public final class VideoDecoder: @unchecked Sendable { codecContext = context lock.lock() usingHardware = hardware + framesFromHardware = nil lock.unlock() return true } diff --git a/Tests/LumeEngineCoreTests/DeinterlaceTests.swift b/Tests/LumeEngineCoreTests/DeinterlaceTests.swift index b1421a4..99ee44c 100644 --- a/Tests/LumeEngineCoreTests/DeinterlaceTests.swift +++ b/Tests/LumeEngineCoreTests/DeinterlaceTests.swift @@ -74,7 +74,15 @@ struct DeinterlaceTests { break } } - guard case .endOfStream? = await decodeEvents.next() else { + // `.downgradedToSoftware` is informational, and expected here: + // VideoToolbox refuses field-coded H.264, so FFmpeg decodes this + // fixture in software and the decoder says so. + var reachedEndOfStream = false + while let event = await decodeEvents.next() { + if case .endOfStream = event { reachedEndOfStream = true; break } + if case .failed = event { break } + } + guard reachedEndOfStream else { decoder.shutdown() throw EngineError(code: .decodeFailed, message: "decoder never reached EOF") } diff --git a/Tests/LumeEngineCoreTests/DolbyVisionTests.swift b/Tests/LumeEngineCoreTests/DolbyVisionTests.swift index 5a2d2f4..ee96d30 100644 --- a/Tests/LumeEngineCoreTests/DolbyVisionTests.swift +++ b/Tests/LumeEngineCoreTests/DolbyVisionTests.swift @@ -179,6 +179,13 @@ struct DolbyVisionTests { let frames = try await decodeFirstFrames(Self.sample("DVprofile20.mp4"), count: 6, policy: policy) #expect(frames.count == 6) for frame in frames { + // Profile 20 is MV-HEVC. Unless the second view is stripped before + // the codec, VideoToolbox refuses the stream and FFmpeg decodes 4K + // in software without saying so — the "super laggy" Apple TV bug. + #expect( + frame.hardwareDecode == (policy == .videoToolbox), + "frames must come from \(policy == .videoToolbox ? "VideoToolbox" : "software")" + ) #expect(frame.converting, "the decoder must report the conversion engaged") #expect(!frame.zeroCopy, "a converted frame is an engine-written surface") #expect(frame.format == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange) @@ -204,6 +211,7 @@ struct DolbyVisionTests { #expect(frames.count == 6) for frame in frames { #expect(!frame.converting, "profile 8's base layer is HDR10 already") + #expect(frame.hardwareDecode) #expect(frame.matrix == kCVImageBufferYCbCrMatrix_ITU_R_2020 as String) } } @@ -214,6 +222,8 @@ struct DolbyVisionTests { let format: OSType let zeroCopy: Bool let converting: Bool + /// The decoder's own report, which reads the frames themselves. + let hardwareDecode: Bool let matrix: String? let transfer: String? /// Frame-average non-linear R'G'B', from a sparse grid of samples. @@ -253,6 +263,7 @@ struct DolbyVisionTests { format: CVPixelBufferGetPixelFormatType(frame.pixelBuffer), zeroCopy: frame.isHardwareDecoded, converting: decoder.isConvertingDolbyVision, + hardwareDecode: decoder.isHardwareActive, matrix: Self.attachment(frame.pixelBuffer, kCVImageBufferYCbCrMatrixKey), transfer: Self.attachment(frame.pixelBuffer, kCVImageBufferTransferFunctionKey), meanRGB: Self.meanRGB(frame.pixelBuffer) diff --git a/Tests/LumeEngineCoreTests/HEVCBaseLayerFilterTests.swift b/Tests/LumeEngineCoreTests/HEVCBaseLayerFilterTests.swift new file mode 100644 index 0000000..a962666 --- /dev/null +++ b/Tests/LumeEngineCoreTests/HEVCBaseLayerFilterTests.swift @@ -0,0 +1,111 @@ +import Testing +@testable import LumeEngineCore + +/// Enhancement layers (MV-HEVC's second view) have to be gone before the codec +/// sees them, or VideoToolbox refuses the stream and FFmpeg silently decodes +/// in software — see `HEVCBaseLayerFilter`. +@Suite("HEVC base layer filter") +struct HEVCBaseLayerFilterTests { + /// A NAL unit: two-byte header for `type`/`layer`, then `payload`. + private static func nal(type: UInt8, layer: UInt8, payload: [UInt8] = [0xAB, 0xCD]) -> [UInt8] { + [type << 1 | layer >> 5, (layer & 0x1F) << 3 | 1] + payload + } + + private static func lengthPrefixed(_ units: [[UInt8]]) -> [UInt8] { + units.flatMap { unit in + let length = unit.count + return [UInt8(length >> 24), UInt8(length >> 16 & 0xFF), UInt8(length >> 8 & 0xFF), UInt8(length & 0xFF)] + unit + } + } + + private static func apply(_ filter: HEVCBaseLayerFilter, _ bytes: [UInt8]) -> [UInt8]? { + bytes.withUnsafeBytes { buffer in + filter.baseLayerRanges(of: buffer).map { ranges in ranges.flatMap { Array(buffer[$0]) } } + } + } + + @Test("header layer ID is the 6 bits spanning both bytes") + func layerID() { + for layer in [0, 1, 31, 32, 63] as [UInt8] { + let unit = Self.nal(type: 1, layer: layer) + #expect(HEVCBaseLayerFilter.layerID(unit[0], unit[1]) == Int(layer)) + } + } + + @Test("length-prefixed: only layer-0 units survive, in order") + func lengthPrefixedPacket() { + let base = [Self.nal(type: 35, layer: 0), Self.nal(type: 19, layer: 0, payload: [1, 2, 3]), Self.nal(type: 62, layer: 0)] + let mixed = [base[0], Self.nal(type: 33, layer: 1), base[1], Self.nal(type: 19, layer: 1), base[2]] + let filter = HEVCBaseLayerFilter(framing: .lengthPrefixed(4)) + + #expect(Self.apply(filter, Self.lengthPrefixed(mixed)) == Self.lengthPrefixed(base)) + #expect(Self.apply(filter, Self.lengthPrefixed(base)) == nil, "nothing to drop must mean no copy") + } + + @Test("length-prefixed: a packet of enhancement data only leaves nothing") + func onlyEnhancement() { + let filter = HEVCBaseLayerFilter(framing: .lengthPrefixed(4)) + #expect(Self.apply(filter, Self.lengthPrefixed([Self.nal(type: 1, layer: 1)])) == []) + } + + @Test("length-prefixed: a truncated packet passes through untouched") + func truncated() { + var bytes = Self.lengthPrefixed([Self.nal(type: 1, layer: 0), Self.nal(type: 1, layer: 1)]) + bytes.removeLast() + #expect(Self.apply(HEVCBaseLayerFilter(framing: .lengthPrefixed(4)), bytes) == nil) + } + + @Test("Annex B: three- and four-byte start codes, leading junk kept") + func annexB() { + let junk: [UInt8] = [0x00] + let keepA = [0, 0, 0, 1] + Self.nal(type: 32, layer: 0) + let dropB = [0, 0, 1] + Self.nal(type: 33, layer: 1, payload: [9, 9, 9]) + let keepC = [0, 0, 1] + Self.nal(type: 1, layer: 0, payload: [0x00, 0x00, 0x03, 0x01]) + let filter = HEVCBaseLayerFilter(framing: .annexB) + + #expect(Self.apply(filter, junk + keepA + dropB + keepC) == junk + keepA + keepC) + #expect(Self.apply(filter, keepA + keepC) == nil) + #expect(Self.apply(filter, [1, 2, 3, 4]) == nil, "no start code: not ours to judge") + } + + @Test("framing follows the extradata the way FFmpeg's decoder decides") + func framing() { + var hvcC = [UInt8](repeating: 0, count: 23) + hvcC[0] = 1 + hvcC[21] = 0xFC | 3 + hvcC.withUnsafeBytes { #expect(HEVCBaseLayerFilter(extradata: $0).framing == .lengthPrefixed(4)) } + hvcC[21] = 0xFC | 1 + hvcC.withUnsafeBytes { #expect(HEVCBaseLayerFilter(extradata: $0).framing == .lengthPrefixed(2)) } + let annexB: [UInt8] = [0, 0, 0, 1] + Self.nal(type: 32, layer: 0) + annexB.withUnsafeBytes { #expect(HEVCBaseLayerFilter(extradata: $0).framing == .annexB) } + [UInt8]().withUnsafeBytes { #expect(HEVCBaseLayerFilter(extradata: $0).framing == .annexB) } + } + + /// An hvcC shaped like DVprofile20.mp4's once FFmpeg's MP4 demuxer has + /// folded its lhvC in: base VPS/SPS/PPS/SEI, then the second view's SPS + /// and PPS as arrays of their own. + @Test("hvcC: enhancement parameter sets and the arrays they empty are removed") + func hvcC() { + func array(_ type: UInt8, _ units: [[UInt8]]) -> [UInt8] { + [0x80 | type, UInt8(units.count >> 8), UInt8(units.count & 0xFF)] + + units.flatMap { [UInt8($0.count >> 8), UInt8($0.count & 0xFF)] + $0 } + } + let header = [UInt8](repeating: 0xEE, count: 22) + let vps = Self.nal(type: 32, layer: 0), sps = Self.nal(type: 33, layer: 0) + let pps = Self.nal(type: 34, layer: 0), sei = Self.nal(type: 39, layer: 0) + let sps1 = Self.nal(type: 33, layer: 1), pps1 = Self.nal(type: 34, layer: 1) + + let merged = header + [6] + + array(32, [vps]) + array(33, [sps]) + array(34, [pps]) + array(39, [sei]) + + array(33, [sps1]) + array(34, [pps1]) + let expected = header + [4] + array(32, [vps]) + array(33, [sps]) + array(34, [pps]) + array(39, [sei]) + merged.withUnsafeBytes { #expect(HEVCBaseLayerFilter.baseLayerHVCC($0) == expected) } + + // Mixed within one array: the array stays, with a corrected count. + let shared = header + [1] + array(33, [sps, sps1]) + shared.withUnsafeBytes { #expect(HEVCBaseLayerFilter.baseLayerHVCC($0) == header + [1] + array(33, [sps])) } + + expected.withUnsafeBytes { #expect(HEVCBaseLayerFilter.baseLayerHVCC($0) == nil, "single-layer hvcC stays as is") } + Array(merged.dropLast()).withUnsafeBytes { #expect(HEVCBaseLayerFilter.baseLayerHVCC($0) == nil, "truncated: untouched") } + } +}