Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,13 @@ This split is the core invariant of the codebase:
- **RAII FFI wrappers:** `FFI/Packet`, `FFI/Frame` etc. wrap `AVPacket*`/`AVFrame*` with deinit-based free. `@unchecked Sendable` is allowed **only** on these audited immutable-after-construction wrappers and on `Channel` — never on state-bearing types. No `Unmanaged` outside FFI.
- **A/V sync is Apple's job:** decoded frames are enqueued into `AVSampleBufferRenderSynchronizer` (`Render/SystemRenderer`). Do not add drop-frame heuristics or hand-rolled clocks.
- **Deinterlacing never costs the hardware decoder.** `Decode/VideoFilterGraph` (bwdif, falling back to yadif) sits between `avcodec_receive_frame` and delivery; VideoToolbox frames are downloaded with `av_hwframe_transfer_data` and filtered on the CPU. Do *not* "simplify" this into switching the codec to software when a stream is interlaced: that costs a mid-stream codec rebuild and keyframe resync at every interlaced/progressive splice, which live broadcasts (ad breaks in a 1080i match) are full of. Engagement is decided per frame from `AV_FRAME_FLAG_INTERLACED` (PLAN.md §3, failure 6) and, once engaged, everything stays on one path — `deint=interlaced` passes progressive frames through. Filter output carries the sink's time base, which a field-doubling filter *halves*; PTS and durations are rescaled back to engine µs in `VideoDecoder`.
- **Multi-layer HEVC reaches the codec as its base layer only.** `Decode/HEVCBaseLayerFilter` drops every `nuh_layer_id > 0` NAL unit from packets and extradata before `avcodec_send_packet`/`avcodec_open2`. FFmpeg files an MV-HEVC stream's second-view SPS/PPS (DV profile 20, spatial video) alongside the base layer's, its VideoToolbox glue packs all of them into one `hvcC`, VideoToolbox rejects that session, and FFmpeg falls back to software *without an error* — 4K HEVC in software, which stutters on Apple TV. The engine presents only the base view, so stripping loses nothing. Relatedly, `VideoDecoder.isHardwareActive` is judged from the frames' pixel format, not from whether a VT device was attached, and such a silent fallback emits `.downgradedToSoftware` — keep it that way, or this class of bug goes invisible again.
- **Dolby Vision with an IPT base layer is converted, not tagged.** Profiles 5, 10.0 and 20 (compatibility ID 0) carry IPTPQc2, which shows pink/green as YCbCr. No colour attachment fixes that — the RPU reshaping changes per scene and Apple's decoders don't apply it to decoded frames — so `Decode/DolbyVisionConverter` turns each frame into HDR10 on the GPU in `VideoDecoder.deliver`, the one point every frame (hardware, software, deinterlaced) passes with its side data. Detection is the stream's DV configuration record *or* an `AVCOL_SPC_IPT_C2` frame; many P5 encodes leave the VUI matrix unspecified, so don't reduce it to the tag. HDR10-compatible profiles (8.1 etc.) must stay on the zero-copy path. GPU refusals are per-frame (iOS refuses a backgrounded app); only a converter that can't exist gives up for the session.
- **Zero global mutable state:** all configuration is `PlayerConfiguration` passed at session creation. No `static var` knobs.
- **No force unwraps / `as!` / `fatalError` in engine paths**; failures return typed `EngineError`. No `assertionFailure` in production channel/data paths — misuse degrades, never crashes.
- **Never scrape FFmpeg log strings for control flow** (`av_log` is diagnostics only). Silence is never a failure mode: pipelines are supervised, stalls are detected from playhead ground truth, and failures surface as typed `PlayerEvent`s on the session's `AsyncStream`.
- Swift 6 strict concurrency (`swiftLanguageModes: [.v6]`) — keep it honest, don't suppress diagnostics.

### Tests

`Tests/LumeEngineCoreTests/` covers the historically crash-prone areas by design: timestamp wraparound (against a real wrap-seam fixture `wrap.ts`), channel backpressure/teardown, fixture-based decode, deinterlacing (against a field-coded `interlaced.ts`), playback, and resilience via `Support/TortureHTTPServer.swift` (a fault-injecting local HTTP server — mid-stream errors, stalls, disconnects). New work in demux/decode/time/concurrency should come with tests here; add new media fixtures to `generate-fixtures.sh`, not as committed binaries.
`Tests/LumeEngineCoreTests/` covers the historically crash-prone areas by design: timestamp wraparound (against a real wrap-seam fixture `wrap.ts`), channel backpressure/teardown, fixture-based decode, deinterlacing (against a field-coded `interlaced.ts`), Dolby Vision conversion (GPU output against an independent CPU reference; the real-stream cases use the non-redistributable samples in the gitignored `TestStreams/DolbyVision/` and skip without them, CI included), playback, and resilience via `Support/TortureHTTPServer.swift` (a fault-injecting local HTTP server — mid-stream errors, stalls, disconnects). New work in demux/decode/time/concurrency should come with tests here; add new media fixtures to `generate-fixtures.sh`, not as committed binaries.
4 changes: 2 additions & 2 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the
- **Zero-delay switching:** `prepare(next:)` opens a second session through first-frame-decoded, then atomically swaps renderer attachment; old session torn down async. (Also powers Lume's next-episode auto-advance and channel zapping.)
- **Precache:** same AVIO disk-cache layer, `precache(url:leadingSeconds:)` background task with LRU budget.
- **Upscaling:** MetalFX spatial scaler stage (D5), quality/perf switch, auto-disable on thermal pressure (`ProcessInfo.thermalState`).
- **HDR:** HDR10/HLG passthrough via sample-buffer attachments; DV P5/P8: preserve `dvcC` config + RPU side data into CMSampleBuffer format description extensions (VideoToolbox tone-maps); HDR10+ dynamic metadata via per-frame attachments. EDR headroom handling on macOS/visionOS Metal path.
- **HDR:** HDR10/HLG passthrough via sample-buffer attachments. DV with an HDR10-compatible base layer (P8.1, P10.1, …) plays as HDR10. DV with an IPTPQc2 base layer (P5, P10.0, P20 — compatibility ID 0) is converted to HDR10 before enqueue by a Metal compute pass (`Decode/DolbyVisionConverter`) that applies each frame's RPU reshaping (polynomial + MMR) and the IPT→LMS→BT.2020 matrices FFmpeg exports as `AV_FRAME_DATA_DOVI_METADATA`; the renderer then tone-maps it like any HDR10 stream. Dynamic DV display management (L1/L2/L8 trims) is not carried over. HDR10+ dynamic metadata via per-frame attachments. EDR headroom handling on macOS/visionOS Metal path.
- **360°/VR:** MetalRenderer projection (equirect sphere / fisheye), CoreMotion + gesture camera; visionOS immersive presentation later.
- **Blu-ray ISO/DVD:** libbluray/libdvdnav behind the `AVIOProtocol` seam; menu-less title playback first (longest-title heuristic), chapters mapped to the chapter API.
- **AI subtitles:** whisper.cpp (Metal) tapping the decoded-audio channel; streaming word-level segments feed the subtitle model as a synthetic track; on-device translation via Translation.framework.
Expand All @@ -208,7 +208,7 @@ Licensing note: engine code is **MIT** (`LICENSE`); FFmpeg is LGPL 2.1+, and the

| Risk | Assessment | Mitigation |
|---|---|---|
| Dolby Vision P5 (proprietary IPT-PQ-C2) | Hard. VideoToolbox handles DV streams only when correctly signaled; P5 has no BL fallback. | Phase-gated spike: test `dvh1/dvhe` signaling through `AVSampleBufferVideoRenderer` on-device early (Phase 3). If VT won't tone-map P5, fall back to libplacebo-style Metal tone mapping (stretch) or document as limitation. |
| Dolby Vision P5/P20 (proprietary IPT-PQ-C2) | Resolved for decoded-frame enqueue: VideoToolbox does *not* reshape — `AVAssetReader` and `AVAssetImageGenerator` both return raw IPT even from a correctly signaled `dvh1` track, and tagging a buffer `IPT_C2` changes nothing. | Metal DV→HDR10 conversion (§6 HDR). Still open: compressed `dvh1` enqueue into `AVSampleBufferVideoRenderer`, the only route to true DV output (TV in DV mode) — needs an on-device spike, and would be a second decode path. |
| Dolby AC-4 | No FFmpeg decoder (patent-encumbered); Apple decodes AC-4 only via its own frameworks. | Investigation ticket: attempt `AVSampleBufferAudioRenderer` passthrough of AC-4 `CMSampleBuffer`s (compressed enqueue). If rejected → out of scope, documented. |
| Audio passthrough (compressed Dolby out) | Apple exposes no raw bitstream-out API; `AVSampleBufferAudioRenderer` *can* accept some compressed formats. | Same compressed-enqueue spike (EAC-3/Atmos-in-EAC3 first — that's the one that matters for AVRs). |
| `AVSampleBufferRenderSynchronizer` edge cases (rate ≠ 1 with filters, very high fps) | Medium. | MetalRenderer is the escape hatch; sync-policy abstraction keeps both backends behind one interface. |
Expand Down
4 changes: 3 additions & 1 deletion Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,9 @@ let package = Package(

.testTarget(
name: "LumeEngineCoreTests",
dependencies: ["LumeEngineCore", "LumeEngine"],
// CFFmpeg directly: some tests build FFmpeg structs (e.g. Dolby
// Vision metadata) to exercise the engine's readers of them.
dependencies: ["LumeEngineCore", "LumeEngine", "CFFmpeg"],
exclude: ["Fixtures/generate-fixtures.sh"]
),
],
Expand Down
Loading
Loading