Skip to content

Guard AV1 sample descriptions in MP4 - #12

Merged
DeepRegular merged 3 commits into
DeepRegular:mainfrom
SteveYu000:codex/av1-mp4-config
Nov 3, 2026
Merged

DeepRegular merged 3 commits into
DeepRegular:mainfrom
SteveYu000:codex/av1-mp4-config

Conversation

@SteveYu000

@SteveYu000 SteveYu000 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

A mixed rav1e/SVT-AV1 MP4 cut now keeps 192 of 214 pictures for the partial selection and 144 of 195 for the two-range selection. Both used to retry with the whole selection re-encoded. The source declares level index 31 (no level-based constraints), while the boundary encoder declares index 0 (Level 2.0). Review reproducer and discussion

The FFmpeg 7.1 writer still extracts each Sequence Header and checks the fields shared by the single MP4 sample description. When the output mixes copied and encoded pictures, the writer can raise the level in an encoded boundary to match the copied configuration. It updates both in-band OBUs and packet extradata, handles the conditional tier bit and reduced still-picture syntax, and rewrites the trailing padding and OBU size when needed. Copied access units stay byte-identical. A wholly encoded selection uses the encoder's actual configuration.

A lower finite target, a change in profile, tier, bit depth or chroma, or conflicting copied sequences still triggers the existing logged full-video retry. Finite promotions must be provably compatible. Explicit decoder models and Level 3.1 → 4.0, which can tighten the minimum compression ratio, take the conservative retry path. AV1 levels

Validation on the final merged tree used an actual Windows x64/MSVC release build, Rust 1.98 and FFmpeg 7.1.1.

  • 21/21 media cases on identical inputs, versus 17/21 on the previous PR and 15/21 on v0.8.39. The stronger assertions require copying on the rav1e and finite Level 4.0 cases, including the key packets carrying the source headers. A real chroma-position mismatch triggers exactly one full-video retry.
  • FFmpeg CBS traces of 40 in-band Sequence Headers across the four promotion cases agree with v0.8.39 on every traced sequence field except the promoted level and its required tier bit. Strict decoding, frame counts, display PTS, DTS, duration, copied packets and decoded pixels passed. The single-picture rav1e selection keeps the encoder's Level 2.0. The three Matroska controls passed on their existing path.
  • The five previous full-video retries in the separate 55-case historical MP4 matrix now all keep their copy regions. On the original base the score stays 45/55. After integrating the upstream SAR merge it is 50/55, with the five remaining failures confined to VFR timing. Every case passes the av1C field checks.
  • Rust workspace 582 passed, four known Windows baseline failures filtered. Clippy/all-targets exits 0 with the existing warnings. The upstream SAR suite also passed 60/60 on the same executable.

This is the AV1/MP4 split from #10. Matroska/WebM policy remains a separate maintainer decision. The branch includes current upstream main with #11 and retains both CI suite entries. Technical details

The FFmpeg 8.x/9.x upgrade remains a separate follow-up for incompatible configurations. Those MOV muxers support mapping packets with new extradata to different sample descriptions. In the earlier 214-packet remux, 9.0.2 selected entries 2 → 3 → 2 with zero mismatches and every packet preserved. FFmpeg 7.1.1 produced one description and two mismatches. The 8.x result comes from source inspection. FFmpeg 8.0 MOV writer · FFmpeg 9.0.2 MOV writer

cargo build --release --locked --manifest-path rust/Cargo.toml -p smartcut-cli
bash tests/run_av1_mp4_tests.sh

Check the profile, first operating-point level/tier and colour-sampling
fields represented by av1C against every emitted Sequence Header. Reuse
FFmpeg's extract_extradata BSF; let the first actual output sequence supply
the MP4 description, including fully re-encoded selections.

Keep compatible smart-rendered middles. If MP4's single stsd entry cannot
describe a changed configuration, close the first attempt and retry the
selected video fully re-encoded, once only. Explain in the log that this
supersedes the initial copy plan. Audio and subtitle modes are unchanged.

Only the mp4 output muxer enables the guard. MKV/WebM keep their existing
logic; their sequence-header handling policy is left to the maintainer
to decide separately. Do not include SAR, source-clock/VFR or M2TS changes.

Add six guard unit tests and an 18-export suite registered in CI. On
identical inputs, the new suite improves from 15/18 to 18/18, and the
historical MP4 matrix from 38/55 to 45/55 without a new failure. Workspace
tests: 576 passed, four known Windows baseline failures filtered; clippy
workspace/all-targets exits 0 with the existing warnings.
@SteveYu000
SteveYu000 marked this pull request as ready for review November 1, 2026 13:28
@DeepRegular

Copy link
Copy Markdown
Owner

Thank you, and for the FFmpeg 8/9 notes. I built this on Linux (Debian 13, FFmpeg 7.1) and ran tests/run_av1_mp4_tests.sh with v0.8.39 and with this branch on the same fixtures: 15/18 → 18/18, as you describe. The three MKV controls are unchanged.

What actually differs is only the level

Looking at sequence_field_prefixes in the results, the three failures are all from the rav1e source, and the only field that disagrees is seq_level_idx:

av1C prefix
rav1e source 81 1f 0c (level 31, unconstrained)
boundary encoder (SVT-AV1) 81 00 0c (level 0, 2.0)

Profile, tier, bit depth, monochrome and chroma position agree in every case, including the 10-bit libaom source. On this branch, two of the three (rav1e-partial, rav1e-multi) become whole-video re-encodes with 0% copied, to fix a level field.

I would rather not re-encode the whole video for that. The v0.8.39 output decodes strictly with no errors, and the copied pictures are the source's own bytes. Since only the level differs, could the boundary sequences be made to carry the source's level instead?

  • either by asking the boundary encoder for that level (SVT-AV1 takes one through svtav1-params; I have not checked what rav1e and libaom accept),
  • or by rewriting seq_level_idx[0] (and adding seq_tier[0] when the level goes above 7) in the boundary's Sequence Header, re-padding trailing_bits and the OBU size.

Level 31 means "no level constraint", so writing it over the boundary's 2.0 does not claim anything untrue. When the source states a real level that is lower than what the boundary encoder chose, the rewrite would be a lie, and that case, along with any profile / bit-depth / chroma mismatch, is where your guard and the full-video fallback make sense.

So my proposal is: keep the extraction and the field check from this PR, align the level at the seams, and fall back to full re-encoding only for the mismatches that cannot be aligned. For those, the automatic fallback as you wrote it, with its log lines, is fine with me.

On FFmpeg 8/9: agreed that this belongs in a separate change. Thank you for checking the 9.0.2 behaviour.

An MP4 cut mixing a rav1e source with SVT-AV1 boundaries can keep its
original pictures when only the boundary's declared level differs. Use
the source or active sequence fields for mixed exports and promote a
compatible boundary level, including the source's unconstrained index 31.
Update both the extracted configuration and the sample OBU, including
the conditional tier bit, trailing bits and OBU size. Preserve packet
properties and copied access units.

Keep the existing single retry for conflicting configurations, a lower
target level, or finite promotions whose constraints cannot be established.
Fully encoded selections use the encoder's actual configuration.

Exercise both seam orders, a finite promotion, a real chroma mismatch,
and the original rav1e partial and multi-range reproductions. Inspect
each sequence with FFmpeg and compare the copied key packets and pixels.
@SteveYu000

Copy link
Copy Markdown
Contributor Author

Thanks for pinpointing the level mismatch. I implemented the Sequence Header rewrite you suggested in f10ae7b. For the mixed rav1e cuts, the SVT-AV1 boundary headers now carry the source’s level index 31, including the required tier bit. The MP4 av1C matches those headers.

On the same fixtures, rav1e-partial now copies 192 of 214 frames, and rav1e-multi copies 144 of 195. I verified that every copied packet and its decoded pixels match the source, including the keyframes. The single-frame rav1e-one output uses the encoder’s Level 2.0 because that selection is entirely re-encoded.

Mixed cuts still fall back to a full re-encode when the source specifies a lower finite level than the boundary encoder, or when fields such as bit depth or chroma position differ. I rebuilt with FFmpeg 7.1.1 on Windows and all 21 cases passed, including the three Matroska controls and a chroma mismatch that exercises the fallback.

@DeepRegular

Copy link
Copy Markdown
Owner

Thank you, this is exactly what I hoped for. I built 0c6e1d5 on Linux (Debian 13, FFmpeg 7.1) and checked it as follows.

Confirmed

  • tests/run_av1_mp4_tests.sh: 21/21. rav1e-partial, rav1e-multi, rav1e-full and rav1e-copy-first carry 81 1f 0c; finite4-partial carries Level 4.0; chroma-partial takes the single full-video retry with both notes in the log.
  • rav1e-partial copies 89.7% and rav1e-multi 73.8% (by duration), and both decode with -err_detect explode.
  • I cut the same rav1e fixture with current main and with this branch and compared them packet by packet. The only differences are the boundary keyframes (2 in partial, 4 in multi), each one byte longer: the level becomes 31 and the tier bit is added. Every other packet and every decoded picture is identical. (SVT-AV1 is not bit-exact from run to run here, even with main, so I compared runs whose re-encoded pictures matched.)
  • The 12 unit tests in av1_mp4 pass, and clippy reports nothing in the new code.

One question on may_raise

The 3.1 → 4.0 exclusion looks unnecessary to me. MinCR is MinCrBasis × (DecodedSampleRate ÷ MaxDisplayRate), and in Table A.3 MainCR ÷ MaxDisplayRate does not increase with the level anywhere: 2 ÷ 31,950,720 at 3.1, 4 ÷ 70,778,880 at 4.0, so the 4.0 bound is looser. Since the exclusion only makes such a cut fall back to a full re-encode, it isn't wrong, and I don't mind keeping it. If you had a different constraint in mind, a word on which one would help me keep the comment accurate; otherwise feel free to drop it.

A tiny nit, no need to act on it: in the timing_info uvlc, the loop stops after 32 zeros without consuming the terminating 1, while the spec keeps reading until the 1. It only matters for a nonsensical num_ticks_per_picture.

I'm happy to merge this as it is. If you'd rather adjust the may_raise comment first, tell me and I'll wait.

@DeepRegular
DeepRegular merged commit 271a99b into DeepRegular:main Nov 3, 2026
3 checks passed
@DeepRegular

Copy link
Copy Markdown
Owner

A follow-up after the merge, so you know what changed in your code. Our review of main with #12 in it turned up six things in or around the new path. They are fixed in 70dda74 (engine) and 5db2874 (GUI); nothing needs doing on your side.

In av1_mp4.rs

  • may_raise: I removed the 3.1 → 4.0 exclusion after all. It wasn't harmless: SVT-AV1 writes 3.1 for 720p30 boundaries, so an ordinary 720p30 source labelled 4.0 fell into the full re-encode where v0.8.39 copied it. Every limit in Table A.3 stays the same or grows from one defined index to the next (I checked each step, and MinCR against libaom's get_min_cr), so any higher defined index is now accepted. The unit test asserts may_raise(5, 8, false). If you did have another constraint in mind, I'd still like to hear it.
  • The timing_info uvlc now reads up to the terminating 1, as in the spec. We also added a test that flips and truncates the real 8-bit, 10-bit and reduced-still-picture headers 20,000 times with random targets: no panic, and every rewrite parses back to its target.

Boundaries written by rav1e

4:4:4, 4:2:2 and 12-bit sources go to rav1e, because SVT-AV1 can't encode them. rav1e writes seq_level_idx 31 unless told otherwise, and 31 over a finite source counts as a lower target, so those cuts also fell into the full re-encode where v0.8.39 copied. open_encoder now passes rav1e the source's level (level=X.Y) when Annex A.3 defines it and the tier is main. If rav1e refuses the level (too low for the picture size or rate), it is opened again exactly as before. One side effect you should know about: open_encoder doesn't know the container, so .mkv/.webm boundaries from rav1e now state the source's level too. Decoded pictures are identical; only the Sequence Header bytes change. I decided to keep that, since it makes the boundary headers agree with the copied ones.

The full-video retry

  • A cut with a transition failed outright: ranges_with_transitions re-plans the body of a touched range with plan_range, which put copies back into the retry, so the second attempt hit the same configuration change. The retry now plans with min_copy at infinity, so every body is one re-encode.
  • A sliver range (thinner than one picture, from CLI ranges given in seconds) was re-planned as a one-segment re-encode and wrote the picture before it again. An empty plan now stays empty, as ranges_with_transitions already does.
  • --verify and the GUI's verify held the output to the discarded copy plan and reported every copied frame as different (the CLI exited 1). join_with_progress now hands back the plans the retry used through CutOptions::replanned, and both checks use them. The GUI also shows a note that the selection was re-encoded whole; until now it was only in the log.

With these, tests/run_av1_mp4_tests.sh is still 21/21, and chroma still takes exactly one retry. Thanks again for the guard; the review found nothing wrong with the description check itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants