Add support for multiple composefs= digests by cgwalters · Pull Request #2248 · bootc-dev/bootc

cgwalters · 2026-06-12T10:42:12Z

This adapts bootc to build on top of the work in composefs/composefs-rs#297

A toplevel goal here is supporting both the v1 and v2 EROFS formats, which means we'll work with RHEL9 era systems.

Right now bootc container ukify still generates composefs= i.e. v2, but I'd like to change that to do both - it's a pretty cheap thing (the main cost is generating the fsverity digests).

cgwalters · 2026-06-22T16:07:48Z

OK I think the chain here should be:

Merge Release 0.7.0 composefs/composefs-rs#314
Rebase on this on top

jmarrero

lgtm

just needs a rebase I think.

The only nit is that

            config.erofs_formats = composefs_ctl::composefs::erofs::format::FormatConfig {
                default: composefs_ctl::composefs::erofs::format::FormatVersion::V1,
                extra: [composefs_ctl::composefs::erofs::format::FormatVersion::V2].into(),
            };

Is duplicated 3 times here. So if we add a new version in the future we need to care about 3 places. Maybe it should be a helper function?

Johan-Liebert1

Lots of .clone() that I believe shouldn't be needed, I might be wrong though. Supporting both V1 and V2 are great, but it is a bit messy, not sure if there's a better way to do this. Also, some inconsistencies here and there (esp in comments) regarding whether V1 is the default or V2

Johan-Liebert1 · 2026-06-29T04:26:58Z

 seal_state=$1
 shift
+# EROFS format version to pass to bootc container ukify (optional, default: v2)
+erofs_version=${1:-v2}


Kind of conflicts with composefs/composefs-rs#330. We'd probably want to have the same defaults everywhere

Johan-Liebert1 · 2026-06-29T04:34:37Z

    os_id: Option<String>,
    boot_digest: String,
+    /// The composefs image digest parsed from (and validated against) the UKI's
+    /// own cmdline.  This is the authoritative deployment key for UKI boots:


This is for every boot right, not just UKIs?

Johan-Liebert1 · 2026-06-29T04:38:22Z

+        let composefs_info = BootComposefsCmdline::<Sha512HashValue>::from_cmdline(&cmdline)
+            .context("Parsing composefs=")?
+            .ok_or_else(|| anyhow::anyhow!("No composefs= or composefs.digest.v1= karg found in UKI cmdline"))?;
+        let composefs_cmdline = composefs_info.digest().clone();


This name is a bit confusing. afaiu this is only the digest and not the entire cmdline?

Johan-Liebert1 · 2026-06-29T04:59:16Z


 if test "${boot_type}" = "uki"; then
-  /run/packaging/seal-uki /run/target /out /run/secrets "${allow_missing_verity}" "${seal_state}"
+  /run/packaging/seal-uki /run/target /out /run/secrets "${allow_missing_verity}" "${seal_state}" "${erofs_version}"


We also need this in tmt/tests/booted/test-install-to-filesystem-var-mount.sh

Johan-Liebert1 · 2026-06-29T05:13:48Z

+    // (see setup_composefs_boot for the full rationale).  Provisional value for
+    // BLS (where bootc writes the karg from this same id); overridden for UKI by
+    // the digest the UKI cmdline actually carries.
+    let provisional_deploy_id = boot_id_v2.clone().unwrap_or_else(|| id.clone());


id here is confusing especially with both boot_id_v1/v2 defined. I believe it's the erofs digest corresponding to the erofs version that the repo is currently using?

Johan-Liebert1 · 2026-06-29T05:16:01Z

+    // Authoritative collision check against the final deploy key.  For UKI this
+    // may differ from the provisional checked above (the UKI may carry a
+    // non-default digest), so this is the load-bearing guarantee.
+    ensure_no_deploy_collision(host, &deploy_id)?;


why do we need to do this again?

Johan-Liebert1 · 2026-06-29T05:18:03Z

+    // setup-root opens `state/deploy/<this>` using that same karg, so we must
+    // key the deployment off exactly this value -- whether the UKI was sealed
+    // with the V2 (default) or V1 EROFS digest.
+    let deploy_id = uki_info.composefs_cmdline.clone();


we shouldn't need to clone this?

cgwalters · 2026-08-31T13:11:56Z

In the general case we may need to add support for "older bootc version" which includes not just the v2 digest but the xattr filtering logic too? See composefs/composefs-rs#337

cgwalters · 2026-09-03T15:45:51Z

OK, this one wants #2290 to land first which fixes our composefs mounts on c9s.

cgwalters · 2026-09-16T11:28:41Z

OK, this one wants #2290 to land first which fixes our composefs mounts on c9s.

OK! Finally that landed. I rebased this, but it's still somewhat raw, especially the last commit.

cgwalters · 2026-09-16T22:56:54Z

+    repo_requires_fsverity: bool,
+    missing_fsverity_allowed: bool,
+    uki_allows_missing_fsverity: bool,


Too many bool here...I think we may need an enum

cgwalters · 2026-09-16T22:58:05Z

+    Ok(())
+}
+
+/// Validate every primary UKI before any bootloader or ESP operation.  The


Hmmm I'm not sure, I think we need to be robust to corrupted state and allow recovery. Bailing early hurts that.

This seems more like a bootc internals fsck style thing.

cgwalters · 2026-09-16T22:58:35Z

+            .file_path
+            .file_name()
+            .ok_or_else(|| anyhow!("Could not get UKI file name"))?
+            .to_string_lossy()


No. We should require UTF-8.

cgwalters · 2026-09-16T23:15:28Z

+const COMPOSEFS_DIGEST_V1_FEATURE: &str = "/usr/lib/bootc/initramfs-features/composefs-digest-v1";
+const COMPOSEFS_DIGEST_V1_FEATURE_CONTENT: &[u8] = b"composefs-digest-v1 state-v1\n";
+
+/// Query `lsinitrd` without unpacking or executing any initramfs contents.


No, this is awful. I don't want ukify to be parsing the initramfs.

Among other things I don't want to hard depend on dracut.

I think we should just default to injecting both EROFS kargs right?

If we just want the cmdline, we have a function in composefs-rs for that. get_uki_cmdline_buffered

cgwalters-bot · 2026-09-22T17:08:09Z

I tested upgrading a sealed composefs RHEL 10.2 system (stock bootc 1.16.4, UKI + systemd-boot + Secure Boot, the rhel-bootc-examples sealing example) to this PR's c10s RPMs, with the new bootc in the compose so the initramfs is regenerated. The upgrade chain works: 1.16.4 bootc switch → PR image, then PR bootc upgrade → second PR image, then bootc rollback, all with Secure Boot, verity=require, /etc and /var state intact, and composefs-gc --assert-no-op clean.

But there's a problem with bare composefs=: every bootc from v1.16.4 through v1.16.13 seals a V1 EROFS digest under composefs=, because composefs-rs a29622c (pinned by v1.16.4) switched the default format to V1 while ukify kept emitting the legacy karg. Using the actual RHEL 10.2 binary against the same rootfs:

bootc 1.16.4 (RHEL 10.2)                 compute-composefs-digest -> 6c60fb07…2af2678f
this PR  --erofs-version=v1              compute-composefs-digest -> 6c60fb07…2af2678f
this PR  --erofs-version=v2              compute-composefs-digest -> 53bab3fa…2b3ca8

(The on-disk image header also has composefs_version=1.) Two consequences:

The documented 1.16 bridge path doesn't happen for ≥1.16.4. The old client already parses composefs.digest= first and stages V1 directly; the V2 fallback is never used and the V2 image isn't created. test-49-composefs-1-16-bridge pins 1.16.0, which predates the V1 default, so it can't catch the 1.16.4+ behavior.
The PR client refuses to deploy images sealed by 1.16.4–1.16.13. On a host running this PR, switching back to the original 1.16.4-sealed image:
```
# bootc switch <1.16.4-sealed image>
error: Switching: Composefs Switching: Performing Upgrade Operation: Pulling composefs repository: Verifying composefs digest against UKI: UKI composefs candidate format does not match recovered EROFS format
```
parse_uki_composefs_candidates (crates/lib/src/bootc_composefs/boot.rs:1036) maps composefs= to V2, so ids_for_candidate (:1100) looks only in the V2 set; ensure_correct_composefs_digest (:1420) then finds the matching V1 image, and resolve_boot_image_match (:1482) rejects it on format. validate_uki_candidates (:1126) has the same assumption. The log also reads confusingly: "Freshly computed composefs digest (sha512:6c60fb…) doesn't match the digest embedded in UKI (sha512:6c60fb…)".

This is deploy-time only: the initramfs mounts images/<hex> by digest regardless of the V1/V2 label, so existing 1.16.x deployments and rollbacks keep booting. But it blocks downgrades and any mixed-version setup, e.g. the common pattern where a separate tools stage with the distro's bootc runs ukify. bootc install of such an image likely goes through the same path (not tested).

Suggested fix: treat bare composefs=<hex> as format-ambiguous (accept a match in either the V1 or V2 expected set) and only enforce the format for the explicit composefs.digest=vN-… form, plus a test case for a composefs=<v1 digest>-only UKI. The docs' bridge section probably also needs to distinguish 1.16.0–1.16.3 (V2 under composefs=) from 1.16.4+ (V1 under composefs=).

Tested with the packages-centos-10 artifact from run 35735900302 (merge of d986021 into main).

Generated-by: https://github.com/cgwalters/#llms

We want to support upgrades from older bootc. bootc 1.16.0 stagers only understand the bare `composefs=` argument and create only V2 images and state. A UKI built with a newly regenerated initramfs carries the V1 digest followed by the V2 fallback, so its first boot selects the V2 image and state. A subsequent upgrade by a current client can then create and select the V1 image. Image builders must regenerate the initramfs when updating bootc; this change does not claim compatibility for stale initramfs artifacts. Explicit V2 remains available as a format control. Always emit both digests without probing initramfs contents. Preserve strict repository requirements when recovering a non-default boot image; changing serialization must not relax integrity. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

Composefs installation must preserve kernel argument policy, validate the target repository before writing boot artifacts, and inspect external images from a target-backed staged copy. Keep the source identity and initial UKI fs-verity policy authoritative across the entire install path. A bare `composefs=<hex>` UKI argument does not identify an EROFS format, even though composefs-rs documents it as V2 shorthand: bootc 1.16.3 sealed a V2 digest there, but 1.16.4 through 1.16.13 seal V1. Accept a bare digest matching a boot image of either format and only enforce the format for the self-describing `composefs.digest=vN-...` form; otherwise switching to any image sealed by those releases fails. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

The compatibility claim depends on regenerated current initramfs artifacts and on keeping strict and missing-verity fixtures distinct. Make the opt-in tests assert those boundaries and inspect the installed target rather than the running installer. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

The previous text read like an internal test log (test plan numbers, TMT variables, evidence lists) and repeated the same compatibility caveat several times. Describe what users need instead: the V1/V2 kernel arguments, what the bare composefs= argument has meant across releases, how to upgrade from 1.16, and what is known not to work yet. The only tested upgrade path remains bootc 1.16.0. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

The separate-/var install test should exercise the format selected by the test plan rather than silently forcing V1. Forward the environment selection while retaining V1 as the standalone default. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

There's a very complex dependency chain between `systemd-tmpfiles-setup.service`, `systemd-random-seed.service` and `systemd-tpm2-setup.service` among others. Basically we need `/var/lib` to be created, which will be done by the tmpfiles setup, but random-seed will normally do it too, but it can't because Fedora SELinux denies it. A surprising discovery is that this happens to be papered over for the OSTree backend by `ostree-remount.service`. Add a test case that no services fail. However this check needs to currently skip due to https://bugzilla.redhat.com/show_bug.cgi?id=2507393. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

The manifest tracks the v0.20.0 tag, but the stale revision-based lock entry makes locked Cargo operations fail. Regenerate the entry so dependency resolution agrees with the declared source. Assisted-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

Johan-Liebert1 · 2026-09-23T10:40:03Z

The old client already parses composefs.digest= first and stages V1 directly

how would the older client parse V1 cmdline arg?

cgwalters · 2026-09-23T10:42:46Z

how would the older client parse V1 cmdline arg?

It just parses composefs= which can historically now be v2 or v1 - where did you see that text?

Johan-Liebert1 · 2026-09-23T10:45:52Z

In the copilot review

The documented 1.16 bridge path doesn't happen for ≥1.16.4. The old client already parses composefs.digest= first and stages V1 directly; the V2 fallback is never used and the V2 image isn't created. test-49-composefs-1-16-bridge pins 1.16.0, which predates the V1 default, so it can't catch the 1.16.4+ behavior.

It just parses composefs= which can historically now be v2 or v1

okay, kinda makes sense then

Johan-Liebert1

Looks good to me now. @cgwalters I've disabled auto merge in case you want to change something here

Release 0.7.0 disabled this job until bootc main caught up with the composefs-rs API. That has now happened: bootc-dev/bootc#2248 (the adapt-composefs-rs-v1-erofs-2 branch we were pinned to) is merged, and bootc main depends on composefs-rs by git tag v0.9.2, so the [patch] override in bootc/Justfile applies to our workspace version again. Test against bootc main as before. This also drops the if: never(), which isn't a valid expression function: the workflow file failed to parse, so every push to any branch recorded a failed run with no jobs instead of skipping it. Generated-by: AI

Release 0.7.0 disabled this job until bootc main caught up with the composefs-rs API. That has now happened: bootc-dev/bootc#2248 (the adapt-composefs-rs-v1-erofs-2 branch we were pinned to) is merged, and bootc main depends on composefs-rs by git tag v0.9.2, so the [patch] override in bootc/Justfile applies to our workspace version again. Test against bootc main as before. This also drops the if: never(), which isn't a valid expression function: the workflow file failed to parse, so every push to any branch recorded a failed run with no jobs instead of skipping it. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

github-actions Bot added the area/install Issues related to `bootc install` label Jun 12, 2026

bootc-bot Bot requested a review from jeckersb June 12, 2026 10:42

cgwalters mentioned this pull request Jun 22, 2026

Update composefs #2262

Merged

jmarrero self-requested a review June 22, 2026 13:28

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch from 3b90847 to f92013b Compare June 22, 2026 17:30

cgwalters marked this pull request as ready for review June 22, 2026 18:51

bootc-bot Bot requested a review from henrywang June 22, 2026 18:51

cgwalters added the ci/tier-1 Run CI for tier-1 OS (centos-10) only label Jun 22, 2026

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch from f92013b to 69757fc Compare June 22, 2026 21:56

jmarrero previously approved these changes Jun 28, 2026

View reviewed changes

Johan-Liebert1 requested changes Jun 29, 2026

View reviewed changes

cgwalters added this to the 1.17 milestone Jul 6, 2026

cgwalters dismissed jmarrero’s stale review via 4ecc241 July 7, 2026 22:55

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch 3 times, most recently from 6ec1bf4 to cbff4b5 Compare July 9, 2026 15:20

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch from cbff4b5 to 8e7417f Compare August 31, 2026 16:16

github-actions Bot added the area/documentation Updates to the documentation label Aug 31, 2026

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch from 4bf145c to 45e92e4 Compare September 1, 2026 11:59

Johan-Liebert1 mentioned this pull request Sep 8, 2026

Remove condition on existance of /run/ostree-booted for bootc systemd units/timers #2437

Draft

cgwalters-bot mentioned this pull request Sep 9, 2026

deploy: Pull bound images before staging via composefs mount #2290

Merged

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch from 45e92e4 to f839490 Compare September 16, 2026 10:59

cgwalters added the ci/merge Run full CI suite (all OSes) — equivalent to merge queue label Sep 16, 2026

Johan-Liebert1 self-assigned this Sep 16, 2026

cgwalters mentioned this pull request Sep 16, 2026

composefs+UKI: Try to generate image with matching digest #2353

Closed

cgwalters commented Sep 16, 2026

View reviewed changes

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch from c30d873 to d986021 Compare September 22, 2026 13:48

cgwalters added 7 commits September 22, 2026 13:29

cgwalters force-pushed the adapt-composefs-rs-v1-erofs-2 branch from d986021 to 51ae138 Compare September 22, 2026 18:51

cgwalters enabled auto-merge (rebase) September 23, 2026 10:38

cgwalters requested a review from Johan-Liebert1 September 23, 2026 10:39

Johan-Liebert1 disabled auto-merge September 23, 2026 12:15

Johan-Liebert1 approved these changes Sep 23, 2026

View reviewed changes

cgwalters merged commit 41049cc into bootc-dev:main Sep 23, 2026
81 of 97 checks passed

cgwalters-bot mentioned this pull request Sep 24, 2026

ci: Re-enable bootc revdep CI cgwalters-forge/composefs-rs#2

Closed

cgwalters-bot mentioned this pull request Sep 25, 2026

ci: Re-enable bootc revdep CI composefs/composefs-rs#402

Merged

jmarrero-bot mentioned this pull request Oct 21, 2026

install: Add --preserve-var and --merge-etc to to-existing-root #2384

Draft

cgwalters-bot mentioned this pull request Oct 22, 2026

escalate: bootc#2437 P0 stalled on third-party author, nudge is public cgwalters-forge/tracker#485

Open

Conversation

cgwalters commented Jun 12, 2026

Uh oh!

cgwalters commented Jun 22, 2026

Uh oh!

jmarrero left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Johan-Liebert1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters commented Aug 31, 2026

Uh oh!

cgwalters commented Sep 3, 2026

Uh oh!

cgwalters commented Sep 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters-bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Johan-Liebert1 commented Sep 23, 2026

Uh oh!

cgwalters commented Sep 23, 2026

Uh oh!

Johan-Liebert1 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Johan-Liebert1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jmarrero left a comment •

edited

Loading

cgwalters-bot commented Sep 22, 2026 •

edited

Loading

Johan-Liebert1 commented Sep 23, 2026 •

edited

Loading