composefs: Support changing the kernel without rebuilding the rootfs image #2584

In the systemd DDI model the UKI is its own artifact: it pins the /usr
image by dm-verity root hash (usrhash=) and is updated by its own
systemd-sysupdate transfer, so in principle a new UKI can be built and
signed against an existing /usr image. In practice the kernel modules live
in /usr, so once /usr is sealed a kernel change still means a new image;
Bluefin Server hit exactly this and settled on one kernel per image (see
prior art). The property we want, a kernel that can be replaced without
touching the sealed rootfs, is one neither side has today. This issue is
about getting it for composefs/UKI systems.

This is about build and publish time, not about swapping kernels on a running
system. The goal is that the rootfs container image is built, rechunked and
(for sealed images) digested once, and a kernel can then be produced or
replaced against it by a separate step or pipeline, for example by a kernel
team that publishes kernel-only OCI images.

Today the kernel, the initramfs and /usr/lib/modules/<kver> all live in the
container image, and the UKI pins that image with composefs=<digest>, so any
kernel change means a new image build.

Use cases:

• shipping a kernel CVE fix without rebuilding and re-rechunking every derived
  image
• building several kernel flavors (debug, hardware enablement, vendor kernel)
  from one rootfs
• letting a kernel be built by a different pipeline than the rootfs, as
  Dakota does with its kernel-only OCI images (see prior art)

What works today

Since #2185 the vmlinuz and initramfs can come from outside the rootfs
(bootc container split-kernel-and-rootfs, bootc container ukify --kernel-dir). So a per-kernel derived build is already possible:

FROM rootfs-rechunked AS base
FROM base AS with-kernel
COPY --from=kernel-image /usr/lib/modules /usr/lib/modules
RUN <build initramfs>
# ... compute digest, bootc container ukify --kernel-dir, sign, copy UKI back

The rootfs layers are shared, so the registry and the client dedup them. What
this does not give us:

1. A stable rootfs digest. The modules go into the rootfs tree, so every
   (rootfs, kernel) pair has its own composefs digest. Anything bound to the
   rootfs digest (the sealing signature, attestations, the UKI cmdline) has
   to be redone per pair.
2. An initramfs recipe. The initramfs mixes kernel modules with userspace
   from the rootfs (systemd, bootc-root-setup, dracut modules). Someone
   has to build it for each pair, and bootc has no documented way to do that
   from a kernel-only image plus a rootfs.
3. Tooling. The flow above is a hand-written multi-stage Containerfile.
   There is no bootc container command that takes a rootfs and a kernel
   image and produces the bootable image.

Strawman

This isn't a proposal yet. It lists the pieces to get discussion going.

• A kernel-only OCI image contract. /usr/lib/modules/<kver>/ with
  vmlinuz, the modules and modules.*, optionally /usr/src/<kver>
  headers, no userspace. Dakota's kernel image already has this shape.
  bootc container lint could check it.
• An assembly command. Something like
  bootc container assemble --rootfs <img> --kernel <img> --to <img>
  (bikeshed) that copies the modules in, builds the initramfs against the
  rootfs userspace, computes the digest, runs ukify, and emits the final
  image. This is the same step Sealed image build UX + implementation #1498 wants for sealing, with the kernel as an
  input instead of taken from the rootfs. Signing stays a separate step as
  Sealed image build UX + implementation #1498 discusses.
• Keeping the rootfs digest stable. If we want the digest to be
  independent of the kernel, the modules can't be part of the digested tree.
  Options:
  • (a) Accept that the digest changes per kernel. Simplest; the cost is one
    digest computation and one signing per pair. This is what the flow above
    does.
  • (b) Ship the modules as a second composefs image with its own digest,
    referenced from the UKI cmdline (e.g. bootc.kmods=<digest>), and have
    bootc-root-setup mount it over /usr/lib/modules/<kver> after
    verifying it. The rootfs digest is then kernel-independent, like
    usrhash=. frontend for systemd-sysext #7 (sysext frontend) would need the same "stack a second
    verified image" mechanism.
  • (c) Keep the UKI generic and have it carry a persistent public key
    instead of a digest; the rootfs digest is then shipped as a signed blob
    that the initramfs verifies. This is the "persistent keypair" variant
    already described in docs/src/relationship-particles.md, with the
    downgrade caveat noted there.

Open questions

• Is (a) enough? If the only cost of a kernel change is re-digesting and
  re-signing, maybe the tooling (assembly command plus a documented kernel
  image contract) is the whole feature and (b)/(c) aren't needed.
• How to build the initramfs. Dakota's kernel image says "build the
  initramfs in the consumer", so the assembly step needs the rootfs
  userspace plus the kernel's modules. Because systemd-stub passes the
  .initrd sections of addons to the kernel alongside the UKI's own initrd,
  another option is a userspace initrd from the rootfs plus a modules initrd
  from the kernel image.
• Out-of-tree modules (nvidia, akmods, anything a derived image builds
  against its kernel) sit in the rootfs and match only one kver. With (a)
  they'd have to be built in the assembly step; with (b) they'd have to move
  into the kernel image. Bluefin Server ships them as image-locked sysexts
  (see prior art).
• Combining separately signed pieces under Secure Boot. With (b) or (c),
  anyone with write access to the ESP can combine independently signed
  pieces, for example an old vulnerable kernel with the current rootfs. Do we
  rely on SBAT generations or a TPM policy on PCR 11/12, or do we require (a)?
• Scope. Should this cover only composefs, or the ostree backend too?
  Starting with composefs/UKI seems simplest.
• Relationship to sysext. Should the modules image in (b) just be a
  bootc-managed system extension (frontend for systemd-sysext #7)? Note the .sysext.raw images the UKI
  spec allows in $UKI.efi.extra.d/ only apply to the initrd and are dropped
  at switch-root, so they don't solve the host-side modules problem on their
  own. systemd-sysext and confext are unsupported on bootc today (docs: Document that systemd-confext and sysext are unsupported #2436).

Prior art

• Kernel-only OCI images. ghcr.io/projectbluefin/dakota-kernel-ubuntu
  (repo) is a
  scratch image with just /usr/lib/modules/<kver>/ (vmlinuz, modules,
  modules.*) and /usr/src/linux-<kver>/ headers, no userspace or
  initramfs; the consumer is expected to build the initramfs. Dakota consumes
  it at build time (BuildStream docker source with runtime/devel split
  rules). This is the input the assembly step would take.
• Kernel modules as a sysext. Bluefin Server ships NVIDIA and ZFS modules
  as image-locked sysexts (VERSION_ID=<image-version>) under
  /usr/lib/modules/<kver>/extra/<name>/, deliberately without modules.*,
  and rebuilds a depmod index under /run before modprobe -d
  (docs).
  Option (b) would hit the same index problem.
• UKI as a separate sysupdate transfer. Bluefin Server's
  files/os/sysupdate.d/{10-usr,11-usr-verity,20-uki}.transfer update /usr,
  its verity partition and the UKI as three artifacts sharing one version.
  They nonetheless settled on one kernel per image
  ("Bluefin server pivot" projectbluefin/server#249), because the modules live in /usr.
• Bluefin Server's history is the same tension. Before
  feat!: FSDK-composed OS with a signed, diskless-first boot chain projectbluefin/server#262 the kernel was its own BuildStream element
  owning /usr/lib/modules (the userspace import deleted that path), a
  kernel.yml workflow exported a kernel-only artifact, and the UKI pinned
  nothing (plain XFS root, no usrhash=), so a new kernel was just a new
  UKI. lsm: exit early if the process already has install_t #262 sealed /usr with verity and usrhash=; the modules were now
  inside the sealed tree, and docs: Extend rpm-ostree relationship a bit #249 was closed as "one kernel per image".
  Sealing is what couples the kernel to the rootfs, not the filesystem
  layout.
• Kernel-independent initrd. Bluefin Server builds its initrd from a
  dedicated minimal userspace stack (initrd-stack.bst), not from /usr,
  and copies in only the modules the kernel image provides
  (bluefin-server-boot.bst). One answer to the initramfs question above.
• Module trust in the kernel. Their module signing cert is compiled into
  the kernel (SYSTEM_TRUSTED_KEYS via a junction override) and the same key
  signs in-tree and sysext modules. A kernel-only image carrying its trusted
  cert lets out-of-tree modules be built by any pipeline holding the key,
  independent of the rootfs.

Related

• Rework UKI generation to enable removing kernel & initrd from image #2185: removing the kernel and initrd from sealed images
  (split-kernel-and-rootfs, ukify --kernel-dir)
• Sealed image build UX + implementation #1498: sealed image build UX (build the UKI, sign elsewhere, seal)
• UKI Addons Support #2448: UKI addons support
• frontend for systemd-sysext #7: frontend for systemd-sysext
• Design initramfs add-on entrypoint for bootc #2414: initramfs add-on entrypoint
• docs: Document that systemd-confext and sysext are unsupported #2436: docs saying sysext/confext are unsupported
• docs/src/relationship-particles.md, "ephemeral key" vs. persistent keypair
  for binding the UKI to the root
• Where to place the UKI in the container or ignoring specific path when building the composefs image composefs/composefs-rs#131: where the UKI goes, and excluding /boot from
  the digest

Assisted-by: AI
I reviewed it carefully.