Skip to content

riscv: implement multikernel lifecycle and safe shutdown - #39

Draft
pro-utkarshM wants to merge 11 commits into
multikernel:multikernel-riscvfrom
pro-utkarshM:riscv/multikernel-23-25-v1
Draft

pro-utkarshM wants to merge 11 commits into
multikernel:multikernel-riscvfrom
pro-utkarshM:riscv/multikernel-23-25-v1

Conversation

@pro-utkarshM

@pro-utkarshM pro-utkarshM commented Sep 3, 2026

Copy link
Copy Markdown

Summary

  • implement the RISC-V multikernel CPU lifecycle with direct SBI HSM calls,
    physical hart IDs, bounded HART_STATUS polling, and parked-state checks
  • add an immutable entry stub that executes fence.i on every start while
    preserving the RISC-V a0/a1 boot ABI
  • contain spawn halt, power-off, reboot, panic, and SMP-stop paths with
    HART_STOP, and serialize lifecycle/resource changes against image teardown
  • retain a hot-added CPU with its instance when a lost ACK leaves ownership
    uncertain, so a possibly running hart is never exposed through the free pool

Addresses #23, #24, and #25.

Base and attribution

This draft targets multikernel-riscv at
fe31d7b0ca32b0f806c88059c88d70600fda45cc, the RISC-V foundation from #37
that Cong applied for follow-up work.

The previous temporary replay of Nikolay Nikolaev's foundation commit has been
dropped. His accepted foundation remains below this series with its original
authorship; every commit in this PR is above that branch point.

Scope

This PR keeps #23-#25 together because lifecycle, entry-cache coherency, and
safe shutdown share the same HSM handoff invariants.

#26 remains a separate follow-up branch. RISC-V instance-DTB device filtering
(#27) and firmware IPI receive support (#28) are out of scope.

Validation

Current head: eef3dcbad234

  • rv64 LLVM vmlinux Image build with CONFIG_MULTIKERNEL=y
  • rv64 LLVM vmlinux Image build with multikernel disabled
  • nommu_virt_defconfig check proving RISCV_M_MODE=y excludes multikernel
  • x86_64 GCC vmlinux build with CONFIG_MULTIKERNEL=y, CONFIG_OF=y, and
    CONFIG_KEXEC_FILE=y
  • linked-symbol check for every RISC-V multikernel architecture hook
  • entry-stub disassembly check: leading fence.i, no a0/a1 clobber, and no
    relocations
  • git diff --check and strict per-commit checkpatch.pl: zero reported
    errors, warnings, or checks
  • focused source-order assertions for reserve-before-handoff, uncertain-CPU
    accounting, RISC-V stopped-state confirmation, and halted-settlement locking

Prior QEMU/OpenSBI runtime results were produced on the pre-rebase series
stacked with #26. The accepted-base head still needs runtime revalidation
before this draft is ready for merge.

Remaining draft gates

  • rebase the separate riscv: KEXEC_TYPE_MULTIKERNEL Image loader, placement in the grant, no purgatory #26 loader branch onto this head
  • rerun QEMU virt + OpenSBI PLIC lifecycle tests on the accepted base
  • rerun halt, power-off, reboot, panic, secondary-hart stop, same-address
    respawn, different-Image respawn, and 50-cycle stress tests
  • inject lost CPU-add ACK and partial-start failures and verify conservative
    ownership/cleanup behavior

AIA runtime validation remains blocked on #27's per-hart IMSIC/APLIC and device
filtering. MKTTY/ring validation remains blocked on #28; neither is duplicated
here.

congwang-mk and others added 11 commits September 4, 2026 20:36
The manifest code reaches into the pool's arch state for the park
slot and calls mk_pool_park_regions(), which only the x86 header
declares. Both are part of what CONFIG_ARCH_HAS_MK_HOST_PARK stands
for, and an architecture that parks CPUs in firmware has neither, so
the generic code does not build there.

Declare mk_pool_park_regions() with the other host park functions and
add mk_pool_park_slot() beside it, with the usual empty fallbacks.

Signed-off-by: Cong Wang <cwang@multikernel.io>
Wire CONFIG_MULTIKERNEL into the 64-bit RISC-V build and add sparse
hart ID translations. Reject the invalid hart sentinel before lookup so
it cannot alias an unused logical CPU slot.

Reserve the architecture control block for the spawn context, DTB and
entry stub. Provide safe stubs for the full architecture interface so the
functional SBI HSM, Image loader and doorbell work can land incrementally.

[Picked from multikernel#37 with the
 Kconfig dependencies split out and the ARCH_HAS_MK_POOL_STATE gate on
 CPU removal dropped: an arch that cannot park a departing CPU should
 refuse in its own takedown path. Added the empty struct mk_pool_arch
 the core now embeds, and dropped a force-stop registration stub that
 nothing declares.]

Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Signed-off-by: Cong Wang <cwang@multikernel.io>
The instance restore and ring handoff paths consume the live OF
tree. Without CONFIG_OF, x86 multikernel configurations compile
references to OF globals that cannot link and cannot restore a spawn at
runtime. Reject that unusable configuration in Kconfig.

Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
RISC-V parks CPUs in firmware with SBI HSM, so multikernel must reuse
the architecture's HART_START, HART_STOP and HART_STATUS wrappers. Make
those helpers available to RISC-V architecture code while retaining the
existing SBI-to-Linux errno mapping.

Add only the per-instance context and entry-stub bookkeeping required by
the lifecycle implementation. Hart IDs remain physical firmware
identifiers and are never used as array indexes.

Link: multikernel#23
Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
OpenSBI does not invalidate a stopped hart's instruction cache when
HART_START restarts it, and an SBI remote fence cannot target a stopped
hart. Reusing an Image address can therefore execute stale instructions.

Add an immutable, relocation-free entry stub whose first instruction is
fence.i and whose target comes from the adjacent data page. Publish its
address through the multikernel manifest and route spawn-kernel secondary
starts through it as well as the primary start. The stub preserves a0 and
a1, so the primary receives its DTB and secondaries receive their normal
SBI boot data.

Flush the local instruction cache before every HART_STOP so a later start
cannot retain an older stub line.

Link: multikernel#24
Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
Implement the RISC-V multikernel CPU lifecycle with direct SBI HSM calls.
A spawn is allowed only after HART_STATUS reaches STOPPED; STARTED,
START_PENDING and STOP_PENDING are polled with a bounded timeout.

Allocate the immutable entry stub and mutable target context from the
instance control block, publish the stub in the manifest, and start the
physical hart with the Image entry and DTB boot ABI. Use physical hart IDs
for host doorbells and confirm firmware stop state before release or memory
reclaim. RISC-V deliberately reports force-stop as unsupported because HSM
has no remote HART_STOP operation.

Link: multikernel#23
Link: multikernel#24
Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
Prime every assigned hart through an immutable host-text fence.i
trampoline before it fetches a rewritten instance stub or Image.
Preserve distinct HSM state errors for bounded lifecycle handling.

Link: multikernel#23

Link: multikernel#24

Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
Reuse the kexec lock for CPU, memory and device transfers so an
Image rewrite cannot race with resource mutation. Recover an active
instance only after every assigned hart is confirmed stopped.

Link: multikernel#23

Link: multikernel#26

Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
A spawned kernel must never invoke SBI SRST or a legacy shutdown
because those operations can reset the host. Detect the spawn handoff
before reset registration and suppress host-wide reset handlers.

Route halt, poweroff, restart, SMP stop and panic shutdown through
local HART_STOP. Cache the parent endpoint for a non-allocating panic
notification, set a safe spawn panic default, and prevent memory reuse
until every assigned hart is confirmed stopped.

Link: multikernel#25

Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
Reserve instance tracking before handing a pool CPU to a running
kernel. If the response is lost and firmware cannot confirm the hart
stopped, conservatively keep the CPU assigned to that instance instead
of exposing a possibly running hart through the free pool.

Link: multikernel#23
Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
Take the kexec lock while confirming parked CPUs and moving an
instance back to the loaded state. This keeps asynchronous halt
notifications from racing CPU ownership changes or image teardown.

Link: multikernel#23

Signed-off-by: Utkarsh Maurya <projects.utkarshMaurya@gmail.com>
@pro-utkarshM
pro-utkarshM force-pushed the riscv/multikernel-23-25-v1 branch from 7fd9938 to eef3dcb Compare September 7, 2026 18:08
@pro-utkarshM
pro-utkarshM changed the base branch from master to multikernel-riscv September 7, 2026 18:09
@pro-utkarshM

Copy link
Copy Markdown
Author

@congwang-mk, before updating this draft, could you confirm the dependency ordering?

We have newer local #23#25 lifecycle commits, with #26 kept as a separate follow-up. Standalone builds passed; QEMU PLIC/AIA validation passed on an integration stack with explicit runtime dependencies, including 50-cycle same-Image and alternating-Image re-spawns.

The tested stack still has the shared receive-ring reset limitation. Nickolaev’s #7 overlaps that work but is not integrated or validated in our stack. Should we depend on #7 first, or submit the bounded RISC-V series with that limitation disclosed?

Which public #27/#28 commits should we use instead of our temporary attributed runtime extracts?

@congwang-mk

Copy link
Copy Markdown

There are no public #27/#28 commits: both are unassigned with no PR or branch, and multikernel-riscv has only Nikolay's #22 skeleton (fe31d7b), nothing for #27/#28.

Please don't carry extracts from Nikolay's July riscv branch. It predates the device tree rework, where the manifest became the spawn's boot tree, so #27 is now "pass the manifest as a1 and add hart/intc nodes to mk_dt_emit_boot_tree()". #27 and #28 are open to claim as separate PRs on multikernel-riscv. Publish your integration stack as a test-only branch so the results are reproducible.

Don't depend on #7; disclose the ring-reset limitation.

For this PR:

  1. Emit the entry stub from mk_manifest_chosen() via an arch hook, under /chosen with the multikernel, prefix, not by reopening the finalized FDT with PAGE_SIZE. The manifest is 256 KB on this base, so any boot tree larger than one page fails there.
  2. Move the SBI IPI send and mk_panic_to_pool() to riscv: multikernel doorbell (IMSIC identity or SBI IPI convention) #28, and send multikernel,host-ipi-cpu to master separately with a CPU 0 fallback, so a new spawn kernel still boots on an older host.
  3. Split the kexec-lock serialization, the hot-add retention and the spawn-state changes into a PR against master, tested on x86, and don't use kexec_trylock() as the resource lock. mk_overlay_mutex already serializes resource changes; image teardown and exec should take the multikernel lock, not the other way round.
  4. Explain in the commit why the prime handshake is needed versus the opaque-record stub from riscv: fence.i entry stub to avoid stale I-cache on re-spawn #24.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants