control: use SEM_UNDO for sandbox name ownership and discovery - #256
Merged
Merged
Conversation
An inherited abstract socket keeps its name bound after the supervisor exits. A host that forks while a sandbox is running can therefore leave its name occupied until the last child closes the descriptor, even though the sandbox itself is gone. Reserve names in a per-user System V semaphore registry with SEM_UNDO. The kernel releases the reservation on supervisor exit, and ordinary forks do not inherit the undo adjustment. Give each instance random socket addresses so that inherited descriptors cannot block name reuse. Discovery reads the active reservations, while requests still use the abstract sockets and authenticate the supervisor with SO_PEERCRED. The registry is diskless and reuses a persistent set with 256 slots. Release claims on normal cleanup and failed or cancelled startup too. Registry exhaustion and transaction failures must fail creation rather than silently leave a sandbox outside control discovery. Signed-off-by: Cong Wang <cwang@multikernel.io>
The process-group socket only supplies the child's kernel-recorded pid. Now that discovery has a semaphore registry, keeping a second socket for that identity also keeps a listen backlog that PID lookups can fill when the control loop is not running. Have the child stamp a dedicated semaphore after setpgid and before confinement. GETPID supplies its identity without a socket connection. Check the instance token in the same atomic semop so that a delayed child cannot stamp a reused slot. Leave the supervisor's ownership semaphore untouched and create no undo adjustment in the child. Keep the request socket for control commands. The default seccomp policy denies SysV IPC, so the confined workload cannot access the registry even with its identifier. Test reads, writes and removal in both supervised and unsupervised modes, and document that explicitly allowing SysV IPC restores same-UID access. Signed-off-by: Cong Wang <cwang@multikernel.io>
Encoding names and tokens in semaphore values limits the registry to 256 live names. Moving metadata out of those values helps, but a fixed registry still caps sandbox density at the size of its semaphore sets. Store names, instance tokens and index links in shared memory, and grow in segments of 4,096 entries as demand increases. Each segment uses one set of 8,196 semaphores for ownership and child pid stamps. A shared hash index directs name lookup and reservation to one bucket, avoiding a scan of every sandbox. Inactive entries are reused within their bucket; there is no compiled-in total sandbox limit. Serialize metadata changes and child generation checks with a root SEM_UNDO lock. Journal new index entries so that a later process can recover publication interrupted by a crash. Segment headers identify the root and segment ordinal before their semaphore sets are used, so IPC key collisions fail without altering another segment's owners. Attach shared memory only during registry operations. Mark attachments MADV_DONTFORK under the sandbox fork lock, and detach the child's own attachment before confinement, including for entrypoints without exec. Default seccomp still denies access to every registry IPC object. Exercise 40,000 simultaneous reservations, indexed lookup and reuse, duplicates behind inactive bucket entries, and recovery from a killed index publisher at each publication stage. The registry remains diskless; allocated objects persist for reuse, with growth bounded by host IPC and memory limits. Signed-off-by: Cong Wang <cwang@multikernel.io>
congwang-mk
force-pushed
the
control-sem-undo
branch
from
September 22, 2026 15:57
5d5ac5f to
cf902c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An abstract socket name remains bound while any inherited descriptor refers to it. A host process that forks while a sandbox is running can therefore keep the sandbox name occupied after the supervisor releases it or exits.
Reserve names in a per-user System V semaphore registry with SEM_UNDO. Ordinary forks do not inherit the undo adjustment, so supervisor exit releases the reservation even if a host child keeps every inherited descriptor. Normal cleanup and failed or cancelled startup release it explicitly. Each instance gets a random request socket address, allowing immediate name reuse despite lingering descriptors.
The registry also replaces the process-group socket. After setpgid and before confinement, the child stamps a dedicated semaphore; GETPID supplies the kernel-recorded PID. An instance-token check under the registry transaction lock prevents a delayed child from stamping a reused slot. The request socket remains for control commands and authenticates the supervisor with SO_PEERCRED.
Supersedes #244. This removes the need to park a descriptor in SCM_RIGHTS, the associated in-flight descriptor limit, and the lifetime backlog budget for PID lookups. Discovery reads the registry instead of /proc/net/unix. No dedicated control thread, helper process, filesystem path, or io_uring is required.
The registry grows in segments of 4,096 entries with no compiled-in total sandbox limit. Each segment has one set of 8,196 semaphores plus shared metadata for names, tokens and index links. A root directory has 65,536 hash buckets; collisions form chains across segments, so the bucket count is not a capacity limit. Name lookup and reservation scan the matching bucket. A small insertion journal recovers interrupted index publication, and segment identity checks reject IPC key collisions before modifying reservations.
Registry objects persist for reuse. Inactive entries are reused within their hash bucket, so storage reflects historical demand per bucket. Host IPC and memory limits bound growth; exhaustion and registry errors fail creation. Transaction waits are bounded to two seconds. Clients must share the relevant IPC and network namespaces, and this versioned protocol does not discover older instances. A host explicitly using CLONE_SYSVSEM can share undo state and delay automatic exit cleanup.
Default seccomp confinement blocks semaphore and shared-memory access even when the workload knows their identifiers, in both supervised and unsupervised modes. Metadata attachments are temporary and marked MADV_DONTFORK while holding the sandbox fork lock, protecting in-process entrypoints as well as applications that exec. Explicitly allowing sysv_ipc restores same-UID access. These boundaries and the registry lifecycle are documented in docs/architecture.md.
Validation covered 832 core unit tests, 488 integration tests, and 102 CLI tests. The full run exposed a missing-name diagnostic regression; after restoring the existing message, all 30 control integration tests passed on rerun. Coverage includes 40,000 simultaneous reservations across growing segments, indexed lookup and reuse, duplicate detection past inactive bucket entries, killed publishers at three index publication stages, segment identity mismatch, lingering host children, supervisor death, failed and cancelled startup, stale child publication, repeated PID lookups, default IPC access denial, and mapping non-inheritance.