seccomp: keep close and fork out of the supervisor so signals cannot fail them

[congwang-mk]

Fixes #240. Fixes #235, except for a rare race that remains when fork is supervised and that only a kernel change can close (see "Known remaining race").

Problem

A syscall routed to SECCOMP_RET_USER_NOTIF sleeps interruptibly until the supervisor has received the notification; SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV only protects it after that. If a signal whose handler lacks SA_RESTART lands first, the syscall fails with EINTR without having run, including syscalls that never return EINTR natively. dash's SIGCHLD handler is one such handler:

• a lost close() leaked a pipe's write end into the last stage of a | b | c, which never saw EOF: the pipeline hung;
• a lost fork() made dash print "Cannot fork".

Nothing in userspace closes that window, so the fix is to keep the syscalls nobody retries out of the supervisor's hands.

Release builds, runs of 100 three-stage dash pipelines:

	complete	hang	"Cannot fork"
main (10 runs)	0	6	4
this branch, default policy (20 runs)	20	0	0
this branch, --max-processes 64 (20 runs)	15	0	5

Commits

1. netlink: stop trapping close(). It was notified only to drop (pid, fd) entries of the netlink cookie set. The set now records the injected socket's SO_COOKIE and checks the slot through pidfd_getfd() on lookup. Not the inode number: those come from a 32-bit counter shared with pipes that wraps without checking live collisions. This also fixes stale entries left by dup2, close_range and O_CLOEXEC on exec.
2. seccomp: refuse clone3 with ENOSYS (seccomp: clone3 bypasses the namespace creation ban #240). clone3 carried CLONE_NEWUSER past the ban that stops clone, because its flags are in a struct BPF cannot read. A supervisor-side check would race a CLONE_VM peer rewriting the struct. glibc and Rust's std fall back to clone on ENOSYS; the arg filter block runs ahead of the notification checks, so nothing routes around it.
3. checkpoint: stop holding fork notifications during a freeze. Redundant: copy_process() makes a signal sent to a group appear after any fork it overlaps. Also broken: thaw() cleared the held ids without answering them, so a process that forked during a freeze hung forever.
4. seccomp: intercept fork only when something needs it. clone/fork/vfork/exit/exit_group are notified only with max_processes, a policy_fn, or an exec handler. The namespace-flag check is the BPF filter's, and the running count of the virtual /proc/loadavg comes from the process index.

An earlier version of this PR also received notifications on a dedicated thread, to shorten the exposed window. It is dropped: with close and fork out of the way it made no measurable difference to the numbers above, while it made every queued task unstoppable until answered (which broke checkpoint and the execve argv freeze), cost a thread per sandbox and about 10 us per notified syscall.

Behavior changes

• clone3 returns ENOSYS in every sandbox.
• resource_peaks() reports a process peak of 1 when fork is not supervised. learn sets a policy_fn, so its profiles are unaffected.
• fork() and close() cost native time again, since neither round-trips through the supervisor.

Known remaining race

Accepted as is; it cannot be eliminated from userspace.

• When: only in a sandbox that still supervises fork, that is with max_processes, a policy_fn, or an exec handler. The default policy is not affected.
• What: a signal whose handler lacks SA_RESTART (dash's SIGCHLD) lands in the few microseconds between fork() entering the kernel and the supervisor receiving its notification. fork() then fails with EINTR, which it never returns natively, and dash prints "Cannot fork".
• How often: between 1 in 1000 and 1 in 400 three-stage dash pipelines across three measurements (1 of 10, 3 of 20 and 5 of 20 runs of 100 pipelines each; the last is the table above). It is a failed fork, not a hang: the hang came from close(), which is no longer notified in any mode.
• Why it stays: the kernel makes the notification wait interruptible until SECCOMP_IOCTL_NOTIF_RECV, and SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV only takes over after that. Receiving sooner does not help measurably: a build of this branch that received notifications on a dedicated thread scored the same (17 of 20 runs against 17 of 20), so that approach was dropped, together with its costs. Enforcing a process limit needs the supervisor to see each fork.
• The fix is in the kernel: restart a notification that was interrupted before it was received with -ERESTARTNOINTR instead of -ERESTARTSYS, or make the wait killable from the start, most likely behind a new filter flag.

Other notified syscalls (openat, ...) have the same window in principle. I could not provoke a failure (0 of 2000 shell redirections), but that test may not be sensitive to it.

Tests

• fork() completes while the supervisor is busy with a slow handler; fails once a process limit is added.
• clone3 with and without CLONE_NEWUSER gets ENOSYS, clone(CLONE_NEWUSER) EPERM, subprocess still works.
• Unit tests: cookie set against a reused slot, notif list with and without fork, close never notified.
• Manually: python thread pool and multiprocessing, node worker threads, JVM, Go, Rust threads and Command all run with clone3 refused; sandlock learn with a pipeline works.
• Locally: sandlock-core 777 lib and 447 integration tests pass, learn_test 40 pass. Python and Go suites were not run locally.
• Two test_control tests fail intermittently here; main fails them at the same rate (5 of 12 loops of its integration binary), so they are not from this branch.

Touches netlink/handlers.rs near the change in #234; whichever lands second needs a small rebase.

🤖 Generated with Claude Code