control: hold the sandbox name with no fd, so no fork can keep it

[congwang-mk]

A sandbox name is an abstract unix socket, and an abstract name stays bound while any fd refers to its socket. fork() copies every fd, and sandlock could only close the copies in the children it forks itself. A host that spawns processes of its own therefore holds the name of every live sandbox in each child until that child execs, and a child that never execs holds it for life. Reusing a name in that window fails with sandbox '<name>' is already running.

Under the parallel integration suite this broke test_control_name_is_free_when_wait_returns and test_control_parked_child_does_not_pin_other_names in about one run out of four: other tests spawn sandlock ps in polling loops, and each spawn sits between fork and exec for a moment with a copy of every control fd. Any embedder that spawns processes while reusing sandbox names can hit the same thing.

Supersedes #243, whose wait for the last copy to go away can block forever on a child that never execs.

Why not something simpler

Userspace cannot stop such a copy from existing. Linux has no close-on-fork, atfork handlers miss vfork, posix_spawn, raw clone and _Fork(), an abstract name cannot be unbound, and seccomp on the host could not touch the child's fd table. The remaining ways to keep the fd out of reach were a thread with a private fd table or a file lock on disk, and this had to stay thread free and diskless.

The design

The name socket is held with no fd. The child creates sandlock/<uid>/<name>, binds it, calls listen() on it, and sends it to the supervisor with SCM_RIGHTS. The supervisor leaves that message unreceived (NameVault). A file in flight is one reference, which fork does not multiply: a forked child copies the socketpair's fds, not what is queued in it. Receiving the message with no room for the fd makes the kernel drop it, so the name is free the moment the supervisor lets go, whoever copied its fds. Drop now frees the name as promptly as wait().

The roles split, because a socket held that way cannot be accepted on:

• The name socket is the mutex and carries the child's pid, which is its process group. This replaces the pgrp socket.
• Requests go to sandlock/<uid>/<name>/<child pid>. It sits in the fd table, since it needs accept(), but its name is never asked for twice, and it refuses connections from the moment it is released, so a lingering copy neither blocks anything nor looks alive.

Polling stays off the name socket. Nobody accepts on it, so every connection to it stays in its backlog until the sandbox ends, and a full backlog makes a blocking connect wait forever. Only kill needs what it offers, a child pid the kernel vouches for. ps, inspect and every other request find the request socket in /proc/net/unix, which ps reads anyway, and cost the sandbox nothing. The connect to the name socket is non-blocking, so a backlog that does fill up is an error message and not a hang.

A child that cannot publish its name gives up by itself. The child binds the name, so a collision is found just after the fork. The losing child exits at once, having run nothing, and the parent reaps it on the spot and clears its pid, so the caller is left where a failure before the fork would have left it.

The live fd registry and fork_without_control_fds are gone.

Known limits

• Every live sandbox keeps one fd in flight, and the kernel caps those per user at the sender's RLIMIT_NOFILE unless it has CAP_SYS_RESOURCE. Past that, creating a sandbox fails with an error that names the limit.
• Each kill stays in the name socket's backlog for the life of the sandbox, which caps kills of one sandbox at somaxconn (4096 by default).
• The child's pid that ps shows comes from the request socket's name, which the supervisor chose, not from a kernel stamp. kill still uses the stamp.
• If a child pid were reused while a never-exec copy still pinned the request socket named after it, binding that request socket would collide. It needs pid wraparound, and is not handled.
• If the supervisor is killed, its name lives until the host's other children exec or exit, as before.
• Requests now depend on /proc/net/unix. A nested sandlock that sees it virtualized was not tested.

What would make this unnecessary

A close-on-fork flag. POSIX.1-2024 has O_CLOFORK, FD_CLOFORK and SOCK_CLOFORK, and FreeBSD 15 and NetBSD 11 implement them. Linux does not: dup_fd() copies every fd unconditionally, and the patches posted in 2011 and 2020 were not merged.

With such a flag both sockets would go back to being ordinary fds that the supervisor binds before the fork, flagged close-on-fork, and no child of the host could copy them, however it was made. Every limit above would go with it:

• nothing is parked in flight, so there is no per-user cap;
• the supervisor accepts on the name socket again, so there is no kill budget and no lookup through /proc/net/unix, and ps reads the child's pid from a kernel stamp again;
• the names are fixed, so no pid sits in one and pid reuse cannot collide;
• a taken name is found before the fork again;
• a killed supervisor's name dies with it, since no other process ever held a copy, which is better than today.

The child would still create the socket that carries its pid and pass it up, received with MSG_CMSG_CLOFORK so that no window opens. This assumes the flag is honoured wherever an fd table is copied, which for Linux includes unshare(CLONE_FILES). Until the minimum supported kernel has such a flag, this PR is the userspace way to get the same property.

Tests

• a_fork_made_by_the_host_cannot_hold_the_name (unit) and test_control_name_survives_a_lingering_host_child (integration): a raw clone child that copies every fd and never execs is alive while the name is reused right after release. The reuse succeeds and nothing waits for the child.
• only_kernel_stamped_lookups_use_up_the_name_socket: fills the name socket's backlog, gets the error instead of a hang, and the listing path still works.
• the_in_flight_limit_is_an_error_and_leaves_the_name_free: pushes the user's in-flight count past a lowered RLIMIT_NOFILE and gets ETOOMANYREFS, with the name left free.
• test_control_name_collision_leaves_no_child: a second create() on a taken name fails, reports no pid, and leaves the owner running.
• a_released_request_socket_refuses_connections, finds_the_request_sockets_of_one_name, and the existing control tests moved to the new API.
• test_control_pgrp_published_before_extra_fd_dup2 is removed: the child no longer inherits a pgrp fd, so the collision it guarded cannot happen. What still matters is kept in test_control_created_sandbox_publishes_its_pids.
• Locally, one run each on the final commit: control unit tests 13 of 13, full integration suite 458 of 458, all CLI suites. Python (426) and Go passed on the first commit of this branch and were not rerun after the two follow-ups.

🤖 Generated with Claude Code

[congwang-mk]

Superseded by #256