[ML] Add Sandbox2 security integration for PyTorch inference - #2873
Draft
valeriy42 wants to merge 93 commits into
Draft
[ML] Add Sandbox2 security integration for PyTorch inference#2873valeriy42 wants to merge 93 commits into
valeriy42 wants to merge 93 commits into
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
valeriy42
changed the title
valeriy42
marked this pull request as draft
October 28, 2025 14:00
- Added new tests for Sandbox2 functionality, including privilege validation, filesystem isolation, and syscall filtering. - Introduced a TestCleanup class for managing temporary files during tests. - Updated the CDetachedProcessSpawner_Linux.cc to support new command line arguments for log handling and model path. - Renamed modelDir to modelPath for clarity and adjusted policy building to accommodate file access. - Improved overall test coverage for Sandbox2 features and ensured graceful degradation when Sandbox2 is not available.
- Improved error handling for cases when Sandbox2 is disabled or unavailable for pytorch_inference processes. - Enhanced logging to provide clearer feedback on spawning failures with Sandbox2. - Updated comments for clarity regarding the fallback to standard posix_spawn for non-pytorch_inference processes.
- Added a new function to apply standard ML syscall restrictions using Sandbox2's PolicyBuilder, ensuring consistent security across ML processes. - Updated the CDetachedProcessSpawner_Linux to utilize the new syscall policy for pytorch_inference, eliminating the need for seccomp filtering in this context. - Enhanced comments and documentation to outline future migration plans for other ML processes to Sandbox2. - Noted the gradual transition from seccomp filters to Sandbox2 policies in the CSystemCallFilter_Linux implementation.
- Removed SetUserAndGroup from PolicyBuilder due to updates in the sandboxed-api. - Updated AddTmpfs to include a size parameter for better resource management. - Refactored Sandbox2 instantiation to use unique_ptr for the executor, improving memory management. - Enhanced comments to clarify changes and provide context for future updates.
- Removed outdated syscall number definitions and replaced them with fallback definitions for newer syscalls, ensuring compatibility with RHEL8 headers. - Updated the handling of input pipes in the sandbox policy to allow read and write access, improving functionality. - Enhanced comments for better clarity on syscall handling and future maintenance.
…l-cpp into enhancement/sandbox2
…for Linux - Deleted Sandbox2SecurityTest.cc as it is no longer needed. - Updated CMakeLists.txt to remove references to the deleted test file. - Introduced CDetachedProcessSpawnerTest_Linux.cc, which includes tests for process spawning and integration with Sandbox2. - Enhanced CMakeLists.txt to link against Sandbox2 libraries for the new tests.
Replace the bare-host ml_test_sandbox re-run with a second docker run against ml-linux-aarch64-native-build:17 using SYS_ADMIN and relaxed seccomp/apparmor so unshare/mount(/proc) works without leaving the GCC13/Boost toolchain environment. Drop boost-host/gcc-host bundling from the aarch64 test bundle now that nothing executes off-image.
SYS_ADMIN plus unconfined seccomp/apparmor still cannot create user namespaces on aarch64 CI agents.
Nested Docker cannot unshare user namespaces even with --privileged.
Ship the complete gcc133 lib{,64} tree so the host ml_test_sandbox re-run
does not pick AlmaLinux 8 /lib64.
The aarch64 test leg failed because ML_SANDBOX2_EXPECT was a two-way assertion: the suite hard-failed whenever the declared mode and the runner's actual user-namespace support disagreed, in either direction. No agent in ml-cpp-pr-builds supports user namespaces, so declaring enforced anywhere could only fail, and the six preceding attempts to relocate the enforced run had no way to learn why the kernel refused. Replace it with a probe the tests dispatch on: - probeUserNamespaces() now reports the first stage the kernel denied - unshare(CLONE_NEWUSER), the uid/gid mapping writes, unshare(CLONE_NEWNS), mount(/, MS_REC|MS_PRIVATE) or mount(proc, /proc, proc) - together with errno, over a pipe from the probe child. - A one-line environment self-check records uid, the userns sysctls, SELinux state, and any mounts covering /proc, which is the usual reason the kernel refuses a fresh procfs inside a new user namespace. - Each test case runs when the host supports its mode and logs a reasoned skip otherwise. The allowlist drift check is mode independent and always runs. - ML_SANDBOX2_REQUIRE=enforced|fail_closed pins the mode a runner must achieve, so a characterised runner cannot silently degrade to the other mode's coverage. It is deliberately unset everywhere for now: no agent has been shown to support enforced, and that gap is recorded in the docs rather than papered over. This also fixes the inverse breakage: docker_entrypoint.sh hardcoded fail_closed, so dev-tools/docker_test.sh failed on any Docker host whose VM does permit user namespaces, such as Docker Desktop. Adds .buildkite/scripts/steps/diagnose_userns.sh, which reports the same stages on the host and in the test container without needing build artifacts, wired in as a step with no depends_on so it answers in about a minute instead of behind the 45 minute compile.
Revert before merge. SANDBOX2_CI_TRIM in .buildkite/pipeline.json.py pins PR CI to the only leg still failing and silences per-push Slack and email notifications, cutting an iteration from ~90 to ~50 minutes. The pytest guard is read from the source flag so it is deleted in the same revert.
The probe was a false negative. mount("proc", "/proc", "proc", ...) inside a
user namespace returns EPERM unless the PID namespace is new as well: the
kernel refuses a procfs instance for a PID namespace that already has one
mounted. Sandbox2 unshares CLONE_NEWPID, the probe did not, so the probe
reported "user namespaces unavailable" on hosts where Sandbox2 can in fact
start - and every attempt to relocate the enforced run was chasing that.
Verified both ways on a 7.0 kernel in a privileged linux/arm64 container:
unshare -U -m --map-root-user sh -c 'mount -t proc proc /proc'
mount: /proc: permission denied
unshare -U -m --map-root-user --pid --fork sh -c 'mount -t proc proc /proc'
MOUNTPROC_OK
So the probe now unshares CLONE_NEWNS|CLONE_NEWPID and performs the mounts in
a fork that is PID 1 of the new namespace, because CLONE_NEWPID only affects
children. Two new reported stages distinguish the unshare from that fork.
The first diagnostic run on the aarch64 agent produced the evidence behind
this. It cleared unshare(CLONE_NEWUSER), the id mapping and CLONE_NEWNS, and
failed only at mount(proc) - so user namespaces are available there
(user.max_user_namespaces=31905) and only the missing PID namespace stood in
the way. It also showed /proc/sys/fs/binfmt_misc covering /proc on the host,
and that Docker's default seccomp profile denies CLONE_NEWUSER outright, which
is why the plain container failed at the first stage instead of the last.
Diagnostics grow accordingly:
- userns_probe.sh carries the per-runner body, so host and container results
are directly comparable, and reports each stage's stderr - an exit status
alone cannot tell a kernel denial from a missing binary.
- diagnose_userns.sh runs it on the host and across five container
configurations, ordered least to most privileged, to find the cheapest one
that clears every stage.
- Both record mounts covering /proc per runner, and a contrast case keeps the
CLONE_NEWPID requirement visible rather than looking like a stray flag.
Also gates the x86_64 debug build/test pair on --build-x86_64. It was appended
regardless of the requested architecture, so an aarch64-only build dragged an
x86_64 pair along with it.
diagnose_userns.sh on core-almalinux-8-aarch64 (kernel 4.18) now clears every stage on the host, so the enforced-only cases - a real sandboxed spawn and the filesystem-policy differential - can run there with no privilege escalation. Pinned with ML_SANDBOX2_REQUIRE=enforced so a silent downgrade to fail-closed coverage fails the build instead of passing quietly. Override with ML_SANDBOX2_HOST_REQUIRE if an agent ever loses namespace support. Measured runner matrix: host all stages OK docker (default) denied at unshare(CLONE_NEWUSER) docker + seccomp=unconfined denied at mount(proc), masked /proc paths docker + seccomp + systempaths all stages OK docker --privileged all stages OK The container run stays unpinned on purpose: Docker's default seccomp profile denies CLONE_NEWUSER, so it supplies the fail-closed half that no other runner here provides, and if a Docker upgrade ever permits CLONE_NEWUSER it will cover enforced instead while the host pin still guarantees enforced coverage exists. Also drops the generic ML_SANDBOX2_REQUIRE forward into the container, which would have pushed the host's value into a runner where it is wrong.
The enforced re-run executes from the build tree, but nothing exports
CPP_SRC_HOME there - set_env.sh does it inside the container only. So
CResourceLocator::cppRootDir() took its fallback of "../../..", which assumes
the cwd is a source unittest directory, and the spawn test looked for
pytorch_inference under cmake-build-docker/ instead of the repository root:
CSandboxedProcessSpawnerTest_Linux.cc(366): fatal error: pytorch_inference
binary not found under ../../../build/distribution/platform/linux-{x86_64,aarch64}/bin/
Under ctest the working directory is the source unittest directory, which is
why the same lookup succeeds in the container run.
testPolicyViolationDifferential passed in the same run - a real sandboxed spawn
on the host, with Sandbox2 terminating the child on a syscall violation - so
enforced coverage does work on this agent now that the probe reports it.
…andbox2" This reverts commit 4439aba.
The enforced coverage pinned in run_tests.sh fails the build if the agent loses user-namespace support. This step is what then says which stage broke, so it stops being temporary scaffolding and becomes the companion to the pin.
Contributor
Author
|
buildkite run_qa_tests |
…rgDirs Four review threads that survived the MlSandbox restructure, plus the duplication that restructure left behind. extractArgDirs was byte-identical in CPytorchInferenceSandboxPolicy.cc and CPytorchInferenceSandboxPolicy_Linux.cc. There was no link error because ml_generate_platform_sources substitutes the _Linux file for the base rather than adding it, so the base copy only ever compiled on macOS - which is also why the same lastSlash fix had to be applied to both copies. The Sandbox2 code in the _Linux file was already guarded by SANDBOX2_AVAILABLE, and that is only defined on Linux, so the two files merge into one shared implementation. linux/futex.h moves inside the guard to keep the merged file portable, since its FUTEX_ constants are only used by the policy builder. The Windows file stays: realpath and PATH_MAX are genuinely absent there. Range-for loops over vectors and sets now spell out const std::string& rather than const auto&, in all seven places rather than only the three that were commented on. libDir now says that it is binDir's sibling and not its child. The suggestion to simplify it to binDir + "/lib" would have broken the mount, which is itself evidence that the line needed a comment. The bare scope block around the monitor thread is gone. It held no lock and no RAII object, so it scoped nothing. spawn() is 116 lines, down from 330 at review time. buildSandboxeeEnvironment, makeConfiguredExecutor and logSandboxeeTermination are named steps in the anonymous namespace now, and the nested #ifdefs the review objected to had already been replaced by a single annotated guard.
Fail-closed coverage could disappear silently. The container run was left unpinned on the reasoning that Docker's default seccomp profile denies unshare(CLONE_NEWUSER), so the sandbox tests would select fail_closed there. That external default was neither asserted nor guaranteed: were it to change, both runners would select enforced, testFailClosedWhenUserNamespacesUnavailable would return without executing, and CI would stay green with the fallback path untested. Calling a loud failure on a Docker upgrade 'brittle' was the wrong trade - losing the coverage quietly is worse. Both modes are now pinned on the host. The fail-closed run happens inside a user namespace whose user.max_user_namespaces has been set to 0. That sysctl is per-user-namespace, so exhausting it in a namespace we own denies every further user namespace in that subtree: clone(CLONE_NEWUSER), which Sandbox2 uses, as well as unshare(CLONE_NEWUSER), which the test's probe uses. Both then fail with ENOSPC, verified on kernels 4.18 and 7.0. A seccomp profile would have been the obvious alternative but is worse here. One denying only unshare(2) leaves clone(CLONE_NEWUSER) working, so the probe would report a capability Sandbox2 does not have; filtering clone's flags argument correctly is architecture-specific, and hand-rolling that is exactly the part Docker's default profile already gets right. The sysctl needs no profile and is architecture-independent. Coverage no longer depends on Docker's defaults in either direction.
The test asserted only that the forbidden $HOME file was absent after the sandboxed child exited. A child terminated by the syscall policy before it ran anything produces exactly that state, so the assertion held without the filesystem policy being exercised at all - and that is what happens: the shell calls getpgid during its own startup, which the pytorch_inference allowlist does not permit. From the passing aarch64 host run in build 3064: SANDBOX VIOLATION : PID: 7977, PROG: 'bash' : [Arm-64] getpgid [155] Unit test timing - CSandboxedProcessSpawnerTest/testPolicyViolationDifferential So the test proved the syscall policy stops a foreign binary, not that the filesystem policy blocks a write. A sentinel under /tmp - a directory the policy does mount read-write - now makes that explicit. The payload touches the sentinel before the forbidden path, the positive control asserts both appear unsandboxed, and the sandboxed case asserts neither does. If Sandbox2 ever lets a foreign binary start, the sentinel appears and the test fails, which is the signal that filesystem-policy coverage needs its own payload. That payload is the remaining gap, recorded in the failure-modes doc: it needs syscalls entirely inside the pytorch_inference allowlist, so a purpose-built helper rather than a shell. Adding getpgid to the allowlist to make a shell survive would weaken the production policy to suit a test.
…pawner spawn() detached a monitor thread that captured a raw this and, once the sandboxee exited, took the spawner's mutex and erased from its PID set. The destructor was defaulted and synchronised nothing, so tearing the spawner down while a sandboxed pytorch_inference was still running left the monitor writing through a dangling pointer whenever the sandboxee later exited. That window is wide in practice: the monitor exists precisely because a sandboxee outlives the spawn call, and it is a long-lived daemon. Joining in the destructor is not an option - it would block controller shutdown until pytorch_inference exits. Instead the mutex and PID set move into an SPidRegistry held by shared_ptr, and the monitor co-owns it. The registry then outlives the spawner exactly as long as some monitor still needs it, and the destructor stays trivial. Also drops the now-unnecessary raw this capture from the lambda.
extractArgDirs treated every key=value argument as a pipe path candidate and reported anything whose value did not start with '/' as rejected. Real pytorch_inference invocations always carry scalar options - --namedPipeConnectTimeout=1, --numAllocations=1, --validElasticLicenseKeyConfirmed=true, --modelid=... - so each of them was listed as "(path not absolute)" on every spawn. The failure-modes doc tells operators to check that rejectedPipeArgs is empty when triaging FIFO and mount problems, and that could never be true, which made the primary triage signal useless. The pipe options cannot be told apart by name: --input, --output and --restore carry paths while --inputIsPipe and friends are bare flags. A value containing no '/' cannot be a pipe path, so those are now skipped silently, and a value that does contain one but is relative is still reported - which is the case the list was meant to catch.
…cation spawnSandboxedWithTimeout captured everything by reference and, on timeout, detached the spawn thread and returned. The thread then kept writing spawnResult and spawnFinished in a frame that had gone away, plus childPid and failureReason in the caller's, and kept using a spawner reference that the test was about to destroy. Detaching is the right way to avoid the hang that ~std::future would cause, but everything the thread touches has to outlive the call: the results now live in a heap-allocated SSpawnAttempt the thread co-owns, the spawner is passed as a shared_ptr, and the path and arguments are copied in. The caller reads the results out only on the success path. Also adds the first tests for extractArgDirs, which had none: that scalar options are ignored rather than reported as rejected, that absolute pipe paths contribute their directory, and that relative and root-level paths are still rejected with the right reason.
extractArgDirs now mounts only input/output/restore/logPipe values and rejects non-absolute paths on those options instead of inferring from value shape, so scalars like --modelid=/etc/passwd never grant mounts.
Replace PID set with generation-tagged child map holding shared_ptr<Sandbox2> and pidfd. Monitor erases before logging with generation check; thread-start failure kills sandbox and cleans registry. terminateChild uses pidfd_send_signal (SIGTERM) with ::kill fallback on kernels without pidfd.
Move the "Spawned" INFO to after successful monitor detach and set childPid to 0 in the std::thread constructor catch path so false returns match the out-param contract of other spawn() failures.
… catch Before assigning a new s_Children entry, close any existing pidfd for the same PID to avoid fd leaks on reuse. Catch std::exception (not only std::system_error) around monitor thread creation so bad_alloc takes the same Kill/AwaitResult cleanup path.
…write The paragraph still described the earlier heuristic - treat a value containing a '/' as a path candidate - which e624f1b replaced with an explicit list of the four path-bearing options. The distinction is visible to an operator: --input=foo is now reported as "(path not absolute)" rather than skipped, and a scalar option is skipped by name whatever its value looks like.
- Wrap the pidfd in an RAII owner and reap the sandboxee if registry insertion throws, so a bad_alloc cannot leak the fd or orphan the process. - Retain the monitor thread in a named variable and join it during cleanup so a detach() failure cannot destroy a joinable thread and std::terminate. - Hardcode ML_SANDBOX2_REQUIRE=enforced in the host re-run; drop the ML_SANDBOX2_HOST_REQUIRE override that let CI pass without enforced coverage. - Add an abandon/terminate handshake to spawnSandboxedWithTimeout so a spawn that succeeds after the test's timeout terminates its sandboxee.
After the policy file merge, comments still pointed at the deleted _Linux.cc path. Update allowlist/sync pointers to CPytorchInferenceSandboxPolicy.cc and attribute ML_SANDBOXED=1 to CSandboxedProcessSpawner_Linux.cc.
valeriy42
marked this pull request as draft
September 8, 2026 19:05
Contributor
Author
|
Frozen at Last green integration evidence: Buildkite #3081. This branch will not be rebased, force-pushed, or merged, and its history is preserved. The Sandbox2 integration is being reconstructed from current
Why replace rather than repair: the recurring defects in this branch share one cause — there was no explicit capability graph and no resource lifecycle state machine, while tests frequently asserted an outcome without proving the mechanism that produced it. Slicing the work lets each boundary be reviewed against a stated contract instead of inferred from a large diff. Security logic is rebuilt from that contract rather than bulk-cherry-picked; only audited vendoring and build material is transplanted mechanically. The established tab-separated This PR stays open and will be closed as superseded once every replacement PR is linked here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PyTorch inference runs untrusted TorchScript models supplied by Elasticsearch users, so the native process must be treated as hostile. Today
pytorch_inferencerelies largely on in-process seccomp and graph validation, but that still leaves a large syscall and filesystem surface exposed inside the same address space as libtorch. This PR hardens the Linux production path by spawningpytorch_inferenceinside Google Sandbox2 from the ML controller, so isolation, syscall policy, and filesystem access are enforced before the model binary starts executing.Architecture
MlSandbox(lib/sandbox/): ownsCSandboxedProcessSpawner,CPytorchInferenceSandboxPolicy, andCSandbox2Diagnostics.SANDBOX2_AVAILABLEis scoped to this library, notMlCore.CProcessSpawnerRouter(bin/controller/): controller-owned routing layer. Nominates./pytorch_inferenceexplicitly, strips--disableSandbox, and chooses sandboxed vs legacy spawn based on the ES kill switch (xpack.ml.trained_models.sandbox_enabled).CDetachedProcessSpawner(lib/core/): generic POSIXposix_spawnspawner again — nopytorch_inference-specific dispatch or Sandbox2 knowledge.pytorch_inference: runtime-only seccomp gate on Linux whenML_SANDBOXED=1; skips in-process BPF when running under Sandbox2.Build support vendors Abseil and the Sandboxed API on Linux, wires them into CMake, and adds the required license files.
Kill switch
When sandboxing cannot be enforced (e.g. user namespaces unavailable) or the operator disables it, the controller falls back to the legacy
posix_spawn+ in-process seccomp path via--disableSandbox.Testing
lib/sandbox/unittest: spawn, allowlist drift guard, fail-closed behaviour, and policy-violation differential tests.enforced(real sandboxed spawn/terminate plus the filesystem-policy differential) orfail_closed(spawn refusal plus kill-switch hint) accordingly. The allowlist drift guard is mode independent and always runs.ML_SANDBOX2_REQUIRE=enforcedpins the aarch64 host runner so enforced coverage cannot silently degrade; the container run is left unpinned and supplies the fail-closed half..buildkite/scripts/steps/diagnose_userns.sh: reports which user-namespace stage a runner denies, on the host and across container configurations, without needing a build. Kept as a regression canary for the pin above.test/test_sandbox2_attack_defense.py: end-to-end attack-defense harness.CPytorchInferenceSyscallAllowlist.h.Review sequence (post-restructure)
lib/sandbox/CSandboxedProcessSpawner_Linux.cc+CPytorchInferenceSandboxPolicy_Linux.cc— spawn and policy.bin/controller/CProcessSpawnerRouter.cc— kill-switch routing.lib/core/CDetachedProcessSpawner.cc— generic legacy spawn baseline.lib/seccomp/CSystemCallFilter_Linux.cc+include/seccomp/CPytorchInferenceSyscallAllowlist.h— legacy vs Sandbox2 syscall coherence.bin/pytorch_inference/Main.cc— runtime seccomp gate.3rd_party/CMakeLists.txt,lib/sandbox/CMakeLists.txt,lib/core/CMakeLists.txt— build wiring.lib/sandbox/unittest/CSandboxedProcessSpawnerTest_Linux.cc— the user-namespace probe and the enforced/fail-closed dispatch it drives;test/test_sandbox2_attack_defense.py..buildkite/scripts/steps/run_tests.sh(where each mode is covered),diagnose_userns.sh+userns_probe.sh.docs/changelog/2873.yaml,docs/sandbox2_production_failure_modes.md.ES companion PRs