Trust and proxy delivery into boxes

.github/workflows/on-pull-request.yml

@@ -145,6 +145,7 @@ jobs:
       DEFAULT_HOST_PROVIDER: docker
       # The sandbox sends its HTTPS through a proxy on the Docker bridge; the API test only checks delivery.
       SECRETS_PROXY_URL: http://172.17.0.1:8880
+      SECRETS_PROXY_CA_FILE: /secrets-proxy-ca.pem
       DOCKER_DEFAULT_IMAGE: drukbox/sandbox:ci
       UVICORN_HOST: 127.0.0.1
 
@@ -163,12 +164,14 @@ jobs:
           docker run --name drukbox-api-test --detach \
             --network host \
             --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
+            --mount type=bind,src=$PWD/env/test-proxy-ca.pem,dst=/secrets-proxy-ca.pem,readonly \
             --group-add "$(stat -c '%g' /var/run/docker.sock)" \
             --env DATABASE_URL \
             --env SECRETS_KEY \
             --env SERVICE_TOKENS \
             --env DEFAULT_HOST_PROVIDER \
             --env SECRETS_PROXY_URL \
+            --env SECRETS_PROXY_CA_FILE \
             --env DOCKER_DEFAULT_IMAGE \
             --env UVICORN_HOST \
             drukbox:api-test \

docs/add-a-provider.md

@@ -82,12 +82,15 @@ installs `--all-extras`, so the shipped image carries every provider.
 
 ## 6. Optional capabilities
 
-Every provider gets the proxy in `secrets` by default. The sandbox gets a placeholder and the proxy address in `env`, like every
-other variable, and the provider has one job: put `env` where every session
-can read it. On Linux that is `/etc/environment`, which PAM reads at login.
-`providers.environment.persist` writes it. The cloud providers get this from
-`environment.cloud_init`. A provider with a secret store of its own, such as
-docker-sbx, implements `SecretInjectionCapability` itself.
+Every provider gets the proxy in `secrets` by default. The sandbox gets a
+placeholder, the proxy address, and the proxy's CA in `env`, like every other
+variable. The provider has two jobs: put `env` where every session can read it,
+and install the CA. On Linux the first is `/etc/environment`, which PAM reads
+at login, and `providers.environment.get_persist` writes it.
+`providers.environment.get_install_ca` writes the second, with `sudo` when the script
+runs as a user. The cloud providers get both from `environment.get_cloud_init`. A
+provider with a secret store of its own, such as docker-sbx, implements
+`SecretInjectionCapability` itself.
 
 Do not add provider-specific fields to the host schema. Add a capability
 instead.

docs/architecture.md

@@ -126,7 +126,8 @@ and the service, `drk.<host id>.<service>.<random>`. The entry keeps only a
 fingerprint of the random part. The sandbox receives the auth variable with
 the placeholder in its boot environment, next to the caller's `env`. On every
 provider but docker-sbx it also receives `HTTPS_PROXY`, `https_proxy`, and
-`NO_PROXY`, so it sends its HTTPS through the proxy at `SECRETS_PROXY_URL`.
+`NO_PROXY`, so it sends its HTTPS through the proxy at `SECRETS_PROXY_URL`,
+and the proxy's CA in `SECRETS_PROXY_CA`, which it installs at boot.
 The proxy is the official mitmproxy image with the addon in `deploy/proxy`.
 It terminates TLS only for the hosts the exchange lists at `/upstreams`, the
 hosts with a registered secret, and tunnels every other host blind. For a

docs/deploy.md

@@ -313,8 +313,25 @@ volumes:
 ```
 
 The exchange binds `0.0.0.0` inside the compose network and publishes no
-port, so only the proxy reaches it. The public certificate of the CA in the
-volume is what a sandbox must trust for the hosts with a secret. Keep `flow_detail` at `1` or below. A
+port, so only the proxy reaches it. The API reads the public certificate of
+the CA from the same volume, at `SECRETS_PROXY_CA_FILE`, and hands it to
+every sandbox with secrets:
+
+```yaml
+  api:
+    image: ghcr.io/czpython/drukbox:latest
+    env_file: drukbox.env
+    environment:
+      SECRETS_PROXY_URL: http://proxy.example:8880
+      SECRETS_PROXY_CA_FILE: /secrets-proxy-ca/mitmproxy-ca-cert.pem
+    volumes:
+      - secrets-proxy-ca:/secrets-proxy-ca:ro
+```
+
+A sandbox with secrets gets the certificate in `SECRETS_PROXY_CA`, base64,
+and installs it at boot with `update-ca-certificates`. `SSL_CERT_FILE`,
+`REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, and `NODE_EXTRA_CA_CERTS` point curl,
+Python, and Node at it. Keep `flow_detail` at `1` or below. A
 higher level prints request headers, and after the swap those carry the
 real credential. Real credentials exist in three places only. They are
 encrypted in Postgres, they pass through the exchange process for one
@@ -409,6 +426,7 @@ Secrets exchange:
 | Variable | Default | Purpose |
 | --- | --- | --- |
 | `SECRETS_PROXY_URL` | — | Proxy a sandbox sends its HTTPS through. Required to create a host with secrets on every provider but docker-sbx. |
+| `SECRETS_PROXY_CA_FILE` | — | Path of the proxy's public CA certificate, from the proxy's volume. Required with `SECRETS_PROXY_URL`. |
 | `SECRETS_EXCHANGE_BIND_HOST` | `127.0.0.1` | Interface the exchange process binds. Bind it where only the proxy can reach it. |
 | `SECRETS_EXCHANGE_PORT` | `8781` | Port the exchange process listens on. |
 

docs/security.md

@@ -113,9 +113,9 @@ header on the host the exchange approved. The proxy refuses a destination
 that resolves to a loopback, private, link-local, or metadata address, so a
 sandbox cannot reach the exchange or the API through it. Bind the exchange
 process where only the proxy can reach it, because its answer is the real
-credential. The sandbox trusts the proxy's CA for the hosts with a secret, so
-whoever holds the CA key can impersonate those hosts to the sandbox. The key
-lives in the proxy's volume.
+credential. A sandbox with secrets installs the proxy's CA at boot, so whoever
+holds the CA key can impersonate any host to that sandbox. The key lives in
+the proxy's volume. The API reads the public certificate only.
 
 Two pieces of material reach the VM through its provider's user-data /
 setup-script mechanism, and that channel is the relevant exposure:

env/test-proxy-ca.pem

@@ -0,0 +1,11 @@
+-----BEGIN CERTIFICATE-----
+MIIBlDCCATugAwIBAgIUb9KZNOdLcC8Znj76t4PPmXdb/p4wCgYIKoZIzj0EAwIw
+IDEeMBwGA1UEAwwVZHJ1a2JveCB0ZXN0IHByb3h5IENBMB4XDTI2MDkwNjE0MjQ1
+NFoXDTM2MDkwMzE0MjQ1NFowIDEeMBwGA1UEAwwVZHJ1a2JveCB0ZXN0IHByb3h5
+IENBMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWW4OH6QzHfZC7kGytzZaLtK4
+TK8bwFaSGy3GO8K/1uMeIQrguXXeUrqHw61nybKeKAsR5j78hkapTZI6wL4mw6NT
+MFEwHQYDVR0OBBYEFHvLb1C5RRD7Ruvyz/cH2JfHdBQjMB8GA1UdIwQYMBaAFHvL
+b1C5RRD7Ruvyz/cH2JfHdBQjMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwID
+RwAwRAIgWK/1s71gqFNXOh+jnhn769pnesgK7EnFL/5R0grBGrgCIAaQ5HCntmqG
+fFZMJLq5HZtCnw/ri3nvlY1fweTudaRI
+-----END CERTIFICATE-----

env/test.env

@@ -9,3 +9,4 @@ TAILSCALE_OAUTH_CLIENT_ID=tailscale-client-id
 TAILSCALE_OAUTH_CLIENT_SECRET=tailscale-client-secret
 TAILSCALE_TAILNET=example.ts.net
 SECRETS_PROXY_URL=http://proxy.test:8880
+SECRETS_PROXY_CA_FILE=env/test-proxy-ca.pem

images/local/entrypoint.sh

@@ -1,6 +1,5 @@
 #!/usr/bin/env bash
-# First-boot entrypoint for the drukbox local sandbox container. Seeds the
-# per-host public key, persists caller env for SSH sessions, then runs sshd.
+# First boot: the host key, the caller env, then sshd.
 set -euo pipefail
 
 : "${DRUKBOX_AUTHORIZED_KEY:?DRUKBOX_AUTHORIZED_KEY is required}"
@@ -9,12 +8,17 @@ install -d -m 700 /root/.ssh
 printf '%s\n' "$DRUKBOX_AUTHORIZED_KEY" > /root/.ssh/authorized_keys
 chmod 600 /root/.ssh/authorized_keys
 
-# Persist the caller-supplied env vars so interactive SSH sessions see them:
-# pam_env reads /etc/environment. Each name is also a real container env var.
+# pam_env reads /etc/environment.
 for name in ${DRUKBOX_ENV_KEYS:-}; do
   printf '%s=%s\n' "$name" "${!name-}" >> /etc/environment
 done
 
+# A box with secrets gets the proxy's CA in SECRETS_PROXY_CA. Install it first.
+if [ -n "${SECRETS_PROXY_CA:-}" ]; then
+  printf '%s' "$SECRETS_PROXY_CA" | base64 -d > /usr/local/share/ca-certificates/drukbox.crt
+  update-ca-certificates >/dev/null
+fi
+
 # Generate host keys if the image doesn't ship any.
 ssh-keygen -A
 

src/core/settings.py

@@ -56,6 +56,14 @@ class Settings(BaseSettings):
             "The proxy swaps the sandbox's placeholder for the real credential."
         ),
     )
+    secrets_proxy_ca_file: str = Field(
+        default="",
+        validation_alias="SECRETS_PROXY_CA_FILE",
+        description=(
+            "Path of the proxy's public CA certificate, from the proxy's volume. "
+            "A sandbox with secrets installs it at boot."
+        ),
+    )
 
     service_label: str = Field(
         default="drukbox",

src/host_secrets/tests/test_api.py

@@ -94,8 +94,11 @@ async def test_a_secret_the_exchange_cannot_serve_is_refused(
     assert "one" not in response.text.replace("exactly one", "")
 
 
-async def test_secrets_on_a_proxy_provider_need_the_proxy_address(client, monkeypatch) -> None:
-    monkeypatch.setattr(get_settings(), "secrets_proxy_url", "")
+@pytest.mark.parametrize("setting", ["secrets_proxy_url", "secrets_proxy_ca_file"])
+async def test_secrets_on_a_proxy_provider_need_the_proxy_and_its_certificate(
+    client, monkeypatch, setting: str
+) -> None:
+    monkeypatch.setattr(get_settings(), setting, "")
 
     response = await client.post(
         "/hosts", headers=AUTH_HEADERS, json={"secrets": {"anthropic": {"value": "sk-ant-real"}}}

src/hosts/schemas.py

@@ -89,7 +89,7 @@ def reject_reserved_env_keys(cls, env: dict[str, str]) -> dict[str, str]:
             raise ValueError(f"reserved env keys are not allowed: {', '.join(reserved_keys)}")
         # Every provider persists env through /etc/environment. persist raises
         # for any entry pam_env would change or drop.
-        environment.persist(env)
+        environment.get_persist(env)
         return env
 
     _validate_expires_at = field_validator("expires_at")(_expires_at_must_be_future_and_tz_aware)

src/hosts/service.py

@@ -28,6 +28,7 @@
     Tailscale,
 )
 from providers.base import VMProvider
+from providers.environment import get_persist
 from providers.exceptions import (
     ProviderCommandError,
     ProviderError,
@@ -233,9 +234,11 @@ async def create_host(
             raise UnsupportedSizingError(
                 f"provider {vm.name!r} does not support a per-request disk_gb"
             )
-        if secrets and not vm.secrets.needs_value and not self.settings.secrets_proxy_url:
+        proxy = self.settings.secrets_proxy_url and self.settings.secrets_proxy_ca_file
+        if secrets and not vm.secrets.needs_value and not proxy:
             raise SecretsProxyNotConfiguredError(
-                "SECRETS_PROXY_URL must name the proxy that sandboxes dial"
+                "SECRETS_PROXY_URL and SECRETS_PROXY_CA_FILE must name the proxy that "
+                "sandboxes dial and its certificate"
             )
         if template and not image:
             image = await self._resolve_template_image(template_id=template, provider=vm.name)
@@ -506,7 +509,8 @@ async def provision(self, host_id: str) -> None:
 
         try:
             environment.update(await self.put_secrets(host, vm))
-        except (IssuerError, ProviderError) as exc:
+            get_persist(environment)
+        except (IssuerError, ProviderError, ValueError) as exc:
             await self.mark_failed(host, exc)
             return
 

src/hosts/tests/test_secrets.py

@@ -52,6 +52,7 @@ async def delete_secret(self, *, vm: str, placeholder: Placeholder) -> None:
 def settings(monkeypatch: pytest.MonkeyPatch) -> Generator[Settings, None, None]:
     monkeypatch.setenv("TAILSCALE_ENABLED", "false")
     monkeypatch.setenv("SECRETS_PROXY_URL", "http://proxy.test:8880")
+    monkeypatch.setenv("SECRETS_PROXY_CA_FILE", "env/test-proxy-ca.pem")
     get_settings.cache_clear()
     yield get_settings()
     get_settings.cache_clear()
@@ -126,6 +127,8 @@ async def test_a_proxy_box_boots_with_its_placeholders_and_the_proxy_and_no_valu
     assert environment["KEEP"] == "me"
     assert environment["HTTPS_PROXY"] == environment["https_proxy"] == "http://proxy.test:8880"
     assert environment["NO_PROXY"].startswith("localhost,")
+    assert environment["SECRETS_PROXY_CA"]
+    assert environment["NODE_EXTRA_CA_CERTS"] == "/usr/local/share/ca-certificates/drukbox.crt"
     for name, variable in (("anthropic", "ANTHROPIC_AUTH_TOKEN"), ("github", "GH_TOKEN")):
         placeholder = Placeholder.read(environment[variable])
         assert (placeholder.host_id, placeholder.service) == (host.id, name)
@@ -155,6 +158,29 @@ async def test_a_provider_that_holds_the_value_gets_it_at_boot(
     assert "HTTPS_PROXY" not in environment
 
 
+class OversizedInjection(RecordingInjection):
+    """Hands the box a value that pam_env cannot read."""
+
+    async def put_secret(
+        self, *, vm: str, service: Service, placeholder: Placeholder, value: str
+    ) -> dict[str, str]:
+        return {service["auth_variable"]: "a" * 9000}
+
+
+async def test_a_boot_environment_pam_cannot_read_fails_provisioning_before_the_vm(
+    settings: Settings, create_vm: AsyncMock, stub_provider: StubVMProvider
+) -> None:
+    stub_provider.secrets = OversizedInjection()
+
+    async with async_session_factory() as session:
+        with pytest.raises(ProvisioningFailedError, match="longer than 8191 bytes"):
+            await HostService(session, settings=settings).create_host(
+                env={}, secrets={"anthropic": {"value": "sk-ant-real"}}, image=None, provider="stub"
+            )
+
+    create_vm.assert_not_awaited()
+
+
 @respx.mock
 async def test_an_issuer_that_gives_no_value_at_boot_fails_provisioning(
     settings: Settings, create_vm: AsyncMock, stub_provider: StubVMProvider

src/providers/aws/provider.py

@@ -107,7 +107,7 @@ async def create_vm(
                 await self.api.delete_key_pair(key_name)
                 raise ProviderTransportError(str(exc)) from exc
 
-        user_data = environment.cloud_init(setup_script or "", env)
+        user_data = environment.get_cloud_init(setup_script or "", env)
         try:
             instance_id = await self.api.run_instance(
                 client_token=name,

src/providers/capabilities.py

@@ -1,15 +1,23 @@
 import abc
+import base64
+import pathlib
 from typing import ClassVar, TypeVar
 
+from cryptography import x509
+from cryptography.hazmat.primitives.serialization import Encoding
+
 from core.settings import get_settings
 from host_secrets.catalog import Service
 from host_secrets.placeholder import Placeholder
-from providers.exceptions import CapabilityUnsupportedError
+from providers import environment
+from providers.exceptions import CapabilityUnsupportedError, ProviderCommandError
 
 CapabilityT = TypeVar("CapabilityT")
 
 # The box itself and the cloud metadata address.
 NO_PROXY = "localhost,127.0.0.1,::1,169.254.169.254"
+# update-ca-certificates writes the system bundle here.
+SYSTEM_CA_BUNDLE = "/etc/ssl/certs/ca-certificates.crt"
 
 
 def resolve_capability(provider, capability: type[CapabilityT]) -> CapabilityT:
@@ -42,7 +50,8 @@ async def delete_secret(self, *, vm: str, placeholder: Placeholder) -> None: ...
 
 
 class ProxyInjection(SecretInjectionCapability):
-    """The box sends its HTTPS through the proxy, which swaps the placeholder."""
+    """The box sends its HTTPS through the proxy, which swaps the placeholder.
+    It gets the proxy's CA and the trust variables."""
 
     needs_value = False
 
@@ -54,17 +63,39 @@ async def put_secret(
         placeholder: Placeholder,
         value: str,
     ) -> dict[str, str]:
-        proxy = get_settings().secrets_proxy_url
+        settings = get_settings()
         return {
             service["auth_variable"]: str(placeholder),
-            "HTTPS_PROXY": proxy,
-            "https_proxy": proxy,
+            "HTTPS_PROXY": settings.secrets_proxy_url,
+            "https_proxy": settings.secrets_proxy_url,
             "NO_PROXY": NO_PROXY,
+            environment.PROXY_CA: base64.b64encode(self.get_public_certificate()).decode(),
+            "SSL_CERT_FILE": SYSTEM_CA_BUNDLE,
+            "REQUESTS_CA_BUNDLE": SYSTEM_CA_BUNDLE,
+            "CURL_CA_BUNDLE": SYSTEM_CA_BUNDLE,
+            "NODE_EXTRA_CA_CERTS": environment.PROXY_CA_PATH,
         }
 
     async def delete_secret(self, *, vm: str, placeholder: Placeholder) -> None:
         return
 
+    def get_public_certificate(self) -> bytes:
+        """mitmproxy writes the key and the certificate into one file and the
+        certificate alone into another. Only the second may reach a box."""
+        path = get_settings().secrets_proxy_ca_file
+        try:
+            pem = pathlib.Path(path).read_bytes()
+            certificate = x509.load_pem_x509_certificate(pem)
+        except (OSError, ValueError) as exc:
+            raise ProviderCommandError(
+                f"SECRETS_PROXY_CA_FILE does not name a readable certificate: {exc}"
+            ) from exc
+        if b"PRIVATE KEY" in pem:
+            raise ProviderCommandError(
+                "SECRETS_PROXY_CA_FILE holds a private key. Name the public certificate only."
+            )
+        return certificate.public_bytes(Encoding.PEM)
+
 
 class TemplateCapability(abc.ABC):
     """An ABC, not a Protocol, so ``resolve_capability`` can test inheritance."""

src/providers/docker_sbx/provider.py

@@ -34,7 +34,7 @@ def _bootstrap_script(*, public_key: str, env: dict[str, str], ssh_username: str
         f"chown {owner}:{owner} {home}/.ssh/authorized_keys",
     ]
     # The runtime takes no environment at create time. pam_env reads this file.
-    return "\n".join([*lines, *environment.persist(env)]) + "\n"
+    return "\n".join([*lines, *environment.get_persist(env)]) + "\n"
 
 
 class DockerSbxProvider(VMProvider, TemplateCapability):
@@ -94,7 +94,7 @@ async def create_vm(
 
         caller_env = env or {}
         try:
-            environment.persist(caller_env)
+            environment.get_persist(caller_env)
         except ValueError as exc:
             raise ProviderCommandError(str(exc)) from exc