From b14412f100383fd183a411890b83cd7109acdb6b Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 16:12:26 -0700 Subject: [PATCH 1/2] docs(agentcompute): record service deployment and Studio access --- .../docs/reference/networking/address-plan.md | 1 + .../reference/networking/tailscale-policy.md | 15 + docs/docs/runbooks/agentcompute-service.md | 580 ++++++++++++++++++ docs/docs/runbooks/agentcompute-studio-ssh.md | 313 ++++++++++ docs/mkdocs.yml | 2 + 5 files changed, 911 insertions(+) create mode 100644 docs/docs/runbooks/agentcompute-service.md create mode 100644 docs/docs/runbooks/agentcompute-studio-ssh.md diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index 97ac9ee..f49c1b3 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -67,6 +67,7 @@ by the `GilmanLab/fleet` `cluster/` project and mirrored in each node's seed. | `sw-mgmt01` management | `10.10.70.2` | Interface address | | `gw01` `glab.lol` mirror | `10.10.10.54` | Local service address | | `ovncentral01` OVN central | `10.10.10.15` | Static VM interface on `nas01`'s unmanaged `mgmt` bridge | +| `agentcompute01` MCP service | `10.10.10.16` | Static `/32` on a routed Incus NIC through `lab01`'s `_vmgmt` interface; host gateway `169.254.0.1` | ### Hosts diff --git a/docs/docs/reference/networking/tailscale-policy.md b/docs/docs/reference/networking/tailscale-policy.md index a137f09..dc0e0f7 100644 --- a/docs/docs/reference/networking/tailscale-policy.md +++ b/docs/docs/reference/networking/tailscale-policy.md @@ -43,6 +43,7 @@ and trailing commas. Section syntax is documented in Tailscale's | --- | --- | --- | | `tag:subnet-router` | Devices that advertise lab and home subnet routes into the tailnet | `autogroup:admin` | | `tag:sandbox` | Sandbox hosts that accept advertised lab subnet routes | `autogroup:admin` | +| `tag:agentcompute` | Durable agentcompute HTTPS service | `autogroup:admin` | A tagged device is owned by its tag, not by the user who registered it. Removing a tag from the policy while a device still carries it leaves that device without @@ -77,6 +78,20 @@ Policy tests require access to all four API endpoints and deny adjacent addresses and ports. Other routed destinations remain denied unless a separate rule permits them. +The agentcompute service uses these additional rules: + +| Source | Destination | Protocol | Port | +| --- | --- | --- | --- | +| `autogroup:member` | `tag:agentcompute` | TCP | `443` | +| `tag:agentcompute` | `studio-1` (`100.122.142.76`) | TCP | `22` | + +`studio-1` is a host alias for a user-owned device, not a device tag. Do not +retag Studio. The existing `autogroup:admin` → `*:*` rule is unchanged. +Policy tests allow the two service flows and deny service-to-Studio VNC, +service-to-other-host SSH, and sandbox-to-Studio SSH. OpenSSH separately +source-pins the service key; see the +[Studio authorization runbook](../../runbooks/agentcompute-studio-ssh.md). + ## Credentials CI authenticates with a Tailscale diff --git a/docs/docs/runbooks/agentcompute-service.md b/docs/docs/runbooks/agentcompute-service.md new file mode 100644 index 0000000..61d08cb --- /dev/null +++ b/docs/docs/runbooks/agentcompute-service.md @@ -0,0 +1,580 @@ +--- +title: Deploy and operate agentcompute +description: Deploy, verify, maintain, and recover the agentcompute HTTP service. +--- + +# Deploy and operate agentcompute + +Use this runbook to deploy and operate the durable `agentcompute` Streamable +HTTP service. The deployed endpoint is +`https://agentcompute01.tailda715.ts.net`. Tailscale Serve terminates HTTPS on +port `443` and proxies to the service on `127.0.0.1:8080`. + +The verified tailnet IPv4 is **`100.65.152.20`**. Studio's SSH policy pins this +address; re-enrollment requires checking both the MagicDNS name and source pin. + +The service runs in the `agentcompute01` Ubuntu 24.04 VM in the Incus `default` +project on `lab01`. OpenTofu in `GilmanLab/fleet/incus/agentcompute` owns the VM, +the `ac-svc-vlan40` OVN network, and the private `agentcompute01.glab.lol` A +record. The service process owns runtime sandbox resources. + +For an initial deployment or replacement, keep this order: apply the +infrastructure, install the verified release, deliver the credentials and start +the service, then enroll the node and publish Tailscale Serve. Publishing last +prevents an unauthenticated or credential-incomplete listener from reaching the +tailnet. + +## Preconditions + +- Administrative access to the Incus cluster, the Tailscale tailnet, the + `glab.lol` private Route 53 zone, and the lab S3 state bucket. +- Current checkouts of `GilmanLab/fleet`, `GilmanLab/agentcompute`, and the + private `GilmanLab/secrets` repository. +- An authenticated `lab-admin` AWS profile and `GLAB_AWS_STATE_BUCKET` set to + the existing fleet state bucket. +- SOPS decryption access to + `services/agentcompute/credentials.sops.yaml`. The file uses the + `agentcompute` encryption scope and follows + [ADR-0003](../decisions/0003-use-kms-with-pgp-recovery-for-secrets.md). +- An Incus administration remote named `nas01` with its existing pinned server + certificate. Do not enable automatic certificate acceptance. +- The `tag:agentcompute` tailnet policy is applied. Its only tag owner is + `autogroup:admin`; tailnet members can reach the tag on TCP `443`, and the tag + can reach the user-owned `studio-1` host only on TCP `22`. +- After enrollment, follow + [Authorize agentcompute SSH to Studio](agentcompute-studio-ssh.md) to pin + Studio access to the service's verified tailnet IPv4. + +## Safety boundary + +> project creation is root-equivalent in Incus 7.4 (OpenFGA model warning; scriptlet cannot see the project name at create time), so this credential is cluster root; the server's only exposure is the tailnet MCP endpoint behind bearer tokens, and agent code never reaches the Incus client. + +The Incus trust entry is the dedicated, unrestricted `agentcompute01` client. +Never substitute `bootstrap-admin`, broaden an image-build or CI certificate, or +reuse one of those identities. The MCP service accepts the static named bearer +identity `omp`; possession of its token grants every currently registered +capability. + +The Phase 2 live restricted-certificate check is the evidence for this +exception: the `image-build` certificate saw an empty filtered list for +`default` instances, received HTTP `403` when updating default-project +configuration, and received HTTP `403` when creating an `ac-` project. It cannot +perform the deployed project's create path. A future design can replace the +root identity with a fleet-managed pool of pre-restricted `ac-NN` projects and +a restricted claim-and-release certificate. That design also needs marker or +profile metadata because a restricted certificate cannot edit project +configuration. Do not implement that deferred pool as an ad hoc deployment +change. + +Do not change the IncusOS bond or disable `strict_hwaddr`. The management NIC is +a routed Incus NIC on the host's `_vmgmt` Layer 3 interface; the `mgmt` bridge +is only Layer 2. The deployed paths are: + +| Purpose | Configuration | +| --- | --- | +| Management and default route | `10.10.10.16/32` through `169.254.0.1` on `mgmt0` | +| Pinned Incus API | `https://10.10.10.14:8443` through `mgmt0` | +| Guest reach | `ac-svc-vlan40`, gateway `10.158.86.1/24`, service guest address `10.158.86.2` | +| Sandbox VLAN route | `10.10.40.0/24` through `10.158.86.1` on `guest0` | +| MCP listener | `127.0.0.1:8080` only | +| Published endpoint | `https://agentcompute01.tailda715.ts.net:443` through Tailscale Serve | + +One service process owns one runtime and one in-process reaper. All HTTP +sessions share it. The reaper scans once at startup and every 30 seconds. Do not +run a second durable service or a long-lived STDIO process against the same +sandbox set. + +## Materialize deployment inputs + +Keep plaintext in one owner-only temporary directory. The trap removes local +material at shell exit; the one-time Tailscale key is unusable after successful +enrollment. + +```bash +export GLAB_SECRETS_DIR="$HOME/code/glab/secrets" +export FLEET_DIR="$HOME/code/lab2/fleet" +export AWS_PROFILE=lab-admin + +umask 077 +AGENTCOMPUTE_CREDENTIALS="$(mktemp -d)" +trap 'rm -rf "$AGENTCOMPUTE_CREDENTIALS"; unset TS_OAUTH_CLIENT_ID TS_OAUTH_CLIENT_SECRET TS_OAUTH_SCOPE TS_OAUTH_TAGS_JSON OMP_TOKEN' EXIT +SECRETS_FILE="$GLAB_SECRETS_DIR/services/agentcompute/credentials.sops.yaml" + +sops --decrypt --extract '["incus"]["client_certificate"]' \ + "$SECRETS_FILE" >"$AGENTCOMPUTE_CREDENTIALS/incus-client.crt" +sops --decrypt --extract '["incus"]["client_key"]' \ + "$SECRETS_FILE" >"$AGENTCOMPUTE_CREDENTIALS/incus-client.key" +sops --decrypt --extract '["studio"]["private_key"]' \ + "$SECRETS_FILE" >"$AGENTCOMPUTE_CREDENTIALS/mac-ssh.key" +sops --decrypt --extract '["auth_tokens"]' --output-type json \ + "$SECRETS_FILE" >"$AGENTCOMPUTE_CREDENTIALS/auth-tokens.json" +pin_version="$( + awk -F'"' ' + $1 ~ /^service_version[[:space:]]*=/ { print $2 } + ' "$FLEET_DIR/incus/agentcompute/release.auto.tfvars" +)" +test -n "$pin_version" +curl -fsSL \ + "https://raw.githubusercontent.com/GilmanLab/agentcompute/v$pin_version/images/catalog.yaml" \ + -o "$AGENTCOMPUTE_CREDENTIALS/catalog.yaml" +chmod 0600 "$AGENTCOMPUTE_CREDENTIALS"/* + +export TF_VAR_images_catalog_file="$AGENTCOMPUTE_CREDENTIALS/catalog.yaml" +export TF_VAR_incus_client_certificate_file="$AGENTCOMPUTE_CREDENTIALS/incus-client.crt" +``` + +The encrypted document also records `incus.url`, `incus.trust_name`, and +`incus.restricted`. Require the reviewed values before using the material: + +```bash +test "$(sops --decrypt --extract '["incus"]["url"]' "$SECRETS_FILE")" = \ + 'https://10.10.10.14:8443' +test "$(sops --decrypt --extract '["incus"]["trust_name"]' "$SECRETS_FILE")" = \ + 'agentcompute01' +test "$(sops --decrypt --extract '["incus"]["restricted"]' "$SECRETS_FILE")" = \ + 'false' +jq -e 'keys == ["omp"] and (.omp | type == "string" and length > 0)' \ + "$AGENTCOMPUTE_CREDENTIALS/auth-tokens.json" >/dev/null +``` + +Nothing private enters cloud-init, OpenTofu input, or OpenTofu state. The public +Incus client certificate, pinned server certificate, image catalog, and +non-secret runtime configuration do enter cloud-init. The three private source +files are delivered after the VM exists as `root:root` mode `0600` files in a +mode `0700` directory. The unit uses four `LoadCredential=` entries: the Incus +client certificate and key, bearer-token file, and Studio SSH key. Treat the +runtime credential mount as unit-private; do not infer its in-unit access from +ownership or mode observed outside the service namespace. + +## Enroll the Incus identity + +Calculate the certificate fingerprint and compare it with any existing entry. +Stop if the name exists with a different fingerprint or is restricted. + +```bash +expected_fingerprint="$( + openssl x509 -in "$AGENTCOMPUTE_CREDENTIALS/incus-client.crt" \ + -noout -fingerprint -sha256 | + cut -d= -f2 | tr -d ':' | tr 'A-F' 'a-f' +)" +existing="$( + incus config trust list nas01: --format=json | + jq -r '.[] | select(.name == "agentcompute01") | [.fingerprint, .restricted] | @tsv' +)" + +if test -z "$existing"; then + incus config trust add-certificate nas01: \ + "$AGENTCOMPUTE_CREDENTIALS/incus-client.crt" \ + --name agentcompute01 \ + --description 'agentcompute01 service runtime' +else + test "$existing" = "$expected_fingerprint"$'\t'false +fi +``` + +Do not modify the existing image-build CI trust while enrolling this identity. + +## Deploy the VM and network + +Run the OpenTofu root from `GilmanLab/fleet`. Review the saved plan before +applying it. It must create or update only this root's VM, OVN network, private +DNS record, and associated state; it must not change host bonds, IncusOS +`strict_hwaddr`, the shared `fast40-uplink`, or other trust entries. + +```bash +cd "$FLEET_DIR/incus/agentcompute" +just init +just check +just plan +tofu show tfplan +just apply +``` + +Wait for the one-time bootstrap. A successful bootstrap installs and holds the +pinned Tailscale package, creates the non-login `agentcompute` service account, +and enables—but does not yet start—the credential-gated service. + +```bash +incus exec --project default nas01:agentcompute01 -- cloud-init status --wait +incus exec --project default nas01:agentcompute01 -- \ + cloud-init status --format json | + jq -e '.status == "done" and .errors == []' >/dev/null +``` + +Cloud-init is not a convergence mechanism. Except for a release pin, whose +installer is refreshed from applied state by `just install-release`, a changed +rendered configuration requires deliberate VM replacement followed by release +installation, credential delivery, and tailnet enrollment. Do not rerun the old +bootstrap on an existing VM. + +## Verify and install the pinned release + +Read the applied release pin instead of choosing a version at the command line. +Download on the administration workstation, verify the GitHub attestation and +immutable release, and compare the exact SHA-256. Check the Linux binary's +embedded commit on the service VM, not on the Mac workstation. + +```bash +pin="$(tofu output -json release_pin)" +version="$(jq -r .version <<<"$pin")" +digest="$(jq -r .sha256 <<<"$pin")" +test "$version" = "$pin_version" +asset="$(jq -r .asset <<<"$pin")" +tag="v$version" +release_dir="$(mktemp -d)" + +cleanup_release() { rm -rf "$release_dir"; } +trap 'cleanup_release; rm -rf "$AGENTCOMPUTE_CREDENTIALS"; unset TS_OAUTH_CLIENT_ID TS_OAUTH_CLIENT_SECRET TS_OAUTH_SCOPE TS_OAUTH_TAGS_JSON OMP_TOKEN' EXIT + +release_state="$(gh release view "$tag" --repo GilmanLab/agentcompute --json isDraft,tagName)" +test "$(jq -r .isDraft <<<"$release_state")" = 'false' +test "$(jq -r .tagName <<<"$release_state")" = "$tag" +gh release download "$tag" --repo GilmanLab/agentcompute \ + --dir "$release_dir" --pattern "$asset" --pattern checksums.txt +tag_commit="$(gh api "repos/GilmanLab/agentcompute/commits/$tag" --jq .sha)" + +gh attestation verify "$release_dir/$asset" \ + --repo GilmanLab/agentcompute \ + --signer-workflow GilmanLab/agentcompute/.github/workflows/attest.yml \ + --source-ref "refs/tags/$tag" \ + --source-digest "$tag_commit" \ + --deny-self-hosted-runners +test "$(shasum -a 256 "$release_dir/$asset" | awk '{print $1}')" = "$digest" +awk -v digest="$digest" -v asset="$asset" ' + $1 == digest && $2 == asset { found = 1 } + END { exit !found } +' "$release_dir/checksums.txt" +gh release verify "$tag" --repo GilmanLab/agentcompute +gh release verify-asset "$tag" "$release_dir/$asset" --repo GilmanLab/agentcompute +just install-release "$release_dir/$asset" +banner="$(incus exec --project default nas01:agentcompute01 -- /usr/local/bin/agentcompute --version)" +case "$banner" in + "agentcompute $version ($tag_commit) built "*) ;; + *) printf 'unexpected release banner: %s\n' "$banner" >&2; exit 1 ;; +esac +``` + +The workstation installer first refreshes the VM-side installer from the +`release_installer` output in applied state. It resolves its module directory +and requires initialized state-backend access. The VM-side installer repeats +the pinned digest and version checks, installs a versioned file under +`/usr/local/lib/agentcompute`, atomically changes +`/usr/local/bin/agentcompute`, and restarts the service only if it is already +running. A rejected asset does not replace the current binary. + +GitHub artifact and OCI attestations and the OCI Cosign signature are release +evidence, not deployed-service acceptance. Do not claim SLSA Build Level 3: +the artifact build occurs outside the reusable attester, so signer isolation +does not establish that level for the build. + +## Deliver credentials and start the service + +Delivery validates that the bearer file is a non-empty JSON identity-to-token +object with unique names and tokens and no whitespace or control characters. +It pushes all three private files as root-owned mode `0600`, then restarts the +unit. Install the release first so the initial service start cannot select an +unverified binary. + +```bash +just deliver-credentials \ + "$AGENTCOMPUTE_CREDENTIALS/incus-client.key" \ + "$AGENTCOMPUTE_CREDENTIALS/auth-tokens.json" \ + "$AGENTCOMPUTE_CREDENTIALS/mac-ssh.key" +``` + +The runtime loads bearer tokens once at process start. Credential delivery uses +`systemctl restart`, so a successful delivery activates all three private +inputs immediately and disconnects existing MCP sessions. + +## Enroll the tailnet node and publish HTTPS + +Mint one non-reusable, non-ephemeral, pre-authorized key from the scoped OAuth +client. The Python process keeps the OAuth secret and access token out of its +arguments; only the one-time auth key is written to the temporary directory. + +```bash +export TS_OAUTH_CLIENT_ID="$(sops --decrypt --extract '["tailscale"]["oauth_client_id"]' "$SECRETS_FILE")" +export TS_OAUTH_CLIENT_SECRET="$(sops --decrypt --extract '["tailscale"]["oauth_client_secret"]' "$SECRETS_FILE")" +export TS_OAUTH_SCOPE="$(sops --decrypt --extract '["tailscale"]["scope"]' "$SECRETS_FILE")" +export TS_OAUTH_TAGS_JSON="$(sops --decrypt --extract '["tailscale"]["tags"]' --output-type json "$SECRETS_FILE")" + +python3 - "$AGENTCOMPUTE_CREDENTIALS/tailscale-auth.key" <<'PY' +import json +import os +import sys +import urllib.parse +import urllib.request + +output = sys.argv[1] +tags = json.loads(os.environ["TS_OAUTH_TAGS_JSON"]) +if os.environ["TS_OAUTH_SCOPE"] != "auth_keys" or tags != ["tag:agentcompute"]: + raise SystemExit("refusing OAuth scope or tags outside the reviewed deployment") + +form = urllib.parse.urlencode({ + "client_id": os.environ["TS_OAUTH_CLIENT_ID"], + "client_secret": os.environ["TS_OAUTH_CLIENT_SECRET"], + "grant_type": "client_credentials", +}).encode() +with urllib.request.urlopen( + urllib.request.Request( + "https://api.tailscale.com/api/v2/oauth/token", + data=form, + headers={"Content-Type": "application/x-www-form-urlencoded"}, + ), + timeout=30, +) as response: + access_token = json.load(response)["access_token"] + +body = json.dumps({ + "keyType": "auth", + "capabilities": { + "devices": { + "create": { + "reusable": False, + "ephemeral": False, + "preauthorized": True, + "tags": tags, + } + } + }, + "expirySeconds": 600, + "description": "agentcompute01 one-time enrollment", +}).encode() +with urllib.request.urlopen( + urllib.request.Request( + "https://api.tailscale.com/api/v2/tailnet/-/keys", + data=body, + headers={ + "Authorization": f"Bearer {access_token}", + "Content-Type": "application/json", + }, + ), + timeout=30, +) as response: + key = json.load(response)["key"] + +with open(output, "x", encoding="utf-8") as handle: + handle.write(key + "\n") +os.chmod(output, 0o600) +PY + +unset TS_OAUTH_CLIENT_ID TS_OAUTH_CLIENT_SECRET TS_OAUTH_SCOPE TS_OAUTH_TAGS_JSON +just enroll-tailscale "$AGENTCOMPUTE_CREDENTIALS/tailscale-auth.key" +rm -f "$AGENTCOMPUTE_CREDENTIALS/tailscale-auth.key" +``` + +The enrollment script moves the key through VM tmpfs, shreds the VM copy on +success or failure, joins as `tag:agentcompute`, does not accept subnet routes, and +requires the exact `agentcompute01.tailda715.ts.net` MagicDNS name before it +publishes HTTPS `443` to `http://127.0.0.1:8080`. It refuses a collision suffix +such as `agentcompute01-1` instead of publishing an endpoint that disagrees +with screenshot URLs. It does not enable Tailscale SSH. On an already enrolled +and correctly tagged node, the VM-side operation reasserts Tailscale Serve only +after the same name check. + +## Verify the deployment + +First verify process, route, listener, node identity, and Serve state from the +administration workstation: + +```bash +incus exec --project default nas01:agentcompute01 -- \ + systemctl is-active agentcompute.service tailscaled.service +incus exec --project default nas01:agentcompute01 -- \ + ss -H -lnt 'sport = :8080' +incus exec --project default nas01:agentcompute01 -- \ + ip route get 10.10.10.14 +incus exec --project default nas01:agentcompute01 -- \ + ip route get 10.10.40.65 +incus exec --project default nas01:agentcompute01 -- \ + tailscale serve status +incus exec --project default nas01:agentcompute01 -- \ + tailscale status --json | + jq -e ' + .BackendState == "Running" and + .Self.DNSName == "agentcompute01.tailda715.ts.net." and + (.Self.TailscaleIPs | index("100.65.152.20") != null) and + (.Self.Tags | index("tag:agentcompute") != null) + ' >/dev/null +``` + +The listener output must name only `127.0.0.1:8080`. The API route must use +`mgmt0` through `169.254.0.1` with source `10.10.10.16`; the VLAN 40 route must +use `guest0` through `10.158.86.1` with source `10.158.86.2`. + +Check private DNS and public HTTPS identity from a tailnet member: + +```bash +set -o pipefail +test "$(dig +short agentcompute01.glab.lol A)" = '10.10.10.16' +openssl s_client \ + -connect agentcompute01.tailda715.ts.net:443 \ + -servername agentcompute01.tailda715.ts.net \ + -verify_return_error /dev/null | + openssl x509 -noout -issuer -subject -dates -ext subjectAltName + +test "$(curl -sS -o /dev/null -w '%{http_code}' \ + https://agentcompute01.tailda715.ts.net/)" = '401' +``` + +The certificate must validate through the workstation's default public trust +and contain `agentcompute01.tailda715.ts.net` in its subject alternative names. +Do not pin the automatically renewed Tailscale Serve leaf. + +Finally, make an authenticated MCP initialization request without putting the +bearer token in the process arguments: + +```bash +OMP_TOKEN="$(sops --decrypt --extract '["auth_tokens"]["omp"]' "$SECRETS_FILE")" +cat >"$AGENTCOMPUTE_CREDENTIALS/initialize.json" <<'JSON' +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"deployment-check","version":"1"}}} +JSON + +printf 'header = "Authorization: Bearer %s"\n' "$OMP_TOKEN" | + curl --config - --fail-with-body --silent --show-error \ + -H 'Accept: application/json, text/event-stream' \ + -H 'Content-Type: application/json' \ + --data-binary @"$AGENTCOMPUTE_CREDENTIALS/initialize.json" \ + https://agentcompute01.tailda715.ts.net/ +unset OMP_TOKEN +``` + +Require a successful `initialize` result naming `agentcompute` before declaring +the MCP endpoint accepted. A healthy systemd unit, valid TLS, and an HTTP `401` +without a token do not prove authenticated MCP handling. + +### Phase 9a qualification — 2026-09-15 + +- Deployed [`v0.1.1`](https://github.com/GilmanLab/agentcompute/releases/tag/v0.1.1), + commit `e1a13b7d9e95c287be3f35c797b1511732d1b0b6`. The Linux amd64 + SHA-256 is `1a3c63ee3fc3edf4a3b65c9198c47aa343ff1361f1f6de91fa0b9b6f027cda60`; + checksum, exact-tag/commit GitHub attestation, and immutable release-asset + verification passed before use. +- OpenTofu reported no changes. The Ubuntu VM runs on `lab01`; routed + management reaches the pinned API without changing host `strict_hwaddr` or + the bond. A sandbox forward at `10.10.40.68:5900` returned + `RFB 003.008` to the service over `guest0`. +- HTTPS validated through the default public trust store with a Let's Encrypt + certificate for the service hostname. Missing and invalid tokens returned + `401`; an authenticated cross-origin request returned `403`. +- Authenticated MCP initialization, `search_api`, `describe_api`, and `execute` + passed. A Linux desktop VM reached readiness, listed applications, launched + Text Editor, and returned a window screenshot through the HTTPS base URL. + Its project recorded `user.agentcompute.subject=omp`. +- Credential redelivery restarted the service. A fresh MCP session rediscovered + the running desktop and a five-minute sandbox. That short-lived sandbox, + created at `23:02:52Z` with expiry `23:07:52Z`, disappeared automatically; + direct Incus lookup then returned `Project not found`. +- Explicit desktop sandbox deletion removed its resources and revoked a fresh + screenshot URL (`404`). No qualification sandboxes remained. +- Studio accepted the dedicated key from `100.65.152.20`, with normal shell + access as `agentcompute`, and rejected the same key from Studio's own source + with `Permission denied (publickey)`. + +Two findings remain relevant to operations: + +1. `v0.1.0` rejected Tailscale Serve's preserved public `Host` header on the + loopback listener. `v0.1.1` permits that path only with configured bearer + authentication, while retaining cross-origin protection and the host guard + for unauthenticated loopback servers. +2. Whole-desktop capture returned a black `1280×800` image even after Text + Editor launched. Explicit window capture rendered the editor correctly + (`822×642`), and accessibility state was available. This is a desktop-capture + finding, not an HTTPS delivery failure; full-desktop capture is not qualified + by this deployment. + +## Routine operations + +### Inspect and restart + +```bash +incus exec --project default nas01:agentcompute01 -- \ + systemctl status agentcompute.service --no-pager +incus exec --project default nas01:agentcompute01 -- \ + journalctl -u agentcompute.service -n 100 --no-pager +incus exec --project default nas01:agentcompute01 -- \ + systemctl restart agentcompute.service +incus exec --project default nas01:agentcompute01 -- \ + systemctl is-active agentcompute.service +``` + +A restart disconnects active MCP sessions. The new process reconnects to Incus, +reconciles the pinned image catalog, rediscovers persisted sandboxes, and runs +an immediate reaper scan before the next 30-second interval. Do not start a +second process to preserve sessions during the restart. + +### Upgrade the release + +Update `service_version` and `service_sha256` together in +`release.auto.tfvars`. Verify the new release with the procedure above, review +and apply the saved OpenTofu plan, then run `just install-release` with the +verified asset. A release-pin plan must not replace the VM. The script resolves +its module directory, reads `release_installer` from applied state, and replaces +the VM-side installer before it stages the asset. It therefore needs the +initialized state backend and `lab-admin` AWS access, but no reboot or VM +replacement. Do not update only the symlink or download from the VM. Keep the +previous versioned binary until the new endpoint passes the complete +verification procedure. + +### Rotate credentials + +Update and merge the SOPS file before changing the running service. + +- **Bearer token:** materialize the new `auth_tokens` object, update the MCP + client for the same maintenance window, then redeliver all three private + inputs. Delivery restarts the unit. The identity name remains `omp`; token + values never belong in logs or documentation. +- **Studio SSH key:** first authorize the new public key with the exact service + source in the [Studio SSH runbook](agentcompute-studio-ssh.md). Redeliver all + three inputs; delivery restarts the unit. Verify the Studio connection, then + remove the old public key. +- **Incus client certificate and key:** enroll the new dedicated certificate + before stopping the old identity. Update the public-certificate OpenTofu + input and perform a deliberate VM replacement so cloud-init installs the new + public half; then install, deliver, enroll, and verify in the documented + order. Remove the old trust entry only after full acceptance. +- **Tailscale OAuth client:** rotate the encrypted OAuth fields. It affects only + future one-time enrollment keys and does not require a service restart. + +## Rollback and recovery + +For a release regression, return `release.auto.tfvars` to the last accepted +version and digest, review and apply the plan, then install that verified asset +through `just install-release`. Confirm the embedded tag commit and repeat the +full verification procedure. Do not point the symlink at an unverified file. + +For a VM replacement, use a saved plan with an explicit replacement and expect +to lose the VM's tailnet node identity and delivered files: + +```bash +tofu plan -replace=incus_instance.service -out=tfplan +tofu show tfplan +tofu apply tfplan +``` + +After cloud-init completes, install the release and deliver credentials. Before +enrollment, remove the old, offline `agentcompute01` device from the tailnet so +the replacement can receive the exact MagicDNS name. Then enroll and run the +full verification. Never accept a collision name such as `agentcompute01-1`; +the enrollment script refuses to publish Serve under that name. + +If exposure must stop immediately, remove Serve without changing the loopback +service or deleting state: + +```bash +incus exec --project default nas01:agentcompute01 -- tailscale serve reset +``` + +After resolving the incident, re-publish the existing loopback service with: + +```bash +incus exec --project default nas01:agentcompute01 -- \ + tailscale serve --bg --https=443 http://127.0.0.1:8080 +``` + +Stop and escalate if the saved plan changes shared OVN or host networking, the +Incus certificate does not match `agentcompute01`, the VM cannot reach the +pinned API, a non-loopback port `8080` listener appears, the node has an +unexpected tag, an unauthenticated request is not `401`, the authenticated MCP +initialization fails, or expired sandboxes survive a service restart and reaper +interval. diff --git a/docs/docs/runbooks/agentcompute-studio-ssh.md b/docs/docs/runbooks/agentcompute-studio-ssh.md new file mode 100644 index 0000000..089cd67 --- /dev/null +++ b/docs/docs/runbooks/agentcompute-studio-ssh.md @@ -0,0 +1,313 @@ +--- +title: Authorize agentcompute SSH to Studio +description: Restrict the agentcompute service key to Studio SSH connections from the deployed service node. +--- + +# Authorize agentcompute SSH to Studio + +Use this runbook to authorize the dedicated `agentcompute01` service key for +native OpenSSH on `studio-1`. This is a narrowly scoped Phase 8 host-access +procedure. It does not install or operate Lume, build a macOS seed, manage +worker VMs, change VNC filtering, or define a Phase 9b macOS lifecycle. + +Studio remains the user-owned tailnet device `studio-1` at `100.122.142.76`. +**Do not assign `tag:macbackend` or any replacement device tag.** The tailnet +policy uses the `studio-1` host alias, allows `tag:agentcompute` to reach only +TCP `22` on that address, and leaves the existing `autogroup:admin` wildcard +rule unchanged. Host OpenSSH provides the source and key restrictions. + +## Preconditions + +- The Studio owner is present with console or existing administrative access. + Keep that access open until a new connection passes. +- Remote Login is enabled and `/usr/sbin/sshd` uses the root-owned drop-in + directory `/etc/ssh/sshd_config.d`. +- The dedicated local `agentcompute` account is hidden, standard rather than + administrator, unable to use `sudo`, and a member of + `com.apple.access_ssh`. +- The owner home directory denies traversal by `agentcompute`. On this Studio, + `/Users/josh` is mode `0700`. +- `lume serve` is already loopback-only on `127.0.0.1:7777` under the + `agentcompute` account. +- SOPS access to + `GilmanLab/secrets/services/agentcompute/credentials.sops.yaml`, whose + `studio.public_key` and `studio.private_key` are the dedicated service SSH + keypair. +- The current tailnet policy containing the `studio-1` alias and + `tag:agentcompute` SSH rule has been applied through + [the policy change procedure](tailscale-policy-change.md). + +## Safety impact + +A malformed OpenSSH drop-in can disable Remote Login. Keep the owner session +open, validate with `sshd -t`, and restore the saved files before closing that +session if validation fails. + +The dedicated account is the containment boundary. Do not make it an +administrator, grant sudo, expose the loopback Lume API directly, copy the +private service key into a guest, or weaken the owner-home permissions. + +The approved transport is a normal shell plus local TCP forwarding. Remove the +old forwarding-only restrictions: no `ForceCommand`, no `PermitOpen`, and no +`restrict` authorized-key option. Agent, X11, tunnel, remote-forward, and +gateway forwarding remain disabled. + +## Confirm the service source + +Read the service node's tailnet IPv4 from the node itself. The approved and +currently deployed source is `100.65.152.20`. + +```bash +SERVICE_TAILNET_IP="$( + incus exec --project default nas01:agentcompute01 -- tailscale ip -4 | + sed -n '1p' +)" +test "$SERVICE_TAILNET_IP" = '100.65.152.20' + +incus exec --project default nas01:agentcompute01 -- tailscale status --json | + jq -e ' + .BackendState == "Running" and + .Self.DNSName == "agentcompute01.tailda715.ts.net." and + (.Self.TailscaleIPs | index("100.65.152.20") != null) and + (.Self.Tags | index("tag:agentcompute") != null) + ' >/dev/null +``` + +If the address changes, stop. Update and re-verify the source pin in both files +below before using the new node. A tag match alone is not an OpenSSH source +pin. + +## Materialize the public key + +On the trusted administration workstation, write only the public half to a +temporary file: + +```bash +export GLAB_SECRETS_DIR="$HOME/code/glab/secrets" +SECRETS_FILE="$GLAB_SECRETS_DIR/services/agentcompute/credentials.sops.yaml" +umask 077 +STUDIO_SSH_WORK="$(mktemp -d)" +trap 'rm -rf "$STUDIO_SSH_WORK"' EXIT + +sops --decrypt --extract '["studio"]["public_key"]' \ + "$SECRETS_FILE" >"$STUDIO_SSH_WORK/agentcompute01.pub" +ssh-keygen -l -f "$STUDIO_SSH_WORK/agentcompute01.pub" +``` + +Transfer that public file to the owner session through the existing trusted +administrative path. Do not transfer the private key to Studio for normal +operation. + +## Install the source-restricted key + +Run this block in the Studio owner session from the directory containing +`agentcompute01.pub`: + +```bash +set -euo pipefail +SERVICE_TAILNET_IP=100.65.152.20 +SERVICE_PUBLIC_KEY="$(cat agentcompute01.pub)" +KEY_BACKUP="$HOME/agentcompute-authorized_keys.$(date +%Y%m%d%H%M%S).bak" +if sudo test -f /Users/agentcompute/.ssh/authorized_keys; then + sudo cp -p /Users/agentcompute/.ssh/authorized_keys "$KEY_BACKUP" + sudo chown "$(id -un):$(id -gn)" "$KEY_BACKUP" +fi + +sudo -u agentcompute -H sh -c ' + set -eu + install -d -m 0700 "$HOME/.ssh" + tmp="$(mktemp "$HOME/.ssh/.authorized_keys.XXXXXX")" + trap '\''rm -f "$tmp"'\'' EXIT + printf '\''from="%s",no-agent-forwarding,no-X11-forwarding %s agentcompute server\n'\'' \ + "$1" "$2" >"$tmp" + chmod 0600 "$tmp" + mv -f "$tmp" "$HOME/.ssh/authorized_keys" + trap - EXIT +' sh "$SERVICE_TAILNET_IP" "$SERVICE_PUBLIC_KEY" +``` + +This dedicated account has one server key. Do not append the same key without a +`from=` restriction, and do not retain an old forced-command form of the key. + +## Install the OpenSSH account policy + +Back up the current drop-in and install the reviewed policy atomically: + +```bash +set -euo pipefail +SERVICE_TAILNET_IP=100.65.152.20 +DROPIN=/etc/ssh/sshd_config.d/110-agentcompute.conf +BACKUP="$HOME/110-agentcompute.conf.$(date +%Y%m%d%H%M%S).bak" + +if sudo test -f "$DROPIN"; then + sudo cp -p "$DROPIN" "$BACKUP" +fi + +candidate="$(mktemp)" +trap 'rm -f "$candidate"' EXIT +cat >"$candidate" <"$STUDIO_SSH_WORK/studio-known-hosts" +incus file push --project default --mode 0600 --uid 0 --gid 0 \ + "$STUDIO_SSH_WORK/studio-known-hosts" \ + nas01:agentcompute01/run/studio-known-hosts +``` + +Run a normal shell command with the delivered private key. Strict host-key +checking must remain enabled. + +```bash +incus exec --project default nas01:agentcompute01 -- \ + ssh -F /dev/null \ + -i /etc/agentcompute/credentials/mac-ssh.key \ + -o BatchMode=yes \ + -o IdentitiesOnly=yes \ + -o StrictHostKeyChecking=yes \ + -o UserKnownHostsFile=/run/studio-known-hosts \ + agentcompute@studio-1 \ + 'printf "user=%s\n" "$(id -un)"; printf "connection=%s\n" "$SSH_CONNECTION"; uname -s; ! sudo -n true' +``` + +Require all of these observations: + +- `user=agentcompute`; +- `SSH_CONNECTION` starts with `100.65.152.20` and ends with + `100.122.142.76 22`; +- `uname -s` prints `Darwin`; +- the remote `sudo -n true` check fails, making the final negation succeed. + +Exercise local forwarding to the loopback Lume API. This check reads the +qualified stopped seed; it does not start or modify a VM. + +```bash +incus exec --project default nas01:agentcompute01 -- sh -eu -c ' + ssh -F /dev/null \ + -i /etc/agentcompute/credentials/mac-ssh.key \ + -o BatchMode=yes \ + -o IdentitiesOnly=yes \ + -o StrictHostKeyChecking=yes \ + -o UserKnownHostsFile=/run/studio-known-hosts \ + -N -L 127.0.0.1:17777:127.0.0.1:7777 \ + agentcompute@studio-1 & + tunnel=$! + trap "kill $tunnel 2>/dev/null || true; wait $tunnel 2>/dev/null || true" EXIT + attempts=0 + while test "$attempts" -lt 10; do + if curl -fsS -o /dev/null \ + http://127.0.0.1:17777/lume/vms/ac-seed-macos-tahoe-desktop; then + exit 0 + fi + attempts=$((attempts + 1)) + sleep 1 + done + exit 1 +' +``` + +Remove the temporary host-key file after the check: + +```bash +incus exec --project default nas01:agentcompute01 -- \ + rm -f /run/studio-known-hosts +``` + +The application must use an independently verified persistent known-hosts file +when this SSH path is wired into runtime configuration. Never replace host-key +verification with `StrictHostKeyChecking=no` or an unverified `ssh-keyscan`. + +## Verify refusal of the same key from another source + +This negative check is required. A different key proves nothing, and a failure +caused only by the tailnet ACL does not prove the OpenSSH source restriction. +Use an admin-owned tailnet device because the existing admin wildcard permits +it to reach Studio TCP `22`. + +In an owner-only temporary directory on that non-service source, materialize the +same private key and confirm its derived public-key fingerprint matches the +installed public key: + +```bash +umask 077 +sops --decrypt --extract '["studio"]["private_key"]' \ + "$SECRETS_FILE" >"$STUDIO_SSH_WORK/mac-ssh.key" +chmod 0600 "$STUDIO_SSH_WORK/mac-ssh.key" +ssh-keygen -y -f "$STUDIO_SSH_WORK/mac-ssh.key" \ + >"$STUDIO_SSH_WORK/derived-mac-ssh.pub" +test "$(ssh-keygen -l -f "$STUDIO_SSH_WORK/derived-mac-ssh.pub" | awk '{print $2}')" = \ + "$(ssh-keygen -l -f "$STUDIO_SSH_WORK/agentcompute01.pub" | awk '{print $2}')" + +set +e +ssh -F /dev/null \ + -i "$STUDIO_SSH_WORK/mac-ssh.key" \ + -o BatchMode=yes \ + -o IdentitiesOnly=yes \ + -o StrictHostKeyChecking=yes \ + -o UserKnownHostsFile="$STUDIO_SSH_WORK/studio-known-hosts" \ + agentcompute@100.122.142.76 true +status=$? +set -e +rm -f "$STUDIO_SSH_WORK/mac-ssh.key" \ + "$STUDIO_SSH_WORK/derived-mac-ssh.pub" +test "$status" -eq 255 +``` + +Require `Permission denied (publickey)` and exit `255`. If the connection +succeeds, restore the previous policy immediately and inspect both +`authorized_keys` and the effective `sshd` configuration. + +## Rollback and escalation + +To roll back before closing the owner session, restore the saved drop-in or +remove the new one when no prior file existed. Restore the previous +`~agentcompute/.ssh/authorized_keys`, then run `sudo /usr/sbin/sshd -t`. Do not +leave a half-applied combination in which the key and account policy have +different source addresses. + +Stop and escalate if the service IPv4 differs from `100.65.152.20`, Studio has +become tagged, the owner home is traversable, the dedicated account can use +sudo, effective policy contains a forced command or destination-specific +`PermitOpen`, the allowed service connection cannot open a normal shell, or the +same key succeeds from any other source. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index ef510dd..dc9f7fc 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -52,6 +52,8 @@ nav: - Tailscale policy: reference/networking/tailscale-policy.md - Runbooks: - Change the Tailscale policy: runbooks/tailscale-policy-change.md + - Deploy and operate agentcompute: runbooks/agentcompute-service.md + - Authorize agentcompute SSH to Studio: runbooks/agentcompute-studio-ssh.md - Commission a lab compute node: runbooks/commission-lab-node.md - Operate OVN central and certificates: runbooks/ovn-central-and-certificates.md - Rebuild nas01: runbooks/rebuild-nas01.md From 15006b0ac36f359973b149d8edf2746940fbe8b0 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 16:14:17 -0700 Subject: [PATCH 2/2] docs(agentcompute): force IPv4 for source-pinned SSH --- docs/docs/runbooks/agentcompute-studio-ssh.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/runbooks/agentcompute-studio-ssh.md b/docs/docs/runbooks/agentcompute-studio-ssh.md index 089cd67..c91ec76 100644 --- a/docs/docs/runbooks/agentcompute-studio-ssh.md +++ b/docs/docs/runbooks/agentcompute-studio-ssh.md @@ -201,7 +201,7 @@ checking must remain enabled. ```bash incus exec --project default nas01:agentcompute01 -- \ - ssh -F /dev/null \ + ssh -4 -F /dev/null \ -i /etc/agentcompute/credentials/mac-ssh.key \ -o BatchMode=yes \ -o IdentitiesOnly=yes \ @@ -224,7 +224,7 @@ qualified stopped seed; it does not start or modify a VM. ```bash incus exec --project default nas01:agentcompute01 -- sh -eu -c ' - ssh -F /dev/null \ + ssh -4 -F /dev/null \ -i /etc/agentcompute/credentials/mac-ssh.key \ -o BatchMode=yes \ -o IdentitiesOnly=yes \ @@ -280,7 +280,7 @@ test "$(ssh-keygen -l -f "$STUDIO_SSH_WORK/derived-mac-ssh.pub" | awk '{print $2 "$(ssh-keygen -l -f "$STUDIO_SSH_WORK/agentcompute01.pub" | awk '{print $2}')" set +e -ssh -F /dev/null \ +ssh -4 -F /dev/null \ -i "$STUDIO_SSH_WORK/mac-ssh.key" \ -o BatchMode=yes \ -o IdentitiesOnly=yes \