docs(agentcompute): record deployed Phase 9b qualification

docs/docs/architecture/agentcompute.md

@@ -0,0 +1,151 @@
+---
+title: Agentcompute architecture
+description: Runtime boundaries, trust, networking, and durable state of the lab compute service.
+---
+
+# Agentcompute architecture
+
+Agentcompute exposes short-lived compute environments to agents through one
+Streamable HTTP MCP endpoint, `https://agentcompute01.tailda715.ts.net/`.
+The agent-facing surface is `search_api`, `describe_api`, and `execute`;
+individual compute and desktop operations are Starlark capabilities, not MCP
+tools. See [ADR-0009](../decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md).
+
+This page describes the deployed boundaries. The
+[implemented design](../designs/agentcompute.md) records the original intent
+and deviations; the [runbook](../runbooks/agentcompute.md) owns deployment,
+credential handling, and recovery.
+
+## Request and execution boundaries
+
+```mermaid
+flowchart LR
+    Agent[Agent on tailnet] -->|HTTPS 443 and bearer| Serve[Tailscale Serve]
+    Serve -->|loopback 8080| MCP[agentcompute HTTP and CodeMode]
+    MCP --> Incus[Incus cluster API]
+    MCP -->|SSH 22 as agentcompute| Studio[Owner's Mac Studio]
+    Studio -->|loopback 7777| Lume[Lume daemon]
+    Studio -->|SSH ProxyJump| Mac[macOS guest]
+    Incus --> Linux[Linux guests]
+    Incus --> Windows[Windows guests]
+```
+
+Tailscale Serve terminates publicly trusted, automatically renewed HTTPS and
+forwards to `127.0.0.1:8080` in the `agentcompute01` VM. The Go HTTP handler
+requires a named static bearer token and rejects cross-origin requests. It
+shares one backend registry and reaper across HTTP sessions. A client reconnect
+or service restart does not create a second inventory.
+
+CodeMode evaluates bounded Starlark, not host Python or a host shell. Only
+registered capabilities enter the backends. Guest `instance.exec` deliberately
+executes commands inside the named sandbox guest; it does not expose Incus
+credentials or a shell on `agentcompute01` to the caller. The `omp` bearer is
+an administrator of this service surface, not a per-sandbox authorization
+boundary. Subject metadata is attribution, not tenant isolation.
+
+## Incus and OVN
+
+The Incus backend owns projects prefixed `ac-`, with sandbox lifetime and
+ownership in `user.agentcompute.*` metadata. Instances, networks, forwards,
+snapshots, and sandbox-scoped published images belong to those projects.
+Discovery reads backend state rather than a process-local inventory.
+
+OVN is the default network kind. NAT networks allocate external addresses from
+`10.10.40.64–10.10.40.127` through `fast40-uplink`. An isolated `nat=false`
+network does not attach to the uplink or consume an external NAT address.
+Inter-network routing requires an explicit guest router or `net.peer`.
+Network forwards allocate their own external listen addresses; they are not
+implicitly the network's NAT address. See
+[ADR-0006](../decisions/0006-ovn-as-the-sandbox-network-fabric.md) and the
+[address plan](../reference/networking/address-plan.md).
+
+The service VM is in the Incus `default` project on `lab01`, outside disposable
+sandbox projects. Its routed management NIC uses `10.10.10.16/32` through
+`169.254.0.1`; its guest-side NIC is `10.158.86.2/24` on `ac-svc-vlan40`.
+The route to VLAN 40 is through `10.158.86.1`. The tailnet address is
+`100.65.152.20`. These paths do not bridge the IncusOS management bond or
+require disabling `strict_hwaddr`.
+
+The dedicated Incus client is **unrestricted cluster root**. Incus 7.4's
+project-creation authorization prevented the intended restricted `ac-*`
+identity: the OpenFGA model warns at project creation, and the authorization
+scriptlet cannot inspect the new project's name. A pre-created project pool
+with a restricted claim/release identity is deferred. The current mitigation
+is the narrow authenticated service boundary, not a claim of least-privilege
+Incus credentials.
+
+## Mac backend
+
+The owner's Mac Studio runs Lume as a separate standard, hidden `agentcompute`
+account. It is not a dedicated machine. Tailnet policy permits the service tag
+to reach Studio only on SSH port 22. Studio's authorized-key rules restrict the
+service key to the verified `100.65.152.20` source. See
+[ADR-0008](../decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md).
+
+The backend reaches the host-local Lume HTTP daemon through SSH; port 7777 is
+not published to the LAN or tailnet. Inventory and host work use the
+account-local `/Users/agentcompute/bin/lume`, never the global install.
+Guest execution uses SSH ProxyJump and SFTP with explicit host-key checking,
+not a nonexistent `lume ssh` command. The qualified Tahoe seed supplies the
+pinned guest host key and machine identifier. Retaining that machine identifier
+preserves the qualified guest's activation and desktop permissions; the server
+enforces a two-running-guest cap. The Mac backend does not pretend to provide
+Incus OVN features.
+
+A pinned source build temporarily supplies upstream Lume's merged
+`--vnc disabled` feature, absent from release 0.5.3. Every backend start,
+including restore and restart, uses `noDisplay: true` and `vnc: disabled`.
+Startup refuses a CLI or daemon that cannot enforce the policy. This avoids a
+wildcard VNC listener at its source; no high-port PF rule is installed on the
+owner's LAN interfaces. `pins/lume.yaml` in agentcompute records the exact
+source commit, installed binary digest, and 0.5.3 as a manual rollback
+reference that backend startup rejects.
+Return to a release pin once a release includes the feature. The global Lume
+install, Internet Sharing, and Continuity services remain outside service
+ownership.
+
+## Desktop transport and screenshots
+
+The desktop API wraps Cua Driver over existing guest-execution channels rather
+than exposing a second guest TCP service. Linux and macOS use the Driver's
+one-shot CLI. Windows uses its persistent MCP transport through the guest
+execution path. This is the trust and transport decision in
+[ADR-0007](../decisions/0007-use-cua-driver-over-guest-execution.md).
+
+Screenshot bytes are written under `/var/lib/agentcompute/screenshots` and
+returned as opaque HTTPS URLs rooted at the service hostname. Screenshot URLs
+are bearer capabilities: do not log or publish them for private workloads.
+Their lifetime is at most five minutes, shortened by sandbox expiry or deletion,
+and they do not survive a service restart. The HTTP service checks that lifetime
+rather than treating a file on disk as perpetual authorization.
+The service's systemd credential directory and Studio keys are not available
+to guest desktop tools.
+
+## Persistence and failure semantics
+
+- Incus projects and Lume sandbox metadata survive an MCP process restart.
+  The reaper reconciles expired sandboxes on its next pass, including those
+  that expired while the process was stopped; it is not an in-memory timer.
+- Incus snapshot restore stops the original and stages a copy carrying current
+  agentcompute metadata, then deletes the original, renames the copy, and
+  starts it. UUID, NIC identity, and DHCP address can change.
+  Deleting the original also deletes its snapshot
+  tree. This is recreate semantics, not an in-place filesystem rollback.
+- A failed create or delete can leave backend resources needing reconciliation.
+  The runbook distinguishes a slow asynchronous operation from a stuck
+  sandbox and requires checking ownership before direct backend cleanup.
+- Fleet OpenTofu owns the long-lived VM/network/DNS and public runtime
+  configuration. Credentials are delivered out of band into root-only files
+  and loaded by systemd. Application release installation converges the
+  public configuration and binary without rerunning cloud-init or replacing
+  the VM and its Tailscale identity.
+
+## Ownership
+
+`GilmanLab/agentcompute` owns the server, capability contracts, backend code,
+image qualification, and release artifacts. `GilmanLab/fleet` owns service
+infrastructure and runtime deployment. `GilmanLab/networking` owns tailnet and
+network policy. `GilmanLab/secrets` owns encrypted credentials. This root
+repository owns architecture, decisions, deployment guidance, and acceptance
+evidence. Changes crossing those boundaries require companion PRs; runtime
+experiments are not a substitute for updating the owning repository.

docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md

@@ -22,6 +22,8 @@ claiming the provider parent. Which fabric should the durable service use?
 - Sandbox traffic cannot initiate connections to management or OOB.
 - Fleet owns infrastructure; agents own only disposable sandbox resources.
 - Management boot and OVN central must not depend on OVN itself.
+- Native `nat=false`, `network=none` isolation must use the same durable
+  control plane as NAT-enabled sandbox networks.
 - Failed creates and expired resources must converge through explicit,
   retryable deletion rather than undocumented service repairs.
 
@@ -43,18 +45,18 @@ to `nas01` and attached to its unmanaged VLAN 10 `mgmt` bridge rather than an
 OVN network. The pinned `ovn-central` package version `26.03.0-2` supplies
 northd and standalone NB/SB databases, without Raft. Per-component TLS gates
 prevent either database from starting without the complete certificate trio.
-Remote database connections use mutual TLS from a dedicated offline OVN CA,
-not the KMS root or the Incus cluster certificate. The CA uses EC P-256, has a
-ten-year validity and no path-length constraint. Central and each chassis have
-separate two-year leaves. Per-chassis keys allow one member's identity to
-rotate without distributing the same private key to every node. The TLS
-cutover occurred only after the cluster carried the CVE-2026-40243 fix.
+Remote database connections use mutual TLS from a dedicated offline OVN
+application CA, not the KMS root or the Incus cluster certificate. The CA uses
+EC P-256, has a ten-year validity and no path-length constraint. Central and
+each chassis have separate two-year leaves. Per-chassis keys allow one member's
+identity to rotate without distributing the same private key to every node.
+The TLS cutover occurred only after the cluster carried the CVE-2026-40243 fix.
 
 The owner approved this application-scoped trust domain on 2026-09-12,
-superseding the draft's requirement to use the ADR-0005 hierarchy.
-**OVN trust domain is application-scoped and offline; ADR-0005 governs the
-KMS root hierarchy and is unaffected.** Revisit the issuance model when Vault
-PKI exists and there is a reason to migrate. ADR-0005 is not amended.
+superseding the draft's requirement to use the ADR-0005 hierarchy. The
+dedicated offline OVN CA is outside ADR-0005's KMS-root hierarchy; ADR-0005 is
+not amended. Revisit the issuance model when Vault PKI exists and there is a
+reason to migrate.
 
 The CA key and every leaf key are escrowed under the `fleet` scope in
 `GilmanLab/secrets`, following ADR-0003's alternative KMS/PGP recipients and
@@ -78,11 +80,11 @@ after rotation; the central private key is never an OpenTofu input.
 Fleet's `cluster/` deployment owns chassis configuration on every member,
 using VLAN 30 tunnel addresses and central's VLAN 10 endpoint. Supported
 settings are mirrored in node seeds. The default-project physical uplink
-exclusively owns its provider parent. Before convergence, fleet checks
-member-specific networks, profiles, and instance NICs, including inherited
-and stopped-instance devices. A competing direct parent attachment or
-physical-uplink NIC aborts deployment with a named conflict; fleet never
-removes it silently.
+`fast40-uplink` exclusively owns the IncusOS `fast40` provider parent. Before
+convergence, fleet checks member-specific networks, profiles, and instance
+NICs, including inherited and stopped-instance devices. A competing direct
+parent attachment or physical-uplink NIC aborts deployment with a named
+conflict; fleet never removes it silently.
 
 The [address plan](../reference/networking/address-plan.md#ovn-external-addresses)
 is authoritative for the approved external allocation and its eight-sandbox
@@ -94,8 +96,8 @@ Reconsider a dedicated VLAN only if OVN needs more than the approved
 64-address block.
 The representative topology has NAT-enabled `default` and `wan` networks, an
 isolated `lan`, and one distinct forward listen address. It consumes three
-external addresses per sandbox: eight sandboxes consume 24 and leave 40 of
-the 64-address reservation.
+external addresses per sandbox. Eight sandboxes consume 24; the service's
+`ac-svc-vlan40` network consumes one more, leaving 39 of the 64-address reservation.
 
 New sandbox projects use project-owned OVN networks and managed-only NICs.
 The default network has NAT. An additional NAT-enabled network consumes one
@@ -105,6 +107,11 @@ consumes no external address, and has no direct path outside its sandbox. It
 becomes reachable only through `net.peer` or a router instance attached to
 another network. A `net.forward` request for an isolated network returns
 `AgentError`.
+Both the overlay control plane and the native isolation contract require
+central. Without central, Incus cannot create the project-owned logical switch
+whose `network=none` setting makes a `nat=false` network isolated. Replacing
+central would therefore require replacing the selected OVN fabric, not only
+the central VM.
 
 The `default_network_kind=bridge` fallback keeps whole sandboxes on
 member-local bridges; explicit bridge networks remain bare wires there. Incus
@@ -165,6 +172,25 @@ central stopped, `central-tls`, central running with the new CA, then fleet's
 OVN client/chassis converge. NB_Global, SB_Global, and two logical-switch UUIDs
 were preserved. The new `lab03` identity authenticated to both NB and SB; the
 old identity was rejected by both before an authenticated response.
+
+Phase 5 also exposed a stale in-memory trust failure after the OVN CA was
+silently re-minted. Stored Incus configuration already named the new CA, but
+the `lab01`, `lab02`, and `nas01` daemons continued reconnecting with the old
+CA. Their failed handshakes filled the central VM's 20 GiB root filesystem.
+After evidence capture and owner approval, recovery truncated only the three
+identified log files and recycled those three Incus daemons serially. `lab03`
+was left running because its daemon already held the new trust. All members
+returned `Online`, central processes retained their PIDs and start times, and
+the measured inbound reconnect rate fell from 1,039 per second to zero.
+
+[Fleet PR #20](https://github.com/GilmanLab/fleet/pull/20) prevents the same
+silent transition: the ceremony refuses to mint an absent CA unless the
+operator supplies `--mint-ca`, delivery paths require the reviewed CA
+fingerprint, an OVN converge reports changed in-memory trust, and the explicit
+trust-roll operation restarts selected daemons one at a time with an online
+gate. [Root PR #34](https://github.com/GilmanLab/root/pull/34) records the
+evidence-first, approval-gated recovery procedure in the canonical runbook.
+
 A cross-member fixture completed three of three pings on each tested path.
 Public OpenTofu cloud-init metadata reconciliation was still pending
 AWS authentication and is not part of that proof.
@@ -191,13 +217,14 @@ fixture residue across all four members and all projects in 3.03 seconds. The
 post-rotation and post-reboot fleet dry run proposed no changes in all 11
 operations.
 
-These checks complete the live technical confirmation recorded here. The record
-remains `proposed`; its status must not be changed without the owner's
-separate acceptance of the decision.
+These checks establish the OVN infrastructure and lifecycle evidence recorded
+here. They do not accept this decision or claim that Phase 9b validation is
+complete. The record remains `proposed`; only the owner may change its status
+to `accepted`.
 
 ## More Information
 
-- [Agentcompute design draft](../designs/drafts/agentcompute.md)
+- [Agentcompute design](../designs/agentcompute.md)
 - [Phase 3 qualification and parent recovery](https://github.com/GilmanLab/agentcompute/blob/spike/ovn-recreate-diagnosis/spikes/ovn/README.md): first post-fixture-deletion lab01-gateway cycle passed in 31.018 seconds, without reboot, central restart, or neighbor repair.
 - [Incus #3985](https://github.com/lxc/incus/issues/3985): unavailable NB creation leaves an `Errored` network; deletion after central recovery releases it.
 - [Incus #3986](https://github.com/lxc/incus/issues/3986): raw macvlan parent contention is separate from the central-outage failure.