From ee7f209e6d155edc2ffe2d62f3c2275651deeeaf Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 18:33:40 -0700 Subject: [PATCH 1/7] docs: record agentcompute platform decisions --- .../0006-ovn-as-the-sandbox-network-fabric.md | 63 +++++--- ...007-use-cua-driver-over-guest-execution.md | 134 +++++++++++++++ ...nfined-account-on-the-owners-mac-studio.md | 152 ++++++++++++++++++ ...odemode-as-the-agent-facing-mcp-surface.md | 135 ++++++++++++++++ docs/docs/reference/hardware-inventory.md | 14 ++ .../docs/reference/networking/address-plan.md | 9 +- 6 files changed, 486 insertions(+), 21 deletions(-) create mode 100644 docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md create mode 100644 docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md create mode 100644 docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md diff --git a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md index a4ac8d2..63aad5e 100644 --- a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md +++ b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md @@ -22,6 +22,8 @@ claiming the provider parent. Which fabric should the durable service use? - Sandbox traffic cannot initiate connections to management or OOB. - Fleet owns infrastructure; agents own only disposable sandbox resources. - Management boot and OVN central must not depend on OVN itself. +- Native `nat=false`, `network=none` isolation must use the same durable + control plane as NAT-enabled sandbox networks. - Failed creates and expired resources must converge through explicit, retryable deletion rather than undocumented service repairs. @@ -43,18 +45,18 @@ to `nas01` and attached to its unmanaged VLAN 10 `mgmt` bridge rather than an OVN network. The pinned `ovn-central` package version `26.03.0-2` supplies northd and standalone NB/SB databases, without Raft. Per-component TLS gates prevent either database from starting without the complete certificate trio. -Remote database connections use mutual TLS from a dedicated offline OVN CA, -not the KMS root or the Incus cluster certificate. The CA uses EC P-256, has a -ten-year validity and no path-length constraint. Central and each chassis have -separate two-year leaves. Per-chassis keys allow one member's identity to -rotate without distributing the same private key to every node. The TLS -cutover occurred only after the cluster carried the CVE-2026-40243 fix. +Remote database connections use mutual TLS from a dedicated offline OVN +application CA, not the KMS root or the Incus cluster certificate. The CA uses +EC P-256, has a ten-year validity and no path-length constraint. Central and +each chassis have separate two-year leaves. Per-chassis keys allow one member's +identity to rotate without distributing the same private key to every node. +The TLS cutover occurred only after the cluster carried the CVE-2026-40243 fix. The owner approved this application-scoped trust domain on 2026-09-12, -superseding the draft's requirement to use the ADR-0005 hierarchy. -**OVN trust domain is application-scoped and offline; ADR-0005 governs the -KMS root hierarchy and is unaffected.** Revisit the issuance model when Vault -PKI exists and there is a reason to migrate. ADR-0005 is not amended. +superseding the draft's requirement to use the ADR-0005 hierarchy. The +dedicated offline OVN CA is outside ADR-0005's KMS-root hierarchy; ADR-0005 is +not amended. Revisit the issuance model when Vault PKI exists and there is a +reason to migrate. The CA key and every leaf key are escrowed under the `fleet` scope in `GilmanLab/secrets`, following ADR-0003's alternative KMS/PGP recipients and @@ -78,11 +80,11 @@ after rotation; the central private key is never an OpenTofu input. Fleet's `cluster/` deployment owns chassis configuration on every member, using VLAN 30 tunnel addresses and central's VLAN 10 endpoint. Supported settings are mirrored in node seeds. The default-project physical uplink -exclusively owns its provider parent. Before convergence, fleet checks -member-specific networks, profiles, and instance NICs, including inherited -and stopped-instance devices. A competing direct parent attachment or -physical-uplink NIC aborts deployment with a named conflict; fleet never -removes it silently. +`fast40-uplink` exclusively owns the IncusOS `fast40` provider parent. Before +convergence, fleet checks member-specific networks, profiles, and instance +NICs, including inherited and stopped-instance devices. A competing direct +parent attachment or physical-uplink NIC aborts deployment with a named +conflict; fleet never removes it silently. The [address plan](../reference/networking/address-plan.md#ovn-external-addresses) is authoritative for the approved external allocation and its eight-sandbox @@ -105,6 +107,11 @@ consumes no external address, and has no direct path outside its sandbox. It becomes reachable only through `net.peer` or a router instance attached to another network. A `net.forward` request for an isolated network returns `AgentError`. +Both the overlay control plane and the native isolation contract require +central. Without central, Incus cannot create the project-owned logical switch +whose `network=none` setting makes a `nat=false` network isolated. Replacing +central would therefore require replacing the selected OVN fabric, not only +the central VM. The `default_network_kind=bridge` fallback keeps whole sandboxes on member-local bridges; explicit bridge networks remain bare wires there. Incus @@ -165,6 +172,25 @@ central stopped, `central-tls`, central running with the new CA, then fleet's OVN client/chassis converge. NB_Global, SB_Global, and two logical-switch UUIDs were preserved. The new `lab03` identity authenticated to both NB and SB; the old identity was rejected by both before an authenticated response. + +Phase 5 also exposed a stale in-memory trust failure after the OVN CA was +silently re-minted. Stored Incus configuration already named the new CA, but +the `lab01`, `lab02`, and `nas01` daemons continued reconnecting with the old +CA. Their failed handshakes filled the central VM's 20 GiB root filesystem. +After evidence capture and owner approval, recovery truncated only the three +identified log files and recycled those three Incus daemons serially. `lab03` +was left running because its daemon already held the new trust. All members +returned `Online`, central processes retained their PIDs and start times, and +the measured inbound reconnect rate fell from 1,039 per second to zero. + +[Fleet PR #20](https://github.com/GilmanLab/fleet/pull/20) prevents the same +silent transition: the ceremony refuses to mint an absent CA unless the +operator supplies `--mint-ca`, delivery paths require the reviewed CA +fingerprint, an OVN converge reports changed in-memory trust, and the explicit +trust-roll operation restarts selected daemons one at a time with an online +gate. [Root PR #34](https://github.com/GilmanLab/root/pull/34) records the +evidence-first, approval-gated recovery procedure in the canonical runbook. + A cross-member fixture completed three of three pings on each tested path. Public OpenTofu cloud-init metadata reconciliation was still pending AWS authentication and is not part of that proof. @@ -191,9 +217,10 @@ fixture residue across all four members and all projects in 3.03 seconds. The post-rotation and post-reboot fleet dry run proposed no changes in all 11 operations. -These checks complete the live technical confirmation recorded here. The record -remains `proposed`; its status must not be changed without the owner's -separate acceptance of the decision. +These checks establish the OVN infrastructure and lifecycle evidence recorded +here. They do not accept this decision or claim that Phase 9b validation is +complete. The record remains `proposed`; only the owner may change its status +to `accepted`. ## More Information diff --git a/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md new file mode 100644 index 0000000..26ebf26 --- /dev/null +++ b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md @@ -0,0 +1,134 @@ +--- +status: proposed +date: 2026-09-15 +decision-makers: + - Josh Gilman +--- + +# ADR-0007: Use Cua Driver over Guest Execution for Desktop Automation + +## Context and Problem Statement + +Agentcompute must expose the same desktop automation vocabulary for Linux, +Windows, and macOS guests without making a guest desktop service reachable from +every sandbox network. The chosen mechanism must support semantic accessibility +actions and screenshots, preserve isolated guest topologies, and keep binary +images out of CodeMode values. How should agentcompute reach and operate a +guest desktop? + +## Decision Drivers + +- Use one native desktop capability set across Linux, Windows, and macOS. +- Keep desktop control available when an Incus guest has no operator-reachable + network path. +- Avoid adding a Driver listener to sandbox network policy. +- Return structured Driver results while moving screenshots through a bounded + binary path. +- Retain a human-accessible fallback for setup, login, and recovery screens. + +## Considered Options + +- Cua Driver in the graphical session, reached through guest execution +- A network-exposed Cua MCP endpoint in every desktop guest +- Hypervisor console automation through SPICE or VNC +- Separate platform-specific automation implementations + +## Decision Outcome + +Proposed choice: **Cua Driver in the graphical session, reached through guest +execution**. This record remains `proposed` until the owner accepts it. + +Curated desktop images install the pinned Cua Driver and start its daemon in the +logged-in graphical session. Agentcompute exposes `desktop.info`, +`desktop.call`, and `desktop.screenshot` capabilities. `desktop.call` forwards a +native Driver tool name and JSON object without defining a second typed API for +each upstream tool. + +Linux and macOS use bounded one-shot Driver CLI calls. Linux calls travel +through Incus exec to the user-session socket. macOS calls travel through the +Lume backend's direct guest SSH path as user `lume`. There is no persistent Mac +Driver bridge. Windows is the exception: agentcompute keeps a Cua MCP session +over one guest exec stream through the in-guest session proxy, because the +Incus agent service identity cannot access the interactive user's named pipe +directly. A lost Windows response closes that session for the next call but +never replays an action whose effect is unknown. + +The Driver writes screenshots to a temporary guest file. Agentcompute pulls the +file through the backend's binary file path—Incus file access or SFTP for +macOS—validates and optionally resizes the PNG, and publishes a short-lived URL. +Screenshot bytes do not cross the CodeMode value boundary. In-guest VNC or +Lume's VM console remains a fallback for a human; it is not the agent automation +API. + +### Consequences + +- Good, because semantic actions, accessibility trees, application discovery, + and screenshots use the Driver's native cross-platform vocabulary. +- Good, because Incus desktop control does not depend on a guest network route + or an exposed Driver port. An isolated `nat=false`, `network=none` guest + remains controllable through the Incus agent. +- Good, because screenshot data uses a bounded binary transfer instead of + base64 or another large value inside a Starlark result. +- Bad, because each desktop image must maintain a logged-in graphical session, + a compatible Driver daemon, and platform-specific startup behavior. +- Bad, because Linux and macOS pay one-shot CLI startup cost for each operation, + while Windows adds persistent-session lifecycle and reconnect handling. +- Bad, because macOS Accessibility and Screen Recording consent must be granted + to the signed Driver app on the private seed; the service cannot grant it + unattended. +- Bad, because pre-login screens and a broken graphical session still require a + separately protected VNC or console path. + +### Confirmation + +Compliance is observable when all of the following remain true: + +- `search_api` finds the stable `desktop.*` capabilities, `describe_api` + reports their typed shapes, and `desktop.info` discovers the installed Driver + version and native tool names. +- A native read such as `list_apps` returns structured Driver content, and a + screenshot capability returns a fetchable PNG URL rather than image bytes. +- Linux and macOS Driver operations execute one-shot guest commands. Windows + reuses a Driver MCP session carried by a guest exec stream and does not replay + a call after a lost response. +- No desktop image exposes a Driver control listener as part of its sandbox + network contract. +- The owner has explicitly changed this record's status to `accepted`; technical + evidence alone does not accept the decision. + +## Pros and Cons of the Options + +### Cua Driver over Guest Execution + +- Good, because it combines a cross-platform semantic API with the existing + authenticated backend control paths. +- Bad, because guest images and graphical-session startup become part of the + automation contract. + +### Network-Exposed Cua MCP Endpoint + +- Good, because every platform could use a long-lived MCP connection. +- Bad, because each guest would need a listener, authentication, address + discovery, and policy exceptions that conflict with arbitrary or isolated + network topologies. + +### Hypervisor Console Automation + +- Good, because a console can cover boot and login screens before the guest + desktop service starts. +- Bad, because console pixels do not provide the Driver's native application, + window, and accessibility semantics. Incus exposes graphical VM consoles + through SPICE, for which this service has no supported client path. + +### Platform-Specific Automation + +- Good, because each platform could use its native automation framework + directly. +- Bad, because agentcompute would own three incompatible vocabularies, image + contracts, permission models, and result formats. + +## More Information + +- [Agentcompute design draft](../designs/drafts/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute-service.md) +- [ADR-0009: Use CodeMode as the Agent-Facing MCP Surface](0009-use-codemode-as-the-agent-facing-mcp-surface.md) diff --git a/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md new file mode 100644 index 0000000..2a33337 --- /dev/null +++ b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md @@ -0,0 +1,152 @@ +--- +status: proposed +date: 2026-09-15 +decision-makers: + - Josh Gilman +--- + +# ADR-0008: Run Lume in a Confined Account on the Owner's Mac Studio + +## Context and Problem Statement + +Mac guests require Apple hardware and Apple's Virtualization framework, so they +cannot run on the Incus cluster. The original agentcompute design assumed a +dedicated Mac mini, but the owner already operates an always-on Mac Studio that +can host the required private seed. Where should Lume run, and which boundary +should separate the backend from the owner's workstation account? + +## Decision Drivers + +- Run macOS guests on owner-controlled Apple Silicon. +- Keep the consented seed host-local and never redistribute it. +- Prevent the backend identity from administering the Mac or traversing the + owner's home directory. +- Expose the loopback Lume lifecycle API only through an authenticated, + source-restricted connection from `agentcompute01`. +- Make clone identity, guest-count enforcement, and cleanup explicit in the + service instead of relying on silent Lume behavior. + +## Considered Options + +- Lume under a confined account on the owner's always-on Mac Studio +- A dedicated Mac mini running Lume +- Tart or a custom Virtualization.framework backend on the Mac Studio +- Hosted macOS workers + +## Decision Outcome + +Proposed choice: **Lume under a confined account on the owner's always-on Mac +Studio**. The host is `studio-1`, a Mac Studio with an M2 Max and 64 GB of +memory. This record remains `proposed` until the owner accepts it. + +Lume and its VM store run as the hidden, standard `agentcompute` account. The +account is not an administrator, has no sudo access, and cannot traverse the +owner's home directory. Its VM store is `/Users/agentcompute/.lume`. Studio +remains a user-owned tailnet device rather than receiving a backend device tag. + +`lume serve` binds only to `127.0.0.1:7777`. Agentcompute reaches that lifecycle +API through the source-restricted Studio SSH account and local TCP forwarding. +Guest exec does not use `lume ssh`: the backend invokes system SSH directly to +the guest's Lume NAT lease with Studio as a `ProxyJump`, a separate per-image +guest key, and pinned host keys for both hops. File transfer uses SFTP over the +same direct guest path. + +The stopped, consented seed stays in the confined account's Lume store and is +not published. Lume assigns a new `machineIdentifier` when it clones a VM, but +a fresh clone with that identifier can enter Setup Assistant. Before the +clone's first boot, agentcompute copies the seed's `machineIdentifier` into the +stopped clone while retaining the clone's generated MAC address. Snapshot and +restore clones follow the same pre-boot identity rule. + +Apple permits at most two additional running macOS guests on a host. Before a +create, start, or restart makes a Lume HTTP mutation, agentcompute serializes +starts and counts every running macOS VM visible in the confined account, +regardless of name. It refuses a third. This check is required because Lume +0.5.3 accepts the third run request but leaves the VM stopped and reports the +limit only in its service log. A separate account can still consume a host-wide +slot, so a run that remains stopped is diagnosed against new Lume log output. + +The backend implementation and host arrangement have qualification evidence, +but the permanent agentcompute service configuration was restored without a +`lume` section after that qualification. The Studio VNC packet-filter boundary +also remains a rollout gate. This decision does not claim that the durable +service is serving Mac guests or that Phase 9b is complete. + +### Consequences + +- Good, because the lab can use an already available, always-on Apple Silicon + host without buying and operating a second Mac now. +- Good, because the backend's normal shell, Lume store, and loopback API are + confined to a non-administrator account separate from the owner's files. +- Good, because lifecycle calls, guest calls, and file transfer use distinct + keys and verify both SSH hops. +- Bad, because a personal host is now a backend availability and capacity + dependency. Owner use, restart, sleep, or maintenance can interrupt Mac + sandboxes. +- Bad, because the two-running-guest host limit is small and shared with any + macOS VM running under another account. +- Bad, because the seed contains an operator's one-time Accessibility and Screen + Recording consent and must remain private, stopped, and recoverable on this + host. +- Bad, because Lume's per-VM VNC listener uses an ephemeral host port and binds + broadly. Workers must not be served until the reviewed host packet-filter + boundary is installed and verified. +- Bad, because moving the backend later requires moving the confined account's + Lume store, reissuing the host key, and re-establishing host policy. + +### Confirmation + +Compliance is observable when all of the following remain true: + +- `agentcompute` is a hidden standard account with no administrator or sudo + access, the owner home denies traversal, and `lume serve` listens only on + `127.0.0.1:7777`. +- The durable service configuration enables Lume only after the source-pinned + SSH policy, separate host and guest keys, known-host pins, and Studio VNC + packet-filter boundary are installed and verified. +- Generated guest SSH configuration uses Studio as `ProxyJump`; guest exec and + SFTP do not invoke `lume ssh` and do not use an SSH agent. +- A fresh clone receives the seed's `machineIdentifier` while stopped and + before first boot, retains a distinct generated MAC address, and reaches the + desktop rather than Setup Assistant. +- A third running macOS guest is refused before a Lume run mutation. A + host-wide slot held by another account produces an explicit capacity error + rather than a false Running state. +- The owner has explicitly changed this record's status to `accepted`; a + qualification run or durable rollout does not accept the decision by itself. + +## Pros and Cons of the Options + +### Confined Account on the Owner's Mac Studio + +- Good, because it uses known hardware and keeps the backend separate from the + owner's login account. +- Bad, because the host is shared with owner activity and was not purchased as + dedicated service infrastructure. + +### Dedicated Mac Mini + +- Good, because service availability, packet filtering, and capacity would be + isolated from the owner's workstation. +- Bad, because it adds hardware and migration work before the two-guest backend + needs more capacity. It remains the migration option if sharing Studio stops + being acceptable. + +### Tart or a Custom Virtualization.framework Backend + +- Good, because another backend could offer different lifecycle or networking + behavior. +- Bad, because agentcompute would need to replace Lume's seed, clone, run, + snapshot, and VNC lifecycle instead of using the implemented backend. + +### Hosted macOS Workers + +- Good, because host maintenance and hardware ownership move to a provider. +- Bad, because the private consented seed, low-latency desktop control, and + sandbox lifecycle would cross a new trust and cost boundary. + +## More Information + +- [Agentcompute design draft](../designs/drafts/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute-service.md) +- [ADR-0007: Use Cua Driver over Guest Execution for Desktop Automation](0007-use-cua-driver-over-guest-execution.md) diff --git a/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md new file mode 100644 index 0000000..b785e05 --- /dev/null +++ b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md @@ -0,0 +1,135 @@ +--- +status: proposed +date: 2026-09-15 +decision-makers: + - Josh Gilman +--- + +# ADR-0009: Use CodeMode as the Agent-Facing MCP Surface + +## Context and Problem Statement + +Agentcompute has many lifecycle, network, file, and desktop operations. Exposing +each operation as a separate MCP tool would enlarge the public tool list and +force intermediate results through the model between dependent calls. How +should agents discover and compose the service's capabilities while keeping the +outer MCP contract small and stable? + +## Decision Drivers + +- Keep the public MCP tool list small as compute capabilities are added. +- Let an agent discover exact capability names and shapes at runtime. +- Compose dependent operations without returning every intermediate value to + the model. +- Validate typed capability inputs and authorize every native call. +- Bound program execution time, native calls, and values. + +## Considered Options + +- CodeMode with three MCP tools and bounded Starlark programs +- One direct MCP tool for every agentcompute operation +- One custom dispatcher tool accepting arrays of JSON operations +- Expose the underlying Incus, Lume, and Driver APIs directly + +## Decision Outcome + +Proposed choice: **CodeMode with three MCP tools and bounded Starlark +programs**. Agentcompute's outer MCP surface is exactly `search_api`, +`describe_api`, and `execute`. This record remains `proposed` until the owner +accepts it. + +`search_api` discovers registered capabilities by name and search metadata. +`describe_api` returns the generated input and output shape for one capability. +`execute` evaluates a bounded Starlark program and returns the converted value +of its zero-argument `main()` function. Capabilities use stable dotted names +under `sandbox`, `image`, `instance`, `net`, and `desktop`. + +Each capability is registered as a typed Go handler. Inputs are flat structs +whose direct exported fields are scalar `string`, `int64`, `bool`, or `float64` +values, with pointers for optional scalars. Input lists and nested objects are +not part of this contract. An agent repeats a capability call inside its +program for multi-value operations. Where the underlying native API itself +requires a free-form object, such as a Cua Driver call, agentcompute accepts an +explicit JSON object string and the handler validates it. Outputs may contain +bounded nested structs and lists. + +CodeMode creates a fresh Starlark worker for each `execute` request and permits +native calls only from `main()`. The worker is a language and execution +boundary, not an operating-system isolation boundary: capability handlers run +in the privileged parent service, and the worker runs as the same operating +system user as the service. Authentication and authorization therefore remain +host responsibilities outside program-controlled arguments. + +### Consequences + +- Good, because adding a capability does not add another outer MCP tool or + require clients to load a growing tool schema list. +- Good, because one program can create resources, wait, inspect results, and + clean up while returning only its final bounded value. +- Good, because generated descriptions and argument binding come from the same + Go types used by handlers. +- Good, because authorization is applied to canonical typed arguments for every + native call, including calls made inside one program. +- Bad, because clients must discover CodeMode and write Starlark instead of + calling an operation-specific MCP tool directly. +- Bad, because flat scalar inputs require repeated calls or an explicitly + validated string encoding when an operation naturally takes a collection or + nested object. +- Bad, because changing a capability's typed shape can invalidate saved + programs even though the outer three-tool list is unchanged. +- Bad, because CodeMode does not isolate handler side effects, filesystem, + credentials, CPU, or memory at the operating-system level. + +### Confirmation + +Compliance is observable when all of the following remain true: + +- MCP tool discovery returns exactly `search_api`, `describe_api`, and + `execute`. +- `search_api` finds an agentcompute capability, `describe_api` reports its + flat typed input, and `execute` can call it from `main()` and return its + result. +- A wrongly typed argument is rejected before handler dispatch. +- New operations are registered through `codemode.Register`; they do not call + `mcp.AddTool` or add another public dispatcher. +- Runtime limits remain configured for execution and native calls, and the + deployment continues to establish trusted subject identity outside program + source and capability arguments. +- The owner has explicitly changed this record's status to `accepted`; passing + discovery or execution checks does not accept the decision. + +## Pros and Cons of the Options + +### CodeMode + +- Good, because it supplies discovery, generated descriptions, typed binding, + per-call authorization, bounded composition, and a fixed MCP surface. +- Bad, because it introduces a worker protocol and Starlark execution model that + operators and contributors must understand. + +### Direct MCP Tool per Operation + +- Good, because ordinary MCP clients can call each operation without writing a + program. +- Bad, because the tool list and schema payload grow with every capability, and + dependent calls repeatedly move intermediate data through the model. + +### Custom JSON Operation Dispatcher + +- Good, because it could keep one public tool and batch several operations. +- Bad, because agentcompute would have to invent discovery, type checking, + references between results, control flow, limits, authorization points, and + error projection already supplied by CodeMode. + +### Direct Infrastructure APIs + +- Good, because it avoids an application-level capability layer. +- Bad, because it exposes provider-specific credentials and mutable + infrastructure primitives instead of the sandbox ownership, TTL, cleanup, + and policy boundaries agentcompute enforces. + +## More Information + +- [Agentcompute design draft](../designs/drafts/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute-service.md) +- [CodeMode documentation](https://meigma.github.io/codemode/) diff --git a/docs/docs/reference/hardware-inventory.md b/docs/docs/reference/hardware-inventory.md index b420cef..3da8e21 100644 --- a/docs/docs/reference/hardware-inventory.md +++ b/docs/docs/reference/hardware-inventory.md @@ -61,6 +61,20 @@ documentation, not this inventory. - `1x10GbE` and `1x5GbE` onboard networking - `2xUSB4`, HDMI, and OCuLink available +### Apple Mac Studio + +- Quantity: `1` +- Device name: `studio-1` +- Role: owner's always-on Apple Silicon host for the Lume macOS backend +- Hardware details: + - Apple M2 Max + - `64GB` memory +- Backend boundary: Lume and its VM store run under the dedicated hidden, + standard `agentcompute` account, not the owner's account. +- Deployment state: the seed and backend have qualification evidence on this + host. Permanent agentcompute service configuration and the Studio VNC packet + filter remain rollout gates. + ### MikroTik CRS309-1G-8S+IN - Quantity: `2` diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index f49c1b3..b70e19e 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -145,15 +145,18 @@ qualification spike central on `sandbox01` has been removed. See the [OVN central and certificate runbook](../../runbooks/ovn-central-and-certificates.md) for deployment, renewal, and recovery procedures. -### Incus-local image runner networks +### Incus-local and service networks -These NAT bridges are local to each Incus member, not routed VLAN prefixes. -Do not advertise them through `gw01` or Tailscale. +These private prefixes are not routed lab VLANs. Do not advertise them through +`gw01` or Tailscale. `incusbr0` and `github-runners` are member-local NAT +bridges. `ac-svc-vlan40` is a managed OVN network whose uplink remains +`fast40-uplink`. | Resource | Address or prefix | Ownership | | --- | --- | --- | | Existing `incusbr0` bridge | `10.158.84.0/24`, gateway `.1` | IncusOS | | `github-runners` bridge | `10.158.85.0/24`, gateway `.1` | Fleet `cluster/` | +| Agentcompute service network `ac-svc-vlan40` | `10.158.86.0/24`, gateway `.1`; `agentcompute01` guest NIC `.2` | Fleet OpenTofu root `incus/agentcompute/` | | Reserved controller VM `ghrunner01` on `nas01` | `10.158.84.50` | Fleet OpenTofu root `incus/incus-gh-runner/` | | Reserved HTTP CONNECT forward | `10.10.10.14:3128` → `10.158.84.50:3128` | Same OpenTofu root, member-local to `nas01` | From e5f50a75df0fe6f35af0a5bd4e93c6e169fa770f Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 19:26:32 -0700 Subject: [PATCH 2/7] docs(agentcompute): record runtime boundaries and operational procedures Promote canonical document paths, preserve proposal deviations, document the account-local no-VNC policy, and add discovery, TTL, recovery, and Mac maintenance procedures. Final Phase9b acceptance evidence and implemented status remain gated on live completion. --- docs/docs/architecture/agentcompute.md | 149 +++++++++++ .../0006-ovn-as-the-sandbox-network-fabric.md | 2 +- ...007-use-cua-driver-over-guest-execution.md | 4 +- ...nfined-account-on-the-owners-mac-studio.md | 36 ++- ...odemode-as-the-agent-facing-mcp-surface.md | 4 +- .../docs/designs/{drafts => }/agentcompute.md | 179 +++++++++++-- ...gentcompute-service.md => agentcompute.md} | 235 ++++++++++++++++-- docs/mkdocs.yml | 9 +- 8 files changed, 563 insertions(+), 55 deletions(-) create mode 100644 docs/docs/architecture/agentcompute.md rename docs/docs/designs/{drafts => }/agentcompute.md (80%) rename docs/docs/runbooks/{agentcompute-service.md => agentcompute.md} (69%) diff --git a/docs/docs/architecture/agentcompute.md b/docs/docs/architecture/agentcompute.md new file mode 100644 index 0000000..b7807ea --- /dev/null +++ b/docs/docs/architecture/agentcompute.md @@ -0,0 +1,149 @@ +--- +title: Agentcompute architecture +description: Runtime boundaries, trust, networking, and durable state of the lab compute service. +--- + +# Agentcompute architecture + +Agentcompute exposes short-lived compute environments to agents through one +Streamable HTTP MCP endpoint, `https://agentcompute01.tailda715.ts.net/`. +The agent-facing surface is `search_api`, `describe_api`, and `execute`; +individual compute and desktop operations are Starlark capabilities, not MCP +tools. See [ADR-0009](../decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md). + +This page describes the deployed boundaries. The +[implemented design](../designs/agentcompute.md) records the original intent +and deviations; the [runbook](../runbooks/agentcompute.md) owns deployment, +credential handling, and recovery. + +## Request and execution boundaries + +```mermaid +flowchart LR + Agent[Agent on tailnet] -->|HTTPS 443 and bearer| Serve[Tailscale Serve] + Serve -->|loopback 8080| MCP[agentcompute HTTP and CodeMode] + MCP --> Incus[Incus cluster API] + MCP -->|SSH 22 as agentcompute| Studio[Owner's Mac Studio] + Studio -->|loopback 7777| Lume[Lume daemon] + Studio -->|SSH ProxyJump| Mac[macOS guest] + Incus --> Linux[Linux guests] + Incus --> Windows[Windows guests] +``` + +Tailscale Serve terminates publicly trusted, automatically renewed HTTPS and +forwards to `127.0.0.1:8080` in the `agentcompute01` VM. The Go HTTP handler +requires a named static bearer token and rejects cross-origin requests. It +shares one backend registry and reaper across HTTP sessions. A client reconnect +or service restart does not create a second inventory. + +CodeMode evaluates bounded Starlark, not host Python or a host shell. Only +registered capabilities enter the backends. Guest `instance.exec` deliberately +executes commands inside the named sandbox guest; it does not expose Incus +credentials or a shell on `agentcompute01` to the caller. The `omp` bearer is +an administrator of this service surface, not a per-sandbox authorization +boundary. Subject metadata is attribution, not tenant isolation. + +## Incus and OVN + +The Incus backend owns projects prefixed `ac-`, with sandbox lifetime and +ownership in `user.agentcompute.*` metadata. Instances, networks, forwards, +snapshots, and sandbox-scoped published images belong to those projects. +Discovery reads backend state rather than a process-local inventory. + +OVN is the default network kind. NAT networks allocate external addresses from +`10.10.40.64–10.10.40.127` through `fast40-uplink`. An isolated `nat=false` +network does not attach to the uplink or consume an external NAT address. +Inter-network routing requires an explicit guest router or `net.peer`. +Network forwards allocate their own external listen addresses; they are not +implicitly the network's NAT address. See +[ADR-0006](../decisions/0006-ovn-as-the-sandbox-network-fabric.md) and the +[address plan](../reference/networking/address-plan.md). + +The service VM is in the Incus `default` project on `lab01`, outside disposable +sandbox projects. Its routed management NIC uses `10.10.10.16/32` through +`169.254.0.1`; its guest-side NIC is `10.158.86.2/24` on `ac-svc-vlan40`. +The route to VLAN 40 is through `10.158.86.1`. The tailnet address is +`100.65.152.20`. These paths do not bridge the IncusOS management bond or +require disabling `strict_hwaddr`. + +The dedicated Incus client is **unrestricted cluster root**. Incus 7.4's +project-creation authorization prevented the intended restricted `ac-*` +identity: the OpenFGA model warns at project creation, and the authorization +scriptlet cannot inspect the new project's name. A pre-created project pool +with a restricted claim/release identity is deferred. The current mitigation +is the narrow authenticated service boundary, not a claim of least-privilege +Incus credentials. + +## Mac backend + +The owner's Mac Studio runs Lume as a separate standard, hidden `agentcompute` +account. It is not a dedicated machine. Tailnet policy permits the service tag +to reach Studio only on SSH port 22. Studio's authorized-key rules restrict the +service key to the verified `100.65.152.20` source. See +[ADR-0008](../decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md). + +The backend reaches the host-local Lume HTTP daemon through SSH; port 7777 is +not published to the LAN or tailnet. Inventory and host work use the +account-local `/Users/agentcompute/bin/lume`, never the global install. +Guest execution uses SSH ProxyJump and SFTP with explicit host-key checking, +not a nonexistent `lume ssh` command. The qualified Tahoe seed supplies the +pinned guest host key and machine identifier. Retaining that machine identifier +preserves the qualified guest's activation and desktop permissions; the server +enforces a two-running-guest cap. The Mac backend does not pretend to provide +Incus OVN features. + +A pinned source build temporarily supplies upstream Lume's merged +`--vnc disabled` feature, absent from release 0.5.3. Every backend start, +including restore and restart, uses `noDisplay: true` and `vnc: disabled`. +Startup refuses a CLI or daemon that cannot enforce the policy. This avoids a +wildcard VNC listener at its source; no high-port PF rule is installed on the +owner's LAN interfaces. `pins/lume.yaml` in agentcompute records the exact +source commit, installed binary digest, and the unchanged 0.5.3 release fallback. +Return to a release pin once a release includes the feature. The global Lume +install, Internet Sharing, and Continuity services remain outside service +ownership. + +## Desktop transport and screenshots + +The desktop API wraps Cua Driver over existing guest-execution channels rather +than exposing a second guest TCP service. Linux and macOS use the Driver's +one-shot CLI. Windows uses its persistent MCP transport through the guest +execution path. This is the trust and transport decision in +[ADR-0007](../decisions/0007-use-cua-driver-over-guest-execution.md). + +Screenshot bytes are written under `/var/lib/agentcompute/screenshots` and +returned as opaque HTTPS URLs rooted at the service hostname. Screenshot URLs +are bearer capabilities: do not log or publish them for private workloads. +Their lifetime is tied to sandbox expiry/deletion; the HTTP service checks +that lifetime rather than treating a file on disk as perpetual authorization. +The service's systemd credential directory and Studio keys are not available +to guest desktop tools. + +## Persistence and failure semantics + +- Incus projects and Lume sandbox metadata survive an MCP process restart. + The reaper reconciles expired sandboxes on its next pass, including those + that expired while the process was stopped; it is not an in-memory timer. +- Incus snapshot restore stages a stopped copy before deleting the original, + then renames the staged instance, reapplies current agentcompute metadata, + and starts it. UUID, NIC identity, and DHCP address can change. + Deleting the original also deletes its snapshot + tree. This is recreate semantics, not an in-place filesystem rollback. +- A failed create or delete can leave backend resources needing reconciliation. + The runbook distinguishes a slow asynchronous operation from a stuck + sandbox and requires checking ownership before direct backend cleanup. +- Fleet OpenTofu owns the long-lived VM/network/DNS and public runtime + configuration. Credentials are delivered out of band into root-only files + and loaded by systemd. Application release installation converges the + public configuration and binary without rerunning cloud-init or replacing + the VM and its Tailscale identity. + +## Ownership + +`GilmanLab/agentcompute` owns the server, capability contracts, backend code, +image qualification, and release artifacts. `GilmanLab/fleet` owns service +infrastructure and runtime deployment. `GilmanLab/networking` owns tailnet and +network policy. `GilmanLab/secrets` owns encrypted credentials. This root +repository owns architecture, decisions, deployment guidance, and acceptance +evidence. Changes crossing those boundaries require companion PRs; runtime +experiments are not a substitute for updating the owning repository. diff --git a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md index 63aad5e..3c0bb6e 100644 --- a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md +++ b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md @@ -224,7 +224,7 @@ to `accepted`. ## More Information -- [Agentcompute design draft](../designs/drafts/agentcompute.md) +- [Agentcompute design](../designs/agentcompute.md) - [Phase 3 qualification and parent recovery](https://github.com/GilmanLab/agentcompute/blob/spike/ovn-recreate-diagnosis/spikes/ovn/README.md): first post-fixture-deletion lab01-gateway cycle passed in 31.018 seconds, without reboot, central restart, or neighbor repair. - [Incus #3985](https://github.com/lxc/incus/issues/3985): unavailable NB creation leaves an `Errored` network; deletion after central recovery releases it. - [Incus #3986](https://github.com/lxc/incus/issues/3986): raw macvlan parent contention is separate from the central-outage failure. diff --git a/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md index 26ebf26..01c5cfb 100644 --- a/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md +++ b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md @@ -129,6 +129,6 @@ Compliance is observable when all of the following remain true: ## More Information -- [Agentcompute design draft](../designs/drafts/agentcompute.md) -- [Deploy and operate agentcompute](../runbooks/agentcompute-service.md) +- [Agentcompute design](../designs/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute.md) - [ADR-0009: Use CodeMode as the Agent-Facing MCP Surface](0009-use-codemode-as-the-agent-facing-mcp-surface.md) diff --git a/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md index 2a33337..31c79d5 100644 --- a/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md +++ b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md @@ -66,11 +66,21 @@ regardless of name. It refuses a third. This check is required because Lume limit only in its service log. A separate account can still consume a host-wide slot, so a run that remains stopped is diagnosed against new Lume log output. -The backend implementation and host arrangement have qualification evidence, -but the permanent agentcompute service configuration was restored without a -`lume` section after that qualification. The Studio VNC packet-filter boundary -also remains a rollout gate. This decision does not claim that the durable -service is serving Mac guests or that Phase 9b is complete. +Lume 0.5.3 starts a wildcard VNC listener even for `--display none`. The owner +rejected a blanket high-port PF rule because it could disrupt LAN Continuity +and `rapportd`. Upstream [cua#3209](https://github.com/trycua/cua/pull/3209) +merged an explicit disabled-VNC policy, but it was not in release 0.5.3 at +rollout. The approved temporary exception is an exact source commit pinned in +`pins/lume.yaml`, with `source_build: true` and the built binary's SHA-256. +It installs only for `agentcompute`; the system-wide Lume remains unchanged. + +Every backend run requests disabled VNC and no display. Startup fails closed +if either the account-local CLI or the running daemon lacks that policy. +Verification samples account-owned TCP listeners from request through Running. +No PF rule is installed, so this change does not alter Internet Sharing or the +owner's Continuity services. Once a release includes the feature, replace the +source pin with that release. The 0.5.3 release pin is retained as a rollback +reference, **not** an automatic fallback that may serve backend guests with VNC. ### Consequences @@ -88,9 +98,9 @@ service is serving Mac guests or that Phase 9b is complete. - Bad, because the seed contains an operator's one-time Accessibility and Screen Recording consent and must remain private, stopped, and recoverable on this host. -- Bad, because Lume's per-VM VNC listener uses an ephemeral host port and binds - broadly. Workers must not be served until the reviewed host packet-filter - boundary is installed and verified. +- Bad, because a temporary source build adds build/signing provenance and an + account-local deployment step until upstream publishes disabled VNC in a + release. Both CLI and daemon must be upgraded together. - Bad, because moving the backend later requires moving the confined account's Lume store, reissuing the host key, and re-establishing host policy. @@ -101,9 +111,9 @@ Compliance is observable when all of the following remain true: - `agentcompute` is a hidden standard account with no administrator or sudo access, the owner home denies traversal, and `lume serve` listens only on `127.0.0.1:7777`. -- The durable service configuration enables Lume only after the source-pinned - SSH policy, separate host and guest keys, known-host pins, and Studio VNC - packet-filter boundary are installed and verified. +- The durable service configuration enables Lume only after source-pinned SSH, + separate host and guest keys, known-host pins, and disabled-VNC enforcement + are verified. No account-owned per-VM VNC listener appears during startup. - Generated guest SSH configuration uses Studio as `ProxyJump`; guest exec and SFTP do not invoke `lume ssh` and do not use an SSH agent. - A fresh clone receives the seed's `machineIdentifier` while stopped and @@ -147,6 +157,6 @@ Compliance is observable when all of the following remain true: ## More Information -- [Agentcompute design draft](../designs/drafts/agentcompute.md) -- [Deploy and operate agentcompute](../runbooks/agentcompute-service.md) +- [Agentcompute design](../designs/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute.md) - [ADR-0007: Use Cua Driver over Guest Execution for Desktop Automation](0007-use-cua-driver-over-guest-execution.md) diff --git a/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md index b785e05..8ed66a9 100644 --- a/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md +++ b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md @@ -130,6 +130,6 @@ Compliance is observable when all of the following remain true: ## More Information -- [Agentcompute design draft](../designs/drafts/agentcompute.md) -- [Deploy and operate agentcompute](../runbooks/agentcompute-service.md) +- [Agentcompute design](../designs/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute.md) - [CodeMode documentation](https://meigma.github.io/codemode/) diff --git a/docs/docs/designs/drafts/agentcompute.md b/docs/docs/designs/agentcompute.md similarity index 80% rename from docs/docs/designs/drafts/agentcompute.md rename to docs/docs/designs/agentcompute.md index e31c1db..14ce893 100644 --- a/docs/docs/designs/drafts/agentcompute.md +++ b/docs/docs/designs/agentcompute.md @@ -5,13 +5,21 @@ authors: - Josh Gilman - agent (session 019) created: 2026-09-11 -updated: 2026-09-13 +updated: 2026-09-16 related-decisions: - ADR-0006 + - ADR-0007 + - ADR-0008 + - ADR-0009 --- # agentcompute — disposable compute for agents +The proposal below is retained so implementation deviations remain visible. +Read [Implementation Outcome](#implementation-outcome) for those deviations, +the [architecture](../architecture/agentcompute.md) for runtime boundaries, +and the [runbook](../runbooks/agentcompute.md) for operations. + ## Summary `agentcompute` is a lab-specific MCP server, built on @@ -52,7 +60,7 @@ The lab already has the substrate: management or OOB). It is carried on the cluster's fast links, and the default-project physical network `fast40-uplink` exclusively owns the IncusOS-owned `fast40` parent. Sandbox NICs use managed logical networks. See the - [address plan](../../reference/networking/address-plan.md). + [address plan](../reference/networking/address-plan.md). - macOS guests cannot run on Incus. They need Apple hardware running Apple's Virtualization framework, driven by [Lume](https://cua.ai/docs/reference/lume/cli-reference) (or Tart; see @@ -134,7 +142,7 @@ but no OVN central. The lab's durable `ovncentral01` VM is pinned to `nas01`'s unmanaged management bridge and runs `ovn-northd` with standalone NB/SB databases. Both databases require mutual TLS from the application-scoped offline OVN CA. See the -[OVN central and certificate runbook](../../runbooks/ovn-central-and-certificates.md). +[OVN central and certificate runbook](../runbooks/ovn-central-and-certificates.md). Each sandbox network is an Incus OVN network: a Geneve overlay that spans the cluster, with a logical router, DHCP, DNS, network ACLs, and peering built in. A NAT-enabled network also has SNAT and can host network forwards. An isolated @@ -372,7 +380,7 @@ behavior were qualified separately and are recorded under Validation. ADR-0003. The CA is not part of ADR-0005's KMS hierarchy; revisit Vault issuance when Vault PKI exists and there is a reason to migrate. The transitional central on `sandbox01` has been purged. See the - [operations runbook](../../runbooks/ovn-central-and-certificates.md). + [operations runbook](../runbooks/ovn-central-and-certificates.md). 2. **OVN chassis on every node — complete.** Fleet converges `/os/1.0/services/ovn` on all four members, with each member's VLAN 30 storage address as its Geneve tunnel address and its own TLS leaf. The @@ -380,7 +388,7 @@ behavior were qualified separately and are recorded under Validation. 3. **An OVN uplink on the sandbox VLAN — complete.** Fleet owns the cluster-wide `physical` uplink on the IncusOS-owned `fast40` parent and the approved external range. The - [address plan](../../reference/networking/address-plan.md#ovn-external-addresses) + [address plan](../reference/networking/address-plan.md#ovn-external-addresses) is the only source for that allocation and its capacity calculation. 4. **An Incus identity for `agentcompute`** with rights to create projects. Fleet `cluster/` concern. @@ -516,7 +524,7 @@ networks and distinct forward listen addresses consume them. The representative `default` NAT, isolated `lan`, `wan` NAT, and one-forward topology consumes three addresses per sandbox: eight sandboxes consume 24 and leave 40 for additional allocations and pending cleanup. The -[address plan](../../reference/networking/address-plan.md#ovn-external-addresses) +[address plan](../reference/networking/address-plan.md#ovn-external-addresses) is authoritative. ## Delivery @@ -665,11 +673,12 @@ fleet dry run was a no-op. - No isolation, no desktops, no topologies, no cleanup. - Not chosen. -## Open Questions +## Design Questions -### Resolved +### Resolved in review -Resolved in review, kept here until the draft is promoted: +These decisions describe the reviewed proposal; later amendments are recorded +in Implementation Outcome rather than silently rewriting the proposal. - **The word `sandbox`** stays. Agents already think in it; the collision is with a host and a repository, neither of which an agent sees. @@ -683,7 +692,7 @@ Resolved in review, kept here until the draft is promoted: - **Blocking operations** stay. Raise `MaxConcurrentExecutions` before considering an asynchronous pattern. - **OVN central** is the OpenTofu-owned `ovncentral01` VM on `nas01`; no Raft. - The [operations runbook](../../runbooks/ovn-central-and-certificates.md) + The [operations runbook](../runbooks/ovn-central-and-certificates.md) defines its deployment, renewal, and recovery. Whether it later serves Talos networks is a T10/T11 question, not this design's. - **Windows and macOS images are not published** to any registry; they @@ -691,7 +700,7 @@ Resolved in review, kept here until the draft is promoted: - **`desktop.call` ergonomics**: pass-through first; typed conveniences only for tools agents demonstrably fumble. - **VLAN 40** is the durable OVN uplink VLAN (owner decision, 2026-09-12). - The [address plan](../../reference/networking/address-plan.md#ovn-external-addresses) + The [address plan](../reference/networking/address-plan.md#ovn-external-addresses) records the approved 64-address allocation and eight-sandbox planning target. Default-plus-forward consumes two external addresses per sandbox; adding an isolated `lan` and NAT-enabled `wan` brings that budget to three. @@ -700,13 +709,141 @@ Resolved in review, kept here until the draft is promoted: needs more than this allocation; DHCP, named endpoints, and routes stay unchanged. -### Still open - -1. **Cua Driver token continuity across one-shot CLI calls.** Decided by - the desktop spike: snapshot in one call, act by token in the next. If - tokens do not survive, `desktop.call` drives a long-lived - `cua-driver serve` per instance over exec; the vocabulary is unchanged. -2. **Cua Driver inside images.** How the daemon is kept alive in the - graphical session on each OS is image-recipe work learned in the - desktop slices. Pin the newest non-prerelease Driver per image; bump - deliberately. +### Resolved during implementation + +1. **Driver token continuity:** Linux and macOS retain the one-shot CLI + transport over guest exec. Windows needs a persistent Driver MCP session + over guest exec. Both retain the `desktop.call` vocabulary; see + [ADR-0007](../decisions/0007-use-cua-driver-over-guest-execution.md). +2. **Driver process lifetime:** image recipes own the graphical-session + service or LaunchAgent, its pinned binary, and readiness qualification. + Guest OSes differ; this is not one universal daemon recipe. +3. **Mac location:** the owner approved the existing Mac Studio under a + separate standard account, not the proposed dedicated Mac mini. Its + availability and two-guest capacity are shared with the owner's workloads. + +## Implementation Outcome + +The service is delivered as a Go CodeMode MCP server in the long-lived +`agentcompute01` Incus VM, with Incus/OVN for Linux and Windows and Lume for +macOS. Runtime and image state remain in their backends, not in MCP sessions. +The root documentation set is authoritative for architecture, decisions, and +operations; source repositories own code, images, and deployment inputs. +ADRs 0006–0009 remain **proposed** until the owner accepts them. Implementation +and successful qualification do not silently accept an ADR. + +### Security and authorization deviations + +- **Incus identity is unrestricted.** The intended `ac-*` project-creation + certificate could not work with Incus 7.4's authorization model. OpenFGA + warns that project creation is root-equivalent, and the scriptlet cannot + inspect the new project's name at creation time. The dedicated + `agentcompute01` identity therefore has cluster-root rights behind the + tailnet, bearer-authenticated MCP boundary. Guest code never receives that + identity. A pre-created restricted project pool with claim/release metadata + is deferred, not partially implemented. See the + [runbook safety boundary](../runbooks/agentcompute.md#safety-boundary). +- **Private-image workflow triggers use a private repository.** The public + implementation repository cannot be the trust gate for self-hosted workers + carrying private Windows/macOS material. The organization's free plan does + not provide the restricted runner-group controls needed for that public + repository arrangement. Private triggers and protected runner access were + used instead. See [private image runners](../runbooks/private-image-runners.md). +- **HTTPS uses Tailscale Serve's public ACME certificate**, not the proposed + internal PKI leaf. Clients use normal public trust and the fixed tailnet + hostname. Serve terminates HTTPS and forwards only to loopback; named bearer + authentication remains mandatory. + +### OVN ownership, PKI, and recovery + +- The default-project physical network `fast40-uplink` exclusively owns the + IncusOS `fast40` parent. Sandbox NICs attach to managed logical networks, + not raw macvlan on that same parent. Raw parent contention was a separate + failure from the unavailable-central network-creation failure. +- The offline OVN CA is an owner-approved application-scoped trust domain + **outside ADR-0005's KMS-root hierarchy**. ADR-0005 itself is unchanged. + Revisit issuance when Vault PKI exists and migration has an operational + reason; do not imply that hierarchy is already deployed for OVN. +- A silently re-minted CA left several Incus daemons reconnecting with stale + in-memory trust even though stored configuration named the new CA. Repeated + failed TLS handshakes filled the central VM's 20 GiB root filesystem. + Owner-approved recovery truncated only the identified logs and recycled + `lab01`, `lab02`, and `nas01` serially; `lab03` already held the new trust. + Central processes stayed up. [Fleet #20](https://github.com/GilmanLab/fleet/pull/20) + now makes CA creation explicit, checks reviewed fingerprints, and provides + an online-gated serial trust roll. [Root #34](https://github.com/GilmanLab/root/pull/34) + records evidence-first recovery. The complete incident and rationale are in + [ADR-0006](../decisions/0006-ovn-as-the-sandbox-network-fabric.md). +- Address accounting remains explicit: default NAT plus forward consumes two + external addresses; adding a NAT-enabled WAN consumes a third. Isolated + `nat=false` LANs consume none. Count failed NAT network allocations until + deletion. The approved range remains `10.10.40.64–10.10.40.127`. + +### Mac implementation amendments + +- The host is the owner's always-on Mac Studio, confined to the hidden + standard `agentcompute` account. Neither administrator rights, the owner's + home, nor a broad host-network permission is part of that account. +- Cloning with a fresh Lume `machineIdentifier` could return the qualified + guest to Setup Assistant. Before first boot, the backend retains the seed's + identifier while keeping the clone's new MAC. Snapshot clones use the same + rule. Consent and activation are seed properties, not permission grants + performed by the server. +- The server serializes starts and enforces the two-running-macOS-guest + limit before Lume mutations. It counts every running macOS VM in the + confined account and diagnoses host-wide capacity occupied elsewhere. +- There is no `lume ssh` transport. Guest exec and SFTP use system SSH through + Studio as `ProxyJump`, with distinct host/guest keys and both host keys + pinned. The guest pin uses the seed name, not a DHCP address. +- Lume 0.5.3's wildcard VNC server cannot be disabled with `--display none`. + The owner rejected a blanket high-port PF block because of Continuity and + `rapportd`. A dynamic port watcher would still permit an initial exposure + window. Instead, [agentcompute #39](https://github.com/GilmanLab/agentcompute/pull/39) + pins the merged upstream no-VNC commit, builds as `agentcompute`, and + installs only in that account. Every backend run disables VNC, and startup + checks both CLI and daemon support. The global install, PF, and Internet + Sharing remain unchanged. This temporary source artifact is SHA-256 pinned + but **not bitwise reproducible after a clean rebuild**. Return to a release + pin once upstream publishes the feature. A human may explicitly enable VNC + for a bounded maintenance console; no automatic fallback does so. + +### Runtime and image contract details + +- Incus snapshot restore is **recreate and start**: stop the original, stage a + snapshot copy, delete the original and its snapshot tree, rename the copy, + preserve current agentcompute metadata, and start it. It is not in-place + rollback. Phase 9b observed the same name with changed UUID, MAC, and DHCP + address, restored file contents, and an empty snapshot list. +- The illustrative NAT program omitted guest configuration needed after a + hot NIC attach. Live qualification explicitly brought up the router WAN, + acquired its DHCP lease, selected the WAN default route, and configured + the isolated client's default route/DNS through that router. A WAN packet + capture proved the client's connection source was the router WAN address; + merely checking NAT rules was not accepted as evidence. +- Fleet release installation now converges the public configuration, catalog, + unit, and SSH pins in place, after artifact checks. It does not replace the + service VM, rerun cloud-init, overwrite private credentials, or re-enroll + Tailscale. Bootstrap/network/certificate changes retain the deliberate + replacement procedure. +- Windows and macOS images remain local and do not have published image + attestations. Binary/container release attestations exist, but the build + still occurs outside the reusable attesting job: **SLSA Level 3 is not + claimed**. This supply-chain gap remains deferred rather than being hidden + by the presence of a signature. + +### Known residuals + +- `sandbox.list` can race expiry/deletion between listing sandbox names and + reading their instance counts. A missing-sandbox error is already converted + to an agent-facing error before the MCP listing loop; a local string-match + suppression would be brittle and could hide real failures. The race is + recorded, not papered over. Sequential lifecycle, expiry, and restart + acceptance remain separate checks. +- A missing executable/exit-127 case can surface as a generic capability error + from Incus and discard the enclosing program's result. Qualification hit + this with deliberately missing commands and an unavailable BusyBox applet; + valid command, routing, and desktop checks used real installed programs. + Error reporting was not broadened as an unrelated Phase 9b change. +- Mac cold-clone Driver scheduling has required an explicit LaunchAgent + kickstart in earlier qualification while TCC grants remained present. + The runbook separates that finding from permission re-consent. diff --git a/docs/docs/runbooks/agentcompute-service.md b/docs/docs/runbooks/agentcompute.md similarity index 69% rename from docs/docs/runbooks/agentcompute-service.md rename to docs/docs/runbooks/agentcompute.md index 61d08cb..91ac5a2 100644 --- a/docs/docs/runbooks/agentcompute-service.md +++ b/docs/docs/runbooks/agentcompute.md @@ -44,6 +44,45 @@ tailnet. - After enrollment, follow [Authorize agentcompute SSH to Studio](agentcompute-studio-ssh.md) to pin Studio access to the service's verified tailnet IPv4. +- On macOS, point the OpenTofu provider at the administration client's actual + configuration: `export INCUS_CONF="$HOME/Library/Application Support/incus"`. + Incus 7.3 uses that directory, while provider 1.2.0 defaults to + `$HOME/.config/incus`. A missing pin caused by that mismatch is not a reason + to enable certificate acceptance. + +## Configure an agent client + +Use a Streamable HTTP MCP connection, not an SSE-only URL or local stdio +server: + +| Setting | Value | +| --- | --- | +| Server name | `agentcompute` | +| Transport | Streamable HTTP | +| URL | `https://agentcompute01.tailda715.ts.net/` | +| HTTP header | `Authorization: Bearer ` | +| Token source | SOPS `auth_tokens.omp` in `services/agentcompute/credentials.sops.yaml` | +| TLS | Default public trust; never disable verification | + +Resolve the token through the client's secret or environment support. Do not +commit its value to client configuration, pass it in a command argument, or +paste it into an agent prompt. The authenticated initialization recipe below +keeps it on curl's standard input. + +The client discovers exactly three MCP tools: `search_api`, `describe_api`, +and `execute`. Start with a resource query such as `{"query":"sandbox"}`; +describe exact returned names before using them in `def main():`. Search is +not free-form planning: if a multi-resource sentence returns no matches, use +one resource or exact capability name. A minimal discovery-driven health +check describes `sandbox.list` and executes: + +```python +def main(): + return sandbox.list() +``` + +Use `sandbox.get` or `instance.list` for inventory details. Creating resources +is not necessary for every client health check. ## Safety boundary @@ -137,13 +176,14 @@ jq -e 'keys == ["omp"] and (.omp | type == "string" and length > 0)' \ "$AGENTCOMPUTE_CREDENTIALS/auth-tokens.json" >/dev/null ``` -Nothing private enters cloud-init, OpenTofu input, or OpenTofu state. The public -Incus client certificate, pinned server certificate, image catalog, and -non-secret runtime configuration do enter cloud-init. The three private source -files are delivered after the VM exists as `root:root` mode `0600` files in a -mode `0700` directory. The unit uses four `LoadCredential=` entries: the Incus -client certificate and key, bearer-token file, and Studio SSH key. Treat the -runtime credential mount as unit-private; do not infer its in-unit access from +Nothing private enters cloud-init, OpenTofu input, or OpenTofu state. Public +certificates, image catalog, SSH host-key pins, and runtime configuration do. +The three base private inputs are delivered after provisioning as `root:root` +mode `0600` files in a mode `0700` directory. Enabling Lume adds a fourth +private input, `mac-guest.key`, delivered separately from the qualified +Studio account. The unit loads the Incus client certificate/key, bearer-token +file, Studio SSH key, and (when enabled) Mac guest key with `LoadCredential=`. +Treat the runtime credential mount as unit-private; do not infer its access from ownership or mode observed outside the service namespace. ## Enroll the Incus identity @@ -483,6 +523,135 @@ Two findings remain relevant to operations: finding, not an HTTPS delivery failure; full-desktop capture is not qualified by this deployment. +## Operate the Mac backend + +The host is the owner's `studio-1` Mac Studio, not a dedicated appliance. +Use only the hidden standard `agentcompute` account, its Lume store, and its +loopback daemon. Do not grant it administrator access or access to the owner's +home. The stopped seed is `ac-seed-macos-tahoe-desktop`; it is private and must +not be published. Two running macOS guests is the host-wide limit. + +### Install and activate the no-VNC build + +Lume 0.5.3 has no VNC-disable option. The temporary source build is pinned in +[`agentcompute/pins/lume.yaml`](https://github.com/GilmanLab/agentcompute/blob/master/pins/lume.yaml). +It records upstream commit `ab957bdb7566f7e137b00654cc01167d9e42af38`, +the actual signed Mach-O SHA-256, toolchain, dependency lock, and release +fallback. `--version` still prints `0.5.3`; that is **not** artifact verification. +The measured clean build is not bitwise reproducible. A checksum mismatch +requires a reviewed re-pin, not skipping the installer check. + +From the owner account, stage only the public build inputs outside the +protected home, then build as `agentcompute`: + +```bash +LUME_STAGE="$(mktemp -d)" +mkdir -p "$LUME_STAGE/images/macos" "$LUME_STAGE/pins" +cp "$AGENTCOMPUTE_DIR/images/macos/build-lume.sh" \ + "$AGENTCOMPUTE_DIR/images/macos/lib.sh" "$LUME_STAGE/images/macos/" +cp "$AGENTCOMPUTE_DIR/pins/lume.yaml" "$LUME_STAGE/pins/" +chmod -R a+rX "$LUME_STAGE" +sudo -u agentcompute -H "$LUME_STAGE/images/macos/build-lume.sh" +rm -rf "$LUME_STAGE" +``` + +The installer refuses root and writes only beneath `/Users/agentcompute`. +The launcher is `/Users/agentcompute/bin/lume`; the signed executable is +`/Users/agentcompute/.local/share/lume/lume.app/Contents/MacOS/lume`. +Leave `/usr/local/bin/lume` and its global app bundle untouched. + +With every backend worker stopped and the seed stopped, repoint only the +existing account daemon. Preserve its `UserName`, `HOME`, log paths, and +KeepAlive settings. Back up the plist first. Replace the whole argument array; +do not insert another executable as an extra argument: + +```bash +plist=/Library/LaunchDaemons/io.gilman.agentcompute.lume-serve.plist +sudo plutil -replace ProgramArguments -json \ + '["/Users/agentcompute/bin/lume","serve","--port","7777"]' "$plist" +sudo launchctl bootout system/io.gilman.agentcompute.lume-serve +sudo launchctl bootstrap system "$plist" +sudo lsof -nP -a -u agentcompute -iTCP -sTCP:LISTEN +``` + +Require only `127.0.0.1:7777` for the lifecycle API. Probe a unique, +nonexistent, colon-free name with a deliberately conflicting policy: + +```bash +curl --silent --show-error --write-out '\nHTTP %{http_code}\n' \ + --json '{"noDisplay":false,"vnc":"disabled"}' \ + "http://127.0.0.1:7777/lume/vms/ac-vnc-policy-probe-$(uuidgen)/run" +``` + +Require HTTP `400` and `VNC is disabled for this run`. The pinned daemon +validates this before any VM operation. Old 0.5.3 ignores the policy and returns +`202`; that is a failed gate, not permission to run a worker. + +No PF configuration is installed. A blanket high-port block was rejected +because it could disrupt LAN Continuity and `rapportd`; dynamic port watching +would leave a pre-discovery exposure window. Preserve Internet Sharing and +Apple's live PF anchors. Sample `lsof` continuously from a real backend run +request through Running: no account-owned per-VM VNC listener may appear, and +Lume inventory must report `vncUrl: null`. + +### Deliver permanent runtime configuration + +Fleet's `lume_host` is Studio's verified tailnet IPv4, `100.122.142.76`. +The fleet module supplies `studio_known_hosts` and `mac_guest_known_hosts`; +the latter pins the qualified seed's SSH key by seed-name `HostKeyAlias`, +not a reusable DHCP address. Verify pins through the already trusted host/seed, +never accept a fresh key merely because `ssh-keyscan` returned it. + +Stage the qualified seed's guest key from Studio in the owner-only credential +directory. It is separate from the service-to-Studio SSH key: + +```bash +sudo cat /Users/agentcompute/.ssh/guest_ed25519 \ + >"$AGENTCOMPUTE_CREDENTIALS/mac-guest.key" +chmod 0600 "$AGENTCOMPUTE_CREDENTIALS/mac-guest.key" +cd "$FLEET_DIR/incus/agentcompute" +just deliver-lume-key "$AGENTCOMPUTE_CREDENTIALS/mac-guest.key" +``` + +Review and apply the fleet plan with `lume_host` enabled and the release's +catalog, including `macos/tahoe/desktop`. Install the verified compatible +release to converge the public runtime bundle and load the new credential. +Require a clean plan, successful authenticated initialization, Mac create, +`sw_vers`, `desktop.screenshot`, and delete through deployed HTTPS. A temporary +systemd override is not permanent backend rollout. + +### Manual console and guest re-consent + +`desktop.screenshot` is the normal observation path. There is no automatic VNC +fallback. If a human needs a console, stop the exact guest, then **start it by +hand with VNC enabled**, for example as the confined account: + +```bash +sudo -u agentcompute -H /Users/agentcompute/bin/lume run \ + --no-display --vnc enabled +``` + +This deliberately opens Lume's wildcard VNC listener for the maintenance +session. Treat its URL/password as a secret; do not leave it running unattended +or expose it on an untrusted LAN. Stop it afterward and return normal workers +to the backend's disabled-VNC start path. + +For lost desktop permissions, have the operator open the **guest's** System +Settings → Privacy & Security. Re-enable CuaDriver under Accessibility and +Screen Recording; add `/Applications/CuaDriver.app` to Screen Recording if +absent. Let the operator authenticate and approve each permission. Do not +modify TCC databases, grant owner-host permissions, or automate consent. +Restart the guest's `com.trycua.cua_driver_daemon` LaunchAgent, then recheck +Driver readiness and a real screenshot. Requalify a disposable clone before +returning the stopped seed to service. A cold clone that needs a LaunchAgent +kickstart is a known qualification finding, not evidence of missing TCC consent. + +Return to an account-local release pin once an upstream Lume release includes +[cua#3209](https://github.com/trycua/cua/pull/3209). Retire the source-build +procedure then, while retaining disabled-VNC enforcement. Falling back to the +old global 0.5.3 binary requires disabling the Mac backend; it is not an +equivalent console-safe release. + ## Routine operations ### Inspect and restart @@ -503,18 +672,56 @@ reconciles the pinned image catalog, rediscovers persisted sandboxes, and runs an immediate reaper scan before the next 30-second interval. Do not start a second process to preserve sessions during the restart. +### Check TTL cleanup + +Discover `sandbox.create`, `sandbox.get`, `instance.create`, and +`sandbox.list`; create a uniquely named one-minute sandbox with a small +running guest. Record the returned `expires_at`. Poll from fresh MCP sessions +until it disappears, then check the backend: + +```bash +incus project list nas01: --format=json +``` + +The corresponding `ac-` project must be absent within two minutes after +expiry, not merely hidden from the MCP list. For Mac sandboxes, also inspect +the confined account's Lume inventory and sandbox metadata. A restart must +still rediscover unexpired sandboxes and reap expired ones. Do not infer +cleanup from a successful HTTP response alone. + +### Recover a stuck sandbox + +1. Describe and call `sandbox.get` and `instance.list`. Capture the sandbox + name, expiry, guest state, and operation error, without bearer tokens or + screenshot URLs. +2. Check `journalctl -u agentcompute.service` and the matching Incus operation + or account-local Lume log. A slow clone/boot is not itself a stuck delete. +3. Use `sandbox.delete` first; it owns ordered resource cleanup. If it fails, + retry after resolving the reported backend problem. Do not create a second + project with the same name or delete shared OVN resources. +4. Before direct backend cleanup, verify the exact `ac-` project and + `user.agentcompute.*` ownership/expiry metadata. Stop or delete only its + instances, forwards, and networks, then its project. On Studio, operate + only on the named backend clone and metadata; never the stopped seed or + another user's VM. +5. Recheck both MCP and backend inventory. A transient not-found error during + concurrent expiry is different from residual backend resources. + ### Upgrade the release Update `service_version` and `service_sha256` together in `release.auto.tfvars`. Verify the new release with the procedure above, review and apply the saved OpenTofu plan, then run `just install-release` with the -verified asset. A release-pin plan must not replace the VM. The script resolves -its module directory, reads `release_installer` from applied state, and replaces -the VM-side installer before it stages the asset. It therefore needs the -initialized state backend and `lab-admin` AWS access, but no reboot or VM -replacement. Do not update only the symlink or download from the VM. Keep the -previous versioned binary until the new endpoint passes the complete -verification procedure. +verified asset. A release-pin plan must not replace the VM. The script reads +`release_installer` from applied state, replaces the VM-side installer, and +stages the asset. After digest and embedded-version verification, it installs +the public runtime bundle (config, catalog, SSH pins, and unit) and binary, +then restarts an already-running service. It never overwrites credentials, +network configuration, certificates, or Tailscale state. It needs initialized +state and `lab-admin` AWS access, but no reboot or VM replacement. Network and +bootstrap changes still require a deliberate replacement; changing cloud-init +metadata alone does not converge the live VM. Keep the previous versioned +binary until the new endpoint passes full verification. ### Rotate credentials diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index dc9f7fc..cc5b511 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -38,9 +38,14 @@ nav: - "ADR-0004: Manage RouterOS Devices with OpenTofu": decisions/0004-manage-routeros-devices-with-opentofu.md - "ADR-0005: Anchor Internal PKI on the KMS Root with Sibling Intermediates": decisions/0005-anchor-internal-pki-on-kms-root-with-sibling-intermediates.md - "ADR-0006: OVN as the Sandbox Network Fabric": decisions/0006-ovn-as-the-sandbox-network-fabric.md + - "ADR-0007: Cua Driver over guest execution": decisions/0007-use-cua-driver-over-guest-execution.md + - "ADR-0008: Lume on the owner's Mac Studio": decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md + - "ADR-0009: CodeMode MCP surface": decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md + - Architecture: + - Agentcompute: architecture/agentcompute.md - Designs: - Lab v2 core network: designs/lab-v2-core-network.md - - "Draft: agentcompute": designs/drafts/agentcompute.md + - Agentcompute: designs/agentcompute.md - Reference: - Device naming: reference/naming.md - Hardware inventory: reference/hardware-inventory.md @@ -52,7 +57,7 @@ nav: - Tailscale policy: reference/networking/tailscale-policy.md - Runbooks: - Change the Tailscale policy: runbooks/tailscale-policy-change.md - - Deploy and operate agentcompute: runbooks/agentcompute-service.md + - Deploy and operate agentcompute: runbooks/agentcompute.md - Authorize agentcompute SSH to Studio: runbooks/agentcompute-studio-ssh.md - Commission a lab compute node: runbooks/commission-lab-node.md - Operate OVN central and certificates: runbooks/ovn-central-and-certificates.md From 5bd0b84e60a13116fc61270daf82b2a9daa7e28c Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 20:31:30 -0700 Subject: [PATCH 3/7] docs(agentcompute): record live qualification and corrected boundaries --- docs/docs/architecture/agentcompute.md | 14 ++-- .../0006-ovn-as-the-sandbox-network-fabric.md | 4 +- ...007-use-cua-driver-over-guest-execution.md | 4 +- ...nfined-account-on-the-owners-mac-studio.md | 13 +-- ...odemode-as-the-agent-facing-mcp-surface.md | 7 +- docs/docs/designs/agentcompute.md | 37 ++++++--- docs/docs/reference/hardware-inventory.md | 7 +- .../docs/reference/networking/address-plan.md | 17 ++-- docs/docs/runbooks/agentcompute.md | 81 ++++++++++++++++--- 9 files changed, 134 insertions(+), 50 deletions(-) diff --git a/docs/docs/architecture/agentcompute.md b/docs/docs/architecture/agentcompute.md index b7807ea..a58a810 100644 --- a/docs/docs/architecture/agentcompute.md +++ b/docs/docs/architecture/agentcompute.md @@ -98,7 +98,8 @@ including restore and restart, uses `noDisplay: true` and `vnc: disabled`. Startup refuses a CLI or daemon that cannot enforce the policy. This avoids a wildcard VNC listener at its source; no high-port PF rule is installed on the owner's LAN interfaces. `pins/lume.yaml` in agentcompute records the exact -source commit, installed binary digest, and the unchanged 0.5.3 release fallback. +source commit, installed binary digest, and 0.5.3 as a manual rollback +reference that backend startup rejects. Return to a release pin once a release includes the feature. The global Lume install, Internet Sharing, and Continuity services remain outside service ownership. @@ -114,8 +115,9 @@ execution path. This is the trust and transport decision in Screenshot bytes are written under `/var/lib/agentcompute/screenshots` and returned as opaque HTTPS URLs rooted at the service hostname. Screenshot URLs are bearer capabilities: do not log or publish them for private workloads. -Their lifetime is tied to sandbox expiry/deletion; the HTTP service checks -that lifetime rather than treating a file on disk as perpetual authorization. +Their lifetime is at most five minutes, shortened by sandbox expiry or deletion, +and they do not survive a service restart. The HTTP service checks that lifetime +rather than treating a file on disk as perpetual authorization. The service's systemd credential directory and Studio keys are not available to guest desktop tools. @@ -124,9 +126,9 @@ to guest desktop tools. - Incus projects and Lume sandbox metadata survive an MCP process restart. The reaper reconciles expired sandboxes on its next pass, including those that expired while the process was stopped; it is not an in-memory timer. -- Incus snapshot restore stages a stopped copy before deleting the original, - then renames the staged instance, reapplies current agentcompute metadata, - and starts it. UUID, NIC identity, and DHCP address can change. +- Incus snapshot restore stops the original and stages a copy carrying current + agentcompute metadata, then deletes the original, renames the copy, and + starts it. UUID, NIC identity, and DHCP address can change. Deleting the original also deletes its snapshot tree. This is recreate semantics, not an in-place filesystem rollback. - A failed create or delete can leave backend resources needing reconciliation. diff --git a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md index 3c0bb6e..18070c1 100644 --- a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md +++ b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md @@ -96,8 +96,8 @@ Reconsider a dedicated VLAN only if OVN needs more than the approved 64-address block. The representative topology has NAT-enabled `default` and `wan` networks, an isolated `lan`, and one distinct forward listen address. It consumes three -external addresses per sandbox: eight sandboxes consume 24 and leave 40 of -the 64-address reservation. +external addresses per sandbox. Eight sandboxes consume 24; the service's +`ac-svc-vlan40` network consumes one more, leaving 39 of the 64-address reservation. New sandbox projects use project-owned OVN networks and managed-only NICs. The default network has NAT. An additional NAT-enabled network consumes one diff --git a/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md index 01c5cfb..3c0ecd3 100644 --- a/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md +++ b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md @@ -40,7 +40,7 @@ execution**. This record remains `proposed` until the owner accepts it. Curated desktop images install the pinned Cua Driver and start its daemon in the logged-in graphical session. Agentcompute exposes `desktop.info`, -`desktop.call`, and `desktop.screenshot` capabilities. `desktop.call` forwards a +`desktop.enable`, `desktop.call`, and `desktop.screenshot`. `desktop.call` forwards a native Driver tool name and JSON object without defining a second typed API for each upstream tool. @@ -87,7 +87,7 @@ Compliance is observable when all of the following remain true: reports their typed shapes, and `desktop.info` discovers the installed Driver version and native tool names. - A native read such as `list_apps` returns structured Driver content, and a - screenshot capability returns a fetchable PNG URL rather than image bytes. + screenshot capability returns a fetchable image URL rather than image bytes. - Linux and macOS Driver operations execute one-shot guest commands. Windows reuses a Driver MCP session carried by a guest exec stream and does not replay a call after a lost response. diff --git a/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md index 31c79d5..3efcbc0 100644 --- a/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md +++ b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md @@ -58,10 +58,11 @@ clone's first boot, agentcompute copies the seed's `machineIdentifier` into the stopped clone while retaining the clone's generated MAC address. Snapshot and restore clones follow the same pre-boot identity rule. -Apple permits at most two additional running macOS guests on a host. Before a -create, start, or restart makes a Lume HTTP mutation, agentcompute serializes -starts and counts every running macOS VM visible in the confined account, -regardless of name. It refuses a third. This check is required because Lume +Apple permits at most two additional running macOS guests on a host. +Agentcompute serializes starts and counts every running macOS VM visible in +the confined account, regardless of name. It refuses a third before the Lume +run mutation; cloning and configuring a stopped VM do not consume a running slot. +This check is required because Lume 0.5.3 accepts the third run request but leaves the VM stopped and reports the limit only in its service log. A separate account can still consume a host-wide slot, so a run that remains stopped is diagnosed against new Lume log output. @@ -76,7 +77,7 @@ It installs only for `agentcompute`; the system-wide Lume remains unchanged. Every backend run requests disabled VNC and no display. Startup fails closed if either the account-local CLI or the running daemon lacks that policy. -Verification samples account-owned TCP listeners from request through Running. +The backend polls Lume until Running and stops any guest reporting a VNC URL. No PF rule is installed, so this change does not alter Internet Sharing or the owner's Continuity services. Once a release includes the feature, replace the source pin with that release. The 0.5.3 release pin is retained as a rollback @@ -114,6 +115,8 @@ Compliance is observable when all of the following remain true: - The durable service configuration enables Lume only after source-pinned SSH, separate host and guest keys, known-host pins, and disabled-VNC enforcement are verified. No account-owned per-VM VNC listener appears during startup. +- Live qualification separately samples account-owned TCP listeners from the + create request through Running; only the daemon's loopback listener may exist. - Generated guest SSH configuration uses Studio as `ProxyJump`; guest exec and SFTP do not invoke `lume ssh` and do not use an SSH agent. - A fresh clone receives the seed's `machineIdentifier` while stopped and diff --git a/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md index 8ed66a9..0ad198a 100644 --- a/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md +++ b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md @@ -21,7 +21,7 @@ outer MCP contract small and stable? - Let an agent discover exact capability names and shapes at runtime. - Compose dependent operations without returning every intermediate value to the model. -- Validate typed capability inputs and authorize every native call. +- Validate typed inputs and expose an authorization point for every native call. - Bound program execution time, native calls, and values. ## Considered Options @@ -68,8 +68,9 @@ host responsibilities outside program-controlled arguments. clean up while returning only its final bounded value. - Good, because generated descriptions and argument binding come from the same Go types used by handlers. -- Good, because authorization is applied to canonical typed arguments for every - native call, including calls made inside one program. +- Good, because CodeMode passes canonical typed arguments to the authorizer for + every native call. The deployed policy is `authz.AllowAll`: bearer + authentication is the current gate, and subject identity is attribution. - Bad, because clients must discover CodeMode and write Starlark instead of calling an operation-specific MCP tool directly. - Bad, because flat scalar inputs require repeated calls or an explicitly diff --git a/docs/docs/designs/agentcompute.md b/docs/docs/designs/agentcompute.md index 14ce893..529bacc 100644 --- a/docs/docs/designs/agentcompute.md +++ b/docs/docs/designs/agentcompute.md @@ -778,6 +778,9 @@ and successful qualification do not silently accept an ADR. external addresses; adding a NAT-enabled WAN consumes a third. Isolated `nat=false` LANs consume none. Count failed NAT network allocations until deletion. The approved range remains `10.10.40.64–10.10.40.127`. + The service's `ac-svc-vlan40` network consumes one address from that same + range. Eight representative sandboxes therefore leave **39**, not the + proposal's 40, after infrastructure use. ### Mac implementation amendments @@ -790,9 +793,9 @@ and successful qualification do not silently accept an ADR. rule. Consent and activation are seed properties, not permission grants performed by the server. - The server serializes starts and enforces the two-running-macOS-guest - limit before Lume mutations. It counts every running macOS VM in the + limit before each Lume run mutation. It counts every running macOS VM in the confined account and diagnoses host-wide capacity occupied elsewhere. -- There is no `lume ssh` transport. Guest exec and SFTP use system SSH through +- The runtime does not use the unavailable `lume ssh`. Guest exec and SFTP use system SSH through Studio as `ProxyJump`, with distinct host/guest keys and both host keys pinned. The guest pin uses the seed name, not a DHCP address. - Lume 0.5.3's wildcard VNC server cannot be disabled with `--display none`. @@ -820,16 +823,26 @@ and successful qualification do not silently accept an ADR. the isolated client's default route/DNS through that router. A WAN packet capture proved the client's connection source was the router WAN address; merely checking NAT rules was not accepted as evidence. +- Linux whole-desktop capture was black because Cua Driver's cosmetic + agent-cursor overlay froze X root-window reads before GNOME's first frame. + Starting it later could freeze a coloured frame instead. Window capture + remained live; VNC was affected too. Driver 0.28.2 reproduced the defect. + [Agentcompute #41](https://github.com/GilmanLab/agentcompute/pull/41) starts + the image's Driver with `--no-overlay`, retaining the reviewed 0.28.1 pin. + This removes synthetic session cursors, not input or the native cursor. + Image qualification now requires whole-desktop pixels to change after + launching Text Editor; a black-frame heuristic would miss a coloured freeze. - Fleet release installation now converges the public configuration, catalog, unit, and SSH pins in place, after artifact checks. It does not replace the service VM, rerun cloud-init, overwrite private credentials, or re-enroll Tailscale. Bootstrap/network/certificate changes retain the deliberate replacement procedure. - Windows and macOS images remain local and do not have published image - attestations. Binary/container release attestations exist, but the build - still occurs outside the reusable attesting job: **SLSA Level 3 is not - claimed**. This supply-chain gap remains deferred rather than being hidden - by the presence of a signature. + attestations. Private Linux image bakes also omit the older hosted + `attest.yml` step: digest verification and boot qualification are not signed + build provenance. Binary/container release attestations exist, but their + build still occurs outside the reusable attesting job: **SLSA Level 3 is not + claimed**. These supply-chain gaps remain deferred. ### Known residuals @@ -844,6 +857,12 @@ and successful qualification do not silently accept an ADR. this with deliberately missing commands and an unavailable BusyBox applet; valid command, routing, and desktop checks used real installed programs. Error reporting was not broadened as an unrelated Phase 9b change. -- Mac cold-clone Driver scheduling has required an explicit LaunchAgent - kickstart in earlier qualification while TCC grants remained present. - The runbook separates that finding from permission re-consent. +- Mac cold-clone readiness now checks the GUI session, kickstarts the existing + Driver LaunchAgent, and verifies the daemon's existing TCC grants before + returning. Phase 9b required no manual intervention or new permission grant. + Missing consent still requires a human; the server does not grant it. +- The existing Mac seed was qualified manually. Legacy + `images/macos/provision.sh` and `verify.sh --clone` still depend on the + unavailable `lume ssh` and are not qualified rebuild automation. The runbook + uses deployed MCP create/readiness/exec/screenshot/delete to requalify clones. + Porting those legacy bootstrap scripts is deferred. diff --git a/docs/docs/reference/hardware-inventory.md b/docs/docs/reference/hardware-inventory.md index 3da8e21..1ea1938 100644 --- a/docs/docs/reference/hardware-inventory.md +++ b/docs/docs/reference/hardware-inventory.md @@ -71,9 +71,10 @@ documentation, not this inventory. - `64GB` memory - Backend boundary: Lume and its VM store run under the dedicated hidden, standard `agentcompute` account, not the owner's account. -- Deployment state: the seed and backend have qualification evidence on this - host. Permanent agentcompute service configuration and the Studio VNC packet - filter remain rollout gates. +- Console boundary: the account-local pinned Lume build supports disabled VNC; + backend starts require it. No broad high-port PF block is installed, so + Internet Sharing and the owner's Continuity services retain their policies. + See the [Mac operations runbook](../runbooks/agentcompute.md#operate-the-mac-backend). ### MikroTik CRS309-1G-8S+IN diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index b70e19e..846b67f 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -126,9 +126,9 @@ isolated networks. The planning target is eight concurrent sandboxes for the single operator and their agents. The representative three-network topology uses 24 of the -64 addresses and leaves 40 for additional NAT-enabled networks, distinct -forward addresses, and pending cleanup. These are address-budget -calculations, not measured scale limits. +64 addresses; the service's `ac-svc-vlan40` network holds one more. This leaves +39 for additional NAT-enabled networks, distinct forward addresses, and +pending cleanup. These are address-budget calculations, not measured scale limits. An `Errored` NAT-enabled OVN network retains its external address until deleted and counts against capacity. If central is unavailable, the reaper @@ -212,11 +212,12 @@ workloads stay off the management plane. These links are not required for IncusOS management boot. The host-side counterpart, converged by the `GilmanLab/fleet` `cluster/` project, is the `vlan_tags` allow-list plus an IncusOS-declared VLAN interface per carried VLAN (`fast30`, `fast40`). -Instances must attach to the IncusOS-owned interface (for example macvlan -with `parent=fast40`), never with an Incus-created `vlan=` sub-interface on -`fast`: the visible `fast` device is an IncusOS-internal VLAN-filtering -bridge, and only IncusOS-declared VLANs receive bridge self-port membership, -so other tagged sub-interfaces pass no traffic. Additional instance VLANs +The default-project physical network `fast40-uplink` exclusively owns `fast40`. +Workload NICs use managed OVN networks, not raw macvlan or bridge attachments +to that same parent. For any additional instance VLAN, use an IncusOS-declared +interface, never an Incus-created `vlan=` sub-interface on `fast`: the visible +`fast` device is an IncusOS-internal VLAN-filtering bridge, and only +IncusOS-declared VLANs receive bridge self-port membership. Additional VLANs join these links when their first consumer arrives. ## DHCP and DNS ownership diff --git a/docs/docs/runbooks/agentcompute.md b/docs/docs/runbooks/agentcompute.md index 91ac5a2..a8e05b6 100644 --- a/docs/docs/runbooks/agentcompute.md +++ b/docs/docs/runbooks/agentcompute.md @@ -132,6 +132,7 @@ enrollment. ```bash export GLAB_SECRETS_DIR="$HOME/code/glab/secrets" export FLEET_DIR="$HOME/code/lab2/fleet" +export AGENTCOMPUTE_DIR="$HOME/code/lab2/agentcompute" export AWS_PROFILE=lab-admin umask 077 @@ -181,8 +182,11 @@ certificates, image catalog, SSH host-key pins, and runtime configuration do. The three base private inputs are delivered after provisioning as `root:root` mode `0600` files in a mode `0700` directory. Enabling Lume adds a fourth private input, `mac-guest.key`, delivered separately from the qualified -Studio account. The unit loads the Incus client certificate/key, bearer-token -file, Studio SSH key, and (when enabled) Mac guest key with `LoadCredential=`. +Studio account by `just deliver-lume-key`. With `lume_host` set, deliver it +**before** `just deliver-credentials` on a first deployment or VM replacement; +the unit is gated on that file too. The unit loads the Incus client +certificate/key, bearer-token file, Studio SSH key, and enabled Mac guest key +with `LoadCredential=`. Treat the runtime credential mount as unit-private; do not infer its access from ownership or mode observed outside the service namespace. @@ -314,6 +318,10 @@ object with unique names and tokens and no whitespace or control characters. It pushes all three private files as root-owned mode `0600`, then restarts the unit. Install the release first so the initial service start cannot select an unverified binary. +With `lume_host` enabled, first follow +[the Mac guest-key delivery procedure](#deliver-permanent-runtime-configuration). +That delivery does not start the service; the base delivery below does. + ```bash just deliver-credentials \ @@ -483,6 +491,45 @@ Require a successful `initialize` result naming `agentcompute` before declaring the MCP endpoint accepted. A healthy systemd unit, valid TLS, and an HTTP `401` without a token do not prove authenticated MCP handling. +### Phase 9b qualification — 2026-09-16 + +Live qualification used the deployed HTTPS endpoint, normal certificate validation, +and the SOPS-managed `omp` identity from fresh workstation processes. The +initial network, TTL, snapshot, and Windows checks ran on v0.1.2; Mac rollout +and the later restart/concurrency checks ran on +[`v0.1.3`](https://github.com/GilmanLab/agentcompute/releases/tag/v0.1.3), +commit `d85426b72257998a339b91f854c8928ca9d3e361`. Its Linux amd64 SHA-256 is +`d7e8e5229e7299612f858ba6d39718d402fc3b80017cbe3da0c7c0e2a6859c63`; +the checksum and exact-tag/commit GitHub attestation passed before installation. + +| Contract | Observed result | +| --- | --- | +| One-minute TTL with a running VM | `p9b-ttl` expired at `02:04:44Z`; absent at `02:05:07.499Z`, 23.499 seconds later. `ac-p9b-ttl` was also absent from Incus project listing. | +| LAN-only client through guest NAT | Client `192.168.50.3` had only its LAN NIC and used router `192.168.50.2`. Verified HTTPS returned `200`; WAN capture showed source `10.99.0.2`, the router's WAN address. | +| Cross-member OVN and workstation forward | `lab01` and `lab02` containers exchanged three of three pings in each direction. Workstation HTTP to `10.10.40.67:8080` returned `p9b-final-forward`. | +| Isolated networks | `network=none`, `ipv4.nat=false`, and no external allocation. Direct lab/internet pings failed; intentional `net.peer` traffic succeeded both ways. A forward on an isolated network returned `AgentError`. | +| External-address budget | The representative sandbox held `.69` for default NAT, `.70` for WAN NAT, and `.71` for its forward; LAN held none. The service network separately held `.65`. Eight such sandboxes use 24 addresses, leaving 39 after the service's one address. | +| Snapshot semantics | Restore recovered the original file contents and returned `Running` with the same name but a different UUID, MAC, and DHCP address; all original snapshots were consumed. Separate create/list/delete checks retained only the named snapshot until its deletion. | +| Windows ready screenshot | Three initial captures took 1.146, 1.150, and 1.148 seconds. After a later service restart, readiness took 6.182 seconds and the subsequent real desktop capture took 1.379 seconds. | +| Restart rediscovery | `systemctl restart agentcompute.service` preserved discovery of all four then-existing sandboxes: three Incus and one Mac, with the same ten-instance total and expiry metadata. | +| Eight blocking executions | Eight simultaneous `sleep 5` programs returned their distinct results successfully. Tool times were 5.671–6.015 seconds; total workstation wall time, including client setup, was 7.381 seconds. | +| Mac lifecycle | HTTPS create returned Running in 44.992 seconds; `sw_vers` reported macOS 26.6.2 / 25G83; Driver 0.28.1 was ready; the 1920×1200 screenshot showed the desktop. Instance and sandbox deletion left only the stopped seed in Lume. | +| No-VNC boundary | Sampling began before the Mac create request and continued through Running. The first 1,000 listener observations contained only account-owned `127.0.0.1:7777`, no guest VNC listener. PF filter/NAT rules, the global Lume digest, and owner `rapportd` listeners were unchanged. | +| Fresh-shell operation | The runbook's service, routing, Serve, DNS, public TLS, unauthenticated `401`, and SOPS-authenticated initialization commands passed under `bash --noprofile --norc` with a minimal environment. | + +A separate isolated HTTP fault fixture exercised the actual Incus SDK and +restore adapter without faulting the deployed cluster. Stop failure retained +the running original and snapshot; copy failure retained them with the original +stopped; delete failure retained the stopped original plus staged copy; +rename failure retained the staged copy; start failure retained the stopped +replacement under the original name. The success control returned that +replacement Running. All six cases passed; the temporary harness was removed. + +Windows's persistent guest MCP session has a cold-start cost. A direct first +screenshot after restart, without a new readiness probe, took 5.275 seconds. +The sub-two-second result is measured **after readiness**, not a cold-call +latency guarantee. + ### Phase 9a qualification — 2026-09-15 - Deployed [`v0.1.1`](https://github.com/GilmanLab/agentcompute/releases/tag/v0.1.1), @@ -511,7 +558,7 @@ without a token do not prove authenticated MCP handling. access as `agentcompute`, and rejected the same key from Studio's own source with `Permission denied (publickey)`. -Two findings remain relevant to operations: +Two findings informed the later qualification: 1. `v0.1.0` rejected Tailscale Serve's preserved public `Host` header on the loopback listener. `v0.1.1` permits that path only with configured bearer @@ -519,9 +566,10 @@ Two findings remain relevant to operations: for unauthenticated loopback servers. 2. Whole-desktop capture returned a black `1280×800` image even after Text Editor launched. Explicit window capture rendered the editor correctly - (`822×642`), and accessibility state was available. This is a desktop-capture - finding, not an HTTPS delivery failure; full-desktop capture is not qualified - by this deployment. + (`822×642`), and accessibility state was available. Phase 9a did not qualify + whole-desktop capture. Phase 9b traced this to the Driver's cosmetic cursor + overlay freezing X root reads, including VNC; see the + [implementation outcome](../designs/agentcompute.md#runtime-and-image-contract-details). ## Operate the Mac backend @@ -545,7 +593,7 @@ From the owner account, stage only the public build inputs outside the protected home, then build as `agentcompute`: ```bash -LUME_STAGE="$(mktemp -d)" +LUME_STAGE="$(mktemp -d /tmp/lume-build.XXXXXX)" mkdir -p "$LUME_STAGE/images/macos" "$LUME_STAGE/pins" cp "$AGENTCOMPUTE_DIR/images/macos/build-lume.sh" \ "$AGENTCOMPUTE_DIR/images/macos/lib.sh" "$LUME_STAGE/images/macos/" @@ -642,9 +690,12 @@ Screen Recording; add `/Applications/CuaDriver.app` to Screen Recording if absent. Let the operator authenticate and approve each permission. Do not modify TCC databases, grant owner-host permissions, or automate consent. Restart the guest's `com.trycua.cua_driver_daemon` LaunchAgent, then recheck -Driver readiness and a real screenshot. Requalify a disposable clone before -returning the stopped seed to service. A cold clone that needs a LaunchAgent -kickstart is a known qualification finding, not evidence of missing TCC consent. +Driver readiness and a real screenshot. Qualify a disposable clone through the +deployed MCP endpoint: `desktop.info` must report readiness, `sw_vers` must +identify the expected release, and `desktop.screenshot` must show the desktop. +Then delete the clone and retain the stopped seed. The legacy +`images/macos/provision.sh` and `verify.sh --clone` scripts still assume +`lume ssh`; they are not a qualified rebuild or recovery path. Return to an account-local release pin once an upstream Lume release includes [cua#3209](https://github.com/trycua/cua/pull/3209). Retire the source-build @@ -749,6 +800,11 @@ For a release regression, return `release.auto.tfvars` to the last accepted version and digest, review and apply the plan, then install that verified asset through `just install-release`. Confirm the embedded tag commit and repeat the full verification procedure. Do not point the symlink at an unverified file. +With `lume_host` enabled, retain agentcompute **v0.1.3 or newer**: older releases +do not enforce disabled VNC. Disable the Mac backend before deliberately +rolling back below that boundary; never silently substitute the global Lume +0.5.3 binary or enable VNC to make a rollback start. + For a VM replacement, use a saved plan with an explicit replacement and expect to lose the VM's tailnet node identity and delivered files: @@ -759,8 +815,9 @@ tofu show tfplan tofu apply tfplan ``` -After cloud-init completes, install the release and deliver credentials. Before -enrollment, remove the old, offline `agentcompute01` device from the tailnet so +After cloud-init completes, install the release, deliver the Mac guest key if +Lume is enabled, then deliver the three base credentials to start the unit. +Before enrollment, remove the old, offline `agentcompute01` device from the tailnet so the replacement can receive the exact MagicDNS name. Then enroll and run the full verification. Never accept a collision name such as `agentcompute01-1`; the enrollment script refuses to publish Serve under that name. From f9709241957b415a9d880d9bc2aac3af0998e523 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 20:43:05 -0700 Subject: [PATCH 4/7] docs(agentcompute): record qualified image and recovery boundaries --- docs/docs/designs/agentcompute.md | 17 +++++++++++++---- docs/docs/runbooks/agentcompute.md | 20 +++++++++++--------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/docs/docs/designs/agentcompute.md b/docs/docs/designs/agentcompute.md index 529bacc..575b424 100644 --- a/docs/docs/designs/agentcompute.md +++ b/docs/docs/designs/agentcompute.md @@ -472,10 +472,13 @@ new volatile NIC state, so its MAC address, DHCP identity, lease, and assigned address can change. Callers must rediscover the instance and its addresses after restore. -Failures before the staged copy completes leave the original stopped with its -snapshots intact; inspect for a generated `restore-*` copy before retrying. If -original deletion fails, the error identifies the retained staged copy. After -original deletion, its snapshots are already consumed: a rename failure leaves +A stop failure preserves the original and its snapshots; inspect its power +state. A copy failure after a successful stop leaves the original stopped with +its snapshots intact; inspect for a generated `restore-*` copy before retrying. +Deletion removes the original's snapshots before deleting the original itself. +If that final deletion fails, the original and staged copy can both remain, +but the snapshots are already consumed; the error identifies the staged copy. +After original deletion, a rename failure leaves the replacement under its generated name. A start or `Running`-wait failure returns an error with the replacement under the original agent-facing name; inspect its status before retrying. @@ -832,6 +835,12 @@ and successful qualification do not silently accept an ADR. This removes synthetic session cursors, not input or the native cursor. Image qualification now requires whole-desktop pixels to change after launching Text Editor; a black-frame heuristic would miss a coloured freeze. + The fresh private bake from commit `cb7f8c6d80bf1e66ea75ce28f07510db9636c34e` + passed that live-pixel check. Its Ubuntu desktop OCI digest is + `sha256:b2a83d7d70de2239de0bb04e255c96079446afb97ebcd36434892acefa5f8b0e`; + [catalog promotion #44](https://github.com/GilmanLab/agentcompute/pull/44) + records the qualified image references. The generated catalog commit was + signed by the integrating operator to satisfy branch policy; no bypass was used. - Fleet release installation now converges the public configuration, catalog, unit, and SSH pins in place, after artifact checks. It does not replace the service VM, rerun cloud-init, overwrite private credentials, or re-enroll diff --git a/docs/docs/runbooks/agentcompute.md b/docs/docs/runbooks/agentcompute.md index a8e05b6..839488a 100644 --- a/docs/docs/runbooks/agentcompute.md +++ b/docs/docs/runbooks/agentcompute.md @@ -245,11 +245,12 @@ incus exec --project default nas01:agentcompute01 -- \ jq -e '.status == "done" and .errors == []' >/dev/null ``` -Cloud-init is not a convergence mechanism. Except for a release pin, whose -installer is refreshed from applied state by `just install-release`, a changed -rendered configuration requires deliberate VM replacement followed by release -installation, credential delivery, and tailnet enrollment. Do not rerun the old -bootstrap on an existing VM. +Cloud-init is not a convergence mechanism. After a reviewed plan is applied, +`just install-release` converges the public runtime bundle—configuration, +catalog, unit, and SSH host pins—alongside the verified release. Bootstrap, +network, and certificate changes still require deliberate VM replacement, +release installation, credential delivery, and tailnet enrollment. Do not +rerun the old bootstrap on an existing VM. ## Verify and install the pinned release @@ -301,10 +302,11 @@ esac The workstation installer first refreshes the VM-side installer from the `release_installer` output in applied state. It resolves its module directory and requires initialized state-backend access. The VM-side installer repeats -the pinned digest and version checks, installs a versioned file under -`/usr/local/lib/agentcompute`, atomically changes -`/usr/local/bin/agentcompute`, and restarts the service only if it is already -running. A rejected asset does not replace the current binary. +the pinned digest and version checks, converges the public runtime bundle, +installs a versioned file under `/usr/local/lib/agentcompute`, atomically +changes `/usr/local/bin/agentcompute`, and restarts the service only if it is +already running. It does not overwrite private credentials. A rejected asset +does not replace the current binary or runtime bundle. GitHub artifact and OCI attestations and the OCI Cosign signature are release evidence, not deployed-service acceptance. Do not claim SLSA Build Level 3: From 43f5fd897de334081f58dfec7ae441eb7d253e73 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 20:59:21 -0700 Subject: [PATCH 5/7] docs(agentcompute): complete deployed Phase 9b qualification --- docs/docs/designs/agentcompute.md | 7 ++++++- docs/docs/runbooks/agentcompute.md | 16 ++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/docs/designs/agentcompute.md b/docs/docs/designs/agentcompute.md index 575b424..30107d0 100644 --- a/docs/docs/designs/agentcompute.md +++ b/docs/docs/designs/agentcompute.md @@ -1,6 +1,6 @@ --- title: agentcompute — disposable compute for agents -status: draft +status: implemented authors: - Josh Gilman - agent (session 019) @@ -734,6 +734,11 @@ The root documentation set is authoritative for architecture, decisions, and operations; source repositories own code, images, and deployment inputs. ADRs 0006–0009 remain **proposed** until the owner accepts them. Implementation and successful qualification do not silently accept an ADR. +The final deployment is v0.1.4 with the qualified live-desktop catalog. +Phase 9b exercised the deployed HTTPS service, including an independent +discovery-only router/Ubuntu-desktop workflow and a visually verified +whole-desktop capture. The [runbook qualification record](../runbooks/agentcompute.md#phase-9b-qualification-2026-09-16) +contains the measurements and release identities. ### Security and authorization deviations diff --git a/docs/docs/runbooks/agentcompute.md b/docs/docs/runbooks/agentcompute.md index 839488a..f6dbada 100644 --- a/docs/docs/runbooks/agentcompute.md +++ b/docs/docs/runbooks/agentcompute.md @@ -308,6 +308,11 @@ changes `/usr/local/bin/agentcompute`, and restarts the service only if it is already running. It does not overwrite private credentials. A rejected asset does not replace the current binary or runtime bundle. +A changed image catalog is reconciled before the MCP listener opens. During +that work, systemd can report `active` while Tailscale Serve returns `502`. +Wait for the service's `listening` log entry, then perform the HTTPS checks +below; `systemctl is-active` alone is not a readiness check. + GitHub artifact and OCI attestations and the OCI Cosign signature are release evidence, not deployed-service acceptance. Do not claim SLSA Build Level 3: the artifact build occurs outside the reusable attester, so signer isolation @@ -504,6 +509,15 @@ commit `d85426b72257998a339b91f854c8928ca9d3e361`. Its Linux amd64 SHA-256 is `d7e8e5229e7299612f858ba6d39718d402fc3b80017cbe3da0c7c0e2a6859c63`; the checksum and exact-tag/commit GitHub attestation passed before installation. +The final deployed release is +[`v0.1.4`](https://github.com/GilmanLab/agentcompute/releases/tag/v0.1.4), +commit `18c881ae6c854030bfed8115ac50d51c221b0151`, Linux amd64 SHA-256 +`fc9394427e573132a2461f7764c23bc0448bbceecd2bbfdd1f901f5b636dc8be`. +Its exact-tag/commit attestation, checksum manifest, immutable release, and +release-asset verification passed. The applied upgrade changed only the +service's public cloud-init metadata and release outputs; the installer +converged the public runtime bundle without replacing the VM or credentials. + | Contract | Observed result | | --- | --- | | One-minute TTL with a running VM | `p9b-ttl` expired at `02:04:44Z`; absent at `02:05:07.499Z`, 23.499 seconds later. `ac-p9b-ttl` was also absent from Incus project listing. | @@ -518,6 +532,8 @@ the checksum and exact-tag/commit GitHub attestation passed before installation. | Mac lifecycle | HTTPS create returned Running in 44.992 seconds; `sw_vers` reported macOS 26.6.2 / 25G83; Driver 0.28.1 was ready; the 1920×1200 screenshot showed the desktop. Instance and sandbox deletion left only the stopped seed in Lume. | | No-VNC boundary | Sampling began before the Mac create request and continued through Running. The first 1,000 listener observations contained only account-owned `127.0.0.1:7777`, no guest VNC listener. PF filter/NAT rules, the global Lume digest, and owner `rapportd` listeners were unchanged. | | Fresh-shell operation | The runbook's service, routing, Serve, DNS, public TLS, unauthenticated `401`, and SOPS-authenticated initialization commands passed under `bash --noprofile --norc` with a minimal environment. | +| Fresh discovery-only workflow | On v0.1.4, an independent agent starting with `search_api` created `p9b-final`: NAT WAN `10.77.9.0/24`, isolated LAN `10.77.10.0/24`, dual-NIC router, and LAN-only Ubuntu desktop at `10.77.10.3`. It configured guest routing/DNS, verified HTTPS to `example.com`, listed applications, and captured the desktop without patching the image. The returned 1280×800 full-desktop image was visually verified over HTTPS. | +| Final fleet drift | OpenTofu reported `No changes. Your infrastructure matches the configuration.` with detailed exit code `0` after the v0.1.4 installation. | A separate isolated HTTP fault fixture exercised the actual Incus SDK and restore adapter without faulting the deployed cluster. Stop failure retained From c3e4b591219947a772319bc2fe25b69175ed5a00 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 21:04:49 -0700 Subject: [PATCH 6/7] docs(agentcompute): record observed expiry listing race --- docs/docs/designs/agentcompute.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/docs/designs/agentcompute.md b/docs/docs/designs/agentcompute.md index 30107d0..48689e3 100644 --- a/docs/docs/designs/agentcompute.md +++ b/docs/docs/designs/agentcompute.md @@ -866,6 +866,9 @@ contains the measurements and release identities. suppression would be brittle and could hide real failures. The race is recorded, not papered over. Sequential lifecycle, expiry, and restart acceptance remain separate checks. + It was observed during Phase 9b cleanup at `2026-09-16T04:03:43Z`: + `capability failed: sandbox "p9b-win" not found` while the reaper removed that + expired sandbox. This remains an explicit deferral, not a successful-list claim. - A missing executable/exit-127 case can surface as a generic capability error from Incus and discard the enclosing program's result. Qualification hit this with deliberately missing commands and an unavailable BusyBox applet; From c54471ed9ce0c86b515247c24c57171d87cee679 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Tue, 15 Sep 2026 21:12:22 -0700 Subject: [PATCH 7/7] docs(agentcompute): align design body with implemented catalog, network, and host boundaries --- docs/docs/designs/agentcompute.md | 44 +++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/docs/docs/designs/agentcompute.md b/docs/docs/designs/agentcompute.md index 48689e3..21e2dec 100644 --- a/docs/docs/designs/agentcompute.md +++ b/docs/docs/designs/agentcompute.md @@ -235,7 +235,7 @@ Indicative catalog: | `router` | container | lab-built | Alpine or Debian with `nftables`, `frr`, `iproute2`/`tc`, `dnsmasq`, `wireguard`, `tcpdump`. The building block for NAT, routing, and impairment topologies. | | `windows/11/desktop` | vm | lab-built, cluster-local | Repacked with `distrobuilder repack-windows`, virtio drivers, autounattend, Cua Driver daemon at logon, VNC fallback. Captured on the cluster; never published to a registry. | | `windows/server-2025` | vm | lab-built, cluster-local | Headless. Same capture and non-publication rule. | -| `macos/sequoia/desktop` | vm | lab-built, Mac-local | `lume create --unattended` from a pinned IPSW; Cua Driver with Accessibility and Screen Recording granted once by an operator; Screen Sharing as the VNC fallback. Kept as a stopped seed on the Mac; never published to a registry. | +| `macos/tahoe/desktop` | vm | lab-built, Mac-local | `lume create --unattended` from a pinned IPSW (Tahoe; Sequoia's last restore image is a year of security updates behind); Cua Driver with Accessibility and Screen Recording granted once by an operator; clones carry the seed's `machineIdentifier` so they boot to a desktop. Kept as a stopped seed on the Mac under a dedicated account; never published to a registry. No VNC console on backend guests; `desktop.screenshot` is the view. | Agents can add to a sandbox's local catalog with `instance.publish` (below). Lab-built images are produced by a pinned recipe in the implementing @@ -269,7 +269,7 @@ records a digest for the former and an alias or seed name for the latter. | Capability | Arguments | Returns | Notes | | --- | --- | --- | --- | -| `net.create` | `sandbox`, `name`, `kind?` (`ovn` default, `bridge`), `cidr?`, `dhcp?`, `nat?`, `dns?` | `{name, kind, cidr, gateway}` | `ovn`: a cluster-wide segment with a logical router at `.1`; `nat` masquerades to the sandbox VLAN; `dhcp` and `dns` are on by default. nat=false networks are unreachable from outside the sandbox; attach a router instance or use net.peer. `bridge`: a bare L2 wire on one member, no router, nothing served; the agent brings its own. | +| `net.create` | `sandbox`, `name`, `kind?` (`ovn` default, `bridge`), `cidr?`, `dhcp?`, `nat?`, `dns?` | `{name, kind, cidr, gateway}` | `ovn`: a cluster-wide segment with a logical router at `.1`; `dhcp` and `dns` are on by default. `nat=true` masquerades to the sandbox VLAN and consumes one external address. `nat=false` is an isolated segment with no uplink: unreachable from outside the sandbox and free of external addresses; reach it through `net.peer` or a two-NIC guest acting as a router. Sandbox prefixes are never routed onto the lab VLAN. `bridge`: a bare L2 wire on one member, no router, nothing served; the agent brings its own. | | `net.list` / `net.get` / `net.delete` | `sandbox` [, `name`] | | Delete fails while NICs are attached. | | `net.attach` | `sandbox`, `instance`, `network`, `nic?`, `ip?`, `mac?` | `{nic, mac}` | Hot-plugs a NIC. `nic` names the device (`eth1`…); `ip` requests a static lease. | | `net.detach` | `sandbox`, `instance`, `nic` | `{}` | | @@ -386,17 +386,39 @@ behavior were qualified separately and are recorded under Validation. storage address as its Geneve tunnel address and its own TLS leaf. The Incus global OVN client uses the `nas01` leaf. 3. **An OVN uplink on the sandbox VLAN — complete.** Fleet owns the - cluster-wide `physical` uplink on the IncusOS-owned `fast40` parent and the - approved external range. The + cluster-wide `physical` uplink `fast40-uplink` on the IncusOS-owned + `fast40` parent and the approved external range. That network is the + exclusive owner of `fast40` on every member: no other network, profile + NIC, or instance NIC may attach to the parent (a raw macvlan there holds + its `rx_handler` and silently disqualifies the member as an OVN gateway + chassis); fleet checks this fail-closed before touching the uplink. The [address plan](../reference/networking/address-plan.md#ovn-external-addresses) is the only source for that allocation and its capacity calculation. -4. **An Incus identity for `agentcompute`** with rights to create - projects. Fleet `cluster/` concern. -5. **A macOS host.** A dedicated Apple Silicon machine in the lab (a - Mac mini is enough) running `lume serve`, reachable from the server, - with SSH. Not a personal workstation: the consented Driver seed is a - security-relevant artifact and the backend must not vanish when a lid - closes. Apple limits a host to two concurrent macOS guests. +4. **An Incus identity for `agentcompute`.** Creating projects is + root-equivalent in Incus 7.4 — a restricted certificate cannot be scoped + to projects that do not yet exist, and the authorization model treats + `can_create_projects` as host-root trust — so the service holds a + dedicated unrestricted identity by explicit exception ("Implementation + Outcome"). The durable shape is a fleet-managed pool of pre-restricted + `ac-NN` projects with a certificate restricted to them. +5. **A macOS host.** An always-on Apple Silicon machine running + `lume serve`, reachable from the server over SSH. The lab's Mac Studio + serves. The containment boundary is a dedicated standard, non-admin, + hidden account that owns Lume, the seed, and every clone; the server + holds an SSH key for that account (source-pinned to the service's + tailnet address, no agent or X11 forwarding, options enforced by a + root-owned sshd `Match` block), and the tailnet policy restricts who + may reach the host's SSH at all. A forwarding-only key was tried and + rejected: stock Lume's HTTP API has no exec, file, or identity route, + so the backend needs the account's shell for guest SSH/scp, the + metadata sidecar, and pinning the seed's `machineIdentifier` on each + clone. The API binds loopback only; backend guests start with VNC + disabled (source-pinned Lume until a release carries it). Migration to + a dedicated Mac mini is one Lume directory and one key. Apple's + two-concurrent-macOS-guest limit is host-wide and shared with any + macOS VMs the operator runs in other accounts, which nothing can + enumerate; the server refuses a third of its own before calling Lume + and detects the cross-account case after `run` with a clear error. 6. **Lab-built images**: the `router` container, one Linux desktop VM, one Windows desktop VM. Built from pinned recipes in the `agentcompute` repository until they earn a product home.