diff --git a/docs/docs/architecture/agentcompute.md b/docs/docs/architecture/agentcompute.md new file mode 100644 index 0000000..a58a810 --- /dev/null +++ b/docs/docs/architecture/agentcompute.md @@ -0,0 +1,151 @@ +--- +title: Agentcompute architecture +description: Runtime boundaries, trust, networking, and durable state of the lab compute service. +--- + +# Agentcompute architecture + +Agentcompute exposes short-lived compute environments to agents through one +Streamable HTTP MCP endpoint, `https://agentcompute01.tailda715.ts.net/`. +The agent-facing surface is `search_api`, `describe_api`, and `execute`; +individual compute and desktop operations are Starlark capabilities, not MCP +tools. See [ADR-0009](../decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md). + +This page describes the deployed boundaries. The +[implemented design](../designs/agentcompute.md) records the original intent +and deviations; the [runbook](../runbooks/agentcompute.md) owns deployment, +credential handling, and recovery. + +## Request and execution boundaries + +```mermaid +flowchart LR + Agent[Agent on tailnet] -->|HTTPS 443 and bearer| Serve[Tailscale Serve] + Serve -->|loopback 8080| MCP[agentcompute HTTP and CodeMode] + MCP --> Incus[Incus cluster API] + MCP -->|SSH 22 as agentcompute| Studio[Owner's Mac Studio] + Studio -->|loopback 7777| Lume[Lume daemon] + Studio -->|SSH ProxyJump| Mac[macOS guest] + Incus --> Linux[Linux guests] + Incus --> Windows[Windows guests] +``` + +Tailscale Serve terminates publicly trusted, automatically renewed HTTPS and +forwards to `127.0.0.1:8080` in the `agentcompute01` VM. The Go HTTP handler +requires a named static bearer token and rejects cross-origin requests. It +shares one backend registry and reaper across HTTP sessions. A client reconnect +or service restart does not create a second inventory. + +CodeMode evaluates bounded Starlark, not host Python or a host shell. Only +registered capabilities enter the backends. Guest `instance.exec` deliberately +executes commands inside the named sandbox guest; it does not expose Incus +credentials or a shell on `agentcompute01` to the caller. The `omp` bearer is +an administrator of this service surface, not a per-sandbox authorization +boundary. Subject metadata is attribution, not tenant isolation. + +## Incus and OVN + +The Incus backend owns projects prefixed `ac-`, with sandbox lifetime and +ownership in `user.agentcompute.*` metadata. Instances, networks, forwards, +snapshots, and sandbox-scoped published images belong to those projects. +Discovery reads backend state rather than a process-local inventory. + +OVN is the default network kind. NAT networks allocate external addresses from +`10.10.40.64–10.10.40.127` through `fast40-uplink`. An isolated `nat=false` +network does not attach to the uplink or consume an external NAT address. +Inter-network routing requires an explicit guest router or `net.peer`. +Network forwards allocate their own external listen addresses; they are not +implicitly the network's NAT address. See +[ADR-0006](../decisions/0006-ovn-as-the-sandbox-network-fabric.md) and the +[address plan](../reference/networking/address-plan.md). + +The service VM is in the Incus `default` project on `lab01`, outside disposable +sandbox projects. Its routed management NIC uses `10.10.10.16/32` through +`169.254.0.1`; its guest-side NIC is `10.158.86.2/24` on `ac-svc-vlan40`. +The route to VLAN 40 is through `10.158.86.1`. The tailnet address is +`100.65.152.20`. These paths do not bridge the IncusOS management bond or +require disabling `strict_hwaddr`. + +The dedicated Incus client is **unrestricted cluster root**. Incus 7.4's +project-creation authorization prevented the intended restricted `ac-*` +identity: the OpenFGA model warns at project creation, and the authorization +scriptlet cannot inspect the new project's name. A pre-created project pool +with a restricted claim/release identity is deferred. The current mitigation +is the narrow authenticated service boundary, not a claim of least-privilege +Incus credentials. + +## Mac backend + +The owner's Mac Studio runs Lume as a separate standard, hidden `agentcompute` +account. It is not a dedicated machine. Tailnet policy permits the service tag +to reach Studio only on SSH port 22. Studio's authorized-key rules restrict the +service key to the verified `100.65.152.20` source. See +[ADR-0008](../decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md). + +The backend reaches the host-local Lume HTTP daemon through SSH; port 7777 is +not published to the LAN or tailnet. Inventory and host work use the +account-local `/Users/agentcompute/bin/lume`, never the global install. +Guest execution uses SSH ProxyJump and SFTP with explicit host-key checking, +not a nonexistent `lume ssh` command. The qualified Tahoe seed supplies the +pinned guest host key and machine identifier. Retaining that machine identifier +preserves the qualified guest's activation and desktop permissions; the server +enforces a two-running-guest cap. The Mac backend does not pretend to provide +Incus OVN features. + +A pinned source build temporarily supplies upstream Lume's merged +`--vnc disabled` feature, absent from release 0.5.3. Every backend start, +including restore and restart, uses `noDisplay: true` and `vnc: disabled`. +Startup refuses a CLI or daemon that cannot enforce the policy. This avoids a +wildcard VNC listener at its source; no high-port PF rule is installed on the +owner's LAN interfaces. `pins/lume.yaml` in agentcompute records the exact +source commit, installed binary digest, and 0.5.3 as a manual rollback +reference that backend startup rejects. +Return to a release pin once a release includes the feature. The global Lume +install, Internet Sharing, and Continuity services remain outside service +ownership. + +## Desktop transport and screenshots + +The desktop API wraps Cua Driver over existing guest-execution channels rather +than exposing a second guest TCP service. Linux and macOS use the Driver's +one-shot CLI. Windows uses its persistent MCP transport through the guest +execution path. This is the trust and transport decision in +[ADR-0007](../decisions/0007-use-cua-driver-over-guest-execution.md). + +Screenshot bytes are written under `/var/lib/agentcompute/screenshots` and +returned as opaque HTTPS URLs rooted at the service hostname. Screenshot URLs +are bearer capabilities: do not log or publish them for private workloads. +Their lifetime is at most five minutes, shortened by sandbox expiry or deletion, +and they do not survive a service restart. The HTTP service checks that lifetime +rather than treating a file on disk as perpetual authorization. +The service's systemd credential directory and Studio keys are not available +to guest desktop tools. + +## Persistence and failure semantics + +- Incus projects and Lume sandbox metadata survive an MCP process restart. + The reaper reconciles expired sandboxes on its next pass, including those + that expired while the process was stopped; it is not an in-memory timer. +- Incus snapshot restore stops the original and stages a copy carrying current + agentcompute metadata, then deletes the original, renames the copy, and + starts it. UUID, NIC identity, and DHCP address can change. + Deleting the original also deletes its snapshot + tree. This is recreate semantics, not an in-place filesystem rollback. +- A failed create or delete can leave backend resources needing reconciliation. + The runbook distinguishes a slow asynchronous operation from a stuck + sandbox and requires checking ownership before direct backend cleanup. +- Fleet OpenTofu owns the long-lived VM/network/DNS and public runtime + configuration. Credentials are delivered out of band into root-only files + and loaded by systemd. Application release installation converges the + public configuration and binary without rerunning cloud-init or replacing + the VM and its Tailscale identity. + +## Ownership + +`GilmanLab/agentcompute` owns the server, capability contracts, backend code, +image qualification, and release artifacts. `GilmanLab/fleet` owns service +infrastructure and runtime deployment. `GilmanLab/networking` owns tailnet and +network policy. `GilmanLab/secrets` owns encrypted credentials. This root +repository owns architecture, decisions, deployment guidance, and acceptance +evidence. Changes crossing those boundaries require companion PRs; runtime +experiments are not a substitute for updating the owning repository. diff --git a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md index a4ac8d2..18070c1 100644 --- a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md +++ b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md @@ -22,6 +22,8 @@ claiming the provider parent. Which fabric should the durable service use? - Sandbox traffic cannot initiate connections to management or OOB. - Fleet owns infrastructure; agents own only disposable sandbox resources. - Management boot and OVN central must not depend on OVN itself. +- Native `nat=false`, `network=none` isolation must use the same durable + control plane as NAT-enabled sandbox networks. - Failed creates and expired resources must converge through explicit, retryable deletion rather than undocumented service repairs. @@ -43,18 +45,18 @@ to `nas01` and attached to its unmanaged VLAN 10 `mgmt` bridge rather than an OVN network. The pinned `ovn-central` package version `26.03.0-2` supplies northd and standalone NB/SB databases, without Raft. Per-component TLS gates prevent either database from starting without the complete certificate trio. -Remote database connections use mutual TLS from a dedicated offline OVN CA, -not the KMS root or the Incus cluster certificate. The CA uses EC P-256, has a -ten-year validity and no path-length constraint. Central and each chassis have -separate two-year leaves. Per-chassis keys allow one member's identity to -rotate without distributing the same private key to every node. The TLS -cutover occurred only after the cluster carried the CVE-2026-40243 fix. +Remote database connections use mutual TLS from a dedicated offline OVN +application CA, not the KMS root or the Incus cluster certificate. The CA uses +EC P-256, has a ten-year validity and no path-length constraint. Central and +each chassis have separate two-year leaves. Per-chassis keys allow one member's +identity to rotate without distributing the same private key to every node. +The TLS cutover occurred only after the cluster carried the CVE-2026-40243 fix. The owner approved this application-scoped trust domain on 2026-09-12, -superseding the draft's requirement to use the ADR-0005 hierarchy. -**OVN trust domain is application-scoped and offline; ADR-0005 governs the -KMS root hierarchy and is unaffected.** Revisit the issuance model when Vault -PKI exists and there is a reason to migrate. ADR-0005 is not amended. +superseding the draft's requirement to use the ADR-0005 hierarchy. The +dedicated offline OVN CA is outside ADR-0005's KMS-root hierarchy; ADR-0005 is +not amended. Revisit the issuance model when Vault PKI exists and there is a +reason to migrate. The CA key and every leaf key are escrowed under the `fleet` scope in `GilmanLab/secrets`, following ADR-0003's alternative KMS/PGP recipients and @@ -78,11 +80,11 @@ after rotation; the central private key is never an OpenTofu input. Fleet's `cluster/` deployment owns chassis configuration on every member, using VLAN 30 tunnel addresses and central's VLAN 10 endpoint. Supported settings are mirrored in node seeds. The default-project physical uplink -exclusively owns its provider parent. Before convergence, fleet checks -member-specific networks, profiles, and instance NICs, including inherited -and stopped-instance devices. A competing direct parent attachment or -physical-uplink NIC aborts deployment with a named conflict; fleet never -removes it silently. +`fast40-uplink` exclusively owns the IncusOS `fast40` provider parent. Before +convergence, fleet checks member-specific networks, profiles, and instance +NICs, including inherited and stopped-instance devices. A competing direct +parent attachment or physical-uplink NIC aborts deployment with a named +conflict; fleet never removes it silently. The [address plan](../reference/networking/address-plan.md#ovn-external-addresses) is authoritative for the approved external allocation and its eight-sandbox @@ -94,8 +96,8 @@ Reconsider a dedicated VLAN only if OVN needs more than the approved 64-address block. The representative topology has NAT-enabled `default` and `wan` networks, an isolated `lan`, and one distinct forward listen address. It consumes three -external addresses per sandbox: eight sandboxes consume 24 and leave 40 of -the 64-address reservation. +external addresses per sandbox. Eight sandboxes consume 24; the service's +`ac-svc-vlan40` network consumes one more, leaving 39 of the 64-address reservation. New sandbox projects use project-owned OVN networks and managed-only NICs. The default network has NAT. An additional NAT-enabled network consumes one @@ -105,6 +107,11 @@ consumes no external address, and has no direct path outside its sandbox. It becomes reachable only through `net.peer` or a router instance attached to another network. A `net.forward` request for an isolated network returns `AgentError`. +Both the overlay control plane and the native isolation contract require +central. Without central, Incus cannot create the project-owned logical switch +whose `network=none` setting makes a `nat=false` network isolated. Replacing +central would therefore require replacing the selected OVN fabric, not only +the central VM. The `default_network_kind=bridge` fallback keeps whole sandboxes on member-local bridges; explicit bridge networks remain bare wires there. Incus @@ -165,6 +172,25 @@ central stopped, `central-tls`, central running with the new CA, then fleet's OVN client/chassis converge. NB_Global, SB_Global, and two logical-switch UUIDs were preserved. The new `lab03` identity authenticated to both NB and SB; the old identity was rejected by both before an authenticated response. + +Phase 5 also exposed a stale in-memory trust failure after the OVN CA was +silently re-minted. Stored Incus configuration already named the new CA, but +the `lab01`, `lab02`, and `nas01` daemons continued reconnecting with the old +CA. Their failed handshakes filled the central VM's 20 GiB root filesystem. +After evidence capture and owner approval, recovery truncated only the three +identified log files and recycled those three Incus daemons serially. `lab03` +was left running because its daemon already held the new trust. All members +returned `Online`, central processes retained their PIDs and start times, and +the measured inbound reconnect rate fell from 1,039 per second to zero. + +[Fleet PR #20](https://github.com/GilmanLab/fleet/pull/20) prevents the same +silent transition: the ceremony refuses to mint an absent CA unless the +operator supplies `--mint-ca`, delivery paths require the reviewed CA +fingerprint, an OVN converge reports changed in-memory trust, and the explicit +trust-roll operation restarts selected daemons one at a time with an online +gate. [Root PR #34](https://github.com/GilmanLab/root/pull/34) records the +evidence-first, approval-gated recovery procedure in the canonical runbook. + A cross-member fixture completed three of three pings on each tested path. Public OpenTofu cloud-init metadata reconciliation was still pending AWS authentication and is not part of that proof. @@ -191,13 +217,14 @@ fixture residue across all four members and all projects in 3.03 seconds. The post-rotation and post-reboot fleet dry run proposed no changes in all 11 operations. -These checks complete the live technical confirmation recorded here. The record -remains `proposed`; its status must not be changed without the owner's -separate acceptance of the decision. +These checks establish the OVN infrastructure and lifecycle evidence recorded +here. They do not accept this decision or claim that Phase 9b validation is +complete. The record remains `proposed`; only the owner may change its status +to `accepted`. ## More Information -- [Agentcompute design draft](../designs/drafts/agentcompute.md) +- [Agentcompute design](../designs/agentcompute.md) - [Phase 3 qualification and parent recovery](https://github.com/GilmanLab/agentcompute/blob/spike/ovn-recreate-diagnosis/spikes/ovn/README.md): first post-fixture-deletion lab01-gateway cycle passed in 31.018 seconds, without reboot, central restart, or neighbor repair. - [Incus #3985](https://github.com/lxc/incus/issues/3985): unavailable NB creation leaves an `Errored` network; deletion after central recovery releases it. - [Incus #3986](https://github.com/lxc/incus/issues/3986): raw macvlan parent contention is separate from the central-outage failure. diff --git a/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md new file mode 100644 index 0000000..3c0ecd3 --- /dev/null +++ b/docs/docs/decisions/0007-use-cua-driver-over-guest-execution.md @@ -0,0 +1,134 @@ +--- +status: proposed +date: 2026-09-15 +decision-makers: + - Josh Gilman +--- + +# ADR-0007: Use Cua Driver over Guest Execution for Desktop Automation + +## Context and Problem Statement + +Agentcompute must expose the same desktop automation vocabulary for Linux, +Windows, and macOS guests without making a guest desktop service reachable from +every sandbox network. The chosen mechanism must support semantic accessibility +actions and screenshots, preserve isolated guest topologies, and keep binary +images out of CodeMode values. How should agentcompute reach and operate a +guest desktop? + +## Decision Drivers + +- Use one native desktop capability set across Linux, Windows, and macOS. +- Keep desktop control available when an Incus guest has no operator-reachable + network path. +- Avoid adding a Driver listener to sandbox network policy. +- Return structured Driver results while moving screenshots through a bounded + binary path. +- Retain a human-accessible fallback for setup, login, and recovery screens. + +## Considered Options + +- Cua Driver in the graphical session, reached through guest execution +- A network-exposed Cua MCP endpoint in every desktop guest +- Hypervisor console automation through SPICE or VNC +- Separate platform-specific automation implementations + +## Decision Outcome + +Proposed choice: **Cua Driver in the graphical session, reached through guest +execution**. This record remains `proposed` until the owner accepts it. + +Curated desktop images install the pinned Cua Driver and start its daemon in the +logged-in graphical session. Agentcompute exposes `desktop.info`, +`desktop.enable`, `desktop.call`, and `desktop.screenshot`. `desktop.call` forwards a +native Driver tool name and JSON object without defining a second typed API for +each upstream tool. + +Linux and macOS use bounded one-shot Driver CLI calls. Linux calls travel +through Incus exec to the user-session socket. macOS calls travel through the +Lume backend's direct guest SSH path as user `lume`. There is no persistent Mac +Driver bridge. Windows is the exception: agentcompute keeps a Cua MCP session +over one guest exec stream through the in-guest session proxy, because the +Incus agent service identity cannot access the interactive user's named pipe +directly. A lost Windows response closes that session for the next call but +never replays an action whose effect is unknown. + +The Driver writes screenshots to a temporary guest file. Agentcompute pulls the +file through the backend's binary file path—Incus file access or SFTP for +macOS—validates and optionally resizes the PNG, and publishes a short-lived URL. +Screenshot bytes do not cross the CodeMode value boundary. In-guest VNC or +Lume's VM console remains a fallback for a human; it is not the agent automation +API. + +### Consequences + +- Good, because semantic actions, accessibility trees, application discovery, + and screenshots use the Driver's native cross-platform vocabulary. +- Good, because Incus desktop control does not depend on a guest network route + or an exposed Driver port. An isolated `nat=false`, `network=none` guest + remains controllable through the Incus agent. +- Good, because screenshot data uses a bounded binary transfer instead of + base64 or another large value inside a Starlark result. +- Bad, because each desktop image must maintain a logged-in graphical session, + a compatible Driver daemon, and platform-specific startup behavior. +- Bad, because Linux and macOS pay one-shot CLI startup cost for each operation, + while Windows adds persistent-session lifecycle and reconnect handling. +- Bad, because macOS Accessibility and Screen Recording consent must be granted + to the signed Driver app on the private seed; the service cannot grant it + unattended. +- Bad, because pre-login screens and a broken graphical session still require a + separately protected VNC or console path. + +### Confirmation + +Compliance is observable when all of the following remain true: + +- `search_api` finds the stable `desktop.*` capabilities, `describe_api` + reports their typed shapes, and `desktop.info` discovers the installed Driver + version and native tool names. +- A native read such as `list_apps` returns structured Driver content, and a + screenshot capability returns a fetchable image URL rather than image bytes. +- Linux and macOS Driver operations execute one-shot guest commands. Windows + reuses a Driver MCP session carried by a guest exec stream and does not replay + a call after a lost response. +- No desktop image exposes a Driver control listener as part of its sandbox + network contract. +- The owner has explicitly changed this record's status to `accepted`; technical + evidence alone does not accept the decision. + +## Pros and Cons of the Options + +### Cua Driver over Guest Execution + +- Good, because it combines a cross-platform semantic API with the existing + authenticated backend control paths. +- Bad, because guest images and graphical-session startup become part of the + automation contract. + +### Network-Exposed Cua MCP Endpoint + +- Good, because every platform could use a long-lived MCP connection. +- Bad, because each guest would need a listener, authentication, address + discovery, and policy exceptions that conflict with arbitrary or isolated + network topologies. + +### Hypervisor Console Automation + +- Good, because a console can cover boot and login screens before the guest + desktop service starts. +- Bad, because console pixels do not provide the Driver's native application, + window, and accessibility semantics. Incus exposes graphical VM consoles + through SPICE, for which this service has no supported client path. + +### Platform-Specific Automation + +- Good, because each platform could use its native automation framework + directly. +- Bad, because agentcompute would own three incompatible vocabularies, image + contracts, permission models, and result formats. + +## More Information + +- [Agentcompute design](../designs/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute.md) +- [ADR-0009: Use CodeMode as the Agent-Facing MCP Surface](0009-use-codemode-as-the-agent-facing-mcp-surface.md) diff --git a/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md new file mode 100644 index 0000000..3efcbc0 --- /dev/null +++ b/docs/docs/decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md @@ -0,0 +1,165 @@ +--- +status: proposed +date: 2026-09-15 +decision-makers: + - Josh Gilman +--- + +# ADR-0008: Run Lume in a Confined Account on the Owner's Mac Studio + +## Context and Problem Statement + +Mac guests require Apple hardware and Apple's Virtualization framework, so they +cannot run on the Incus cluster. The original agentcompute design assumed a +dedicated Mac mini, but the owner already operates an always-on Mac Studio that +can host the required private seed. Where should Lume run, and which boundary +should separate the backend from the owner's workstation account? + +## Decision Drivers + +- Run macOS guests on owner-controlled Apple Silicon. +- Keep the consented seed host-local and never redistribute it. +- Prevent the backend identity from administering the Mac or traversing the + owner's home directory. +- Expose the loopback Lume lifecycle API only through an authenticated, + source-restricted connection from `agentcompute01`. +- Make clone identity, guest-count enforcement, and cleanup explicit in the + service instead of relying on silent Lume behavior. + +## Considered Options + +- Lume under a confined account on the owner's always-on Mac Studio +- A dedicated Mac mini running Lume +- Tart or a custom Virtualization.framework backend on the Mac Studio +- Hosted macOS workers + +## Decision Outcome + +Proposed choice: **Lume under a confined account on the owner's always-on Mac +Studio**. The host is `studio-1`, a Mac Studio with an M2 Max and 64 GB of +memory. This record remains `proposed` until the owner accepts it. + +Lume and its VM store run as the hidden, standard `agentcompute` account. The +account is not an administrator, has no sudo access, and cannot traverse the +owner's home directory. Its VM store is `/Users/agentcompute/.lume`. Studio +remains a user-owned tailnet device rather than receiving a backend device tag. + +`lume serve` binds only to `127.0.0.1:7777`. Agentcompute reaches that lifecycle +API through the source-restricted Studio SSH account and local TCP forwarding. +Guest exec does not use `lume ssh`: the backend invokes system SSH directly to +the guest's Lume NAT lease with Studio as a `ProxyJump`, a separate per-image +guest key, and pinned host keys for both hops. File transfer uses SFTP over the +same direct guest path. + +The stopped, consented seed stays in the confined account's Lume store and is +not published. Lume assigns a new `machineIdentifier` when it clones a VM, but +a fresh clone with that identifier can enter Setup Assistant. Before the +clone's first boot, agentcompute copies the seed's `machineIdentifier` into the +stopped clone while retaining the clone's generated MAC address. Snapshot and +restore clones follow the same pre-boot identity rule. + +Apple permits at most two additional running macOS guests on a host. +Agentcompute serializes starts and counts every running macOS VM visible in +the confined account, regardless of name. It refuses a third before the Lume +run mutation; cloning and configuring a stopped VM do not consume a running slot. +This check is required because Lume +0.5.3 accepts the third run request but leaves the VM stopped and reports the +limit only in its service log. A separate account can still consume a host-wide +slot, so a run that remains stopped is diagnosed against new Lume log output. + +Lume 0.5.3 starts a wildcard VNC listener even for `--display none`. The owner +rejected a blanket high-port PF rule because it could disrupt LAN Continuity +and `rapportd`. Upstream [cua#3209](https://github.com/trycua/cua/pull/3209) +merged an explicit disabled-VNC policy, but it was not in release 0.5.3 at +rollout. The approved temporary exception is an exact source commit pinned in +`pins/lume.yaml`, with `source_build: true` and the built binary's SHA-256. +It installs only for `agentcompute`; the system-wide Lume remains unchanged. + +Every backend run requests disabled VNC and no display. Startup fails closed +if either the account-local CLI or the running daemon lacks that policy. +The backend polls Lume until Running and stops any guest reporting a VNC URL. +No PF rule is installed, so this change does not alter Internet Sharing or the +owner's Continuity services. Once a release includes the feature, replace the +source pin with that release. The 0.5.3 release pin is retained as a rollback +reference, **not** an automatic fallback that may serve backend guests with VNC. + +### Consequences + +- Good, because the lab can use an already available, always-on Apple Silicon + host without buying and operating a second Mac now. +- Good, because the backend's normal shell, Lume store, and loopback API are + confined to a non-administrator account separate from the owner's files. +- Good, because lifecycle calls, guest calls, and file transfer use distinct + keys and verify both SSH hops. +- Bad, because a personal host is now a backend availability and capacity + dependency. Owner use, restart, sleep, or maintenance can interrupt Mac + sandboxes. +- Bad, because the two-running-guest host limit is small and shared with any + macOS VM running under another account. +- Bad, because the seed contains an operator's one-time Accessibility and Screen + Recording consent and must remain private, stopped, and recoverable on this + host. +- Bad, because a temporary source build adds build/signing provenance and an + account-local deployment step until upstream publishes disabled VNC in a + release. Both CLI and daemon must be upgraded together. +- Bad, because moving the backend later requires moving the confined account's + Lume store, reissuing the host key, and re-establishing host policy. + +### Confirmation + +Compliance is observable when all of the following remain true: + +- `agentcompute` is a hidden standard account with no administrator or sudo + access, the owner home denies traversal, and `lume serve` listens only on + `127.0.0.1:7777`. +- The durable service configuration enables Lume only after source-pinned SSH, + separate host and guest keys, known-host pins, and disabled-VNC enforcement + are verified. No account-owned per-VM VNC listener appears during startup. +- Live qualification separately samples account-owned TCP listeners from the + create request through Running; only the daemon's loopback listener may exist. +- Generated guest SSH configuration uses Studio as `ProxyJump`; guest exec and + SFTP do not invoke `lume ssh` and do not use an SSH agent. +- A fresh clone receives the seed's `machineIdentifier` while stopped and + before first boot, retains a distinct generated MAC address, and reaches the + desktop rather than Setup Assistant. +- A third running macOS guest is refused before a Lume run mutation. A + host-wide slot held by another account produces an explicit capacity error + rather than a false Running state. +- The owner has explicitly changed this record's status to `accepted`; a + qualification run or durable rollout does not accept the decision by itself. + +## Pros and Cons of the Options + +### Confined Account on the Owner's Mac Studio + +- Good, because it uses known hardware and keeps the backend separate from the + owner's login account. +- Bad, because the host is shared with owner activity and was not purchased as + dedicated service infrastructure. + +### Dedicated Mac Mini + +- Good, because service availability, packet filtering, and capacity would be + isolated from the owner's workstation. +- Bad, because it adds hardware and migration work before the two-guest backend + needs more capacity. It remains the migration option if sharing Studio stops + being acceptable. + +### Tart or a Custom Virtualization.framework Backend + +- Good, because another backend could offer different lifecycle or networking + behavior. +- Bad, because agentcompute would need to replace Lume's seed, clone, run, + snapshot, and VNC lifecycle instead of using the implemented backend. + +### Hosted macOS Workers + +- Good, because host maintenance and hardware ownership move to a provider. +- Bad, because the private consented seed, low-latency desktop control, and + sandbox lifecycle would cross a new trust and cost boundary. + +## More Information + +- [Agentcompute design](../designs/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute.md) +- [ADR-0007: Use Cua Driver over Guest Execution for Desktop Automation](0007-use-cua-driver-over-guest-execution.md) diff --git a/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md new file mode 100644 index 0000000..0ad198a --- /dev/null +++ b/docs/docs/decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md @@ -0,0 +1,136 @@ +--- +status: proposed +date: 2026-09-15 +decision-makers: + - Josh Gilman +--- + +# ADR-0009: Use CodeMode as the Agent-Facing MCP Surface + +## Context and Problem Statement + +Agentcompute has many lifecycle, network, file, and desktop operations. Exposing +each operation as a separate MCP tool would enlarge the public tool list and +force intermediate results through the model between dependent calls. How +should agents discover and compose the service's capabilities while keeping the +outer MCP contract small and stable? + +## Decision Drivers + +- Keep the public MCP tool list small as compute capabilities are added. +- Let an agent discover exact capability names and shapes at runtime. +- Compose dependent operations without returning every intermediate value to + the model. +- Validate typed inputs and expose an authorization point for every native call. +- Bound program execution time, native calls, and values. + +## Considered Options + +- CodeMode with three MCP tools and bounded Starlark programs +- One direct MCP tool for every agentcompute operation +- One custom dispatcher tool accepting arrays of JSON operations +- Expose the underlying Incus, Lume, and Driver APIs directly + +## Decision Outcome + +Proposed choice: **CodeMode with three MCP tools and bounded Starlark +programs**. Agentcompute's outer MCP surface is exactly `search_api`, +`describe_api`, and `execute`. This record remains `proposed` until the owner +accepts it. + +`search_api` discovers registered capabilities by name and search metadata. +`describe_api` returns the generated input and output shape for one capability. +`execute` evaluates a bounded Starlark program and returns the converted value +of its zero-argument `main()` function. Capabilities use stable dotted names +under `sandbox`, `image`, `instance`, `net`, and `desktop`. + +Each capability is registered as a typed Go handler. Inputs are flat structs +whose direct exported fields are scalar `string`, `int64`, `bool`, or `float64` +values, with pointers for optional scalars. Input lists and nested objects are +not part of this contract. An agent repeats a capability call inside its +program for multi-value operations. Where the underlying native API itself +requires a free-form object, such as a Cua Driver call, agentcompute accepts an +explicit JSON object string and the handler validates it. Outputs may contain +bounded nested structs and lists. + +CodeMode creates a fresh Starlark worker for each `execute` request and permits +native calls only from `main()`. The worker is a language and execution +boundary, not an operating-system isolation boundary: capability handlers run +in the privileged parent service, and the worker runs as the same operating +system user as the service. Authentication and authorization therefore remain +host responsibilities outside program-controlled arguments. + +### Consequences + +- Good, because adding a capability does not add another outer MCP tool or + require clients to load a growing tool schema list. +- Good, because one program can create resources, wait, inspect results, and + clean up while returning only its final bounded value. +- Good, because generated descriptions and argument binding come from the same + Go types used by handlers. +- Good, because CodeMode passes canonical typed arguments to the authorizer for + every native call. The deployed policy is `authz.AllowAll`: bearer + authentication is the current gate, and subject identity is attribution. +- Bad, because clients must discover CodeMode and write Starlark instead of + calling an operation-specific MCP tool directly. +- Bad, because flat scalar inputs require repeated calls or an explicitly + validated string encoding when an operation naturally takes a collection or + nested object. +- Bad, because changing a capability's typed shape can invalidate saved + programs even though the outer three-tool list is unchanged. +- Bad, because CodeMode does not isolate handler side effects, filesystem, + credentials, CPU, or memory at the operating-system level. + +### Confirmation + +Compliance is observable when all of the following remain true: + +- MCP tool discovery returns exactly `search_api`, `describe_api`, and + `execute`. +- `search_api` finds an agentcompute capability, `describe_api` reports its + flat typed input, and `execute` can call it from `main()` and return its + result. +- A wrongly typed argument is rejected before handler dispatch. +- New operations are registered through `codemode.Register`; they do not call + `mcp.AddTool` or add another public dispatcher. +- Runtime limits remain configured for execution and native calls, and the + deployment continues to establish trusted subject identity outside program + source and capability arguments. +- The owner has explicitly changed this record's status to `accepted`; passing + discovery or execution checks does not accept the decision. + +## Pros and Cons of the Options + +### CodeMode + +- Good, because it supplies discovery, generated descriptions, typed binding, + per-call authorization, bounded composition, and a fixed MCP surface. +- Bad, because it introduces a worker protocol and Starlark execution model that + operators and contributors must understand. + +### Direct MCP Tool per Operation + +- Good, because ordinary MCP clients can call each operation without writing a + program. +- Bad, because the tool list and schema payload grow with every capability, and + dependent calls repeatedly move intermediate data through the model. + +### Custom JSON Operation Dispatcher + +- Good, because it could keep one public tool and batch several operations. +- Bad, because agentcompute would have to invent discovery, type checking, + references between results, control flow, limits, authorization points, and + error projection already supplied by CodeMode. + +### Direct Infrastructure APIs + +- Good, because it avoids an application-level capability layer. +- Bad, because it exposes provider-specific credentials and mutable + infrastructure primitives instead of the sandbox ownership, TTL, cleanup, + and policy boundaries agentcompute enforces. + +## More Information + +- [Agentcompute design](../designs/agentcompute.md) +- [Deploy and operate agentcompute](../runbooks/agentcompute.md) +- [CodeMode documentation](https://meigma.github.io/codemode/) diff --git a/docs/docs/designs/drafts/agentcompute.md b/docs/docs/designs/agentcompute.md similarity index 71% rename from docs/docs/designs/drafts/agentcompute.md rename to docs/docs/designs/agentcompute.md index e31c1db..21e2dec 100644 --- a/docs/docs/designs/drafts/agentcompute.md +++ b/docs/docs/designs/agentcompute.md @@ -1,17 +1,25 @@ --- title: agentcompute — disposable compute for agents -status: draft +status: implemented authors: - Josh Gilman - agent (session 019) created: 2026-09-11 -updated: 2026-09-13 +updated: 2026-09-16 related-decisions: - ADR-0006 + - ADR-0007 + - ADR-0008 + - ADR-0009 --- # agentcompute — disposable compute for agents +The proposal below is retained so implementation deviations remain visible. +Read [Implementation Outcome](#implementation-outcome) for those deviations, +the [architecture](../architecture/agentcompute.md) for runtime boundaries, +and the [runbook](../runbooks/agentcompute.md) for operations. + ## Summary `agentcompute` is a lab-specific MCP server, built on @@ -52,7 +60,7 @@ The lab already has the substrate: management or OOB). It is carried on the cluster's fast links, and the default-project physical network `fast40-uplink` exclusively owns the IncusOS-owned `fast40` parent. Sandbox NICs use managed logical networks. See the - [address plan](../../reference/networking/address-plan.md). + [address plan](../reference/networking/address-plan.md). - macOS guests cannot run on Incus. They need Apple hardware running Apple's Virtualization framework, driven by [Lume](https://cua.ai/docs/reference/lume/cli-reference) (or Tart; see @@ -134,7 +142,7 @@ but no OVN central. The lab's durable `ovncentral01` VM is pinned to `nas01`'s unmanaged management bridge and runs `ovn-northd` with standalone NB/SB databases. Both databases require mutual TLS from the application-scoped offline OVN CA. See the -[OVN central and certificate runbook](../../runbooks/ovn-central-and-certificates.md). +[OVN central and certificate runbook](../runbooks/ovn-central-and-certificates.md). Each sandbox network is an Incus OVN network: a Geneve overlay that spans the cluster, with a logical router, DHCP, DNS, network ACLs, and peering built in. A NAT-enabled network also has SNAT and can host network forwards. An isolated @@ -227,7 +235,7 @@ Indicative catalog: | `router` | container | lab-built | Alpine or Debian with `nftables`, `frr`, `iproute2`/`tc`, `dnsmasq`, `wireguard`, `tcpdump`. The building block for NAT, routing, and impairment topologies. | | `windows/11/desktop` | vm | lab-built, cluster-local | Repacked with `distrobuilder repack-windows`, virtio drivers, autounattend, Cua Driver daemon at logon, VNC fallback. Captured on the cluster; never published to a registry. | | `windows/server-2025` | vm | lab-built, cluster-local | Headless. Same capture and non-publication rule. | -| `macos/sequoia/desktop` | vm | lab-built, Mac-local | `lume create --unattended` from a pinned IPSW; Cua Driver with Accessibility and Screen Recording granted once by an operator; Screen Sharing as the VNC fallback. Kept as a stopped seed on the Mac; never published to a registry. | +| `macos/tahoe/desktop` | vm | lab-built, Mac-local | `lume create --unattended` from a pinned IPSW (Tahoe; Sequoia's last restore image is a year of security updates behind); Cua Driver with Accessibility and Screen Recording granted once by an operator; clones carry the seed's `machineIdentifier` so they boot to a desktop. Kept as a stopped seed on the Mac under a dedicated account; never published to a registry. No VNC console on backend guests; `desktop.screenshot` is the view. | Agents can add to a sandbox's local catalog with `instance.publish` (below). Lab-built images are produced by a pinned recipe in the implementing @@ -261,7 +269,7 @@ records a digest for the former and an alias or seed name for the latter. | Capability | Arguments | Returns | Notes | | --- | --- | --- | --- | -| `net.create` | `sandbox`, `name`, `kind?` (`ovn` default, `bridge`), `cidr?`, `dhcp?`, `nat?`, `dns?` | `{name, kind, cidr, gateway}` | `ovn`: a cluster-wide segment with a logical router at `.1`; `nat` masquerades to the sandbox VLAN; `dhcp` and `dns` are on by default. nat=false networks are unreachable from outside the sandbox; attach a router instance or use net.peer. `bridge`: a bare L2 wire on one member, no router, nothing served; the agent brings its own. | +| `net.create` | `sandbox`, `name`, `kind?` (`ovn` default, `bridge`), `cidr?`, `dhcp?`, `nat?`, `dns?` | `{name, kind, cidr, gateway}` | `ovn`: a cluster-wide segment with a logical router at `.1`; `dhcp` and `dns` are on by default. `nat=true` masquerades to the sandbox VLAN and consumes one external address. `nat=false` is an isolated segment with no uplink: unreachable from outside the sandbox and free of external addresses; reach it through `net.peer` or a two-NIC guest acting as a router. Sandbox prefixes are never routed onto the lab VLAN. `bridge`: a bare L2 wire on one member, no router, nothing served; the agent brings its own. | | `net.list` / `net.get` / `net.delete` | `sandbox` [, `name`] | | Delete fails while NICs are attached. | | `net.attach` | `sandbox`, `instance`, `network`, `nic?`, `ip?`, `mac?` | `{nic, mac}` | Hot-plugs a NIC. `nic` names the device (`eth1`…); `ip` requests a static lease. | | `net.detach` | `sandbox`, `instance`, `nic` | `{}` | | @@ -372,23 +380,45 @@ behavior were qualified separately and are recorded under Validation. ADR-0003. The CA is not part of ADR-0005's KMS hierarchy; revisit Vault issuance when Vault PKI exists and there is a reason to migrate. The transitional central on `sandbox01` has been purged. See the - [operations runbook](../../runbooks/ovn-central-and-certificates.md). + [operations runbook](../runbooks/ovn-central-and-certificates.md). 2. **OVN chassis on every node — complete.** Fleet converges `/os/1.0/services/ovn` on all four members, with each member's VLAN 30 storage address as its Geneve tunnel address and its own TLS leaf. The Incus global OVN client uses the `nas01` leaf. 3. **An OVN uplink on the sandbox VLAN — complete.** Fleet owns the - cluster-wide `physical` uplink on the IncusOS-owned `fast40` parent and the - approved external range. The - [address plan](../../reference/networking/address-plan.md#ovn-external-addresses) + cluster-wide `physical` uplink `fast40-uplink` on the IncusOS-owned + `fast40` parent and the approved external range. That network is the + exclusive owner of `fast40` on every member: no other network, profile + NIC, or instance NIC may attach to the parent (a raw macvlan there holds + its `rx_handler` and silently disqualifies the member as an OVN gateway + chassis); fleet checks this fail-closed before touching the uplink. The + [address plan](../reference/networking/address-plan.md#ovn-external-addresses) is the only source for that allocation and its capacity calculation. -4. **An Incus identity for `agentcompute`** with rights to create - projects. Fleet `cluster/` concern. -5. **A macOS host.** A dedicated Apple Silicon machine in the lab (a - Mac mini is enough) running `lume serve`, reachable from the server, - with SSH. Not a personal workstation: the consented Driver seed is a - security-relevant artifact and the backend must not vanish when a lid - closes. Apple limits a host to two concurrent macOS guests. +4. **An Incus identity for `agentcompute`.** Creating projects is + root-equivalent in Incus 7.4 — a restricted certificate cannot be scoped + to projects that do not yet exist, and the authorization model treats + `can_create_projects` as host-root trust — so the service holds a + dedicated unrestricted identity by explicit exception ("Implementation + Outcome"). The durable shape is a fleet-managed pool of pre-restricted + `ac-NN` projects with a certificate restricted to them. +5. **A macOS host.** An always-on Apple Silicon machine running + `lume serve`, reachable from the server over SSH. The lab's Mac Studio + serves. The containment boundary is a dedicated standard, non-admin, + hidden account that owns Lume, the seed, and every clone; the server + holds an SSH key for that account (source-pinned to the service's + tailnet address, no agent or X11 forwarding, options enforced by a + root-owned sshd `Match` block), and the tailnet policy restricts who + may reach the host's SSH at all. A forwarding-only key was tried and + rejected: stock Lume's HTTP API has no exec, file, or identity route, + so the backend needs the account's shell for guest SSH/scp, the + metadata sidecar, and pinning the seed's `machineIdentifier` on each + clone. The API binds loopback only; backend guests start with VNC + disabled (source-pinned Lume until a release carries it). Migration to + a dedicated Mac mini is one Lume directory and one key. Apple's + two-concurrent-macOS-guest limit is host-wide and shared with any + macOS VMs the operator runs in other accounts, which nothing can + enumerate; the server refuses a third of its own before calling Lume + and detects the cross-account case after `run` with a clear error. 6. **Lab-built images**: the `router` container, one Linux desktop VM, one Windows desktop VM. Built from pinned recipes in the `agentcompute` repository until they earn a product home. @@ -464,10 +494,13 @@ new volatile NIC state, so its MAC address, DHCP identity, lease, and assigned address can change. Callers must rediscover the instance and its addresses after restore. -Failures before the staged copy completes leave the original stopped with its -snapshots intact; inspect for a generated `restore-*` copy before retrying. If -original deletion fails, the error identifies the retained staged copy. After -original deletion, its snapshots are already consumed: a rename failure leaves +A stop failure preserves the original and its snapshots; inspect its power +state. A copy failure after a successful stop leaves the original stopped with +its snapshots intact; inspect for a generated `restore-*` copy before retrying. +Deletion removes the original's snapshots before deleting the original itself. +If that final deletion fails, the original and staged copy can both remain, +but the snapshots are already consumed; the error identifies the staged copy. +After original deletion, a rename failure leaves the replacement under its generated name. A start or `Running`-wait failure returns an error with the replacement under the original agent-facing name; inspect its status before retrying. @@ -516,7 +549,7 @@ networks and distinct forward listen addresses consume them. The representative `default` NAT, isolated `lan`, `wan` NAT, and one-forward topology consumes three addresses per sandbox: eight sandboxes consume 24 and leave 40 for additional allocations and pending cleanup. The -[address plan](../../reference/networking/address-plan.md#ovn-external-addresses) +[address plan](../reference/networking/address-plan.md#ovn-external-addresses) is authoritative. ## Delivery @@ -665,11 +698,12 @@ fleet dry run was a no-op. - No isolation, no desktops, no topologies, no cleanup. - Not chosen. -## Open Questions +## Design Questions -### Resolved +### Resolved in review -Resolved in review, kept here until the draft is promoted: +These decisions describe the reviewed proposal; later amendments are recorded +in Implementation Outcome rather than silently rewriting the proposal. - **The word `sandbox`** stays. Agents already think in it; the collision is with a host and a repository, neither of which an agent sees. @@ -683,7 +717,7 @@ Resolved in review, kept here until the draft is promoted: - **Blocking operations** stay. Raise `MaxConcurrentExecutions` before considering an asynchronous pattern. - **OVN central** is the OpenTofu-owned `ovncentral01` VM on `nas01`; no Raft. - The [operations runbook](../../runbooks/ovn-central-and-certificates.md) + The [operations runbook](../runbooks/ovn-central-and-certificates.md) defines its deployment, renewal, and recovery. Whether it later serves Talos networks is a T10/T11 question, not this design's. - **Windows and macOS images are not published** to any registry; they @@ -691,7 +725,7 @@ Resolved in review, kept here until the draft is promoted: - **`desktop.call` ergonomics**: pass-through first; typed conveniences only for tools agents demonstrably fumble. - **VLAN 40** is the durable OVN uplink VLAN (owner decision, 2026-09-12). - The [address plan](../../reference/networking/address-plan.md#ovn-external-addresses) + The [address plan](../reference/networking/address-plan.md#ovn-external-addresses) records the approved 64-address allocation and eight-sandbox planning target. Default-plus-forward consumes two external addresses per sandbox; adding an isolated `lan` and NAT-enabled `wan` brings that budget to three. @@ -700,13 +734,174 @@ Resolved in review, kept here until the draft is promoted: needs more than this allocation; DHCP, named endpoints, and routes stay unchanged. -### Still open - -1. **Cua Driver token continuity across one-shot CLI calls.** Decided by - the desktop spike: snapshot in one call, act by token in the next. If - tokens do not survive, `desktop.call` drives a long-lived - `cua-driver serve` per instance over exec; the vocabulary is unchanged. -2. **Cua Driver inside images.** How the daemon is kept alive in the - graphical session on each OS is image-recipe work learned in the - desktop slices. Pin the newest non-prerelease Driver per image; bump - deliberately. +### Resolved during implementation + +1. **Driver token continuity:** Linux and macOS retain the one-shot CLI + transport over guest exec. Windows needs a persistent Driver MCP session + over guest exec. Both retain the `desktop.call` vocabulary; see + [ADR-0007](../decisions/0007-use-cua-driver-over-guest-execution.md). +2. **Driver process lifetime:** image recipes own the graphical-session + service or LaunchAgent, its pinned binary, and readiness qualification. + Guest OSes differ; this is not one universal daemon recipe. +3. **Mac location:** the owner approved the existing Mac Studio under a + separate standard account, not the proposed dedicated Mac mini. Its + availability and two-guest capacity are shared with the owner's workloads. + +## Implementation Outcome + +The service is delivered as a Go CodeMode MCP server in the long-lived +`agentcompute01` Incus VM, with Incus/OVN for Linux and Windows and Lume for +macOS. Runtime and image state remain in their backends, not in MCP sessions. +The root documentation set is authoritative for architecture, decisions, and +operations; source repositories own code, images, and deployment inputs. +ADRs 0006–0009 remain **proposed** until the owner accepts them. Implementation +and successful qualification do not silently accept an ADR. +The final deployment is v0.1.4 with the qualified live-desktop catalog. +Phase 9b exercised the deployed HTTPS service, including an independent +discovery-only router/Ubuntu-desktop workflow and a visually verified +whole-desktop capture. The [runbook qualification record](../runbooks/agentcompute.md#phase-9b-qualification-2026-09-16) +contains the measurements and release identities. + +### Security and authorization deviations + +- **Incus identity is unrestricted.** The intended `ac-*` project-creation + certificate could not work with Incus 7.4's authorization model. OpenFGA + warns that project creation is root-equivalent, and the scriptlet cannot + inspect the new project's name at creation time. The dedicated + `agentcompute01` identity therefore has cluster-root rights behind the + tailnet, bearer-authenticated MCP boundary. Guest code never receives that + identity. A pre-created restricted project pool with claim/release metadata + is deferred, not partially implemented. See the + [runbook safety boundary](../runbooks/agentcompute.md#safety-boundary). +- **Private-image workflow triggers use a private repository.** The public + implementation repository cannot be the trust gate for self-hosted workers + carrying private Windows/macOS material. The organization's free plan does + not provide the restricted runner-group controls needed for that public + repository arrangement. Private triggers and protected runner access were + used instead. See [private image runners](../runbooks/private-image-runners.md). +- **HTTPS uses Tailscale Serve's public ACME certificate**, not the proposed + internal PKI leaf. Clients use normal public trust and the fixed tailnet + hostname. Serve terminates HTTPS and forwards only to loopback; named bearer + authentication remains mandatory. + +### OVN ownership, PKI, and recovery + +- The default-project physical network `fast40-uplink` exclusively owns the + IncusOS `fast40` parent. Sandbox NICs attach to managed logical networks, + not raw macvlan on that same parent. Raw parent contention was a separate + failure from the unavailable-central network-creation failure. +- The offline OVN CA is an owner-approved application-scoped trust domain + **outside ADR-0005's KMS-root hierarchy**. ADR-0005 itself is unchanged. + Revisit issuance when Vault PKI exists and migration has an operational + reason; do not imply that hierarchy is already deployed for OVN. +- A silently re-minted CA left several Incus daemons reconnecting with stale + in-memory trust even though stored configuration named the new CA. Repeated + failed TLS handshakes filled the central VM's 20 GiB root filesystem. + Owner-approved recovery truncated only the identified logs and recycled + `lab01`, `lab02`, and `nas01` serially; `lab03` already held the new trust. + Central processes stayed up. [Fleet #20](https://github.com/GilmanLab/fleet/pull/20) + now makes CA creation explicit, checks reviewed fingerprints, and provides + an online-gated serial trust roll. [Root #34](https://github.com/GilmanLab/root/pull/34) + records evidence-first recovery. The complete incident and rationale are in + [ADR-0006](../decisions/0006-ovn-as-the-sandbox-network-fabric.md). +- Address accounting remains explicit: default NAT plus forward consumes two + external addresses; adding a NAT-enabled WAN consumes a third. Isolated + `nat=false` LANs consume none. Count failed NAT network allocations until + deletion. The approved range remains `10.10.40.64–10.10.40.127`. + The service's `ac-svc-vlan40` network consumes one address from that same + range. Eight representative sandboxes therefore leave **39**, not the + proposal's 40, after infrastructure use. + +### Mac implementation amendments + +- The host is the owner's always-on Mac Studio, confined to the hidden + standard `agentcompute` account. Neither administrator rights, the owner's + home, nor a broad host-network permission is part of that account. +- Cloning with a fresh Lume `machineIdentifier` could return the qualified + guest to Setup Assistant. Before first boot, the backend retains the seed's + identifier while keeping the clone's new MAC. Snapshot clones use the same + rule. Consent and activation are seed properties, not permission grants + performed by the server. +- The server serializes starts and enforces the two-running-macOS-guest + limit before each Lume run mutation. It counts every running macOS VM in the + confined account and diagnoses host-wide capacity occupied elsewhere. +- The runtime does not use the unavailable `lume ssh`. Guest exec and SFTP use system SSH through + Studio as `ProxyJump`, with distinct host/guest keys and both host keys + pinned. The guest pin uses the seed name, not a DHCP address. +- Lume 0.5.3's wildcard VNC server cannot be disabled with `--display none`. + The owner rejected a blanket high-port PF block because of Continuity and + `rapportd`. A dynamic port watcher would still permit an initial exposure + window. Instead, [agentcompute #39](https://github.com/GilmanLab/agentcompute/pull/39) + pins the merged upstream no-VNC commit, builds as `agentcompute`, and + installs only in that account. Every backend run disables VNC, and startup + checks both CLI and daemon support. The global install, PF, and Internet + Sharing remain unchanged. This temporary source artifact is SHA-256 pinned + but **not bitwise reproducible after a clean rebuild**. Return to a release + pin once upstream publishes the feature. A human may explicitly enable VNC + for a bounded maintenance console; no automatic fallback does so. + +### Runtime and image contract details + +- Incus snapshot restore is **recreate and start**: stop the original, stage a + snapshot copy, delete the original and its snapshot tree, rename the copy, + preserve current agentcompute metadata, and start it. It is not in-place + rollback. Phase 9b observed the same name with changed UUID, MAC, and DHCP + address, restored file contents, and an empty snapshot list. +- The illustrative NAT program omitted guest configuration needed after a + hot NIC attach. Live qualification explicitly brought up the router WAN, + acquired its DHCP lease, selected the WAN default route, and configured + the isolated client's default route/DNS through that router. A WAN packet + capture proved the client's connection source was the router WAN address; + merely checking NAT rules was not accepted as evidence. +- Linux whole-desktop capture was black because Cua Driver's cosmetic + agent-cursor overlay froze X root-window reads before GNOME's first frame. + Starting it later could freeze a coloured frame instead. Window capture + remained live; VNC was affected too. Driver 0.28.2 reproduced the defect. + [Agentcompute #41](https://github.com/GilmanLab/agentcompute/pull/41) starts + the image's Driver with `--no-overlay`, retaining the reviewed 0.28.1 pin. + This removes synthetic session cursors, not input or the native cursor. + Image qualification now requires whole-desktop pixels to change after + launching Text Editor; a black-frame heuristic would miss a coloured freeze. + The fresh private bake from commit `cb7f8c6d80bf1e66ea75ce28f07510db9636c34e` + passed that live-pixel check. Its Ubuntu desktop OCI digest is + `sha256:b2a83d7d70de2239de0bb04e255c96079446afb97ebcd36434892acefa5f8b0e`; + [catalog promotion #44](https://github.com/GilmanLab/agentcompute/pull/44) + records the qualified image references. The generated catalog commit was + signed by the integrating operator to satisfy branch policy; no bypass was used. +- Fleet release installation now converges the public configuration, catalog, + unit, and SSH pins in place, after artifact checks. It does not replace the + service VM, rerun cloud-init, overwrite private credentials, or re-enroll + Tailscale. Bootstrap/network/certificate changes retain the deliberate + replacement procedure. +- Windows and macOS images remain local and do not have published image + attestations. Private Linux image bakes also omit the older hosted + `attest.yml` step: digest verification and boot qualification are not signed + build provenance. Binary/container release attestations exist, but their + build still occurs outside the reusable attesting job: **SLSA Level 3 is not + claimed**. These supply-chain gaps remain deferred. + +### Known residuals + +- `sandbox.list` can race expiry/deletion between listing sandbox names and + reading their instance counts. A missing-sandbox error is already converted + to an agent-facing error before the MCP listing loop; a local string-match + suppression would be brittle and could hide real failures. The race is + recorded, not papered over. Sequential lifecycle, expiry, and restart + acceptance remain separate checks. + It was observed during Phase 9b cleanup at `2026-09-16T04:03:43Z`: + `capability failed: sandbox "p9b-win" not found` while the reaper removed that + expired sandbox. This remains an explicit deferral, not a successful-list claim. +- A missing executable/exit-127 case can surface as a generic capability error + from Incus and discard the enclosing program's result. Qualification hit + this with deliberately missing commands and an unavailable BusyBox applet; + valid command, routing, and desktop checks used real installed programs. + Error reporting was not broadened as an unrelated Phase 9b change. +- Mac cold-clone readiness now checks the GUI session, kickstarts the existing + Driver LaunchAgent, and verifies the daemon's existing TCC grants before + returning. Phase 9b required no manual intervention or new permission grant. + Missing consent still requires a human; the server does not grant it. +- The existing Mac seed was qualified manually. Legacy + `images/macos/provision.sh` and `verify.sh --clone` still depend on the + unavailable `lume ssh` and are not qualified rebuild automation. The runbook + uses deployed MCP create/readiness/exec/screenshot/delete to requalify clones. + Porting those legacy bootstrap scripts is deferred. diff --git a/docs/docs/reference/hardware-inventory.md b/docs/docs/reference/hardware-inventory.md index b420cef..1ea1938 100644 --- a/docs/docs/reference/hardware-inventory.md +++ b/docs/docs/reference/hardware-inventory.md @@ -61,6 +61,21 @@ documentation, not this inventory. - `1x10GbE` and `1x5GbE` onboard networking - `2xUSB4`, HDMI, and OCuLink available +### Apple Mac Studio + +- Quantity: `1` +- Device name: `studio-1` +- Role: owner's always-on Apple Silicon host for the Lume macOS backend +- Hardware details: + - Apple M2 Max + - `64GB` memory +- Backend boundary: Lume and its VM store run under the dedicated hidden, + standard `agentcompute` account, not the owner's account. +- Console boundary: the account-local pinned Lume build supports disabled VNC; + backend starts require it. No broad high-port PF block is installed, so + Internet Sharing and the owner's Continuity services retain their policies. + See the [Mac operations runbook](../runbooks/agentcompute.md#operate-the-mac-backend). + ### MikroTik CRS309-1G-8S+IN - Quantity: `2` diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index f49c1b3..846b67f 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -126,9 +126,9 @@ isolated networks. The planning target is eight concurrent sandboxes for the single operator and their agents. The representative three-network topology uses 24 of the -64 addresses and leaves 40 for additional NAT-enabled networks, distinct -forward addresses, and pending cleanup. These are address-budget -calculations, not measured scale limits. +64 addresses; the service's `ac-svc-vlan40` network holds one more. This leaves +39 for additional NAT-enabled networks, distinct forward addresses, and +pending cleanup. These are address-budget calculations, not measured scale limits. An `Errored` NAT-enabled OVN network retains its external address until deleted and counts against capacity. If central is unavailable, the reaper @@ -145,15 +145,18 @@ qualification spike central on `sandbox01` has been removed. See the [OVN central and certificate runbook](../../runbooks/ovn-central-and-certificates.md) for deployment, renewal, and recovery procedures. -### Incus-local image runner networks +### Incus-local and service networks -These NAT bridges are local to each Incus member, not routed VLAN prefixes. -Do not advertise them through `gw01` or Tailscale. +These private prefixes are not routed lab VLANs. Do not advertise them through +`gw01` or Tailscale. `incusbr0` and `github-runners` are member-local NAT +bridges. `ac-svc-vlan40` is a managed OVN network whose uplink remains +`fast40-uplink`. | Resource | Address or prefix | Ownership | | --- | --- | --- | | Existing `incusbr0` bridge | `10.158.84.0/24`, gateway `.1` | IncusOS | | `github-runners` bridge | `10.158.85.0/24`, gateway `.1` | Fleet `cluster/` | +| Agentcompute service network `ac-svc-vlan40` | `10.158.86.0/24`, gateway `.1`; `agentcompute01` guest NIC `.2` | Fleet OpenTofu root `incus/agentcompute/` | | Reserved controller VM `ghrunner01` on `nas01` | `10.158.84.50` | Fleet OpenTofu root `incus/incus-gh-runner/` | | Reserved HTTP CONNECT forward | `10.10.10.14:3128` → `10.158.84.50:3128` | Same OpenTofu root, member-local to `nas01` | @@ -209,11 +212,12 @@ workloads stay off the management plane. These links are not required for IncusOS management boot. The host-side counterpart, converged by the `GilmanLab/fleet` `cluster/` project, is the `vlan_tags` allow-list plus an IncusOS-declared VLAN interface per carried VLAN (`fast30`, `fast40`). -Instances must attach to the IncusOS-owned interface (for example macvlan -with `parent=fast40`), never with an Incus-created `vlan=` sub-interface on -`fast`: the visible `fast` device is an IncusOS-internal VLAN-filtering -bridge, and only IncusOS-declared VLANs receive bridge self-port membership, -so other tagged sub-interfaces pass no traffic. Additional instance VLANs +The default-project physical network `fast40-uplink` exclusively owns `fast40`. +Workload NICs use managed OVN networks, not raw macvlan or bridge attachments +to that same parent. For any additional instance VLAN, use an IncusOS-declared +interface, never an Incus-created `vlan=` sub-interface on `fast`: the visible +`fast` device is an IncusOS-internal VLAN-filtering bridge, and only +IncusOS-declared VLANs receive bridge self-port membership. Additional VLANs join these links when their first consumer arrives. ## DHCP and DNS ownership diff --git a/docs/docs/runbooks/agentcompute-service.md b/docs/docs/runbooks/agentcompute.md similarity index 55% rename from docs/docs/runbooks/agentcompute-service.md rename to docs/docs/runbooks/agentcompute.md index 61d08cb..f6dbada 100644 --- a/docs/docs/runbooks/agentcompute-service.md +++ b/docs/docs/runbooks/agentcompute.md @@ -44,6 +44,45 @@ tailnet. - After enrollment, follow [Authorize agentcompute SSH to Studio](agentcompute-studio-ssh.md) to pin Studio access to the service's verified tailnet IPv4. +- On macOS, point the OpenTofu provider at the administration client's actual + configuration: `export INCUS_CONF="$HOME/Library/Application Support/incus"`. + Incus 7.3 uses that directory, while provider 1.2.0 defaults to + `$HOME/.config/incus`. A missing pin caused by that mismatch is not a reason + to enable certificate acceptance. + +## Configure an agent client + +Use a Streamable HTTP MCP connection, not an SSE-only URL or local stdio +server: + +| Setting | Value | +| --- | --- | +| Server name | `agentcompute` | +| Transport | Streamable HTTP | +| URL | `https://agentcompute01.tailda715.ts.net/` | +| HTTP header | `Authorization: Bearer ` | +| Token source | SOPS `auth_tokens.omp` in `services/agentcompute/credentials.sops.yaml` | +| TLS | Default public trust; never disable verification | + +Resolve the token through the client's secret or environment support. Do not +commit its value to client configuration, pass it in a command argument, or +paste it into an agent prompt. The authenticated initialization recipe below +keeps it on curl's standard input. + +The client discovers exactly three MCP tools: `search_api`, `describe_api`, +and `execute`. Start with a resource query such as `{"query":"sandbox"}`; +describe exact returned names before using them in `def main():`. Search is +not free-form planning: if a multi-resource sentence returns no matches, use +one resource or exact capability name. A minimal discovery-driven health +check describes `sandbox.list` and executes: + +```python +def main(): + return sandbox.list() +``` + +Use `sandbox.get` or `instance.list` for inventory details. Creating resources +is not necessary for every client health check. ## Safety boundary @@ -93,6 +132,7 @@ enrollment. ```bash export GLAB_SECRETS_DIR="$HOME/code/glab/secrets" export FLEET_DIR="$HOME/code/lab2/fleet" +export AGENTCOMPUTE_DIR="$HOME/code/lab2/agentcompute" export AWS_PROFILE=lab-admin umask 077 @@ -137,13 +177,17 @@ jq -e 'keys == ["omp"] and (.omp | type == "string" and length > 0)' \ "$AGENTCOMPUTE_CREDENTIALS/auth-tokens.json" >/dev/null ``` -Nothing private enters cloud-init, OpenTofu input, or OpenTofu state. The public -Incus client certificate, pinned server certificate, image catalog, and -non-secret runtime configuration do enter cloud-init. The three private source -files are delivered after the VM exists as `root:root` mode `0600` files in a -mode `0700` directory. The unit uses four `LoadCredential=` entries: the Incus -client certificate and key, bearer-token file, and Studio SSH key. Treat the -runtime credential mount as unit-private; do not infer its in-unit access from +Nothing private enters cloud-init, OpenTofu input, or OpenTofu state. Public +certificates, image catalog, SSH host-key pins, and runtime configuration do. +The three base private inputs are delivered after provisioning as `root:root` +mode `0600` files in a mode `0700` directory. Enabling Lume adds a fourth +private input, `mac-guest.key`, delivered separately from the qualified +Studio account by `just deliver-lume-key`. With `lume_host` set, deliver it +**before** `just deliver-credentials` on a first deployment or VM replacement; +the unit is gated on that file too. The unit loads the Incus client +certificate/key, bearer-token file, Studio SSH key, and enabled Mac guest key +with `LoadCredential=`. +Treat the runtime credential mount as unit-private; do not infer its access from ownership or mode observed outside the service namespace. ## Enroll the Incus identity @@ -201,11 +245,12 @@ incus exec --project default nas01:agentcompute01 -- \ jq -e '.status == "done" and .errors == []' >/dev/null ``` -Cloud-init is not a convergence mechanism. Except for a release pin, whose -installer is refreshed from applied state by `just install-release`, a changed -rendered configuration requires deliberate VM replacement followed by release -installation, credential delivery, and tailnet enrollment. Do not rerun the old -bootstrap on an existing VM. +Cloud-init is not a convergence mechanism. After a reviewed plan is applied, +`just install-release` converges the public runtime bundle—configuration, +catalog, unit, and SSH host pins—alongside the verified release. Bootstrap, +network, and certificate changes still require deliberate VM replacement, +release installation, credential delivery, and tailnet enrollment. Do not +rerun the old bootstrap on an existing VM. ## Verify and install the pinned release @@ -257,10 +302,16 @@ esac The workstation installer first refreshes the VM-side installer from the `release_installer` output in applied state. It resolves its module directory and requires initialized state-backend access. The VM-side installer repeats -the pinned digest and version checks, installs a versioned file under -`/usr/local/lib/agentcompute`, atomically changes -`/usr/local/bin/agentcompute`, and restarts the service only if it is already -running. A rejected asset does not replace the current binary. +the pinned digest and version checks, converges the public runtime bundle, +installs a versioned file under `/usr/local/lib/agentcompute`, atomically +changes `/usr/local/bin/agentcompute`, and restarts the service only if it is +already running. It does not overwrite private credentials. A rejected asset +does not replace the current binary or runtime bundle. + +A changed image catalog is reconciled before the MCP listener opens. During +that work, systemd can report `active` while Tailscale Serve returns `502`. +Wait for the service's `listening` log entry, then perform the HTTPS checks +below; `systemctl is-active` alone is not a readiness check. GitHub artifact and OCI attestations and the OCI Cosign signature are release evidence, not deployed-service acceptance. Do not claim SLSA Build Level 3: @@ -274,6 +325,10 @@ object with unique names and tokens and no whitespace or control characters. It pushes all three private files as root-owned mode `0600`, then restarts the unit. Install the release first so the initial service start cannot select an unverified binary. +With `lume_host` enabled, first follow +[the Mac guest-key delivery procedure](#deliver-permanent-runtime-configuration). +That delivery does not start the service; the base delivery below does. + ```bash just deliver-credentials \ @@ -443,6 +498,56 @@ Require a successful `initialize` result naming `agentcompute` before declaring the MCP endpoint accepted. A healthy systemd unit, valid TLS, and an HTTP `401` without a token do not prove authenticated MCP handling. +### Phase 9b qualification — 2026-09-16 + +Live qualification used the deployed HTTPS endpoint, normal certificate validation, +and the SOPS-managed `omp` identity from fresh workstation processes. The +initial network, TTL, snapshot, and Windows checks ran on v0.1.2; Mac rollout +and the later restart/concurrency checks ran on +[`v0.1.3`](https://github.com/GilmanLab/agentcompute/releases/tag/v0.1.3), +commit `d85426b72257998a339b91f854c8928ca9d3e361`. Its Linux amd64 SHA-256 is +`d7e8e5229e7299612f858ba6d39718d402fc3b80017cbe3da0c7c0e2a6859c63`; +the checksum and exact-tag/commit GitHub attestation passed before installation. + +The final deployed release is +[`v0.1.4`](https://github.com/GilmanLab/agentcompute/releases/tag/v0.1.4), +commit `18c881ae6c854030bfed8115ac50d51c221b0151`, Linux amd64 SHA-256 +`fc9394427e573132a2461f7764c23bc0448bbceecd2bbfdd1f901f5b636dc8be`. +Its exact-tag/commit attestation, checksum manifest, immutable release, and +release-asset verification passed. The applied upgrade changed only the +service's public cloud-init metadata and release outputs; the installer +converged the public runtime bundle without replacing the VM or credentials. + +| Contract | Observed result | +| --- | --- | +| One-minute TTL with a running VM | `p9b-ttl` expired at `02:04:44Z`; absent at `02:05:07.499Z`, 23.499 seconds later. `ac-p9b-ttl` was also absent from Incus project listing. | +| LAN-only client through guest NAT | Client `192.168.50.3` had only its LAN NIC and used router `192.168.50.2`. Verified HTTPS returned `200`; WAN capture showed source `10.99.0.2`, the router's WAN address. | +| Cross-member OVN and workstation forward | `lab01` and `lab02` containers exchanged three of three pings in each direction. Workstation HTTP to `10.10.40.67:8080` returned `p9b-final-forward`. | +| Isolated networks | `network=none`, `ipv4.nat=false`, and no external allocation. Direct lab/internet pings failed; intentional `net.peer` traffic succeeded both ways. A forward on an isolated network returned `AgentError`. | +| External-address budget | The representative sandbox held `.69` for default NAT, `.70` for WAN NAT, and `.71` for its forward; LAN held none. The service network separately held `.65`. Eight such sandboxes use 24 addresses, leaving 39 after the service's one address. | +| Snapshot semantics | Restore recovered the original file contents and returned `Running` with the same name but a different UUID, MAC, and DHCP address; all original snapshots were consumed. Separate create/list/delete checks retained only the named snapshot until its deletion. | +| Windows ready screenshot | Three initial captures took 1.146, 1.150, and 1.148 seconds. After a later service restart, readiness took 6.182 seconds and the subsequent real desktop capture took 1.379 seconds. | +| Restart rediscovery | `systemctl restart agentcompute.service` preserved discovery of all four then-existing sandboxes: three Incus and one Mac, with the same ten-instance total and expiry metadata. | +| Eight blocking executions | Eight simultaneous `sleep 5` programs returned their distinct results successfully. Tool times were 5.671–6.015 seconds; total workstation wall time, including client setup, was 7.381 seconds. | +| Mac lifecycle | HTTPS create returned Running in 44.992 seconds; `sw_vers` reported macOS 26.6.2 / 25G83; Driver 0.28.1 was ready; the 1920×1200 screenshot showed the desktop. Instance and sandbox deletion left only the stopped seed in Lume. | +| No-VNC boundary | Sampling began before the Mac create request and continued through Running. The first 1,000 listener observations contained only account-owned `127.0.0.1:7777`, no guest VNC listener. PF filter/NAT rules, the global Lume digest, and owner `rapportd` listeners were unchanged. | +| Fresh-shell operation | The runbook's service, routing, Serve, DNS, public TLS, unauthenticated `401`, and SOPS-authenticated initialization commands passed under `bash --noprofile --norc` with a minimal environment. | +| Fresh discovery-only workflow | On v0.1.4, an independent agent starting with `search_api` created `p9b-final`: NAT WAN `10.77.9.0/24`, isolated LAN `10.77.10.0/24`, dual-NIC router, and LAN-only Ubuntu desktop at `10.77.10.3`. It configured guest routing/DNS, verified HTTPS to `example.com`, listed applications, and captured the desktop without patching the image. The returned 1280×800 full-desktop image was visually verified over HTTPS. | +| Final fleet drift | OpenTofu reported `No changes. Your infrastructure matches the configuration.` with detailed exit code `0` after the v0.1.4 installation. | + +A separate isolated HTTP fault fixture exercised the actual Incus SDK and +restore adapter without faulting the deployed cluster. Stop failure retained +the running original and snapshot; copy failure retained them with the original +stopped; delete failure retained the stopped original plus staged copy; +rename failure retained the staged copy; start failure retained the stopped +replacement under the original name. The success control returned that +replacement Running. All six cases passed; the temporary harness was removed. + +Windows's persistent guest MCP session has a cold-start cost. A direct first +screenshot after restart, without a new readiness probe, took 5.275 seconds. +The sub-two-second result is measured **after readiness**, not a cold-call +latency guarantee. + ### Phase 9a qualification — 2026-09-15 - Deployed [`v0.1.1`](https://github.com/GilmanLab/agentcompute/releases/tag/v0.1.1), @@ -471,7 +576,7 @@ without a token do not prove authenticated MCP handling. access as `agentcompute`, and rejected the same key from Studio's own source with `Permission denied (publickey)`. -Two findings remain relevant to operations: +Two findings informed the later qualification: 1. `v0.1.0` rejected Tailscale Serve's preserved public `Host` header on the loopback listener. `v0.1.1` permits that path only with configured bearer @@ -479,9 +584,142 @@ Two findings remain relevant to operations: for unauthenticated loopback servers. 2. Whole-desktop capture returned a black `1280×800` image even after Text Editor launched. Explicit window capture rendered the editor correctly - (`822×642`), and accessibility state was available. This is a desktop-capture - finding, not an HTTPS delivery failure; full-desktop capture is not qualified - by this deployment. + (`822×642`), and accessibility state was available. Phase 9a did not qualify + whole-desktop capture. Phase 9b traced this to the Driver's cosmetic cursor + overlay freezing X root reads, including VNC; see the + [implementation outcome](../designs/agentcompute.md#runtime-and-image-contract-details). + +## Operate the Mac backend + +The host is the owner's `studio-1` Mac Studio, not a dedicated appliance. +Use only the hidden standard `agentcompute` account, its Lume store, and its +loopback daemon. Do not grant it administrator access or access to the owner's +home. The stopped seed is `ac-seed-macos-tahoe-desktop`; it is private and must +not be published. Two running macOS guests is the host-wide limit. + +### Install and activate the no-VNC build + +Lume 0.5.3 has no VNC-disable option. The temporary source build is pinned in +[`agentcompute/pins/lume.yaml`](https://github.com/GilmanLab/agentcompute/blob/master/pins/lume.yaml). +It records upstream commit `ab957bdb7566f7e137b00654cc01167d9e42af38`, +the actual signed Mach-O SHA-256, toolchain, dependency lock, and release +fallback. `--version` still prints `0.5.3`; that is **not** artifact verification. +The measured clean build is not bitwise reproducible. A checksum mismatch +requires a reviewed re-pin, not skipping the installer check. + +From the owner account, stage only the public build inputs outside the +protected home, then build as `agentcompute`: + +```bash +LUME_STAGE="$(mktemp -d /tmp/lume-build.XXXXXX)" +mkdir -p "$LUME_STAGE/images/macos" "$LUME_STAGE/pins" +cp "$AGENTCOMPUTE_DIR/images/macos/build-lume.sh" \ + "$AGENTCOMPUTE_DIR/images/macos/lib.sh" "$LUME_STAGE/images/macos/" +cp "$AGENTCOMPUTE_DIR/pins/lume.yaml" "$LUME_STAGE/pins/" +chmod -R a+rX "$LUME_STAGE" +sudo -u agentcompute -H "$LUME_STAGE/images/macos/build-lume.sh" +rm -rf "$LUME_STAGE" +``` + +The installer refuses root and writes only beneath `/Users/agentcompute`. +The launcher is `/Users/agentcompute/bin/lume`; the signed executable is +`/Users/agentcompute/.local/share/lume/lume.app/Contents/MacOS/lume`. +Leave `/usr/local/bin/lume` and its global app bundle untouched. + +With every backend worker stopped and the seed stopped, repoint only the +existing account daemon. Preserve its `UserName`, `HOME`, log paths, and +KeepAlive settings. Back up the plist first. Replace the whole argument array; +do not insert another executable as an extra argument: + +```bash +plist=/Library/LaunchDaemons/io.gilman.agentcompute.lume-serve.plist +sudo plutil -replace ProgramArguments -json \ + '["/Users/agentcompute/bin/lume","serve","--port","7777"]' "$plist" +sudo launchctl bootout system/io.gilman.agentcompute.lume-serve +sudo launchctl bootstrap system "$plist" +sudo lsof -nP -a -u agentcompute -iTCP -sTCP:LISTEN +``` + +Require only `127.0.0.1:7777` for the lifecycle API. Probe a unique, +nonexistent, colon-free name with a deliberately conflicting policy: + +```bash +curl --silent --show-error --write-out '\nHTTP %{http_code}\n' \ + --json '{"noDisplay":false,"vnc":"disabled"}' \ + "http://127.0.0.1:7777/lume/vms/ac-vnc-policy-probe-$(uuidgen)/run" +``` + +Require HTTP `400` and `VNC is disabled for this run`. The pinned daemon +validates this before any VM operation. Old 0.5.3 ignores the policy and returns +`202`; that is a failed gate, not permission to run a worker. + +No PF configuration is installed. A blanket high-port block was rejected +because it could disrupt LAN Continuity and `rapportd`; dynamic port watching +would leave a pre-discovery exposure window. Preserve Internet Sharing and +Apple's live PF anchors. Sample `lsof` continuously from a real backend run +request through Running: no account-owned per-VM VNC listener may appear, and +Lume inventory must report `vncUrl: null`. + +### Deliver permanent runtime configuration + +Fleet's `lume_host` is Studio's verified tailnet IPv4, `100.122.142.76`. +The fleet module supplies `studio_known_hosts` and `mac_guest_known_hosts`; +the latter pins the qualified seed's SSH key by seed-name `HostKeyAlias`, +not a reusable DHCP address. Verify pins through the already trusted host/seed, +never accept a fresh key merely because `ssh-keyscan` returned it. + +Stage the qualified seed's guest key from Studio in the owner-only credential +directory. It is separate from the service-to-Studio SSH key: + +```bash +sudo cat /Users/agentcompute/.ssh/guest_ed25519 \ + >"$AGENTCOMPUTE_CREDENTIALS/mac-guest.key" +chmod 0600 "$AGENTCOMPUTE_CREDENTIALS/mac-guest.key" +cd "$FLEET_DIR/incus/agentcompute" +just deliver-lume-key "$AGENTCOMPUTE_CREDENTIALS/mac-guest.key" +``` + +Review and apply the fleet plan with `lume_host` enabled and the release's +catalog, including `macos/tahoe/desktop`. Install the verified compatible +release to converge the public runtime bundle and load the new credential. +Require a clean plan, successful authenticated initialization, Mac create, +`sw_vers`, `desktop.screenshot`, and delete through deployed HTTPS. A temporary +systemd override is not permanent backend rollout. + +### Manual console and guest re-consent + +`desktop.screenshot` is the normal observation path. There is no automatic VNC +fallback. If a human needs a console, stop the exact guest, then **start it by +hand with VNC enabled**, for example as the confined account: + +```bash +sudo -u agentcompute -H /Users/agentcompute/bin/lume run \ + --no-display --vnc enabled +``` + +This deliberately opens Lume's wildcard VNC listener for the maintenance +session. Treat its URL/password as a secret; do not leave it running unattended +or expose it on an untrusted LAN. Stop it afterward and return normal workers +to the backend's disabled-VNC start path. + +For lost desktop permissions, have the operator open the **guest's** System +Settings → Privacy & Security. Re-enable CuaDriver under Accessibility and +Screen Recording; add `/Applications/CuaDriver.app` to Screen Recording if +absent. Let the operator authenticate and approve each permission. Do not +modify TCC databases, grant owner-host permissions, or automate consent. +Restart the guest's `com.trycua.cua_driver_daemon` LaunchAgent, then recheck +Driver readiness and a real screenshot. Qualify a disposable clone through the +deployed MCP endpoint: `desktop.info` must report readiness, `sw_vers` must +identify the expected release, and `desktop.screenshot` must show the desktop. +Then delete the clone and retain the stopped seed. The legacy +`images/macos/provision.sh` and `verify.sh --clone` scripts still assume +`lume ssh`; they are not a qualified rebuild or recovery path. + +Return to an account-local release pin once an upstream Lume release includes +[cua#3209](https://github.com/trycua/cua/pull/3209). Retire the source-build +procedure then, while retaining disabled-VNC enforcement. Falling back to the +old global 0.5.3 binary requires disabling the Mac backend; it is not an +equivalent console-safe release. ## Routine operations @@ -503,18 +741,56 @@ reconciles the pinned image catalog, rediscovers persisted sandboxes, and runs an immediate reaper scan before the next 30-second interval. Do not start a second process to preserve sessions during the restart. +### Check TTL cleanup + +Discover `sandbox.create`, `sandbox.get`, `instance.create`, and +`sandbox.list`; create a uniquely named one-minute sandbox with a small +running guest. Record the returned `expires_at`. Poll from fresh MCP sessions +until it disappears, then check the backend: + +```bash +incus project list nas01: --format=json +``` + +The corresponding `ac-` project must be absent within two minutes after +expiry, not merely hidden from the MCP list. For Mac sandboxes, also inspect +the confined account's Lume inventory and sandbox metadata. A restart must +still rediscover unexpired sandboxes and reap expired ones. Do not infer +cleanup from a successful HTTP response alone. + +### Recover a stuck sandbox + +1. Describe and call `sandbox.get` and `instance.list`. Capture the sandbox + name, expiry, guest state, and operation error, without bearer tokens or + screenshot URLs. +2. Check `journalctl -u agentcompute.service` and the matching Incus operation + or account-local Lume log. A slow clone/boot is not itself a stuck delete. +3. Use `sandbox.delete` first; it owns ordered resource cleanup. If it fails, + retry after resolving the reported backend problem. Do not create a second + project with the same name or delete shared OVN resources. +4. Before direct backend cleanup, verify the exact `ac-` project and + `user.agentcompute.*` ownership/expiry metadata. Stop or delete only its + instances, forwards, and networks, then its project. On Studio, operate + only on the named backend clone and metadata; never the stopped seed or + another user's VM. +5. Recheck both MCP and backend inventory. A transient not-found error during + concurrent expiry is different from residual backend resources. + ### Upgrade the release Update `service_version` and `service_sha256` together in `release.auto.tfvars`. Verify the new release with the procedure above, review and apply the saved OpenTofu plan, then run `just install-release` with the -verified asset. A release-pin plan must not replace the VM. The script resolves -its module directory, reads `release_installer` from applied state, and replaces -the VM-side installer before it stages the asset. It therefore needs the -initialized state backend and `lab-admin` AWS access, but no reboot or VM -replacement. Do not update only the symlink or download from the VM. Keep the -previous versioned binary until the new endpoint passes the complete -verification procedure. +verified asset. A release-pin plan must not replace the VM. The script reads +`release_installer` from applied state, replaces the VM-side installer, and +stages the asset. After digest and embedded-version verification, it installs +the public runtime bundle (config, catalog, SSH pins, and unit) and binary, +then restarts an already-running service. It never overwrites credentials, +network configuration, certificates, or Tailscale state. It needs initialized +state and `lab-admin` AWS access, but no reboot or VM replacement. Network and +bootstrap changes still require a deliberate replacement; changing cloud-init +metadata alone does not converge the live VM. Keep the previous versioned +binary until the new endpoint passes full verification. ### Rotate credentials @@ -542,6 +818,11 @@ For a release regression, return `release.auto.tfvars` to the last accepted version and digest, review and apply the plan, then install that verified asset through `just install-release`. Confirm the embedded tag commit and repeat the full verification procedure. Do not point the symlink at an unverified file. +With `lume_host` enabled, retain agentcompute **v0.1.3 or newer**: older releases +do not enforce disabled VNC. Disable the Mac backend before deliberately +rolling back below that boundary; never silently substitute the global Lume +0.5.3 binary or enable VNC to make a rollback start. + For a VM replacement, use a saved plan with an explicit replacement and expect to lose the VM's tailnet node identity and delivered files: @@ -552,8 +833,9 @@ tofu show tfplan tofu apply tfplan ``` -After cloud-init completes, install the release and deliver credentials. Before -enrollment, remove the old, offline `agentcompute01` device from the tailnet so +After cloud-init completes, install the release, deliver the Mac guest key if +Lume is enabled, then deliver the three base credentials to start the unit. +Before enrollment, remove the old, offline `agentcompute01` device from the tailnet so the replacement can receive the exact MagicDNS name. Then enroll and run the full verification. Never accept a collision name such as `agentcompute01-1`; the enrollment script refuses to publish Serve under that name. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index dc9f7fc..cc5b511 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -38,9 +38,14 @@ nav: - "ADR-0004: Manage RouterOS Devices with OpenTofu": decisions/0004-manage-routeros-devices-with-opentofu.md - "ADR-0005: Anchor Internal PKI on the KMS Root with Sibling Intermediates": decisions/0005-anchor-internal-pki-on-kms-root-with-sibling-intermediates.md - "ADR-0006: OVN as the Sandbox Network Fabric": decisions/0006-ovn-as-the-sandbox-network-fabric.md + - "ADR-0007: Cua Driver over guest execution": decisions/0007-use-cua-driver-over-guest-execution.md + - "ADR-0008: Lume on the owner's Mac Studio": decisions/0008-run-lume-in-a-confined-account-on-the-owners-mac-studio.md + - "ADR-0009: CodeMode MCP surface": decisions/0009-use-codemode-as-the-agent-facing-mcp-surface.md + - Architecture: + - Agentcompute: architecture/agentcompute.md - Designs: - Lab v2 core network: designs/lab-v2-core-network.md - - "Draft: agentcompute": designs/drafts/agentcompute.md + - Agentcompute: designs/agentcompute.md - Reference: - Device naming: reference/naming.md - Hardware inventory: reference/hardware-inventory.md @@ -52,7 +57,7 @@ nav: - Tailscale policy: reference/networking/tailscale-policy.md - Runbooks: - Change the Tailscale policy: runbooks/tailscale-policy-change.md - - Deploy and operate agentcompute: runbooks/agentcompute-service.md + - Deploy and operate agentcompute: runbooks/agentcompute.md - Authorize agentcompute SSH to Studio: runbooks/agentcompute-studio-ssh.md - Commission a lab compute node: runbooks/commission-lab-node.md - Operate OVN central and certificates: runbooks/ovn-central-and-certificates.md