From 47d5b455f43a9bdffe14e10eb98cbc7f63cbf5f4 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Sun, 13 Sep 2026 20:05:55 -0700 Subject: [PATCH 1/3] docs(ovn): record durable fabric and lifecycle qualification --- .../0006-ovn-as-the-sandbox-network-fabric.md | 205 +++++ docs/docs/designs/drafts/agentcompute.md | 706 ++++++++++++++++++ .../docs/reference/networking/address-plan.md | 70 +- docs/docs/runbooks/commission-lab-node.md | 16 +- .../runbooks/ovn-central-and-certificates.md | 638 ++++++++++++++++ docs/mkdocs.yml | 3 + 6 files changed, 1619 insertions(+), 19 deletions(-) create mode 100644 docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md create mode 100644 docs/docs/designs/drafts/agentcompute.md create mode 100644 docs/docs/runbooks/ovn-central-and-certificates.md diff --git a/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md new file mode 100644 index 0000000..a4ac8d2 --- /dev/null +++ b/docs/docs/decisions/0006-ovn-as-the-sandbox-network-fabric.md @@ -0,0 +1,205 @@ +--- +status: proposed +date: 2026-09-12 +decision-makers: + - Josh Gilman +--- + +# ADR-0006: OVN as the Sandbox Network Fabric + +## Context and Problem Statement + +Agentcompute needs disposable networks spanning Incus members, with NAT, +peering, ACLs, and operator-reachable forwards. Phase 2 bridges proved the +sandbox lifecycle but provide separate member-local L2 domains. Phase 3 +qualified OVN after removing a raw macvlan fixture that prevented OVS from +claiming the provider parent. Which fabric should the durable service use? + +## Decision Drivers + +- Cross-member guest connectivity without a router instance for ordinary + NAT-enabled and peered segments. +- Sandbox traffic cannot initiate connections to management or OOB. +- Fleet owns infrastructure; agents own only disposable sandbox resources. +- Management boot and OVN central must not depend on OVN itself. +- Failed creates and expired resources must converge through explicit, + retryable deletion rather than undocumented service repairs. + +## Considered Options + +- OVN with one independent central VM and the existing workload VLAN +- Member-local bridges only +- OVN with a dedicated provider VLAN +- A replicated OVN central cluster + +## Decision Outcome + +Proposed choice: **OVN with one independent central VM and the existing +workload VLAN**. The owner has approved the VLAN and address budget; this +record remains `proposed` until the owner accepts the fabric decision. + +Fleet's `incus/ovn-central/` OpenTofu root owns `ovncentral01`, one VM pinned +to `nas01` and attached to its unmanaged VLAN 10 `mgmt` bridge rather than an +OVN network. The pinned `ovn-central` package version `26.03.0-2` supplies +northd and standalone NB/SB databases, without Raft. Per-component TLS gates +prevent either database from starting without the complete certificate trio. +Remote database connections use mutual TLS from a dedicated offline OVN CA, +not the KMS root or the Incus cluster certificate. The CA uses EC P-256, has a +ten-year validity and no path-length constraint. Central and each chassis have +separate two-year leaves. Per-chassis keys allow one member's identity to +rotate without distributing the same private key to every node. The TLS +cutover occurred only after the cluster carried the CVE-2026-40243 fix. + +The owner approved this application-scoped trust domain on 2026-09-12, +superseding the draft's requirement to use the ADR-0005 hierarchy. +**OVN trust domain is application-scoped and offline; ADR-0005 governs the +KMS root hierarchy and is unaffected.** Revisit the issuance model when Vault +PKI exists and there is a reason to migrate. ADR-0005 is not amended. + +The CA key and every leaf key are escrowed under the `fleet` scope in +`GilmanLab/secrets`, following ADR-0003's alternative KMS/PGP recipients and +scoped encryption context. The CA key is used only in a controlled fleet +issuance ceremony and is never delivered to central or a chassis. Private leaf +material is delivered outside OpenTofu state and plaintext node seeds. + +A chassis leaf rotates in one fleet deploy. A central-leaf renewal and a +complete CA plus all-five-leaf replacement use the fleet `central-tls` +delivery while all central components are stopped. The delivery validates and +installs changed TLS files without touching `/var/lib/ovn` or controlling +service lifecycle. A full trust-set replacement starts central with the new CA +before fleet updates the Incus client and four chassis identities, producing a +brief fail-closed mismatch without a dual-CA interval. TLS renewal does not +replace the VM. Deliberate VM replacement remains reserved for bootstrap or +package changes and has the empty-database consequences described in the +[OVN central and certificate runbook](../runbooks/ovn-central-and-certificates.md). +OpenTofu's public CA and central-certificate metadata must be updated in place +after rotation; the central private key is never an OpenTofu input. + +Fleet's `cluster/` deployment owns chassis configuration on every member, +using VLAN 30 tunnel addresses and central's VLAN 10 endpoint. Supported +settings are mirrored in node seeds. The default-project physical uplink +exclusively owns its provider parent. Before convergence, fleet checks +member-specific networks, profiles, and instance NICs, including inherited +and stopped-instance devices. A competing direct parent attachment or +physical-uplink NIC aborts deployment with a named conflict; fleet never +removes it silently. + +The [address plan](../reference/networking/address-plan.md#ovn-external-addresses) +is authoritative for the approved external allocation and its eight-sandbox +budget. VLAN 40, DHCP allocations, and named endpoints stay in place. The +reservation is inside the uplink gateway subnet, not a routed subnet. Fleet +sets `ipv4.ovn.ranges` but does not add a redundant `ipv4.routes` allowlist; +the configured uplink gateway subnet already authorizes its addresses. +Reconsider a dedicated VLAN only if OVN needs more than the approved +64-address block. +The representative topology has NAT-enabled `default` and `wan` networks, an +isolated `lan`, and one distinct forward listen address. It consumes three +external addresses per sandbox: eight sandboxes consume 24 and leave 40 of +the 64-address reservation. + +New sandbox projects use project-owned OVN networks and managed-only NICs. +The default network has NAT. An additional NAT-enabled network consumes one +external address and has a direct path toward the lab and internet, subject to +ACL and gateway policy. An isolated `nat=false` network uses `network=none`, +consumes no external address, and has no direct path outside its sandbox. It +becomes reachable only through `net.peer` or a router instance attached to +another network. A `net.forward` request for an isolated network returns +`AgentError`. + +The `default_network_kind=bridge` fallback keeps whole sandboxes on +member-local bridges; explicit bridge networks remain bare wires there. Incus +cannot expose default-project bridges to an OVN project's managed-only NICs, +so mixed OVN/bridge requests return `AgentError`, not a rewritten network kind +or an unmanaged NIC. Existing bridge sandboxes expire and are deleted, never +converted in place. + +The server installs management/OOB baseline ACL denies that agent ACL calls +cannot remove. The current software contract also rejects every `allow` rule +unless `dst` is an explicit IP address or CIDR that does not overlap either +protected range. This applies to ingress and egress because Incus evaluates +native ingress fields from the opposite endpoint perspective. The restriction +remains conservative while native ACL interaction is qualified; this record +does not assume that a native priority relationship makes a broader allow +safe. + +### Consequences + +- OVN networks span members and provide logical routing, DHCP, DNS, ACLs, and + peering through Incus rather than guest-specific configuration. +- NAT-enabled networks provide SNAT and may host forwards. Isolated + `nat=false` networks provide neither an external allocation nor a direct lab + or internet path; a peer or dual-NIC router must supply intentional + reachability. +- One central VM is a single control-plane failure domain. Every OVN create, + update, and delete requires central availability. Existing installed flows + may continue during an outage; that does not permit mutations. +- An `Errored` NAT-enabled network keeps its external address until deletion. + Capacity accounting includes failed NAT-enabled networks and pending + teardown. +- During a central outage, the reaper leaves expired projects pending. After + central recovers, a later scan continues dependency-ordered deletion: + forwards and peers, NIC references, instances and snapshots, sandbox images, + profiles, networks and their ACLs, then the project. Recovery does not + restart OVS, chassis, the uplink, or central again. +- A provider-parent conflict is a separate infrastructure preflight failure. + Fleet names the competing network, profile, or instance NIC and refuses + convergence without deleting it. The sandbox reaper cannot repair that + conflict. +- NB database backups are optional because the consumers are disposable; + this is not a durability promise for long-lived workloads. +- Phase 3 observed guest MTU 1442 on the 1500-byte Geneve underlay. Forward + paths may have a smaller operator-side MTU; later desktop and deployment + work must qualify their actual paths. + +### Confirmation + +Phase 5 has established the infrastructure portion of this proposal: +OpenTofu produced a clean plan for the standalone central VM; the durable +central serves only mutual TLS; an issued client certificate completed +`ovn-nbctl show`; and a plaintext connection was refused. All four chassis +completed the TLS cutover. A subsequent 11-operation fleet dry run was a +no-op, and the transitional `sandbox01` central was purged through fleet. + +The complete CA and all-five-leaf rotation also passed live. The sequence was +central stopped, `central-tls`, central running with the new CA, then fleet's +OVN client/chassis converge. NB_Global, SB_Global, and two logical-switch UUIDs +were preserved. The new `lab03` identity authenticated to both NB and SB; the +old identity was rejected by both before an authenticated response. +A cross-member fixture completed three of three pings on each tested path. +Public OpenTofu cloud-init metadata reconciliation was still pending +AWS authentication and is not part of that proof. + +The application lifecycle acceptance run completed in 308.22 seconds. It +covered the dual-NIC router NAT path, forwards, native isolated OVN networks, +sandbox-local publication and clone, the approved snapshot-recreation +contract, and dependency-ordered deletion. + +The final central-outage qualification also passed. With central stopped, +fleet accepted one receipt-backed reboot of `lab03`. Its management API was +unavailable from `02:34:27.844Z` until `02:36:19.475Z`, approximately 112 +seconds, then the member returned `Online` and `Fully operational`. The +surviving `lab01`-to-`nas01` guest path completed three of three pings while +central was down and after the reboot. A guest on the rebooted member +completed zero of three during the outage, as expected. + +A separate sandbox create during the outage left its owned `default` network +`Errored` while holding `10.10.40.65`. A reaper scan returned backend +unavailable in 480 ms and retained the expired project and network. Central +was then started once, with no other repair or restart. Both guest paths +recovered to three of three pings, and a later reaper scan removed all owned +fixture residue across all four members and all projects in 3.03 seconds. The +post-rotation and post-reboot fleet dry run proposed no changes in all 11 +operations. + +These checks complete the live technical confirmation recorded here. The record +remains `proposed`; its status must not be changed without the owner's +separate acceptance of the decision. + +## More Information + +- [Agentcompute design draft](../designs/drafts/agentcompute.md) +- [Phase 3 qualification and parent recovery](https://github.com/GilmanLab/agentcompute/blob/spike/ovn-recreate-diagnosis/spikes/ovn/README.md): first post-fixture-deletion lab01-gateway cycle passed in 31.018 seconds, without reboot, central restart, or neighbor repair. +- [Incus #3985](https://github.com/lxc/incus/issues/3985): unavailable NB creation leaves an `Errored` network; deletion after central recovery releases it. +- [Incus #3986](https://github.com/lxc/incus/issues/3986): raw macvlan parent contention is separate from the central-outage failure. +- [Operate OVN central and certificates](../runbooks/ovn-central-and-certificates.md) +- [CVE-2026-40243 advisory](https://github.com/lxc/incus/security/advisories/GHSA-c839-4qxr-j4x3): patched in Incus 7.0.0 and later. diff --git a/docs/docs/designs/drafts/agentcompute.md b/docs/docs/designs/drafts/agentcompute.md new file mode 100644 index 0000000..33c7c11 --- /dev/null +++ b/docs/docs/designs/drafts/agentcompute.md @@ -0,0 +1,706 @@ +--- +title: agentcompute — disposable compute for agents +status: draft +authors: + - Josh Gilman + - agent (session 019) +created: 2026-09-11 +updated: 2026-09-13 +related-decisions: + - ADR-0006 +--- + +# agentcompute — disposable compute for agents + +## Summary + +`agentcompute` is a lab-specific MCP server, built on +[CodeMode](https://github.com/meigma/codemode) from +[`template-mcp-codemode`](https://github.com/meigma/template-mcp-codemode), +that lets coding agents create and drive throwaway containers and VMs on the +lab. An agent writes one short Starlark program against a small, stable +vocabulary (`sandbox.*`, `image.*`, `instance.*`, `net.*`, `desktop.*`) to +stand up an isolated environment, run commands in it, wire arbitrary network +topologies between instances, and see and operate a graphical desktop on +Linux, Windows, and macOS guests. Everything an agent creates lives inside a +named, time-limited sandbox that is destroyed as a unit. + +This is not a general product. It targets exactly the lab stack: the +four-node IncusOS cluster for Linux and Windows guests, and an Apple Silicon +host for macOS guests. + +## Context and Scope + +Agents doing lab and product work repeatedly need a machine that is not the +workstation: to test an installer, reproduce a networking bug behind a NAT, +verify a GUI, or try something destructive. Today that means `sandbox01` +(one shared Ubuntu host) or hand-driven `incus` commands. Neither gives an +agent a self-serve, isolated, disposable environment, and neither gives it a +desktop. + +The lab already has the substrate: + +- The Incus cluster (`nas01`, `lab01`–`lab03`, cluster pool `data`) + runs containers and KVM VMs, including Windows VMs from a repacked ISO. + Runtime cluster configuration is owned by the `GilmanLab/fleet` + `cluster/` pyinfra project; sandbox workloads are not cluster + configuration and are created imperatively through the Incus API. +- Incus projects give per-sandbox namespacing and resource limits; managed + bridge networks give per-sandbox L2 segments on one cluster member. +- VLAN 40 (`10.10.40.0/24`) is the routed sandbox/workload VLAN whose + firewall posture already fits untrusted workloads (it cannot initiate to + management or OOB). It is carried on the cluster's fast links, and + the default-project physical network `fast40-uplink` exclusively owns + the IncusOS-owned `fast40` parent. Sandbox NICs use managed logical networks. See the + [address plan](../../reference/networking/address-plan.md). +- macOS guests cannot run on Incus. They need Apple hardware running + Apple's Virtualization framework, driven by + [Lume](https://cua.ai/docs/reference/lume/cli-reference) (or Tart; see + Alternatives). +- [Cua Driver](https://cua.ai/docs/reference/cua-driver/platform-support) + is an in-guest desktop automation runtime for Windows, macOS, and Linux: + accessibility-tree snapshots, semantic and pixel actions, screenshots, + and browser control, exposed as MCP tools and as one-shot CLI calls with + JSON output. It is the same model the operator's agent harness already + uses for the host desktop. + +CodeMode shapes the vocabulary directly, so its constraints are design +inputs, not implementation detail: + +- Capabilities are dotted names called with keyword arguments only. +- Inputs are flat structs of scalars (`str`, `int`, `bool`, `float`, each + optionally `None`). No lists or nested objects on input. Multi-valued + operations are expressed as repeated calls inside the agent's program. +- Outputs may be rich (nested structs, lists, dicts) but are bounded (1 MiB + per value by default). Binary payloads such as screenshots must not cross + this boundary as values. +- One program is bounded by execution time and native-call count. Slow + operations (VM boot, image fetch) need either raised limits or explicit + wait capabilities. + +In scope: the agent-facing vocabulary, the mapping of that vocabulary onto +Incus and Lume, how agents reach guests, and the lab prerequisites. + +Out of scope for this draft: the exact Go package layout, the authorization +policy, and the deployment form of the server. Those follow the prototype. + +## Goals + +- An agent can create an isolated environment, launch Linux/Windows/macOS + instances into it, run commands, and destroy it, in one or two `execute` + calls and with no operator involvement. +- An agent has root-equivalent, unrestricted control inside its sandbox. +- An agent can build multi-segment topologies (NAT, routed, isolated, + impaired links) between instances in a sandbox. +- An agent can take screenshots of and send keyboard and pointer input to a + guest desktop. +- Sandboxes expire. Nothing an agent forgets outlives its TTL. +- The vocabulary is small enough to be discoverable through `search_api` and + stable enough that agents' saved programs keep working. + +## Non-goals + +- Durable workloads. Anything meant to survive the day belongs in OpenTofu + or CAPI, not here. +- Generality. No second-user abstraction over hypervisors; the two backends + (Incus, Lume) are named and their differences are exposed, not hidden. +- Cluster configuration. Storage pools, cluster-wide networks, and profiles + stay with `fleet`. +- Multi-tenant isolation between agents beyond project separation. The lab + has one operator. + +## Design Overview + +```mermaid +flowchart LR + A[Agent] -- MCP search/describe/execute --> S[agentcompute] + S -- Incus REST --> C[(Incus cluster)] + S -- lume serve HTTP --> M[(macOS host)] + C -- exec --> D[cua-driver in guest] + M -- ssh --> D +``` + +`agentcompute` is one Go binary from `template-mcp-codemode`. Each +capability handler is a thin translation onto either the Incus client +library or Lume's HTTP API. The server keeps almost no state of its own: +the sandbox registry is the set of Incus projects (and Lume VM name +prefixes) carrying `user.agentcompute.*` metadata, so a restarted server +sees the same world. + +Sandbox networking is **OVN**. IncusOS ships the OVN chassis (`ovn-controller` +and OVS, configured through the +[OVN service](https://linuxcontainers.org/incus-os/docs/main/reference/services/ovn/)) +but no OVN central. The lab's durable `ovncentral01` VM is pinned to `nas01`'s +unmanaged management bridge and runs `ovn-northd` with standalone NB/SB +databases. Both databases require mutual TLS from the application-scoped +offline OVN CA. See the +[OVN central and certificate runbook](../../runbooks/ovn-central-and-certificates.md). +Each sandbox network is an Incus OVN network: a Geneve overlay that spans the +cluster, with a logical router, DHCP, DNS, network ACLs, and peering built in. +A NAT-enabled network also has SNAT and can host network forwards. An isolated +`nat=false` network uses Incus `network=none`, has no external allocation, and +has no direct lab or internet path. It becomes reachable from another network +only through `net.peer` or a router instance attached to both networks. +Instances in one sandbox can land on different members, and ordinary +NAT-enabled and peered topologies need no router instance. + +The `default_network_kind=bridge` fallback keeps an entire sandbox on +member-local bridges, including bare L2 wires. Incus does not support mixing +those default-project bridges with project-owned OVN networks while retaining +managed-only NICs. A mixed-kind request returns `AgentError`; the server never +changes an explicit kind or relaxes the NIC restriction. + +Every sandbox starts with a **default network**: an OVN network with NAT +to the sandbox VLAN. Instances land on it unless the agent says otherwise. +That is how guests reach the internet and, through forwards, how agents +reach services they start. Experimental topologies are built on additional +networks. A guest that must have no side channel is attached only to an +isolated `nat=false` network; `instance.exec` and everything built on it still +work through the Incus agent. + +Desktop work rides **Cua Driver inside the guest, invoked over exec**. +Curated `/desktop` images ship a desktop environment with the Driver +daemon running in the logged-in graphical session. `agentcompute` runs +`cua-driver call --screenshot-out-file ` through the Incus agent (or `lume ssh`), +returns the structured result, and pulls any screenshot file through the +instance file API to serve it over HTTP. No guest network reachability is +required, so desktops work behind an agent's NAT topology. The hypervisor +console is not used: Incus exposes VGA only as SPICE, which has no usable +Go client. In-guest VNC remains available as the fallback for pre-login +screens, installers, and human viewers. + +## Detailed Design + +### Vocabulary + +The vocabulary is the contract. Everything else in this document exists to +make these names true. Conventions: + +- Five roots. `sandbox` is the lifecycle unit; the rest take `sandbox` as + their first argument. +- Verbs are the boring ones: `create`, `delete`, `list`, `get`, `start`, + `stop`, `restart`, `exec`, `wait`, `attach`, `detach`. +- Names identify things. Sandboxes, instances, networks, NICs, and + snapshots are addressed by the name the agent gave them, never by an + opaque ID. Names match `[a-z0-9]([a-z0-9-]{0,30}[a-z0-9])?`; `default` + and `none` are reserved. An omitted sandbox name is generated. +- Blocking by default. `instance.create` returns when the instance is + running; there is no operation-ID plumbing for the agent to poll. + Explicit `wait` capabilities cover the readiness stages that are slower + than "running". +- Errors are Starlark errors with a short, actionable message, via + `codemode.AgentError` ([meigma/codemode#59](https://github.com/meigma/codemode/pull/59)): + unknown or expired resources, unsupported kinds, and not-ready + instances say so by name. A capability an image or backend does not + support (for example `net.*` on the Mac) fails with a message that says + so; it is not silently no-op'd. Backend faults stay a bare + `capability failed` and are logged server-side. + +Optional arguments are marked `?`. Output shapes are indicative; the exact +shape is whatever `describe_api` reports from the Go types. + +#### `sandbox` — the lifecycle, isolation, and placement unit + +| Capability | Arguments | Returns | Notes | +| --- | --- | --- | --- | +| `sandbox.create` | `name?`, `platform?` (`incus` default, `mac`), `ttl_minutes?` | `{name, platform, expires_at, network}` | Creates an Incus project (or Lume name prefix) named after the sandbox, plus its `default` NAT'd network. Default TTL 240 minutes. | +| `sandbox.list` | — | `list[{name, platform, created_at, expires_at, instances: int}]` | | +| `sandbox.get` | `name` | `{name, platform, created_at, expires_at, instances: list[...], networks: list[...]}` | One call for an agent to re-orient. | +| `sandbox.extend` | `name`, `ttl_minutes` | `{expires_at}` | Extends from now. | +| `sandbox.delete` | `name` | `{}` | Records expiry, then destroys owned resources in dependency order. A partial failure remains discoverable for reaper retry; an unknown sandbox returns `AgentError`. | + +Expired sandboxes are deleted by a reaper inside `agentcompute`. + +#### `image` — the curated catalog + +| Capability | Arguments | Returns | Notes | +| --- | --- | --- | --- | +| `image.list` | `os?`, `desktop?`, `platform?` | `list[{name, os, version, kind, desktop, platform, description}]` | Catalog is a static list in the server config, mapping names to Incus aliases or Lume OCI references. | + +Indicative catalog: + +| Name | Kind | Backend | Notes | +| --- | --- | --- | --- | +| `ubuntu/24.04` | container or vm | Incus `images:` remote | Default Linux. | +| `ubuntu/24.04/desktop` | vm | lab-built | GNOME on Xorg, Cua Driver daemon in the user session, VNC fallback. X11 because the Driver's Linux support is strongest there. | +| `debian/13`, `fedora/43`, `alpine/3.22` | container or vm | Incus `images:` remote | | +| `router` | container | lab-built | Alpine or Debian with `nftables`, `frr`, `iproute2`/`tc`, `dnsmasq`, `wireguard`, `tcpdump`. The building block for NAT, routing, and impairment topologies. | +| `windows/11/desktop` | vm | lab-built, cluster-local | Repacked with `distrobuilder repack-windows`, virtio drivers, autounattend, Cua Driver daemon at logon, VNC fallback. Captured on the cluster; never published to a registry. | +| `windows/server-2025` | vm | lab-built, cluster-local | Headless. Same capture and non-publication rule. | +| `macos/sequoia/desktop` | vm | lab-built, Mac-local | `lume create --unattended` from a pinned IPSW; Cua Driver with Accessibility and Screen Recording granted once by an operator; Screen Sharing as the VNC fallback. Kept as a stopped seed on the Mac; never published to a registry. | + +Agents can add to a sandbox's local catalog with `instance.publish` (below). +Lab-built images are produced by a pinned recipe in the implementing +repository, consistent with the image-distribution principle, but they are +not part of this draft. Linux images are published to GHCR as imgoci +releases and imported by digest; Windows and macOS images stay lab-local +(an Incus alias on the cluster, a stopped Lume seed on the Mac) because +their licenses do not grant registry redistribution. The catalog entry +records a digest for the former and an alias or seed name for the latter. + +#### `instance` — containers and VMs + +| Capability | Arguments | Returns | Notes | +| --- | --- | --- | --- | +| `instance.create` | `sandbox`, `name`, `image`, `kind?` (`container`/`vm`; default from image), `cpus?`, `memory_mb?`, `disk_gb?`, `network?` (default `default`; `none` for no NIC), `host?`, `start?` (default `true`) | `{name, kind, host, status, addresses: dict[str, list[str]]}` | Blocks until the instance is running. Desktop images also wait for the Driver daemon. `host` pins a cluster member; default is least-loaded. | +| `instance.list` | `sandbox` | `list[{name, kind, image, status, addresses}]` | | +| `instance.get` | `sandbox`, `name` | `{name, kind, image, status, cpus, memory_mb, nics: list[{name, network, mac, addresses}], desktop: bool, snapshots: list[str]}` | | +| `instance.start` / `instance.stop` / `instance.restart` | `sandbox`, `name`, `force?` | `{status}` | Blocking. | +| `instance.delete` | `sandbox`, `name` | `{}` | Stops first if needed. | +| `instance.wait` | `sandbox`, `name`, `until` (`running`, `agent`, `network`, `desktop`, `stopped`), `timeout_seconds?` | `{status, elapsed_seconds}` | The one place slow readiness is spent. | +| `instance.exec` | `sandbox`, `name`, `command`, `timeout_seconds?`, `user?`, `cwd?`, `stdin?`, `env?` | `{exit_code, stdout, stderr, timed_out}` | `command` is a shell string run by the guest's native shell (`sh -c`, `cmd.exe /c`, `zsh -c`). `env` is a `KEY=VALUE\n` string because inputs are flat. Output is truncated to a stated byte cap. | +| `instance.file.write` | `sandbox`, `name`, `path`, `content`, `mode?` | `{bytes}` | Text-sized files. Large transfers use `instance.exec` with `curl`. | +| `instance.file.read` | `sandbox`, `name`, `path`, `max_bytes?` | `{content, truncated}` | | +| `instance.snapshot.create` | `sandbox`, `name`, `snapshot` | `{}` | Creates a stateless named snapshot. | +| `instance.snapshot.restore` | `sandbox`, `name`, `snapshot` | `{}` | Recreates the instance from the snapshot, retains the agent-facing instance name and metadata, starts it, and waits for `Running`. The UUID changes and the NIC MAC can change; successful restore consumes every snapshot on the original instance. | +| `instance.snapshot.delete` | `sandbox`, `name`, `snapshot` | `{}` | Deletes one named snapshot without restoring it. | +| `instance.snapshot.list` | `sandbox`, `name` | `list[{name, created_at}]` | Lists snapshots still attached to the current instance. | +| `instance.publish` | `sandbox`, `name`, `image` | `{image}` | Produces a sandbox-scoped image the agent can `instance.create` from. Dies with the sandbox. | + +#### `net` — sandbox networks and links + +| Capability | Arguments | Returns | Notes | +| --- | --- | --- | --- | +| `net.create` | `sandbox`, `name`, `kind?` (`ovn` default, `bridge`), `cidr?`, `dhcp?`, `nat?`, `dns?` | `{name, kind, cidr, gateway}` | `ovn`: a cluster-wide segment with a logical router at `.1`; `nat` masquerades to the sandbox VLAN; `dhcp` and `dns` are on by default. nat=false networks are unreachable from outside the sandbox; attach a router instance or use net.peer. `bridge`: a bare L2 wire on one member, no router, nothing served; the agent brings its own. | +| `net.list` / `net.get` / `net.delete` | `sandbox` [, `name`] | | Delete fails while NICs are attached. | +| `net.attach` | `sandbox`, `instance`, `network`, `nic?`, `ip?`, `mac?` | `{nic, mac}` | Hot-plugs a NIC. `nic` names the device (`eth1`…); `ip` requests a static lease. | +| `net.detach` | `sandbox`, `instance`, `nic` | `{}` | | +| `net.peer` | `sandbox`, `network`, `peer` | `{}` | Routes between two OVN networks in the sandbox without a router instance. | +| `net.acl.add` | `sandbox`, `network`, `direction` (`ingress`/`egress`), `action` (`allow`/`drop`/`reject`), `protocol?`, `src?`, `dst?`, `port?` | `{rule}` | Stateful rule on the network's logical router. `agentcompute` installs immutable management/OOB denies. An `allow` requires an explicit IP/CIDR `dst` outside both protected ranges, for ingress and egress. | +| `net.acl.remove` | `sandbox`, `network`, `rule` | `{}` | | +| `net.forward` | `sandbox`, `network`, `instance`, `port`, `listen_port?`, `protocol?` | `{address, port}` | Exposes an instance port on an address from the uplink's OVN range. Available only on NAT-enabled networks; an isolated-network request returns `AgentError`. | +| `net.impair` | `sandbox`, `instance`, `nic`, `latency_ms?`, `jitter_ms?`, `loss_percent?`, `rate_mbit?`, `clear?` | `{}` | Applies `tc netem` inside a Linux guest on that NIC. Intended for `router` instances so guests under test stay untouched. Fails on non-Linux guests. | + +OVN covers the floor. Routing protocols, exotic NAT (hairpin, port-restricted), +link impairment, VPN endpoints, and anything else OVN cannot express are +not capabilities: they are `instance.exec` against a `router` instance +attached to two or more networks. The `router` image is the extension +point: as recurring agent patterns show up, they become helper scripts +baked into the image before they become new vocabulary. + +#### `desktop` — Cua Driver, proxied + +All `desktop.*` capabilities require a desktop-capable instance: one whose +image is `/desktop`, or one the agent has prepared by installing Cua +Driver into the graphical session and calling `desktop.enable`. + +| Capability | Arguments | Returns | Notes | +| --- | --- | --- | --- | +| `desktop.info` | `sandbox`, `instance` | `{ready, os, driver_version, tools: list[str], vnc?: str}` | `tools` is the Driver tool catalog for that guest's OS. `vnc` is the fallback viewer endpoint when one exists. | +| `desktop.enable` | `sandbox`, `instance` | `{ready}` | Verifies the Driver daemon answers inside the guest. | +| `desktop.call` | `sandbox`, `instance`, `tool`, `args?` | `{ok, summary, result, screenshot_url?}` | Runs `cua-driver call --screenshot-out-file ` in the guest. `args` is a JSON object as a string because inputs are flat; `result` is the structured content as a JSON string for `json.decode`. Any screenshot is written in-guest, pulled, and served over HTTP. | +| `desktop.screenshot` | `sandbox`, `instance`, `pid?`, `window_id?`, `max_dimension?` | `{url, width, height, scale}` | Convenience over `get_desktop_state` / `get_window_state(include_accessibility_tree=false)`. | + +The Driver's own vocabulary is the desktop vocabulary: `list_apps`, +`launch_app`, `list_windows`, `get_window_state` (AX tree with element +tokens plus a screenshot), `click`, `type_text`, `press_key`, browser +tools, and the rest. `agentcompute` does not re-model those 50-odd tools +as flat-scalar capabilities; `desktop.call` is the pass-through, and +typed conveniences are added only where agents demonstrably fumble the +JSON string. The agent follows the Driver's action policy: prefer an +element token from a fresh `get_window_state`, fall back to pixels, and +escalate to foreground delivery last. Snapshot tokens are per +`(pid, window_id)` and expire on the next snapshot; `agentcompute` holds +no desktop state. + +### A representative program + +```python +def main(): + sb = sandbox.create(name="nat-repro", ttl_minutes=120) + lan = net.create(sandbox="nat-repro", name="lan", cidr="192.168.50.0/24", nat=False) + # The isolated LAN reaches outside only through the router's WAN NIC. + wan = net.create(sandbox="nat-repro", name="wan", cidr="10.99.0.0/24", nat=True) + + # Port-restricted NAT is beyond OVN's SNAT, so a router instance does it. + instance.create(sandbox="nat-repro", name="rtr", image="router", network="lan") + net.attach(sandbox="nat-repro", instance="rtr", network="wan") + instance.exec(sandbox="nat-repro", name="rtr", command="/opt/router/nat --mode port-restricted --inside eth0 --outside eth1") + + instance.create(sandbox="nat-repro", name="client", image="ubuntu/24.04/desktop", network="lan") + instance.wait(sandbox="nat-repro", name="client", until="desktop", timeout_seconds=300) + + shot = desktop.screenshot(sandbox="nat-repro", instance="client") + apps = json.decode(desktop.call(sandbox="nat-repro", instance="client", tool="list_apps")["result"]) + return {"router": instance.get(sandbox="nat-repro", name="rtr")["nics"], "screenshot": shot["url"], "apps": apps} +``` + +`client` sits only on `lan` behind `rtr`, which makes the NAT test honest. +Its desktop is still reachable because `desktop.*` rides `instance.exec`, +not the network. + +### Backend mapping + +| Concept | Incus | Lume (macOS host) | +| --- | --- | --- | +| Sandbox | Project `ac-` with `features.networks=true`, `features.images=true`, `user.agentcompute.expires_at`, `restricted` limits; instances placed per call with `--target` | Name prefix `ac--`; a JSON sidecar for metadata | +| Instance | Container or VM in the project | `lume clone` + `lume run --detach --display none` via `lume serve` | +| Default network | OVN network with `ipv4.nat=true` on the sandbox VLAN uplink | Lume is NAT-only ([trycua/cua#1007](https://github.com/trycua/cua/issues/1007)); guests reach out but are reachable only from the Mac | +| Additional networks | OVN networks in the project (cluster-wide); or managed `bridge` networks, which Incus instantiates on every member as separate L2 domains and which live in the `default` project under short generated names mapped by metadata (a bridge-backed sandbox keeps its instances on one member); peers, ACLs, and forwards are the Incus objects of the same names | Unsupported (`net.*` errors) | +| Exec | `/1.0/instances//exec` via the Incus agent | `lume ssh ` | +| Desktop | `cua-driver` CLI over exec; screenshot files pulled with the file API | `cua-driver` CLI over `lume ssh`; files pulled with `scp` | +| Snapshot | Instance snapshots | `lume clone` of a stopped VM | + +### Where the server runs + +The durable `agentcompute` deployment target is a **Streamable HTTP service** +in a VM on the cluster (OpenTofu-owned, per the one-off-VM decision), with a +client certificate for Incus and an SSH key for the Mac host. That is the form +that makes it usable from any agent host, keeps the reaper alive +independently of any agent session, and is the only form under which +the TTL guarantee holds. + +The stdio transport is a development convenience: run on the operator +workstation, reaching the Incus API on VLAN 10 and guests on VLAN 40 +through the existing Tailscale subnet routes. A stdio process exits with +its client, so sandboxes it created are reaped only when some +agentcompute is next running. Do not rely on stdio for anything with a +TTL that matters. + +### Lab prerequisites + +These changes live in other repositories. The first three infrastructure +prerequisites were completed in Phase 5. The application lifecycle and outage +behavior were qualified separately and are recorded under Validation. + +1. **OVN central — complete.** The OpenTofu-owned `ovncentral01` VM is pinned + to `nas01` on its unmanaged VLAN 10 management bridge. It runs the pinned + `ovn-central` package with standalone NB/SB databases and per-component TLS + gates. Remote database access is mutual TLS only. The dedicated offline OVN + CA is EC P-256 and valid for ten years; central and each chassis have + separate two-year leaves. Fleet owns issuance, delivery, and renewal under + ADR-0003. The CA is not part of ADR-0005's KMS hierarchy; revisit Vault + issuance when Vault PKI exists and there is a reason to migrate. The + transitional central on `sandbox01` has been purged. See the + [operations runbook](../../runbooks/ovn-central-and-certificates.md). +2. **OVN chassis on every node — complete.** Fleet converges + `/os/1.0/services/ovn` on all four members, with each member's VLAN 30 + storage address as its Geneve tunnel address and its own TLS leaf. The + Incus global OVN client uses the `nas01` leaf. +3. **An OVN uplink on the sandbox VLAN — complete.** Fleet owns the + cluster-wide `physical` uplink on the IncusOS-owned `fast40` parent and the + approved external range. The + [address plan](../../reference/networking/address-plan.md#ovn-external-addresses) + is the only source for that allocation and its capacity calculation. +4. **An Incus identity for `agentcompute`** with rights to create + projects. Fleet `cluster/` concern. +5. **A macOS host.** A dedicated Apple Silicon machine in the lab (a + Mac mini is enough) running `lume serve`, reachable from the server, + with SSH. Not a personal workstation: the consented Driver seed is a + security-relevant artifact and the backend must not vanish when a lid + closes. Apple limits a host to two concurrent macOS guests. +6. **Lab-built images**: the `router` container, one Linux desktop VM, one + Windows desktop VM. Built from pinned recipes in the `agentcompute` + repository until they earn a product home. +7. **CodeMode ≥ v0.2.1** for `AgentError` + ([meigma/codemode#59](https://github.com/meigma/codemode/pull/59), + [v0.2.1](https://github.com/meigma/codemode/releases/tag/v0.2.1)). + Without it every failure is a bare `capability failed`. + +## Cross-cutting Concerns + +### Security and Privacy + +Sandboxes are untrusted by construction: agents run arbitrary code as root +and can be prompt-injected by what they test. The sandbox VLAN's existing +firewall posture (no initiation toward management or OOB) is the boundary, +and it must hold for the new attachment path. Incus `restricted` project +settings prevent privileged containers, host device passthrough, and +nesting escapes. The Mac host is a softer boundary; it should hold nothing +but Lume. + +An isolated `nat=false` network has no direct path to the lab or internet. +This absence of an external path complements, rather than replaces, the +baseline ACLs. Peering or attaching a dual-NIC router intentionally adds a +path and remains subject to those ACLs. + +The server installs baseline egress ACL drops for the protected management and +OOB ranges. Agents cannot remove those rules. As a conservative current +contract, `net.acl.add` rejects every `allow` unless `dst` is a literal IP +address or CIDR that excludes both protected ranges. The same restriction +applies to ingress because Incus evaluates native ingress rules from the +opposite endpoint perspective. This restriction remains in place while native +ACL interaction is qualified; the design does not rely on an unproven priority +relationship. Sandbox projects retain `restricted.devices.nic=managed`. +`restricted.containers.lowlevel=block` is non-negotiable. Snapshot restore +recreates the instance rather than weakening this project restriction to make +Incus's native in-place restore succeed. +Before uplink reconciliation, fleet checks default-project networks (including +member-specific configuration), profile NICs, and instance NICs across all +members. Any competing parent attachment or direct physical-uplink NIC aborts +the deploy with the resource named; fleet never deletes a conflict. + +`agentcompute` itself is single-operator. Authorization starts as +`AllowAll`; the subject is recorded on each sandbox so a later policy can +scope agents to their own sandboxes. + +### Reliability and Failure Modes + +The server is stateless with respect to sandboxes, so a crash loses at +most in-flight screenshot files. In the durable deployment, the reaper runs +on process start and on a timer inside the long-running HTTP service. That +deployment, not an agent session, keeps the TTL promise. A sandbox pinned to a +member that goes offline is broken until the member returns; sandboxes are +not migrated. Deletion proceeds in dependency order: forwards and peers, NIC +references, instances and snapshots, sandbox images, profile references, +networks and owned ACLs, then the project. A partial failure leaves the +sandbox discoverable and is retried on every reaper scan until nothing owned +remains. + +Snapshot restore is a destructive replacement operation. Incus cannot natively +restore this running-instance snapshot under +`restricted.containers.lowlevel=block` because the saved +`volatile.eth0.host_name` is low-level configuration. The approved path stages +a stopped copy from the snapshot **before** deleting the parent, because parent +deletion also deletes the source snapshot. It then deletes the original and all +of its snapshots, renames the staged copy to the original agent-facing name, +reapplies the current `user.agentcompute.*` metadata, starts the replacement, +and waits for `Running` under the same sandbox service gate. See +[Incus issue #3993](https://github.com/lxc/incus/issues/3993). + +A successful restore consumes every snapshot that belonged to the original, +not only the selected snapshot. The replacement has a new instance UUID and +new volatile NIC state, so its MAC address, DHCP identity, lease, and assigned +address can change. Callers must rediscover the instance and its addresses +after restore. + +Failures before the staged copy completes leave the original stopped with its +snapshots intact; inspect for a generated `restore-*` copy before retrying. If +original deletion fails, the error identifies the retained staged copy. After +original deletion, its snapshots are already consumed: a rename failure leaves +the replacement under its generated name. A start or `Running`-wait failure +returns an error with the replacement under the original agent-facing name; +inspect its status before retrying. + +One live restricted-project probe on 2026-09-13 reached `Running` with the +agent-facing name preserved. Its instance UUID changed, and its source MAC +changed from `10:66:6a:71:f1:63` to `10:66:6a:f0:88:bc`. This confirms the +recreation and identity-change behavior; it is not full Phase 5 lifecycle +acceptance. + +OVN central must be available for every OVN create, update, and delete. +An owned `Errored` NAT-enabled network holds its external address until +deletion. During a central outage, an expired project stays pending; a later +reaper scan resumes dependency-ordered cleanup after central recovers. +The retry does not restart OVS, a chassis, the uplink, or central. Existing +installed flows may survive a central outage; that dataplane behavior does not +make control-plane mutations safe. + +A provider-parent conflict is separate from central unavailability. Fleet +preflight names a competing network, profile, or instance NIC and refuses +convergence without deleting it. The sandbox deletion retry does not own or +repair infrastructure parent conflicts. Phase 3 demonstrated this distinction: +after fleet deleted the raw macvlan fixture `default/soak01`, the first +lab01-gateway cycle passed in 31.018 seconds without reboot or +service/neighbor repair. See the +[qualification report](https://github.com/GilmanLab/agentcompute/blob/spike/ovn-recreate-diagnosis/spikes/ovn/README.md). + +### Performance and Capacity + +CodeMode's default 5 s execution budget and 100 native calls per program +are wrong for this workload. VM creation and desktop readiness are +minutes. The server raises `MaxExecutionTime` to the order of 15 minutes +and native calls to 1,000; `instance.wait` is where the time goes. +Screenshot images never cross the value boundary. Capacity is bounded by +project limits (indicative: 8 vCPU, 16 GiB, 100 GiB per sandbox) and by +the TTL reaper. Geneve tunnels ride the VLAN 30 storage links; sandbox +east-west traffic is small. + +The approved OVN range contains 64 external addresses. Only NAT-enabled +networks and distinct forward listen addresses consume them. The +representative `default` NAT, isolated `lan`, `wan` NAT, and one-forward +topology consumes three addresses per sandbox: eight sandboxes consume 24 and +leave 40 for additional allocations and pending cleanup. The +[address plan](../../reference/networking/address-plan.md#ovn-external-addresses) +is authoritative. + +## Delivery + +Agile: each step is a working slice that an agent uses before the next is +designed in detail. + +1. Vocabulary review (this document). Agree the nouns and verbs; rename + before code exists. +2. Prototype on the cluster with Linux containers only: `sandbox.*`, + `image.list`, `instance.create/exec/delete`, `net.create/attach` with + `kind="bridge"` and a temporary NAT'd bridge as the default network, so + neither the VLAN nor OVN blocks learning. Throwaway is acceptable. +3. OVN infrastructure: Phase 3 qualified the mechanism with the temporary + `sandbox01` central, four chassis using VLAN 30 tunnel addresses, a temporary + uplink, cross-node ping, and a forward. Phase 5 replaced that infrastructure + with the durable OpenTofu central, mutual TLS, the fleet-owned uplink, and + the approved external range, then purged the transitional central. The live + MCP lifecycle and central-outage acceptance evidence is recorded below; + infrastructure delivery alone did not establish that result. +4. Linux desktop image with Cua Driver, and `desktop.*` over exec. This is + the step most likely to change the vocabulary. +5. Windows desktop image. `net.impair` and the `router` image. +6. macOS via Lume. +7. Promote the draft; deploy the server as a cluster VM; write the + runbook; record the durable choices as decision records (OVN as the + sandbox fabric is one). + +### Validation + +- An agent, given only `search_api`, completes the representative program + above without operator help. +- A sandbox with `ttl_minutes=1` and one running VM is gone within two + minutes of expiry with no residue in `incus project list`. +- A `client` on `lan` behind `rtr` reaches the internet and shows a + routed source of `rtr`'s `wan` address from the `wan` side. +- Two instances on one OVN network placed on different members ping each + other; a `net.forward` on the default network is reachable from the + operator workstation. +- An isolated `nat=false` network receives no external allocation and has no + direct lab or internet path. `net.peer` and a dual-NIC router each provide + intentional reachability; `net.forward` on the isolated network returns + `AgentError`. +- The representative NAT-enabled `default`, isolated `lan`, NAT-enabled + `wan`, and one-forward topology consumes three external addresses per + sandbox: 24 for eight sandboxes, leaving 40 of the 64-address reservation. +- Snapshot create, list, and delete retain their named-resource behavior. + Restore returns only after a replacement with the same agent-facing name is + `Running`; the UUID changes, callers tolerate MAC/DHCP identity change, all + original snapshots are consumed, and each injected failure preserves the + staged or replacement instance described above. +- `desktop.screenshot` on a Windows guest returns an image in under 2 s + after the desktop is ready. + +### Phase 5 acceptance evidence + +The application lifecycle run completed in 308.22 seconds. It exercised the +dual-NIC router NAT path, forwards, native `nat=false` isolation, +sandbox-local publication and clone, the approved snapshot-recreation +contract, and dependency-ordered deletion. + +The central-outage qualification stopped central before rebooting only +`lab03` through fleet's receipt-backed reboot command. The member management +API was unavailable for approximately 112 seconds, then returned `Online` and +`Fully operational` while central remained stopped. A surviving +`lab01`-to-`nas01` guest path completed three of three pings throughout. A +guest on the rebooted member completed zero of three during the outage; after +central was started once, with no other repair or restart, both paths +completed three of three. + +A separate create during the outage left an owned expired project and an +`Errored` NAT-enabled `default` network holding `10.10.40.65`. The reaper +returned backend unavailable in 480 ms and retained both for retry. After the +single central start, a later scan removed all owned fixture residue across +all four members and all projects in 3.03 seconds. The subsequent 11-operation +fleet dry run was a no-op. + +## Alternatives Considered + +### One MCP tool per operation (no CodeMode) + +- Familiar; no Starlark. +- Every topology needs a dozen round trips, each dragging intermediate + output into context. +- Not chosen: composition in-program is exactly what multi-instance, + multi-network setups need, and codemode is the house style. + +### Raw VNC as the primary desktop path + +- Backend-agnostic; works at the login screen and inside installers. +- Pixels only: no accessibility tree, no semantic actions, no browser + control; needs a reachable guest network; agents must reinvent + window discovery from screenshots. +- Not chosen as primary. Kept as the fallback for pre-login screens and + human viewers. + +### Hypervisor console (SPICE) for desktops + +- Works before any guest agent exists; works for any OS on Incus. +- No usable Go SPICE client; would need a headless native client as a + sidecar. Not applicable to Lume. +- Not chosen for now. + +### Tart instead of Lume for macOS + +- Bridged and softnet networking, a Packer plugin for image recipes, and + a longer CI track record. +- Fair Source license; CLI only, no API server; no first-party pairing + with an in-guest agent. +- Not chosen provisionally. Lume's HTTP API, MIT license, and Cua Driver + images fit better now that guest networking is not needed for desktop + work. Swap if Lume's NAT-only networking or stability disappoints in + step 6; the adapter is thin. + +### Cua Sandbox SDK or Cua Fleets as the sandbox layer + +- The Sandbox SDK already models images, exec, and screenshots; Fleets + are managed cloud desktops with zero lab work. +- Python SDK with local runtimes of QEMU, Docker, Hyper-V, Lume, and the + Android emulator; no Incus runtime, no multi-segment networking, + snapshots not implemented. Fleets are off-lab, paid, Linux and Windows + only. +- Not chosen. It would replace the substrate rather than complement it, + and the lab is the point. Fleets remain the buy option if the lab path + stalls. + +### Per-member bridge networks only (no OVN) + +- Nothing new to run; managed bridges already work on the cluster. +- Sandboxes pinned to one member; NAT, DHCP, and firewalling every time + through a router instance; agents reach services only via a management + NIC on the raw VLAN. +- Not chosen as the target; it is the step-2 prototype and the fallback if + the OVN spike is not smooth. Bridges stay available as `kind="bridge"`. + +### Kubernetes-hosted sandboxes (KubeVirt, vcluster) + +- Would land on the eventual platform cluster. +- The platform cluster does not exist yet; desktops and Windows on KubeVirt + are a much longer road; macOS impossible. +- Not chosen. + +### Doing nothing (keep using `sandbox01` by hand) + +- Zero work. +- No isolation, no desktops, no topologies, no cleanup. +- Not chosen. + +## Open Questions + +### Resolved + +Resolved in review, kept here until the draft is promoted: + +- **The word `sandbox`** stays. Agents already think in it; the collision + is with a host and a repository, neither of which an agent sees. +- **Screenshot return path** is the HTTP URL only. No base64 fallback: + the deployed service is reachable from agent hosts the same way + everything else in the lab is, and the agent harness reads image URLs. +- **macOS host** is a dedicated Apple Silicon box in the lab (prerequisite + 5). +- **Deployment form** is Streamable HTTP in a cluster VM; stdio is + development-only ("Where the server runs"). +- **Blocking operations** stay. Raise `MaxConcurrentExecutions` before + considering an asynchronous pattern. +- **OVN central** is the OpenTofu-owned `ovncentral01` VM on `nas01`; no Raft. + The [operations runbook](../../runbooks/ovn-central-and-certificates.md) + defines its deployment, renewal, and recovery. Whether it later serves Talos + networks is a T10/T11 question, not this design's. +- **Windows and macOS images are not published** to any registry; they + are cluster-local and Mac-local respectively ("image" catalog). +- **`desktop.call` ergonomics**: pass-through first; typed conveniences + only for tools agents demonstrably fumble. +- **VLAN 40** is the durable OVN uplink VLAN (owner decision, 2026-09-12). + The [address plan](../../reference/networking/address-plan.md#ovn-external-addresses) + records the approved 64-address allocation and eight-sandbox planning + target. Default-plus-forward consumes two external addresses per sandbox; + adding `lan` and `wan` brings that budget to four. Count `Errored` + networks until deletion. A dedicated VLAN is reconsidered only if OVN + needs more than this allocation; DHCP, named endpoints, and routes stay + unchanged. + +### Still open + +1. **Cua Driver token continuity across one-shot CLI calls.** Decided by + the desktop spike: snapshot in one call, act by token in the next. If + tokens do not survive, `desktop.call` drives a long-lived + `cua-driver serve` per instance over exec; the vocabulary is unchanged. +2. **Cua Driver inside images.** How the daemon is kept alive in the + graphical session on each OS is image-recipe work learned in the + desktop slices. Pin the newest non-prerelease Driver per image; bump + deliberately. diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index b2b1be8..97ac9ee 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -66,6 +66,7 @@ by the `GilmanLab/fleet` `cluster/` project and mirrored in each node's seed. | `gw01` OOB gateway | `10.10.70.1` | Interface address | | `sw-mgmt01` management | `10.10.70.2` | Interface address | | `gw01` `glab.lol` mirror | `10.10.10.54` | Local service address | +| `ovncentral01` OVN central | `10.10.10.15` | Static VM interface on `nas01`'s unmanaged `mgmt` bridge | ### Hosts @@ -87,26 +88,61 @@ configuration. Infrastructure endpoints do not depend on DHCP: gateway and managed-switch interface addresses and the local DNS mirror address are static interface -configuration, IncusOS node management addresses are static in each node's -seed (bound to the management NIC's hardware MAC in `GilmanLab/fleet`), -storage-network addresses are static IncusOS runtime configuration (converged -by the fleet `cluster/` project and mirrored in the seeds), and lab-node AMT -addresses are static in MEBx so out-of-band access survives a gateway outage. -The AMT interfaces have no DHCP reservations. +configuration; the OVN central address is static VM network configuration; +IncusOS node management addresses are static in each node's seed (bound to the +management NIC's hardware MAC in `GilmanLab/fleet`); storage-network addresses +are static IncusOS runtime configuration (converged by the fleet `cluster/` +project and mirrored in the seeds); and lab-node AMT addresses are static in +MEBx so out-of-band access survives a gateway outage. The AMT interfaces have +no DHCP reservations. ### OVN external addresses -Reserve `10.10.40.64` through `10.10.40.79` for the agentcompute OVN -spike and its Phase 5 handoff. Fleet owns the uplink allocation. These are -external router and network-forward addresses on VLAN 40, not a separate -routed subnet or an extension of the `.200`–`.250` DHCP pool. Do not assign -them to other endpoints. - -Temporary OVN central runs on `sandbox01` at `10.10.40.10`, with northbound -TCP port `6641` and southbound TCP port `6642`; it remains running until -Phase 5 replaces it. Chassis encapsulation uses the members' VLAN 30 -addresses. See the [spike report](https://github.com/GilmanLab/agentcompute/blob/spike/ovn-mechanism/spikes/ovn/README.md) -for measurements and the northbound/chassis sequencing constraint. +Reserve `10.10.40.64` through `10.10.40.127` (64 addresses) for OVN +virtual-router external addresses and network forwards. The owner approved +this durable allocation on 2026-09-12, replacing the `.64`–`.79` spike +reservation. Fleet owns `ipv4.ovn.ranges` on the default-project physical +network `fast40-uplink`. + +The allocation stays inside the physical uplink's existing +`10.10.40.0/24` gateway subnet; it is not a new routed subnet and requires no +new route or VLAN. Do not add this reservation to `ipv4.routes` on +`fast40-uplink`: Incus authorizes addresses from the uplink's configured +gateway subnet. The `.200`–`.250` DHCP pool and named endpoints remain +unchanged. Do not assign the reserved addresses to other endpoints. + +Capacity accounting charges one external address for each NAT-enabled OVN +network and one for each distinct forward listen address. An isolated +`nat=false` network uses `network=none` and consumes no external address. +Guests use internal addresses, and additional ports sharing a listen address +do not consume another external address. Forwards are not available on +isolated networks. + +| Sandbox topology | External addresses per sandbox | Eight concurrent sandboxes | +| --- | ---: | ---: | +| NAT-enabled `default` network and one forward listen address | 2 | 16 | +| NAT-enabled `default`, isolated `lan`, NAT-enabled `wan`, and one forward listen address | 3 | 24 | + +The planning target is eight concurrent sandboxes for the single operator +and their agents. The representative three-network topology uses 24 of the +64 addresses and leaves 40 for additional NAT-enabled networks, distinct +forward addresses, and pending cleanup. These are address-budget +calculations, not measured scale limits. + +An `Errored` NAT-enabled OVN network retains its external address until +deleted and counts against capacity. If central is unavailable, the reaper +leaves cleanup pending and retries deletion after central recovers; it does not +repair the network with service restarts. Revisit a dedicated VLAN only +when OVN needs more than this 64-address allocation. + +Durable OVN central uses the `ovncentral01` allocation listed above. The VM is +pinned to `nas01` and attaches directly to its unmanaged VLAN 10 `mgmt` +bridge, so management boot and central do not depend on OVN. The standalone +northbound and southbound databases accept mutual TLS only on ports `6641` +and `6642`; chassis encapsulation uses the members' VLAN 30 addresses. The +qualification spike central on `sandbox01` has been removed. See the +[OVN central and certificate runbook](../../runbooks/ovn-central-and-certificates.md) +for deployment, renewal, and recovery procedures. ### Incus-local image runner networks diff --git a/docs/docs/runbooks/commission-lab-node.md b/docs/docs/runbooks/commission-lab-node.md index ec95c09..135a1a0 100644 --- a/docs/docs/runbooks/commission-lab-node.md +++ b/docs/docs/runbooks/commission-lab-node.md @@ -17,8 +17,9 @@ canonical for every address used here. Seed configuration lives in ## Preconditions and required access - A `GilmanLab/fleet` config for the node with its **verified** 10GbE - management NIC MAC (`strict_hwaddr` binding; a wrong MAC leaves the node - unreachable after install). See "Harvest the management MAC" below. + management NIC MAC. Lab compute nodes keep `strict_hwaddr: true`; a wrong + `hwaddr` leaves the node unreachable after install. See "Harvest the + management MAC" below. - `incusos-builder` at the pin recorded in the fleet README, and a sacrificial USB stick. Install media MUST be a raw image written with `dd`; never boot the installer through Ventoy, and never attempt an AMT IDER/USB-R boot (see @@ -107,6 +108,17 @@ as a prediction only; verify before building. Record the MAC in `GilmanLab/fleet` `nodes//config.yaml` through the normal PR flow. +The `hwaddr` value selects the physical NIC that becomes `mgmt`. +`strict_hwaddr` is a separate guest-egress policy: when enabled, it prevents +frames with another source MAC from leaving that bridge. It does not create or +remove the permanent interface identity binding. + +Keep `strict_hwaddr: true` on `lab01`–`lab03`. The only exception is +`nas01`, where fleet sets it to `false` so the bridged `ovncentral01` VM can +egress with its own source MAC. The VM NIC still sets +`security.mac_filtering=true`, which confines the guest to its assigned MAC. +Do not copy the `nas01` exception to a lab compute node. + ### 3. Build and write install media ```sh diff --git a/docs/docs/runbooks/ovn-central-and-certificates.md b/docs/docs/runbooks/ovn-central-and-certificates.md new file mode 100644 index 0000000..bb6678b --- /dev/null +++ b/docs/docs/runbooks/ovn-central-and-certificates.md @@ -0,0 +1,638 @@ +--- +title: Operate OVN central and certificates +description: Issue and renew OVN certificates, deploy the durable central VM and chassis configuration, and recover the OVN control plane. +--- + +# Operate OVN central and certificates + +Use this runbook to issue or renew OVN certificates, deploy or rebuild +`ovncentral01`, converge the four IncusOS chassis, and recover after an OVN +control-plane outage. The [network address and VLAN +plan](../reference/networking/address-plan.md#ovn-external-addresses) is the +only source for central endpoints, the provider uplink, and external address +allocations. + +The durable central is an OpenTofu-owned VM pinned to `nas01`'s unmanaged +`mgmt` bridge. It runs `ovn-northd` and standalone northbound and southbound +databases. Both databases accept mutual TLS only. Fleet owns the VM, the +central service lifecycle, Incus client TLS, all four chassis, and the +physical uplink. + +## Preconditions and required access + +- Trusted administration workstation with `openssl`, `ovn-nbctl`, `sops`, + `jq`, OpenTofu, `just`, Moon, and the fleet-pinned Python environment. +- `GilmanLab/fleet` and `GilmanLab/secrets` checkouts. Use the reviewed revision + of each repository. +- `lab-admin` AWS credentials for the SOPS KMS key and the OpenTofu state + backend. Set `GLAB_AWS_STATE_BUCKET` to the existing fleet state bucket. +- An authenticated `nas01` Incus remote. Set `INCUS_CONF` to the existing + administrator configuration so OpenTofu uses the pinned cluster certificate + and client identity; do not enable automatic certificate acceptance. +- An owner-only absolute directory for plaintext ceremony material. Keep it + outside every repository, set mode `0700`, and remove it after deployment. +- Before a planned IncusOS member reboot, create the persistent receipt + directory on the execution host. It must already exist, be writable by its + owner, and have mode `0700`; `fleet-cluster reboot` refuses the request + before contacting the API when the directory is absent. +- A maintenance window for TLS rotation or central replacement. TLS rotation + creates a bounded control-plane outage without replacing the VM. A central + replacement is reserved for bootstrap or package changes and has the + database consequences described below. + +## Safety impact + +- The OVN CA is an application-scoped offline CA: EC P-256, ten-year validity, + `CA:TRUE`, and no path-length constraint. It is not the ADR-0005 KMS-root + hierarchy. Here, offline means that no CA signing service or CA private key + is deployed. A controlled issuance ceremony may use a connected + administration workstation because KMS-backed SOPS escrow can require + connectivity. Revisit Vault-backed issuance only when Vault PKI exists and + there is a reason to migrate. +- Central and each chassis have separate two-year leaves. The Incus daemon + uses the `nas01` leaf as its global OVN client identity; rotating `nas01` + changes both that identity and the `nas01` chassis identity in one converge. +- Never deliver the CA private key to a node. Never put a leaf private key in + OpenTofu variables, state, cloud-init, an IncusOS seed, a command argument, + or a log. +- `fleet-cluster central-tls` is the only central TLS delivery path. It + validates the complete CA, certificate, and key set, installs changed files + only, refuses a change while any central component is active, and never + starts, stops, or restarts a component. It does not touch `/var/lib/ovn`. +- Do not restart central as a repair. A restart disconnects every chassis and + Incus client at once but does not repair the cause. Planned service work is + an explicit `stopped` transition followed by `running`. +- Replacing `ovncentral01` starts the standalone databases empty unless a + reviewed, compatible database backup is restored. Automatic reconstruction + of existing sandbox network state has not been qualified. Without a backup, + treat existing sandbox networking as invalid and recreate the disposable + sandboxes through the normal lifecycle after central is healthy. This is not + a durability promise for long-lived workloads. +- The retired `sandbox01` spike central is not a rollback target. Do not + reinstall or restart it. + +## Issue the initial certificate set + +Run the controlled issuance ceremony from fleet's `cluster/` directory: + +```sh +export OVN_TLS_DIR=/absolute/path/to/owner-only-directory +install -d -m 0700 "$OVN_TLS_DIR" + +cd /path/to/fleet/cluster +uv run --locked python -m fleet_cluster ovn-ceremony \ + --dir "$OVN_TLS_DIR" +``` + +The command creates `ca.crt` and `ca.key`, the `ovncentral01` certificate and +key, and one certificate and key for each chassis. It prints JSON containing +subjects, expiry dates, SHA-256 fingerprints, and the required SOPS escrow +path for every artifact. It never prints a private key or contacts the +cluster. + +Review the JSON and escrow every certificate/key pair at the paths it reports: + +- CA: `fleet/shared/ovn-ca.sops.yaml` +- central leaf: `fleet/shared/ovn-central.sops.yaml` +- chassis leaves: `fleet//ovn.sops.yaml` + +Use the secrets repository's normal SOPS and review flow. KMS-backed SOPS +escrow can require network access; the offline issuer model does not require +physical workstation disconnection. Each OVN file has `certificate` and `key` +fields. Do not deploy until all six identities can be materialized back into +one mode-`0700` directory with private keys mode `0600`. After escrow, remove +the plaintext CA key before starting deployment: + +```sh +rm -f -- "$OVN_TLS_DIR/ca.key" +``` + +The cluster converge needs `ca.crt` and the four chassis pairs; central +deployment also needs the central pair. + +### Materialize the escrowed set + +Create a fresh owner-only directory, then decrypt the public CA certificate, +the central pair, and all four chassis pairs from the reviewed secrets +checkout: + +```sh +export OVN_TLS_DIR=/absolute/path/to/owner-only-directory +install -d -m 0700 "$OVN_TLS_DIR" +cd /path/to/secrets + +AWS_PROFILE=lab-admin sops -d --extract '["certificate"]' \ + fleet/shared/ovn-ca.sops.yaml >"$OVN_TLS_DIR/ca.crt" +AWS_PROFILE=lab-admin sops -d --extract '["certificate"]' \ + fleet/shared/ovn-central.sops.yaml >"$OVN_TLS_DIR/ovncentral01.crt" +AWS_PROFILE=lab-admin sops -d --extract '["key"]' \ + fleet/shared/ovn-central.sops.yaml >"$OVN_TLS_DIR/ovncentral01.key" + +for member in nas01 lab01 lab02 lab03; do + AWS_PROFILE=lab-admin sops -d --extract '["certificate"]' \ + "fleet/$member/ovn.sops.yaml" >"$OVN_TLS_DIR/$member.crt" + AWS_PROFILE=lab-admin sops -d --extract '["key"]' \ + "fleet/$member/ovn.sops.yaml" >"$OVN_TLS_DIR/$member.key" +done + +chmod 0644 "$OVN_TLS_DIR"/*.crt +chmod 0600 "$OVN_TLS_DIR"/*.key +``` + +For a controlled leaf-issuance ceremony only, also materialize the CA key: + +```sh +AWS_PROFILE=lab-admin sops -d --extract '["key"]' \ + fleet/shared/ovn-ca.sops.yaml >"$OVN_TLS_DIR/ca.key" +chmod 0600 "$OVN_TLS_DIR/ca.key" +``` + +Remove `ca.key` again after the new leaf pair is escrowed and before +deployment. + +## Deploy the durable central and chassis + +### 1. Plan and apply the central VM + +Point OpenTofu at the public CA and central certificates. The private central +key is deliberately not an input: + +```sh +cd /path/to/fleet/incus/ovn-central +export AWS_PROFILE=lab-admin +export GLAB_AWS_STATE_BUCKET= +export INCUS_CONF= +export TF_VAR_ovn_ca_certificate_file="$OVN_TLS_DIR/ca.crt" +export TF_VAR_central_certificate_file="$OVN_TLS_DIR/ovncentral01.crt" + +just init +just plan +tofu show tfplan +just apply +``` + +Review the saved plan before `just apply`. The root owns only `ovncentral01`. +It must not change chassis configuration or the provider uplink. The VM's TLS +gates keep the database components stopped until all three central TLS files +exist. + +### 2. Deliver central TLS and start the components + +Wait for the Incus guest agent and cloud-init before delivering TLS material. +The agent check stops after 150 seconds; the in-guest cloud-init wait stops after +10 minutes: + +```sh +( + set -eu + for attempt in $(seq 1 30); do + if incus exec --project default nas01:ovncentral01 -- true \ + >/dev/null 2>&1; then + ovn_central_agent_ready=true + break + fi + sleep 5 + done + test "${ovn_central_agent_ready:-false}" = true || { + echo "ovncentral01 guest agent did not become ready" >&2 + exit 1 + } + incus exec --project default nas01:ovncentral01 -- \ + timeout 600 cloud-init status --wait +) +``` + +Stop if either check fails. From fleet's `cluster/` directory, validate and +review the TLS delivery plan before applying it. The directory must be an +absolute owner-only path and contain `ca.crt`, `ovncentral01.crt`, and +`ovncentral01.key`: + +```sh +cd ../../cluster +uv run --locked python -m fleet_cluster central-tls \ + --dir "$OVN_TLS_DIR" \ + --dry-run +uv run --locked python -m fleet_cluster central-tls \ + --dir "$OVN_TLS_DIR" +uv run --locked python -m fleet_cluster central --state running +``` + +On the initial deployment, the TLS gates leave every central component +stopped. `central-tls` validates the certificate chain, validity periods, +central identity, matching key, directory mode `0700`, and key mode `0600`. +It stages and verifies changed files before installing them as `root:root` +with the CA and leaf mode `0644` and the key mode `0600`. No private key is an +OpenTofu input. + +The lifecycle command starts the northbound database, southbound database, +`ovn-northd`, and the remote-listener publisher in dependency order. It +refuses to report success when the CA certificate, central certificate, or +central key is missing. A rerun with the requested state already present is a +no-op. + +### 3. Verify central before touching a chassis + +Print the root's read-only acceptance commands and run each command exactly as +rendered: + +```sh +cd ../incus/ovn-central +tofu output -json acceptance_commands | jq -r '.[]' +``` + +Require these observations: + +- Each database reports exactly one `pssl:` connection from + `get-connection`. +- The database sockets match the TLS endpoints in the address plan; no `ptcp:` + connection exists. +- The central and remote-listener services are active. + +From the trusted workstation, verify a real client connection with the +`nas01` identity. Copy the northbound endpoint from the address plan rather +than recording it in this runbook: + +```sh +export OVN_NB_ENDPOINT='ssl:' +ovn-nbctl --db="$OVN_NB_ENDPOINT" \ + --private-key="$OVN_TLS_DIR/nas01.key" \ + --certificate="$OVN_TLS_DIR/nas01.crt" \ + --ca-cert="$OVN_TLS_DIR/ca.crt" \ + show +``` + +The command must return northbound state without a trust prompt. Repeat with a +`tcp:` endpoint only as a negative, read-only probe and require refusal; do not +weaken the server configuration to make that probe connect. + +### 4. Converge Incus and all four chassis + +Run the read-only provider-parent preflight first, then the complete OVN +converge: + +```sh +cd /path/to/fleet/cluster +moon run fleet-cluster:ovn-preflight + +GLAB_SECRETS_DIR=/path/to/secrets \ +FLEET_OVN_TLS_DIR="$OVN_TLS_DIR" \ + moon run fleet-cluster:ovn +``` + +`fleet-cluster ovn` validates the complete certificate set before planning any +change. It sets the Incus client TLS trio and northbound connection before it +updates a chassis, then converges each chassis and the physical uplink. It +never disables a chassis, clears the northbound connection, changes IncusOS +system network configuration, or reboots a member. + +Run the same command again and require a no-op. Repeat the northbound TLS check +and the OpenTofu acceptance commands. Full sandbox lifecycle acceptance is a +separate agentcompute qualification; central and chassis checks do not prove +it. + +Remove the plaintext directory after the escrow revision and deployment +evidence are reviewed: + +```sh +rm -rf -- "$OVN_TLS_DIR" +unset OVN_TLS_DIR TF_VAR_ovn_ca_certificate_file TF_VAR_central_certificate_file +``` + +## Renew a chassis leaf + +Use "Materialize the escrowed set," including the CA key, to create a fresh +owner-only directory. Rotation must use the existing CA; the ceremony refuses +to mint a replacement CA during `--rotate`. + +Issue exactly the named chassis leaf: + +```sh +export OVN_TLS_DIR=/absolute/path/to/owner-only-directory +cd /path/to/fleet/cluster +uv run --locked python -m fleet_cluster ovn-ceremony \ + --dir "$OVN_TLS_DIR" \ + --leaf lab03 \ + --rotate +``` + +Replace `lab03` with the intended member. Compare the reported CA fingerprint +with the escrowed CA, verify that only the named leaf reports `reissued`, and +escrow the new certificate/key pair. Remove the plaintext CA key after escrow +and before deployment: + +```sh +rm -f -- "$OVN_TLS_DIR/ca.key" +``` + +Run the converge from fleet's `cluster/` directory: + +```sh +GLAB_SECRETS_DIR=/path/to/secrets \ +FLEET_OVN_TLS_DIR="$OVN_TLS_DIR" \ + moon run fleet-cluster:ovn +``` + +Only the named chassis should change. For `nas01`, the same leaf also updates +the Incus global OVN client identity. Require northbound connectivity, verify +all four chassis remain configured, and rerun the deploy to a no-op before +removing the plaintext directory. + +## Rotate the central leaf or complete OVN trust set + +TLS-only renewal does not replace `ovncentral01`. The owner approved both a +central-leaf renewal under the current CA and a complete replacement of the CA +and all five leaves. Both use `central-tls` while the central components are +stopped, preserve `/var/lib/ovn`, and leave the northbound and southbound +databases in place. + +### Prepare and escrow the new material + +For central-leaf renewal, use "Materialize the escrowed set," including the +current CA key, and rotate only the central leaf: + +```sh +export OVN_TLS_DIR=/absolute/path/to/owner-only-directory +cd /path/to/fleet/cluster +uv run --locked python -m fleet_cluster ovn-ceremony \ + --dir "$OVN_TLS_DIR" \ + --leaf central \ + --rotate +``` + +Compare the reported CA fingerprint with escrow, confirm only the central leaf +reports `reissued`, and escrow the new central certificate/key pair. + +For a complete CA replacement, use a fresh empty owner-only directory and run +the ceremony without `--rotate` or a leaf selector: + +```sh +export OVN_TLS_DIR=/absolute/path/to/new-owner-only-directory +install -d -m 0700 "$OVN_TLS_DIR" +cd /path/to/fleet/cluster +uv run --locked python -m fleet_cluster ovn-ceremony \ + --dir "$OVN_TLS_DIR" +``` + +Review and escrow the new CA, central pair, and four chassis pairs. Confirm all +six identities can be materialized from the reviewed secrets checkout. In +either procedure, remove the plaintext CA key after escrow and before +deployment: + +```sh +rm -f -- "$OVN_TLS_DIR/ca.key" +``` + +Keep the prior deployment set in a separate owner-only directory through +verification so the old client identity can be tested for rejection. The new +deployment directory must contain `ca.crt`, `ovncentral01.crt`, +`ovncentral01.key`, and each member's certificate/key pair. + +### Review and apply the rotation + +Before the outage, record the current NB_Global, SB_Global, and representative +logical-switch UUIDs. Validate the central delivery while the service is still +running: + +```sh +cd /path/to/fleet/cluster +uv run --locked python -m fleet_cluster central-tls \ + --dir "$OVN_TLS_DIR" \ + --dry-run +``` + +The dry run writes nothing. For a central-leaf renewal it should report the +central certificate and key as changed and the CA as unchanged. For a complete +CA replacement it should report all three central TLS files as changed and +note that applying them while components are active would be refused. + +Apply the bounded outage in this order: + +```sh +uv run --locked python -m fleet_cluster central --state stopped +uv run --locked python -m fleet_cluster central-tls \ + --dir "$OVN_TLS_DIR" +uv run --locked python -m fleet_cluster central --state running +GLAB_SECRETS_DIR=/path/to/secrets \ +FLEET_OVN_TLS_DIR="$OVN_TLS_DIR" \ + moon run fleet-cluster:ovn +``` + +Do not run the chassis/client converge while northbound is stopped: Incus must +reach northbound to update its global client TLS configuration. During a +complete CA replacement, starting central before that converge creates a +brief fail-closed trust mismatch. Do not add a dual-CA interval. The subsequent +fleet deploy updates the Incus client identity and all four chassis to the new +trust set. + +Repeat the central acceptance commands, the northbound TLS check, and the +chassis checks. Require the recorded NB_Global, SB_Global, and logical-switch +UUIDs to be unchanged. For a complete CA replacement, require a new chassis +identity to authenticate to both database endpoints and the matching old +identity to be rejected by both. Verify a cross-member guest path after the +rotation, including a guest restart, then rerun `central-tls --dry-run` and the +fleet deploy to no-ops. + +The live complete-set rotation replaced the central's three TLS files, the +Incus global client identity, and all four chassis identities. It preserved +NB_Global, SB_Global, and two logical-switch UUIDs. The new `lab03` identity +authenticated to both NB `6641` and SB `6642`; the old identity was rejected +by both before an authenticated response. Each of the two fixture paths +completed three of three pings. + +### Update OpenTofu public metadata + +Keep OpenTofu's public cloud-init metadata aligned with the active CA and +central certificate so a later deliberate VM replacement starts from the +current public trust set. The central private key remains outside OpenTofu: + +```sh +cd /path/to/fleet/incus/ovn-central +export AWS_PROFILE=lab-admin +export GLAB_AWS_STATE_BUCKET= +export INCUS_CONF= +export TF_VAR_ovn_ca_certificate_file="$OVN_TLS_DIR/ca.crt" +export TF_VAR_central_certificate_file="$OVN_TLS_DIR/ovncentral01.crt" +just plan +tofu show tfplan +just apply +``` + +Apply only an in-place public metadata update. Stop if the plan replaces the VM +or includes any private key. Remove the materialized deployment directories +after this update and all rotation checks pass. + +## Rebuild central + +Use deliberate VM replacement for bootstrap or package changes, not for TLS +renewal. Replacing the VM starts the standalone databases empty unless a +reviewed, compatible database backup is restored. Before replacement, decide +whether to restore such a backup through its established recovery procedure. +Without one, plan to recreate all disposable sandboxes through the normal +lifecycle. + +For a planned replacement, stop a reachable central through fleet: + +```sh +cd /path/to/fleet/cluster +uv run --locked python -m fleet_cluster central --state stopped +``` + +During recovery, attempt the same transition only when the VM answers through +Incus. If the VM is absent or cannot execute commands, record that failure and +continue with deliberate replacement; do not add another access path. + +Plan and apply the replacement using public TLS material only: + +```sh +cd /path/to/fleet/incus/ovn-central +export AWS_PROFILE=lab-admin +export GLAB_AWS_STATE_BUCKET= +export INCUS_CONF= +export TF_VAR_ovn_ca_certificate_file="$OVN_TLS_DIR/ca.crt" +export TF_VAR_central_certificate_file="$OVN_TLS_DIR/ovncentral01.crt" +tofu plan -replace=incus_instance.central -out=tfplan +tofu show tfplan +just apply +``` + +After apply, wait for the guest agent and cloud-init as in the initial +deployment. Restore a reviewed database backup, when available, before +starting central. Then deliver the complete active TLS set and converge in +order: + +```sh +cd ../../cluster +uv run --locked python -m fleet_cluster central-tls \ + --dir "$OVN_TLS_DIR" \ + --dry-run +uv run --locked python -m fleet_cluster central-tls \ + --dir "$OVN_TLS_DIR" +uv run --locked python -m fleet_cluster central --state running +GLAB_SECRETS_DIR=/path/to/secrets \ +FLEET_OVN_TLS_DIR="$OVN_TLS_DIR" \ + moon run fleet-cluster:ovn +``` + +Run all central acceptance commands and the northbound TLS check. If a reviewed +database backup was restored, validate it through its recovery procedure. +Otherwise, the rebuilt databases remain empty. Automatic reconstruction by +Incus has not been qualified. Treat every pre-rebuild sandbox network as +invalid and delete and recreate its disposable sandbox through the normal +lifecycle. Do not restore an unreviewed database copy. + +## Qualify a member reboot during a central outage + +This is a planned acceptance procedure, not routine outage repair. Create and +verify the persistent receipt directory **before** stopping central. On the +administration Mac, use: + +```sh +RECEIPT_DIR="$HOME/Library/Application Support/GilmanLab/fleet/reboot-receipts" +install -d -m 0700 "$RECEIPT_DIR" +test -d "$RECEIPT_DIR" && test -w "$RECEIPT_DIR" +test "$(stat -f '%Su' "$RECEIPT_DIR")" = "$USER" +test "$(stat -f '%Lp' "$RECEIPT_DIR")" = 700 +``` + +Choose a new stable request ID for each intended reboot. After recording +management and guest-path baselines, stop central and submit exactly one +confirmed reboot: + +```sh +cd /path/to/fleet/cluster +REQUEST_ID="ovn-central-outage-$(date -u +%Y%m%dT%H%M%SZ)" +uv run --locked python -m fleet_cluster central --state stopped +uv run --locked python -m fleet_cluster reboot \ + --member lab03 \ + --request-id "$REQUEST_ID" \ + --receipt-dir "$RECEIPT_DIR" \ + --confirm +``` + +The receipt proves that fleet accepted the request; it does not prove that the +member rebooted or returned healthy. Require the member management API to +return and the cluster to report the member `Online` and `Fully operational` +while central remains stopped. Measure a guest path that does not use the +rebooted member and one that does. Then start central exactly once: + +```sh +uv run --locked python -m fleet_cluster central --state running +``` + +Do not restart central, a chassis, OVS, or the uplink. Require both guest paths +to recover naturally, allow a later reaper scan to clear outage residue, and +rerun the fleet deploy to a no-op. Reusing the same request ID must not submit +another reboot. + +The Phase 5 qualification used request +`phase5-central-outage-lab03-01` and the persistent directory +`/Users/josh/Library/Application Support/GilmanLab/fleet/reboot-receipts`. +The first invocation was refused before an API call because that directory did +not exist. After it was created with mode `0700`, the same request ID was +accepted once. The management +API returned after approximately 112 seconds while central was still stopped. +A surviving path completed three of three pings throughout; the rebooted +guest completed zero of three during the outage and three of three after the +single central start. + +## Recover from control-plane unavailability + +Existing installed dataplane flows may continue while central is down. Treat +all OVN creates, updates, and deletes as unavailable regardless of that +traffic. Diagnose with read-only commands first: + +```sh +cd /path/to/fleet/incus/ovn-central +tofu output -json acceptance_commands | jq -r '.[]' +incus exec --project default nas01:ovncentral01 -- \ + journalctl --no-pager -u ovn-ovsdb-server-nb -u ovn-ovsdb-server-sb \ + -u ovn-northd -u ovn-central-remotes +``` + +If the VM and complete TLS trio exist, converge the running state: + +```sh +cd ../../cluster +uv run --locked python -m fleet_cluster central --state running +``` + +Do not use a restart, disable a chassis, clear +`network.ovn.northbound_connection`, or recreate the uplink. If the lifecycle +command reports missing or mismatched TLS material, leave the components +stopped and use `central-tls --dry-run` followed by `central-tls` with the +complete active set. Start central only after that delivery succeeds. Use +deliberate VM replacement only when bootstrap or package state is inconsistent; +missing TLS files alone do not require replacement. + +An outage-created `Errored` NAT-enabled network retains its external address. +An isolated `nat=false` network has no external address to retain. Explicit +`sandbox.delete` first records expiry; if cleanup fails, the project and +remaining resources stay discoverable. The reaper retries in dependency order +on later scans. After central recovers, allow a later scan to delete forwards +and peers, NIC references, instances and snapshots, sandbox images, profiles, +networks and their ACLs, then the project. Do not restart infrastructure to +force cleanup, and do not report capacity as released until the network is +gone. + +A provider-parent conflict is a different failure. Fleet preflight names the +competing default-project network, profile NIC, or instance NIC and stops +before changing the uplink. Identify the owner and remove or migrate that +consumer through its owning workflow. The reaper owns sandbox resources only; +it cannot repair an infrastructure parent conflict. + +## Escalation + +Stop and investigate rather than broadening the procedure when: + +- The CA key is missing, compromised, expired, or does not match `ca.crt`. A + new CA is a coordinated trust-domain replacement for central, Incus, and all + four chassis; leaf rotation is not sufficient. +- A central plan changes resources other than the intended VM, or a chassis + converge proposes a reboot, a northbound clear, a chassis disable, or an + unrelated network change. +- Mutual TLS fails after certificate fingerprints and SANs match the reviewed + escrow. Do not add plaintext listeners as a diagnostic bypass. +- Owned sandbox resources remain after central is healthy and a later reaper + scan. Capture the named resource and backend error before any manual delete. +- Recovery would require the retired `sandbox01` central or an unreviewed OVN + database backup. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index f48bcd6..221899f 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -37,8 +37,10 @@ nav: - "ADR-0003: Use AWS KMS with PGP Recovery for Secrets": decisions/0003-use-kms-with-pgp-recovery-for-secrets.md - "ADR-0004: Manage RouterOS Devices with OpenTofu": decisions/0004-manage-routeros-devices-with-opentofu.md - "ADR-0005: Anchor Internal PKI on the KMS Root with Sibling Intermediates": decisions/0005-anchor-internal-pki-on-kms-root-with-sibling-intermediates.md + - "ADR-0006: OVN as the Sandbox Network Fabric": decisions/0006-ovn-as-the-sandbox-network-fabric.md - Designs: - Lab v2 core network: designs/lab-v2-core-network.md + - "Draft: agentcompute": designs/drafts/agentcompute.md - Reference: - Device naming: reference/naming.md - Hardware inventory: reference/hardware-inventory.md @@ -50,6 +52,7 @@ nav: - Runbooks: - Change the Tailscale policy: runbooks/tailscale-policy-change.md - Commission a lab compute node: runbooks/commission-lab-node.md + - Operate OVN central and certificates: runbooks/ovn-central-and-certificates.md - Rebuild nas01: runbooks/rebuild-nas01.md - Manage the sw-core01 configuration: runbooks/sw-core01-configuration.md - Manage the sw-mgmt01 configuration: runbooks/sw-mgmt01-configuration.md From 207a0e910189f306e86bc08a71d3b5e0644b5948 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Sun, 13 Sep 2026 20:34:10 -0700 Subject: [PATCH 2/3] docs(ovn): distinguish renewal from compromised-key containment --- docs/docs/runbooks/ovn-central-and-certificates.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/docs/runbooks/ovn-central-and-certificates.md b/docs/docs/runbooks/ovn-central-and-certificates.md index bb6678b..1ab1bf6 100644 --- a/docs/docs/runbooks/ovn-central-and-certificates.md +++ b/docs/docs/runbooks/ovn-central-and-certificates.md @@ -300,6 +300,11 @@ unset OVN_TLS_DIR TF_VAR_ovn_ca_certificate_file TF_VAR_central_certificate_file ## Renew a chassis leaf +Renewing a leaf under the same CA does not revoke its old certificate. This +deployment does not configure per-leaf revocation. If a leaf private key is +exposed, use the complete CA and all-five-leaf replacement below; a leaf-only +renewal is not containment. + Use "Materialize the escrowed set," including the CA key, to create a fresh owner-only directory. Rotation must use the existing CA; the ceremony refuses to mint a replacement CA during `--rotate`. From 6c1987507dd03b850cc10d29250b74cc32149f5c Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Sun, 13 Sep 2026 20:44:34 -0700 Subject: [PATCH 3/3] docs(ovn): align resolved capacity and record MTU handoff --- docs/docs/designs/drafts/agentcompute.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/docs/designs/drafts/agentcompute.md b/docs/docs/designs/drafts/agentcompute.md index 33c7c11..e31c1db 100644 --- a/docs/docs/designs/drafts/agentcompute.md +++ b/docs/docs/designs/drafts/agentcompute.md @@ -506,6 +506,11 @@ project limits (indicative: 8 vCPU, 16 GiB, 100 GiB per sandbox) and by the TTL reaper. Geneve tunnels ride the VLAN 30 storage links; sandbox east-west traffic is small. +Phase 3 measured a 1442-byte OVN guest MTU over the 1500-byte underlay. +The operator's Tailscale path used a 1280-byte tunnel MTU. Desktop and file +transfers must not assume a 1500-byte end-to-end path; see the +[Phase 3 measurements](https://github.com/GilmanLab/agentcompute/blob/master/spikes/ovn/README.md). + The approved OVN range contains 64 external addresses. Only NAT-enabled networks and distinct forward listen addresses consume them. The representative `default` NAT, isolated `lan`, `wan` NAT, and one-forward @@ -689,7 +694,8 @@ Resolved in review, kept here until the draft is promoted: The [address plan](../../reference/networking/address-plan.md#ovn-external-addresses) records the approved 64-address allocation and eight-sandbox planning target. Default-plus-forward consumes two external addresses per sandbox; - adding `lan` and `wan` brings that budget to four. Count `Errored` + adding an isolated `lan` and NAT-enabled `wan` brings that budget to three. + Count `Errored` NAT-enabled networks until deletion. A dedicated VLAN is reconsidered only if OVN needs more than this allocation; DHCP, named endpoints, and routes stay unchanged.