From 0946b91ee7403573c8ae3efbf44702dbd5014318 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Sat, 12 Sep 2026 15:08:49 -0700 Subject: [PATCH] docs(runners): document private image runner deployment --- .../docs/reference/networking/address-plan.md | 20 ++ docs/docs/runbooks/private-image-runners.md | 247 ++++++++++++++++++ docs/mkdocs.yml | 1 + 3 files changed, 268 insertions(+) create mode 100644 docs/docs/runbooks/private-image-runners.md diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index 23d87ed..b2b1be8 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -108,6 +108,26 @@ Phase 5 replaces it. Chassis encapsulation uses the members' VLAN 30 addresses. See the [spike report](https://github.com/GilmanLab/agentcompute/blob/spike/ovn-mechanism/spikes/ovn/README.md) for measurements and the northbound/chassis sequencing constraint. +### Incus-local image runner networks + +These NAT bridges are local to each Incus member, not routed VLAN prefixes. +Do not advertise them through `gw01` or Tailscale. + +| Resource | Address or prefix | Ownership | +| --- | --- | --- | +| Existing `incusbr0` bridge | `10.158.84.0/24`, gateway `.1` | IncusOS | +| `github-runners` bridge | `10.158.85.0/24`, gateway `.1` | Fleet `cluster/` | +| Reserved controller VM `ghrunner01` on `nas01` | `10.158.84.50` | Fleet OpenTofu root `incus/incus-gh-runner/` | +| Reserved HTTP CONNECT forward | `10.10.10.14:3128` → `10.158.84.50:3128` | Same OpenTofu root, member-local to `nas01` | + +Runner egress permits the management gateway's DNS service, the pinned Incus +API at `10.10.10.14:8443`, and the proxy port. The ACL also permits the exact +proxy destination after DNAT because network-forward translation precedes +the network ACL on the member hosting the forward. No other controller +port is exposed by that forward. See the +[private image runner runbook](../../runbooks/private-image-runners.md) +for deployment and cutover checks. + ## Gateway interface mapping | `gw01` chassis port | VyOS interface | Mode | Network | diff --git a/docs/docs/runbooks/private-image-runners.md b/docs/docs/runbooks/private-image-runners.md new file mode 100644 index 0000000..6b9ca43 --- /dev/null +++ b/docs/docs/runbooks/private-image-runners.md @@ -0,0 +1,247 @@ +--- +title: Deploy private image runners +description: Deploy and qualify the private image publisher and its Incus controller. +--- + +# Deploy private image runners + +Deploy the image publisher for the private `GilmanLab/agentcompute-images` +repository. Recipes, immutable image references, and the catalog remain in +[`GilmanLab/agentcompute`](https://github.com/GilmanLab/agentcompute/tree/master/images). +Do not register a self-hosted runner with that public repository. + +## Preconditions + +- Administrative access to the Incus cluster, GitHub organization and packages, + and the lab S3 state bucket. +- An authenticated `lab-admin` AWS profile, plus the repository-pinned tools in + a trusted, interactive operator shell. Fleet deployment tasks do not run in CI. +- OpenTofu `>= 1.10`, `just`, and the public `images:` Incus remote used to fetch + the controller VM image. +- Point `INCUS_CONF` at the existing administrator CLI configuration, including + its client identity and pinned server certificate. On macOS this is usually + `$HOME/Library/Application Support/incus`; the provider otherwise defaults to + `$HOME/.config/incus`. Do not enable automatic certificate acceptance. +- Reviewed companion changes in `agentcompute`, `agentcompute-images`, `fleet`, + and the secrets repository. Keep the private repository out of `init.sh`. +- The guest lifecycle smoke passes for both runner image variants before any + controller is started. +- Escrow both App private keys and both Incus client keys under the appropriate + `services/incus-gh-runner/` and `services/agentcompute-images/` scopes. Follow + [ADR-0003](../decisions/0003-use-kms-with-pgp-recovery-for-secrets.md); use KMS and + the recovery PGP recipient, not a local-only encrypted copy. + +## Safety boundary + +The public publisher runs on a GitHub-hosted machine, validates its definitions, +then dispatches a full commit SHA. The private bake checks public-master ancestry +before running source code. The root-owned build wrapper repeats that check in +its own checkout. Neither gate accepts a pull-request branch merely because its +workflow run received approval. + +The `agentcompute-publisher` scale set uses the `default` runner group and is +repository-scoped to `agentcompute-images`. The general `runner` image is not +registered with a public pool. All outside contributors require workflow approval +in the public repository, and every action requires a full commit SHA. These +settings supplement repository-scoped runner registration; they do not replace it. + +Source maintainers, private workflow maintainers, and the dispatch/catalog App +remain trusted publishers. The publisher VM has an explicit build-wrapper sudo +grant; the general runner has no such grant. Neither image permits nested +virtualization. The controller Incus identity can use only `github-runners`; +CI can use `github-runners` and `image-build`. Neither can read default-project +instances or mutate default-project resources. Jobs reach external services only +through the proxy's hostname allowlist; they also have explicit DNS and Incus API +access. + +The private bake does **not** mint the GitHub artifact attestations produced by +the previous public image workflow. Digest verification and boot qualification +are not a substitute for that provenance record. Do not claim that new private +bakes carry the old attestation. + +## Prepare GitHub identities + +1. Create a dedicated controller App with repository administration-write and + metadata-read permissions. Install it only on `agentcompute-images`. +2. Create a separate dispatch/catalog App with contents-write and + pull-requests-write permissions. Install it on the public source repository + and the private bake repository only. The public workflow requests a + contents-write token for the private repository; the private catalog step + requests contents-write and pull-requests-write for the public repository. +3. Set `IMAGES_APP_CLIENT_ID` and the escrowed `IMAGES_APP_PRIVATE_KEY` in both + repositories. Never reuse the controller App key as an Actions secret. +4. Grant the private repository Actions write access to the existing GHCR + `agentcompute/router` package. After the first publication creates + `agentcompute/runner` and `agentcompute/runner-publisher`, verify that access + on both new packages too. Keep all three packages public: the catalog + reconciler pulls without registry credentials. Linking a package to the + public source repository does not make its visibility public or grant the + private workflow access. +5. Remove `IMAGES_RUNNER` from the public repository. Confirm its runner inventory + is empty and its publisher triggers only on protected-master pushes or manual + dispatches. Keep the public `image-publish` environment restricted to protected + branches. + +## Qualify the images + +From the source repository on a Linux amd64 build host, validate the pins and +build each runner variant with `images/build.py build --image runner` or +`--image runner-publisher`. Each build requires fresh work and output directories. +The recipe uses a checksummed Ubuntu base and CA bootstrap package, a fixed +Ubuntu snapshot, a checksummed Actions Runner, and the unmodified v2.0.0 guest +files. + +Run `images/runner/smoke.py` against a disposable, non-default Incus project with +a `runner-smoke` profile. Supply its split `incus.tar.xz` and `disk.qcow2` files, +remote, project, profile, and evidence directory. The smoke uses a temporary +Linux status observer to capture brief file transitions without changing the +shipped guest implementation. Require: + +- An active guest path unit and a grown root filesystem. +- Actions Runner executes as `actions-runner`, and the guest obtains a DHCP + default route. A root-owned Incus agent responding is not enough. +- Status-file transitions `starting` → `running` → `exited` after a bogus JIT + payload, and deletion of both payload files. +- Corresponding serial-console lifecycle lines with no JIT payload disclosure. +- Guest-initiated poweroff and deletion of the test VM and any image the smoke + imported. An existing image is not owned by the smoke and is not deleted. + +Capture the serial buffer during the guest's 30-second diagnostics grace period, +not after poweroff: the tested cluster returns an empty post-poweroff VM console. +The smoke stages the zero-byte readiness marker and renames it atomically; +directly pushing the watched filename can race the guest's immediate deletion. + +## Converge and deploy + +1. Run `moon run fleet-cluster:image-build` with the existing public CI certificate + in `FLEET_IMAGE_BUILD_CERT_FILE`. This creates the project before the CI trust + references it, enables bounded VM qualification, and supplies `runner-smoke` + without changing the router's default profile. +2. Export the two **public** Incus client certificate paths as + `FLEET_GITHUB_RUNNERS_CONTROLLER_CERT_FILE` and + `FLEET_GITHUB_RUNNERS_CI_CERT_FILE`. Run + `moon run fleet-cluster:github-runners` from the fleet repository. The deploy + owns the restricted project, bridge, ACL, publisher profile, and both trusts. +3. Bootstrap the released v2.0.0 controller once on `sandbox01`, using HTTPS, the + pinned server certificate, and the restricted controller identity. Require a + GitHub session and record the unmodified validator's rejection. Keep + `capacity.min_runners=0` for this spike until the managed proxy is available. +4. Stop the manual controller. Do not run two controllers with the same owner and + scale set during the move. +5. In fleet's `incus/incus-gh-runner/`, configure the real App identifiers, the + qualified publisher image, and the public controller certificate path. Check + address and port conflicts against the + [address plan](../reference/networking/address-plan.md). +6. Authenticate AWS, export `AWS_PROFILE=lab-admin` and + `GLAB_AWS_STATE_BUCKET` with the existing lab bucket name, then run + `just check`, `just init`, and `just plan`. Review the saved plan before + `just apply`. +7. Wait for cloud-init to complete. Deliver the two private keys using the + `credential_delivery_commands` OpenTofu output. The packaged systemd credential + drop-ins load them; private keys must not enter cloud-init, configuration YAML, + or OpenTofu state. +8. Start the controller and require its GitHub session and standby runner logs. + Configure the private workflow's restricted CI certificate, key, and server + pin as `INCUS_CLIENT_CERT`, `INCUS_CLIENT_KEY`, and `INCUS_SERVER_CERT`. + +### Controller inputs and credentials + +Set these real, non-secret values in the root's `terraform.tfvars`: + +| Variable | Value | +| --- | --- | +| `github_app_client_id` | Controller App client ID | +| `github_app_installation_id` | Its private-repository installation ID | +| `runner_image` | Qualified publisher alias or fingerprint imported into `github-runners` | +| `controller_client_certificate_file` | Local path to its public client certificate | + +The server pin defaults to fleet's committed `cluster/tls/incus-cluster.crt`. +Verify that fingerprint against the cluster before first use. The provider's +`incus_address` must be the address of the member hosting the controller, not a +round-robin alias: bridge network forwards belong to the member serving the API +request. Keep `controller_address` equal to fleet's +`GITHUB_RUNNERS_PROXY_TARGET_ADDRESS` so same-member post-DNAT traffic matches +the runner ACL. + +Bootstrap downloads the official v2.0.0 amd64 DEB, verifies its pinned SHA-256, +installs that exact version, and holds it against unattended upgrades. The +package's GitHub attestation was verified against `meigma/incus-gh-runner`. +The upstream apt example is not usable: its key URL returns 404 and the live +repository does not contain this package. Update `controller_package_version` +and `controller_package_sha256` together after verifying a replacement release. +The unit is enabled but skipped until both credential files exist. +After cloud-init, deliver the App key to +`/etc/incus-gh-runner/github-app-private-key.pem` and the Incus key to +`/etc/incus-gh-runner/client.key`, both `root:root` mode `0600`. systemd's +`LoadCredential=` supplies them to the dynamic service user. Rotation requires +replacing the file and restarting the controller. + +The proxy's `proxy_allowed_hosts` permits GitHub/Actions, GHCR, and the specific +Ubuntu, Alpine, Go, and Python download hosts needed by the pinned recipes. +Its source allowlist accounts for both local runner addresses and other members' +NAT addresses. Changes require controller-VM replacement, not an in-place edit +that the next OpenTofu apply cannot reproduce. + +## Verify the cutover + +A protected-master image change must produce the full chain: hosted validation, +private dispatch, a lab VM running the bake, qualified GHCR releases, a public +catalog digest PR from the dedicated App, and deletion of the completed job VM. +The public dispatch job succeeding is not evidence that the private bake succeeded. + +An unchanged definition tree skips assembly and publication. It still fetches +and qualifies existing digests; authentication and registry failures are not +cache misses. Catalog and README changes do not change the definition identity. + +Verify a non-master SHA refuses before assembly, an incorrect server pin fails +TLS, both restricted identities return `[]` for default-project instance reads, +and default-project mutations are denied. Record runner startup latency and +per-image wall time; compare router work against the earlier approximately +3.5-minute hosted router workflow, not against a three-image aggregate. + +After success, retire the old public image-build CI trust and unused public Incus +secrets. Repeat both fleet dry runs and require no changes, then require a clean +OpenTofu plan. Keep the bake's metrics, lifecycle transcripts, and release JSON. + +## Recovery and remaining validation gap + +The released v2.0.0 runtime supports clustered HTTPS connections, but its drift +validator does not support this cluster baseline. Track +[upstream issue 68](https://github.com/meigma/incus-gh-runner/issues/68) and +[PR 69](https://github.com/meigma/incus-gh-runner/pull/69). The contribution adds +cluster-member topology checks; the lab's wildcard core listener, shared storage, +and IncusOS IPv6 filtering constraints still require explicit qualification. +Do not install a local validator patch or weaken a check to obtain a passing +result. Until a compatible upstream release is qualified, fleet convergence and +the negative checks above enforce the reviewed baseline. + +The lab-specific controls differ from the single-host baseline: + +- Each cluster member has its own NAT bridge; the runner project does not own + those host networks. +- Cluster members also host unrelated workloads. A VM escape would reach a shared + compute host, not the upstream baseline's dedicated, single-purpose host. + Project restrictions and network controls do not eliminate that residual risk. +- Runner storage uses a bounded slice of the shared `data` pool rather than a + dedicated pool. +- IncusOS does not load `br_netfilter`, so enabling `security.ipv6_filtering` + prevents VM startup. The bridge has no configured IPv6 service. The NIC ACL + denies IPv6 traffic, including link-local and manually assigned addresses, + and port isolation prevents same-bridge peers from communicating. Qualify + this with live peer listeners and explicit neighbors; failed ARP or NDP + alone is not evidence of packet filtering. +- Incus 7.4 supports project-level VM nesting restrictions. The project blocks + nesting in addition to the profile's `security.nesting=false`. + +The unmodified v2.0.0 validator returned +`clustered Incus is outside this dedicated-host baseline`. The upstream +cluster-member contribution progressed to `core.https_address drift detected`: +the lab listens on `:8443`, whereas that baseline requires an exact member +address. These are failed validation results, not waived checks. + +To stop scheduling, stop the controller service first and inspect active jobs +before removing VMs. Do not move publication back onto public self-hosted runners. +For controller configuration changes, drain jobs and explicitly replace the VM: +cloud-init does not rewrite an existing guest. Restore escrowed keys afterward. +Stop the rollout if credential escrow, package permissions, lifecycle qualification, +restricted access, or the private runner boundary cannot be verified. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index d31cbf8..f48bcd6 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -54,6 +54,7 @@ nav: - Manage the sw-core01 configuration: runbooks/sw-core01-configuration.md - Manage the sw-mgmt01 configuration: runbooks/sw-mgmt01-configuration.md - Deploy the VyOS gateway configuration: runbooks/vyos-gateway-deployment.md + - Deploy private image runners: runbooks/private-image-runners.md plugins: - search