diff --git a/README.md b/README.md index 061ab603..8d26e2b7 100644 --- a/README.md +++ b/README.md @@ -136,7 +136,7 @@ documents for different readers. at all. Every pinned digest is also scanned weekly for fixable HIGH and CRITICAL CVEs, and each image with findings has an open issue until a scan finds it clean. -- **Documented decisions and runbooks.** 116 ADRs covering what was chosen +- **Documented decisions and runbooks.** 117 ADRs covering what was chosen and what was rejected — including the costs accepted knowingly; 54 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. Every critical alert links to @@ -236,7 +236,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md changelog.md │ ├── diagrams/ # the network diagram (SVG) and its predecessors -│ ├── adr/ # 116 ADRs — architecture decision records +│ ├── adr/ # 117 ADRs — architecture decision records │ ├── exercises/ # purple-team exercise records — see its README │ └── runbooks/ # 54 runbooks; successor-handover.md is the front door └── Makefile # make help diff --git a/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md index faf80a0e..f1086e04 100644 --- a/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md +++ b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md @@ -35,6 +35,12 @@ and amends the first consequence of > one host instead of the segment. Nothing here is amended, per ADR-0001; > [`build-the-playground.md`](../runbooks/build-the-playground.md) §4 carries > the procedure and the finding. +> +> [!NOTE] +> Amended by [ADR-0117](0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): +> `phoenix`'s own `sshd` is key-only, and its inbound policy is default-deny +> with `ufw`. It admits `22` from Hicks and the tunnel's peers only. The text +> here is left as written, per ADR-0001 and ADR-0115. ## Context diff --git a/docs/adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md b/docs/adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md new file mode 100644 index 00000000..fdfdb3ee --- /dev/null +++ b/docs/adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md @@ -0,0 +1,130 @@ +# ADR-0117: Give `phoenix` a key-only `sshd` and a default-deny inbound policy + +**Status:** Accepted · 2026-11 · amends [ADR-0043](0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) + +## Context + +[ADR-0043](0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) +made `phoenix` the one machine whose purpose is to hold credentials for other +machines. It holds the Proxmox tokens for `Saruman` and `gandalf`, the +domain's passwords, `tofu/`'s state passphrase, and the only key that +ADR-0029's six accept for administration +([ADR-0077](0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). +[`security.md`](../security.md) accepts the residual: whoever owns `phoenix` +owns the lab domain. + +That acceptance assumes the way into `phoenix` is narrow, but nothing in the +repository says what it is. The runbook that built it, +[`build-the-jumpbox.md`](../runbooks/build-the-jumpbox.md), installs OpenSSH +and says nothing more. The six, which `phoenix` configures, are held to more +than that: `sshd` refuses passwords, and one rule admits `10.0.30.70` alone. +The audit of the host on 2026-11-13 +([#1496](https://github.com/Gerrrt/HomeLab/issues/1496)) found what that gap +leaves in place: + +- the Ubuntu installer's `PasswordAuthentication yes`, in + `sshd_config.d/50-cloud-init.conf`; +- no host firewall; +- `sshd` answering every address that can route to it. + +Those addresses are all of VLAN 30, all of Hicks +([`firewall-claims.yaml`](../firewall-claims.yaml): nothing blocks Hicks from +ImaginationLAN), and the tunnel's peers. A password guessed or reused from any +of those owns the lab domain. The key-only rule on the six then protects +nothing, because the key is on the machine the password opens. + +Three things must still reach `phoenix`: + +- **The operator,** from a Hicks workstation or through the tunnel. The + Mac's `deploy-agent.sh` run and the CA copy in + [`build-the-lab-templates.md`](../runbooks/build-the-lab-templates.md) both + come from Hicks. +- **WireGuard's handshake,** from the internet through `morpheus`'s one + `rdr` ([ADR-0042](0042-terminate-the-remote-path-on-the-lab-and-route-it.md)). +- **A Debian or Kali installer under construction,** fetching its preseed from + Packer's HTTP server on `8800`, for the length of that build + ([ADR-0090](0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)). + +Nothing on VLAN 30 needs to log in to `phoenix`. Its work all flows outward: +API calls to `8006`, SSH and WinRM to the guests it builds and configures, +and Alloy's push to `alexander`. + +## Decision + +**`sshd` on `phoenix` is key-only.** A drop-in, +`/etc/ssh/sshd_config.d/10-phoenix.conf`, sorts ahead of the installer's +`50-cloud-init.conf`. `sshd` keeps the first value it reads, so the drop-in +wins without the installer's file being edited. It sets: + +- `PasswordAuthentication no` and `KbdInteractiveAuthentication no`; +- `PermitRootLogin no`; +- `X11Forwarding no`. + +Every workstation that logs in has its key in `authorized_keys` before the +drop-in is applied. That ordering is the whole risk of the change, so the +runbook makes it a check rather than a reminder. + +**Inbound is default-deny, with `ufw`.** `ufw` because +[`build-the-lab-templates.md`](../runbooks/build-the-lab-templates.md) +already opens `8800` with it. One tool, not two. The rules: + +- `22/tcp` from Hicks, `10.0.50.0/24`; +- `22/tcp` arriving on `wg0` from the peers, `172.31.0.0/24`; +- the WireGuard listen port, `udp`, from anywhere. `morpheus` decides who + reaches it, and the port number stays out of this repository; +- `8800/tcp` from `10.0.30.0/24`, added for a Debian or Kali build and + deleted after it, as that runbook already says; +- outbound allowed, as `ufw` defaults. + +**Routing from the tunnel is left as ADR-0042 made it.** `ufw`'s forward +policy is `DROP`, so one route rule, `wg0` to `ens18`, carries the peers' +traffic as before, with replies admitted as established. It does not narrow +the destinations. ADR-0042 makes `morpheus` the control and its blocks the +attribution. A second filter on `phoenix` would stop that traffic before +`morpheus` could log it, and the remote-path runbook's negative tests would +then prove the wrong firewall. `ufw` does not touch `net.ipv4.ip_forward`, so +the systemd drop-in still scopes forwarding to the tunnel's lifetime. + +**Nothing converges `phoenix`, so this is applied by hand** from +[`build-the-jumpbox.md`](../runbooks/build-the-jumpbox.md) §6a, and a rebuild +applies it in the same place. The runbook ends with the checks that prove +each rule from the side it applies to. + +This amends ADR-0043, which placed `phoenix` and named what it holds but not +how it is reached. + +## Alternatives considered + +- **`nftables` directly,** with a ruleset file in the repository. It is more + exact, and [ADR-0093](0093-run-openvas-on-garuda-on-demand-scoped-to-the-domain-by-nftables.md) + uses it on `garuda`. Declined because the template runbook's `ufw` step + would then be a second tool on the same host. The policy here is a short + allow-list, which `ufw` states in five lines. +- **Admit `22` from VLAN 30 as well.** It is the segment `phoenix` is on, and + the rule would be shorter. Declined: nothing on it needs to log in. VLAN 30 + includes the lab domain and the guests `phoenix` builds, which are the + machines least trusted to reach the one that owns them. +- **Narrow the routed traffic on `phoenix` to `10.0.30.0/24`.** Declined + for the reason above: it moves ADR-0042's control and its evidence off + `morpheus`. +- **Leave passwords on and rely on the firewall.** Declined. Hicks is a + whole segment of household machines, and the firewall admits all of it. + +## Consequences + +- **A lost key now locks a workstation out,** where before it fell back to a + password. The recovery path is `phoenix`'s console in `Saruman`'s web UI, + reached from Hicks, where a local login still takes a password. +- **A rebuilt `phoenix` is closed before its first login from anywhere but + Hicks and the tunnel.** `deploy-agent.sh` from the Mac is unaffected. +- **Peers keep their access to `phoenix`'s `22`,** as before. A peer is a + device with a key and no second factor (`security.md`), and `sshd` asks for + a second key. +- **`8800` is a step again, not a no-op.** The template runbook's "if `ufw` is + active" becomes unconditional. +- **Reopened by:** + - anything on VLAN 30 that needs to reach `phoenix` inbound; + - `phoenix` gaining a converger, which would carry this config instead of + the runbook; + - Hicks being narrowed by `firewall-claims.yaml`, which would let the + `22` rule name workstations instead of the segment. diff --git a/docs/architecture.md b/docs/architecture.md index 3846f510..dddf4509 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -204,7 +204,7 @@ hole from the monitoring VLAN into the monitored one. | `dot-gentoo` (10.0.30.96) | 🟢 30 | *(none — a test machine)* | Built 2026-10-09. Gentoo Linux on the 23.0 systemd profile, cloned from `tpl-gentoo` (908), which Packer made from Gentoo's own signed cloud image, for the dotfiles' Gentoo layer (whose default is OpenRC). Secure Boot is off. A guest on `Saruman`, VMID 196, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy | | `dot-nixos` (10.0.30.97) | 🟢 30 | *(none — a test machine)* | Built 2026-10-09. NixOS 26.05, cloned from `tpl-nixos` (909), which Packer installed from the minimal ISO with a committed `configuration.nix`, for the dotfiles' NixOS layer. Secure Boot is off. A guest on `Saruman`, VMID 197, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy | | `dot-windows` (10.0.30.98) | 🟢 30 | *(none — Windows)* | Built 2026-10-08. Windows 11 Pro, cloned from `tpl-win11-pro` (911) and left unactivated, for the dotfiles' Windows layer. A guest on `Saruman`, VMID 198, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy | -| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and runs no continuous convergence (what it configures, it configures on demand, below); it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). The first toolchain on it is [`packer/`](../packer/README.md), the lab's VM templates at VMIDs 901–912, with 903–909 for the dotfiles OS layers ([ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)) ([ADR-0074](adr/0074-build-the-lab-templates-with-packer-from-phoenix.md), [`build-the-lab-templates.md`](runbooks/build-the-lab-templates.md)). The second is [`ansible/`](../ansible/README.md), which configures ADR-0029's six over SSH and touches nothing else ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md), [#448](https://github.com/Gerrrt/HomeLab/issues/448)). Alloy agent (native package, `scripts/deploy-agent.sh`) | +| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and runs no continuous convergence (what it configures, it configures on demand, below); it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). The first toolchain on it is [`packer/`](../packer/README.md), the lab's VM templates at VMIDs 901–912, with 903–909 for the dotfiles OS layers ([ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)) ([ADR-0074](adr/0074-build-the-lab-templates-with-packer-from-phoenix.md), [`build-the-lab-templates.md`](runbooks/build-the-lab-templates.md)). The second is [`ansible/`](../ansible/README.md), which configures ADR-0029's six over SSH and touches nothing else ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md), [#448](https://github.com/Gerrrt/HomeLab/issues/448)). Its own way in is decided by [ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): `sshd` takes keys only, and `ufw` admits `22` from Hicks and the tunnel's peers, the tunnel's port, and `8800` from VLAN 30 for the length of a Debian or Kali template build. Applying it on the host is [#1496](https://github.com/Gerrrt/HomeLab/issues/1496). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `golem` (10.0.30.80) | 🟢 30 | *(none — an appliance, no Docker)* | Proxmox Backup Server 4.2, built 2026-10-03 by [`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md) ([#485](https://github.com/Gerrrt/HomeLab/issues/485)): the lab's backups, placed by [ADR-0053](adr/0053-run-pbs-on-saruman-with-its-datastore-on-smaug-over-nfs.md). A guest on `Saruman`, VMID 180, whose only datastore, `erebor`, is `smaug`'s `erebor/pbs` over NFSv4 through one `2049` pass, with `atime` on because garbage collection depends on it, and the mountpoint made immutable so an unmounted share cannot fill the OS disk. Every backup is encrypted on `Saruman` before it leaves, with a key made on `Saruman` and copied, on 2026-10-03, into `secrets/lab.sops.yaml` and onto paper (the runbook's §7), so a lost `Saruman` does not take the backups' readability with it; `Saruman` writes through a token that can back up and cannot prune. PBS prunes (7 daily, 4 weekly, 3 monthly), collects garbage on Saturdays and verifies on Sundays; TrueNAS keeps fourteen daily snapshots of the dataset as the copy PBS cannot prune. The nightly job takes `odin`, `garuda` and the domain's six, the guests tagged `backup`; `alexander`, `phoenix`, `fenrir` and `golem` itself are rebuilt from this repository and stay out, and so does `eden`, whose graph is one collection run from rebuilt (the runbook's §8). `scripts/collect-pbs-task-state.py` publishes every job's last outcome and the snapshots' verify state hourly, and seven rules in `stacks/lab` read them; the first nightly run, 2026-10-03, was verified the next morning. Its Alloy pushes to `alexander`, never to VLAN 99, since 2026-10-03. Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | [`stacks/wiki`](../stacks/wiki) | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)), hand-run containers until [#251](https://github.com/Gerrrt/HomeLab/issues/251) wrote them down as a stack deployed by hand from `main`, like `smaug`'s. **Cut over 2026-10-07** ([#847](https://github.com/Gerrrt/HomeLab/issues/847), [ADR-0082](adr/0082-serve-the-wiki-over-tls-and-deploy-it-from-a-verified-checkout.md)): deployed by hand from a checkout whose signature is verified first, and served over https from a Caddy on a leaf from the estate's CA, with port 80 only redirecting ([`stacks/wiki/README.md`](../stacks/wiki/README.md#tls)); its database is pulled to `prometheus` nightly as a `pg_dump` (`make backup-wiki`, [ADR-0065](adr/0065-pull-the-wikis-database-to-prometheus-as-a-dump.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)), and of `trinity`'s nightly sensitive-tier volume sets and Immich's library ([ADR-0064](adr/0064-copy-immichs-library-to-oracle-until-the-off-estate-copy-exists.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Holding real data since 2026-09-28, Immich's first 615 photographs, which passed the #404 step 10 gate while most of it was open. The library is copied nightly to `oracle`, off-host and not off-estate, until [#455](https://github.com/Gerrrt/HomeLab/issues/455) ([ADR-0064](adr/0064-copy-immichs-library-to-oracle-until-the-off-estate-copy-exists.md)). The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) run beside them, with the household's page of links to them at `home.matrix.elysium` and its live numbers on read-only tokens ([#137](https://github.com/Gerrrt/HomeLab/issues/137)) — Homepage until [ADR-0103](adr/0103-retire-homepage-and-serve-the-household-page-from-glance.md) made it a page of Glance's. ntfy ([#136](https://github.com/Gerrrt/HomeLab/issues/136)) is the in-house receiver for Alertmanager's three real channels, **deployed and cut over 2026-09-29** by [`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md#cutting-over-to-the-in-house-ntfy), with `urgent` and `security` also sent to ntfy.sh for a phone away from home. Miniflux, a feed reader with a Postgres of its own, is the tier's first service beyond ADR-0008's nine, **deployed 2026-09-29** ([#147](https://github.com/Gerrrt/HomeLab/issues/147), [ADR-0057](adr/0057-add-miniflux-to-the-sensitive-tier-with-its-fetcher-kept-off-winterfell.md)): the one container here that reaches outward on a timer, with its fetcher refused every private address. Memos, the household's notes, is the second, decided by [ADR-0059](adr/0059-add-memos-to-the-sensitive-tier-for-notes-and-keep-documentation-in-docs.md) ([#145](https://github.com/Gerrrt/HomeLab/issues/145)) and deployed behind Caddy the same day, all sixteen containers healthy after that `make up`. Miniflux holds no real data; Memos holds the household's first real notes since 2026-10-22, ahead of ADR-0023's off-estate copy by the operator's accepted exception, with the nightly volume set on `oracle` its only copy off the host, since the household copy under [#455](https://github.com/Gerrrt/HomeLab/issues/455) does not carry `memos-data` yet. Mealie, the household's recipes at `recipes.matrix.elysium`, is the third, **deployed 2026-09-29** ([#146](https://github.com/Gerrrt/HomeLab/issues/146), [ADR-0060](adr/0060-add-mealie-to-the-sensitive-tier-as-recipes.md)), behind the same Caddy with no new rule, its URL import refused every private address. linkding, a bookmark manager, is the fourth, **deployed 2026-09-29** ([#144](https://github.com/Gerrrt/HomeLab/issues/144), [ADR-0061](adr/0061-add-linkding-to-the-sensitive-tier-behind-one-factor.md)). Actual ([#142](https://github.com/Gerrrt/HomeLab/issues/142)), the household's budget, is the fifth beyond ADR-0008's nine, **deployed 2026-09-29** ([ADR-0062](adr/0062-add-actual-to-the-sensitive-tier.md)), claimed from SOPS before its first start. It holds no real budget before #404 step 10. Stirling-PDF, the household's PDF editor at `pdf.matrix.elysium`, is the sixth, **deployed 2026-09-29** ([#143](https://github.com/Gerrrt/HomeLab/issues/143), [ADR-0063](adr/0063-add-stirling-pdf-to-the-sensitive-tier-and-keep-its-documents-in-memory.md)): its documents live on a tmpfs and never reach a disk. Glance, the operator's browser start page at `start.matrix.elysium`, is **deployed 2026-10-21** ([#1223](https://github.com/Gerrrt/HomeLab/issues/1223), [ADR-0098](adr/0098-add-glance-as-the-operators-start-page-beside-homepage.md)): the estate's alerts, probes and targets read from `prometheus` with the reader token, and feeds read from a Miniflux user of its own. Since [ADR-0100](adr/0100-grow-glance-into-a-homelab-page-and-keep-the-operators-lists-in-sops.md) it has a page per subject — Home, Read, Watch, Sports & Markets, Homelab, Network & Storage and Maintenance — the estate's read from `prometheus`, and the operator's own lists come from SOPS. It also serves the household's page at `home.matrix.elysium` ([ADR-0103](adr/0103-retire-homepage-and-serve-the-household-page-from-glance.md)), which took over from Homepage. Mosquitto and ring-mqtt bring the Ring Alarm into Home Assistant, the broker on an internal network that only those two join, **deployed 2026-10-22** ([ADR-0104](adr/0104-bring-the-ring-alarm-into-home-assistant-through-ring-mqtt.md)). Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | diff --git a/docs/network.md b/docs/network.md index 5dc2cac7..112e06d2 100644 --- a/docs/network.md +++ b/docs/network.md @@ -741,7 +741,13 @@ Where things get broken on purpose. [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s WireGuard tunnel terminates, built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the peers reach the - lab through it and nothing else. Its Alloy pushes to + lab through it and nothing else. Its inbound policy is decided and not yet + applied: [ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md) + makes it default-deny, with `22` admitting Hicks and the peers and nothing + else on this segment, and `8800` open to the segment only while a Debian or + Kali template builds. Until [#1496](https://github.com/Gerrrt/HomeLab/issues/1496) + applies it, `phoenix` has no host firewall and `22` answers the whole + segment. Its Alloy pushes to `alexander` and not to Winterfell, and it gets no pass out of this segment: a rule from it into 99 would make it the bastion ADR-0002 and ADR-0012 declined. Everything diff --git a/docs/roadmap.md b/docs/roadmap.md index b2600b24..2af00519 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -104,6 +104,12 @@ Closes when it is empty. `smaug`.** Decided by [ADR-0114](adr/0114-rip-discs-on-smaug-with-makemkv-started-only-to-rip.md): deploy it, measure what capabilities it needs, and create the `5800` pass. +- **[#1496](https://github.com/Gerrrt/HomeLab/issues/1496) Close `phoenix`'s + way in.** Decided by + [ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): + give every workstation that logs in a key, then run + [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md) §6a on the host. + First of these, because `phoenix` owns the lab domain. The rest of the milestone has no order between its issues. diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index 05e19a21..5ab836f3 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -384,6 +384,86 @@ to check the pass to `10.0.30.40:9443`, the answer is that there is no pass to place — go back to §5, because that message was written for hosts on another VLAN and this one is not. +## 6a. Close the way in + +[ADR-0117](../adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): +`sshd` takes keys only, and inbound is default-deny. Nothing converges this +host, so this section is how it gets there, on a rebuild as on the first +build. + +**First, every workstation that logs in has a key.** This is the step that +locks you out if it is skipped. From each Hicks machine that uses `phoenix`, +add its public key to `~/.ssh/authorized_keys` on `phoenix`, then prove it +with passwords refused on the client side: + +```bash +ssh -o ControlMaster=no -o ControlPath=none \ + -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no \ + @10.0.30.70 true && echo key-ok +``` + +`ControlMaster=no` and `ControlPath=none` force a new connection. Without +them, a client that multiplexes reuses an open session that logged in by +password, and prints `key-ok` without testing a key. Every machine prints +`key-ok` before you go on. Keep one session open on +`phoenix` until the end of this section. + +**Then `sshd`.** On `phoenix`. The `10-` prefix sorts ahead of the installer's +`50-cloud-init.conf`, and `sshd` keeps the first value it reads: + +```bash +sudo tee /etc/ssh/sshd_config.d/10-phoenix.conf >/dev/null <<'EOF' +# ADR-0117: phoenix owns the lab domain, so its own login is key-only. +PasswordAuthentication no +KbdInteractiveAuthentication no +PermitRootLogin no +X11Forwarding no +EOF +sudo sshd -t && sudo systemctl reload ssh +sudo sshd -T | grep -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin|x11forwarding) ' +``` + +All four lines read `no`. A fresh login from a second terminal, with the +first still open, is the check that the reload did not shut you out: the +same command as above, with its `ControlMaster=no` and `ControlPath=none`. + +**Then the firewall.** `ufw` does not touch `net.ipv4.ip_forward`, and its +forward policy is `DROP`, so the tunnel needs the one `route` line. On a +first build the tunnel does not exist yet: leave out the three lines marked +`tunnel`, and [`open-the-remote-path.md`](open-the-remote-path.md) §3 adds +them. + +```bash +sudo ufw default deny incoming +sudo ufw default allow outgoing +sudo ufw default deny routed +sudo ufw allow from 10.0.50.0/24 to any port 22 proto tcp comment 'Hicks' +sudo ufw allow in on wg0 from 172.31.0.0/24 to any port 22 proto tcp comment 'tunnel peers' # tunnel +sudo ufw allow /udp comment 'WireGuard' # tunnel +sudo ufw route allow in on wg0 out on ens18 comment 'ADR-0042: morpheus filters' # tunnel +sudo ufw enable +sudo ufw status verbose +``` + +`` is the one `open-the-remote-path.md` §0 chose. It stays out +of this repository. `ens18` is the NIC `ip -br addr` shows with +`10.0.30.70`. Port `8800` is not here: a Debian or Kali template build opens +it for its own length +([`build-the-lab-templates.md`](build-the-lab-templates.md) §5). + +**Check each rule from the side it applies to.** + +- From Hicks: `ssh -o ControlMaster=no -o ControlPath=none @10.0.30.70 true` + succeeds. +- From a VLAN 30 host, `alexander` for example: + `nc -zv -w 3 10.0.30.70 22` times out. +- From a peer, with the tunnel up: the same `ssh` succeeds. + `ping -c 1 10.0.30.40` answers at `ttl=63`, which shows `phoenix` still + forwards. Then run [`open-the-remote-path.md`](open-the-remote-path.md) §8's + negative tests, which must still be blocked and logged on `morpheus`. +- On `phoenix`: §4's `curl` still lists the guests, and §7's Alloy checks + still pass. Both are outbound, and `ufw` allows outbound. + ## 7. Verify — including the thing that fails quietly ```bash diff --git a/docs/runbooks/build-the-lab-templates.md b/docs/runbooks/build-the-lab-templates.md index 27fed6b8..a6f52c25 100644 --- a/docs/runbooks/build-the-lab-templates.md +++ b/docs/runbooks/build-the-lab-templates.md @@ -509,11 +509,15 @@ DHCP server dhcpcd adds a `169.254.x.x` address and a link-local default route ([`build-the-playground.md`](build-the-playground.md) §8). Rebuild 902 and 910 with this section and §5b to clear it. -1. Admit the guest to that port on `phoenix` for the length of the build. If - `ufw` is active, run `sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp`, - and delete the rule afterwards. Debian's build (§4) needs this step and - no other here. On 2026-11-09 `ufw` was inactive on `phoenix`, so there - was nothing to open. +1. Admit the guest to that port on `phoenix` for the length of the build: + `sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp`. Afterwards, + `sudo ufw delete allow from 10.0.30.0/24 to any port 8800 proto tcp`. + Debian's build (§4) needs this step and no other here. `ufw` is + default-deny on `phoenix` + ([ADR-0117](../adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md)), + so without the rule the installer times out fetching its preseed. On + 2026-11-09, before that ADR, `ufw` was inactive and there was nothing to + open. 2. Give `phoenix` a user on `gandalf`, and trust `gandalf`'s CA on `phoenix`, as §2 did for `Saruman`. Unless `gandalf` joins `Saruman` in a cluster, it is its own API with its own users, so `phoenix@pve` there is a diff --git a/docs/runbooks/open-the-remote-path.md b/docs/runbooks/open-the-remote-path.md index d37d1a74..09c3a3b9 100644 --- a/docs/runbooks/open-the-remote-path.md +++ b/docs/runbooks/open-the-remote-path.md @@ -269,6 +269,15 @@ sudo sysctl -w net.ipv4.ip_forward=0 \ # expect 1, then 0, then 1 ``` +Then the jumpbox's own firewall. `ufw` is default-deny on `phoenix` +([ADR-0117](../adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md)), +so add the three lines +[`build-the-jumpbox.md`](build-the-jumpbox.md) §6a marks `tunnel`: the listen +port, `22` from the peers on `wg0`, and the `route` rule from `wg0` to the +lab. Without the `route` rule the handshake succeeds and nothing forwards, +the same symptom as the CAUTION above. `sudo ufw status verbose` lists all +three. + ## 4. The client configuration On the **client device**, generate its own keypair and build this. Send the diff --git a/docs/security.md b/docs/security.md index b5124b95..fa84dd7d 100644 --- a/docs/security.md +++ b/docs/security.md @@ -17,7 +17,7 @@ What this network is actually built to survive: | A lost or stolen tunnel peer | **Accepted.** A WireGuard peer is a device with a key and no second factor ([ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)); the firewall, not the key, bounds what it reaches — the lab and nothing else — with a `/32` pin on the server and a preshared key per peer. Revocation is a `wg0.conf` edit on `phoenix` ([runbook](runbooks/open-the-remote-path.md#rollback)), proportionate at two or three devices and recorded as the thing that stops being so | | An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | An attacker on the lab segment reaching the hypervisor's management plane | **Closed at the host, and watched.** `Saruman`'s Proxmox firewall admits `8006` and `22` from Hicks and `8006` from the deployment host only ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md)). It was found disabled and turned on on 2026-09-20 ([#566](https://github.com/Gerrrt/HomeLab/issues/566)); since [#576](https://github.com/Gerrrt/HomeLab/issues/576) `homelab_pve_firewall_enabled` is read every five minutes and `PveFirewallDisabled` pages after ten, so a `pve-firewall stop` left in place is noticed rather than found. `PveFirewallPolicyAccept` does the same for `policy_in` left at `ACCEPT`, which reads as enabled and admits the whole segment | -| Someone on the lab segment reaching the domain's configuration path | **Scoped, key-only, and accepted for `phoenix` itself.** Each of ADR-0029's six listens on `22` for `ansible/` ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). The rule admits `10.0.30.70` alone, `sshd` refuses passwords, and the only administrators' key is `phoenix`'s, installed by the template and never by hand on a clone. The residual: whoever owns `phoenix` owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them | +| Someone on the lab segment reaching the domain's configuration path | **Scoped, key-only, and accepted for `phoenix` itself.** Each of ADR-0029's six listens on `22` for `ansible/` ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). The rule admits `10.0.30.70` alone, `sshd` refuses passwords, and the only administrators' key is `phoenix`'s, installed by the template and never by hand on a clone. The residual: whoever owns `phoenix` owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them. `phoenix`'s own way in is narrowed to match by [ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): its `sshd` takes keys only, and its `ufw` admits `22` from Hicks and the tunnel's peers and nothing from the rest of VLAN 30. Until [#1496](https://github.com/Gerrrt/HomeLab/issues/1496) applies it on the host, the installer's password login and an open segment are what stand there | | Someone on the lab segment logging in to the sensor | **Authored, and live only once applied.** `fenrir` listens on `22` to the whole segment and has no host firewall. [`build-the-sensor-guest.md`](runbooks/build-the-sensor-guest.md) §3 makes `sshd` key-only for one user, refuses root and X11 forwarding, and keeps `authorized_keys` to the machines that administer the guest. Until that step has run on `fenrir`, the installer's `sshd` defaults are the running state. The source address stays open: anyone on the segment can still offer a key | | Someone on the lab segment logging in to `alexander` | **Closed at `sshd`.** `alexander` holds the lab's only age key, and its `sshd` answers all of VLAN 30 with no host firewall. It is installed by hand from the ISO, so it does not inherit the template's `allow-pw: false`. [`build-the-lab-guest.md`](runbooks/build-the-lab-guest.md) §2 makes `sshd` key-only through a drop-in, refuses root and X11 forwarding, and keeps `garnet`'s `authorized_keys` to Hicks's key and `phoenix`'s; §7 checks it with `sshd -T`. A read on 2026-11-15 found it taking passwords; the step was applied the same day, and `sshd -T` read `no` for all four settings. The source address stays open: anyone on the segment can still offer a key | | Someone on the lab segment reading the domain's metrics | **Scoped, unauthenticated, and accepted.** `windows_exporter` listens on `9182` on all six of ADR-0029's guests, on the segment that exists to hold attackers. Each guest's Windows Firewall admits `10.0.30.40` (`alexander`) alone ([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) §7). That rule is the difference between an endpoint a `/24` sweep finds and one you have to go looking for, and it is a residual, not a control: the exporter has no authentication, so whoever owns `alexander` reads every gauge. It has no write API, so what leaks is the shape of the host (services, disks, uptime, the licence clock), the same class as `node_exporter` on the NAS below | @@ -653,7 +653,10 @@ assumption consistent with what they are. genuinely encrypted, and that no state, plan or `.terraform/` is tracked. - **`phoenix` is a secret-bearing host, and holds no age key.** Its secrets are all mode 600 and owned by its operator: - - the Proxmox API token `phoenix@pve!builder`; + - the Proxmox API token `phoenix@pve!builder` on `Saruman`, and its + namesake on `gandalf`, the range hypervisor, which is in `phoenix.env` as + `TF_VAR_gandalf_api_token` and in `gandalf.env` for the curl scripts + ([ADR-0116](adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)); - the Windows build password, which is also a fresh clone's Administrator password until #448 rotates it; - `tofu/`'s state passphrase, all three in `~/.config/proxmox/phoenix.env` @@ -667,6 +670,12 @@ assumption consistent with what they are. and `LAB_POPULATION_SEED`, from which every population password is derived ([ADR-0078](adr/0078-populate-the-lab-domain-from-a-committed-file-and-a-seed.md)). The seed adds no reach: the first of the three already owns the domain. + Four more, also in `phoenix.env`, add none either + ([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md)): + `LAB_TIER_ADMIN_PASSWORD` and `LAB_ENDPOINT_ADMIN_PASSWORD`, which a + rebuild creates accounts with, `LAB_WEAK_PASSWORD`, the deliberately + crackable one (ADR-0083), and `LAB_WAZUH_REGISTRATION_PASSWORD`, a copy + of the SOC manager's enrolment password. The state is the one that would have been quiet. A Terraform state holds every value a provider touched in cleartext, the cloud-init password of every