From e513afac67ca3437179a47a95c9e723addc8236e Mon Sep 17 00:00:00 2001 From: Your Name Date: Mon, 16 Nov 2026 08:35:27 +0000 Subject: [PATCH 1/2] feat(observability): alert when the wiki drift check cannot run DriftCheckStopped reads only the timestamp, which the collector writes whatever the run found, so the 2026-11-15 failure (exit 128 at the wiki fetch) raised nothing. DriftCheckFailing fires on an exit of 2 or higher. Exit 1, drift, already files a wiki issue. Also corrects four stale current-state lines found by the oracle audit: oracle's cell (replaced and measured 2026-09-29), patch-state on agent hosts, oracle's agent deploy running from prometheus, and the host rule count. Refs #1531 Co-Authored-By: Claude Opus 5.5 --- README.md | 4 +- docs/changelog.md | 20 ++++++ docs/observability.md | 11 ++-- docs/runbooks/deploy-merged-changes.md | 6 +- docs/runbooks/deploy-stack.md | 2 +- docs/runbooks/fit-the-ups-battery.md | 8 ++- docs/runbooks/schedule-maintenance.md | 10 ++- docs/security.md | 4 +- stacks/observability/README.md | 2 +- .../prometheus/rules/host.rules.yaml | 35 +++++++++++ .../prometheus/tests/host.test.yaml | 62 +++++++++++++++++++ 11 files changed, 146 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index d8debbc5..2189fa62 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 8 provisioned dashboards, 156 panels, and - 184 alert rules — 163 metric-based in Prometheus, 21 log-based in Loki — sharing + 185 alert rules — 164 metric-based in Prometheus, 21 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -206,7 +206,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology . ├── stacks/ │ ├── observability/ # the estate's stack on prometheus — ten services -│ │ ├── prometheus/ # config, file_sd targets, 163 alert rules +│ │ ├── prometheus/ # config, file_sd targets, 164 alert rules │ │ ├── alertmanager/ # routing and inhibition │ │ ├── loki/ # single-binary config + 21 LogQL rules │ │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/changelog.md b/docs/changelog.md index 71c353f9..06ce9774 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,26 @@ docstring gives: it is a record, not a claim about now. ## 2026-11-16 +- **A wiki drift check that cannot run now alerts** + ([#1531](https://github.com/Gerrrt/HomeLab/issues/1531)). On 2026-11-15 + the run on `oracle` failed at the wiki fetch, exit 128. The checker never + started, and nothing alerted: `DriftCheckStopped` reads only the timestamp, + and the collector writes it whatever the run found. The next run, at 06:40 + UTC on 2026-11-16, passed all 16 claims, and the cause is not known. + `DriftCheckFailing` fires on an exit of 2 or higher, an unreadable claim or + a checker that did not start. Exit 1 is left out: drift files its own wiki + issue. It is the 164th Prometheus rule, with three promtool tests. + - **Four stale current-state lines, from the same audit of `oracle`.** + `fit-the-ups-battery.md` and `security.md` still had `oracle`'s cell as + the original at 72 % with its runtime never measured. It was replaced and + measured on 2026-09-29. `schedule-maintenance.md` said `patch-state` would + need shipping to `oracle`, which the agent installer already does. + `deploy-merged-changes.md` §6 ran `oracle`'s agent deploy from the Mac, + which cannot decrypt its token. It now runs from `prometheus`. + - **`observability.md` said forty-one rules in `host.rules.yaml`.** There + were sixty and are now sixty-one. `check_docs.py` checks the totals, not + that number. + - **Template 901 is rebuilt with both of `eden`'s fixes.** The Ubuntu build of `build-the-lab-templates.md` §4 ran from `phoenix` on main after #1525, in 19 min 40 s. `prune.sh` purged all eight services, and the diff --git a/docs/observability.md b/docs/observability.md index a65092d4..ed836857 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -352,7 +352,7 @@ separates a quiet stream from a stopped one. ## Alerting -184 rules in total: 163 metric-based in `prometheus/rules/`, and 21 log-based in +185 rules in total: 164 metric-based in `prometheus/rules/`, and 21 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -513,11 +513,11 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -163 rules across twelve files in `prometheus/rules/`: +164 rules across twelve files in `prometheus/rules/`: | File | Covers | | --- | --- | -| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github.com/Gerrrt/HomeLab/issues/538), [#576](https://github.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)); and a guest tagged `backup` that no enabled backup job selects, as `GuestNotInBackupJob`, after the domain's six fell out of `golem-nightly` through purged tofu destroys (2026-10-10); and how many days the four management consoles' certificates have left — the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI, none of which the blackbox exporter can reach — `ManagementCertificateExpiringSoon` at 30 days and `ManagementCertificateExpiryImminent` at 7, read from a handshake made inside each console's own segment, with `ManagementCertificateUnchecked` for a console that would not answer and `CertExpiryStateStale` for a collector that stopped ([#857](https://github.com/Gerrrt/HomeLab/issues/857), [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) | +| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running, and still able to read its claims ([#470](https://github.com/Gerrrt/HomeLab/issues/470), [#1531](https://github.com/Gerrrt/HomeLab/issues/1531)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github.com/Gerrrt/HomeLab/issues/538), [#576](https://github.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)); and a guest tagged `backup` that no enabled backup job selects, as `GuestNotInBackupJob`, after the domain's six fell out of `golem-nightly` through purged tofu destroys (2026-10-10); and how many days the four management consoles' certificates have left — the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI, none of which the blackbox exporter can reach — `ManagementCertificateExpiringSoon` at 30 days and `ManagementCertificateExpiryImminent` at 7, read from a handshake made inside each console's own segment, with `ManagementCertificateUnchecked` for a console that would not answer and `CertExpiryStateStale` for a collector that stopped ([#857](https://github.com/Gerrrt/HomeLab/issues/857), [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, a switch port whose VLAN changed (`SwitchPortVlanChanged`, from `dot1qPvid`, [#1387](https://github.com/Gerrrt/HomeLab/issues/1387)), iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature, and whether the shutdown order has time to run: `UpsLowBatteryDurationShort` when the card's low-battery duration is under the 16 minutes [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) needs, `UpsRideThroughShort` when a charged pack on mains would start the halt under 15 minutes into a cut. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | @@ -549,7 +549,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is 163 rules of 163 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all twelve +against the broken rule too. Coverage is 164 rules of 164 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all twelve in `blackbox.rules.yaml`, all three in `dns.rules.yaml`, `GatewayFilesystemCritical`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled`, the three container-state rules from @@ -562,7 +562,7 @@ rules from [#840](https://github.com/Gerrrt/HomeLab/issues/840), `RemoteWriteJob `ScrapeTargetDisappeared`, `SuricataStopped`, the two gateway rules from [#353](https://github.com/Gerrrt/HomeLab/issues/353), the two dynamic DNS rules from -[#604](https://github.com/Gerrrt/HomeLab/issues/604), and all forty-one in +[#604](https://github.com/Gerrrt/HomeLab/issues/604), and all sixty-one in `host.rules.yaml` — `HostDiskWillFillIn24h` from [#189](https://github.com/Gerrrt/HomeLab/issues/189), six more from [#320](https://github.com/Gerrrt/HomeLab/issues/320), the four @@ -570,6 +570,7 @@ SMART rules from [#351](https://github.com/Gerrrt/HomeLab/issues/351), `PatchStateStopped` from [#360](https://github.com/Gerrrt/HomeLab/issues/360), `SystemUpdateAvailable` from [#378](https://github.com/Gerrrt/HomeLab/issues/378), `DriftCheckStopped` from [#470](https://github.com/Gerrrt/HomeLab/issues/470), +`DriftCheckFailing` from [#1531](https://github.com/Gerrrt/HomeLab/issues/1531), the two guest rules from [#257](https://github.com/Gerrrt/HomeLab/issues/257), the three laptop-battery rules from [#454](https://github.com/Gerrrt/HomeLab/issues/454), diff --git a/docs/runbooks/deploy-merged-changes.md b/docs/runbooks/deploy-merged-changes.md index 46024408..8704574a 100644 --- a/docs/runbooks/deploy-merged-changes.md +++ b/docs/runbooks/deploy-merged-changes.md @@ -129,7 +129,11 @@ Grafana is minutes old. From the Mac, after `git pull`: - **The Alloy agent.** `make deploy-agent ARGS="@"` for `oracle` - and `Saruman`, which push to `prometheus`. For `phoenix` and `golem`, which + and `Saruman`, which push to `prometheus`. For `oracle`, run it on + `prometheus` instead, which holds the SOPS key the Mac does not: + `ssh prometheus 'cd ~/code/Gerrrt/HomeLab && scripts/deploy-agent.sh atropos@10.0.99.30'`. + `prometheus` does not resolve `oracle` by name + ([#1526](https://github.com/Gerrrt/HomeLab/issues/1526)). For `phoenix` and `golem`, which push to the lab, add `--monitoring-host 10.0.30.40`. - **Tokens.** For a lab host, nothing is decrypted on the Mac: export `INGEST_TOKEN` and `INGEST_TOKEN_READER` from `secrets/lab.sops.yaml` on diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 61a4eef8..adc8343e 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -151,7 +151,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. 2. **Prometheus → Status → Rules.** Every rule loaded, none in error. The - page lists 169: the 163 alert rules this repository counts everywhere + page lists 170: the 164 alert rules this repository counts everywhere else, plus the six recording rules: `homelab_suricata_expected_interface`, `homelab_battery_runtime_seconds`, and the two directions each of diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index 62066daa..b1c0ce53 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -86,7 +86,8 @@ is measured rather than assumed ([#454](https://github.com/Gerrrt/HomeLab/issues/454)). No collector was needed: Alloy's node collector already exports `node_power_supply_*` from both hosts, and `host.rules.yaml` reads it — `HostBatteryHealthLow` when a cell's -`charge_full` falls below 80 % of `charge_full_design` (`oracle` reads 72 %; +`charge_full` falls below 80 % of `charge_full_design` (`oracle` read 72 % until +its cell was replaced on 2026-09-29, and now reads 100 %, its design figure; `prometheus` read 94 % after 108 cycles until its cell was replaced on 2026-09-18, and now reads 101 % at one cycle), `HostOnBattery` when the mains adapter loses @@ -110,8 +111,9 @@ above design, `cyclecount` 1. **The second half was proved on 2026-09-19**: the brick pulled on a full pack, `HostOnBattery` firing for `prometheus` alone inside three minutes with `UpsOnBattery` quiet, the host up throughout, and a measured draw of 2.72 Ah/h — about 2.5 hours from full at the stack's load, one -measurement on one day. That closed #454. `oracle`'s half is untested and its -cell is [#531](https://github.com/Gerrrt/HomeLab/issues/531). +measurement on one day. That closed #454. `oracle`'s cell was replaced on +2026-09-29 and measured the same day at 0.54 Ah/h, about 5.2 hours from full +([#531](https://github.com/Gerrrt/HomeLab/issues/531)). ## Before you start diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index e0cc5cec..86102957 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -185,8 +185,9 @@ it has no log store ([#327](https://github.com/Gerrrt/HomeLab/issues/327)). happened OR that nothing diverged — it collects a fact the estate did not previously have, which is how far behind this host's packages are ([#152](https://github.com/Gerrrt/HomeLab/issues/152)). It needs no root, and it -covers **this host only**: `oracle` and `Saruman` would need it shipped to them -and `morpheus` is FreeBSD with no apt at all. +covers **this host** from this table. An agent host with apt, `oracle` among +them, gets it from the installer below, and `morpheus` is FreeBSD with no apt +at all. **Agent hosts run these collectors too, and not through this table.** A host that runs Alloy but has no checkout of this repository — `oracle` — gets the @@ -274,7 +275,10 @@ login, via `runuser -l` from a root unit that exists only to write the textfile directory — reads the checker off the wiki's `origin/main`, and records `homelab_drift_check_last_run_timestamp_seconds`, the exit code and the four claim counts. `DriftCheckStopped` fires when the timestamp is more than a day -old ([#470](https://github.com/Gerrrt/HomeLab/issues/470)). Its requirement is +old ([#470](https://github.com/Gerrrt/HomeLab/issues/470)), and `DriftCheckFailing` +when the last run exited 2 or higher: a claim it could not read, or a checker +that never started. Exit 1, drift, files its own wiki issue +([#1531](https://github.com/Gerrrt/HomeLab/issues/1531)). Its requirement is the wiki checkout's guard script, so every other host correctly reports that it cannot run this one. It replaced a crontab entry that nothing watched. diff --git a/docs/security.md b/docs/security.md index ece6a6b4..b5124b95 100644 --- a/docs/security.md +++ b/docs/security.md @@ -29,9 +29,9 @@ What this network is actually built to survive: | Someone on the lab segment reaching the lab's Grafana or BloodHound, or a detonation reaching `diabolos`'s UIs | **Scoped by address, as the SOC's are.** `alexander`'s Grafana (`3000`) is published by the lab stack's `caddy`, and `eden`'s BloodHound (`8443`) by the bloodhound stack's own `caddy`; each answers Hicks's `10.0.50.0/24` and refuses the rest of VLAN 30, the six domain guests included. `diabolos`, not yet built, has soc's fence for its dashboard and GUI. Until then Grafana and BloodHound answered all of VLAN 30, where BloodHound's graph is the domain's attack-path map ([#1428](https://github.com/Gerrrt/HomeLab/issues/1428)). The residuals are the SOC row's: Hicks is a whole /24, and an address on VLAN 30 can be spoofed for a one-way packet but not for the TLS session the UIs need | | A range target with a path out | It has none — `gandalf`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 184 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 185 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | -| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What happens when the cut outlasts the pack is armed and not proved.** Since 2026-09-23 the firewall's NUT server reads `mjolnir`, and `Saruman` and `smaug` subscribe to it on their own gateway addresses, so a real cut would now try to run the sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides: `Saruman` and `smaug` halt first and `morpheus` last. Nobody has watched it run. Since the move into the 32U rack on 2026-11-06 the PDU also carries `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room. [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) makes `trinity` a third subscriber and bounds `Saruman`'s guest stop. `Saruman`'s half was applied on 2026-11-13 (`down=90` on the four ordered guests, and its monitor stays logged in until they are down); `trinity`'s subscription, `HOSTSYNC 600` on `morpheus` and the card's 16-minute threshold are not, so `trinity` still stops uncleanly when the pack empties and the card still holds 8 minutes; `gandalf`, the console and `neo` are left to the pack by decision. The pack itself is unmeasured: about 47 minutes at 21 % load is the card's own estimate. [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §5 to §7 are what is left — the forced-shutdown proof and the one mains pull that measures the pack ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | +| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s was replaced on 2026-09-29 and reads 100 %, its design figure rather than a learned one ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **each runtime was measured once: `prometheus`'s on 2026-09-19, about 2.5 hours from full at the stack's load, and `oracle`'s on 2026-09-29, about 5.2 hours**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for both hosts' runtime and for neither's cell temperature — see below. **What happens when the cut outlasts the pack is armed and not proved.** Since 2026-09-23 the firewall's NUT server reads `mjolnir`, and `Saruman` and `smaug` subscribe to it on their own gateway addresses, so a real cut would now try to run the sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides: `Saruman` and `smaug` halt first and `morpheus` last. Nobody has watched it run. Since the move into the 32U rack on 2026-11-06 the PDU also carries `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room. [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) makes `trinity` a third subscriber and bounds `Saruman`'s guest stop. `Saruman`'s half was applied on 2026-11-13 (`down=90` on the four ordered guests, and its monitor stays logged in until they are down); `trinity`'s subscription, `HOSTSYNC 600` on `morpheus` and the card's 16-minute threshold are not, so `trinity` still stops uncleanly when the pack empties and the card still holds 8 minutes; `gandalf`, the console and `neo` are left to the pack by decision. The pack itself is unmeasured: about 47 minutes at 21 % load is the card's own estimate. [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §5 to §7 are what is left — the forced-shutdown proof and the one mains pull that measures the pack ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below | What it explicitly does **not** defend against: a determined attacker with diff --git a/stacks/observability/README.md b/stacks/observability/README.md index b9e2ee17..e2ab0ba2 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -53,7 +53,7 @@ prometheus/ targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/node.yaml node_exporter scrapes, for the host that runs no Alloy (smaug) targets/blackbox*.yaml probe targets, http, dns and latency — hot-reloaded, no restart - rules/*.rules.yaml 163 alert rules: host, network, ups, containers, blackbox, + rules/*.rules.yaml 164 alert rules: host, network, ups, containers, blackbox, dns, backup, ids, deploy, stack, internet and watchdog tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said diff --git a/stacks/observability/prometheus/rules/host.rules.yaml b/stacks/observability/prometheus/rules/host.rules.yaml index e0fc2fa7..bab34ba3 100644 --- a/stacks/observability/prometheus/rules/host.rules.yaml +++ b/stacks/observability/prometheus/rules/host.rules.yaml @@ -353,6 +353,41 @@ groups: maintainer's user, so a missing checkout at /home/atropos/code/Gerrrt/Lemmiwinks or an expired gh login is the usual cause. + - alert: DriftCheckFailing + # The wiki's drift check on oracle runs but cannot do its job (#1531). + # + # DriftCheckStopped above reads only the timestamp, and the collector + # writes that whatever the run found, so a check that runs every day and + # fails every day stays quiet under it. On 2026-11-15 the wiki fetch + # failed with exit 128, the checker never started, and the journal was + # the only record. + # + # TWO AND UP, NOT ONE. Exit 1 is drift or a missing anchor, and the + # checker already reports that itself, as an issue on the wiki. Exit 2 + # is a claim it could not read (#312's shape: two weeks of UNREADABLE + # rows after the ingest proxy took a token). Anything above 2 means the + # checker did not run: a fetch, a login or the checkout failed under it. + # Neither of those files an issue. + # + # The gauge holds its value until the next daily run, so `for: 30m` + # waits out nothing but the scrape. A failure clears on the next run + # that exits 0 or 1. + expr: homelab_drift_check_last_exit_code >= 2 + for: 30m + labels: + component: host + severity: warning + category: correctness + annotations: + summary: "{{ $labels.host }}'s wiki drift check ran and could not finish (exit {{ $value }})" + description: >- + Exit 2 means a claim could not be read, so part of the wiki is + unchecked. A higher exit means the checker never started. Its last + run is in Loki under unit="homelab-drift-check.service". At 2, + the usual cause is a missing or expired ~/.ingest-reader-token for + atropos. Above 2, it is the wiki fetch or the gh login in + /home/atropos/code/Gerrrt/Lemmiwinks; a one-off clears itself at + the next run, at 06:30 UTC. # --- SMART, for the disks the iLO cannot see (#351) ------------------- # # #151 already covers Saruman's array: cpqDaPhyDrvSmartStatus arrives over diff --git a/stacks/observability/prometheus/tests/host.test.yaml b/stacks/observability/prometheus/tests/host.test.yaml index acd2b8a1..5fdabfc9 100644 --- a/stacks/observability/prometheus/tests/host.test.yaml +++ b/stacks/observability/prometheus/tests/host.test.yaml @@ -763,6 +763,68 @@ tests: alertname: DriftCheckStopped exp_alerts: [] + # --- DriftCheckFailing: fires when the checker could not start ------------ + # 2026-11-15's shape: the wiki fetch failed, exit 128, and the timestamp kept + # moving, so DriftCheckStopped stayed quiet. + - interval: 5m + input_series: + - series: 'homelab_drift_check_last_exit_code{host="oracle"}' + values: "0x12 128x24" + alert_rule_test: + - eval_time: 2h + alertname: DriftCheckFailing + exp_alerts: + - exp_labels: + host: oracle + component: host + severity: warning + category: correctness + exp_annotations: + summary: "oracle's wiki drift check ran and could not finish (exit 128)" + description: >- + Exit 2 means a claim could not be read, so part of the wiki is + unchecked. A higher exit means the checker never started. Its last + run is in Loki under unit="homelab-drift-check.service". At 2, + the usual cause is a missing or expired ~/.ingest-reader-token for + atropos. Above 2, it is the wiki fetch or the gh login in + /home/atropos/code/Gerrrt/Lemmiwinks; a one-off clears itself at + the next run, at 06:30 UTC. + + # --- DriftCheckFailing: fires on an unreadable claim ------------------------ + - interval: 5m + input_series: + - series: 'homelab_drift_check_last_exit_code{host="oracle"}' + values: "2x24" + alert_rule_test: + - eval_time: 1h + alertname: DriftCheckFailing + exp_alerts: + - exp_labels: + host: oracle + component: host + severity: warning + category: correctness + exp_annotations: + summary: "oracle's wiki drift check ran and could not finish (exit 2)" + description: >- + Exit 2 means a claim could not be read, so part of the wiki is + unchecked. A higher exit means the checker never started. Its last + run is in Loki under unit="homelab-drift-check.service". At 2, + the usual cause is a missing or expired ~/.ingest-reader-token for + atropos. Above 2, it is the wiki fetch or the gh login in + /home/atropos/code/Gerrrt/Lemmiwinks; a one-off clears itself at + the next run, at 06:30 UTC. + + # --- DriftCheckFailing: quiet on drift, which files its own wiki issue ----- + - interval: 5m + input_series: + - series: 'homelab_drift_check_last_exit_code{host="oracle"}' + values: "0x12 1x24" + alert_rule_test: + - eval_time: 3h + alertname: DriftCheckFailing + exp_alerts: [] + # --- PatchStateStopped: quiet for a host being onboarded ------------------- # A host that has never reported must stay silent, or every apt host not yet # covered by #360 — `Saruman` today — alerts as a fault. The series starts at From 88fc6e2380357794ce2c03979af55d7a7c8376fe Mon Sep 17 00:00:00 2001 From: Your Name Date: Mon, 16 Nov 2026 08:53:18 +0000 Subject: [PATCH 2/2] docs(runbooks): deploy oracle's agent only after prometheus has converged Refs #1531 Co-Authored-By: Claude Opus 5.5 --- docs/runbooks/deploy-merged-changes.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/deploy-merged-changes.md b/docs/runbooks/deploy-merged-changes.md index 8704574a..a4c40130 100644 --- a/docs/runbooks/deploy-merged-changes.md +++ b/docs/runbooks/deploy-merged-changes.md @@ -130,7 +130,10 @@ From the Mac, after `git pull`: - **The Alloy agent.** `make deploy-agent ARGS="@"` for `oracle` and `Saruman`, which push to `prometheus`. For `oracle`, run it on - `prometheus` instead, which holds the SOPS key the Mac does not: + `prometheus` instead, which holds the SOPS key the Mac does not. That + ships `prometheus`'s checkout, not the Mac's, so finish §2 there first and + check with `git -C ~/code/Gerrrt/HomeLab log -1` that it holds the merge. + Then `ssh prometheus 'cd ~/code/Gerrrt/HomeLab && scripts/deploy-agent.sh atropos@10.0.99.30'`. `prometheus` does not resolve `oracle` by name ([#1526](https://github.com/Gerrrt/HomeLab/issues/1526)). For `phoenix` and `golem`, which