Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ documents for different readers.
can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
identically everywhere. [How](docs/architecture.md#observability-data-flow)
- **Dashboards and alerting as code.** 8 provisioned dashboards, 156 panels, and
184 alert rules — 163 metric-based in Prometheus, 21 log-based in Loki — sharing
185 alert rules — 164 metric-based in Prometheus, 21 log-based in Loki — sharing
one Alertmanager routing tree. No dashboard exists only in a database.
- **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
decrypted at deploy time into gitignored paths, with `git log` showing which
Expand Down Expand Up @@ -206,7 +206,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology
.
├── stacks/
│ ├── observability/ # the estate's stack on prometheus — ten services
│ │ ├── prometheus/ # config, file_sd targets, 163 alert rules
│ │ ├── prometheus/ # config, file_sd targets, 164 alert rules
│ │ ├── alertmanager/ # routing and inhibition
│ │ ├── loki/ # single-binary config + 21 LogQL rules
│ │ ├── alloy/ # the agent config directory, shipped to every host
Expand Down
20 changes: 20 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,26 @@ docstring gives: it is a record, not a claim about now.

## 2026-11-16

- **A wiki drift check that cannot run now alerts**
([#1531](https://github.com/Gerrrt/HomeLab/issues/1531)). On 2026-11-15
the run on `oracle` failed at the wiki fetch, exit 128. The checker never
started, and nothing alerted: `DriftCheckStopped` reads only the timestamp,
and the collector writes it whatever the run found. The next run, at 06:40
UTC on 2026-11-16, passed all 16 claims, and the cause is not known.
`DriftCheckFailing` fires on an exit of 2 or higher, an unreadable claim or
a checker that did not start. Exit 1 is left out: drift files its own wiki
issue. It is the 164th Prometheus rule, with three promtool tests.
- **Four stale current-state lines, from the same audit of `oracle`.**
`fit-the-ups-battery.md` and `security.md` still had `oracle`'s cell as
the original at 72 % with its runtime never measured. It was replaced and
measured on 2026-09-29. `schedule-maintenance.md` said `patch-state` would
need shipping to `oracle`, which the agent installer already does.
`deploy-merged-changes.md` §6 ran `oracle`'s agent deploy from the Mac,
which cannot decrypt its token. It now runs from `prometheus`.
- **`observability.md` said forty-one rules in `host.rules.yaml`.** There
were sixty and are now sixty-one. `check_docs.py` checks the totals, not
that number.

- **Template 901 is rebuilt with both of `eden`'s fixes.** The Ubuntu build
of `build-the-lab-templates.md` §4 ran from `phoenix` on main after #1525,
in 19 min 40 s. `prune.sh` purged all eight services, and the
Expand Down
11 changes: 6 additions & 5 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,7 @@ separates a quiet stream from a stopped one.

## Alerting

184 rules in total: 163 metric-based in `prometheus/rules/`, and 21 log-based in
185 rules in total: 164 metric-based in `prometheus/rules/`, and 21 log-based in
`loki/rules/`.

### Log-based (Loki ruler)
Expand Down Expand Up @@ -513,11 +513,11 @@ argument and for what to do when it exits 1.

### Metric-based (Prometheus)

163 rules across twelve files in `prometheus/rules/`:
164 rules across twelve files in `prometheus/rules/`:

| File | Covers |
| --- | --- |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github.com/Gerrrt/HomeLab/issues/538), [#576](https://github.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)); and a guest tagged `backup` that no enabled backup job selects, as `GuestNotInBackupJob`, after the domain's six fell out of `golem-nightly` through purged tofu destroys (2026-10-10); and how many days the four management consoles' certificates have left — the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI, none of which the blackbox exporter can reach — `ManagementCertificateExpiringSoon` at 30 days and `ManagementCertificateExpiryImminent` at 7, read from a handshake made inside each console's own segment, with `ManagementCertificateUnchecked` for a console that would not answer and `CertExpiryStateStale` for a collector that stopped ([#857](https://github.com/Gerrrt/HomeLab/issues/857), [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running, and still able to read its claims ([#470](https://github.com/Gerrrt/HomeLab/issues/470), [#1531](https://github.com/Gerrrt/HomeLab/issues/1531)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github.com/Gerrrt/HomeLab/issues/538), [#576](https://github.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)); and a guest tagged `backup` that no enabled backup job selects, as `GuestNotInBackupJob`, after the domain's six fell out of `golem-nightly` through purged tofu destroys (2026-10-10); and how many days the four management consoles' certificates have left — the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI, none of which the blackbox exporter can reach — `ManagementCertificateExpiringSoon` at 30 days and `ManagementCertificateExpiryImminent` at 7, read from a handshake made inside each console's own segment, with `ManagementCertificateUnchecked` for a console that would not answer and `CertExpiryStateStale` for a collector that stopped ([#857](https://github.com/Gerrrt/HomeLab/issues/857), [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) |
| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, a switch port whose VLAN changed (`SwitchPortVlanChanged`, from `dot1qPvid`, [#1387](https://github.com/Gerrrt/HomeLab/issues/1387)), iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below |
| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature, and whether the shutdown order has time to run: `UpsLowBatteryDurationShort` when the card's low-battery duration is under the 16 minutes [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) needs, `UpsRideThroughShort` when a charged pack on mains would start the halt under 15 minutes into a cut. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
Expand Down Expand Up @@ -549,7 +549,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
feed a rule synthetic series and assert it fires — paired with a case asserting
it stays quiet, because a test that only ever expects silence would have passed
against the broken rule too. Coverage is 163 rules of 163 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all twelve
against the broken rule too. Coverage is 164 rules of 164 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all twelve
in `blackbox.rules.yaml`, all three in `dns.rules.yaml`, `GatewayFilesystemCritical`, `ContainerHighMemory`,
`ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled`,
the three container-state rules from
Expand All @@ -562,14 +562,15 @@ rules from [#840](https://github.com/Gerrrt/HomeLab/issues/840), `RemoteWriteJob
`ScrapeTargetDisappeared`,
`SuricataStopped`, the two gateway rules from
[#353](https://github.com/Gerrrt/HomeLab/issues/353), the two dynamic DNS rules from
[#604](https://github.com/Gerrrt/HomeLab/issues/604), and all forty-one in
[#604](https://github.com/Gerrrt/HomeLab/issues/604), and all sixty-one in
`host.rules.yaml` —
`HostDiskWillFillIn24h` from [#189](https://github.com/Gerrrt/HomeLab/issues/189),
six more from [#320](https://github.com/Gerrrt/HomeLab/issues/320), the four
SMART rules from [#351](https://github.com/Gerrrt/HomeLab/issues/351),
`PatchStateStopped` from [#360](https://github.com/Gerrrt/HomeLab/issues/360),
`SystemUpdateAvailable` from [#378](https://github.com/Gerrrt/HomeLab/issues/378),
`DriftCheckStopped` from [#470](https://github.com/Gerrrt/HomeLab/issues/470),
`DriftCheckFailing` from [#1531](https://github.com/Gerrrt/HomeLab/issues/1531),
the two guest rules from [#257](https://github.com/Gerrrt/HomeLab/issues/257),
the three laptop-battery rules from
[#454](https://github.com/Gerrrt/HomeLab/issues/454),
Expand Down
9 changes: 8 additions & 1 deletion docs/runbooks/deploy-merged-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,14 @@ Grafana is minutes old.
From the Mac, after `git pull`:

- **The Alloy agent.** `make deploy-agent ARGS="<user>@<host>"` for `oracle`
and `Saruman`, which push to `prometheus`. For `phoenix` and `golem`, which
and `Saruman`, which push to `prometheus`. For `oracle`, run it on
`prometheus` instead, which holds the SOPS key the Mac does not. That
ships `prometheus`'s checkout, not the Mac's, so finish §2 there first and
check with `git -C ~/code/Gerrrt/HomeLab log -1` that it holds the merge.
Then
`ssh prometheus 'cd ~/code/Gerrrt/HomeLab && scripts/deploy-agent.sh atropos@10.0.99.30'`.
`prometheus` does not resolve `oracle` by name
([#1526](https://github.com/Gerrrt/HomeLab/issues/1526)). For `phoenix` and `golem`, which
push to the lab, add `--monitoring-host 10.0.30.40`.
- **Tokens.** For a lab host, nothing is decrypted on the Mac: export
`INGEST_TOKEN` and `INGEST_TOKEN_READER` from `secrets/lab.sops.yaml` on
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/deploy-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ Then in the UI:
1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
take up to 45 seconds on their first scrape.
2. **Prometheus → Status → Rules.** Every rule loaded, none in error. The
page lists 169: the 163 alert rules this repository counts everywhere
page lists 170: the 164 alert rules this repository counts everywhere
else, plus the six recording rules:
`homelab_suricata_expected_interface`,
`homelab_battery_runtime_seconds`, and the two directions each of
Expand Down
8 changes: 5 additions & 3 deletions docs/runbooks/fit-the-ups-battery.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,8 @@ is measured rather than assumed
([#454](https://github.com/Gerrrt/HomeLab/issues/454)). No collector was
needed: Alloy's node collector already exports `node_power_supply_*` from both
hosts, and `host.rules.yaml` reads it — `HostBatteryHealthLow` when a cell's
`charge_full` falls below 80 % of `charge_full_design` (`oracle` reads 72 %;
`charge_full` falls below 80 % of `charge_full_design` (`oracle` read 72 % until
its cell was replaced on 2026-09-29, and now reads 100 %, its design figure;
`prometheus` read 94 % after 108 cycles until its cell was replaced on
2026-09-18, and now reads 101 % at one cycle), `HostOnBattery` when the mains
adapter loses
Expand All @@ -110,8 +111,9 @@ above design, `cyclecount` 1. **The second half was proved on 2026-09-19**: the
brick pulled on a full pack, `HostOnBattery` firing for `prometheus` alone
inside three minutes with `UpsOnBattery` quiet, the host up throughout, and a
measured draw of 2.72 Ah/h — about 2.5 hours from full at the stack's load, one
measurement on one day. That closed #454. `oracle`'s half is untested and its
cell is [#531](https://github.com/Gerrrt/HomeLab/issues/531).
measurement on one day. That closed #454. `oracle`'s cell was replaced on
2026-09-29 and measured the same day at 0.54 Ah/h, about 5.2 hours from full
([#531](https://github.com/Gerrrt/HomeLab/issues/531)).

## Before you start

Expand Down
10 changes: 7 additions & 3 deletions docs/runbooks/schedule-maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,9 @@ it has no log store ([#327](https://github.com/Gerrrt/HomeLab/issues/327)).
happened OR that nothing diverged — it collects a fact the estate did not
previously have, which is how far behind this host's packages are
([#152](https://github.com/Gerrrt/HomeLab/issues/152)). It needs no root, and it
covers **this host only**: `oracle` and `Saruman` would need it shipped to them
and `morpheus` is FreeBSD with no apt at all.
covers **this host** from this table. An agent host with apt, `oracle` among
them, gets it from the installer below, and `morpheus` is FreeBSD with no apt
at all.

**Agent hosts run these collectors too, and not through this table.** A host that
runs Alloy but has no checkout of this repository — `oracle` — gets the
Expand Down Expand Up @@ -274,7 +275,10 @@ login, via `runuser -l` from a root unit that exists only to write the textfile
directory — reads the checker off the wiki's `origin/main`, and records
`homelab_drift_check_last_run_timestamp_seconds`, the exit code and the four
claim counts. `DriftCheckStopped` fires when the timestamp is more than a day
old ([#470](https://github.com/Gerrrt/HomeLab/issues/470)). Its requirement is
old ([#470](https://github.com/Gerrrt/HomeLab/issues/470)), and `DriftCheckFailing`
when the last run exited 2 or higher: a claim it could not read, or a checker
that never started. Exit 1, drift, files its own wiki issue
([#1531](https://github.com/Gerrrt/HomeLab/issues/1531)). Its requirement is
the wiki checkout's guard script, so every other host correctly reports that it
cannot run this one. It replaced a crontab entry that nothing watched.

Expand Down
Loading
Loading