Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ documents for different readers.
`promtool`, `amtool`, `alloy fmt`, a real Loki boot to parse the LogQL rules,
dashboard-JSON and datasource checks, every dashboard's PromQL parsed, plus
`gitleaks` over the full history.
- **CI that validates the documentation too.** Thirteen assertions cross-check
- **CI that validates the documentation too.** Fourteen assertions cross-check
this prose against the configs it describes — counted claims (rules,
dashboards, panels, Alloy agents, VLANs, ADRs, runbooks), the SNMP
inventory and the compute table against `docs/network.md`, the host/stack
Expand All @@ -120,7 +120,8 @@ documents for different readers.
file's outstanding-purchase count against the roadmap's buy table, every
critical alert's `runbook_url` against the runbook and heading it names,
every relative Markdown link against the file and heading it points at, the
network diagram's PNG against its SVG, and
network diagram's PNG against its SVG, each stack README's per-file rule
counts against the rule files, and
a ban on image versions in prose — Dependabot edits only `compose.yaml`, so
a version written anywhere else is stale from the next bump. A document
that disagrees with the repository fails the build. That opening count is
Expand Down
19 changes: 19 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,25 @@ docstring gives: it is a record, not a claim about now.
`fenrir`'s uses. It is not applied yet: the change on `alexander` is the
operator's to make, by §2, and §7's `sshd -T` is the check. This follows the
same weakness closed on `golem` by the audit of 2026-11-13.
- **The lab's Prometheus now reads its guests' patch and reboot state.**
`homelab-patch-state` has written it on `alexander`, `odin`, `garuda` and
`eden` all along, and no lab rule read it. The audit of `alexander` on
2026-11-13 found libc6 waiting for a reboot that nothing could see, and the
garuda entry below found the same gap. `lab.rules.yaml` gains the estate's
`SecurityUpdatesPending`, `RebootRequired` and `PatchStateStopped`, with
tests. `PatchStateStopped` also requires the host to still be pushing node
metrics, so `eden` being off between sessions stays quiet, and fires when
`apt-patch-state.prom` has not been rewritten in 26 hours, because a
stopped timer leaves the file's last gauges being served as current. None of them
pages (ADR-0020). The same audit found `compose.yaml` saying the `loki`
scrape job observes readiness. It does not, and the comment now says so.
- **`stacks/lab/README.md`'s rule counts are checked now.** The audit of
`alexander` on 2026-11-13 found its layout block saying "9 rules" and
"12 rules" against files holding 19 and 13, and its ingest section naming
three clients and six tokens when there are six clients and seven tokens.
Every global count in `check_docs.py` passed throughout, because those are
totals across all rule files. A fourteenth assertion now reads each stack
README's per-file count against its rule file.
- **`garuda` is patched by hand, monthly, because nothing else patches it.**
After the audit of 2026-11-13 closed, a re-check of the box found a gap that
audit missed. `unattended-upgrades` runs daily on `garuda` and applies
Expand Down
63 changes: 62 additions & 1 deletion scripts/check_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
The device list must live in exactly one place. It is currently spread
across five ... and --check asserts the other copies still agree.

Thirteen assertions, each comparing prose against something machine-readable:
Fourteen assertions, each comparing prose against something machine-readable:

1. Counted claims rules, unit-test coverage, dashboards, panels,
Alloy agents, ADRs, runbooks, and the size of this
Expand Down Expand Up @@ -53,6 +53,11 @@
13. Diagram render docs/diagrams/current/network.png changes whenever
network.svg does, in the working tree and in the
last commit that touched either.
14. Stack rule files a stack README's "rules/<file>.rules.yaml N rules"
against the rules in that file. The totals above
are global; this is the per-file count that stood
at 9 and 12 in stacks/lab/README.md while the files
held 19 and 13.

Only present-tense documents are checked for their claims — the relative-link
assertion (12) is the exception, and scans every tracked file including the
Expand Down Expand Up @@ -1559,6 +1564,34 @@ def anchors_of(path: pathlib.Path) -> set[str]:
return problems


# A stack README's layout block names each rule file with its count beside it:
# "rules/lab.rules.yaml 19 rules". check_counts cannot see these, because
# they are per-file and its facts are totals across every file. Found by the
# audit of alexander on 2026-11-13, which read "9 rules" and "12 rules" against
# files holding 19 and 13 while every global count passed.
STACK_RULE_FILE_CLAIM = re.compile(r"rules/([\w.-]+\.rules\.yaml)[ \t]+" + COUNT + WS + r"rules\b")


def check_stack_rule_files(repo: pathlib.Path = REPO) -> list[str]:
problems = []
for readme in sorted(repo.glob("stacks/*/README.md")):
text = readme.read_text(encoding="utf-8")
rel = readme.relative_to(repo)
for match in STACK_RULE_FILE_CLAIM.finditer(text):
name, claimed = match.group(1), number(match.group(2))
n = text.count("\n", 0, match.start()) + 1
files = sorted(readme.parent.glob(f"**/rules/{name}"))
if len(files) != 1:
problems.append(
f"{rel}:{n} counts rules/{name}, which names {len(files)} files under {readme.parent.relative_to(repo)}/; want exactly one"
)
continue
have = count_alerts(files) + count_recording_rules(files)
if claimed != have:
problems.append(f"{rel}:{n} claims {claimed} rules in {name}; the file has {have}")
return problems


def check_diagram_render() -> list[str]:
"""The PNG is re-rendered whenever the SVG changes.

Expand Down Expand Up @@ -1641,6 +1674,7 @@ def main() -> int:
("critical alerts' runbook_url against docs/runbooks/", check_runbook_urls),
("relative Markdown links resolve to a file and an anchor", check_relative_links),
("network.png rendered with network.svg", check_diagram_render),
("stack READMEs' per-file rule counts against the rule files", check_stack_rule_files),
)

# The registry is the source for README's "N assertions", so adding a
Expand Down Expand Up @@ -1759,6 +1793,33 @@ def adrs(status: str, first_mentions: bool) -> list[str]:
),
]

# Stack READMEs' per-file rule counts. Built as a temp tree so the glob and
# the count are exercised, not only the regex.
with tempfile.TemporaryDirectory() as d:
repo = pathlib.Path(d)
stack = repo / "stacks" / "demo"
(stack / "prometheus" / "rules").mkdir(parents=True)
(stack / "prometheus" / "rules" / "a.rules.yaml").write_text(
"groups:\n - rules:\n - alert: One\n - alert: Two\n - record: three\n"
)

def stack_rules(claim: str) -> str:
(stack / "README.md").write_text(claim + "\n")
return " | ".join(check_stack_rule_files(repo))

cases += [
("a matching per-file rule count passes", not stack_rules("rules/a.rules.yaml 3 rules")),
("a count in words passes too", not stack_rules("rules/a.rules.yaml three rules")),
(
"a stale count is caught",
"claims 2 rules in a.rules.yaml; the file has 3" in stack_rules("rules/a.rules.yaml 2 rules"),
),
("a missing rule file is caught", "names 0 files" in stack_rules("rules/gone.rules.yaml 1 rules")),
]
(stack / "other" / "rules").mkdir(parents=True)
(stack / "other" / "rules" / "a.rules.yaml").write_text("groups: []\n")
cases.append(("an ambiguous rule file is caught", "names 2 files" in stack_rules("rules/a.rules.yaml 3 rules")))

failed = 0
for name, ok in cases:
failed += not ok
Expand Down
26 changes: 16 additions & 10 deletions stacks/lab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,16 +59,20 @@ prometheus/
prometheus.yaml six scrape jobs — this stack's four, the domain's
windows_exporter and odin's Velociraptor; no
alerting block, no file_sd
rules/lab.rules.yaml 9 rules — four for this stack watching itself,
two for the guests' disks, three for the domain
ADR-0029 sized
rules/soc.rules.yaml 12 rules — the SOC's indexer on odin, whose health is
rules/lab.rules.yaml 22 rules — four for this stack watching itself,
two for the guests' disks, three for their patch
and reboot state, three for the domain
ADR-0029 sized, seven for golem's backups
(ADR-0053) and three for fenrir's Zeek archive
prune (#850)
rules/soc.rules.yaml 13 rules — the SOC's indexer on odin, whose health is
pushed here by stacks/soc's Alloy (ADR-0030); the
manager's agents, queues and drops, from odin's
wazuh-manager-state collector; and Zeek's conn
stream, from fenrir's Alloy (#1038)
tests/lab.test.yaml promtool unit tests; all nineteen rules, firing + quiet
tests/soc.test.yaml the same for the twelve
tests/lab.test.yaml promtool unit tests; every rule in lab.rules.yaml,
firing + quiet
tests/soc.test.yaml the same for soc.rules.yaml
loki/loki-config.yaml single-binary, filesystem, 15-day retention, no ruler
grafana/
provisioning/ two datasources + dashboard provider
Expand Down Expand Up @@ -100,9 +104,9 @@ delete API, and any query without the reader token.
token early loses nothing. A proxy that goes up before its clients refuses
their pushes until each one catches up.

1. **On `alexander`**, generate six tokens with `openssl rand -hex 32` and add
1. **On `alexander`**, generate seven tokens with `openssl rand -hex 32` and add
them with `make secrets-edit STACK=lab`. The keys are `INGEST_TOKEN_ODIN`,
`_PHOENIX`, `_FENRIR`, `_GOLEM`, `_EDEN` and `_READER`, and
`_PHOENIX`, `_FENRIR`, `_GOLEM`, `_EDEN`, `_GARUDA` and `_READER`, and
[`secrets/lab.example.yaml`](../../secrets/lab.example.yaml) says where each
one goes. Do not `make up` yet.
2. **On `odin`**, set `INGEST_TOKEN` to `INGEST_TOKEN_ODIN` with `make
Expand Down Expand Up @@ -133,8 +137,10 @@ curl -s -o /dev/null -w '%{http_code}\n' --cacert stacks/observability/alloy/ing
curl -s -o /dev/null -w '%{http_code}\n' --cacert stacks/observability/alloy/ingest-ca.pem -X POST https://10.0.30.40:9443/api/v1/write
```

The three clients must still deliver. In the lab's Grafana, `up` and the
newest log line for `odin`, `phoenix` and `fenrir` should be minutes old.
The six clients must still deliver. In the lab's Grafana, `up` and the
newest log line for `odin`, `phoenix`, `fenrir`, `golem`, `eden` and `garuda`
should be minutes old. `eden` is off between sessions, so its newest data is
from its last session.
`docker logs lab-ingest-proxy` lists any refused request, with the client its
token mapped to. A client still on its old config shows there as `-`.

Expand Down
7 changes: 5 additions & 2 deletions stacks/lab/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,11 @@ services:
# ---------------------------------------------------------------------------
# Log store. No healthcheck, for the reason written out against the `loki`
# service in stacks/observability/compose.yaml: the image is distroless and
# holds no shell, wget or curl to probe with. Readiness is observed by the
# `loki` scrape job in prometheus.yaml instead.
# holds no shell, wget or curl to probe with. Nothing here observes /ready:
# the `loki` scrape job reads /metrics, which answers whatever /ready says,
# and the audit of alexander on 2026-11-13 found /ready returning 503 for
# four days while ingest ran normally. What is watched is the outcome —
# InstanceDown on the scrape, and LokiIngestionStalled on lines arriving.
# ---------------------------------------------------------------------------
loki:
<<: *service-defaults
Expand Down
99 changes: 97 additions & 2 deletions stacks/lab/prometheus/rules/lab.rules.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
---
# The lab stack watching itself.
#
# Nineteen rules, and the number is still the point. The estate has 48; ADR-0007
# Twenty-two rules, and the number is still the point. The estate has 48; ADR-0007
# names giving up "Loki, the dashboards and 40 alert rules" as the price of
# keeping lab telemetry out of the estate, and this file does not pretend to
# replace them. Four cover this stack's own working — the things that would make
# every other signal here silently wrong. Two cover the disks of the guests that
# push here, in the detail only their own node_exporter has; the estate pages on
# the same disks, read another way (ADR-0070). Three cover the domain ADR-0029 sized,
# the same disks, read another way (ADR-0070). Three cover the same guests'
# patch state, which only reaches this Prometheus. Three cover the domain ADR-0029 sized,
# and they are here rather than in a stack of their own because this Prometheus
# is the only thing on the segment that scrapes it. Seven cover golem's backups
# (ADR-0053), because a verify nobody watches is a green check with nothing
Expand Down Expand Up @@ -171,6 +172,100 @@ groups:
annotations:
summary: "{{ $labels.instance }}:{{ $labels.mountpoint }} below 10% free"

# The guests' patch state. homelab-patch-state writes it on every apt
# guest that pushes here (scripts/install-agent-collectors.sh), and until
# 2026-11-15 nothing in this file read it: the audit of alexander on
# 2026-11-13 found libc6 asking for a reboot that no rule could see, and
# the 2026-11-15 changelog found the same on garuda. The estate's three
# rules in host.rules.yaml cannot see these guests either, because none
# of them pushes to VLAN 99 (ADR-0007).
#
# Same names, expressions and windows as the estate's three, per the
# header; host.rules.yaml carries the reasoning for each window. One
# difference, in PatchStateStopped, is stated against it.
- alert: SecurityUpdatesPending
expr: homelab_apt_security_upgrades_pending > 0
for: 7d
labels:
component: host
severity: warning
category: security
annotations:
summary: "{{ $labels.host }} has had {{ $value }} security update(s) pending for a week"
description: >-
Unattended-upgrades should have applied these. A week of them sitting
there means they are held, phased, or waiting on a reboot. Check with
`apt list --upgradable` and
`systemctl status unattended-upgrades.service`. On garuda a 0 is not
good news: Kali has no security pocket (schedule-maintenance.md).

- alert: RebootRequired
expr: homelab_reboot_required == 1
for: 3d
labels:
component: host
severity: warning
category: security
annotations:
summary: "{{ $labels.host }} has needed a reboot for three days"
description: >-
/var/run/reboot-required.pkgs names which packages asked, and
homelab_reboot_required_packages counts them. Pick a window outside
an exercise: rebooting alexander blinds the lab for its duration,
and rebooting odin stops the SOC.

# The estate's expression, plus two conditions.
#
# The host must still be pushing node metrics. eden is off between
# sessions (ADR-0029's per-session shape), and when it is off its patch
# state stops with everything else. Without the last line that is a
# permanently recurring alert for a correct state, which ADR-0017
# refuses. A guest that is off entirely is the estate's business
# (ADR-0028); this asks only about a guest that is up and has stopped
# collecting.
#
# And the file must still be rewritten. Presence alone misses the
# commonest way this breaks: when homelab-patch-state.timer stops,
# apt-patch-state.prom stays on disk and the textfile collector keeps
# re-serving its last gauges, so the series never goes absent. The
# wiki drift check in the estate's host.rules.yaml says the same of its
# own file. Twenty-six hours against a daily 08:00 timer with fifteen
# minutes of jitter allows one late run and not a missed one. eden
# coming back after days off has a stale file until Persistent=true
# runs the missed timer at boot, inside the fifteen-minute jitter, and
# `for: 30m` waits that out.
- alert: PatchStateStopped
expr: >-
(
(
count by (host) (present_over_time(homelab_apt_upgrades_pending[1d] offset 6h))
unless
count by (host) (present_over_time(homelab_apt_upgrades_pending[6h]))
)
or
(
time() - max by (host) (node_textfile_mtime_seconds{file=~".*/apt-patch-state[.]prom"})
> 26 * 3600
)
)
and on (host)
count by (host) (present_over_time(node_boot_time_seconds[6h]))
for: 30m
labels:
component: host
severity: warning
category: security
annotations:
summary: "{{ $labels.host }} has stopped reporting patch state"
description: >-
Its patch state has either stopped arriving or is being re-served
from an apt-patch-state.prom not rewritten in over 26 hours, while
its node metrics still arrive. SecurityUpdatesPending and
RebootRequired are both blind to this host until it is fixed.
Check `systemctl status homelab-patch-state.timer` on the host and
that /var/lib/node_exporter/textfile_collector/apt-patch-state.prom
is being rewritten daily and is world-readable.

# -----------------------------------------------------------------------
# The domain (ADR-0029). Scraped, never pushed — publishing a remote-write
# receiver on the segment that exists to hold attackers would hand it a
Expand Down
Loading
Loading