diff --git a/README.md b/README.md index 2189fa62..061ab603 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,7 @@ documents for different readers. `promtool`, `amtool`, `alloy fmt`, a real Loki boot to parse the LogQL rules, dashboard-JSON and datasource checks, every dashboard's PromQL parsed, plus `gitleaks` over the full history. -- **CI that validates the documentation too.** Thirteen assertions cross-check +- **CI that validates the documentation too.** Fourteen assertions cross-check this prose against the configs it describes — counted claims (rules, dashboards, panels, Alloy agents, VLANs, ADRs, runbooks), the SNMP inventory and the compute table against `docs/network.md`, the host/stack @@ -120,7 +120,8 @@ documents for different readers. file's outstanding-purchase count against the roadmap's buy table, every critical alert's `runbook_url` against the runbook and heading it names, every relative Markdown link against the file and heading it points at, the - network diagram's PNG against its SVG, and + network diagram's PNG against its SVG, each stack README's per-file rule + counts against the rule files, and a ban on image versions in prose — Dependabot edits only `compose.yaml`, so a version written anywhere else is stale from the next bump. A document that disagrees with the repository fails the build. That opening count is diff --git a/docs/changelog.md b/docs/changelog.md index 0e6ec4ad..3d40a51f 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -282,6 +282,13 @@ docstring gives: it is a record, not a claim about now. `fenrir`'s uses. It is not applied yet: the change on `alexander` is the operator's to make, by §2, and §7's `sshd -T` is the check. This follows the same weakness closed on `golem` by the audit of 2026-11-13. +- **`stacks/lab/README.md`'s rule counts are checked now.** The audit of + `alexander` on 2026-11-13 found its layout block saying "9 rules" and + "12 rules" against files holding 19 and 13, and its ingest section naming + three clients and six tokens when there are six clients and seven tokens. + Every global count in `check_docs.py` passed throughout, because those are + totals across all rule files. A fourteenth assertion now reads each stack + README's per-file count against its rule file. - **`garuda` is patched by hand, monthly, because nothing else patches it.** After the audit of 2026-11-13 closed, a re-check of the box found a gap that audit missed. `unattended-upgrades` runs daily on `garuda` and applies diff --git a/scripts/check_docs.py b/scripts/check_docs.py index e1f5aafe..81cb163b 100755 --- a/scripts/check_docs.py +++ b/scripts/check_docs.py @@ -17,7 +17,7 @@ The device list must live in exactly one place. It is currently spread across five ... and --check asserts the other copies still agree. -Thirteen assertions, each comparing prose against something machine-readable: +Fourteen assertions, each comparing prose against something machine-readable: 1. Counted claims rules, unit-test coverage, dashboards, panels, Alloy agents, ADRs, runbooks, and the size of this @@ -53,6 +53,11 @@ 13. Diagram render docs/diagrams/current/network.png changes whenever network.svg does, in the working tree and in the last commit that touched either. + 14. Stack rule files a stack README's "rules/.rules.yaml N rules" + against the rules in that file. The totals above + are global; this is the per-file count that stood + at 9 and 12 in stacks/lab/README.md while the files + held 19 and 13. Only present-tense documents are checked for their claims — the relative-link assertion (12) is the exception, and scans every tracked file including the @@ -1559,6 +1564,34 @@ def anchors_of(path: pathlib.Path) -> set[str]: return problems +# A stack README's layout block names each rule file with its count beside it: +# "rules/lab.rules.yaml 19 rules". check_counts cannot see these, because +# they are per-file and its facts are totals across every file. Found by the +# audit of alexander on 2026-11-13, which read "9 rules" and "12 rules" against +# files holding 19 and 13 while every global count passed. +STACK_RULE_FILE_CLAIM = re.compile(r"rules/([\w.-]+\.rules\.yaml)[ \t]+" + COUNT + WS + r"rules\b") + + +def check_stack_rule_files(repo: pathlib.Path = REPO) -> list[str]: + problems = [] + for readme in sorted(repo.glob("stacks/*/README.md")): + text = readme.read_text(encoding="utf-8") + rel = readme.relative_to(repo) + for match in STACK_RULE_FILE_CLAIM.finditer(text): + name, claimed = match.group(1), number(match.group(2)) + n = text.count("\n", 0, match.start()) + 1 + files = sorted(readme.parent.glob(f"**/rules/{name}")) + if len(files) != 1: + problems.append( + f"{rel}:{n} counts rules/{name}, which names {len(files)} files under {readme.parent.relative_to(repo)}/; want exactly one" + ) + continue + have = count_alerts(files) + count_recording_rules(files) + if claimed != have: + problems.append(f"{rel}:{n} claims {claimed} rules in {name}; the file has {have}") + return problems + + def check_diagram_render() -> list[str]: """The PNG is re-rendered whenever the SVG changes. @@ -1641,6 +1674,7 @@ def main() -> int: ("critical alerts' runbook_url against docs/runbooks/", check_runbook_urls), ("relative Markdown links resolve to a file and an anchor", check_relative_links), ("network.png rendered with network.svg", check_diagram_render), + ("stack READMEs' per-file rule counts against the rule files", check_stack_rule_files), ) # The registry is the source for README's "N assertions", so adding a @@ -1759,6 +1793,33 @@ def adrs(status: str, first_mentions: bool) -> list[str]: ), ] + # Stack READMEs' per-file rule counts. Built as a temp tree so the glob and + # the count are exercised, not only the regex. + with tempfile.TemporaryDirectory() as d: + repo = pathlib.Path(d) + stack = repo / "stacks" / "demo" + (stack / "prometheus" / "rules").mkdir(parents=True) + (stack / "prometheus" / "rules" / "a.rules.yaml").write_text( + "groups:\n - rules:\n - alert: One\n - alert: Two\n - record: three\n" + ) + + def stack_rules(claim: str) -> str: + (stack / "README.md").write_text(claim + "\n") + return " | ".join(check_stack_rule_files(repo)) + + cases += [ + ("a matching per-file rule count passes", not stack_rules("rules/a.rules.yaml 3 rules")), + ("a count in words passes too", not stack_rules("rules/a.rules.yaml three rules")), + ( + "a stale count is caught", + "claims 2 rules in a.rules.yaml; the file has 3" in stack_rules("rules/a.rules.yaml 2 rules"), + ), + ("a missing rule file is caught", "names 0 files" in stack_rules("rules/gone.rules.yaml 1 rules")), + ] + (stack / "other" / "rules").mkdir(parents=True) + (stack / "other" / "rules" / "a.rules.yaml").write_text("groups: []\n") + cases.append(("an ambiguous rule file is caught", "names 2 files" in stack_rules("rules/a.rules.yaml 3 rules"))) + failed = 0 for name, ok in cases: failed += not ok diff --git a/stacks/lab/README.md b/stacks/lab/README.md index c915717e..872643be 100644 --- a/stacks/lab/README.md +++ b/stacks/lab/README.md @@ -59,16 +59,19 @@ prometheus/ prometheus.yaml six scrape jobs — this stack's four, the domain's windows_exporter and odin's Velociraptor; no alerting block, no file_sd - rules/lab.rules.yaml 9 rules — four for this stack watching itself, + rules/lab.rules.yaml 19 rules — four for this stack watching itself, two for the guests' disks, three for the domain - ADR-0029 sized - rules/soc.rules.yaml 12 rules — the SOC's indexer on odin, whose health is + ADR-0029 sized, seven for golem's backups + (ADR-0053) and three for fenrir's Zeek archive + prune (#850) + rules/soc.rules.yaml 13 rules — the SOC's indexer on odin, whose health is pushed here by stacks/soc's Alloy (ADR-0030); the manager's agents, queues and drops, from odin's wazuh-manager-state collector; and Zeek's conn stream, from fenrir's Alloy (#1038) - tests/lab.test.yaml promtool unit tests; all nineteen rules, firing + quiet - tests/soc.test.yaml the same for the twelve + tests/lab.test.yaml promtool unit tests; every rule in lab.rules.yaml, + firing + quiet + tests/soc.test.yaml the same for soc.rules.yaml loki/loki-config.yaml single-binary, filesystem, 15-day retention, no ruler grafana/ provisioning/ two datasources + dashboard provider @@ -100,9 +103,9 @@ delete API, and any query without the reader token. token early loses nothing. A proxy that goes up before its clients refuses their pushes until each one catches up. -1. **On `alexander`**, generate six tokens with `openssl rand -hex 32` and add +1. **On `alexander`**, generate seven tokens with `openssl rand -hex 32` and add them with `make secrets-edit STACK=lab`. The keys are `INGEST_TOKEN_ODIN`, - `_PHOENIX`, `_FENRIR`, `_GOLEM`, `_EDEN` and `_READER`, and + `_PHOENIX`, `_FENRIR`, `_GOLEM`, `_EDEN`, `_GARUDA` and `_READER`, and [`secrets/lab.example.yaml`](../../secrets/lab.example.yaml) says where each one goes. Do not `make up` yet. 2. **On `odin`**, set `INGEST_TOKEN` to `INGEST_TOKEN_ODIN` with `make @@ -133,8 +136,10 @@ curl -s -o /dev/null -w '%{http_code}\n' --cacert stacks/observability/alloy/ing curl -s -o /dev/null -w '%{http_code}\n' --cacert stacks/observability/alloy/ingest-ca.pem -X POST https://10.0.30.40:9443/api/v1/write ``` -The three clients must still deliver. In the lab's Grafana, `up` and the -newest log line for `odin`, `phoenix` and `fenrir` should be minutes old. +The six clients must still deliver. In the lab's Grafana, `up` and the +newest log line for `odin`, `phoenix`, `fenrir`, `golem`, `eden` and `garuda` +should be minutes old. `eden` is off between sessions, so its newest data is +from its last session. `docker logs lab-ingest-proxy` lists any refused request, with the client its token mapped to. A client still on its old config shows there as `-`.