Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ documents for different readers.
`promtool`, `amtool`, `alloy fmt`, a real Loki boot to parse the LogQL rules,
dashboard-JSON and datasource checks, every dashboard's PromQL parsed, plus
`gitleaks` over the full history.
- **CI that validates the documentation too.** Thirteen assertions cross-check
- **CI that validates the documentation too.** Fourteen assertions cross-check
this prose against the configs it describes — counted claims (rules,
dashboards, panels, Alloy agents, VLANs, ADRs, runbooks), the SNMP
inventory and the compute table against `docs/network.md`, the host/stack
Expand All @@ -120,7 +120,8 @@ documents for different readers.
file's outstanding-purchase count against the roadmap's buy table, every
critical alert's `runbook_url` against the runbook and heading it names,
every relative Markdown link against the file and heading it points at, the
network diagram's PNG against its SVG, and
network diagram's PNG against its SVG, each stack README's per-file rule
counts against the rule files, and
a ban on image versions in prose — Dependabot edits only `compose.yaml`, so
a version written anywhere else is stale from the next bump. A document
that disagrees with the repository fails the build. That opening count is
Expand Down
7 changes: 7 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,13 @@ docstring gives: it is a record, not a claim about now.
`fenrir`'s uses. It is not applied yet: the change on `alexander` is the
operator's to make, by §2, and §7's `sshd -T` is the check. This follows the
same weakness closed on `golem` by the audit of 2026-11-13.
- **`stacks/lab/README.md`'s rule counts are checked now.** The audit of
`alexander` on 2026-11-13 found its layout block saying "9 rules" and
"12 rules" against files holding 19 and 13, and its ingest section naming
three clients and six tokens when there are six clients and seven tokens.
Every global count in `check_docs.py` passed throughout, because those are
totals across all rule files. A fourteenth assertion now reads each stack
README's per-file count against its rule file.
- **`garuda` is patched by hand, monthly, because nothing else patches it.**
After the audit of 2026-11-13 closed, a re-check of the box found a gap that
audit missed. `unattended-upgrades` runs daily on `garuda` and applies
Expand Down
63 changes: 62 additions & 1 deletion scripts/check_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
The device list must live in exactly one place. It is currently spread
across five ... and --check asserts the other copies still agree.

Thirteen assertions, each comparing prose against something machine-readable:
Fourteen assertions, each comparing prose against something machine-readable:

1. Counted claims rules, unit-test coverage, dashboards, panels,
Alloy agents, ADRs, runbooks, and the size of this
Expand Down Expand Up @@ -53,6 +53,11 @@
13. Diagram render docs/diagrams/current/network.png changes whenever
network.svg does, in the working tree and in the
last commit that touched either.
14. Stack rule files a stack README's "rules/<file>.rules.yaml N rules"
against the rules in that file. The totals above
are global; this is the per-file count that stood
at 9 and 12 in stacks/lab/README.md while the files
held 19 and 13.

Only present-tense documents are checked for their claims — the relative-link
assertion (12) is the exception, and scans every tracked file including the
Expand Down Expand Up @@ -1559,6 +1564,34 @@ def anchors_of(path: pathlib.Path) -> set[str]:
return problems


# A stack README's layout block names each rule file with its count beside it:
# "rules/lab.rules.yaml 19 rules". check_counts cannot see these, because
# they are per-file and its facts are totals across every file. Found by the
# audit of alexander on 2026-11-13, which read "9 rules" and "12 rules" against
# files holding 19 and 13 while every global count passed.
STACK_RULE_FILE_CLAIM = re.compile(r"rules/([\w.-]+\.rules\.yaml)[ \t]+" + COUNT + WS + r"rules\b")


def check_stack_rule_files(repo: pathlib.Path = REPO) -> list[str]:
problems = []
for readme in sorted(repo.glob("stacks/*/README.md")):
text = readme.read_text(encoding="utf-8")
rel = readme.relative_to(repo)
for match in STACK_RULE_FILE_CLAIM.finditer(text):
name, claimed = match.group(1), number(match.group(2))
n = text.count("\n", 0, match.start()) + 1
files = sorted(readme.parent.glob(f"**/rules/{name}"))
if len(files) != 1:
problems.append(
f"{rel}:{n} counts rules/{name}, which names {len(files)} files under {readme.parent.relative_to(repo)}/; want exactly one"
)
continue
have = count_alerts(files) + count_recording_rules(files)
if claimed != have:
problems.append(f"{rel}:{n} claims {claimed} rules in {name}; the file has {have}")
return problems


def check_diagram_render() -> list[str]:
"""The PNG is re-rendered whenever the SVG changes.

Expand Down Expand Up @@ -1641,6 +1674,7 @@ def main() -> int:
("critical alerts' runbook_url against docs/runbooks/", check_runbook_urls),
("relative Markdown links resolve to a file and an anchor", check_relative_links),
("network.png rendered with network.svg", check_diagram_render),
("stack READMEs' per-file rule counts against the rule files", check_stack_rule_files),
)

# The registry is the source for README's "N assertions", so adding a
Expand Down Expand Up @@ -1759,6 +1793,33 @@ def adrs(status: str, first_mentions: bool) -> list[str]:
),
]

# Stack READMEs' per-file rule counts. Built as a temp tree so the glob and
# the count are exercised, not only the regex.
with tempfile.TemporaryDirectory() as d:
repo = pathlib.Path(d)
stack = repo / "stacks" / "demo"
(stack / "prometheus" / "rules").mkdir(parents=True)
(stack / "prometheus" / "rules" / "a.rules.yaml").write_text(
"groups:\n - rules:\n - alert: One\n - alert: Two\n - record: three\n"
)

def stack_rules(claim: str) -> str:
(stack / "README.md").write_text(claim + "\n")
return " | ".join(check_stack_rule_files(repo))

cases += [
("a matching per-file rule count passes", not stack_rules("rules/a.rules.yaml 3 rules")),
("a count in words passes too", not stack_rules("rules/a.rules.yaml three rules")),
(
"a stale count is caught",
"claims 2 rules in a.rules.yaml; the file has 3" in stack_rules("rules/a.rules.yaml 2 rules"),
),
("a missing rule file is caught", "names 0 files" in stack_rules("rules/gone.rules.yaml 1 rules")),
]
(stack / "other" / "rules").mkdir(parents=True)
(stack / "other" / "rules" / "a.rules.yaml").write_text("groups: []\n")
cases.append(("an ambiguous rule file is caught", "names 2 files" in stack_rules("rules/a.rules.yaml 3 rules")))

failed = 0
for name, ok in cases:
failed += not ok
Expand Down
23 changes: 14 additions & 9 deletions stacks/lab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,16 +59,19 @@ prometheus/
prometheus.yaml six scrape jobs — this stack's four, the domain's
windows_exporter and odin's Velociraptor; no
alerting block, no file_sd
rules/lab.rules.yaml 9 rules — four for this stack watching itself,
rules/lab.rules.yaml 19 rules — four for this stack watching itself,
two for the guests' disks, three for the domain
ADR-0029 sized
rules/soc.rules.yaml 12 rules — the SOC's indexer on odin, whose health is
ADR-0029 sized, seven for golem's backups
(ADR-0053) and three for fenrir's Zeek archive
prune (#850)
rules/soc.rules.yaml 13 rules — the SOC's indexer on odin, whose health is
pushed here by stacks/soc's Alloy (ADR-0030); the
manager's agents, queues and drops, from odin's
wazuh-manager-state collector; and Zeek's conn
stream, from fenrir's Alloy (#1038)
tests/lab.test.yaml promtool unit tests; all nineteen rules, firing + quiet
tests/soc.test.yaml the same for the twelve
tests/lab.test.yaml promtool unit tests; every rule in lab.rules.yaml,
firing + quiet
tests/soc.test.yaml the same for soc.rules.yaml
loki/loki-config.yaml single-binary, filesystem, 15-day retention, no ruler
grafana/
provisioning/ two datasources + dashboard provider
Expand Down Expand Up @@ -100,9 +103,9 @@ delete API, and any query without the reader token.
token early loses nothing. A proxy that goes up before its clients refuses
their pushes until each one catches up.

1. **On `alexander`**, generate six tokens with `openssl rand -hex 32` and add
1. **On `alexander`**, generate seven tokens with `openssl rand -hex 32` and add
them with `make secrets-edit STACK=lab`. The keys are `INGEST_TOKEN_ODIN`,
`_PHOENIX`, `_FENRIR`, `_GOLEM`, `_EDEN` and `_READER`, and
`_PHOENIX`, `_FENRIR`, `_GOLEM`, `_EDEN`, `_GARUDA` and `_READER`, and
[`secrets/lab.example.yaml`](../../secrets/lab.example.yaml) says where each
one goes. Do not `make up` yet.
2. **On `odin`**, set `INGEST_TOKEN` to `INGEST_TOKEN_ODIN` with `make
Expand Down Expand Up @@ -133,8 +136,10 @@ curl -s -o /dev/null -w '%{http_code}\n' --cacert stacks/observability/alloy/ing
curl -s -o /dev/null -w '%{http_code}\n' --cacert stacks/observability/alloy/ingest-ca.pem -X POST https://10.0.30.40:9443/api/v1/write
```

The three clients must still deliver. In the lab's Grafana, `up` and the
newest log line for `odin`, `phoenix` and `fenrir` should be minutes old.
The six clients must still deliver. In the lab's Grafana, `up` and the
newest log line for `odin`, `phoenix`, `fenrir`, `golem`, `eden` and `garuda`
should be minutes old. `eden` is off between sessions, so its newest data is
from its last session.
`docker logs lab-ingest-proxy` lists any refused request, with the client its
token mapped to. A client still on its old config shows there as `-`.

Expand Down
Loading