Repository navigation
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The patch-state alert misses timer failures when stale file gauges continue to be scraped; it needs a file-age check and a test covering that scenario.
1 open issue
What changed in this PR
This PR adds patch and reboot monitoring to the lab’s Prometheus and strengthens documentation checks to catch stale per-file rule counts.
Changes:
- Adds three host-state alerts with firing and quiet test cases.
- Adds a fourteenth documentation assertion for stack README rule counts.
- Updates ingest documentation and clarifies that Loki’s metrics scrape does not observe readiness.
| File | Description |
|---|---|
stacks/lab/README.md |
Corrects rule counts, ingest tokens, and client coverage. |
stacks/lab/prometheus/tests/lab.test.yaml |
Adds fixtures for the three host-state alerts. |
stacks/lab/prometheus/rules/lab.rules.yaml |
Adds security-update, reboot, and patch-state reporting alerts. |
stacks/lab/compose.yaml |
Corrects the comment describing Loki readiness monitoring. |
scripts/check_docs.py |
Checks per-file rule counts in stack READMEs. |
README.md |
Documents the new validation assertion. |
docs/changelog.md |
Records the monitoring and documentation changes. |
Review effort: Balanced
…k them The audit of alexander on 2026-11-13 found stacks/lab/README.md saying 9 and 12 rules against files holding 19 and 13, and naming three ingest clients and six tokens when there are six clients and seven. Correct both, and add a fourteenth check_docs assertion that reads each stack README's per-file rule count against its rule file, since the global totals passed throughout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt
force-pushed
the
feat/lab-host-and-readiness-rules
branch
from
November 16, 2026 09:51
96518ef to
01e2c91
Compare
check_stack_rule_files now takes the tree it reads, and --self-test runs it against a temp stack: a matching count, one in words, a stale count, a missing rule file and an ambiguous one. self-tests.sh already discovers check_docs.py's suite, so CI runs these. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oki comment homelab-patch-state writes reboot and apt state on alexander, odin, garuda and eden, and no lab rule read it: the audit of alexander on 2026-11-13 found libc6 waiting on a reboot nothing could see. Add the estate's SecurityUpdatesPending, RebootRequired and PatchStateStopped to lab.rules.yaml with firing and quiet tests. PatchStateStopped also requires node metrics still arriving, so eden being off between sessions stays quiet. The loki service's comment said the scrape job observes readiness; it reads /metrics, which answers whatever /ready says. Say what is watched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ewritten When homelab-patch-state.timer stops, apt-patch-state.prom stays on disk and its gauges keep being scraped, so the presence check never fired. The rule now also fires when node_textfile_mtime_seconds for that file is more than 26 hours old (daily 08:00 timer, 15 min jitter), still behind the node-metrics guard so eden being off stays quiet. Tests cover a stale file whose gauges keep arriving and a file rewritten on schedule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt
force-pushed
the
feat/lab-host-and-readiness-rules
branch
from
November 16, 2026 10:08
01e2c91 to
3747cef
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What changed
Why
Blast radius
secrets/*.sops.yamlVerification
make validatepassesdocs/, service README, ordocs/roadmap.md)