Skip to content

Feat/lab host and readiness rules - #1529

Open
Gerrrt wants to merge 4 commits into
mainfrom
feat/lab-host-and-readiness-rules
Open

Gerrrt wants to merge 4 commits into
mainfrom
feat/lab-host-and-readiness-rules

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Nov 16, 2026

Copy link
Copy Markdown
Owner

What changed

Why

Blast radius

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

  • make validate passes
  • Deployed to the lab and confirmed working
  • Docs updated (docs/, service README, or docs/roadmap.md)

Copilot AI balanced review requested due to automatic review settings November 16, 2026 05:43
@Gerrrt
Gerrrt enabled auto-merge (squash) November 16, 2026 05:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The patch-state alert misses timer failures when stale file gauges continue to be scraped; it needs a file-age check and a test covering that scenario.

1 open issue
What changed in this PR

This PR adds patch and reboot monitoring to the lab’s Prometheus and strengthens documentation checks to catch stale per-file rule counts.

Changes:

  • Adds three host-state alerts with firing and quiet test cases.
  • Adds a fourteenth documentation assertion for stack README rule counts.
  • Updates ingest documentation and clarifies that Loki’s metrics scrape does not observe readiness.
File Description
stacks/​lab/​README.md Corrects rule counts, ingest tokens, and client coverage.
stacks/​lab/​prometheus/​tests/​lab.test.yaml Adds fixtures for the three host-state alerts.
stacks/​lab/​prometheus/​rules/​lab.rules.yaml Adds security-update, reboot, and patch-state reporting alerts.
stacks/​lab/​compose.yaml Corrects the comment describing Loki readiness monitoring.
scripts/​check_docs.py Checks per-file rule counts in stack READMEs.
README.md Documents the new validation assertion.
docs/​changelog.md Records the monitoring and documentation changes.

Review effort: Balanced

…k them

The audit of alexander on 2026-11-13 found stacks/lab/README.md saying
9 and 12 rules against files holding 19 and 13, and naming three ingest
clients and six tokens when there are six clients and seven. Correct
both, and add a fourteenth check_docs assertion that reads each stack
README's per-file rule count against its rule file, since the global
totals passed throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt force-pushed the feat/lab-host-and-readiness-rules branch from 96518ef to 01e2c91 Compare November 16, 2026 09:51
Gerrrt and others added 3 commits November 16, 2026 10:00
check_stack_rule_files now takes the tree it reads, and --self-test runs
it against a temp stack: a matching count, one in words, a stale count,
a missing rule file and an ambiguous one. self-tests.sh already
discovers check_docs.py's suite, so CI runs these.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oki comment

homelab-patch-state writes reboot and apt state on alexander, odin,
garuda and eden, and no lab rule read it: the audit of alexander on
2026-11-13 found libc6 waiting on a reboot nothing could see. Add the
estate's SecurityUpdatesPending, RebootRequired and PatchStateStopped
to lab.rules.yaml with firing and quiet tests. PatchStateStopped also
requires node metrics still arriving, so eden being off between
sessions stays quiet.

The loki service's comment said the scrape job observes readiness; it
reads /metrics, which answers whatever /ready says. Say what is watched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ewritten

When homelab-patch-state.timer stops, apt-patch-state.prom stays on disk
and its gauges keep being scraped, so the presence check never fired.
The rule now also fires when node_textfile_mtime_seconds for that file is
more than 26 hours old (daily 08:00 timer, 15 min jitter), still behind
the node-metrics guard so eden being off stays quiet. Tests cover a
stale file whose gauges keep arriving and a file rewritten on schedule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt force-pushed the feat/lab-host-and-readiness-rules branch from 01e2c91 to 3747cef Compare November 16, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants