Skip to content

feat(observability): alert when the wiki drift check cannot run - #1532

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/drift-check-failing-1531
Nov 16, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/drift-check-failing-1531

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Nov 16, 2026

Copy link
Copy Markdown
Owner

What changed

A new Prometheus rule, DriftCheckFailing, fires when the wiki drift check on oracle exits 2 or higher, with three promtool tests. Four stale current-state lines found by the oracle audit are corrected, and the rule counts are updated.

Why

On 2026-11-15 the drift check failed at the wiki fetch (exit 128), and nothing alerted. DriftCheckStopped reads only the timestamp, which the collector writes whatever the run found. Exit 2 (an unreadable claim) and anything above it (the checker never started) reached no one. Exit 1, drift, is left out: the checker already files its own wiki issue for that.

Doc fixes:

Closes #1531.

Blast radius

The change adds one warning-level rule on the monitoring host, and it can only fire for oracle, the one host that exports homelab_drift_check_last_exit_code. Its current value is 0 (16/16 passed at 06:40 UTC today), so it will be quiet after deploy.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

  • promtool test rules tests/host.test.yaml with prom/prometheus:v3.16.0: SUCCESS. The new cases are exit 128 (fires), exit 2 (fires) and exit 1 (quiet).

  • scripts/check_rule_tests.py: every one of 196 alert rules is selected by a test.

  • scripts/check_docs.py: docs OK, 164 Prometheus + 21 Loki rules.

  • ./scripts/lint.sh: passed, nothing skipped.

  • make validate passes. Not run; the docs-only checks and the promtool run above were run instead.

  • Deployed to the lab and confirmed working. Needs make up STACK=observability on prometheus after merge.

  • Docs updated (docs/, service README, or docs/roadmap.md)

🤖 Generated with Claude Code

DriftCheckStopped reads only the timestamp, which the collector writes
whatever the run found, so the 2026-11-15 failure (exit 128 at the wiki
fetch) raised nothing. DriftCheckFailing fires on an exit of 2 or higher.
Exit 1, drift, already files a wiki issue.

Also corrects four stale current-state lines found by the oracle audit:
oracle's cell (replaced and measured 2026-09-29), patch-state on agent
hosts, oracle's agent deploy running from prometheus, and the host rule
count.

Refs #1531

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 08:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The deployment command uses prometheus’s checkout without confirming it contains the merged revision, so an immediate redeploy can ship stale agent configuration and image pins.

1 open issue
What changed in this PR

Adds monitoring for wiki drift-check failures on oracle, covering runs that update their timestamp but cannot complete the check.

Changes:

  • Adds warning-level DriftCheckFailing for exit codes ≥2, with tests for exits 128, 2, and 1.
  • Corrects stale battery, collector-installation, and agent-deployment documentation.
  • Updates alert counts and records the changes in the changelog.
File Description
stacks/​observability/​README.md Updates the Prometheus alert count.
stacks/​observability/​prometheus/​tests/​host.test.yaml Tests failure alerts and silence for drift-only results.
stacks/​observability/​prometheus/​rules/​host.rules.yaml Adds DriftCheckFailing and troubleshooting guidance.
README.md Updates overall and Prometheus alert counts.
docs/​security.md Corrects battery status and total alert count.
docs/​runbooks/​schedule-maintenance.md Documents collector installation and the new alert.
docs/​runbooks/​fit-the-ups-battery.md Records oracle’s replacement cell and measured runtime.
docs/​runbooks/​deploy-stack.md Updates expected loaded-rule counts.
docs/​runbooks/​deploy-merged-changes.md Moves oracle agent deployment to prometheus.
docs/​observability.md Updates alert coverage and rule counts.
docs/​changelog.md Records the alert and documentation corrections.

Review effort: Balanced

Comment thread docs/runbooks/deploy-merged-changes.md Resolved
@Gerrrt
Gerrrt enabled auto-merge (squash) November 16, 2026 08:52
…rged

Refs #1531

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit de4e7d3 into main Nov 16, 2026
8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/drift-check-failing-1531 branch November 16, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Alert when the wiki drift check cannot run

2 participants