Repository navigation
feat(observability): alert when the wiki drift check cannot run - #1532
Merged
Merged
Conversation
DriftCheckStopped reads only the timestamp, which the collector writes whatever the run found, so the 2026-11-15 failure (exit 128 at the wiki fetch) raised nothing. DriftCheckFailing fires on an exit of 2 or higher. Exit 1, drift, already files a wiki issue. Also corrects four stale current-state lines found by the oracle audit: oracle's cell (replaced and measured 2026-09-29), patch-state on agent hosts, oracle's agent deploy running from prometheus, and the host rule count. Refs #1531 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The deployment command uses prometheus’s checkout without confirming it contains the merged revision, so an immediate redeploy can ship stale agent configuration and image pins.
1 open issue
What changed in this PR
Adds monitoring for wiki drift-check failures on oracle, covering runs that update their timestamp but cannot complete the check.
Changes:
- Adds warning-level
DriftCheckFailingfor exit codes ≥2, with tests for exits 128, 2, and 1. - Corrects stale battery, collector-installation, and agent-deployment documentation.
- Updates alert counts and records the changes in the changelog.
| File | Description |
|---|---|
| stacks/observability/README.md | Updates the Prometheus alert count. |
| stacks/observability/prometheus/tests/host.test.yaml | Tests failure alerts and silence for drift-only results. |
| stacks/observability/prometheus/rules/host.rules.yaml | Adds DriftCheckFailing and troubleshooting guidance. |
| README.md | Updates overall and Prometheus alert counts. |
| docs/security.md | Corrects battery status and total alert count. |
| docs/runbooks/schedule-maintenance.md | Documents collector installation and the new alert. |
| docs/runbooks/fit-the-ups-battery.md | Records oracle’s replacement cell and measured runtime. |
| docs/runbooks/deploy-stack.md | Updates expected loaded-rule counts. |
| docs/runbooks/deploy-merged-changes.md | Moves oracle agent deployment to prometheus. |
| docs/observability.md | Updates alert coverage and rule counts. |
| docs/changelog.md | Records the alert and documentation corrections. |
Review effort: Balanced
Gerrrt
enabled auto-merge (squash)
November 16, 2026 08:52
…rged Refs #1531 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What changed
A new Prometheus rule,
DriftCheckFailing, fires when the wiki drift check onoracleexits 2 or higher, with three promtool tests. Four stale current-state lines found by theoracleaudit are corrected, and the rule counts are updated.Why
On 2026-11-15 the drift check failed at the wiki fetch (exit 128), and nothing alerted.
DriftCheckStoppedreads only the timestamp, which the collector writes whatever the run found. Exit 2 (an unreadable claim) and anything above it (the checker never started) reached no one. Exit 1, drift, is left out: the checker already files its own wiki issue for that.Doc fixes:
fit-the-ups-battery.mdandsecurity.mdhadoracle's cell as the original at 72 %, with its runtime never measured. It was replaced and measured on 2026-09-29 (oracle's laptop cell reads 72 % and its alert is silenced to 2026-10-08: buy and fit it, second in line after prometheus #531).schedule-maintenance.mdsaidpatch-statewould need shipping tooracle. The agent installer already ships it.deploy-merged-changes.md§6 ranoracle's agent deploy from the Mac. The Mac has no sops, so it can't decrypt the token (deploy-agent.sh: say why the token could not be decrypted #1526). The deploy now runs fromprometheus.observability.mdsaid "all forty-one inhost.rules.yaml". Before this PR there were already sixty; there are now sixty-one.check_docs.pydoesn't check that number.Closes #1531.
Blast radius
The change adds one warning-level rule on the monitoring host, and it can only fire for
oracle, the one host that exportshomelab_drift_check_last_exit_code. Its current value is 0 (16/16 passed at 06:40 UTC today), so it will be quiet after deploy.secrets/*.sops.yamlVerification
promtool test rules tests/host.test.yamlwithprom/prometheus:v3.16.0: SUCCESS. The new cases are exit 128 (fires), exit 2 (fires) and exit 1 (quiet).scripts/check_rule_tests.py: every one of 196 alert rules is selected by a test.scripts/check_docs.py: docs OK, 164 Prometheus + 21 Loki rules../scripts/lint.sh: passed, nothing skipped.make validatepasses. Not run; the docs-only checks and the promtool run above were run instead.Deployed to the lab and confirmed working. Needs
make up STACK=observabilityonprometheusafter merge.Docs updated (
docs/, service README, ordocs/roadmap.md)🤖 Generated with Claude Code