Repository navigation
feat(telemetry): add blackbox prober, multi-window SLO alerts, and self-hosted ntfy - #66
Merged
Merged
Conversation
…lf-hosted ntfy - Add Prometheus Blackbox Exporter service for outside-in uptime probing of research endpoints - Implement Google SRE multi-window burn rate alerts (fast and slow burn) for 99.5% availability targets - Add BlackboxProbeFailed critical alert for probed endpoint outages - Add self-hosted ntfy service with zero-trust ACLs, persistent cache, and Cloudflare tunnel ingress
…gelog - Add 18-rule alert catalog reference table in RUNBOOK.md - Document self-hosted ntfy user provisioning and Grafana contact point setup - Add NTFY_BASE_URL and ntfy webhook documentation to .env.example - Record changes under [Unreleased] in CHANGELOG.md
- Bind ntfy metrics endpoint to internal port 9090 instead of public port 80 - Prevents unauthenticated internal telemetry exposure via the public Cloudflare tunnel
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds outside-in uptime probing, multi-window SLO burn rate alerts, and self-hosted alert delivery via ntfy so we can track and alert on service availability without third-party push services. Closes #19.
prom/blackbox-exporter:v0.26.0) withhttp_2xxprobing for internal and external endpoints.ErrorBudgetBurnFastandErrorBudgetBurnSlow) targeting 99.5% availability, plus aBlackboxProbeFailedcritical alert.ntfy(binwiederhier/ntfy:v2.11.0) with zero-trust topic permissions, internal Prometheus metrics on:9090, and Cloudflare tunnel ingress.scripts/smoke.sh,justfilevalidation, and documentation indocs/RUNBOOK.mdand.env.example.Tests:
just checkpasses with config validation for prometheus, blackbox-exporter, otel-collector, loki, and tempo.just smokepasses end-to-end against the local sandbox with ntfy health and blackbox probe assertions.