Repository navigation
Alert only when the health check fails from three runners - #52
Merged
Merged
Conversation
GitHub-hosted runners are sometimes blackholed by the network in front of the VPS, which from one runner looks the same as an outage. The check now probes from a runner, re-checks from a second after 60 s and a third after 120 s when the previous one saw the app down, and fails the workflow only if all three did. The probe lives in a reusable health-probe.yml that reports up or down with the reason, such as a DNS failure, a TCP timeout or a non-200 answer.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The workflow mislabels every curl exit 28 as a TCP connection timeout, potentially attributing a stalled application response to a runner or network problem.
1 open issue
What changed in this PR
This PR reduces false outage alerts caused by individual GitHub-hosted runners being unable to reach production. It replaces the single-runner health check with three sequential probes on separate runners.
Changes:
- Adds a reusable probe with retries, health-response validation, diagnostic outputs, and egress-IP logging.
- Rechecks failures after 60 seconds, then confirms after another 120 seconds.
- Reports an outage only when all three probes return
down.
| File | Description |
|---|---|
.github/workflows/health-probe.yml |
Adds the reusable health probe and diagnostic reporting. |
.github/workflows/health-check.yml |
Orchestrates the three-stage check and final outage alert. |
Review effort: Balanced
| case "$1" in | ||
| 6) echo "DNS lookup failed" ;; | ||
| 7) echo "could not connect to the host on port 443" ;; | ||
| 28) echo "no answer: the TCP connection timed out" ;; |
Exit 28 also fires when the 30 s request limit expires after the connection is up, so it can't be called a TCP connect timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub-hosted runners are sometimes blackholed by the network in front of the VPS. From a single runner that looks the same as an outage. Between 2026-08-20 and 2026-09-22 the old check failed 30 times, mostly as isolated runs between successes.
health-probe.yml(new, reusable): probes/api/healthup to three times and reportsupordownwith the reason (DNS failure, TCP timeout, TLS failure, non-200 answer) through outputs instead of the exit code. It also logs the runner's egress IP.health-check.yml: probe, then re-check from a fresh runner after 60 s, then confirm from a third after 120 s, each only if the previous stage saw the app down. The workflow fails and notifies only when all three did.Note: GitHub disabled the scheduled Health Check workflow for inactivity, so it needs re-enabling after merge.