Skip to content

fix(metrics): publish no restore-drill verdict before a drill has ever run - #1203

Open
HugoXK wants to merge 2 commits into
mainfrom
dev4/1199-drill-alert-fresh-install
Open

HugoXK wants to merge 2 commits into
mainfrom
dev4/1199-drill-alert-fresh-install

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Written-by: Fledge Platform Dev-4

Fixes #1199

The defect

Shape (a): the source published a verdict it did not have. With no backup-drill.json, publishBackupMetrics set fledge_restore_drill_ok 0 unconditionally (src/lib/backupMetrics.ts:78), and the shipped FledgeRestoreDrillFailing rule selects exactly that value (deploy/prometheus/fledge-alerts.yml:115, expr: fledge_restore_drill_ok == 0, for: 30m). So a host that had never drilled paged half an hour after its first scrape and kept paging. docs/observability.md promised a fresh install says nothing, and the in-app ops alert was already guarded by readings.drill !== null (src/jobs/service.ts:3806); only the scrape and the rule disagreed with them.

What changed

  • src/lib/backupMetrics.ts: when readings.drill === null, the verdict is cleared instead of set to 0 — the same treatment fledge_backup_age_hours, fledge_restore_drill_age_hours and fledge_clock_skew_seconds already take. A record that exists still publishes 0 when it failed or aged past BACKUP_EXPECTED_HOURS, beside its age series.
  • The rule's expression is unchanged. Absence means Prometheus has no series to compare, so the fresh-install case disappears without a presence guard every future rule would have to remember. Only the rule's comment changed, because it claimed a missing record "reads 0 like a fresh install does".
  • docs/observability.md and docs/deploy.md now say the verdict and the age series are absent before the first drill, and why a 0 is still the failed/aged drill case.

Why (a) rather than (b)

(a) fixes the producer, so the invariant holds for every consumer and every future rule; (b) would have made the single shipped rule correct while leaving a 0 on the scrape that reads as "drilled and failed" to anything else that scrapes it. I checked the consumers before choosing: the only reader of the series is /metrics and this one rule; the /admin card reads readings.drillOk from the same reader, not the gauge, and the in-app ops alert is already guarded by readings.drill !== null. Nothing else in the repo reads fledge_restore_drill_ok.

The case that must not regress

A drill that ran and failed, and a drill that passed but aged out, both still publish fledge_restore_drill_ok 0 and an age series. test/backupMetrics.test.ts now pins both, so silencing the fresh install cannot silence the drill that stopped running.

Ran:

  • npm run check — pass (lint 11 baseline warnings in 8 entries, typecheck, 3265/3265 tests, build)
  • node scripts/version.mjs check — pass (version: 0.4.4 ok)
  • npx tsx --test test/backupMetrics.test.ts at the base f1c48ed3 — red: no drill record is unknown, not a zero verdict, expected: ~ (absent), actual: 0
  • npm run mock:snapshot — no diff (no view change)
    Not run: Docker image, a real Prometheus with a 30-minute for: window, a production deploy.

Uncertainty and residual risk

  • The rule is severity: warning (deploy/prometheus/fledge-alerts.yml:118), so "pages" only holds through Alertmanager routing; the change does not touch severity.
  • A record that exists but whose at cannot be parsed still publishes 0 with no age series, so it alerts. That is deliberate — a record exists and cannot prove a passing drill — but it means a malformed record still counts as a failure rather than as unknown.
  • A scrape reads the gauge list at scrape time; between the drill file being deleted and the next scrape, a previously published 0 could still be in the scrape response. The series is rebuilt per scrape, so the window is one request.
  • No new metric, exporter, scheduler or rule; FledgeBackupStale, the 26-hour threshold, the in-app ops alert and the /admin card are untouched.

Alternatives considered

  • Keep the gauge and add absent_over_time-style presence logic to the rule (rejected: every future consumer of the series keeps seeing a misleading 0, and the guard has to be remembered by each new rule).
  • Publish fledge_restore_drill_ok 1 on a fresh install (rejected outright: a green nobody proved).

Re-tip after main v0.4.5

main moved to 044c7515 (release: v0.4.5 (#1200)) and gained the column-check reading of the drill gauge, so this branch no longer merged (mergeable_state: dirty). It is re-tipped as 3c98a6c53ea50d5aafeac5c19e7ecd1ec55b52d9 with a forward merge of 044c7515 — same branch, same PR, no force-push — because rewriting the reviewed tip is not allowed.

Three conflicts, all in this change's own words rather than in behaviour:

  • src/lib/backupMetrics.ts — both comment blocks kept: no record publishes no verdict (bug: the shipped restore-drill alert pages every fresh install that has never run a drill #1199), and a fail/not-reached column check still publishes 0 even beside ok: true.
  • test/backupMetrics.test.ts — both tests kept: the failed/aged-out drill publishes 0 beside its age series, and the column-check test main added.
  • docs/deploy.md — one paragraph now states both: absent verdict and age for a fresh install, and the column-check rule.

At this tip: npm run check 3294/3294 (lint baseline, typecheck, build), node scripts/version.mjs check → version: 0.4.5 ok, npm run mock:check → ok (14 files). The red half is unchanged: at f1c48ed3 the fresh-install assertion expected no verdict and got 0.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS at cf90782

  • A fresh install is now silent at the producer: src/lib/backupMetrics.ts:84 clears the verdict when readings.drill === null, so /metrics has no fledge_restore_drill_ok series and expr: fledge_restore_drill_ok == 0 (deploy/prometheus/fledge-alerts.yml:116) cannot select it. readRestoreDrill returns null for a missing, unreadable or malformed record (src/lib/backupStatus.ts:83-90), which is its documented "no record".
  • The must-not-regress cases hold: a record that failed and one that aged past BACKUP_EXPECTED_HOURS still publish 0 beside the age series, pinned by the new assertions (test/backupMetrics.test.ts:119-147); I ran 6/6 at this tip, and the updated fresh-install assertion is red at base f1c48ed3 exactly as the body says (expected: null, actual: 0).
  • The rule change is comment-only: expr, for: 30m, labels and annotations are unchanged at :116-122; the old comment ("a missing record reads 0 like a fresh install") was the wrong claim and is replaced, and the new text matches the cleared-gauge behaviour.
  • No other reader of the series: grep fledge_restore_drill_ok over src, scripts and test finds only the metrics lib, this rule and its tests; /admin reads readings.drillOk and the in-app ops alert keeps the readings.drill !== null guard (src/jobs/service.ts:3806), so scrape and card now agree.
  • Non-blocking: the body's residual-risk bullet says a record "whose at cannot be parsed still publishes 0 … so it alerts" — that holds only for a shape-valid record with an unparsable date string; malformed JSON or a missing at/ok reads as null at src/lib/backupStatus.ts:83-90 and now clears the verdict, so that narrower subset is silent (it was already silent in the in-app alert). The body's rule citations also drifted by its own +1 comment line: at this tip expr is :116 and severity: warning is :119, not :115/:118.
    Ran: npx tsx --test test/backupMetrics.test.ts test/sloDefinitions.test.ts at this tip (6/6 and 3/3 pass); the updated backupMetrics test at base f1c48ed3 fails once — "no drill record is unknown, not a zero verdict" expected null, actual 0; read the rule block at the tip to confirm expression, for, labels and annotations unchanged.
    Not run: npm run check locally (CI ci-ok 136734444700 and coverage green at this tip); a real Prometheus with the 30-minute for: window; Docker and a production deploy (out of scope per the body).
    Written-by: HugoXK-bot

@HugoXK-bot HugoXK-bot added the review:approved Issue reviewed and approved by the Fledge Platform Reviewer label Nov 21, 2026
…r run

A fresh install published `fledge_restore_drill_ok 0`, and the shipped
`FledgeRestoreDrillFailing` rule selects exactly that value, so every deployment
paged half an hour after its first scrape and kept paging until someone ran a
drill. `docs/observability.md` promised a fresh install says nothing, and the
in-app ops alert was already guarded by `readings.drill !== null`; only the
scrape and the rule disagreed with them.

The gauge is now cleared when there is no drill record, the treatment
`fledge_backup_age_hours`, `fledge_restore_drill_age_hours` and
`fledge_clock_skew_seconds` already use. A record that exists still publishes 0
when it failed or aged past the expected interval, beside its age series, so the
case the rule was written for keeps paging. The rule's expression is unchanged;
its comment, `docs/observability.md` and `docs/deploy.md` now describe the series
as absent rather than zero.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS at 3c98a6c

  • The forward merge is honest: the commit's parents are cf907824 and 044c7515 with the old tip first, so history only moved forward (no force). git diff 044c7515..HEAD --stat is exactly the six files (+65/−18), and for each of the three conflicted files the head differs from 044c7515 only by this branch's own hunks — main's column-check reader (src/lib/backupMetrics.ts:66-68) and its test (test/backupMetrics.test.ts:150) survive verbatim, so "keep both conclusions" is a real merge and not a text squash.
  • Both rules compose at src/lib/backupMetrics.ts:96-97: readings.drill === null clears fledge_restore_drill_ok, while any record that exists publishes 1/0, so main's fail/not-reached zero at :66-68 still fires beside an ok: true; the new assertions (test/backupMetrics.test.ts:122-148) pin the failed and aged-out records, and main's column-check test is untouched at :150.
  • The docs state both rules: docs/deploy.md:1332-1339 in one paragraph, docs/observability.md:66-69 ("absent is never measured, 0 is measured and not passing"), and the rule comment (deploy/prometheus/fledge-alerts.yml:110-116) is the only change there — expr: fledge_restore_drill_ok == 0, for: 30m, labels and annotations are unchanged.
  • Non-blocking, carried from the earlier verdict and untouched by the merge: a shape-valid record with an unparsable at has drill !== null but no age series, so it publishes the 0 the docs describe as carrying an age; malformed JSON or a missing at/ok reads as null and is silent.
  • No UI surface (无 UI 面): metrics, an alert comment and docs only.

Ran: npx tsx --test test/backupMetrics.test.ts test/sloDefinitions.test.ts in a worktree at 3c98a6c — 10/10 pass; git diff 044c751..HEAD --stat and a main-vs-head diff of the three conflicted files; curl reads of the check-runs (filter=latest: 32 checks, ci-ok 136926505191 and test (24) 136925571170 green, only review-gate 136925569949 red for the stale cf907824 approve).
Not run: npm run check locally (CI green at this tip); a real Prometheus with the 30-minute for: window; Docker or a production deploy.
Written-by: HugoXK-bot

Main moved to the v0.4.5 release and gained the column-check reading in
src/lib/backupMetrics.ts, test/backupMetrics.test.ts and docs/deploy.md, so
this branch could no longer merge. Both conflicts keep both conclusions: no
drill record publishes neither the verdict nor the age (#1199), while a
recorded fail/not-reached column check still publishes 0 beside its age
series, and a recorded failed or aged-out drill keeps the 0 the rule pages on.

Ran: npm run check (3294/3294, lint baseline, build), node scripts/version.mjs
check (0.4.5 ok), npm run mock:check (ok, 14 files).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved Issue reviewed and approved by the Fledge Platform Reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: the shipped restore-drill alert pages every fresh install that has never run a drill

2 participants