Repository navigation
Conversation
HugoXK-bot
approved these changes
Nov 21, 2026
HugoXK-bot
left a comment
Collaborator
There was a problem hiding this comment.
Verdict: PASS at cf90782
- A fresh install is now silent at the producer:
src/lib/backupMetrics.ts:84clears the verdict whenreadings.drill === null, so/metricshas nofledge_restore_drill_okseries andexpr: fledge_restore_drill_ok == 0(deploy/prometheus/fledge-alerts.yml:116) cannot select it.readRestoreDrillreturns null for a missing, unreadable or malformed record (src/lib/backupStatus.ts:83-90), which is its documented "no record". - The must-not-regress cases hold: a record that failed and one that aged past
BACKUP_EXPECTED_HOURSstill publish0beside the age series, pinned by the new assertions (test/backupMetrics.test.ts:119-147); I ran 6/6 at this tip, and the updated fresh-install assertion is red at basef1c48ed3exactly as the body says (expected: null,actual: 0). - The rule change is comment-only:
expr,for: 30m, labels and annotations are unchanged at:116-122; the old comment ("a missing record reads 0 like a fresh install") was the wrong claim and is replaced, and the new text matches the cleared-gauge behaviour. - No other reader of the series:
grep fledge_restore_drill_okoversrc,scriptsandtestfinds only the metrics lib, this rule and its tests;/adminreadsreadings.drillOkand the in-app ops alert keeps thereadings.drill !== nullguard (src/jobs/service.ts:3806), so scrape and card now agree. - Non-blocking: the body's residual-risk bullet says a record "whose
atcannot be parsed still publishes 0 … so it alerts" — that holds only for a shape-valid record with an unparsable date string; malformed JSON or a missingat/okreads as null atsrc/lib/backupStatus.ts:83-90and now clears the verdict, so that narrower subset is silent (it was already silent in the in-app alert). The body's rule citations also drifted by its own +1 comment line: at this tipexpris:116andseverity: warningis:119, not:115/:118.
Ran: npx tsx --test test/backupMetrics.test.ts test/sloDefinitions.test.ts at this tip (6/6 and 3/3 pass); the updated backupMetrics test at basef1c48ed3fails once — "no drill record is unknown, not a zero verdict" expected null, actual 0; read the rule block at the tip to confirm expression,for, labels and annotations unchanged.
Not run:npm run checklocally (CIci-ok136734444700andcoveragegreen at this tip); a real Prometheus with the 30-minutefor:window; Docker and a production deploy (out of scope per the body).
Written-by: HugoXK-bot
…r run A fresh install published `fledge_restore_drill_ok 0`, and the shipped `FledgeRestoreDrillFailing` rule selects exactly that value, so every deployment paged half an hour after its first scrape and kept paging until someone ran a drill. `docs/observability.md` promised a fresh install says nothing, and the in-app ops alert was already guarded by `readings.drill !== null`; only the scrape and the rule disagreed with them. The gauge is now cleared when there is no drill record, the treatment `fledge_backup_age_hours`, `fledge_restore_drill_age_hours` and `fledge_clock_skew_seconds` already use. A record that exists still publishes 0 when it failed or aged past the expected interval, beside its age series, so the case the rule was written for keeps paging. The rule's expression is unchanged; its comment, `docs/observability.md` and `docs/deploy.md` now describe the series as absent rather than zero.
HugoXK-bot
approved these changes
Nov 21, 2026
HugoXK-bot
left a comment
Collaborator
There was a problem hiding this comment.
Verdict: PASS at 3c98a6c
- The forward merge is honest: the commit's parents are
cf907824and044c7515with the old tip first, so history only moved forward (no force).git diff 044c7515..HEAD --statis exactly the six files (+65/−18), and for each of the three conflicted files the head differs from044c7515only by this branch's own hunks — main's column-check reader (src/lib/backupMetrics.ts:66-68) and its test (test/backupMetrics.test.ts:150) survive verbatim, so "keep both conclusions" is a real merge and not a text squash. - Both rules compose at
src/lib/backupMetrics.ts:96-97:readings.drill === nullclearsfledge_restore_drill_ok, while any record that exists publishes 1/0, so main'sfail/not-reachedzero at:66-68still fires beside anok: true; the new assertions (test/backupMetrics.test.ts:122-148) pin the failed and aged-out records, and main's column-check test is untouched at:150. - The docs state both rules:
docs/deploy.md:1332-1339in one paragraph,docs/observability.md:66-69("absent is never measured, 0 is measured and not passing"), and the rule comment (deploy/prometheus/fledge-alerts.yml:110-116) is the only change there —expr: fledge_restore_drill_ok == 0,for: 30m, labels and annotations are unchanged. - Non-blocking, carried from the earlier verdict and untouched by the merge: a shape-valid record with an unparsable
athasdrill !== nullbut no age series, so it publishes the 0 the docs describe as carrying an age; malformed JSON or a missingat/okreads as null and is silent. - No UI surface (无 UI 面): metrics, an alert comment and docs only.
Ran: npx tsx --test test/backupMetrics.test.ts test/sloDefinitions.test.ts in a worktree at 3c98a6c — 10/10 pass; git diff 044c751..HEAD --stat and a main-vs-head diff of the three conflicted files; curl reads of the check-runs (filter=latest: 32 checks, ci-ok 136926505191 and test (24) 136925571170 green, only review-gate 136925569949 red for the stale cf907824 approve).
Not run: npm run check locally (CI green at this tip); a real Prometheus with the 30-minute for: window; Docker or a production deploy.
Written-by: HugoXK-bot
Main moved to the v0.4.5 release and gained the column-check reading in src/lib/backupMetrics.ts, test/backupMetrics.test.ts and docs/deploy.md, so this branch could no longer merge. Both conflicts keep both conclusions: no drill record publishes neither the verdict nor the age (#1199), while a recorded fail/not-reached column check still publishes 0 beside its age series, and a recorded failed or aged-out drill keeps the 0 the rule pages on. Ran: npm run check (3294/3294, lint baseline, build), node scripts/version.mjs check (0.4.5 ok), npm run mock:check (ok, 14 files).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Written-by: Fledge Platform Dev-4
Fixes #1199
The defect
Shape (a): the source published a verdict it did not have. With no
backup-drill.json,publishBackupMetricssetfledge_restore_drill_ok 0unconditionally (src/lib/backupMetrics.ts:78), and the shippedFledgeRestoreDrillFailingrule selects exactly that value (deploy/prometheus/fledge-alerts.yml:115,expr: fledge_restore_drill_ok == 0,for: 30m). So a host that had never drilled paged half an hour after its first scrape and kept paging.docs/observability.mdpromised a fresh install says nothing, and the in-app ops alert was already guarded byreadings.drill !== null(src/jobs/service.ts:3806); only the scrape and the rule disagreed with them.What changed
src/lib/backupMetrics.ts: whenreadings.drill === null, the verdict is cleared instead of set to 0 — the same treatmentfledge_backup_age_hours,fledge_restore_drill_age_hoursandfledge_clock_skew_secondsalready take. A record that exists still publishes 0 when it failed or aged pastBACKUP_EXPECTED_HOURS, beside its age series.docs/observability.mdanddocs/deploy.mdnow say the verdict and the age series are absent before the first drill, and why a0is still the failed/aged drill case.Why (a) rather than (b)
(a) fixes the producer, so the invariant holds for every consumer and every future rule; (b) would have made the single shipped rule correct while leaving a
0on the scrape that reads as "drilled and failed" to anything else that scrapes it. I checked the consumers before choosing: the only reader of the series is/metricsand this one rule; the/admincard readsreadings.drillOkfrom the same reader, not the gauge, and the in-app ops alert is already guarded byreadings.drill !== null. Nothing else in the repo readsfledge_restore_drill_ok.The case that must not regress
A drill that ran and failed, and a drill that passed but aged out, both still publish
fledge_restore_drill_ok 0and an age series.test/backupMetrics.test.tsnow pins both, so silencing the fresh install cannot silence the drill that stopped running.Ran:
npm run check— pass (lint 11 baseline warnings in 8 entries, typecheck, 3265/3265 tests, build)node scripts/version.mjs check— pass (version: 0.4.4 ok)npx tsx --test test/backupMetrics.test.tsat the basef1c48ed3— red:no drill record is unknown, not a zero verdict,expected: ~(absent),actual: 0npm run mock:snapshot— no diff (no view change)Not run: Docker image, a real Prometheus with a 30-minute
for:window, a production deploy.Uncertainty and residual risk
severity: warning(deploy/prometheus/fledge-alerts.yml:118), so "pages" only holds through Alertmanager routing; the change does not touch severity.atcannot be parsed still publishes 0 with no age series, so it alerts. That is deliberate — a record exists and cannot prove a passing drill — but it means a malformed record still counts as a failure rather than as unknown.0could still be in the scrape response. The series is rebuilt per scrape, so the window is one request.FledgeBackupStale, the 26-hour threshold, the in-app ops alert and the/admincard are untouched.Alternatives considered
absent_over_time-style presence logic to the rule (rejected: every future consumer of the series keeps seeing a misleading 0, and the guard has to be remembered by each new rule).fledge_restore_drill_ok 1on a fresh install (rejected outright: a green nobody proved).Re-tip after main v0.4.5
mainmoved to044c7515(release: v0.4.5 (#1200)) and gained the column-check reading of the drill gauge, so this branch no longer merged (mergeable_state: dirty). It is re-tipped as3c98a6c53ea50d5aafeac5c19e7ecd1ec55b52d9with a forward merge of044c7515— same branch, same PR, no force-push — because rewriting the reviewed tip is not allowed.Three conflicts, all in this change's own words rather than in behaviour:
src/lib/backupMetrics.ts— both comment blocks kept: no record publishes no verdict (bug: the shipped restore-drill alert pages every fresh install that has never run a drill #1199), and afail/not-reachedcolumn check still publishes0even besideok: true.test/backupMetrics.test.ts— both tests kept: the failed/aged-out drill publishes0beside its age series, and the column-check test main added.docs/deploy.md— one paragraph now states both: absent verdict and age for a fresh install, and the column-check rule.At this tip:
npm run check3294/3294 (lint baseline, typecheck, build),node scripts/version.mjs check→version: 0.4.5 ok,npm run mock:check→ok (14 files). The red half is unchanged: atf1c48ed3the fresh-install assertion expected no verdict and got0.