Drift check-ins carry no metadata.fault: a change of kind inside one fault waits for the daily reminder #58

Activity

1. gregoryfoster commented on Oct 21, 2026

   

   gregoryfoster

   on Oct 21, 2026

   ContributorAuthor

   More actions

   Hand-off from the burn-down epic #24, 2026-10-21. This is next, ahead of #57, as a deploy-safety item: it is about how quickly a drift alert reaches a person. Size S. It touches src/, so it ends with a deploy. The pre-flight is against origin/main (540975e).

   What the coordinator checked:

   • The gap is as stated. post_checkin (src/processor/checkin.py:68) sends json={"status": status, "variables": variables} and nothing else. Its only drift caller is __main__._drift (__main__.py:117), which passes verdict.status and verdict.variables().
   • The contract is co-status's, already in use by two siblings. On an alert, send metadata.fault; on an ok, send no metadata. Drift monitor for broker (co-broker / co-broker-drift) status#51 (closed 2026-10-20) states it for co-broker-drift, and #22 C6: broker-drift timer - live lagging origin/main alerts through co-status broker#110 (closed 2026-10-20) implements it. The issue also cites status#32 for co-archiver-drift. No change is needed on co-status's side.
   • The fault key is verdict.kind. The Kind literal is at drift.py:69 (ok, lag, off_main, unstamped, test, silent). status maps ok → ok, silent → no check-in, and everything else → alert (drift.py:83-87). So the alerting kinds are lag, off_main, unstamped and test.
   • Liveness needs no change. liveness.py only ever checks in ok: when the loop is stale it skips the check-in, and Status's grace pages. Keep metadata optional with a None default, so liveness's call is unchanged.

   The fix (the issue's suggestion, which the coordinator agrees with):

   • Give post_checkin an optional metadata: dict[str, str] | None = None. Add the key to the JSON body only when it is given.
   • Have _drift pass {"fault": verdict.kind} when verdict.status == "alert", and nothing on ok. Putting the fault on the verdict (for example a metadata() beside variables()) is your call.
   • Update checkin.py's module docstring (line 5 describes the body). Add one line to DEPLOYMENT.md's drift section, saying each alert names its kind as metadata.fault, so a change of kind reports at once.

   Tests (TDD; model them on broker's tests/test_drift.py::TestVerdict::test_every_alert_s_fault_is_its_kind and ::test_an_ok_carries_no_metadata):

   • Every alerting kind's check-in carries metadata.fault == kind, and --test-alert's carries "test".
   • An ok check-in has no metadata key at all: not null, not {}.
   • post_checkin with no metadata sends exactly today's body. Extend tests/test_checkin.py:19's stub assertion.
   • Liveness's body is unchanged.

   Traps:

   1. "No metadata" means the key is absent. co-status compares metadata.fault. A {"metadata": null} or {} on an ok is harmless there, but it isn't the sibling contract, and the tests should pin absence.
   2. A --test-alert sent while a real fault is open now reports at once, as a separate fault (test). That is the intended behaviour, so say it in DEPLOYMENT.md where the test alert is documented (around line 292).
   3. Don't put anything secret in fault. It is the kind token only, never body or a URL.

   Live checks after the deploy:

   • The next hourly processor-drift run logs kind ok / checkin 202, and the unit runs from the new release.
   • The alert path (operator gate): one processor drift --test-alert, as in DEPLOYMENT.md § the drift check. It should get 202, and the operator should confirm the alert arrived on the drift channels. The next hourly ok then clears it, which is a second notification. This pages people, so ask the operator before sending it. If they decline, say in the closing comment that the alert path is proven only in tests.

   Cross-repo: none needed. A courtesy reply on CannObserv/status#51 is optional and needs approval. The operator's approval of 2026-10-20 covered #37's two posts only.

   Done when:

   • every drift alert carries metadata.fault = its kind, an ok carries no metadata, and liveness is unchanged, all with tests;
   • the docstring and DEPLOYMENT.md are updated;
   • the change is deployed through the gate, and the next drift run is ok/202 from the new release;
   • the test alert has been sent with approval, or its absence is stated.

   Out of scope:

   • Co-core bump rule: resolve_dispatch_essence is notice, not coordination #57;
   • any change to co-status;
   • liveness's check-in shape.

   Ship: issue → branch in a worktree → plan (short) → TDD → PR (Refs #58: the deploy and live checks come after the merge) → FF merge → git pull --ff-only → scripts/deploy.sh → the live checks. The closing comment gives each box with its evidence, the known limits, and the follow-ups (or "none").

   Kickoff prompt:

   Pick up processor#58. Read its hand-off comment from epic #24 first.
2. gregoryfoster commented on Oct 21, 2026

   

   gregoryfoster

   on Oct 21, 2026

   ContributorAuthor

   More actions

   Shipped in #59 (FF merge, a1a9e91), live 2026-10-21 16:14Z.

   Done when:

   • Every drift alert carries metadata.fault = its kind, an ok carries no metadata, and liveness is unchanged, all with tests. Verdict.metadata() is {"fault": kind} on an alert and None otherwise (ok, silent). post_checkin(..., metadata=None) adds the key only when it is given, never as null or {}. Tests went red first (f1a75ac): TestVerdict::test_every_alert_s_fault_is_its_kind[lag|off_main|unstamped|test] and ::test_an_ok_carries_no_metadata[ok|silent]; the command's lag and --test-alert check-ins carry the fault, and its ok check-in has no metadata key; test_no_metadata_is_no_key_at_all[none|empty]. test_liveness.py's body-equality test passes unchanged. CI on the final SHA: lint ✓, test ✓ (run 39632225886).
   • Docstring and DEPLOYMENT.md. The checkin.py module docstring now describes metadata. DEPLOYMENT.md § The drift check gets a bullet for the fault, and the test-alert paragraph now says it reports at once during an open fault, as its own fault test (trap 2).
   • Deployed through the gate, and the next drift run is ok/202 from the new release. scripts/deploy.sh: CI passed for a1a9e91ef684, and live is on a1a9e91ef684. A manual run at 16:14:26Z returned ok/202. The hourly timer run at 17:15:04Z returned {"level":"INFO","build":"a1a9e91ef684","kind":"ok","checkin":202}. processor is active, and starting shows build a1a9e91ef684 with liveness on.
   • Test alert, with the operator's approval. It was sent at 17:50:16Z with kind test, got checkin 202, and exited 0. The operator confirmed it arrived on the drift channels. The next hourly ok (~18:15Z) clears it.

   Known limits: the fault is the kind token only. Two alerts of the same kind with different bodies (for example a lag whose CI result changes) are one fault, which is intended. I didn't observe the clearing ok after the test alert before closing.

   Follow-ups: none. The optional courtesy reply on CannObserv/status#51 has not been posted.
3. gregoryfoster

   closed this as completedon Oct 21, 2026