Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,7 @@ A stop mid-reclaim finishes the command in hand and leaves the rest pending. A c
| unstamped: the release has no `REVISION` | `alert` (`unstamped`) | `live` names something that isn't a finished release: deploy |
| GitHub can't answer: a rate limit, a 5xx, a timeout, a wrong-shaped answer | none; the run exits 1 | nothing. The monitor's grace absorbs one missed run |

- **Each `alert` names its kind as `metadata.fault`** (#58; status#28, the contract of `co-archiver-drift` and `co-broker-drift`); an `ok` sends no `metadata`. Status groups a run of alerts into one fault by it, so a change of kind (`lag` → `off_main`, say) reports at once, not at the next daily reminder.
- **What runs:** `src/`, `deploy/`, `scripts/deploy.sh`, `pyproject.toml`, `uv.lock`. A rename counts by either name. Docs, tests, CI, skills and the other scripts never count, however old.
- **The check is itself runtime:** it lives in `src/` and its units in `deploy/`. A change to it needs a deploy, and lags like any other.
- **The clock** starts at the push, from its CI run's `created_at`, never the commit date. Within the grace, the body names the oldest push since live, which may be docs only.
Expand All @@ -283,7 +284,7 @@ The level is INFO for `ok`, WARNING for an alert or a silent GitHub, and ERROR w

**Silence a lag you mean to keep.** It alerts hourly until deployed. `sudo systemctl stop processor-drift.timer` stops the alerts, and the monitor then goes `missing`, which is honest. `start` resumes them. A deploy never re-enables a timer it installed before, so `disable --now` keeps it stopped across reboots too.

**A deliberate alert** proves the channels end to end. It sends `kind: test`, and asks GitHub nothing:
**A deliberate alert** proves the channels end to end. It sends `kind: test`, and asks GitHub nothing. Its fault is `test`, so sent while a real fault is open it reports at once, as a fault of its own; the next hourly run's alert, back to the real kind, reports at once too:

```bash
sudo systemd-run --quiet --pipe --wait --collect -p User=processor -p Group=processor \
Expand Down
35 changes: 35 additions & 0 deletions docs/plans/2026-10-21-58-drift-fault.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Drift alerts carry metadata.fault (#58)
date: 2026-10-21
status: per #58's hand-off from epic #24 (2026-10-21)
---

# Drift alerts carry `metadata.fault`

## Problem

`post_checkin` sends `{"status", "variables"}` only. Since status#28 co-status groups a run of `alert`s into one fault by `metadata.fault`; a change of it reports at once. Without the key a change of drift `kind` inside one fault (`lag` → `off_main`, `unstamped`) waits for the daily reminder, up to ~23.5 h. Same for a `--test-alert` sent while a real fault is open.

## Approach

The sibling contract (status#32, #51; broker#110): every `alert` sends `"metadata": {"fault": <kind>}`; an `ok` sends no `metadata` key at all.

- `checkin.post_checkin` gains `metadata: dict[str, str] | None = None`; the key enters the body only when given. Liveness's call is unchanged (it only checks in `ok`).
- `drift.Verdict.metadata()` beside `variables()`, as broker's: `{"fault": kind}` when `status == "alert"`, else `None`. The fault is the kind token only, never `body` or a URL.
- `__main__._drift` passes `metadata=verdict.metadata()`.
- Docs: `checkin.py`'s module docstring, `drift.py`'s `Verdict`, DEPLOYMENT.md § The drift check (each `alert` names its kind as `metadata.fault`; a `--test-alert` during an open fault reports at once as its own fault, `test`).

## Tests (red first)

- `test_checkin.py`: no `metadata` → exactly today's body (no key); `metadata` given → in the body.
- `test_drift.py::TestVerdict`: every alerting kind's `metadata()` is `{"fault": kind}`; `ok` and `silent` are `None`.
- `test_drift.py::TestTheCommand`: a `lag` check-in carries `metadata.fault == "lag"`; an `ok` has no `metadata` key; `--test-alert`'s carries `"test"`.
- `test_liveness.py`: body already pinned by equality (no `metadata`); unchanged.

## Ship

PR `Refs #58` → FF merge → pull → `scripts/deploy.sh` → next hourly drift run `ok`/202 from the new release → `--test-alert` only with the operator's approval.

## Open questions

None.
7 changes: 6 additions & 1 deletion src/processor/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,12 @@ def _drift(settings: DriftSettings, *, test_alert: bool) -> int:
return 1
try:
code = post_checkin(
settings.status_url, settings.drift_monitor_id, key, verdict.status, verdict.variables()
settings.status_url,
settings.drift_monitor_id,
key,
verdict.status,
verdict.variables(),
metadata=verdict.metadata(),
)
except CheckinFailed as exc:
logger.error("drift check", extra=fields | {"checkin": f"failed: {exc}"})
Expand Down
9 changes: 8 additions & 1 deletion src/processor/checkin.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
"variables": {...}}`` and an ``X-API-Key``, answered 202. An ``ok`` records the
check-in; an ``alert`` renders the monitor's templates against ``variables``
and dispatches them; silence past the interval and grace goes ``missing``.
``"metadata": {"fault": ...}`` rides along only when given (status#28): co-status
groups a run of alerts into one fault by it, and a change of it reports at once.
With none, the key is absent, never ``null`` or ``{}`` (the siblings', status#51).

**One attempt, never a loop** (#35 trap 4): the next run is the retry, and the
monitor's grace absorbs one missed check-in. A failure raises
Expand Down Expand Up @@ -53,6 +56,7 @@ def post_checkin(
status: Literal["ok", "alert"],
variables: dict[str, str],
*,
metadata: dict[str, str] | None = None,
timeout: float = TIMEOUT_SECONDS,
) -> int:
"""Check in once; return Status's answer code (202), or raise :class:`CheckinFailed`."""
Expand All @@ -62,10 +66,13 @@ def post_checkin(
"reinstall it (docs/DEPLOYMENT.md § The drift check)"
)
url = f"{base_url.rstrip('/')}/api/v1/monitors/{monitor_id}/checkin"
body: dict[str, object] = {"status": status, "variables": variables}
if metadata:
body["metadata"] = metadata
try:
response = requests.post(
url,
json={"status": status, "variables": variables},
json=body,
headers={"X-API-Key": key},
timeout=timeout,
)
Expand Down
8 changes: 8 additions & 0 deletions src/processor/drift.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,14 @@ def variables(self) -> dict[str, str]:
"""Every variable on every check-in, so the monitor's template always renders."""
return {"kind": self.kind, "live": self.live, "main": self.main, "body": self.body}

def metadata(self) -> dict[str, str] | None:
"""An alert's fault, its kind; ``None`` on an ``ok``, so the key is absent (#58).

co-status compares ``metadata.fault`` and opens a new fault, reported at
once, when it changes (status#28). The kind token only: never ``body``.
"""
return {"fault": self.kind} if self.status == "alert" else None


@dataclass(frozen=True)
class Push:
Expand Down
25 changes: 23 additions & 2 deletions tests/test_checkin.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""``processor.checkin``: one check-in to a Status monitor (#35), against a local stub.

Status's contract (CannObserv/status spec D6, notifier's unchanged):
``POST /api/v1/monitors/{id}/checkin`` with ``{"status", "variables"}`` and an
``X-API-Key``, answered 202.
``POST /api/v1/monitors/{id}/checkin`` with ``{"status", "variables"}``, and
``metadata`` only when given (status#28), and an ``X-API-Key``, answered 202.
"""

import json
Expand All @@ -25,6 +25,27 @@ def test_a_check_in_posts_status_and_variables_with_the_key_as_a_header(http_stu
assert request["headers"]["Content-Type"] == "application/json"


def test_metadata_given_is_sent_beside_status_and_variables(http_stub):
"""co-status compares ``metadata.fault``: a change of it reports at once (status#28)."""
http_stub.route("POST", PATH, 202)
post_checkin(http_stub.url, MONITOR, KEY, "alert", {"kind": "lag"}, metadata={"fault": "lag"})
(request,) = http_stub.requests
assert json.loads(request["body"]) == {
"status": "alert",
"variables": {"kind": "lag"},
"metadata": {"fault": "lag"},
}


@pytest.mark.parametrize("metadata", [None, {}], ids=["none", "empty"])
def test_no_metadata_is_no_key_at_all(http_stub, metadata):
"""Absent, never ``null`` or ``{}``: the siblings' contract on an ``ok`` (status#51)."""
http_stub.route("POST", PATH, 202)
post_checkin(http_stub.url, MONITOR, KEY, "ok", {}, metadata=metadata)
(request,) = http_stub.requests
assert json.loads(request["body"]) == {"status": "ok", "variables": {}}


def test_a_trailing_slash_on_the_base_url_is_harmless(http_stub):
http_stub.route("POST", PATH, 202)
assert post_checkin(http_stub.url + "/", MONITOR, KEY, "ok", {}) == 202
Expand Down
12 changes: 12 additions & 0 deletions tests/test_drift.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,15 @@ def test_every_check_in_carries_every_variable(self):
"body": "b",
}

@pytest.mark.parametrize("kind", ["lag", "off_main", "unstamped", "test"])
def test_every_alert_s_fault_is_its_kind(self, kind):
"""The one value co-status compares: a change of fault reports at once (status#28)."""
assert Verdict(kind, "b", LIVE).metadata() == {"fault": kind}

@pytest.mark.parametrize("kind", ["ok", "silent"])
def test_an_ok_carries_no_metadata(self, kind):
assert Verdict(kind, "b", LIVE).metadata() is None


class TestFirstLook:
def test_unstamped_alerts(self):
Expand Down Expand Up @@ -611,6 +620,7 @@ def test_a_runtime_lag_past_the_grace_alerts(self, world, capsys):
assert checkin["variables"]["live"] == LIVE
assert checkin["variables"]["main"] == sha(1)[:12]
assert "main's CI: success — scripts/deploy.sh" in checkin["variables"]["body"]
assert checkin["metadata"] == {"fault": "lag"}
record = outcome(capsys)
assert record["level"] == "WARNING"
assert (record["build"], record["kind"], record["checkin"]) == (LIVE, "lag", 202)
Expand All @@ -621,6 +631,7 @@ def test_a_docs_only_lag_is_ok_however_old(self, world, capsys):
(checkin,) = world.checkins()
assert checkin["status"] == "ok"
assert checkin["variables"]["kind"] == "ok"
assert "metadata" not in checkin
assert outcome(capsys)["level"] == "INFO"

def test_a_runtime_lag_within_the_grace_is_ok(self, world):
Expand Down Expand Up @@ -723,4 +734,5 @@ def test_test_alert_sends_one_alert_and_asks_github_nothing(self, world, capsys)
assert checkin["status"] == "alert"
assert checkin["variables"]["kind"] == "test"
assert set(checkin["variables"]) == {"kind", "live", "main", "body"}
assert checkin["metadata"] == {"fault": "test"}
assert outcome(capsys)["kind"] == "test"
Loading