Skip to content

Burn-down: open issues after v1 go-live (closed 2026-10-23; #54 continues alone) #24

Closed

Description

@gregoryfoster

Burn-down of processor's open issues after v1 went live (#1, 2026-10-02). Protocol: gregoryfoster/skills#348 (burning-down-an-epic, a draft). The operator implements one issue at a time in a separate agent thread on co-processor. At each check-in, the coordinator thread verifies the work, ticks it here, re-scans for new issues, and names the next one.

Milestones (read first at every check-in):

Measured state (2026-10-15T15:01Z)

The heading's time is the section's last restructure, which v1 can't rename; each bullet carries its own.

Parking-lot mode (since 2026-10-11T00:51Z; operator, 2026-10-10T23:36Z)

#5 was verified at 2026-10-11T00:51Z, so #24 stays open as a parking lot. A check-in happens only when:

Protocol (since 2026-10-15, #52): coordinating-epics v1, vendored at .claude/skills/coordinating-epics. Every body edit goes through its epic-edit.sh, and every check-in starts with scan.sh. scan.sh reads CannObserv/cannobserv refs as 404 with this repo's token, so check those by hand with the cannobserv token. The intro paragraph's "burning-down-an-epic, a draft" predates this; v1 has no op that edits it.

Each check-in still runs the milestone read and the probe from Last probe. The co-core catch-up (v0.30.0 against a 0.19.7 pin) is folded into whichever of #29/#37 bumps first (operator, 2026-10-10T23:36Z). Done by #29 (2026-10-12: 0.19.7 → 0.32.0, the whole range read). The next bump starts from 0.38.0 (#37).

Ordering rule

The operator gave no rule this time. The coordinator applied the following; challenge it here.

  1. Time windows first. The window before shadow closes when Watcher flips WATCHER_EXTRACT_MODE=shadow. Work that is cheapest with the bus empty goes in it.
  2. Then work that must land before the cutover (Switch WATCHER_EXTRACT_MODE to processor: processor-decided apply path, flip, soak, delete local extraction and the co-core [extract] extra watcher#326, spec §6 step 5), smallest first. Tier complete: the cutover happened 2026-10-06T21:35:24Z.
  3. Then deploy safety (added 2026-10-06 for deploy.sh: gate a deploy on the commit's CI (status#11's CI gate) #34 and A drift signal when live lags origin/main (status#12's drift check) #35): guards on how code reaches production. Not cutover preconditions, but production-facing, so ahead of agent tooling.
  4. Then agent tooling, which has no effect on production.
  5. Blocked items stay parked, each with a re-entry test.

Security first (since 2026-10-22): a security fix goes ahead of every tier above, below only the standing rule (#65, #66).

Standing rule: any Processor failure under real traffic goes above everything else in this list. Since 2026-10-06T21:35Z that is a production failure: a failed command takes the item's failure path in Watcher.
The probe, run at every check-in:

  • read Switch WATCHER_EXTRACT_MODE to processor: processor-decided apply path, flip, soak, delete local extraction and the co-core [extract] extra watcher#326's newest comment (the soak; shadow verdicts ended at the switch);
  • summarise the journal: journalctl -u processor --since <last probe> -o cat, counting outcomes by message and listing every record above INFO;
  • run the probe last, just before writing the epic, and stamp Last probe with its time;
  • look for new output digests per info_source_id (a real change) and check that Watcher reports no failure for it;
  • check the broker's and Watcher's operational issues for planned events (reboots, rehearsals) since the last check-in;
  • liveness: the starting record says liveness: on …, and no liveness WARNING; Status's co-processor-live ok;
  • journalctl -u processor-drift --since <last probe> -o cat: every run kind: ok and checkin: 202 (a lag, an alert, or a gap of more than an hour gets triaged).

Any record above INFO, a processing_failed for a live item, or NRestarts > 0 is triaged before the next hand-off.

Tailnet flap rule (operator, 2026-10-23, #67), word for word:

A tailnet flap gets an issue only if it delays a command, pages, or outlasts liveness's 900 s grace. Any other flap is logged on the epic with its window and signature.

It replaces the rule set at the first flap (2026-10-10: "a second flap, or any delayed command, gets an issue"). Telling a flap apart, and the triage commands, are in DEPLOYMENT.md § Tailnet flaps: a host pause shows virtio_balloon … Out of puff and tailscaled's rebind-reason=[time-jumped(…)]; path churn shows neither; a broker-side event or a tailscaled restart shows no LinkChange.

Tiebreaks:

Checklist

Tags: type · size · surface. Gate = an approval the implementer will need partway through; ask for it early.

Before shadow

Before cutover (CannObserv/watcher#326)

Security (since 2026-10-22: the eMudhra compromise)

Ahead of every other tier: security fixes, and co-processor is the cohort's canary in CannObserv/provisioner#77's order.

  • Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) #65 Tailscale 1.104.2 for TS-2026-013 · ops/security · S. Filed 2026-10-22 from CannObserv/provisioner#77 (co-processor is the cohort's canary). Hand run 2026-10-22, simulated first (tailscale alone), pinned, no -y, NEEDRESTART_MODE=l: 1.102.4 → 1.104.2, Go 1.26.6 → 1.27.3, client and daemon. tailscaled restarted 23:40:29Z (MainPID 292 → 1080778); tailnet down 2.3 s. Coordinator re-checked: version and Go, the new MainPID, the OOM drop-in at -950 on the new PID, /etc/resolv.conf → 100.100.100.100, broker resolving; processor unrestarted (NRestarts=0) with no WARNING since (the broker socket survived); drift ok/202 at 23:41:12Z; the next command acked 23:44:02Z. No repo change, no deploy. Posts (approved): CannObserv/provisioner#77 (issuecomment-6354166256, canary clear; this repo's token can't resolve the link); the bus-health ask on CannObserv/broker#119, reply pending (it gates the cohort's next nodes, not Processor; re-checked at the next check-in). Run, closing comment. Follow-ups: certifi ≥ 2026.10.19: the live venv still trusts the eMudhra-compromised emSign G3 chain #66. · Bus-health reading in (CannObserv/broker#119, 2026-10-23T12:34Z): finding_count: 0 before (23:40:10Z) and after (23:50:50Z) the restart and every summary since; processor.process 0 pending, last-delivered = last-generated (1792756561337-0). Coordinator cross-checked: 15 ack: complete here since 23:40:30Z, the last 1792756561337-0, matching broker's 15 entries.
  • certifi ≥ 2026.10.19: the live venv still trusts the eMudhra-compromised emSign G3 chain #66 certifi ≥ 2026.10.19 · security · S. Filed 2026-10-22 by the coordinator while assessing Tailscale 1.104.2 for TS-2026-013 (eMudhra intermediates) #65. Shipped in #66 fix: certifi ≥ 2026.10.19 — drop the emSign ECC G3 trust #68 (130f769..ba1262d, FF, merged 2026-10-23T00:50:47Z), CI green by job (lint, test) on the PR run and the main push run. The lock moves certifi alone (2026.7.22 → 2026.10.19), with a [tool.uv] constraint-dependencies floor; tests/test_tls_trust.py (red first) parses the bundle's subjects for emSign ECC Root CA - G3; DEPLOYMENT.md § TLS trust. Deployed through the gate 2026-10-23T00:55:47Z (CI passed for ba1262dd9344, then live -> ba1262dd9344); starting has the build, 0.38.0+1 (unchanged, so no Watcher notice), child_containment: required; NRestarts=0. Coordinator re-probed the live venv: certifi 2026.10.19, no G3 in the bundle, the advisory's noct host now rejects (SSLError; HTTP 200 before), storage.googleapis.com still verifies. The next 3 commands acked (GCS over the new bundle) and 3 drift runs read ok/202 from the new build (GitHub). Closing comment. Known limit: the floor covers this root only; a future distrust is the same bump by hand. Follow-ups: none.

Deploy safety (since #2: a merge no longer deploys)

Agent tooling

Parked (waiting on others)

How a check-in verifies a ship

  1. The issue is closed and its PR merged to main (FF, never squashed).
  2. CI is green on the merge commit and on origin/main's head, when docs-only commits came after it.
  3. Since Contain the extraction child: it runs as the service user and can reach the service's secrets #2: cat /srv/processor/live/REVISION equals git -C ~/processor rev-parse --short=12 origin/main, and journalctl -t processor-deploy shows live -> <build>. For a docs-only change, live may lag, which is expected until A drift signal when live lags origin/main (status#12's drift check) #35 defines "runtime diff". (~/processor is the operator's clone; HEAD = origin/main there still matters for the skills hook's push.)
  4. For a change under src/, deploy/, scripts/deploy.sh, pyproject.toml or uv.lock: systemctl show processor -p ActiveEnterTimestamp -p NRestarts shows a start after the deploy, 0 restarts, and the starting record has that build with child_containment: required.
  5. Each "Done when" item has its evidence in the closing comment. Every "known limit" and "follow-up: none" gets a recorded answer.
  6. The merged PR body is searched for follow-ups that weren't filed (follow-up|out of scope|not filed|deferred).

Verification debt

Log

Activity

  1. changed the title [-]Burn-down: open issues after v1 go-live (#2, #5, #15, #20, #21, #23; #1 parked)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #29, #37)[/+] on Oct 11, 2026
  2. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #29, #37)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37)[/+] on Oct 12, 2026
  3. gregoryfoster commented on Oct 15, 2026

    @gregoryfoster
    ContributorAuthor

    Adoption issue: #52. coordinating-epics v1 shipped in gregoryfoster/skills (gregoryfoster/skills#348). #52 moves this epic onto it: vendoring the skill, the adoption call for the next check-in, and what a dry run on this body found. Until then, nothing here changes.

  4. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37; adopting coordinating-epics: #52)[/+] on Oct 15, 2026
  5. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37; adopting coordinating-epics: #52)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37)[/+] on Oct 15, 2026
  6. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37, #54)[/+] on Oct 16, 2026
  7. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #21, #37, #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; active: #21; parking lot: #37, #54)[/+] on Oct 17, 2026
  8. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; active: #21; parking lot: #37, #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #37, #54)[/+] on Oct 19, 2026
  9. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #37, #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #37; parked: #54)[/+] on Oct 20, 2026
  10. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #37; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #58, #57; parked: #54)[/+] on Oct 21, 2026
  11. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #58, #57; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #57; parked: #54)[/+] on Oct 21, 2026
  12. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #57; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #62; parked: #54)[/+] on Oct 21, 2026
  13. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #62; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #65, #66, #67; parked: #54)[/+] on Oct 22, 2026
  14. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #65, #66, #67; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #66, #67; parked: #54)[/+] on Oct 23, 2026
  15. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #66, #67; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #67; parked: #54)[/+] on Oct 23, 2026
  16. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #67; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #70, optional; parked: #54)[/+] on Oct 23, 2026
  17. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; next: #70, optional; parked: #54)[/-] [+]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #54)[/+] on Oct 23, 2026
  18. changed the title [-]Burn-down: open issues after v1 go-live (done 2026-10-11; parking lot: #54)[/-] [+]Burn-down: open issues after v1 go-live (closed 2026-10-23; #54 continues alone)[/+] on Oct 23, 2026
  19. gregoryfoster commented on Oct 23, 2026

    @gregoryfoster
    ContributorAuthor

    Closing summary: the post-go-live burn-down, 2026-10-03 to 2026-10-23

    Closed by the operator on 2026-10-23. 26 items resolved: 25 shipped, plus #70 closed as not planned. #54 continues on its own. The closing checks passed:

    • the re-scan is empty;
    • the backlog accounting (--account) lists no open issue the epic misses;
    • the final state was re-verified independently, below;
    • the one open verification debt is carried and listed under "For the operator".

    What changed, by theme

    Live state left behind (re-verified 2026-10-23T17:05Z)

    • The release: live → releases/ba1262dd9344; origin/main a23970b differs in no runtime path.
    • The service: processor active, with NRestarts=0 since 2026-10-23T00:55:52Z. processor_version 0.38.0+1, child_containment: required.
    • Monitors: drift and liveness ok.
    • Trust: Tailscale 1.104.2. certifi 2026.10.19, and the noct test host is rejected.

    Continues on its own

    For the operator: carried, not filed

    Process

    The epic ran on coordinating-epics v1 from 2026-10-15: every body edit went through epic-edit.sh, and every check-in started with scan.sh. 175 process findings were posted to gregoryfoster/skills#348, cited processor#24-F<n>.

    This thread no longer coordinates. #54 is picked up directly when its trigger fires, in its own implementation thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions