You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A node that ran the #302 build cannot start on the previous plugin: the checkpoint state changes one way and the older build takes the node down #308
What happened. On 2026-11-18 the loop lab ran capsules-plugin 2cc0974 (#302) on lab-3 (14:02Z) and lab-1 (14:10Z). At 15:07:54Z I put lab-1 back on the plugin it ran before, 64c518e (main at the #294 merge), by restoring its config.toml and restarting the node. The older plugin did not start on the ledger the #302 build had written, and because capsules is a required plugin the whole node went down (fatal) until 15:11:40Z, about 4 minutes, when I put the #302 build back.
The log line (lab-1's node log, 2026-11-18T15:08Z): Error: failed to load checkpoint state at <node>/.local/share/capsules/ledger: checkpoint error: the CLL state's checkpoint or MMR does not match the log's entries: the checkpoint's COSE does not verify: CWT subject Some("… then Plugin failed to load plugin=capsules error=Timed out waiting for plugin 'capsules' and Required plugin 'capsules' failed to start.
Why it matters. The first start of the #302 build changes the ledger's checkpoint state in a way the previous plugin cannot read (I did not diagnose which part; #302 moves the evidence stack to evidencebook 0.0.21 / checkpointed-local-log 0.6). It is one way: lab-1 and lab-3 cannot go back to 64c518e without restoring their ledgers from a backup, and I know of none. A node operator who tries to roll back a release takes their node down.
Wanted (either). A downgrade that refuses cleanly: the older plugin names the cause ("this ledger was written by a newer plugin; downgrade is not supported") and the node keeps running without the plugin or the plugin runs read-only; or release notes for the release that carries #302 that say plainly that there is no downgrade and to back up the ledger before upgrading.
Measured: the failure, its log line and its timing (lab-1). Not established: which file or field of the ledger changed, whether a ledger copy taken before the upgrade restores a working older node.
(Ethan's agent) Started: caps-308; (1) a ledger format marker (ledger/format) that a build writes on open and a build that meets a newer marker refuses on, naming the cause, instead of failing deep in the checkpoint COSE check; (2) a plain no-downgrade and back-up-first warning in the Unreleased changelog entry that carries #300/#302.
Owning repository: Mesh-LLM/capsules-plugin (the ledger and its open path are this plugin's: CapsuleState::open, crates/evidence-plugin/src/capsule_emit.rs; no boundary ruling is involved).
Cause (code):#300 (f9c16d5) moves to checkpointed-local-log 0.6, whose checkpoints carry the bare log id as the CWT sub; the previous build's checkpoint check rejects them (the checkpoint's COSE does not verify: CWT subject ...), and capsules is a required plugin, so the node dies.
A format marker: ledger/format, one integer. This build writes it (1 = the 0.6 checkpoint stack) when it opens the ledger and refuses to open a ledger whose marker is higher than it knows, with this ledger was written by a newer plugin (format N, this build reads up to M); downgrade is not supported: restore a backup of the ledger or run the newer plugin. It guards the next format change and any later rollback to a build with this check; it cannot help a rollback to a build before it, which is why (1) matters.
Constraints: no change to ledger contents, checkpoints, wire formats or any signed or served artifact; the marker is a local file no peer reads. capsules stays a required plugin, so the refusal is a clear start error, not a read-only mode (a read-only mode would be new machinery).
One-way doors: a new on-disk file in the ledger directory is a local storage format with no product meaning. It is engineering's call under tracking.md; the release-note wording is Sherlock2's. Both go to Sherlock2 (and the twin) for a door confirmation and the wording review; merge waits for the answers.
The bar: this moves no bar (THE_LOOP.md names nothing on downgrade or rollback); no Steven ruling applies ("none yet").
A clean refusal: the plugin writes ledger/format on open and a build that meets a higher number stops at start with this ledger was written by a newer plugin (ledger format N, this build reads up to M); downgrade is not supported ..., before a pending new history can move the ledger. Tests: ledger_format unit tests, open_refuses_a_ledger_written_in_a_newer_format, a_newer_format_ledger_is_refused_before_a_pending_new_history_moves_it; cargo test in crates/evidence-plugin exit 0 (1518 passed).
Limit, as the changelog says: builds before this one do not read the marker, so a rollback to them (the lab-1 case, 64c518e) is still unsafe; only the backup protects it. No release or tag was cut. Not established, as in the issue: whether a pre-upgrade copy restores a working older node.
Activity
ethanyzhang commentedon Nov 18, 2026
(Ethan's agent) Started: caps-308; (1) a ledger format marker (
ledger/format) that a build writes on open and a build that meets a newer marker refuses on, naming the cause, instead of failing deep in the checkpoint COSE check; (2) a plain no-downgrade and back-up-first warning in the Unreleased changelog entry that carries #300/#302.ethanyzhang commentedon Nov 18, 2026
(Ethan's agent) How ack.
Owning repository: Mesh-LLM/capsules-plugin (the ledger and its open path are this plugin's:
CapsuleState::open,crates/evidence-plugin/src/capsule_emit.rs; no boundary ruling is involved).Cause (code): #300 (f9c16d5) moves to checkpointed-local-log 0.6, whose checkpoints carry the bare log id as the CWT
sub; the previous build's checkpoint check rejects them (the checkpoint's COSE does not verify: CWT subject ...), andcapsulesis a required plugin, so the node dies.Approach.
ledger/andkeys/) before upgrading, and what the failure looks like. A build that is already released (64c518e and earlier) cannot be changed, so this is the only protection for a rollback to those builds.ledger/format, one integer. This build writes it (1 = the 0.6 checkpoint stack) when it opens the ledger and refuses to open a ledger whose marker is higher than it knows, withthis ledger was written by a newer plugin (format N, this build reads up to M); downgrade is not supported: restore a backup of the ledger or run the newer plugin. It guards the next format change and any later rollback to a build with this check; it cannot help a rollback to a build before it, which is why (1) matters.Constraints: no change to ledger contents, checkpoints, wire formats or any signed or served artifact; the marker is a local file no peer reads.
capsulesstays a required plugin, so the refusal is a clear start error, not a read-only mode (a read-only mode would be new machinery).One-way doors: a new on-disk file in the ledger directory is a local storage format with no product meaning. It is engineering's call under tracking.md; the release-note wording is Sherlock2's. Both go to Sherlock2 (and the twin) for a door confirmation and the wording review; merge waits for the answers.
The bar: this moves no bar (THE_LOOP.md names nothing on downgrade or rollback); no Steven ruling applies ("none yet").
fix(evidence-plugin): check the ledger format before a pending new hi…
docs(evidence-plugin): put the end-to-end restart test's doc back on …
docs: say the keys backup is private and that a rollback or a refusal…
docs: back up the whole data directory; drop a repeated sentence; set…
feat(evidence-plugin): mark the ledger's format and refuse a newer on…
ethanyzhang commentedon Nov 18, 2026
(Ethan's agent) Done: merged in #309 (c115512, head 39544b2, 13 checks pass, one skipped; review record #309 (comment)).
Both wanted parts are in:
ledger/formaton open and a build that meets a higher number stops at start withthis ledger was written by a newer plugin (ledger format N, this build reads up to M); downgrade is not supported ..., before a pending new history can move the ledger. Tests:ledger_formatunit tests,open_refuses_a_ledger_written_in_a_newer_format,a_newer_format_ledger_is_refused_before_a_pending_new_history_moves_it;cargo testin crates/evidence-plugin exit 0 (1518 passed).capsules export-flatis no way back.Limit, as the changelog says: builds before this one do not read the marker, so a rollback to them (the lab-1 case, 64c518e) is still unsafe; only the backup protects it. No release or tag was cut. Not established, as in the issue: whether a pre-upgrade copy restores a working older node.