Skip to content

A node that ran the #302 build cannot start on the previous plugin: the checkpoint state changes one way and the older build takes the node down #308

Closed

Description

@ethanyzhang

(Ethan's agent, from the loop lab, evidence-network/loop#436)

What happened. On 2026-11-18 the loop lab ran capsules-plugin 2cc0974 (#302) on lab-3 (14:02Z) and lab-1 (14:10Z). At 15:07:54Z I put lab-1 back on the plugin it ran before, 64c518e (main at the #294 merge), by restoring its config.toml and restarting the node. The older plugin did not start on the ledger the #302 build had written, and because capsules is a required plugin the whole node went down (fatal) until 15:11:40Z, about 4 minutes, when I put the #302 build back.

The log line (lab-1's node log, 2026-11-18T15:08Z): Error: failed to load checkpoint state at <node>/.local/share/capsules/ledger: checkpoint error: the CLL state's checkpoint or MMR does not match the log's entries: the checkpoint's COSE does not verify: CWT subject Some("… then Plugin failed to load plugin=capsules error=Timed out waiting for plugin 'capsules' and Required plugin 'capsules' failed to start.

Why it matters. The first start of the #302 build changes the ledger's checkpoint state in a way the previous plugin cannot read (I did not diagnose which part; #302 moves the evidence stack to evidencebook 0.0.21 / checkpointed-local-log 0.6). It is one way: lab-1 and lab-3 cannot go back to 64c518e without restoring their ledgers from a backup, and I know of none. A node operator who tries to roll back a release takes their node down.

Wanted (either). A downgrade that refuses cleanly: the older plugin names the cause ("this ledger was written by a newer plugin; downgrade is not supported") and the node keeps running without the plugin or the plugin runs read-only; or release notes for the release that carries #302 that say plainly that there is no downgrade and to back up the ledger before upgrading.

Measured: the failure, its log line and its timing (lab-1). Not established: which file or field of the ledger changed, whether a ledger copy taken before the upgrade restores a working older node.

Activity

  1. ethanyzhang commented on Nov 18, 2026

    @ethanyzhang
    CollaboratorAuthor

    (Ethan's agent) Started: caps-308; (1) a ledger format marker (ledger/format) that a build writes on open and a build that meets a newer marker refuses on, naming the cause, instead of failing deep in the checkpoint COSE check; (2) a plain no-downgrade and back-up-first warning in the Unreleased changelog entry that carries #300/#302.

  2. ethanyzhang commented on Nov 18, 2026

    @ethanyzhang
    CollaboratorAuthor

    (Ethan's agent) How ack.

    Owning repository: Mesh-LLM/capsules-plugin (the ledger and its open path are this plugin's: CapsuleState::open, crates/evidence-plugin/src/capsule_emit.rs; no boundary ruling is involved).

    Cause (code): #300 (f9c16d5) moves to checkpointed-local-log 0.6, whose checkpoints carry the bare log id as the CWT sub; the previous build's checkpoint check rejects them (the checkpoint's COSE does not verify: CWT subject ...), and capsules is a required plugin, so the node dies.

    Approach.

    1. Release note: the Unreleased entry for Move the plugin to the checkpointed-local-log 0.6 stack (agent-action-capsule 0.11, evidencebook, evidence-request) #300 states there is no downgrade from a build with it, to back up the whole data directory (ledger/ and keys/) before upgrading, and what the failure looks like. A build that is already released (64c518e and earlier) cannot be changed, so this is the only protection for a rollback to those builds.
    2. A format marker: ledger/format, one integer. This build writes it (1 = the 0.6 checkpoint stack) when it opens the ledger and refuses to open a ledger whose marker is higher than it knows, with this ledger was written by a newer plugin (format N, this build reads up to M); downgrade is not supported: restore a backup of the ledger or run the newer plugin. It guards the next format change and any later rollback to a build with this check; it cannot help a rollback to a build before it, which is why (1) matters.

    Constraints: no change to ledger contents, checkpoints, wire formats or any signed or served artifact; the marker is a local file no peer reads. capsules stays a required plugin, so the refusal is a clear start error, not a read-only mode (a read-only mode would be new machinery).

    One-way doors: a new on-disk file in the ledger directory is a local storage format with no product meaning. It is engineering's call under tracking.md; the release-note wording is Sherlock2's. Both go to Sherlock2 (and the twin) for a door confirmation and the wording review; merge waits for the answers.

    The bar: this moves no bar (THE_LOOP.md names nothing on downgrade or rollback); no Steven ruling applies ("none yet").

  3. ethanyzhang commented on Nov 18, 2026

    @ethanyzhang
    CollaboratorAuthor

    (Ethan's agent) Done: merged in #309 (c115512, head 39544b2, 13 checks pass, one skipped; review record #309 (comment)).

    Both wanted parts are in:

    • A clean refusal: the plugin writes ledger/format on open and a build that meets a higher number stops at start with this ledger was written by a newer plugin (ledger format N, this build reads up to M); downgrade is not supported ..., before a pending new history can move the ledger. Tests: ledger_format unit tests, open_refuses_a_ledger_written_in_a_newer_format, a_newer_format_ledger_is_refused_before_a_pending_new_history_moves_it; cargo test in crates/evidence-plugin exit 0 (1518 passed).
    • Release notes: the Unreleased CHANGELOG entry for Move the plugin to the checkpointed-local-log 0.6 stack (agent-action-capsule 0.11, evidencebook, evidence-request) #300 says there is no downgrade from this build, to copy the whole data directory first (keys/ is private), and that capsules export-flat is no way back.

    Limit, as the changelog says: builds before this one do not read the marker, so a rollback to them (the lab-1 case, 64c518e) is still unsafe; only the backup protects it. No release or tag was cut. Not established, as in the issue: whether a pre-upgrade copy restores a working older node.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions