Dead-letter give-ups: adopt co-core 0.37.0's dead_lettered reason (cannobserv#536), with the 0.32.0 → 0.37.0 bump

Context. Since #17, a command Processor gives up on (the escape cap) gets processing_failed with terminal=true and reason=extraction_error, with the cause in detail (dead-lettered: gave up on attempt 3: …). Watcher (CannObserv/watcher#325, issuecomment-5961451300) treats reason and detail as opaque, so it routes these down its extraction-failure path: ERROR on one item, which suggests the item's spec is bad when the fault is Processor's.

At the next content.process contract bump, Watcher will ask cannobserv for a reason of its own, e.g. processor_error (terminal, no re-fetch). Nothing is needed from Processor before then.

When cannobserv ships it:

Bump co-core in lockstep with Watcher (D5): the parity corpus, real and synthetic, passes unchanged, and the bump note says whether output moved.
handler.publish_gave_up publishes the new reason instead of extraction_error; update its unit tests and the consumer's cap tests.
Spec §4 (the Dead-lettering a decodable command publishes a terminal processing_failed, so Watcher isn't left waiting #17 note and both escape rows) and DEPLOYMENT.md's dead-letter line.

Blocked on cannobserv (the contract) and Watcher (the ask). Refs #17.

From the burn-down epic #24, 2026-10-11. Watcher's ask is filed: CannObserv/cannobserv#536 (ProcessingFailureReason: add a terminal processor_error, open, filed 2026-10-06). It asks to ship in the release that carries cannobserv#523.

Re-entry: a cannobserv release containing #536. A new ProcessingFailureReason is a contract change, so the bump is coordinated with Watcher (D5 as amended by #47: Processor owns the pin; Watcher on contract changes). If #29 or #37 hasn't already done the co-core catch-up from 0.19.7, this bump does it (see the notes there).

The body's "Watcher will ask cannobserv" is now done; nothing else here has moved.

From the burn-down epic #24, 2026-10-12. The co-core catch-up is done: #29 (#51) took co-core 0.19.7 → 0.32.0, read the whole range, and deployed as processor_version 0.32.0+1. This issue's bump now starts from 0.32.0, so it reads only the CHANGELOGs after that. Ignore the earlier note saying this bump might carry the catch-up.

#29 also set the pattern to follow:

regenerate goldens from co-core directly, with a new fixture added before regenerating;
list moved digests in the bump note (Take co-core's XML and feed extractors in step with Watcher, and re-run parity with a feed fixture #29 moved none of the 13);
a refused spec is a golden ({"refused": …});
give Watcher notice of the new processor_version, and coordinate first if the contract moves.

Its trigger was not met by v0.32.0: the upstream issue is still open.

cannobserv v0.37.0 is published (tag, release commit 49d42aca, Publish run green with 4 wheels + 4 sdists; CHANGELOG). It is a MINOR, but every entry is additive: the owner chose MINOR to mark the headless-Watcher contract set (CannObserv/cannobserv#568). The sibling floors are >=0.37,<0.38, so a pin <0.37 (or ==0.32.0) needs an explicit bump to pick it up.

The reason this issue waits for is released (CannObserv/cannobserv#536). Two changes to the body's plan:

The token shipped as dead_lettered, not processor_error (owner, 2026-10-17, on CannObserv/cannobserv#536). It is terminal, with no re-fetch. It's named for the one act that path performs, so no broader processor fault reads into it.
- Only handler.publish_gave_up moves to it. It covers a command whose attempt at the cap raised past the handler's own error handling: a bug outside that handling, or a non-transient publish refusal. It's published just before the dead-letter, unless an earlier fact for the command already went out. After a refused ack, that fact stands and no dead_lettered follows.
- The handler's strike cap stays extraction_error. Every attempt timing out, crashing or raising outside the parser can't be told from a document that defeats the parser. Accepted limit: a raise outside the parser is Processor's fault but still reads extraction_error.
The "Bump co-core in lockstep with Watcher (D5)" checkbox is superseded (Close out the cutover: spec §6 step 5 done, and re-decide D5 (lockstep pin, Watcher-generated goldens) now that Watcher doesn't extract #47; content.process issuer + content.derived consumer: process_commands, PROCESSING status, reaper, shadow comparator watcher#325's restatement, 2026-10-10). Processor owns its pin, and Watcher gets notice before the bump. An un-upgraded Watcher decodes dead_lettered (reason is a str on the consumer class) and routes it like a give-up today, so nothing regresses. Watcher's processor-blaming branch follows on its own schedule. The coordinator raises that issue.

The bump runs from ==0.32.0 (processor main 4b3a500) to 0.37.0, so it reads the CHANGELOG entries for 0.33.0 through 0.37.0. The rest of the #29 pattern above still applies.

Hand-off from the burn-down epic #24, 2026-10-17. This is next: the trigger fired. cannobserv v0.37.0 (2026-10-17) contains CannObserv/cannobserv#536 (closed 13:11Z). Read the release comment above first. This brief adds what the coordinator verified, and the pre-flight against origin/main (4b3a500).

What changed from the body's plan (verified against v0.37.0's CHANGELOG):

The token is dead_lettered, not processor_error. It is terminal, with no re-fetch. It is one new member of the ProcessingFailureReason Literal on ProcessingFailedEmit; schema_version stays 1.
Scope: the dead-letter path only.
- handler.publish_gave_up (handler.py:276; its fact is built at line 284) moves to dead_lettered. It is called from consumer._publish_gave_up (consumer.py:278).
- The handler's own strike-cap give-up stays extraction_error (handler.py:145: every attempt timed out, crashed or raised outside the parser). That's upstream's accepted limit, so don't widen the token to it.
D5 as amended by Close out the cutover: spec §6 step 5 done, and re-decide D5 (lockstep pin, Watcher-generated goldens) now that Watcher doesn't extract #47 applies, not the body's "lockstep" box. A new token is a contract change, so Watcher gets notice before the bump, with the new processor_version. Verified: in co-core 0.32.0, Watcher's pin, ProcessingFailedEvent.reason is a plain str (co_core/pure/models/changes.py:1292). So an un-upgraded Watcher decodes dead_lettered and routes it as it routes a dead-letter today [measured from the installed 0.32.0 source]. Watcher's processor-blaming route is already filed: Route content.derived's dead_lettered to a processor-blaming failure, not the extraction-failure path (cannobserv#536) watcher#386 (open, 2026-10-17). Post the notice there, or on content.process issuer + content.derived consumer: process_commands, PROCESSING status, reaper, shadow comparator watcher#325 as Take co-core's XML and feed extractors in step with Watcher, and re-run parity with a feed fixture #29 did on its own issue. That cross-repo post needs the operator's approval.

The bump: 0.32.0 → 0.37.0 (co-core[extract], co-core-aio[bus], co-core-sync[gcs] in pyproject.toml:11-13), so processor_version becomes 0.37.0+1. LOCAL_GENERATION stays 1. Follow #29's pattern and DEPLOYMENT.md's "Bumping co-core".

Pins held by tests: tests/test_pin.py:18 (EXPECTED), and scripts/build_wheelhouse.sh:11's default tag (Take co-core's XML and feed extractors in step with Watcher, and re-run parity with a feed fixture #29's CR 3 holds it to the pin).
Catch-up reading, 0.33.0–0.37.0: 0.33.0–0.36.0's Downstream lines are cli-only, as read at each check-in. 0.37.0 is additive: the info.watch-events and info.watch-check streams, three WatchStatusState levels, FetchPolicyState.generation, and dead_lettered. Processor consumes none of the new streams. Still check it in code, not only in the CHANGELOG (Take co-core's XML and feed extractors in step with Watcher, and re-run parity with a feed fixture #29's rule): diff ContentProcessCommand, ProcessingFailedEmit, ContentDerivedEmit, canonical_text, resolve_dispatch_essence and the extract package across v0.32.0..v0.37.0.
Goldens: regenerate from co-core directly, and list every moved digest in the bump note. None are expected to move (no extract change in range); a move means stop and explain. The real corpus must pass unchanged.

Pre-flight: what moves, and who reads it today:

tests/test_handler.py:418: test_publish_gave_up_publishes_one_terminal_extraction_error asserts the old token (rename it, and assert dead_lettered).
tests/test_consumer.py: the give-up facts around lines 467 and 705–709 (fact.detail == f"dead-lettered: {meta.reason}"), plus the escaped-strike case at line 539 ([f.reason for f in await facts(admin)] == ["extraction_error"]). Sort them: a give-up through publish_gave_up becomes dead_lettered, and the handler's own cap stays extraction_error. Keep detail's dead-lettered: … prefix: the DLQ and the fact agree on it (line 709), and Watcher stores detail as written.
Docstrings: handler.py:11–14, consumer.py:17 ("a decodable command whose fact never went out gets a terminal extraction_error").
Spec §4's failure table (lines 313 and 315: an escape past the handler, a non-transient publish failure) and the give-up prose around lines 361–387 (reason=extraction_error). The rows for the child's timeout or crash (line 312) and the extractor raising (line 311) stay extraction_error.
Watcher's reading of detail is unchanged, since it never parses it.

Traps:

"An earlier fact already went out" stays as it is. After a refused ack, the fact that went out stands and no dead_lettered follows (upstream's scope, and Consumer: a re-run at the cap after the failure fact went out re-extracts instead of dead-lettering #28's give-up memory). Check that no test now expects a second fact.
Live proof has a floor. The dead-letter path doesn't fire under normal traffic, so the live evidence is the deploy recipe: the starting record shows processor_version: 0.37.0+1, each source's digest is unchanged across the bump (F139), and NRestarts=0. The new token is proven on the scratch Redis by the integration tests. Tests never touch the real broker.
Interaction with Follow the broker to Valkey 7.2 (CannObserv/broker#85): CI image, scratch server, 7.0.15-only claims #54 (the broker's Valkey window): none. They're separate windows. If the broker's window lands first, finish whichever deploy is in flight before starting the other.

Gates (operator): approval of the Watcher notice before the deploy. The deploy itself goes through #34's gate.

Done when:

the pin is ==0.37.0 on all three packages, and processor_version is 0.37.0+1;
publish_gave_up publishes dead_lettered, the handler's cap still publishes extraction_error, and the tests above are updated, with integration tests run on the scratch Redis;
the bump note gives the catch-up reading (checked in code), the goldens (moved: none, or each one explained) and the real corpus;
the spec, the docstrings and DEPLOYMENT.md are updated where they name the give-up's token;
Watcher's notice is posted, with approval, before the deploy;
deployed: the live checks in trap 2 pass.

Out of scope: #37 (cannobserv#354 is still open), #54, and widening dead_lettered beyond the dead-letter path.

Ship: issue → branch in a worktree → plan (short) → TDD → PR (Refs #21: the notice and the deploy come after the merge) → FF merge → git pull --ff-only → the notice → scripts/deploy.sh → the live checks. The closing comment gives each box with its evidence, the known limits, and the follow-ups (or "none").

Kickoff prompt:

Pick up processor#21. Read its hand-off comment from epic #24 first.

Bump note: co-core 0.32.0 → 0.37.0 and dead_lettered. Shipped in #55, which was fast-forward merged as 5829e4b. Deployed at 2026-10-17T23:08:56Z as processor_version 0.37.0+1.

Catch-up reading, checked in code (git diff v0.32.0 v0.37.0 -- packages/) and against the CHANGELOG:

Unchanged: the extract package, canonical_text, spec_fingerprint, resolve_dispatch_essence, util.blobstore / util.hashing, co_core_sync.drivers.blobstore, effects.bus, dead_letter, the bus exceptions, and co_core_aio.bus (its __init__ changes only the version).
changes.py: ProcessingFailureReason gains "dead_lettered", and ProcessingFailedEvent's docstring changes. The fields and validators of ContentProcessCommand, ProcessingCompleteEmit and ProcessingFailedEmit are unchanged. The rest of the file covers the watch pair, FetchPolicyState.generation and the WatchStatusState levels, none of which Processor reads.
envelope.py / streams.py: additive only, the info.watch-events / info.watch-check streams and their key rules.
Dependencies: each package's pyproject.toml moves only its version and the sibling floor.
CHANGELOG: the Downstream lines for 0.33.0–0.36.0 are cli-only. 0.37.0 is additive. The range has no security entries.

Goldens: co-core 0.37.0: 0 moved. Only the co_core stamp changed. Real corpus: passes unchanged.

Token: publish_gave_up now publishes dead_lettered, and detail keeps its dead-lettered: … prefix. The handler's strike cap still publishes extraction_error. A refused ack at the cap still adds no fact (test_a_refused_ack_at_the_cap_adds_no_failure_fact, unchanged). Full suite, integration included on the scratch Redis, with containment required: 650 passed. CI on main: run 39077616568, lint and test green.

Watcher notice: posted on watcher#386 before the deploy, with approval: CannObserv/watcher#386 (comment)

Live so far: REVISION is 5829e4b3a0aa. starting shows processor_version 0.37.0+1, child_containment required, Landlock ABI 6. NRestarts=0. Since go-live, the journal has no dead-lettering record. Each source's first post-bump digest is still pending: the sources run from hourly to daily. The closing comment will compare all 5 sources against their last 0.32.0+1 digests.

Closing: shipped, deployed, and checked live. PR #55, fast-forward merged as 5829e4b. Deployed at 2026-10-17T23:08:56Z. The bump note is above.

Done when:

The pin is ==0.37.0 on all three packages, and processor_version is 0.37.0+1. pyproject.toml, uv.lock, tests/test_pin.py EXPECTED and scripts/build_wheelhouse.sh's default tag all moved. LOCAL_GENERATION stays 1.
publish_gave_up publishes dead_lettered, and the handler's cap still publishes extraction_error. The tests are updated: test_publish_gave_up_publishes_one_terminal_dead_lettered and the consumer's five give-up tests that go through it. The handler-cap tests still assert extraction_error. A refused ack at the cap still adds no fact. The full suite passed with integration tests on the scratch Redis and containment required: 650 passed. CI on main passed: run 39077616568, lint and test.
Bump note. The catch-up reading was checked in code, no golden moved (co-core 0.37.0: 0 moved), and the real corpus passes unchanged.
Docs. The spec has a 2026-10-17 amendment and updates to §4's two escape rows and the Dead-lettering a decodable command publishes a terminal processing_failed, so Watcher isn't left waiting #17 prose, D5, §2, §5, §7's pin test, the quick reference and OQ2. Also updated: the docstrings in handler.py and consumer.py, DEPLOYMENT.md's dead-letter line, and AGENTS.md.
Watcher's notice was posted, with approval, before the deploy: Route content.derived's dead_lettered to a processor-blaming failure, not the extraction-failure path (cannobserv#536) watcher#386 (comment)
Deployed, and the live checks pass.
- REVISION is 5829e4b3a0aa, and starting shows processor_version 0.37.0+1, child_containment required, Landlock ABI 6.
- NRestarts=0.
- Every command since the deploy completed: 29 of 29, with no ERROR or WARNING records.
- Each source's first 0.37.0+1 digest equals its last 0.32.0+1 digest:

`info_source_id`	First fact on `0.37.0+1`	`output_digest`
`01KRYSPTN633JZ8582NHHSAH9A`	2026-10-17T23:46:02Z	`sha256:80ea56dd…` unchanged
`01M4RS9CWFZCDNRJXTPTEC6WZ5`	2026-10-18T04:42:02Z	`sha256:f5339c1d…` unchanged
`01KTCNXKXN7Z94G495YEGM6JX3`	2026-10-18T19:33:02Z	`sha256:f77c3471…` unchanged
`01KTVPAV7RW8TDZH88A43AFT04`	2026-10-18T19:33:02Z	`sha256:8f0f09f6…` unchanged
`01KVP9B9HVZ9NK1HRWT5569R62`	2026-10-18T22:31:02Z	`sha256:36e8c9f2…` unchanged

Known limits:

dead_lettered is not proven live. The dead-letter path has never fired in production: the journal since go-live (2026-10-02T23:27Z) has no dead-lettering record. The token is proven only by the integration tests on the scratch Redis.
The handler's strike cap still reads extraction_error, including a raise outside the parser that is Processor's fault. That is upstream's accepted scope (cannobserv#536).

Follow-ups: none on Processor's side. Watcher's processor-blaming route for dead_lettered is CannObserv/watcher#386, on Watcher's schedule. Until it lands, Watcher routes the token as it routes a give-up today.

Navigation Menu

Repository navigation

Dead-letter give-ups: adopt co-core 0.37.0's dead_lettered reason (cannobserv#536), with the 0.32.0 → 0.37.0 bump #21

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Dead-letter give-ups: adopt co-core 0.37.0's dead_lettered reason (cannobserv#536), with the 0.32.0 → 0.37.0 bump #21

Description

Activity

gregoryfoster commented on Oct 11, 2026

gregoryfoster commented on Oct 12, 2026

gregoryfoster commented on Oct 17, 2026

gregoryfoster commented on Oct 17, 2026

gregoryfoster commented on Oct 17, 2026

gregoryfoster commented on Oct 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions