Skip to content

fix(harness): harden durable runs after the head/tail review - #147

Merged
sanchitrk merged 1 commit into
canaryfrom
fix/durable-hardening
Nov 9, 2026
Merged

sanchitrk merged 1 commit into
canaryfrom
fix/durable-hardening

Conversation

@sanchitrk

@sanchitrk sanchitrk commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A review of the head/tail design (#146) against the reference durable harness found two bugs, two narrow races and several gaps:

  • The ACP prompt waited on execution.succeeded/failed/interrupted only. A run ending in session.run.failed (truncated, interrupted with no retry left, crashloop) never settled it, so ACP hung forever.
  • resumeUnfinished used Effect.ignore, which lets defects through: one thread failing to resume cancelled every other thread's resume.
  • A stop arriving while a drain settled, or before a cut-off thread was claimed at startup, was dropped; the prompt or run then ran by itself at the next start.
  • A stop landing between run.failed and the drain's close rewrote the run's outcome.
  • A hook end left runnable items with no wake.
  • A parked follow-up ran after a newer steer, against the doc's "oldest first".
  • A clean shutdown reset the retry budget; partial text after a crash was marked error, not aborted.
  • A crash while asleep in retry backoff counted toward the restart limit.
  • A lifecycle publish racing a session delete could recreate the session's log.
  • An old database failed with an obscure missing-column error.

Design

Each fix follows one rule from an existing hardened system. The acceptance bar is the reference runtime; nothing goes past it.

Rule Source Change
One completion per submission io_uring: one CQE per SQE EventList.RunEnded terminal union; ACP and Client.run wait on it. ACP maps run.failed to a JSON-RPC error with the reason
Re-check the condition before you sleep futex FUTEX_WAIT addr, expected Control.interrupt with nothing to interrupt decides from resumable, records execution.interrupted { user }, re-checks for a run that woke meanwhile, returns true
The loop stays alive while work is ref'd libuv uv__loop_alive A drain re-checks runnable on exit and wakes again
Isolate failures per unit Erlang one_for_one resumeUnfinished uses ignoreCause per thread
Return partial progress, restart the rest Linux ERESTARTSYS Shutdown keeps attempt and resends it as the same attempt, no backoff, no charge. Partial text healed as aborted
Receive count, dead-letter at the limit SQS ReceiveCount A crash counts unless the run was asleep in retry backoff (attempt set, no open draft). No new column
Submission order io_uring SQE order; reference applyBoundary At a final boundary steers and the oldest follow-up promote together in admitted_seq order
One terminal per run A terminal on a run already ended only moves head; cut-off runs still owed their end are excepted
Fail loudly on a schema you cannot serve Startup guard names the database file and says to recreate it

Also: lifecycle projections check the thread inside the write (RETURNING id, die rolls the event back); subagent threads are no longer excluded from unfinished.

Tests

E2E, spawning the real CLI and SIGKILLing or SIGTERMing it, each ending in an __artifacts__ snapshot:

Test Covers
acp.run-failed ACP prompt returns an error on a truncated run; the old handler timed out at 60s
recovery.backoff SIGKILL in retry backoff counts no restart; SIGTERM in backoff resends the same attempt at once
recovery.isolation two threads cut off, one dies on resume, the other resumes
cli.old-database a database missing head/tail exits 1 with the message
recovery.stop, recovery.shutdown, healed-partial artifacts regenerated: promotion order, carried attempt, aborted reason

Isolated tests only for races an E2E cannot time, each checked to fail on the old logic: stop with no drain left to interrupt; stop between run.failed and close; hook end with a queued follow-up; delete racing a lifecycle publish; restart charging by run state; subagent thread listed as unfinished.

Checks: vp check and vp run typecheck clean; codework recovery/ACP/CLI/client/server/sync 108 passed (6 live OpenRouter cases fail for lack of credits); harness 627/628 excluding live files (one bash output timing flake that passes alone).

Reviewed by an independent Opus 5.5 pass; its one likely bug (crashes between turns uncharged) and three gaps are fixed here.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

A review of the head/tail design against the reference durable harness found
two bugs, two narrow races and several gaps. Each fix follows a rule from an
existing hardened system; the acceptance bar is the reference runtime.

- One completion per run: `EventList.RunEnded` is the single terminal union.
  The ACP `prompt` waited on three of the four terminals and hung forever on
  `session.run.failed`; it now returns a JSON-RPC error carrying the reason.
  `Client.run` uses the same predicate.
- A stop decides from durable state: when nothing here owns the thread (a
  cut-off run not yet resumed, a runnable prompt with no bell, a drain already
  settling), `Control.interrupt` records `execution.interrupted { user }` when
  the thread is resumable, re-checks for a run that woke meanwhile, and
  returns true.
- Level-triggered drain end: a drain re-checks `runnable` as it exits and
  wakes again, so a hook `end` or a lost lifecycle publish never strands a
  prompt.
- Resume isolation: one thread failing to resume, defects included, no longer
  cancels the others.
- One terminal per run: a terminal landing on a run already ended only moves
  `head`; the run row keeps the outcome that ended it. Cut-off runs still
  owed their end are excepted.
- Shutdown keeps the in-flight attempt and resends it as the same attempt, no
  backoff and no retry charged. A crash still counts as a failed attempt.
  `attempt` now always names the request in flight or the next to send.
- Restarts are a receive count: a crash counts unless the run was asleep in
  retry backoff (`attempt` set, no open draft). Crashes between turns and at
  run start count again.
- Lane order at a final boundary follows the reference: steers and the oldest
  follow-up are promoted together in admission order; boundaries after tool
  results promote steers only.
- Partial text after a crash is healed as `aborted`.
- Lifecycle publishes check the thread inside the write, so a deleted session
  cannot regrow its log. Subagent threads are no longer excluded from resume.
- A database created before `head` and `tail` fails at startup with one
  message naming the file and saying to recreate it.

Tests: E2E, spawning the real CLI and killing it. New `recovery.backoff`,
`recovery.isolation`, `acp.run-failed`, `cli.old-database`; `recovery.stop`,
`recovery.shutdown` and the healed-partial artifacts regenerated for the new
order, the carried attempt and the `aborted` reason. Isolated tests only for
the races an E2E cannot time, each checked to fail on the old logic.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sanchitrk
sanchitrk merged commit 3cab529 into canary Nov 9, 2026
1 check passed
@sanchitrk
sanchitrk deleted the fix/durable-hardening branch November 9, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant