Repository navigation
fix(harness): harden durable runs after the head/tail review - #147
Merged
Merged
Conversation
A review of the head/tail design against the reference durable harness found
two bugs, two narrow races and several gaps. Each fix follows a rule from an
existing hardened system; the acceptance bar is the reference runtime.
- One completion per run: `EventList.RunEnded` is the single terminal union.
The ACP `prompt` waited on three of the four terminals and hung forever on
`session.run.failed`; it now returns a JSON-RPC error carrying the reason.
`Client.run` uses the same predicate.
- A stop decides from durable state: when nothing here owns the thread (a
cut-off run not yet resumed, a runnable prompt with no bell, a drain already
settling), `Control.interrupt` records `execution.interrupted { user }` when
the thread is resumable, re-checks for a run that woke meanwhile, and
returns true.
- Level-triggered drain end: a drain re-checks `runnable` as it exits and
wakes again, so a hook `end` or a lost lifecycle publish never strands a
prompt.
- Resume isolation: one thread failing to resume, defects included, no longer
cancels the others.
- One terminal per run: a terminal landing on a run already ended only moves
`head`; the run row keeps the outcome that ended it. Cut-off runs still
owed their end are excepted.
- Shutdown keeps the in-flight attempt and resends it as the same attempt, no
backoff and no retry charged. A crash still counts as a failed attempt.
`attempt` now always names the request in flight or the next to send.
- Restarts are a receive count: a crash counts unless the run was asleep in
retry backoff (`attempt` set, no open draft). Crashes between turns and at
run start count again.
- Lane order at a final boundary follows the reference: steers and the oldest
follow-up are promoted together in admission order; boundaries after tool
results promote steers only.
- Partial text after a crash is healed as `aborted`.
- Lifecycle publishes check the thread inside the write, so a deleted session
cannot regrow its log. Subagent threads are no longer excluded from resume.
- A database created before `head` and `tail` fails at startup with one
message naming the file and saying to recreate it.
Tests: E2E, spawning the real CLI and killing it. New `recovery.backoff`,
`recovery.isolation`, `acp.run-failed`, `cli.old-database`; `recovery.stop`,
`recovery.shutdown` and the healed-partial artifacts regenerated for the new
order, the carried attempt and the `aborted` reason. Isolated tests only for
the races an E2E cannot time, each checked to fail on the old logic.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A review of the head/tail design (#146) against the reference durable harness found two bugs, two narrow races and several gaps:
promptwaited onexecution.succeeded/failed/interruptedonly. A run ending insession.run.failed(truncated, interrupted with no retry left, crashloop) never settled it, so ACP hung forever.resumeUnfinishedusedEffect.ignore, which lets defects through: one thread failing to resume cancelled every other thread's resume.run.failedand the drain's close rewrote the run's outcome.endleft runnable items with no wake.error, notaborted.Design
Each fix follows one rule from an existing hardened system. The acceptance bar is the reference runtime; nothing goes past it.
EventList.RunEndedterminal union; ACP andClient.runwait on it. ACP mapsrun.failedto a JSON-RPC error with the reasonFUTEX_WAIT addr, expectedControl.interruptwith nothing to interrupt decides fromresumable, recordsexecution.interrupted { user }, re-checks for a run that woke meanwhile, returns trueuv__loop_aliverunnableon exit and wakes againone_for_oneresumeUnfinishedusesignoreCauseper threadERESTARTSYSattemptand resends it as the same attempt, no backoff, no charge. Partial text healed asabortedReceiveCountattemptset, no open draft). No new columnapplyBoundaryadmitted_seqorderhead; cut-off runs still owed their end are exceptedAlso: lifecycle projections check the thread inside the write (
RETURNING id, die rolls the event back); subagent threads are no longer excluded fromunfinished.Tests
E2E, spawning the real CLI and SIGKILLing or SIGTERMing it, each ending in an
__artifacts__snapshot:acp.run-failedrecovery.backoffrecovery.isolationcli.old-databasehead/tailexits 1 with the messagerecovery.stop,recovery.shutdown, healed-partial artifactsattempt,abortedreasonIsolated tests only for races an E2E cannot time, each checked to fail on the old logic: stop with no drain left to interrupt; stop between
run.failedand close; hookendwith a queued follow-up; delete racing a lifecycle publish; restart charging by run state; subagent thread listed as unfinished.Checks:
vp checkandvp run typecheckclean; codework recovery/ACP/CLI/client/server/sync 108 passed (6 live OpenRouter cases fail for lack of credits); harness 627/628 excluding live files (one bash output timing flake that passes alone).Reviewed by an independent Opus 5.5 pass; its one likely bug (crashes between turns uncharged) and three gaps are fixed here.
🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.