[sanchitrk]

Problem

Each thread row had a recovery mark: execution IS NOT NULL meant "resume me on the next start". A run that ended visibly (truncated, interrupted, crashloop) cleared it unconditionally. But a prompt sent while that run was still going had already woken the thread, so its run followed straight away in the same busy period, without the mark. A host crash in that run lost the prompt, because nothing resumed it.

A first fix kept the mark for prompts admitted after the run began. Review found two holes in it: the run-start position was read after sandbox/MCP setup, and a move to steer or a same-id resend wakes the thread without a new admission. Both come from the same root cause: the real question is "did the in-memory doorbell ring?", and a stored copy of that drifts.

Design: head and tail on the session log

Two positions on the session log decide what runs:

Position	Where	Meaning
tail	inbox.tail	seq of a prompt's admission or its latest move to steer, written in that event's commit
head	thread.head	seq of the last unanswered run end (run.failed, execution.failed, user execution.interrupted), written only by the projector, as MAX(head, seq)

• runnable: a pending prompt with tail > head.
• unfinished: the last run was interrupted by crash or shutdown, or the thread has a runnable prompt.
• Startup: the sweep turns open runs into crash. A run a crash or shutdown cut off is claimed (claimRestart counts only crashes), then resumes, or ends as crashloop at the limit. A thread listed only for a runnable prompt gets a plain wake, as a live prompt would.
• Move to steer: asking for the steer lane is recorded even for a prompt already in it, so "send now" moves its tail and runs a parked steer.
• One run per drain that runs: execution.started is published before anything is promoted, and the run gets exactly one terminal. When a drain fails or is stopped, the stored run state decides whether that terminal is still owed.
• A failed run parks its queue: a run that ends without an answer never starts the next one. Prompts queued before or during its end stay parked, and run with your next prompt. A user stop parks too, even one accepted before its run opened or landing after its run ended. A stop parks the queue rather than withdrawing it; to clear it, cancel first, then stop.
• Removed: the execution column, markRunning, endExecution, idle, the in-memory stop position, and the coordinator's started hook.

updated_at. A thread's updated_at moves with any write to it or anything under it (entries, parts, queued prompts, documents). A session's moves only when its own row is written or a thread is created in it; session lists order by their newest thread's updated_at, read at query time. It is one rule in Store.write: rows a write touches carry its revision, so before commit it moves the threads that own them, and the sessions whose own row it wrote, and stamps the moves for sync. The sync.updated E2E records which rows each change moves.

Two related client fixes in codework run --server (Client.run):

• It now ends on session.run.failed. Before, it waited forever, and the E2E reproduces that hang on the old client.
• It attributes a failure from before promotion by log seq, not by notification order.

Design doc: "Durable Loop Execution & Interrupts Handling" (Linear).

Tests

E2E, all spawning the real CLI and SIGKILLing or SIGTERMing it:

Test	Covers
recovery.parked	prompts and a steer move during a failing run stay parked across a kill; the next prompt runs them all
recovery.after-failure	a prompt after run.failed runs and resumes after a kill
recovery.stop	a stop parks the queued prompt and one sent while it winds down, across a kill
recovery.moved	a parked prompt moved to steer runs at once and resumes after a kill; a same-id resend changes nothing
recovery.shutdown	a clean shutdown resumes with restarts 0, checked while the resumed request is in flight
cli.run-failed	codework run --server exits with error[run.truncated]

The existing recovery E2Es now wait for each request to reach the provider before killing the host, which fixes kill-versus-request races.

Real machines on OrbStack (apps/server/test/recovery.machines.live.test.ts, VM cw-a), all passing:

• a SIGKILLed server;
• a real machine shutdown and boot (orb stop / orb start);
• a stop then crash;
• a crash loop to the restart limit;
• a network cut and a slow link.

The sync machines suite also passes, 12/12.

Isolated tests, only for races an E2E can't time:

• a stop landing while run.failed commits;
• a stop accepted before the run opened;
• a stop before the run opened on a thread a crash cut off, with nothing runnable: it stands across a restart;
• a stop landing after a run's success, before its drain returns;
• a parked steer asked for the steer lane again;
• a prompt accepted after a failure whose host died before its run: it resumes as its wake would, parked follow-up first;
• client notifications reordered.

Each was checked to fail against the old logic.

Checks:

• harness: 616 passed, with 4 settings.llm tests needing a generated model catalog locally;
• codework recovery, client and CLI E2Es: 47 passed;
• vp check and vp run typecheck: clean.

Reviewed three times by Codex gpt-6.1-sol at high effort, the last for runtime parity with the reference durable harness, and once by Fable 5.1 as a distributed systems review; all findings in scope are fixed except the migration note below. The rest predate this PR and are tracked separately.

Note

head and tail are folded into the single init migration, following the repo's convention (as attempt was in #124). A database created before this change must be recreated.

🤖 Generated with Claude Code