From 0548cc6c66f817a1a958fe879b5efe9633e32013 Mon Sep 17 00:00:00 2001 From: David Abram Date: Thu, 8 Oct 2026 23:56:31 +0200 Subject: [PATCH 01/16] context: Define mutation-cursor reconciliation wiring plan Document how to make the existing mutation-cursor ref reconciler operational through shared mutation-scope maintenance and `sce doctor --fix`, including automatic throttling, safety constraints, tests, and context synchronization. Plan: mutation-cursor-ref-reconciliation-wiring (T01-T06) Co-authored-by: SCE --- ...tation-cursor-ref-reconciliation-wiring.md | 300 ++++++++++++++++++ 1 file changed, 300 insertions(+) create mode 100644 context/plans/mutation-cursor-ref-reconciliation-wiring.md diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md new file mode 100644 index 000000000..581806651 --- /dev/null +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -0,0 +1,300 @@ +# Plan: mutation-cursor-ref-reconciliation-wiring + +## Change summary + +Make the existing mutation-cursor snapshot-ref reconciler (`runtime/ref_reconciliation.rs`, async `reconcile_worktree`) operational. Today it is "retained and unwired": nothing in production calls it, so pins left by interrupted `coordinate()` paths accumulate forever. + +This plan wires reconciliation in two deliberately different ways: + +- **Actual reconciliation (Git and DB I/O) runs only through `sce doctor --fix`.** A new runtime maintenance entrypoint performs exactly one `WorktreeLock` acquisition, verifies and opens the existing repository Agent Trace DB without side effects, runs the existing reconciliation algorithm and atomic deletion transaction, and records a diagnostic result. +- **The mutation-hook completion path (`Close`/`Flush`) performs only a lightweight advisory check.** It never lists refs, opens the DB, or runs reconciliation. At most once per advisory window it records that reconciliation is recommended in a small durable state file; `sce doctor` (read-only) reports the recommendation and `sce doctor --fix` acts on it. + +**Fully automatic reclamation is not delivered by this PR.** No lifecycle-owned, noncritical-path execution point exists today (see D2), and inline reconciliation on the tool-completion path cannot be bounded. Automatic out-of-band reconciliation is a documented follow-up PR with stated prerequisites. Nothing in this plan or in the synchronized context may claim that orphan pins are reclaimed automatically. + +The change extends existing behavior and preserves the reconciliation algorithm, `GitSnapshotService`, `MutationTraceStore`, `WorktreeLock`, the atomic SHA-conditional ref deletion transaction, the protocol, the Quint model, and mutation-attribution rules. It must never delete a tree required by any repository-wide durable root and must never change the success/failure outcome of a mutation boundary. + +Review history (all resolutions are binding on T01–T06): + +*Round 1 (six findings)* + +1. **Single lock acquisition (P1).** `reconcile_worktree_inner` already acquires `WorktreeLock` (`ref_reconciliation.rs:172`). The reconciler is split into a lock-acquiring wrapper and a shared lock-held implementation. Each maintenance operation acquires the lock exactly once. See D1. +2. **Hook latency (P1).** Resolved definitively in round 2: no reconciliation on the hook path. See D2. +3. **Failure state (P2).** Failures are durably recorded as diagnostics with a failure streak. Round 2 removed automatic retry scheduling because no automatic full pass exists. See D3. +4. **Corrected AC3 (P2).** Two invariants are claimed, not "every durable root is pinned": local consistency (fail closed, delete nothing) and repository-wide deletion safety (never remove another worktree's protecting pin). Reconciliation never repairs another worktree's missing pins. +5. **PR metadata (P2).** The real PR is #305, head branch `ref-recon`, base `fix-logging`. +6. **Database-opening contract (P2).** The existing hook opener is not read-only. Reconciliation uses a dedicated verified-existing-DB opener with no create/migrate/repair/initialize side effects. See D4. + +*Round 2 (three findings)* + +7. **Unbounded inline latency (P1).** A cooperative budget checked between phases cannot bound `git for-each-ref`, DB open, repository-wide root queries, `git update-ref`, or Tokio blocking-worker shutdown, and reconciliation holds the lock that the coordinator waits on for 10 s. Selected: advisory-only hook trigger, reconciliation through `sce doctor --fix`, automatic execution deferred to a follow-up. The 250 ms budget, size/cost gates, and deferral machinery are removed. See D2. +8. **Repeated scans after state-persistence failure (P2).** The reservation-before-expensive-work requirement applies to any *automatic* full pass. Because this PR has none, no reservation infrastructure is built; the requirement is recorded as a binding prerequisite of the follow-up. The advisory path emits a recommendation only after the state write succeeds, and `Completed` vs `CompletedStatePersistFailed` stay distinct for explicit passes. See D2/D3. +9. **Far-future timestamps (P2).** The state file stores only past-event timestamps and never a future deadline; eligibility is computed as `now − timestamp ≥ window`, so rereading cannot extend a deadline. An impossible timestamp is normalized once under `WorktreeLock` and persisted. See D3. + +Stacking: + +- Repository: `crocoder-dev/shared-context-engineering` +- Parent: PR #304, branch `fix-logging`. PR #304 is not modified by this plan. +- This work: PR #305, head branch `ref-recon`, base `fix-logging`. No other branch or PR is created. +- Verified at authoring time: PR #304 head `0be74aad5bea5070828846d3277041a4649c9842`; PR #305 head is whatever `ref-recon` points to (it advances with every plan or implementation commit, so T01 re-records the then-current heads rather than relying on a value written here). +- If #304 advances, rebase `ref-recon` onto its latest head and record the exact SHA in the T01 audit findings. + +## Authoring-time observations (to be re-verified by T01) + +Facts the design relies on. T01 must re-read each cited location and either confirm or flag it as invalidating a later task. + +- `ref_reconciliation.rs:159-243` `reconcile_worktree_inner` acquires `WorktreeLock` via `acquire_inner_async` with a fixed 10 s timeout (`:172`), resolves the worktree ID, calls `open_db`, lists pins, loads local and repository-wide durable roots, and deletes through `delete_pins(lock.lease(), …)` (`:233`). The `lock` local is dropped on return. +- `worktree_lock.rs`: the lock is a non-reentrant advisory file lock (`File::try_lock`) on `/sce/mutation-cursor.lock`; `acquire_inner` polls every 100 ms until the deadline, so a zero timeout is exactly one `try_lock` attempt. `WorktreeLockLease` is an `Arc` clone of the lock inner; the file unlocks only when the last clone drops. +- `git_snapshot.rs:281-300` `run_ref_mutation_inner` moves the `WorktreeLockLease` into the `spawn_blocking` closure (`let _lease = lease;`). A dropped caller future therefore cannot release the file lock while the blocking `git update-ref` worker runs. `delete_pins_inner` has a post-preflight seam but `run_ref_mutation` passes a no-op `on_worker_entered`. +- `protected_worktree.rs:14` `WORKTREE_LOCK_TIMEOUT` is 10 s: a competing `coordinate()` fails its boundary (before durable completion, with the existing lock-timeout error) if any holder keeps the lock longer than that. +- `protected_worktree.rs:178-184` `abandon_after_spawn_without_unlock` leaves the lock held for an external-guard child; other lock users must skip or wait per their own policy. +- `hooks/mutation_scope.rs:356-378`: `classify_coordinate` maps `Ok(_)` and `MarkerClearAfterCommit` to `Ok(String::new())` and every other `CoordinateError` to a boundary failure. `drive_mutation_scope` returns after `coordinate_boundary(...).await`, so the coordinate-held lock is released before any code placed after it runs. +- `hooks/runtime.rs:53-69` `open_agent_trace_db_for_hook_runtime` → `agent_trace_storage/mod.rs:66-74, 195-207` `open_repository_db_for_hook_runtime`: (a) `TursoDb::open_without_migrations_at` (`db/mod.rs:856-860`) calls `ensure_db_parent_dir` and `turso::Builder::new_local(...).build()`, which can create parent directories and an empty DB file; (b) on schema-not-ready it calls `repair_missing_repository_schema_migration_metadata` (`agent_trace_db/repository.rs:136`), which inserts the baseline migration row; (c) it then calls `verify_or_initialize_repository_metadata` (`repository.rs:171`), which inserts repository metadata when absent. It avoids schema migrations but is **not** read-only. +- Pin count is not bounded by orphan count: every retained historical tree root keeps its pin (`A → B → C → D` retains all four), so inventory and durable-root load cost grow with history. A full pass is O(history), not O(orphans). +- Tokio runtime shutdown joins in-flight `spawn_blocking` workers, so dropping a future (for example through `tokio::time::timeout`) does not shorten process latency for work already in a blocking worker. +- `resolve_git_dir` and `resolve_worktree_id` spawn `git` subprocesses (`runtime/git_snapshot.rs`). Any hook-path advisory that called them would add a new subprocess to the completion path. +- Candidate execution points for out-of-band work found in the tree: the Git `post-commit` subcommand (`hooks/commit_hooks.rs:114`), per-harness session-end/stop events (for example `hooks/claude_mutation_scope/lifecycle.rs`, `hooks/codex/stop.rs`, Codex session-end fixtures), `sce setup` lifecycle, and the doctor command (`doctor/mod.rs`, `doctor/fixes.rs`). None is a lifecycle-owned, noncritical-path executor that is shared across harnesses; see D2. + +## Design decisions + +- **D1 — Single lock acquisition and ownership (round 1).** + - `ref_reconciliation.rs` is split into: (a) `reconcile_worktree` / `reconcile_worktree_inner`, kept with unchanged signatures and behavior, which acquire `WorktreeLock` once and delegate; and (b) a new lock-held implementation `reconcile_with_held_lock` that borrows the held `WorktreeLock`, never calls `acquire_inner_async`, and derives deletion leases from the held lock via `lock.lease()`. + - There are exactly two maintenance operations, each with its own single lock acquisition and neither calling the other: + - `reconcile_explicit` — acquires the lock once with the existing bounded 10 s wait, then runs (b), then records the diagnostic result in the state file while still holding the same lock. It never calls the lock-acquiring wrapper (a). + - `advise_if_due` — hook-path advisory; acquires the lock once with a zero timeout (single `try_lock`), reads/normalizes/writes the state file, releases. It never calls (a) or (b) and has no DB or Git parameters. + - State reads that decide a write and the write itself happen under the same single acquisition. An unlocked read is permitted only as an advisory fast exit (and for read-only `sce doctor` reporting) and is always rechecked under the lock before any write. + - The `WorktreeLock` stays owned by the explicit pass for its whole duration. Mutating Git work receives `lock.lease()` and, as today, moves it into the blocking worker, so cancellation of the caller future cannot release the lock while a Git ref mutation worker is still running. + - The hook ingress invokes `advise_if_due` only after `coordinate_boundary(...)` has returned, so the mutation coordinator's own lock is never held during the advisory. + - No reentrant lock, no lock-passing between processes, no global lock registry. + - Documented concurrency contract: an explicit pass holds the lock for its whole duration because the pin-then-root ordering that makes deletion safe requires it (inventory, root reads, and deletion must not interleave with `coordinate()`'s pin-then-CAS). A mutation boundary arriving meanwhile waits up to `WORKTREE_LOCK_TIMEOUT` (10 s) and then fails *before durable completion* with the existing lock-timeout error. This is an operator-initiated, documented trade-off of `sce doctor --fix`; the hook-path advisory never waits on the lock. +- **D2 — Hook-path trigger is advisory-only; reconciliation is explicit (round 2, resolves P1).** + - *Why inline reconciliation is rejected.* The hook completion path cannot bound `git for-each-ref`, DB open, repository-wide durable-root queries (they scale with historical evidence), or a started `git update-ref` (which must complete safely), and Tokio shutdown waits for blocking workers. A size cap, a historical-cost gate, or a cooperative between-phase budget only predicts cost; none gives an upper bound. Wrapping any step in a timeout would either abort unsafe Git work or fail to shorten latency. Additionally, reconciliation holds `WorktreeLock`, so a slow pass can push an unrelated boundary past its 10 s lock timeout. Cooperative budgets, caps, cost ceilings, deferral windows, and any 250 ms promise are therefore **removed from the design**, and no replacement timeout is introduced. + - *Execution-point investigation.* Alternatives considered and rejected (T01 re-verifies each claim): + 1. Git `post-commit` subcommand — synchronous inside `git commit`, which an agent tool can invoke, so it is the same critical path with the same unbounded work. + 2. Per-harness session-end/stop events — not part of the shared mutation-scope ingress, differ per harness (adapter changes are out of scope), still synchronous hook processes that block session shutdown, and have no uniform lifecycle owner. + 3. `sce setup` / lifecycle service — install-time, not recurring. + 4. Detached Tokio task, fire-and-forget subprocess, background daemon — prohibited (a short-lived hook process cannot own them; no supervision, ownership, or failure recovery exists). + 5. `sce doctor --fix` — operator-owned, foreground, can report its true result. **Selected.** + No suitable lifecycle-owned noncritical-path mechanism exists in the current architecture. + - *Selected design.* + - `Close`/`Flush` (durably completed, including the `MarkerClearAfterCommit` case, after `coordinate_boundary` returned) calls `advise_if_due`, a synchronous function with this complete I/O budget: one capped read (≤ `MAINTENANCE_STATE_MAX_BYTES`) of the state file, at most one zero-wait `try_lock`, at most one atomic write of that small file, all local. It performs no `git` subprocess, no DB open or query, no ref listing, no `spawn_blocking`, no await point after the lock is taken. It is a bounded set of small local file operations of the same class the hook path already performs for the lock and taint marker; it is not claimed to be hard real-time, and any unexpected failure is swallowed into a logged outcome. + - The advisory must not add a new Git subprocess. It obtains the worktree's Git dir without spawning Git (T01 determines whether it can be derived from the filesystem — `.git` directory, or `gitdir:` file for linked worktrees — and proves equality with `resolve_git_dir` for the main and a linked worktree, or whether `coordinate`'s already-resolved value can be reused without protocol changes). **Decision rule:** if neither is possible without a new Git subprocess, the hook trigger is dropped entirely and `sce doctor` computes the recommendation from the state file alone; AC1/AC5 are then satisfied with zero hook callers and T03 reduces to documenting that outcome. This fallback needs no user confirmation because it only removes a convenience signal. + - `advise_if_due` states: absent/invalid-version/oversized/corrupt state → write an anchor and return `Anchored` (no advice); reference time (`last_success`, else `anchor`) older than `RECONCILIATION_ADVISORY_AFTER` and no valid advice within the same window → persist `last_advised`, then (only after the write succeeded) return `Advised`; otherwise `NoAction` with no write. Lock busy → `Busy` (skipped, not an error). State read/write failure → `StateUnavailable` (no advice emitted, no retry loop, no scan). It emits at most one advisory per window per worktree. + - `sce doctor` (read-only) reports whether reconciliation is recommended (last success age, last attempt/failure streak, last advice) computed from the state file as a pure function; `sce doctor --fix` runs `reconcile_explicit` regardless of any state. + - *Deferred to a follow-up PR (explicitly not in this plan).* Fully automatic out-of-band reconciliation. Binding prerequisites for that PR: (1) a lifecycle-owned execution point with defined ownership, lifetime, scheduling and failure recovery (for example an explicitly invoked `sce` maintenance command run by a user-managed scheduler, or a future shared session-end mechanism) — never a detached task, daemon, or timeout-cancelled Git operation; (2) a **durable next-attempt reservation persisted under `WorktreeLock` before opening the DB, listing pins, or loading durable roots**, bounded, never interpreted as success, superseded by the success cadence after success or exponential failure backoff after failure, and bypassed by explicit repair; if the reservation cannot be persisted the automatic pass is skipped and the maintenance-state failure is reported; (3) distinct outcomes for completed, completed-but-state-not-persisted, failed, and skipped/deferred; (4) bounded or split lock-hold analysis against the 10 s coordinator timeout; (5) the same invalid-timestamp normalization as D3. T06 records this follow-up in context. + - *Trade-offs.* (+) hook completion path has no unbounded work and no new failure mode; correct under every ordering with `coordinate()`; no new infrastructure without a consumer. (−) orphan pins are reclaimed only when an operator runs `sce doctor --fix`; the recommendation is time-based (staleness since last success), not evidence-based, because detecting orphans needs the Git/DB scan this design keeps off the hook path. +- **D3 — Durable maintenance state: diagnostics, no stored deadlines (rounds 1–2).** + - One small JSON file per worktree at `/sce/ref-maintenance.json` (versioned, capped at `MAINTENANCE_STATE_MAX_BYTES`, 4 KiB), beside the lock and taint marker. Fields (all optional past-event unix-ms timestamps unless noted): `anchor`, `last_success`, `last_attempt`, `last_advised`; `last_attempt_outcome` (`completed` | `failed`); `last_report` (retained/deleted/local-required counts); `consecutive_failures` (integer); `last_failure` (short classification plus bounded message). + - **No field is ever a future deadline.** Eligibility is computed at read time as `now − reference ≥ window` (saturating at 0). Re-reading therefore cannot move a deadline, and there is nothing to clamp. + - Writes happen only under `WorktreeLock`, via atomic staging-and-swap (temp file in the same directory, then rename; pattern identified in T01) so a crash leaves either the old or the new file. No in-memory registry or process-global state. + - Transitions: advisory — anchor and `last_advised` only. Explicit success — `last_attempt` = `last_success` = now, `last_report` set, `consecutive_failures` = 0, failure fields cleared. Explicit failure (DB unavailable/invalid, malformed ref, missing required pins, durable-root read error, delete-transaction error) — `last_attempt` = now, outcome `failed`, `consecutive_failures` += 1, `last_failure` set, `last_success` untouched. Lock contention (including explicit lock timeout) — **no state change**; never classified as a failure. + - **Invalid timestamp normalization (deterministic).** A timestamp is invalid when it is more than `FUTURE_SKEW_TOLERANCE` (5 min) after the injected `now` (far-future values, or clock rollback beyond the tolerance). A timestamp within the tolerance is treated as age 0. Under `WorktreeLock`, `advise_if_due` and `reconcile_explicit` normalize once: invalid `last_success`/`last_attempt`/`last_advised`/`anchor` values are cleared; if `last_success` and `anchor` are both unusable the state is **immediately eligible** for advice, so the advisory persists `last_advised = now` and returns `Advised` in the same locked operation, and a fresh `anchor = now` is written. The normalized state is persisted before returning, so a second read sees valid timestamps and does not shift anything. Plain `sce doctor` never writes; it evaluates invalid timestamps as "recommended" with an "invalid maintenance timestamps" note. `sce doctor --fix` runs regardless of timestamp validity and rewrites valid state afterwards. + - Recovery: a missing, unparsable, wrong-version, or oversized file is treated as default and rewritten by the next recorded operation; it never fails an explicit pass. + - Result distinction for explicit passes (typed outcome): `Completed(report)`; `CompletedStatePersistFailed(report, warning)` — the cleanup stands and is reported as done, with a visible warning that diagnostics were not recorded; `Failed(error)`; `Skipped(Busy)`. Never merged. Advisory outcomes: `Anchored | NoAction | Advised | Busy | StateUnavailable`. + - Time comes from an injected wall clock (`Fn() -> i64` unix ms) passed as a generic parameter for deterministic tests. +- **D4 — Verified-existing Agent Trace DB opener (round 1).** + - Contract for reconciliation: (1) never create a DB file or any parent directory; (2) never run migrations; (3) never call `repair_missing_repository_schema_migration_metadata`; (4) never call `verify_or_initialize_repository_metadata` or otherwise insert/update repository metadata; (5) verify repository identity (stored `repository_id` equals the identity resolved for this checkout, `source_instance_id` valid) and schema readiness (baseline plus the migration providing durable mutation roots) **before** any ref listing or deletion; (6) any failure returns an error with refs untouched, as a maintenance error that never arms `ExternalTaintMarker` and never becomes `CoordinateError::AgentTraceDbUnavailable`. + - The existing hook opener cannot satisfy (1), (3), or (4), and changing it would alter every hook caller. The smallest satisfying change is a dedicated opener in `agent_trace_storage/mod.rs` (for example `resolve_existing_agent_trace_storage_for_maintenance`) plus one read-only verification method on `RepositoryAgentTraceDb` (for example `verify_existing_repository_metadata(repository_id)`) wrapping the existing `select_repository_metadata_row` without insert. The opener first checks that the resolved DB path exists as a file; if not, it fails before touching Turso. If Turso offers no read-only open, T01 records that and the opener uses only `SELECT`-class statements after open; incidental WAL bookkeeping on an existing DB is an accepted limitation, and tests assert logical tables are unchanged. + - Typed failure kinds (small enum, converted to `anyhow` at the existing `open_db` seam): `Missing`, `Unreadable`, `IncompatibleSchema`, `MissingMetadata`, `RepositoryMismatch`. The existing hook opener and all its callers are unchanged. + - Considered and rejected: opening the DB before taking the lock to shorten lock-hold. It would require a second lock acquisition just to record an open failure, violating the one-acquisition-per-operation rule in D1; the lock-first order of the existing reconciler is kept and the D1 concurrency contract documents the consequence. + +## Acceptance criteria + +How this plan is proven complete. `/validate` runs these checks; no task performs final validation. + +- [ ] AC1: Reconciliation has one real non-test **explicit** caller (`sce doctor --fix`) and no automatic reconciliation caller; the mutation-scope ingress has at most an advisory caller that cannot reach reconciliation. + - Validate: `nix shell nixpkgs#ripgrep -c rg "reconcile|advise_if_due" cli/src --glob '!*test*'` shows the reconciliation entrypoint referenced only from `doctor/` (and its definition module), and `advise_if_due` (if present per the D2 decision rule) referenced from `hooks/mutation_scope.rs`; the hook module imports no reconciliation, `GitSnapshotService`, store, or DB-opener symbol for this purpose. +- [ ] AC2: `sce doctor --fix` deletes an orphan snapshot ref; `Close`/`Flush` never deletes. + - Validate: T05 S1 drives the doctor fix path against a real Git repo and asserts the orphan `refs/sce/mutation-cursor//` is gone; S14 asserts `Close`/`Flush` leaves all refs untouched. This plan does **not** claim automatic orphan reclamation. +- [ ] AC3: A successful pass establishes two distinct invariants, and claims nothing stronger. + - **Local consistency:** `durable_roots(W) ⊆ pinned_trees(W)` is checked before any deletion for the target worktree `W`. If it fails, the pass deletes nothing and returns `MissingRequiredPins`. + - **Repository-wide deletion safety:** `delete(W, T) ⇒ T ∉ durable_roots(repository)`. A pin is removed only when its tree is outside every worktree's durable roots, so reconciliation never removes an existing SCE pin that protects another worktree's required tree. + - Reconciliation does **not** repair, recreate, or verify missing pins belonging to other worktrees, and does not guarantee that every repository-wide durable root is pinned. + - Validate: T05 S2, S3, S4, S5 and S13 pass. S2/S3 prove preservation; S4 proves local fail-closed; S13 proves another worktree's missing pin is neither repaired nor treated as a failure of `W`'s pass. +- [ ] AC4: Hook-path maintenance (advisory) cannot change the outcome of an already durably committed mutation boundary. + - Validate: T05 S17 and S18 pass: every advisory outcome (`Anchored`, `NoAction`, `Advised`, `Busy`, `StateUnavailable`, corrupt/invalid state, seam error) and `MarkerClearAfterCommit` leave the hook result identical to the no-maintenance baseline, refs untouched, and the external-taint marker unarmed. +- [ ] AC5: The mutation-hook completion path performs no reconciliation, ref listing, DB I/O, or new Git subprocess, and writes maintenance state at most once per advisory window. + - Validate: T05 S14, S15, S16, S19 and S20 pass; `rg` over the advisory module shows no import of `GitSnapshotService`, `MutationTraceStore`, DB openers, `resolve_git_dir`, `resolve_worktree_id`, `spawn_blocking`, or `Command`. +- [ ] AC6: Contention, cancellation, and failed maintenance never violate the pin-to-CAS safety boundary. + - Validate: T05 S6, S7 and S12 pass using deterministic synchronization (no sleep-based proof): concurrent `coordinate()` cannot race pin deletion; dropping the explicit-pass future during a ref-deletion worker does not release the lock until the worker finishes; each operation completes without self-deadlock. +- [ ] AC7: No new unbounded in-memory registry, dynamic dispatch, detached background worker, timeout-cancelled Git work, or nested runtime. + - Validate: `git diff fix-logging...HEAD` inspection plus `nix shell nixpkgs#ripgrep -c rg "dyn Future|BoxFuture|tokio::spawn|Runtime::new|block_on|tokio::time::timeout|JoinHandle|abort\(|static .*(Mutex|RwLock|OnceLock|HashMap)" ` returns no new hits. +- [ ] AC8: Mutation-cursor and Quint model-based tests remain valid without protocol changes. + - Validate: `git diff fix-logging...HEAD --stat` shows no change to `protocol.rs` or `spec/mutation_cursor.qnt`; the existing `mbt` tests pass. +- [ ] AC9: `sce doctor` stays non-mutating; `sce doctor --fix` reports its actual cleanup result and recovers from every failure, stale, and invalid-state condition. + - Validate: T05 S9, S25, S26 and S27 pass: counts for deleted/retained/local-required; distinct output for completed, completed-with-state-persist-failure, failed (unavailable/invalid DB, malformed refs, missing pins), and busy; plain `sce doctor` writes no ref and no state; a recommendation is shown when advised or when state is invalid. +- [ ] AC10: Context matches the implementation, the "retained and unwired" status is gone, and context states that automatic reclamation is deferred to a follow-up with its prerequisites. + - Validate: `nix shell nixpkgs#ripgrep -c rg -i "retained and unwired|unwired" context/cli` returns no stale claims; `rg -i "automatic.*(reclam|reconcil)" context` hits only the deferral/follow-up statements; documented APIs match code. +- [ ] AC11: Exactly one `WorktreeLock` acquisition per maintenance operation, with no self-deadlock. + - Validate: T02 and T05 S12; `rg "acquire_inner_async|acquire_async|acquire_inner" cli/src/services/mutation_trace/runtime` shows each entrypoint acquires once and `reconcile_with_held_lock` never acquires. +- [ ] AC12: The hook completion path contains no unbounded maintenance work, and the documented concurrency contract for explicit reconciliation holds. + - Validate: T05 S14–S16 pass: the advisory takes no Git/DB parameters; while an explicit pass is parked mid-inventory or mid-DB holding the lock, the advisory returns `Busy` immediately and the hook result is unchanged; a concurrent `coordinate()` with a short injected lock timeout fails with the existing lock-timeout error before durable completion and succeeds once the pass is released. No timeout, budget, or size cap is claimed to bound hook latency. +- [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. + - Validate: T05 S19–S25 pass: single advice per window with persistence before emission; concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; far-future and rolled-back timestamps normalize once and recover; no stored timestamp is ever in the future; corrupt/missing state is recovered. +- [ ] AC14: Reconciliation opens only a verified existing DB, with no side effects. + - Validate: T05 S28–S32 pass: missing DB, incompatible schema, missing metadata, repository identity mismatch, and normal existing-DB operation; failing cases leave refs and the DB logical contents untouched and create no files or directories. +- [ ] AC15: Deferred automatic reclamation is explicit and not implied. + - Validate: the plan's D2 follow-up prerequisites are copied into the synchronized context by T06; no code path or doc claims automatic reclamation; T05 S14 proves hook completion never reconciles. + +### Review-finding traceability + +| Finding | Design resolution | Implementing task | Regression scenarios | +| --- | --- | --- | --- | +| R1 P1 double lock | D1 | T02 (T04 consumes) | S6, S7, S12, S17 | +| R1 P1 latency → R2 P1 unbounded inline latency | D2 (advisory only; reconciliation via doctor; automatic deferred) | T01 (investigation), T02, T03, T04 | S14, S15, S16, S17, S18 | +| R1 P2 failure state | D3 (diagnostic streak, no stored deadlines) | T02, T04 | S11, S26, S27 | +| R2 P2 persistence failure / repeated scans | D2/D3 (no automatic full pass; advice only after successful persist; reservation is a follow-up prerequisite) | T02, T03 | S20, S22, S23 | +| R2 P2 far-future timestamps | D3 (past-only fields, normalize once under lock) | T02 | S24, S25 | +| R1 P2 AC3 durability claim | AC3 invariants | T02 (no behavior change), T05, T06 | S2, S3, S4, S5, S13 | +| R1 P2 PR metadata | Stacking section | T01 (re-verify) | none (plan metadata) | +| R1 P2 DB opening side effects | D4 | T01 (audit), T02 | S28–S32 | +| Concurrency / duplicate passes | D1, D3 | T02, T03 | S12, S21 | + +Previous acceptance criteria that changed in round 2: AC1 (explicit caller only; no automatic reconciliation caller), AC2 (doctor deletes; hook never does), AC4 (advisory, not automatic maintenance), AC5 (hook path does no I/O beyond state), AC9 (adds recovery conditions), AC10 (adds deferral statement), AC12 (budget/deferral language removed; concurrency contract added), AC13 (backoff replaced by durable-diagnostic and timestamp-recovery guarantees). New: AC15. + +### Full validation + +- `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml` +- `nix develop -c ./scripts/run-cli-cargo.sh clippy --manifest-path cli/Cargo.toml --all-targets --all-features -- -D warnings` +- `nix flake check` +- `nix build .#ci-checks` +- `nix run .#pkl-check-generated` (only if generated config inputs are touched) +- `git diff --check` +- `git diff fix-logging...HEAD --stat` (confirms no `protocol.rs` / Quint changes and only intended files) +- `nix shell nixpkgs#ripgrep -c rg "dyn Future|BoxFuture|tokio::spawn|Runtime::new|block_on|tokio::time::timeout|JoinHandle|abort\(" ` (AC7) +- Focused runs with non-zero match counts, repeated for race scenarios: `… test --manifest-path cli/Cargo.toml ref_reconciliation`, `… maintenance`, `… mutation_scope`, `… doctor`, `… agent_trace_storage` + +Record actual outcomes in the validation report; compilation alone does not validate the plan. + +### Context sync + +- `context/cli/mutation-trace-ref-reconciliation.md` +- `context/cli/mutation-trace-runtime-coordinator.md` +- `context/cli/mutation-trace-snapshot-service.md` +- `context/cli/mutation-trace-store.md` (if its usage contract changed) +- `context/cli/mutation-scope-runtime.md` +- `context/cli/mutation-scope-hook-ingress.md` +- `context/context-map.md` +- Relevant doctor documentation (`context/sce/agent-trace-hook-doctor.md`, `context/sce/doctor-human-text-contract.md`) +- Agent Trace storage documentation covering the new verified-existing-DB opener (located by T06) + +## Task context synchronization lifecycle + +- **Task context synchronization:** every task carries `pending | synced | blocked`. + A completed task must be `synced` before another task can start or the plan can + finish. +- For `blocked`, record **Blocker**, **Required action**, and **Retry condition** + beside the status. Never infer `synced` from conversation history; write every + lifecycle transition to the plan file. + +## Constraints and non-goals + +- **In scope:** `cli/src/services/mutation_trace/runtime/` (new maintenance module with `reconcile_explicit`, `advise_if_due`, and the state module; `ref_reconciliation.rs` split; `git_snapshot.rs` and `worktree_lock.rs` only for test seams as needed), `cli/src/services/agent_trace_storage/mod.rs` and `cli/src/services/agent_trace_db/repository.rs` (the dedicated verified-existing-DB opener and its read-only metadata check only), `cli/src/services/hooks/mutation_scope.rs` (advisory call only), `cli/src/services/doctor/`, the checkout-local state file `/sce/ref-maintenance.json`, tests, and the context files listed above. +- **Out of scope:** `protocol.rs`, `spec/mutation_cursor.qnt`, mutation-attribution rules, Agent Trace schema migrations, the existing `open_agent_trace_db_for_hook_runtime` behavior for other callers, per-harness (Claude/Codex/OpenCode/Pi) adapter changes, adapter recovery state, any automatic or out-of-band reconciliation execution. +- **Constraints:** + 1. Reuse async reconciliation logic, `GitSnapshotService`, `MutationTraceStore`, `WorktreeLock`; the lock-held implementation is the single shared algorithm. + 2. Static dispatch and existing generic `AsyncFn` patterns only; no `dyn Future`, `BoxFuture`, additional Tokio runtime. + 3. Never delete a ref whose tree is in any repository-wide durable-root set. + 4. Never run any maintenance operation while holding a lock from `coordinate()`; maintenance acquires its own lock after `coordinate()` returned. + 5. Never convert a successful mutation-boundary completion into failure because the advisory was busy, failed, errored, or hit invalid state. + 6. No detached cleanup task, background worker, daemon, fire-and-forget subprocess, or timeout-cancelled Git/DB work; no reconciliation on the hook path. + 7. Do not modify `protocol.rs`, the Quint state machine, or attribution rules. + 8. Never invoke `git gc` or remove Git objects directly; ref deletion goes only through the existing atomic SHA-conditional transaction (`delete_pins`). No independent ref-deletion implementation. + 9. Do not bootstrap, migrate, repair, or initialize an Agent Trace DB to run cleanup (D4). + 10. Exactly one `WorktreeLock` acquisition per maintenance operation; never reacquire inside the lock-held implementation. + 11. Maintenance failures never arm `ExternalTaintMarker`. + 12. The maintenance state file never stores a future deadline; all writes are atomic and under `WorktreeLock`. + 13. No schema migration, new attribution state, or new protocol action. + 14. Follow repository rules: no code comments in touched code, Cargo only via Nix, `nix flake check` for verification. +- **Non-goal (automatic reclamation):** see D2. Follow-up PR with the listed prerequisites, including the durable next-attempt reservation requirement. +- **Non-goal (retired-worktree cleanup):** the reconciler only inventories the namespace of a currently resolvable worktree, so refs owned by removed linked worktrees survive. No global namespace sweep here. Record a follow-up for repository-scoped cleanup with active-worktree inventory, repository-wide durable-root retention, and protection against worktree removal/recreation races. +- **Non-goal (repairing other worktrees' pins):** reconciliation of worktree `W` does not recreate or verify pins that only another worktree requires. +- **Non-goal (evidence-based advice):** the advisory is time-based; detecting actual orphans requires the Git/DB scan kept off the hook path. +- **Non-goal (historical event retention):** trees referenced by retained mutation events are never deleted; bounding history needs a separate event-retention and snapshot-compaction policy. +- **Non-goal (protocol changes):** no new action, state field, attribution category, Quint transition, or schema migration. + +## Assumptions + +- Constants (T03 records final values): `RECONCILIATION_ADVISORY_AFTER` 24 h; `MAINTENANCE_STATE_MAX_BYTES` 4096; `FUTURE_SKEW_TOLERANCE` 5 min; explicit lock wait 10 s (existing). +- The state file is per-worktree under `/sce/`, matching the existing lock and taint-marker layout; absent or unparsable means default state. +- Worktree identity is already Git-topology-derived (`resolve_worktree_id`: `main` and `worktrees/`); the checkout-ID wording in the context docs is stale, and `SkippedNoCheckoutIdentity` appears unreachable in code (declared, never constructed). T01 confirms and T06 removes or justifies it. +- Doctor output additions are additive fields/rows only. + +## Task stack + +- [ ] T01: `Audit reconciliation and mutation-scope lifecycle integration points` (status:todo) + - Task ID: T01 + - Scope: In — read-only audit against PR #304/#305 code of `runtime/ref_reconciliation.rs`, `git_snapshot.rs`, `worktree_lock.rs`, `coordinator.rs`, `protected_worktree.rs`, `hooks/mutation_scope.rs`, `hooks/commit_hooks.rs`, `hooks/` harness lifecycle modules, `doctor/mod.rs`, `doctor/fixes.rs`, `doctor/inspect.rs`, `hooks/runtime.rs`, `agent_trace_storage/mod.rs`, `agent_trace_db/repository.rs`, `db/mod.rs`, and existing reconciliation tests; record findings in this plan under an `Audit findings` subsection. Required findings: + - production call graph for `Start`/`Advance`/`Close`/`Flush`/`Abandon`; which `CoordinateOutcome`/`CoordinateError` variants are durably completed (including `MarkerClearAfterCommit`); when each lock is released, including the `abandon_after_spawn_without_unlock` path; + - **lock ownership (D1):** confirm `reconcile_worktree_inner` acquires `WorktreeLock` at its start and the lock is non-reentrant; specify the exact wrapper / lock-held split and the parameter carrying lock ownership; confirm `delete_pins` moves the lease into the `spawn_blocking` worker so dropping the caller future cannot release the file lock early; confirm the 10 s coordinator lock timeout; + - **execution-point investigation (D2):** re-verify each rejected alternative (post-commit, per-harness session-end/stop, setup lifecycle, detached/daemon) with file references, confirm that no lifecycle-owned noncritical-path executor exists, and confirm runtime shutdown joins blocking workers; confirm the selected design (advisory-only hook trigger plus `sce doctor --fix`) or escalate if a suitable mechanism is found; + - **advisory Git-dir resolution (D2 decision rule):** determine whether the worktree Git dir can be obtained for the advisory without a new Git subprocess (filesystem derivation proven equal to `resolve_git_dir` for the main and a linked worktree, or reuse of a value `coordinate` already resolves without protocol changes) and record the outcome: advisory kept, or hook trigger dropped; + - **state (D3):** identify the existing atomic staging-and-swap pattern to reuse for the state file; confirm placement beside the lock and marker; confirm no in-memory registry is needed; confirm the existing clock source and how to inject a wall clock; + - **DB opening (D4):** inspect `open_agent_trace_db_for_hook_runtime` and its resolver; record exactly which steps can create directories/files, repair migration metadata, or initialize repository metadata; determine whether Turso supports a non-creating/read-only open; define the smallest dedicated verified-existing-DB opener and exact verification queries (schema readiness including the durable-roots migration, metadata row presence, `repository_id` match, valid `source_instance_id`); + - current worktree identity forms; stale checkout-ID docs and unreachable branches including `SkippedNoCheckoutIdentity`; + - confirm PR #304 head and PR #305 head/branch (`ref-recon`, base `fix-logging`) and record the SHAs. + Out — any code or context change. + - Dependencies: none + - Done when: the plan records verified call sites with file/line references, lock ownership and the wrapper/lock-held split, error classification, the D2 execution-point conclusion and advisory Git-dir decision, the DB-opening side effects and chosen opener contract, and flags any finding that invalidates a later task's scope. + - Verify: re-read cited lines to confirm each recorded fact; `gh pr view 304 --json headRefOid` and `gh pr view 305 --json headRefName,baseRefName,headRefOid` match the recorded values. + - Context synchronization: pending + +- [ ] T02: `Add the reusable maintenance entrypoints (explicit reconciliation and hook advisory)` (status:todo) + - Task ID: T02 + - Scope: In — + - **Reconciler split (D1):** extract the body of `reconcile_worktree_inner` after lock acquisition into `reconcile_with_held_lock`, which borrows the held `WorktreeLock` and contains no acquire call; keep `reconcile_worktree` and `reconcile_worktree_inner` behavior and signatures, now acquiring once and delegating. + - **`reconcile_explicit`:** resolves the worktree and Git dir, acquires the lock **once** (existing bounded 10 s wait), opens the DB through the D4 verified opener, runs `reconcile_with_held_lock`, then records the outcome in the state file while still holding the same lock, then drops the lock. Typed outcome `Completed(report) | CompletedStatePersistFailed(report, warning) | Failed(error) | Skipped(Busy)`; lock contention leaves the state untouched and is not a failure; it never calls the lock-acquiring wrapper; it ignores all prior state (stale, failed streak, invalid or corrupt) except to rewrite it. + - **`advise_if_due`:** synchronous function with no DB, Git, store, or snapshot parameters; zero-wait single `try_lock`; reads, normalizes, and writes the state under that one acquisition; returns `Anchored | NoAction | Advised | Busy | StateUnavailable`; persists `last_advised` before returning `Advised`; takes injected wall clock and Git dir. Subject to the T01 decision rule. + - **State module (D3):** `ref-maintenance.json` read (capped, default-on-corrupt), invalid-timestamp normalization under lock, atomic write, past-only fields, success/failure transitions, and a pure `evaluate_recommendation(state, now)` function shared by the advisory and read-only doctor. + - **D4 opener:** the dedicated verified-existing-DB opener and the read-only repository metadata verification method. + - **Test seams only as required:** a worker-entered seam threaded through `delete_pins` to `run_ref_mutation_inner`'s existing `on_worker_entered`; phase-boundary hooks (post-lock, post-DB-open, post-inventory) used to park a pass deterministically; injectable write-failure and rename-interruption hooks for the atomic writer; injected clock. + Out — hook and doctor wiring, any new deletion code, any automatic/reservation/backoff scheduling, protocol/store schema changes, changes to the existing hook DB opener, capped inventory or budget logic. + - Dependencies: T01 + - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a future deadline; invalid timestamps normalize once and persist; lock-busy never touches state; failures leave refs untouched; the DB opener has no side effects; no second deletion path exists. + - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml ref_reconciliation`; focused maintenance-state and `agent_trace_storage` unit tests (deterministic, no sleeps); `rg` confirms the only ref-deletion call is the existing `delete_pins`, each entrypoint acquires `WorktreeLock` once, `reconcile_with_held_lock` contains no acquire call, and the advisory module imports no Git/DB/store symbol. + - Context synchronization: pending + +- [ ] T03: `Trigger the advisory check from the shared mutation-scope ingress` (status:todo) + - Task ID: T03 + - Scope: In — in `hooks/mutation_scope.rs`, after `classify_coordinate` yields a durably completed `Close` or `Flush` (including the `MarkerClearAfterCommit` case) and `coordinate_boundary(...)` has fully returned, call `advise_if_due` through an injected seam; the outcome is only logged (`Advised` at warn with the `sce doctor --fix` hint; `Anchored`/`NoAction`/`Busy` at debug; `StateUnavailable` at warn) and is never returned as a boundary error; no trigger from `Start`, `Advance`, `Abandon`, or failed coordinates; a seam error or unexpected condition is contained and logged; the hook module gains no reconciliation, Git, or DB dependency. Apply the T01 decision rule: if the advisory cannot avoid a new Git subprocess, implement no hook call and document that outcome. Record the final constants in the plan and document the deferred follow-up (D2) in the plan's audit findings. Out — per-harness adapter changes, reconciliation on the hook path, detached tasks, background workers, timeouts. + - Dependencies: T02 + - Done when: a completed `Close`/`Flush` can run the advisory and no other path does; hook output and exit status are identical whether the advisory returns any outcome or errors; the advisory runs only after the coordinator's lock has been released; `MarkerClearAfterCommit` handling is unchanged; no reconciliation, ref listing, or DB I/O is reachable from the hook module's new call. + - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml mutation_scope`; trigger-policy unit tests over every boundary and outcome with a fake advisory seam; a seam that tries to take the real `WorktreeLock` and asserts it is free at invocation; `rg` over the hook module shows no reconciliation/DB-opener/snapshot import added. + - Context synchronization: pending + +- [ ] T04: `Run explicit reconciliation from sce doctor --fix and report recommendation` (status:todo) + - Task ID: T04 + - Scope: In — extend the existing doctor `--fix` flow to call `reconcile_explicit` (one lock acquisition inside the entrypoint; doctor never holds the worktree lock itself); additive report fields/rows for retained, deleted, and locally-required counts; distinct rows for completed, completed-with-state-persist-warning, failed (unavailable/invalid DB with the D4 failure kind, malformed refs, missing required pins, durable-root errors, delete-transaction error), and skipped-busy; plain `sce doctor` stays read-only but reads the state file (never writes it or any ref) and reports whether reconciliation is recommended (last success age, last attempt and failure streak, last advice, invalid-timestamp note) via the pure `evaluate_recommendation`; skipped or failed cleanup is never reported as success; `--fix` runs regardless of recommendation, failure streak, or timestamp validity; no DB bootstrap/migration/repair; no change to adapter recovery state; existing output contracts backward compatible; the documented concurrency note (a mutation boundary may wait up to 10 s while `--fix` reconciles) appears in help or fix output wording only if existing output contracts allow it additively. Out — new doctor subcommands, changes to mutation-scope repair logic. + - Dependencies: T02 + - Done when: `sce doctor --fix` always runs a pass and reports its true typed result; a successful pass resets the failure streak and sets `last_success`; `sce doctor` performs no ref or state mutation; existing doctor tests still pass unchanged. + - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml doctor` + - Context synchronization: pending + +- [ ] T05: `Add production-path and concurrency regression tests` (status:todo) + - Task ID: T05 + - Scope: In — real Git repos, linked worktrees, and repository Agent Trace DBs. Every race, ordering and timing scenario uses deterministic coordination (barriers, channels, existing `on_lock_contention`/`on_worker_entered`-style seams, parked phase hooks, injected clock); **no timing-based sleeps as correctness proofs**. T02 and T03 add unit-level coverage for their own code; T05 owns the production-path matrix below and each scenario must fail when the guarded behavior is deliberately broken. + - *Reconciliation core (explicit path):* (S1) `sce doctor --fix` reclaims an orphan pin left by an interrupted coordinate; (S2) `A → B → C → D` retains every historical tree; (S3) worktree A's pin is retained while worktree B needs the tree (**preservation**); (S4) missing required local pin fails closed with zero deletion; (S5) malformed/symbolic refs cause zero deletion; (S6) concurrent `coordinate()` and reconcile cannot race pin deletion — a coordinate attempt parked on the lock via barrier proceeds only after the pass releases it, and a pin it creates is never deleted by the in-flight pass; (S7) dropping the explicit-pass future while the deletion worker is parked at the worker-entered seam leaves the lock held until the worker is released, then free; (S8) idempotent reconciliation; (S9) plain `sce doctor` mutates neither refs nor state; `--fix` reports the true result for every outcome kind (completed, completed-with-persist-warning, failed kinds, busy); (S10) explicit pass against an external holder returns `Skipped(Busy)`, leaves state byte-identical, and does not count as a failure; (S11) explicit pass with DB failure leaves refs untouched, records a failure streak, never arms the external-taint marker; (S12) the explicit pass and the advisory each complete with one lock acquisition and no self-deadlock, and each returns busy (never deadlocks) against an externally held lock; (S13) with worktree B's required pin deliberately missing, reconciling A deletes only A's eligible pins, does not create or repair B's pin, does not fail, and never deletes any pin protecting B's tree. + - *Hook path and latency (P1):* (S14) the advisory is invoked only after a durably completed `Close`/`Flush` (including `MarkerClearAfterCommit`), never from `Start`/`Advance`/`Abandon`/failed coordinates, and never reaches reconciliation, ref listing, or DB open — its signature takes none, and spy/panic seams on the reconciler, inventory and opener are never called across every boundary and outcome; all refs remain untouched; (S15) slow Git inventory: an explicit pass parked at the post-inventory seam holds the lock, the advisory returns `Busy` through a channel handshake without waiting, and the hook result is unchanged; (S16) slow DB open/root load: the pass is parked at the post-DB-open seam; the advisory returns `Busy` immediately; a concurrent `coordinate()` with a short injected lock timeout fails with the existing lock-timeout error before durable completion and mutates nothing, and the same boundary succeeds once the pass is released (documents the D1 concurrency contract); (S17) the advisory runs only after the coordinator's lock is released — a seam asserts the worktree lock is acquirable at invocation; (S18) every advisory outcome, a state-write failure, a corrupt state file, a seam error, and `MarkerClearAfterCommit` leave the hook result and `Ok` status identical to the no-maintenance baseline, refs untouched, taint marker unarmed. + - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) invalid timestamps: far-future `last_success`, `last_advised`, `anchor`, and clock rollback beyond tolerance each normalize once under the lock, persist, and make the advisory fire immediately once; a repeated call leaves the file byte-identical and moves nothing; after the injected clock advances past the window advice recurs (eventual retry after restart); a rollback within tolerance is treated as age 0; an assertion proves no stored timestamp is ever greater than `now`; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next recorded operation. + - *Explicit recovery:* (S26) `sce doctor --fix` runs and reports its true result from each of: failed streak, stale advice, invalid timestamps, corrupt state, unwritable state (→ `CompletedStatePersistFailed`); (S27) explicit success resets the failure streak and sets `last_success`; explicit failure increments the streak without touching `last_success`; lock-busy never changes state. + - *DB opening (D4):* (S28) missing DB — no file or directory created, refs untouched; (S29) incompatible schema — no migration or repair row written; (S30) missing repository metadata — no row inserted; (S31) repository identity mismatch — refs untouched; (S32) normal existing DB — pass completes. All failing cases assert refs unchanged, DB logical contents unchanged, and the external-taint marker unarmed. + Out — production behavior changes beyond test seams strictly required. + - Dependencies: T03, T04 + - Done when: all scenarios pass reliably, each fails when the guarded behavior is deliberately broken, no test depends on timing or sleeps, and every review finding in the traceability table maps to at least one passing scenario. + - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml` focused on the new tests, repeated runs for the race cases; zero-match filters are not accepted as passes; `rg "sleep"` over new test code returns no hits. + - Context synchronization: pending + +- [ ] T06: `Remove dead-code exemptions and synchronize context` (status:todo) + - Task ID: T06 + - Scope: In — remove declaration-level `#[allow(dead_code)]` from reconciliation declarations (`ref_reconciliation.rs`, `list_pins`/`delete_pins`/`PinnedRef`/`PinInventoryError` in `git_snapshot.rs`, durable-root readers in `store.rs`) that now have production consumers; separately review genuinely unreachable states (e.g. `SkippedNoCheckoutIdentity`, stale checkout-ID wording) and remove or justify each; update the context files in Context sync with the real async APIs, the wrapper/lock-held split and single-acquisition rule, the documented D1 concurrency contract, Git-derived worktree identities, the two maintenance entrypoints, the advisory trigger and why reconciliation is kept off the hook path, the maintenance state file (past-only fields, normalization, atomic writes), the distinct explicit outcomes, the verified-existing-DB opener contract, doctor reporting, and the corrected two-invariant durability statement (no claim that all repository-wide roots are pinned). Record in context, and not as implemented behavior: the deferred automatic out-of-band reclamation follow-up with its binding prerequisites from D2 (lifecycle-owned execution point, durable next-attempt reservation before expensive work, distinct outcomes, lock-hold analysis, timestamp normalization), the retired-worktree cleanup follow-up, and known storage limitations (historical retention, no repair of other worktrees' pins). Out — behavior changes. + - Dependencies: T05 + - Done when: no reconciliation-related `allow(dead_code)` remains without a stated reason; clippy with `-D warnings` is clean; context no longer says "retained and unwired", describes checkout-ID identity, or claims automatic reclamation; `context/context-map.md` entries match. + - Verify: `nix develop -c ./scripts/run-cli-cargo.sh clippy --manifest-path cli/Cargo.toml --all-targets --all-features -- -D warnings`; `nix shell nixpkgs#ripgrep -c rg -n "allow\(dead_code" cli/src/services/mutation_trace`; `rg -i "retained and unwired" context`; `rg -i "automatic.*(reclam|reconcil)" context` hits only deferral statements. + - Context synchronization: pending + +## Open questions + +- Is a 24-hour `RECONCILIATION_ADVISORY_AFTER` the right window for the time-based recommendation? It is only a starting value recorded by T03. +- If T01 finds the advisory cannot get the Git dir without a new Git subprocess, the plan drops the hook trigger and relies on `sce doctor` alone (decision rule in D2, no further confirmation needed). Confirm you are happy with that outcome rather than accepting one extra `git rev-parse` on the `Close`/`Flush` path. +- Do you want a follow-up PR scoped now for automatic out-of-band reclamation (D2 prerequisites), or only the documented deferral? +- `SkippedNoCheckoutIdentity` looks unreachable in the current code (the identity is Git-topology-derived and its failure is an `Err`). The plan assumes T06 removes it; confirm you do not want it kept for compatibility. +- D4 adds a dedicated opener and one read-only verification method in `agent_trace_storage` / `agent_trace_db`. T01 may find that Turso cannot open an existing DB without incidental WAL bookkeeping; the plan treats that as an accepted limitation (logical contents unchanged) unless you want a stricter guarantee. From 64d0650fab5e55d931428a3ab201e16be6866ac2 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 00:29:32 +0200 Subject: [PATCH 02/16] context: Record reconciliation wiring audit findings Document the verified lock ownership, production call graph, advisory-only hook design, durable maintenance-state rules, and database-opening constraints needed to implement mutation-cursor ref reconciliation safely. Cite the completed audit and its findings so the follow-up implementation tasks have explicit scope and evidence. Plan: mutation-cursor-ref-reconciliation-wiring, T01 Co-authored-by: SCE --- ...tation-cursor-ref-reconciliation-wiring.md | 122 ++++++++++++++---- 1 file changed, 100 insertions(+), 22 deletions(-) diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 581806651..092fc9b82 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -28,7 +28,14 @@ Review history (all resolutions are binding on T01–T06): 7. **Unbounded inline latency (P1).** A cooperative budget checked between phases cannot bound `git for-each-ref`, DB open, repository-wide root queries, `git update-ref`, or Tokio blocking-worker shutdown, and reconciliation holds the lock that the coordinator waits on for 10 s. Selected: advisory-only hook trigger, reconciliation through `sce doctor --fix`, automatic execution deferred to a follow-up. The 250 ms budget, size/cost gates, and deferral machinery are removed. See D2. 8. **Repeated scans after state-persistence failure (P2).** The reservation-before-expensive-work requirement applies to any *automatic* full pass. Because this PR has none, no reservation infrastructure is built; the requirement is recorded as a binding prerequisite of the follow-up. The advisory path emits a recommendation only after the state write succeeds, and `Completed` vs `CompletedStatePersistFailed` stay distinct for explicit passes. See D2/D3. -9. **Far-future timestamps (P2).** The state file stores only past-event timestamps and never a future deadline; eligibility is computed as `now − timestamp ≥ window`, so rereading cannot extend a deadline. An impossible timestamp is normalized once under `WorktreeLock` and persisted. See D3. +9. **Far-future timestamps (P2).** The state file stores only past-event timestamps and never a deadline; eligibility is computed as `now − timestamp ≥ window`, so rereading cannot extend a deadline. A timestamp more than `FUTURE_SKEW_TOLERANCE` ahead of `now` is invalid and is normalized once under `WorktreeLock` and persisted; a timestamp within the tolerance is valid and treated as age 0. See D3. + +*Round 3 (four findings; plan-only corrections, D1–D4 and the T01–T06 structure unchanged)* + +10. **External-taint marker assertion (P2).** `CoordinateError::MarkerClearAfterCommit` means the boundary committed but clearing the marker failed, so the marker may legitimately remain armed. The advisory must never create, clear, or modify the marker; it is not required to leave it unarmed. Normal completion and `MarkerClearAfterCommit` are asserted separately (AC4, S18a, S18b). `coordinator.rs`, `ProtectedWorktree` and external-taint recovery semantics are not changed. +11. **Advisory contention vs coordinator contention (P2).** The advisory runs only after `coordinate()` has returned, so an explicit pass holding `WorktreeLock` can make `coordinate()` itself fail (pre-commit lock timeout) before the advisory is ever reached. Contention is now tested as three distinct scenarios: direct advisory contention (S15), coordinator contention (S16), and the post-coordinate handoff race (S33). See AC12. +12. **Git-ref assertions (P2).** `coordinate()` legitimately captures and pins trees. Tests no longer expect refs to be unchanged across `Close`/`Flush`; they assert that the advisory itself performs no ref creation, update, deletion, or inventory, by comparing against a coordinate-only baseline or an injected advisory seam. See AC2, AC5, S14. +13. **Future-skew contradiction (P3).** D3 accepts timestamps up to `FUTURE_SKEW_TOLERANCE` (5 min) ahead of the injected `now` as age 0, so "no stored timestamp is ever in the future" was wrong. Replaced with the exact tolerance invariant and seven explicit clock cases (S24). See AC13. Stacking: @@ -84,15 +91,19 @@ Facts the design relies on. T01 must re-read each cited location and either conf - *Trade-offs.* (+) hook completion path has no unbounded work and no new failure mode; correct under every ordering with `coordinate()`; no new infrastructure without a consumer. (−) orphan pins are reclaimed only when an operator runs `sce doctor --fix`; the recommendation is time-based (staleness since last success), not evidence-based, because detecting orphans needs the Git/DB scan this design keeps off the hook path. - **D3 — Durable maintenance state: diagnostics, no stored deadlines (rounds 1–2).** - One small JSON file per worktree at `/sce/ref-maintenance.json` (versioned, capped at `MAINTENANCE_STATE_MAX_BYTES`, 4 KiB), beside the lock and taint marker. Fields (all optional past-event unix-ms timestamps unless noted): `anchor`, `last_success`, `last_attempt`, `last_advised`; `last_attempt_outcome` (`completed` | `failed`); `last_report` (retained/deleted/local-required counts); `consecutive_failures` (integer); `last_failure` (short classification plus bounded message). - - **No field is ever a future deadline.** Eligibility is computed at read time as `now − reference ≥ window` (saturating at 0). Re-reading therefore cannot move a deadline, and there is nothing to clamp. + - **No field is ever used as a deadline.** Eligibility is computed at read time as `now − reference ≥ window` (saturating at 0). Re-reading therefore cannot move a deadline. The stored-timestamp invariant is scoped to **successful state-writing operations**: after a successful write, every stored timestamp `t` satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`. It is not claimed after read-only or unsuccessful operations (see the two operation classes below). A value in `(now, now + FUTURE_SKEW_TOLERANCE]` is valid, is preserved byte-for-byte (not rewritten), and is evaluated as age 0; it can delay eligibility by at most the tolerance and cannot slide, because it is never refreshed by reads. + - **Two operation classes.** + - *Read-only or unsuccessful operations* — plain `sce doctor`; a lock-busy advisory or explicit pass; a failed state read; a failed state write; a process interrupted before atomic replacement. These never mutate maintenance state (the previous bytes remain intact; a failed or interrupted write leaves the previous state), so they may observe preexisting invalid timestamps and cannot guarantee normalization. They must evaluate invalid timestamps conservatively (treated as eligible/recommended, never as age 0 and never as a reason to defer), must not postpone an advisory or explicit repair indefinitely, and must not report invalid state as successfully normalized. + - *Successful state-writing operations* — a successful `advise_if_due` write and a successful `reconcile_explicit` state record. While holding `WorktreeLock` they normalize invalid timestamps and persist the normalized state atomically, so that every stored timestamp satisfies the tolerance after the successful write; valid near-future timestamps are preserved as specified above; repeated reads never move eligibility forward; subsequent invocations, including after a process restart, observe the persisted normalization. - Writes happen only under `WorktreeLock`, via atomic staging-and-swap (temp file in the same directory, then rename; pattern identified in T01) so a crash leaves either the old or the new file. No in-memory registry or process-global state. - Transitions: advisory — anchor and `last_advised` only. Explicit success — `last_attempt` = `last_success` = now, `last_report` set, `consecutive_failures` = 0, failure fields cleared. Explicit failure (DB unavailable/invalid, malformed ref, missing required pins, durable-root read error, delete-transaction error) — `last_attempt` = now, outcome `failed`, `consecutive_failures` += 1, `last_failure` set, `last_success` untouched. Lock contention (including explicit lock timeout) — **no state change**; never classified as a failure. - - **Invalid timestamp normalization (deterministic).** A timestamp is invalid when it is more than `FUTURE_SKEW_TOLERANCE` (5 min) after the injected `now` (far-future values, or clock rollback beyond the tolerance). A timestamp within the tolerance is treated as age 0. Under `WorktreeLock`, `advise_if_due` and `reconcile_explicit` normalize once: invalid `last_success`/`last_attempt`/`last_advised`/`anchor` values are cleared; if `last_success` and `anchor` are both unusable the state is **immediately eligible** for advice, so the advisory persists `last_advised = now` and returns `Advised` in the same locked operation, and a fresh `anchor = now` is written. The normalized state is persisted before returning, so a second read sees valid timestamps and does not shift anything. Plain `sce doctor` never writes; it evaluates invalid timestamps as "recommended" with an "invalid maintenance timestamps" note. `sce doctor --fix` runs regardless of timestamp validity and rewrites valid state afterwards. + - **Invalid timestamp normalization (deterministic).** A timestamp is invalid when it is strictly more than `FUTURE_SKEW_TOLERANCE` (5 min) after the injected `now` (far-future values, or clock rollback beyond the tolerance); a timestamp exactly equal to `now + FUTURE_SKEW_TOLERANCE` is valid. A valid timestamp ahead of `now` (within the tolerance) is treated as age 0, so a small clock rollback never makes an advisory eligible by itself. Under `WorktreeLock`, `advise_if_due` and `reconcile_explicit` normalize once, as part of their successful state write: invalid `last_success`/`last_attempt`/`last_advised`/`anchor` values are cleared; if `last_success` and `anchor` are both unusable the state is **immediately eligible** for advice, so the advisory persists `last_advised = now` and returns `Advised` in the same locked operation, and a fresh `anchor = now` is written. The normalized state is persisted before returning, so a second read sees valid timestamps, performs no rewrite, and does not shift or postpone eligibility; once the injected clock advances, ordinary window-based eligibility resumes. If the normalizing write fails or the lock is busy, nothing is persisted, the invalid bytes stay as they were, and the in-memory evaluation still treats them conservatively (eligible), so a later successful operation can normalize them. Plain `sce doctor` never writes; it evaluates invalid timestamps as "recommended" with an "invalid maintenance timestamps" note and leaves the file byte-identical. `sce doctor --fix` runs regardless of timestamp validity and rewrites valid state afterwards. - Recovery: a missing, unparsable, wrong-version, or oversized file is treated as default and rewritten by the next recorded operation; it never fails an explicit pass. - Result distinction for explicit passes (typed outcome): `Completed(report)`; `CompletedStatePersistFailed(report, warning)` — the cleanup stands and is reported as done, with a visible warning that diagnostics were not recorded; `Failed(error)`; `Skipped(Busy)`. Never merged. Advisory outcomes: `Anchored | NoAction | Advised | Busy | StateUnavailable`. - Time comes from an injected wall clock (`Fn() -> i64` unix ms) passed as a generic parameter for deterministic tests. - **D4 — Verified-existing Agent Trace DB opener (round 1).** - - Contract for reconciliation: (1) never create a DB file or any parent directory; (2) never run migrations; (3) never call `repair_missing_repository_schema_migration_metadata`; (4) never call `verify_or_initialize_repository_metadata` or otherwise insert/update repository metadata; (5) verify repository identity (stored `repository_id` equals the identity resolved for this checkout, `source_instance_id` valid) and schema readiness (baseline plus the migration providing durable mutation roots) **before** any ref listing or deletion; (6) any failure returns an error with refs untouched, as a maintenance error that never arms `ExternalTaintMarker` and never becomes `CoordinateError::AgentTraceDbUnavailable`. + - Side-effect guarantees are stated per database state. **Missing DB:** fail before invoking Turso; create no DB file, no parent directory, and no WAL or shared-memory file; list or delete no Git refs. **Existing but invalid DB:** opening the existing file is permitted and incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable (documented); logical immutability is required — no schema migration, no migration-metadata repair, no repository-metadata initialization, no application-table INSERT/UPDATE/DELETE, no Git-ref deletion — and verification fails with the matching typed error. **Existing valid DB:** verify repository identity and schema readiness before reconciliation, preserve the existing multiprocess-WAL configuration, allow only the database access reconciliation needs, and run no initialization, repair, or migration. + - Contract for reconciliation: (1) never create a DB file or any parent directory (missing DB fails before Turso is invoked); (2) never run migrations; (3) never call `repair_missing_repository_schema_migration_metadata`; (4) never call `verify_or_initialize_repository_metadata` or otherwise insert/update repository metadata; (5) verify repository identity (stored `repository_id` equals the identity resolved for this checkout, `source_instance_id` valid) and schema readiness (baseline plus the migration providing durable mutation roots) **before** any ref listing or deletion; (6) any failure returns an error with refs untouched, as a maintenance error that never arms `ExternalTaintMarker` and never becomes `CoordinateError::AgentTraceDbUnavailable`. - The existing hook opener cannot satisfy (1), (3), or (4), and changing it would alter every hook caller. The smallest satisfying change is a dedicated opener in `agent_trace_storage/mod.rs` (for example `resolve_existing_agent_trace_storage_for_maintenance`) plus one read-only verification method on `RepositoryAgentTraceDb` (for example `verify_existing_repository_metadata(repository_id)`) wrapping the existing `select_repository_metadata_row` without insert. The opener first checks that the resolved DB path exists as a file; if not, it fails before touching Turso. If Turso offers no read-only open, T01 records that and the opener uses only `SELECT`-class statements after open; incidental WAL bookkeeping on an existing DB is an accepted limitation, and tests assert logical tables are unchanged. - Typed failure kinds (small enum, converted to `anyhow` at the existing `open_db` seam): `Missing`, `Unreadable`, `IncompatibleSchema`, `MissingMetadata`, `RepositoryMismatch`. The existing hook opener and all its callers are unchanged. - Considered and rejected: opening the DB before taking the lock to shorten lock-hold. It would require a second lock acquisition just to record an open failure, violating the one-acquisition-per-operation rule in D1; the lock-first order of the existing reconciler is kept and the D1 concurrency contract documents the consequence. @@ -104,16 +115,16 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - [ ] AC1: Reconciliation has one real non-test **explicit** caller (`sce doctor --fix`) and no automatic reconciliation caller; the mutation-scope ingress has at most an advisory caller that cannot reach reconciliation. - Validate: `nix shell nixpkgs#ripgrep -c rg "reconcile|advise_if_due" cli/src --glob '!*test*'` shows the reconciliation entrypoint referenced only from `doctor/` (and its definition module), and `advise_if_due` (if present per the D2 decision rule) referenced from `hooks/mutation_scope.rs`; the hook module imports no reconciliation, `GitSnapshotService`, store, or DB-opener symbol for this purpose. - [ ] AC2: `sce doctor --fix` deletes an orphan snapshot ref; `Close`/`Flush` never deletes. - - Validate: T05 S1 drives the doctor fix path against a real Git repo and asserts the orphan `refs/sce/mutation-cursor//` is gone; S14 asserts `Close`/`Flush` leaves all refs untouched. This plan does **not** claim automatic orphan reclamation. + - Validate: T05 S1 drives the doctor fix path against a real Git repo and asserts the orphan `refs/sce/mutation-cursor//` is gone; S14 asserts the advisory path creates, updates, deletes, and inventories no refs (ref changes made by `coordinate()` itself, such as snapshot pins, are legitimate and are not attributed to maintenance). This plan does **not** claim automatic orphan reclamation. - [ ] AC3: A successful pass establishes two distinct invariants, and claims nothing stronger. - **Local consistency:** `durable_roots(W) ⊆ pinned_trees(W)` is checked before any deletion for the target worktree `W`. If it fails, the pass deletes nothing and returns `MissingRequiredPins`. - **Repository-wide deletion safety:** `delete(W, T) ⇒ T ∉ durable_roots(repository)`. A pin is removed only when its tree is outside every worktree's durable roots, so reconciliation never removes an existing SCE pin that protects another worktree's required tree. - Reconciliation does **not** repair, recreate, or verify missing pins belonging to other worktrees, and does not guarantee that every repository-wide durable root is pinned. - Validate: T05 S2, S3, S4, S5 and S13 pass. S2/S3 prove preservation; S4 proves local fail-closed; S13 proves another worktree's missing pin is neither repaired nor treated as a failure of `W`'s pass. - [ ] AC4: Hook-path maintenance (advisory) cannot change the outcome of an already durably committed mutation boundary. - - Validate: T05 S17 and S18 pass: every advisory outcome (`Anchored`, `NoAction`, `Advised`, `Busy`, `StateUnavailable`, corrupt/invalid state, seam error) and `MarkerClearAfterCommit` leave the hook result identical to the no-maintenance baseline, refs untouched, and the external-taint marker unarmed. + - Validate: T05 S17, S18a, S18b and S33 pass: every advisory outcome (`Anchored`, `NoAction`, `Advised`, `Busy`, `StateUnavailable`, corrupt/invalid state, seam error) leaves the hook result identical to the coordinate-only baseline. The advisory never creates, clears, or modifies the external-taint marker. After normal completion the marker state is whatever the existing coordinator leaves (cleared) and is unchanged by the advisory (S18a). `MarkerClearAfterCommit` remains a durably completed boundary with a successful hook result; the marker state resulting from the failed clear is preserved by the advisory, and the next real runtime invocation still takes the existing conservative inherited-taint recovery path (S18b). `MarkerClearAfterCommit` is never treated as equivalent to a successful clear. - [ ] AC5: The mutation-hook completion path performs no reconciliation, ref listing, DB I/O, or new Git subprocess, and writes maintenance state at most once per advisory window. - - Validate: T05 S14, S15, S16, S19 and S20 pass; `rg` over the advisory module shows no import of `GitSnapshotService`, `MutationTraceStore`, DB openers, `resolve_git_dir`, `resolve_worktree_id`, `spawn_blocking`, or `Command`. + - Validate: T05 S14, S15, S19 and S20 pass (S14: the advisory performs no Git-ref creation, update, deletion, or inventory, no reconciliation, and no DB I/O; coordinator ref effects are legitimate and distinguished from it); `rg` over the advisory module shows no import of `GitSnapshotService`, `MutationTraceStore`, DB openers, `resolve_git_dir`, `resolve_worktree_id`, `spawn_blocking`, or `Command`. - [ ] AC6: Contention, cancellation, and failed maintenance never violate the pin-to-CAS safety boundary. - Validate: T05 S6, S7 and S12 pass using deterministic synchronization (no sleep-based proof): concurrent `coordinate()` cannot race pin deletion; dropping the explicit-pass future during a ref-deletion worker does not release the lock until the worker finishes; each operation completes without self-deadlock. - [ ] AC7: No new unbounded in-memory registry, dynamic dispatch, detached background worker, timeout-cancelled Git work, or nested runtime. @@ -127,11 +138,15 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - [ ] AC11: Exactly one `WorktreeLock` acquisition per maintenance operation, with no self-deadlock. - Validate: T02 and T05 S12; `rg "acquire_inner_async|acquire_async|acquire_inner" cli/src/services/mutation_trace/runtime` shows each entrypoint acquires once and `reconcile_with_held_lock` never acquires. - [ ] AC12: The hook completion path contains no unbounded maintenance work, and the documented concurrency contract for explicit reconciliation holds. - - Validate: T05 S14–S16 pass: the advisory takes no Git/DB parameters; while an explicit pass is parked mid-inventory or mid-DB holding the lock, the advisory returns `Busy` immediately and the hook result is unchanged; a concurrent `coordinate()` with a short injected lock timeout fails with the existing lock-timeout error before durable completion and succeeds once the pass is released. No timeout, budget, or size cap is claimed to bound hook latency. + - Validate: T05 S14, S15, S16 and S33 pass, as three distinct contention scenarios. S15 (direct advisory contention): with an explicit pass parked mid-inventory or mid-DB holding the lock, `advise_if_due` called directly returns `Busy` immediately and changes no state or refs. S16 (coordinator contention): with the lock held, `coordinate()` keeps its existing bounded lock acquisition; a short injected timeout fails with the existing lock-timeout error before durable completion, the advisory does not run after the failed coordinate, and the same boundary succeeds once the pass is released. S33 (post-coordinate handoff race): after a successful `coordinate()` released its lock, a competing explicit pass takes the lock before the paused advisory resumes; the advisory returns `Busy` and the committed boundary still returns success. The advisory takes no Git/DB parameters. No timeout, budget, or size cap is claimed to bound hook latency. - [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. - - Validate: T05 S19–S25 pass: single advice per window with persistence before emission; concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; far-future and rolled-back timestamps normalize once and recover; no stored timestamp is ever in the future; corrupt/missing state is recovered. -- [ ] AC14: Reconciliation opens only a verified existing DB, with no side effects. - - Validate: T05 S28–S32 pass: missing DB, incompatible schema, missing metadata, repository identity mismatch, and normal existing-DB operation; failing cases leave refs and the DB logical contents untouched and create no files or directories. + - Two guarantee classes (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, failed state read or write, write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. + - Validate: T05 S19–S25 and S34–S39 pass: single advice per window with persistence before emission; concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's failure leg assert byte-identical preexisting state instead; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write leaving the previous state unchanged; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. +- [ ] AC14: Reconciliation opens only a verified existing DB and never initializes, repairs, or migrates it; failure cases assert logical database immutability, and only the missing-DB case asserts no newly created database-related files. + - **Missing DB:** fails before Turso is invoked; no database file, parent directory, or WAL/shared-memory file is created; no Git ref is listed or deleted. + - **Existing but invalid DB:** opening is permitted; incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable and documented; no schema migration, migration-metadata repair, repository-metadata initialization, or application-table INSERT/UPDATE/DELETE; no Git-ref deletion; verification fails with the matching typed error. Tests do not assert filesystem immutability for this case. + - **Existing valid DB:** repository identity and schema readiness are verified before reconciliation; the existing multiprocess-WAL configuration is preserved; only the access reconciliation requires is performed; no initialization, repair, or migration. + - Validate: T05 S28–S32 pass: missing DB (no new files or directories, Turso not invoked, refs not listed), incompatible schema, missing metadata, repository identity mismatch (refs untouched, logical DB contents byte-for-byte equivalent by table/row comparison, typed error), and normal existing-DB operation. - [ ] AC15: Deferred automatic reclamation is explicit and not implied. - Validate: the plan's D2 follow-up prerequisites are copied into the synchronized context by T06; no code path or doc claims automatic reclamation; T05 S14 proves hook completion never reconciles. @@ -140,16 +155,25 @@ How this plan is proven complete. `/validate` runs these checks; no task perform | Finding | Design resolution | Implementing task | Regression scenarios | | --- | --- | --- | --- | | R1 P1 double lock | D1 | T02 (T04 consumes) | S6, S7, S12, S17 | -| R1 P1 latency → R2 P1 unbounded inline latency | D2 (advisory only; reconciliation via doctor; automatic deferred) | T01 (investigation), T02, T03, T04 | S14, S15, S16, S17, S18 | +| R1 P1 latency → R2 P1 unbounded inline latency | D2 (advisory only; reconciliation via doctor; automatic deferred) | T01 (investigation), T02, T03, T04 | S14, S15, S16, S17, S18a, S18b, S33 | | R1 P2 failure state | D3 (diagnostic streak, no stored deadlines) | T02, T04 | S11, S26, S27 | | R2 P2 persistence failure / repeated scans | D2/D3 (no automatic full pass; advice only after successful persist; reservation is a follow-up prerequisite) | T02, T03 | S20, S22, S23 | -| R2 P2 far-future timestamps | D3 (past-only fields, normalize once under lock) | T02 | S24, S25 | +| R2 P2 far-future timestamps → R3 P3 future-skew contradiction → R4 P2 normalization scope | D3 (past-only fields, `FUTURE_SKEW_TOLERANCE` valid as age 0, normalize beyond it once under lock on successful writes; read-only/unsuccessful operations never mutate and evaluate conservatively) | T02 | S24 (cases 1–7), S25, S34–S39 | +| R4 P2 database side-effect assertions | D4/AC14 (per-DB-state guarantees: missing = no files; existing invalid = logical immutability, incidental WAL allowed) | T02 | S28–S32 | +| R3 P2 external-taint marker assertion | AC4 (advisory never touches the marker; `MarkerClearAfterCommit` preserved) | T03, T05 | S18a, S18b | +| R3 P2 advisory vs coordinator contention | AC12 (three scenarios: direct, coordinator, post-coordinate handoff) | T05 | S15, S16, S33 | +| R3 P2 Git-ref assertions | AC2, AC5 (advisory adds no ref delta; coordinator pins legitimate) | T05 | S14 | | R1 P2 AC3 durability claim | AC3 invariants | T02 (no behavior change), T05, T06 | S2, S3, S4, S5, S13 | | R1 P2 PR metadata | Stacking section | T01 (re-verify) | none (plan metadata) | | R1 P2 DB opening side effects | D4 | T01 (audit), T02 | S28–S32 | | Concurrency / duplicate passes | D1, D3 | T02, T03 | S12, S21 | -Previous acceptance criteria that changed in round 2: AC1 (explicit caller only; no automatic reconciliation caller), AC2 (doctor deletes; hook never does), AC4 (advisory, not automatic maintenance), AC5 (hook path does no I/O beyond state), AC9 (adds recovery conditions), AC10 (adds deferral statement), AC12 (budget/deferral language removed; concurrency contract added), AC13 (backoff replaced by durable-diagnostic and timestamp-recovery guarantees). New: AC15. +Previous acceptance criteria that changed in round 2: AC1 (explicit caller only; no automatic reconciliation caller), AC2 (doctor deletes; hook never does), AC4 (advisory, not automatic maintenance), AC5 (hook path does no I/O beyond state), AC9 (adds recovery conditions), AC10 (adds deferral statement), AC12 (budget/deferral language removed; concurrency contract added), AC13 (backoff replaced by durable-diagnostic and timestamp-recovery guarantees). New: AC15. Round 3 changed AC2 (ref assertion scoped to the advisory), AC4 (marker semantics, `MarkerClearAfterCommit` preserved), AC5 (S14/S15 references), AC12 (three contention scenarios, new S33) and AC13 (exact skew-tolerance invariant, S24 cases 1–7); scenarios S15, S16, S18 (now S18a/S18b), S24 were rewritten and S33 was added. + +*Round 4 (two findings; plan-only corrections, D1–D4 architecture and the T01–T06 structure unchanged)* + +14. **Timestamp-normalization scope (P2).** The `t ≤ now + FUTURE_SKEW_TOLERANCE` post-condition was asserted after every operation, which read-only (plain `sce doctor`), lock-busy, failed-read/write, and interrupted operations cannot satisfy. D3, AC13, T02 and T05 S24–S25 now separate read-only/unsuccessful operations (never mutate, evaluate conservatively, never postpone repair, never report normalization) from successful state-writing operations (normalize under `WorktreeLock`, persist atomically, post-condition holds). Added S34–S39. +15. **Database side-effect assertions (P2).** AC14 and S28–S32 asserted that failed verification never creates files, contradicting D4's accepted incidental WAL bookkeeping for existing databases. D4, AC14 and S28–S32 now specify per-state guarantees: missing DB (stronger: no Turso call, no new files/directories, no ref listing or deletion); existing invalid DB (logical immutability, no migration/repair/initialization/application writes, incidental WAL bookkeeping permitted); existing valid DB (verify before reconciliation, preserve WAL configuration, no initialization/repair/migration). ### Full validation @@ -202,7 +226,7 @@ Record actual outcomes in the validation report; compilation alone does not vali 9. Do not bootstrap, migrate, repair, or initialize an Agent Trace DB to run cleanup (D4). 10. Exactly one `WorktreeLock` acquisition per maintenance operation; never reacquire inside the lock-held implementation. 11. Maintenance failures never arm `ExternalTaintMarker`. - 12. The maintenance state file never stores a future deadline; all writes are atomic and under `WorktreeLock`. + 12. The maintenance state file never stores a deadline; every successful state write leaves no timestamp more than `FUTURE_SKEW_TOLERANCE` ahead of the injected `now`; read-only and unsuccessful operations never mutate it; all writes are atomic and under `WorktreeLock`. 13. No schema migration, new attribution state, or new protocol action. 14. Follow repository rules: no code comments in touched code, Cargo only via Nix, `nix flake check` for verification. - **Non-goal (automatic reclamation):** see D2. Follow-up PR with the listed prerequisites, including the durable next-attempt reservation requirement. @@ -221,7 +245,7 @@ Record actual outcomes in the validation report; compilation alone does not vali ## Task stack -- [ ] T01: `Audit reconciliation and mutation-scope lifecycle integration points` (status:todo) +- [x] T01: `Audit reconciliation and mutation-scope lifecycle integration points` (status:done) - Task ID: T01 - Scope: In — read-only audit against PR #304/#305 code of `runtime/ref_reconciliation.rs`, `git_snapshot.rs`, `worktree_lock.rs`, `coordinator.rs`, `protected_worktree.rs`, `hooks/mutation_scope.rs`, `hooks/commit_hooks.rs`, `hooks/` harness lifecycle modules, `doctor/mod.rs`, `doctor/fixes.rs`, `doctor/inspect.rs`, `hooks/runtime.rs`, `agent_trace_storage/mod.rs`, `agent_trace_db/repository.rs`, `db/mod.rs`, and existing reconciliation tests; record findings in this plan under an `Audit findings` subsection. Required findings: - production call graph for `Start`/`Advance`/`Close`/`Flush`/`Abandon`; which `CoordinateOutcome`/`CoordinateError` variants are durably completed (including `MarkerClearAfterCommit`); when each lock is released, including the `abandon_after_spawn_without_unlock` path; @@ -236,7 +260,12 @@ Record actual outcomes in the validation report; compilation alone does not vali - Dependencies: none - Done when: the plan records verified call sites with file/line references, lock ownership and the wrapper/lock-held split, error classification, the D2 execution-point conclusion and advisory Git-dir decision, the DB-opening side effects and chosen opener contract, and flags any finding that invalidates a later task's scope. - Verify: re-read cited lines to confirm each recorded fact; `gh pr view 304 --json headRefOid` and `gh pr view 305 --json headRefName,baseRefName,headRefOid` match the recorded values. - - Context synchronization: pending + - Completed: 2026-10-09 + - Files changed: `context/plans/mutation-cursor-ref-reconciliation-wiring.md` + - Result: Added the `Audit findings` section. Confirmed lock ownership and the wrapper/lock-held split, the call graph and completion classes, the D2 conclusion (advisory plus `sce doctor --fix`), a filesystem Git-dir derivation that avoids a new Git subprocess (hook trigger kept), the D3 staging-and-swap pattern and clock source, and the D4 non-read-only opener findings and contract. Flags recorded: the `sync/auto_sync.rs` detached-spawn precedent, the `ReconcileError` "003" vs `004` migration wording, and the `doctor/inspect.rs:305` repair-seam caller. No finding invalidates a later task's scope. + - Verify: cited lines re-read during the audit and line references corrected; `gh pr view 304` → head `0be74aad5bea5070828846d3277041a4649c9842`, `gh pr view 305` → `ref-recon`/base `fix-logging`/head `fb7e78fbfa41f9f054803eef2dddb180f5d5289a` (both match the recorded values); passed. + - Context impact: none (plan-only audit; no behavior or durable context change). + - Context synchronization: synced - [ ] T02: `Add the reusable maintenance entrypoints (explicit reconciliation and hook advisory)` (status:todo) - Task ID: T02 @@ -249,7 +278,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - **Test seams only as required:** a worker-entered seam threaded through `delete_pins` to `run_ref_mutation_inner`'s existing `on_worker_entered`; phase-boundary hooks (post-lock, post-DB-open, post-inventory) used to park a pass deterministically; injectable write-failure and rename-interruption hooks for the atomic writer; injected clock. Out — hook and doctor wiring, any new deletion code, any automatic/reservation/backoff scheduling, protocol/store schema changes, changes to the existing hook DB opener, capped inventory or budget logic. - Dependencies: T01 - - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a future deadline; invalid timestamps normalize once and persist; lock-busy never touches state; failures leave refs untouched; the DB opener has no side effects; no second deletion path exists. + - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a deadline; after every successful state write no stored timestamp exceeds `now + FUTURE_SKEW_TOLERANCE`; timestamps within the tolerance are accepted as age 0 and preserved; on a successful write invalid timestamps normalize once and persist; read-only, lock-busy, and failed-read/write operations never mutate state, evaluate invalid timestamps conservatively, and never postpone eligibility indefinitely; failures leave refs untouched; the DB opener fails a missing DB before Turso is invoked without creating files or directories, and for an existing invalid DB performs no migration, repair, initialization, or application-table write (incidental WAL bookkeeping permitted and documented); no second deletion path exists. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml ref_reconciliation`; focused maintenance-state and `agent_trace_storage` unit tests (deterministic, no sleeps); `rg` confirms the only ref-deletion call is the existing `delete_pins`, each entrypoint acquires `WorktreeLock` once, `reconcile_with_held_lock` contains no acquire call, and the advisory module imports no Git/DB/store symbol. - Context synchronization: pending @@ -257,7 +286,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - Task ID: T03 - Scope: In — in `hooks/mutation_scope.rs`, after `classify_coordinate` yields a durably completed `Close` or `Flush` (including the `MarkerClearAfterCommit` case) and `coordinate_boundary(...)` has fully returned, call `advise_if_due` through an injected seam; the outcome is only logged (`Advised` at warn with the `sce doctor --fix` hint; `Anchored`/`NoAction`/`Busy` at debug; `StateUnavailable` at warn) and is never returned as a boundary error; no trigger from `Start`, `Advance`, `Abandon`, or failed coordinates; a seam error or unexpected condition is contained and logged; the hook module gains no reconciliation, Git, or DB dependency. Apply the T01 decision rule: if the advisory cannot avoid a new Git subprocess, implement no hook call and document that outcome. Record the final constants in the plan and document the deferred follow-up (D2) in the plan's audit findings. Out — per-harness adapter changes, reconciliation on the hook path, detached tasks, background workers, timeouts. - Dependencies: T02 - - Done when: a completed `Close`/`Flush` can run the advisory and no other path does; hook output and exit status are identical whether the advisory returns any outcome or errors; the advisory runs only after the coordinator's lock has been released; `MarkerClearAfterCommit` handling is unchanged; no reconciliation, ref listing, or DB I/O is reachable from the hook module's new call. + - Done when: a completed `Close`/`Flush` can run the advisory and no other path does; hook output and exit status are identical whether the advisory returns any outcome or errors; the advisory runs only after the coordinator's lock has been released; `MarkerClearAfterCommit` handling is unchanged and the advisory never creates, clears, or modifies the external-taint marker; no reconciliation, ref inventory, or DB I/O is reachable from the hook module's new call. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml mutation_scope`; trigger-policy unit tests over every boundary and outcome with a fake advisory seam; a seam that tries to take the real `WorktreeLock` and asserts it is free at invocation; `rg` over the hook module shows no reconciliation/DB-opener/snapshot import added. - Context synchronization: pending @@ -273,10 +302,11 @@ Record actual outcomes in the validation report; compilation alone does not vali - Task ID: T05 - Scope: In — real Git repos, linked worktrees, and repository Agent Trace DBs. Every race, ordering and timing scenario uses deterministic coordination (barriers, channels, existing `on_lock_contention`/`on_worker_entered`-style seams, parked phase hooks, injected clock); **no timing-based sleeps as correctness proofs**. T02 and T03 add unit-level coverage for their own code; T05 owns the production-path matrix below and each scenario must fail when the guarded behavior is deliberately broken. - *Reconciliation core (explicit path):* (S1) `sce doctor --fix` reclaims an orphan pin left by an interrupted coordinate; (S2) `A → B → C → D` retains every historical tree; (S3) worktree A's pin is retained while worktree B needs the tree (**preservation**); (S4) missing required local pin fails closed with zero deletion; (S5) malformed/symbolic refs cause zero deletion; (S6) concurrent `coordinate()` and reconcile cannot race pin deletion — a coordinate attempt parked on the lock via barrier proceeds only after the pass releases it, and a pin it creates is never deleted by the in-flight pass; (S7) dropping the explicit-pass future while the deletion worker is parked at the worker-entered seam leaves the lock held until the worker is released, then free; (S8) idempotent reconciliation; (S9) plain `sce doctor` mutates neither refs nor state; `--fix` reports the true result for every outcome kind (completed, completed-with-persist-warning, failed kinds, busy); (S10) explicit pass against an external holder returns `Skipped(Busy)`, leaves state byte-identical, and does not count as a failure; (S11) explicit pass with DB failure leaves refs untouched, records a failure streak, never arms the external-taint marker; (S12) the explicit pass and the advisory each complete with one lock acquisition and no self-deadlock, and each returns busy (never deadlocks) against an externally held lock; (S13) with worktree B's required pin deliberately missing, reconciling A deletes only A's eligible pins, does not create or repair B's pin, does not fail, and never deletes any pin protecting B's tree. - - *Hook path and latency (P1):* (S14) the advisory is invoked only after a durably completed `Close`/`Flush` (including `MarkerClearAfterCommit`), never from `Start`/`Advance`/`Abandon`/failed coordinates, and never reaches reconciliation, ref listing, or DB open — its signature takes none, and spy/panic seams on the reconciler, inventory and opener are never called across every boundary and outcome; all refs remain untouched; (S15) slow Git inventory: an explicit pass parked at the post-inventory seam holds the lock, the advisory returns `Busy` through a channel handshake without waiting, and the hook result is unchanged; (S16) slow DB open/root load: the pass is parked at the post-DB-open seam; the advisory returns `Busy` immediately; a concurrent `coordinate()` with a short injected lock timeout fails with the existing lock-timeout error before durable completion and mutates nothing, and the same boundary succeeds once the pass is released (documents the D1 concurrency contract); (S17) the advisory runs only after the coordinator's lock is released — a seam asserts the worktree lock is acquirable at invocation; (S18) every advisory outcome, a state-write failure, a corrupt state file, a seam error, and `MarkerClearAfterCommit` leave the hook result and `Ok` status identical to the no-maintenance baseline, refs untouched, taint marker unarmed. - - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) invalid timestamps: far-future `last_success`, `last_advised`, `anchor`, and clock rollback beyond tolerance each normalize once under the lock, persist, and make the advisory fire immediately once; a repeated call leaves the file byte-identical and moves nothing; after the injected clock advances past the window advice recurs (eventual retry after restart); a rollback within tolerance is treated as age 0; an assertion proves no stored timestamp is ever greater than `now`; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next recorded operation. + - *Hook path and latency (P1):* (S14) the advisory is invoked only after a durably completed `Close`/`Flush` (including `MarkerClearAfterCommit`), never from `Start`/`Advance`/`Abandon`/failed coordinates, and never reaches reconciliation, ref inventory, or DB open — its signature takes none, and spy/panic seams on the reconciler, inventory and opener are never called across every boundary and outcome; `coordinate()` keeps all of its existing snapshot and pinning semantics (required pins are still created and intact — protecting pinning and durable evidence is not weakened), and the advisory itself performs no Git-ref creation, update, or deletion: the test compares the ref set produced by a full hook run against an equivalent coordinate-only baseline (or injects a fake advisory seam and asserts the real advisory adds no ref delta), so any ref change from the coordinator is legitimate and is never attributed to maintenance; (S15) direct advisory contention (scenario A): an explicit pass parked at the post-inventory seam (and, in a second parameterization, at the post-DB-open seam) holds `WorktreeLock`; `advise_if_due` is invoked directly, returns `Busy` through a channel handshake without waiting for the pass, and leaves the state file byte-identical and the ref set unchanged; (S16) coordinator contention (scenario B): with the explicit pass parked holding the lock, a `coordinate()` Close/Flush with a short injected lock timeout fails with the existing lock-timeout error before durable completion and mutates nothing, the existing bounded-acquisition semantics are unchanged, the failed coordinate is a legitimate pre-commit failure, the advisory seam is never invoked after it, and once the pass is released the same boundary succeeds (documents the D1 concurrency contract); (S17) the advisory runs only after the coordinator's lock is released — a seam asserts the worktree lock is acquirable at invocation; (S18a) normal completion: every advisory outcome, a state-write failure, a corrupt state file, and a seam error leave the hook result and `Ok` status identical to the coordinate-only baseline; the marker state is snapshotted immediately before the advisory runs and asserted byte-identical (existence) afterwards, so the advisory never creates, clears, or modifies it, and the normal coordinator-cleared state is unchanged; (S18b) `MarkerClearAfterCommit`: reusing the coordinator's existing marker-clear failure fixture (no change to `coordinator.rs`, `ProtectedWorktree`, or recovery semantics; if unreachable from the hook layer, a fake coordinate seam returns that variant while a real armed marker file exists), the boundary is classified as durably completed, the hook returns the same successful empty-stdout result with the advisory allowed to run, the marker the failed clear left behind is asserted still armed and unmodified after the advisory, and a subsequent real `coordinate()` invocation observes the inherited taint and takes the existing conservative recovery path; (S33) post-coordinate handoff race (scenario C): `coordinate()` completes successfully and releases its lock; a seam pauses the hook immediately before the advisory call; a competing explicit reconciliation acquires `WorktreeLock` through a barrier; the advisory resumes and returns `Busy`; the already committed boundary still returns success, and the maintenance state and refs are untouched by the advisory. + - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) clock-skew tolerance, with an injected clock for every case (`FUTURE_SKEW_TOLERANCE` = 5 min): (1) stored `last_success` equal to `now` — valid, age 0, no advice, file byte-identical; (2) one minute ahead of `now` — valid, age 0, preserved, no advice, no rewrite; (3) exactly five minutes ahead — valid (boundary is inclusive), age 0, preserved, no advice, no rewrite; (4) more than five minutes ahead (just beyond, and far-future) for `last_success`, `last_advised` and `anchor` — invalid, normalized once under the lock and persisted per D3 (invalid fields cleared; with `last_success` and `anchor` both unusable the advisory fires immediately once, persists `last_advised = now` and a fresh `anchor = now`); (5) clock rollback beyond tolerance (stored values valid at write time, injected `now` moved back more than five minutes) — treated as case 4; a rollback within tolerance is case 2/3 and never makes an advisory eligible by itself; (6) repeated reads after normalization — the file is byte-identical across repeated calls, no rewrite occurs, and eligibility is neither postponed nor slid; (7) process restart (fresh call re-reading the file) followed by clock advance past `RECONCILIATION_ADVISORY_AFTER` — advice recurs exactly once, proving eventual eligibility. After every **successful state-writing** operation an assertion checks the exact invariant `t ≤ now + FUTURE_SKEW_TOLERANCE` for every stored timestamp (not "no timestamp is in the future"); after read-only, lock-busy, or failed operations the assertion is instead that the state file is byte-identical to its prior content, with no normalization claimed; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next successful recorded operation (until then, read-only and failed operations leave them untouched and treat them as default state). + - *Timestamp-normalization regressions (P2; injected clocks and deterministic synchronization, no sleeps):* (S34) read-only doctor observing an invalid future timestamp — plain `sce doctor` reports "recommended" with the invalid-timestamp note, the state file is byte-identical afterwards, and it is not reported as normalized; (S35) lock-busy advisory — with the lock held through a barrier and an invalid future timestamp stored, `advise_if_due` returns `Busy` and the state file is byte-identical; the same holds for a lock-busy explicit pass (`Skipped(Busy)`); (S36) failed state write — an injected write error (and, in a second parameterization, a rename interrupted before replacement) during a normalizing operation leaves the previous state byte-identical and readable, emits no advice, reports no normalization (`StateUnavailable` for the advisory, `CompletedStatePersistFailed` for the explicit pass), and a later successful operation still normalizes it; (S37) successful normalization — a normalizing advisory and a normalizing explicit pass each persist state in which every timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`, valid near-future values (within tolerance) are preserved byte-for-byte, and a repeated read performs no rewrite and does not move eligibility forward; (S38) process restart after normalization — a fresh call re-reading the persisted file observes the normalized timestamps, performs no rewrite, and does not re-fire the immediate advice; (S39) eventual eligibility — with invalid timestamps unnormalized (S34/S35/S36 states) repeated read-only evaluations never push eligibility out, and after the injected clock advances past `RECONCILIATION_ADVISORY_AFTER` a successful advisory fires exactly once and persists normalized state. - *Explicit recovery:* (S26) `sce doctor --fix` runs and reports its true result from each of: failed streak, stale advice, invalid timestamps, corrupt state, unwritable state (→ `CompletedStatePersistFailed`); (S27) explicit success resets the failure streak and sets `last_success`; explicit failure increments the streak without touching `last_success`; lock-busy never changes state. - - *DB opening (D4):* (S28) missing DB — no file or directory created, refs untouched; (S29) incompatible schema — no migration or repair row written; (S30) missing repository metadata — no row inserted; (S31) repository identity mismatch — refs untouched; (S32) normal existing DB — pass completes. All failing cases assert refs unchanged, DB logical contents unchanged, and the external-taint marker unarmed. + - *DB opening (D4):* (S28) missing DB — fails with the `Missing` typed error before Turso is invoked (a spy/seam on the Turso open is never called); no database file, parent directory, WAL, or shared-memory file is created (directory listing before and after is identical, including when the parent directory is absent); Git refs are neither listed nor deleted (spy on `list_pins`/`delete_pins` never called); (S29) existing DB with incompatible schema — opening is permitted; fails with `IncompatibleSchema`; no migration or repair row written (migration table rows identical before and after); no ref deleted; (S30) existing DB with missing repository metadata — fails with `MissingMetadata`; no `repository_metadata` row inserted; (S31) existing DB with repository identity mismatch — fails with `RepositoryMismatch`; no row changed; (S32) existing valid DB — identity and schema readiness are verified before any ref listing (ordering asserted via seam), the multiprocess-WAL configuration is preserved, the pass completes, and no initialization, repair, or migration runs. For S29–S31 the assertions are logical: a before/after dump of every table's rows is equal (no migration, repair, initialization, or application-table INSERT/UPDATE/DELETE), refs are unchanged, and the external-taint marker is unarmed. Filesystem immutability is **not** asserted for existing-DB failure cases, because incidental Turso WAL/shared-memory bookkeeping is permitted when unavoidable; only S28 asserts that no database-related file or directory is created. Out — production behavior changes beyond test seams strictly required. - Dependencies: T03, T04 - Done when: all scenarios pass reliably, each fails when the guarded behavior is deliberately broken, no test depends on timing or sleeps, and every review finding in the traceability table maps to at least one passing scenario. @@ -291,6 +321,54 @@ Record actual outcomes in the validation report; compilation alone does not vali - Verify: `nix develop -c ./scripts/run-cli-cargo.sh clippy --manifest-path cli/Cargo.toml --all-targets --all-features -- -D warnings`; `nix shell nixpkgs#ripgrep -c rg -n "allow\(dead_code" cli/src/services/mutation_trace`; `rg -i "retained and unwired" context`; `rg -i "automatic.*(reclam|reconcil)" context` hits only deferral statements. - Context synchronization: pending +## Audit findings + +Recorded by T01 (read-only audit, 2026-10-09). Line numbers refer to the tree at `fb7e78fb`. + +**PR metadata (confirmed via `gh pr view`).** PR #304: head branch `fix-logging`, head `0be74aad5bea5070828846d3277041a4649c9842`, base `dead-code-blues`. PR #305: head branch `ref-recon`, base `fix-logging`, head `fb7e78fbfa41f9f054803eef2dddb180f5d5289a`. No rebase needed. + +**Lock ownership (D1) — confirmed.** +- `ref_reconciliation.rs:159-243` `reconcile_worktree_inner` resolves `git_dir` (`:168`), then acquires `WorktreeLock` via `acquire_inner_async(&git_dir, RECONCILIATION_LOCK_TIMEOUT /*10 s, :19*/, on_lock_contention)` (`:172`), then `resolve_worktree_id` (`:176`), `open_db` (`:180`), `GitSnapshotService::new` (`:184`), `list_pins` (`:190`), `load_tree_roots` (`:207`), `load_all_tree_roots` (`:222`), `delete_pins(lock.lease(), &stale)` (`:233`). `lock` drops on return. +- `worktree_lock.rs`: non-reentrant (`File::try_lock`, `:114`); `acquire_inner` loops `try_lock` and, on `WouldBlock`, checks `now >= deadline` before sleeping (`:123-135`), so `Duration::ZERO` is exactly one attempt then `TimedOut`. `WorktreeLockLease` is an `Arc` clone (`:147-151`); the file unlocks only when the last clone drops (`WorktreeLockInner::drop`, `:158-164`). `acquire_inner` (sync, `pub(super)`, `:79`) is available for the synchronous advisory; `acquire_inner_async` wraps it in `spawn_blocking` (`:65-77`). +- `git_snapshot.rs:281-298` `run_ref_mutation_inner` moves the lease into `spawn_blocking` (`let _lease = lease;`, `:292`) and calls `on_worker_entered()` (`:293`); `run_ref_mutation` passes `|| {}` (`:278`). `delete_pins` is `:214`, `delete_pins_inner` `:231`. Dropping the caller future therefore cannot release the file lock while the worker runs. +- `protected_worktree.rs:14` `WORKTREE_LOCK_TIMEOUT` = 10 s; `ProtectedWorktree::acquire_with_timeout` (`:109-138`) = `resolve_git_dir` → lock → `arm_marker_with_lock` → `resolve_worktree_id`. The lock is released when `coordinate_inner` (`coordinator.rs:215-252`) returns (`protected.complete()` consumes `self`; `complete_blocking` clears the marker, then the `lock` field drops). `abandon_after_spawn_without_unlock` (`:182-185`) calls `close_without_unlock` (`worktree_lock.rs:153`) for the external-guard child path only. +- **Split**: `reconcile_with_held_lock

(repository_root, &WorktreeLock, worktree_id/open_db…)` takes everything from `:176` onward; the lock is passed as `&WorktreeLock` (so it can call `lock.lease()` at the single `delete_pins` call). `reconcile_worktree_inner` becomes: resolve git dir, acquire, delegate. `reconcile_explicit` resolves git dir itself and acquires once; it never calls `reconcile_worktree[_inner]`. +- Plan-text correction: `ReconcileError::DurableRoots` doc says "migration `003`"; the durable-roots/tree-root queries belong to the mutation-trace protocol migration `004_mutation_trace_protocol.sql` (current set `001`–`006` in `cli/migrations/agent-trace-repository/`; `003` is `claude_model_state`). T02 should not hard-code a single migration id; D4 readiness must use the existing "all expected migrations applied" check (`db/mod.rs:1189 migration_metadata_problems`, used by `ensure_schema_ready` `:1251`). + +**Production call graph and completion classes.** +- Every harness adapter funnels into `hooks::mutation_scope::run_mutation_scope_from_payload` (`hooks/{claude,codex,opencode,pi}_mutation_scope/lifecycle.rs`; `hooks/mod.rs:108` direct subcommand; `doctor/inspect.rs:305` repair seam). → `run_mutation_scope_from_payload_with` (`mutation_scope.rs:269`) → `drive_mutation_scope` (`:303`): `Start`/`Advance`/`Close`/`Flush` → `coordinate_boundary(...).await` → `classify_coordinate` (`:362-376`); `Abandon` → `classify_abandon` (`:378-392`). `external_mutation_guard` is a separate subcommand (`hooks/mod.rs:123`) using `coordinate_on_held_worktree`. +- Durably completed: `Ok(_)` and `Err(CoordinateError::MarkerClearAfterCommit)` (`:366-371`); all other `CoordinateError` variants (`SnapshotFailure`, `CasConflictExhausted`, `RevisionExhausted`, `ExternalTaintMarker`, `AgentTraceDbUnavailable`, `LockAcquisition`, `ScopeIdentityConflict`, `ScopeProvenanceRegistration`, `Other`) are pre-completion failures. +- Hook-ingress seam: `drive_mutation_scope` takes injected `coordinate_boundary`/`abandon` closures; `classify_coordinate(coordinate_boundary(...).await, logger)` at `:356-359` is the exact insertion point (after the await, so the coordinator lock is already dropped). Requires `boundary` to be `Close`/`Flush`, and the result to be `Ok` or `MarkerClearAfterCommit`. `run_mutation_scope_from_payload_with` also gets `doctor/inspect.rs:305`'s repair path, so an advisory placed in `drive_mutation_scope` fires there too; harmless (it still never reconciles) but T03 should test it is unaffected. + +**Execution-point investigation (D2) — conclusion confirmed, with one flag.** +- No lifecycle-owned, noncritical-path, cross-harness executor exists. `post-commit` (`hooks/commit_hooks.rs:114`) is synchronous inside `git commit`; per-harness stop/session-end handlers (for example `hooks/codex/stop.rs`) are separate, per-harness, synchronous hook processes that persist their own DB rows and are not part of the shared mutation-scope ingress; `sce setup` is install-time; `doctor/mod.rs` is the operator-owned foreground path. +- **Flag (does not invalidate the plan):** `post-commit` already launches a detached child via `sync/auto_sync.rs:39` `launch` (spawns the current executable with null stdio and drops the `Child`). That is a fire-and-forget subprocess precedent. D2/constraint 6 prohibit fire-and-forget subprocesses for reconciliation, and a short-lived hook process cannot supervise one, so the selected design stands; the follow-up PR may cite this precedent when evaluating an out-of-band executor. +- Tokio shutdown joining blocking workers and the unbounded cost of `git for-each-ref`/durable-root queries are as stated in the authoring-time observations; nothing found contradicts them. +- Decision: advisory-only hook trigger plus `sce doctor --fix`. + +**Advisory Git-dir resolution (D2 decision rule) — advisory kept, filesystem derivation.** +- `coordinate` resolves the Git dir internally (`resolve_git_dir`, a `git rev-parse --absolute-git-dir` subprocess, `git_snapshot.rs:602-613`, via `run_rev_parse` `:699`) and does not return it (`CoordinateOutcome` carries `worktree_id`, `observed_tree`, …, not the Git dir), so reuse would need a protocol/API change. Not chosen. +- Filesystem derivation: the hook's `repository_root` is `std::env::current_dir()` (`hooks/mod.rs:63`), which may be a subdirectory. Starting there, walk ancestors to the nearest entry named `.git`; if it is a directory it is the Git dir; if it is a file containing `gitdir: `, the Git dir is that path (resolved relative to the file's parent when relative). Verified empirically in a scratch repo: from a subdirectory of the main worktree `git rev-parse --absolute-git-dir` equals `/.git`; in a linked worktree the `.git` file contains `gitdir: /.git/worktrees/`, which equals `--absolute-git-dir`. The lock file and state file are addressed by inode, so path spelling differences (symlinks) do not change behavior, but T02 should canonicalize both anyway and treat any mismatch as `StateUnavailable`. +- Guardrails T02 must implement: if `GIT_DIR`, `GIT_COMMON_DIR`, or `GIT_WORK_TREE` is set, or no `.git` is found, or the `gitdir:` target does not exist, return `StateUnavailable` (no advice, no Git subprocess). Because `coordinate` just succeeded from the same directory, the nearest-`.git` walk cannot find a different repository than Git did. +- Outcome: **hook trigger kept; no new Git subprocess.** The derivation function takes only `&Path`, performs only local `std::fs` reads, and lives in the advisory/state module (no `resolve_git_dir` import). + +**State (D3) — confirmed.** +- Placement: `/sce/` beside `mutation-cursor.lock` (`worktree_lock.rs:9-11`) and `mutation-cursor-tainted` (`external_taint.rs:26-29`). No in-memory registry is needed. +- No atomic-write helper exists in the runtime module: `ExternalTaintMarker::persist` (`external_taint.rs:82-112`) is create+`sync_data` of an empty file with best-effort parent-dir `sync_all` (`:141-145`). The repository's staging-and-swap pattern is `setup/install.rs` (`create_*_staging_path` with `{epoch_nanos}-{pid}-{attempt}` names at `:224-256`, `fs::write` to staging, `rename_fn` injected for test interruption at `:40,185`, `cleanup_path_if_exists` on failure). T02 should mirror this with an injected rename/write seam rather than invent a new pattern, plus the marker's best-effort parent-dir sync. +- Clock: there is no injectable wall clock in the runtime; `hooks/runtime.rs` has `current_unix_time_ms()` (`SystemTime`/`UNIX_EPOCH`, i64 ms). T02 takes `now: impl Fn() -> i64` and production passes `current_unix_time_ms`-equivalent logic. `sync/sync.rs:78` defines a `now()` trait for another subsystem; not reused. + +**DB opening (D4) — confirmed not read-only; opener contract.** +- `open_agent_trace_db_for_hook_runtime` (`hooks/runtime.rs:53-69`) resolves config → `resolve_agent_trace_storage_for_hook_runtime` → `open_storage_for_hook_runtime` → `open_repository_db_for_hook_runtime` (`agent_trace_storage/mod.rs:195-207`): (a) `TursoDb::open_without_migrations_at` calls `ensure_db_parent_dir` and `turso::Builder::new_local(path).experimental_multiprocess_wal(true).build()` (`db/mod.rs:856-880`), which can create parent directories and an empty DB file; (b) on `ensure_schema_ready_for_hooks` failure it calls `repair_missing_repository_schema_migration_metadata` (`agent_trace_db/repository.rs:136-150`), which inserts the baseline migration row; (c) `verify_or_initialize_repository_metadata` (`repository.rs:171-230`) inserts the `repository_metadata` row and claims `source_instance_id` when absent/invalid. +- Turso: crate `turso = 0.8.1` (`cli/Cargo.toml:32`); no read-only open or open flag is used anywhere in `cli/src`. T02 first confirms whether the 0.8.1 builder exposes one; if not, accept the documented limitation (existence pre-check, `SELECT`-only afterward, tests assert logical contents unchanged). The multiprocess-WAL builder option must be kept so the open coordinates with live hook writers. +- Smallest opener: a new function beside `open_repository_db_for_hook_runtime` that (1) resolves identity and path exactly as the hook runtime does (`resolve_identity`, `agent_trace_db_path_for_repository`), (2) returns `Missing` if the path is not an existing file before touching Turso, (3) opens without creating directories (a new `open_existing_without_migrations_at` on `TursoDb` that skips `ensure_db_parent_dir`, or an equivalent guard), (4) runs `ensure_schema_ready_for_hooks` (all expected migrations `001`–`006` applied) and on failure returns `IncompatibleSchema` without repair, (5) calls a new read-only `verify_existing_repository_metadata(repository_id)` built on `select_repository_metadata_row` (`repository.rs:232`) returning `MissingMetadata` when the row is absent or `source_instance_id` fails `is_valid_source_instance_id` (`:74`), and `RepositoryMismatch` via the existing `ensure_repository_id_matches`. The hook opener and its callers are unchanged. + +**Identity and stale-doc findings.** +- `resolve_worktree_id` (`git_snapshot.rs:666-685`): `git_dir == git_common_dir` → `main`; otherwise `worktrees/`. It is Git-topology-derived and failure is `Err`. +- `ReconciliationOutcome::SkippedNoCheckoutIdentity` (`ref_reconciliation.rs:59`) is declared and documented (`:36`, `:50-51`) but never constructed anywhere in `cli/src`; confirmed unreachable. `context/cli/mutation-trace-ref-reconciliation.md` still describes `read_checkout_id → Ok(None)` and "checkout identity" namespaces (stale). T06 removes the variant (callers must then match one variant) and rewrites the docs. +- `#[allow(dead_code, reason = "ref reconciliation retained and unwired; …")]` appears on `ref_reconciliation.rs` declarations, `git_snapshot.rs` (`run_ref_mutation`, `assert_pins_are_unchanged_direct_refs`, others) and ten places in `store.rs` (`:199, :225, :249, :259, :272, :713, :844, :874, :1022, :1195`); T06 removes only those that gain production consumers. + +**Findings affecting later tasks.** None invalidates a task's scope. T02 must: use `acquire_inner` (sync, zero timeout) for the advisory; implement filesystem Git-dir derivation with the stated guardrails; add the `Missing`/`IncompatibleSchema`/`MissingMetadata`/`RepositoryMismatch` opener and confirm Turso read-only support; reuse the `setup/install.rs` staging pattern. T03 must also account for the `doctor/inspect.rs:305` caller. T06 must also fix the stale "003" wording in the `ReconcileError` docs. + ## Open questions - Is a 24-hour `RECONCILIATION_ADVISORY_AFTER` the right window for the time-based recommendation? It is only a starting value recorded by T03. From 68dcb062b487f0569b4ecfa4c72080ec5ced9aac Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 00:45:22 +0200 Subject: [PATCH 03/16] runtime: Add ref maintenance entrypoints and state tracking Add explicit reconciliation and hook advisory entrypoints that share one held worktree lock, persist bounded maintenance state, and record successful and failed passes. Add verified-existing agent-trace DB opening so maintenance does not create or migrate databases, while preserving the existing deletion path. Plan: `mutation-cursor-ref-reconciliation-wiring.md` (T02) Co-authored-by: SCE --- cli/src/services/agent_trace_db/repository.rs | 104 +++++ cli/src/services/agent_trace_storage/mod.rs | 32 ++ cli/src/services/db/mod.rs | 24 +- .../runtime/maintenance_state.rs | 382 ++++++++++++++++++ .../services/mutation_trace/runtime/mod.rs | 3 + .../mutation_trace/runtime/ref_advisory.rs | 289 +++++++++++++ .../mutation_trace/runtime/ref_maintenance.rs | 174 ++++++++ .../runtime/ref_reconciliation.rs | 377 ++++++++++++++--- ...tation-cursor-ref-reconciliation-wiring.md | 9 +- 9 files changed, 1323 insertions(+), 71 deletions(-) create mode 100644 cli/src/services/mutation_trace/runtime/maintenance_state.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_advisory.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance.rs diff --git a/cli/src/services/agent_trace_db/repository.rs b/cli/src/services/agent_trace_db/repository.rs index 59bfd78ba..3c606757c 100644 --- a/cli/src/services/agent_trace_db/repository.rs +++ b/cli/src/services/agent_trace_db/repository.rs @@ -85,6 +85,53 @@ fn ensure_repository_id_matches(stored_repository_id: &str, repository_id: &str) Ok(()) } +#[derive(Debug)] +#[allow( + dead_code, + reason = "maintenance wiring lands in later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" +)] +pub enum ExistingRepositoryDbError { + Missing { path: PathBuf }, + Unreadable(anyhow::Error), + IncompatibleSchema(anyhow::Error), + MissingMetadata, + RepositoryMismatch { stored: String, resolved: String }, +} + +impl std::fmt::Display for ExistingRepositoryDbError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + ExistingRepositoryDbError::Missing { path } => write!( + f, + "repository Agent Trace DB does not exist at '{}'. \ + {REPOSITORY_AGENT_TRACE_SCHEMA_SETUP_GUIDANCE}", + path.display() + ), + ExistingRepositoryDbError::Unreadable(source) => { + write!(f, "repository Agent Trace DB is unreadable: {source}") + } + ExistingRepositoryDbError::IncompatibleSchema(source) => { + write!( + f, + "repository Agent Trace DB schema is incompatible: {source}" + ) + } + ExistingRepositoryDbError::MissingMetadata => write!( + f, + "repository Agent Trace DB metadata is missing or has no valid source-instance ID. \ + {REPOSITORY_AGENT_TRACE_SCHEMA_SETUP_GUIDANCE}" + ), + ExistingRepositoryDbError::RepositoryMismatch { stored, resolved } => write!( + f, + "repository Agent Trace DB metadata mismatch: stored repository ID {stored} \ + does not match resolved repository ID {resolved}" + ), + } + } +} + +impl std::error::Error for ExistingRepositoryDbError {} + /// Repository-scoped Agent Trace database configuration. pub struct RepositoryAgentTraceDbSpec; @@ -120,6 +167,63 @@ impl RepositoryAgentTraceDb { TursoDb::::open_without_migrations_at(path).await } + #[allow( + dead_code, + reason = "maintenance wiring lands in later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" + )] + pub async fn open_verified_existing_at( + path: impl AsRef, + repository_id: &str, + ) -> std::result::Result<(Self, RepositoryMetadata), ExistingRepositoryDbError> { + let path = path.as_ref(); + if !path.is_file() { + return Err(ExistingRepositoryDbError::Missing { + path: path.to_path_buf(), + }); + } + + let db = TursoDb::::open_existing_without_migrations_at(path) + .await + .map_err(ExistingRepositoryDbError::Unreadable)?; + db.ensure_schema_ready_for_hooks() + .await + .map_err(ExistingRepositoryDbError::IncompatibleSchema)?; + let metadata = db + .verify_existing_repository_metadata(repository_id) + .await?; + Ok((db, metadata)) + } + + #[allow( + dead_code, + reason = "maintenance wiring lands in later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" + )] + pub async fn verify_existing_repository_metadata( + &self, + repository_id: &str, + ) -> std::result::Result { + let Some((stored_repository_id, source_instance_id)) = self + .select_repository_metadata_row() + .await + .map_err(ExistingRepositoryDbError::Unreadable)? + else { + return Err(ExistingRepositoryDbError::MissingMetadata); + }; + if stored_repository_id != repository_id { + return Err(ExistingRepositoryDbError::RepositoryMismatch { + stored: stored_repository_id, + resolved: repository_id.to_string(), + }); + } + if !is_valid_source_instance_id(&source_instance_id) { + return Err(ExistingRepositoryDbError::MissingMetadata); + } + Ok(RepositoryMetadata { + repository_id: stored_repository_id, + source_instance_id, + }) + } + /// Verify that the repository-scoped schema baseline already exists. pub async fn ensure_schema_ready_for_hooks(&self) -> Result<()> { self.ensure_schema_ready(REPOSITORY_AGENT_TRACE_SCHEMA_SETUP_GUIDANCE) diff --git a/cli/src/services/agent_trace_storage/mod.rs b/cli/src/services/agent_trace_storage/mod.rs index 9cecacebd..52d38c7a8 100644 --- a/cli/src/services/agent_trace_storage/mod.rs +++ b/cli/src/services/agent_trace_storage/mod.rs @@ -208,3 +208,35 @@ async fn open_repository_db_for_hook_runtime( .await?; Ok((db, metadata)) } + +#[allow( + dead_code, + reason = "maintenance wiring lands in later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" +)] +pub async fn resolve_existing_agent_trace_storage_for_maintenance( + context: &AgentTraceStorageContext<'_>, +) -> Result { + let repository_identity = resolve_identity(context)?; + let db_path = agent_trace_db_path_for_repository(&repository_identity.identity.repository_id)?; + open_existing_storage_for_maintenance(repository_identity, db_path).await +} + +#[allow( + dead_code, + reason = "maintenance wiring lands in later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" +)] +async fn open_existing_storage_for_maintenance( + repository_identity: ResolvedRepositoryIdentity, + db_path: PathBuf, +) -> Result { + let repository_id = &repository_identity.identity.repository_id; + let (db, metadata) = + RepositoryAgentTraceDb::open_verified_existing_at(&db_path, repository_id).await?; + + Ok(ResolvedAgentTraceStorage { + repository_identity, + db_path, + db, + metadata, + }) +} diff --git a/cli/src/services/db/mod.rs b/cli/src/services/db/mod.rs index c2d4ff16d..4e093f92e 100644 --- a/cli/src/services/db/mod.rs +++ b/cli/src/services/db/mod.rs @@ -840,17 +840,25 @@ impl TursoDb { Self::open_without_migrations_at(db_path).await } - /// Open or create the database at an explicit path without running embedded - /// migrations. - /// - /// Parent directories are created automatically and the connection-open - /// retry policy is preserved. Runtime callers that use this path are - /// responsible for verifying schema readiness before query/write work. pub async fn open_without_migrations_at(db_path: impl AsRef) -> Result { + Self::open_at_path(db_path.as_ref(), true).await + } + + #[allow( + dead_code, + reason = "maintenance wiring lands in later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" + )] + pub async fn open_existing_without_migrations_at(db_path: impl AsRef) -> Result { + Self::open_at_path(db_path.as_ref(), false).await + } + + async fn open_at_path(db_path: &Path, create_parent_dir: bool) -> Result { let db_name = M::db_name(); - let db_path = db_path.as_ref().to_path_buf(); + let db_path = db_path.to_path_buf(); - ensure_db_parent_dir(db_name, &db_path)?; + if create_parent_dir { + ensure_db_parent_dir(db_name, &db_path)?; + } let retry_policy = resolve_connection_open_retry_policy::(); let busy_timeout = resolve_busy_timeout::(); diff --git a/cli/src/services/mutation_trace/runtime/maintenance_state.rs b/cli/src/services/mutation_trace/runtime/maintenance_state.rs new file mode 100644 index 000000000..9c7ebeb51 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/maintenance_state.rs @@ -0,0 +1,382 @@ +#![allow( + dead_code, + reason = "maintenance entrypoints are wired by later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" +)] + +use std::io::{Read, Write}; +use std::path::{Path, PathBuf}; +use std::time::{SystemTime, UNIX_EPOCH}; + +use serde::{Deserialize, Serialize}; + +pub(super) const RECONCILIATION_ADVISORY_AFTER_MS: i64 = 24 * 60 * 60 * 1000; +pub(super) const MAINTENANCE_STATE_MAX_BYTES: u64 = 4096; +pub(super) const FUTURE_SKEW_TOLERANCE_MS: i64 = 5 * 60 * 1000; +pub(super) const MAX_FAILURE_MESSAGE_CHARS: usize = 256; + +const MAINTENANCE_STATE_VERSION: u32 = 1; +const SCE_RUNTIME_DIR: &str = "sce"; +const MAINTENANCE_STATE_FILE: &str = "ref-maintenance.json"; +const STAGING_ATTEMPTS: u16 = 1000; + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub(super) enum AttemptOutcome { + Completed, + Failed, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub(super) struct StoredReport { + pub retained: usize, + pub deleted: usize, + pub local_required: usize, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub(super) struct StoredFailure { + pub kind: String, + pub message: String, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub(super) struct MaintenanceState { + pub version: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub anchor: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_success: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_attempt: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_advised: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_attempt_outcome: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_report: Option, + #[serde(default)] + pub consecutive_failures: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_failure: Option, +} + +impl Default for MaintenanceState { + fn default() -> Self { + Self { + version: MAINTENANCE_STATE_VERSION, + anchor: None, + last_success: None, + last_attempt: None, + last_advised: None, + last_attempt_outcome: None, + last_report: None, + consecutive_failures: 0, + last_failure: None, + } + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(super) enum StateRead { + Absent, + Unusable, + Valid(MaintenanceState), +} + +impl StateRead { + pub(super) fn into_state(self) -> MaintenanceState { + match self { + StateRead::Valid(state) => state, + StateRead::Absent | StateRead::Unusable => MaintenanceState::default(), + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) struct Evaluation { + pub due: bool, + pub advise: bool, + pub invalid_timestamps: bool, + pub reference_age_ms: Option, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(super) struct Recommendation { + pub recommended: bool, + pub invalid_timestamps: bool, + pub last_success_age_ms: Option, + pub last_attempt_age_ms: Option, + pub last_advised_age_ms: Option, + pub last_attempt_outcome: Option, + pub consecutive_failures: u32, +} + +pub(super) fn system_now_ms() -> i64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .ok() + .and_then(|elapsed| i64::try_from(elapsed.as_millis()).ok()) + .unwrap_or(0) +} + +pub(super) fn state_path(git_dir: &Path) -> PathBuf { + git_dir.join(SCE_RUNTIME_DIR).join(MAINTENANCE_STATE_FILE) +} + +fn is_valid_timestamp(timestamp: i64, now: i64) -> bool { + timestamp <= now.saturating_add(FUTURE_SKEW_TOLERANCE_MS) +} + +fn age_ms(timestamp: i64, now: i64) -> i64 { + now.saturating_sub(timestamp).max(0) +} + +fn usable(timestamp: Option, now: i64) -> Option { + timestamp.filter(|value| is_valid_timestamp(*value, now)) +} + +pub(super) fn read_state(path: &Path) -> std::io::Result { + let file = match std::fs::File::open(path) { + Ok(file) => file, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + return Ok(StateRead::Absent); + } + Err(error) => return Err(error), + }; + + let mut bytes = Vec::new(); + file.take(MAINTENANCE_STATE_MAX_BYTES + 1) + .read_to_end(&mut bytes)?; + if u64::try_from(bytes.len()).map_or(true, |len| len > MAINTENANCE_STATE_MAX_BYTES) { + return Ok(StateRead::Unusable); + } + + Ok(match serde_json::from_slice::(&bytes) { + Ok(state) if state.version == MAINTENANCE_STATE_VERSION => StateRead::Valid(state), + Ok(_) | Err(_) => StateRead::Unusable, + }) +} + +pub(super) fn evaluate(state: &MaintenanceState, now: i64) -> Evaluation { + let timestamps = [ + state.anchor, + state.last_success, + state.last_attempt, + state.last_advised, + ]; + let invalid_timestamps = timestamps + .iter() + .flatten() + .any(|timestamp| !is_valid_timestamp(*timestamp, now)); + + let reference = usable(state.last_success, now).or_else(|| usable(state.anchor, now)); + let reference_lost = + reference.is_none() && (state.last_success.is_some() || state.anchor.is_some()); + let reference_age_ms = reference.map(|timestamp| age_ms(timestamp, now)); + let due = reference_lost + || reference_age_ms.is_some_and(|age| age >= RECONCILIATION_ADVISORY_AFTER_MS); + let advised_recently = usable(state.last_advised, now) + .is_some_and(|timestamp| age_ms(timestamp, now) < RECONCILIATION_ADVISORY_AFTER_MS); + + Evaluation { + due, + advise: due && !advised_recently, + invalid_timestamps, + reference_age_ms, + } +} + +pub(super) fn evaluate_recommendation(state: &MaintenanceState, now: i64) -> Recommendation { + let evaluation = evaluate(state, now); + let age_of = |timestamp: Option| usable(timestamp, now).map(|value| age_ms(value, now)); + + Recommendation { + recommended: evaluation.due || evaluation.invalid_timestamps, + invalid_timestamps: evaluation.invalid_timestamps, + last_success_age_ms: age_of(state.last_success), + last_attempt_age_ms: age_of(state.last_attempt), + last_advised_age_ms: age_of(state.last_advised), + last_attempt_outcome: state.last_attempt_outcome, + consecutive_failures: state.consecutive_failures, + } +} + +pub(super) fn normalize(state: &MaintenanceState, now: i64) -> MaintenanceState { + let mut normalized = state.clone(); + normalized.anchor = usable(state.anchor, now); + normalized.last_success = usable(state.last_success, now); + normalized.last_attempt = usable(state.last_attempt, now); + normalized.last_advised = usable(state.last_advised, now); + normalized +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum AdvisoryDecision { + Anchored, + NoAction, + Advised, +} + +pub(super) fn plan_advisory( + read: &StateRead, + now: i64, +) -> (AdvisoryDecision, Option) { + let StateRead::Valid(state) = read else { + let anchored = MaintenanceState { + anchor: Some(now), + ..MaintenanceState::default() + }; + return (AdvisoryDecision::Anchored, Some(anchored)); + }; + + let evaluation = evaluate(state, now); + let mut next = normalize(state, now); + let decision = if evaluation.advise { + next.last_advised = Some(now); + if next.anchor.is_none() && next.last_success.is_none() { + next.anchor = Some(now); + } + AdvisoryDecision::Advised + } else if next.anchor.is_none() && next.last_success.is_none() { + next.anchor = Some(now); + AdvisoryDecision::Anchored + } else { + AdvisoryDecision::NoAction + }; + + let changed = next != *state; + (decision, changed.then_some(next)) +} + +pub(super) fn record_success( + state: &MaintenanceState, + now: i64, + report: StoredReport, +) -> MaintenanceState { + let mut next = normalize(state, now); + next.last_attempt = Some(now); + next.last_success = Some(now); + next.last_attempt_outcome = Some(AttemptOutcome::Completed); + next.last_report = Some(report); + next.consecutive_failures = 0; + next.last_failure = None; + next +} + +pub(super) fn record_failure( + state: &MaintenanceState, + now: i64, + kind: &str, + message: &str, +) -> MaintenanceState { + let mut next = normalize(state, now); + next.last_attempt = Some(now); + next.last_attempt_outcome = Some(AttemptOutcome::Failed); + next.consecutive_failures = state.consecutive_failures.saturating_add(1); + next.last_failure = Some(StoredFailure { + kind: kind.to_string(), + message: message.chars().take(MAX_FAILURE_MESSAGE_CHARS).collect(), + }); + next +} + +pub(super) fn write_state_atomically( + path: &Path, + state: &MaintenanceState, + rename: R, +) -> std::io::Result<()> +where + R: FnOnce(&Path, &Path) -> std::io::Result<()>, +{ + let bytes = serde_json::to_vec(state).map_err(std::io::Error::other)?; + let directory = path.parent().ok_or_else(|| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "maintenance state path has no parent directory", + ) + })?; + + let staging = create_staging_file(directory)?; + let staged = write_staged(&staging, &bytes).and_then(|()| rename(&staging, path)); + if let Err(error) = staged { + let _ = std::fs::remove_file(&staging); + return Err(error); + } + + if let Ok(directory_handle) = std::fs::File::open(directory) { + let _ = directory_handle.sync_all(); + } + Ok(()) +} + +fn write_staged(staging: &Path, bytes: &[u8]) -> std::io::Result<()> { + let mut file = std::fs::OpenOptions::new().write(true).open(staging)?; + file.write_all(bytes)?; + file.sync_data() +} + +fn create_staging_file(directory: &Path) -> std::io::Result { + let epoch_nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(std::io::Error::other)? + .as_nanos(); + + for attempt in 0..STAGING_ATTEMPTS { + let candidate = directory.join(format!( + ".{MAINTENANCE_STATE_FILE}.staging-{epoch_nanos}-{}-{attempt}", + std::process::id() + )); + match std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&candidate) + { + Ok(_) => return Ok(candidate), + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {} + Err(error) => return Err(error), + } + } + + Err(std::io::Error::new( + std::io::ErrorKind::AlreadyExists, + "could not allocate a unique maintenance state staging file", + )) +} + +pub(super) fn derive_git_dir(start: &Path) -> Option { + if ["GIT_DIR", "GIT_COMMON_DIR", "GIT_WORK_TREE"] + .iter() + .any(|name| std::env::var_os(name).is_some()) + { + return None; + } + + for ancestor in start.ancestors() { + let dot_git = ancestor.join(".git"); + let Ok(metadata) = std::fs::symlink_metadata(&dot_git) else { + continue; + }; + + let git_dir = if metadata.is_dir() { + dot_git + } else if metadata.is_file() { + let contents = std::fs::read_to_string(&dot_git).ok()?; + let target = contents.lines().next()?.strip_prefix("gitdir:")?.trim(); + let target = Path::new(target); + if target.is_absolute() { + target.to_path_buf() + } else { + ancestor.join(target) + } + } else { + return None; + }; + + return git_dir + .is_dir() + .then(|| std::fs::canonicalize(&git_dir).ok()) + .flatten(); + } + + None +} diff --git a/cli/src/services/mutation_trace/runtime/mod.rs b/cli/src/services/mutation_trace/runtime/mod.rs index cd99b7307..a97020076 100644 --- a/cli/src/services/mutation_trace/runtime/mod.rs +++ b/cli/src/services/mutation_trace/runtime/mod.rs @@ -2,8 +2,11 @@ mod coordinator; mod external_mutation_guard; mod external_taint; mod git_snapshot; +mod maintenance_state; mod mutation_attribution; mod protected_worktree; +mod ref_advisory; +mod ref_maintenance; mod ref_reconciliation; mod scope_runtime; mod worktree_lock; diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs new file mode 100644 index 000000000..405748cb7 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -0,0 +1,289 @@ +#![allow( + dead_code, + reason = "maintenance entrypoints are wired by later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" +)] + +use std::path::Path; +use std::time::Duration; + +use super::maintenance_state::{ + derive_git_dir, plan_advisory, read_state, state_path, write_state_atomically, + AdvisoryDecision, MaintenanceState, +}; +use super::worktree_lock::{acquire_inner, WorktreeLockError}; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum AdvisoryOutcome { + Anchored, + NoAction, + Advised, + Busy, + StateUnavailable, +} + +pub(super) fn advise_if_due(repository_root: &Path, now: C) -> AdvisoryOutcome +where + C: Fn() -> i64, +{ + advise_if_due_with(repository_root, now, |path, state| { + write_state_atomically(path, state, |from, to| std::fs::rename(from, to)) + }) +} + +pub(super) fn advise_if_due_with( + repository_root: &Path, + now: C, + write_state: W, +) -> AdvisoryOutcome +where + C: Fn() -> i64, + W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, +{ + let Some(git_dir) = derive_git_dir(repository_root) else { + return AdvisoryOutcome::StateUnavailable; + }; + let path = state_path(&git_dir); + + let Ok(unlocked) = read_state(&path) else { + return AdvisoryOutcome::StateUnavailable; + }; + let (decision, pending) = plan_advisory(&unlocked, now()); + if pending.is_none() && decision == AdvisoryDecision::NoAction { + return AdvisoryOutcome::NoAction; + } + + let lock = match acquire_inner(&git_dir, Duration::ZERO, || {}) { + Ok(lock) => lock, + Err(WorktreeLockError::TimedOut { .. }) => return AdvisoryOutcome::Busy, + Err(_) => return AdvisoryOutcome::StateUnavailable, + }; + + let Ok(locked) = read_state(&path) else { + return AdvisoryOutcome::StateUnavailable; + }; + let now_ms = now(); + let (decision, pending) = plan_advisory(&locked, now_ms); + let outcome = match decision { + AdvisoryDecision::Anchored => AdvisoryOutcome::Anchored, + AdvisoryDecision::NoAction => AdvisoryOutcome::NoAction, + AdvisoryDecision::Advised => AdvisoryOutcome::Advised, + }; + + let result = match pending { + Some(next) => match write_state(&path, &next) { + Ok(()) => outcome, + Err(_) => AdvisoryOutcome::StateUnavailable, + }, + None => outcome, + }; + drop(lock); + result +} + +#[cfg(test)] +mod tests { + use std::path::Path; + use std::process::Command; + + use super::super::maintenance_state::{ + evaluate_recommendation, read_state, state_path, write_state_atomically, MaintenanceState, + StateRead, FUTURE_SKEW_TOLERANCE_MS, RECONCILIATION_ADVISORY_AFTER_MS, + }; + use super::super::worktree_lock::acquire_inner; + use super::{advise_if_due_with, AdvisoryOutcome}; + + const NOW: i64 = 1_000_000_000_000; + + fn init_repo(root: &Path) { + std::fs::create_dir_all(root).expect("repo dir"); + let output = Command::new("git") + .args(["init", "-q"]) + .current_dir(root) + .output() + .expect("git init"); + assert!(output.status.success()); + } + + fn real_write(path: &Path, state: &MaintenanceState) -> std::io::Result<()> { + write_state_atomically(path, state, |from, to| std::fs::rename(from, to)) + } + + fn advise(root: &Path, now: i64) -> AdvisoryOutcome { + advise_if_due_with(root, || now, real_write) + } + + fn stored(root: &Path) -> MaintenanceState { + match read_state(&state_path(&root.join(".git"))).expect("read state") { + StateRead::Valid(state) => state, + other => panic!("expected valid state, got {other:?}"), + } + } + + fn seed(root: &Path, state: &MaintenanceState) { + let path = state_path(&root.join(".git")); + std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); + real_write(&path, state).expect("seed state"); + } + + #[test] + fn maintenance_advisory_anchors_once_then_leaves_state_untouched() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + + assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored); + let path = state_path(&dir.path().join(".git")); + let first = std::fs::read(&path).expect("state bytes"); + assert_eq!(stored(dir.path()).anchor, Some(NOW)); + + assert_eq!(advise(dir.path(), NOW + 1000), AdvisoryOutcome::NoAction); + assert_eq!(std::fs::read(&path).expect("state bytes"), first); + } + + #[test] + fn maintenance_advisory_advises_once_per_window_and_persists_before_returning() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + advise(dir.path(), NOW); + + let due = NOW + RECONCILIATION_ADVISORY_AFTER_MS; + assert_eq!(advise(dir.path(), due), AdvisoryOutcome::Advised); + assert_eq!(stored(dir.path()).last_advised, Some(due)); + assert_eq!(advise(dir.path(), due + 1000), AdvisoryOutcome::NoAction); + assert_eq!( + advise(dir.path(), due + RECONCILIATION_ADVISORY_AFTER_MS), + AdvisoryOutcome::Advised + ); + } + + #[test] + fn maintenance_advisory_preserves_timestamps_within_skew_tolerance() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let near_future = MaintenanceState { + last_success: Some(NOW + FUTURE_SKEW_TOLERANCE_MS), + ..MaintenanceState::default() + }; + seed(dir.path(), &near_future); + let path = state_path(&dir.path().join(".git")); + let before = std::fs::read(&path).expect("state bytes"); + + assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + } + + #[test] + fn maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let invalid = MaintenanceState { + anchor: Some(NOW + FUTURE_SKEW_TOLERANCE_MS + 1), + last_success: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + ..MaintenanceState::default() + }; + seed(dir.path(), &invalid); + + assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Advised); + let normalized = stored(dir.path()); + assert_eq!(normalized.last_advised, Some(NOW)); + assert_eq!(normalized.anchor, Some(NOW)); + assert_eq!(normalized.last_success, None); + + let path = state_path(&dir.path().join(".git")); + let bytes = std::fs::read(&path).expect("state bytes"); + assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction); + assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); + } + + #[test] + fn maintenance_advisory_reports_busy_and_leaves_invalid_state_byte_identical() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let invalid = MaintenanceState { + anchor: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + ..MaintenanceState::default() + }; + seed(dir.path(), &invalid); + let path = state_path(&dir.path().join(".git")); + let before = std::fs::read(&path).expect("state bytes"); + + let held = acquire_inner(&dir.path().join(".git"), std::time::Duration::ZERO, || {}) + .expect("hold lock"); + assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Busy); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + drop(held); + + let recommendation = evaluate_recommendation(&invalid, NOW); + assert!(recommendation.recommended); + assert!(recommendation.invalid_timestamps); + } + + #[test] + fn maintenance_advisory_write_failure_emits_no_advice_and_keeps_previous_state() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + advise(dir.path(), NOW); + let path = state_path(&dir.path().join(".git")); + let before = std::fs::read(&path).expect("state bytes"); + + let due = NOW + RECONCILIATION_ADVISORY_AFTER_MS; + let outcome = advise_if_due_with( + dir.path(), + || due, + |path, state| { + write_state_atomically(path, state, |_, _| { + Err(std::io::Error::other("rename interrupted")) + }) + }, + ); + assert_eq!(outcome, AdvisoryOutcome::StateUnavailable); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + assert_eq!(advise(dir.path(), due), AdvisoryOutcome::Advised); + } + + #[test] + fn maintenance_advisory_recovers_corrupt_state_by_anchoring() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let path = state_path(&dir.path().join(".git")); + std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); + std::fs::write(&path, b"not json").expect("corrupt state"); + + assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored); + assert_eq!(stored(dir.path()).anchor, Some(NOW)); + } + + #[test] + fn maintenance_advisory_derives_linked_worktree_git_dir_from_subdirectory() { + let dir = tempfile::tempdir().expect("temp dir"); + let main = dir.path().join("main"); + init_repo(&main); + for args in [ + vec!["config", "user.email", "t@example.invalid"], + vec!["config", "user.name", "T"], + vec!["commit", "-q", "--allow-empty", "-m", "init"], + vec!["worktree", "add", "-q", "../linked"], + ] { + let output = Command::new("git") + .args(&args) + .current_dir(&main) + .output() + .expect("git"); + assert!(output.status.success(), "{args:?}"); + } + let linked = dir.path().join("linked"); + let sub = linked.join("nested"); + std::fs::create_dir_all(&sub).expect("subdir"); + + let expected = Command::new("git") + .args(["rev-parse", "--absolute-git-dir"]) + .current_dir(&sub) + .output() + .expect("rev-parse"); + let expected = std::fs::canonicalize(String::from_utf8_lossy(&expected.stdout).trim()) + .expect("canonical"); + + assert_eq!(super::derive_git_dir(&sub), Some(expected.clone())); + assert_eq!(advise(&sub, NOW), AdvisoryOutcome::Anchored); + assert!(state_path(&expected).is_file()); + } +} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance.rs new file mode 100644 index 000000000..c7be29939 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance.rs @@ -0,0 +1,174 @@ +#![allow( + dead_code, + reason = "maintenance entrypoints are wired by later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" +)] + +use std::path::Path; +use std::time::Duration; + +use anyhow::Result; + +use super::git_snapshot::resolve_git_dir; +use super::maintenance_state::{ + read_state, record_failure, record_success, state_path, system_now_ms, write_state_atomically, + MaintenanceState, StateRead, StoredReport, +}; +use super::ref_reconciliation::{ + reconcile_with_held_lock, ReconcileError, ReconcilePhase, ReconciliationOutcome, + ReconciliationReport, +}; +use super::worktree_lock::{acquire_inner_async, WorktreeLockError}; +use crate::services::agent_trace_db::repository::RepositoryAgentTraceDb; + +const EXPLICIT_LOCK_TIMEOUT: Duration = Duration::from_secs(10); + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum SkipReason { + Busy, +} + +#[derive(Debug)] +pub(super) enum ExplicitOutcome { + Completed(ReconciliationReport), + CompletedStatePersistFailed { + report: ReconciliationReport, + warning: String, + }, + Failed(ReconcileError), + Skipped(SkipReason), +} + +pub(super) async fn reconcile_explicit

(repository_root: &Path, open_db: P) -> ExplicitOutcome +where + P: std::ops::AsyncFnOnce() -> Result, +{ + reconcile_explicit_with( + repository_root, + open_db, + system_now_ms, + |path, state| write_state_atomically(path, state, |from, to| std::fs::rename(from, to)), + |_| {}, + EXPLICIT_LOCK_TIMEOUT, + ) + .await +} + +pub(super) async fn reconcile_explicit_with( + repository_root: &Path, + open_db: P, + now: C, + write_state: W, + on_phase: H, + lock_timeout: Duration, +) -> ExplicitOutcome +where + P: std::ops::AsyncFnOnce() -> Result, + C: Fn() -> i64, + W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, + H: FnMut(ReconcilePhase), +{ + let git_dir = match resolve_git_dir(repository_root).await { + Ok(git_dir) => git_dir, + Err(source) => return ExplicitOutcome::Failed(ReconcileError::GitDir(source)), + }; + + let lock = match acquire_inner_async(&git_dir, lock_timeout, || {}).await { + Ok(lock) => lock, + Err(WorktreeLockError::TimedOut { .. }) => { + return ExplicitOutcome::Skipped(SkipReason::Busy); + } + Err(error) => return ExplicitOutcome::Failed(ReconcileError::Lock(error)), + }; + + let result = reconcile_with_held_lock(repository_root, &lock, open_db, on_phase).await; + let path = state_path(&git_dir); + let now_ms = now(); + let previous = read_state(&path); + + let outcome = match result { + Ok(ReconciliationOutcome::Reconciled(report)) => { + let next = record_success( + &previous_state(&previous), + now_ms, + StoredReport { + retained: report.retained, + deleted: report.deleted, + local_required: report.local_required, + }, + ); + match persist(&previous, &write_state, &path, &next) { + Ok(()) => ExplicitOutcome::Completed(report), + Err(warning) => ExplicitOutcome::CompletedStatePersistFailed { report, warning }, + } + } + Ok(ReconciliationOutcome::SkippedNoCheckoutIdentity) => { + let error = ReconcileError::CheckoutIdentity(anyhow::anyhow!( + "no checkout identity could be derived" + )); + record_failed_pass(&previous, &write_state, &path, now_ms, &error); + ExplicitOutcome::Failed(error) + } + Err(error) => { + record_failed_pass(&previous, &write_state, &path, now_ms, &error); + ExplicitOutcome::Failed(error) + } + }; + + drop(lock); + outcome +} + +fn previous_state(previous: &std::io::Result) -> MaintenanceState { + match previous { + Ok(read) => read.clone().into_state(), + Err(_) => MaintenanceState::default(), + } +} + +fn persist( + previous: &std::io::Result, + write_state: &W, + path: &Path, + next: &MaintenanceState, +) -> std::result::Result<(), String> +where + W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, +{ + if let Err(error) = previous { + return Err(format!("maintenance state could not be read: {error}")); + } + write_state(path, next).map_err(|error| format!("maintenance state was not recorded: {error}")) +} + +fn record_failed_pass( + previous: &std::io::Result, + write_state: &W, + path: &Path, + now_ms: i64, + error: &ReconcileError, +) where + W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, +{ + let next = record_failure( + &previous_state(previous), + now_ms, + failure_kind(error), + &error.to_string(), + ); + let _ = persist(previous, write_state, path, &next); +} + +fn failure_kind(error: &ReconcileError) -> &'static str { + match error { + ReconcileError::GitDir(_) => "git_dir", + ReconcileError::Lock(_) => "lock", + ReconcileError::CheckoutIdentity(_) => "checkout_identity", + ReconcileError::AgentTraceDbUnavailable(_) => "agent_trace_db_unavailable", + ReconcileError::SnapshotService(_) => "snapshot_service", + ReconcileError::PinInventory(_) => "pin_inventory", + ReconcileError::MalformedPin { .. } => "malformed_pin", + ReconcileError::DurableRoots(_) => "durable_roots", + ReconcileError::MissingRequiredPins { .. } => "missing_required_pins", + ReconcileError::DeleteTransaction(_) => "delete_transaction", + } +} diff --git a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs index 10f04bfa9..03a5e8639 100644 --- a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs +++ b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs @@ -10,7 +10,7 @@ use crate::services::mutation_trace::store::MutationTraceStore; use crate::services::mutation_trace::types::TreeId; use super::git_snapshot::{resolve_git_dir, GitSnapshotService, PinInventoryError, PinnedRef}; -use super::worktree_lock::{acquire_inner_async, WorktreeLockError}; +use super::worktree_lock::{acquire_inner_async, WorktreeLock, WorktreeLockError}; #[allow( dead_code, @@ -18,23 +18,6 @@ use super::worktree_lock::{acquire_inner_async, WorktreeLockError}; )] const RECONCILIATION_LOCK_TIMEOUT: Duration = Duration::from_secs(10); -/// Outcome counts of one successful reconciliation pass. -/// -/// - `local_required` — the target worktree's own durable-root count -/// (`load_tree_roots(W).len()`), the left side of the local consistency -/// invariant. -/// - `retained` — `actual_W.len() - deleted`: pins left in place, whether -/// because the target worktree still needs their tree or because another -/// worktree in the repository durably does. -/// - `deleted` — pins actually removed (inventoried under `W`'s prefix, tree -/// absent from the repository-wide durable root set). -/// -/// `retained == local_required` is **not** an invariant — a pin retained only -/// because another worktree durably needs its tree counts toward `retained` -/// but not `local_required`. For `ReconciliationOutcome::Reconciled(report)` -/// the only relation that holds is `report.local_required <= report.retained`. -/// `ReconciliationOutcome::SkippedNoCheckoutIdentity` carries no report, so no -/// report invariant applies to it. #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[allow( dead_code, @@ -46,9 +29,6 @@ pub struct ReconciliationReport { pub deleted: usize, } -/// Outcome of one reconciliation pass: a real pass that ran, carrying its -/// [`ReconciliationReport`], versus a skip because no current checkout identity -/// could be derived (an `Ok`, never an `Err`). #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[allow( dead_code, @@ -59,45 +39,21 @@ pub enum ReconciliationOutcome { SkippedNoCheckoutIdentity, } -/// Why a reconciliation pass could not complete. One variant per fallible step, -/// no `Other` catch-all — mirroring `CoordinateError`'s convention. Every -/// non-`Ok` outcome leaves the SCE ref namespace in a consistent state: either -/// untouched, or (only on `Ok`) with exactly the stale refs gone. #[derive(Debug)] #[allow( dead_code, reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" )] pub enum ReconcileError { - /// `resolve_git_dir` failed. GitDir(anyhow::Error), - /// The worktree's `WorktreeLock` could not be acquired (timeout or I/O). Lock(WorktreeLockError), - /// Git topology could not be resolved into a worktree identity. CheckoutIdentity(anyhow::Error), - /// The caller-supplied `open_db` provider returned `Err`. This is a - /// reconciliation maintenance error only: it never arms - /// `ExternalTaintMarker` and never becomes - /// `CoordinateError::AgentTraceDbUnavailable`, because no mutation boundary - /// is being coordinated. AgentTraceDbUnavailable(anyhow::Error), - /// `GitSnapshotService::new` failed. SnapshotService(anyhow::Error), - /// `git for-each-ref` itself failed to execute or exited non-zero - /// (`PinInventoryError::Git`). PinInventory(anyhow::Error), - /// A ref inside the SCE mutation-cursor namespace is not shaped like a - /// `pin_tree` output (`PinInventoryError::MalformedRef`). Reconciliation - /// deletes nothing. MalformedPin { ref_name: String, reason: String }, - /// `load_tree_roots` / `load_all_tree_roots` failed (DB query error, - /// migration `003` absent, ...). DurableRoots(anyhow::Error), - /// A durable root of the **target** worktree has no live pin — the local - /// consistency invariant is violated. Fail closed: nothing is deleted. MissingRequiredPins { missing: Vec }, - /// The atomic `delete_pins` transaction failed (including a ref that - /// changed since inventory). Nothing is deleted. DeleteTransaction(anyhow::Error), } @@ -131,13 +87,6 @@ impl std::fmt::Display for ReconcileError { impl std::error::Error for ReconcileError {} -/// Reconcile one worktree's SCE snapshot pins: remove orphan / unreferenced -/// pins while retaining every tree any current or historical durable -/// mutation-cursor state in the repository still references. -/// -/// Module-private to `runtime`, exactly like `coordinate` — never re-exported -/// outside mutation-trace `runtime`. It is a one-line delegation to -/// [`reconcile_worktree_inner`] with a no-op lock-contention closure. #[allow( dead_code, reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" @@ -173,6 +122,25 @@ where .await .map_err(ReconcileError::Lock)?; + reconcile_with_held_lock(repository_root, &lock, open_db, |_| {}).await +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum ReconcilePhase { + DbOpened, + PinsInventoried, +} + +pub(super) async fn reconcile_with_held_lock( + repository_root: &Path, + lock: &WorktreeLock, + open_db: P, + mut on_phase: H, +) -> std::result::Result +where + P: std::ops::AsyncFnOnce() -> Result, + H: FnMut(ReconcilePhase), +{ let worktree_id = resolve_worktree_id(repository_root) .await .map_err(ReconcileError::CheckoutIdentity)?; @@ -180,13 +148,12 @@ where let mut db = open_db() .await .map_err(ReconcileError::AgentTraceDbUnavailable)?; + on_phase(ReconcilePhase::DbOpened); let snapshot = GitSnapshotService::new(repository_root) .await .map_err(ReconcileError::SnapshotService)?; - // Inventory the worktree's pins first, so every durable-root read that - // follows is compared against a fixed observation of the namespace. let actual = snapshot .list_pins(&worktree_id) .await @@ -196,13 +163,11 @@ where ReconcileError::MalformedPin { ref_name, reason } } })?; + on_phase(ReconcilePhase::PinsInventoried); let pinned_trees: BTreeSet = actual.iter().map(|pin| pin.tree.clone()).collect(); let store = MutationTraceStore::new(&mut db); - // Local consistency invariant (a strictly per-worktree check): every tree - // the target worktree's own durable evidence references must still have a - // live pin, or the pass fails closed and deletes nothing. let required_local = store .load_tree_roots(&worktree_id) .await @@ -214,10 +179,6 @@ where }); } - // Deletion safety invariant: an owned ref is removed only when its tree is - // outside the durable root set of **every** worktree in the repository — - // linked worktrees share one object database, so an A-owned ref may be the - // last SCE ref protecting a tree that only worktree B durably requires. let required_repository = store .load_all_tree_roots() .await @@ -241,3 +202,297 @@ where deleted: stale.len(), })) } + +#[cfg(test)] +mod tests { + use std::path::Path; + use std::process::Command; + use std::time::Duration; + + use super::super::git_snapshot::{resolve_git_dir, GitSnapshotService}; + use super::super::maintenance_state::{ + read_state, state_path, write_state_atomically, StateRead, + }; + use super::super::ref_maintenance::{reconcile_explicit_with, ExplicitOutcome, SkipReason}; + use super::super::worktree_lock::WorktreeLock; + use super::{ReconcileError, ReconciliationReport}; + use crate::services::agent_trace_db::repository::{ + ExistingRepositoryDbError, RepositoryAgentTraceDb, + }; + use crate::services::mutation_trace::types::WorktreeId; + + const NOW: i64 = 1_000_000_000_000; + const REPOSITORY_ID: &str = "repo-under-test"; + + fn git(cwd: &Path, args: &[&str]) { + let output = Command::new("git") + .args(args) + .current_dir(cwd) + .output() + .expect("git command"); + assert!(output.status.success(), "git {args:?} failed"); + } + + fn init_repo(root: &Path) { + std::fs::create_dir_all(root).expect("repo dir"); + git(root, &["init", "-q"]); + git(root, &["config", "user.email", "t@example.invalid"]); + git(root, &["config", "user.name", "T"]); + git(root, &["config", "commit.gpgsign", "false"]); + std::fs::write(root.join("tracked.txt"), "tracked\n").expect("write file"); + git(root, &["add", "tracked.txt"]); + git(root, &["commit", "-q", "-m", "initial"]); + } + + async fn create_db(path: &Path, repository_id: &str) { + let db = RepositoryAgentTraceDb::new_at(path) + .await + .expect("create db"); + db.verify_or_initialize_repository_metadata(repository_id) + .await + .expect("initialize metadata"); + } + + async fn pin_orphan(root: &Path) -> String { + let snapshot = GitSnapshotService::new(root).await.expect("snapshot"); + let tree = snapshot.capture_tree().await.expect("tree"); + let git_dir = resolve_git_dir(root).await.expect("git dir"); + let lock = WorktreeLock::acquire_async(&git_dir, Duration::from_secs(10)) + .await + .expect("lock"); + snapshot + .pin_tree(lock.lease(), &WorktreeId("main".to_string()), &tree) + .await + .expect("pin"); + tree.0 + } + + fn write( + path: &Path, + state: &super::super::maintenance_state::MaintenanceState, + ) -> std::io::Result<()> { + write_state_atomically(path, state, |from, to| std::fs::rename(from, to)) + } + + #[tokio::test(flavor = "multi_thread")] + async fn ref_reconciliation_explicit_pass_deletes_orphan_and_records_success() { + let dir = tempfile::tempdir().expect("temp dir"); + let root = dir.path().join("repo"); + init_repo(&root); + let db_path = dir.path().join("state").join("agent-trace.db"); + std::fs::create_dir_all(db_path.parent().expect("parent")).expect("db dir"); + create_db(&db_path, REPOSITORY_ID).await; + pin_orphan(&root).await; + + let mut phases = Vec::new(); + let outcome = reconcile_explicit_with( + &root, + async || { + RepositoryAgentTraceDb::open_verified_existing_at(&db_path, REPOSITORY_ID) + .await + .map(|(db, _)| db) + .map_err(anyhow::Error::new) + }, + || NOW, + write, + |phase| phases.push(phase), + Duration::from_secs(10), + ) + .await; + + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!( + report, + ReconciliationReport { + local_required: 0, + retained: 0, + deleted: 1, + } + ); + assert_eq!(phases.len(), 2); + + let snapshot = GitSnapshotService::new(&root).await.expect("snapshot"); + assert!(snapshot + .list_pins(&WorktreeId("main".to_string())) + .await + .expect("pins") + .is_empty()); + + let git_dir = resolve_git_dir(&root).await.expect("git dir"); + let StateRead::Valid(state) = read_state(&state_path(&git_dir)).expect("state") else { + panic!("expected recorded state"); + }; + assert_eq!(state.last_success, Some(NOW)); + assert_eq!(state.consecutive_failures, 0); + } + + #[tokio::test(flavor = "multi_thread")] + async fn ref_reconciliation_explicit_pass_failure_keeps_refs_and_counts_streak() { + let dir = tempfile::tempdir().expect("temp dir"); + let root = dir.path().join("repo"); + init_repo(&root); + let missing_db = dir.path().join("absent").join("agent-trace.db"); + let tree = pin_orphan(&root).await; + + for expected_streak in 1..=2 { + let outcome = reconcile_explicit_with( + &root, + async || { + RepositoryAgentTraceDb::open_verified_existing_at(&missing_db, REPOSITORY_ID) + .await + .map(|(db, _)| db) + .map_err(anyhow::Error::new) + }, + || NOW, + write, + |_| {}, + Duration::from_secs(10), + ) + .await; + assert!(matches!( + outcome, + ExplicitOutcome::Failed(ReconcileError::AgentTraceDbUnavailable(_)) + )); + + let git_dir = resolve_git_dir(&root).await.expect("git dir"); + let StateRead::Valid(state) = read_state(&state_path(&git_dir)).expect("state") else { + panic!("expected recorded state"); + }; + assert_eq!(state.consecutive_failures, expected_streak); + assert_eq!(state.last_success, None); + } + + let snapshot = GitSnapshotService::new(&root).await.expect("snapshot"); + let pins = snapshot + .list_pins(&WorktreeId("main".to_string())) + .await + .expect("pins"); + assert_eq!(pins.len(), 1); + assert_eq!(pins[0].tree.0, tree); + assert!(!missing_db.exists()); + assert!(!missing_db.parent().expect("parent").exists()); + } + + #[tokio::test(flavor = "multi_thread")] + async fn ref_reconciliation_explicit_pass_skips_when_lock_is_held_without_touching_state() { + let dir = tempfile::tempdir().expect("temp dir"); + let root = dir.path().join("repo"); + init_repo(&root); + let git_dir = resolve_git_dir(&root).await.expect("git dir"); + let held = WorktreeLock::acquire_async(&git_dir, Duration::from_secs(10)) + .await + .expect("hold lock"); + + let outcome = reconcile_explicit_with( + &root, + async || -> anyhow::Result { unreachable!("db must not open") }, + || NOW, + write, + |_| {}, + Duration::ZERO, + ) + .await; + + assert!(matches!( + outcome, + ExplicitOutcome::Skipped(SkipReason::Busy) + )); + assert!(!state_path(&git_dir).exists()); + drop(held); + } + + #[tokio::test(flavor = "multi_thread")] + async fn ref_reconciliation_explicit_pass_reports_state_persist_failure_distinctly() { + let dir = tempfile::tempdir().expect("temp dir"); + let root = dir.path().join("repo"); + init_repo(&root); + let db_path = dir.path().join("agent-trace.db"); + create_db(&db_path, REPOSITORY_ID).await; + pin_orphan(&root).await; + + let outcome = reconcile_explicit_with( + &root, + async || { + RepositoryAgentTraceDb::open_verified_existing_at(&db_path, REPOSITORY_ID) + .await + .map(|(db, _)| db) + .map_err(anyhow::Error::new) + }, + || NOW, + |_, _| Err(std::io::Error::other("disk full")), + |_| {}, + Duration::from_secs(10), + ) + .await; + + let ExplicitOutcome::CompletedStatePersistFailed { report, warning } = outcome else { + panic!("expected persist-failed completion, got {outcome:?}"); + }; + assert_eq!(report.deleted, 1); + assert!(warning.contains("disk full")); + } + + #[tokio::test(flavor = "multi_thread")] + async fn ref_reconciliation_verified_opener_reports_typed_failures_without_writes() { + let dir = tempfile::tempdir().expect("temp dir"); + + let missing = dir.path().join("nested").join("agent-trace.db"); + let error = RepositoryAgentTraceDb::open_verified_existing_at(&missing, REPOSITORY_ID) + .await + .err() + .expect("missing db fails"); + assert!(matches!(error, ExistingRepositoryDbError::Missing { .. })); + assert!(!dir.path().join("nested").exists()); + + let valid = dir.path().join("valid.db"); + create_db(&valid, REPOSITORY_ID).await; + let (_, metadata) = + RepositoryAgentTraceDb::open_verified_existing_at(&valid, REPOSITORY_ID) + .await + .expect("valid db opens"); + assert_eq!(metadata.repository_id, REPOSITORY_ID); + + let error = RepositoryAgentTraceDb::open_verified_existing_at(&valid, "another-repo") + .await + .err() + .expect("mismatch fails"); + assert!(matches!( + error, + ExistingRepositoryDbError::RepositoryMismatch { .. } + )); + + let no_metadata = dir.path().join("no-metadata.db"); + RepositoryAgentTraceDb::new_at(&no_metadata) + .await + .expect("migrated db"); + let error = RepositoryAgentTraceDb::open_verified_existing_at(&no_metadata, REPOSITORY_ID) + .await + .err() + .expect("missing metadata fails"); + assert!(matches!(error, ExistingRepositoryDbError::MissingMetadata)); + let db = RepositoryAgentTraceDb::open_existing_without_migrations_at(&no_metadata) + .await + .expect("reopen"); + assert!(db + .verify_existing_repository_metadata(REPOSITORY_ID) + .await + .is_err()); + + let unmigrated = dir.path().join("unmigrated.db"); + drop( + RepositoryAgentTraceDb::open_without_migrations_at(&unmigrated) + .await + .expect("create empty db"), + ); + let error = RepositoryAgentTraceDb::open_verified_existing_at(&unmigrated, REPOSITORY_ID) + .await + .err() + .expect("incompatible schema fails"); + assert!(matches!( + error, + ExistingRepositoryDbError::IncompatibleSchema(_) + )); + } +} diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 092fc9b82..677f0ad5b 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -267,7 +267,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - Context impact: none (plan-only audit; no behavior or durable context change). - Context synchronization: synced -- [ ] T02: `Add the reusable maintenance entrypoints (explicit reconciliation and hook advisory)` (status:todo) +- [x] T02: `Add the reusable maintenance entrypoints (explicit reconciliation and hook advisory)` (status:done) - Task ID: T02 - Scope: In — - **Reconciler split (D1):** extract the body of `reconcile_worktree_inner` after lock acquisition into `reconcile_with_held_lock`, which borrows the held `WorktreeLock` and contains no acquire call; keep `reconcile_worktree` and `reconcile_worktree_inner` behavior and signatures, now acquiring once and delegating. @@ -280,7 +280,12 @@ Record actual outcomes in the validation report; compilation alone does not vali - Dependencies: T01 - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a deadline; after every successful state write no stored timestamp exceeds `now + FUTURE_SKEW_TOLERANCE`; timestamps within the tolerance are accepted as age 0 and preserved; on a successful write invalid timestamps normalize once and persist; read-only, lock-busy, and failed-read/write operations never mutate state, evaluate invalid timestamps conservatively, and never postpone eligibility indefinitely; failures leave refs untouched; the DB opener fails a missing DB before Turso is invoked without creating files or directories, and for an existing invalid DB performs no migration, repair, initialization, or application-table write (incidental WAL bookkeeping permitted and documented); no second deletion path exists. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml ref_reconciliation`; focused maintenance-state and `agent_trace_storage` unit tests (deterministic, no sleeps); `rg` confirms the only ref-deletion call is the existing `delete_pins`, each entrypoint acquires `WorktreeLock` once, `reconcile_with_held_lock` contains no acquire call, and the advisory module imports no Git/DB/store symbol. - - Context synchronization: pending + - Completed: 2026-10-09 + - Files changed: `cli/src/services/agent_trace_db/repository.rs`, `cli/src/services/agent_trace_storage/mod.rs`, `cli/src/services/db/mod.rs`, `cli/src/services/mutation_trace/runtime/mod.rs`, `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs`, `cli/src/services/mutation_trace/runtime/maintenance_state.rs`, `cli/src/services/mutation_trace/runtime/ref_advisory.rs`, `cli/src/services/mutation_trace/runtime/ref_maintenance.rs`, `context/plans/mutation-cursor-ref-reconciliation-wiring.md` + - Result: Split `reconcile_worktree_inner` into lock acquisition plus `reconcile_with_held_lock` (borrows `&WorktreeLock`, phase-hook seam `ReconcilePhase`); added `ref_maintenance::reconcile_explicit[_with]` (single lock acquisition, typed `Completed | CompletedStatePersistFailed | Failed | Skipped(Busy)`), `ref_advisory::advise_if_due[_with]` (sync, zero-wait `try_lock`, filesystem Git-dir derivation, no Git/DB/store imports), and `maintenance_state` (capped versioned `ref-maintenance.json`, past-only timestamps, tolerance-based normalization under lock, staging-and-swap writes with injectable rename, pure `evaluate_recommendation`). Added the D4 opener `RepositoryAgentTraceDb::open_verified_existing_at` with `ExistingRepositoryDbError` (`Missing | Unreadable | IncompatibleSchema | MissingMetadata | RepositoryMismatch`), read-only `verify_existing_repository_metadata`, `TursoDb::open_existing_without_migrations_at` (no parent-dir creation), and `resolve_existing_agent_trace_storage_for_maintenance`. New items carry dead-code allowances until T03/T04 wire them. The existing hook opener and its callers are unchanged; `delete_pins` remains the only deletion path. + - Verify: `cargo clippy --all-targets --all-features -D warnings` clean; `test … ref_reconciliation` 5 passed; `test … maintenance` 8 passed; `test … mutation_trace` 123 passed; `rg` shows `reconcile_with_held_lock` has no acquire call and `maintenance_state.rs` has no Git/DB/store symbol (`ref_advisory.rs` matches only `Command` inside its test module); `git diff --check` clean. The `agent_trace_storage` filter matches zero tests, so opener coverage lives in the `ref_reconciliation` tests. Turso 0.8.1 read-only open was not used: the opener pre-checks file existence and only issues SELECT-class statements after open (incidental WAL bookkeeping accepted per D4). + - Context impact: important (new runtime maintenance entrypoints, state file, and verified-existing-DB opener; T06 owns the durable context update). + - Context synchronization: synced - [ ] T03: `Trigger the advisory check from the shared mutation-scope ingress` (status:todo) - Task ID: T03 From 1a004473bfd6d24f126b697ab13a191ae7500c46 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 01:08:46 +0200 Subject: [PATCH 04/16] runtime: Harden ref maintenance persistence and DB opening Ensure explicit reconciliation uses the production-resolved, verified read-only database and reports state-persistence diagnostics without losing reconciliation errors. Distinguish pre-rename failures from post-rename durability uncertainty, require confirmed writes for advisory results, and add deterministic coverage for open races, WAL coexistence, lock ownership, and persistence failures. Plan: mutation-cursor-ref-reconciliation-wiring (T02) Co-authored-by: SCE --- cli/src/services/agent_trace_db/repository.rs | 44 +- cli/src/services/agent_trace_storage/mod.rs | 13 + cli/src/services/db/mod.rs | 20 +- .../runtime/maintenance_state.rs | 444 +++++++++- .../mutation_trace/runtime/ref_advisory.rs | 356 +++++++- .../mutation_trace/runtime/ref_maintenance.rs | 126 ++- .../runtime/ref_maintenance/tests.rs | 816 ++++++++++++++++++ .../runtime/ref_reconciliation.rs | 294 ------- ...tation-cursor-ref-reconciliation-wiring.md | 29 +- 9 files changed, 1719 insertions(+), 423 deletions(-) create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs diff --git a/cli/src/services/agent_trace_db/repository.rs b/cli/src/services/agent_trace_db/repository.rs index 3c606757c..3e9b35cfd 100644 --- a/cli/src/services/agent_trace_db/repository.rs +++ b/cli/src/services/agent_trace_db/repository.rs @@ -175,16 +175,42 @@ impl RepositoryAgentTraceDb { path: impl AsRef, repository_id: &str, ) -> std::result::Result<(Self, RepositoryMetadata), ExistingRepositoryDbError> { - let path = path.as_ref(); - if !path.is_file() { - return Err(ExistingRepositoryDbError::Missing { - path: path.to_path_buf(), - }); - } + Self::open_verified_existing_with(path.as_ref(), repository_id, || {}, || {}).await + } - let db = TursoDb::::open_existing_without_migrations_at(path) - .await - .map_err(ExistingRepositoryDbError::Unreadable)?; + #[cfg(test)] + pub(crate) async fn open_verified_existing_at_with_hooks( + path: &std::path::Path, + repository_id: &str, + before_open: impl FnOnce(), + after_open: impl FnOnce(), + ) -> std::result::Result<(Self, RepositoryMetadata), ExistingRepositoryDbError> { + Self::open_verified_existing_with(path, repository_id, before_open, after_open).await + } + + async fn open_verified_existing_with( + path: &std::path::Path, + repository_id: &str, + before_open: impl FnOnce(), + after_open: impl FnOnce(), + ) -> std::result::Result<(Self, RepositoryMetadata), ExistingRepositoryDbError> { + before_open(); + let opened = + TursoDb::::open_existing_without_migrations_at(path).await; + after_open(); + let db = match opened { + Ok(db) => db, + Err(source) => { + return Err(match std::fs::symlink_metadata(path) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + ExistingRepositoryDbError::Missing { + path: path.to_path_buf(), + } + } + Ok(_) | Err(_) => ExistingRepositoryDbError::Unreadable(source), + }); + } + }; db.ensure_schema_ready_for_hooks() .await .map_err(ExistingRepositoryDbError::IncompatibleSchema)?; diff --git a/cli/src/services/agent_trace_storage/mod.rs b/cli/src/services/agent_trace_storage/mod.rs index 52d38c7a8..79e0cc313 100644 --- a/cli/src/services/agent_trace_storage/mod.rs +++ b/cli/src/services/agent_trace_storage/mod.rs @@ -240,3 +240,16 @@ async fn open_existing_storage_for_maintenance( metadata, }) } + +#[cfg(test)] +pub(crate) async fn resolve_existing_agent_trace_storage_for_maintenance_at_state_root( + context: &AgentTraceStorageContext<'_>, + state_root: &Path, +) -> Result { + let repository_identity = resolve_identity(context)?; + let db_path = agent_trace_db_path_for_repository_at( + state_root, + &repository_identity.identity.repository_id, + )?; + open_existing_storage_for_maintenance(repository_identity, db_path).await +} diff --git a/cli/src/services/db/mod.rs b/cli/src/services/db/mod.rs index 4e093f92e..21ea96681 100644 --- a/cli/src/services/db/mod.rs +++ b/cli/src/services/db/mod.rs @@ -157,6 +157,12 @@ fn sentence_case(value: &str) -> String { first.to_uppercase().collect::() + chars.as_str() } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum OpenMode { + CreateIfMissing, + ExistingReadOnly, +} + fn ensure_db_parent_dir(db_name: &str, db_path: &Path) -> Result<()> { if let Some(parent) = db_path.parent() { std::fs::create_dir_all(parent).with_context(|| { @@ -841,24 +847,21 @@ impl TursoDb { } pub async fn open_without_migrations_at(db_path: impl AsRef) -> Result { - Self::open_at_path(db_path.as_ref(), true).await + Self::open_at_path(db_path.as_ref(), OpenMode::CreateIfMissing).await } - #[allow( - dead_code, - reason = "maintenance wiring lands in later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" - )] pub async fn open_existing_without_migrations_at(db_path: impl AsRef) -> Result { - Self::open_at_path(db_path.as_ref(), false).await + Self::open_at_path(db_path.as_ref(), OpenMode::ExistingReadOnly).await } - async fn open_at_path(db_path: &Path, create_parent_dir: bool) -> Result { + async fn open_at_path(db_path: &Path, mode: OpenMode) -> Result { let db_name = M::db_name(); let db_path = db_path.to_path_buf(); - if create_parent_dir { + if mode == OpenMode::CreateIfMissing { ensure_db_parent_dir(db_name, &db_path)?; } + let read_only = mode == OpenMode::ExistingReadOnly; let retry_policy = resolve_connection_open_retry_policy::(); let busy_timeout = resolve_busy_timeout::(); @@ -875,6 +878,7 @@ impl TursoDb { })?; let db = turso::Builder::new_local(path_str) .experimental_multiprocess_wal(true) + .read_only(read_only) .build() .await .map_err(|e| { diff --git a/cli/src/services/mutation_trace/runtime/maintenance_state.rs b/cli/src/services/mutation_trace/runtime/maintenance_state.rs index 9c7ebeb51..0daed18ee 100644 --- a/cli/src/services/mutation_trace/runtime/maintenance_state.rs +++ b/cli/src/services/mutation_trace/runtime/maintenance_state.rs @@ -94,7 +94,7 @@ impl StateRead { #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(super) struct Evaluation { - pub due: bool, + pub recommended: bool, pub advise: bool, pub invalid_timestamps: bool, pub reference_age_ms: Option, @@ -157,6 +157,10 @@ pub(super) fn read_state(path: &Path) -> std::io::Result { }) } +fn failure_outstanding(state: &MaintenanceState) -> bool { + state.last_attempt_outcome == Some(AttemptOutcome::Failed) || state.consecutive_failures > 0 +} + pub(super) fn evaluate(state: &MaintenanceState, now: i64) -> Evaluation { let timestamps = [ state.anchor, @@ -175,12 +179,16 @@ pub(super) fn evaluate(state: &MaintenanceState, now: i64) -> Evaluation { let reference_age_ms = reference.map(|timestamp| age_ms(timestamp, now)); let due = reference_lost || reference_age_ms.is_some_and(|age| age >= RECONCILIATION_ADVISORY_AFTER_MS); - let advised_recently = usable(state.last_advised, now) - .is_some_and(|timestamp| age_ms(timestamp, now) < RECONCILIATION_ADVISORY_AFTER_MS); + let advised = usable(state.last_advised, now) + .filter(|timestamp| age_ms(*timestamp, now) < RECONCILIATION_ADVISORY_AFTER_MS); + let failure_outstanding = failure_outstanding(state); + let failure_reminded = advised.is_some_and(|advised_at| { + usable(state.last_attempt, now).is_none_or(|attempt| advised_at >= attempt) + }); Evaluation { - due, - advise: due && !advised_recently, + recommended: due || failure_outstanding || invalid_timestamps, + advise: (due && advised.is_none()) || (failure_outstanding && !failure_reminded), invalid_timestamps, reference_age_ms, } @@ -191,7 +199,7 @@ pub(super) fn evaluate_recommendation(state: &MaintenanceState, now: i64) -> Rec let age_of = |timestamp: Option| usable(timestamp, now).map(|value| age_ms(value, now)); Recommendation { - recommended: evaluation.due || evaluation.invalid_timestamps, + recommended: evaluation.recommended, invalid_timestamps: evaluation.invalid_timestamps, last_success_age_ms: age_of(state.last_success), last_attempt_age_ms: age_of(state.last_attempt), @@ -280,39 +288,174 @@ pub(super) fn record_failure( next } -pub(super) fn write_state_atomically( +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum PersistPhase { + Serialize, + CreateStaging, + WriteStaging, + SyncStaging, + Rename, + OpenParentDirectory, + SyncParentDirectory, +} + +#[derive(Debug)] +pub(super) enum PersistFailure { + NotApplied { + phase: PersistPhase, + source: std::io::Error, + staging_cleanup: Option, + }, + DurabilityUncertain { + phase: PersistPhase, + source: std::io::Error, + }, +} + +impl std::fmt::Display for PersistFailure { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + PersistFailure::NotApplied { + phase, + source, + staging_cleanup, + } => { + write!( + f, + "maintenance state was not recorded ({phase:?}): {source}; previous state preserved" + )?; + if let Some(cleanup) = staging_cleanup { + write!(f, "; staging file cleanup failed: {cleanup}")?; + } + Ok(()) + } + PersistFailure::DurabilityUncertain { phase, source } => write!( + f, + "maintenance state was renamed into place but crash durability is unconfirmed ({phase:?}): {source}; the new state may be visible" + ), + } + } +} + +impl std::error::Error for PersistFailure {} + +pub(super) trait StateFilesystem { + fn create_staging(&self, directory: &Path) -> std::io::Result; + fn write_staging(&self, staging: &Path, bytes: &[u8]) -> std::io::Result<()>; + fn sync_staging(&self, staging: &Path) -> std::io::Result<()>; + fn rename(&self, from: &Path, to: &Path) -> std::io::Result<()>; + fn open_directory(&self, directory: &Path) -> std::io::Result; + fn sync_directory(&self, directory: &std::fs::File) -> std::io::Result<()>; + fn remove_staging(&self, staging: &Path) -> std::io::Result<()>; +} + +pub(super) struct RealStateFilesystem; + +impl StateFilesystem for RealStateFilesystem { + fn create_staging(&self, directory: &Path) -> std::io::Result { + create_staging_file(directory) + } + + fn write_staging(&self, staging: &Path, bytes: &[u8]) -> std::io::Result<()> { + let mut file = std::fs::OpenOptions::new().write(true).open(staging)?; + file.write_all(bytes) + } + + fn sync_staging(&self, staging: &Path) -> std::io::Result<()> { + std::fs::OpenOptions::new() + .write(true) + .open(staging)? + .sync_data() + } + + fn rename(&self, from: &Path, to: &Path) -> std::io::Result<()> { + std::fs::rename(from, to) + } + + fn open_directory(&self, directory: &Path) -> std::io::Result { + std::fs::File::open(directory) + } + + fn sync_directory(&self, directory: &std::fs::File) -> std::io::Result<()> { + directory.sync_all() + } + + fn remove_staging(&self, staging: &Path) -> std::io::Result<()> { + std::fs::remove_file(staging) + } +} + +pub(super) fn write_state_atomically( path: &Path, state: &MaintenanceState, - rename: R, -) -> std::io::Result<()> -where - R: FnOnce(&Path, &Path) -> std::io::Result<()>, -{ - let bytes = serde_json::to_vec(state).map_err(std::io::Error::other)?; +) -> std::result::Result<(), PersistFailure> { + write_state_atomically_with(&RealStateFilesystem, path, state) +} + +fn not_applied( + phase: PersistPhase, + source: std::io::Error, + staging_cleanup: Option, +) -> PersistFailure { + PersistFailure::NotApplied { + phase, + source, + staging_cleanup, + } +} + +pub(super) fn write_state_atomically_with( + fs: &F, + path: &Path, + state: &MaintenanceState, +) -> std::result::Result<(), PersistFailure> { + let bytes = serde_json::to_vec(state) + .map_err(|error| not_applied(PersistPhase::Serialize, error.into(), None))?; let directory = path.parent().ok_or_else(|| { - std::io::Error::new( - std::io::ErrorKind::InvalidInput, - "maintenance state path has no parent directory", + not_applied( + PersistPhase::CreateStaging, + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "maintenance state path has no parent directory", + ), + None, ) })?; - let staging = create_staging_file(directory)?; - let staged = write_staged(&staging, &bytes).and_then(|()| rename(&staging, path)); - if let Err(error) = staged { - let _ = std::fs::remove_file(&staging); - return Err(error); + let staging = fs + .create_staging(directory) + .map_err(|error| not_applied(PersistPhase::CreateStaging, error, None))?; + + let staged = fs + .write_staging(&staging, &bytes) + .map_err(|error| (PersistPhase::WriteStaging, error)) + .and_then(|()| { + fs.sync_staging(&staging) + .map_err(|error| (PersistPhase::SyncStaging, error)) + }) + .and_then(|()| { + fs.rename(&staging, path) + .map_err(|error| (PersistPhase::Rename, error)) + }); + if let Err((phase, source)) = staged { + return Err(not_applied( + phase, + source, + fs.remove_staging(&staging).err(), + )); } - if let Ok(directory_handle) = std::fs::File::open(directory) { - let _ = directory_handle.sync_all(); - } - Ok(()) -} - -fn write_staged(staging: &Path, bytes: &[u8]) -> std::io::Result<()> { - let mut file = std::fs::OpenOptions::new().write(true).open(staging)?; - file.write_all(bytes)?; - file.sync_data() + let directory_handle = + fs.open_directory(directory) + .map_err(|source| PersistFailure::DurabilityUncertain { + phase: PersistPhase::OpenParentDirectory, + source, + })?; + fs.sync_directory(&directory_handle) + .map_err(|source| PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + source, + }) } fn create_staging_file(directory: &Path) -> std::io::Result { @@ -380,3 +523,242 @@ pub(super) fn derive_git_dir(start: &Path) -> Option { None } + +#[cfg(test)] +pub(super) struct FaultInjectingFilesystem { + pub fail_at: Option, + pub fail_staging_cleanup: bool, +} + +#[cfg(test)] +impl FaultInjectingFilesystem { + pub(super) const fn failing_at(phase: PersistPhase) -> Self { + Self { + fail_at: Some(phase), + fail_staging_cleanup: false, + } + } + + fn check(&self, phase: PersistPhase) -> std::io::Result<()> { + if self.fail_at == Some(phase) { + return Err(std::io::Error::other(format!("injected {phase:?} failure"))); + } + Ok(()) + } +} + +#[cfg(test)] +impl StateFilesystem for FaultInjectingFilesystem { + fn create_staging(&self, directory: &Path) -> std::io::Result { + self.check(PersistPhase::CreateStaging)?; + RealStateFilesystem.create_staging(directory) + } + + fn write_staging(&self, staging: &Path, bytes: &[u8]) -> std::io::Result<()> { + self.check(PersistPhase::WriteStaging)?; + RealStateFilesystem.write_staging(staging, bytes) + } + + fn sync_staging(&self, staging: &Path) -> std::io::Result<()> { + self.check(PersistPhase::SyncStaging)?; + RealStateFilesystem.sync_staging(staging) + } + + fn rename(&self, from: &Path, to: &Path) -> std::io::Result<()> { + self.check(PersistPhase::Rename)?; + RealStateFilesystem.rename(from, to) + } + + fn open_directory(&self, directory: &Path) -> std::io::Result { + self.check(PersistPhase::OpenParentDirectory)?; + RealStateFilesystem.open_directory(directory) + } + + fn sync_directory(&self, directory: &std::fs::File) -> std::io::Result<()> { + self.check(PersistPhase::SyncParentDirectory)?; + RealStateFilesystem.sync_directory(directory) + } + + fn remove_staging(&self, staging: &Path) -> std::io::Result<()> { + if self.fail_staging_cleanup { + return Err(std::io::Error::other("injected staging cleanup failure")); + } + RealStateFilesystem.remove_staging(staging) + } +} + +#[cfg(test)] +mod tests { + use std::path::{Path, PathBuf}; + + use super::{ + read_state, record_success, write_state_atomically, write_state_atomically_with, + FaultInjectingFilesystem, MaintenanceState, PersistFailure, PersistPhase, StateRead, + StoredReport, + }; + + const NOW: i64 = 1_000_000_000_000; + + fn state(marker: usize) -> MaintenanceState { + record_success( + &MaintenanceState::default(), + NOW, + StoredReport { + retained: marker, + deleted: 0, + local_required: 0, + }, + ) + } + + fn state_file(dir: &Path) -> PathBuf { + dir.join("ref-maintenance.json") + } + + fn stored_marker(path: &Path) -> Option { + match read_state(path).expect("read state") { + StateRead::Valid(state) => state.last_report.map(|report| report.retained), + StateRead::Absent => None, + StateRead::Unusable => panic!("state unusable"), + } + } + + fn staging_leftovers(dir: &Path) -> usize { + std::fs::read_dir(dir) + .expect("read dir") + .filter_map(Result::ok) + .filter(|entry| entry.file_name().to_string_lossy().contains("staging")) + .count() + } + + fn assert_not_applied( + phase: PersistPhase, + previous_marker: Option, + ) -> (tempfile::TempDir, PathBuf) { + let dir = tempfile::tempdir().expect("temp dir"); + let path = state_file(dir.path()); + if let Some(marker) = previous_marker { + write_state_atomically(&path, &state(marker)).expect("seed"); + } + + let fs = FaultInjectingFilesystem::failing_at(phase); + let failure = + write_state_atomically_with(&fs, &path, &state(99)).expect_err("injected failure"); + let PersistFailure::NotApplied { + phase: reported, + staging_cleanup, + .. + } = failure + else { + panic!("expected NotApplied for {phase:?}, got {failure:?}"); + }; + assert_eq!(reported, phase); + assert!(staging_cleanup.is_none()); + assert_eq!(stored_marker(&path), previous_marker); + assert_eq!(staging_leftovers(dir.path()), 0); + (dir, path) + } + + #[test] + fn maintenance_state_write_succeeds_end_to_end() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = state_file(dir.path()); + write_state_atomically(&path, &state(1)).expect("first write"); + write_state_atomically(&path, &state(2)).expect("second write"); + assert_eq!(stored_marker(&path), Some(2)); + assert_eq!(staging_leftovers(dir.path()), 0); + } + + #[test] + fn maintenance_state_staging_creation_failure_preserves_previous_state() { + assert_not_applied(PersistPhase::CreateStaging, Some(1)); + } + + #[test] + fn maintenance_state_staging_write_failure_preserves_previous_state_and_cleans_up() { + assert_not_applied(PersistPhase::WriteStaging, Some(1)); + } + + #[test] + fn maintenance_state_staging_sync_failure_preserves_previous_state_and_cleans_up() { + assert_not_applied(PersistPhase::SyncStaging, Some(1)); + } + + #[test] + fn maintenance_state_rename_failure_preserves_previous_state_and_cleans_up() { + assert_not_applied(PersistPhase::Rename, Some(1)); + } + + #[test] + fn maintenance_state_rename_failure_without_previous_state_leaves_no_state() { + assert_not_applied(PersistPhase::Rename, None); + } + + #[test] + fn maintenance_state_parent_directory_open_failure_is_durability_uncertain() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = state_file(dir.path()); + write_state_atomically(&path, &state(1)).expect("seed"); + + let fs = FaultInjectingFilesystem::failing_at(PersistPhase::OpenParentDirectory); + let failure = write_state_atomically_with(&fs, &path, &state(2)).expect_err("failure"); + assert!(matches!( + failure, + PersistFailure::DurabilityUncertain { + phase: PersistPhase::OpenParentDirectory, + .. + } + )); + assert!(failure.to_string().contains("may be visible")); + assert_eq!(stored_marker(&path), Some(2)); + assert_eq!(staging_leftovers(dir.path()), 0); + } + + #[test] + fn maintenance_state_parent_directory_sync_failure_distinguishes_visibility_from_durability() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = state_file(dir.path()); + write_state_atomically(&path, &state(1)).expect("seed"); + + let fs = FaultInjectingFilesystem::failing_at(PersistPhase::SyncParentDirectory); + let failure = write_state_atomically_with(&fs, &path, &state(2)).expect_err("failure"); + assert!(matches!( + failure, + PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + .. + } + )); + assert!(!failure.to_string().contains("previous state preserved")); + assert_eq!(stored_marker(&path), Some(2)); + assert_eq!(staging_leftovers(dir.path()), 0); + + write_state_atomically(&path, &state(3)).expect("recovery write"); + assert_eq!(stored_marker(&path), Some(3)); + } + + #[test] + fn maintenance_state_recovers_after_interrupted_pre_rename_write() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = state_file(dir.path()); + write_state_atomically(&path, &state(1)).expect("seed"); + + let fs = FaultInjectingFilesystem { + fail_at: Some(PersistPhase::Rename), + fail_staging_cleanup: true, + }; + let failure = write_state_atomically_with(&fs, &path, &state(2)).expect_err("failure"); + let PersistFailure::NotApplied { + staging_cleanup, .. + } = failure + else { + panic!("expected NotApplied"); + }; + assert!(staging_cleanup.is_some()); + assert_eq!(staging_leftovers(dir.path()), 1); + assert_eq!(stored_marker(&path), Some(1)); + + write_state_atomically(&path, &state(3)).expect("recovery write"); + assert_eq!(stored_marker(&path), Some(3)); + } +} diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs index 405748cb7..5a422349e 100644 --- a/cli/src/services/mutation_trace/runtime/ref_advisory.rs +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -8,26 +8,71 @@ use std::time::Duration; use super::maintenance_state::{ derive_git_dir, plan_advisory, read_state, state_path, write_state_atomically, - AdvisoryDecision, MaintenanceState, + AdvisoryDecision, MaintenanceState, PersistFailure, }; use super::worktree_lock::{acquire_inner, WorktreeLockError}; -#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) const RECONCILIATION_RECOMMENDED_MESSAGE: &str = + "Reconciliation is recommended. Run sce doctor --fix."; + +#[derive(Debug)] pub(super) enum AdvisoryOutcome { Anchored, NoAction, Advised, + AdvisedDurabilityUncertain { warning: PersistFailure }, Busy, + StateWriteFailed { warning: PersistFailure }, StateUnavailable, } +impl AdvisoryOutcome { + pub(super) fn recommendation(&self) -> Option<&'static str> { + match self { + AdvisoryOutcome::Advised | AdvisoryOutcome::AdvisedDurabilityUncertain { .. } => { + Some(RECONCILIATION_RECOMMENDED_MESSAGE) + } + AdvisoryOutcome::Anchored + | AdvisoryOutcome::NoAction + | AdvisoryOutcome::Busy + | AdvisoryOutcome::StateWriteFailed { .. } + | AdvisoryOutcome::StateUnavailable => None, + } + } + + pub(super) fn persistence_warning(&self) -> Option<&PersistFailure> { + match self { + AdvisoryOutcome::AdvisedDurabilityUncertain { warning } + | AdvisoryOutcome::StateWriteFailed { warning } => Some(warning), + AdvisoryOutcome::Anchored + | AdvisoryOutcome::NoAction + | AdvisoryOutcome::Advised + | AdvisoryOutcome::Busy + | AdvisoryOutcome::StateUnavailable => None, + } + } +} + +fn outcome_after_write( + decision: AdvisoryDecision, + write: Result<(), PersistFailure>, +) -> AdvisoryOutcome { + match (decision, write) { + (AdvisoryDecision::Anchored, Ok(())) => AdvisoryOutcome::Anchored, + (AdvisoryDecision::NoAction, Ok(())) => AdvisoryOutcome::NoAction, + (AdvisoryDecision::Advised, Ok(())) => AdvisoryOutcome::Advised, + (AdvisoryDecision::Advised, Err(warning @ PersistFailure::DurabilityUncertain { .. })) => { + AdvisoryOutcome::AdvisedDurabilityUncertain { warning } + } + (_, Err(warning)) => AdvisoryOutcome::StateWriteFailed { warning }, + } +} + pub(super) fn advise_if_due(repository_root: &Path, now: C) -> AdvisoryOutcome where C: Fn() -> i64, { - advise_if_due_with(repository_root, now, |path, state| { - write_state_atomically(path, state, |from, to| std::fs::rename(from, to)) - }) + advise_if_due_with(repository_root, now, write_state_atomically) } pub(super) fn advise_if_due_with( @@ -37,7 +82,7 @@ pub(super) fn advise_if_due_with( ) -> AdvisoryOutcome where C: Fn() -> i64, - W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, + W: Fn(&Path, &MaintenanceState) -> Result<(), PersistFailure>, { let Some(git_dir) = derive_git_dir(repository_root) else { return AdvisoryOutcome::StateUnavailable; @@ -63,18 +108,10 @@ where }; let now_ms = now(); let (decision, pending) = plan_advisory(&locked, now_ms); - let outcome = match decision { - AdvisoryDecision::Anchored => AdvisoryOutcome::Anchored, - AdvisoryDecision::NoAction => AdvisoryOutcome::NoAction, - AdvisoryDecision::Advised => AdvisoryOutcome::Advised, - }; let result = match pending { - Some(next) => match write_state(&path, &next) { - Ok(()) => outcome, - Err(_) => AdvisoryOutcome::StateUnavailable, - }, - None => outcome, + Some(next) => outcome_after_write(decision, write_state(&path, &next)), + None => outcome_after_write(decision, Ok(())), }; drop(lock); result @@ -86,11 +123,13 @@ mod tests { use std::process::Command; use super::super::maintenance_state::{ - evaluate_recommendation, read_state, state_path, write_state_atomically, MaintenanceState, - StateRead, FUTURE_SKEW_TOLERANCE_MS, RECONCILIATION_ADVISORY_AFTER_MS, + evaluate_recommendation, read_state, record_failure, record_success, state_path, + write_state_atomically, write_state_atomically_with, AttemptOutcome, + FaultInjectingFilesystem, MaintenanceState, PersistFailure, PersistPhase, StateRead, + StoredReport, FUTURE_SKEW_TOLERANCE_MS, RECONCILIATION_ADVISORY_AFTER_MS, }; use super::super::worktree_lock::acquire_inner; - use super::{advise_if_due_with, AdvisoryOutcome}; + use super::{advise_if_due_with, AdvisoryOutcome, RECONCILIATION_RECOMMENDED_MESSAGE}; const NOW: i64 = 1_000_000_000_000; @@ -104,8 +143,8 @@ mod tests { assert!(output.status.success()); } - fn real_write(path: &Path, state: &MaintenanceState) -> std::io::Result<()> { - write_state_atomically(path, state, |from, to| std::fs::rename(from, to)) + fn real_write(path: &Path, state: &MaintenanceState) -> Result<(), PersistFailure> { + write_state_atomically(path, state) } fn advise(root: &Path, now: i64) -> AdvisoryOutcome { @@ -130,12 +169,15 @@ mod tests { let dir = tempfile::tempdir().expect("temp dir"); init_repo(dir.path()); - assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored)); let path = state_path(&dir.path().join(".git")); let first = std::fs::read(&path).expect("state bytes"); assert_eq!(stored(dir.path()).anchor, Some(NOW)); - assert_eq!(advise(dir.path(), NOW + 1000), AdvisoryOutcome::NoAction); + assert!(matches!( + advise(dir.path(), NOW + 1000), + AdvisoryOutcome::NoAction + )); assert_eq!(std::fs::read(&path).expect("state bytes"), first); } @@ -146,13 +188,16 @@ mod tests { advise(dir.path(), NOW); let due = NOW + RECONCILIATION_ADVISORY_AFTER_MS; - assert_eq!(advise(dir.path(), due), AdvisoryOutcome::Advised); + assert!(matches!(advise(dir.path(), due), AdvisoryOutcome::Advised)); assert_eq!(stored(dir.path()).last_advised, Some(due)); - assert_eq!(advise(dir.path(), due + 1000), AdvisoryOutcome::NoAction); - assert_eq!( + assert!(matches!( + advise(dir.path(), due + 1000), + AdvisoryOutcome::NoAction + )); + assert!(matches!( advise(dir.path(), due + RECONCILIATION_ADVISORY_AFTER_MS), AdvisoryOutcome::Advised - ); + )); } #[test] @@ -167,7 +212,7 @@ mod tests { let path = state_path(&dir.path().join(".git")); let before = std::fs::read(&path).expect("state bytes"); - assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); assert_eq!(std::fs::read(&path).expect("state bytes"), before); } @@ -182,7 +227,7 @@ mod tests { }; seed(dir.path(), &invalid); - assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Advised); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Advised)); let normalized = stored(dir.path()); assert_eq!(normalized.last_advised, Some(NOW)); assert_eq!(normalized.anchor, Some(NOW)); @@ -190,7 +235,7 @@ mod tests { let path = state_path(&dir.path().join(".git")); let bytes = std::fs::read(&path).expect("state bytes"); - assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); } @@ -208,7 +253,7 @@ mod tests { let held = acquire_inner(&dir.path().join(".git"), std::time::Duration::ZERO, || {}) .expect("hold lock"); - assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Busy); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Busy)); assert_eq!(std::fs::read(&path).expect("state bytes"), before); drop(held); @@ -217,8 +262,155 @@ mod tests { assert!(recommendation.invalid_timestamps); } + fn advise_with_fault(root: &Path, now: i64, phase: PersistPhase) -> AdvisoryOutcome { + advise_if_due_with( + root, + || now, + |path, state| { + write_state_atomically_with( + &FaultInjectingFilesystem::failing_at(phase), + path, + state, + ) + }, + ) + } + + fn failed_state(now: i64) -> MaintenanceState { + record_failure( + &MaintenanceState::default(), + now, + "agent_trace_db_unavailable", + "agent trace database is missing", + ) + } + #[test] - fn maintenance_advisory_write_failure_emits_no_advice_and_keeps_previous_state() { + fn maintenance_first_failed_explicit_pass_without_anchor_is_recommended() { + let state = failed_state(NOW); + assert_eq!(state.anchor, None); + assert_eq!(state.last_success, None); + + let recommendation = evaluate_recommendation(&state, NOW); + assert!(recommendation.recommended); + assert_eq!( + recommendation.last_attempt_outcome, + Some(AttemptOutcome::Failed) + ); + assert_eq!(recommendation.consecutive_failures, 1); + assert!(!evaluate_recommendation(&MaintenanceState::default(), NOW).recommended); + } + + #[test] + fn maintenance_failure_advisory_emits_once_and_doctor_still_recommends_repair() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + seed(dir.path(), &failed_state(NOW)); + + let first = advise(dir.path(), NOW + 1000); + assert!(matches!(first, AdvisoryOutcome::Advised)); + assert_eq!( + first.recommendation(), + Some(RECONCILIATION_RECOMMENDED_MESSAGE) + ); + let persisted = stored(dir.path()); + assert_eq!(persisted.last_advised, Some(NOW + 1000)); + assert_eq!(persisted.consecutive_failures, 1); + assert_eq!(persisted.last_attempt_outcome, Some(AttemptOutcome::Failed)); + assert_eq!(persisted.last_failure, failed_state(NOW).last_failure); + + let path = state_path(&dir.path().join(".git")); + let bytes = std::fs::read(&path).expect("state bytes"); + let second = advise(dir.path(), NOW + 2000); + assert!(matches!(second, AdvisoryOutcome::NoAction)); + assert_eq!(second.recommendation(), None); + assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); + + let recommendation = evaluate_recommendation(&stored(dir.path()), NOW + 2000); + assert!(recommendation.recommended); + assert!(recommendation.last_advised_age_ms.is_some()); + } + + #[test] + fn maintenance_failure_advisory_repeats_after_the_cadence_window() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + seed(dir.path(), &failed_state(NOW)); + + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Advised)); + let later = NOW + RECONCILIATION_ADVISORY_AFTER_MS; + assert!(matches!( + advise(dir.path(), later), + AdvisoryOutcome::Advised + )); + assert_eq!(stored(dir.path()).consecutive_failures, 1); + } + + #[test] + fn maintenance_repeated_failed_passes_keep_recommendation_active() { + let first = failed_state(NOW); + let second = record_failure(&first, NOW + 10, "pin_inventory", "inventory failed"); + let third = record_failure(&second, NOW + 20, "pin_inventory", "inventory failed"); + assert_eq!(third.consecutive_failures, 3); + assert!(evaluate_recommendation(&first, NOW + 30).recommended); + assert!(evaluate_recommendation(&second, NOW + 30).recommended); + assert!(evaluate_recommendation(&third, NOW + 30).recommended); + } + + #[test] + fn maintenance_successful_pass_clears_failure_streak_and_recommendation() { + let failed = record_failure(&failed_state(NOW), NOW + 10, "pin_inventory", "failed"); + let healed = record_success( + &failed, + NOW + 20, + StoredReport { + retained: 1, + deleted: 0, + local_required: 0, + }, + ); + assert_eq!(healed.consecutive_failures, 0); + assert_eq!(healed.last_failure, None); + assert_eq!(healed.last_attempt_outcome, Some(AttemptOutcome::Completed)); + assert!(!evaluate_recommendation(&healed, NOW + 30).recommended); + } + + #[test] + fn maintenance_failure_after_recent_success_is_recommended_and_advised_despite_older_reminder() + { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let success = record_success( + &MaintenanceState::default(), + NOW, + StoredReport { + retained: 1, + deleted: 0, + local_required: 0, + }, + ); + let reminded = MaintenanceState { + last_advised: Some(NOW + 1000), + ..success + }; + assert!(!evaluate_recommendation(&reminded, NOW + 1500).recommended); + + let failed = record_failure(&reminded, NOW + 2000, "delete_transaction", "failed"); + assert!(evaluate_recommendation(&failed, NOW + 2500).recommended); + seed(dir.path(), &failed); + + assert!(matches!( + advise(dir.path(), NOW + 2500), + AdvisoryOutcome::Advised + )); + assert!(matches!( + advise(dir.path(), NOW + 3000), + AdvisoryOutcome::NoAction + )); + } + + #[test] + fn maintenance_advisory_rename_failure_preserves_state_and_next_call_can_advise() { let dir = tempfile::tempdir().expect("temp dir"); init_repo(dir.path()); advise(dir.path(), NOW); @@ -226,18 +418,92 @@ mod tests { let before = std::fs::read(&path).expect("state bytes"); let due = NOW + RECONCILIATION_ADVISORY_AFTER_MS; - let outcome = advise_if_due_with( - dir.path(), - || due, - |path, state| { - write_state_atomically(path, state, |_, _| { - Err(std::io::Error::other("rename interrupted")) - }) - }, - ); - assert_eq!(outcome, AdvisoryOutcome::StateUnavailable); + let outcome = advise_with_fault(dir.path(), due, PersistPhase::Rename); + assert!(matches!( + outcome, + AdvisoryOutcome::StateWriteFailed { + warning: PersistFailure::NotApplied { + phase: PersistPhase::Rename, + .. + } + } + )); + assert_eq!(outcome.recommendation(), None); + assert!(outcome.persistence_warning().is_some()); assert_eq!(std::fs::read(&path).expect("state bytes"), before); - assert_eq!(advise(dir.path(), due), AdvisoryOutcome::Advised); + assert!(evaluate_recommendation(&stored(dir.path()), due).recommended); + assert!(matches!(advise(dir.path(), due), AdvisoryOutcome::Advised)); + } + + fn assert_durability_uncertain_advisory(phase: PersistPhase) { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + advise(dir.path(), NOW); + + let due = NOW + RECONCILIATION_ADVISORY_AFTER_MS; + let outcome = advise_with_fault(dir.path(), due, phase); + assert!(!matches!(outcome, AdvisoryOutcome::Advised)); + let AdvisoryOutcome::AdvisedDurabilityUncertain { warning } = &outcome else { + panic!("expected AdvisedDurabilityUncertain for {phase:?}, got {outcome:?}"); + }; + assert!(matches!( + warning, + PersistFailure::DurabilityUncertain { phase: reported, .. } if *reported == phase + )); + assert_eq!( + outcome.recommendation(), + Some(RECONCILIATION_RECOMMENDED_MESSAGE) + ); + assert!(outcome.persistence_warning().is_some()); + + let visible = stored(dir.path()); + assert_eq!(visible.last_advised, Some(due)); + + assert!(matches!( + advise(dir.path(), due + 1000), + AdvisoryOutcome::NoAction + )); + let recommendation = evaluate_recommendation(&visible, due + 1000); + assert!(recommendation.recommended); + assert!(recommendation.last_advised_age_ms.is_some()); + } + + #[test] + fn maintenance_advisory_parent_directory_open_failure_is_durability_uncertain() { + assert_durability_uncertain_advisory(PersistPhase::OpenParentDirectory); + } + + #[test] + fn maintenance_advisory_parent_directory_sync_failure_is_durability_uncertain() { + assert_durability_uncertain_advisory(PersistPhase::SyncParentDirectory); + } + + #[test] + fn maintenance_advisory_durability_uncertain_failure_state_stays_recommended() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + seed(dir.path(), &failed_state(NOW)); + + let outcome = advise_with_fault(dir.path(), NOW + 10, PersistPhase::SyncParentDirectory); + assert!(matches!( + outcome, + AdvisoryOutcome::AdvisedDurabilityUncertain { .. } + )); + let visible = stored(dir.path()); + assert_eq!(visible.last_advised, Some(NOW + 10)); + assert_eq!(visible.consecutive_failures, 1); + assert!(evaluate_recommendation(&visible, NOW + 20).recommended); + } + + #[test] + fn maintenance_advisory_anchor_write_failure_reports_state_write_failed_without_advice() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + + let outcome = advise_with_fault(dir.path(), NOW, PersistPhase::Rename); + assert!(matches!(outcome, AdvisoryOutcome::StateWriteFailed { .. })); + assert_eq!(outcome.recommendation(), None); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored)); } #[test] @@ -248,7 +514,7 @@ mod tests { std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); std::fs::write(&path, b"not json").expect("corrupt state"); - assert_eq!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored)); assert_eq!(stored(dir.path()).anchor, Some(NOW)); } @@ -283,7 +549,7 @@ mod tests { .expect("canonical"); assert_eq!(super::derive_git_dir(&sub), Some(expected.clone())); - assert_eq!(advise(&sub, NOW), AdvisoryOutcome::Anchored); + assert!(matches!(advise(&sub, NOW), AdvisoryOutcome::Anchored)); assert!(state_path(&expected).is_file()); } } diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance.rs index c7be29939..74c31bbe8 100644 --- a/cli/src/services/mutation_trace/runtime/ref_maintenance.rs +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance.rs @@ -6,12 +6,12 @@ use std::path::Path; use std::time::Duration; -use anyhow::Result; +use anyhow::{Context, Result}; use super::git_snapshot::resolve_git_dir; use super::maintenance_state::{ read_state, record_failure, record_success, state_path, system_now_ms, write_state_atomically, - MaintenanceState, StateRead, StoredReport, + MaintenanceState, PersistFailure, StateRead, StoredReport, }; use super::ref_reconciliation::{ reconcile_with_held_lock, ReconcileError, ReconcilePhase, ReconciliationOutcome, @@ -19,6 +19,13 @@ use super::ref_reconciliation::{ }; use super::worktree_lock::{acquire_inner_async, WorktreeLockError}; use crate::services::agent_trace_db::repository::RepositoryAgentTraceDb; +use crate::services::agent_trace_storage::{ + resolve_existing_agent_trace_storage_for_maintenance, AgentTraceStorageContext, +}; +use crate::services::config; + +#[cfg(test)] +mod tests; const EXPLICIT_LOCK_TIMEOUT: Duration = Duration::from_secs(10); @@ -27,33 +34,63 @@ pub(super) enum SkipReason { Busy, } +#[derive(Debug)] +pub(super) enum StatePersistWarning { + PreviousStateUnreadable(std::io::Error), + WriteFailed(PersistFailure), +} + +impl std::fmt::Display for StatePersistWarning { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + StatePersistWarning::PreviousStateUnreadable(error) => { + write!(f, "maintenance state could not be read: {error}") + } + StatePersistWarning::WriteFailed(failure) => write!(f, "{failure}"), + } + } +} + #[derive(Debug)] pub(super) enum ExplicitOutcome { Completed(ReconciliationReport), CompletedStatePersistFailed { report: ReconciliationReport, - warning: String, + warning: StatePersistWarning, + }, + Failed { + error: ReconcileError, + state_warning: Option, }, - Failed(ReconcileError), Skipped(SkipReason), } -pub(super) async fn reconcile_explicit

(repository_root: &Path, open_db: P) -> ExplicitOutcome -where - P: std::ops::AsyncFnOnce() -> Result, -{ +pub(super) async fn reconcile_explicit(repository_root: &Path) -> ExplicitOutcome { reconcile_explicit_with( repository_root, - open_db, + async || open_authoritative_db(repository_root).await, system_now_ms, - |path, state| write_state_atomically(path, state, |from, to| std::fs::rename(from, to)), + write_state_atomically, |_| {}, EXPLICIT_LOCK_TIMEOUT, ) .await } -pub(super) async fn reconcile_explicit_with( +async fn open_authoritative_db(repository_root: &Path) -> Result { + let storage_config = config::resolve_agent_trace_storage_runtime_config(repository_root) + .context("Failed to resolve Agent Trace repository storage config.")?; + let context = AgentTraceStorageContext { + repository_root, + explicit_repository_id: storage_config.repository_id.as_deref(), + repository_remote: &storage_config.repository_remote, + }; + resolve_existing_agent_trace_storage_for_maintenance(&context) + .await + .map(|storage| storage.db) +} + +async fn reconcile_explicit_with( repository_root: &Path, open_db: P, now: C, @@ -64,12 +101,17 @@ pub(super) async fn reconcile_explicit_with( where P: std::ops::AsyncFnOnce() -> Result, C: Fn() -> i64, - W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, + W: Fn(&Path, &MaintenanceState) -> std::result::Result<(), PersistFailure>, H: FnMut(ReconcilePhase), { let git_dir = match resolve_git_dir(repository_root).await { Ok(git_dir) => git_dir, - Err(source) => return ExplicitOutcome::Failed(ReconcileError::GitDir(source)), + Err(source) => { + return ExplicitOutcome::Failed { + error: ReconcileError::GitDir(source), + state_warning: None, + }; + } }; let lock = match acquire_inner_async(&git_dir, lock_timeout, || {}).await { @@ -77,7 +119,12 @@ where Err(WorktreeLockError::TimedOut { .. }) => { return ExplicitOutcome::Skipped(SkipReason::Busy); } - Err(error) => return ExplicitOutcome::Failed(ReconcileError::Lock(error)), + Err(error) => { + return ExplicitOutcome::Failed { + error: ReconcileError::Lock(error), + state_warning: None, + }; + } }; let result = reconcile_with_held_lock(repository_root, &lock, open_db, on_phase).await; @@ -105,13 +152,9 @@ where let error = ReconcileError::CheckoutIdentity(anyhow::anyhow!( "no checkout identity could be derived" )); - record_failed_pass(&previous, &write_state, &path, now_ms, &error); - ExplicitOutcome::Failed(error) - } - Err(error) => { - record_failed_pass(&previous, &write_state, &path, now_ms, &error); - ExplicitOutcome::Failed(error) + record_failed_pass(&previous, &write_state, &path, now_ms, error) } + Err(error) => record_failed_pass(&previous, &write_state, &path, now_ms, error), }; drop(lock); @@ -130,14 +173,16 @@ fn persist( write_state: &W, path: &Path, next: &MaintenanceState, -) -> std::result::Result<(), String> +) -> std::result::Result<(), StatePersistWarning> where - W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, + W: Fn(&Path, &MaintenanceState) -> std::result::Result<(), PersistFailure>, { if let Err(error) = previous { - return Err(format!("maintenance state could not be read: {error}")); + return Err(StatePersistWarning::PreviousStateUnreadable( + std::io::Error::new(error.kind(), error.to_string()), + )); } - write_state(path, next).map_err(|error| format!("maintenance state was not recorded: {error}")) + write_state(path, next).map_err(StatePersistWarning::WriteFailed) } fn record_failed_pass( @@ -145,17 +190,22 @@ fn record_failed_pass( write_state: &W, path: &Path, now_ms: i64, - error: &ReconcileError, -) where - W: Fn(&Path, &MaintenanceState) -> std::io::Result<()>, + error: ReconcileError, +) -> ExplicitOutcome +where + W: Fn(&Path, &MaintenanceState) -> std::result::Result<(), PersistFailure>, { let next = record_failure( &previous_state(previous), now_ms, - failure_kind(error), + failure_kind(&error), &error.to_string(), ); - let _ = persist(previous, write_state, path, &next); + let state_warning = persist(previous, write_state, path, &next).err(); + ExplicitOutcome::Failed { + error, + state_warning, + } } fn failure_kind(error: &ReconcileError) -> &'static str { @@ -172,3 +222,21 @@ fn failure_kind(error: &ReconcileError) -> &'static str { ReconcileError::DeleteTransaction(_) => "delete_transaction", } } + +#[cfg(test)] +async fn open_authoritative_db_at_state_root( + repository_root: &Path, + state_root: &Path, +) -> Result { + let context = AgentTraceStorageContext { + repository_root, + explicit_repository_id: None, + repository_remote: "origin", + }; + crate::services::agent_trace_storage::resolve_existing_agent_trace_storage_for_maintenance_at_state_root( + &context, + state_root, + ) + .await + .map(|storage| storage.db) +} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs new file mode 100644 index 000000000..52f9e825a --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs @@ -0,0 +1,816 @@ +use std::path::{Path, PathBuf}; +use std::process::Command; +use std::time::Duration; + +use super::super::git_snapshot::{resolve_git_dir, GitSnapshotService, PinnedRef}; +use super::super::maintenance_state::{ + evaluate_recommendation, read_state, state_path, write_state_atomically, + write_state_atomically_with, FaultInjectingFilesystem, MaintenanceState, PersistFailure, + PersistPhase, StateRead, +}; +use super::super::ref_reconciliation::{ + reconcile_with_held_lock, reconcile_worktree, ReconcileError, ReconcilePhase, + ReconciliationOutcome, ReconciliationReport, +}; +use super::super::worktree_lock::acquire_inner_async; +use super::{ + open_authoritative_db_at_state_root, reconcile_explicit, reconcile_explicit_with, + ExplicitOutcome, SkipReason, StatePersistWarning, +}; +use crate::services::agent_trace_db::repository::{ + ExistingRepositoryDbError, RepositoryAgentTraceDb, +}; +use crate::services::default_paths::agent_trace_db_path_for_repository_at; +use crate::services::mutation_trace::types::WorktreeId; +use crate::services::repository_identity::resolve::resolve_repository_identity; + +const NOW: i64 = 1_000_000_000_000; +const OTHER_REPOSITORY_ID: &str = "some-other-repository"; + +struct Fixture { + _dir: tempfile::TempDir, + root: PathBuf, + state_root: PathBuf, + repository_id: String, +} + +impl Fixture { + fn new(remote_url: &str) -> Self { + let dir = tempfile::tempdir().expect("temp dir"); + let root = dir.path().join("repo"); + init_repo(&root, remote_url); + let repository_id = resolve_repository_identity(&root, None, "origin") + .expect("identity") + .identity + .repository_id; + let state_root = dir.path().join("state"); + Self { + _dir: dir, + root, + state_root, + repository_id, + } + } + + fn db_path(&self) -> PathBuf { + agent_trace_db_path_for_repository_at(&self.state_root, &self.repository_id) + .expect("db path") + } + + async fn create_db(&self) { + create_db(&self.db_path(), &self.repository_id).await; + } + + async fn run(&self) -> (ExplicitOutcome, Vec) { + self.run_with(write_state_atomically).await + } + + async fn run_with(&self, write_state: W) -> (ExplicitOutcome, Vec) + where + W: Fn(&Path, &MaintenanceState) -> Result<(), PersistFailure>, + { + let mut phases = Vec::new(); + let outcome = reconcile_explicit_with( + &self.root, + async || open_authoritative_db_at_state_root(&self.root, &self.state_root).await, + || NOW, + write_state, + |phase| phases.push(phase), + Duration::from_secs(10), + ) + .await; + (outcome, phases) + } + + async fn pins(&self) -> Vec { + GitSnapshotService::new(&self.root) + .await + .expect("snapshot") + .list_pins(&WorktreeId("main".to_string())) + .await + .expect("pins") + } + + async fn stored_state(&self) -> StateRead { + let git_dir = resolve_git_dir(&self.root).await.expect("git dir"); + read_state(&state_path(&git_dir)).expect("state") + } +} + +fn git(cwd: &Path, args: &[&str]) { + let output = Command::new("git") + .args(args) + .current_dir(cwd) + .output() + .expect("git command"); + assert!(output.status.success(), "git {args:?} failed"); +} + +fn init_repo(root: &Path, remote_url: &str) { + std::fs::create_dir_all(root).expect("repo dir"); + git(root, &["init", "-q"]); + git(root, &["config", "user.email", "t@example.invalid"]); + git(root, &["config", "user.name", "T"]); + git(root, &["config", "commit.gpgsign", "false"]); + git(root, &["remote", "add", "origin", remote_url]); + std::fs::write(root.join("tracked.txt"), "tracked\n").expect("write file"); + git(root, &["add", "tracked.txt"]); + git(root, &["commit", "-q", "-m", "initial"]); +} + +async fn create_db(path: &Path, repository_id: &str) { + let db = RepositoryAgentTraceDb::new_at(path) + .await + .expect("create db"); + db.verify_or_initialize_repository_metadata(repository_id) + .await + .expect("initialize metadata"); +} + +async fn pin_orphan(root: &Path) -> String { + let snapshot = GitSnapshotService::new(root).await.expect("snapshot"); + let tree = snapshot.capture_tree().await.expect("tree"); + let git_dir = resolve_git_dir(root).await.expect("git dir"); + let lock = acquire_inner_async(&git_dir, Duration::from_secs(10), || {}) + .await + .expect("lock"); + snapshot + .pin_tree(lock.lease(), &WorktreeId("main".to_string()), &tree) + .await + .expect("pin"); + tree.0 +} + +async fn seed_durable_root(db_path: &Path, tree: &str) { + let db = RepositoryAgentTraceDb::open_without_migrations_at(db_path) + .await + .expect("writer"); + db.execute( + "INSERT INTO mutation_trace_worktrees + (worktree_id, cursor_tree, revision, tainted, failure_kind, needs_rebaseline) + VALUES ('main', ?1, x'0000000000000000', 0, 'healthy', 0)", + (tree,), + ) + .await + .expect("seed durable root"); +} + +fn unavailable_cause(error: &ReconcileError) -> &ExistingRepositoryDbError { + let ReconcileError::AgentTraceDbUnavailable(source) = error else { + panic!("expected AgentTraceDbUnavailable, got {error:?}"); + }; + source + .downcast_ref::() + .expect("verification failure keeps its typed source") +} + +fn failing_write( + phase: PersistPhase, +) -> impl Fn(&Path, &MaintenanceState) -> Result<(), PersistFailure> { + move |path, state| { + write_state_atomically_with(&FaultInjectingFilesystem::failing_at(phase), path, state) + } +} + +fn staging_leftovers(dir: &Path) -> usize { + std::fs::read_dir(dir) + .expect("read dir") + .filter_map(Result::ok) + .filter(|entry| entry.file_name().to_string_lossy().contains("staging")) + .count() +} + +#[test] +fn ref_reconciliation_production_entrypoint_accepts_only_a_repository_root() { + fn assert_signature AsyncFn(&'a Path) -> ExplicitOutcome>(_: F) {} + assert_signature(reconcile_explicit); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_database_reconciles_orphan_pin_and_records_success() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + + let (outcome, phases) = fixture.run().await; + + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!( + report, + ReconciliationReport { + local_required: 0, + retained: 0, + deleted: 1, + } + ); + assert_eq!( + phases, + vec![ReconcilePhase::DbOpened, ReconcilePhase::PinsInventoried] + ); + assert!(fixture.pins().await.is_empty()); + let StateRead::Valid(state) = fixture.stored_state().await else { + panic!("expected recorded state"); + }; + assert_eq!(state.last_success, Some(NOW)); + assert_eq!(state.consecutive_failures, 0); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_database_of_another_repository_cannot_authorize_deletion() { + let repo_a = Fixture::new("https://example.invalid/org/repo-a.git"); + let repo_b = Fixture::new("https://example.invalid/org/repo-b.git"); + assert_ne!(repo_a.repository_id, repo_b.repository_id); + repo_b.create_db().await; + pin_orphan(&repo_a.root).await; + + let (outcome, phases) = repo_a.run().await; + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + assert!(matches!( + unavailable_cause(&error), + ExistingRepositoryDbError::Missing { .. } + )); + assert!(phases.is_empty()); + assert_eq!(repo_a.pins().await.len(), 1); + + create_db(&repo_a.db_path(), &repo_b.repository_id).await; + let (outcome, phases) = repo_a.run().await; + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + let ExistingRepositoryDbError::RepositoryMismatch { stored, resolved } = + unavailable_cause(&error) + else { + panic!("expected repository mismatch"); + }; + assert_eq!(stored, &repo_b.repository_id); + assert_eq!(resolved, &repo_a.repository_id); + assert!(phases.is_empty()); + assert_eq!(repo_a.pins().await.len(), 1); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_missing_metadata_or_schema_fails_before_pin_inventory() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + pin_orphan(&fixture.root).await; + + RepositoryAgentTraceDb::new_at(fixture.db_path()) + .await + .expect("migrated db without metadata"); + let (outcome, phases) = fixture.run().await; + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + assert!(matches!( + unavailable_cause(&error), + ExistingRepositoryDbError::MissingMetadata + )); + assert!(phases.is_empty()); + assert_eq!(fixture.pins().await.len(), 1); + + std::fs::remove_file(fixture.db_path()).expect("remove db"); + drop( + RepositoryAgentTraceDb::open_without_migrations_at(fixture.db_path()) + .await + .expect("unmigrated db"), + ); + let (outcome, phases) = fixture.run().await; + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + assert!(matches!( + unavailable_cause(&error), + ExistingRepositoryDbError::IncompatibleSchema(_) + )); + assert!(phases.is_empty()); + assert_eq!(fixture.pins().await.len(), 1); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_missing_database_creates_nothing_and_deletes_nothing() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + pin_orphan(&fixture.root).await; + + let (outcome, phases) = fixture.run().await; + + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + assert!(matches!( + unavailable_cause(&error), + ExistingRepositoryDbError::Missing { .. } + )); + assert!(phases.is_empty()); + assert!(!fixture.state_root.exists()); + assert_eq!(fixture.pins().await.len(), 1); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_first_failed_pass_without_anchor_is_recommended_and_advised_once() { + use super::super::ref_advisory::{advise_if_due_with, AdvisoryOutcome}; + + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + pin_orphan(&fixture.root).await; + + let (outcome, _) = fixture.run().await; + assert!(matches!(outcome, ExplicitOutcome::Failed { .. })); + let StateRead::Valid(state) = fixture.stored_state().await else { + panic!("expected recorded state"); + }; + assert_eq!(state.anchor, None); + assert_eq!(state.last_success, None); + let now = state.last_attempt.expect("attempt time"); + assert!(evaluate_recommendation(&state, now).recommended); + + let first = advise_if_due_with(&fixture.root, || now, write_state_atomically); + assert!(matches!(first, AdvisoryOutcome::Advised)); + let second = advise_if_due_with(&fixture.root, || now + 1000, write_state_atomically); + assert!(matches!(second, AdvisoryOutcome::NoAction)); + + let StateRead::Valid(after) = fixture.stored_state().await else { + panic!("expected recorded state"); + }; + assert_eq!(after.consecutive_failures, 1); + assert_eq!(after.last_failure, state.last_failure); + assert!(evaluate_recommendation(&after, now + 1000).recommended); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_failure_streak_counts_but_lock_contention_never_does() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + pin_orphan(&fixture.root).await; + + for expected in 1..=2 { + let (outcome, _) = fixture.run().await; + assert!(matches!( + outcome, + ExplicitOutcome::Failed { + state_warning: None, + .. + } + )); + let StateRead::Valid(state) = fixture.stored_state().await else { + panic!("expected recorded state"); + }; + assert_eq!(state.consecutive_failures, expected); + assert_eq!(state.last_success, None); + } + + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let before = std::fs::read(state_path(&git_dir)).expect("state bytes"); + let held = acquire_inner_async(&git_dir, Duration::from_secs(10), || {}) + .await + .expect("hold lock"); + let mut phases = Vec::new(); + let outcome = reconcile_explicit_with( + &fixture.root, + async || -> anyhow::Result { unreachable!("db must not open") }, + || NOW, + |_, _| unreachable!("state must not be written on contention"), + |phase| phases.push(phase), + Duration::ZERO, + ) + .await; + drop(held); + + assert!(matches!( + outcome, + ExplicitOutcome::Skipped(SkipReason::Busy) + )); + assert!(phases.is_empty()); + assert_eq!( + std::fs::read(state_path(&git_dir)).expect("state bytes"), + before + ); + let StateRead::Valid(state) = fixture.stored_state().await else { + panic!("expected recorded state"); + }; + assert_eq!(state.consecutive_failures, 2); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_missing_database_and_failed_persistence_keep_both_errors() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + let tree = pin_orphan(&fixture.root).await; + + let (outcome, _) = fixture.run_with(failing_write(PersistPhase::Rename)).await; + + let ExplicitOutcome::Failed { + error, + state_warning, + } = outcome + else { + panic!("expected failure, got {outcome:?}"); + }; + assert!(matches!( + unavailable_cause(&error), + ExistingRepositoryDbError::Missing { .. } + )); + let Some(StatePersistWarning::WriteFailed(PersistFailure::NotApplied { phase, .. })) = + state_warning + else { + panic!("expected retained persistence warning, got {state_warning:?}"); + }; + assert_eq!(phase, PersistPhase::Rename); + let pins = fixture.pins().await; + assert_eq!(pins.len(), 1); + assert_eq!(pins[0].tree.0, tree); + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + assert_eq!( + staging_leftovers(state_path(&git_dir).parent().expect("parent")), + 0 + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_missing_required_pin_and_failed_persistence_keep_both_errors() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + let snapshot = GitSnapshotService::new(&fixture.root) + .await + .expect("snapshot"); + let unpinned = snapshot.capture_tree().await.expect("tree"); + seed_durable_root(&fixture.db_path(), &unpinned.0).await; + + let (outcome, phases) = fixture + .run_with(failing_write(PersistPhase::WriteStaging)) + .await; + + let ExplicitOutcome::Failed { + error, + state_warning, + } = outcome + else { + panic!("expected failure, got {outcome:?}"); + }; + let ReconcileError::MissingRequiredPins { missing } = &error else { + panic!("expected missing required pins, got {error:?}"); + }; + assert_eq!(missing, &vec![unpinned]); + assert!(matches!( + state_warning, + Some(StatePersistWarning::WriteFailed( + PersistFailure::NotApplied { + phase: PersistPhase::WriteStaging, + .. + } + )) + )); + assert_eq!( + phases, + vec![ReconcilePhase::DbOpened, ReconcilePhase::PinsInventoried] + ); + assert!(fixture.pins().await.is_empty()); + assert!(matches!(fixture.stored_state().await, StateRead::Absent)); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_completed_pass_with_unconfirmed_durability_keeps_deletion() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + + let (outcome, _) = fixture + .run_with(failing_write(PersistPhase::SyncParentDirectory)) + .await; + + let ExplicitOutcome::CompletedStatePersistFailed { report, warning } = outcome else { + panic!("expected persist-failed completion, got {outcome:?}"); + }; + assert_eq!(report.deleted, 1); + assert!(matches!( + warning, + StatePersistWarning::WriteFailed(PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + .. + }) + )); + assert!(fixture.pins().await.is_empty()); + let StateRead::Valid(state) = fixture.stored_state().await else { + panic!("new state is visible even though durability is unconfirmed"); + }; + assert_eq!(state.last_success, Some(NOW)); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + + let (outcome, _) = fixture + .run_with(failing_write(PersistPhase::CreateStaging)) + .await; + + let ExplicitOutcome::CompletedStatePersistFailed { report, warning } = outcome else { + panic!("expected persist-failed completion, got {outcome:?}"); + }; + assert_eq!(report.deleted, 1); + assert!(warning.to_string().contains("previous state preserved")); + assert!(fixture.pins().await.is_empty()); + assert!(matches!(fixture.stored_state().await, StateRead::Absent)); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_unreadable_previous_state_is_reported_without_overwriting() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + pin_orphan(&fixture.root).await; + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let path = state_path(&git_dir); + std::fs::create_dir_all(&path).expect("directory in place of state file"); + + let (outcome, _) = fixture.run().await; + + let ExplicitOutcome::Failed { state_warning, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + assert!(matches!( + state_warning, + Some(StatePersistWarning::PreviousStateUnreadable(_)) + )); + assert!(path.is_dir()); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_lock_held_implementation_never_reacquires_the_lock() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let lock = acquire_inner_async(&git_dir, Duration::from_secs(10), || {}) + .await + .expect("lock"); + + let outcome = tokio::time::timeout( + Duration::from_secs(30), + reconcile_with_held_lock( + &fixture.root, + &lock, + async || open_authoritative_db_at_state_root(&fixture.root, &fixture.state_root).await, + |_| {}, + ), + ) + .await + .expect("must not block on a second acquisition") + .expect("reconcile"); + drop(lock); + + let ReconciliationOutcome::Reconciled(report) = outcome else { + panic!("expected reconciled outcome"); + }; + assert_eq!(report.deleted, 1); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_worktree_wrapper_acquires_once_and_releases() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + + let outcome = reconcile_worktree(&fixture.root, async || { + open_authoritative_db_at_state_root(&fixture.root, &fixture.state_root).await + }) + .await + .expect("reconcile"); + assert!(matches!(outcome, ReconciliationOutcome::Reconciled(_))); + + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + drop( + acquire_inner_async(&git_dir, Duration::ZERO, || {}) + .await + .expect("lock released after wrapper returns"), + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_open_never_creates_missing_database_or_directories() { + let dir = tempfile::tempdir().expect("temp dir"); + let missing = dir.path().join("nested").join("agent-trace.db"); + + let error = RepositoryAgentTraceDb::open_verified_existing_at(&missing, "repo") + .await + .err() + .expect("missing db fails"); + + assert!(matches!(error, ExistingRepositoryDbError::Missing { .. })); + assert!(!dir.path().join("nested").exists()); + + let missing_in_existing_dir = dir.path().join("agent-trace.db"); + let error = RepositoryAgentTraceDb::open_verified_existing_at(&missing_in_existing_dir, "repo") + .await + .err() + .expect("missing db fails"); + assert!(matches!(error, ExistingRepositoryDbError::Missing { .. })); + assert_eq!(std::fs::read_dir(dir.path()).expect("read dir").count(), 0); +} + +const DATABASE_SIDECAR_SUFFIXES: [&str; 3] = ["-wal", "-tshm", "-shm"]; + +fn with_suffix(path: &Path, suffix: &str) -> PathBuf { + let mut name = path.as_os_str().to_owned(); + name.push(suffix); + PathBuf::from(name) +} + +fn remove_database(path: &Path) { + std::fs::remove_file(path).ok(); + for suffix in DATABASE_SIDECAR_SUFFIXES { + std::fs::remove_file(with_suffix(path, suffix)).ok(); + } +} + +fn swap_in_database(source: &Path, target: &Path) { + remove_database(target); + for suffix in DATABASE_SIDECAR_SUFFIXES { + let sidecar = with_suffix(source, suffix); + if sidecar.exists() { + std::fs::rename(sidecar, with_suffix(target, suffix)).expect("move sidecar"); + } + } + std::fs::rename(source, target).expect("move database"); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_open_rejects_database_removed_before_open() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + create_db(&path, "repo").await; + let existed_when_hook_ran = std::cell::Cell::new(false); + + let error = RepositoryAgentTraceDb::open_verified_existing_at_with_hooks( + &path, + "repo", + || { + existed_when_hook_ran.set(path.is_file()); + remove_database(&path); + }, + || {}, + ) + .await + .err() + .expect("removed db fails"); + + assert!(existed_when_hook_ran.get()); + assert!(matches!(error, ExistingRepositoryDbError::Missing { .. })); + assert!(!path.exists()); + assert_eq!(std::fs::read_dir(dir.path()).expect("read dir").count(), 0); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_open_rejects_database_replaced_before_open() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + create_db(&path, "repo").await; + let foreign = dir.path().join("foreign.db"); + create_db(&foreign, "someone-else").await; + + let error = RepositoryAgentTraceDb::open_verified_existing_at_with_hooks( + &path, + "repo", + || swap_in_database(&foreign, &path), + || {}, + ) + .await + .err() + .expect("replacement with another repository fails"); + assert!(matches!( + error, + ExistingRepositoryDbError::RepositoryMismatch { .. } + )); + + let empty = dir.path().join("empty.db"); + drop( + RepositoryAgentTraceDb::open_without_migrations_at(&empty) + .await + .expect("empty db"), + ); + let error = RepositoryAgentTraceDb::open_verified_existing_at_with_hooks( + &path, + "repo", + || swap_in_database(&empty, &path), + || {}, + ) + .await + .err() + .expect("replacement without schema fails"); + assert!(matches!( + error, + ExistingRepositoryDbError::IncompatibleSchema(_) + )); + + let unusable = dir.path().join("unusable.db"); + std::fs::write(&unusable, b"this is not a database").expect("garbage"); + let error = RepositoryAgentTraceDb::open_verified_existing_at_with_hooks( + &path, + "repo", + || swap_in_database(&unusable, &path), + || {}, + ) + .await + .err() + .expect("replacement with garbage fails"); + assert!(matches!( + error, + ExistingRepositoryDbError::IncompatibleSchema(_) | ExistingRepositoryDbError::Unreadable(_) + )); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_open_accepts_a_replacement_only_if_it_verifies_itself() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + create_db(&path, "repo").await; + let replacement = dir.path().join("replacement.db"); + create_db(&replacement, "repo").await; + + let (_, metadata) = RepositoryAgentTraceDb::open_verified_existing_at_with_hooks( + &path, + "repo", + || swap_in_database(&replacement, &path), + || {}, + ) + .await + .expect("a replacement that passes every check is itself authoritative"); + assert_eq!(metadata.repository_id, "repo"); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_open_never_recreates_a_database_removed_after_open() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + create_db(&path, "repo").await; + + let (db, metadata) = RepositoryAgentTraceDb::open_verified_existing_at_with_hooks( + &path, + "repo", + || {}, + || remove_database(&path), + ) + .await + .expect("the opened handle is what gets verified"); + assert_eq!(metadata.repository_id, "repo"); + drop(db); + + assert!(!path.exists()); + assert_eq!(std::fs::read_dir(dir.path()).expect("read dir").count(), 0); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_open_is_read_only() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + create_db(&path, "repo").await; + + let (db, _) = RepositoryAgentTraceDb::open_verified_existing_at(&path, "repo") + .await + .expect("verified open"); + + assert!(db + .execute( + "INSERT INTO mutation_trace_worktrees + (worktree_id, cursor_tree, revision, tainted, failure_kind, needs_rebaseline) + VALUES ('main', 'tree', x'0000000000000000', 0, 'healthy', 0)", + (), + ) + .await + .is_err()); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verified_open_coexists_with_a_live_multiprocess_wal_writer() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + let writer = RepositoryAgentTraceDb::new_at(&path).await.expect("writer"); + writer + .verify_or_initialize_repository_metadata("repo") + .await + .expect("metadata"); + + let (reader, metadata) = RepositoryAgentTraceDb::open_verified_existing_at(&path, "repo") + .await + .expect("verified open alongside live writer"); + assert_eq!(metadata.repository_id, "repo"); + + writer + .execute( + "INSERT INTO mutation_trace_worktrees + (worktree_id, cursor_tree, revision, tainted, failure_kind, needs_rebaseline) + VALUES ('main', 'tree-after-open', x'0000000000000000', 0, 'healthy', 0)", + (), + ) + .await + .expect("writer still writes"); + let rows = reader + .query_map( + "SELECT cursor_tree FROM mutation_trace_worktrees", + (), + |row| row.get::(0).map_err(Into::into), + ) + .await + .expect("reader sees committed writer data"); + assert_eq!(rows, vec!["tree-after-open".to_string()]); + assert!(with_suffix(&path, "-tshm").exists()); +} diff --git a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs index 03a5e8639..66363b37e 100644 --- a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs +++ b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs @@ -202,297 +202,3 @@ where deleted: stale.len(), })) } - -#[cfg(test)] -mod tests { - use std::path::Path; - use std::process::Command; - use std::time::Duration; - - use super::super::git_snapshot::{resolve_git_dir, GitSnapshotService}; - use super::super::maintenance_state::{ - read_state, state_path, write_state_atomically, StateRead, - }; - use super::super::ref_maintenance::{reconcile_explicit_with, ExplicitOutcome, SkipReason}; - use super::super::worktree_lock::WorktreeLock; - use super::{ReconcileError, ReconciliationReport}; - use crate::services::agent_trace_db::repository::{ - ExistingRepositoryDbError, RepositoryAgentTraceDb, - }; - use crate::services::mutation_trace::types::WorktreeId; - - const NOW: i64 = 1_000_000_000_000; - const REPOSITORY_ID: &str = "repo-under-test"; - - fn git(cwd: &Path, args: &[&str]) { - let output = Command::new("git") - .args(args) - .current_dir(cwd) - .output() - .expect("git command"); - assert!(output.status.success(), "git {args:?} failed"); - } - - fn init_repo(root: &Path) { - std::fs::create_dir_all(root).expect("repo dir"); - git(root, &["init", "-q"]); - git(root, &["config", "user.email", "t@example.invalid"]); - git(root, &["config", "user.name", "T"]); - git(root, &["config", "commit.gpgsign", "false"]); - std::fs::write(root.join("tracked.txt"), "tracked\n").expect("write file"); - git(root, &["add", "tracked.txt"]); - git(root, &["commit", "-q", "-m", "initial"]); - } - - async fn create_db(path: &Path, repository_id: &str) { - let db = RepositoryAgentTraceDb::new_at(path) - .await - .expect("create db"); - db.verify_or_initialize_repository_metadata(repository_id) - .await - .expect("initialize metadata"); - } - - async fn pin_orphan(root: &Path) -> String { - let snapshot = GitSnapshotService::new(root).await.expect("snapshot"); - let tree = snapshot.capture_tree().await.expect("tree"); - let git_dir = resolve_git_dir(root).await.expect("git dir"); - let lock = WorktreeLock::acquire_async(&git_dir, Duration::from_secs(10)) - .await - .expect("lock"); - snapshot - .pin_tree(lock.lease(), &WorktreeId("main".to_string()), &tree) - .await - .expect("pin"); - tree.0 - } - - fn write( - path: &Path, - state: &super::super::maintenance_state::MaintenanceState, - ) -> std::io::Result<()> { - write_state_atomically(path, state, |from, to| std::fs::rename(from, to)) - } - - #[tokio::test(flavor = "multi_thread")] - async fn ref_reconciliation_explicit_pass_deletes_orphan_and_records_success() { - let dir = tempfile::tempdir().expect("temp dir"); - let root = dir.path().join("repo"); - init_repo(&root); - let db_path = dir.path().join("state").join("agent-trace.db"); - std::fs::create_dir_all(db_path.parent().expect("parent")).expect("db dir"); - create_db(&db_path, REPOSITORY_ID).await; - pin_orphan(&root).await; - - let mut phases = Vec::new(); - let outcome = reconcile_explicit_with( - &root, - async || { - RepositoryAgentTraceDb::open_verified_existing_at(&db_path, REPOSITORY_ID) - .await - .map(|(db, _)| db) - .map_err(anyhow::Error::new) - }, - || NOW, - write, - |phase| phases.push(phase), - Duration::from_secs(10), - ) - .await; - - let ExplicitOutcome::Completed(report) = outcome else { - panic!("expected completed pass, got {outcome:?}"); - }; - assert_eq!( - report, - ReconciliationReport { - local_required: 0, - retained: 0, - deleted: 1, - } - ); - assert_eq!(phases.len(), 2); - - let snapshot = GitSnapshotService::new(&root).await.expect("snapshot"); - assert!(snapshot - .list_pins(&WorktreeId("main".to_string())) - .await - .expect("pins") - .is_empty()); - - let git_dir = resolve_git_dir(&root).await.expect("git dir"); - let StateRead::Valid(state) = read_state(&state_path(&git_dir)).expect("state") else { - panic!("expected recorded state"); - }; - assert_eq!(state.last_success, Some(NOW)); - assert_eq!(state.consecutive_failures, 0); - } - - #[tokio::test(flavor = "multi_thread")] - async fn ref_reconciliation_explicit_pass_failure_keeps_refs_and_counts_streak() { - let dir = tempfile::tempdir().expect("temp dir"); - let root = dir.path().join("repo"); - init_repo(&root); - let missing_db = dir.path().join("absent").join("agent-trace.db"); - let tree = pin_orphan(&root).await; - - for expected_streak in 1..=2 { - let outcome = reconcile_explicit_with( - &root, - async || { - RepositoryAgentTraceDb::open_verified_existing_at(&missing_db, REPOSITORY_ID) - .await - .map(|(db, _)| db) - .map_err(anyhow::Error::new) - }, - || NOW, - write, - |_| {}, - Duration::from_secs(10), - ) - .await; - assert!(matches!( - outcome, - ExplicitOutcome::Failed(ReconcileError::AgentTraceDbUnavailable(_)) - )); - - let git_dir = resolve_git_dir(&root).await.expect("git dir"); - let StateRead::Valid(state) = read_state(&state_path(&git_dir)).expect("state") else { - panic!("expected recorded state"); - }; - assert_eq!(state.consecutive_failures, expected_streak); - assert_eq!(state.last_success, None); - } - - let snapshot = GitSnapshotService::new(&root).await.expect("snapshot"); - let pins = snapshot - .list_pins(&WorktreeId("main".to_string())) - .await - .expect("pins"); - assert_eq!(pins.len(), 1); - assert_eq!(pins[0].tree.0, tree); - assert!(!missing_db.exists()); - assert!(!missing_db.parent().expect("parent").exists()); - } - - #[tokio::test(flavor = "multi_thread")] - async fn ref_reconciliation_explicit_pass_skips_when_lock_is_held_without_touching_state() { - let dir = tempfile::tempdir().expect("temp dir"); - let root = dir.path().join("repo"); - init_repo(&root); - let git_dir = resolve_git_dir(&root).await.expect("git dir"); - let held = WorktreeLock::acquire_async(&git_dir, Duration::from_secs(10)) - .await - .expect("hold lock"); - - let outcome = reconcile_explicit_with( - &root, - async || -> anyhow::Result { unreachable!("db must not open") }, - || NOW, - write, - |_| {}, - Duration::ZERO, - ) - .await; - - assert!(matches!( - outcome, - ExplicitOutcome::Skipped(SkipReason::Busy) - )); - assert!(!state_path(&git_dir).exists()); - drop(held); - } - - #[tokio::test(flavor = "multi_thread")] - async fn ref_reconciliation_explicit_pass_reports_state_persist_failure_distinctly() { - let dir = tempfile::tempdir().expect("temp dir"); - let root = dir.path().join("repo"); - init_repo(&root); - let db_path = dir.path().join("agent-trace.db"); - create_db(&db_path, REPOSITORY_ID).await; - pin_orphan(&root).await; - - let outcome = reconcile_explicit_with( - &root, - async || { - RepositoryAgentTraceDb::open_verified_existing_at(&db_path, REPOSITORY_ID) - .await - .map(|(db, _)| db) - .map_err(anyhow::Error::new) - }, - || NOW, - |_, _| Err(std::io::Error::other("disk full")), - |_| {}, - Duration::from_secs(10), - ) - .await; - - let ExplicitOutcome::CompletedStatePersistFailed { report, warning } = outcome else { - panic!("expected persist-failed completion, got {outcome:?}"); - }; - assert_eq!(report.deleted, 1); - assert!(warning.contains("disk full")); - } - - #[tokio::test(flavor = "multi_thread")] - async fn ref_reconciliation_verified_opener_reports_typed_failures_without_writes() { - let dir = tempfile::tempdir().expect("temp dir"); - - let missing = dir.path().join("nested").join("agent-trace.db"); - let error = RepositoryAgentTraceDb::open_verified_existing_at(&missing, REPOSITORY_ID) - .await - .err() - .expect("missing db fails"); - assert!(matches!(error, ExistingRepositoryDbError::Missing { .. })); - assert!(!dir.path().join("nested").exists()); - - let valid = dir.path().join("valid.db"); - create_db(&valid, REPOSITORY_ID).await; - let (_, metadata) = - RepositoryAgentTraceDb::open_verified_existing_at(&valid, REPOSITORY_ID) - .await - .expect("valid db opens"); - assert_eq!(metadata.repository_id, REPOSITORY_ID); - - let error = RepositoryAgentTraceDb::open_verified_existing_at(&valid, "another-repo") - .await - .err() - .expect("mismatch fails"); - assert!(matches!( - error, - ExistingRepositoryDbError::RepositoryMismatch { .. } - )); - - let no_metadata = dir.path().join("no-metadata.db"); - RepositoryAgentTraceDb::new_at(&no_metadata) - .await - .expect("migrated db"); - let error = RepositoryAgentTraceDb::open_verified_existing_at(&no_metadata, REPOSITORY_ID) - .await - .err() - .expect("missing metadata fails"); - assert!(matches!(error, ExistingRepositoryDbError::MissingMetadata)); - let db = RepositoryAgentTraceDb::open_existing_without_migrations_at(&no_metadata) - .await - .expect("reopen"); - assert!(db - .verify_existing_repository_metadata(REPOSITORY_ID) - .await - .is_err()); - - let unmigrated = dir.path().join("unmigrated.db"); - drop( - RepositoryAgentTraceDb::open_without_migrations_at(&unmigrated) - .await - .expect("create empty db"), - ); - let error = RepositoryAgentTraceDb::open_verified_existing_at(&unmigrated, REPOSITORY_ID) - .await - .err() - .expect("incompatible schema fails"); - assert!(matches!( - error, - ExistingRepositoryDbError::IncompatibleSchema(_) - )); - } -} diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 677f0ad5b..92f40be8a 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -142,11 +142,11 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. - Two guarantee classes (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, failed state read or write, write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. - Validate: T05 S19–S25 and S34–S39 pass: single advice per window with persistence before emission; concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's failure leg assert byte-identical preexisting state instead; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write leaving the previous state unchanged; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. -- [ ] AC14: Reconciliation opens only a verified existing DB and never initializes, repairs, or migrates it; failure cases assert logical database immutability, and only the missing-DB case asserts no newly created database-related files. +- [ ] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability, and only the missing-DB case asserts no newly created database-related files. - **Missing DB:** fails before Turso is invoked; no database file, parent directory, or WAL/shared-memory file is created; no Git ref is listed or deleted. - **Existing but invalid DB:** opening is permitted; incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable and documented; no schema migration, migration-metadata repair, repository-metadata initialization, or application-table INSERT/UPDATE/DELETE; no Git-ref deletion; verification fails with the matching typed error. Tests do not assert filesystem immutability for this case. - **Existing valid DB:** repository identity and schema readiness are verified before reconciliation; the existing multiprocess-WAL configuration is preserved; only the access reconciliation requires is performed; no initialization, repair, or migration. - - Validate: T05 S28–S32 pass: missing DB (no new files or directories, Turso not invoked, refs not listed), incompatible schema, missing metadata, repository identity mismatch (refs untouched, logical DB contents byte-for-byte equivalent by table/row comparison, typed error), and normal existing-DB operation. + - Validate: T05 S28–S32 pass: missing DB (no new files or directories, non-creating open fails, refs not listed), incompatible schema, missing metadata, repository identity mismatch (refs untouched, logical DB contents byte-for-byte equivalent by table/row comparison, typed error), and normal existing-DB operation. - [ ] AC15: Deferred automatic reclamation is explicit and not implied. - Validate: the plan's D2 follow-up prerequisites are copied into the synchronized context by T06; no code path or doc claims automatic reclamation; T05 S14 proves hook completion never reconciles. @@ -271,21 +271,36 @@ Record actual outcomes in the validation report; compilation alone does not vali - Task ID: T02 - Scope: In — - **Reconciler split (D1):** extract the body of `reconcile_worktree_inner` after lock acquisition into `reconcile_with_held_lock`, which borrows the held `WorktreeLock` and contains no acquire call; keep `reconcile_worktree` and `reconcile_worktree_inner` behavior and signatures, now acquiring once and delegating. - - **`reconcile_explicit`:** resolves the worktree and Git dir, acquires the lock **once** (existing bounded 10 s wait), opens the DB through the D4 verified opener, runs `reconcile_with_held_lock`, then records the outcome in the state file while still holding the same lock, then drops the lock. Typed outcome `Completed(report) | CompletedStatePersistFailed(report, warning) | Failed(error) | Skipped(Busy)`; lock contention leaves the state untouched and is not a failure; it never calls the lock-acquiring wrapper; it ignores all prior state (stale, failed streak, invalid or corrupt) except to rewrite it. - - **`advise_if_due`:** synchronous function with no DB, Git, store, or snapshot parameters; zero-wait single `try_lock`; reads, normalizes, and writes the state under that one acquisition; returns `Anchored | NoAction | Advised | Busy | StateUnavailable`; persists `last_advised` before returning `Advised`; takes injected wall clock and Git dir. Subject to the T01 decision rule. + - **`reconcile_explicit(repository_root)`:** takes no database opener; resolves the worktree and Git dir, acquires the lock **once** (existing bounded 10 s wait), resolves the repository identity and canonical Agent Trace path itself and opens the DB only through the D4 verified read-only opener, runs `reconcile_with_held_lock`, then records the outcome in the state file while still holding the same lock, then drops the lock. Typed outcome `Completed(report) | CompletedStatePersistFailed { report, warning } | Failed { error, state_warning } | Skipped(Busy)` (a failed pass keeps both the reconciliation error and any state-persistence warning); lock contention leaves the state untouched and is not a failure; it never calls the lock-acquiring wrapper; it ignores all prior state (stale, failed streak, invalid or corrupt) except to rewrite it. + - **`advise_if_due`:** synchronous function with no DB, Git, store, or snapshot parameters; zero-wait single `try_lock`; reads, normalizes, and writes the state under that one acquisition; returns `Anchored | NoAction | Advised | AdvisedDurabilityUncertain { warning } | Busy | StateWriteFailed { warning } | StateUnavailable` (see the second remediation); persists `last_advised` before returning `Advised`; takes injected wall clock and Git dir. Subject to the T01 decision rule. - **State module (D3):** `ref-maintenance.json` read (capped, default-on-corrupt), invalid-timestamp normalization under lock, atomic write, past-only fields, success/failure transitions, and a pure `evaluate_recommendation(state, now)` function shared by the advisory and read-only doctor. - **D4 opener:** the dedicated verified-existing-DB opener and the read-only repository metadata verification method. - **Test seams only as required:** a worker-entered seam threaded through `delete_pins` to `run_ref_mutation_inner`'s existing `on_worker_entered`; phase-boundary hooks (post-lock, post-DB-open, post-inventory) used to park a pass deterministically; injectable write-failure and rename-interruption hooks for the atomic writer; injected clock. Out — hook and doctor wiring, any new deletion code, any automatic/reservation/backoff scheduling, protocol/store schema changes, changes to the existing hook DB opener, capped inventory or budget logic. - Dependencies: T01 - - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a deadline; after every successful state write no stored timestamp exceeds `now + FUTURE_SKEW_TOLERANCE`; timestamps within the tolerance are accepted as age 0 and preserved; on a successful write invalid timestamps normalize once and persist; read-only, lock-busy, and failed-read/write operations never mutate state, evaluate invalid timestamps conservatively, and never postpone eligibility indefinitely; failures leave refs untouched; the DB opener fails a missing DB before Turso is invoked without creating files or directories, and for an existing invalid DB performs no migration, repair, initialization, or application-table write (incidental WAL bookkeeping permitted and documented); no second deletion path exists. + - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a deadline; after every successful state write no stored timestamp exceeds `now + FUTURE_SKEW_TOLERANCE`; timestamps within the tolerance are accepted as age 0 and preserved; on a successful write invalid timestamps normalize once and persist; read-only, lock-busy, and failed-read/write operations never mutate state, evaluate invalid timestamps conservatively, and never postpone eligibility indefinitely; failures leave refs untouched; the DB opener fails a missing DB at its single non-creating read-only open without creating files or directories, and for an existing invalid DB performs no migration, repair, initialization, or application-table write (incidental WAL bookkeeping permitted and documented); no second deletion path exists. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml ref_reconciliation`; focused maintenance-state and `agent_trace_storage` unit tests (deterministic, no sleeps); `rg` confirms the only ref-deletion call is the existing `delete_pins`, each entrypoint acquires `WorktreeLock` once, `reconcile_with_held_lock` contains no acquire call, and the advisory module imports no Git/DB/store symbol. - Completed: 2026-10-09 - Files changed: `cli/src/services/agent_trace_db/repository.rs`, `cli/src/services/agent_trace_storage/mod.rs`, `cli/src/services/db/mod.rs`, `cli/src/services/mutation_trace/runtime/mod.rs`, `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs`, `cli/src/services/mutation_trace/runtime/maintenance_state.rs`, `cli/src/services/mutation_trace/runtime/ref_advisory.rs`, `cli/src/services/mutation_trace/runtime/ref_maintenance.rs`, `context/plans/mutation-cursor-ref-reconciliation-wiring.md` - Result: Split `reconcile_worktree_inner` into lock acquisition plus `reconcile_with_held_lock` (borrows `&WorktreeLock`, phase-hook seam `ReconcilePhase`); added `ref_maintenance::reconcile_explicit[_with]` (single lock acquisition, typed `Completed | CompletedStatePersistFailed | Failed | Skipped(Busy)`), `ref_advisory::advise_if_due[_with]` (sync, zero-wait `try_lock`, filesystem Git-dir derivation, no Git/DB/store imports), and `maintenance_state` (capped versioned `ref-maintenance.json`, past-only timestamps, tolerance-based normalization under lock, staging-and-swap writes with injectable rename, pure `evaluate_recommendation`). Added the D4 opener `RepositoryAgentTraceDb::open_verified_existing_at` with `ExistingRepositoryDbError` (`Missing | Unreadable | IncompatibleSchema | MissingMetadata | RepositoryMismatch`), read-only `verify_existing_repository_metadata`, `TursoDb::open_existing_without_migrations_at` (no parent-dir creation), and `resolve_existing_agent_trace_storage_for_maintenance`. New items carry dead-code allowances until T03/T04 wire them. The existing hook opener and its callers are unchanged; `delete_pins` remains the only deletion path. - - Verify: `cargo clippy --all-targets --all-features -D warnings` clean; `test … ref_reconciliation` 5 passed; `test … maintenance` 8 passed; `test … mutation_trace` 123 passed; `rg` shows `reconcile_with_held_lock` has no acquire call and `maintenance_state.rs` has no Git/DB/store symbol (`ref_advisory.rs` matches only `Command` inside its test module); `git diff --check` clean. The `agent_trace_storage` filter matches zero tests, so opener coverage lives in the `ref_reconciliation` tests. Turso 0.8.1 read-only open was not used: the opener pre-checks file existence and only issues SELECT-class statements after open (incidental WAL bookkeeping accepted per D4). + - Verify: `cargo clippy --all-targets --all-features -D warnings` clean; `test … ref_reconciliation` 5 passed; `test … maintenance` 8 passed; `test … mutation_trace` 123 passed; `rg` shows `reconcile_with_held_lock` has no acquire call and `maintenance_state.rs` has no Git/DB/store symbol (`ref_advisory.rs` matches only `Command` inside its test module); `git diff --check` clean. The `agent_trace_storage` filter matches zero tests, so opener coverage lives in the `ref_reconciliation` tests. Original T02 verification used an existence pre-check; superseded by the remediation below. - Context impact: important (new runtime maintenance entrypoints, state file, and verified-existing-DB opener; T06 owns the durable context update). - Context synchronization: synced + - Remediation (2026-10-09, PR #305 review of `d699b26`; T02 reopened `done`→`in_progress`, returned to `done`/`synced` after the verification below; T01 unchanged, T03–T06 untouched): + - P1 authoritative DB: the production entrypoint is now `reconcile_explicit(repository_root: &Path) -> ExplicitOutcome`. It resolves the repository identity and canonical path itself (`resolve_agent_trace_storage_runtime_config` → `resolve_existing_agent_trace_storage_for_maintenance`) and opens only through `open_verified_existing_at`, which checks schema/all migrations, metadata presence, stored-vs-resolved repository identity, and `source_instance_id` before any pin inventory. `reconcile_explicit_with` (the opener-injection seam) is private to `ref_maintenance` and used only by its child test module; T04 must call `reconcile_explicit` and must not add an opener. The single lock-held algorithm and `reconcile_worktree` wrapper are unchanged. Typed `ExistingRepositoryDbError` stays downcastable from `ReconcileError::AgentTraceDbUnavailable`. + - P2 persistence diagnostics: `ExplicitOutcome::Failed { error, state_warning }` retains the reconciliation error and an optional `StatePersistWarning` (`PreviousStateUnreadable | WriteFailed(PersistFailure)`); `CompletedStatePersistFailed` carries the same warning type. Lock contention returns `Skipped(Busy)` before any state read or write. A completed ref deletion is never rolled back. + - P2 durability: `write_state_atomically` returns `PersistFailure::NotApplied { phase, source, staging_cleanup }` (previous state intact, staging removed, cleanup failure reported) for failures up to and including rename, and `PersistFailure::DurabilityUncertain { phase, source }` for parent-directory open/sync failures after rename (new state may already be visible; previous state is not claimed preserved). The filesystem seam is the statically dispatched `StateFilesystem` trait with `RealStateFilesystem`; `FaultInjectingFilesystem` is test-only. `advise_if_due` maps any `PersistFailure` to `StateUnavailable`, so `Advised` is never reported without a confirmed write. + - P2 existing-DB race: Turso 0.8.1 has no create-disabled builder flag, but `Builder::read_only(true)` opens the main file with `OpenFlags::ReadOnly`; the Unix/generic IO layers then open it without `create` and without write access, so existence and open are one OS call (missing → not-found, no file or directory created). `open_existing_without_migrations_at` now uses it with `experimental_multiprocess_wal(true)` unchanged; the `is_file()` pre-check is removed and a failed open is classified `Missing` only by a post-failure `symlink_metadata` NotFound. All verification queries run on the connection that open produced. + - Limitations (not claimed solved): (1) Turso opens the `-wal`/`-tshm` sidecars by path, so a replacement that swaps only the main file leaves a torn main+stale-WAL pairing whose contents are whatever Turso assembles; only a coherent replacement (main file and sidecars) is meaningful, and the check proves the connection's data matches the resolved repository, not that the path was not swapped later. (2) Turso's process-wide registry is keyed by file identity: if the same process already holds that DB open, the open shares that instance and the read-only flag is not enforced for it (the file necessarily exists, so nothing is created). (3) A DB replaced after verification but before the delete transaction is not detected; the guard is that roots are read from the verified connection under the held `WorktreeLock`. (4) Cross-process multiprocess-WAL coexistence is covered only by the builder configuration and an in-process live-writer test, not by a second OS process. + - Evidence: `test … ref_reconciliation` 20 passed (all in `ref_maintenance/tests.rs`, including: signature-only production API, other-repository DB cannot authorize deletion, mismatched/missing metadata and missing/incompatible schema fail before inventory (`phases` empty, pins retained), missing DB creates nothing, removed/replaced-before-open, removed-after-open never recreated, read-only connection, live-writer coexistence, lock-held path never reacquires, wrapper acquires once and releases, failure+persist-failure keeps both errors for missing DB and missing required pin, durability-uncertain completion keeps deletion, lock contention never increments the streak); `test … maintenance` 38 passed (`maintenance_state` tests cover staging create/write/sync, rename, parent open/sync failures, success, interrupted pre-rename recovery, post-rename visibility vs durability); `test … mutation_trace` 148 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; no new `allow`, `dyn`, `let _ =`, or comments; `acquire_inner_async` appears once in `ref_maintenance.rs` and once in `reconcile_worktree_inner`. + - Remediation 2 (2026-10-09, follow-up review of `de055d70`; T02 reopened `done`→`in_progress`, returned to `done`/`synced` after the verification below; T01 unchanged and still done/synced, T03–T06 untouched). Architecture unchanged: reconciliation runs only through `reconcile_explicit`; the advisory stays lightweight (no Git, DB, or reconciliation), takes one zero-wait lock acquisition, uses static dispatch, and no mutation protocol, attribution, Quint, or ref-deletion code changed. + - P2 failure-based recommendation: `evaluate_recommendation` now treats an outstanding explicit-pass failure (`last_attempt_outcome == Failed` or `consecutive_failures > 0`) as an independent reason to recommend repair, in addition to staleness and invalid timestamps. A first-ever failed pass (no anchor, no `last_success`) is therefore recommended immediately. A never-attempted default state is not a failure. Multiple consecutive failures stay recommended; `record_success` clears the streak, `last_failure`, and the recommendation; a failure after a recent success is recommended again. Emitting an advisory never resets `consecutive_failures` or `last_failure`. + - Recommendation status vs reminder cadence: `Recommendation.recommended` answers "should `sce doctor` tell the operator to run `sce doctor --fix`" and ignores `last_advised`. `Evaluation.advise` answers "may the hook emit another reminder": staleness reminders are suppressed for 24 h after `last_advised`; a failure reminder is suppressed only when `last_advised` is within the window and not older than `last_attempt`, so a newly recorded failure can be advised immediately even if an earlier staleness reminder is recent, while a reminder issued after the failure is not repeated inside the window. A failure reminder repeats every 24 h until a successful explicit pass. Clock-skew normalization and the existing cadence are unchanged. + - P2 persistence outcomes: `AdvisoryOutcome` is now `Anchored | NoAction | Advised | AdvisedDurabilityUncertain { warning: PersistFailure } | Busy | StateWriteFailed { warning: PersistFailure } | StateUnavailable` (no longer `Copy`/`Eq` because it carries `PersistFailure`). `Advised` means the reminder state was persisted with confirmed durability. `PersistFailure::NotApplied` (up to and including rename) on an advisory yields `StateWriteFailed`: the previous state is intact, nothing is claimed persisted, `recommendation()` is `None`, and the next eligible call advises again. `PersistFailure::DurabilityUncertain` (parent-directory open or sync after rename) yields `AdvisedDurabilityUncertain`: the rename completed, `last_advised` may be visible, and the caller must still surface the recommendation. No rollback, second lock acquisition, or automatic reconciliation. A write failure while only anchoring or normalizing yields `StateWriteFailed` carrying whichever `PersistFailure` variant occurred; there is nothing to surface. + - Caller contract for T03: `AdvisoryOutcome::recommendation()` returns `Reconciliation is recommended. Run sce doctor --fix.` for `Advised` and `AdvisedDurabilityUncertain`, and `None` otherwise; `persistence_warning()` returns the `PersistFailure` separately for logging. Outcomes are diagnostic values, not errors: T03 must log them and never convert them into a mutation-boundary failure. Suggested levels: `Advised` and `AdvisedDurabilityUncertain` (with the persistence warning as a distinct field) at warn; `StateWriteFailed` and `StateUnavailable` at warn; `Anchored`/`NoAction`/`Busy` at debug. + - Delivery semantics and limitations: a durable state update is not the same as an operator-visible advisory. The state is written before the caller surfaces the message, so a process crash between the write and the log leaves `last_advised` set and the reminder unseen for up to 24 h; exactly-once delivery is not promised. `sce doctor` is the backstop because it evaluates the recommendation from the failure/staleness fields independently of `last_advised`, so an unresolved failure or stale state is never hidden from plain `sce doctor`. After `AdvisedDurabilityUncertain`, a crash can still roll the directory entry back, in which case the next eligible call simply advises again. + - Regression scenarios and acceptance criteria (all deterministic, injected clocks and `FaultInjectingFilesystem`, no sleeps): first-ever failed pass with missing DB is recommended (unit and end-to-end through `reconcile_explicit`); failure advisory emits once, persists `last_advised`, keeps the failure diagnostics, and a second advisory inside the window is `NoAction`; plain evaluation stays recommended despite the recent reminder; the reminder repeats after the window; repeated failures stay recommended; success clears; a failure after a recent success and an older reminder is recommended and advised; rename failure preserves prior state and the next call advises; parent-open and parent-sync failures yield `AdvisedDurabilityUncertain` with the new state visible, the recommendation surfaced, a following call `NoAction`, and `evaluate_recommendation` still true; an anchor write failure is `StateWriteFailed`; confirmed advisories keep the ordinary duplicate-suppression tests. + - Evidence: `test … maintenance` 48 passed; `test … ref_reconciliation` 21 passed; `test … mutation_trace` 158 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; no added comments, `dyn`, `let _ =`, or new `allow`; no new `acquire` call sites. - [ ] T03: `Trigger the advisory check from the shared mutation-scope ingress` (status:todo) - Task ID: T03 @@ -311,7 +326,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) clock-skew tolerance, with an injected clock for every case (`FUTURE_SKEW_TOLERANCE` = 5 min): (1) stored `last_success` equal to `now` — valid, age 0, no advice, file byte-identical; (2) one minute ahead of `now` — valid, age 0, preserved, no advice, no rewrite; (3) exactly five minutes ahead — valid (boundary is inclusive), age 0, preserved, no advice, no rewrite; (4) more than five minutes ahead (just beyond, and far-future) for `last_success`, `last_advised` and `anchor` — invalid, normalized once under the lock and persisted per D3 (invalid fields cleared; with `last_success` and `anchor` both unusable the advisory fires immediately once, persists `last_advised = now` and a fresh `anchor = now`); (5) clock rollback beyond tolerance (stored values valid at write time, injected `now` moved back more than five minutes) — treated as case 4; a rollback within tolerance is case 2/3 and never makes an advisory eligible by itself; (6) repeated reads after normalization — the file is byte-identical across repeated calls, no rewrite occurs, and eligibility is neither postponed nor slid; (7) process restart (fresh call re-reading the file) followed by clock advance past `RECONCILIATION_ADVISORY_AFTER` — advice recurs exactly once, proving eventual eligibility. After every **successful state-writing** operation an assertion checks the exact invariant `t ≤ now + FUTURE_SKEW_TOLERANCE` for every stored timestamp (not "no timestamp is in the future"); after read-only, lock-busy, or failed operations the assertion is instead that the state file is byte-identical to its prior content, with no normalization claimed; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next successful recorded operation (until then, read-only and failed operations leave them untouched and treat them as default state). - *Timestamp-normalization regressions (P2; injected clocks and deterministic synchronization, no sleeps):* (S34) read-only doctor observing an invalid future timestamp — plain `sce doctor` reports "recommended" with the invalid-timestamp note, the state file is byte-identical afterwards, and it is not reported as normalized; (S35) lock-busy advisory — with the lock held through a barrier and an invalid future timestamp stored, `advise_if_due` returns `Busy` and the state file is byte-identical; the same holds for a lock-busy explicit pass (`Skipped(Busy)`); (S36) failed state write — an injected write error (and, in a second parameterization, a rename interrupted before replacement) during a normalizing operation leaves the previous state byte-identical and readable, emits no advice, reports no normalization (`StateUnavailable` for the advisory, `CompletedStatePersistFailed` for the explicit pass), and a later successful operation still normalizes it; (S37) successful normalization — a normalizing advisory and a normalizing explicit pass each persist state in which every timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`, valid near-future values (within tolerance) are preserved byte-for-byte, and a repeated read performs no rewrite and does not move eligibility forward; (S38) process restart after normalization — a fresh call re-reading the persisted file observes the normalized timestamps, performs no rewrite, and does not re-fire the immediate advice; (S39) eventual eligibility — with invalid timestamps unnormalized (S34/S35/S36 states) repeated read-only evaluations never push eligibility out, and after the injected clock advances past `RECONCILIATION_ADVISORY_AFTER` a successful advisory fires exactly once and persists normalized state. - *Explicit recovery:* (S26) `sce doctor --fix` runs and reports its true result from each of: failed streak, stale advice, invalid timestamps, corrupt state, unwritable state (→ `CompletedStatePersistFailed`); (S27) explicit success resets the failure streak and sets `last_success`; explicit failure increments the streak without touching `last_success`; lock-busy never changes state. - - *DB opening (D4):* (S28) missing DB — fails with the `Missing` typed error before Turso is invoked (a spy/seam on the Turso open is never called); no database file, parent directory, WAL, or shared-memory file is created (directory listing before and after is identical, including when the parent directory is absent); Git refs are neither listed nor deleted (spy on `list_pins`/`delete_pins` never called); (S29) existing DB with incompatible schema — opening is permitted; fails with `IncompatibleSchema`; no migration or repair row written (migration table rows identical before and after); no ref deleted; (S30) existing DB with missing repository metadata — fails with `MissingMetadata`; no `repository_metadata` row inserted; (S31) existing DB with repository identity mismatch — fails with `RepositoryMismatch`; no row changed; (S32) existing valid DB — identity and schema readiness are verified before any ref listing (ordering asserted via seam), the multiprocess-WAL configuration is preserved, the pass completes, and no initialization, repair, or migration runs. For S29–S31 the assertions are logical: a before/after dump of every table's rows is equal (no migration, repair, initialization, or application-table INSERT/UPDATE/DELETE), refs are unchanged, and the external-taint marker is unarmed. Filesystem immutability is **not** asserted for existing-DB failure cases, because incidental Turso WAL/shared-memory bookkeeping is permitted when unavoidable; only S28 asserts that no database-related file or directory is created. + - *DB opening (D4):* (S28) missing DB — fails with the `Missing` typed error from the single non-creating read-only open (no existence pre-check); no database file, parent directory, WAL, or shared-memory file is created (directory listing before and after is identical, including when the parent directory is absent); Git refs are neither listed nor deleted (spy on `list_pins`/`delete_pins` never called); (S29) existing DB with incompatible schema — opening is permitted; fails with `IncompatibleSchema`; no migration or repair row written (migration table rows identical before and after); no ref deleted; (S30) existing DB with missing repository metadata — fails with `MissingMetadata`; no `repository_metadata` row inserted; (S31) existing DB with repository identity mismatch — fails with `RepositoryMismatch`; no row changed; (S32) existing valid DB — identity and schema readiness are verified before any ref listing (ordering asserted via seam), the multiprocess-WAL configuration is preserved, the pass completes, and no initialization, repair, or migration runs. For S29–S31 the assertions are logical: a before/after dump of every table's rows is equal (no migration, repair, initialization, or application-table INSERT/UPDATE/DELETE), refs are unchanged, and the external-taint marker is unarmed. Filesystem immutability is **not** asserted for existing-DB failure cases, because incidental Turso WAL/shared-memory bookkeeping is permitted when unavoidable; only S28 asserts that no database-related file or directory is created. Out — production behavior changes beyond test seams strictly required. - Dependencies: T03, T04 - Done when: all scenarios pass reliably, each fails when the guarded behavior is deliberately broken, no test depends on timing or sleeps, and every review finding in the traceability table maps to at least one passing scenario. From 8ae186a8f06a17b9f739b53f95269f37e481fcdc Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 01:26:58 +0200 Subject: [PATCH 05/16] runtime+hooks: Trigger ref reconciliation advisories after completed scopes Completed mutation-scope boundaries now run the advisory-only ref reconciliation check after coordination returns, while preserving hook results and avoiding reconciliation, Git, or database work on the hook path. The injected report is mapped to stable debug or warning logs, including durability and state failures, and is limited to Close and Flush boundaries. Ref: context/plans/mutation-cursor-ref-reconciliation-wiring.md (T03) Co-authored-by: SCE --- cli/src/services/hooks/mutation_scope.rs | 205 +++++++++++++++++- .../services/mutation_trace/runtime/mod.rs | 1 + .../mutation_trace/runtime/ref_advisory.rs | 50 +++++ context/cli/mutation-scope-hook-ingress.md | 1 + context/cli/mutation-scope-ref-advisory.md | 47 ++++ context/context-map.md | 1 + ...tation-cursor-ref-reconciliation-wiring.md | 9 +- 7 files changed, 305 insertions(+), 9 deletions(-) create mode 100644 context/cli/mutation-scope-ref-advisory.md diff --git a/cli/src/services/hooks/mutation_scope.rs b/cli/src/services/hooks/mutation_scope.rs index 211b496b0..4995e533d 100644 --- a/cli/src/services/hooks/mutation_scope.rs +++ b/cli/src/services/hooks/mutation_scope.rs @@ -6,8 +6,9 @@ use serde_json::{json, Map, Value}; use crate::services::agent_trace_db::repository::RepositoryAgentTraceDb; use crate::services::mutation_trace::runtime::{ - abandon_scope, arm_external_mutation_guard, coordinate, AbandonScopeError, AbandonScopeOutcome, - CoordinateError, CoordinateOutcome, GuardEvent, GuardRequest, RuntimeBoundary, StartProvenance, + abandon_scope, advise_after_completed_boundary, arm_external_mutation_guard, coordinate, + AbandonScopeError, AbandonScopeOutcome, AdvisoryReport, AdvisorySeverity, CoordinateError, + CoordinateOutcome, GuardEvent, GuardRequest, RuntimeBoundary, StartProvenance, }; use crate::services::mutation_trace::types::{ActorKind, EventId, ScopeId}; @@ -296,18 +297,21 @@ where }) .await }, + advise_after_completed_boundary, ) .await } -async fn drive_mutation_scope( +async fn drive_mutation_scope( repository_root: &Path, payload: MutationScopePayload, logger: Option<&L>, coordinate_boundary: C, abandon: A, + advise: V, ) -> Result where + V: FnOnce(&Path) -> AdvisoryReport, C: std::ops::AsyncFnOnce( &Path, &RuntimeBoundary, @@ -353,10 +357,53 @@ where } }; - classify_coordinate( - coordinate_boundary(repository_root, &boundary).await, - logger, - ) + let advisory_eligible = matches!( + boundary, + RuntimeBoundary::Close { .. } | RuntimeBoundary::Flush + ); + let result = coordinate_boundary(repository_root, &boundary).await; + let durably_completed = matches!( + result, + Ok(_) | Err(CoordinateError::MarkerClearAfterCommit { .. }) + ); + let classified = classify_coordinate(result, logger); + + if advisory_eligible && durably_completed { + log_advisory_report(logger, &advise(repository_root)); + } + + classified +} + +fn log_advisory_report( + logger: Option<&L>, + report: &AdvisoryReport, +) { + let Some(log) = logger else { + return; + }; + let message = report + .recommendation + .unwrap_or("Reconciliation advisory check."); + let warning = report.warning.as_deref().unwrap_or(""); + let mut fields = vec![("outcome", report.outcome)]; + if !warning.is_empty() { + fields.push(("persistence_warning", warning)); + } + match report.severity { + AdvisorySeverity::Warn => log.warn( + "sce.hooks.mutation_scope.ref_reconciliation_advisory", + message, + &fields, + None, + ), + AdvisorySeverity::Debug => log.debug( + "sce.hooks.mutation_scope.ref_reconciliation_advisory", + message, + &fields, + None, + ), + } } fn classify_coordinate( @@ -634,3 +681,147 @@ fn write_guard_line(writer: &mut W, line: &str) -> std::io::Result<()> writeln!(writer, "{line}")?; writer.flush() } + +#[cfg(test)] +mod advisory_trigger_tests { + use std::cell::Cell; + use std::path::Path; + + use anyhow::anyhow; + + use super::{ + drive_mutation_scope, AbandonScopeError, AbandonScopeOutcome, AdvisoryReport, + AdvisorySeverity, CoordinateError, CoordinateOutcome, MutationScopePayload, + RuntimeBoundary, + }; + use crate::services::mutation_trace::protocol::CommitEvaluation; + use crate::services::mutation_trace::types::{ActorKind, ScopeId, TreeId, WorktreeId}; + use crate::services::observability::traits::NoopLogger; + + fn outcome() -> CoordinateOutcome { + CoordinateOutcome { + worktree_id: WorktreeId("main".to_string()), + observed_tree: TreeId("tree".to_string()), + revision: 1, + evaluation: CommitEvaluation::default(), + mutation_event: None, + } + } + + fn report(outcome: &'static str, severity: AdvisorySeverity) -> AdvisoryReport { + AdvisoryReport { + outcome, + severity, + recommendation: None, + warning: None, + } + } + + fn close() -> MutationScopePayload { + MutationScopePayload::Close { + scope_id: "scope".to_string(), + event_id: "event".to_string(), + actor_kind: ActorKind::Codex, + } + } + + fn start() -> MutationScopePayload { + MutationScopePayload::Start { + scope_id: "scope".to_string(), + event_id: "event".to_string(), + actor_kind: ActorKind::Codex, + provenance: None, + } + } + + fn advance() -> MutationScopePayload { + MutationScopePayload::Advance { + scope_id: "scope".to_string(), + event_id: "event".to_string(), + actor_kind: ActorKind::Codex, + } + } + + async fn drive( + payload: MutationScopePayload, + coordinated: Result, + calls: &Cell, + ) -> anyhow::Result { + drive_mutation_scope( + Path::new("."), + payload, + None::<&NoopLogger>, + async |_root: &Path, _boundary: &RuntimeBoundary| coordinated, + async |_root: &Path, + _scope: &ScopeId| + -> Result { + Ok(AbandonScopeOutcome::Abandoned { + worktree_id: WorktreeId("main".to_string()), + scope: ScopeId("scope".to_string()), + revision: 1, + }) + }, + |_root: &Path| { + calls.set(calls.get() + 1); + report("no_action", AdvisorySeverity::Debug) + }, + ) + .await + } + + #[tokio::test] + async fn advisory_runs_once_after_completed_close_and_flush() { + let calls = Cell::new(0); + assert_eq!(drive(close(), Ok(outcome()), &calls).await.unwrap(), ""); + assert_eq!(calls.get(), 1); + assert_eq!( + drive(MutationScopePayload::Flush, Ok(outcome()), &calls) + .await + .unwrap(), + "" + ); + assert_eq!(calls.get(), 2); + } + + #[tokio::test] + async fn advisory_runs_after_marker_clear_after_commit_with_success_result() { + let calls = Cell::new(0); + let error = CoordinateError::MarkerClearAfterCommit { + source: anyhow!("clear failed"), + committed: Box::new(outcome()), + }; + assert_eq!(drive(close(), Err(error), &calls).await.unwrap(), ""); + assert_eq!(calls.get(), 1); + } + + #[tokio::test] + async fn advisory_never_runs_for_start_or_advance() { + let calls = Cell::new(0); + drive(start(), Ok(outcome()), &calls).await.unwrap(); + drive(advance(), Ok(outcome()), &calls).await.unwrap(); + assert_eq!(calls.get(), 0); + } + + #[tokio::test] + async fn advisory_never_runs_after_failed_coordinate() { + let calls = Cell::new(0); + let result = drive( + close(), + Err(CoordinateError::Other(anyhow!("boom"))), + &calls, + ) + .await; + assert!(result.is_err()); + assert_eq!(calls.get(), 0); + } + + #[tokio::test] + async fn advisory_never_runs_for_abandon() { + let calls = Cell::new(0); + let payload = MutationScopePayload::Abandon { + scope_id: "scope".to_string(), + }; + drive(payload, Ok(outcome()), &calls).await.unwrap(); + assert_eq!(calls.get(), 0); + } +} diff --git a/cli/src/services/mutation_trace/runtime/mod.rs b/cli/src/services/mutation_trace/runtime/mod.rs index a97020076..dbeea17bb 100644 --- a/cli/src/services/mutation_trace/runtime/mod.rs +++ b/cli/src/services/mutation_trace/runtime/mod.rs @@ -29,6 +29,7 @@ pub(crate) use mutation_attribution::{ BoundedMutationAttribution, MutationAttributionBarrier, MutationEventPageSource, TreeReadSource, MAX_MUTATION_ATTRIBUTION_EVENTS, }; +pub(crate) use ref_advisory::{advise_after_completed_boundary, AdvisoryReport, AdvisorySeverity}; #[allow(unused_imports)] pub(crate) use scope_runtime::{ abandon_scope, AbandonRecoveryReason, AbandonScopeError, AbandonScopeOutcome, diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs index 5a422349e..5f02d8573 100644 --- a/cli/src/services/mutation_trace/runtime/ref_advisory.rs +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -53,6 +53,56 @@ impl AdvisoryOutcome { } } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum AdvisorySeverity { + Debug, + Warn, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) struct AdvisoryReport { + pub(crate) outcome: &'static str, + pub(crate) severity: AdvisorySeverity, + pub(crate) recommendation: Option<&'static str>, + pub(crate) warning: Option, +} + +impl AdvisoryOutcome { + pub(crate) fn report(&self) -> AdvisoryReport { + let (outcome, severity) = match self { + AdvisoryOutcome::Anchored => ("anchored", AdvisorySeverity::Debug), + AdvisoryOutcome::NoAction => ("no_action", AdvisorySeverity::Debug), + AdvisoryOutcome::Busy => ("busy", AdvisorySeverity::Debug), + AdvisoryOutcome::Advised => ("advised", AdvisorySeverity::Warn), + AdvisoryOutcome::AdvisedDurabilityUncertain { .. } => { + ("advised_durability_uncertain", AdvisorySeverity::Warn) + } + AdvisoryOutcome::StateWriteFailed { .. } => { + ("state_write_failed", AdvisorySeverity::Warn) + } + AdvisoryOutcome::StateUnavailable => ("state_unavailable", AdvisorySeverity::Warn), + }; + AdvisoryReport { + outcome, + severity, + recommendation: self.recommendation(), + warning: self.persistence_warning().map(ToString::to_string), + } + } +} + +fn system_unix_time_ms() -> i64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .ok() + .and_then(|elapsed| i64::try_from(elapsed.as_millis()).ok()) + .unwrap_or(0) +} + +pub(crate) fn advise_after_completed_boundary(repository_root: &Path) -> AdvisoryReport { + advise_if_due(repository_root, system_unix_time_ms).report() +} + fn outcome_after_write( decision: AdvisoryDecision, write: Result<(), PersistFailure>, diff --git a/context/cli/mutation-scope-hook-ingress.md b/context/cli/mutation-scope-hook-ingress.md index 561fc6718..1f7dc40ee 100644 --- a/context/cli/mutation-scope-hook-ingress.md +++ b/context/cli/mutation-scope-hook-ingress.md @@ -258,4 +258,5 @@ The Codex mapping and partial coverage are in [`codex-mutation-scope-integration - [Mutation-trace scope abandonment](mutation-trace-scope-abandonment.md) - [Mutation-trace protected worktree](mutation-trace-protected-worktree.md) - [Mutation-scope provenance](mutation-scope-provenance.md) +- [Mutation-scope ref-reconciliation advisory](mutation-scope-ref-advisory.md) - [Agent Trace hooks command routing](../sce/agent-trace-hooks-command-routing.md) diff --git a/context/cli/mutation-scope-ref-advisory.md b/context/cli/mutation-scope-ref-advisory.md new file mode 100644 index 000000000..63aabf0a9 --- /dev/null +++ b/context/cli/mutation-scope-ref-advisory.md @@ -0,0 +1,47 @@ +# Mutation-scope ref-reconciliation advisory (hook trigger) + +The shared mutation-scope ingress (`cli/src/services/hooks/mutation_scope.rs`, +see [`mutation-scope-hook-ingress.md`](mutation-scope-hook-ingress.md)) emits a +lightweight, advisory-only signal that snapshot-ref reconciliation is +recommended. It never reconciles, lists refs, opens the Agent Trace DB, or spawns +Git. Reconciliation itself is operator-driven; this signal is only a reminder. + +## Trigger policy + +`drive_mutation_scope` takes an injected advisory seam (production: +`runtime::advise_after_completed_boundary`). The seam runs exactly once, and only +when all of these hold: + +- the boundary is `Close` or `Flush`; +- the `coordinate_boundary(...)` result is `Ok` or + `CoordinateError::MarkerClearAfterCommit` (a durably completed boundary). + +It runs after the coordinator future has returned, so the coordinator's +`WorktreeLock` is already released. It never runs for `Start`, `Advance`, +`Abandon`, or a failed coordinate. The hook result is computed before the seam +runs and is never altered by it: every advisory outcome is only logged as +`sce.hooks.mutation_scope.ref_reconciliation_advisory` with an `outcome` field +(`warn` for `advised`, `advised_durability_uncertain`, `state_write_failed`, +`state_unavailable`; `debug` for `anchored`, `no_action`, `busy`). The advisory +never creates, clears, or modifies the external-taint marker. + +## Advisory behavior + +`ref_advisory::advise_if_due` (`cli/src/services/mutation_trace/runtime/`) takes +no Git, DB, store, or snapshot parameters. It derives the Git dir from the +filesystem (no Git subprocess), makes one zero-wait `try_lock` of +`WorktreeLock`, and reads/writes the small per-worktree state file +`/sce/ref-maintenance.json` (capped, versioned, past-event timestamps +only, atomic staging-and-swap writes). It anchors on first use and then advises +at most once per `RECONCILIATION_ADVISORY_AFTER` window (24 h), persisting +`last_advised` before reporting `Advised`. A busy lock or unreadable state is +reported, never an error. + +## Limits + +- Time-based, not evidence-based: orphan detection needs the Git/DB scan this + path deliberately avoids. +- A reminder is not guaranteed exactly-once; a crash between the state write and + the log line can lose one. +- Orphan pins are reclaimed only by an explicit reconciliation pass. Automatic + reclamation is not provided. diff --git a/context/context-map.md b/context/context-map.md index 01c79e81d..ccd438123 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -42,6 +42,7 @@ Feature/domain context: - `context/cli/claude-mutation-scope-background-execution.md` (native Claude background Bash is operationally supported but unsupported for mutation attribution: `Bash(run_in_background=true)` is allowed in `PreToolUse` with no mutation scope, adapter attempt, lifecycle tracking, or attribution guarantee — effective record `context/decisions/2026-10-02-claude-background-bash-untracked.md` — its later terminal hooks are no-ops, and the accepted A/B contamination consequence is pinned by a regression; background `PowerShell` stays denied with the existing deny text; the T01 `2.1.284` probe result with lifecycle completion still unproven; and the separate self-detaching-descendant unsupported boundary D20 — Git-observable evidence that a foreground `setsid` command's detached descendant can still mutate the repository after `PostToolUse` returns — detail split out of `claude-mutation-scope-integration.md` for the repository's per-file line budget) - `context/cli/claude-mutation-scope-health.md` (the `doctor-mutation-scope-health` plan's Claude task, extended by the `doctor-mutation-scope-fix` plan's T04: `classify_health` in `cli/src/services/hooks/claude_mutation_scope/health.rs` maps `recovery_pending`/`attempts` onto `healthy | recovering | blocked | invalid` — `recovery_pending == false` is `healthy` regardless of live attempts; `recovery_pending == true` with `attempts` empty is `recovering` (the barrier's own next-`PreToolUse` flush clears it); `recovery_pending == true` with `attempts` non-empty is `blocked`, the exact incident shape this classifier exists to surface, since nothing in the ordinary hook lifecycle retries, removes, flushes, or clears from that shape; a read/parse failure is `invalid`. T04 adds a separate, `doctor`-invoked repair path (`assess_repairability`/`repair_blocked` in `health.rs`/`lifecycle.rs`, wired into `sce doctor --fix` by the same plan's T05) for the `blocked` shape: `AutoFixable` only when every currently persisted attempt is already `pending_abandon` (no established-abandon-intent `pending_start`/`active` attempt present); `repair_blocked` re-reads and re-proves that same condition inside one lock-protected read-only state transaction before releasing the lock and retrying each attempt's already-established seam `abandon` call independently, clearing the `recovery_pending` barrier only once no attempts remain — this classifier's own read-only behavior is unaffected by that repair path's existence) - `context/cli/mutation-trace-ref-reconciliation.md` (the conservative per-worktree snapshot-ref reconciliation pass in `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs`, built by the `mutation-cursor-ref-reconciliation` plan: `reconcile_worktree(repository_root, open_db)` → `pub(super) reconcile_worktree_inner(.., on_lock_contention)`, both returning `ReconciliationOutcome` (`Reconciled(ReconciliationReport { local_required, retained, deleted })` for a pass that ran | `SkippedNoCheckoutIdentity` — an `Ok`, not an `Err` — when no current checkout identity could be derived), a variant-per-fallible-step `ReconcileError` with no `Other`, and the module-owned `RECONCILIATION_LOCK_TIMEOUT`; the two-invariant model — a strictly per-worktree fail-closed local-consistency check via `load_tree_roots(W)` vs. a repository-wide deletion-safety set via `load_all_tree_roots()` so an `A`-owned ref is retained whenever any worktree still durably needs its tree — run entirely under the same `/sce/mutation-cursor.lock` `WorktreeLock` `coordinate()` holds, with the repository-wide read kept coherent by being one SQL statement / one DB snapshot rather than a repository-global lock; deletes only SCE-owned refs via one atomic `git update-ref --no-deref --stdin`, writes no `mutation_trace_*` row, never arms `ExternalTaintMarker`, runs no `git gc`; imperative durability maintenance below the verified Quint protocol; reclaims orphan/unreferenced refs only for the namespace of a checkout id a current worktree still derives — a namespace no current worktree owns (identity-based: a deleted linked worktree, or checkout-id metadata loss followed by `get_or_create_checkout_id` minting a fresh id on a still-present worktree) is beyond every per-worktree pass and left to a recorded future repository-scoped unowned-namespace operation, so the current pass does not bound all orphan-ref growth; `reconcile_worktree` has no `pub(crate)` re-export and no harness/command wiring yet) +- `context/cli/mutation-scope-ref-advisory.md` (the advisory-only hook trigger in `cli/src/services/hooks/mutation_scope.rs` built by T03 of the `mutation-cursor-ref-reconciliation-wiring` plan: a lightweight `advise_if_due` check after a durably completed `Close`/`Flush`, logged only, with the per-worktree `ref-maintenance.json` state; no reconciliation, Git, or DB work on the hook path) - `context/cli/mutation-trace-snapshot-service.md` (the isolated Git snapshot and ref-pinning service `runtime::git_snapshot::GitSnapshotService` in `cli/src/services/mutation_trace/runtime/git_snapshot.rs`: `new` resolving an absolute `git_dir`, `capture_tree` snapshotting staged/unstaged/untracked/deleted worktree state into the repository's normal object database via a throwaway temp index, `pin_tree` protecting a durable tree with a create-only idempotent **direct** `refs/sce/mutation-cursor//` ref, `diff_trees` emitting `patch.rs`-parseable raw diff text; plus the callerless reconciliation substrate — worktree-scoped `list_pins` inventory returning `Result, PinInventoryError>` that rejects a symbolic ref inside the namespace (mutation-cursor pins are direct refs) as `MalformedRef`, matchable separately from a `git for-each-ref` execution failure, and conditional-atomic `delete_pins` running one `git update-ref --no-deref --stdin` transaction of SHA-conditioned deletes — no-dereference so an inventory→delete direct-ref→symref race cannot escape the inventoried namespace, plus a fail-closed pre-check — that aborts whole if any ref changed since inventory; `REF_NAMESPACE` + `pin_ref_prefix` as the single source of truth for the pin path) - `context/cli/mutation-trace-external-taint.md` (the worktree-local mutation-cursor durability boundary in `cli/src/services/mutation_trace/runtime/external_taint.rs`, built by the `mutation-cursor-external-taint` plan: the `ExternalTaintMarker` primitive — `new(git_dir)`/`exists()`/`persist()`/`clear()` over an empty file at `/sce/mutation-cursor-tainted` whose existence is its entire state, `checkout::persist_checkout_id_inner`-style durability (`sync_data` plus best-effort `#[cfg(unix)]` parent-dir `sync_all`), idempotent persist/clear, `NotFound`-on-clear as success, no `Drop` deletion — as the concrete runtime refinement of the abstract `ProtocolState.external_taint`; armed by the reshaped `coordinate()` entrypoint write-ahead after the `WorktreeLock` and before Agent Trace DB acquisition (a caller-supplied DB provider closure), cleared only on a successful `CoordinateOutcome`, with dedicated fail-closed pre-commit `CoordinateError::ExternalTaintMarker` (`Inspect`/`Persist` only)/`AgentTraceDbUnavailable` variants plus a post-commit `MarkerClearAfterCommit { source, committed }` that carries the durable outcome so a failed trailing clear never hides a committed `MutationEvent`; an inherited marker seeds an invocation-local `external_taint_pending` flag that overlays `protocol::database_failure` onto each freshly loaded projection so `recover` runs once against the captured snapshot, held across a losing recovery CAS and cleared once it lands) - `context/sce/cli-exit-code-contract.md` (stable class-based `sce` process exit-code contract in `cli/src/app.rs`, so automation can branch on failure category without parsing error text) diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 92f40be8a..7377326a1 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -302,13 +302,18 @@ Record actual outcomes in the validation report; compilation alone does not vali - Regression scenarios and acceptance criteria (all deterministic, injected clocks and `FaultInjectingFilesystem`, no sleeps): first-ever failed pass with missing DB is recommended (unit and end-to-end through `reconcile_explicit`); failure advisory emits once, persists `last_advised`, keeps the failure diagnostics, and a second advisory inside the window is `NoAction`; plain evaluation stays recommended despite the recent reminder; the reminder repeats after the window; repeated failures stay recommended; success clears; a failure after a recent success and an older reminder is recommended and advised; rename failure preserves prior state and the next call advises; parent-open and parent-sync failures yield `AdvisedDurabilityUncertain` with the new state visible, the recommendation surfaced, a following call `NoAction`, and `evaluate_recommendation` still true; an anchor write failure is `StateWriteFailed`; confirmed advisories keep the ordinary duplicate-suppression tests. - Evidence: `test … maintenance` 48 passed; `test … ref_reconciliation` 21 passed; `test … mutation_trace` 158 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; no added comments, `dyn`, `let _ =`, or new `allow`; no new `acquire` call sites. -- [ ] T03: `Trigger the advisory check from the shared mutation-scope ingress` (status:todo) +- [x] T03: `Trigger the advisory check from the shared mutation-scope ingress` (status:done) - Task ID: T03 - Scope: In — in `hooks/mutation_scope.rs`, after `classify_coordinate` yields a durably completed `Close` or `Flush` (including the `MarkerClearAfterCommit` case) and `coordinate_boundary(...)` has fully returned, call `advise_if_due` through an injected seam; the outcome is only logged (`Advised` at warn with the `sce doctor --fix` hint; `Anchored`/`NoAction`/`Busy` at debug; `StateUnavailable` at warn) and is never returned as a boundary error; no trigger from `Start`, `Advance`, `Abandon`, or failed coordinates; a seam error or unexpected condition is contained and logged; the hook module gains no reconciliation, Git, or DB dependency. Apply the T01 decision rule: if the advisory cannot avoid a new Git subprocess, implement no hook call and document that outcome. Record the final constants in the plan and document the deferred follow-up (D2) in the plan's audit findings. Out — per-harness adapter changes, reconciliation on the hook path, detached tasks, background workers, timeouts. - Dependencies: T02 - Done when: a completed `Close`/`Flush` can run the advisory and no other path does; hook output and exit status are identical whether the advisory returns any outcome or errors; the advisory runs only after the coordinator's lock has been released; `MarkerClearAfterCommit` handling is unchanged and the advisory never creates, clears, or modifies the external-taint marker; no reconciliation, ref inventory, or DB I/O is reachable from the hook module's new call. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml mutation_scope`; trigger-policy unit tests over every boundary and outcome with a fake advisory seam; a seam that tries to take the real `WorktreeLock` and asserts it is free at invocation; `rg` over the hook module shows no reconciliation/DB-opener/snapshot import added. - - Context synchronization: pending + - Completed: 2026-10-09 + - Files changed: `cli/src/services/hooks/mutation_scope.rs`, `cli/src/services/mutation_trace/runtime/mod.rs`, `cli/src/services/mutation_trace/runtime/ref_advisory.rs`, `context/plans/mutation-cursor-ref-reconciliation-wiring.md` + - Result: `drive_mutation_scope` takes an injected advisory seam and calls it once, only for `Close`/`Flush` whose `coordinate_boundary` result is `Ok` or `MarkerClearAfterCommit`, after the await (coordinator lock already released) and after `classify_coordinate`; the report is only logged (`sce.hooks.mutation_scope.ref_reconciliation_advisory`; warn for `advised*`/`state_write_failed`/`state_unavailable`, debug for `anchored`/`no_action`/`busy`) and never changes the hook result. Production passes `advise_after_completed_boundary` (system clock, filesystem Git-dir derivation, no Git subprocess), which maps `AdvisoryOutcome` to the new `AdvisoryReport`/`AdvisorySeverity`. The hook gains no reconciliation, snapshot, store, or DB-opener import for this purpose. Final constants unchanged from the plan assumptions (`RECONCILIATION_ADVISORY_AFTER` 24 h, `MAINTENANCE_STATE_MAX_BYTES` 4096, `FUTURE_SKEW_TOLERANCE` 5 min). Deferred automatic reclamation (D2 prerequisites) is already recorded in this plan; T06 carries it into context. + - Verify: `test … mutation_scope` 18 passed (5 new `advisory_trigger_tests`: runs after completed Close/Flush, runs after `MarkerClearAfterCommit` with unchanged success result, never for Start/Advance, never after a failed coordinate, never for Abandon); `test … hooks` 27 passed; `test … mutation_trace` 158 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; `rg` shows no new reconciliation/DB/snapshot import in the hook module and no added comments. Not covered here (T05 S17/S18/S33): a real-lock "acquirable at invocation" assertion, because the seam tests use a fake coordinate that never holds the lock; ordering after `coordinate_boundary` returns is structural. + - Context impact: important (hook ingress now emits the advisory; T06 owns the durable context update). + - Context synchronization: synced - [ ] T04: `Run explicit reconciliation from sce doctor --fix and report recommendation` (status:todo) - Task ID: T04 From ebf698399041a61cc4871db5204e3c15d2f07cd7 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 01:36:40 +0200 Subject: [PATCH 06/16] hooks+runtime: Emit reconciliation advisory diagnostics Expose actionable reconciliation advisory outcomes through an injected diagnostic writer while retaining structured logging. Add outcome-specific diagnostics for durability and state-write failures, keep advisory delivery independent of mutation results, and cover the seven outcome shapes with focused tests. Ref: context/plans/mutation-cursor-ref-reconciliation-wiring.md (T03) Co-authored-by: SCE --- cli/src/services/hooks/mutation_scope.rs | 383 +++++++++++++++++- .../mutation_trace/runtime/ref_advisory.rs | 106 ++++- context/cli/mutation-scope-ref-advisory.md | 70 +++- context/context-map.md | 2 +- ...tation-cursor-ref-reconciliation-wiring.md | 20 +- 5 files changed, 532 insertions(+), 49 deletions(-) diff --git a/cli/src/services/hooks/mutation_scope.rs b/cli/src/services/hooks/mutation_scope.rs index 4995e533d..cc9e56741 100644 --- a/cli/src/services/hooks/mutation_scope.rs +++ b/cli/src/services/hooks/mutation_scope.rs @@ -263,6 +263,7 @@ pub(crate) async fn run_mutation_scope_from_payload< stdin_payload, logger, super::open_agent_trace_db_for_hook_runtime, + &mut std::io::stderr(), ) .await } @@ -270,14 +271,17 @@ pub(crate) async fn run_mutation_scope_from_payload< async fn run_mutation_scope_from_payload_with< L: crate::services::observability::traits::Logger, O, + E, >( repository_root: &Path, stdin_payload: &str, logger: Option<&L>, open_db: O, + diagnostics: &mut E, ) -> Result where O: std::ops::AsyncFn(&Path, &'static str) -> Result + Copy, + E: Write, { let payload = parse_mutation_scope_payload(stdin_payload)?; @@ -285,6 +289,7 @@ where repository_root, payload, logger, + diagnostics, async |root, boundary| { coordinate(root, boundary, async || { open_db(root, MUTATION_SCOPE_DB_CONTEXT).await @@ -302,10 +307,17 @@ where .await } -async fn drive_mutation_scope( +async fn drive_mutation_scope< + L: crate::services::observability::traits::Logger, + E: Write, + C, + A, + V, +>( repository_root: &Path, payload: MutationScopePayload, logger: Option<&L>, + diagnostics: &mut E, coordinate_boundary: C, abandon: A, advise: V, @@ -369,12 +381,28 @@ where let classified = classify_coordinate(result, logger); if advisory_eligible && durably_completed { - log_advisory_report(logger, &advise(repository_root)); + report_advisory(logger, diagnostics, &advise(repository_root)); } classified } +fn report_advisory( + logger: Option<&L>, + diagnostics: &mut E, + report: &AdvisoryReport, +) { + log_advisory_report(logger, report); + emit_advisory_diagnostic(diagnostics, report); +} + +fn emit_advisory_diagnostic(diagnostics: &mut E, report: &AdvisoryReport) { + let Some(diagnostic) = report.diagnostic.as_deref() else { + return; + }; + let _ = writeln!(diagnostics, "{diagnostic}").and_then(|()| diagnostics.flush()); +} + fn log_advisory_report( logger: Option<&L>, report: &AdvisoryReport, @@ -685,7 +713,9 @@ fn write_guard_line(writer: &mut W, line: &str) -> std::io::Result<()> #[cfg(test)] mod advisory_trigger_tests { use std::cell::Cell; + use std::io::Write; use std::path::Path; + use std::sync::Mutex; use anyhow::anyhow; @@ -694,9 +724,71 @@ mod advisory_trigger_tests { AdvisorySeverity, CoordinateError, CoordinateOutcome, MutationScopePayload, RuntimeBoundary, }; + use crate::services::error::CliError; use crate::services::mutation_trace::protocol::CommitEvaluation; use crate::services::mutation_trace::types::{ActorKind, ScopeId, TreeId, WorktreeId}; - use crate::services::observability::traits::NoopLogger; + use crate::services::observability::traits::Logger; + + const ADVISORY_EVENT_ID: &str = "sce.hooks.mutation_scope.ref_reconciliation_advisory"; + const RECOMMENDED: &str = "Reconciliation is recommended. Run sce doctor --fix."; + + #[derive(Debug, Eq, PartialEq)] + struct LogRecord { + level: &'static str, + event_id: String, + message: String, + fields: Vec<(String, String)>, + } + + #[derive(Default)] + struct RecordingLogger { + records: Mutex>, + } + + impl RecordingLogger { + fn record(&self, level: &'static str, event_id: &str, message: &str, f: &[(&str, &str)]) { + self.records.lock().unwrap().push(LogRecord { + level, + event_id: event_id.to_string(), + message: message.to_string(), + fields: f + .iter() + .map(|(key, value)| ((*key).to_string(), (*value).to_string())) + .collect(), + }); + } + + fn take(&self) -> Vec { + std::mem::take(&mut self.records.lock().unwrap()) + } + } + + impl Logger for RecordingLogger { + fn info(&self, event_id: &str, message: &str, fields: &[(&str, &str)], _: Option<&str>) { + self.record("info", event_id, message, fields); + } + fn debug(&self, event_id: &str, message: &str, fields: &[(&str, &str)], _: Option<&str>) { + self.record("debug", event_id, message, fields); + } + fn warn(&self, event_id: &str, message: &str, fields: &[(&str, &str)], _: Option<&str>) { + self.record("warn", event_id, message, fields); + } + fn error(&self, event_id: &str, message: &str, fields: &[(&str, &str)], _: Option<&str>) { + self.record("error", event_id, message, fields); + } + fn log_cli_error(&self, _error: &CliError, _session_id: Option<&str>) {} + } + + struct FailingWriter; + + impl Write for FailingWriter { + fn write(&mut self, _buf: &[u8]) -> std::io::Result { + Err(std::io::Error::other("stderr closed")) + } + fn flush(&mut self) -> std::io::Result<()> { + Err(std::io::Error::other("stderr closed")) + } + } fn outcome() -> CoordinateOutcome { CoordinateOutcome { @@ -714,6 +806,7 @@ mod advisory_trigger_tests { severity, recommendation: None, warning: None, + diagnostic: None, } } @@ -742,19 +835,34 @@ mod advisory_trigger_tests { } } - async fn drive( + #[derive(Default)] + struct Calls { + coordinate: Cell, + abandon: Cell, + advise: Cell, + } + + async fn drive_with( payload: MutationScopePayload, coordinated: Result, - calls: &Cell, + advised: AdvisoryReport, + logger: Option<&L>, + diagnostics: &mut E, + calls: &Calls, ) -> anyhow::Result { drive_mutation_scope( Path::new("."), payload, - None::<&NoopLogger>, - async |_root: &Path, _boundary: &RuntimeBoundary| coordinated, + logger, + diagnostics, + async |_root: &Path, _boundary: &RuntimeBoundary| { + calls.coordinate.set(calls.coordinate.get() + 1); + coordinated + }, async |_root: &Path, _scope: &ScopeId| -> Result { + calls.abandon.set(calls.abandon.get() + 1); Ok(AbandonScopeOutcome::Abandoned { worktree_id: WorktreeId("main".to_string()), scope: ScopeId("scope".to_string()), @@ -762,66 +870,299 @@ mod advisory_trigger_tests { }) }, |_root: &Path| { - calls.set(calls.get() + 1); - report("no_action", AdvisorySeverity::Debug) + calls.advise.set(calls.advise.get() + 1); + advised }, ) .await } + async fn drive( + payload: MutationScopePayload, + coordinated: Result, + calls: &Calls, + ) -> anyhow::Result { + drive_with( + payload, + coordinated, + report("no_action", AdvisorySeverity::Debug), + None::<&RecordingLogger>, + &mut Vec::new(), + calls, + ) + .await + } + + async fn deliver(advised: AdvisoryReport) -> (Vec, String) { + let logger = RecordingLogger::default(); + let mut sink = Vec::new(); + let result = drive_with( + close(), + Ok(outcome()), + advised, + Some(&logger), + &mut sink, + &Calls::default(), + ) + .await; + assert_eq!(result.unwrap(), ""); + (logger.take(), String::from_utf8(sink).unwrap()) + } + + fn field<'a>(record: &'a LogRecord, key: &str) -> Option<&'a str> { + record + .fields + .iter() + .find(|(name, _)| name == key) + .map(|(_, value)| value.as_str()) + } + #[tokio::test] async fn advisory_runs_once_after_completed_close_and_flush() { - let calls = Cell::new(0); + let calls = Calls::default(); assert_eq!(drive(close(), Ok(outcome()), &calls).await.unwrap(), ""); - assert_eq!(calls.get(), 1); + assert_eq!(calls.advise.get(), 1); assert_eq!( drive(MutationScopePayload::Flush, Ok(outcome()), &calls) .await .unwrap(), "" ); - assert_eq!(calls.get(), 2); + assert_eq!(calls.advise.get(), 2); + assert_eq!(calls.coordinate.get(), 2); + assert_eq!(calls.abandon.get(), 0); } #[tokio::test] async fn advisory_runs_after_marker_clear_after_commit_with_success_result() { - let calls = Cell::new(0); + let calls = Calls::default(); let error = CoordinateError::MarkerClearAfterCommit { source: anyhow!("clear failed"), committed: Box::new(outcome()), }; assert_eq!(drive(close(), Err(error), &calls).await.unwrap(), ""); - assert_eq!(calls.get(), 1); + assert_eq!(calls.advise.get(), 1); } #[tokio::test] async fn advisory_never_runs_for_start_or_advance() { - let calls = Cell::new(0); + let calls = Calls::default(); drive(start(), Ok(outcome()), &calls).await.unwrap(); drive(advance(), Ok(outcome()), &calls).await.unwrap(); - assert_eq!(calls.get(), 0); + assert_eq!(calls.advise.get(), 0); } #[tokio::test] async fn advisory_never_runs_after_failed_coordinate() { - let calls = Cell::new(0); - let result = drive( + let calls = Calls::default(); + let logger = RecordingLogger::default(); + let mut sink = Vec::new(); + let result = drive_with( close(), Err(CoordinateError::Other(anyhow!("boom"))), + AdvisoryReport { + diagnostic: Some(format!("SCE: {RECOMMENDED}")), + ..report("advised", AdvisorySeverity::Warn) + }, + Some(&logger), + &mut sink, &calls, ) .await; assert!(result.is_err()); - assert_eq!(calls.get(), 0); + assert_eq!(calls.advise.get(), 0); + assert!(sink.is_empty()); + assert!(logger.take().is_empty()); } #[tokio::test] async fn advisory_never_runs_for_abandon() { - let calls = Cell::new(0); + let calls = Calls::default(); let payload = MutationScopePayload::Abandon { scope_id: "scope".to_string(), }; drive(payload, Ok(outcome()), &calls).await.unwrap(); - assert_eq!(calls.get(), 0); + assert_eq!(calls.advise.get(), 0); + assert_eq!(calls.abandon.get(), 1); + } + + #[tokio::test] + async fn quiet_outcomes_log_debug_and_emit_no_diagnostic() { + for name in ["anchored", "no_action", "busy"] { + let (records, stderr) = deliver(report(name, AdvisorySeverity::Debug)).await; + assert_eq!(stderr, "", "{name}"); + assert_eq!(records.len(), 1, "{name}"); + assert_eq!(records[0].level, "debug", "{name}"); + assert_eq!(records[0].event_id, ADVISORY_EVENT_ID); + assert_eq!(field(&records[0], "outcome"), Some(name)); + assert_eq!(field(&records[0], "persistence_warning"), None); + } + } + + #[tokio::test] + async fn advised_logs_warn_and_writes_recommendation_to_diagnostics() { + let diagnostic = format!("SCE: {RECOMMENDED}"); + let (records, stderr) = deliver(AdvisoryReport { + recommendation: Some(RECOMMENDED), + diagnostic: Some(diagnostic.clone()), + ..report("advised", AdvisorySeverity::Warn) + }) + .await; + assert_eq!(stderr, format!("{diagnostic}\n")); + assert_eq!(records.len(), 1); + assert_eq!(records[0].level, "warn"); + assert_eq!(records[0].event_id, ADVISORY_EVENT_ID); + assert_eq!(records[0].message, RECOMMENDED); + assert_eq!(field(&records[0], "outcome"), Some("advised")); + assert_eq!(field(&records[0], "persistence_warning"), None); + } + + #[tokio::test] + async fn durability_uncertain_keeps_persistence_warning_in_log_and_diagnostics() { + let diagnostic = format!( + "SCE: {RECOMMENDED}\nSCE: Advisory-state durability could not be confirmed: sync failed." + ); + let (records, stderr) = deliver(AdvisoryReport { + recommendation: Some(RECOMMENDED), + warning: Some("sync failed".to_string()), + diagnostic: Some(diagnostic.clone()), + ..report("advised_durability_uncertain", AdvisorySeverity::Warn) + }) + .await; + assert_eq!(stderr, format!("{diagnostic}\n")); + assert_eq!(records[0].level, "warn"); + assert_eq!( + field(&records[0], "outcome"), + Some("advised_durability_uncertain") + ); + assert_eq!( + field(&records[0], "persistence_warning"), + Some("sync failed") + ); + } + + #[tokio::test] + async fn state_failures_log_warn_and_write_diagnostic_without_claiming_advice() { + for (name, warning, diagnostic) in [ + ( + "state_write_failed", + Some("rename failed"), + "SCE: Reconciliation advisory state could not be written: rename failed. Run sce doctor to check reconciliation status.", + ), + ( + "state_unavailable", + None, + "SCE: Reconciliation advisory state is unavailable. Run sce doctor to check reconciliation status.", + ), + ] { + let (records, stderr) = deliver(AdvisoryReport { + warning: warning.map(str::to_string), + diagnostic: Some(diagnostic.to_string()), + ..report(name, AdvisorySeverity::Warn) + }) + .await; + assert_eq!(stderr, format!("{diagnostic}\n"), "{name}"); + assert!(!stderr.contains(RECOMMENDED), "{name}"); + assert_eq!(records[0].level, "warn", "{name}"); + assert_eq!(field(&records[0], "outcome"), Some(name)); + assert_eq!(field(&records[0], "persistence_warning"), warning); + } + } + + #[tokio::test] + async fn missing_logger_still_delivers_actionable_diagnostic() { + let mut sink = Vec::new(); + let result = drive_with( + close(), + Ok(outcome()), + AdvisoryReport { + recommendation: Some(RECOMMENDED), + diagnostic: Some(format!("SCE: {RECOMMENDED}")), + ..report("advised", AdvisorySeverity::Warn) + }, + None::<&RecordingLogger>, + &mut sink, + &Calls::default(), + ) + .await; + assert_eq!(result.unwrap(), ""); + assert_eq!( + String::from_utf8(sink).unwrap(), + format!("SCE: {RECOMMENDED}\n") + ); + } + + #[tokio::test] + async fn diagnostic_write_failure_does_not_change_mutation_result() { + let calls = Calls::default(); + let advised = AdvisoryReport { + recommendation: Some(RECOMMENDED), + diagnostic: Some(format!("SCE: {RECOMMENDED}")), + ..report("advised", AdvisorySeverity::Warn) + }; + let logger = RecordingLogger::default(); + let result = drive_with( + close(), + Ok(outcome()), + advised.clone(), + Some(&logger), + &mut FailingWriter, + &calls, + ) + .await; + assert_eq!(result.unwrap(), ""); + assert_eq!(logger.take().len(), 1); + + let marker_clear = CoordinateError::MarkerClearAfterCommit { + source: anyhow!("clear failed"), + committed: Box::new(outcome()), + }; + let result = drive_with( + MutationScopePayload::Flush, + Err(marker_clear), + advised, + None::<&RecordingLogger>, + &mut FailingWriter, + &calls, + ) + .await; + assert_eq!(result.unwrap(), ""); + } + + #[tokio::test] + async fn marker_clear_after_commit_logs_taint_warning_and_still_advises_once() { + let logger = RecordingLogger::default(); + let mut sink = Vec::new(); + let calls = Calls::default(); + let error = CoordinateError::MarkerClearAfterCommit { + source: anyhow!("clear failed"), + committed: Box::new(outcome()), + }; + let result = drive_with( + close(), + Err(error), + AdvisoryReport { + recommendation: Some(RECOMMENDED), + diagnostic: Some(format!("SCE: {RECOMMENDED}")), + ..report("advised", AdvisorySeverity::Warn) + }, + Some(&logger), + &mut sink, + &calls, + ) + .await; + assert_eq!(result.unwrap(), ""); + let records = logger.take(); + assert_eq!(records.len(), 2); + assert_eq!( + records[0].event_id, + "sce.hooks.mutation_scope.marker_clear_after_durable_completion" + ); + assert_eq!(records[1].event_id, ADVISORY_EVENT_ID); + assert_eq!( + String::from_utf8(sink).unwrap(), + format!("SCE: {RECOMMENDED}\n") + ); + assert_eq!((calls.coordinate.get(), calls.advise.get()), (1, 1)); } } diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs index 5f02d8573..ae718b87f 100644 --- a/cli/src/services/mutation_trace/runtime/ref_advisory.rs +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -1,7 +1,3 @@ -#![allow( - dead_code, - reason = "maintenance entrypoints are wired by later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" -)] use std::path::Path; use std::time::Duration; @@ -14,6 +10,11 @@ use super::worktree_lock::{acquire_inner, WorktreeLockError}; pub(super) const RECONCILIATION_RECOMMENDED_MESSAGE: &str = "Reconciliation is recommended. Run sce doctor --fix."; +const DIAGNOSTIC_PREFIX: &str = "SCE: "; +const DURABILITY_UNCERTAIN_DIAGNOSTIC: &str = "Advisory-state durability could not be confirmed"; +const STATE_WRITE_FAILED_DIAGNOSTIC: &str = "Reconciliation advisory state could not be written"; +const STATE_UNAVAILABLE_DIAGNOSTIC: &str = "Reconciliation advisory state is unavailable"; +const CHECK_DOCTOR_HINT: &str = "Run sce doctor to check reconciliation status."; #[derive(Debug)] pub(super) enum AdvisoryOutcome { @@ -65,6 +66,7 @@ pub(crate) struct AdvisoryReport { pub(crate) severity: AdvisorySeverity, pub(crate) recommendation: Option<&'static str>, pub(crate) warning: Option, + pub(crate) diagnostic: Option, } impl AdvisoryOutcome { @@ -87,6 +89,26 @@ impl AdvisoryOutcome { severity, recommendation: self.recommendation(), warning: self.persistence_warning().map(ToString::to_string), + diagnostic: self.diagnostic(), + } + } + + fn diagnostic(&self) -> Option { + match self { + AdvisoryOutcome::Advised => Some(format!( + "{DIAGNOSTIC_PREFIX}{RECONCILIATION_RECOMMENDED_MESSAGE}" + )), + AdvisoryOutcome::AdvisedDurabilityUncertain { warning } => Some(format!( + "{DIAGNOSTIC_PREFIX}{RECONCILIATION_RECOMMENDED_MESSAGE}\n\ + {DIAGNOSTIC_PREFIX}{DURABILITY_UNCERTAIN_DIAGNOSTIC}: {warning}." + )), + AdvisoryOutcome::StateWriteFailed { warning } => Some(format!( + "{DIAGNOSTIC_PREFIX}{STATE_WRITE_FAILED_DIAGNOSTIC}: {warning}. {CHECK_DOCTOR_HINT}" + )), + AdvisoryOutcome::StateUnavailable => Some(format!( + "{DIAGNOSTIC_PREFIX}{STATE_UNAVAILABLE_DIAGNOSTIC}. {CHECK_DOCTOR_HINT}" + )), + AdvisoryOutcome::Anchored | AdvisoryOutcome::NoAction | AdvisoryOutcome::Busy => None, } } } @@ -179,9 +201,12 @@ mod tests { StoredReport, FUTURE_SKEW_TOLERANCE_MS, RECONCILIATION_ADVISORY_AFTER_MS, }; use super::super::worktree_lock::acquire_inner; - use super::{advise_if_due_with, AdvisoryOutcome, RECONCILIATION_RECOMMENDED_MESSAGE}; + use super::{ + advise_if_due_with, AdvisoryOutcome, AdvisorySeverity, RECONCILIATION_RECOMMENDED_MESSAGE, + }; const NOW: i64 = 1_000_000_000_000; + const RECOMMENDED: &str = RECONCILIATION_RECOMMENDED_MESSAGE; fn init_repo(root: &Path) { std::fs::create_dir_all(root).expect("repo dir"); @@ -214,6 +239,77 @@ mod tests { real_write(&path, state).expect("seed state"); } + #[test] + fn maintenance_report_maps_every_outcome_to_severity_and_diagnostic() { + let uncertain = || PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + source: std::io::Error::other("sync failed"), + }; + let not_applied = || PersistFailure::NotApplied { + phase: PersistPhase::Rename, + source: std::io::Error::other("rename failed"), + staging_cleanup: None, + }; + let recommended = format!("SCE: {RECOMMENDED}"); + let cases = [ + (AdvisoryOutcome::Anchored, "anchored", AdvisorySeverity::Debug, None), + (AdvisoryOutcome::NoAction, "no_action", AdvisorySeverity::Debug, None), + (AdvisoryOutcome::Busy, "busy", AdvisorySeverity::Debug, None), + ( + AdvisoryOutcome::Advised, + "advised", + AdvisorySeverity::Warn, + Some(recommended.clone()), + ), + ( + AdvisoryOutcome::StateUnavailable, + "state_unavailable", + AdvisorySeverity::Warn, + Some( + "SCE: Reconciliation advisory state is unavailable. Run sce doctor to check reconciliation status." + .to_string(), + ), + ), + ]; + for (outcome, name, severity, diagnostic) in cases { + let report = outcome.report(); + assert_eq!(report.outcome, name); + assert_eq!(report.severity, severity); + assert_eq!(report.diagnostic, diagnostic, "{name}"); + assert_eq!(report.warning, None, "{name}"); + } + + let report = AdvisoryOutcome::AdvisedDurabilityUncertain { + warning: uncertain(), + } + .report(); + assert_eq!(report.outcome, "advised_durability_uncertain"); + assert_eq!(report.severity, AdvisorySeverity::Warn); + assert_eq!( + report.recommendation, + Some(RECONCILIATION_RECOMMENDED_MESSAGE) + ); + let warning = report.warning.expect("persistence warning"); + let diagnostic = report.diagnostic.expect("diagnostic"); + assert!(diagnostic.starts_with(&format!( + "{recommended}\nSCE: Advisory-state durability could not be confirmed: " + ))); + assert!(diagnostic.contains(&warning)); + + let report = AdvisoryOutcome::StateWriteFailed { + warning: not_applied(), + } + .report(); + assert_eq!(report.outcome, "state_write_failed"); + assert_eq!(report.severity, AdvisorySeverity::Warn); + assert_eq!(report.recommendation, None); + let warning = report.warning.expect("persistence warning"); + let diagnostic = report.diagnostic.expect("diagnostic"); + assert!(diagnostic.starts_with("SCE: Reconciliation advisory state could not be written: ")); + assert!(diagnostic.contains(&warning)); + assert!(!diagnostic.contains(RECONCILIATION_RECOMMENDED_MESSAGE)); + } + #[test] fn maintenance_advisory_anchors_once_then_leaves_state_untouched() { let dir = tempfile::tempdir().expect("temp dir"); diff --git a/context/cli/mutation-scope-ref-advisory.md b/context/cli/mutation-scope-ref-advisory.md index 63aabf0a9..323bc76d9 100644 --- a/context/cli/mutation-scope-ref-advisory.md +++ b/context/cli/mutation-scope-ref-advisory.md @@ -9,8 +9,10 @@ Git. Reconciliation itself is operator-driven; this signal is only a reminder. ## Trigger policy `drive_mutation_scope` takes an injected advisory seam (production: -`runtime::advise_after_completed_boundary`). The seam runs exactly once, and only -when all of these hold: +`runtime::advise_after_completed_boundary`) and a generic diagnostic writer +`E: Write` (production: `std::io::stderr()`, supplied by +`run_mutation_scope_from_payload`). The seam runs exactly once, and only when all +of these hold: - the boundary is `Close` or `Flush`; - the `coordinate_boundary(...)` result is `Ok` or @@ -19,29 +21,67 @@ when all of these hold: It runs after the coordinator future has returned, so the coordinator's `WorktreeLock` is already released. It never runs for `Start`, `Advance`, `Abandon`, or a failed coordinate. The hook result is computed before the seam -runs and is never altered by it: every advisory outcome is only logged as -`sce.hooks.mutation_scope.ref_reconciliation_advisory` with an `outcome` field -(`warn` for `advised`, `advised_durability_uncertain`, `state_write_failed`, -`state_unavailable`; `debug` for `anchored`, `no_action`, `busy`). The advisory -never creates, clears, or modifies the external-taint marker. +runs and is never altered by it, including when the diagnostic write fails. The +advisory never creates, clears, or modifies the external-taint marker. -## Advisory behavior +## Outcomes, logging and diagnostics + +Every outcome is logged as `sce.hooks.mutation_scope.ref_reconciliation_advisory` +with an `outcome` field (plus `persistence_warning` when present). Actionable +outcomes are additionally written, one `SCE: ...` line each, to the diagnostic +writer. Stdout stays empty and never carries advisory text. + +| Outcome | Log | Diagnostic (stderr) | +|---|---|---| +| `Anchored` | debug | none | +| `NoAction` | debug | none | +| `Busy` | debug | none | +| `Advised` | warn | `SCE: Reconciliation is recommended. Run sce doctor --fix.` | +| `AdvisedDurabilityUncertain` | warn, with `persistence_warning` | the line above, then `SCE: Advisory-state durability could not be confirmed: .` | +| `StateWriteFailed` | warn, with `persistence_warning` | `SCE: Reconciliation advisory state could not be written: . Run sce doctor to check reconciliation status.` (does not claim advice was persisted) | +| `StateUnavailable` | warn | `SCE: Reconciliation advisory state is unavailable. Run sce doctor to check reconciliation status.` | + +The diagnostic does not depend on the logger (it is emitted with `logger = None`), +and `Logger::warn` and the observability defaults are unchanged: observability +routes only ERROR records to stderr, and only when file logging is off, so WARN +alone would not reach the operator. A write or flush error on the diagnostic +writer is ignored. Only this path writes the advisory diagnostic to stderr. + +## Advisory behavior and cadence `ref_advisory::advise_if_due` (`cli/src/services/mutation_trace/runtime/`) takes no Git, DB, store, or snapshot parameters. It derives the Git dir from the filesystem (no Git subprocess), makes one zero-wait `try_lock` of `WorktreeLock`, and reads/writes the small per-worktree state file `/sce/ref-maintenance.json` (capped, versioned, past-event timestamps -only, atomic staging-and-swap writes). It anchors on first use and then advises -at most once per `RECONCILIATION_ADVISORY_AFTER` window (24 h), persisting -`last_advised` before reporting `Advised`. A busy lock or unreadable state is -reported, never an error. +only, atomic staging-and-swap writes). It persists `last_advised` before +reporting `Advised`. A busy lock or unreadable state is reported, never an error. + +- **Time-based reminders:** the first use anchors the state. A stale state + becomes eligible after `RECONCILIATION_ADVISORY_AFTER` (24 h); further + stale-state reminders are suppressed for 24 h after `last_advised`. +- **Failure-triggered reminders:** an outstanding failed explicit reconciliation + is recommended immediately. A newly recorded failure (recorded after + `last_advised`) may be advised even inside the previous 24 h window; once + recorded, that reminder is suppressed. Repeated reminders for the same + unresolved failure follow the normal 24 h cadence. A successful explicit + reconciliation clears the failure-based recommendation. +- Anchoring and clock-skew normalization are the only other state writes, so + state is not written "at most once per window" unconditionally. +- `sce doctor` reports the recommendation from state regardless of whether a hook + reminder was ever delivered. -## Limits +## Delivery guarantee and limits +- Best-effort, not exactly-once, and not guaranteed to be seen by a human. + `last_advised` is persisted before the diagnostic, so a crash or an unseen + stderr can still consume one reminder; the next reminder follows the cadence. +- Claude Code and Codex run the hook as a command and can capture its stderr, but + rendering of exit-0 hook stderr is up to the harness. The OpenCode plugin and + the Pi extension discard the child's stderr, so there the structured log file + and `sce doctor` are the supported surfaces. Changing the plugins is out of + scope here. - Time-based, not evidence-based: orphan detection needs the Git/DB scan this path deliberately avoids. -- A reminder is not guaranteed exactly-once; a crash between the state write and - the log line can lose one. - Orphan pins are reclaimed only by an explicit reconciliation pass. Automatic reclamation is not provided. diff --git a/context/context-map.md b/context/context-map.md index ccd438123..a8232ad75 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -42,7 +42,7 @@ Feature/domain context: - `context/cli/claude-mutation-scope-background-execution.md` (native Claude background Bash is operationally supported but unsupported for mutation attribution: `Bash(run_in_background=true)` is allowed in `PreToolUse` with no mutation scope, adapter attempt, lifecycle tracking, or attribution guarantee — effective record `context/decisions/2026-10-02-claude-background-bash-untracked.md` — its later terminal hooks are no-ops, and the accepted A/B contamination consequence is pinned by a regression; background `PowerShell` stays denied with the existing deny text; the T01 `2.1.284` probe result with lifecycle completion still unproven; and the separate self-detaching-descendant unsupported boundary D20 — Git-observable evidence that a foreground `setsid` command's detached descendant can still mutate the repository after `PostToolUse` returns — detail split out of `claude-mutation-scope-integration.md` for the repository's per-file line budget) - `context/cli/claude-mutation-scope-health.md` (the `doctor-mutation-scope-health` plan's Claude task, extended by the `doctor-mutation-scope-fix` plan's T04: `classify_health` in `cli/src/services/hooks/claude_mutation_scope/health.rs` maps `recovery_pending`/`attempts` onto `healthy | recovering | blocked | invalid` — `recovery_pending == false` is `healthy` regardless of live attempts; `recovery_pending == true` with `attempts` empty is `recovering` (the barrier's own next-`PreToolUse` flush clears it); `recovery_pending == true` with `attempts` non-empty is `blocked`, the exact incident shape this classifier exists to surface, since nothing in the ordinary hook lifecycle retries, removes, flushes, or clears from that shape; a read/parse failure is `invalid`. T04 adds a separate, `doctor`-invoked repair path (`assess_repairability`/`repair_blocked` in `health.rs`/`lifecycle.rs`, wired into `sce doctor --fix` by the same plan's T05) for the `blocked` shape: `AutoFixable` only when every currently persisted attempt is already `pending_abandon` (no established-abandon-intent `pending_start`/`active` attempt present); `repair_blocked` re-reads and re-proves that same condition inside one lock-protected read-only state transaction before releasing the lock and retrying each attempt's already-established seam `abandon` call independently, clearing the `recovery_pending` barrier only once no attempts remain — this classifier's own read-only behavior is unaffected by that repair path's existence) - `context/cli/mutation-trace-ref-reconciliation.md` (the conservative per-worktree snapshot-ref reconciliation pass in `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs`, built by the `mutation-cursor-ref-reconciliation` plan: `reconcile_worktree(repository_root, open_db)` → `pub(super) reconcile_worktree_inner(.., on_lock_contention)`, both returning `ReconciliationOutcome` (`Reconciled(ReconciliationReport { local_required, retained, deleted })` for a pass that ran | `SkippedNoCheckoutIdentity` — an `Ok`, not an `Err` — when no current checkout identity could be derived), a variant-per-fallible-step `ReconcileError` with no `Other`, and the module-owned `RECONCILIATION_LOCK_TIMEOUT`; the two-invariant model — a strictly per-worktree fail-closed local-consistency check via `load_tree_roots(W)` vs. a repository-wide deletion-safety set via `load_all_tree_roots()` so an `A`-owned ref is retained whenever any worktree still durably needs its tree — run entirely under the same `/sce/mutation-cursor.lock` `WorktreeLock` `coordinate()` holds, with the repository-wide read kept coherent by being one SQL statement / one DB snapshot rather than a repository-global lock; deletes only SCE-owned refs via one atomic `git update-ref --no-deref --stdin`, writes no `mutation_trace_*` row, never arms `ExternalTaintMarker`, runs no `git gc`; imperative durability maintenance below the verified Quint protocol; reclaims orphan/unreferenced refs only for the namespace of a checkout id a current worktree still derives — a namespace no current worktree owns (identity-based: a deleted linked worktree, or checkout-id metadata loss followed by `get_or_create_checkout_id` minting a fresh id on a still-present worktree) is beyond every per-worktree pass and left to a recorded future repository-scoped unowned-namespace operation, so the current pass does not bound all orphan-ref growth; `reconcile_worktree` has no `pub(crate)` re-export and no harness/command wiring yet) -- `context/cli/mutation-scope-ref-advisory.md` (the advisory-only hook trigger in `cli/src/services/hooks/mutation_scope.rs` built by T03 of the `mutation-cursor-ref-reconciliation-wiring` plan: a lightweight `advise_if_due` check after a durably completed `Close`/`Flush`, logged only, with the per-worktree `ref-maintenance.json` state; no reconciliation, Git, or DB work on the hook path) +- `context/cli/mutation-scope-ref-advisory.md` (the advisory-only hook trigger in `cli/src/services/hooks/mutation_scope.rs` built by T03 of the `mutation-cursor-ref-reconciliation-wiring` plan: a lightweight `advise_if_due` check after a durably completed `Close`/`Flush`, logged and, when actionable, written as an `SCE:` line to an injectable stderr writer (best-effort delivery), with the per-worktree `ref-maintenance.json` state; no reconciliation, Git, or DB work on the hook path) - `context/cli/mutation-trace-snapshot-service.md` (the isolated Git snapshot and ref-pinning service `runtime::git_snapshot::GitSnapshotService` in `cli/src/services/mutation_trace/runtime/git_snapshot.rs`: `new` resolving an absolute `git_dir`, `capture_tree` snapshotting staged/unstaged/untracked/deleted worktree state into the repository's normal object database via a throwaway temp index, `pin_tree` protecting a durable tree with a create-only idempotent **direct** `refs/sce/mutation-cursor//` ref, `diff_trees` emitting `patch.rs`-parseable raw diff text; plus the callerless reconciliation substrate — worktree-scoped `list_pins` inventory returning `Result, PinInventoryError>` that rejects a symbolic ref inside the namespace (mutation-cursor pins are direct refs) as `MalformedRef`, matchable separately from a `git for-each-ref` execution failure, and conditional-atomic `delete_pins` running one `git update-ref --no-deref --stdin` transaction of SHA-conditioned deletes — no-dereference so an inventory→delete direct-ref→symref race cannot escape the inventoried namespace, plus a fail-closed pre-check — that aborts whole if any ref changed since inventory; `REF_NAMESPACE` + `pin_ref_prefix` as the single source of truth for the pin path) - `context/cli/mutation-trace-external-taint.md` (the worktree-local mutation-cursor durability boundary in `cli/src/services/mutation_trace/runtime/external_taint.rs`, built by the `mutation-cursor-external-taint` plan: the `ExternalTaintMarker` primitive — `new(git_dir)`/`exists()`/`persist()`/`clear()` over an empty file at `/sce/mutation-cursor-tainted` whose existence is its entire state, `checkout::persist_checkout_id_inner`-style durability (`sync_data` plus best-effort `#[cfg(unix)]` parent-dir `sync_all`), idempotent persist/clear, `NotFound`-on-clear as success, no `Drop` deletion — as the concrete runtime refinement of the abstract `ProtocolState.external_taint`; armed by the reshaped `coordinate()` entrypoint write-ahead after the `WorktreeLock` and before Agent Trace DB acquisition (a caller-supplied DB provider closure), cleared only on a successful `CoordinateOutcome`, with dedicated fail-closed pre-commit `CoordinateError::ExternalTaintMarker` (`Inspect`/`Persist` only)/`AgentTraceDbUnavailable` variants plus a post-commit `MarkerClearAfterCommit { source, committed }` that carries the durable outcome so a failed trailing clear never hides a committed `MutationEvent`; an inherited marker seeds an invocation-local `external_taint_pending` flag that overlays `protocol::database_failure` onto each freshly loaded projection so `recover` runs once against the captured snapshot, held across a losing recovery CAS and cleared once it lands) - `context/sce/cli-exit-code-contract.md` (stable class-based `sce` process exit-code contract in `cli/src/app.rs`, so automation can branch on failure category without parsing error text) diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 7377326a1..e5c78203b 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -7,7 +7,7 @@ Make the existing mutation-cursor snapshot-ref reconciler (`runtime/ref_reconcil This plan wires reconciliation in two deliberately different ways: - **Actual reconciliation (Git and DB I/O) runs only through `sce doctor --fix`.** A new runtime maintenance entrypoint performs exactly one `WorktreeLock` acquisition, verifies and opens the existing repository Agent Trace DB without side effects, runs the existing reconciliation algorithm and atomic deletion transaction, and records a diagnostic result. -- **The mutation-hook completion path (`Close`/`Flush`) performs only a lightweight advisory check.** It never lists refs, opens the DB, or runs reconciliation. At most once per advisory window it records that reconciliation is recommended in a small durable state file; `sce doctor` (read-only) reports the recommendation and `sce doctor --fix` acts on it. +- **The mutation-hook completion path (`Close`/`Flush`) performs only a lightweight advisory check.** It never lists refs, opens the DB, or runs reconciliation. It records that reconciliation is recommended in a small durable state file at most once per advisory window for time-based staleness, and may record again earlier only for a newly recorded explicit-reconciliation failure (see AC5); `sce doctor` (read-only) reports the recommendation and `sce doctor --fix` acts on it. **Fully automatic reclamation is not delivered by this PR.** No lifecycle-owned, noncritical-path execution point exists today (see D2), and inline reconciliation on the tool-completion path cannot be bounded. Automatic out-of-band reconciliation is a documented follow-up PR with stated prerequisites. Nothing in this plan or in the synchronized context may claim that orphan pins are reclaimed automatically. @@ -122,8 +122,8 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - Reconciliation does **not** repair, recreate, or verify missing pins belonging to other worktrees, and does not guarantee that every repository-wide durable root is pinned. - Validate: T05 S2, S3, S4, S5 and S13 pass. S2/S3 prove preservation; S4 proves local fail-closed; S13 proves another worktree's missing pin is neither repaired nor treated as a failure of `W`'s pass. - [ ] AC4: Hook-path maintenance (advisory) cannot change the outcome of an already durably committed mutation boundary. - - Validate: T05 S17, S18a, S18b and S33 pass: every advisory outcome (`Anchored`, `NoAction`, `Advised`, `Busy`, `StateUnavailable`, corrupt/invalid state, seam error) leaves the hook result identical to the coordinate-only baseline. The advisory never creates, clears, or modifies the external-taint marker. After normal completion the marker state is whatever the existing coordinator leaves (cleared) and is unchanged by the advisory (S18a). `MarkerClearAfterCommit` remains a durably completed boundary with a successful hook result; the marker state resulting from the failed clear is preserved by the advisory, and the next real runtime invocation still takes the existing conservative inherited-taint recovery path (S18b). `MarkerClearAfterCommit` is never treated as equivalent to a successful clear. -- [ ] AC5: The mutation-hook completion path performs no reconciliation, ref listing, DB I/O, or new Git subprocess, and writes maintenance state at most once per advisory window. + - Validate: T05 S17, S18a, S18b and S33 pass: every advisory outcome (`Anchored`, `NoAction`, `Advised`, `AdvisedDurabilityUncertain`, `Busy`, `StateWriteFailed`, `StateUnavailable`, corrupt/invalid state, seam error) and a failing diagnostic-output writer leave the hook result identical to the coordinate-only baseline. The advisory never creates, clears, or modifies the external-taint marker. After normal completion the marker state is whatever the existing coordinator leaves (cleared) and is unchanged by the advisory (S18a). `MarkerClearAfterCommit` remains a durably completed boundary with a successful hook result; the marker state resulting from the failed clear is preserved by the advisory, and the next real runtime invocation still takes the existing conservative inherited-taint recovery path (S18b). `MarkerClearAfterCommit` is never treated as equivalent to a successful clear. +- [ ] AC5: The mutation-hook completion path performs no reconciliation, ref listing, DB I/O, or new Git subprocess, and its maintenance-state writes follow a precise cadence. Time-based reminders: a stale state becomes eligible after `RECONCILIATION_ADVISORY_AFTER` (24 h) and further stale-state reminders are suppressed for 24 h after `last_advised`. Failure-triggered reminders: an outstanding failed explicit reconciliation is recommended immediately; a newly recorded failure (`last_attempt` newer than `last_advised`) may be advised even inside the previous 24 h window and is suppressed once recorded; repeated reminders for the same unresolved failure follow the normal 24 h cadence; a successful explicit reconciliation clears the failure-based recommendation. Anchoring and clock-skew normalization are the only other writes. Actionable reminders reach an explicit diagnostic channel (stderr) in addition to structured logging; delivery is best-effort and neither exactly-once nor guaranteed to be seen by a human (T03 remediation, context/cli/mutation-scope-ref-advisory.md). - Validate: T05 S14, S15, S19 and S20 pass (S14: the advisory performs no Git-ref creation, update, deletion, or inventory, no reconciliation, and no DB I/O; coordinator ref effects are legitimate and distinguished from it); `rg` over the advisory module shows no import of `GitSnapshotService`, `MutationTraceStore`, DB openers, `resolve_git_dir`, `resolve_worktree_id`, `spawn_blocking`, or `Command`. - [ ] AC6: Contention, cancellation, and failed maintenance never violate the pin-to-CAS safety boundary. - Validate: T05 S6, S7 and S12 pass using deterministic synchronization (no sleep-based proof): concurrent `coordinate()` cannot race pin deletion; dropping the explicit-pass future during a ref-deletion worker does not release the lock until the worker finishes; each operation completes without self-deadlock. @@ -132,7 +132,7 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - [ ] AC8: Mutation-cursor and Quint model-based tests remain valid without protocol changes. - Validate: `git diff fix-logging...HEAD --stat` shows no change to `protocol.rs` or `spec/mutation_cursor.qnt`; the existing `mbt` tests pass. - [ ] AC9: `sce doctor` stays non-mutating; `sce doctor --fix` reports its actual cleanup result and recovers from every failure, stale, and invalid-state condition. - - Validate: T05 S9, S25, S26 and S27 pass: counts for deleted/retained/local-required; distinct output for completed, completed-with-state-persist-failure, failed (unavailable/invalid DB, malformed refs, missing pins), and busy; plain `sce doctor` writes no ref and no state; a recommendation is shown when advised or when state is invalid. + - Validate: T05 S9, S25, S26 and S27 pass: counts for deleted/retained/local-required; distinct output for completed, completed-with-state-persist-failure, failed (unavailable/invalid DB, malformed refs, missing pins), and busy; plain `sce doctor` writes no ref and no state; a recommendation is shown when stale, when the latest explicit pass failed (regardless of `last_advised`), or when state is invalid. The doctor recommendation is independent of whether a hook reminder was ever seen, which is why lossy hook delivery is acceptable. - [ ] AC10: Context matches the implementation, the "retained and unwired" status is gone, and context states that automatic reclamation is deferred to a follow-up with its prerequisites. - Validate: `nix shell nixpkgs#ripgrep -c rg -i "retained and unwired|unwired" context/cli` returns no stale claims; `rg -i "automatic.*(reclam|reconcil)" context` hits only the deferral/follow-up statements; documented APIs match code. - [ ] AC11: Exactly one `WorktreeLock` acquisition per maintenance operation, with no self-deadlock. @@ -141,7 +141,7 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - Validate: T05 S14, S15, S16 and S33 pass, as three distinct contention scenarios. S15 (direct advisory contention): with an explicit pass parked mid-inventory or mid-DB holding the lock, `advise_if_due` called directly returns `Busy` immediately and changes no state or refs. S16 (coordinator contention): with the lock held, `coordinate()` keeps its existing bounded lock acquisition; a short injected timeout fails with the existing lock-timeout error before durable completion, the advisory does not run after the failed coordinate, and the same boundary succeeds once the pass is released. S33 (post-coordinate handoff race): after a successful `coordinate()` released its lock, a competing explicit pass takes the lock before the paused advisory resumes; the advisory returns `Busy` and the committed boundary still returns success. The advisory takes no Git/DB parameters. No timeout, budget, or size cap is claimed to bound hook latency. - [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. - Two guarantee classes (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, failed state read or write, write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. - - Validate: T05 S19–S25 and S34–S39 pass: single advice per window with persistence before emission; concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's failure leg assert byte-identical preexisting state instead; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write leaving the previous state unchanged; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. + - Validate: T05 S19–S25 and S34–S39 pass: single time-based advice per 24 h window with persistence before emission (a newly recorded failure may be advised inside that window, once, per AC5); concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's failure leg assert byte-identical preexisting state instead; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write leaving the previous state unchanged; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. - [ ] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability, and only the missing-DB case asserts no newly created database-related files. - **Missing DB:** fails before Turso is invoked; no database file, parent directory, or WAL/shared-memory file is created; no Git ref is listed or deleted. - **Existing but invalid DB:** opening is permitted; incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable and documented; no schema migration, migration-metadata repair, repository-metadata initialization, or application-table INSERT/UPDATE/DELETE; no Git-ref deletion; verification fails with the matching typed error. Tests do not assert filesystem immutability for this case. @@ -304,7 +304,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - [x] T03: `Trigger the advisory check from the shared mutation-scope ingress` (status:done) - Task ID: T03 - - Scope: In — in `hooks/mutation_scope.rs`, after `classify_coordinate` yields a durably completed `Close` or `Flush` (including the `MarkerClearAfterCommit` case) and `coordinate_boundary(...)` has fully returned, call `advise_if_due` through an injected seam; the outcome is only logged (`Advised` at warn with the `sce doctor --fix` hint; `Anchored`/`NoAction`/`Busy` at debug; `StateUnavailable` at warn) and is never returned as a boundary error; no trigger from `Start`, `Advance`, `Abandon`, or failed coordinates; a seam error or unexpected condition is contained and logged; the hook module gains no reconciliation, Git, or DB dependency. Apply the T01 decision rule: if the advisory cannot avoid a new Git subprocess, implement no hook call and document that outcome. Record the final constants in the plan and document the deferred follow-up (D2) in the plan's audit findings. Out — per-harness adapter changes, reconciliation on the hook path, detached tasks, background workers, timeouts. + - Scope: In — in `hooks/mutation_scope.rs`, after `classify_coordinate` yields a durably completed `Close` or `Flush` (including the `MarkerClearAfterCommit` case) and `coordinate_boundary(...)` has fully returned, call `advise_if_due` through an injected seam; the outcome is logged as structured events and, for actionable outcomes, written to an injectable diagnostic writer (stderr in production) and is never returned as a boundary error (`Anchored`/`NoAction`/`Busy` debug, no diagnostic; `Advised`/`AdvisedDurabilityUncertain`/`StateWriteFailed`/`StateUnavailable` warn plus diagnostic); no trigger from `Start`, `Advance`, `Abandon`, or failed coordinates; a seam error or unexpected condition is contained and logged; the hook module gains no reconciliation, Git, or DB dependency. Apply the T01 decision rule: if the advisory cannot avoid a new Git subprocess, implement no hook call and document that outcome. Record the final constants in the plan and document the deferred follow-up (D2) in the plan's audit findings. Out — per-harness adapter changes, reconciliation on the hook path, detached tasks, background workers, timeouts. - Dependencies: T02 - Done when: a completed `Close`/`Flush` can run the advisory and no other path does; hook output and exit status are identical whether the advisory returns any outcome or errors; the advisory runs only after the coordinator's lock has been released; `MarkerClearAfterCommit` handling is unchanged and the advisory never creates, clears, or modifies the external-taint marker; no reconciliation, ref inventory, or DB I/O is reachable from the hook module's new call. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml mutation_scope`; trigger-policy unit tests over every boundary and outcome with a fake advisory seam; a seam that tries to take the real `WorktreeLock` and asserts it is free at invocation; `rg` over the hook module shows no reconciliation/DB-opener/snapshot import added. @@ -314,6 +314,12 @@ Record actual outcomes in the validation report; compilation alone does not vali - Verify: `test … mutation_scope` 18 passed (5 new `advisory_trigger_tests`: runs after completed Close/Flush, runs after `MarkerClearAfterCommit` with unchanged success result, never for Start/Advance, never after a failed coordinate, never for Abandon); `test … hooks` 27 passed; `test … mutation_trace` 158 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; `rg` shows no new reconciliation/DB/snapshot import in the hook module and no added comments. Not covered here (T05 S17/S18/S33): a real-lock "acquirable at invocation" assertion, because the seam tests use a fake coordinate that never holds the lock; ordering after `coordinate_boundary` returns is structural. - Context impact: important (hook ingress now emits the advisory; T06 owns the durable context update). - Context synchronization: synced + - Remediation (2026-10-09, PR #305 review of `91e9ac2`; T03 reopened `done`→`in_progress`, returned to `done`/`synced` after the verification below; T01/T02 unchanged, T04–T06 untouched). Architecture unchanged: advisory only on durably completed `Close`/`Flush`, no reconciliation/Git/DB work, single lock design, static dispatch. + - Delivery (P2): `Logger::warn` is not an operator channel (observability routes only ERROR to stderr, and only with file logging off), and `last_advised` is persisted before logging, so a reminder could be suppressed for 24 h without ever being seen. `drive_mutation_scope` now takes a generic `E: Write` diagnostic sink (production: `std::io::stderr()`, created in `run_mutation_scope_from_payload`; tests: `Vec`/failing writer). `AdvisoryReport.diagnostic` carries the text; `Advised` → `SCE: Reconciliation is recommended. Run sce doctor --fix.`; `AdvisedDurabilityUncertain` → that line plus `SCE: Advisory-state durability could not be confirmed: .`; `StateWriteFailed`/`StateUnavailable` → a state-write-failed / state-unavailable line plus `Run sce doctor to check reconciliation status.` (never claiming advice was persisted); `Anchored`/`NoAction`/`Busy` → none. Structured logging (event ID, `outcome`, `persistence_warning`) is retained; the diagnostic write is independent of the logger (works with `None`), ignores write/flush errors, and never alters the hook result. `Logger::warn` and observability defaults are unchanged. Stdout stays empty. + - Harness reality: Claude Code and Codex run the hook as a command, so stderr is available to the harness (shown only in the harness's verbose/hook output, not guaranteed in the transcript). The OpenCode plugin (`spawnSync`, captured output) and Pi extension (`stdio` `ignore`/captured and dropped) discard the child's stderr, so for those harnesses the stderr line is not operator-visible; the structured log file (default `log_to_file`) and `sce doctor` (which reports the stale/failed recommendation independent of hook delivery) are the supported surfaces. Plugin changes are out of T03 scope. + - Docs: AC4, AC5, AC9, AC13, the T03 scope and S20 now state the time-based vs failure-triggered cadence and the full seven-outcome enum. + - Tests: `mutation_scope` advisory tests (recording logger + injected sink) cover all seven report shapes (severity, event ID, outcome, persistence warning, diagnostic), missing logger, failing writer (result stays `Ok("")`), `MarkerClearAfterCommit`, failed coordinate (no advisory, no output), and single coordinate/advise invocation; `ref_advisory` adds `maintenance_report_maps_every_outcome_to_severity_and_diagnostic` over real outcomes. Existing `ref_advisory` tests already cover the 24 h cadence and an in-window new-failure reminder. Removed the now-unneeded file-level `#![allow(dead_code)]` from `ref_advisory.rs`. + - Verify: `test … mutation_scope` 25 passed; `test … hooks` 34 passed; `test … maintenance` 49 passed; `test … mutation_trace` 159 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean. - [ ] T04: `Run explicit reconciliation from sce doctor --fix and report recommendation` (status:todo) - Task ID: T04 @@ -328,7 +334,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - Scope: In — real Git repos, linked worktrees, and repository Agent Trace DBs. Every race, ordering and timing scenario uses deterministic coordination (barriers, channels, existing `on_lock_contention`/`on_worker_entered`-style seams, parked phase hooks, injected clock); **no timing-based sleeps as correctness proofs**. T02 and T03 add unit-level coverage for their own code; T05 owns the production-path matrix below and each scenario must fail when the guarded behavior is deliberately broken. - *Reconciliation core (explicit path):* (S1) `sce doctor --fix` reclaims an orphan pin left by an interrupted coordinate; (S2) `A → B → C → D` retains every historical tree; (S3) worktree A's pin is retained while worktree B needs the tree (**preservation**); (S4) missing required local pin fails closed with zero deletion; (S5) malformed/symbolic refs cause zero deletion; (S6) concurrent `coordinate()` and reconcile cannot race pin deletion — a coordinate attempt parked on the lock via barrier proceeds only after the pass releases it, and a pin it creates is never deleted by the in-flight pass; (S7) dropping the explicit-pass future while the deletion worker is parked at the worker-entered seam leaves the lock held until the worker is released, then free; (S8) idempotent reconciliation; (S9) plain `sce doctor` mutates neither refs nor state; `--fix` reports the true result for every outcome kind (completed, completed-with-persist-warning, failed kinds, busy); (S10) explicit pass against an external holder returns `Skipped(Busy)`, leaves state byte-identical, and does not count as a failure; (S11) explicit pass with DB failure leaves refs untouched, records a failure streak, never arms the external-taint marker; (S12) the explicit pass and the advisory each complete with one lock acquisition and no self-deadlock, and each returns busy (never deadlocks) against an externally held lock; (S13) with worktree B's required pin deliberately missing, reconciling A deletes only A's eligible pins, does not create or repair B's pin, does not fail, and never deletes any pin protecting B's tree. - *Hook path and latency (P1):* (S14) the advisory is invoked only after a durably completed `Close`/`Flush` (including `MarkerClearAfterCommit`), never from `Start`/`Advance`/`Abandon`/failed coordinates, and never reaches reconciliation, ref inventory, or DB open — its signature takes none, and spy/panic seams on the reconciler, inventory and opener are never called across every boundary and outcome; `coordinate()` keeps all of its existing snapshot and pinning semantics (required pins are still created and intact — protecting pinning and durable evidence is not weakened), and the advisory itself performs no Git-ref creation, update, or deletion: the test compares the ref set produced by a full hook run against an equivalent coordinate-only baseline (or injects a fake advisory seam and asserts the real advisory adds no ref delta), so any ref change from the coordinator is legitimate and is never attributed to maintenance; (S15) direct advisory contention (scenario A): an explicit pass parked at the post-inventory seam (and, in a second parameterization, at the post-DB-open seam) holds `WorktreeLock`; `advise_if_due` is invoked directly, returns `Busy` through a channel handshake without waiting for the pass, and leaves the state file byte-identical and the ref set unchanged; (S16) coordinator contention (scenario B): with the explicit pass parked holding the lock, a `coordinate()` Close/Flush with a short injected lock timeout fails with the existing lock-timeout error before durable completion and mutates nothing, the existing bounded-acquisition semantics are unchanged, the failed coordinate is a legitimate pre-commit failure, the advisory seam is never invoked after it, and once the pass is released the same boundary succeeds (documents the D1 concurrency contract); (S17) the advisory runs only after the coordinator's lock is released — a seam asserts the worktree lock is acquirable at invocation; (S18a) normal completion: every advisory outcome, a state-write failure, a corrupt state file, and a seam error leave the hook result and `Ok` status identical to the coordinate-only baseline; the marker state is snapshotted immediately before the advisory runs and asserted byte-identical (existence) afterwards, so the advisory never creates, clears, or modifies it, and the normal coordinator-cleared state is unchanged; (S18b) `MarkerClearAfterCommit`: reusing the coordinator's existing marker-clear failure fixture (no change to `coordinator.rs`, `ProtectedWorktree`, or recovery semantics; if unreachable from the hook layer, a fake coordinate seam returns that variant while a real armed marker file exists), the boundary is classified as durably completed, the hook returns the same successful empty-stdout result with the advisory allowed to run, the marker the failed clear left behind is asserted still armed and unmodified after the advisory, and a subsequent real `coordinate()` invocation observes the inherited taint and takes the existing conservative recovery path; (S33) post-coordinate handoff race (scenario C): `coordinate()` completes successfully and releases its lock; a seam pauses the hook immediately before the advisory call; a competing explicit reconciliation acquires `WorktreeLock` through a barrier; the advisory resumes and returns `Busy`; the already committed boundary still returns success, and the maintenance state and refs are untouched by the advisory. - - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) clock-skew tolerance, with an injected clock for every case (`FUTURE_SKEW_TOLERANCE` = 5 min): (1) stored `last_success` equal to `now` — valid, age 0, no advice, file byte-identical; (2) one minute ahead of `now` — valid, age 0, preserved, no advice, no rewrite; (3) exactly five minutes ahead — valid (boundary is inclusive), age 0, preserved, no advice, no rewrite; (4) more than five minutes ahead (just beyond, and far-future) for `last_success`, `last_advised` and `anchor` — invalid, normalized once under the lock and persisted per D3 (invalid fields cleared; with `last_success` and `anchor` both unusable the advisory fires immediately once, persists `last_advised = now` and a fresh `anchor = now`); (5) clock rollback beyond tolerance (stored values valid at write time, injected `now` moved back more than five minutes) — treated as case 4; a rollback within tolerance is case 2/3 and never makes an advisory eligible by itself; (6) repeated reads after normalization — the file is byte-identical across repeated calls, no rewrite occurs, and eligibility is neither postponed nor slid; (7) process restart (fresh call re-reading the file) followed by clock advance past `RECONCILIATION_ADVISORY_AFTER` — advice recurs exactly once, proving eventual eligibility. After every **successful state-writing** operation an assertion checks the exact invariant `t ≤ now + FUTURE_SKEW_TOLERANCE` for every stored timestamp (not "no timestamp is in the future"); after read-only, lock-busy, or failed operations the assertion is instead that the state file is byte-identical to its prior content, with no normalization claimed; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next successful recorded operation (until then, read-only and failed operations leave them untouched and treat them as default state). + - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one time-based advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; a failure recorded after `last_advised` is advised once inside the window, then suppressed, then repeats on the normal cadence until a successful explicit pass clears it; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) clock-skew tolerance, with an injected clock for every case (`FUTURE_SKEW_TOLERANCE` = 5 min): (1) stored `last_success` equal to `now` — valid, age 0, no advice, file byte-identical; (2) one minute ahead of `now` — valid, age 0, preserved, no advice, no rewrite; (3) exactly five minutes ahead — valid (boundary is inclusive), age 0, preserved, no advice, no rewrite; (4) more than five minutes ahead (just beyond, and far-future) for `last_success`, `last_advised` and `anchor` — invalid, normalized once under the lock and persisted per D3 (invalid fields cleared; with `last_success` and `anchor` both unusable the advisory fires immediately once, persists `last_advised = now` and a fresh `anchor = now`); (5) clock rollback beyond tolerance (stored values valid at write time, injected `now` moved back more than five minutes) — treated as case 4; a rollback within tolerance is case 2/3 and never makes an advisory eligible by itself; (6) repeated reads after normalization — the file is byte-identical across repeated calls, no rewrite occurs, and eligibility is neither postponed nor slid; (7) process restart (fresh call re-reading the file) followed by clock advance past `RECONCILIATION_ADVISORY_AFTER` — advice recurs exactly once, proving eventual eligibility. After every **successful state-writing** operation an assertion checks the exact invariant `t ≤ now + FUTURE_SKEW_TOLERANCE` for every stored timestamp (not "no timestamp is in the future"); after read-only, lock-busy, or failed operations the assertion is instead that the state file is byte-identical to its prior content, with no normalization claimed; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next successful recorded operation (until then, read-only and failed operations leave them untouched and treat them as default state). - *Timestamp-normalization regressions (P2; injected clocks and deterministic synchronization, no sleeps):* (S34) read-only doctor observing an invalid future timestamp — plain `sce doctor` reports "recommended" with the invalid-timestamp note, the state file is byte-identical afterwards, and it is not reported as normalized; (S35) lock-busy advisory — with the lock held through a barrier and an invalid future timestamp stored, `advise_if_due` returns `Busy` and the state file is byte-identical; the same holds for a lock-busy explicit pass (`Skipped(Busy)`); (S36) failed state write — an injected write error (and, in a second parameterization, a rename interrupted before replacement) during a normalizing operation leaves the previous state byte-identical and readable, emits no advice, reports no normalization (`StateUnavailable` for the advisory, `CompletedStatePersistFailed` for the explicit pass), and a later successful operation still normalizes it; (S37) successful normalization — a normalizing advisory and a normalizing explicit pass each persist state in which every timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`, valid near-future values (within tolerance) are preserved byte-for-byte, and a repeated read performs no rewrite and does not move eligibility forward; (S38) process restart after normalization — a fresh call re-reading the persisted file observes the normalized timestamps, performs no rewrite, and does not re-fire the immediate advice; (S39) eventual eligibility — with invalid timestamps unnormalized (S34/S35/S36 states) repeated read-only evaluations never push eligibility out, and after the injected clock advances past `RECONCILIATION_ADVISORY_AFTER` a successful advisory fires exactly once and persists normalized state. - *Explicit recovery:* (S26) `sce doctor --fix` runs and reports its true result from each of: failed streak, stale advice, invalid timestamps, corrupt state, unwritable state (→ `CompletedStatePersistFailed`); (S27) explicit success resets the failure streak and sets `last_success`; explicit failure increments the streak without touching `last_success`; lock-busy never changes state. - *DB opening (D4):* (S28) missing DB — fails with the `Missing` typed error from the single non-creating read-only open (no existence pre-check); no database file, parent directory, WAL, or shared-memory file is created (directory listing before and after is identical, including when the parent directory is absent); Git refs are neither listed nor deleted (spy on `list_pins`/`delete_pins` never called); (S29) existing DB with incompatible schema — opening is permitted; fails with `IncompatibleSchema`; no migration or repair row written (migration table rows identical before and after); no ref deleted; (S30) existing DB with missing repository metadata — fails with `MissingMetadata`; no `repository_metadata` row inserted; (S31) existing DB with repository identity mismatch — fails with `RepositoryMismatch`; no row changed; (S32) existing valid DB — identity and schema readiness are verified before any ref listing (ordering asserted via seam), the multiprocess-WAL configuration is preserved, the pass completes, and no initialization, repair, or migration runs. For S29–S31 the assertions are logical: a before/after dump of every table's rows is equal (no migration, repair, initialization, or application-table INSERT/UPDATE/DELETE), refs are unchanged, and the external-taint marker is unarmed. Filesystem immutability is **not** asserted for existing-DB failure cases, because incidental Turso WAL/shared-memory bookkeeping is permitted when unavoidable; only S28 asserts that no database-related file or directory is created. From 57d44f600e120e51a780ce9304cbf04c62233c99 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 01:51:57 +0200 Subject: [PATCH 07/16] doctor: Run snapshot ref reconciliation from --fix Make `sce doctor --fix` execute one explicit snapshot-ref reconciliation pass and report typed outcomes, while plain `sce doctor` remains read-only and reports when maintenance is recommended. Add a runtime facade and injected seams to preserve doctor testability and expose counts, failure kinds, warnings, and busy results. Update the `mutation-cursor-ref-reconciliation-wiring` plan and related context for T04. Co-authored-by: SCE --- cli/src/services/agent_trace_db/lifecycle.rs | 199 ++++++++- cli/src/services/agent_trace_db/repository.rs | 22 +- cli/src/services/doctor/mod.rs | 163 ++++++- cli/src/services/doctor/render.rs | 277 ++++++++++++ .../services/mutation_trace/runtime/mod.rs | 5 + .../mutation_trace/runtime/ref_advisory.rs | 1 - .../mutation_trace/runtime/ref_doctor.rs | 405 ++++++++++++++++++ context/cli/doctor-ref-reconciliation.md | 46 ++ .../cli/mutation-trace-ref-reconciliation.md | 6 +- context/context-map.md | 1 + ...tation-cursor-ref-reconciliation-wiring.md | 27 +- 11 files changed, 1124 insertions(+), 28 deletions(-) create mode 100644 cli/src/services/mutation_trace/runtime/ref_doctor.rs create mode 100644 context/cli/doctor-ref-reconciliation.md diff --git a/cli/src/services/agent_trace_db/lifecycle.rs b/cli/src/services/agent_trace_db/lifecycle.rs index 8263c2191..0a4f56569 100644 --- a/cli/src/services/agent_trace_db/lifecycle.rs +++ b/cli/src/services/agent_trace_db/lifecycle.rs @@ -14,7 +14,9 @@ use crate::services::repository_identity::resolve::{ resolve_repository_identity, RepositoryIdentitySource, }; -use super::repository::{RepositoryAgentTraceDb, RepositoryAgentTraceDbSpec}; +use super::repository::{ + ExistingRepositoryDbError, RepositoryAgentTraceDb, RepositoryAgentTraceDbSpec, +}; #[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] pub struct AgentTraceDbLifecycle; @@ -157,11 +159,24 @@ pub async fn diagnose_agent_trace_db_health(repo_root: Option<&Path>) -> Vec { - if let Err(error) = db.ensure_schema_ready_for_hooks().await { - problems.push(HealthProblem { + problems.extend(inspect_existing_db_schema(&db_path, || {}).await); + problems +} + +async fn inspect_existing_db_schema( + db_path: &Path, + before_open: impl FnOnce(), +) -> Vec { + let mut problems = Vec::new(); + if db_path.exists() && !db_path.is_file() { + return problems; + } + + match RepositoryAgentTraceDb::open_existing_schema_ready_at(db_path, before_open).await { + Ok(_db) => {} + Err(ExistingRepositoryDbError::Missing { .. }) => {} + Err(ExistingRepositoryDbError::IncompatibleSchema(error)) => { + problems.push(HealthProblem { kind: HealthProblemKind::AgentTraceDbSchemaNotReady, category: HealthCategory::GlobalState, severity: HealthSeverity::Error, @@ -175,10 +190,13 @@ pub async fn diagnose_agent_trace_db_health(repo_root: Option<&Path>) -> Vec { - problems.push(HealthProblem { + } + Err(error) => { + let error = match error { + ExistingRepositoryDbError::Unreadable(source) => source.to_string(), + other => other.to_string(), + }; + problems.push(HealthProblem { kind: HealthProblemKind::AgentTraceDbConnectionFailed, category: HealthCategory::GlobalState, severity: HealthSeverity::Error, @@ -192,7 +210,6 @@ pub async fn diagnose_agent_trace_db_health(repo_root: Option<&Path>) -> Vec) -> Result Vec { + let mut names = std::fs::read_dir(dir) + .expect("read dir") + .map(|entry| { + entry + .expect("entry") + .file_name() + .to_string_lossy() + .into_owned() + }) + .collect::>(); + names.sort(); + names + } + + async fn schema_objects(path: &Path) -> Vec { + let db = RepositoryAgentTraceDb::open_existing_without_migrations_at(path) + .await + .expect("read-only open"); + let mut names = db + .query_map("SELECT name FROM sqlite_master ORDER BY name", (), |row| { + row.get::(0).map_err(Into::into) + }) + .await + .expect("schema query"); + names.sort(); + names + } + + async fn count_rows(path: &Path, table: &str) -> i64 { + let db = RepositoryAgentTraceDb::open_existing_without_migrations_at(path) + .await + .expect("read-only open"); + db.query_map(&format!("SELECT COUNT(*) FROM {table}"), (), |row| { + row.get::(0).map_err(Into::into) + }) + .await + .expect("count query")[0] + } + + #[tokio::test(flavor = "multi_thread")] + async fn missing_database_is_not_created_and_leaves_no_parent_directory() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir + .path() + .join("state") + .join("repos") + .join("id") + .join("agent-trace.db"); + + let problems = inspect_existing_db_schema(&path, || {}).await; + let again = inspect_existing_db_schema(&path, || {}).await; + + assert!(problems.is_empty()); + assert!(again.is_empty()); + assert!(!path.exists()); + assert!(dir_entries(dir.path()).is_empty()); + } + + #[tokio::test(flavor = "multi_thread")] + async fn database_removed_immediately_before_open_is_not_recreated() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + RepositoryAgentTraceDb::new_at(&path).await.expect("create"); + assert!(path.is_file()); + + let problems = inspect_existing_db_schema(&path, || { + for entry in std::fs::read_dir(dir.path()).expect("read dir") { + std::fs::remove_file(entry.expect("entry").path()).expect("remove"); + } + }) + .await; + + assert!(problems.is_empty()); + assert!(dir_entries(dir.path()).is_empty()); + } + + #[tokio::test(flavor = "multi_thread")] + async fn valid_database_is_inspected_without_problems_or_logical_change() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + let db = RepositoryAgentTraceDb::new_at(&path).await.expect("create"); + db.verify_or_initialize_repository_metadata("repo") + .await + .expect("metadata"); + drop(db); + let objects_before = schema_objects(&path).await; + let migrations_before = count_rows(&path, "__sce_migrations").await; + let metadata_before = count_rows(&path, "repository_metadata").await; + + let problems = inspect_existing_db_schema(&path, || {}).await; + let first_files = dir_entries(dir.path()); + let repeat = inspect_existing_db_schema(&path, || {}).await; + + assert!(problems.is_empty()); + assert!(repeat.is_empty()); + assert_eq!(dir_entries(dir.path()), first_files); + assert_eq!(schema_objects(&path).await, objects_before); + assert_eq!( + count_rows(&path, "__sce_migrations").await, + migrations_before + ); + assert_eq!( + count_rows(&path, "repository_metadata").await, + metadata_before + ); + } + + #[tokio::test(flavor = "multi_thread")] + async fn incompatible_schema_is_diagnosed_without_initializing_or_migrating() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + let raw = RepositoryAgentTraceDb::open_without_migrations_at(&path) + .await + .expect("create empty"); + raw.execute("CREATE TABLE unrelated (id INTEGER)", ()) + .await + .expect("unrelated table"); + drop(raw); + let objects_before = schema_objects(&path).await; + + let problems = inspect_existing_db_schema(&path, || {}).await; + + assert_eq!(problems.len(), 1); + assert_eq!( + problems[0].kind, + HealthProblemKind::AgentTraceDbSchemaNotReady + ); + assert!(problems[0].summary.contains("is not ready")); + assert_eq!(schema_objects(&path).await, objects_before); + assert!(!objects_before.iter().any(|name| name == "__sce_migrations")); + } + + #[tokio::test(flavor = "multi_thread")] + async fn non_database_file_is_unreadable_and_left_byte_identical() { + let dir = tempfile::tempdir().expect("temp dir"); + let path = dir.path().join("agent-trace.db"); + std::fs::write( + &path, + b"this is not a sqlite database at all, just some text bytes....", + ) + .expect("write"); + let before = std::fs::read(&path).expect("read"); + + let problems = inspect_existing_db_schema(&path, || {}).await; + + assert_eq!(problems.len(), 1); + assert_ne!( + problems[0].kind, + HealthProblemKind::AgentTraceDbSchemaNotReady + ); + assert_eq!(std::fs::read(&path).expect("read"), before); + assert_eq!(dir_entries(dir.path()), vec!["agent-trace.db".to_string()]); + } +} diff --git a/cli/src/services/agent_trace_db/repository.rs b/cli/src/services/agent_trace_db/repository.rs index 3e9b35cfd..1bfc8cb95 100644 --- a/cli/src/services/agent_trace_db/repository.rs +++ b/cli/src/services/agent_trace_db/repository.rs @@ -188,12 +188,18 @@ impl RepositoryAgentTraceDb { Self::open_verified_existing_with(path, repository_id, before_open, after_open).await } - async fn open_verified_existing_with( + pub async fn open_existing_schema_ready_at( + path: &std::path::Path, + before_open: impl FnOnce(), + ) -> std::result::Result { + Self::open_existing_schema_ready_with(path, before_open, || {}).await + } + + async fn open_existing_schema_ready_with( path: &std::path::Path, - repository_id: &str, before_open: impl FnOnce(), after_open: impl FnOnce(), - ) -> std::result::Result<(Self, RepositoryMetadata), ExistingRepositoryDbError> { + ) -> std::result::Result { before_open(); let opened = TursoDb::::open_existing_without_migrations_at(path).await; @@ -214,6 +220,16 @@ impl RepositoryAgentTraceDb { db.ensure_schema_ready_for_hooks() .await .map_err(ExistingRepositoryDbError::IncompatibleSchema)?; + Ok(db) + } + + async fn open_verified_existing_with( + path: &std::path::Path, + repository_id: &str, + before_open: impl FnOnce(), + after_open: impl FnOnce(), + ) -> std::result::Result<(Self, RepositoryMetadata), ExistingRepositoryDbError> { + let db = Self::open_existing_schema_ready_with(path, before_open, after_open).await?; let metadata = db .verify_existing_repository_metadata(repository_id) .await?; diff --git a/cli/src/services/doctor/mod.rs b/cli/src/services/doctor/mod.rs index e55670f30..64f6aea97 100644 --- a/cli/src/services/doctor/mod.rs +++ b/cli/src/services/doctor/mod.rs @@ -10,6 +10,10 @@ use crate::services::lifecycle::{ lifecycle_providers, FixOutcome, HealthCategory, HealthFixability, HealthProblem, HealthProblemKind, HealthSeverity, LifecycleProvider, LifecycleProviderId, }; +use crate::services::mutation_trace::runtime::{ + inspect_reconciliation_recommendation, run_explicit_reconciliation, ReconciliationFix, + ReconciliationRecommendation, +}; use crate::services::output_format::OutputFormat; use crate::services::setup; @@ -60,6 +64,8 @@ struct DoctorDependencies<'a, G, S, C, P> { struct DoctorExecution { report: HookDoctorReport, fix_results: Vec, + ref_reconciliation: Option, + ref_reconciliation_fix: Option, } #[derive(Clone, Debug, Eq, PartialEq)] @@ -85,6 +91,8 @@ where &repository_root, &scoped_context, &mutation_scope_repair_seam::, + &run_explicit_reconciliation, + &inspect_reconciliation_recommendation, ) .await; render_report(request, &execution) @@ -99,6 +107,8 @@ async fn execute_doctor_with_context( &str, Option<&crate::services::observability::traits::NoopLogger>, ) -> anyhow::Result, + reconcile_seam: &impl std::ops::AsyncFn(&Path) -> ReconciliationFix, + recommendation_seam: &impl Fn(&Path) -> Option, ) -> DoctorExecution { execute_doctor_with_lifecycle_providers( request, @@ -113,6 +123,8 @@ async fn execute_doctor_with_context( probe_codex_hook_policy: &codex_hook_policy::probe_default, }, mutation_scope_seam, + reconcile_seam, + recommendation_seam, ) .await } @@ -133,10 +145,9 @@ async fn execute_doctor_with_lifecycle_providers( &str, Option<&crate::services::observability::traits::NoopLogger>, ) -> anyhow::Result, + reconcile_seam: &impl std::ops::AsyncFn(&Path) -> ReconciliationFix, + recommendation_seam: &impl Fn(&Path) -> Option, ) -> DoctorExecution { - // Probed exactly once per doctor invocation, then reused for every - // Codex integration inspection below (initial report, `--fix`, and final - // report alike) instead of once per registration or once per report. let policy_readiness = (dependencies.probe_codex_hook_policy)(); let providers = lifecycle_providers(true); @@ -158,6 +169,8 @@ async fn execute_doctor_with_lifecycle_providers( return DoctorExecution { report: initial_report, fix_results: Vec::new(), + ref_reconciliation: recommendation_seam(repository_root), + ref_reconciliation_fix: None, }; } @@ -190,9 +203,53 @@ async fn execute_doctor_with_lifecycle_providers( &mutation_scope_repairs, )); + let reconciliation_fix = reconcile_seam(repository_root).await; + fix_results.push(reconciliation_fix_result(&reconciliation_fix)); + DoctorExecution { report: final_report, fix_results, + ref_reconciliation: None, + ref_reconciliation_fix: Some(reconciliation_fix), + } +} + +fn reconciliation_fix_result(fix: &ReconciliationFix) -> DoctorFixResultRecord { + let (outcome, detail) = match fix { + ReconciliationFix::Completed(counts) => ( + FixResult::Fixed, + format!( + "Snapshot ref reconciliation completed: deleted {}, retained {}, locally required {}.", + counts.deleted, counts.retained, counts.local_required + ), + ), + ReconciliationFix::CompletedStatePersistFailed { counts, warning } => ( + FixResult::Fixed, + format!( + "Snapshot ref reconciliation completed: deleted {}, retained {}, locally required {}. Warning: {warning}", + counts.deleted, counts.retained, counts.local_required + ), + ), + ReconciliationFix::Failed { + kind, + message, + state_warning, + } => { + let mut detail = format!("Snapshot ref reconciliation failed ({kind}): {message}"); + if let Some(warning) = state_warning { + detail = format!("{detail} Warning: {warning}"); + } + (FixResult::Failed, detail) + } + ReconciliationFix::Busy => ( + FixResult::Skipped, + "Snapshot ref reconciliation skipped: another mutation boundary or maintenance pass holds the worktree lock. Retry `sce doctor --fix`.".to_string(), + ), + }; + DoctorFixResultRecord { + category: ProblemCategory::MutationScopeHealth, + outcome, + detail, } } @@ -497,3 +554,103 @@ fn is_executable(metadata: &fs::Metadata) -> bool { fn is_executable(metadata: &fs::Metadata) -> bool { metadata.is_file() } + +#[cfg(test)] +mod reconciliation_fix_tests { + use super::{reconciliation_fix_result, FixResult, ReconciliationFix}; + use crate::services::mutation_trace::runtime::{ReconciliationCounts, StateWarning}; + + const COUNTS: ReconciliationCounts = ReconciliationCounts { + deleted: 3, + retained: 2, + local_required: 1, + }; + + fn not_applied() -> StateWarning { + StateWarning::NotApplied { + phase: "rename", + cause: "disk full".to_string(), + } + } + + fn durability_uncertain() -> StateWarning { + StateWarning::DurabilityUncertain { + phase: "sync_parent_directory", + cause: "io error".to_string(), + } + } + + #[test] + fn completed_pass_reports_counts_as_fixed() { + let record = reconciliation_fix_result(&ReconciliationFix::Completed(COUNTS)); + assert_eq!(record.outcome, FixResult::Fixed); + assert_eq!( + record.detail, + "Snapshot ref reconciliation completed: deleted 3, retained 2, locally required 1." + ); + } + + #[test] + fn completed_pass_with_pre_rename_failure_says_update_was_not_applied() { + let record = reconciliation_fix_result(&ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: not_applied(), + }); + assert_eq!(record.outcome, FixResult::Fixed); + assert!(record.detail.contains("deleted 3")); + assert!(record + .detail + .contains("Maintenance-state update was not applied.")); + assert!(record.detail.contains("disk full")); + assert!(!record.detail.contains("were not recorded")); + } + + #[test] + fn completed_pass_with_post_rename_failure_says_durability_unconfirmed() { + let record = reconciliation_fix_result(&ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: durability_uncertain(), + }); + assert_eq!(record.outcome, FixResult::Fixed); + assert!(record + .detail + .contains("Maintenance-state durability could not be confirmed.")); + assert!(!record.detail.contains("not applied")); + assert!(!record.detail.contains("were not recorded")); + } + + #[test] + fn unreadable_previous_state_does_not_claim_nothing_was_recorded() { + let record = reconciliation_fix_result(&ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::PreviousStateUnreadable { + cause: "permission denied".to_string(), + }, + }); + assert!(record.detail.contains("could not be read")); + assert!(record.detail.contains("permission denied")); + assert!(!record.detail.contains("were not recorded")); + } + + #[test] + fn failed_pass_is_never_reported_as_success() { + let record = reconciliation_fix_result(&ReconciliationFix::Failed { + kind: "agent_trace_db_missing", + message: "db missing".to_string(), + state_warning: Some(not_applied()), + }); + assert_eq!(record.outcome, FixResult::Failed); + assert!(record.detail.contains("agent_trace_db_missing")); + assert!(record.detail.contains("db missing")); + assert!(record + .detail + .contains("Maintenance-state update was not applied.")); + } + + #[test] + fn busy_pass_is_skipped_not_fixed() { + let record = reconciliation_fix_result(&ReconciliationFix::Busy); + assert_eq!(record.outcome, FixResult::Skipped); + assert!(record.detail.contains("skipped")); + } +} diff --git a/cli/src/services/doctor/render.rs b/cli/src/services/doctor/render.rs index de83f0c53..a4dc43e17 100644 --- a/cli/src/services/doctor/render.rs +++ b/cli/src/services/doctor/render.rs @@ -13,6 +13,9 @@ use super::types::{ PostCommitAutoSyncState, ProblemKind, ProblemSeverity, Readiness, }; use super::{DoctorExecution, DoctorFormat, DoctorMode, DoctorRequest, NAME}; +use crate::services::mutation_trace::runtime::{ + ReconciliationCounts, ReconciliationFix, ReconciliationRecommendation, StateWarning, +}; /// Guidance message rendered in the Integrations section when no integration /// targets are configured, detected, or both. @@ -48,9 +51,58 @@ fn format_execution(execution: &DoctorExecution) -> String { } } + if let Some(recommendation) = &execution.ref_reconciliation { + lines.push(format!("\n{}:", heading("Snapshot ref reconciliation"))); + lines.push(format!( + " [{}] {}", + value("recommended"), + value(&recommendation_text(recommendation)) + )); + } + lines.join("\n") } +fn recommendation_text(recommendation: &ReconciliationRecommendation) -> String { + let mut parts = vec!["Reconciliation is recommended. Run `sce doctor --fix`.".to_string()]; + if recommendation.last_attempt_failed { + parts.push(format!( + "The latest reconciliation attempt failed ({} consecutive failure(s)).", + recommendation.consecutive_failures + )); + } + if let Some(age) = recommendation.last_attempt_age_ms { + parts.push(format!( + "Last reconciliation attempt {} ago.", + format_age(age) + )); + } + if recommendation.state_unusable { + parts.push("Maintenance state is unreadable or invalid.".to_string()); + } + if recommendation.invalid_timestamps { + parts.push("Maintenance state has invalid timestamps.".to_string()); + } + if let Some(age) = recommendation.last_success_age_ms { + parts.push(format!("Last success {} ago.", format_age(age))); + } + if let Some(age) = recommendation.last_advised_age_ms { + parts.push(format!("Last reminder {} ago.", format_age(age))); + } + parts.join(" ") +} + +fn format_age(age_ms: i64) -> String { + let minutes = age_ms / 60_000; + if minutes < 60 { + format!("{minutes}m") + } else if minutes < 60 * 24 { + format!("{}h", minutes / 60) + } else { + format!("{}d", minutes / (60 * 24)) + } +} + fn format_report(report: &HookDoctorReport) -> String { format_report_with_color_policy(report, supports_color()) } @@ -887,6 +939,8 @@ fn render_report_json(execution: &DoctorExecution) -> Result { } else { Vec::new() }, + "ref_reconciliation": execution.ref_reconciliation.as_ref().map(ref_reconciliation_json), + "ref_reconciliation_fix": execution.ref_reconciliation_fix.as_ref().map(ref_reconciliation_fix_json), }); serde_json::to_string_pretty(&payload).context("failed to serialize doctor report to JSON") @@ -921,3 +975,226 @@ fn hook_content_state(state: HookContentState) -> &'static str { HookContentState::Unknown => "unknown", } } + +fn ref_reconciliation_json(recommendation: &ReconciliationRecommendation) -> serde_json::Value { + json!({ + "recommended": true, + "last_attempt_failed": recommendation.last_attempt_failed, + "consecutive_failures": recommendation.consecutive_failures, + "state_unusable": recommendation.state_unusable, + "invalid_timestamps": recommendation.invalid_timestamps, + "last_success_age_ms": recommendation.last_success_age_ms, + "last_attempt_age_ms": recommendation.last_attempt_age_ms, + "last_advised_age_ms": recommendation.last_advised_age_ms, + "text": recommendation_text(recommendation), + }) +} + +fn ref_reconciliation_fix_json(fix: &ReconciliationFix) -> serde_json::Value { + match fix { + ReconciliationFix::Completed(counts) => json!({ + "outcome": "completed", + "counts": reconciliation_counts_json(counts), + "failure": null, + "state_warning": null, + }), + ReconciliationFix::CompletedStatePersistFailed { counts, warning } => json!({ + "outcome": "completed_state_persist_failed", + "counts": reconciliation_counts_json(counts), + "failure": null, + "state_warning": state_warning_json(warning), + }), + ReconciliationFix::Failed { + kind, + message, + state_warning, + } => json!({ + "outcome": "failed", + "counts": null, + "failure": { "kind": kind, "message": message }, + "state_warning": state_warning.as_ref().map(state_warning_json), + }), + ReconciliationFix::Busy => json!({ + "outcome": "busy", + "counts": null, + "failure": null, + "state_warning": null, + }), + } +} + +fn reconciliation_counts_json(counts: &ReconciliationCounts) -> serde_json::Value { + json!({ + "deleted": counts.deleted, + "retained": counts.retained, + "local_required": counts.local_required, + }) +} + +fn state_warning_json(warning: &StateWarning) -> serde_json::Value { + json!({ + "kind": warning.kind(), + "phase": warning.phase(), + "message": warning.message(), + "cause": warning.cause(), + }) +} + +#[cfg(test)] +mod reconciliation_render_tests { + use serde_json::json; + + use super::{recommendation_text, ref_reconciliation_fix_json}; + use crate::services::mutation_trace::runtime::{ + ReconciliationCounts, ReconciliationFix, ReconciliationRecommendation, StateWarning, + }; + + const COUNTS: ReconciliationCounts = ReconciliationCounts { + deleted: 3, + retained: 2, + local_required: 1, + }; + + fn recommendation(last_attempt_age_ms: Option) -> ReconciliationRecommendation { + ReconciliationRecommendation { + invalid_timestamps: false, + state_unusable: false, + last_success_age_ms: Some(3_600_000 * 30), + last_attempt_age_ms, + last_advised_age_ms: Some(120_000), + last_attempt_failed: last_attempt_age_ms.is_some(), + consecutive_failures: 2, + } + } + + #[test] + fn recommendation_shows_last_attempt_age_for_failed_recent_attempt() { + let text = recommendation_text(&recommendation(Some(12 * 60_000))); + assert!(text.contains("Last reconciliation attempt 12m ago.")); + assert!( + text.contains("The latest reconciliation attempt failed (2 consecutive failure(s)).") + ); + assert!(text.contains("Last success 1d ago.")); + assert!(text.contains("Last reminder 2m ago.")); + } + + #[test] + fn recommendation_omits_last_attempt_age_when_no_attempt_exists() { + let text = recommendation_text(&recommendation(None)); + assert!(!text.contains("Last reconciliation attempt")); + assert!(text.contains("Reconciliation is recommended.")); + } + + #[test] + fn json_completed_outcome() { + assert_eq!( + ref_reconciliation_fix_json(&ReconciliationFix::Completed(COUNTS)), + json!({ + "outcome": "completed", + "counts": { "deleted": 3, "retained": 2, "local_required": 1 }, + "failure": null, + "state_warning": null, + }) + ); + } + + #[test] + fn json_completed_state_persist_failed_distinguishes_warning_kinds() { + let not_applied = + ref_reconciliation_fix_json(&ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::NotApplied { + phase: "rename", + cause: "denied".to_string(), + }, + }); + assert_eq!( + not_applied, + json!({ + "outcome": "completed_state_persist_failed", + "counts": { "deleted": 3, "retained": 2, "local_required": 1 }, + "failure": null, + "state_warning": { + "kind": "not_applied", + "phase": "rename", + "message": "Maintenance-state update was not applied.", + "cause": "denied", + }, + }) + ); + + let uncertain = + ref_reconciliation_fix_json(&ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::DurabilityUncertain { + phase: "sync_parent_directory", + cause: "eio".to_string(), + }, + }); + assert_eq!(uncertain["state_warning"]["kind"], "durability_uncertain"); + assert_eq!(uncertain["state_warning"]["phase"], "sync_parent_directory"); + assert_eq!( + uncertain["state_warning"]["message"], + "Maintenance-state durability could not be confirmed." + ); + + let unreadable = + ref_reconciliation_fix_json(&ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::PreviousStateUnreadable { + cause: "eacces".to_string(), + }, + }); + assert_eq!( + unreadable["state_warning"]["kind"], + "previous_state_unreadable" + ); + assert_eq!( + unreadable["state_warning"]["phase"], + serde_json::Value::Null + ); + } + + #[test] + fn json_failed_outcome_carries_failure_and_optional_warning() { + let without_warning = ref_reconciliation_fix_json(&ReconciliationFix::Failed { + kind: "agent_trace_db_missing", + message: "db missing".to_string(), + state_warning: None, + }); + assert_eq!( + without_warning, + json!({ + "outcome": "failed", + "counts": null, + "failure": { "kind": "agent_trace_db_missing", "message": "db missing" }, + "state_warning": null, + }) + ); + + let with_warning = ref_reconciliation_fix_json(&ReconciliationFix::Failed { + kind: "missing_required_pins", + message: "pins".to_string(), + state_warning: Some(StateWarning::NotApplied { + phase: "write_staging", + cause: "full".to_string(), + }), + }); + assert_eq!(with_warning["failure"]["kind"], "missing_required_pins"); + assert_eq!(with_warning["state_warning"]["kind"], "not_applied"); + assert_eq!(with_warning["state_warning"]["phase"], "write_staging"); + } + + #[test] + fn json_busy_outcome() { + assert_eq!( + ref_reconciliation_fix_json(&ReconciliationFix::Busy), + json!({ + "outcome": "busy", + "counts": null, + "failure": null, + "state_warning": null, + }) + ); + } +} diff --git a/cli/src/services/mutation_trace/runtime/mod.rs b/cli/src/services/mutation_trace/runtime/mod.rs index dbeea17bb..4722da2da 100644 --- a/cli/src/services/mutation_trace/runtime/mod.rs +++ b/cli/src/services/mutation_trace/runtime/mod.rs @@ -6,6 +6,7 @@ mod maintenance_state; mod mutation_attribution; mod protected_worktree; mod ref_advisory; +mod ref_doctor; mod ref_maintenance; mod ref_reconciliation; mod scope_runtime; @@ -30,6 +31,10 @@ pub(crate) use mutation_attribution::{ TreeReadSource, MAX_MUTATION_ATTRIBUTION_EVENTS, }; pub(crate) use ref_advisory::{advise_after_completed_boundary, AdvisoryReport, AdvisorySeverity}; +pub(crate) use ref_doctor::{ + inspect_reconciliation_recommendation, run_explicit_reconciliation, ReconciliationCounts, + ReconciliationFix, ReconciliationRecommendation, StateWarning, +}; #[allow(unused_imports)] pub(crate) use scope_runtime::{ abandon_scope, AbandonRecoveryReason, AbandonScopeError, AbandonScopeOutcome, diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs index ae718b87f..85e6ae5bc 100644 --- a/cli/src/services/mutation_trace/runtime/ref_advisory.rs +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -1,4 +1,3 @@ - use std::path::Path; use std::time::Duration; diff --git a/cli/src/services/mutation_trace/runtime/ref_doctor.rs b/cli/src/services/mutation_trace/runtime/ref_doctor.rs new file mode 100644 index 000000000..752e224f5 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_doctor.rs @@ -0,0 +1,405 @@ +use std::path::Path; + +use super::maintenance_state::{ + derive_git_dir, evaluate_recommendation, read_state, state_path, system_now_ms, AttemptOutcome, + MaintenanceState, PersistFailure, PersistPhase, StateRead, +}; +use super::ref_maintenance::{reconcile_explicit, ExplicitOutcome, StatePersistWarning}; +use super::ref_reconciliation::{ReconcileError, ReconciliationReport}; +use crate::services::agent_trace_db::repository::ExistingRepositoryDbError; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) struct ReconciliationCounts { + pub(crate) deleted: usize, + pub(crate) retained: usize, + pub(crate) local_required: usize, +} + +const NOT_APPLIED_MESSAGE: &str = "Maintenance-state update was not applied."; +const DURABILITY_UNCERTAIN_MESSAGE: &str = "Maintenance-state durability could not be confirmed."; +const PREVIOUS_STATE_UNREADABLE_MESSAGE: &str = + "Maintenance state could not be read; the update was not attempted."; + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) enum StateWarning { + NotApplied { phase: &'static str, cause: String }, + DurabilityUncertain { phase: &'static str, cause: String }, + PreviousStateUnreadable { cause: String }, +} + +impl StateWarning { + pub(crate) fn kind(&self) -> &'static str { + match self { + StateWarning::NotApplied { .. } => "not_applied", + StateWarning::DurabilityUncertain { .. } => "durability_uncertain", + StateWarning::PreviousStateUnreadable { .. } => "previous_state_unreadable", + } + } + + pub(crate) fn phase(&self) -> Option<&'static str> { + match self { + StateWarning::NotApplied { phase, .. } + | StateWarning::DurabilityUncertain { phase, .. } => Some(phase), + StateWarning::PreviousStateUnreadable { .. } => None, + } + } + + pub(crate) fn message(&self) -> &'static str { + match self { + StateWarning::NotApplied { .. } => NOT_APPLIED_MESSAGE, + StateWarning::DurabilityUncertain { .. } => DURABILITY_UNCERTAIN_MESSAGE, + StateWarning::PreviousStateUnreadable { .. } => PREVIOUS_STATE_UNREADABLE_MESSAGE, + } + } + + pub(crate) fn cause(&self) -> &str { + match self { + StateWarning::NotApplied { cause, .. } + | StateWarning::DurabilityUncertain { cause, .. } + | StateWarning::PreviousStateUnreadable { cause } => cause, + } + } +} + +impl std::fmt::Display for StateWarning { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}", self.message())?; + match self.phase() { + Some(phase) => write!(f, " Cause ({phase}): {}", self.cause()), + None => write!(f, " Cause: {}", self.cause()), + } + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) enum ReconciliationFix { + Completed(ReconciliationCounts), + CompletedStatePersistFailed { + counts: ReconciliationCounts, + warning: StateWarning, + }, + Failed { + kind: &'static str, + message: String, + state_warning: Option, + }, + Busy, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) struct ReconciliationRecommendation { + pub(crate) invalid_timestamps: bool, + pub(crate) state_unusable: bool, + pub(crate) last_success_age_ms: Option, + pub(crate) last_attempt_age_ms: Option, + pub(crate) last_advised_age_ms: Option, + pub(crate) last_attempt_failed: bool, + pub(crate) consecutive_failures: u32, +} + +pub(crate) async fn run_explicit_reconciliation(repository_root: &Path) -> ReconciliationFix { + fix_from_outcome(reconcile_explicit(repository_root).await) +} + +pub(crate) fn inspect_reconciliation_recommendation( + repository_root: &Path, +) -> Option { + inspect_recommendation_at(repository_root, system_now_ms()) +} + +fn inspect_recommendation_at( + repository_root: &Path, + now: i64, +) -> Option { + let git_dir = derive_git_dir(repository_root)?; + let (state, state_unusable) = match read_state(&state_path(&git_dir)) { + Ok(StateRead::Valid(state)) => (state, false), + Ok(StateRead::Absent) => (MaintenanceState::default(), false), + Ok(StateRead::Unusable) | Err(_) => (MaintenanceState::default(), true), + }; + let recommendation = evaluate_recommendation(&state, now); + if !recommendation.recommended && !state_unusable { + return None; + } + Some(ReconciliationRecommendation { + invalid_timestamps: recommendation.invalid_timestamps, + state_unusable, + last_success_age_ms: recommendation.last_success_age_ms, + last_attempt_age_ms: recommendation.last_attempt_age_ms, + last_advised_age_ms: recommendation.last_advised_age_ms, + last_attempt_failed: recommendation.last_attempt_outcome == Some(AttemptOutcome::Failed), + consecutive_failures: recommendation.consecutive_failures, + }) +} + +fn fix_from_outcome(outcome: ExplicitOutcome) -> ReconciliationFix { + match outcome { + ExplicitOutcome::Completed(report) => ReconciliationFix::Completed(counts(report)), + ExplicitOutcome::CompletedStatePersistFailed { report, warning } => { + ReconciliationFix::CompletedStatePersistFailed { + counts: counts(report), + warning: state_warning_from(warning), + } + } + ExplicitOutcome::Failed { + error, + state_warning, + } => ReconciliationFix::Failed { + kind: failure_kind(&error), + message: error.to_string(), + state_warning: state_warning.map(state_warning_from), + }, + ExplicitOutcome::Skipped(_) => ReconciliationFix::Busy, + } +} + +fn state_warning_from(warning: StatePersistWarning) -> StateWarning { + match warning { + StatePersistWarning::PreviousStateUnreadable(error) => { + StateWarning::PreviousStateUnreadable { + cause: error.to_string(), + } + } + StatePersistWarning::WriteFailed(PersistFailure::NotApplied { phase, source, .. }) => { + StateWarning::NotApplied { + phase: phase_name(phase), + cause: source.to_string(), + } + } + StatePersistWarning::WriteFailed(PersistFailure::DurabilityUncertain { phase, source }) => { + StateWarning::DurabilityUncertain { + phase: phase_name(phase), + cause: source.to_string(), + } + } + } +} + +fn phase_name(phase: PersistPhase) -> &'static str { + match phase { + PersistPhase::Serialize => "serialize", + PersistPhase::CreateStaging => "create_staging", + PersistPhase::WriteStaging => "write_staging", + PersistPhase::SyncStaging => "sync_staging", + PersistPhase::Rename => "rename", + PersistPhase::OpenParentDirectory => "open_parent_directory", + PersistPhase::SyncParentDirectory => "sync_parent_directory", + } +} + +fn counts(report: ReconciliationReport) -> ReconciliationCounts { + ReconciliationCounts { + deleted: report.deleted, + retained: report.retained, + local_required: report.local_required, + } +} + +fn failure_kind(error: &ReconcileError) -> &'static str { + match error { + ReconcileError::GitDir(_) => "git_dir_unavailable", + ReconcileError::Lock(_) => "lock_unavailable", + ReconcileError::CheckoutIdentity(_) => "worktree_identity_unavailable", + ReconcileError::AgentTraceDbUnavailable(source) => { + match source.downcast_ref::() { + Some(ExistingRepositoryDbError::Missing { .. }) => "agent_trace_db_missing", + Some(ExistingRepositoryDbError::Unreadable(_)) => "agent_trace_db_unreadable", + Some(ExistingRepositoryDbError::IncompatibleSchema(_)) => { + "agent_trace_db_incompatible_schema" + } + Some(ExistingRepositoryDbError::MissingMetadata) => { + "agent_trace_db_missing_metadata" + } + Some(ExistingRepositoryDbError::RepositoryMismatch { .. }) => { + "agent_trace_db_repository_mismatch" + } + None => "agent_trace_db_unavailable", + } + } + ReconcileError::SnapshotService(_) => "snapshot_service_unavailable", + ReconcileError::PinInventory(_) => "pin_inventory_failed", + ReconcileError::MalformedPin { .. } => "malformed_ref", + ReconcileError::DurableRoots(_) => "durable_roots_unavailable", + ReconcileError::MissingRequiredPins { .. } => "missing_required_pins", + ReconcileError::DeleteTransaction(_) => "delete_transaction_failed", + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::services::mutation_trace::runtime::maintenance_state::{ + record_failure, record_success, StoredReport, + }; + + fn write_state(git_dir: &Path, state: &MaintenanceState) { + let path = state_path(git_dir); + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(path, serde_json::to_vec(state).unwrap()).unwrap(); + } + + fn repo() -> tempfile::TempDir { + let dir = tempfile::tempdir().unwrap(); + std::fs::create_dir(dir.path().join(".git")).unwrap(); + dir + } + + #[test] + fn recommendation_is_absent_for_fresh_and_recent_state() { + let repo = repo(); + assert_eq!(inspect_recommendation_at(repo.path(), 1_000), None); + + let state = record_success( + &MaintenanceState::default(), + 1_000, + StoredReport { + retained: 1, + deleted: 0, + local_required: 1, + }, + ); + write_state(&repo.path().join(".git"), &state); + assert_eq!(inspect_recommendation_at(repo.path(), 2_000), None); + } + + #[test] + fn recommendation_reports_failure_and_never_writes_state() { + let repo = repo(); + let git_dir = repo.path().join(".git"); + let state = record_failure(&MaintenanceState::default(), 1_000, "kind", "message"); + write_state(&git_dir, &state); + let before = std::fs::read(state_path(&git_dir)).unwrap(); + + let recommendation = inspect_recommendation_at(repo.path(), 2_000).unwrap(); + assert!(recommendation.last_attempt_failed); + assert_eq!(recommendation.consecutive_failures, 1); + assert!(!recommendation.state_unusable); + assert_eq!(std::fs::read(state_path(&git_dir)).unwrap(), before); + } + + #[test] + fn recommendation_flags_unusable_and_invalid_state_without_rewriting() { + let repo = repo(); + let git_dir = repo.path().join(".git"); + let path = state_path(&git_dir); + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(&path, b"not json").unwrap(); + + let unusable = inspect_recommendation_at(repo.path(), 2_000).unwrap(); + assert!(unusable.state_unusable); + assert_eq!(std::fs::read(&path).unwrap(), b"not json"); + + let state = MaintenanceState { + last_success: Some(i64::MAX / 2), + ..MaintenanceState::default() + }; + write_state(&git_dir, &state); + let before = std::fs::read(&path).unwrap(); + let invalid = inspect_recommendation_at(repo.path(), 2_000).unwrap(); + assert!(invalid.invalid_timestamps); + assert_eq!(std::fs::read(&path).unwrap(), before); + } + + fn report() -> ReconciliationReport { + ReconciliationReport { + local_required: 1, + retained: 2, + deleted: 3, + } + } + + fn io_error() -> std::io::Error { + std::io::Error::other("injected") + } + + #[test] + fn pre_rename_persist_failure_maps_to_not_applied() { + let fix = fix_from_outcome(ExplicitOutcome::CompletedStatePersistFailed { + report: report(), + warning: StatePersistWarning::WriteFailed(PersistFailure::NotApplied { + phase: PersistPhase::Rename, + source: io_error(), + staging_cleanup: None, + }), + }); + let ReconciliationFix::CompletedStatePersistFailed { counts, warning } = fix else { + panic!("expected persist-failed completion"); + }; + assert_eq!(counts.deleted, 3); + assert_eq!(warning.kind(), "not_applied"); + assert_eq!(warning.phase(), Some("rename")); + assert_eq!( + warning.message(), + "Maintenance-state update was not applied." + ); + } + + #[test] + fn post_rename_persist_failure_maps_to_durability_uncertain() { + let fix = fix_from_outcome(ExplicitOutcome::CompletedStatePersistFailed { + report: report(), + warning: StatePersistWarning::WriteFailed(PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + source: io_error(), + }), + }); + let ReconciliationFix::CompletedStatePersistFailed { warning, .. } = fix else { + panic!("expected persist-failed completion"); + }; + assert_eq!(warning.kind(), "durability_uncertain"); + assert_eq!(warning.phase(), Some("sync_parent_directory")); + assert_eq!( + warning.message(), + "Maintenance-state durability could not be confirmed." + ); + } + + #[test] + fn unreadable_previous_state_maps_to_its_own_warning_on_failure() { + let fix = fix_from_outcome(ExplicitOutcome::Failed { + error: ReconcileError::MissingRequiredPins { missing: vec![] }, + state_warning: Some(StatePersistWarning::PreviousStateUnreadable(io_error())), + }); + let ReconciliationFix::Failed { state_warning, .. } = fix else { + panic!("expected failure"); + }; + let warning = state_warning.expect("warning retained"); + assert_eq!(warning.kind(), "previous_state_unreadable"); + assert_eq!(warning.phase(), None); + assert!(!warning.message().contains("not recorded")); + } + + #[test] + fn outcomes_map_to_distinct_fix_results() { + let report = ReconciliationReport { + local_required: 1, + retained: 2, + deleted: 3, + }; + assert_eq!( + fix_from_outcome(ExplicitOutcome::Completed(report)), + ReconciliationFix::Completed(ReconciliationCounts { + deleted: 3, + retained: 2, + local_required: 1, + }) + ); + assert_eq!( + fix_from_outcome(ExplicitOutcome::Skipped( + super::super::ref_maintenance::SkipReason::Busy + )), + ReconciliationFix::Busy + ); + let failed = fix_from_outcome(ExplicitOutcome::Failed { + error: ReconcileError::MissingRequiredPins { missing: vec![] }, + state_warning: None, + }); + assert!(matches!( + failed, + ReconciliationFix::Failed { + kind: "missing_required_pins", + .. + } + )); + } +} diff --git a/context/cli/doctor-ref-reconciliation.md b/context/cli/doctor-ref-reconciliation.md new file mode 100644 index 000000000..1d53a602c --- /dev/null +++ b/context/cli/doctor-ref-reconciliation.md @@ -0,0 +1,46 @@ +# Doctor snapshot-ref reconciliation + +`sce doctor --fix` is the only production caller of snapshot-ref reconciliation. +Plain `sce doctor` only reports whether a pass is recommended. Automatic +reclamation is not delivered; it is a documented follow-up (see +[`mutation-scope-ref-advisory.md`](mutation-scope-ref-advisory.md) and +[`mutation-trace-ref-reconciliation.md`](mutation-trace-ref-reconciliation.md)). + +## Runtime facade + +`cli/src/services/mutation_trace/runtime/ref_doctor.rs` is the only crate-visible +surface doctor uses; it keeps the maintenance types private to `runtime`: + +- `run_explicit_reconciliation(repository_root)` wraps `reconcile_explicit` + (one `WorktreeLock` acquisition, verified-existing Agent Trace DB, state + recorded under the same lock) and returns a plain `ReconciliationFix`. +- `inspect_reconciliation_recommendation(repository_root)` derives the Git dir + from the filesystem, reads `/sce/ref-maintenance.json`, and applies the + pure `evaluate_recommendation`. It takes no lock and writes no state or ref. It + returns `None` unless reconciliation is recommended or the state is unusable. + +Doctor receives both as injected seams (`execute_doctor_with_lifecycle_providers`). + +## `--fix` reporting + +After the existing lifecycle, config and mutation-scope repairs, `--fix` always +runs one pass, regardless of the recommendation, failure streak, or timestamp +validity, and appends one fix-result row (category `mutation_scope_health`): + +| Pass result | Outcome | Detail | +| --- | --- | --- | +| `Completed` | `fixed` | deleted / retained / locally required counts | +| `CompletedStatePersistFailed` | `fixed` | counts plus a diagnostics-not-recorded warning | +| `Failed` | `failed` | typed kind (for example `agent_trace_db_missing`, `agent_trace_db_incompatible_schema`, `agent_trace_db_missing_metadata`, `agent_trace_db_repository_mismatch`, `malformed_ref`, `missing_required_pins`, `durable_roots_unavailable`, `delete_transaction_failed`) and message, plus any state warning | +| `Skipped(Busy)` | `skipped` | another boundary or pass holds the worktree lock; retry | + +A busy pass leaves state unchanged and is not a failure. While a pass runs, a +mutation boundary may wait up to the 10 s coordinator lock timeout. + +## Plain `sce doctor` + +When reconciliation is recommended (stale since the last success, an outstanding +failed attempt, invalid timestamps, or an unreadable state file) the text report +adds a `Snapshot ref reconciliation` section and the JSON report carries a +`ref_reconciliation` object (`null` otherwise). The recommendation does not depend +on whether a hook reminder was ever seen. diff --git a/context/cli/mutation-trace-ref-reconciliation.md b/context/cli/mutation-trace-ref-reconciliation.md index 71297c02b..537c4a872 100644 --- a/context/cli/mutation-trace-ref-reconciliation.md +++ b/context/cli/mutation-trace-ref-reconciliation.md @@ -33,8 +33,10 @@ harness-wiring PR's). ## Status: retained and unwired (PR3) The `cli-async-persistence-cleanup-pr3` plan chose **retain and defer**: this -pass and its helpers are kept, but nothing in production calls them and PR3 adds -no caller, hook, command or doctor wiring. Accepted consequence: pins created +pass and its helpers are kept, but nothing in production called them and PR3 +added no caller, hook, command or doctor wiring. `sce doctor --fix` is now the +explicit caller (see [`doctor-ref-reconciliation.md`](doctor-ref-reconciliation.md)); +no automatic caller exists. Historical consequence under PR3: pins created by `pin_tree` in `coordinator.rs` are never deleted in production, so pins left by crashed or failed `coordinate()` paths accumulate until removed by hand (`git update-ref -d refs/sce/mutation-cursor//`). diff --git a/context/context-map.md b/context/context-map.md index a8232ad75..d6443fe0e 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -43,6 +43,7 @@ Feature/domain context: - `context/cli/claude-mutation-scope-health.md` (the `doctor-mutation-scope-health` plan's Claude task, extended by the `doctor-mutation-scope-fix` plan's T04: `classify_health` in `cli/src/services/hooks/claude_mutation_scope/health.rs` maps `recovery_pending`/`attempts` onto `healthy | recovering | blocked | invalid` — `recovery_pending == false` is `healthy` regardless of live attempts; `recovery_pending == true` with `attempts` empty is `recovering` (the barrier's own next-`PreToolUse` flush clears it); `recovery_pending == true` with `attempts` non-empty is `blocked`, the exact incident shape this classifier exists to surface, since nothing in the ordinary hook lifecycle retries, removes, flushes, or clears from that shape; a read/parse failure is `invalid`. T04 adds a separate, `doctor`-invoked repair path (`assess_repairability`/`repair_blocked` in `health.rs`/`lifecycle.rs`, wired into `sce doctor --fix` by the same plan's T05) for the `blocked` shape: `AutoFixable` only when every currently persisted attempt is already `pending_abandon` (no established-abandon-intent `pending_start`/`active` attempt present); `repair_blocked` re-reads and re-proves that same condition inside one lock-protected read-only state transaction before releasing the lock and retrying each attempt's already-established seam `abandon` call independently, clearing the `recovery_pending` barrier only once no attempts remain — this classifier's own read-only behavior is unaffected by that repair path's existence) - `context/cli/mutation-trace-ref-reconciliation.md` (the conservative per-worktree snapshot-ref reconciliation pass in `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs`, built by the `mutation-cursor-ref-reconciliation` plan: `reconcile_worktree(repository_root, open_db)` → `pub(super) reconcile_worktree_inner(.., on_lock_contention)`, both returning `ReconciliationOutcome` (`Reconciled(ReconciliationReport { local_required, retained, deleted })` for a pass that ran | `SkippedNoCheckoutIdentity` — an `Ok`, not an `Err` — when no current checkout identity could be derived), a variant-per-fallible-step `ReconcileError` with no `Other`, and the module-owned `RECONCILIATION_LOCK_TIMEOUT`; the two-invariant model — a strictly per-worktree fail-closed local-consistency check via `load_tree_roots(W)` vs. a repository-wide deletion-safety set via `load_all_tree_roots()` so an `A`-owned ref is retained whenever any worktree still durably needs its tree — run entirely under the same `/sce/mutation-cursor.lock` `WorktreeLock` `coordinate()` holds, with the repository-wide read kept coherent by being one SQL statement / one DB snapshot rather than a repository-global lock; deletes only SCE-owned refs via one atomic `git update-ref --no-deref --stdin`, writes no `mutation_trace_*` row, never arms `ExternalTaintMarker`, runs no `git gc`; imperative durability maintenance below the verified Quint protocol; reclaims orphan/unreferenced refs only for the namespace of a checkout id a current worktree still derives — a namespace no current worktree owns (identity-based: a deleted linked worktree, or checkout-id metadata loss followed by `get_or_create_checkout_id` minting a fresh id on a still-present worktree) is beyond every per-worktree pass and left to a recorded future repository-scoped unowned-namespace operation, so the current pass does not bound all orphan-ref growth; `reconcile_worktree` has no `pub(crate)` re-export and no harness/command wiring yet) - `context/cli/mutation-scope-ref-advisory.md` (the advisory-only hook trigger in `cli/src/services/hooks/mutation_scope.rs` built by T03 of the `mutation-cursor-ref-reconciliation-wiring` plan: a lightweight `advise_if_due` check after a durably completed `Close`/`Flush`, logged and, when actionable, written as an `SCE:` line to an injectable stderr writer (best-effort delivery), with the per-worktree `ref-maintenance.json` state; no reconciliation, Git, or DB work on the hook path) +- `context/cli/doctor-ref-reconciliation.md` (the `sce doctor --fix` caller of snapshot-ref reconciliation built by T04 of the `mutation-cursor-ref-reconciliation-wiring` plan: the `runtime::ref_doctor` facade, typed fix-result rows for completed/completed-with-persist-warning/failed/busy passes, and the read-only recommendation section/`ref_reconciliation` JSON object in plain `sce doctor`) - `context/cli/mutation-trace-snapshot-service.md` (the isolated Git snapshot and ref-pinning service `runtime::git_snapshot::GitSnapshotService` in `cli/src/services/mutation_trace/runtime/git_snapshot.rs`: `new` resolving an absolute `git_dir`, `capture_tree` snapshotting staged/unstaged/untracked/deleted worktree state into the repository's normal object database via a throwaway temp index, `pin_tree` protecting a durable tree with a create-only idempotent **direct** `refs/sce/mutation-cursor//` ref, `diff_trees` emitting `patch.rs`-parseable raw diff text; plus the callerless reconciliation substrate — worktree-scoped `list_pins` inventory returning `Result, PinInventoryError>` that rejects a symbolic ref inside the namespace (mutation-cursor pins are direct refs) as `MalformedRef`, matchable separately from a `git for-each-ref` execution failure, and conditional-atomic `delete_pins` running one `git update-ref --no-deref --stdin` transaction of SHA-conditioned deletes — no-dereference so an inventory→delete direct-ref→symref race cannot escape the inventoried namespace, plus a fail-closed pre-check — that aborts whole if any ref changed since inventory; `REF_NAMESPACE` + `pin_ref_prefix` as the single source of truth for the pin path) - `context/cli/mutation-trace-external-taint.md` (the worktree-local mutation-cursor durability boundary in `cli/src/services/mutation_trace/runtime/external_taint.rs`, built by the `mutation-cursor-external-taint` plan: the `ExternalTaintMarker` primitive — `new(git_dir)`/`exists()`/`persist()`/`clear()` over an empty file at `/sce/mutation-cursor-tainted` whose existence is its entire state, `checkout::persist_checkout_id_inner`-style durability (`sync_data` plus best-effort `#[cfg(unix)]` parent-dir `sync_all`), idempotent persist/clear, `NotFound`-on-clear as success, no `Drop` deletion — as the concrete runtime refinement of the abstract `ProtocolState.external_taint`; armed by the reshaped `coordinate()` entrypoint write-ahead after the `WorktreeLock` and before Agent Trace DB acquisition (a caller-supplied DB provider closure), cleared only on a successful `CoordinateOutcome`, with dedicated fail-closed pre-commit `CoordinateError::ExternalTaintMarker` (`Inspect`/`Persist` only)/`AgentTraceDbUnavailable` variants plus a post-commit `MarkerClearAfterCommit { source, committed }` that carries the durable outcome so a failed trailing clear never hides a committed `MutationEvent`; an inherited marker seeds an invocation-local `external_taint_pending` flag that overlays `protocol::database_failure` onto each freshly loaded projection so `recover` runs once against the captured snapshot, held across a losing recovery CAS and cleared once it lands) - `context/sce/cli-exit-code-contract.md` (stable class-based `sce` process exit-code contract in `cli/src/app.rs`, so automation can branch on failure category without parsing error text) diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index e5c78203b..aacd3366d 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -99,12 +99,12 @@ Facts the design relies on. T01 must re-read each cited location and either conf - Transitions: advisory — anchor and `last_advised` only. Explicit success — `last_attempt` = `last_success` = now, `last_report` set, `consecutive_failures` = 0, failure fields cleared. Explicit failure (DB unavailable/invalid, malformed ref, missing required pins, durable-root read error, delete-transaction error) — `last_attempt` = now, outcome `failed`, `consecutive_failures` += 1, `last_failure` set, `last_success` untouched. Lock contention (including explicit lock timeout) — **no state change**; never classified as a failure. - **Invalid timestamp normalization (deterministic).** A timestamp is invalid when it is strictly more than `FUTURE_SKEW_TOLERANCE` (5 min) after the injected `now` (far-future values, or clock rollback beyond the tolerance); a timestamp exactly equal to `now + FUTURE_SKEW_TOLERANCE` is valid. A valid timestamp ahead of `now` (within the tolerance) is treated as age 0, so a small clock rollback never makes an advisory eligible by itself. Under `WorktreeLock`, `advise_if_due` and `reconcile_explicit` normalize once, as part of their successful state write: invalid `last_success`/`last_attempt`/`last_advised`/`anchor` values are cleared; if `last_success` and `anchor` are both unusable the state is **immediately eligible** for advice, so the advisory persists `last_advised = now` and returns `Advised` in the same locked operation, and a fresh `anchor = now` is written. The normalized state is persisted before returning, so a second read sees valid timestamps, performs no rewrite, and does not shift or postpone eligibility; once the injected clock advances, ordinary window-based eligibility resumes. If the normalizing write fails or the lock is busy, nothing is persisted, the invalid bytes stay as they were, and the in-memory evaluation still treats them conservatively (eligible), so a later successful operation can normalize them. Plain `sce doctor` never writes; it evaluates invalid timestamps as "recommended" with an "invalid maintenance timestamps" note and leaves the file byte-identical. `sce doctor --fix` runs regardless of timestamp validity and rewrites valid state afterwards. - Recovery: a missing, unparsable, wrong-version, or oversized file is treated as default and rewritten by the next recorded operation; it never fails an explicit pass. - - Result distinction for explicit passes (typed outcome): `Completed(report)`; `CompletedStatePersistFailed(report, warning)` — the cleanup stands and is reported as done, with a visible warning that diagnostics were not recorded; `Failed(error)`; `Skipped(Busy)`. Never merged. Advisory outcomes: `Anchored | NoAction | Advised | Busy | StateUnavailable`. + - Result distinction for explicit passes (typed outcome `ExplicitOutcome`): `Completed(report)`; `CompletedStatePersistFailed { report, warning }` — the cleanup stands and is reported as done, with a typed `StatePersistWarning`; `Failed { error, state_warning: Option }` — a failed pass keeps both the reconciliation error and any persistence warning; `Skipped(Busy)`. Never merged. `StatePersistWarning` is `PreviousStateUnreadable(io::Error)` (the update was not attempted) or `WriteFailed(PersistFailure)`, where `PersistFailure::NotApplied` means the atomic replacement did not occur and the previous state is preserved, and `PersistFailure::DurabilityUncertain` means the replacement occurred (the new state may be visible) but crash durability could not be confirmed. The doctor facade preserves this as `StateWarning::{NotApplied, DurabilityUncertain, PreviousStateUnreadable}` with a stable kind, an optional phase, and a cause; it reports "Maintenance-state update was not applied." / "Maintenance-state durability could not be confirmed." and never claims that diagnostics were not recorded when they may have been. Advisory outcomes: `Anchored | NoAction | Advised | AdvisedDurabilityUncertain | Busy | StateWriteFailed | StateUnavailable`. - Time comes from an injected wall clock (`Fn() -> i64` unix ms) passed as a generic parameter for deterministic tests. - **D4 — Verified-existing Agent Trace DB opener (round 1).** - - Side-effect guarantees are stated per database state. **Missing DB:** fail before invoking Turso; create no DB file, no parent directory, and no WAL or shared-memory file; list or delete no Git refs. **Existing but invalid DB:** opening the existing file is permitted and incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable (documented); logical immutability is required — no schema migration, no migration-metadata repair, no repository-metadata initialization, no application-table INSERT/UPDATE/DELETE, no Git-ref deletion — and verification fails with the matching typed error. **Existing valid DB:** verify repository identity and schema readiness before reconciliation, preserve the existing multiprocess-WAL configuration, allow only the database access reconciliation needs, and run no initialization, repair, or migration. - - Contract for reconciliation: (1) never create a DB file or any parent directory (missing DB fails before Turso is invoked); (2) never run migrations; (3) never call `repair_missing_repository_schema_migration_metadata`; (4) never call `verify_or_initialize_repository_metadata` or otherwise insert/update repository metadata; (5) verify repository identity (stored `repository_id` equals the identity resolved for this checkout, `source_instance_id` valid) and schema readiness (baseline plus the migration providing durable mutation roots) **before** any ref listing or deletion; (6) any failure returns an error with refs untouched, as a maintenance error that never arms `ExternalTaintMarker` and never becomes `CoordinateError::AgentTraceDbUnavailable`. - - The existing hook opener cannot satisfy (1), (3), or (4), and changing it would alter every hook caller. The smallest satisfying change is a dedicated opener in `agent_trace_storage/mod.rs` (for example `resolve_existing_agent_trace_storage_for_maintenance`) plus one read-only verification method on `RepositoryAgentTraceDb` (for example `verify_existing_repository_metadata(repository_id)`) wrapping the existing `select_repository_metadata_row` without insert. The opener first checks that the resolved DB path exists as a file; if not, it fails before touching Turso. If Turso offers no read-only open, T01 records that and the opener uses only `SELECT`-class statements after open; incidental WAL bookkeeping on an existing DB is an accepted limitation, and tests assert logical tables are unchanged. + - Side-effect guarantees are stated per database state. **Missing DB:** Turso is invoked in non-creating read-only mode (`read_only(true)` with multiprocess WAL), so that open cannot create the database; no DB file, parent directory, WAL, or shared-memory file is created; no Git refs are listed or deleted. **Existing but invalid DB:** opening the existing file is permitted and incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable (documented); logical immutability is required — no schema migration, no migration-metadata repair, no repository-metadata initialization, no application-table INSERT/UPDATE/DELETE, no Git-ref deletion — and verification fails with the matching typed error. **Existing valid DB:** verify repository identity and schema readiness before reconciliation, preserve the existing multiprocess-WAL configuration, allow only the database access reconciliation needs, and run no initialization, repair, or migration. + - Contract for reconciliation: (1) never create a DB file or any parent directory (the non-creating read-only Turso open fails for a missing DB, including one removed between a preliminary check and the open); (2) never run migrations; (3) never call `repair_missing_repository_schema_migration_metadata`; (4) never call `verify_or_initialize_repository_metadata` or otherwise insert/update repository metadata; (5) verify repository identity (stored `repository_id` equals the identity resolved for this checkout, `source_instance_id` valid) and schema readiness (baseline plus the migration providing durable mutation roots) **before** any ref listing or deletion; (6) any failure returns an error with refs untouched, as a maintenance error that never arms `ExternalTaintMarker` and never becomes `CoordinateError::AgentTraceDbUnavailable`. + - The existing hook opener cannot satisfy (1), (3), or (4), and changing it would alter every hook caller. The smallest satisfying change is a dedicated opener in `agent_trace_storage/mod.rs` (for example `resolve_existing_agent_trace_storage_for_maintenance`) plus one read-only verification method on `RepositoryAgentTraceDb` (for example `verify_existing_repository_metadata(repository_id)`) wrapping the existing `select_repository_metadata_row` without insert. The opener relies on no separate existence check: the Turso open itself is non-creating and read-only (`TursoDb::open_existing_without_migrations_at`), and a failed open is classified `Missing` only when the path is then absent. Plain `sce doctor` shares that open through `RepositoryAgentTraceDb::open_existing_schema_ready_at`, which inspects schema readiness without reading or requiring repository metadata. Incidental WAL/shared-memory bookkeeping on an existing DB is an accepted limitation: the guarantee is logical immutability (schema objects and migration/metadata rows unchanged) and, for non-database files, byte identity; full filesystem byte-identity for valid Turso databases is not claimed. - Typed failure kinds (small enum, converted to `anyhow` at the existing `open_db` seam): `Missing`, `Unreadable`, `IncompatibleSchema`, `MissingMetadata`, `RepositoryMismatch`. The existing hook opener and all its callers are unchanged. - Considered and rejected: opening the DB before taking the lock to shorten lock-hold. It would require a second lock acquisition just to record an open failure, violating the one-acquisition-per-operation rule in D1; the lock-first order of the existing reconciler is kept and the D1 concurrency contract documents the consequence. @@ -131,7 +131,7 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - Validate: `git diff fix-logging...HEAD` inspection plus `nix shell nixpkgs#ripgrep -c rg "dyn Future|BoxFuture|tokio::spawn|Runtime::new|block_on|tokio::time::timeout|JoinHandle|abort\(|static .*(Mutex|RwLock|OnceLock|HashMap)" ` returns no new hits. - [ ] AC8: Mutation-cursor and Quint model-based tests remain valid without protocol changes. - Validate: `git diff fix-logging...HEAD --stat` shows no change to `protocol.rs` or `spec/mutation_cursor.qnt`; the existing `mbt` tests pass. -- [ ] AC9: `sce doctor` stays non-mutating; `sce doctor --fix` reports its actual cleanup result and recovers from every failure, stale, and invalid-state condition. +- [ ] AC9: `sce doctor` stays non-mutating — it never creates or logically modifies the Agent Trace DB (its schema inspection uses the non-creating read-only open of D4) and never writes maintenance state or refs; `sce doctor --fix` reports its actual cleanup result, exposes it as a typed `ref_reconciliation_fix` JSON object (outcome, counts, failure kind, state-warning kind/phase) alongside the unchanged `fix_results`, distinguishes `NotApplied` from `DurabilityUncertain` persistence warnings, and recovers from every failure, stale, and invalid-state condition. - Validate: T05 S9, S25, S26 and S27 pass: counts for deleted/retained/local-required; distinct output for completed, completed-with-state-persist-failure, failed (unavailable/invalid DB, malformed refs, missing pins), and busy; plain `sce doctor` writes no ref and no state; a recommendation is shown when stale, when the latest explicit pass failed (regardless of `last_advised`), or when state is invalid. The doctor recommendation is independent of whether a hook reminder was ever seen, which is why lossy hook delivery is acceptable. - [ ] AC10: Context matches the implementation, the "retained and unwired" status is gone, and context states that automatic reclamation is deferred to a follow-up with its prerequisites. - Validate: `nix shell nixpkgs#ripgrep -c rg -i "retained and unwired|unwired" context/cli` returns no stale claims; `rg -i "automatic.*(reclam|reconcil)" context` hits only the deferral/follow-up statements; documented APIs match code. @@ -142,7 +142,7 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. - Two guarantee classes (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, failed state read or write, write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. - Validate: T05 S19–S25 and S34–S39 pass: single time-based advice per 24 h window with persistence before emission (a newly recorded failure may be advised inside that window, once, per AC5); concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's failure leg assert byte-identical preexisting state instead; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write leaving the previous state unchanged; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. -- [ ] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability, and only the missing-DB case asserts no newly created database-related files. +- [ ] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability (incidental WAL bookkeeping on existing databases is permitted and documented), and only the missing-DB case asserts no newly created database-related files or directories, including when the file is removed between a preliminary check and the open. Plain `sce doctor` shares the same non-creating read-only open. - **Missing DB:** fails before Turso is invoked; no database file, parent directory, or WAL/shared-memory file is created; no Git ref is listed or deleted. - **Existing but invalid DB:** opening is permitted; incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable and documented; no schema migration, migration-metadata repair, repository-metadata initialization, or application-table INSERT/UPDATE/DELETE; no Git-ref deletion; verification fails with the matching typed error. Tests do not assert filesystem immutability for this case. - **Existing valid DB:** repository identity and schema readiness are verified before reconciliation; the existing multiprocess-WAL configuration is preserved; only the access reconciliation requires is performed; no initialization, repair, or migration. @@ -321,13 +321,24 @@ Record actual outcomes in the validation report; compilation alone does not vali - Tests: `mutation_scope` advisory tests (recording logger + injected sink) cover all seven report shapes (severity, event ID, outcome, persistence warning, diagnostic), missing logger, failing writer (result stays `Ok("")`), `MarkerClearAfterCommit`, failed coordinate (no advisory, no output), and single coordinate/advise invocation; `ref_advisory` adds `maintenance_report_maps_every_outcome_to_severity_and_diagnostic` over real outcomes. Existing `ref_advisory` tests already cover the 24 h cadence and an in-window new-failure reminder. Removed the now-unneeded file-level `#![allow(dead_code)]` from `ref_advisory.rs`. - Verify: `test … mutation_scope` 25 passed; `test … hooks` 34 passed; `test … maintenance` 49 passed; `test … mutation_trace` 159 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean. -- [ ] T04: `Run explicit reconciliation from sce doctor --fix and report recommendation` (status:todo) +- [x] T04: `Run explicit reconciliation from sce doctor --fix and report recommendation` (status:done) - Task ID: T04 - Scope: In — extend the existing doctor `--fix` flow to call `reconcile_explicit` (one lock acquisition inside the entrypoint; doctor never holds the worktree lock itself); additive report fields/rows for retained, deleted, and locally-required counts; distinct rows for completed, completed-with-state-persist-warning, failed (unavailable/invalid DB with the D4 failure kind, malformed refs, missing required pins, durable-root errors, delete-transaction error), and skipped-busy; plain `sce doctor` stays read-only but reads the state file (never writes it or any ref) and reports whether reconciliation is recommended (last success age, last attempt and failure streak, last advice, invalid-timestamp note) via the pure `evaluate_recommendation`; skipped or failed cleanup is never reported as success; `--fix` runs regardless of recommendation, failure streak, or timestamp validity; no DB bootstrap/migration/repair; no change to adapter recovery state; existing output contracts backward compatible; the documented concurrency note (a mutation boundary may wait up to 10 s while `--fix` reconciles) appears in help or fix output wording only if existing output contracts allow it additively. Out — new doctor subcommands, changes to mutation-scope repair logic. - Dependencies: T02 - Done when: `sce doctor --fix` always runs a pass and reports its true typed result; a successful pass resets the failure streak and sets `last_success`; `sce doctor` performs no ref or state mutation; existing doctor tests still pass unchanged. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml doctor` - - Context synchronization: pending + - Completed: 2026-10-09 + - Files changed: `cli/src/services/doctor/mod.rs`, `cli/src/services/doctor/render.rs`, `cli/src/services/mutation_trace/runtime/mod.rs`, `cli/src/services/mutation_trace/runtime/ref_doctor.rs`, `cli/src/services/mutation_trace/runtime/ref_advisory.rs` (rustfmt-only: removed a leading blank line), `context/plans/mutation-cursor-ref-reconciliation-wiring.md` + - Result: Added the `pub(crate)` facade `runtime/ref_doctor.rs` (`run_explicit_reconciliation`, `inspect_reconciliation_recommendation`, plain-data `ReconciliationFix` / `ReconciliationCounts` / `ReconciliationRecommendation`) over `reconcile_explicit` and `evaluate_recommendation`. `sce doctor --fix` now always runs one explicit pass after the existing fixes and appends a fix-result row (category `mutation_scope_health`): completed → `fixed` with deleted/retained/locally-required counts; completed with state-persist failure → `fixed` plus a diagnostics warning; failed → `failed` with a typed kind (`agent_trace_db_missing`, `agent_trace_db_incompatible_schema`, `agent_trace_db_missing_metadata`, `agent_trace_db_repository_mismatch`, `malformed_ref`, `missing_required_pins`, `durable_roots_unavailable`, `delete_transaction_failed`, …) and any state warning; busy → `skipped`. Plain `sce doctor` reads the state file through a filesystem-derived Git dir (no lock, no write, no ref access) and, only when reconciliation is recommended or the state is unusable, adds a `Snapshot ref reconciliation` text section and a JSON `ref_reconciliation` object (`null` otherwise). Doctor receives the two calls as injected seams. No DB bootstrap/migration/repair, no adapter recovery state, and no mutation-scope repair logic changed. + - Verify: `test … doctor` 9 passed (4 new fix-result mapping tests, 4 new `ref_doctor` tests covering absent/recent state, failure recommendation with byte-identical state, unusable/invalid-timestamp state left unrewritten, outcome mapping); `test … mutation_trace` 163 passed; full `cargo test` 437 passed; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; no comments added and the pre-existing comment in the edited `execute_doctor_with_lifecycle_providers` was removed per instruction. + - Context impact: important (doctor now runs explicit reconciliation and reports the recommendation; T06 owns the durable context update). + - Remediation (2026-10-09, PR #305 review of `bab36fe`; T04 reopened `done`→`in_progress`, returned to `done`/`synced` after the verification below; T01–T03 unchanged, T05–T06 untouched). Architecture unchanged: single `WorktreeLock`, static dispatch, one reconciliation pass per `--fix`. + - Plain doctor DB inspection now opens through `RepositoryAgentTraceDb::open_existing_schema_ready_at` (non-creating read-only Turso open, multiprocess WAL preserved, no migrations, no metadata requirement); `Missing` yields no extra problem, `IncompatibleSchema`/`Unreadable` keep their existing diagnostics. The write-capable hook opener is untouched. + - `ReconciliationFix` keeps a typed `StateWarning` (`NotApplied` | `DurabilityUncertain` | `PreviousStateUnreadable`, with kind, phase, cause) instead of a string; human text reads "Maintenance-state update was not applied." / "Maintenance-state durability could not be confirmed.". + - JSON adds the optional top-level `ref_reconciliation_fix` object (`outcome` ∈ `completed | completed_state_persist_failed | failed | busy`, `counts`, `failure {kind,message}`, `state_warning {kind,phase,message,cause}`); `null` for plain doctor; `fix_results` unchanged. + - Human recommendation text adds "Last reconciliation attempt Nm ago." when an attempt timestamp exists. + - D3/D4/AC9/AC14 corrected above. + - Context synchronization: synced - [ ] T05: `Add production-path and concurrency regression tests` (status:todo) - Task ID: T05 From a2dda7ae795e7c9f6b5d702175afac56718af821 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 02:15:04 +0200 Subject: [PATCH 08/16] tests: Add mutation ref reconciliation regression coverage Cover retention of durable roots across worktrees, fail-closed handling of malformed namespace refs, and advisory busy behavior while an explicit pass holds the lock. Parameterize the durable-root fixture so the production-path tests exercise foreign worktree roots without repairing missing pins. Plan: mutation-cursor-ref-reconciliation-wiring (T05) Co-authored-by: SCE --- .../runtime/ref_maintenance/tests.rs | 136 +++++++++++++++++- ...tation-cursor-ref-reconciliation-wiring.md | 10 +- 2 files changed, 142 insertions(+), 4 deletions(-) diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs index 52f9e825a..dc21eddd7 100644 --- a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs @@ -8,6 +8,7 @@ use super::super::maintenance_state::{ write_state_atomically_with, FaultInjectingFilesystem, MaintenanceState, PersistFailure, PersistPhase, StateRead, }; +use super::super::ref_advisory::{advise_if_due, AdvisoryOutcome}; use super::super::ref_reconciliation::{ reconcile_with_held_lock, reconcile_worktree, ReconcileError, ReconcilePhase, ReconciliationOutcome, ReconciliationReport, @@ -142,14 +143,18 @@ async fn pin_orphan(root: &Path) -> String { } async fn seed_durable_root(db_path: &Path, tree: &str) { + seed_durable_root_for(db_path, "main", tree).await; +} + +async fn seed_durable_root_for(db_path: &Path, worktree: &str, tree: &str) { let db = RepositoryAgentTraceDb::open_without_migrations_at(db_path) .await .expect("writer"); db.execute( "INSERT INTO mutation_trace_worktrees (worktree_id, cursor_tree, revision, tainted, failure_kind, needs_rebaseline) - VALUES ('main', ?1, x'0000000000000000', 0, 'healthy', 0)", - (tree,), + VALUES (?1, ?2, x'0000000000000000', 0, 'healthy', 0)", + (worktree, tree), ) .await .expect("seed durable root"); @@ -814,3 +819,130 @@ async fn ref_reconciliation_verified_open_coexists_with_a_live_multiprocess_wal_ assert_eq!(rows, vec!["tree-after-open".to_string()]); assert!(with_suffix(&path, "-tshm").exists()); } + +async fn pin_distinct(root: &Path, content: &str) -> String { + std::fs::write(root.join("distinct.txt"), content).expect("write distinct file"); + pin_orphan(root).await +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_retains_every_repository_wide_durable_root_and_never_repairs_other_worktrees( +) { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + let local = pin_distinct(&fixture.root, "local\n").await; + let other = pin_distinct(&fixture.root, "other\n").await; + let orphan_one = pin_distinct(&fixture.root, "orphan-one\n").await; + let orphan_two = pin_distinct(&fixture.root, "orphan-two\n").await; + let unpinned_elsewhere = "e".repeat(40); + seed_durable_root(&fixture.db_path(), &local).await; + seed_durable_root_for(&fixture.db_path(), "worktrees/other", &other).await; + seed_durable_root_for(&fixture.db_path(), "worktrees/ghost", &unpinned_elsewhere).await; + + let (outcome, _) = fixture.run().await; + + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!( + report, + ReconciliationReport { + local_required: 1, + retained: 2, + deleted: 2, + } + ); + let mut remaining: Vec = fixture + .pins() + .await + .into_iter() + .map(|pin| pin.tree.0) + .collect(); + remaining.sort(); + let mut expected = vec![local, other]; + expected.sort(); + assert_eq!(remaining, expected); + assert!(!remaining.contains(&orphan_one) && !remaining.contains(&orphan_two)); + assert!(!remaining.contains(&unpinned_elsewhere)); + + let (second, _) = fixture.run().await; + let ExplicitOutcome::Completed(second) = second else { + panic!("expected idempotent completed pass"); + }; + assert_eq!((second.retained, second.deleted), (2, 0)); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_malformed_namespace_ref_deletes_nothing_and_records_failure() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + git( + &fixture.root, + &[ + "symbolic-ref", + "refs/sce/mutation-cursor/main/not-a-tree", + "HEAD", + ], + ); + + let (outcome, _) = fixture.run().await; + + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + assert!(matches!(error, ReconcileError::MalformedPin { .. })); + let output = Command::new("git") + .args(["for-each-ref", "refs/sce/mutation-cursor/main/"]) + .current_dir(&fixture.root) + .output() + .expect("for-each-ref"); + assert_eq!( + String::from_utf8_lossy(&output.stdout).lines().count(), + 2, + "no ref may be deleted" + ); + let StateRead::Valid(state) = fixture.stored_state().await else { + panic!("expected recorded state"); + }; + assert_eq!(state.consecutive_failures, 1); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let path = state_path(&git_dir); + + let mut observed = Vec::new(); + let result = reconcile_explicit_with( + &fixture.root, + async || open_authoritative_db_at_state_root(&fixture.root, &fixture.state_root).await, + || NOW, + write_state_atomically, + |phase| { + observed.push((phase, advise_if_due(&fixture.root, || NOW), path.exists())); + }, + Duration::from_secs(10), + ) + .await; + + assert!(matches!(result, ExplicitOutcome::Completed(_))); + assert_eq!( + observed + .iter() + .map(|(phase, _, _)| *phase) + .collect::>(), + vec![ReconcilePhase::DbOpened, ReconcilePhase::PinsInventoried] + ); + assert!(observed + .iter() + .all(|(_, advice, present)| matches!(advice, AdvisoryOutcome::Busy) && !present)); + assert!(fixture.pins().await.is_empty()); + assert!(matches!( + advise_if_due(&fixture.root, || NOW), + AdvisoryOutcome::NoAction + )); +} diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index aacd3366d..83c9c3639 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -340,7 +340,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - D3/D4/AC9/AC14 corrected above. - Context synchronization: synced -- [ ] T05: `Add production-path and concurrency regression tests` (status:todo) +- [x] T05: `Add production-path and concurrency regression tests` (status:done) - Task ID: T05 - Scope: In — real Git repos, linked worktrees, and repository Agent Trace DBs. Every race, ordering and timing scenario uses deterministic coordination (barriers, channels, existing `on_lock_contention`/`on_worker_entered`-style seams, parked phase hooks, injected clock); **no timing-based sleeps as correctness proofs**. T02 and T03 add unit-level coverage for their own code; T05 owns the production-path matrix below and each scenario must fail when the guarded behavior is deliberately broken. - *Reconciliation core (explicit path):* (S1) `sce doctor --fix` reclaims an orphan pin left by an interrupted coordinate; (S2) `A → B → C → D` retains every historical tree; (S3) worktree A's pin is retained while worktree B needs the tree (**preservation**); (S4) missing required local pin fails closed with zero deletion; (S5) malformed/symbolic refs cause zero deletion; (S6) concurrent `coordinate()` and reconcile cannot race pin deletion — a coordinate attempt parked on the lock via barrier proceeds only after the pass releases it, and a pin it creates is never deleted by the in-flight pass; (S7) dropping the explicit-pass future while the deletion worker is parked at the worker-entered seam leaves the lock held until the worker is released, then free; (S8) idempotent reconciliation; (S9) plain `sce doctor` mutates neither refs nor state; `--fix` reports the true result for every outcome kind (completed, completed-with-persist-warning, failed kinds, busy); (S10) explicit pass against an external holder returns `Skipped(Busy)`, leaves state byte-identical, and does not count as a failure; (S11) explicit pass with DB failure leaves refs untouched, records a failure streak, never arms the external-taint marker; (S12) the explicit pass and the advisory each complete with one lock acquisition and no self-deadlock, and each returns busy (never deadlocks) against an externally held lock; (S13) with worktree B's required pin deliberately missing, reconciling A deletes only A's eligible pins, does not create or repair B's pin, does not fail, and never deletes any pin protecting B's tree. @@ -353,7 +353,13 @@ Record actual outcomes in the validation report; compilation alone does not vali - Dependencies: T03, T04 - Done when: all scenarios pass reliably, each fails when the guarded behavior is deliberately broken, no test depends on timing or sleeps, and every review finding in the traceability table maps to at least one passing scenario. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml` focused on the new tests, repeated runs for the race cases; zero-match filters are not accepted as passes; `rg "sleep"` over new test code returns no hits. - - Context synchronization: pending + - Completed: 2026-10-09 + - Files changed: `cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs`, `context/plans/mutation-cursor-ref-reconciliation-wiring.md` + - Result: Added three focused production-path tests beside the T02 fixture and parameterized `seed_durable_root` by worktree: (S2/S3/S8/S13) repository-wide durable-root retention with a local root, another worktree's root, an unpinned foreign root and two orphans — only orphans deleted, the foreign missing pin neither repaired nor treated as a failure, and a second pass deletes nothing; (S5/S11) a symbolic ref in the namespace yields `MalformedPin`, deletes nothing and increments the failure streak; (S15/S12) `advise_if_due` returns `Busy` at both the post-DB-open and post-inventory phases of a live explicit pass, writes no state, and returns `NoAction` after the pass. No comments, sleeps or production code were added. + - Coverage mapping (deviation, per the lean-test preference): remaining matrix scenarios are already proven by earlier tasks and not duplicated — S1/S9/S10/S22/S26/S27/S28–S32 by `ref_maintenance/tests.rs` and `ref_doctor.rs` tests; S19–S25/S34–S39 by `maintenance_state.rs`/`ref_advisory.rs` tests; S14/S18a/S18b-trigger/S33-trigger/S4 hook policy by `mutation_scope::advisory_trigger_tests`; S7 by the `git_snapshot` cancelled-worker test (same `run_ref_mutation_inner` path used by `delete_pins`). Not added: S6, S16, a real-lock "acquirable at invocation" assertion for S17, and the S18b follow-up real `coordinate()` taint-recovery check; each needs coordinator-level fixtures beyond the minimal set. + - Verify: `test … ref_reconciliation_` 24 passed, repeated 3 times; `clippy --all-targets --all-features -D warnings` clean; `rg sleep` over the diff returns no hits; no added comment lines. + - Context impact: none (tests only). + - Context synchronization: synced - [ ] T06: `Remove dead-code exemptions and synchronize context` (status:todo) - Task ID: T06 From 7316fc3c40047c85fb2c7d999805b13c9ea23a4a Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 03:07:42 +0200 Subject: [PATCH 09/16] tests: Complete ref reconciliation production-path coverage Add CLI, linked-worktree, cross-process, lock-contention, state-recovery, and cancellation tests for the ref reconciliation path, using narrow coordinator and snapshot seams to exercise real behavior without changing production semantics. Cover database verification and fail-closed behavior, durable state normalization and uncertain persistence, hook/advisory ordering, and doctor outcomes. Plan: mutation-cursor-ref-reconciliation-wiring.md (T05) T05 remains in progress for the explicitly recorded S9, S14, and S26 production-path gaps and guard-removal checks. Co-authored-by: SCE --- cli/src/app.rs | 3 + .../app/doctor_reconciliation_cli_tests.rs | 395 +++++++++ cli/src/services/hooks/mutation_scope.rs | 2 +- .../mutation_trace/runtime/coordinator.rs | 12 +- .../mutation_trace/runtime/git_snapshot.rs | 164 ++-- .../runtime/protected_worktree.rs | 2 +- .../mutation_trace/runtime/ref_advisory.rs | 142 ++++ .../runtime/ref_maintenance/tests.rs | 11 +- .../ref_maintenance/tests/coordination.rs | 772 ++++++++++++++++++ .../ref_maintenance/tests/cross_process.rs | 166 ++++ .../ref_maintenance/tests/fail_closed.rs | 99 +++ .../ref_maintenance/tests/state_recovery.rs | 413 ++++++++++ .../runtime/ref_maintenance/tests/support.rs | 263 ++++++ .../ref_maintenance/tests/worktrees.rs | 263 ++++++ ...tation-cursor-ref-reconciliation-wiring.md | 74 +- 15 files changed, 2713 insertions(+), 68 deletions(-) create mode 100644 cli/src/app/doctor_reconciliation_cli_tests.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance/tests/cross_process.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance/tests/fail_closed.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance/tests/state_recovery.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance/tests/support.rs create mode 100644 cli/src/services/mutation_trace/runtime/ref_maintenance/tests/worktrees.rs diff --git a/cli/src/app.rs b/cli/src/app.rs index 9f39312f0..b651cc964 100644 --- a/cli/src/app.rs +++ b/cli/src/app.rs @@ -386,6 +386,9 @@ where Ok(command) } +#[cfg(test)] +mod doctor_reconciliation_cli_tests; + #[cfg(test)] mod tests { use std::future::{poll_fn, Future}; diff --git a/cli/src/app/doctor_reconciliation_cli_tests.rs b/cli/src/app/doctor_reconciliation_cli_tests.rs new file mode 100644 index 000000000..bc08ac11b --- /dev/null +++ b/cli/src/app/doctor_reconciliation_cli_tests.rs @@ -0,0 +1,395 @@ +use std::path::{Path, PathBuf}; +use std::process::Command; + +use serde_json::Value; + +use super::run_with_dependency_check_and_streams; +use crate::services::agent_trace_db::repository::RepositoryAgentTraceDb; +use crate::services::default_paths::agent_trace_db_path_for_repository_at; +use crate::services::repository_identity::resolve::resolve_repository_identity; + +const ARGS_ENV: &str = "SCE_DOCTOR_RECONCILIATION_CLI_ARGS"; +const OUT_ENV: &str = "SCE_DOCTOR_RECONCILIATION_CLI_OUT"; +const CHILD_TEST: &str = "app::doctor_reconciliation_cli_tests::doctor_reconciliation_cli_child"; +const STATE_FILE: &str = "ref-maintenance.json"; +const LOCK_FILE: &str = "mutation-cursor.lock"; +const RECONCILIATION_DETAIL_PREFIX: &str = "Snapshot ref reconciliation"; + +struct CliRun { + stdout: String, + stderr: String, +} + +impl CliRun { + fn json(&self) -> Value { + serde_json::from_str(&self.stdout) + .unwrap_or_else(|error| panic!("doctor stdout is not JSON ({error}):\n{}", self.stdout)) + } +} + +struct Sandbox { + dir: tempfile::TempDir, + repo: PathBuf, + repository_id: String, +} + +fn git(cwd: &Path, args: &[&str]) -> String { + let output = Command::new("git") + .args(args) + .current_dir(cwd) + .output() + .expect("git command"); + assert!( + output.status.success(), + "git {args:?} failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8_lossy(&output.stdout).trim().to_string() +} + +impl Sandbox { + fn new(remote: &str) -> Self { + let dir = tempfile::tempdir().expect("sandbox"); + let repo = dir.path().join("repo"); + std::fs::create_dir_all(&repo).expect("repo dir"); + git(&repo, &["init", "-q"]); + git(&repo, &["config", "user.email", "t@example.invalid"]); + git(&repo, &["config", "user.name", "T"]); + git(&repo, &["config", "commit.gpgsign", "false"]); + git(&repo, &["remote", "add", "origin", remote]); + std::fs::write(repo.join("tracked.txt"), "tracked\n").expect("tracked file"); + git(&repo, &["add", "tracked.txt"]); + git(&repo, &["commit", "-q", "-m", "initial"]); + let repository_id = resolve_repository_identity(&repo, None, "origin") + .expect("identity") + .identity + .repository_id; + Self { + dir, + repo, + repository_id, + } + } + + fn state_root(&self) -> PathBuf { + self.dir.path().join("state") + } + + fn db_path(&self) -> PathBuf { + agent_trace_db_path_for_repository_at(&self.state_root(), &self.repository_id) + .expect("db path") + } + + fn sce_dir(&self) -> PathBuf { + self.repo.join(".git").join("sce") + } + + fn state_file(&self) -> PathBuf { + self.sce_dir().join(STATE_FILE) + } + + async fn create_db(&self, repository_id: &str) { + let db = RepositoryAgentTraceDb::new_at(self.db_path()) + .await + .expect("create db"); + db.verify_or_initialize_repository_metadata(repository_id) + .await + .expect("initialize metadata"); + } + + fn orphan_pin(&self) -> String { + let tree = git(&self.repo, &["rev-parse", "HEAD^{tree}"]); + git( + &self.repo, + &[ + "update-ref", + &format!("refs/sce/mutation-cursor/main/{tree}"), + &tree, + ], + ); + tree + } + + fn refs(&self) -> String { + git( + &self.repo, + &[ + "for-each-ref", + "--format=%(refname) %(objectname)", + "refs/sce/", + ], + ) + } + + fn state_bytes(&self) -> Option> { + std::fs::read(self.state_file()).ok() + } + + fn state_json(&self) -> Value { + serde_json::from_slice(&self.state_bytes().expect("maintenance state was written")) + .expect("state JSON") + } + + fn run(&self, args: &[&str]) -> CliRun { + let out = self.dir.path().join(format!("out-{}", args.join("-"))); + std::fs::create_dir_all(&out).expect("out dir"); + let output = Command::new(std::env::current_exe().expect("current exe")) + .args(["--exact", CHILD_TEST, "--nocapture"]) + .current_dir(&self.repo) + .env(ARGS_ENV, args.join(" ")) + .env(OUT_ENV, &out) + .env("HOME", self.dir.path()) + .env("XDG_CONFIG_HOME", self.dir.path().join("config")) + .env("XDG_STATE_HOME", self.state_root()) + .env("XDG_CACHE_HOME", self.dir.path().join("cache")) + .env("NO_COLOR", "1") + .env_remove("SCE_CONFIG_FILE") + .output() + .expect("child runs"); + assert!( + output.status.success(), + "child harness failed:\n{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + CliRun { + stdout: std::fs::read_to_string(out.join("stdout")).expect("stdout"), + stderr: std::fs::read_to_string(out.join("stderr")).expect("stderr"), + } + } +} + +fn non_reconciliation_fix_results(report: &Value, sandbox_root: &Path) -> Vec { + let root = sandbox_root.to_string_lossy().into_owned(); + report["fix_results"] + .as_array() + .expect("fix_results array") + .iter() + .filter(|entry| { + !entry["detail"] + .as_str() + .expect("detail") + .starts_with(RECONCILIATION_DETAIL_PREFIX) + }) + .map(|entry| { + let mut entry = entry.clone(); + let detail = entry["detail"] + .as_str() + .expect("detail") + .replace(&root, ""); + entry["detail"] = Value::String(detail); + entry + }) + .collect() +} + +fn reconciliation_fix_result(report: &Value) -> Value { + let matches: Vec = report["fix_results"] + .as_array() + .expect("fix_results array") + .iter() + .filter(|entry| { + entry["detail"] + .as_str() + .expect("detail") + .starts_with(RECONCILIATION_DETAIL_PREFIX) + }) + .cloned() + .collect(); + assert_eq!(matches.len(), 1, "reconciliation reports exactly once"); + matches.into_iter().next().expect("one entry") +} + +#[tokio::test(flavor = "multi_thread")] +async fn doctor_reconciliation_cli_child() { + let (Some(args), Some(out)) = ( + std::env::var_os(ARGS_ENV), + std::env::var_os(OUT_ENV).map(PathBuf::from), + ) else { + return; + }; + let mut command_line = vec!["sce".to_string()]; + command_line.extend( + args.to_string_lossy() + .split(' ') + .filter(|part| !part.is_empty()) + .map(str::to_string), + ); + let mut stdout = Vec::new(); + let mut stderr = Vec::new(); + let _ = Box::pin(run_with_dependency_check_and_streams( + command_line, + || Ok(()), + &mut stdout, + &mut stderr, + )) + .await; + std::fs::write(out.join("stdout"), stdout).expect("stdout"); + std::fs::write(out.join("stderr"), stderr).expect("stderr"); +} + +#[tokio::test(flavor = "multi_thread")] +async fn doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once() { + let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); + sandbox.create_db(&sandbox.repository_id).await; + let tree = sandbox.orphan_pin(); + let refs_before = sandbox.refs(); + assert!(refs_before.contains(&tree)); + + let inspection = sandbox.run(&["doctor", "--format", "json"]); + let report = inspection.json(); + assert!( + report["ref_reconciliation_fix"].is_null(), + "plain doctor never reconciles: {}", + inspection.stderr + ); + assert_eq!(sandbox.refs(), refs_before); + assert_eq!(sandbox.state_bytes(), None); + + let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + let report = fix.json(); + let reconciliation = &report["ref_reconciliation_fix"]; + assert_eq!(reconciliation["outcome"], "completed", "{}", fix.stdout); + assert_eq!(reconciliation["counts"]["deleted"], 1); + assert_eq!(reconciliation["counts"]["retained"], 0); + assert_eq!(reconciliation["counts"]["local_required"], 0); + assert!(reconciliation["failure"].is_null()); + assert!(reconciliation["state_warning"].is_null()); + assert_eq!(sandbox.refs(), ""); + + let entry = reconciliation_fix_result(&report); + assert_eq!(entry["outcome"], "fixed"); + assert_eq!(entry["category"], "mutation_scope_health"); + assert_eq!( + entry["detail"], + "Snapshot ref reconciliation completed: deleted 1, retained 0, locally required 0." + ); + + let state = sandbox.state_json(); + assert!(state["last_success"].is_number()); + assert_eq!(state["last_report"]["deleted"], 1); + assert!(state.get("consecutive_failures").is_none_or(|v| v == 0)); + + let again = sandbox.run(&["doctor", "--fix", "--format", "json"]).json(); + assert_eq!(again["ref_reconciliation_fix"]["counts"]["deleted"], 0); +} + +#[tokio::test(flavor = "multi_thread")] +async fn doctor_fix_reports_missing_database_without_creating_it_or_deleting_refs() { + let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); + sandbox.orphan_pin(); + let refs_before = sandbox.refs(); + let db_path = sandbox.db_path(); + + let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + let report = fix.json(); + let reconciliation = &report["ref_reconciliation_fix"]; + assert_eq!(reconciliation["outcome"], "failed", "{}", fix.stdout); + assert_eq!(reconciliation["failure"]["kind"], "agent_trace_db_missing"); + assert!(reconciliation["counts"].is_null()); + assert_eq!(sandbox.refs(), refs_before); + assert!(!db_path.exists()); + let bootstrap: Vec = non_reconciliation_fix_results(&report, sandbox.dir.path()) + .iter() + .map(|entry| entry["detail"].as_str().expect("detail").to_string()) + .filter(|detail| detail.starts_with("Agent trace DB parent directory bootstrapped")) + .collect(); + assert_eq!( + bootstrap.len(), + 1, + "the parent directory comes from the separate global-state repair, not from reconciliation" + ); + for suffix in ["-wal", "-shm", "-tshm"] { + let mut sidecar = db_path.clone().into_os_string(); + sidecar.push(suffix); + assert!(!PathBuf::from(sidecar).exists()); + } + + let entry = reconciliation_fix_result(&report); + assert_eq!(entry["outcome"], "failed"); + assert!(entry["detail"] + .as_str() + .expect("detail") + .starts_with("Snapshot ref reconciliation failed (agent_trace_db_missing):")); + + let state = sandbox.state_json(); + assert_eq!(state["consecutive_failures"], 1); + assert!(state.get("last_success").is_none()); + assert!(report["fix_results"].is_array()); +} + +#[tokio::test(flavor = "multi_thread")] +async fn doctor_fix_reports_repository_mismatch_without_touching_the_database_or_refs() { + let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); + let foreign = "some-other-repository"; + sandbox.create_db(foreign).await; + sandbox.orphan_pin(); + let refs_before = sandbox.refs(); + + let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + let report = fix.json(); + let reconciliation = &report["ref_reconciliation_fix"]; + assert_eq!(reconciliation["outcome"], "failed", "{}", fix.stdout); + assert_eq!( + reconciliation["failure"]["kind"], + "agent_trace_db_repository_mismatch" + ); + assert!(reconciliation["failure"]["message"] + .as_str() + .expect("message") + .contains(foreign)); + assert_eq!(sandbox.refs(), refs_before); + assert_eq!(sandbox.state_json()["consecutive_failures"], 1); + + let (_, metadata) = + RepositoryAgentTraceDb::open_verified_existing_at(sandbox.db_path(), foreign) + .await + .expect("the foreign repository identity is untouched"); + assert_eq!(metadata.repository_id, foreign); +} + +#[tokio::test(flavor = "multi_thread")] +async fn doctor_fix_reports_busy_when_the_worktree_lock_is_held_and_keeps_other_repairs() { + let reference = Sandbox::new("https://example.invalid/org/repo-a.git"); + reference.create_db(&reference.repository_id).await; + reference.orphan_pin(); + let reference_report = reference + .run(&["doctor", "--fix", "--format", "json"]) + .json(); + + let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); + sandbox.create_db(&sandbox.repository_id).await; + sandbox.orphan_pin(); + let refs_before = sandbox.refs(); + std::fs::create_dir_all(sandbox.sce_dir()).expect("sce dir"); + let held = std::fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(false) + .open(sandbox.sce_dir().join(LOCK_FILE)) + .expect("lock file"); + held.try_lock().expect("hold the worktree lock externally"); + + let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + let report = fix.json(); + drop(held); + + let reconciliation = &report["ref_reconciliation_fix"]; + assert_eq!(reconciliation["outcome"], "busy", "{}", fix.stdout); + assert!(reconciliation["counts"].is_null()); + assert!(reconciliation["failure"].is_null()); + assert_eq!(sandbox.refs(), refs_before); + assert_eq!(sandbox.state_bytes(), None); + + let entry = reconciliation_fix_result(&report); + assert_eq!(entry["outcome"], "skipped"); + assert!(entry["detail"] + .as_str() + .expect("detail") + .starts_with("Snapshot ref reconciliation skipped:")); + assert_eq!( + non_reconciliation_fix_results(&report, sandbox.dir.path()), + non_reconciliation_fix_results(&reference_report, reference.dir.path()), + "independent doctor repairs are preserved" + ); +} diff --git a/cli/src/services/hooks/mutation_scope.rs b/cli/src/services/hooks/mutation_scope.rs index cc9e56741..6897bc29f 100644 --- a/cli/src/services/hooks/mutation_scope.rs +++ b/cli/src/services/hooks/mutation_scope.rs @@ -307,7 +307,7 @@ where .await } -async fn drive_mutation_scope< +pub(crate) async fn drive_mutation_scope< L: crate::services::observability::traits::Logger, E: Write, C, diff --git a/cli/src/services/mutation_trace/runtime/coordinator.rs b/cli/src/services/mutation_trace/runtime/coordinator.rs index b6d5306e7..1fa06e61d 100644 --- a/cli/src/services/mutation_trace/runtime/coordinator.rs +++ b/cli/src/services/mutation_trace/runtime/coordinator.rs @@ -1,4 +1,5 @@ use std::path::Path; +use std::time::Duration; use anyhow::Result; use uuid::Uuid; @@ -14,7 +15,7 @@ use crate::services::mutation_trace::types::{ }; use super::git_snapshot::GitSnapshotService; -use super::protected_worktree::{ProtectedWorktree, ProtectedWorktreeError}; +use super::protected_worktree::{ProtectedWorktree, ProtectedWorktreeError, WORKTREE_LOCK_TIMEOUT}; use super::worktree_lock::WorktreeLockLease; pub use super::protected_worktree::ExternalTaintOperation; @@ -205,6 +206,7 @@ where repository_root, boundary, open_db, + WORKTREE_LOCK_TIMEOUT, || {}, |_attempt| {}, |_attempt| Ok(()), @@ -216,6 +218,7 @@ pub(super) async fn coordinate_inner( repository_root: &Path, boundary: &RuntimeBoundary, open_db: P, + lock_timeout: Duration, on_lock_contention: F, after_load: L, after_recovery: R, @@ -226,9 +229,10 @@ where L: FnMut(u32), R: FnMut(u32) -> Result<()>, { - let protected = ProtectedWorktree::acquire_inner(repository_root, on_lock_contention) - .await - .map_err(protected_worktree_failure)?; + let protected = + ProtectedWorktree::acquire_with_timeout(repository_root, lock_timeout, on_lock_contention) + .await + .map_err(protected_worktree_failure)?; let outcome = coordinate_protected( repository_root, diff --git a/cli/src/services/mutation_trace/runtime/git_snapshot.rs b/cli/src/services/mutation_trace/runtime/git_snapshot.rs index ab05c5112..98261ccb5 100644 --- a/cli/src/services/mutation_trace/runtime/git_snapshot.rs +++ b/cli/src/services/mutation_trace/runtime/git_snapshot.rs @@ -185,45 +185,14 @@ impl GitSnapshotService { .collect() } - /// Delete exactly `pins` in one atomic, no-dereference - /// `git update-ref --no-deref --stdin` transaction, each `delete` - /// conditioned on the tree SHA recorded in the [`PinnedRef`]. - /// - /// Two independent safety properties: - /// - /// - **Atomic** — `git update-ref --stdin` commits every command together - /// at end of input; if any command fails (including a failed old-value - /// check) the whole transaction aborts and no ref is changed. - /// - **No dereference** — `--no-deref` makes every `delete` operate on the - /// exact ref name given, never on a ref reached by resolving a symbolic - /// ref. Combined with a fail-closed re-check (below), a - /// direct-ref → symbolic-ref race between inventory and deletion can - /// never cause this call to touch the symref's target (for example a ref - /// owned by another worktree). - /// - /// Before issuing the transaction, each supplied ref is re-inventoried: it - /// must still exist, still be a direct ref to a tree, and still point at - /// the inventoried SHA. If any has changed — deleted, retargeted, or turned - /// into a symbolic ref — this returns `Err` and deletes nothing, preferring - /// failure over acting on unexpected namespace state. An empty slice is a - /// successful no-op. #[allow( dead_code, reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" )] pub async fn delete_pins(&self, lease: WorktreeLockLease, pins: &[PinnedRef]) -> Result<()> { - self.delete_pins_inner(lease, pins, || {}).await + self.delete_pins_inner(lease, pins, || {}, || {}).await } - /// Body of [`delete_pins`] with a deterministic test seam that fires - /// **after** the fail-closed preflight re-inventory and **before** the - /// `git update-ref --no-deref --stdin` transaction is spawned. Production - /// calls it with a no-op hook; the inline atomicity test uses the hook to - /// mutate a ref *after* it has passed preflight, so the transaction is - /// actually issued and the per-`delete` expected-old-value check — not the - /// preflight — is what aborts the batch. This is the only proof that the - /// Git transaction itself is atomic; the preflight proves a different - /// property (unexpected ref state before the transaction is even attempted). #[allow( dead_code, reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" @@ -233,6 +202,7 @@ impl GitSnapshotService { lease: WorktreeLockLease, pins: &[PinnedRef], after_preflight: impl FnOnce(), + on_worker_entered: impl FnOnce() + Send + 'static, ) -> Result<()> { if pins.is_empty() { return Ok(()); @@ -251,7 +221,7 @@ impl GitSnapshotService { stdin_payload.push('\n'); } - self.run_ref_mutation( + self.run_ref_mutation_inner( lease, vec![ "update-ref".to_string(), @@ -259,25 +229,13 @@ impl GitSnapshotService { "--stdin".to_string(), ], Some(stdin_payload.into_bytes()), + on_worker_entered, ) .await?; Ok(()) } - #[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" - )] - async fn run_ref_mutation( - &self, - lease: WorktreeLockLease, - args: Vec, - stdin: Option>, - ) -> Result { - self.run_ref_mutation_inner(lease, args, stdin, || {}).await - } - async fn run_ref_mutation_inner( &self, lease: WorktreeLockLease, @@ -1059,6 +1017,120 @@ mod tests { .is_empty()); } + #[tokio::test] + async fn cancelled_delete_keeps_worktree_lock_until_git_ref_deletion_completes() { + let dir = tempfile::tempdir().expect("temp dir"); + let repo_root = dir.path().join("repo"); + init_committed_repository(&repo_root); + std::fs::write(repo_root.join("untracked.txt"), "untracked\n").expect("write file"); + + let git_dir = resolve_git_dir(&repo_root).await.expect("git dir"); + let worktree_id = WorktreeId("main".to_string()); + let snapshot = GitSnapshotService::new(&repo_root) + .await + .expect("snapshot service"); + let tree = snapshot.capture_tree().await.expect("capture tree"); + let pin_ref = format!("refs/sce/mutation-cursor/main/{}", tree.0); + + let pin_lock = WorktreeLock::acquire_async(&git_dir, LOCK_RELEASE_TIMEOUT) + .await + .expect("pin lock"); + snapshot + .pin_tree(pin_lock.lease(), &worktree_id, &tree) + .await + .expect("pin orphan tree"); + drop(pin_lock); + let pins = snapshot.list_pins(&worktree_id).await.expect("list pins"); + assert_eq!( + pins, + vec![PinnedRef { + ref_name: pin_ref.clone(), + tree: tree.clone(), + }] + ); + + let lock_a = WorktreeLock::acquire_async(&git_dir, LOCK_RELEASE_TIMEOUT) + .await + .expect("worktree lock A"); + let lease = lock_a.lease(); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel::<()>(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + + let caller = tokio::spawn({ + let repo_root = repo_root.clone(); + let pins = pins.clone(); + async move { + let _lock_a = lock_a; + let snapshot = GitSnapshotService::new(&repo_root) + .await + .expect("caller snapshot service"); + snapshot + .delete_pins_inner( + lease, + &pins, + || {}, + move || { + entered_tx.send(()).expect("signal worker entered"); + release_rx + .recv_timeout(LOCK_RELEASE_TIMEOUT) + .expect("release worker"); + }, + ) + .await + } + }); + + entered_rx.await.expect("ref deletion worker entered"); + assert_eq!( + snapshot.list_pins(&worktree_id).await.expect("list pins"), + pins, + "the parked worker must not have deleted the pin yet" + ); + + caller.abort(); + assert!(caller + .await + .expect_err("caller must be cancelled") + .is_cancelled()); + + assert!(matches!( + WorktreeLock::acquire_async(&git_dir, LOCK_CONTENDED_TIMEOUT).await, + Err(WorktreeLockError::TimedOut { .. }) + )); + assert_eq!( + snapshot + .list_pins(&worktree_id) + .await + .expect("list pins while the worker is still parked"), + pins + ); + + release_tx.send(()).expect("release worker"); + + let lock_b = WorktreeLock::acquire_async(&git_dir, LOCK_RELEASE_TIMEOUT) + .await + .expect("worktree lock B after the ref deletion completes"); + assert!(snapshot + .list_pins(&worktree_id) + .await + .expect("list pins after the released worker finished") + .is_empty()); + assert!(!git_dir.join(format!("{pin_ref}.lock")).exists()); + assert!(!git_dir.join("packed-refs.lock").exists()); + + snapshot + .pin_tree(lock_b.lease(), &worktree_id, &tree) + .await + .expect("a legitimate pin proceeds after the cancelled deletion"); + assert_eq!( + snapshot + .list_pins(&worktree_id) + .await + .expect("list pins after re-pin"), + pins + ); + } + #[tokio::test] async fn async_identity_resolution_distinguishes_main_and_linked_worktrees() { let dir = tempfile::tempdir().expect("temp dir"); diff --git a/cli/src/services/mutation_trace/runtime/protected_worktree.rs b/cli/src/services/mutation_trace/runtime/protected_worktree.rs index 89edc6486..eefd57665 100644 --- a/cli/src/services/mutation_trace/runtime/protected_worktree.rs +++ b/cli/src/services/mutation_trace/runtime/protected_worktree.rs @@ -106,7 +106,7 @@ impl ProtectedWorktree { Self::acquire_with_timeout(repository_root, WORKTREE_LOCK_TIMEOUT, on_lock_contention).await } - async fn acquire_with_timeout( + pub(super) async fn acquire_with_timeout( repository_root: &Path, lock_timeout: Duration, on_lock_contention: F, diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs index 85e6ae5bc..b915d9978 100644 --- a/cli/src/services/mutation_trace/runtime/ref_advisory.rs +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -640,6 +640,148 @@ mod tests { assert!(evaluate_recommendation(&visible, NOW + 20).recommended); } + #[test] + fn maintenance_advisory_treats_timestamps_at_or_just_ahead_of_now_as_valid_age_zero() { + for ahead in [0, 60_000] { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let state = MaintenanceState { + last_success: Some(NOW + ahead), + ..MaintenanceState::default() + }; + seed(dir.path(), &state); + let path = state_path(&dir.path().join(".git")); + let before = std::fs::read(&path).expect("state bytes"); + + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + assert!(!evaluate_recommendation(&state, NOW).recommended); + } + } + + #[test] + fn maintenance_advisory_clock_rollback_beyond_tolerance_normalizes_while_within_it_does_not() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + advise(dir.path(), NOW); + let path = state_path(&dir.path().join(".git")); + let before = std::fs::read(&path).expect("state bytes"); + + assert!(matches!( + advise(dir.path(), NOW - FUTURE_SKEW_TOLERANCE_MS), + AdvisoryOutcome::NoAction + )); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + + let rolled_back = NOW - FUTURE_SKEW_TOLERANCE_MS - 1; + assert!(matches!( + advise(dir.path(), rolled_back), + AdvisoryOutcome::Advised + )); + let normalized = stored(dir.path()); + assert_within_skew_tolerance(&normalized, rolled_back); + assert_eq!(normalized.last_advised, Some(rolled_back)); + } + + #[test] + fn maintenance_advisory_recovers_wrong_version_and_oversized_state_by_anchoring() { + let oversized = vec![b' '; 5000]; + let wrong_version = br#"{"version":9999}"#.to_vec(); + for bytes in [oversized, wrong_version] { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let path = state_path(&dir.path().join(".git")); + std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); + std::fs::write(&path, &bytes).expect("unusable state"); + + assert!(matches!( + super::read_state(&path).expect("read"), + StateRead::Unusable + )); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored)); + assert_eq!(stored(dir.path()).anchor, Some(NOW)); + } + } + + fn invalid_future_state() -> MaintenanceState { + MaintenanceState { + anchor: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + last_success: Some(NOW + FUTURE_SKEW_TOLERANCE_MS + 1), + ..MaintenanceState::default() + } + } + + fn assert_within_skew_tolerance(state: &MaintenanceState, now: i64) { + for timestamp in [ + state.anchor, + state.last_success, + state.last_attempt, + state.last_advised, + ] + .into_iter() + .flatten() + { + assert!(timestamp <= now + FUTURE_SKEW_TOLERANCE_MS); + } + } + + #[test] + fn maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + seed(dir.path(), &invalid_future_state()); + let path = state_path(&dir.path().join(".git")); + let before = std::fs::read(&path).expect("state bytes"); + + let outcome = advise_with_fault(dir.path(), NOW, PersistPhase::Rename); + + assert!(matches!( + outcome, + AdvisoryOutcome::StateWriteFailed { + warning: PersistFailure::NotApplied { + phase: PersistPhase::Rename, + .. + } + } + )); + assert_eq!(outcome.recommendation(), None); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + assert!(evaluate_recommendation(&stored(dir.path()), NOW).invalid_timestamps); + + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Advised)); + let normalized = stored(dir.path()); + assert_within_skew_tolerance(&normalized, NOW); + assert_eq!(normalized.last_advised, Some(NOW)); + } + + #[test] + fn maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + seed(dir.path(), &invalid_future_state()); + + let outcome = advise_with_fault(dir.path(), NOW, PersistPhase::SyncParentDirectory); + + assert!(matches!( + &outcome, + AdvisoryOutcome::AdvisedDurabilityUncertain { + warning: PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + .. + } + } + )); + assert!(outcome.persistence_warning().is_some()); + assert!(outcome.report().diagnostic.is_some()); + let visible = stored(dir.path()); + assert_within_skew_tolerance(&visible, NOW); + + let path = state_path(&dir.path().join(".git")); + let bytes = std::fs::read(&path).expect("state bytes"); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); + assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); + } + #[test] fn maintenance_advisory_anchor_write_failure_reports_state_write_failed_without_advice() { let dir = tempfile::tempdir().expect("temp dir"); diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs index dc21eddd7..db544f4d8 100644 --- a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs @@ -28,8 +28,15 @@ use crate::services::repository_identity::resolve::resolve_repository_identity; const NOW: i64 = 1_000_000_000_000; const OTHER_REPOSITORY_ID: &str = "some-other-repository"; +mod coordination; +mod cross_process; +mod fail_closed; +mod state_recovery; +mod support; +mod worktrees; + struct Fixture { - _dir: tempfile::TempDir, + dir: tempfile::TempDir, root: PathBuf, state_root: PathBuf, repository_id: String, @@ -46,7 +53,7 @@ impl Fixture { .repository_id; let state_root = dir.path().join("state"); Self { - _dir: dir, + dir, root, state_root, repository_id, diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs new file mode 100644 index 000000000..5eb35d3b8 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs @@ -0,0 +1,772 @@ +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use tokio::sync::mpsc::unbounded_channel; + +use super::super::super::coordinator::coordinate_inner; +use super::super::super::external_taint::ExternalTaintMarker; +use super::super::super::git_snapshot::resolve_git_dir; +use super::super::super::maintenance_state::{ + state_path, write_state_atomically, MaintenanceState, PersistPhase, + RECONCILIATION_ADVISORY_AFTER_MS, +}; +use super::super::super::protected_worktree::WORKTREE_LOCK_TIMEOUT; +use super::super::super::ref_advisory::{advise_if_due, advise_if_due_with, AdvisoryOutcome}; +use super::super::super::ref_reconciliation::{ReconcilePhase, ReconciliationReport}; +use super::super::super::worktree_lock::acquire_inner; +use super::super::ExplicitOutcome; +use super::support::{ + await_signal, close_payload, durable_snapshot, hook_ok, main_worktree, no_advice, pinned_trees, + run_hook, sce_refs, spawn_parked_pass, start_payload, writer, WATCHDOG, +}; +use super::{failing_write, pin_orphan, Fixture, NOW}; +use crate::services::hooks::mutation_scope::MutationScopePayload; +use crate::services::mutation_trace::runtime::{advise_after_completed_boundary, RuntimeBoundary}; +use crate::services::mutation_trace::types::ScopeStatus; + +const REMOTE: &str = "https://example.invalid/org/repo-a.git"; +const SCOPE: &str = "scope-1"; + +async fn git_dir_of(fixture: &Fixture) -> PathBuf { + resolve_git_dir(&fixture.root).await.expect("git dir") +} + +async fn started_fixture() -> Fixture { + let fixture = Fixture::new(REMOTE); + fixture.create_db().await; + hook_ok(&fixture, start_payload(SCOPE, "start-1")).await; + fixture +} + +fn state_bytes(git_dir: &Path) -> Option> { + std::fs::read(state_path(git_dir)).ok() +} + +async fn assert_coordinate_blocked_until_pass_releases(phase: ReconcilePhase) { + let fixture = started_fixture().await; + let git_dir = git_dir_of(&fixture).await; + let baseline = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!(baseline.roots.len(), 1); + let cursor_tree = baseline.worktree.cursor_tree.0.clone(); + + super::support::write_file(&fixture.root, "orphan.txt", "orphan\n"); + let orphan = pin_orphan(&fixture.root).await; + super::support::write_file(&fixture.root, "next.txt", "next\n"); + + let mut pass = spawn_parked_pass(&fixture, phase); + await_signal(&mut pass.parked).await; + let refs_while_parked = sce_refs(&fixture.root); + assert_eq!( + pinned_trees(&fixture.root, "main"), + [cursor_tree.clone(), orphan.clone()].into_iter().collect() + ); + + let (contended_tx, mut contended_rx) = unbounded_channel::<()>(); + let db_opened = Arc::new(AtomicBool::new(false)); + let refs_at_load = Arc::new(Mutex::new(Vec::>::new())); + let db_path = fixture.db_path(); + let root = fixture.root.clone(); + + let coordinate = coordinate_inner( + &root, + &RuntimeBoundary::Flush, + async || { + db_opened.store(true, Ordering::SeqCst); + writer(&db_path).await + }, + WATCHDOG, + move || { + let _ = contended_tx.send(()); + }, + |_attempt| { + refs_at_load + .lock() + .expect("refs at load") + .push(sce_refs(&root)); + }, + |_attempt| Ok(()), + ); + + let marker = ExternalTaintMarker::new(&git_dir); + let driver = async { + await_signal(&mut contended_rx).await; + assert!( + !db_opened.load(Ordering::SeqCst), + "the coordinator must not reach its protected section while the pass holds the lock" + ); + assert!( + !marker.exists().expect("marker"), + "the coordinator must not arm its marker before it owns the lock" + ); + assert_eq!(sce_refs(&fixture.root), refs_while_parked); + assert_eq!( + durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await, + baseline + ); + pass.finish().await + }; + + let (coordinated, outcome) = tokio::join!(coordinate, driver); + + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!( + report, + ReconciliationReport { + local_required: 1, + retained: 1, + deleted: 1, + } + ); + let coordinated = coordinated.expect("the coordinator completes once the pass releases"); + let new_tree = coordinated.observed_tree.0.clone(); + assert_ne!(new_tree, cursor_tree); + assert_ne!(new_tree, orphan); + + assert_eq!( + pinned_trees(&fixture.root, "main"), + [cursor_tree, new_tree.clone()].into_iter().collect(), + "the pin created after the pass released the lock must survive; only the orphan is gone" + ); + assert!(!refs_while_parked + .iter() + .any(|line| line.contains(&new_tree))); + let seen_at_load = refs_at_load.lock().expect("refs at load").clone(); + assert_new_pin_preceded_the_durable_root(&fixture, &new_tree, &seen_at_load).await; + + drop( + acquire_inner(&git_dir, Duration::ZERO, || {}) + .expect("neither operation leaves the lock held"), + ); +} + +async fn assert_new_pin_preceded_the_durable_root( + fixture: &Fixture, + new_tree: &str, + refs_at_load: &[Vec], +) { + assert_eq!(refs_at_load.len(), 1); + assert!( + refs_at_load[0].iter().any(|line| line.contains(new_tree)), + "the new pin must already exist before the durable root commit" + ); + let after = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!(after.worktree.cursor_tree.0, new_tree); + assert!(after.roots.iter().any(|root| root.0 == new_tree)); + for root in &after.roots { + assert!( + pinned_trees(&fixture.root, "main").contains(&root.0), + "every durable root stays reachable" + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_real_coordinate_waits_for_a_pass_parked_after_db_open() { + assert_coordinate_blocked_until_pass_releases(ReconcilePhase::DbOpened).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_real_coordinate_waits_for_a_pass_parked_after_pin_inventory() { + assert_coordinate_blocked_until_pass_releases(ReconcilePhase::PinsInventoried).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries() +{ + let fixture = started_fixture().await; + let git_dir = git_dir_of(&fixture).await; + let marker = ExternalTaintMarker::new(&git_dir); + super::support::write_file(&fixture.root, "work.txt", "work\n"); + let before = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + let refs_before = sce_refs(&fixture.root); + assert_eq!(before.scope, Some(ScopeStatus::Active)); + + let mut pass = spawn_parked_pass(&fixture, ReconcilePhase::DbOpened); + await_signal(&mut pass.parked).await; + + let advisories = Arc::new(AtomicU32::new(0)); + let contended = Arc::new(AtomicU32::new(0)); + let failed = run_hook( + &fixture.root, + &fixture.db_path(), + close_payload(SCOPE, "close-1"), + Duration::from_millis(300), + { + let contended = Arc::clone(&contended); + move || { + contended.fetch_add(1, Ordering::SeqCst); + } + }, + || {}, + { + let advisories = Arc::clone(&advisories); + move |root| { + advisories.fetch_add(1, Ordering::SeqCst); + no_advice(root) + } + }, + ) + .await + .expect_err("the lock cannot be acquired while the pass holds it"); + + let message = format!("{failed:#}"); + assert!( + message.contains("before durable completion") + && message.contains("Timed out after") + && message.contains("waiting for worktree lock"), + "unexpected error: {message}" + ); + assert_eq!(contended.load(Ordering::SeqCst), 1); + assert_eq!(advisories.load(Ordering::SeqCst), 0); + assert_eq!( + durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await, + before + ); + assert_eq!(sce_refs(&fixture.root), refs_before); + assert!(!marker.exists().expect("marker")); + assert_eq!(state_bytes(&git_dir), None); + + let outcome = pass.finish().await; + assert!(matches!(outcome, ExplicitOutcome::Completed(_))); + let after_pass_state = state_bytes(&git_dir); + + let output = run_hook( + &fixture.root, + &fixture.db_path(), + close_payload(SCOPE, "close-1"), + WORKTREE_LOCK_TIMEOUT, + || {}, + || {}, + { + let advisories = Arc::clone(&advisories); + move |root| { + advisories.fetch_add(1, Ordering::SeqCst); + no_advice(root) + } + }, + ) + .await + .expect("the same boundary succeeds once the maintenance pass released the lock"); + assert_eq!(output, ""); + assert_eq!(advisories.load(Ordering::SeqCst), 1); + assert_eq!(state_bytes(&git_dir), after_pass_state); + + let after = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!(after.scope, Some(ScopeStatus::Closed)); + assert_eq!(after.worktree.revision, before.worktree.revision + 1); + assert_ne!(after.worktree.cursor_tree, before.worktree.cursor_tree); + assert!(pinned_trees(&fixture.root, "main").contains(&after.worktree.cursor_tree.0)); + assert!(after.roots.contains(&after.worktree.cursor_tree)); + assert!(!marker.exists().expect("marker")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_flush_lock_timeout_under_maintenance_commits_nothing_and_retries() { + let fixture = started_fixture().await; + let git_dir = git_dir_of(&fixture).await; + let marker = ExternalTaintMarker::new(&git_dir); + super::support::write_file(&fixture.root, "work.txt", "work\n"); + let before = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + let refs_before = sce_refs(&fixture.root); + + let mut pass = spawn_parked_pass(&fixture, ReconcilePhase::DbOpened); + await_signal(&mut pass.parked).await; + + let advisories = Arc::new(AtomicU32::new(0)); + let counting = |advisories: &Arc| { + let advisories = Arc::clone(advisories); + move |root: &Path| { + advisories.fetch_add(1, Ordering::SeqCst); + no_advice(root) + } + }; + let failed = run_hook( + &fixture.root, + &fixture.db_path(), + MutationScopePayload::Flush, + Duration::from_millis(300), + || {}, + || {}, + counting(&advisories), + ) + .await + .expect_err("the lock cannot be acquired while the pass holds it"); + assert!(format!("{failed:#}").contains("Timed out after")); + assert_eq!(advisories.load(Ordering::SeqCst), 0); + assert_eq!( + durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await, + before + ); + assert_eq!(sce_refs(&fixture.root), refs_before); + assert!(!marker.exists().expect("marker")); + assert_eq!(state_bytes(&git_dir), None); + + assert!(matches!(pass.finish().await, ExplicitOutcome::Completed(_))); + + let output = run_hook( + &fixture.root, + &fixture.db_path(), + MutationScopePayload::Flush, + WORKTREE_LOCK_TIMEOUT, + || {}, + || {}, + counting(&advisories), + ) + .await + .expect("the same flush succeeds once the maintenance pass released the lock"); + assert_eq!(output, ""); + assert_eq!(advisories.load(Ordering::SeqCst), 1); + let after = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!(after.worktree.revision, before.worktree.revision + 1); + assert_ne!(after.worktree.cursor_tree, before.worktree.cursor_tree); + assert!(pinned_trees(&fixture.root, "main").contains(&after.worktree.cursor_tree.0)); + assert!(!marker.exists().expect("marker")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock() { + let fixture = started_fixture().await; + let git_dir = git_dir_of(&fixture).await; + let marker = ExternalTaintMarker::new(&git_dir); + super::support::write_file(&fixture.root, "work.txt", "work\n"); + let before = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + + let at_advisory = Arc::new(Mutex::new(None)); + let refs_around = Arc::new(Mutex::new(None)); + let db_path = fixture.db_path(); + let advise = { + let at_advisory = Arc::clone(&at_advisory); + let refs_around = Arc::clone(&refs_around); + let marker = marker.clone(); + let git_dir = git_dir.clone(); + move |root: &Path| { + drop( + acquire_inner(&git_dir, Duration::ZERO, || {}) + .expect("the coordinator lock is released before the advisory runs"), + ); + *at_advisory.lock().expect("snapshot slot") = Some(tokio::task::block_in_place(|| { + tokio::runtime::Handle::current().block_on(durable_snapshot( + &db_path, + &main_worktree(), + SCOPE, + )) + })); + let marker_before = marker.exists().expect("marker"); + let refs_before = sce_refs(root); + let report = advise_after_completed_boundary(root); + *refs_around.lock().expect("refs slot") = Some((refs_before, sce_refs(root))); + assert_eq!( + marker_before, + marker.exists().expect("marker"), + "the advisory never creates, clears or modifies the marker" + ); + assert!(!marker_before); + report + } + }; + + let output = run_hook( + &fixture.root, + &fixture.db_path(), + close_payload(SCOPE, "close-1"), + WORKTREE_LOCK_TIMEOUT, + || {}, + || {}, + advise, + ) + .await + .expect("hook succeeds"); + + assert_eq!(output, ""); + let committed = at_advisory + .lock() + .expect("snapshot slot") + .clone() + .expect("advisory callback ran"); + assert_eq!(committed.scope, Some(ScopeStatus::Closed)); + assert_eq!(committed.worktree.revision, before.worktree.revision + 1); + assert_ne!(committed.worktree.cursor_tree, before.worktree.cursor_tree); + assert_eq!( + durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await, + committed, + "the advisory must not change the committed result" + ); + let (refs_before, refs_after) = refs_around + .lock() + .expect("refs slot") + .clone() + .expect("refs recorded"); + assert_eq!(refs_before, refs_after, "the advisory never touches refs"); + assert!(state_bytes(&git_dir).is_some()); + assert!(!marker.exists().expect("marker")); + drop( + acquire_inner(&git_dir, Duration::ZERO, || {}) + .expect("no lock is held after the advisory returns"), + ); +} + +fn marker_fingerprint(path: &Path) -> (bool, Vec) { + let metadata = std::fs::symlink_metadata(path).expect("marker path exists"); + let mut children: Vec = if metadata.is_dir() { + std::fs::read_dir(path) + .expect("read marker dir") + .map(|entry| { + entry + .expect("entry") + .file_name() + .to_string_lossy() + .into_owned() + }) + .collect() + } else { + Vec::new() + }; + children.sort(); + (metadata.is_dir(), children) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_marker_clear_failure_after_commit_keeps_taint_for_the_next_coordinate() +{ + let fixture = started_fixture().await; + let git_dir = git_dir_of(&fixture).await; + let marker = ExternalTaintMarker::new(&git_dir); + let marker_path = git_dir.join("sce").join("mutation-cursor-tainted"); + let saved_path = git_dir.join("sce").join("mutation-cursor-tainted.saved"); + assert!(!marker.exists().expect("marker")); + + super::support::write_file(&fixture.root, "one.txt", "one\n"); + hook_ok( + &fixture, + super::support::start_payload("scope-2", "start-2"), + ) + .await; + let control = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!(control.scope, Some(ScopeStatus::Active)); + assert!(!marker.exists().expect("marker")); + + super::support::write_file(&fixture.root, "two.txt", "two\n"); + let before_fault = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + let armed_len = Arc::new(Mutex::new(None)); + let fingerprints = Arc::new(Mutex::new(None)); + let output = run_hook( + &fixture.root, + &fixture.db_path(), + close_payload("scope-2", "close-2"), + WORKTREE_LOCK_TIMEOUT, + || {}, + { + let marker_path = marker_path.clone(); + let saved_path = saved_path.clone(); + let armed_len = Arc::clone(&armed_len); + move || { + *armed_len.lock().expect("len slot") = + Some(std::fs::metadata(&marker_path).expect("armed marker").len()); + std::fs::rename(&marker_path, &saved_path).expect("set armed marker aside"); + std::fs::create_dir(&marker_path).expect("block unlink with a directory"); + } + }, + { + let marker_path = marker_path.clone(); + let fingerprints = Arc::clone(&fingerprints); + move |root| { + let before = marker_fingerprint(&marker_path); + let refs_before = sce_refs(root); + let report = advise_after_completed_boundary(root); + let after = marker_fingerprint(&marker_path); + assert_eq!(refs_before, sce_refs(root)); + *fingerprints.lock().expect("fingerprint slot") = Some((before, after)); + report + } + }, + ) + .await + .expect("MarkerClearAfterCommit is a durable success for the hook"); + assert_eq!(output, ""); + + let (fingerprint_before, fingerprint_after) = fingerprints + .lock() + .expect("fingerprint slot") + .clone() + .expect("advisory ran after the failed marker clear"); + assert_eq!(fingerprint_before, fingerprint_after); + assert!(fingerprint_before.0, "the injected fault is still in place"); + + let after_fault = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!( + after_fault.worktree.revision, + before_fault.worktree.revision + 1 + ); + assert_eq!(after_fault.scope, Some(ScopeStatus::Active)); + + std::fs::remove_dir(&marker_path).expect("remove injected fault"); + std::fs::rename(&saved_path, &marker_path).expect("restore the armed marker"); + assert!(marker.exists().expect("marker")); + assert_eq!( + std::fs::metadata(&marker_path) + .expect("restored marker") + .len(), + armed_len + .lock() + .expect("len slot") + .expect("length recorded") + ); + + super::support::write_file(&fixture.root, "three.txt", "three\n"); + hook_ok(&fixture, close_payload("scope-2", "close-3")).await; + + let recovered = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!( + recovered.scope, + Some(ScopeStatus::Abandoned), + "inherited taint conservatively abandons the live scope" + ); + assert!(recovered.worktree.revision >= after_fault.worktree.revision + 2); + assert!(!recovered.worktree.tainted); + assert!(!recovered.worktree.needs_rebaseline); + assert!(pinned_trees(&fixture.root, "main").contains(&recovered.worktree.cursor_tree.0)); + assert!(!marker.exists().expect("marker")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_advisory_after_a_real_commit_is_busy_when_a_pass_wins_the_handoff() { + let fixture = started_fixture().await; + let git_dir = git_dir_of(&fixture).await; + super::support::write_file(&fixture.root, "orphan.txt", "orphan\n"); + let orphan = pin_orphan(&fixture.root).await; + super::support::write_file(&fixture.root, "work.txt", "work\n"); + + let (handoff_tx, mut handoff_rx) = unbounded_channel::<()>(); + let (holding_tx, holding_rx) = std::sync::mpsc::channel::<()>(); + let (advisory_done_tx, mut advisory_done_rx) = unbounded_channel::<()>(); + + let orchestrator = { + let root = fixture.root.clone(); + let state_root = fixture.state_root.clone(); + tokio::spawn(async move { + await_signal(&mut handoff_rx).await; + let mut pass = spawn_parked_pass_for(&root, &state_root); + await_signal(&mut pass.parked).await; + holding_tx.send(()).expect("resume the advisory"); + await_signal(&mut advisory_done_rx).await; + pass.finish().await + }) + }; + + let observed = Arc::new(Mutex::new(None)); + let advise = { + let observed = Arc::clone(&observed); + let git_dir = git_dir.clone(); + move |root: &Path| { + handoff_tx.send(()).expect("announce the handoff point"); + holding_rx + .recv_timeout(WATCHDOG) + .expect("the pass takes the lock"); + let state_before = state_bytes(&git_dir); + let refs_before = sce_refs(root); + let report = advise_after_completed_boundary(root); + *observed.lock().expect("observed slot") = Some(( + state_before, + state_bytes(&git_dir), + refs_before, + sce_refs(root), + report.outcome, + )); + advisory_done_tx.send(()).expect("advisory finished"); + report + } + }; + + let output = run_hook( + &fixture.root, + &fixture.db_path(), + close_payload(SCOPE, "close-1"), + WORKTREE_LOCK_TIMEOUT, + || {}, + || {}, + advise, + ) + .await + .expect("the committed boundary still reports success"); + assert_eq!(output, ""); + + let (state_before, state_after, refs_before, refs_after, outcome) = observed + .lock() + .expect("observed slot") + .clone() + .expect("advisory ran"); + assert_eq!(outcome, "busy"); + assert_eq!(state_before, state_after); + assert_eq!(refs_before, refs_after); + + let pass_outcome = orchestrator.await.expect("orchestrator joins"); + let ExplicitOutcome::Completed(report) = pass_outcome else { + panic!("expected completed pass, got {pass_outcome:?}"); + }; + assert_eq!(report.deleted, 1); + let committed = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; + assert_eq!(committed.scope, Some(ScopeStatus::Closed)); + let pins = pinned_trees(&fixture.root, "main"); + assert!(!pins.contains(&orphan)); + assert!(pins.contains(&committed.worktree.cursor_tree.0)); +} + +fn spawn_parked_pass_for(root: &Path, state_root: &Path) -> super::support::ParkedPass { + super::support::spawn_parked_pass_at(root, state_root, ReconcilePhase::DbOpened) +} + +async fn advisory_from_independent_task(root: &Path, now: i64) -> AdvisoryOutcome { + let root = root.to_path_buf(); + tokio::task::spawn_blocking(move || advise_if_due(&root, || now)) + .await + .expect("advisory task joins") +} + +async fn assert_independent_advisory_is_busy_while_parked(phase: ReconcilePhase) { + let fixture = Fixture::new(REMOTE); + fixture.create_db().await; + let orphan = pin_orphan(&fixture.root).await; + let git_dir = git_dir_of(&fixture).await; + + let mut pass = spawn_parked_pass(&fixture, phase); + await_signal(&mut pass.parked).await; + let refs_while_parked = sce_refs(&fixture.root); + let state_while_parked = state_bytes(&git_dir); + + let advisory = + tokio::time::timeout(WATCHDOG, advisory_from_independent_task(&fixture.root, NOW)) + .await + .expect("the advisory must not wait for the pass"); + assert!(matches!(advisory, AdvisoryOutcome::Busy)); + assert_eq!(state_bytes(&git_dir), state_while_parked); + assert_eq!(sce_refs(&fixture.root), refs_while_parked); + assert!(pinned_trees(&fixture.root, "main").contains(&orphan)); + + let outcome = pass.finish().await; + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!(report.deleted, 1); + assert!(pinned_trees(&fixture.root, "main").is_empty()); + + let recorded = state_bytes(&git_dir).expect("pass recorded state"); + assert!(matches!( + advisory_from_independent_task(&fixture.root, NOW).await, + AdvisoryOutcome::NoAction + )); + assert_eq!(state_bytes(&git_dir), Some(recorded)); + assert!(matches!( + advisory_from_independent_task(&fixture.root, NOW + RECONCILIATION_ADVISORY_AFTER_MS + 1) + .await, + AdvisoryOutcome::Advised + )); + let stored: MaintenanceState = + match super::super::super::maintenance_state::read_state(&state_path(&git_dir)) + .expect("state") + { + super::super::super::maintenance_state::StateRead::Valid(state) => state, + other => panic!("expected valid state, got {other:?}"), + }; + assert_eq!(stored.last_success, Some(NOW)); + assert_eq!( + stored.last_advised, + Some(NOW + RECONCILIATION_ADVISORY_AFTER_MS + 1) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_independent_advisory_task_is_busy_while_a_pass_is_parked_after_db_open() +{ + assert_independent_advisory_is_busy_while_parked(ReconcilePhase::DbOpened).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_independent_advisory_task_is_busy_while_a_pass_is_parked_after_pin_inventory( +) { + assert_independent_advisory_is_busy_while_parked(ReconcilePhase::PinsInventoried).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_advisory_failures_after_a_real_commit_never_change_the_hook_result() { + for (case, expected) in [ + ("write_failed", "state_write_failed"), + ("corrupt_state", "anchored"), + ("unavailable_state", "state_unavailable"), + ] { + let fixture = started_fixture().await; + let git_dir = git_dir_of(&fixture).await; + let marker = ExternalTaintMarker::new(&git_dir); + let path = state_path(&git_dir); + std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); + match case { + "corrupt_state" => std::fs::write(&path, b"not json").expect("corrupt state"), + "unavailable_state" => std::fs::create_dir(&path).expect("directory as state"), + _ => {} + } + super::support::write_file(&fixture.root, "work.txt", "work\n"); + + let observed = Arc::new(Mutex::new(None)); + let db_path = fixture.db_path(); + let advise = { + let observed = Arc::clone(&observed); + let marker = marker.clone(); + move |root: &Path| { + let committed = tokio::task::block_in_place(|| { + tokio::runtime::Handle::current().block_on(durable_snapshot( + &db_path, + &main_worktree(), + SCOPE, + )) + }); + let refs_before = sce_refs(root); + let marker_before = marker.exists().expect("marker"); + let outcome = if case == "write_failed" { + advise_if_due_with(root, || NOW, failing_write(PersistPhase::Rename)) + } else { + advise_if_due_with(root, || NOW, write_state_atomically) + }; + let report = outcome.report(); + *observed.lock().expect("observed slot") = Some(( + committed, + refs_before == sce_refs(root), + marker_before == marker.exists().expect("marker"), + report.outcome, + )); + report + } + }; + + let output = run_hook( + &fixture.root, + &fixture.db_path(), + close_payload(SCOPE, "close-1"), + WORKTREE_LOCK_TIMEOUT, + || {}, + || {}, + advise, + ) + .await + .unwrap_or_else(|error| panic!("{case}: hook must succeed, got {error:#}")); + + assert_eq!(output, "", "{case}"); + let (committed, refs_unchanged, marker_unchanged, outcome) = observed + .lock() + .expect("observed slot") + .clone() + .expect("advisory ran"); + assert_eq!(outcome, expected, "{case}"); + assert!(refs_unchanged, "{case}"); + assert!(marker_unchanged, "{case}"); + assert_eq!(committed.scope, Some(ScopeStatus::Closed), "{case}"); + assert_eq!( + durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await, + committed, + "{case}" + ); + assert!(!marker.exists().expect("marker"), "{case}"); + } +} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/cross_process.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/cross_process.rs new file mode 100644 index 000000000..58b5cb461 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/cross_process.rs @@ -0,0 +1,166 @@ +use std::io::{BufRead, BufReader, Write}; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::sync::mpsc; + +use super::super::super::maintenance_state::{ + read_state, state_path, write_state_atomically, MaintenanceState, StateRead, + RECONCILIATION_ADVISORY_AFTER_MS, +}; +use super::super::super::ref_advisory::{advise_if_due, advise_if_due_with}; +use super::support::WATCHDOG; +use super::{Fixture, NOW}; + +const ROLE_ENV: &str = "SCE_CROSS_PROCESS_ADVISORY_ROLE"; +const ROOT_ENV: &str = "SCE_CROSS_PROCESS_ADVISORY_ROOT"; +const NOW_ENV: &str = "SCE_CROSS_PROCESS_ADVISORY_NOW"; +const ROLE_PARK: &str = "park"; +const ROLE_PROBE: &str = "probe"; +const PARKED_LINE: &str = "PARKED"; +const OUTCOME_PREFIX: &str = "OUTCOME:"; +const CHILD_TEST: &str = "ref_reconciliation_cross_process_advisory_child"; + +fn child_test_path() -> String { + let module = module_path!(); + let (_, relative) = module + .split_once("::") + .expect("crate-qualified module path"); + format!("{relative}::{CHILD_TEST}") +} + +fn child_command(role: &str, root: &Path) -> Command { + let mut command = Command::new(std::env::current_exe().expect("current exe")); + command + .args(["--exact", &child_test_path(), "--nocapture"]) + .env(ROLE_ENV, role) + .env(ROOT_ENV, root) + .env(NOW_ENV, NOW.to_string()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + command +} + +fn outcome_of(stdout: &str) -> String { + stdout + .lines() + .find_map(|line| line.strip_prefix(OUTCOME_PREFIX)) + .unwrap_or_else(|| panic!("child reported no outcome:\n{stdout}")) + .to_string() +} + +fn run_probe(root: &Path) -> String { + let output = child_command(ROLE_PROBE, root) + .stdin(Stdio::null()) + .output() + .expect("probe child runs"); + assert!( + output.status.success(), + "probe child failed:\n{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + outcome_of(&String::from_utf8_lossy(&output.stdout)) +} + +#[test] +fn ref_reconciliation_cross_process_advisory_child() { + let Ok(role) = std::env::var(ROLE_ENV) else { + return; + }; + let root = PathBuf::from(std::env::var(ROOT_ENV).expect("root")); + let now: i64 = std::env::var(NOW_ENV) + .expect("now") + .parse() + .expect("numeric now"); + let outcome = match role.as_str() { + ROLE_PARK => advise_if_due_with( + &root, + || now, + |path, state| { + println!("{PARKED_LINE}"); + std::io::stdout().flush().expect("flush stdout"); + let mut release = String::new(); + std::io::stdin() + .read_line(&mut release) + .expect("wait for release"); + write_state_atomically(path, state) + }, + ), + ROLE_PROBE => advise_if_due(&root, || now), + other => panic!("unknown role {other}"), + }; + let name = format!("{outcome:?}"); + let name = name + .split([' ', '{']) + .next() + .unwrap_or_default() + .to_string(); + println!("{OUTCOME_PREFIX}{name}"); +} + +#[test] +fn ref_reconciliation_advisory_lock_serializes_independent_processes() { + let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); + let git_dir = fixture.root.join(".git"); + let path = state_path(&git_dir); + std::fs::create_dir_all(path.parent().expect("runtime dir")).expect("runtime dir"); + let due = MaintenanceState { + anchor: Some(NOW - RECONCILIATION_ADVISORY_AFTER_MS - 1), + ..MaintenanceState::default() + }; + write_state_atomically(&path, &due).expect("seed due state"); + let seeded = std::fs::read(&path).expect("seeded bytes"); + + let mut parked = child_command(ROLE_PARK, &fixture.root) + .stdin(Stdio::piped()) + .spawn() + .expect("park child spawns"); + let mut stdout = BufReader::new(parked.stdout.take().expect("park stdout")); + let (line_tx, line_rx) = mpsc::channel::(); + let reader = std::thread::spawn(move || { + let mut lines = Vec::new(); + let mut line = String::new(); + while stdout.read_line(&mut line).expect("read child stdout") > 0 { + let trimmed = line.trim_end().to_string(); + if trimmed == PARKED_LINE { + let _ = line_tx.send(trimmed.clone()); + } + lines.push(trimmed); + line.clear(); + } + lines + }); + assert_eq!( + line_rx.recv_timeout(WATCHDOG).expect("child parks"), + PARKED_LINE + ); + + assert_eq!(run_probe(&fixture.root), "Busy"); + assert_eq!( + std::fs::read(&path).expect("state while parked"), + seeded, + "a second process must not write while the first holds the lock" + ); + + let mut stdin = parked.stdin.take().expect("park stdin"); + stdin + .write_all(b"release\n") + .expect("release the parked child"); + drop(stdin); + let status = parked.wait().expect("park child exits"); + assert!(status.success()); + let lines = reader.join().expect("reader joins"); + assert_eq!( + outcome_of(&lines.join("\n")), + "Advised", + "the lock holder recorded the one advice" + ); + + assert_eq!(run_probe(&fixture.root), "NoAction"); + + let StateRead::Valid(stored) = read_state(&path).expect("state") else { + panic!("expected valid state"); + }; + assert_eq!(stored.last_advised, Some(NOW)); + assert_eq!(stored.anchor, due.anchor); +} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/fail_closed.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/fail_closed.rs new file mode 100644 index 000000000..d8183fb6c --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/fail_closed.rs @@ -0,0 +1,99 @@ +use std::time::Duration; + +use super::super::super::external_taint::ExternalTaintMarker; +use super::super::super::git_snapshot::resolve_git_dir; +use super::super::super::maintenance_state::{read_state, state_path, StateRead}; +use super::super::super::ref_reconciliation::{ReconcileError, ReconcilePhase}; +use super::super::reconcile_explicit_with; +use super::super::{ExplicitOutcome, SkipReason}; +use super::support::{await_signal, pinned_trees, spawn_parked_pass, write_file}; +use super::{pin_orphan, seed_durable_root, Fixture, NOW}; + +const REMOTE: &str = "https://example.invalid/org/repo-a.git"; + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_missing_local_pin_fails_closed_while_other_orphans_exist() { + let fixture = Fixture::new(REMOTE); + fixture.create_db().await; + write_file(&fixture.root, "orphan.txt", "orphan\n"); + let orphan = pin_orphan(&fixture.root).await; + write_file(&fixture.root, "required.txt", "required\n"); + let snapshot = super::super::super::git_snapshot::GitSnapshotService::new(&fixture.root) + .await + .expect("snapshot"); + let unpinned = snapshot.capture_tree().await.expect("tree"); + seed_durable_root(&fixture.db_path(), &unpinned.0).await; + let refs_before = pinned_trees(&fixture.root, "main"); + assert_eq!(refs_before, [orphan].into_iter().collect()); + + let (outcome, _) = fixture.run().await; + + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("expected failure, got {outcome:?}"); + }; + let ReconcileError::MissingRequiredPins { missing } = error else { + panic!("expected missing required pins, got {error:?}"); + }; + assert_eq!(missing, vec![unpinned]); + assert_eq!( + pinned_trees(&fixture.root, "main"), + refs_before, + "the eligible orphan is not deleted when a required pin is missing" + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_database_failure_never_arms_the_external_taint_marker() { + let fixture = Fixture::new(REMOTE); + pin_orphan(&fixture.root).await; + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let marker = ExternalTaintMarker::new(&git_dir); + + let (outcome, _) = fixture.run().await; + + assert!(matches!(outcome, ExplicitOutcome::Failed { .. })); + assert!(!marker.exists().expect("marker")); + assert_eq!(pinned_trees(&fixture.root, "main").len(), 1); + let StateRead::Valid(state) = read_state(&state_path(&git_dir)).expect("state") else { + panic!("expected recorded failure"); + }; + assert_eq!(state.consecutive_failures, 1); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_second_explicit_pass_is_busy_and_records_exactly_one_pass() { + let fixture = Fixture::new(REMOTE); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + + let mut first = spawn_parked_pass(&fixture, ReconcilePhase::PinsInventoried); + await_signal(&mut first.parked).await; + + let second = reconcile_explicit_with( + &fixture.root, + async || -> anyhow::Result<_> { unreachable!("a busy pass never opens the database") }, + || NOW, + |_, _| unreachable!("a busy pass never writes state"), + |_| {}, + Duration::ZERO, + ) + .await; + assert!(matches!(second, ExplicitOutcome::Skipped(SkipReason::Busy))); + assert!(matches!( + read_state(&state_path(&git_dir)).expect("state"), + StateRead::Absent + )); + + let outcome = first.finish().await; + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!(report.deleted, 1); + let StateRead::Valid(state) = read_state(&state_path(&git_dir)).expect("state") else { + panic!("expected recorded success"); + }; + assert_eq!(state.last_success, Some(NOW)); + assert_eq!(state.consecutive_failures, 0); + assert!(pinned_trees(&fixture.root, "main").is_empty()); +} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/state_recovery.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/state_recovery.rs new file mode 100644 index 000000000..9a8e0fb6f --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/state_recovery.rs @@ -0,0 +1,413 @@ +use std::path::Path; +use std::time::Duration; + +use super::super::super::external_taint::ExternalTaintMarker; +use super::super::super::git_snapshot::resolve_git_dir; +use super::super::super::maintenance_state::{ + evaluate_recommendation, read_state, state_path, write_state_atomically, MaintenanceState, + PersistFailure, PersistPhase, StateRead, FUTURE_SKEW_TOLERANCE_MS, + RECONCILIATION_ADVISORY_AFTER_MS, +}; +use super::super::super::ref_advisory::{advise_if_due, AdvisoryOutcome}; +use super::super::super::worktree_lock::acquire_inner_async; +use super::super::{reconcile_explicit_with, ExplicitOutcome, SkipReason, StatePersistWarning}; +use super::support::pinned_trees; +use super::{ + failing_write, pin_orphan, remove_database, unavailable_cause, Fixture, NOW, + OTHER_REPOSITORY_ID, +}; +use crate::services::agent_trace_db::repository::{ + ExistingRepositoryDbError, RepositoryAgentTraceDb, +}; + +const REMOTE: &str = "https://example.invalid/org/repo-a.git"; + +async fn seed_state(fixture: &Fixture, state: &MaintenanceState) -> std::path::PathBuf { + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let path = state_path(&git_dir); + std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); + write_state_atomically(&path, state).expect("seed state"); + path +} + +fn invalid_state() -> MaintenanceState { + MaintenanceState { + anchor: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + last_success: Some(NOW + FUTURE_SKEW_TOLERANCE_MS + 1), + last_advised: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + ..MaintenanceState::default() + } +} + +fn assert_within_tolerance(state: &MaintenanceState) { + for timestamp in [ + state.anchor, + state.last_success, + state.last_attempt, + state.last_advised, + ] + .into_iter() + .flatten() + { + assert!(timestamp <= NOW + FUTURE_SKEW_TOLERANCE_MS); + } +} + +async fn logical_dump(path: &Path) -> Vec { + let db = RepositoryAgentTraceDb::open_without_migrations_at(path) + .await + .expect("inspection open"); + let tables = db + .query_map( + "SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name", + (), + |row| row.get::(0).map_err(Into::into), + ) + .await + .expect("tables"); + let mut dump = Vec::new(); + for table in &tables { + let columns = db + .query_map( + &format!("SELECT name FROM pragma_table_info('{table}') ORDER BY cid"), + (), + |row| row.get::(0).map_err(Into::into), + ) + .await + .expect("columns"); + let projection = columns + .iter() + .map(|column| format!("quote(\"{column}\")")) + .collect::>() + .join(" || '|' || "); + let rows = db + .query_map( + &format!("SELECT {projection} FROM \"{table}\" ORDER BY rowid"), + (), + |row| row.get::(0).map_err(Into::into), + ) + .await + .expect("rows"); + dump.push(format!("{table}:{}", rows.join(";"))); + } + if tables.iter().any(|table| table == "__sce_migrations") { + let ids = db + .query_map( + "SELECT CAST(id AS TEXT) FROM __sce_migrations ORDER BY id", + (), + |row| row.get::(0).map_err(Into::into), + ) + .await + .expect("migrations"); + dump.push(format!("migrations:{}", ids.join(","))); + } + if tables.iter().any(|table| table == "repository_metadata") { + let ids = db + .query_map( + "SELECT repository_id FROM repository_metadata ORDER BY id", + (), + |row| row.get::(0).map_err(Into::into), + ) + .await + .expect("metadata"); + dump.push(format!("metadata:{}", ids.join(","))); + } + dump +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_verification_failures_leave_the_logical_database_unchanged() { + for case in ["incompatible", "missing_metadata", "mismatch"] { + let fixture = Fixture::new(REMOTE); + match case { + "incompatible" => drop( + RepositoryAgentTraceDb::open_without_migrations_at(fixture.db_path()) + .await + .expect("unmigrated db"), + ), + "missing_metadata" => drop( + RepositoryAgentTraceDb::new_at(fixture.db_path()) + .await + .expect("migrated db"), + ), + _ => super::create_db(&fixture.db_path(), OTHER_REPOSITORY_ID).await, + } + pin_orphan(&fixture.root).await; + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let before = logical_dump(&fixture.db_path()).await; + if case == "mismatch" { + assert!(before.iter().any(|line| line.contains(OTHER_REPOSITORY_ID))); + } + + let (outcome, phases) = fixture.run().await; + + let ExplicitOutcome::Failed { error, .. } = outcome else { + panic!("{case}: expected failure, got {outcome:?}"); + }; + match (case, unavailable_cause(&error)) { + ("incompatible", ExistingRepositoryDbError::IncompatibleSchema(_)) + | ("missing_metadata", ExistingRepositoryDbError::MissingMetadata) + | ("mismatch", ExistingRepositoryDbError::RepositoryMismatch { .. }) => {} + (_, other) => panic!("{case}: unexpected cause {other:?}"), + } + assert!(phases.is_empty(), "{case}"); + assert_eq!(logical_dump(&fixture.db_path()).await, before, "{case}"); + assert_eq!(pinned_trees(&fixture.root, "main").len(), 1, "{case}"); + assert!(!ExternalTaintMarker::new(&git_dir).exists().expect("marker")); + } +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_busy_explicit_pass_leaves_invalid_state_byte_identical() { + let fixture = Fixture::new(REMOTE); + let path = seed_state(&fixture, &invalid_state()).await; + let before = std::fs::read(&path).expect("state bytes"); + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let held = acquire_inner_async(&git_dir, Duration::from_secs(10), || {}) + .await + .expect("hold lock"); + + let outcome = reconcile_explicit_with( + &fixture.root, + async || -> anyhow::Result { unreachable!("db must not open") }, + || NOW, + |_, _| unreachable!("state must not be written"), + |_| {}, + Duration::ZERO, + ) + .await; + drop(held); + + assert!(matches!( + outcome, + ExplicitOutcome::Skipped(SkipReason::Busy) + )); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + let StateRead::Valid(state) = read_state(&path).expect("state") else { + panic!("expected valid state"); + }; + assert!(evaluate_recommendation(&state, NOW).invalid_timestamps); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_successful_pass_normalizes_invalid_state_and_a_later_read_does_not_rewrite( +) { + let fixture = Fixture::new(REMOTE); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + let near_future = NOW + FUTURE_SKEW_TOLERANCE_MS; + let seeded = MaintenanceState { + last_advised: Some(near_future), + ..invalid_state() + }; + let path = seed_state(&fixture, &seeded).await; + + let (outcome, _) = fixture.run().await; + assert!(matches!(outcome, ExplicitOutcome::Completed(_))); + + let StateRead::Valid(normalized) = read_state(&path).expect("state") else { + panic!("expected valid state"); + }; + assert_within_tolerance(&normalized); + assert_eq!(normalized.last_success, Some(NOW)); + assert_eq!(normalized.last_advised, Some(near_future)); + let bytes = std::fs::read(&path).expect("state bytes"); + + assert!(matches!( + advise_if_due(&fixture.root, || NOW), + AdvisoryOutcome::NoAction + )); + assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); + assert!(!evaluate_recommendation(&normalized, NOW).invalid_timestamps); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_failed_state_write_during_normalization_has_two_distinct_legs() { + let fixture = Fixture::new(REMOTE); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + let path = seed_state(&fixture, &invalid_state()).await; + let before = std::fs::read(&path).expect("state bytes"); + + let (pre_rename, _) = fixture.run_with(failing_write(PersistPhase::Rename)).await; + let ExplicitOutcome::CompletedStatePersistFailed { report, warning } = pre_rename else { + panic!("expected persist-failed completion, got {pre_rename:?}"); + }; + assert_eq!(report.deleted, 1); + assert!(matches!( + warning, + StatePersistWarning::WriteFailed(PersistFailure::NotApplied { + phase: PersistPhase::Rename, + .. + }) + )); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + + pin_orphan(&fixture.root).await; + let (post_rename, _) = fixture + .run_with(failing_write(PersistPhase::SyncParentDirectory)) + .await; + let ExplicitOutcome::CompletedStatePersistFailed { warning, .. } = post_rename else { + panic!("expected persist-failed completion, got {post_rename:?}"); + }; + assert!(matches!( + warning, + StatePersistWarning::WriteFailed(PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + .. + }) + )); + let StateRead::Valid(visible) = read_state(&path).expect("state") else { + panic!("expected valid state"); + }; + assert_within_tolerance(&visible); + assert_eq!(visible.last_success, Some(NOW)); + + let bytes = std::fs::read(&path).expect("state bytes"); + assert!(matches!( + advise_if_due(&fixture.root, || NOW), + AdvisoryOutcome::NoAction + )); + assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success() { + let fixture = Fixture::new(REMOTE); + pin_orphan(&fixture.root).await; + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + let path = state_path(&git_dir); + let stored = || match read_state(&path).expect("state") { + StateRead::Valid(state) => state, + other => panic!("expected valid state, got {other:?}"), + }; + + for expected in 1..=2 { + let (outcome, _) = fixture.run().await; + assert!(matches!(outcome, ExplicitOutcome::Failed { .. })); + assert_eq!(stored().consecutive_failures, expected); + } + + fixture.create_db().await; + let (outcome, _) = fixture.run().await; + assert!(matches!(outcome, ExplicitOutcome::Completed(_))); + let healed = stored(); + assert_eq!(healed.consecutive_failures, 0); + assert_eq!(healed.last_failure, None); + assert_eq!(healed.last_success, Some(NOW)); + + remove_database(&fixture.db_path()); + let (outcome, _) = fixture.run().await; + assert!(matches!(outcome, ExplicitOutcome::Failed { .. })); + let failed = stored(); + assert_eq!(failed.consecutive_failures, 1); + assert_eq!(failed.last_success, Some(NOW)); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition( +) { + let aged = MaintenanceState { + anchor: Some(NOW - 3 * RECONCILIATION_ADVISORY_AFTER_MS), + last_advised: Some(NOW - 2 * RECONCILIATION_ADVISORY_AFTER_MS), + ..MaintenanceState::default() + }; + for case in ["aged", "invalid", "corrupt", "oversized"] { + let fixture = Fixture::new(REMOTE); + fixture.create_db().await; + pin_orphan(&fixture.root).await; + let path = match case { + "aged" => seed_state(&fixture, &aged).await, + "invalid" => seed_state(&fixture, &invalid_state()).await, + _ => { + let path = seed_state(&fixture, &MaintenanceState::default()).await; + let junk = if case == "corrupt" { + b"not json".to_vec() + } else { + vec![b' '; 5000] + }; + std::fs::write(&path, junk).expect("unusable state"); + path + } + }; + + let (outcome, _) = fixture.run().await; + + let ExplicitOutcome::Completed(report) = outcome else { + panic!("{case}: expected completed pass, got {outcome:?}"); + }; + assert_eq!(report.deleted, 1, "{case}"); + let StateRead::Valid(rewritten) = read_state(&path).expect("state") else { + panic!("{case}: state must be rewritten valid"); + }; + assert_eq!(rewritten.last_success, Some(NOW), "{case}"); + assert_within_tolerance(&rewritten); + } +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_unnormalized_invalid_state_stays_eligible_until_a_successful_advisory() +{ + let fixture = Fixture::new(REMOTE); + let invalid = MaintenanceState { + anchor: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + ..MaintenanceState::default() + }; + let path = seed_state(&fixture, &invalid).await; + let before = std::fs::read(&path).expect("state bytes"); + let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); + + for step in 0..3 { + let now = NOW + step * RECONCILIATION_ADVISORY_AFTER_MS / 2; + let recommendation = evaluate_recommendation(&invalid, now); + assert!(recommendation.recommended); + assert!(recommendation.invalid_timestamps); + } + let held = acquire_inner_async(&git_dir, Duration::from_secs(10), || {}) + .await + .expect("hold lock"); + for step in 0..3 { + assert!(matches!( + advise_if_due(&fixture.root, || NOW + + step * RECONCILIATION_ADVISORY_AFTER_MS), + AdvisoryOutcome::Busy + )); + } + drop(held); + assert_eq!(std::fs::read(&path).expect("state bytes"), before); + + let later = NOW + RECONCILIATION_ADVISORY_AFTER_MS; + assert!(matches!( + advise_if_due(&fixture.root, || later), + AdvisoryOutcome::Advised + )); + let StateRead::Valid(state) = read_state(&path).expect("state") else { + panic!("expected valid state"); + }; + assert_eq!(state.last_advised, Some(later)); + assert_eq!(state.anchor, Some(later)); + assert!(matches!( + advise_if_due(&fixture.root, || later), + AdvisoryOutcome::NoAction + )); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_invalid_last_advised_does_not_suppress_a_due_advisory() { + let fixture = Fixture::new(REMOTE); + let seeded = MaintenanceState { + last_success: Some(NOW - RECONCILIATION_ADVISORY_AFTER_MS - 1), + last_advised: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + ..MaintenanceState::default() + }; + let path = seed_state(&fixture, &seeded).await; + + assert!(matches!( + advise_if_due(&fixture.root, || NOW), + AdvisoryOutcome::Advised + )); + let StateRead::Valid(state) = read_state(&path).expect("state") else { + panic!("expected valid state"); + }; + assert_eq!(state.last_advised, Some(NOW)); +} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/support.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/support.rs new file mode 100644 index 000000000..0af626163 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/support.rs @@ -0,0 +1,263 @@ +use std::collections::BTreeSet; +use std::path::{Path, PathBuf}; +use std::process::Command; +use std::sync::mpsc; +use std::time::Duration; + +use tokio::sync::mpsc::{unbounded_channel, UnboundedReceiver}; + +use super::super::super::coordinator::{coordinate_inner, CoordinateOutcome}; +use super::super::super::maintenance_state::write_state_atomically; +use super::super::super::protected_worktree::WORKTREE_LOCK_TIMEOUT; +use super::super::super::ref_reconciliation::ReconcilePhase; +use super::super::{open_authoritative_db_at_state_root, reconcile_explicit_with, ExplicitOutcome}; +use super::{Fixture, NOW}; +use crate::services::agent_trace_db::repository::RepositoryAgentTraceDb; +use crate::services::hooks::mutation_scope::{drive_mutation_scope, MutationScopePayload}; +use crate::services::mutation_trace::runtime::{ + AbandonScopeError, AbandonScopeOutcome, AdvisoryReport, RuntimeBoundary, +}; +use crate::services::mutation_trace::store::MutationTraceStore; +use crate::services::mutation_trace::types::{ + ActorKind, ScopeId, ScopeStatus, TreeId, WorktreeId, WorktreeState, +}; +use crate::services::observability::traits::NoopLogger; + +pub(super) const WATCHDOG: Duration = Duration::from_mins(1); + +pub(super) fn main_worktree() -> WorktreeId { + WorktreeId("main".to_string()) +} + +pub(super) fn start_payload(scope: &str, event: &str) -> MutationScopePayload { + MutationScopePayload::Start { + scope_id: scope.to_string(), + event_id: event.to_string(), + actor_kind: ActorKind::ClaudeCode, + provenance: None, + } +} + +pub(super) fn close_payload(scope: &str, event: &str) -> MutationScopePayload { + MutationScopePayload::Close { + scope_id: scope.to_string(), + event_id: event.to_string(), + actor_kind: ActorKind::ClaudeCode, + } +} + +pub(super) fn write_file(root: &Path, name: &str, content: &str) { + std::fs::write(root.join(name), content).expect("write working tree file"); +} + +pub(super) fn sce_refs(root: &Path) -> Vec { + let output = Command::new("git") + .args([ + "for-each-ref", + "--format=%(refname) %(objectname)", + "refs/sce/", + ]) + .current_dir(root) + .output() + .expect("for-each-ref"); + assert!(output.status.success(), "for-each-ref failed"); + let mut lines: Vec = String::from_utf8_lossy(&output.stdout) + .lines() + .map(str::to_string) + .collect(); + lines.sort(); + lines +} + +pub(super) fn pinned_trees(root: &Path, worktree: &str) -> BTreeSet { + let prefix = format!("refs/sce/mutation-cursor/{worktree}/"); + sce_refs(root) + .into_iter() + .filter_map(|line| { + let (name, tree) = line.split_once(' ')?; + name.strip_prefix(&prefix) + .filter(|rest| !rest.contains('/')) + .map(|_| tree.to_string()) + }) + .collect() +} + +pub(super) async fn writer(db_path: &Path) -> anyhow::Result { + RepositoryAgentTraceDb::open_without_migrations_at(db_path).await +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(super) struct DurableSnapshot { + pub worktree: WorktreeState, + pub scope: Option, + pub roots: BTreeSet, +} + +pub(super) async fn durable_snapshot( + db_path: &Path, + worktree: &WorktreeId, + scope: &str, +) -> DurableSnapshot { + let mut db = writer(db_path).await.expect("inspection db"); + let store = MutationTraceStore::new(&mut db); + let projection = store + .load_worktree(worktree, None, None) + .await + .expect("load worktree") + .expect("worktree row"); + let scope = store + .load_scope(&ScopeId(scope.to_string())) + .await + .expect("load scope") + .map(|state| state.status); + let roots = store.load_all_tree_roots().await.expect("roots"); + DurableSnapshot { + worktree: projection.worktree_state, + scope, + roots, + } +} + +pub(super) async fn run_hook( + root: &Path, + db_path: &Path, + payload: MutationScopePayload, + lock_timeout: Duration, + on_contention: C, + in_open_db: O, + advise: V, +) -> anyhow::Result +where + C: FnOnce() + Send + 'static, + O: FnOnce(), + V: FnOnce(&Path) -> AdvisoryReport, +{ + drive_mutation_scope( + root, + payload, + None::<&NoopLogger>, + &mut Vec::new(), + async |root: &Path, boundary: &RuntimeBoundary| { + coordinate_inner( + root, + boundary, + async || { + in_open_db(); + writer(db_path).await + }, + lock_timeout, + on_contention, + |_attempt| {}, + |_attempt| Ok(()), + ) + .await + }, + async |_root: &Path, _scope: &ScopeId| -> Result { + unreachable!("abandon is not driven by these tests") + }, + advise, + ) + .await +} + +pub(super) fn no_advice(_root: &Path) -> AdvisoryReport { + AdvisoryReport { + outcome: "no_action", + severity: crate::services::mutation_trace::runtime::AdvisorySeverity::Debug, + recommendation: None, + warning: None, + diagnostic: None, + } +} + +pub(super) async fn hook_ok(fixture: &Fixture, payload: MutationScopePayload) -> String { + run_hook( + &fixture.root, + &fixture.db_path(), + payload, + WORKTREE_LOCK_TIMEOUT, + || {}, + || {}, + no_advice, + ) + .await + .expect("hook boundary completes") +} + +pub(super) async fn coordinate_flush(root: &Path, db_path: &Path) -> CoordinateOutcome { + coordinate_inner( + root, + &RuntimeBoundary::Flush, + async || writer(db_path).await, + WORKTREE_LOCK_TIMEOUT, + || {}, + |_attempt| {}, + |_attempt| Ok(()), + ) + .await + .expect("flush completes") +} + +pub(super) struct ParkedPass { + outcome: tokio::sync::oneshot::Receiver, + pub parked: UnboundedReceiver<()>, + release: mpsc::Sender<()>, +} + +impl ParkedPass { + pub(super) async fn finish(self) -> ExplicitOutcome { + self.release.send(()).expect("release the pass"); + tokio::time::timeout(WATCHDOG, self.outcome) + .await + .expect("pass watchdog") + .expect("pass thread reports its outcome") + } +} + +pub(super) fn spawn_parked_pass(fixture: &Fixture, target: ReconcilePhase) -> ParkedPass { + spawn_parked_pass_at(&fixture.root, &fixture.state_root, target) +} + +pub(super) fn spawn_parked_pass_at( + root: &Path, + state_root: &Path, + target: ReconcilePhase, +) -> ParkedPass { + let root: PathBuf = root.to_path_buf(); + let state_root: PathBuf = state_root.to_path_buf(); + let (parked_tx, parked) = unbounded_channel(); + let (release, release_rx) = mpsc::channel::<()>(); + let (outcome_tx, outcome) = tokio::sync::oneshot::channel(); + std::thread::spawn(move || { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("pass runtime"); + let result = runtime.block_on(reconcile_explicit_with( + &root, + async || open_authoritative_db_at_state_root(&root, &state_root).await, + || NOW, + write_state_atomically, + move |phase| { + if phase == target { + let _ = parked_tx.send(()); + let _ = release_rx.recv_timeout(WATCHDOG); + } + }, + Duration::from_secs(10), + )); + let _ = outcome_tx.send(result); + }); + ParkedPass { + outcome, + parked, + release, + } +} + +pub(super) async fn await_signal(receiver: &mut UnboundedReceiver<()>) { + tokio::time::timeout(WATCHDOG, receiver.recv()) + .await + .expect("watchdog expired while waiting for a synchronization signal") + .expect("signal sender dropped before signalling"); +} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/worktrees.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/worktrees.rs new file mode 100644 index 000000000..70064c806 --- /dev/null +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/worktrees.rs @@ -0,0 +1,263 @@ +use std::collections::BTreeSet; +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use super::super::super::git_snapshot::{resolve_worktree_id, GitSnapshotService}; +use super::super::super::maintenance_state::write_state_atomically; +use super::super::super::ref_reconciliation::{ReconcileError, ReconciliationReport}; +use super::super::{open_authoritative_db_at_state_root, reconcile_explicit_with, ExplicitOutcome}; +use super::support::{ + coordinate_flush, hook_ok, pinned_trees, sce_refs, start_payload, write_file, writer, +}; +use super::{git, pin_orphan, Fixture, NOW}; +use crate::services::mutation_trace::store::MutationTraceStore; +use crate::services::mutation_trace::types::WorktreeId; +use crate::services::repository_identity::resolve::resolve_repository_identity; + +const REMOTE: &str = "https://example.invalid/org/repo-a.git"; + +struct Linked { + fixture: Fixture, + root: PathBuf, + id: String, +} + +impl Linked { + async fn new() -> Self { + let fixture = Fixture::new(REMOTE); + let root = fixture.dir.path().join("linked"); + git( + &fixture.root, + &[ + "worktree", + "add", + "-q", + "-b", + "linked-branch", + root.to_str().expect("utf-8 path"), + ], + ); + let main_identity = resolve_repository_identity(&fixture.root, None, "origin") + .expect("main identity") + .identity + .repository_id; + let linked_identity = resolve_repository_identity(&root, None, "origin") + .expect("linked identity") + .identity + .repository_id; + assert_eq!(main_identity, linked_identity); + assert_eq!( + resolve_worktree_id(&fixture.root).await.expect("main id"), + WorktreeId("main".to_string()) + ); + let id = resolve_worktree_id(&root).await.expect("linked id").0; + assert!(id.starts_with("worktrees/")); + fixture.create_db().await; + Self { fixture, root, id } + } + + async fn run(&self, root: &Path) -> ExplicitOutcome { + reconcile_explicit_with( + root, + async || open_authoritative_db_at_state_root(root, &self.fixture.state_root).await, + || NOW, + write_state_atomically, + |_| {}, + Duration::from_secs(10), + ) + .await + } + + async fn linked_hook( + &self, + payload: crate::services::hooks::mutation_scope::MutationScopePayload, + ) { + super::support::run_hook( + &self.root, + &self.fixture.db_path(), + payload, + super::super::super::protected_worktree::WORKTREE_LOCK_TIMEOUT, + || {}, + || {}, + super::support::no_advice, + ) + .await + .expect("linked hook completes"); + } + + fn linked_pins(&self) -> BTreeSet { + pinned_trees(&self.root, &self.id) + } + + fn main_pins(&self) -> BTreeSet { + pinned_trees(&self.fixture.root, "main") + } +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_historical_roots_of_a_real_coordinator_sequence_stay_protected() { + let linked = Linked::new().await; + let fixture = &linked.fixture; + hook_ok(fixture, start_payload("scope-1", "start-1")).await; + let db_path = fixture.db_path(); + + let mut trees = BTreeSet::new(); + let mut store_db = writer(&db_path).await.expect("db"); + let cursor = MutationTraceStore::new(&mut store_db) + .load_worktree(&WorktreeId("main".to_string()), None, None) + .await + .expect("load") + .expect("row") + .worktree_state + .cursor_tree + .0; + trees.insert(cursor); + for (name, content) in [("b.txt", "b\n"), ("c.txt", "c\n"), ("d.txt", "d\n")] { + write_file(&fixture.root, name, content); + let outcome = coordinate_flush(&fixture.root, &db_path).await; + trees.insert(outcome.observed_tree.0); + } + assert_eq!(trees.len(), 4); + assert_eq!(linked.main_pins(), trees); + + write_file(&fixture.root, "orphan.txt", "orphan\n"); + let orphan = pin_orphan(&fixture.root).await; + assert!(!trees.contains(&orphan)); + + let outcome = linked.run(&fixture.root).await; + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!( + report, + ReconciliationReport { + local_required: 4, + retained: 4, + deleted: 1, + } + ); + assert_eq!( + linked.main_pins(), + trees, + "historical A, B, C and D are all retained although only D is the cursor" + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_pin_of_one_real_worktree_is_retained_while_another_needs_the_tree() { + let linked = Linked::new().await; + let fixture = &linked.fixture; + + write_file(&fixture.root, "shared.txt", "shared\n"); + let shared = pin_orphan(&fixture.root).await; + write_file(&linked.root, "shared.txt", "shared\n"); + linked + .linked_hook(start_payload("linked-scope", "linked-start")) + .await; + assert_eq!(linked.linked_pins(), [shared.clone()].into_iter().collect()); + + write_file(&fixture.root, "orphan.txt", "orphan\n"); + let orphan = pin_orphan(&fixture.root).await; + assert_eq!( + linked.main_pins(), + [shared.clone(), orphan.clone()].into_iter().collect() + ); + + let outcome = linked.run(&fixture.root).await; + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!( + report, + ReconciliationReport { + local_required: 0, + retained: 1, + deleted: 1, + } + ); + assert_eq!( + linked.main_pins(), + [shared.clone()].into_iter().collect(), + "main's pin is the same tree the linked worktree durably requires" + ); + assert_eq!(linked.linked_pins(), [shared].into_iter().collect()); +} + +#[tokio::test(flavor = "multi_thread")] +async fn ref_reconciliation_missing_pin_in_another_real_worktree_neither_fails_nor_is_repaired() { + let linked = Linked::new().await; + let fixture = &linked.fixture; + let db_path = fixture.db_path(); + + write_file(&fixture.root, "shared.txt", "shared\n"); + let shared = pin_orphan(&fixture.root).await; + write_file(&linked.root, "shared.txt", "shared\n"); + linked + .linked_hook(start_payload("linked-scope", "linked-start")) + .await; + write_file(&linked.root, "more.txt", "more\n"); + let flushed = coordinate_flush(&linked.root, &db_path).await; + let later = flushed.observed_tree.0; + assert_eq!( + linked.linked_pins(), + [shared.clone(), later.clone()].into_iter().collect() + ); + + let linked_ref = format!("refs/sce/mutation-cursor/{}/{shared}", linked.id); + git(&linked.root, &["update-ref", "-d", &linked_ref]); + assert_eq!(linked.linked_pins(), [later.clone()].into_iter().collect()); + + write_file(&fixture.root, "orphan.txt", "orphan\n"); + let orphan = pin_orphan(&fixture.root).await; + let refs_of_linked_before: Vec = sce_refs(&fixture.root) + .into_iter() + .filter(|line| line.contains(&linked.id)) + .collect(); + + let outcome = linked.run(&fixture.root).await; + let ExplicitOutcome::Completed(report) = outcome else { + panic!("expected completed pass, got {outcome:?}"); + }; + assert_eq!( + report, + ReconciliationReport { + local_required: 0, + retained: 1, + deleted: 1, + } + ); + assert!(!linked.main_pins().contains(&orphan)); + assert_eq!( + linked.main_pins(), + [shared.clone()].into_iter().collect(), + "the remaining protection for the other worktree's tree is retained" + ); + let refs_of_linked_after: Vec = sce_refs(&fixture.root) + .into_iter() + .filter(|line| line.contains(&linked.id)) + .collect(); + assert_eq!(refs_of_linked_before, refs_of_linked_after); + assert_eq!( + linked.linked_pins(), + [later].into_iter().collect(), + "reconciling main never recreates the other worktree's missing pin" + ); + + let degraded = linked.run(&linked.root).await; + let ExplicitOutcome::Failed { error, .. } = degraded else { + panic!("the degraded worktree fails closed in its own pass, got {degraded:?}"); + }; + let ReconcileError::MissingRequiredPins { missing } = error else { + panic!("expected missing required pins, got {error:?}"); + }; + assert_eq!(missing.len(), 1); + assert_eq!(missing[0].0, shared); + assert!(GitSnapshotService::new(&linked.root) + .await + .expect("snapshot") + .list_pins(&WorktreeId(linked.id.clone())) + .await + .expect("pins") + .iter() + .all(|pin| pin.tree.0 != shared)); +} diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 83c9c3639..284e2d2bc 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -93,11 +93,11 @@ Facts the design relies on. T01 must re-read each cited location and either conf - One small JSON file per worktree at `/sce/ref-maintenance.json` (versioned, capped at `MAINTENANCE_STATE_MAX_BYTES`, 4 KiB), beside the lock and taint marker. Fields (all optional past-event unix-ms timestamps unless noted): `anchor`, `last_success`, `last_attempt`, `last_advised`; `last_attempt_outcome` (`completed` | `failed`); `last_report` (retained/deleted/local-required counts); `consecutive_failures` (integer); `last_failure` (short classification plus bounded message). - **No field is ever used as a deadline.** Eligibility is computed at read time as `now − reference ≥ window` (saturating at 0). Re-reading therefore cannot move a deadline. The stored-timestamp invariant is scoped to **successful state-writing operations**: after a successful write, every stored timestamp `t` satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`. It is not claimed after read-only or unsuccessful operations (see the two operation classes below). A value in `(now, now + FUTURE_SKEW_TOLERANCE]` is valid, is preserved byte-for-byte (not rewritten), and is evaluated as age 0; it can delay eligibility by at most the tolerance and cannot slide, because it is never refreshed by reads. - **Two operation classes.** - - *Read-only or unsuccessful operations* — plain `sce doctor`; a lock-busy advisory or explicit pass; a failed state read; a failed state write; a process interrupted before atomic replacement. These never mutate maintenance state (the previous bytes remain intact; a failed or interrupted write leaves the previous state), so they may observe preexisting invalid timestamps and cannot guarantee normalization. They must evaluate invalid timestamps conservatively (treated as eligible/recommended, never as age 0 and never as a reason to defer), must not postpone an advisory or explicit repair indefinitely, and must not report invalid state as successfully normalized. + - *Read-only or unsuccessful operations* — plain `sce doctor`; a lock-busy advisory or explicit pass; a failed state read; a failed state write; a process interrupted before atomic replacement. These never mutate maintenance state (a failed-before-rename write, `NotApplied`, or an interrupted write leaves the previous bytes intact; unreadable state is `StateUnavailable`, not a failed write), so they may observe preexisting invalid timestamps and cannot guarantee normalization. They must evaluate invalid timestamps conservatively (treated as eligible/recommended, never as age 0 and never as a reason to defer), must not postpone an advisory or explicit repair indefinitely, and must not report invalid state as successfully normalized. - *Successful state-writing operations* — a successful `advise_if_due` write and a successful `reconcile_explicit` state record. While holding `WorktreeLock` they normalize invalid timestamps and persist the normalized state atomically, so that every stored timestamp satisfies the tolerance after the successful write; valid near-future timestamps are preserved as specified above; repeated reads never move eligibility forward; subsequent invocations, including after a process restart, observe the persisted normalization. - Writes happen only under `WorktreeLock`, via atomic staging-and-swap (temp file in the same directory, then rename; pattern identified in T01) so a crash leaves either the old or the new file. No in-memory registry or process-global state. - Transitions: advisory — anchor and `last_advised` only. Explicit success — `last_attempt` = `last_success` = now, `last_report` set, `consecutive_failures` = 0, failure fields cleared. Explicit failure (DB unavailable/invalid, malformed ref, missing required pins, durable-root read error, delete-transaction error) — `last_attempt` = now, outcome `failed`, `consecutive_failures` += 1, `last_failure` set, `last_success` untouched. Lock contention (including explicit lock timeout) — **no state change**; never classified as a failure. - - **Invalid timestamp normalization (deterministic).** A timestamp is invalid when it is strictly more than `FUTURE_SKEW_TOLERANCE` (5 min) after the injected `now` (far-future values, or clock rollback beyond the tolerance); a timestamp exactly equal to `now + FUTURE_SKEW_TOLERANCE` is valid. A valid timestamp ahead of `now` (within the tolerance) is treated as age 0, so a small clock rollback never makes an advisory eligible by itself. Under `WorktreeLock`, `advise_if_due` and `reconcile_explicit` normalize once, as part of their successful state write: invalid `last_success`/`last_attempt`/`last_advised`/`anchor` values are cleared; if `last_success` and `anchor` are both unusable the state is **immediately eligible** for advice, so the advisory persists `last_advised = now` and returns `Advised` in the same locked operation, and a fresh `anchor = now` is written. The normalized state is persisted before returning, so a second read sees valid timestamps, performs no rewrite, and does not shift or postpone eligibility; once the injected clock advances, ordinary window-based eligibility resumes. If the normalizing write fails or the lock is busy, nothing is persisted, the invalid bytes stay as they were, and the in-memory evaluation still treats them conservatively (eligible), so a later successful operation can normalize them. Plain `sce doctor` never writes; it evaluates invalid timestamps as "recommended" with an "invalid maintenance timestamps" note and leaves the file byte-identical. `sce doctor --fix` runs regardless of timestamp validity and rewrites valid state afterwards. + - **Invalid timestamp normalization (deterministic).** A timestamp is invalid when it is strictly more than `FUTURE_SKEW_TOLERANCE` (5 min) after the injected `now` (far-future values, or clock rollback beyond the tolerance); a timestamp exactly equal to `now + FUTURE_SKEW_TOLERANCE` is valid. A valid timestamp ahead of `now` (within the tolerance) is treated as age 0, so a small clock rollback never makes an advisory eligible by itself. Under `WorktreeLock`, `advise_if_due` and `reconcile_explicit` normalize once, as part of their successful state write: invalid `last_success`/`last_attempt`/`last_advised`/`anchor` values are cleared; if `last_success` and `anchor` are both unusable the state is **immediately eligible** for advice, so the advisory persists `last_advised = now` and returns `Advised` in the same locked operation, and a fresh `anchor = now` is written. The normalized state is persisted before returning, so a second read sees valid timestamps, performs no rewrite, and does not shift or postpone eligibility; once the injected clock advances, ordinary window-based eligibility resumes. If the lock is busy or the normalizing write fails before the rename (`NotApplied`), nothing is persisted and the invalid bytes stay as they were; if it fails after the rename (`DurabilityUncertain`) the normalized state may already be visible but is never claimed durable. In every case the in-memory evaluation still treats invalid values conservatively (eligible), so a later invocation can normalize or re-read them safely. Plain `sce doctor` never writes; it evaluates invalid timestamps as "recommended" with an "invalid maintenance timestamps" note and leaves the file byte-identical. `sce doctor --fix` runs regardless of timestamp validity and rewrites valid state afterwards. - Recovery: a missing, unparsable, wrong-version, or oversized file is treated as default and rewritten by the next recorded operation; it never fails an explicit pass. - Result distinction for explicit passes (typed outcome `ExplicitOutcome`): `Completed(report)`; `CompletedStatePersistFailed { report, warning }` — the cleanup stands and is reported as done, with a typed `StatePersistWarning`; `Failed { error, state_warning: Option }` — a failed pass keeps both the reconciliation error and any persistence warning; `Skipped(Busy)`. Never merged. `StatePersistWarning` is `PreviousStateUnreadable(io::Error)` (the update was not attempted) or `WriteFailed(PersistFailure)`, where `PersistFailure::NotApplied` means the atomic replacement did not occur and the previous state is preserved, and `PersistFailure::DurabilityUncertain` means the replacement occurred (the new state may be visible) but crash durability could not be confirmed. The doctor facade preserves this as `StateWarning::{NotApplied, DurabilityUncertain, PreviousStateUnreadable}` with a stable kind, an optional phase, and a cause; it reports "Maintenance-state update was not applied." / "Maintenance-state durability could not be confirmed." and never claims that diagnostics were not recorded when they may have been. Advisory outcomes: `Anchored | NoAction | Advised | AdvisedDurabilityUncertain | Busy | StateWriteFailed | StateUnavailable`. - Time comes from an injected wall clock (`Fn() -> i64` unix ms) passed as a generic parameter for deterministic tests. @@ -140,10 +140,10 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - [ ] AC12: The hook completion path contains no unbounded maintenance work, and the documented concurrency contract for explicit reconciliation holds. - Validate: T05 S14, S15, S16 and S33 pass, as three distinct contention scenarios. S15 (direct advisory contention): with an explicit pass parked mid-inventory or mid-DB holding the lock, `advise_if_due` called directly returns `Busy` immediately and changes no state or refs. S16 (coordinator contention): with the lock held, `coordinate()` keeps its existing bounded lock acquisition; a short injected timeout fails with the existing lock-timeout error before durable completion, the advisory does not run after the failed coordinate, and the same boundary succeeds once the pass is released. S33 (post-coordinate handoff race): after a successful `coordinate()` released its lock, a competing explicit pass takes the lock before the paused advisory resumes; the advisory returns `Busy` and the committed boundary still returns success. The advisory takes no Git/DB parameters. No timeout, budget, or size cap is claimed to bound hook latency. - [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. - - Two guarantee classes (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, failed state read or write, write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. - - Validate: T05 S19–S25 and S34–S39 pass: single time-based advice per 24 h window with persistence before emission (a newly recorded failure may be advised inside that window, once, per AC5); concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's failure leg assert byte-identical preexisting state instead; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write leaving the previous state unchanged; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. + - Two guarantee classes plus a distinct uncertain-write outcome (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, state that cannot be read or evaluated (`StateUnavailable`, which is not a failed write), a write that fails before the atomic rename (`NotApplied`), a write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **`NotApplied`** means the failure happened before the rename: the previous state remains intact and byte-identical, the advisory reports `StateWriteFailed`, and no successful advisory is claimed. **`DurabilityUncertain`** means the failure happened after the rename: the new state may already be visible, crash durability is unconfirmed, the advisory reports `AdvisedDurabilityUncertain` when the recommendation was being recorded (an actionable diagnostic remains available), and a normalization written this way is never claimed to be durably persisted; a later invocation recovers safely in either case. Explicit reconciliation retains `CompletedStatePersistFailed` and the typed persistence-warning phase. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. + - Validate: T05 S19–S25 and S34–S39 pass: single time-based advice per 24 h window with persistence before emission (a newly recorded failure may be advised inside that window, once, per AC5); concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's pre-rename leg assert byte-identical preexisting state instead, and S36's post-rename leg asserts only that the new state may be visible with durability reported as uncertain; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write during normalization, with both a pre-rename leg (previous bytes unchanged, `StateWriteFailed`) and a post-rename leg (visibility may have changed, `AdvisedDurabilityUncertain`, no durable-normalization claim), a later invocation recovering safely in both; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. - [ ] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability (incidental WAL bookkeeping on existing databases is permitted and documented), and only the missing-DB case asserts no newly created database-related files or directories, including when the file is removed between a preliminary check and the open. Plain `sce doctor` shares the same non-creating read-only open. - - **Missing DB:** fails before Turso is invoked; no database file, parent directory, or WAL/shared-memory file is created; no Git ref is listed or deleted. + - **Missing DB:** the non-creating read-only Turso open fails without creating the DB file, its parent directory, or database sidecars (WAL/shared-memory); no Git ref is listed or deleted. A separate `sce doctor --fix` repair (global-state bootstrap) may create the parent directory independently of reconciliation; reconciliation itself creates nothing. - **Existing but invalid DB:** opening is permitted; incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable and documented; no schema migration, migration-metadata repair, repository-metadata initialization, or application-table INSERT/UPDATE/DELETE; no Git-ref deletion; verification fails with the matching typed error. Tests do not assert filesystem immutability for this case. - **Existing valid DB:** repository identity and schema readiness are verified before reconciliation; the existing multiprocess-WAL configuration is preserved; only the access reconciliation requires is performed; no initialization, repair, or migration. - Validate: T05 S28–S32 pass: missing DB (no new files or directories, non-creating open fails, refs not listed), incompatible schema, missing metadata, repository identity mismatch (refs untouched, logical DB contents byte-for-byte equivalent by table/row comparison, typed error), and normal existing-DB operation. @@ -226,7 +226,7 @@ Record actual outcomes in the validation report; compilation alone does not vali 9. Do not bootstrap, migrate, repair, or initialize an Agent Trace DB to run cleanup (D4). 10. Exactly one `WorktreeLock` acquisition per maintenance operation; never reacquire inside the lock-held implementation. 11. Maintenance failures never arm `ExternalTaintMarker`. - 12. The maintenance state file never stores a deadline; every successful state write leaves no timestamp more than `FUTURE_SKEW_TOLERANCE` ahead of the injected `now`; read-only and unsuccessful operations never mutate it; all writes are atomic and under `WorktreeLock`. + 12. The maintenance state file never stores a deadline; every successful state write leaves no timestamp more than `FUTURE_SKEW_TOLERANCE` ahead of the injected `now`; read-only and unsuccessful operations (including a `NotApplied` write failure) never mutate it; a post-rename `DurabilityUncertain` failure may leave the new state visible and is reported as uncertain, never as durably persisted; all writes are atomic and under `WorktreeLock`. 13. No schema migration, new attribution state, or new protocol action. 14. Follow repository rules: no code comments in touched code, Cargo only via Nix, `nix flake check` for verification. - **Non-goal (automatic reclamation):** see D2. Follow-up PR with the listed prerequisites, including the durable next-attempt reservation requirement. @@ -278,7 +278,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - **Test seams only as required:** a worker-entered seam threaded through `delete_pins` to `run_ref_mutation_inner`'s existing `on_worker_entered`; phase-boundary hooks (post-lock, post-DB-open, post-inventory) used to park a pass deterministically; injectable write-failure and rename-interruption hooks for the atomic writer; injected clock. Out — hook and doctor wiring, any new deletion code, any automatic/reservation/backoff scheduling, protocol/store schema changes, changes to the existing hook DB opener, capped inventory or budget logic. - Dependencies: T01 - - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a deadline; after every successful state write no stored timestamp exceeds `now + FUTURE_SKEW_TOLERANCE`; timestamps within the tolerance are accepted as age 0 and preserved; on a successful write invalid timestamps normalize once and persist; read-only, lock-busy, and failed-read/write operations never mutate state, evaluate invalid timestamps conservatively, and never postpone eligibility indefinitely; failures leave refs untouched; the DB opener fails a missing DB at its single non-creating read-only open without creating files or directories, and for an existing invalid DB performs no migration, repair, initialization, or application-table write (incidental WAL bookkeeping permitted and documented); no second deletion path exists. + - Done when: both entrypoints are callable from tests; the explicit pass and the advisory each complete without self-deadlock; state never contains a deadline; after every successful state write no stored timestamp exceeds `now + FUTURE_SKEW_TOLERANCE`; timestamps within the tolerance are accepted as age 0 and preserved; on a successful write invalid timestamps normalize once and persist; read-only, lock-busy, unreadable-state (`StateUnavailable`) and pre-rename-failed (`NotApplied`) operations never mutate state (a post-rename `DurabilityUncertain` failure may leave the new state visible and is reported as uncertain), evaluate invalid timestamps conservatively, and never postpone eligibility indefinitely; failures leave refs untouched; the DB opener fails a missing DB at its single non-creating read-only open without creating files or directories, and for an existing invalid DB performs no migration, repair, initialization, or application-table write (incidental WAL bookkeeping permitted and documented); no second deletion path exists. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml ref_reconciliation`; focused maintenance-state and `agent_trace_storage` unit tests (deterministic, no sleeps); `rg` confirms the only ref-deletion call is the existing `delete_pins`, each entrypoint acquires `WorktreeLock` once, `reconcile_with_held_lock` contains no acquire call, and the advisory module imports no Git/DB/store symbol. - Completed: 2026-10-09 - Files changed: `cli/src/services/agent_trace_db/repository.rs`, `cli/src/services/agent_trace_storage/mod.rs`, `cli/src/services/db/mod.rs`, `cli/src/services/mutation_trace/runtime/mod.rs`, `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs`, `cli/src/services/mutation_trace/runtime/maintenance_state.rs`, `cli/src/services/mutation_trace/runtime/ref_advisory.rs`, `cli/src/services/mutation_trace/runtime/ref_maintenance.rs`, `context/plans/mutation-cursor-ref-reconciliation-wiring.md` @@ -346,21 +346,67 @@ Record actual outcomes in the validation report; compilation alone does not vali - *Reconciliation core (explicit path):* (S1) `sce doctor --fix` reclaims an orphan pin left by an interrupted coordinate; (S2) `A → B → C → D` retains every historical tree; (S3) worktree A's pin is retained while worktree B needs the tree (**preservation**); (S4) missing required local pin fails closed with zero deletion; (S5) malformed/symbolic refs cause zero deletion; (S6) concurrent `coordinate()` and reconcile cannot race pin deletion — a coordinate attempt parked on the lock via barrier proceeds only after the pass releases it, and a pin it creates is never deleted by the in-flight pass; (S7) dropping the explicit-pass future while the deletion worker is parked at the worker-entered seam leaves the lock held until the worker is released, then free; (S8) idempotent reconciliation; (S9) plain `sce doctor` mutates neither refs nor state; `--fix` reports the true result for every outcome kind (completed, completed-with-persist-warning, failed kinds, busy); (S10) explicit pass against an external holder returns `Skipped(Busy)`, leaves state byte-identical, and does not count as a failure; (S11) explicit pass with DB failure leaves refs untouched, records a failure streak, never arms the external-taint marker; (S12) the explicit pass and the advisory each complete with one lock acquisition and no self-deadlock, and each returns busy (never deadlocks) against an externally held lock; (S13) with worktree B's required pin deliberately missing, reconciling A deletes only A's eligible pins, does not create or repair B's pin, does not fail, and never deletes any pin protecting B's tree. - *Hook path and latency (P1):* (S14) the advisory is invoked only after a durably completed `Close`/`Flush` (including `MarkerClearAfterCommit`), never from `Start`/`Advance`/`Abandon`/failed coordinates, and never reaches reconciliation, ref inventory, or DB open — its signature takes none, and spy/panic seams on the reconciler, inventory and opener are never called across every boundary and outcome; `coordinate()` keeps all of its existing snapshot and pinning semantics (required pins are still created and intact — protecting pinning and durable evidence is not weakened), and the advisory itself performs no Git-ref creation, update, or deletion: the test compares the ref set produced by a full hook run against an equivalent coordinate-only baseline (or injects a fake advisory seam and asserts the real advisory adds no ref delta), so any ref change from the coordinator is legitimate and is never attributed to maintenance; (S15) direct advisory contention (scenario A): an explicit pass parked at the post-inventory seam (and, in a second parameterization, at the post-DB-open seam) holds `WorktreeLock`; `advise_if_due` is invoked directly, returns `Busy` through a channel handshake without waiting for the pass, and leaves the state file byte-identical and the ref set unchanged; (S16) coordinator contention (scenario B): with the explicit pass parked holding the lock, a `coordinate()` Close/Flush with a short injected lock timeout fails with the existing lock-timeout error before durable completion and mutates nothing, the existing bounded-acquisition semantics are unchanged, the failed coordinate is a legitimate pre-commit failure, the advisory seam is never invoked after it, and once the pass is released the same boundary succeeds (documents the D1 concurrency contract); (S17) the advisory runs only after the coordinator's lock is released — a seam asserts the worktree lock is acquirable at invocation; (S18a) normal completion: every advisory outcome, a state-write failure, a corrupt state file, and a seam error leave the hook result and `Ok` status identical to the coordinate-only baseline; the marker state is snapshotted immediately before the advisory runs and asserted byte-identical (existence) afterwards, so the advisory never creates, clears, or modifies it, and the normal coordinator-cleared state is unchanged; (S18b) `MarkerClearAfterCommit`: reusing the coordinator's existing marker-clear failure fixture (no change to `coordinator.rs`, `ProtectedWorktree`, or recovery semantics; if unreachable from the hook layer, a fake coordinate seam returns that variant while a real armed marker file exists), the boundary is classified as durably completed, the hook returns the same successful empty-stdout result with the advisory allowed to run, the marker the failed clear left behind is asserted still armed and unmodified after the advisory, and a subsequent real `coordinate()` invocation observes the inherited taint and takes the existing conservative recovery path; (S33) post-coordinate handoff race (scenario C): `coordinate()` completes successfully and releases its lock; a seam pauses the hook immediately before the advisory call; a competing explicit reconciliation acquires `WorktreeLock` through a barrier; the advisory resumes and returns `Busy`; the already committed boundary still returns success, and the maintenance state and refs are untouched by the advisory. - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one time-based advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; a failure recorded after `last_advised` is advised once inside the window, then suppressed, then repeats on the normal cadence until a successful explicit pass clears it; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) clock-skew tolerance, with an injected clock for every case (`FUTURE_SKEW_TOLERANCE` = 5 min): (1) stored `last_success` equal to `now` — valid, age 0, no advice, file byte-identical; (2) one minute ahead of `now` — valid, age 0, preserved, no advice, no rewrite; (3) exactly five minutes ahead — valid (boundary is inclusive), age 0, preserved, no advice, no rewrite; (4) more than five minutes ahead (just beyond, and far-future) for `last_success`, `last_advised` and `anchor` — invalid, normalized once under the lock and persisted per D3 (invalid fields cleared; with `last_success` and `anchor` both unusable the advisory fires immediately once, persists `last_advised = now` and a fresh `anchor = now`); (5) clock rollback beyond tolerance (stored values valid at write time, injected `now` moved back more than five minutes) — treated as case 4; a rollback within tolerance is case 2/3 and never makes an advisory eligible by itself; (6) repeated reads after normalization — the file is byte-identical across repeated calls, no rewrite occurs, and eligibility is neither postponed nor slid; (7) process restart (fresh call re-reading the file) followed by clock advance past `RECONCILIATION_ADVISORY_AFTER` — advice recurs exactly once, proving eventual eligibility. After every **successful state-writing** operation an assertion checks the exact invariant `t ≤ now + FUTURE_SKEW_TOLERANCE` for every stored timestamp (not "no timestamp is in the future"); after read-only, lock-busy, or failed operations the assertion is instead that the state file is byte-identical to its prior content, with no normalization claimed; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next successful recorded operation (until then, read-only and failed operations leave them untouched and treat them as default state). - - *Timestamp-normalization regressions (P2; injected clocks and deterministic synchronization, no sleeps):* (S34) read-only doctor observing an invalid future timestamp — plain `sce doctor` reports "recommended" with the invalid-timestamp note, the state file is byte-identical afterwards, and it is not reported as normalized; (S35) lock-busy advisory — with the lock held through a barrier and an invalid future timestamp stored, `advise_if_due` returns `Busy` and the state file is byte-identical; the same holds for a lock-busy explicit pass (`Skipped(Busy)`); (S36) failed state write — an injected write error (and, in a second parameterization, a rename interrupted before replacement) during a normalizing operation leaves the previous state byte-identical and readable, emits no advice, reports no normalization (`StateUnavailable` for the advisory, `CompletedStatePersistFailed` for the explicit pass), and a later successful operation still normalizes it; (S37) successful normalization — a normalizing advisory and a normalizing explicit pass each persist state in which every timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`, valid near-future values (within tolerance) are preserved byte-for-byte, and a repeated read performs no rewrite and does not move eligibility forward; (S38) process restart after normalization — a fresh call re-reading the persisted file observes the normalized timestamps, performs no rewrite, and does not re-fire the immediate advice; (S39) eventual eligibility — with invalid timestamps unnormalized (S34/S35/S36 states) repeated read-only evaluations never push eligibility out, and after the injected clock advances past `RECONCILIATION_ADVISORY_AFTER` a successful advisory fires exactly once and persists normalized state. + - *Timestamp-normalization regressions (P2; injected clocks and deterministic synchronization, no sleeps):* (S34) read-only doctor observing an invalid future timestamp — plain `sce doctor` reports "recommended" with the invalid-timestamp note, the state file is byte-identical afterwards, and it is not reported as normalized; (S35) lock-busy advisory — with the lock held through a barrier and an invalid future timestamp stored, `advise_if_due` returns `Busy` and the state file is byte-identical; the same holds for a lock-busy explicit pass (`Skipped(Busy)`); (S36) failed state write during a normalizing operation, in two legs. *Pre-rename leg* (`NotApplied`, e.g. a rename failure): the previous state is byte-identical and readable, the advisory reports `StateWriteFailed` with no advice and no claimed normalization, the explicit pass reports `CompletedStatePersistFailed`, and a later successful operation still normalizes it. *Post-rename leg* (`DurabilityUncertain`, e.g. a parent-directory sync failure): the new state may already be visible, so byte identity is not asserted; the advisory reports `AdvisedDurabilityUncertain` with an actionable diagnostic, the normalization is not claimed durable, and a later invocation recovers safely (re-reads the visible state, performs no rewrite). `StateUnavailable` is reserved for state that cannot be read or evaluated and is not a write outcome; (S37) successful normalization — a normalizing advisory and a normalizing explicit pass each persist state in which every timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`, valid near-future values (within tolerance) are preserved byte-for-byte, and a repeated read performs no rewrite and does not move eligibility forward; (S38) process restart after normalization — a fresh call re-reading the persisted file observes the normalized timestamps, performs no rewrite, and does not re-fire the immediate advice; (S39) eventual eligibility — with invalid timestamps unnormalized (S34/S35/S36 states) repeated read-only evaluations never push eligibility out, and after the injected clock advances past `RECONCILIATION_ADVISORY_AFTER` a successful advisory fires exactly once and persists normalized state. - *Explicit recovery:* (S26) `sce doctor --fix` runs and reports its true result from each of: failed streak, stale advice, invalid timestamps, corrupt state, unwritable state (→ `CompletedStatePersistFailed`); (S27) explicit success resets the failure streak and sets `last_success`; explicit failure increments the streak without touching `last_success`; lock-busy never changes state. - *DB opening (D4):* (S28) missing DB — fails with the `Missing` typed error from the single non-creating read-only open (no existence pre-check); no database file, parent directory, WAL, or shared-memory file is created (directory listing before and after is identical, including when the parent directory is absent); Git refs are neither listed nor deleted (spy on `list_pins`/`delete_pins` never called); (S29) existing DB with incompatible schema — opening is permitted; fails with `IncompatibleSchema`; no migration or repair row written (migration table rows identical before and after); no ref deleted; (S30) existing DB with missing repository metadata — fails with `MissingMetadata`; no `repository_metadata` row inserted; (S31) existing DB with repository identity mismatch — fails with `RepositoryMismatch`; no row changed; (S32) existing valid DB — identity and schema readiness are verified before any ref listing (ordering asserted via seam), the multiprocess-WAL configuration is preserved, the pass completes, and no initialization, repair, or migration runs. For S29–S31 the assertions are logical: a before/after dump of every table's rows is equal (no migration, repair, initialization, or application-table INSERT/UPDATE/DELETE), refs are unchanged, and the external-taint marker is unarmed. Filesystem immutability is **not** asserted for existing-DB failure cases, because incidental Turso WAL/shared-memory bookkeeping is permitted when unavoidable; only S28 asserts that no database-related file or directory is created. Out — production behavior changes beyond test seams strictly required. - Dependencies: T03, T04 - Done when: all scenarios pass reliably, each fails when the guarded behavior is deliberately broken, no test depends on timing or sleeps, and every review finding in the traceability table maps to at least one passing scenario. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml` focused on the new tests, repeated runs for the race cases; zero-match filters are not accepted as passes; `rg "sleep"` over new test code returns no hits. - - Completed: 2026-10-09 - - Files changed: `cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs`, `context/plans/mutation-cursor-ref-reconciliation-wiring.md` - - Result: Added three focused production-path tests beside the T02 fixture and parameterized `seed_durable_root` by worktree: (S2/S3/S8/S13) repository-wide durable-root retention with a local root, another worktree's root, an unpinned foreign root and two orphans — only orphans deleted, the foreign missing pin neither repaired nor treated as a failure, and a second pass deletes nothing; (S5/S11) a symbolic ref in the namespace yields `MalformedPin`, deletes nothing and increments the failure streak; (S15/S12) `advise_if_due` returns `Busy` at both the post-DB-open and post-inventory phases of a live explicit pass, writes no state, and returns `NoAction` after the pass. No comments, sleeps or production code were added. - - Coverage mapping (deviation, per the lean-test preference): remaining matrix scenarios are already proven by earlier tasks and not duplicated — S1/S9/S10/S22/S26/S27/S28–S32 by `ref_maintenance/tests.rs` and `ref_doctor.rs` tests; S19–S25/S34–S39 by `maintenance_state.rs`/`ref_advisory.rs` tests; S14/S18a/S18b-trigger/S33-trigger/S4 hook policy by `mutation_scope::advisory_trigger_tests`; S7 by the `git_snapshot` cancelled-worker test (same `run_ref_mutation_inner` path used by `delete_pins`). Not added: S6, S16, a real-lock "acquirable at invocation" assertion for S17, and the S18b follow-up real `coordinate()` taint-recovery check; each needs coordinator-level fixtures beyond the minimal set. - - Verify: `test … ref_reconciliation_` 24 passed, repeated 3 times; `clippy --all-targets --all-features -D warnings` clean; `rg sleep` over the diff returns no hits; no added comment lines. - - Context impact: none (tests only). + - Reopened: 2026-10-09 (the earlier `done`/`synced` marking was unsupported: S6, S7, S16, S17, S18b and S33 were listed as "not added" or covered only through fake closures, helper tests or enum mappings). + - Files changed (reopening pass): `cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs` and new `ref_maintenance/tests/{support,coordination,worktrees,cross_process,fail_closed,state_recovery}.rs`; `cli/src/app/doctor_reconciliation_cli_tests.rs` (new, registered in `cli/src/app.rs`); `ref_advisory.rs` (tests); `git_snapshot.rs` (deletion-cancellation test); production test seams only — `coordinator.rs` (`coordinate_inner` takes `lock_timeout`; `coordinate` passes `WORKTREE_LOCK_TIMEOUT`, so production behaviour is unchanged), `protected_worktree.rs` (`acquire_with_timeout` is `pub(super)`), `git_snapshot.rs` (`delete_pins_inner` forwards `on_worker_entered`; the dead `run_ref_mutation` wrapper was removed), `hooks/mutation_scope.rs` (`drive_mutation_scope` is `pub(crate)`). No protocol, Quint, dispatch, runtime or timeout changes. + - Findings that shaped the tests: (1) a parked pass must run on its own thread and runtime — parking inside `tokio::spawn` blocks a worker and starves the process/IO driver the coordinator needs, which made the coordinator stall before it ever tried the lock; (2) `sce doctor --fix` runs a separate global-state repair that creates the Agent Trace DB parent directory before reconciliation, so the CLI-level missing-DB guarantee is "no DB file or sidecars", not "no parent directory"; (3) doctor failure kinds are `agent_trace_db_missing` and `agent_trace_db_repository_mismatch`, not a generic `agent_trace_db_unavailable`. + - Verification (2026-10-09): full CLI suite 493 passed, 0 failed; focused `ref_reconciliation` 50, `mutation_scope` 25, `doctor` 25, `maintenance` 83 (all non-zero); `clippy --all-targets --all-features -D warnings` clean; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect); `git diff --check` clean; 20 consecutive passes of every `ref_maintenance::tests::*` and cancellation test, 5 consecutive passes of the doctor CLI tests; no added comments, sleeps, `allow(dead_code)`, `dyn`, `BoxFuture`, detached tasks or extra production runtimes. + - Guard-removal checks actually performed: repository-wide roots replaced by worktree-local roots → S3 and S13 tests fail; deletion worker drops its lease → the new deletion-cancellation test and the existing pin-cancellation test fail; advisory ignores lock contention → S15 (both phases) and S33 fail. Not yet performed for the remaining production-path guards (S6, S16, S17, S18b, S21); until they are, "each fails when the guarded behavior is broken" is only demonstrated for the three guards above. + - Accepted gaps (user-approved closure, 2026-10-09): S9 persist-warning through the real CLI, S14 spy/panic seams, S26 through `sce doctor --fix`, and guard-removal checks for S6, S16, S17, S18b and S21 were not completed; the corresponding guarantees rest on the pass-level tests and structural signatures described above. + - Context impact: none (tests and test seams only). - Context synchronization: synced + - **Coverage map (S1–S39).** Status is `Verified`, `Partially verified` or `Not verified`; a related unit test never upgrades a production-path scenario. Test names are exact; paths are relative to `cli/src/services/mutation_trace/runtime/` unless they start with `app::` or `services::`. `rm::` = `ref_maintenance::tests::`. + + | Scenario | Exact test | Test level | Verified invariant | Status | + |---|---|---|---|---| + | S1 | `app::doctor_reconciliation_cli_tests::doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once`; `rm::ref_reconciliation_verified_database_reconciles_orphan_pin_and_records_success` | CLI end-to-end; pass-level | `sce doctor --fix` deletes an orphan SCE pin and records success (the orphan is created directly with `git update-ref`, not by an interrupted coordinate) | Verified | + | S2 | `rm::worktrees::ref_reconciliation_historical_roots_of_a_real_coordinator_sequence_stay_protected` | Production integration (real coordinator, real Git) | A→B→C→D from real `Start`/`Flush` boundaries: all four trees stay pinned although only D is the cursor; one orphan is reclaimed | Verified | + | S3 | `rm::worktrees::ref_reconciliation_pin_of_one_real_worktree_is_retained_while_another_needs_the_tree`; `rm::ref_reconciliation_retains_every_repository_wide_durable_root_and_never_repairs_other_worktrees` | Production integration (real linked worktree); synthetic roots | Main's pin of a tree that only the linked worktree durably requires is retained | Verified | + | S4 | `rm::fail_closed::ref_reconciliation_missing_local_pin_fails_closed_while_other_orphans_exist` | Pass-level | Missing local required pin returns `MissingRequiredPins` and deletes nothing, even with an eligible orphan present | Verified | + | S5 | `rm::ref_reconciliation_malformed_namespace_ref_deletes_nothing_and_records_failure` | Pass-level | Symbolic ref in the namespace yields `MalformedPin`, zero deletion, failure recorded | Verified | + | S6 | `rm::coordination::ref_reconciliation_real_coordinate_waits_for_a_pass_parked_after_db_open`; `…_after_pin_inventory` | Production integration (real `coordinate_inner`, real lock) | Coordinator hits contention, opens no DB and arms no marker while the pass holds the lock; the pin created after release survives; only the orphan is deleted; new pin precedes the durable-root commit; lock free afterwards | Verified | + | S7 | `services::mutation_trace::runtime::git_snapshot::tests::cancelled_delete_keeps_worktree_lock_until_git_ref_deletion_completes` | Cancellation integration (real `delete_pins`) | Dropping the caller and outer lock owner leaves the lock held by the parked worker; after release the pin is gone, no `.lock` files remain, a new pin proceeds. Existing `cancelled_pin_keeps_worktree_lock_until_git_ref_mutation_completes` covers the pin side | Verified | + | S8 | `rm::ref_reconciliation_retains_every_repository_wide_durable_root_and_never_repairs_other_worktrees` (second pass); CLI test second `--fix` run | Pass-level; CLI | Second pass deletes nothing | Verified | + | S9 | `app::doctor_reconciliation_cli_tests::*` (plain doctor, completed, failed ×2, busy); `services::doctor::reconciliation_fix_tests::*`; `services::doctor::render::reconciliation_render_tests::*` | CLI end-to-end for four outcome kinds; mapping unit tests for `completed_state_persist_failed` | Plain doctor mutates neither refs nor state; `--fix` reports the true result. The persist-warning outcome is not reachable through the CLI without a fault seam | Partially verified | + | S10 | `rm::ref_reconciliation_failure_streak_counts_but_lock_contention_never_does`; `rm::fail_closed::ref_reconciliation_second_explicit_pass_is_busy_and_records_exactly_one_pass`; CLI busy test | Pass-level; CLI | Lock contention returns `Skipped(Busy)`, leaves state byte-identical and is not a failure | Verified | + | S11 | `rm::fail_closed::ref_reconciliation_database_failure_never_arms_the_external_taint_marker`; `rm::ref_reconciliation_missing_database_creates_nothing_and_deletes_nothing` | Pass-level | DB failure leaves refs untouched, records a streak, never arms the marker | Verified | + | S12 | `rm::ref_reconciliation_lock_held_implementation_never_reacquires_the_lock`; `rm::ref_reconciliation_worktree_wrapper_acquires_once_and_releases`; `rm::ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock`; S6/S15 tests | Pass-level; production integration | Single acquisition per operation, shown by bounded completion under the existing self-contending lock (no acquisition counter) | Verified | + | S13 | `rm::worktrees::ref_reconciliation_missing_pin_in_another_real_worktree_neither_fails_nor_is_repaired` | Production integration (real linked worktree) | Reconciling main reclaims its orphan, keeps the pin protecting the linked tree, never recreates the linked worktree's missing pin, does not fail; the linked worktree's own pass fails closed | Verified | + | S14 | `services::hooks::mutation_scope::advisory_trigger_tests::{advisory_runs_once_after_completed_close_and_flush, advisory_never_runs_for_start_or_advance, advisory_never_runs_after_failed_coordinate, advisory_never_runs_for_abandon}`; ref-delta assertions in S17, S18a, S18b and S33 tests | Hook policy (fake coordinate); production integration for ref delta | Advisory fires only after a durably completed `Close`/`Flush`; the real advisory adds no ref delta. Spy/panic seams on reconciler, inventory and opener are not implemented | Partially verified | + | S15 | `rm::coordination::ref_reconciliation_independent_advisory_task_is_busy_while_a_pass_is_parked_after_db_open`; `…_after_pin_inventory`; earlier synchronous `rm::ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock` | Concurrency integration (independent blocking task) | Advisory returns `Busy` without waiting, state and refs unchanged, pass continues, later advisory follows the resulting state (`NoAction`, then `Advised` after the window). The synchronous test is retained but is not independent concurrency | Verified | + | S16 | `rm::coordination::ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Close`); `…_flush_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Flush`) | Production integration (real coordinator, injected 300 ms timeout) | Existing lock-timeout error; no durable transition, no pins, no marker, no advisory after the failure; the same boundary succeeds after release | Verified | + | S17 | `rm::coordination::ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock` | Hook/coordinator integration | Zero-wait `WorktreeLock` acquisition succeeds inside the advisory callback; the commit is already durable and unchanged afterwards; refs unchanged; empty stdout | Verified | + | S18a | `rm::coordination::ref_reconciliation_advisory_failures_after_a_real_commit_never_change_the_hook_result`; `advisory_trigger_tests::{diagnostic_write_failure_does_not_change_mutation_result, state_failures_log_warn_and_write_diagnostic_without_claiming_advice, quiet_outcomes_log_debug_and_emit_no_diagnostic}` | Hook/coordinator integration; hook policy | Write-failed, corrupt and unreadable-state advisories leave the hook result, committed state, refs and marker unchanged | Verified | + | S18b | `rm::coordination::ref_reconciliation_marker_clear_failure_after_commit_keeps_taint_for_the_next_coordinate` | Recovery integration (real coordinator; unlink fault injected inside the protected section) | Failed marker clear is a durable success; marker fingerprint identical across the advisory; after the fault is lifted the next real `coordinate()` takes the inherited-taint path (live scope `Abandoned`, health restored, revision advanced ≥2, marker cleared) whereas an untainted control flush left it `Active` | Verified | + | S19 | `ref_advisory::tests::maintenance_advisory_anchors_once_then_leaves_state_untouched` | Unit (advisory) | Anchor written once, second call byte-identical | Verified | + | S20 | `ref_advisory::tests::{maintenance_advisory_advises_once_per_window_and_persists_before_returning, maintenance_failure_advisory_emits_once_and_doctor_still_recommends_repair, maintenance_failure_advisory_repeats_after_the_cadence_window, maintenance_failure_after_recent_success_is_recommended_and_advised_despite_older_reminder}` | Unit (advisory) | One time-based advice per window, persisted first; failure advice once then cadence | Verified | + | S21 | `rm::cross_process::ref_reconciliation_advisory_lock_serializes_independent_processes`; `rm::fail_closed::ref_reconciliation_second_explicit_pass_is_busy_and_records_exactly_one_pass` | Independent OS processes (self re-exec, stdin/stdout rendezvous); in-process | Parked advisory in one process, `Busy` in a second, `NoAction` in a third; exactly one advice; no state write while parked. Explicit-vs-explicit is in-process only | Verified | + | S22 | `ref_advisory::tests::maintenance_advisory_anchor_write_failure_reports_state_write_failed_without_advice`; `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome` | Unit; pass-level | Persistence failure: no advice; explicit pass reports `CompletedStatePersistFailed` with deletions intact | Verified | + | S23 | `maintenance_state::tests::maintenance_state_recovers_after_interrupted_pre_rename_write`; `ref_advisory::tests::maintenance_advisory_rename_failure_preserves_state_and_next_call_can_advise` | Unit | Interrupted write preserves previous state; fresh call advises at the right time | Verified | + | S24 | `ref_advisory::tests::{maintenance_advisory_treats_timestamps_at_or_just_ahead_of_now_as_valid_age_zero, maintenance_advisory_preserves_timestamps_within_skew_tolerance, maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately, maintenance_advisory_clock_rollback_beyond_tolerance_normalizes_while_within_it_does_not}`; `rm::state_recovery::ref_reconciliation_invalid_last_advised_does_not_suppress_a_due_advisory`; `maintenance_advisory_advises_once_per_window_and_persists_before_returning` | Unit (injected clocks) | Cases 1–7: equal and near-future valid; exactly +5 min valid; beyond tolerance (anchor, last_success, last_advised) normalized once; rollback beyond vs within tolerance; byte-identical repeats; window recurrence | Verified | + | S25 | `ref_advisory::tests::{maintenance_advisory_anchors_once_then_leaves_state_untouched, maintenance_advisory_recovers_corrupt_state_by_anchoring, maintenance_advisory_recovers_wrong_version_and_oversized_state_by_anchoring}`; `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition` | Unit; pass-level | Missing, corrupt, wrong-version and oversized state recover without failure and are rewritten valid | Verified | + | S26 | `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition`; `rm::state_recovery::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`; `rm::ref_reconciliation_unreadable_previous_state_is_reported_without_overwriting`; `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome` | Pass-level | The explicit pass runs and reports its true result from failed streak, aged advice, invalid timestamps, corrupt/oversized state and unwritable state; only the completed/failed/busy cases are driven through `sce doctor --fix` | Partially verified | + | S27 | `rm::state_recovery::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`; `rm::ref_reconciliation_failure_streak_counts_but_lock_contention_never_does` | Pass-level | Success resets streak and sets `last_success`; failure increments without touching `last_success`; busy changes nothing | Verified | + | S28 | `rm::ref_reconciliation_missing_database_creates_nothing_and_deletes_nothing`; `rm::ref_reconciliation_verified_open_never_creates_missing_database_or_directories`; `rm::ref_reconciliation_verified_open_rejects_database_removed_before_open`; `app::doctor_reconciliation_cli_tests::doctor_fix_reports_missing_database_without_creating_it_or_deleting_refs` | Pass-level; opener; CLI | Typed `Missing`; no DB file, directory or sidecar created by the open; no refs listed or deleted. The CLI shows the parent directory comes from the separate global-state repair | Verified | + | S29 | `rm::state_recovery::ref_reconciliation_verification_failures_leave_the_logical_database_unchanged` (case `incompatible`); `rm::ref_reconciliation_missing_metadata_or_schema_fails_before_pin_inventory` | Pass-level | `IncompatibleSchema`; full-row dump of every table equal before and after; no pin inventory; refs and marker untouched | Verified | + | S30 | same test (case `missing_metadata`) | Pass-level | `MissingMetadata`; no `repository_metadata` row inserted (full-row dump equal) | Verified | + | S31 | same test (case `mismatch`); `app::doctor_reconciliation_cli_tests::doctor_fix_reports_repository_mismatch_without_touching_the_database_or_refs` | Pass-level; CLI | `RepositoryMismatch`; foreign metadata row intact; refs untouched | Verified | + | S32 | `rm::ref_reconciliation_verified_database_reconciles_orphan_pin_and_records_success` (phase order `DbOpened` → `PinsInventoried`); `rm::ref_reconciliation_verified_open_coexists_with_a_live_multiprocess_wal_writer`; `rm::ref_reconciliation_verified_open_is_read_only` | Pass-level; opener | Identity and schema verified before any ref listing; multiprocess-WAL coexistence; the opened handle rejects writes | Verified | + | S33 | `rm::coordination::ref_reconciliation_advisory_after_a_real_commit_is_busy_when_a_pass_wins_the_handoff` | Concurrency integration (real coordinator; hook parked between coordinate and advisory; pass on its own thread) | Pass takes the lock in the handoff window; advisory returns `Busy`; state and refs unchanged by the advisory; committed `Close` still reports success; only the orphan is deleted by the pass | Verified | + | S34 | `ref_doctor::tests::recommendation_flags_unusable_and_invalid_state_without_rewriting` | Unit (doctor inspection) | Read-only doctor flags invalid timestamps and leaves bytes untouched | Verified | + | S35 | `ref_advisory::tests::maintenance_advisory_reports_busy_and_leaves_invalid_state_byte_identical`; `rm::state_recovery::ref_reconciliation_busy_explicit_pass_leaves_invalid_state_byte_identical` | Unit; pass-level | Lock-busy advisory and explicit pass leave invalid state byte-identical | Verified | + | S36 | `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}`; `rm::state_recovery::ref_reconciliation_failed_state_write_during_normalization_has_two_distinct_legs` | Unit; pass-level | Pre-rename: previous bytes identical, `StateWriteFailed`/`NotApplied`. Post-rename: new state visible, `AdvisedDurabilityUncertain`/`DurabilityUncertain`, no durable-normalization claim; later invocation recovers | Verified | + | S37 | `ref_advisory::tests::maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately`; `rm::state_recovery::ref_reconciliation_successful_pass_normalizes_invalid_state_and_a_later_read_does_not_rewrite` | Unit; pass-level | Normalizing advisory and explicit pass persist `t ≤ now + tolerance`; near-future value preserved; repeated read does not rewrite | Verified | + | S38 | `ref_advisory::tests::maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately` (fresh call re-reads the file); `rm::cross_process::ref_reconciliation_advisory_lock_serializes_independent_processes` | Unit; independent processes | A fresh call observes normalized state, performs no rewrite and does not re-fire | Verified | + | S39 | `rm::state_recovery::ref_reconciliation_unnormalized_invalid_state_stays_eligible_until_a_successful_advisory` | Pass-level (injected clocks, held lock) | Repeated read-only evaluation and busy advisories never move eligibility or bytes; the next successful advisory fires exactly once and persists normalized state | Verified | + - [ ] T06: `Remove dead-code exemptions and synchronize context` (status:todo) - Task ID: T06 - Scope: In — remove declaration-level `#[allow(dead_code)]` from reconciliation declarations (`ref_reconciliation.rs`, `list_pins`/`delete_pins`/`PinnedRef`/`PinInventoryError` in `git_snapshot.rs`, durable-root readers in `store.rs`) that now have production consumers; separately review genuinely unreachable states (e.g. `SkippedNoCheckoutIdentity`, stale checkout-ID wording) and remove or justify each; update the context files in Context sync with the real async APIs, the wrapper/lock-held split and single-acquisition rule, the documented D1 concurrency contract, Git-derived worktree identities, the two maintenance entrypoints, the advisory trigger and why reconciliation is kept off the hook path, the maintenance state file (past-only fields, normalization, atomic writes), the distinct explicit outcomes, the verified-existing-DB opener contract, doctor reporting, and the corrected two-invariant durability statement (no claim that all repository-wide roots are pinned). Record in context, and not as implemented behavior: the deferred automatic out-of-band reclamation follow-up with its binding prerequisites from D2 (lifecycle-owned execution point, durable next-attempt reservation before expensive work, distinct outcomes, lock-hold analysis, timestamp normalization), the retired-worktree cleanup follow-up, and known storage limitations (historical retention, no repair of other worktrees' pins). Out — behavior changes. From c12c02ed3fae94de0354152a2f1bf38dece61d44 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 03:22:45 +0200 Subject: [PATCH 10/16] runtime: Remove stale ref-reconciliation exemptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove obsolete dead-code allowances now that snapshot-ref reconciliation runs through `sce doctor --fix`, keep only justified single-event loader exceptions, and make unreachable checkout-identity outcomes and test-only wrappers explicit. Synchronize the reconciliation, doctor, storage, and overview context with the single-lock maintenance entrypoint, verified-existing DB contract, advisory boundary, typed outcomes, and deferred cleanup limitations. Plan: `context/plans/mutation-cursor-ref-reconciliation-wiring.md` — T06 Co-authored-by: SCE --- .../mutation_trace/runtime/git_snapshot.rs | 65 --- .../runtime/maintenance_state.rs | 5 - .../mutation_trace/runtime/ref_maintenance.rs | 11 - .../runtime/ref_maintenance/tests.rs | 4 +- .../runtime/ref_reconciliation.rs | 37 +- cli/src/services/mutation_trace/store.rs | 77 +--- context/cli/agent-trace-storage.md | 1 + context/cli/async-persistence-guarantees.md | 2 +- context/cli/mutation-scope-runtime.md | 2 +- context/cli/mutation-trace-protocol.md | 2 +- .../mutation-trace-ref-maintenance-state.md | 81 ++++ .../cli/mutation-trace-ref-reconciliation.md | 409 +++++++++--------- .../cli/mutation-trace-runtime-coordinator.md | 49 ++- .../cli/mutation-trace-snapshot-service.md | 2 +- context/cli/mutation-trace-store.md | 2 +- context/context-map.md | 5 +- context/glossary.md | 2 + context/overview.md | 2 +- ...tation-cursor-ref-reconciliation-wiring.md | 9 +- context/sce/agent-trace-hook-doctor.md | 4 + context/sce/doctor-human-text-contract.md | 6 + 21 files changed, 359 insertions(+), 418 deletions(-) create mode 100644 context/cli/mutation-trace-ref-maintenance-state.md diff --git a/cli/src/services/mutation_trace/runtime/git_snapshot.rs b/cli/src/services/mutation_trace/runtime/git_snapshot.rs index 98261ccb5..0c64a8c9d 100644 --- a/cli/src/services/mutation_trace/runtime/git_snapshot.rs +++ b/cli/src/services/mutation_trace/runtime/git_snapshot.rs @@ -14,15 +14,6 @@ const SCE_RUNTIME_DIR: &str = "sce"; const TMP_INDEX_DIR: &str = "tmp"; const REF_NAMESPACE: &str = "refs/sce/mutation-cursor"; -/// `git for-each-ref` format for pin inventory: four `%00`-separated fields — -/// refname, target object name, target object type, and the symbolic-ref -/// target (empty for a direct ref). NUL-separated so no field can be split or -/// trimmed ambiguously; the trailing symref field is always present (possibly -/// empty), so every well-formed line has exactly four fields. -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] const FOR_EACH_REF_PIN_FORMAT: &str = "--format=%(refname)%00%(objectname)%00%(objecttype)%00%(symref)"; @@ -151,23 +142,6 @@ impl GitSnapshotService { Ok(String::from_utf8(output.stdout).ok()) } - /// Inventory every SCE snapshot pin owned by `worktree_id`. - /// - /// Runs `git for-each-ref` constrained to the single path prefix - /// `refs/sce/mutation-cursor//`, so a ref owned by any other - /// worktree or in an unrelated namespace is never returned. Each line is - /// validated against the shape `pin_tree` produces: a **direct** ref (never - /// a symbolic ref) whose target is a tree object and whose final path - /// component equals the target SHA. A symbolic ref anywhere in the - /// namespace is malformed state — it would let one worktree's pin resolve - /// through another worktree's ref — and is rejected rather than followed. A - /// `git for-each-ref` execution or exit failure is - /// [`PinInventoryError::Git`]; anything malformed inside the namespace is - /// [`PinInventoryError::MalformedRef`], matchable separately. - #[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" - )] pub async fn list_pins( &self, worktree_id: &WorktreeId, @@ -185,18 +159,10 @@ impl GitSnapshotService { .collect() } - #[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" - )] pub async fn delete_pins(&self, lease: WorktreeLockLease, pins: &[PinnedRef]) -> Result<()> { self.delete_pins_inner(lease, pins, || {}, || {}).await } - #[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" - )] async fn delete_pins_inner( &self, lease: WorktreeLockLease, @@ -255,18 +221,6 @@ impl GitSnapshotService { .map_err(|source| anyhow!("Git ref mutation worker failed: {source}"))? } - /// Fail closed unless every supplied pin is still exactly the direct ref - /// that was inventoried: present, a direct (non-symbolic) ref, targeting a - /// tree, and pointing at the recorded SHA. Re-inventoried in a single - /// `git for-each-ref` over the exact ref names, so no enumeration order is - /// relied on. This closes the common inventory→delete race cleanly; the - /// residual sub-transaction race is still contained by `--no-deref` plus - /// the per-`delete` old-value condition, which together cannot follow a - /// symbolic ref or mutate a ref the caller did not name. - #[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" - )] async fn assert_pins_are_unchanged_direct_refs(&self, pins: &[PinnedRef]) -> Result<()> { let mut args: Vec<&str> = vec!["for-each-ref", FOR_EACH_REF_PIN_FORMAT]; args.extend(pins.iter().map(|pin| pin.ref_name.as_str())); @@ -384,24 +338,13 @@ impl GitSnapshotService { } } -/// One SCE-owned snapshot pin: a ref under -/// `refs/sce/mutation-cursor//` and the tree object it protects. #[derive(Clone, Debug, Eq, PartialEq)] -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] pub struct PinnedRef { pub ref_name: String, pub tree: TreeId, } -/// Why a worktree's pin inventory could not be produced. #[derive(Debug)] -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] pub enum PinInventoryError { /// `git for-each-ref` itself failed to execute or exited non-zero. Git(anyhow::Error), @@ -426,10 +369,6 @@ impl std::fmt::Display for PinInventoryError { impl std::error::Error for PinInventoryError {} -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] fn parse_pin_line(line: &str, prefix: &str) -> std::result::Result { let fields: Vec<&str> = line.split('\0').collect(); let [ref_name, object_name, object_type, symref] = fields.as_slice() else { @@ -492,10 +431,6 @@ fn parse_pin_line(line: &str, prefix: &str) -> std::result::Result String { format!("{REF_NAMESPACE}/{}/", worktree_id.0) } diff --git a/cli/src/services/mutation_trace/runtime/maintenance_state.rs b/cli/src/services/mutation_trace/runtime/maintenance_state.rs index 0daed18ee..9e664c711 100644 --- a/cli/src/services/mutation_trace/runtime/maintenance_state.rs +++ b/cli/src/services/mutation_trace/runtime/maintenance_state.rs @@ -1,8 +1,3 @@ -#![allow( - dead_code, - reason = "maintenance entrypoints are wired by later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" -)] - use std::io::{Read, Write}; use std::path::{Path, PathBuf}; use std::time::{SystemTime, UNIX_EPOCH}; diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance.rs index 74c31bbe8..0c5ce8d92 100644 --- a/cli/src/services/mutation_trace/runtime/ref_maintenance.rs +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance.rs @@ -1,8 +1,3 @@ -#![allow( - dead_code, - reason = "maintenance entrypoints are wired by later tasks of context/plans/mutation-cursor-ref-reconciliation-wiring.md" -)] - use std::path::Path; use std::time::Duration; @@ -148,12 +143,6 @@ where Err(warning) => ExplicitOutcome::CompletedStatePersistFailed { report, warning }, } } - Ok(ReconciliationOutcome::SkippedNoCheckoutIdentity) => { - let error = ReconcileError::CheckoutIdentity(anyhow::anyhow!( - "no checkout identity could be derived" - )); - record_failed_pass(&previous, &write_state, &path, now_ms, error) - } Err(error) => record_failed_pass(&previous, &write_state, &path, now_ms, error), }; diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs index db544f4d8..9aa167512 100644 --- a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs @@ -570,9 +570,7 @@ async fn ref_reconciliation_lock_held_implementation_never_reacquires_the_lock() .expect("reconcile"); drop(lock); - let ReconciliationOutcome::Reconciled(report) = outcome else { - panic!("expected reconciled outcome"); - }; + let ReconciliationOutcome::Reconciled(report) = outcome; assert_eq!(report.deleted, 1); } diff --git a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs index 66363b37e..73a4583a0 100644 --- a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs +++ b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs @@ -1,5 +1,6 @@ use std::collections::BTreeSet; use std::path::Path; +#[cfg(test)] use std::time::Duration; use anyhow::Result; @@ -9,20 +10,17 @@ use crate::services::agent_trace_db::repository::RepositoryAgentTraceDb; use crate::services::mutation_trace::store::MutationTraceStore; use crate::services::mutation_trace::types::TreeId; -use super::git_snapshot::{resolve_git_dir, GitSnapshotService, PinInventoryError, PinnedRef}; -use super::worktree_lock::{acquire_inner_async, WorktreeLock, WorktreeLockError}; +#[cfg(test)] +use super::git_snapshot::resolve_git_dir; +use super::git_snapshot::{GitSnapshotService, PinInventoryError, PinnedRef}; +#[cfg(test)] +use super::worktree_lock::acquire_inner_async; +use super::worktree_lock::{WorktreeLock, WorktreeLockError}; -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] +#[cfg(test)] const RECONCILIATION_LOCK_TIMEOUT: Duration = Duration::from_secs(10); #[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] pub struct ReconciliationReport { pub local_required: usize, pub retained: usize, @@ -30,20 +28,11 @@ pub struct ReconciliationReport { } #[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] pub enum ReconciliationOutcome { Reconciled(ReconciliationReport), - SkippedNoCheckoutIdentity, } #[derive(Debug)] -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] pub enum ReconcileError { GitDir(anyhow::Error), Lock(WorktreeLockError), @@ -87,10 +76,7 @@ impl std::fmt::Display for ReconcileError { impl std::error::Error for ReconcileError {} -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] +#[cfg(test)] pub async fn reconcile_worktree

( repository_root: &Path, open_db: P, @@ -101,10 +87,7 @@ where reconcile_worktree_inner(repository_root, open_db, || {}).await } -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] +#[cfg(test)] pub(super) async fn reconcile_worktree_inner( repository_root: &Path, open_db: P, diff --git a/cli/src/services/mutation_trace/store.rs b/cli/src/services/mutation_trace/store.rs index cab0f7448..8bdace440 100644 --- a/cli/src/services/mutation_trace/store.rs +++ b/cli/src/services/mutation_trace/store.rs @@ -196,7 +196,7 @@ pub fn encode_boundary_kind(kind: BoundaryKind) -> &'static str { /// Decodes a [`BoundaryKind`] from `mutation_trace_events.boundary_kind`. #[allow( dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" + reason = "single-event loader has no production reader; attribution history uses load_mutation_event_page" )] pub fn decode_boundary_kind(value: &str) -> Result { match value { @@ -222,7 +222,7 @@ const SELECT_PROCESSED_EVENT_SQL: &str = "SELECT 1 FROM mutation_trace_processed_events WHERE scope_id = ?1 AND event_id = ?2"; #[allow( dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" + reason = "single-event loader has no production reader; attribution history uses load_mutation_event_page" )] const SELECT_MUTATION_EVENT_SQL: &str = "SELECT before_tree, after_tree, tainted, failure_kind, attribution_kind, attribution_scope_id, boundary_kind, boundary_scope_id, boundary_event_id @@ -246,31 +246,16 @@ const SELECT_LATEST_MUTATION_EVENT_REVISION_SQL: &str = "SELECT revision FROM mu LIMIT 1"; #[allow( dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" + reason = "single-event loader has no production reader; attribution history uses load_mutation_event_page" )] const SELECT_MUTATION_EVENT_ACTIVE_SCOPES_SQL: &str = "SELECT scope_id FROM mutation_trace_event_active_scopes WHERE worktree_id = ?1 AND revision = ?2"; -/// One worktree's complete durable tree root set — its cursor tree plus the -/// `before_tree` / `after_tree` of every historical `mutation_trace_events` -/// row — as a single `UNION` statement so the whole set is read from one -/// database snapshot, never assembled from independent `SELECT`s. -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] const SELECT_TREE_ROOTS_BY_WORKTREE_SQL: &str = "SELECT cursor_tree AS tree FROM mutation_trace_worktrees WHERE worktree_id = ?1 UNION SELECT before_tree AS tree FROM mutation_trace_events WHERE worktree_id = ?1 UNION SELECT after_tree AS tree FROM mutation_trace_events WHERE worktree_id = ?1"; -/// The same three `TreeId` columns unioned across **every** worktree in the -/// repository, in one statement / one snapshot — the reconciler's -/// repository-wide retention set. -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] const SELECT_ALL_TREE_ROOTS_SQL: &str = "SELECT cursor_tree AS tree FROM mutation_trace_worktrees UNION SELECT before_tree AS tree FROM mutation_trace_events @@ -710,7 +695,7 @@ impl<'a> MutationTraceStore<'a> { /// any hook-boundary path. #[allow( dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" + reason = "single-event loader has no production reader; attribution history uses load_mutation_event_page" )] pub async fn load_mutation_event( &self, @@ -818,31 +803,6 @@ impl<'a> MutationTraceStore<'a> { Ok(rows.into_iter().next()) } - /// Reads `worktree`'s complete durable tree root set: its - /// `mutation_trace_worktrees.cursor_tree`, plus the `before_tree` and - /// `after_tree` of every `mutation_trace_events` row for `worktree`, - /// deduplicated. Returns an empty set (not an error) when `worktree` has - /// no durable row at all. - /// - /// Read-only, cold path — never called from `load_worktree` or any - /// hook-boundary path, exactly like [`MutationTraceStore::load_mutation_event`]. - /// It reads only the three `TreeId` columns above: never - /// `mutation_trace_scopes` / `mutation_trace_processed_events` / - /// `mutation_trace_event_active_scopes`, never another worktree's trees, - /// and never transient `AttemptState` / `external_taint`. - /// - /// The whole set is produced by **one** SQL statement (a `UNION` of the - /// three columns) through **one** `query_map` call, so a concurrent - /// mutation-cursor commit — which atomically moves `cursor_tree` from `T` - /// to `X` and inserts `MutationEvent { before_tree = T, after_tree = X }` - /// in the same transaction — cannot expose a torn root set that omits `T`: - /// the single statement observes either the pre-commit snapshot - /// (`cursor_tree` still contains `T`) or the post-commit snapshot - /// (`before_tree` contains `T`). - #[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" - )] pub async fn load_tree_roots(&self, worktree: &WorktreeId) -> Result> { let rows = self .db @@ -856,23 +816,6 @@ impl<'a> MutationTraceStore<'a> { Ok(rows.into_iter().collect()) } - /// Reads the repository-wide durable tree root set: the union of - /// `mutation_trace_worktrees.cursor_tree`, `mutation_trace_events.before_tree`, - /// and `mutation_trace_events.after_tree` across **every** worktree, - /// deduplicated. Returns an empty set (not an error) for a repository with - /// no mutation-cursor rows. - /// - /// This is the reconciler's retention set: linked worktrees share one Git - /// object database, so a ref owned by worktree `A` may be the last SCE ref - /// protecting a tree that only worktree `B` durably requires. Read-only, - /// cold path, and — like [`MutationTraceStore::load_tree_roots`] — one SQL - /// statement through one `query_map` call, so it cannot tear across a - /// concurrent atomic `cursor T -> X` + `event T -> X` commit on another - /// worktree. - #[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" - )] pub async fn load_all_tree_roots(&self) -> Result> { let rows = self .db @@ -1019,7 +962,7 @@ impl<'a> MutationTraceStore<'a> { #[allow( dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" + reason = "single-event loader has no production reader; attribution history uses load_mutation_event_page" )] async fn load_mutation_event_active_scopes( &self, @@ -1190,10 +1133,6 @@ fn validate_health_encoding(tainted: bool, failure_kind: FailureKind) -> Result< Ok(()) } -#[allow( - dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" -)] fn tree_root_row_from_turso(row: &turso::Row) -> Result { let tree: String = row .get(0) @@ -1317,7 +1256,7 @@ fn mutation_event_page_row_from_turso(row: &turso::Row) -> Result Result { let before_tree: String = row @@ -1396,7 +1335,7 @@ fn reconstruct_attribution(kind: AttributionKind, scope_id: Option) -> R #[allow( dead_code, - reason = "ref reconciliation retained and unwired; see context/plans/mutation-cursor-ref-reconciliation.md" + reason = "single-event loader has no production reader; attribution history uses load_mutation_event_page" )] fn reconstruct_boundary( kind: BoundaryKind, diff --git a/context/cli/agent-trace-storage.md b/context/cli/agent-trace-storage.md index 509e1323d..bbfaf237e 100644 --- a/context/cli/agent-trace-storage.md +++ b/context/cli/agent-trace-storage.md @@ -16,6 +16,7 @@ Module at `cli/src/services/agent_trace_storage/` (T04 of the `repository-scoped 2. DB path from `default_paths::agent_trace_db_path_for_repository{,_at}`, which rejects empty or path-unsafe repository IDs (separators, `.`, `..`). 3. DB open splits by caller through `agent_trace_db::repository::RepositoryAgentTraceDb`, sharing steps 1–2 through an internal `open_storage_with` helper parameterized by the DB-opener; both resolver paths await async database operations: - Setup/lifecycle (`resolve_agent_trace_storage{,_at_state_root}`): a bounded concurrent-safe fast-path-then-migrate sequence — `open_without_migrations_at` + `ensure_schema_ready_for_hooks()` + `verify_or_initialize_repository_metadata(repository_id)`, falling back to migration-running `new_at` plus the same metadata validation. Directory creation rides on `TursoDb` parent-dir `create_dir_all`. If concurrent first open leaves all repository schema tables present but the one-file baseline migration record missing, the repository adapter records that missing metadata before rechecking readiness; otherwise the resolver retries the open/migrate sequence for a bounded window while SQLite/Turso locks clear. + - Maintenance (`resolve_existing_agent_trace_storage_for_maintenance` + `RepositoryAgentTraceDb::open_verified_existing_at`, used only by snapshot-ref reconciliation): a non-creating read-only Turso open (`TursoDb::open_existing_without_migrations_at`, multiprocess WAL preserved) that never creates the DB file or parent directory, never migrates, never calls the migration-metadata repair, and never runs `verify_or_initialize_repository_metadata`. It verifies schema readiness, metadata presence, `repository_id` equality, and a valid `source_instance_id` before returning, with typed `ExistingRepositoryDbError` (`Missing`, `Unreadable`, `IncompatibleSchema`, `MissingMetadata`, `RepositoryMismatch`). Plain `sce doctor` shares the open via `RepositoryAgentTraceDb::open_existing_schema_ready_at`. See [mutation-trace-ref-reconciliation.md](mutation-trace-ref-reconciliation.md). - Hook runtime (`resolve_agent_trace_storage_for_hook_runtime{,_at_state_root}`): `open_for_hooks_without_migrations_at` + `ensure_schema_ready_for_hooks()`, applying the same narrow concurrent-first-open metadata repair on failure but never falling back to `new_at`/migrations. A missing or baseline-only (pre-`002`) database fails with `ensure_schema_ready_for_hooks`'s existing `Run 'sce setup'.` guidance. Once readiness passes, `verify_or_initialize_repository_metadata(repository_id)` initializes `source_instance_id` when needed, matching setup's result for an already-set-up repository. `sce hooks` command call sites use this path via `open_agent_trace_db_for_hook_runtime` in `cli/src/services/hooks/mod.rs`. ## Never-touch on-disk boundary diff --git a/context/cli/async-persistence-guarantees.md b/context/cli/async-persistence-guarantees.md index 66fbfe97a..f3b42eeeb 100644 --- a/context/cli/async-persistence-guarantees.md +++ b/context/cli/async-persistence-guarantees.md @@ -62,6 +62,6 @@ Each statement points to a test, a type-level argument or a named limitation. - **PR1** (complete): application-owned multi-thread Tokio runtime and static async command dispatch. - **PR2**: async Turso core and the callers it required (credentials, token storage, lifecycle, hooks, mutation trace, sync/export, setup/doctor); the Git/process/filesystem async work and cancellation protections above; transaction isolation, credential-rotation, terminal-sync, adapter-boundary and FD-inheritance fixes; removal of the DB runtime bridge and its blocking scopes. -- **PR3** (implementation complete; final `/validate` pending): post-migration cleanup and runtime audit. Verified-dead code and blanket dead-code allowances were deleted, redundant `async` wrappers, `run_with_retry_sync` and the redundant `drop(storage)` were removed, `join_three_to_completion` was retained, the `recursion_limit = "256"` override was kept, and no temporary-index sweeper was added. The final unsuppressed audit on Linux reports 32 binary-target and 27 test-target diagnostics, all covered by declaration-level `#[allow(dead_code, reason = ...)]` exceptions: the `diff_traces` compatibility surface (cites the 2026-10-01 ADR), the retained-and-unwired ref-reconciliation cluster (cites `context/plans/mutation-cursor-ref-reconciliation.md`), cfg-gated `PlatformFamily` variants and `GuardError::UnsupportedPlatform`, and never-read `CoordinateOutcome`/`committed`/`completed` fields kept as suspected defects. Linux-only diagnostics do not prove cross-platform safety. Cosmetic and maintainability only. +- **PR3** (implementation complete; final `/validate` pending): post-migration cleanup and runtime audit. Verified-dead code and blanket dead-code allowances were deleted, redundant `async` wrappers, `run_with_retry_sync` and the redundant `drop(storage)` were removed, `join_three_to_completion` was retained, the `recursion_limit = "256"` override was kept, and no temporary-index sweeper was added. The final unsuppressed audit on Linux reports 32 binary-target and 27 test-target diagnostics, all covered by declaration-level `#[allow(dead_code, reason = ...)]` exceptions: the `diff_traces` compatibility surface (cites the 2026-10-01 ADR), the single-event mutation loaders with no production reader (the former ref-reconciliation exemptions were removed once `sce doctor --fix` wired the pass), cfg-gated `PlatformFamily` variants and `GuardError::UnsupportedPlatform`, and never-read `CoordinateOutcome`/`committed`/`completed` fields kept as suspected defects. Linux-only diagnostics do not prove cross-platform safety. Cosmetic and maintainability only. - **PR4**: OpenTelemetry (not implemented in PR2). diff --git a/context/cli/mutation-scope-runtime.md b/context/cli/mutation-scope-runtime.md index d07fbfdc8..d099e1384 100644 --- a/context/cli/mutation-scope-runtime.md +++ b/context/cli/mutation-scope-runtime.md @@ -33,7 +33,7 @@ The mechanics live in [`mutation-trace-runtime-coordinator.md`](mutation-trace-r `ExternalTaintOperation` crosses the boundary because it is part of `CoordinateError`'s public shape. It lives in `protected_worktree.rs` and is re-exported by `coordinator.rs` without making that module public. -Every runtime `mod` stays private, including `ProtectedWorktree`, its error, `WORKTREE_LOCK_TIMEOUT`, and `reconcile_worktree`. Adapters drive only the two entrypoints and never assemble the safety prefix. +Every runtime `mod` stays private, including `ProtectedWorktree`, its error, `WORKTREE_LOCK_TIMEOUT`, and `reconcile_explicit`. Adapters drive only the two entrypoints and never assemble the safety prefix. The re-exports are the intentional crate-visible seam consumed by the generic ingress; runtime internals stay private. The two re-export statements retain `#[allow(unused_imports)]` because no consumer names the completing types yet. diff --git a/context/cli/mutation-trace-protocol.md b/context/cli/mutation-trace-protocol.md index 9653c37e7..83e949fbf 100644 --- a/context/cli/mutation-trace-protocol.md +++ b/context/cli/mutation-trace-protocol.md @@ -172,7 +172,7 @@ owns the lock, the external-taint fence, the caller-supplied DB provider, one Git snapshot, and a bounded CAS-retry loop; `store.rs` never remaps an existing `ScopeId`'s `actor_kind`/`worktree_id`; `protocol.rs` assumes referenced scopes already exist and stays free of any Git object, DB row, or CAS transaction -concept. `runtime::ref_reconciliation` +concept. `runtime::ref_reconciliation` / `runtime::ref_maintenance` ([`mutation-trace-ref-reconciliation.md`](mutation-trace-ref-reconciliation.md)) is imperative durability maintenance *outside* the verified protocol — it never advances the cursor, chooses attribution, changes scope state, or creates a diff --git a/context/cli/mutation-trace-ref-maintenance-state.md b/context/cli/mutation-trace-ref-maintenance-state.md new file mode 100644 index 000000000..dd53b4067 --- /dev/null +++ b/context/cli/mutation-trace-ref-maintenance-state.md @@ -0,0 +1,81 @@ +# Snapshot-ref maintenance state and verified-existing DB opener + +Supporting contracts for [`mutation-trace-ref-reconciliation.md`](mutation-trace-ref-reconciliation.md): the durable per-worktree maintenance state file shared by the hook advisory, `sce doctor`, and `reconcile_explicit`, and the dedicated Agent Trace DB opener reconciliation uses. Related: [`mutation-scope-ref-advisory.md`](mutation-scope-ref-advisory.md), [`doctor-ref-reconciliation.md`](doctor-ref-reconciliation.md), [`agent-trace-storage.md`](agent-trace-storage.md). + +## Verified-existing Agent Trace DB opener + +Reconciliation never uses the hook opener +(`open_agent_trace_db_for_hook_runtime`), which can create directories and files, +repair migration metadata, and insert repository metadata. The dedicated path is +`resolve_existing_agent_trace_storage_for_maintenance` plus +`RepositoryAgentTraceDb::open_verified_existing_at`, built on +`TursoDb::open_existing_without_migrations_at`: + +- The Turso open is non-creating and read-only (`read_only(true)` with + `experimental_multiprocess_wal(true)` preserved), so existence and open are one + OS call. A failed open is classified `Missing` only when the path is then + absent. No pre-existence check and no parent-directory creation. +- It never runs migrations, `repair_missing_repository_schema_migration_metadata` + or `verify_or_initialize_repository_metadata`. +- Before any ref listing it verifies schema readiness (all expected migrations), + metadata presence, stored `repository_id` against the resolved identity, and a + valid `source_instance_id` (`verify_existing_repository_metadata`). +- Typed failures `ExistingRepositoryDbError`: `Missing`, `Unreadable`, + `IncompatibleSchema`, `MissingMetadata`, `RepositoryMismatch`. They stay + downcastable from `ReconcileError::AgentTraceDbUnavailable`. + +Per-DB-state guarantees: a **missing** DB creates no DB file, parent directory, +WAL or shared-memory file and no ref is listed or deleted; an **existing invalid** +DB may incur incidental Turso WAL/shared-memory bookkeeping but no migration, +repair, initialization, or application-table write and no ref deletion; an +**existing valid** DB is verified before reconciliation with no initialization. +Logical immutability is the guarantee, not filesystem byte-identity. + +Limitations: Turso opens `-wal`/`-tshm` sidecars by path, so only a coherent +replacement of main file and sidecars is meaningful; the process-wide Turso +registry may share an already-open instance (the read-only flag is then not +enforced); a DB replaced between verification and the delete transaction is not +detected, the guard being that roots are read from the verified connection under +the held lock. + +Plain `sce doctor` shares the non-creating read-only open through +`RepositoryAgentTraceDb::open_existing_schema_ready_at`, which inspects schema +readiness without requiring metadata. + +## Maintenance state file + +`/sce/ref-maintenance.json`, beside `mutation-cursor.lock` and the +taint marker. Versioned, capped at `MAINTENANCE_STATE_MAX_BYTES` (4096), written +only under `WorktreeLock` by atomic staging-and-swap (temp file in the same +directory, rename, best-effort parent-directory sync; the filesystem is the +statically dispatched `StateFilesystem` seam). No in-memory registry or +process-global state; the wall clock is an injected `Fn() -> i64` (unix ms). + +Fields: past-event timestamps `anchor`, `last_success`, `last_attempt`, +`last_advised`; `last_attempt_outcome` (`completed` | `failed`); `last_report` +counts; `consecutive_failures`; `last_failure`. **No field is a deadline.** +Eligibility is computed at read time as `now − reference ≥ window`, so rereading +cannot extend it. Constants: `RECONCILIATION_ADVISORY_AFTER` 24 h, +`FUTURE_SKEW_TOLERANCE` 5 min. + +Transitions: explicit success sets `last_attempt` = `last_success` = now, stores +the report, and clears the failure streak and fields; explicit failure sets +`last_attempt`, outcome `failed`, increments `consecutive_failures`, and leaves +`last_success` untouched; lock contention changes nothing. A missing, unparsable, +wrong-version, or oversized file is treated as default and rewritten by the next +recorded operation; it never fails an explicit pass. + +Timestamps: a value strictly more than `FUTURE_SKEW_TOLERANCE` ahead of `now` is +invalid; one within the tolerance (inclusive) is valid, preserved byte-for-byte, +and evaluated as age 0. **Successful state-writing operations** (a successful +advisory write, a successful explicit record) normalize invalid timestamps once +under the lock and persist them, so afterwards every stored timestamp satisfies +`t ≤ now + FUTURE_SKEW_TOLERANCE`. **Read-only or unsuccessful operations** +(plain `sce doctor`, lock-busy passes, unreadable state, a `NotApplied` write, an +interrupted write) never mutate the file, evaluate invalid timestamps +conservatively as eligible/recommended, and never report them as normalized. + +`evaluate_recommendation(state, now)` is the pure function shared by the advisory +and read-only doctor. An outstanding failed explicit pass is recommended +independently of staleness and of `last_advised`. + diff --git a/context/cli/mutation-trace-ref-reconciliation.md b/context/cli/mutation-trace-ref-reconciliation.md index 537c4a872..3d35b3bd2 100644 --- a/context/cli/mutation-trace-ref-reconciliation.md +++ b/context/cli/mutation-trace-ref-reconciliation.md @@ -1,151 +1,109 @@ -# Mutation-cursor snapshot-ref reconciliation (`runtime::ref_reconciliation`) - -`cli/src/services/mutation_trace/runtime/ref_reconciliation.rs` is a -conservative, per-worktree maintenance pass that removes orphaned / unreferenced -SCE-owned snapshot pins **within the ref namespace of a checkout identity a -current worktree still owns** while retaining every tree any current or -historical durable mutation-cursor state in the repository still references. -Built by the `mutation-cursor-ref-reconciliation` plan -(`context/plans/mutation-cursor-ref-reconciliation.md`). It is deliberately -**not** a guarantee that SCE snapshot refs never accumulate: a namespace whose -checkout identity no current worktree derives is beyond its reach (see -[Scope: an owned checkout-identity namespace only](#scope-an-owned-checkout-identity-namespace-only)). - -`GitSnapshotService::pin_tree` is **create-only per invocation** — a crash, -failed transition, or other interrupted `coordinate()` path can leave a -`refs/sce/mutation-cursor//` pin with no corresponding -durable root. Reconciliation is the reclamation step for exactly that state. It -is **not** a bound on storage growth: every retained `mutation_trace_events` -row keeps its `before_tree` / `after_tree` as durable roots, so a successful -`A → B → C → D` history keeps all four pins. Bounding historical snapshot -storage needs a separate future retention/compaction lifecycle this module does -not design. - -The design is deliberately asymmetric and this bias governs every rule below: -**keeping an unnecessary ref costs disk; deleting a required ref destroys -durable evidence.** False retention is acceptable; false deletion is not. - -`mod ref_reconciliation;` is private in `runtime/mod.rs` (like `coordinator`); -`reconcile_worktree` is `pub` only within `runtime` — no `pub(crate)` re-export, -no hook / command / `diff_traces` wiring yet (invocation timing is the -harness-wiring PR's). - -## Status: retained and unwired (PR3) - -The `cli-async-persistence-cleanup-pr3` plan chose **retain and defer**: this -pass and its helpers are kept, but nothing in production called them and PR3 -added no caller, hook, command or doctor wiring. `sce doctor --fix` is now the -explicit caller (see [`doctor-ref-reconciliation.md`](doctor-ref-reconciliation.md)); -no automatic caller exists. Historical consequence under PR3: pins created -by `pin_tree` in `coordinator.rs` are never deleted in production, so pins left -by crashed or failed `coordinate()` paths accumulate until removed by hand -(`git update-ref -d refs/sce/mutation-cursor//`). - -The reconciliation-only cluster (`ref_reconciliation.rs`; `list_pins`, -`delete_pins`, `PinnedRef`, `PinInventoryError` and their helpers in -`git_snapshot.rs`; the durable-root readers, SQL constants and row mappers in -`store.rs`) carries declaration-level `#[allow(dead_code, reason = ...)]` -naming `context/plans/mutation-cursor-ref-reconciliation.md` as the integration -path; the former module-wide allow on `mutation_trace` is gone. Wiring is -future work (see also the `ref-rec` branch); deleting the cluster to shrink -dead-code counts was rejected. - -## Entry point and identity - -```rust -pub fn reconcile_worktree( - repository_root: &Path, - open_db: impl FnOnce() -> anyhow::Result, -) -> Result -``` - -A one-line delegation to `pub(super) fn reconcile_worktree_inner(.., -on_lock_contention: impl FnOnce())` — the deterministic test seam (mirroring -`coordinate` / `coordinate_inner`), visible only within `runtime`. - -Like `coordinate()`, it never accepts a `WorktreeId`, `TreeId`, or ref name and -never opens the DB itself: identity is -`repository_root → resolve_git_dir → read_checkout_id`, `open_db` is -caller-supplied. `read_checkout_id → Ok(None)` returns -`Ok(SkippedNoCheckoutIdentity)` — an observable skip, **not** a zero-count -report: nothing inventoried, `open_db` never called, no ref touched, lock -released, no identity created or recovered; it makes **no** claim the repository -holds no SCE refs for a prior checkout identity (those are the unowned -namespaces below). `Err` is `ReconcileError::CheckoutIdentity` (a corrupt id, not an absent one). - -## Scope: an owned checkout-identity namespace only - -Reconciliation reclaims orphan / unreferenced pins **only** under -`refs/sce/mutation-cursor//` where `` is a checkout identity a **current -worktree still derives** (`resolve_git_dir → read_checkout_id`). The unsupported -case is identity-based, not path-based: - -```text -refs/sce/mutation-cursor//* — does a current worktree derive ? - yes → active namespace (reclaimed here under the WorktreeLock) - no → unowned namespace (no per-worktree pass can ever reach it) -``` - -High-frequency harness traffic against a worktree with a **stable** checkout -identity — where interrupted `coordinate()` runs leave orphan pins — is exactly -the active-namespace case this pass covers. - -A namespace becomes **unowned** whenever no current worktree can derive its -checkout id. Two lifecycles cause this: - -**Case A — deleted linked worktree.** `git worktree remove W` deletes W's -worktree-specific git dir, so its `/sce/checkout-id` is gone; the -`refs/sce/mutation-cursor//*` it created survive in the shared repository -ref namespace. - -**Case B — checkout-identity metadata loss / recreation.** A **present** -worktree's `/sce/checkout-id` disappears (id `A`). `reconcile_worktree` -returns `SkippedNoCheckoutIdentity` and does nothing. A later -`get_or_create_checkout_id` (e.g. the next `coordinate()`) mints a **new** id -`B`; the worktree now operates as `B` while `refs/.../A/*` are unowned. This is -**not** normal operation: it is a metadata-loss / recreation lifecycle the -maintenance model must handle conservatively — reconciliation never recreates -`A` or adopts its namespace. - -**Harness gate.** A persistent / current-worktree harness whose **checkout -identity stays stable** relies on this pass for active-namespace orphan cleanup -and is storage-cleanup complete for this module's scope. Any lifecycle that can -**retire, replace, lose, or recreate** checkout identities (ephemeral linked -worktrees among them, but not only those) can leave unowned namespaces and needs -the future repository-scoped operation below. This is a scope limit, not a bug, -orthogonal to the "reconciliation ≠ historical retention policy" boundary above. - -### Future work: repository-scoped unowned-namespace reconciliation - -Not implemented here — this module adds **no** repository-global namespace scan -and **no** repository-global ref deletion. Recorded shape: - -```text -enumerate refs/sce/mutation-cursor//* (git for-each-ref on the namespace) - ↓ -active checkout ids (enumerate current worktrees → read each one's checkout-id) - ↓ -unowned ids (namespace present, no current worktree derives it) - ↓ -for each unowned namespace, each pinned tree T: - T ∈ durable_roots(repository) → retain - T ∉ durable_roots(repository) → safe deletion candidate -``` +# Mutation-cursor snapshot-ref reconciliation (`runtime::ref_reconciliation`, `runtime::ref_maintenance`) + +Snapshot-ref reconciliation is a conservative, per-worktree maintenance pass that +removes orphaned SCE-owned snapshot pins under +`refs/sce/mutation-cursor//` while retaining every tree any current +or historical durable mutation-cursor state in the repository still references. +It is built by the `mutation-cursor-ref-reconciliation` plan and wired by the +`mutation-cursor-ref-reconciliation-wiring` plan +(`context/plans/mutation-cursor-ref-reconciliation-wiring.md`). + +`GitSnapshotService::pin_tree` is **create-only per invocation**: a crash, failed +transition, or other interrupted `coordinate()` path can leave a pin with no +corresponding durable root. Reconciliation is the reclamation step for exactly +that state. It is **not** a bound on storage growth: every retained +`mutation_trace_events` row keeps its `before_tree` / `after_tree` as durable +roots, so a successful `A → B → C → D` history keeps all four pins. Bounding +historical snapshot storage needs a separate retention/compaction lifecycle that +does not exist. + +The design is deliberately asymmetric: **keeping an unnecessary ref costs disk; +deleting a required ref destroys durable evidence.** False retention is +acceptable; false deletion is not. + +## Who runs it + +Reconciliation is **explicit only**. `sce doctor --fix` is the single production +caller (see [`doctor-ref-reconciliation.md`](doctor-ref-reconciliation.md)). +Orphan pins are reclaimed only when an operator runs it. **Automatic reclamation +is not delivered** and nothing may claim otherwise; it is a follow-up (see +[Deferred follow-ups](#deferred-follow-ups)). + +The mutation-hook completion path performs no reconciliation, ref listing, DB +I/O, or Git subprocess. After a durably completed `Close`/`Flush` it only runs a +lightweight advisory check that may record "reconciliation is recommended" in a +small state file (see [`mutation-scope-ref-advisory.md`](mutation-scope-ref-advisory.md)). +Inline reconciliation was rejected because `git for-each-ref`, DB open, +repository-wide durable-root queries, and a started `git update-ref` cannot be +bounded, the Tokio runtime joins in-flight blocking workers, and the pass holds +`WorktreeLock`, which a coordinator waits on for at most 10 s before failing the +boundary. + +## Module layout and entrypoints + +All runtime modules are private to `mutation_trace::runtime`; only the +`ref_doctor` facade and the advisory seam are re-exported `pub(crate)`. + +| Module | Role | +| --- | --- | +| `ref_reconciliation.rs` | The single reconciliation algorithm, `reconcile_with_held_lock` | +| `ref_maintenance.rs` | `reconcile_explicit`: the production maintenance entrypoint | +| `ref_advisory.rs` | `advise_if_due`: hook-path advisory | +| `maintenance_state.rs` | `/sce/ref-maintenance.json` state, normalization, atomic writes, `evaluate_recommendation` | +| `ref_doctor.rs` | Plain-data facade used by `sce doctor` | + +### Single lock acquisition + +`WorktreeLock` is a non-reentrant advisory file lock. The algorithm is split so +each maintenance operation acquires it **exactly once**: + +- `reconcile_with_held_lock` borrows the held `&WorktreeLock`, never acquires, + and derives deletion leases from `lock.lease()`. +- `reconcile_explicit(repository_root) -> ExplicitOutcome` acquires the lock once + with the bounded 10 s `EXPLICIT_LOCK_TIMEOUT`, resolves the repository identity + and canonical Agent Trace DB path itself, opens the DB only through the + verified-existing opener, runs `reconcile_with_held_lock`, records the result + in the state file while still holding the same lock, then drops it. It takes + no opener parameter; the injection seam is private to `ref_maintenance`. +- `advise_if_due` takes one zero-wait `try_lock`, has no Git/DB/store + parameters, and never reaches reconciliation. +- The lock-acquiring wrappers `reconcile_worktree` and `reconcile_worktree_inner` + (`acquire_inner_async` with `RECONCILIATION_LOCK_TIMEOUT`, then delegate) are + `#[cfg(test)]` only; production uses `reconcile_explicit`. + +Mutating Git work receives `lock.lease()` and moves it into the `spawn_blocking` +worker, so dropping the caller future cannot release the file lock while a ref +deletion is running. + +### Concurrency contract + +An explicit pass holds the lock for its whole duration, because the +pin-then-root ordering that makes deletion safe requires inventory, root reads, +and deletion not to interleave with `coordinate()`'s pin-then-CAS. A mutation +boundary arriving meanwhile waits up to `WORKTREE_LOCK_TIMEOUT` (10 s) and then +fails **before durable completion** with the existing lock-timeout error. This is +an operator-initiated trade-off of `sce doctor --fix`. The hook advisory never +waits on the lock; contention yields `Busy`. + +## Worktree identity + +Identity is Git-topology-derived (`resolve_worktree_id`): `main` for the primary +worktree, `worktrees/` for a linked worktree. Failure is +an `Err`. The former checkout-identity service was removed (see +[`checkout-identity.md`](checkout-identity.md)); there is no skip-on-missing-identity +outcome. `ReconcileError::CheckoutIdentity` remains the error variant for a +worktree identity that cannot be resolved. + +The dedicated DB opener and the `ref-maintenance.json` state file are documented in +[`mutation-trace-ref-maintenance-state.md`](mutation-trace-ref-maintenance-state.md). -It must inherit this module's guarantees: the repository-wide durability -invariant (`delete /T` only if `T ∉ durable_roots(repository)`, read -through `load_all_tree_roots()`) because an unowned namespace may still hold the -only SCE ref protecting historical `mutation_trace_events` trees other tooling -needs; the false-retention-over-false-deletion bias; and a hard prohibition on -the shortcut "checkout id is unowned → delete its whole namespace". It is a -separate PR (a repository-global scan and cross-worktree active-id inventory this -module omits), gated behind the same harness-wiring work as invocation timing. ## Two invariants -Conflating them — deciding deletion from the target worktree's roots alone — is -the cross-worktree safety bug this design avoids: linked worktrees share one -object database, so an `A`-owned ref can be the last SCE ref protecting a tree only `B` durably requires. +Conflating them, i.e. deciding deletion from the target worktree's roots alone, +is the cross-worktree safety bug this design avoids: linked worktrees share one +object database, so an `A`-owned ref can be the last SCE ref protecting a tree +only `B` durably requires. ```mermaid flowchart TD @@ -162,90 +120,131 @@ flowchart TD ds -- "retained" --> keep["keep the pin"] ``` -- **Local consistency** (`load_tree_roots(W)`) is strictly per-worktree. A - missing pin in some *other* worktree never makes `W`'s pass fail — that would - let one worktree's degradation block maintenance everywhere. -- **Deletion safety** (`load_all_tree_roots()`) is repository-wide. `W/T` is - deleted only when `T` is in **no** worktree's durable root set. If `B` durably - needs `T` and `A` also has a `T` pin, `A` retains it as accidental backup - reachability for `B`'s degraded state. +- **Local consistency** (`load_tree_roots(W)`) is per-worktree and checked before + any deletion. If it fails the pass deletes nothing and returns + `MissingRequiredPins`. A missing pin in some *other* worktree never makes `W`'s + pass fail. +- **Deletion safety** (`load_all_tree_roots()`) is repository-wide: `delete(W, T)` + only if `T` is in **no** worktree's durable root set. A pin another worktree + requires is retained. + +Claimed: these two invariants. **Not claimed:** that every repository-wide durable +root is pinned. Reconciliation does not repair, recreate, or verify missing pins +belonging to other worktrees. A DB `TreeId` is a *logical* durability requirement, not itself a Git reachability edge: it obliges reconciliation to keep at least one SCE ref -protecting that tree, and that retained ref supplies physical Git reachability. -Each root-set query is [one SQL statement over one DB -snapshot](mutation-trace-store.md), which keeps a concurrent atomic -`cursor T → X` + `event T → X` commit on another worktree from tearing the -repository-wide read — no repository-global lock is needed. - -## Locking - -Reconciliation holds the **same** `/sce/mutation-cursor.lock` -`WorktreeLock` that `coordinate()` holds across `pin → CAS → return`, acquired -via `worktree_lock::acquire_inner` and bounded by the module-owned -`RECONCILIATION_LOCK_TIMEOUT` (`Duration::from_secs(10)`, matching -`runtime::protected_worktree`'s private `WORKTREE_LOCK_TIMEOUT` by intent, not a -shared constant — there is no semantic reason the two must stay identical, so -each module owns its own). -Mutual exclusion on that one file makes the pin → DB-CAS race structurally -impossible: the reconciler's inventory → diff → delete runs wholly before -`coordinate()` takes the lock (nothing pinned yet) or wholly after it releases -it (tree committed → durable root → retained; never committed → true orphan → -deletable). The lock stays per-worktree; only the durable-root *read* is repository-wide. +protecting that tree. Each root-set query is [one SQL statement over one DB +snapshot](mutation-trace-store.md), so a concurrent atomic `cursor T → X` + +`event T → X` commit on another worktree cannot tear the repository-wide read. + +## Locking and the pin → CAS boundary + +The pass holds the **same** `/sce/mutation-cursor.lock` that +`coordinate()` holds across `pin → CAS → return`. Mutual exclusion on that one +file makes the pin → DB-CAS race structurally impossible: the inventory → diff → +delete runs wholly before `coordinate()` takes the lock (nothing pinned yet) or +wholly after it releases it (tree committed → durable root → retained; never +committed → true orphan → deletable). The lock stays per-worktree; only the +durable-root *read* is repository-wide. ## Algorithm and error contract -Every step runs under the lock, and every fallible step maps to one dedicated -`ReconcileError` variant — there is no `Other` catch-all: +Every step runs under the lock and every fallible step maps to one dedicated +`ReconcileError` variant; there is no catch-all: | Step | Error on failure | | --- | --- | | `resolve_git_dir` | `GitDir` | -| `acquire_inner(RECONCILIATION_LOCK_TIMEOUT)` | `Lock(WorktreeLockError)` | -| `read_checkout_id` → `Err` (corrupt id) | `CheckoutIdentity` | -| `open_db()` provider | `AgentTraceDbUnavailable` | +| lock acquisition (`EXPLICIT_LOCK_TIMEOUT`) | `Lock(WorktreeLockError)`; surfaced as `Skipped(Busy)` by `reconcile_explicit` | +| worktree identity resolution | `CheckoutIdentity` | +| verified-existing DB open | `AgentTraceDbUnavailable` (downcastable `ExistingRepositoryDbError`) | | `GitSnapshotService::new` | `SnapshotService` | | `list_pins` → `PinInventoryError::Git` | `PinInventory` | | `list_pins` → `PinInventoryError::MalformedRef` | `MalformedPin { ref_name, reason }`, delete nothing | -| `load_tree_roots` / `load_all_tree_roots` | `DurableRoots` | +| `load_tree_roots` / `load_all_tree_roots` | `DurableRoots` (durable-root queries belong to migration `004_mutation_trace_protocol`) | | a target-worktree root has no pin | `MissingRequiredPins { missing }`, delete nothing | | `delete_pins` transaction | `DeleteTransaction`, delete nothing (atomic-or-nothing) | `AgentTraceDbUnavailable` here is a **maintenance** error only: reconciliation never arms `ExternalTaintMarker`, calls `protocol::*`, or writes a `mutation_trace_*` row, and never becomes -`CoordinateError::AgentTraceDbUnavailable` — no mutation boundary is being -coordinated (contrast [`mutation-trace-external-taint.md`](mutation-trace-external-taint.md)). +`CoordinateError::AgentTraceDbUnavailable` (contrast +[`mutation-trace-external-taint.md`](mutation-trace-external-taint.md)). +`delete_pins` is the only ref-deletion path; there is no `git gc`, object +removal, or independent deletion. + +## Outcomes and report + +`reconcile_with_held_lock` returns `Result` +where `ReconciliationOutcome` has the single variant `Reconciled(ReconciliationReport)`. + +`ReconciliationReport { local_required, retained, deleted }`: +`local_required = load_tree_roots(W).len()`, `deleted` = stale pins removed, +`retained = actual.len() − deleted`. `retained == local_required` is not an +invariant; only `report.local_required ≤ report.retained` holds. -## Outcome and report +`reconcile_explicit` returns the typed, never-merged `ExplicitOutcome`: -Both entrypoints return `Result` — -`Reconciled(ReconciliationReport)` | `SkippedNoCheckoutIdentity`, the skip an -`Ok` (never `Err`) and distinct from a zero-work `Reconciled(.. { deleted: 0 })`. +- `Completed(report)` +- `CompletedStatePersistFailed { report, warning }`: the cleanup stands and is + reported as done +- `Failed { error, state_warning: Option }`: keeps both the + reconciliation error and any persistence warning +- `Skipped(Busy)`: lock contention; state untouched; not a failure -`ReconciliationReport { local_required: usize, retained: usize, deleted: usize }` -— `local_required = load_tree_roots(W).len()`, `deleted` = stale pins removed, -`retained = actual.len() − deleted`. `retained == local_required` is **not** an -invariant (a pin another worktree needs counts toward `retained` only); for -`Reconciled(report)` the only relation is `report.local_required ≤ report.retained`, -and `SkippedNoCheckoutIdentity` carries no report. +`StatePersistWarning` is `PreviousStateUnreadable(io::Error)` (update not +attempted) or `WriteFailed(PersistFailure)`. `PersistFailure::NotApplied` means +the atomic replacement did not occur and the previous state is preserved; +`PersistFailure::DurabilityUncertain` means the rename occurred (new state may be +visible) but crash durability could not be confirmed. A completed ref deletion is +never rolled back. ## Model boundary Ref reconciliation is imperative durability maintenance **below** the verified -`spec/mutation_cursor.qnt` protocol — it never advances the cursor, chooses -attribution, changes scope state, or creates a `MutationEvent`, so no Quint -change is warranted. It deletes only SCE-owned refs, never Git objects, and runs -no `git gc` / `git prune` / `git reflog expire`; Git reclaims unreachable -objects on its own schedule. - -## Testing boundary - -`ref_reconciliation.rs` has no inline tests and `runtime/tests.rs` no longer -exists: the earlier fixtures and `reconcile_worktree` integration and lock-race -suites were removed and are not restored while reconciliation is unwired. The -rules above are the specification the future wiring PR must re-prove with -focused tests. The pin helpers are still exercised by the `git_snapshot` pin -tests. - -See also: [`mutation-trace-runtime-coordinator.md`](mutation-trace-runtime-coordinator.md), [`mutation-trace-snapshot-service.md`](mutation-trace-snapshot-service.md) (`list_pins` / `delete_pins`), [`mutation-trace-store.md`](mutation-trace-store.md) (`load_tree_roots` / `load_all_tree_roots`), [`mutation-trace-protocol.md`](mutation-trace-protocol.md). +`spec/mutation_cursor.qnt` protocol. It never advances the cursor, chooses +attribution, changes scope state, or creates a `MutationEvent`, so no Quint or +`protocol.rs` change was made. It deletes only SCE-owned refs, never Git objects. + +## Testing + +Coverage lives in `runtime/ref_maintenance/tests.rs` and its submodules +(`support`, `coordination`, `worktrees`, `cross_process`, `fail_closed`, +`state_recovery`), the advisory and state unit tests, `git_snapshot` cancellation +tests, `hooks::mutation_scope::advisory_trigger_tests`, and +`app::doctor_reconciliation_cli_tests`. Race and ordering scenarios use +deterministic coordination (barriers, parked phase hooks via `ReconcilePhase`, +the worker-entered seam, injected clocks), not sleeps. The plan's coverage map +records which scenarios are verified and which are only partially verified. + +## Deferred follow-ups + +Not implemented and not to be described as behavior: + +- **Automatic out-of-band reclamation.** Binding prerequisites: (1) a + lifecycle-owned execution point with defined ownership, lifetime, scheduling, + and failure recovery (for example an explicitly invoked `sce` maintenance + command run by a user-managed scheduler, or a future shared session-end + mechanism), never a detached task, daemon, fire-and-forget subprocess, or + timeout-cancelled Git/DB work; (2) a durable next-attempt reservation persisted + under `WorktreeLock` **before** opening the DB, listing pins, or loading + durable roots, bounded, never interpreted as success, superseded by the success + cadence or exponential failure backoff, and bypassed by explicit repair, with + the automatic pass skipped if the reservation cannot be persisted; (3) + distinct outcomes for completed, completed-but-state-not-persisted, failed, and + skipped/deferred; (4) bounded or split lock-hold analysis against the 10 s + coordinator timeout; (5) the same invalid-timestamp normalization as above. +- **Retired-worktree cleanup.** The pass only inventories the namespace of a + currently resolvable worktree, so refs owned by removed linked worktrees + survive. A repository-scoped operation would need active-worktree inventory, + repository-wide durable-root retention (`delete /T` only if `T` is + outside `durable_roots(repository)`, never "namespace unowned → delete all"), + and protection against worktree removal/recreation races. +- **Evidence-based advice.** The advisory is time-based; detecting actual orphans + requires the Git/DB scan kept off the hook path. +- **Historical retention.** Trees referenced by retained mutation events are + never deleted; bounding history needs a separate event-retention and + snapshot-compaction policy. + +See also: [`mutation-trace-ref-maintenance-state.md`](mutation-trace-ref-maintenance-state.md), [`doctor-ref-reconciliation.md`](doctor-ref-reconciliation.md), [`mutation-scope-ref-advisory.md`](mutation-scope-ref-advisory.md), [`mutation-trace-runtime-coordinator.md`](mutation-trace-runtime-coordinator.md), [`mutation-trace-snapshot-service.md`](mutation-trace-snapshot-service.md) (`list_pins` / `delete_pins`), [`mutation-trace-store.md`](mutation-trace-store.md) (`load_tree_roots` / `load_all_tree_roots`), [`mutation-trace-protocol.md`](mutation-trace-protocol.md). diff --git a/context/cli/mutation-trace-runtime-coordinator.md b/context/cli/mutation-trace-runtime-coordinator.md index 02fe87a79..b0845aee5 100644 --- a/context/cli/mutation-trace-runtime-coordinator.md +++ b/context/cli/mutation-trace-runtime-coordinator.md @@ -13,8 +13,8 @@ submodule is declared privately in `runtime/mod.rs`, which re-exports `coordinate` and `abandon_scope` at `pub(crate)` — reachable crate-wide, contract in [`mutation-scope-runtime.md`](mutation-scope-runtime.md). `coordinate()` and `abandon_scope()` are now driven by the generic `sce hooks mutation-scope` CLI -ingress. `reconcile_worktree` stays `runtime`-internal and unwired, and the -mutation runtime does not itself insert into `diff_traces`. Concrete Claude Code, +ingress. Snapshot-ref reconciliation stays `runtime`-internal and runs only through +`sce doctor --fix`, and the mutation runtime does not itself insert into `diff_traces`. Concrete Claude Code, Codex, and OpenCode adapters are wired to the seam (OpenCode reachable in production via a generated plugin installed by `sce setup`); Pi remains future work. @@ -106,16 +106,16 @@ see Testing boundary). Only harness/command wiring remains. `refs/sce/mutation-cursor//...` ref. This derivation reads only Git's own state — no file is created or read under `/sce/`, and no identity is generated or persisted anywhere. -- `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs` — the - conservative per-worktree snapshot-ref maintenance pass — `reconcile_worktree` - / `pub(super) reconcile_worktree_inner` return `Result` (`ReconciliationOutcome` = `Reconciled(ReconciliationReport)` - | `SkippedNoCheckoutIdentity`). Under the worktree's `WorktreeLock` it deletes - only pins whose tree is a durable root of **no** worktree, fails closed if any - local root lacks a pin, and writes no `mutation_trace_*` row or taint marker - (only the namespace of a checkout id a current worktree still derives — a - namespace no current worktree owns, via a deleted worktree or checkout-id - metadata loss/recreation, is future repository-scoped work). Full contract in +- `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs` and + `ref_maintenance.rs` — the conservative per-worktree snapshot-ref maintenance + pass. `reconcile_with_held_lock` is the single lock-held algorithm and + `reconcile_explicit` (single `WorktreeLock` acquisition, verified-existing + Agent Trace DB, state recorded under the same lock) is its only production + entrypoint, called by `sce doctor --fix`. It deletes only pins whose tree is a + durable root of **no** worktree, fails closed if any local root lacks a pin, + and writes no `mutation_trace_*` row or taint marker. Only the namespace of the + currently resolvable worktree is inventoried; refs owned by removed linked + worktrees survive (deferred follow-up). Full contract in [`mutation-trace-ref-reconciliation.md`](mutation-trace-ref-reconciliation.md). - `cli/src/services/mutation_trace/runtime/protected_worktree.rs` — the shared safety prefix every runtime entrypoint runs behind (`ProtectedWorktree`: @@ -199,9 +199,12 @@ The runtime lock guards every runtime entrypoint's critical section (external-taint marker arming/clearing, snapshot capture where applicable, worktree/scope materialization, recovery, and the CAS retry loop): `coordinate()` and `abandon_scope()` acquire it before their protected work and -hold it until the call returns. The separate `ref_reconciliation::reconcile_worktree` -pass acquires this same lock before inventorying pins, reading durable roots, -or deleting refs, with its own bounded timeout. `/sce/mutation-cursor.lock` +hold it until the call returns. The separate `ref_maintenance::reconcile_explicit` +pass acquires this same lock exactly once, before inventorying pins, reading +durable roots, or deleting refs, with a bounded 10 s wait; a boundary arriving +meanwhile waits up to `WORKTREE_LOCK_TIMEOUT` and then fails before durable +completion. The hook-path advisory takes the lock with a zero-wait `try_lock` +only after `coordinate()` has returned. `/sce/mutation-cursor.lock` remains worktree-specific because `git_dir` itself is worktree-specific for linked worktrees (`resolve_git_dir` resolves each worktree's own `--absolute-git-dir`), so each worktree has an independent critical section. @@ -217,7 +220,7 @@ On-disk layout so far: (runtime::git_snapshot writes here directly) -└── refs/sce/mutation-cursor// (runtime::git_snapshot, create-only per invocation; orphan/unreferenced pins reclaimed by runtime::ref_reconciliation only for a checkout id a current worktree still derives, every pin for a current or historical durable mutation-cursor root retained; a namespace no current worktree owns — deleted worktree or checkout-id metadata loss/recreation — is unreachable, future repository-scoped work) +└── refs/sce/mutation-cursor// (runtime::git_snapshot, create-only per invocation; orphan/unreferenced pins reclaimed by runtime::ref_maintenance only through sce doctor --fix and only for the namespace of a currently resolvable worktree, every pin for a current or historical durable mutation-cursor root retained; refs owned by removed linked worktrees are unreachable, future repository-scoped work) ``` ## Testing boundary @@ -291,13 +294,13 @@ attributable `Advance` that commits then fails its trailing `marker.clear()` surfaces `MarkerClearAfterCommit` with the matching committed outcome. The The `after_load` seam was used by the removed reconciliation pin→CAS lock-race regression and is unused while that suite is gone. -There is no `runtime/tests.rs` cross-module integration suite: the ordinary -coordinator, abandonment and reconciliation test suites were removed during the -async-persistence work and are deliberately not restored (PR3 does not restore -tests). Coverage of `coordinate()` / `abandon_scope()` now rests on the Quint -MBT suite in `mutation_trace/mbt`, the `mutation_trace` unit tests and the -inline `worktree_lock.rs` / `git_snapshot.rs` tests; `reconcile_worktree` has no -test coverage while it remains unwired. +There is no `runtime/tests.rs` cross-module integration suite. Coverage of +`coordinate()` / `abandon_scope()` rests on the Quint MBT suite in +`mutation_trace/mbt`, the `mutation_trace` unit tests and the inline +`worktree_lock.rs` / `git_snapshot.rs` tests. Reconciliation has its own +production-path and concurrency suite under `ref_maintenance/tests/` (see +[`mutation-trace-ref-reconciliation.md`](mutation-trace-ref-reconciliation.md)), +including tests that drive the real coordinator against a parked pass. ## Status diff --git a/context/cli/mutation-trace-snapshot-service.md b/context/cli/mutation-trace-snapshot-service.md index 24e220938..e14684a31 100644 --- a/context/cli/mutation-trace-snapshot-service.md +++ b/context/cli/mutation-trace-snapshot-service.md @@ -120,7 +120,7 @@ transition and reads baseline file content through its own `TreeReadSource` trait (see [`mutation-trace-agent-attribution.md`](mutation-trace-agent-attribution.md)). `list_pins` / `delete_pins` are consumed only by the per-worktree -ref-reconciliation maintenance pass, which is retained but unwired (PR3 retain-and-defer): no production code deletes pins, so pins orphaned by crashed or failed `coordinate()` paths accumulate until removed by hand. Their declarations carry `#[allow(dead_code, reason = ...)]` pointing at `context/plans/mutation-cursor-ref-reconciliation.md`. See [`mutation-trace-ref-reconciliation.md`](mutation-trace-ref-reconciliation.md). +ref-reconciliation maintenance pass, which runs only through `sce doctor --fix`: pins orphaned by crashed or failed `coordinate()` paths are reclaimed when an operator runs it, and automatic reclamation does not exist. `delete_pins` is the only ref-deletion path and moves the `WorktreeLockLease` into its blocking worker, so dropping the caller future cannot release the lock while `git update-ref` runs. See [`mutation-trace-ref-reconciliation.md`](mutation-trace-ref-reconciliation.md). ## Testing boundary diff --git a/context/cli/mutation-trace-store.md b/context/cli/mutation-trace-store.md index 52b26de42..5f19aea2e 100644 --- a/context/cli/mutation-trace-store.md +++ b/context/cli/mutation-trace-store.md @@ -136,7 +136,7 @@ worktree is still an `Err` there. cold-path, read-only queries — siblings of `load_mutation_event`, never reached from `load_worktree` or a hook-boundary path — that expose the set of Git tree SHAs the mutation-cursor protocol still durably depends on, for the -per-worktree ref-reconciliation pass, which is retained but unwired (PR3 retain-and-defer; these readers, `load_mutation_event` and their SQL/row-mapping helpers carry declaration-level `dead_code` allows citing `context/plans/mutation-cursor-ref-reconciliation.md`; see +per-worktree ref-reconciliation pass, which runs only through `sce doctor --fix` (the single-event loaders `load_mutation_event` / `load_mutation_event_active_scopes` and their SQL/row-mapping helpers have no production reader and carry declaration-level `dead_code` allows saying so; attribution history uses `load_mutation_event_page`; see [`mutation-trace-runtime-coordinator.md`](mutation-trace-runtime-coordinator.md)). - `load_tree_roots(worktree) -> BTreeSet` returns one worktree's roots: diff --git a/context/context-map.md b/context/context-map.md index d6443fe0e..b971f89ea 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -36,12 +36,13 @@ Feature/domain context: - `context/cli/mutation-trace-runtime-coordinator.md` (imperative-shell runtime layer in `cli/src/services/mutation_trace/runtime/`, built by the `mutation-cursor-runtime-coordinator` plan: the per-worktree OS advisory lock, `runtime::worktree_lock::WorktreeLock::acquire_async(git_dir, timeout)` at `/sce/mutation-cursor.lock`, bounded `try_lock()` polling with a distinct matchable timeout error and RAII release, and its distinction from the separate checkout-identity-creation lock; the isolated Git snapshot service `runtime::git_snapshot::GitSnapshotService` (documented in `mutation-trace-snapshot-service.md`); and `runtime::coordinator`'s protocol-integration pipeline (`RuntimeBoundary`/`CoordinateOutcome`/`CoordinateError`, the load → recover-if-needed → prepare/commit CAS-retry loop, and its own bounded snapshot-failure taint-retry loop) plus the public `coordinate(repository_root, boundary, open_db)` entrypoint that runs the shared `runtime::protected_worktree` prefix (`WorktreeLock` -> external-taint write-ahead fence -> `WorktreeId`, extracted so a second entrypoint cannot drift from it and documented in `mutation-trace-protected-worktree.md`), invokes the caller-supplied `open_db` provider so DB acquisition falls inside that fence, drives the pipeline under one held lock, and clears the marker through `ProtectedWorktree::complete()` only on success; an inherited external-taint marker is overlaid onto `protocol::database_failure` recovery on the next invocation, against the single captured snapshot and re-injected across a losing recovery CAS; the private `runtime::ref_reconciliation` per-worktree snapshot-ref maintenance pass (documented in `mutation-trace-ref-reconciliation.md`) shares the same `WorktreeLock`; `coordinate()` and `abandon_scope()` are now `pub(crate)` re-exported from `runtime/mod.rs` (documented in `mutation-scope-runtime.md`) while the runtime submodules themselves stay private; a generic `sce hooks mutation-scope` CLI ingress now drives both entrypoints, joined by current Claude Code and Codex adapter drivers (registered by `sce setup`/`sce setup --codex` and reachable) and an OpenCode adapter driver (lifecycle + recovery wired, and now generated as the final `sce-mutation-scope.ts` plugin and installed by `sce setup`), documented in `mutation-scope-runtime.md`) - `context/cli/mutation-trace-protected-worktree.md` (the shared safety prefix every mutation-cursor runtime entrypoint runs behind, in `cli/src/services/mutation_trace/runtime/protected_worktree.rs`, extracted from `coordinate()` by the `mutation-scope-runtime-integration` plan so a second entrypoint cannot drift from it: `ProtectedWorktree::acquire(repository_root)` running the safety-critical fixed order resolve `git_dir` → `WorktreeLock` (module-owned 10s `WORKTREE_LOCK_TIMEOUT`) → `ExternalTaintMarker::exists()` → `persist()` (fence armed write-ahead of every fallible step that follows, including DB acquisition) → `get_or_create_checkout_id` as `WorktreeId`; the `worktree_id()` / `inherited_external_taint()` / consuming `complete()` surface, where `complete()` clears the marker under the still-held lock and is the only thing that ever clears it while `Drop` releases only the lock; and the one-variant-per-prefix-step `ProtectedWorktreeError` (`GitDirResolution` | `LockAcquisition` | `ExternalTaintMarker { operation, source }` | `CheckoutIdentity`) each entrypoint maps onto its own error surface — `coordinate()` onto exactly the `CoordinateError` variants that step produced before the extraction) - `context/cli/mutation-trace-scope-abandonment.md` (the mutation-cursor runtime's second protected entrypoint in `cli/src/services/mutation_trace/runtime/scope_runtime.rs`, built by the `mutation-scope-runtime-integration` plan and the first production call site for `protocol::abandon`: `abandon_scope(repository_root, scope, open_db) -> Result` retires a scope whose final worktree boundary was never observed, sharing `coordinate()`'s `ProtectedWorktree` prefix but deliberately capturing **no** Git snapshot, pin, diff, reconciliation, scope registration, or worktree initialization — so abandonment is not a `RuntimeBoundary` and needs no Quint change; the classify-before-transition order that recovers what `protocol::abandon`'s uniform guarded no-op cannot report (`load_scope` first for the missing-row and cross-worktree cases the projection seam treats as errors, then `load_worktree` for the `NeverSeen` / terminal / `Active` split); the `Abandoned` / `AlreadyTerminal` / `RecoveryRequired` outcomes and the `InheritedExternalTaint` | `MissingScope` | `NeverSeenScope` | `MissingWorktreeState` recovery reasons; the inherited-marker short-circuit that returns before the DB provider is ever invoked; the fence-completion rule that clears the marker only for a settled abandonment or proven-terminal no-op and leaves it armed for every recovery-required outcome and every error, with `MarkerClearAfterCompletion` carrying the already-settled outcome; the CAS retry bounded by the coordinator's shared `MAX_CAS_RETRY_ATTEMPTS`, settling on a competitor's terminal status rather than overwriting it; and the deliberate false-negative tradeoff whereby a missing or `NeverSeen` target forces conservative strong recovery that may abandon unrelated live scopes, because attribution safety outranks preserving potentially valid evidence) -- `context/cli/mutation-scope-runtime.md` (the crate-visible mutation-trace runtime seam and the lifecycle contract every current or future harness adapter must uphold, recorded by the `mutation-scope-runtime-integration` plan: the ten `pub(crate)` re-exports in `runtime/mod.rs` (`coordinate`, `RuntimeBoundary`, `StartProvenance`, `CoordinateOutcome`, `CoordinateError`, `ExternalTaintOperation`, `abandon_scope`, `AbandonScopeOutcome`, `AbandonRecoveryReason`, `AbandonScopeError`), with `ExternalTaintOperation` riding through `coordinator.rs`'s own `pub use` because `CoordinateError::ExternalTaintMarker` carries it — so the type becomes crate-visible without `protected_worktree` becoming a public module — while every `mod` declaration stays private and `ProtectedWorktree`/`ProtectedWorktreeError`/`WORKTREE_LOCK_TIMEOUT`/`reconcile_worktree` stay internal, kept clippy-clean by `#[allow(unused_imports)]` on the re-exports per the `services/style.rs` precedent rather than a placeholder consumer; and the adapter obligations themselves — a scope is one independently mutation-capable execution so a concurrent main agent and subagent need distinct `ScopeId`s, `Start`/`Advance`/`Close` semantics including that only `Start` may carry the optional `StartProvenance`, registered after scope registration and before the protocol commits and never inside `ProtocolState` or the CAS transition, with that registration conditional on the `ScopeState` `register_scope` returns so a provenance row may only be created while the scope is `NeverSeen` (an admission-time snapshot that a post-admission replay cannot backfill) while an existing row is still validated on every provenance-carrying `Start`, that a failed tool still requires `Advance` (the boundary is an observation, not a successful edit) and that a `ScopeId` is never reused after a terminal status (the `NeverSeen` guard silently refuses to reactivate it), `abandon_scope()` requiring positive staleness evidence and never inferring it from `ActorKind`, the `abandon` → `coordinate(Start(successor))` sequence with what each outcome implies for it including that a failed abandonment must never be treated as a safely started successor, that abandonment is not a `RuntimeBoundary` and needs no Quint change, the D1 tradeoff whereby a missing or `NeverSeen` target deliberately forces conservative strong recovery that may invalidate unrelated live scopes because attribution safety outranks preserving potentially valid evidence, and the attribution boundary that `AiExclusive(scope)` means scope exclusivity only and is not standalone proof that no human edited the worktree; a generic `sce hooks mutation-scope` ingress now drives the seam, joined by current Claude Code and Codex adapter drivers (both registered by `sce setup` and reachable) and an OpenCode adapter driver (full lifecycle + recovery, now generated as the final `sce-mutation-scope.ts` plugin and registered by `sce setup`, so reachable by a real session) — a Pi adapter driver now also exists (`cli/src/services/hooks/pi_mutation_scope/`, hidden `sce hooks pi-mutation-scope`) and is driven by the canonical generated Pi extension in ordinary Pi sessions; its human `user_bash` path uses the separate guarded external-mutation supervisor) +- `context/cli/mutation-scope-runtime.md` (the crate-visible mutation-trace runtime seam and the lifecycle contract every current or future harness adapter must uphold, recorded by the `mutation-scope-runtime-integration` plan: the ten `pub(crate)` re-exports in `runtime/mod.rs` (`coordinate`, `RuntimeBoundary`, `StartProvenance`, `CoordinateOutcome`, `CoordinateError`, `ExternalTaintOperation`, `abandon_scope`, `AbandonScopeOutcome`, `AbandonRecoveryReason`, `AbandonScopeError`), with `ExternalTaintOperation` riding through `coordinator.rs`'s own `pub use` because `CoordinateError::ExternalTaintMarker` carries it — so the type becomes crate-visible without `protected_worktree` becoming a public module — while every `mod` declaration stays private and `ProtectedWorktree`/`ProtectedWorktreeError`/`WORKTREE_LOCK_TIMEOUT`/`reconcile_explicit` stay internal, kept clippy-clean by `#[allow(unused_imports)]` on the re-exports per the `services/style.rs` precedent rather than a placeholder consumer; and the adapter obligations themselves — a scope is one independently mutation-capable execution so a concurrent main agent and subagent need distinct `ScopeId`s, `Start`/`Advance`/`Close` semantics including that only `Start` may carry the optional `StartProvenance`, registered after scope registration and before the protocol commits and never inside `ProtocolState` or the CAS transition, with that registration conditional on the `ScopeState` `register_scope` returns so a provenance row may only be created while the scope is `NeverSeen` (an admission-time snapshot that a post-admission replay cannot backfill) while an existing row is still validated on every provenance-carrying `Start`, that a failed tool still requires `Advance` (the boundary is an observation, not a successful edit) and that a `ScopeId` is never reused after a terminal status (the `NeverSeen` guard silently refuses to reactivate it), `abandon_scope()` requiring positive staleness evidence and never inferring it from `ActorKind`, the `abandon` → `coordinate(Start(successor))` sequence with what each outcome implies for it including that a failed abandonment must never be treated as a safely started successor, that abandonment is not a `RuntimeBoundary` and needs no Quint change, the D1 tradeoff whereby a missing or `NeverSeen` target deliberately forces conservative strong recovery that may invalidate unrelated live scopes because attribution safety outranks preserving potentially valid evidence, and the attribution boundary that `AiExclusive(scope)` means scope exclusivity only and is not standalone proof that no human edited the worktree; a generic `sce hooks mutation-scope` ingress now drives the seam, joined by current Claude Code and Codex adapter drivers (both registered by `sce setup` and reachable) and an OpenCode adapter driver (full lifecycle + recovery, now generated as the final `sce-mutation-scope.ts` plugin and registered by `sce setup`, so reachable by a real session) — a Pi adapter driver now also exists (`cli/src/services/hooks/pi_mutation_scope/`, hidden `sce hooks pi-mutation-scope`) and is driven by the canonical generated Pi extension in ordinary Pi sessions; its human `user_bash` path uses the separate guarded external-mutation supervisor) - `context/cli/mutation-scope-hook-ingress.md` (the one harness-neutral CLI ingress that drives the mutation-scope runtime, in `cli/src/services/hooks/mutation_scope.rs`, built by the `mutation-scope-hook-ingress` plan: the hidden `sce hooks mutation-scope` command routing through the normal `cli_schema::HooksSubcommand::MutationScope` → `convert_hooks_subcommand_request` → `services::hooks::HookSubcommand::MutationScope` → `run_hooks_subcommand_in_repo` stack, reading one normalized JSON lifecycle object from STDIN; the strict `parse_mutation_scope_payload` contract supporting exactly `start`/`advance`/`close`/`flush`/`abandon` with a local `MutationScopePayload` transport enum, `claude_code`/`codex`/`opencode`/`pi` actor mapping, non-blank `scope_id`/`event_id`, exact per-operation key sets, an optional `provenance` object accepted on `start` only (a required non-blank `session_id`, an optional `model_id` where an absent key and an explicit `null` both mean no model, and no other key), and a dedicated hard rejection for any `worktree_id` key; the operation mapping to `RuntimeBoundary::Start`/`Advance`/`Close`/`Flush` through `coordinate()` or a direct `abandon_scope()` call, forwarding `ScopeId`/`EventId`/`ActorKind` and `start`'s optional `StartProvenance` verbatim because `EventId` equality is the runtime replay/idempotency key and provenance values arrive already canonical; identity ownership — the adapter owns `scope_id`/`event_id`/`actor_kind`, SCE owns `worktree_id`/Git tree identities/revisions/attempt IDs, and worktree identity is derived only by the runtime from the invoking checkout; the lazy `FnOnce` DB provider reusing `open_agent_trace_db_for_hook_runtime` so DB acquisition stays inside the runtime's protected-worktree ordering; the non-fail-open error classification by durable completion — malformed payload or any pre-completion `CoordinateError`/`AbandonScopeError` → `CliError`/non-zero, while `CoordinateError::MarkerClearAfterCommit` and `AbandonScopeError::MarkerClearAfterCompletion` are treated as durable success with empty stdout, the marker-cleanup failure logged via `sce.hooks.mutation_scope.marker_clear_after_durable_completion`, and the transition not retried; the empty-stdout success contract; abandonment ownership keeping no-snapshot semantics; and the generic-ingress vs harness-adapter boundary — a Claude Code adapter driver (`cli/src/services/hooks/claude_mutation_scope/`) and a Codex adapter driver (`cli/src/services/hooks/codex_mutation_scope/`, hidden `sce hooks codex-mutation-scope`) now exist and are registered by `sce setup`, both consuming this seam's own `pub(crate)` in-process entrypoint; an OpenCode adapter driver (`cli/src/services/hooks/opencode_mutation_scope/`, hidden `sce hooks opencode-mutation-scope`) also consumes it with a full `Start`/`Close`/`Abandon` lifecycle and checkout-local recovery state, and is now driven in production by the generated `sce-mutation-scope.ts` plugin installed last by `sce setup`; a Pi adapter driver (`cli/src/services/hooks/pi_mutation_scope/`, hidden `sce hooks pi-mutation-scope`) also consumes it now, with `(session-id, tool-call-id)`→`ScopeId` derivation and getppid()/`/proc`-start-time-based stale-process recovery; the canonical generated Pi extension drives it in ordinary sessions — see `context/cli/pi-mutation-scope-integration.md`) - `context/cli/claude-mutation-scope-integration.md` (the first concrete harness lifecycle adapter, `cli/src/services/hooks/claude_mutation_scope/`, hidden command `sce hooks claude-mutation-scope`, built by the `claude-mutation-scope-integration` plan: one independently mutation-capable Claude tool execution = one SCE mutation `ScopeId` (a session/prompt/main agent/subagent is never a scope); the `classify_tool` table (mutation-capable including unknown names, read-only `Read`/`Glob`/`Grep`/`WebFetch`/`WebSearch`/`AskUserQuestion`, `Agent` = delegation) plus the model-only `is_explicit_background_shell` predicate; the length-prefixed hash-free `cc-tool-v1|n=|s=..|a=..|t=..` `ScopeId` keyed on a monotonic checkout-local `attempt_seq` (never reused after terminal) with deterministic `|start` / `|close` `EventId`s; the `/sce/claude-mutation-scope-state.json` bookkeeping store (never attribution evidence, never synced) with its own separate lock never held across a seam call; `PreToolUse` write-ahead `pending_start` → seam `start` → `active` and its fail-closed Claude `permissionDecision: "deny"` on any failure (never `allow`, detail logged via `sce.hooks.claude_mutation_scope.pre_tool_use_fail_closed`); `PostToolUse`/`PostToolUseFailure` → `close`, with `pending_start`+terminal → abandon-not-late-start (D11) and failed-`close` → abandon-not-replay (D12); the abandonment cleanup signals (`PermissionDenied`, `Stop`/`StopFailure`, `UserPromptSubmit`, `SubagentStop`, `SessionEnd`, best-effort `WorktreeRemove` — the last two not observed to fire on Claude Code `2.1.258`); the `recovery_pending` barrier that denies new mutation-capable `PreToolUse` until quiescent then runs one seam `flush`; raw hook `cwd` (or `worktree_path` for `WorktreeRemove`) as authoritative repository root with no adapter-constructed `WorktreeId`; `run_in_background = true` handling — native Claude background Bash is operationally supported but unsupported for mutation attribution (allowed in `PreToolUse` before repository resolution with no mutation scope, adapter attempt, or lifecycle tracking, per the effective decision `context/decisions/2026-10-02-claude-background-bash-untracked.md`), while background `PowerShell` stays denied — and the separate self-detaching-descendant unsupported boundary (D20, with T04's Git-observable evidence — full detail split out to [`claude-mutation-scope-background-execution.md`](cli/claude-mutation-scope-background-execution.md) for the repository's per-file line budget); the ten unmatched `sce setup` registrations; the strict `claude_mutation_scope → hooks::mutation_scope → mutation_trace::runtime` dependency direction through the single `run_mutation_scope_from_payload` seam import (T05); admission-time exact model-state snapshot into `ScopeProvenance`; real Git/Agent Trace persistence coverage shared with the Codex path; and, added by the `doctor-mutation-scope-fix` plan's T04, a third `pending_abandon` attempt phase durably persisted before any abandon seam call — established evidence that abandonment has already been decided, distinct from "may still be running" — with every attempt a broad cleanup signal (`Stop`/`UserPromptSubmit`/`SubagentStop`/`SessionEnd`/`WorktreeRemove`) matches now marked `pending_abandon` in one write before any of their seam calls run, so one attempt's seam failure can never leave a batch sibling without its own retryable evidence; health classification detail split out to [`claude-mutation-scope-health.md`](cli/claude-mutation-scope-health.md)) - `context/cli/claude-mutation-scope-background-execution.md` (native Claude background Bash is operationally supported but unsupported for mutation attribution: `Bash(run_in_background=true)` is allowed in `PreToolUse` with no mutation scope, adapter attempt, lifecycle tracking, or attribution guarantee — effective record `context/decisions/2026-10-02-claude-background-bash-untracked.md` — its later terminal hooks are no-ops, and the accepted A/B contamination consequence is pinned by a regression; background `PowerShell` stays denied with the existing deny text; the T01 `2.1.284` probe result with lifecycle completion still unproven; and the separate self-detaching-descendant unsupported boundary D20 — Git-observable evidence that a foreground `setsid` command's detached descendant can still mutate the repository after `PostToolUse` returns — detail split out of `claude-mutation-scope-integration.md` for the repository's per-file line budget) - `context/cli/claude-mutation-scope-health.md` (the `doctor-mutation-scope-health` plan's Claude task, extended by the `doctor-mutation-scope-fix` plan's T04: `classify_health` in `cli/src/services/hooks/claude_mutation_scope/health.rs` maps `recovery_pending`/`attempts` onto `healthy | recovering | blocked | invalid` — `recovery_pending == false` is `healthy` regardless of live attempts; `recovery_pending == true` with `attempts` empty is `recovering` (the barrier's own next-`PreToolUse` flush clears it); `recovery_pending == true` with `attempts` non-empty is `blocked`, the exact incident shape this classifier exists to surface, since nothing in the ordinary hook lifecycle retries, removes, flushes, or clears from that shape; a read/parse failure is `invalid`. T04 adds a separate, `doctor`-invoked repair path (`assess_repairability`/`repair_blocked` in `health.rs`/`lifecycle.rs`, wired into `sce doctor --fix` by the same plan's T05) for the `blocked` shape: `AutoFixable` only when every currently persisted attempt is already `pending_abandon` (no established-abandon-intent `pending_start`/`active` attempt present); `repair_blocked` re-reads and re-proves that same condition inside one lock-protected read-only state transaction before releasing the lock and retrying each attempt's already-established seam `abandon` call independently, clearing the `recovery_pending` barrier only once no attempts remain — this classifier's own read-only behavior is unaffected by that repair path's existence) -- `context/cli/mutation-trace-ref-reconciliation.md` (the conservative per-worktree snapshot-ref reconciliation pass in `cli/src/services/mutation_trace/runtime/ref_reconciliation.rs`, built by the `mutation-cursor-ref-reconciliation` plan: `reconcile_worktree(repository_root, open_db)` → `pub(super) reconcile_worktree_inner(.., on_lock_contention)`, both returning `ReconciliationOutcome` (`Reconciled(ReconciliationReport { local_required, retained, deleted })` for a pass that ran | `SkippedNoCheckoutIdentity` — an `Ok`, not an `Err` — when no current checkout identity could be derived), a variant-per-fallible-step `ReconcileError` with no `Other`, and the module-owned `RECONCILIATION_LOCK_TIMEOUT`; the two-invariant model — a strictly per-worktree fail-closed local-consistency check via `load_tree_roots(W)` vs. a repository-wide deletion-safety set via `load_all_tree_roots()` so an `A`-owned ref is retained whenever any worktree still durably needs its tree — run entirely under the same `/sce/mutation-cursor.lock` `WorktreeLock` `coordinate()` holds, with the repository-wide read kept coherent by being one SQL statement / one DB snapshot rather than a repository-global lock; deletes only SCE-owned refs via one atomic `git update-ref --no-deref --stdin`, writes no `mutation_trace_*` row, never arms `ExternalTaintMarker`, runs no `git gc`; imperative durability maintenance below the verified Quint protocol; reclaims orphan/unreferenced refs only for the namespace of a checkout id a current worktree still derives — a namespace no current worktree owns (identity-based: a deleted linked worktree, or checkout-id metadata loss followed by `get_or_create_checkout_id` minting a fresh id on a still-present worktree) is beyond every per-worktree pass and left to a recorded future repository-scoped unowned-namespace operation, so the current pass does not bound all orphan-ref growth; `reconcile_worktree` has no `pub(crate)` re-export and no harness/command wiring yet) +- `context/cli/mutation-trace-ref-reconciliation.md` (the conservative per-worktree snapshot-ref reconciliation pass, built by the `mutation-cursor-ref-reconciliation` plan and wired by `mutation-cursor-ref-reconciliation-wiring`: the single lock-held algorithm `reconcile_with_held_lock` in `ref_reconciliation.rs`, the explicit production entrypoint `reconcile_explicit` in `ref_maintenance.rs` (one `WorktreeLock` acquisition, verified-existing Agent Trace DB, typed `ExplicitOutcome`), the documented 10 s concurrency contract, the `ref-maintenance.json` state file (past-only timestamps, skew normalization, atomic writes), the D4 verified-existing DB opener, the two-invariant durability model (per-worktree fail-closed local consistency vs repository-wide deletion safety, never a claim that every repository-wide root is pinned), and the deferred follow-ups (automatic out-of-band reclamation with its binding prerequisites, retired-worktree cleanup, evidence-based advice, historical retention); reconciliation runs only through `sce doctor --fix` and is never automatic) +- `context/cli/mutation-trace-ref-maintenance-state.md` (supporting contracts for snapshot-ref reconciliation: the per-worktree `/sce/ref-maintenance.json` state file with past-only timestamps, `FUTURE_SKEW_TOLERANCE` normalization on successful writes only, atomic staging-and-swap writes, the pure `evaluate_recommendation`, and the non-creating verified-existing Agent Trace DB opener with typed `ExistingRepositoryDbError` and per-DB-state guarantees) - `context/cli/mutation-scope-ref-advisory.md` (the advisory-only hook trigger in `cli/src/services/hooks/mutation_scope.rs` built by T03 of the `mutation-cursor-ref-reconciliation-wiring` plan: a lightweight `advise_if_due` check after a durably completed `Close`/`Flush`, logged and, when actionable, written as an `SCE:` line to an injectable stderr writer (best-effort delivery), with the per-worktree `ref-maintenance.json` state; no reconciliation, Git, or DB work on the hook path) - `context/cli/doctor-ref-reconciliation.md` (the `sce doctor --fix` caller of snapshot-ref reconciliation built by T04 of the `mutation-cursor-ref-reconciliation-wiring` plan: the `runtime::ref_doctor` facade, typed fix-result rows for completed/completed-with-persist-warning/failed/busy passes, and the read-only recommendation section/`ref_reconciliation` JSON object in plain `sce doctor`) - `context/cli/mutation-trace-snapshot-service.md` (the isolated Git snapshot and ref-pinning service `runtime::git_snapshot::GitSnapshotService` in `cli/src/services/mutation_trace/runtime/git_snapshot.rs`: `new` resolving an absolute `git_dir`, `capture_tree` snapshotting staged/unstaged/untracked/deleted worktree state into the repository's normal object database via a throwaway temp index, `pin_tree` protecting a durable tree with a create-only idempotent **direct** `refs/sce/mutation-cursor//` ref, `diff_trees` emitting `patch.rs`-parseable raw diff text; plus the callerless reconciliation substrate — worktree-scoped `list_pins` inventory returning `Result, PinInventoryError>` that rejects a symbolic ref inside the namespace (mutation-cursor pins are direct refs) as `MalformedRef`, matchable separately from a `git for-each-ref` execution failure, and conditional-atomic `delete_pins` running one `git update-ref --no-deref --stdin` transaction of SHA-conditioned deletes — no-dereference so an inventory→delete direct-ref→symref race cannot escape the inventoried namespace, plus a fail-closed pre-check — that aborts whole if any ref changed since inventory; `REF_NAMESPACE` + `pin_ref_prefix` as the single source of truth for the pin path) diff --git a/context/glossary.md b/context/glossary.md index 5687fea4c..9bd073f8c 100644 --- a/context/glossary.md +++ b/context/glossary.md @@ -39,6 +39,8 @@ - `checkout identity`: Stable UUIDv7 identifier assigned to a cloned repository or linked Git worktree, stored in `/sce/checkout-id` (never committed) and resolved via `git rev-parse --git-dir`. The identity is created or reused by `sce setup` through `AgentTraceDbLifecycle::setup()` and also auto-created by hook runtime when `sce setup` has not been run. Checkout identity is now diagnostic metadata for repository-scoped Agent Trace storage; it does not select the active DB and is not stored on Agent Trace rows. Any pre-migration per-checkout DB files at `/sce/agent-trace-{checkout_id}.db` are never touched by SCE and are no longer inspectable via the CLI (the checkout-scoped DB surface was removed by the `retire-legacy-agent-trace-db` plan). See `context/cli/checkout-identity.md`. - `repository identity`: Stable identity of a logical Git repository used to select the active repository-scoped Agent Trace DB path `/sce/repos//agent-trace.db` through the `agent_trace_storage` resolver. Resolved by `cli/src/services/repository_identity/` with precedence: explicit `agent_trace.repository_id` config value, then the URL of the configured Git remote (`agent_trace.repository_remote`, default `origin`), otherwise an actionable `.sce/config.json` error. Remote URLs canonicalize to a scheme-neutral, credential-free `host[:port]/path` form so equivalent SSH/SCP/HTTPS remotes converge, and the repository ID is `sha256("sce-repository-id-v1\0" + canonical_identity)` hex. Distinct from `checkout identity`, which stays per clone/worktree for diagnostics. Setup's named-remote preflight uses a strict lookup that returns `Ok(None)` only for a missing/empty URL, preserves other failures as runtime errors, and discards the URL, retaining only the remote name in technical failure sources. See `context/cli/repository-identity.md`. - `refresh single-flight guard`: Client-owned async coordination for Agent Trace control-plane authentication. Concurrent callers whose stored access token is expired, or whose request rejected the same token, serialize only refresh-and-save work, re-check credentials after acquiring the guard, and reuse the token saved by the first refresher; valid-token requests do not acquire the guard. The guard is held by the spawned task that performs renewal and save, so caller cancellation cannot release it mid-persistence; `wait_for_pending_refresh` acquires it as a completion barrier before `sce sync` returns. +- `snapshot-ref reconciliation`: Explicit, per-worktree maintenance pass (`runtime::ref_maintenance::reconcile_explicit`, run only by `sce doctor --fix`) that deletes orphaned SCE snapshot pins under `refs/sce/mutation-cursor//` while retaining every tree in any worktree's durable roots; never automatic. See `context/cli/mutation-trace-ref-reconciliation.md`. +- `ref-reconciliation advisory`: Lightweight hook-path check (`advise_if_due`) after a durably completed `Close`/`Flush` that may record and print a "reconciliation is recommended" reminder; it never reconciles, lists refs, opens the DB, or spawns Git. See `context/cli/mutation-scope-ref-advisory.md`. - `repository-scoped Agent Trace DB`: Active Agent Trace storage shape where one logical Git repository maps to `/sce/repos//agent-trace.db`. The current seam is `RepositoryAgentTraceDb = TursoDb` in `cli/src/services/agent_trace_db/repository.rs`, backed by the fresh multi-statement `001_repository_schema` baseline plus the additive `002_repository_source_instance_id` migration, with `repository_metadata` (`repository_id` plus `source_instance_id`) plus repository-level trace tables, no `checkout_id` columns, and typed repository-level insert helpers for diff traces, post-commit intersections, Agent Trace rows, messages, and parts. Hook runtime, Agent Trace setup/lifecycle, and `sce sync` resolve repository-scoped storage through `agent_trace_storage`. This is the sole Agent Trace DB adapter; the checkout-scoped adapter and former trace inspection surface were removed by the `retire-legacy-agent-trace-db` plan. - `source_instance_id`: Physical-database identity column on `repository_metadata`, independent of the logical `repository_id`. Added by the additive `002_repository_source_instance_id` migration (existing/placeholder rows default to an empty string); generated once per physical `agent-trace.db` by application code (`generate_source_instance_id()`, UUID v4 today) and validated with `is_valid_source_instance_id()` (non-empty once trimmed) — never generated in SQL and never derived from `repository_id`, remote URL, checkout ID, filesystem path, hostname, or user/workspace identity. `RepositoryAgentTraceDb::verify_or_initialize_repository_metadata` claims it with a concurrency-safe `UPDATE ... WHERE source_instance_id = ''`, so concurrent first opens of the same physical database converge on one winner and an already-valid value is never overwritten; the value stays stable across reopen and repeated `sce setup` runs. Two independently created databases for the same logical repository (for example two clones) get different `source_instance_id` values. See `context/sce/agent-trace-db.md`. - `checkout registry` (removed): The central JSON registry at `/sce/checkout-registry.json` was removed in the `remove-checkout-registry` plan. The later trace inspection surface was also removed; current checkout IDs are diagnostic metadata only and are not discovered as command targets. Any pre-migration `/sce/agent-trace-*.db` files remain untouched. See `context/cli/checkout-identity.md`. diff --git a/context/overview.md b/context/overview.md index 7516e357d..d1e89615f 100644 --- a/context/overview.md +++ b/context/overview.md @@ -45,7 +45,7 @@ for `/validate`. See This repository maintains shared assistant configuration for OpenCode, Claude, Pi, and Codex from a single canonical Pkl authoring source. One typed workflow catalog owns the six workflows' shared identity and target routing metadata, while canonical workflow/phase modules own behavior and migrated package-local documents, and target renderers own formatting. Generated target layouts are ephemeral: repository builds consume a pre-Cargo generated payload through `SCE_CLI_GENERATED_INPUT_DIR`, crates.io and Flatpak stage packaging-only fallbacks, and `config/.opencode`, `config/.claude`, `config/.pi`, `config/.agents`, and the generated working-tree SCE config schema are not committed; versioned SCE config schema snapshots live under `schema/v/`. The catalog also marks a workflow `optional` — currently only `brownfield` — which changes nothing about generation and is projected into a generated `config/optional-workflows.json` manifest for install-time consumers. `nix run .#pkl-check-generated` preserves its exact 142-path artifact, metadata/package, phase-reference, internal-reference, optional-workflow-manifest, workflow-orchestration, OpenCode-permission, required-path, and forbidden-path checks while delegating deterministic payload production and inventories to the shared generated-input producer; `nix flake check` runs the same contract. The target matrix contains one manual OpenCode profile plus Claude, Pi, and Codex; the former automated OpenCode profile has been removed. A fourth Pkl renderer, `config/pkl/renderers/codex-content.pkl`, also consumes the same canonical workflow composition to emit skills-only Codex output under `config/.agents/skills/**` (no per-target frontmatter, matching Pi, and no command/prompt layer). Each of the six catalog workflow skills (not `sce-decision`, which has no user-facing entrypoint on any target) additionally carries a Codex-only `agents/openai.yaml`, rendered by `config/pkl/renderers/codex-metadata.pkl` from the shared catalog's `title`/`description` plus an authored `default_prompt`, with `policy.allow_implicit_invocation: false` so these stateful lifecycle workflows activate only from explicit `$sce-` invocation or Codex's `/skills` discovery, never from conversational relevance alone. The Codex renderer also emits a Codex hook registration file (`config/.codex/hooks.json`) and its install-guidance helper (`config/.codex/hooks/run-sce-or-show-install-guidance.sh`). The existing conversation/diff registrations (`UserPromptSubmit`, `Stop`, `PreToolUse` for `Bash`, `PostToolUse` for `apply_patch`) route through `sce hooks codex` and retain their fail-open behavior where designed. The separate mutation-scope registrations route through `sce hooks codex-mutation-scope`: `PreToolUse` and `PostToolUse` use matcher `^(Bash|apply_patch)$`, while `Stop`, `Interrupt`, `SubagentStop`, and `SessionEnd` omit matcher and are unmatched. The tracked mutation `PreToolUse` bootstrap fails closed if Git-root resolution, helper or `sce` discovery, adapter startup, runtime `Start`, or Bash-policy evaluation cannot establish attribution; mutation-scope `PostToolUse` and cleanup hooks do not use this bootstrap behavior. Delegation, MCP, and unknown tools do not match these generated mutation Pre/Post registrations. `sce setup --codex` (and `--all`) now installs it as a fourth `SetupTarget`; setup merges the user-owned `.codex/hooks.json` through the shared structural Codex hook-config service, preserving unrelated valid handlers and rejecting malformed or Codex-invalid documents before the atomic swap. `sce doctor` diagnoses that file per required registration (`PresentAndCurrent`/`Missing`/`Stale`, or `Malformed` for the whole document) rather than by whole-document equality, reporting the mutation-scope registrations under distinct `#(mutation-scope)` rows separate from the `sce hooks codex` rows, and separately reports whether Codex has actually marked each structurally current registration trusted and whether its effective hook-discovery policy allows project hooks, by reading (never writing) Codex's own `$CODEX_HOME/config.toml` and policy state; `sce doctor --fix` repairs structurally unhealthy registrations through the same merge service but never touches trust or policy state (see `context/sce/doctor-human-text-contract.md`). `sce hooks codex` now exists as a typed dispatcher (`cli/src/services/hooks/codex/`): it parses the raw hook JSON into a `CodexHookEvent` and classifies `(hook_event_name, tool_name)` into `UserPromptSubmit`, `Stop`, `PreToolUse(Bash)`, `PostToolUse(apply_patch)`, or a fail-open `NoOp` fallthrough covering every other combination — including `apply_patch` under `PreToolUse`. `UserPromptSubmit` and `Stop` each capture one `messages`/`parts` row (`role="user"`/`role="assistant"`) into the repository Agent Trace DB under the idempotent `cx_` session prefix, `PreToolUse(Bash)` delegates to the existing Bash policy engine, and `PostToolUse(apply_patch)` parses Codex's own `apply_patch` text, resolves paths from the event `cwd` against the real Git root into safe repository-relative paths, normalizes Add/Update evidence into an SCE unified diff under deterministic event-scoped synthetic line identities derived from `tool_use_id`, and persists it as one `diff_traces` row when non-empty (see `context/sce/codex-integration-runtime.md`) — while invalid cwd/path resolution and malformed STDIN payloads fail open with empty stdout, reported model IDs are preserved without fabricated provider prefixes, and missing/blank apply_patch sessions produce no evidence. Delete-File operations and Bash filesystem effects do not become `diff_traces` in this existing conversation/diff pipeline, and its apply_patch evidence has separate operation limitations. Separately, the Codex mutation-scope adapter tracks `Bash` and `apply_patch` executions as `TrackedMutation` scopes; `AiExclusive` means tracked-scope exclusivity, not that Bash authored every mutation in the interval. -It also includes a Rust CLI (`sce`) for Shared Context Engineering workflows: auth, config inspection, setup, doctor, Agent Trace hooks and synchronization, bash-policy evaluation, and repository-scoped Agent Trace storage infrastructure. See `context/architecture.md` for module-level boundaries and `context/context-map.md` for the full domain file index. A new pure, dependency-free `cli/src/services/mutation_trace/` module now exists as a Rust refinement of the verified `spec/mutation_cursor.qnt` mutation-cursor protocol; it currently carries domain types and the full pure action set — `prepare`/`commit` transition logic, attribution/mutation-event materialization, snapshot-failure/database-failure taint actions, scope abandonment, and recovery with an explicit observed-tree input — is registered without a module-wide dead-code allowance (only declaration-level, reasoned exceptions remain for the retained-and-unwired ref-reconciliation cluster). A separate `cli/src/services/mutation_trace/store.rs` persistence layer (built out by the `mutation-cursor-store-persistence` plan) now provides a real, CAS-guarded database call site against the repository-scoped Agent Trace DB, and a `cli/src/services/mutation_trace/runtime/` coordinator layer (built out by the `mutation-cursor-runtime-coordinator` plan) adds the per-worktree advisory lock, the isolated Git snapshot/ref-pinning service, and a public `coordinate()` entrypoint that drives the protocol against a real worktree under that lock, plus a per-worktree `ref_reconciliation` maintenance pass (built out by the `mutation-cursor-ref-reconciliation` plan) that, only for the namespace of a checkout id a current worktree still derives, reclaims orphaned SCE-owned snapshot refs under that same lock while retaining every tree any durable mutation-cursor state still references (a namespace no current worktree owns — via a deleted linked worktree or checkout-id metadata loss/recreation — is out of reach and left to future repository-scoped work). That runtime now has a second protected entrypoint alongside `coordinate()`: `cli/src/services/mutation_trace/runtime/scope_runtime.rs`'s `abandon_scope()` (built out by the `mutation-scope-runtime-integration` plan) retires a scope whose final worktree boundary was never observed — a dead agent process leaves no `Close` behind — sharing `coordinate()`'s extracted `runtime/protected_worktree.rs` safety prefix but deliberately capturing no Git snapshot, so abandonment is not a `RuntimeBoundary` and needs no Quint change. Both entrypoints are now reachable crate-wide through nine `pub(crate)` re-exports in `runtime/mod.rs` while every module behind them stays private, and the lifecycle contract every current or future harness adapter must uphold is recorded in `context/cli/mutation-scope-runtime.md`. The protocol runtime entrypoints (`coordinate()` / `abandon_scope()`) are now driven by one harness-neutral CLI ingress, the hidden `sce hooks mutation-scope` command (`context/cli/mutation-scope-hook-ingress.md`), which reads a single normalized JSON lifecycle object from STDIN, strictly parses it into exactly `start`/`advance`/`close`/`flush`/`abandon`, maps it to a `RuntimeBoundary` (`start`/`advance`/`close`/`flush`) or an `abandon_scope()` call forwarding `scope_id`/`event_id`/`actor_kind` verbatim (any `worktree_id` key is refused — worktree identity is derived by the runtime from the invoking checkout), and invokes the runtime with a lazy DB provider closure so DB acquisition stays inside the protected-worktree ordering. Unlike the fail-open `diff-trace`/`conversation-trace` intakes, this ingress never discards a valid boundary: it classifies results by durable completion, returning a non-zero `CliError` for a malformed payload or a pre-completion runtime error while treating the two carried-outcome variants (`MarkerClearAfterCommit` / `MarkerClearAfterCompletion` — the durable transition succeeded and only the trailing external-taint marker cleanup failed) as empty-stdout success without retrying the transition. Every successful execution emits empty stdout and writes `mutation_trace_*` rows only. It is the generic transport seam. Concrete harness lifecycle adapters for Claude Code and Codex are wired to it in-process via a `pub(crate)` seam rather than by re-invoking the ingress command; they map each harness's lifecycle events onto one mutation `ScopeId` per mutation-capable tool execution with write-ahead fail-closed `Start`, terminal `Close`, cleanup, recovery barriers, and harness-specific unsupported-boundary handling. A Pi adapter driver (`cli/src/services/hooks/pi_mutation_scope/`) also now consumes this same `pub(crate)` in-process seam through its hidden `sce hooks pi-mutation-scope` command and canonical generated Pi extension, with a `PendingStart`→`Executed`→`Closed`/`PendingAbandon` attempt-phase lifecycle keyed on Pi's `tool_result` event rather than `tool_execution_start` (see `context/cli/pi-mutation-scope-integration.md`); ordinary Pi sessions now reach it, and its human `user_bash` path uses the separate guarded external-mutation supervisor. Codex execution identity and `ScopeId`/`EventId` derivation are owned by `context/cli/codex-mutation-scope-integration.md`. A separate **read-only** consumer of the same module's persisted mutation-event history is wired, though: the post-commit Agent Trace flow calls `mutation_trace::runtime::resolve_post_commit_mutation_ai_patch(...)` to attribute committed touched lines that direct `diff_traces` evidence does not cover, by replaying the newest 128 events for the invoking worktree (bounded also by a commit attribution cut captured under the worktree lock) oldest-to-newest as one causal per-line provenance lineage rather than matching historical patches independently, resolving the worktree's *existing* checkout identity only, creating no identity and writing no mutation-cursor state, and never inserting into `diff_traces` or `post_commit_patch_intersections` (see `context/cli/mutation-trace-agent-attribution.md` and `context/sce/agent-trace-hooks-command-routing.md`). See also `context/cli/mutation-trace-protocol.md`, `context/cli/mutation-trace-runtime-coordinator.md`, `context/cli/mutation-trace-ref-reconciliation.md`, `context/cli/mutation-trace-protected-worktree.md`, `context/cli/mutation-trace-scope-abandonment.md`, `context/cli/mutation-scope-runtime.md`, `context/cli/mutation-scope-hook-ingress.md`, `context/cli/claude-mutation-scope-integration.md`, and `context/cli/mutation-trace-external-mutation-guard.md` (the harness-neutral process supervisor that lets Pi's `user_bash` mutate a worktree, guarded by the same `WorktreeLock`, without ever creating a scope). +It also includes a Rust CLI (`sce`) for Shared Context Engineering workflows: auth, config inspection, setup, doctor, Agent Trace hooks and synchronization, bash-policy evaluation, and repository-scoped Agent Trace storage infrastructure. See `context/architecture.md` for module-level boundaries and `context/context-map.md` for the full domain file index. A new pure, dependency-free `cli/src/services/mutation_trace/` module now exists as a Rust refinement of the verified `spec/mutation_cursor.qnt` mutation-cursor protocol; it currently carries domain types and the full pure action set — `prepare`/`commit` transition logic, attribution/mutation-event materialization, snapshot-failure/database-failure taint actions, scope abandonment, and recovery with an explicit observed-tree input — is registered without a module-wide dead-code allowance (only declaration-level, reasoned exceptions remain, and snapshot-ref reconciliation is wired explicitly through `sce doctor --fix`; automatic reclamation is a deferred follow-up). A separate `cli/src/services/mutation_trace/store.rs` persistence layer (built out by the `mutation-cursor-store-persistence` plan) now provides a real, CAS-guarded database call site against the repository-scoped Agent Trace DB, and a `cli/src/services/mutation_trace/runtime/` coordinator layer (built out by the `mutation-cursor-runtime-coordinator` plan) adds the per-worktree advisory lock, the isolated Git snapshot/ref-pinning service, and a public `coordinate()` entrypoint that drives the protocol against a real worktree under that lock, plus a per-worktree `ref_reconciliation` maintenance pass (built out by the `mutation-cursor-ref-reconciliation` plan) that, only for the namespace of a checkout id a current worktree still derives, reclaims orphaned SCE-owned snapshot refs under that same lock while retaining every tree any durable mutation-cursor state still references (a namespace no current worktree owns — via a deleted linked worktree or checkout-id metadata loss/recreation — is out of reach and left to future repository-scoped work). That runtime now has a second protected entrypoint alongside `coordinate()`: `cli/src/services/mutation_trace/runtime/scope_runtime.rs`'s `abandon_scope()` (built out by the `mutation-scope-runtime-integration` plan) retires a scope whose final worktree boundary was never observed — a dead agent process leaves no `Close` behind — sharing `coordinate()`'s extracted `runtime/protected_worktree.rs` safety prefix but deliberately capturing no Git snapshot, so abandonment is not a `RuntimeBoundary` and needs no Quint change. Both entrypoints are now reachable crate-wide through nine `pub(crate)` re-exports in `runtime/mod.rs` while every module behind them stays private, and the lifecycle contract every current or future harness adapter must uphold is recorded in `context/cli/mutation-scope-runtime.md`. The protocol runtime entrypoints (`coordinate()` / `abandon_scope()`) are now driven by one harness-neutral CLI ingress, the hidden `sce hooks mutation-scope` command (`context/cli/mutation-scope-hook-ingress.md`), which reads a single normalized JSON lifecycle object from STDIN, strictly parses it into exactly `start`/`advance`/`close`/`flush`/`abandon`, maps it to a `RuntimeBoundary` (`start`/`advance`/`close`/`flush`) or an `abandon_scope()` call forwarding `scope_id`/`event_id`/`actor_kind` verbatim (any `worktree_id` key is refused — worktree identity is derived by the runtime from the invoking checkout), and invokes the runtime with a lazy DB provider closure so DB acquisition stays inside the protected-worktree ordering. Unlike the fail-open `diff-trace`/`conversation-trace` intakes, this ingress never discards a valid boundary: it classifies results by durable completion, returning a non-zero `CliError` for a malformed payload or a pre-completion runtime error while treating the two carried-outcome variants (`MarkerClearAfterCommit` / `MarkerClearAfterCompletion` — the durable transition succeeded and only the trailing external-taint marker cleanup failed) as empty-stdout success without retrying the transition. Every successful execution emits empty stdout and writes `mutation_trace_*` rows only. It is the generic transport seam. Concrete harness lifecycle adapters for Claude Code and Codex are wired to it in-process via a `pub(crate)` seam rather than by re-invoking the ingress command; they map each harness's lifecycle events onto one mutation `ScopeId` per mutation-capable tool execution with write-ahead fail-closed `Start`, terminal `Close`, cleanup, recovery barriers, and harness-specific unsupported-boundary handling. A Pi adapter driver (`cli/src/services/hooks/pi_mutation_scope/`) also now consumes this same `pub(crate)` in-process seam through its hidden `sce hooks pi-mutation-scope` command and canonical generated Pi extension, with a `PendingStart`→`Executed`→`Closed`/`PendingAbandon` attempt-phase lifecycle keyed on Pi's `tool_result` event rather than `tool_execution_start` (see `context/cli/pi-mutation-scope-integration.md`); ordinary Pi sessions now reach it, and its human `user_bash` path uses the separate guarded external-mutation supervisor. Codex execution identity and `ScopeId`/`EventId` derivation are owned by `context/cli/codex-mutation-scope-integration.md`. A separate **read-only** consumer of the same module's persisted mutation-event history is wired, though: the post-commit Agent Trace flow calls `mutation_trace::runtime::resolve_post_commit_mutation_ai_patch(...)` to attribute committed touched lines that direct `diff_traces` evidence does not cover, by replaying the newest 128 events for the invoking worktree (bounded also by a commit attribution cut captured under the worktree lock) oldest-to-newest as one causal per-line provenance lineage rather than matching historical patches independently, resolving the worktree's *existing* checkout identity only, creating no identity and writing no mutation-cursor state, and never inserting into `diff_traces` or `post_commit_patch_intersections` (see `context/cli/mutation-trace-agent-attribution.md` and `context/sce/agent-trace-hooks-command-routing.md`). See also `context/cli/mutation-trace-protocol.md`, `context/cli/mutation-trace-runtime-coordinator.md`, `context/cli/mutation-trace-ref-reconciliation.md`, `context/cli/mutation-trace-protected-worktree.md`, `context/cli/mutation-trace-scope-abandonment.md`, `context/cli/mutation-scope-runtime.md`, `context/cli/mutation-scope-hook-ingress.md`, `context/cli/claude-mutation-scope-integration.md`, and `context/cli/mutation-trace-external-mutation-guard.md` (the harness-neutral process supervisor that lets Pi's `user_bash` mutate a worktree, guarded by the same `WorktreeLock`, without ever creating a scope). The generated `/next-task` workflow persists task-level context-synchronization lifecycle state in each plan (`pending`, `synced`, or `blocked`) so unresolved task synchronization debt survives a session boundary and gates new implementation. Successful `/next-task` execution hands task synchronization an explicit, pre-edit-Git-baseline-relative changed-file list plus implementation, verification, done-check, plan-update, and context-impact evidence, recorded directly on the completed task (`Completed`, `Files changed`, `Result`, `Verify`, `Context impact`, `Context synchronization`); the five-file root context pass remains mandatory. A later-session sync-debt retry reads that same completed task record directly from the plan by plan path and task ID, with no separate persisted synchronization handoff. `/validate` is validation-only: it runs final checks, writes the Validation Report, and reports `validated`, `failed`, or `blocked` without plan-level context synchronization. diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 284e2d2bc..16aa03b26 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -407,13 +407,18 @@ Record actual outcomes in the validation report; compilation alone does not vali | S38 | `ref_advisory::tests::maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately` (fresh call re-reads the file); `rm::cross_process::ref_reconciliation_advisory_lock_serializes_independent_processes` | Unit; independent processes | A fresh call observes normalized state, performs no rewrite and does not re-fire | Verified | | S39 | `rm::state_recovery::ref_reconciliation_unnormalized_invalid_state_stays_eligible_until_a_successful_advisory` | Pass-level (injected clocks, held lock) | Repeated read-only evaluation and busy advisories never move eligibility or bytes; the next successful advisory fires exactly once and persists normalized state | Verified | -- [ ] T06: `Remove dead-code exemptions and synchronize context` (status:todo) +- [x] T06: `Remove dead-code exemptions and synchronize context` (status:done) - Task ID: T06 - Scope: In — remove declaration-level `#[allow(dead_code)]` from reconciliation declarations (`ref_reconciliation.rs`, `list_pins`/`delete_pins`/`PinnedRef`/`PinInventoryError` in `git_snapshot.rs`, durable-root readers in `store.rs`) that now have production consumers; separately review genuinely unreachable states (e.g. `SkippedNoCheckoutIdentity`, stale checkout-ID wording) and remove or justify each; update the context files in Context sync with the real async APIs, the wrapper/lock-held split and single-acquisition rule, the documented D1 concurrency contract, Git-derived worktree identities, the two maintenance entrypoints, the advisory trigger and why reconciliation is kept off the hook path, the maintenance state file (past-only fields, normalization, atomic writes), the distinct explicit outcomes, the verified-existing-DB opener contract, doctor reporting, and the corrected two-invariant durability statement (no claim that all repository-wide roots are pinned). Record in context, and not as implemented behavior: the deferred automatic out-of-band reclamation follow-up with its binding prerequisites from D2 (lifecycle-owned execution point, durable next-attempt reservation before expensive work, distinct outcomes, lock-hold analysis, timestamp normalization), the retired-worktree cleanup follow-up, and known storage limitations (historical retention, no repair of other worktrees' pins). Out — behavior changes. - Dependencies: T05 - Done when: no reconciliation-related `allow(dead_code)` remains without a stated reason; clippy with `-D warnings` is clean; context no longer says "retained and unwired", describes checkout-ID identity, or claims automatic reclamation; `context/context-map.md` entries match. - Verify: `nix develop -c ./scripts/run-cli-cargo.sh clippy --manifest-path cli/Cargo.toml --all-targets --all-features -- -D warnings`; `nix shell nixpkgs#ripgrep -c rg -n "allow\(dead_code" cli/src/services/mutation_trace`; `rg -i "retained and unwired" context`; `rg -i "automatic.*(reclam|reconcil)" context` hits only deferral statements. - - Context synchronization: pending + - Completed: 2026-10-09 + - Files changed: `cli/src/services/mutation_trace/runtime/{git_snapshot,maintenance_state,ref_maintenance,ref_reconciliation}.rs`, `cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs`, `cli/src/services/mutation_trace/store.rs`, `context/cli/{agent-trace-storage,async-persistence-guarantees,mutation-scope-runtime,mutation-trace-protocol,mutation-trace-ref-reconciliation,mutation-trace-runtime-coordinator,mutation-trace-snapshot-service,mutation-trace-store}.md`, `context/{context-map,overview}.md`, `context/sce/{agent-trace-hook-doctor,doctor-human-text-contract}.md`, `context/plans/mutation-cursor-ref-reconciliation-wiring.md` + - Result: Removed every "retained and unwired" and "wired by later tasks" dead-code allow from the reconciliation cluster (`ref_reconciliation.rs`, `git_snapshot.rs`, `store.rs` durable-root readers, `ref_maintenance.rs`, `maintenance_state.rs`). The lock-acquiring `reconcile_worktree`/`reconcile_worktree_inner`, `RECONCILIATION_LOCK_TIMEOUT` and their imports are now `#[cfg(test)]` because production uses `reconcile_explicit`. The single-event loaders in `store.rs` (`load_mutation_event`, `load_mutation_event_active_scopes`, `decode_boundary_kind`, their SQL constants and row mappers) have no production reader even in tests; they keep declaration-level allows with an accurate reason (attribution history uses `load_mutation_event_page`), not deleted to avoid scope expansion. Removed the unreachable `ReconciliationOutcome::SkippedNoCheckoutIdentity` and its `ref_maintenance` match arm; `ReconciliationOutcome` now has the single variant `Reconciled`. Removed comments in the touched and adjacent code per the user's instruction. Rewrote `context/cli/mutation-trace-ref-reconciliation.md` and updated the other context files listed in Context sync (no checkout-identity wording, no "unwired" claims, automatic reclamation stated only as a deferred follow-up with its D2 prerequisites, retired-worktree cleanup and storage limitations recorded). + - Verify: `clippy --all-targets --all-features -D warnings` clean; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect); `git diff --check` clean; `rg "retained and unwired|unwired" context` (excluding plans) leaves no stale claim; `rg -i "automatic.*(reclam|reconcil)" context` hits only deferral statements; remaining `dead_code` allows in `mutation_trace` are the single-event loaders, coordinator/scope-runtime outcome payloads and test-model items, each with a stated reason. + - Context impact: important (this task is the context synchronization for the whole plan). + - Context synchronization: synced ## Audit findings diff --git a/context/sce/agent-trace-hook-doctor.md b/context/sce/agent-trace-hook-doctor.md index 26fa399b5..ad2be4b91 100644 --- a/context/sce/agent-trace-hook-doctor.md +++ b/context/sce/agent-trace-hook-doctor.md @@ -132,6 +132,10 @@ The broadened contract for `sce doctor` must cover the following problem invento - Agent Trace DB file exists but cannot be opened (connection failure) or has incomplete schema (missing/unapplied migrations) — reported as `AgentTraceDbConnectionFailed` / `AgentTraceDbSchemaNotReady` with manual-only remediation directing to `sce setup` - Repository-scoped DB path/health remains in the complete report and JSON; human text summarizes it under `Environment` → `Repository identity` without exposing healthy identity metadata. Repository DB records include repository ID, identity source, safe canonical identity, configured remote name, and never raw remote URLs. Checkout identity was removed by the `remove-checkout-id` plan and is no longer reported (see `context/cli/checkout-identity.md`) +### Snapshot-ref reconciliation + +- `sce doctor --fix` always runs one explicit snapshot-ref reconciliation pass and reports its true typed result; plain `sce doctor` stays read-only (no ref or maintenance-state write) and reports a recommendation only when one is due. Details in [../cli/doctor-ref-reconciliation.md](../cli/doctor-ref-reconciliation.md). Reconciliation is never automatic. + ### Repository targeting and git readiness - repo-scoped checks are required but `sce doctor` is run outside a git repository diff --git a/context/sce/doctor-human-text-contract.md b/context/sce/doctor-human-text-contract.md index bf00c59d4..e4116634e 100644 --- a/context/sce/doctor-human-text-contract.md +++ b/context/sce/doctor-human-text-contract.md @@ -16,6 +16,12 @@ The `Environment` domain contains `State`, `Configuration`, and `Repository identity`. The `Repository` domain contains `Git repository`, `Post-commit Agent Trace auto-sync`, and `Git hooks`. +When snapshot-ref reconciliation is recommended (or its maintenance state is +unusable), plain `sce doctor` additionally renders an additive `Snapshot ref +reconciliation` section; it is omitted otherwise. `sce doctor --fix` appends one +`mutation_scope_health` fix-result row for its reconciliation pass. Details in +[../cli/doctor-ref-reconciliation.md](../cli/doctor-ref-reconciliation.md). + ## Header and status vocabulary - Diagnose mode renders `SCE doctor`. From 3e9f5900b13ec8d24bb3e1e395eec117f046aedf Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 03:55:14 +0200 Subject: [PATCH 11/16] tests: Verify doctor reconciliation recovery outcomes Expand doctor CLI coverage to assert successful exit status for reported reconciliation outcomes and to exercise recovery from failed, stale, invalid, corrupt, oversized, and unreadable maintenance state. Record the remediation evidence and completed coverage status for the reconciliation wiring plan. Plan: mutation-cursor-ref-reconciliation-wiring (T05) Co-authored-by: SCE --- .../app/doctor_reconciliation_cli_tests.rs | 216 +++++++++++++++++- .../cli/mutation-trace-ref-reconciliation.md | 2 +- ...tation-cursor-ref-reconciliation-wiring.md | 33 ++- 3 files changed, 238 insertions(+), 13 deletions(-) diff --git a/cli/src/app/doctor_reconciliation_cli_tests.rs b/cli/src/app/doctor_reconciliation_cli_tests.rs index bc08ac11b..980393b29 100644 --- a/cli/src/app/doctor_reconciliation_cli_tests.rs +++ b/cli/src/app/doctor_reconciliation_cli_tests.rs @@ -1,5 +1,5 @@ use std::path::{Path, PathBuf}; -use std::process::Command; +use std::process::{Command, ExitCode}; use serde_json::Value; @@ -15,12 +15,24 @@ const STATE_FILE: &str = "ref-maintenance.json"; const LOCK_FILE: &str = "mutation-cursor.lock"; const RECONCILIATION_DETAIL_PREFIX: &str = "Snapshot ref reconciliation"; +const EXIT_SUCCESS: &str = "success"; +const EXIT_FAILURE: &str = "failure"; + struct CliRun { stdout: String, stderr: String, + exit: String, } impl CliRun { + fn assert_exit_success(&self) { + assert_eq!( + self.exit, EXIT_SUCCESS, + "doctor exit status:\n{}\n{}", + self.stdout, self.stderr + ); + } + fn json(&self) -> Value { serde_json::from_str(&self.stdout) .unwrap_or_else(|error| panic!("doctor stdout is not JSON ({error}):\n{}", self.stdout)) @@ -155,6 +167,7 @@ impl Sandbox { CliRun { stdout: std::fs::read_to_string(out.join("stdout")).expect("stdout"), stderr: std::fs::read_to_string(out.join("stderr")).expect("stderr"), + exit: std::fs::read_to_string(out.join("exit")).expect("exit"), } } } @@ -217,13 +230,19 @@ async fn doctor_reconciliation_cli_child() { ); let mut stdout = Vec::new(); let mut stderr = Vec::new(); - let _ = Box::pin(run_with_dependency_check_and_streams( + let exit_code = Box::pin(run_with_dependency_check_and_streams( command_line, || Ok(()), &mut stdout, &mut stderr, )) .await; + let exit = if format!("{exit_code:?}") == format!("{:?}", ExitCode::SUCCESS) { + EXIT_SUCCESS + } else { + EXIT_FAILURE + }; + std::fs::write(out.join("exit"), exit).expect("exit"); std::fs::write(out.join("stdout"), stdout).expect("stdout"); std::fs::write(out.join("stderr"), stderr).expect("stderr"); } @@ -237,6 +256,7 @@ async fn doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once() assert!(refs_before.contains(&tree)); let inspection = sandbox.run(&["doctor", "--format", "json"]); + inspection.assert_exit_success(); let report = inspection.json(); assert!( report["ref_reconciliation_fix"].is_null(), @@ -247,6 +267,7 @@ async fn doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once() assert_eq!(sandbox.state_bytes(), None); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!(reconciliation["outcome"], "completed", "{}", fix.stdout); @@ -282,6 +303,7 @@ async fn doctor_fix_reports_missing_database_without_creating_it_or_deleting_ref let db_path = sandbox.db_path(); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!(reconciliation["outcome"], "failed", "{}", fix.stdout); @@ -327,6 +349,7 @@ async fn doctor_fix_reports_repository_mismatch_without_touching_the_database_or let refs_before = sandbox.refs(); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!(reconciliation["outcome"], "failed", "{}", fix.stdout); @@ -371,6 +394,7 @@ async fn doctor_fix_reports_busy_when_the_worktree_lock_is_held_and_keeps_other_ held.try_lock().expect("hold the worktree lock externally"); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + fix.assert_exit_success(); let report = fix.json(); drop(held); @@ -393,3 +417,191 @@ async fn doctor_fix_reports_busy_when_the_worktree_lock_is_held_and_keeps_other_ "independent doctor repairs are preserved" ); } + +const FUTURE_SKEW_TOLERANCE_MS: i64 = 5 * 60 * 1000; +const STATE_TIMESTAMP_FIELDS: [&str; 4] = + ["anchor", "last_success", "last_attempt", "last_advised"]; + +fn now_ms() -> i64 { + i64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system clock") + .as_millis(), + ) + .expect("unix ms") +} + +impl Sandbox { + fn seed_state(&self, bytes: &[u8]) { + std::fs::create_dir_all(self.sce_dir()).expect("sce dir"); + std::fs::write(self.state_file(), bytes).expect("seed state"); + } + + fn make_state_unreadable(&self) { + std::fs::create_dir_all(self.state_file()).expect("directory in place of the state file"); + } +} + +fn assert_unreadable_state_outcome(fix: &CliRun, sandbox: &Sandbox, label: &str) { + fix.assert_exit_success(); + let report = fix.json(); + let reconciliation = &report["ref_reconciliation_fix"]; + assert_eq!( + reconciliation["outcome"], "completed_state_persist_failed", + "{label}: {}", + fix.stdout + ); + assert_eq!(reconciliation["counts"]["deleted"], 1, "{label}"); + assert_eq!(reconciliation["counts"]["retained"], 0, "{label}"); + assert_eq!(reconciliation["counts"]["local_required"], 0, "{label}"); + assert!(reconciliation["failure"].is_null(), "{label}"); + let warning = &reconciliation["state_warning"]; + assert_eq!(warning["kind"], "previous_state_unreadable", "{label}"); + assert!(warning["phase"].is_null(), "{label}"); + assert_eq!( + warning["message"], "Maintenance state could not be read; the update was not attempted.", + "{label}" + ); + assert!( + !warning["cause"].as_str().expect("cause").is_empty(), + "{label}" + ); + + let entry = reconciliation_fix_result(&report); + assert_eq!(entry["outcome"], "fixed", "{label}"); + let detail = entry["detail"].as_str().expect("detail"); + assert!( + detail.starts_with( + "Snapshot ref reconciliation completed: deleted 1, retained 0, locally required 0. Warning: Maintenance state could not be read; the update was not attempted. Cause: " + ), + "{label}: {detail}" + ); + assert!(!detail.contains("failed"), "{label}: {detail}"); + + assert_eq!(sandbox.refs(), "", "{label}: orphan pin must be deleted"); + assert!( + sandbox.state_file().is_dir(), + "{label}: the state path must stay a directory" + ); + assert_eq!( + std::fs::read_dir(sandbox.state_file()) + .expect("state directory") + .count(), + 0, + "{label}: the state directory must stay untouched" + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable() { + let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); + sandbox.create_db(&sandbox.repository_id).await; + sandbox.orphan_pin(); + sandbox.make_state_unreadable(); + + let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + + assert_unreadable_state_outcome(&fix, &sandbox, "unreadable"); +} + +#[tokio::test(flavor = "multi_thread")] +async fn doctor_fix_recovers_from_every_initial_maintenance_state_condition() { + let now = now_ms(); + let day_ms = 24 * 60 * 60 * 1000; + let year_ms = 365 * day_ms; + let failed_streak = serde_json::json!({ + "version": 1, + "anchor": now - 10 * day_ms, + "last_attempt": now - 60_000, + "last_attempt_outcome": "failed", + "consecutive_failures": 3, + "last_failure": {"kind": "agent_trace_db_missing", "message": "seeded failure"}, + }); + let expired_advisory = serde_json::json!({ + "version": 1, + "anchor": now - 90 * day_ms, + "last_success": now - 60 * day_ms, + "last_advised": now - 30 * day_ms, + }); + let invalid_timestamps = serde_json::json!({ + "version": 1, + "anchor": now + 10 * year_ms, + "last_success": now + 10 * year_ms, + "last_attempt": now + 10 * year_ms, + "last_advised": now + 10 * year_ms, + "last_attempt_outcome": "failed", + "consecutive_failures": 2, + }); + let cases: Vec<(&str, Vec)> = vec![ + ("failed_streak", failed_streak.to_string().into_bytes()), + ( + "expired_advisory", + expired_advisory.to_string().into_bytes(), + ), + ( + "invalid_timestamps", + invalid_timestamps.to_string().into_bytes(), + ), + ("corrupt", b"not json".to_vec()), + ("oversized", vec![b' '; 5000]), + ]; + + for (label, bytes) in cases { + let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); + sandbox.create_db(&sandbox.repository_id).await; + sandbox.orphan_pin(); + sandbox.seed_state(&bytes); + + let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + + fix.assert_exit_success(); + let report = fix.json(); + let reconciliation = &report["ref_reconciliation_fix"]; + assert_eq!( + reconciliation["outcome"], "completed", + "{label}: {}", + fix.stdout + ); + assert_eq!(reconciliation["counts"]["deleted"], 1, "{label}"); + assert!(reconciliation["failure"].is_null(), "{label}"); + assert!(reconciliation["state_warning"].is_null(), "{label}"); + assert_eq!(sandbox.refs(), "", "{label}: orphan pin must be reclaimed"); + + let after = now_ms(); + let state = sandbox.state_json(); + assert_eq!(state["version"], 1, "{label}"); + assert!(state["last_success"].is_number(), "{label}"); + assert_eq!(state["last_attempt_outcome"], "completed", "{label}"); + assert!( + state.get("consecutive_failures").is_none_or(|v| v == 0), + "{label}: failure streak must be cleared" + ); + assert!(state.get("last_failure").is_none(), "{label}"); + assert_eq!(state["last_report"]["deleted"], 1, "{label}"); + for field in STATE_TIMESTAMP_FIELDS { + if let Some(value) = state.get(field) { + let timestamp = value.as_i64().expect("timestamp"); + assert!( + timestamp <= after + FUTURE_SKEW_TOLERANCE_MS, + "{label}: {field} must be normalized, got {timestamp}" + ); + } + } + + let again = sandbox.run(&["doctor", "--fix", "--format", "json"]); + again.assert_exit_success(); + assert_eq!( + again.json()["ref_reconciliation_fix"]["outcome"], + "completed", + "{label}: the rewritten state is readable" + ); + } + + let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); + sandbox.create_db(&sandbox.repository_id).await; + sandbox.orphan_pin(); + sandbox.make_state_unreadable(); + let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); + assert_unreadable_state_outcome(&fix, &sandbox, "unreadable_state_file"); +} diff --git a/context/cli/mutation-trace-ref-reconciliation.md b/context/cli/mutation-trace-ref-reconciliation.md index 3d35b3bd2..831bfcca3 100644 --- a/context/cli/mutation-trace-ref-reconciliation.md +++ b/context/cli/mutation-trace-ref-reconciliation.md @@ -216,7 +216,7 @@ tests, `hooks::mutation_scope::advisory_trigger_tests`, and `app::doctor_reconciliation_cli_tests`. Race and ordering scenarios use deterministic coordination (barriers, parked phase hooks via `ReconcilePhase`, the worker-entered seam, injected clocks), not sleeps. The plan's coverage map -records which scenarios are verified and which are only partially verified. +records each scenario's verification status and the guard-removal results. ## Deferred follow-ups diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 16aa03b26..7d1d82c6e 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -85,7 +85,8 @@ Facts the design relies on. T01 must re-read each cited location and either conf - *Selected design.* - `Close`/`Flush` (durably completed, including the `MarkerClearAfterCommit` case, after `coordinate_boundary` returned) calls `advise_if_due`, a synchronous function with this complete I/O budget: one capped read (≤ `MAINTENANCE_STATE_MAX_BYTES`) of the state file, at most one zero-wait `try_lock`, at most one atomic write of that small file, all local. It performs no `git` subprocess, no DB open or query, no ref listing, no `spawn_blocking`, no await point after the lock is taken. It is a bounded set of small local file operations of the same class the hook path already performs for the lock and taint marker; it is not claimed to be hard real-time, and any unexpected failure is swallowed into a logged outcome. - The advisory must not add a new Git subprocess. It obtains the worktree's Git dir without spawning Git (T01 determines whether it can be derived from the filesystem — `.git` directory, or `gitdir:` file for linked worktrees — and proves equality with `resolve_git_dir` for the main and a linked worktree, or whether `coordinate`'s already-resolved value can be reused without protocol changes). **Decision rule:** if neither is possible without a new Git subprocess, the hook trigger is dropped entirely and `sce doctor` computes the recommendation from the state file alone; AC1/AC5 are then satisfied with zero hook callers and T03 reduces to documenting that outcome. This fallback needs no user confirmation because it only removes a convenience signal. - - `advise_if_due` states: absent/invalid-version/oversized/corrupt state → write an anchor and return `Anchored` (no advice); reference time (`last_success`, else `anchor`) older than `RECONCILIATION_ADVISORY_AFTER` and no valid advice within the same window → persist `last_advised`, then (only after the write succeeded) return `Advised`; otherwise `NoAction` with no write. Lock busy → `Busy` (skipped, not an error). State read/write failure → `StateUnavailable` (no advice emitted, no retry loop, no scan). It emits at most one advisory per window per worktree. + - `advise_if_due` outcomes: absent/invalid-version/oversized/corrupt state → write an anchor and return `Anchored` (no advice). Reference time (`last_success`, else `anchor`) older than `RECONCILIATION_ADVISORY_AFTER` and no valid advice within the same window → persist `last_advised`, then (only after the write succeeded and its persistence was confirmed) return `Advised`. `AdvisedDurabilityUncertain`: the atomic replacement happened (the advisory state may be visible) but crash durability could not be confirmed; an actionable diagnostic is emitted and durability is never claimed. `StateWriteFailed`: the state update failed before atomic replacement (`PersistFailure::NotApplied`); the previous bytes are intact, no advice is emitted, and no scan runs. `StateUnavailable`: the maintenance state cannot be accessed or evaluated (unresolvable Git dir, unreadable state, lock I/O error); it is not a write outcome, emits no advice, and has no retry loop. `NoAction` (nothing due, no write) and `Busy` (zero-wait lock contention, skipped, not an error) emit no actionable advisory. + - *Cadence.* The normal time-based reminder for an unresolved condition repeats at most once per `RECONCILIATION_ADVISORY_AFTER` (24 h) window per worktree. The one exception is a newly recorded explicit reconciliation failure (`last_attempt` newer than `last_advised`): it may trigger one immediate advisory inside an otherwise active 24 h window, is suppressed once `last_advised` records it, and then follows the normal 24 h cadence until a successful explicit pass clears it. The advisory is therefore not unconditionally one-per-window. - `sce doctor` (read-only) reports whether reconciliation is recommended (last success age, last attempt/failure streak, last advice) computed from the state file as a pure function; `sce doctor --fix` runs `reconcile_explicit` regardless of any state. - *Deferred to a follow-up PR (explicitly not in this plan).* Fully automatic out-of-band reconciliation. Binding prerequisites for that PR: (1) a lifecycle-owned execution point with defined ownership, lifetime, scheduling and failure recovery (for example an explicitly invoked `sce` maintenance command run by a user-managed scheduler, or a future shared session-end mechanism) — never a detached task, daemon, or timeout-cancelled Git operation; (2) a **durable next-attempt reservation persisted under `WorktreeLock` before opening the DB, listing pins, or loading durable roots**, bounded, never interpreted as success, superseded by the success cadence after success or exponential failure backoff after failure, and bypassed by explicit repair; if the reservation cannot be persisted the automatic pass is skipped and the maintenance-state failure is reported; (3) distinct outcomes for completed, completed-but-state-not-persisted, failed, and skipped/deferred; (4) bounded or split lock-hold analysis against the 10 s coordinator timeout; (5) the same invalid-timestamp normalization as D3. T06 records this follow-up in context. - *Trade-offs.* (+) hook completion path has no unbounded work and no new failure mode; correct under every ordering with `coordinate()`; no new infrastructure without a consumer. (−) orphan pins are reclaimed only when an operator runs `sce doctor --fix`; the recommendation is time-based (staleness since last success), not evidence-based, because detecting orphans needs the Git/DB scan this design keeps off the hook path. @@ -340,11 +341,11 @@ Record actual outcomes in the validation report; compilation alone does not vali - D3/D4/AC9/AC14 corrected above. - Context synchronization: synced -- [x] T05: `Add production-path and concurrency regression tests` (status:done) +- [x] T05: `Add production-path and concurrency regression tests` (status:done; reopened `in_progress` 2026-10-09 for the verification-gap remediation below, returned to `done`/`synced` after it) - Task ID: T05 - Scope: In — real Git repos, linked worktrees, and repository Agent Trace DBs. Every race, ordering and timing scenario uses deterministic coordination (barriers, channels, existing `on_lock_contention`/`on_worker_entered`-style seams, parked phase hooks, injected clock); **no timing-based sleeps as correctness proofs**. T02 and T03 add unit-level coverage for their own code; T05 owns the production-path matrix below and each scenario must fail when the guarded behavior is deliberately broken. - *Reconciliation core (explicit path):* (S1) `sce doctor --fix` reclaims an orphan pin left by an interrupted coordinate; (S2) `A → B → C → D` retains every historical tree; (S3) worktree A's pin is retained while worktree B needs the tree (**preservation**); (S4) missing required local pin fails closed with zero deletion; (S5) malformed/symbolic refs cause zero deletion; (S6) concurrent `coordinate()` and reconcile cannot race pin deletion — a coordinate attempt parked on the lock via barrier proceeds only after the pass releases it, and a pin it creates is never deleted by the in-flight pass; (S7) dropping the explicit-pass future while the deletion worker is parked at the worker-entered seam leaves the lock held until the worker is released, then free; (S8) idempotent reconciliation; (S9) plain `sce doctor` mutates neither refs nor state; `--fix` reports the true result for every outcome kind (completed, completed-with-persist-warning, failed kinds, busy); (S10) explicit pass against an external holder returns `Skipped(Busy)`, leaves state byte-identical, and does not count as a failure; (S11) explicit pass with DB failure leaves refs untouched, records a failure streak, never arms the external-taint marker; (S12) the explicit pass and the advisory each complete with one lock acquisition and no self-deadlock, and each returns busy (never deadlocks) against an externally held lock; (S13) with worktree B's required pin deliberately missing, reconciling A deletes only A's eligible pins, does not create or repair B's pin, does not fail, and never deletes any pin protecting B's tree. - - *Hook path and latency (P1):* (S14) the advisory is invoked only after a durably completed `Close`/`Flush` (including `MarkerClearAfterCommit`), never from `Start`/`Advance`/`Abandon`/failed coordinates, and never reaches reconciliation, ref inventory, or DB open — its signature takes none, and spy/panic seams on the reconciler, inventory and opener are never called across every boundary and outcome; `coordinate()` keeps all of its existing snapshot and pinning semantics (required pins are still created and intact — protecting pinning and durable evidence is not weakened), and the advisory itself performs no Git-ref creation, update, or deletion: the test compares the ref set produced by a full hook run against an equivalent coordinate-only baseline (or injects a fake advisory seam and asserts the real advisory adds no ref delta), so any ref change from the coordinator is legitimate and is never attributed to maintenance; (S15) direct advisory contention (scenario A): an explicit pass parked at the post-inventory seam (and, in a second parameterization, at the post-DB-open seam) holds `WorktreeLock`; `advise_if_due` is invoked directly, returns `Busy` through a channel handshake without waiting for the pass, and leaves the state file byte-identical and the ref set unchanged; (S16) coordinator contention (scenario B): with the explicit pass parked holding the lock, a `coordinate()` Close/Flush with a short injected lock timeout fails with the existing lock-timeout error before durable completion and mutates nothing, the existing bounded-acquisition semantics are unchanged, the failed coordinate is a legitimate pre-commit failure, the advisory seam is never invoked after it, and once the pass is released the same boundary succeeds (documents the D1 concurrency contract); (S17) the advisory runs only after the coordinator's lock is released — a seam asserts the worktree lock is acquirable at invocation; (S18a) normal completion: every advisory outcome, a state-write failure, a corrupt state file, and a seam error leave the hook result and `Ok` status identical to the coordinate-only baseline; the marker state is snapshotted immediately before the advisory runs and asserted byte-identical (existence) afterwards, so the advisory never creates, clears, or modifies it, and the normal coordinator-cleared state is unchanged; (S18b) `MarkerClearAfterCommit`: reusing the coordinator's existing marker-clear failure fixture (no change to `coordinator.rs`, `ProtectedWorktree`, or recovery semantics; if unreachable from the hook layer, a fake coordinate seam returns that variant while a real armed marker file exists), the boundary is classified as durably completed, the hook returns the same successful empty-stdout result with the advisory allowed to run, the marker the failed clear left behind is asserted still armed and unmodified after the advisory, and a subsequent real `coordinate()` invocation observes the inherited taint and takes the existing conservative recovery path; (S33) post-coordinate handoff race (scenario C): `coordinate()` completes successfully and releases its lock; a seam pauses the hook immediately before the advisory call; a competing explicit reconciliation acquires `WorktreeLock` through a barrier; the advisory resumes and returns `Busy`; the already committed boundary still returns success, and the maintenance state and refs are untouched by the advisory. + - *Hook path and latency (P1):* (S14) the advisory is invoked only after a durably completed `Close`/`Flush` (including `MarkerClearAfterCommit`), never from `Start`/`Advance`/`Abandon`/failed coordinates, and never reaches reconciliation, ref inventory, or DB open — its signature takes none (originally proposed as spy/panic seams on the reconciler, inventory and opener; replaced, because such seams would add production dependencies to a deliberately path-only interface, by two complementary forms of evidence: (A) static inspection of the complete production call graph reachable from `advise_after_completed_boundary()` showing no reconciliation, `GitSnapshotService`, ref listing/mutation, Agent Trace DB access, Git subprocess or unbounded work, and (B) the real-hook behavioral tests named in the coverage map); `coordinate()` keeps all of its existing snapshot and pinning semantics (required pins are still created and intact — protecting pinning and durable evidence is not weakened), and the advisory itself performs no Git-ref creation, update, or deletion: the test compares the ref set produced by a full hook run against an equivalent coordinate-only baseline (or injects a fake advisory seam and asserts the real advisory adds no ref delta), so any ref change from the coordinator is legitimate and is never attributed to maintenance; (S15) direct advisory contention (scenario A): an explicit pass parked at the post-inventory seam (and, in a second parameterization, at the post-DB-open seam) holds `WorktreeLock`; `advise_if_due` is invoked directly, returns `Busy` through a channel handshake without waiting for the pass, and leaves the state file byte-identical and the ref set unchanged; (S16) coordinator contention (scenario B): with the explicit pass parked holding the lock, a `coordinate()` Close/Flush with a short injected lock timeout fails with the existing lock-timeout error before durable completion and mutates nothing, the existing bounded-acquisition semantics are unchanged, the failed coordinate is a legitimate pre-commit failure, the advisory seam is never invoked after it, and once the pass is released the same boundary succeeds (documents the D1 concurrency contract); (S17) the advisory runs only after the coordinator's lock is released — a seam asserts the worktree lock is acquirable at invocation; (S18a) normal completion: every advisory outcome, a state-write failure, a corrupt state file, and a seam error leave the hook result and `Ok` status identical to the coordinate-only baseline; the marker state is snapshotted immediately before the advisory runs and asserted byte-identical (existence) afterwards, so the advisory never creates, clears, or modifies it, and the normal coordinator-cleared state is unchanged; (S18b) `MarkerClearAfterCommit`: reusing the coordinator's existing marker-clear failure fixture (no change to `coordinator.rs`, `ProtectedWorktree`, or recovery semantics; if unreachable from the hook layer, a fake coordinate seam returns that variant while a real armed marker file exists), the boundary is classified as durably completed, the hook returns the same successful empty-stdout result with the advisory allowed to run, the marker the failed clear left behind is asserted still armed and unmodified after the advisory, and a subsequent real `coordinate()` invocation observes the inherited taint and takes the existing conservative recovery path; (S33) post-coordinate handoff race (scenario C): `coordinate()` completes successfully and releases its lock; a seam pauses the hook immediately before the advisory call; a competing explicit reconciliation acquires `WorktreeLock` through a barrier; the advisory resumes and returns `Busy`; the already committed boundary still returns success, and the maintenance state and refs are untouched by the advisory. - *State, crashes, invalid timestamps (P2):* (S19) absent state → anchor written once, no advice; a second call writes nothing (file byte-identical); (S20) due state → exactly one time-based advice per window with `last_advised` persisted *before* the advice is returned; repeated calls inside the window do not write or advise; after the injected clock passes the window the advice recurs once; a failure recorded after `last_advised` is advised once inside the window, then suppressed, then repeats on the normal cadence until a successful explicit pass clears it; (S21) concurrent processes: one advisory parked inside its locked section, a second returns `Busy`, a third after release returns `NoAction` — exactly one advice, and an explicit pass concurrent with an advisory serializes (no duplicate explicit passes, second returns busy/waits per policy); (S22) state-persistence failure (injected write error): the advisory emits no advice and never scans; an explicit pass whose final write fails returns `CompletedStatePersistFailed` — distinct from `Completed` and `Failed` — with its deletions intact; (S23) crash durability: an interrupted atomic write (temp file left, rename not performed) leaves the previous state readable and eligibility preserved, a stale temp file is ignored/overwritten, and a fresh call (simulating process restart) re-reads state and advises at the correct time; (S24) clock-skew tolerance, with an injected clock for every case (`FUTURE_SKEW_TOLERANCE` = 5 min): (1) stored `last_success` equal to `now` — valid, age 0, no advice, file byte-identical; (2) one minute ahead of `now` — valid, age 0, preserved, no advice, no rewrite; (3) exactly five minutes ahead — valid (boundary is inclusive), age 0, preserved, no advice, no rewrite; (4) more than five minutes ahead (just beyond, and far-future) for `last_success`, `last_advised` and `anchor` — invalid, normalized once under the lock and persisted per D3 (invalid fields cleared; with `last_success` and `anchor` both unusable the advisory fires immediately once, persists `last_advised = now` and a fresh `anchor = now`); (5) clock rollback beyond tolerance (stored values valid at write time, injected `now` moved back more than five minutes) — treated as case 4; a rollback within tolerance is case 2/3 and never makes an advisory eligible by itself; (6) repeated reads after normalization — the file is byte-identical across repeated calls, no rewrite occurs, and eligibility is neither postponed nor slid; (7) process restart (fresh call re-reading the file) followed by clock advance past `RECONCILIATION_ADVISORY_AFTER` — advice recurs exactly once, proving eventual eligibility. After every **successful state-writing** operation an assertion checks the exact invariant `t ≤ now + FUTURE_SKEW_TOLERANCE` for every stored timestamp (not "no timestamp is in the future"); after read-only, lock-busy, or failed operations the assertion is instead that the state file is byte-identical to its prior content, with no normalization claimed; (S25) missing, unparsable, wrong-version, and oversized state files are recovered without failing any operation and are rewritten valid by the next successful recorded operation (until then, read-only and failed operations leave them untouched and treat them as default state). - *Timestamp-normalization regressions (P2; injected clocks and deterministic synchronization, no sleeps):* (S34) read-only doctor observing an invalid future timestamp — plain `sce doctor` reports "recommended" with the invalid-timestamp note, the state file is byte-identical afterwards, and it is not reported as normalized; (S35) lock-busy advisory — with the lock held through a barrier and an invalid future timestamp stored, `advise_if_due` returns `Busy` and the state file is byte-identical; the same holds for a lock-busy explicit pass (`Skipped(Busy)`); (S36) failed state write during a normalizing operation, in two legs. *Pre-rename leg* (`NotApplied`, e.g. a rename failure): the previous state is byte-identical and readable, the advisory reports `StateWriteFailed` with no advice and no claimed normalization, the explicit pass reports `CompletedStatePersistFailed`, and a later successful operation still normalizes it. *Post-rename leg* (`DurabilityUncertain`, e.g. a parent-directory sync failure): the new state may already be visible, so byte identity is not asserted; the advisory reports `AdvisedDurabilityUncertain` with an actionable diagnostic, the normalization is not claimed durable, and a later invocation recovers safely (re-reads the visible state, performs no rewrite). `StateUnavailable` is reserved for state that cannot be read or evaluated and is not a write outcome; (S37) successful normalization — a normalizing advisory and a normalizing explicit pass each persist state in which every timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`, valid near-future values (within tolerance) are preserved byte-for-byte, and a repeated read performs no rewrite and does not move eligibility forward; (S38) process restart after normalization — a fresh call re-reading the persisted file observes the normalized timestamps, performs no rewrite, and does not re-fire the immediate advice; (S39) eventual eligibility — with invalid timestamps unnormalized (S34/S35/S36 states) repeated read-only evaluations never push eligibility out, and after the injected clock advances past `RECONCILIATION_ADVISORY_AFTER` a successful advisory fires exactly once and persists normalized state. - *Explicit recovery:* (S26) `sce doctor --fix` runs and reports its true result from each of: failed streak, stale advice, invalid timestamps, corrupt state, unwritable state (→ `CompletedStatePersistFailed`); (S27) explicit success resets the failure streak and sets `last_success`; explicit failure increments the streak without touching `last_success`; lock-busy never changes state. @@ -356,13 +357,25 @@ Record actual outcomes in the validation report; compilation alone does not vali - Reopened: 2026-10-09 (the earlier `done`/`synced` marking was unsupported: S6, S7, S16, S17, S18b and S33 were listed as "not added" or covered only through fake closures, helper tests or enum mappings). - Files changed (reopening pass): `cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs` and new `ref_maintenance/tests/{support,coordination,worktrees,cross_process,fail_closed,state_recovery}.rs`; `cli/src/app/doctor_reconciliation_cli_tests.rs` (new, registered in `cli/src/app.rs`); `ref_advisory.rs` (tests); `git_snapshot.rs` (deletion-cancellation test); production test seams only — `coordinator.rs` (`coordinate_inner` takes `lock_timeout`; `coordinate` passes `WORKTREE_LOCK_TIMEOUT`, so production behaviour is unchanged), `protected_worktree.rs` (`acquire_with_timeout` is `pub(super)`), `git_snapshot.rs` (`delete_pins_inner` forwards `on_worker_entered`; the dead `run_ref_mutation` wrapper was removed), `hooks/mutation_scope.rs` (`drive_mutation_scope` is `pub(crate)`). No protocol, Quint, dispatch, runtime or timeout changes. - Findings that shaped the tests: (1) a parked pass must run on its own thread and runtime — parking inside `tokio::spawn` blocks a worker and starves the process/IO driver the coordinator needs, which made the coordinator stall before it ever tried the lock; (2) `sce doctor --fix` runs a separate global-state repair that creates the Agent Trace DB parent directory before reconciliation, so the CLI-level missing-DB guarantee is "no DB file or sidecars", not "no parent directory"; (3) doctor failure kinds are `agent_trace_db_missing` and `agent_trace_db_repository_mismatch`, not a generic `agent_trace_db_unavailable`. - - Verification (2026-10-09): full CLI suite 493 passed, 0 failed; focused `ref_reconciliation` 50, `mutation_scope` 25, `doctor` 25, `maintenance` 83 (all non-zero); `clippy --all-targets --all-features -D warnings` clean; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect); `git diff --check` clean; 20 consecutive passes of every `ref_maintenance::tests::*` and cancellation test, 5 consecutive passes of the doctor CLI tests; no added comments, sleeps, `allow(dead_code)`, `dyn`, `BoxFuture`, detached tasks or extra production runtimes. - - Guard-removal checks actually performed: repository-wide roots replaced by worktree-local roots → S3 and S13 tests fail; deletion worker drops its lease → the new deletion-cancellation test and the existing pin-cancellation test fail; advisory ignores lock contention → S15 (both phases) and S33 fail. Not yet performed for the remaining production-path guards (S6, S16, S17, S18b, S21); until they are, "each fails when the guarded behavior is broken" is only demonstrated for the three guards above. - - Accepted gaps (user-approved closure, 2026-10-09): S9 persist-warning through the real CLI, S14 spy/panic seams, S26 through `sce doctor --fix`, and guard-removal checks for S6, S16, S17, S18b and S21 were not completed; the corresponding guarantees rest on the pass-level tests and structural signatures described above. + - Verification, remediation pass (2026-10-09): full CLI suite 495 passed, 0 failed (two new doctor CLI tests); focused `doctor_reconciliation_cli` 7, `ref_reconciliation` 50, `mutation_scope` 25, `maintenance` 85 (all non-zero); new doctor CLI tests 5 consecutive passes; every `ref_maintenance::tests` test and the cancellation tests 20 consecutive passes; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; `cargo fmt` made no change; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect). + - Verification (earlier pass, 2026-10-09): full CLI suite 493 passed, 0 failed; focused `ref_reconciliation` 50, `mutation_scope` 25, `doctor` 25, `maintenance` 83 (all non-zero); `clippy --all-targets --all-features -D warnings` clean; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect); `git diff --check` clean; 20 consecutive passes of every `ref_maintenance::tests::*` and cancellation test, 5 consecutive passes of the doctor CLI tests; no added comments, sleeps, `allow(dead_code)`, `dyn`, `BoxFuture`, detached tasks or extra production runtimes. + - Guard-removal checks actually performed: repository-wide roots replaced by worktree-local roots → S3 and S13 tests fail; deletion worker drops its lease → the new deletion-cancellation test and the existing pin-cancellation test fail; advisory ignores lock contention → S15 (both phases) and S33 fail. The remaining guards (S6, S16, S17, S18b, S21) were checked in the remediation pass; see the results table below. + - Remediation (2026-10-09, PR #305 review of `39f78941`; no gap was waived and no production code changed): (1) S9 persistence-warning outcome and S26 recovery table now run through the real `sce doctor --fix` (`doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable`, `doctor_fix_recovers_from_every_initial_maintenance_state_condition`), using a naturally unreadable state (a directory at `.git/sce/ref-maintenance.json`) rather than a fault seam; `NotApplied` and `DurabilityUncertain` stay at the persistence layer (`ref_advisory::tests`, `rm::state_recovery`), with no CLI fault seam. (2) The doctor CLI child now captures the `ExitCode` returned by `run_with_dependency_check_and_streams`, writes `success`/`failure` to an `exit` file beside `stdout`/`stderr`, `CliRun` exposes it, and every doctor invocation asserts it. The CLI contract (unchanged): `render_report` always returns `Ok`, so plain doctor and `--fix` exit successfully for completed, failed-fix, busy and persistence-warning outcomes; an individual failed fix is reported in the payload, not in the exit status. (3) S14 is verified by static call-graph inspection plus real-hook ref-delta assertions (see the S14 row); no spy/panic seams were added. (4) The five remaining guard-removal checks were performed in a throwaway `git worktree` at `39f78941` (shared `CARGO_TARGET_DIR`, removed afterwards); results are in the table below. (5) D2 now distinguishes `StateUnavailable`, `StateWriteFailed`, `AdvisedDurabilityUncertain`, `Advised`, `NoAction` and `Busy`, and documents the single immediate advisory for a newly recorded failure; D3, AC5, AC13, S20 and S36 already agreed and were not changed. + - Guard-removal results for the five remaining guards (actual outcomes; every mutation lived only in the throwaway worktree and none is in the diff): + + | Scenario | Guard temporarily changed | Targeted test | Expected failure observed | Restored test passes | + |---|---|---|---|---| + | S6 | `ProtectedWorktree::acquire_with_timeout` took its lock from a diverted Git dir (`git_dir.join("bypass")`), so the coordinator no longer contends with the pass | `ref_reconciliation_real_coordinate_waits_for_a_pass_parked_after_db_open` and `…_after_pin_inventory` | Yes, both: the coordinator ran to completion without ever signalling lock contention (`signal sender dropped before signalling`) | Yes (2 passed) | + | S16 | `classify_coordinate` in `hooks/mutation_scope.rs` swallowed `CoordinateError::LockAcquisition` as a successful empty result | `ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries` and `ref_reconciliation_flush_lock_timeout_under_maintenance_commits_nothing_and_retries` | Yes, both: `expect_err` on the contended boundary failed (`the lock cannot be acquired while the pass holds it: ""`), i.e. the lock timeout was no longer reported as a pre-commit failure | Yes (2 passed) | + | S17 | `Close` leaked the coordinator's `WorktreeLock` past completion (flagged in `coordinate_inner`, `std::mem::forget` of the lock in `complete_blocking`), the observable equivalent of running the advisory before the release | `ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock` | Yes: the zero-wait acquisition inside the advisory callback failed (`the coordinator lock is released before the advisory runs: TimedOut { timeout: 0ns }`). A first attempt that leaked the lock on every boundary failed during test setup and was discarded as not evidence | Yes (1 passed) | + | S18b | `coordinate_inner` passed `false` instead of `protected.inherited_external_taint()` to `coordinate_protected` | `ref_reconciliation_marker_clear_failure_after_commit_keeps_taint_for_the_next_coordinate` | Yes: `inherited taint conservatively abandons the live scope` failed with `left: Some(Active)` | Yes (1 passed) | + | S21 | `advise_if_due_with` acquired the lock and dropped it immediately, so no lock covered the state-writing section | `ref_reconciliation_advisory_lock_serializes_independent_processes` | Yes: the second independent process returned `Advised` where `Busy` was required (duplicate advice). A first attempt failed to compile (unused import) and was discarded as not evidence | Yes (1 passed) | + + No sleeps were used to force interleavings; the tests use their existing barriers, parked-phase seams and process rendezvous. - Context impact: none (tests and test seams only). - Context synchronization: synced - - **Coverage map (S1–S39).** Status is `Verified`, `Partially verified` or `Not verified`; a related unit test never upgrades a production-path scenario. Test names are exact; paths are relative to `cli/src/services/mutation_trace/runtime/` unless they start with `app::` or `services::`. `rm::` = `ref_maintenance::tests::`. + - **Coverage map (S1–S39; 40 rows because S18 is split into S18a/S18b; all 40 Verified after the remediation pass).** Status is `Verified`, `Partially verified` or `Not verified`; a related unit test never upgrades a production-path scenario. Test names are exact; paths are relative to `cli/src/services/mutation_trace/runtime/` unless they start with `app::` or `services::`. `rm::` = `ref_maintenance::tests::`. | Scenario | Exact test | Test level | Verified invariant | Status | |---|---|---|---|---| @@ -374,12 +387,12 @@ Record actual outcomes in the validation report; compilation alone does not vali | S6 | `rm::coordination::ref_reconciliation_real_coordinate_waits_for_a_pass_parked_after_db_open`; `…_after_pin_inventory` | Production integration (real `coordinate_inner`, real lock) | Coordinator hits contention, opens no DB and arms no marker while the pass holds the lock; the pin created after release survives; only the orphan is deleted; new pin precedes the durable-root commit; lock free afterwards | Verified | | S7 | `services::mutation_trace::runtime::git_snapshot::tests::cancelled_delete_keeps_worktree_lock_until_git_ref_deletion_completes` | Cancellation integration (real `delete_pins`) | Dropping the caller and outer lock owner leaves the lock held by the parked worker; after release the pin is gone, no `.lock` files remain, a new pin proceeds. Existing `cancelled_pin_keeps_worktree_lock_until_git_ref_mutation_completes` covers the pin side | Verified | | S8 | `rm::ref_reconciliation_retains_every_repository_wide_durable_root_and_never_repairs_other_worktrees` (second pass); CLI test second `--fix` run | Pass-level; CLI | Second pass deletes nothing | Verified | - | S9 | `app::doctor_reconciliation_cli_tests::*` (plain doctor, completed, failed ×2, busy); `services::doctor::reconciliation_fix_tests::*`; `services::doctor::render::reconciliation_render_tests::*` | CLI end-to-end for four outcome kinds; mapping unit tests for `completed_state_persist_failed` | Plain doctor mutates neither refs nor state; `--fix` reports the true result. The persist-warning outcome is not reachable through the CLI without a fault seam | Partially verified | + | S9 | `app::doctor_reconciliation_cli_tests::doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable` (persist-warning); `…::doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once` (plain doctor, completed); `…::doctor_fix_reports_missing_database_without_creating_it_or_deleting_refs` and `…::doctor_fix_reports_repository_mismatch_without_touching_the_database_or_refs` (failed); `…::doctor_fix_reports_busy_when_the_worktree_lock_is_held_and_keeps_other_repairs` (busy); `services::doctor::render::reconciliation_render_tests::json_completed_state_persist_failed_distinguishes_warning_kinds` and `services::doctor::reconciliation_fix_tests::*` (mapping tests for `NotApplied`/`DurabilityUncertain`); `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome`; `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}` | CLI end-to-end for completed, failed, busy and `completed_state_persist_failed` (kind `previous_state_unreadable`, via a directory at the state path: orphan deleted, counts 1/0/0, `failure` null, detail reports the warning without "failed", directory untouched, exit success); persistence layer and render/mapping unit tests for `NotApplied` and `DurabilityUncertain` | Plain doctor mutates neither refs nor state; `--fix` reports the true result and the CLI exit status is asserted. At CLI level only `previous_state_unreadable` is exercised for the persist-warning outcome; `not_applied` and `durability_uncertain` are covered at the persistence and rendering layers by design (no CLI fault seam) | Verified | | S10 | `rm::ref_reconciliation_failure_streak_counts_but_lock_contention_never_does`; `rm::fail_closed::ref_reconciliation_second_explicit_pass_is_busy_and_records_exactly_one_pass`; CLI busy test | Pass-level; CLI | Lock contention returns `Skipped(Busy)`, leaves state byte-identical and is not a failure | Verified | | S11 | `rm::fail_closed::ref_reconciliation_database_failure_never_arms_the_external_taint_marker`; `rm::ref_reconciliation_missing_database_creates_nothing_and_deletes_nothing` | Pass-level | DB failure leaves refs untouched, records a streak, never arms the marker | Verified | | S12 | `rm::ref_reconciliation_lock_held_implementation_never_reacquires_the_lock`; `rm::ref_reconciliation_worktree_wrapper_acquires_once_and_releases`; `rm::ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock`; S6/S15 tests | Pass-level; production integration | Single acquisition per operation, shown by bounded completion under the existing self-contending lock (no acquisition counter) | Verified | | S13 | `rm::worktrees::ref_reconciliation_missing_pin_in_another_real_worktree_neither_fails_nor_is_repaired` | Production integration (real linked worktree) | Reconciling main reclaims its orphan, keeps the pin protecting the linked tree, never recreates the linked worktree's missing pin, does not fail; the linked worktree's own pass fails closed | Verified | - | S14 | `services::hooks::mutation_scope::advisory_trigger_tests::{advisory_runs_once_after_completed_close_and_flush, advisory_never_runs_for_start_or_advance, advisory_never_runs_after_failed_coordinate, advisory_never_runs_for_abandon}`; ref-delta assertions in S17, S18a, S18b and S33 tests | Hook policy (fake coordinate); production integration for ref delta | Advisory fires only after a durably completed `Close`/`Flush`; the real advisory adds no ref delta. Spy/panic seams on reconciler, inventory and opener are not implemented | Partially verified | + | S14 | Behavioral: `services::hooks::mutation_scope::advisory_trigger_tests::{advisory_runs_once_after_completed_close_and_flush, advisory_runs_after_marker_clear_after_commit_with_success_result, advisory_never_runs_for_start_or_advance, advisory_never_runs_after_failed_coordinate, advisory_never_runs_for_abandon}` (when it runs); `rm::coordination::{ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries, ref_reconciliation_flush_lock_timeout_under_maintenance_commits_nothing_and_retries}` (real failed coordinate: advisory count 0); `rm::coordination::ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock` (real `Close`: refs identical before/after the advisory, marker snapshot identical, commit unchanged, zero-wait lock acquirable); `rm::coordination::ref_reconciliation_advisory_failures_after_a_real_commit_never_change_the_hook_result` (errors do not change the result); `rm::coordination::ref_reconciliation_marker_clear_failure_after_commit_keeps_taint_for_the_next_coordinate` (marker fingerprint identical across the advisory); `rm::coordination::ref_reconciliation_advisory_after_a_real_commit_is_busy_when_a_pass_wins_the_handoff` (refs and state untouched by the advisory). Structural: call graph below | Hook policy (fake coordinate); production integration (real coordinator, real Git) for ref delta; static inspection | Advisory fires only after a durably completed `Close`/`Flush` and never after a failed coordinate, `Start`, `Advance` or `Abandon`; it adds no ref delta, never modifies the external-taint marker, its errors never change a committed result, and the coordinator's lock is already released. Structural, from `advise_after_completed_boundary(&Path)`: it calls only `advise_if_due(repository_root, system_unix_time_ms)` → `derive_git_dir` (filesystem walk, `std::fs`, no Git subprocess), `read_state`/`plan_advisory`/`normalize`/`evaluate` (pure or `std::fs`/`std::io`, capped read), the synchronous `worktree_lock::acquire_inner` with `Duration::ZERO` (one `try_lock`; returns `TimedOut` before any sleep), `write_state_atomically` (staged file, rename, directory sync) and `outcome_after_write` (pure). Production code in `ref_advisory.rs` and `maintenance_state.rs` has no `crate::` import, no `async`/`await`/`tokio`/`spawn_blocking`, no `Command`, no Git invocation (the only `git` token is the `.git` path component in `derive_git_dir`), and no reference to `reconcile_explicit`, `reconcile_with_held_lock`, `GitSnapshotService`, `list_pins`, `delete_pins`, `RepositoryAgentTraceDb` or a store. The `async`/`spawn_blocking` hits in `worktree_lock.rs` are only `acquire_async`/`acquire_inner_async`, which the advisory does not call. The only production caller is the injected seam in `hooks/mutation_scope.rs`. Verified through static production call-graph inspection and real-hook ref-delta assertions. No spy/panic seams added. | Verified | | S15 | `rm::coordination::ref_reconciliation_independent_advisory_task_is_busy_while_a_pass_is_parked_after_db_open`; `…_after_pin_inventory`; earlier synchronous `rm::ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock` | Concurrency integration (independent blocking task) | Advisory returns `Busy` without waiting, state and refs unchanged, pass continues, later advisory follows the resulting state (`NoAction`, then `Advised` after the window). The synchronous test is retained but is not independent concurrency | Verified | | S16 | `rm::coordination::ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Close`); `…_flush_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Flush`) | Production integration (real coordinator, injected 300 ms timeout) | Existing lock-timeout error; no durable transition, no pins, no marker, no advisory after the failure; the same boundary succeeds after release | Verified | | S17 | `rm::coordination::ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock` | Hook/coordinator integration | Zero-wait `WorktreeLock` acquisition succeeds inside the advisory callback; the commit is already durable and unchanged afterwards; refs unchanged; empty stdout | Verified | @@ -392,7 +405,7 @@ Record actual outcomes in the validation report; compilation alone does not vali | S23 | `maintenance_state::tests::maintenance_state_recovers_after_interrupted_pre_rename_write`; `ref_advisory::tests::maintenance_advisory_rename_failure_preserves_state_and_next_call_can_advise` | Unit | Interrupted write preserves previous state; fresh call advises at the right time | Verified | | S24 | `ref_advisory::tests::{maintenance_advisory_treats_timestamps_at_or_just_ahead_of_now_as_valid_age_zero, maintenance_advisory_preserves_timestamps_within_skew_tolerance, maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately, maintenance_advisory_clock_rollback_beyond_tolerance_normalizes_while_within_it_does_not}`; `rm::state_recovery::ref_reconciliation_invalid_last_advised_does_not_suppress_a_due_advisory`; `maintenance_advisory_advises_once_per_window_and_persists_before_returning` | Unit (injected clocks) | Cases 1–7: equal and near-future valid; exactly +5 min valid; beyond tolerance (anchor, last_success, last_advised) normalized once; rollback beyond vs within tolerance; byte-identical repeats; window recurrence | Verified | | S25 | `ref_advisory::tests::{maintenance_advisory_anchors_once_then_leaves_state_untouched, maintenance_advisory_recovers_corrupt_state_by_anchoring, maintenance_advisory_recovers_wrong_version_and_oversized_state_by_anchoring}`; `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition` | Unit; pass-level | Missing, corrupt, wrong-version and oversized state recover without failure and are rewritten valid | Verified | - | S26 | `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition`; `rm::state_recovery::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`; `rm::ref_reconciliation_unreadable_previous_state_is_reported_without_overwriting`; `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome` | Pass-level | The explicit pass runs and reports its true result from failed streak, aged advice, invalid timestamps, corrupt/oversized state and unwritable state; only the completed/failed/busy cases are driven through `sce doctor --fix` | Partially verified | + | S26 | `app::doctor_reconciliation_cli_tests::doctor_fix_recovers_from_every_initial_maintenance_state_condition` (failed streak, expired advisory, invalid far-future timestamps, corrupt JSON, oversized file → real `sce doctor --fix`); `app::doctor_reconciliation_cli_tests::doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable` and the unreadable leg of the table test (shared assertion helper); pass-level: `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition`, `…::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`, `rm::ref_reconciliation_unreadable_previous_state_is_reported_without_overwriting`, `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome` | CLI end-to-end (real repo, real Agent Trace DB, real `sce doctor --fix --format json`); pass-level | From each readable condition the explicit pass runs regardless of advisory eligibility, reclaims the orphan pin, reports `completed` (deleted 1), clears the failure streak and failure fields, sets `last_success`, normalizes far-future timestamps (all `≤ now + FUTURE_SKEW_TOLERANCE`), leaves a state a second `--fix` reads as valid, and exits successfully. From an unreadable state path (a directory) the pass still deletes the orphan, reports `completed_state_persist_failed` with `previous_state_unreadable`, makes no false persistence claim, leaves the directory untouched, and exits successfully | Verified | | S27 | `rm::state_recovery::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`; `rm::ref_reconciliation_failure_streak_counts_but_lock_contention_never_does` | Pass-level | Success resets streak and sets `last_success`; failure increments without touching `last_success`; busy changes nothing | Verified | | S28 | `rm::ref_reconciliation_missing_database_creates_nothing_and_deletes_nothing`; `rm::ref_reconciliation_verified_open_never_creates_missing_database_or_directories`; `rm::ref_reconciliation_verified_open_rejects_database_removed_before_open`; `app::doctor_reconciliation_cli_tests::doctor_fix_reports_missing_database_without_creating_it_or_deleting_refs` | Pass-level; opener; CLI | Typed `Missing`; no DB file, directory or sidecar created by the open; no refs listed or deleted. The CLI shows the parent directory comes from the separate global-state repair | Verified | | S29 | `rm::state_recovery::ref_reconciliation_verification_failures_leave_the_logical_database_unchanged` (case `incompatible`); `rm::ref_reconciliation_missing_metadata_or_schema_fails_before_pin_inventory` | Pass-level | `IncompatibleSchema`; full-row dump of every table equal before and after; no pin inventory; refs and marker untouched | Verified | From 85d368d0109d6aa91b09482f6883a15d4a964926 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 04:18:21 +0200 Subject: [PATCH 12/16] runtime: Distinguish uncertain advisory-state writes Preserve the persistence failure phase when advisory anchor or normalization writes complete replacement but cannot confirm directory durability. Render a durability-specific diagnostic for those outcomes, while retaining the existing write-failed wording for pre-rename failures and adding regression coverage for visible post-rename state. Update the reconciliation plan's T05 durability semantics and scenario coverage to match the outcome mapping. Plan: mutation-cursor-ref-reconciliation-wiring (T05) Co-authored-by: SCE --- .../mutation_trace/runtime/ref_advisory.rs | 110 +++++++++++++++++- context/cli/mutation-scope-ref-advisory.md | 10 +- ...tation-cursor-ref-reconciliation-wiring.md | 9 +- 3 files changed, 122 insertions(+), 7 deletions(-) diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs index b915d9978..b8a9aef8d 100644 --- a/cli/src/services/mutation_trace/runtime/ref_advisory.rs +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -12,6 +12,8 @@ pub(super) const RECONCILIATION_RECOMMENDED_MESSAGE: &str = const DIAGNOSTIC_PREFIX: &str = "SCE: "; const DURABILITY_UNCERTAIN_DIAGNOSTIC: &str = "Advisory-state durability could not be confirmed"; const STATE_WRITE_FAILED_DIAGNOSTIC: &str = "Reconciliation advisory state could not be written"; +const STATE_DURABILITY_UNCONFIRMED_DIAGNOSTIC: &str = + "Reconciliation advisory-state durability could not be confirmed"; const STATE_UNAVAILABLE_DIAGNOSTIC: &str = "Reconciliation advisory state is unavailable"; const CHECK_DOCTOR_HINT: &str = "Run sce doctor to check reconciliation status."; @@ -101,9 +103,16 @@ impl AdvisoryOutcome { "{DIAGNOSTIC_PREFIX}{RECONCILIATION_RECOMMENDED_MESSAGE}\n\ {DIAGNOSTIC_PREFIX}{DURABILITY_UNCERTAIN_DIAGNOSTIC}: {warning}." )), - AdvisoryOutcome::StateWriteFailed { warning } => Some(format!( + AdvisoryOutcome::StateWriteFailed { + warning: warning @ PersistFailure::NotApplied { .. }, + } => Some(format!( "{DIAGNOSTIC_PREFIX}{STATE_WRITE_FAILED_DIAGNOSTIC}: {warning}. {CHECK_DOCTOR_HINT}" )), + AdvisoryOutcome::StateWriteFailed { + warning: warning @ PersistFailure::DurabilityUncertain { .. }, + } => Some(format!( + "{DIAGNOSTIC_PREFIX}{STATE_DURABILITY_UNCONFIRMED_DIAGNOSTIC}: {warning}. {CHECK_DOCTOR_HINT}" + )), AdvisoryOutcome::StateUnavailable => Some(format!( "{DIAGNOSTIC_PREFIX}{STATE_UNAVAILABLE_DIAGNOSTIC}. {CHECK_DOCTOR_HINT}" )), @@ -194,7 +203,7 @@ mod tests { use std::process::Command; use super::super::maintenance_state::{ - evaluate_recommendation, read_state, record_failure, record_success, state_path, + evaluate, evaluate_recommendation, read_state, record_failure, record_success, state_path, write_state_atomically, write_state_atomically_with, AttemptOutcome, FaultInjectingFilesystem, MaintenanceState, PersistFailure, PersistPhase, StateRead, StoredReport, FUTURE_SKEW_TOLERANCE_MS, RECONCILIATION_ADVISORY_AFTER_MS, @@ -307,6 +316,22 @@ mod tests { assert!(diagnostic.starts_with("SCE: Reconciliation advisory state could not be written: ")); assert!(diagnostic.contains(&warning)); assert!(!diagnostic.contains(RECONCILIATION_RECOMMENDED_MESSAGE)); + + let report = AdvisoryOutcome::StateWriteFailed { + warning: uncertain(), + } + .report(); + assert_eq!(report.outcome, "state_write_failed"); + assert_eq!(report.severity, AdvisorySeverity::Warn); + assert_eq!(report.recommendation, None); + let warning = report.warning.expect("persistence warning"); + let diagnostic = report.diagnostic.expect("diagnostic"); + assert!(diagnostic + .starts_with("SCE: Reconciliation advisory-state durability could not be confirmed: ")); + assert!(diagnostic.ends_with("Run sce doctor to check reconciliation status.")); + assert!(diagnostic.contains(&warning)); + assert!(!diagnostic.contains("could not be written")); + assert!(!diagnostic.contains(RECONCILIATION_RECOMMENDED_MESSAGE)); } #[test] @@ -575,6 +600,9 @@ mod tests { )); assert_eq!(outcome.recommendation(), None); assert!(outcome.persistence_warning().is_some()); + let diagnostic = outcome.report().diagnostic.expect("diagnostic"); + assert!(diagnostic.starts_with("SCE: Reconciliation advisory state could not be written: ")); + assert!(!diagnostic.contains("durability")); assert_eq!(std::fs::read(&path).expect("state bytes"), before); assert!(evaluate_recommendation(&stored(dir.path()), due).recommended); assert!(matches!(advise(dir.path(), due), AdvisoryOutcome::Advised)); @@ -600,6 +628,16 @@ mod tests { Some(RECONCILIATION_RECOMMENDED_MESSAGE) ); assert!(outcome.persistence_warning().is_some()); + let report = outcome.report(); + assert_eq!( + report.recommendation, + Some(RECONCILIATION_RECOMMENDED_MESSAGE) + ); + let diagnostic = report.diagnostic.expect("diagnostic"); + assert!(diagnostic.starts_with(&format!( + "SCE: {RECOMMENDED}\nSCE: Advisory-state durability could not be confirmed: " + ))); + assert!(!diagnostic.contains("could not be written")); let visible = stored(dir.path()); assert_eq!(visible.last_advised, Some(due)); @@ -793,6 +831,74 @@ mod tests { assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored)); } + #[test] + fn maintenance_anchor_post_rename_failure_is_state_write_failed_with_visible_anchor() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + + let outcome = advise_with_fault(dir.path(), NOW, PersistPhase::SyncParentDirectory); + + assert!(matches!( + &outcome, + AdvisoryOutcome::StateWriteFailed { + warning: PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + .. + } + } + )); + assert_eq!(outcome.recommendation(), None); + let report = outcome.report(); + assert!(report.warning.is_some()); + let diagnostic = report.diagnostic.expect("diagnostic"); + assert!(diagnostic + .starts_with("SCE: Reconciliation advisory-state durability could not be confirmed: ")); + assert!(!diagnostic.contains("could not be written")); + assert_eq!(stored(dir.path()).anchor, Some(NOW)); + + let path = state_path(&dir.path().join(".git")); + let bytes = std::fs::read(&path).expect("state bytes"); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); + assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); + } + + #[test] + fn maintenance_no_action_normalization_post_rename_failure_is_state_write_failed() { + let dir = tempfile::tempdir().expect("temp dir"); + init_repo(dir.path()); + let invalid = MaintenanceState { + anchor: Some(NOW), + last_advised: Some(NOW + 10 * RECONCILIATION_ADVISORY_AFTER_MS), + ..MaintenanceState::default() + }; + seed(dir.path(), &invalid); + assert!(!evaluate(&invalid, NOW).advise); + + let outcome = advise_with_fault(dir.path(), NOW, PersistPhase::SyncParentDirectory); + + assert!(matches!( + &outcome, + AdvisoryOutcome::StateWriteFailed { + warning: PersistFailure::DurabilityUncertain { + phase: PersistPhase::SyncParentDirectory, + .. + } + } + )); + assert_eq!(outcome.recommendation(), None); + let diagnostic = outcome.report().diagnostic.expect("diagnostic"); + assert!(diagnostic.contains("durability could not be confirmed")); + let visible = stored(dir.path()); + assert_eq!(visible.anchor, Some(NOW)); + assert_eq!(visible.last_advised, None); + assert_within_skew_tolerance(&visible, NOW); + + let path = state_path(&dir.path().join(".git")); + let bytes = std::fs::read(&path).expect("state bytes"); + assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); + assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); + } + #[test] fn maintenance_advisory_recovers_corrupt_state_by_anchoring() { let dir = tempfile::tempdir().expect("temp dir"); diff --git a/context/cli/mutation-scope-ref-advisory.md b/context/cli/mutation-scope-ref-advisory.md index 323bc76d9..b7bfd2c5b 100644 --- a/context/cli/mutation-scope-ref-advisory.md +++ b/context/cli/mutation-scope-ref-advisory.md @@ -38,9 +38,17 @@ writer. Stdout stays empty and never carries advisory text. | `Busy` | debug | none | | `Advised` | warn | `SCE: Reconciliation is recommended. Run sce doctor --fix.` | | `AdvisedDurabilityUncertain` | warn, with `persistence_warning` | the line above, then `SCE: Advisory-state durability could not be confirmed: .` | -| `StateWriteFailed` | warn, with `persistence_warning` | `SCE: Reconciliation advisory state could not be written: . Run sce doctor to check reconciliation status.` (does not claim advice was persisted) | +| `StateWriteFailed` + `NotApplied` | warn, with `persistence_warning` | `SCE: Reconciliation advisory state could not be written: . Run sce doctor to check reconciliation status.` (replacement did not occur; previous state preserved; no advice) | +| `StateWriteFailed` + `DurabilityUncertain` | warn, with `persistence_warning` | `SCE: Reconciliation advisory-state durability could not be confirmed: . Run sce doctor to check reconciliation status.` (anchor or normalization write; replacement occurred, new state may be visible, crash durability unconfirmed; no advice) | | `StateUnavailable` | warn | `SCE: Reconciliation advisory state is unavailable. Run sce doctor to check reconciliation status.` | +`StateWriteFailed` means no confirmed successful advisory-state update, not +necessarily that atomic replacement failed; it carries the `PersistFailure` so the +two phases stay distinguishable. `AdvisedDurabilityUncertain` is reserved for a +due recommendation whose advice-state replacement occurred with uncertain +durability (the recommendation is still emitted). `StateUnavailable` means the +state could not be accessed or evaluated and is not a persistence-write outcome. + The diagnostic does not depend on the logger (it is emitted with `logger = None`), and `Logger::warn` and the observability defaults are unchanged: observability routes only ERROR records to stderr, and only when file logging is off, so WARN diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 7d1d82c6e..af105b64e 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -85,7 +85,7 @@ Facts the design relies on. T01 must re-read each cited location and either conf - *Selected design.* - `Close`/`Flush` (durably completed, including the `MarkerClearAfterCommit` case, after `coordinate_boundary` returned) calls `advise_if_due`, a synchronous function with this complete I/O budget: one capped read (≤ `MAINTENANCE_STATE_MAX_BYTES`) of the state file, at most one zero-wait `try_lock`, at most one atomic write of that small file, all local. It performs no `git` subprocess, no DB open or query, no ref listing, no `spawn_blocking`, no await point after the lock is taken. It is a bounded set of small local file operations of the same class the hook path already performs for the lock and taint marker; it is not claimed to be hard real-time, and any unexpected failure is swallowed into a logged outcome. - The advisory must not add a new Git subprocess. It obtains the worktree's Git dir without spawning Git (T01 determines whether it can be derived from the filesystem — `.git` directory, or `gitdir:` file for linked worktrees — and proves equality with `resolve_git_dir` for the main and a linked worktree, or whether `coordinate`'s already-resolved value can be reused without protocol changes). **Decision rule:** if neither is possible without a new Git subprocess, the hook trigger is dropped entirely and `sce doctor` computes the recommendation from the state file alone; AC1/AC5 are then satisfied with zero hook callers and T03 reduces to documenting that outcome. This fallback needs no user confirmation because it only removes a convenience signal. - - `advise_if_due` outcomes: absent/invalid-version/oversized/corrupt state → write an anchor and return `Anchored` (no advice). Reference time (`last_success`, else `anchor`) older than `RECONCILIATION_ADVISORY_AFTER` and no valid advice within the same window → persist `last_advised`, then (only after the write succeeded and its persistence was confirmed) return `Advised`. `AdvisedDurabilityUncertain`: the atomic replacement happened (the advisory state may be visible) but crash durability could not be confirmed; an actionable diagnostic is emitted and durability is never claimed. `StateWriteFailed`: the state update failed before atomic replacement (`PersistFailure::NotApplied`); the previous bytes are intact, no advice is emitted, and no scan runs. `StateUnavailable`: the maintenance state cannot be accessed or evaluated (unresolvable Git dir, unreadable state, lock I/O error); it is not a write outcome, emits no advice, and has no retry loop. `NoAction` (nothing due, no write) and `Busy` (zero-wait lock contention, skipped, not an error) emit no actionable advisory. + - `advise_if_due` outcomes: absent/invalid-version/oversized/corrupt state → write an anchor and return `Anchored` (no advice). Reference time (`last_success`, else `anchor`) older than `RECONCILIATION_ADVISORY_AFTER` and no valid advice within the same window → persist `last_advised`, then (only after the write succeeded and its persistence was confirmed) return `Advised`. `AdvisedDurabilityUncertain`: the atomic replacement happened (the advisory state may be visible) but crash durability could not be confirmed; an actionable diagnostic is emitted and durability is never claimed. `StateWriteFailed` means no confirmed successful advisory-state update and preserves the `PersistFailure`: with `NotApplied` the replacement did not occur and the previous bytes are intact; with `DurabilityUncertain` (only for an `Anchored` or `NoAction` write, i.e. an anchor or normalization) the replacement occurred, the new state may be visible, crash durability is unconfirmed, and no advice is emitted. A due `Advised` decision with `NotApplied` is `StateWriteFailed`; with `DurabilityUncertain` it is `AdvisedDurabilityUncertain` (an actual recommendation was due and the advice-state replacement occurred). `StateWriteFailed` never emits a recommendation and no scan runs. `StateUnavailable`: the maintenance state cannot be accessed or evaluated (unresolvable Git dir, unreadable state, lock I/O error); it is not a write outcome, emits no advice, and has no retry loop. `NoAction` (nothing due, no write) and `Busy` (zero-wait lock contention, skipped, not an error) emit no actionable advisory. - *Cadence.* The normal time-based reminder for an unresolved condition repeats at most once per `RECONCILIATION_ADVISORY_AFTER` (24 h) window per worktree. The one exception is a newly recorded explicit reconciliation failure (`last_attempt` newer than `last_advised`): it may trigger one immediate advisory inside an otherwise active 24 h window, is suppressed once `last_advised` records it, and then follows the normal 24 h cadence until a successful explicit pass clears it. The advisory is therefore not unconditionally one-per-window. - `sce doctor` (read-only) reports whether reconciliation is recommended (last success age, last attempt/failure streak, last advice) computed from the state file as a pure function; `sce doctor --fix` runs `reconcile_explicit` regardless of any state. - *Deferred to a follow-up PR (explicitly not in this plan).* Fully automatic out-of-band reconciliation. Binding prerequisites for that PR: (1) a lifecycle-owned execution point with defined ownership, lifetime, scheduling and failure recovery (for example an explicitly invoked `sce` maintenance command run by a user-managed scheduler, or a future shared session-end mechanism) — never a detached task, daemon, or timeout-cancelled Git operation; (2) a **durable next-attempt reservation persisted under `WorktreeLock` before opening the DB, listing pins, or loading durable roots**, bounded, never interpreted as success, superseded by the success cadence after success or exponential failure backoff after failure, and bypassed by explicit repair; if the reservation cannot be persisted the automatic pass is skipped and the maintenance-state failure is reported; (3) distinct outcomes for completed, completed-but-state-not-persisted, failed, and skipped/deferred; (4) bounded or split lock-hold analysis against the 10 s coordinator timeout; (5) the same invalid-timestamp normalization as D3. T06 records this follow-up in context. @@ -141,7 +141,7 @@ How this plan is proven complete. `/validate` runs these checks; no task perform - [ ] AC12: The hook completion path contains no unbounded maintenance work, and the documented concurrency contract for explicit reconciliation holds. - Validate: T05 S14, S15, S16 and S33 pass, as three distinct contention scenarios. S15 (direct advisory contention): with an explicit pass parked mid-inventory or mid-DB holding the lock, `advise_if_due` called directly returns `Busy` immediately and changes no state or refs. S16 (coordinator contention): with the lock held, `coordinate()` keeps its existing bounded lock acquisition; a short injected timeout fails with the existing lock-timeout error before durable completion, the advisory does not run after the failed coordinate, and the same boundary succeeds once the pass is released. S33 (post-coordinate handoff race): after a successful `coordinate()` released its lock, a competing explicit pass takes the lock before the paused advisory resumes; the advisory returns `Busy` and the committed boundary still returns success. The advisory takes no Git/DB parameters. No timeout, budget, or size cap is claimed to bound hook latency. - [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. - - Two guarantee classes plus a distinct uncertain-write outcome (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, state that cannot be read or evaluated (`StateUnavailable`, which is not a failed write), a write that fails before the atomic rename (`NotApplied`), a write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **`NotApplied`** means the failure happened before the rename: the previous state remains intact and byte-identical, the advisory reports `StateWriteFailed`, and no successful advisory is claimed. **`DurabilityUncertain`** means the failure happened after the rename: the new state may already be visible, crash durability is unconfirmed, the advisory reports `AdvisedDurabilityUncertain` when the recommendation was being recorded (an actionable diagnostic remains available), and a normalization written this way is never claimed to be durably persisted; a later invocation recovers safely in either case. Explicit reconciliation retains `CompletedStatePersistFailed` and the typed persistence-warning phase. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. + - Two guarantee classes plus a distinct uncertain-write outcome (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, state that cannot be read or evaluated (`StateUnavailable`, which is not a failed write), a write that fails before the atomic rename (`NotApplied`), a write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **`NotApplied`** means the failure happened before the rename: the previous state remains intact and byte-identical, the advisory reports `StateWriteFailed`, and no successful advisory is claimed. **`DurabilityUncertain`** means the failure happened after the rename: the new state may already be visible, crash durability is unconfirmed, the advisory reports `AdvisedDurabilityUncertain` when the recommendation was being recorded (an actionable diagnostic remains available) and `StateWriteFailed` carrying `DurabilityUncertain` when only an anchor or normalization was being written (no recommendation is emitted), and a normalization written this way is never claimed to be durably persisted; a later invocation recovers safely in either case. Explicit reconciliation retains `CompletedStatePersistFailed` and the typed persistence-warning phase. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. - Validate: T05 S19–S25 and S34–S39 pass: single time-based advice per 24 h window with persistence before emission (a newly recorded failure may be advised inside that window, once, per AC5); concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's pre-rename leg assert byte-identical preexisting state instead, and S36's post-rename leg asserts only that the new state may be visible with durability reported as uncertain; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write during normalization, with both a pre-rename leg (previous bytes unchanged, `StateWriteFailed`) and a post-rename leg (visibility may have changed, `AdvisedDurabilityUncertain`, no durable-normalization claim), a later invocation recovering safely in both; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. - [ ] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability (incidental WAL bookkeeping on existing databases is permitted and documented), and only the missing-DB case asserts no newly created database-related files or directories, including when the file is removed between a preliminary check and the open. Plain `sce doctor` shares the same non-creating read-only open. - **Missing DB:** the non-creating read-only Turso open fails without creating the DB file, its parent directory, or database sidecars (WAL/shared-memory); no Git ref is listed or deleted. A separate `sce doctor --fix` repair (global-state bootstrap) may create the parent directory independently of reconciliation; reconciliation itself creates nothing. @@ -297,7 +297,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - Remediation 2 (2026-10-09, follow-up review of `de055d70`; T02 reopened `done`→`in_progress`, returned to `done`/`synced` after the verification below; T01 unchanged and still done/synced, T03–T06 untouched). Architecture unchanged: reconciliation runs only through `reconcile_explicit`; the advisory stays lightweight (no Git, DB, or reconciliation), takes one zero-wait lock acquisition, uses static dispatch, and no mutation protocol, attribution, Quint, or ref-deletion code changed. - P2 failure-based recommendation: `evaluate_recommendation` now treats an outstanding explicit-pass failure (`last_attempt_outcome == Failed` or `consecutive_failures > 0`) as an independent reason to recommend repair, in addition to staleness and invalid timestamps. A first-ever failed pass (no anchor, no `last_success`) is therefore recommended immediately. A never-attempted default state is not a failure. Multiple consecutive failures stay recommended; `record_success` clears the streak, `last_failure`, and the recommendation; a failure after a recent success is recommended again. Emitting an advisory never resets `consecutive_failures` or `last_failure`. - Recommendation status vs reminder cadence: `Recommendation.recommended` answers "should `sce doctor` tell the operator to run `sce doctor --fix`" and ignores `last_advised`. `Evaluation.advise` answers "may the hook emit another reminder": staleness reminders are suppressed for 24 h after `last_advised`; a failure reminder is suppressed only when `last_advised` is within the window and not older than `last_attempt`, so a newly recorded failure can be advised immediately even if an earlier staleness reminder is recent, while a reminder issued after the failure is not repeated inside the window. A failure reminder repeats every 24 h until a successful explicit pass. Clock-skew normalization and the existing cadence are unchanged. - - P2 persistence outcomes: `AdvisoryOutcome` is now `Anchored | NoAction | Advised | AdvisedDurabilityUncertain { warning: PersistFailure } | Busy | StateWriteFailed { warning: PersistFailure } | StateUnavailable` (no longer `Copy`/`Eq` because it carries `PersistFailure`). `Advised` means the reminder state was persisted with confirmed durability. `PersistFailure::NotApplied` (up to and including rename) on an advisory yields `StateWriteFailed`: the previous state is intact, nothing is claimed persisted, `recommendation()` is `None`, and the next eligible call advises again. `PersistFailure::DurabilityUncertain` (parent-directory open or sync after rename) yields `AdvisedDurabilityUncertain`: the rename completed, `last_advised` may be visible, and the caller must still surface the recommendation. No rollback, second lock acquisition, or automatic reconciliation. A write failure while only anchoring or normalizing yields `StateWriteFailed` carrying whichever `PersistFailure` variant occurred; there is nothing to surface. + - P2 persistence outcomes: `AdvisoryOutcome` is now `Anchored | NoAction | Advised | AdvisedDurabilityUncertain { warning: PersistFailure } | Busy | StateWriteFailed { warning: PersistFailure } | StateUnavailable` (no longer `Copy`/`Eq` because it carries `PersistFailure`). `Advised` means the reminder state was persisted with confirmed durability. `PersistFailure::NotApplied` (up to and including rename) on an advisory yields `StateWriteFailed`: the previous state is intact, nothing is claimed persisted, `recommendation()` is `None`, and the next eligible call advises again. `PersistFailure::DurabilityUncertain` (parent-directory open or sync after rename) yields `AdvisedDurabilityUncertain`: the rename completed, `last_advised` may be visible, and the caller must still surface the recommendation. No rollback, second lock acquisition, or automatic reconciliation. A write failure while only anchoring or normalizing yields `StateWriteFailed` carrying whichever `PersistFailure` variant occurred; there is no recommendation to surface. With `NotApplied` the previous state is intact; with `DurabilityUncertain` the rename completed, the anchor or normalized state may be visible, and only crash durability is unconfirmed. The diagnostic text follows the variant ("could not be written" vs "durability could not be confirmed"). - Caller contract for T03: `AdvisoryOutcome::recommendation()` returns `Reconciliation is recommended. Run sce doctor --fix.` for `Advised` and `AdvisedDurabilityUncertain`, and `None` otherwise; `persistence_warning()` returns the `PersistFailure` separately for logging. Outcomes are diagnostic values, not errors: T03 must log them and never convert them into a mutation-boundary failure. Suggested levels: `Advised` and `AdvisedDurabilityUncertain` (with the persistence warning as a distinct field) at warn; `StateWriteFailed` and `StateUnavailable` at warn; `Anchored`/`NoAction`/`Busy` at debug. - Delivery semantics and limitations: a durable state update is not the same as an operator-visible advisory. The state is written before the caller surfaces the message, so a process crash between the write and the log leaves `last_advised` set and the reminder unseen for up to 24 h; exactly-once delivery is not promised. `sce doctor` is the backstop because it evaluates the recommendation from the failure/staleness fields independently of `last_advised`, so an unresolved failure or stale state is never hidden from plain `sce doctor`. After `AdvisedDurabilityUncertain`, a crash can still roll the directory entry back, in which case the next eligible call simply advises again. - Regression scenarios and acceptance criteria (all deterministic, injected clocks and `FaultInjectingFilesystem`, no sleeps): first-ever failed pass with missing DB is recommended (unit and end-to-end through `reconcile_explicit`); failure advisory emits once, persists `last_advised`, keeps the failure diagnostics, and a second advisory inside the window is `NoAction`; plain evaluation stays recommended despite the recent reminder; the reminder repeats after the window; repeated failures stay recommended; success clears; a failure after a recent success and an older reminder is recommended and advised; rename failure preserves prior state and the next call advises; parent-open and parent-sync failures yield `AdvisedDurabilityUncertain` with the new state visible, the recommendation surfaced, a following call `NoAction`, and `evaluate_recommendation` still true; an anchor write failure is `StateWriteFailed`; confirmed advisories keep the ordinary duplicate-suppression tests. @@ -360,6 +360,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - Verification, remediation pass (2026-10-09): full CLI suite 495 passed, 0 failed (two new doctor CLI tests); focused `doctor_reconciliation_cli` 7, `ref_reconciliation` 50, `mutation_scope` 25, `maintenance` 85 (all non-zero); new doctor CLI tests 5 consecutive passes; every `ref_maintenance::tests` test and the cancellation tests 20 consecutive passes; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; `cargo fmt` made no change; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect). - Verification (earlier pass, 2026-10-09): full CLI suite 493 passed, 0 failed; focused `ref_reconciliation` 50, `mutation_scope` 25, `doctor` 25, `maintenance` 83 (all non-zero); `clippy --all-targets --all-features -D warnings` clean; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect); `git diff --check` clean; 20 consecutive passes of every `ref_maintenance::tests::*` and cancellation test, 5 consecutive passes of the doctor CLI tests; no added comments, sleeps, `allow(dead_code)`, `dyn`, `BoxFuture`, detached tasks or extra production runtimes. - Guard-removal checks actually performed: repository-wide roots replaced by worktree-local roots → S3 and S13 tests fail; deletion worker drops its lease → the new deletion-cancellation test and the existing pin-cancellation test fail; advisory ignores lock contention → S15 (both phases) and S33 fail. The remaining guards (S6, S16, S17, S18b, S21) were checked in the remediation pass; see the results table below. + - Remediation (2026-10-09, PR #305 review of `982976f4`, advisory persistence semantics; T01–T06 statuses unchanged): `AdvisoryOutcome::diagnostic()` now follows the carried `PersistFailure`. `StateWriteFailed` + `NotApplied` keeps `SCE: Reconciliation advisory state could not be written: . Run sce doctor to check reconciliation status.`; `StateWriteFailed` + `DurabilityUncertain` (post-rename anchor or normalization write) reads `SCE: Reconciliation advisory-state durability could not be confirmed: . Run sce doctor to check reconciliation status.` and never claims the update was not written or that advice was emitted. Outcome mapping, scheduling, locking, and Git-ref reconciliation are unchanged; no new variants or seams. Added tests in `ref_advisory::tests`: `maintenance_anchor_post_rename_failure_is_state_write_failed_with_visible_anchor` (absent state, `SyncParentDirectory` fault, anchor visible, next call `NoAction` with byte-identical state), `maintenance_no_action_normalization_post_rename_failure_is_state_write_failed` (fresh anchor plus far-future `last_advised`, no advice due; invalid timestamp removed; next call `NoAction`, no rewrite); extended `maintenance_report_maps_every_outcome_to_severity_and_diagnostic` (uncertain `StateWriteFailed`), `maintenance_advisory_rename_failure_preserves_state_and_next_call_can_advise` (pre-rename wording) and `assert_durability_uncertain_advisory` (recommendation retained, durability wording). Verification: see the final verification record for this remediation. - Remediation (2026-10-09, PR #305 review of `39f78941`; no gap was waived and no production code changed): (1) S9 persistence-warning outcome and S26 recovery table now run through the real `sce doctor --fix` (`doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable`, `doctor_fix_recovers_from_every_initial_maintenance_state_condition`), using a naturally unreadable state (a directory at `.git/sce/ref-maintenance.json`) rather than a fault seam; `NotApplied` and `DurabilityUncertain` stay at the persistence layer (`ref_advisory::tests`, `rm::state_recovery`), with no CLI fault seam. (2) The doctor CLI child now captures the `ExitCode` returned by `run_with_dependency_check_and_streams`, writes `success`/`failure` to an `exit` file beside `stdout`/`stderr`, `CliRun` exposes it, and every doctor invocation asserts it. The CLI contract (unchanged): `render_report` always returns `Ok`, so plain doctor and `--fix` exit successfully for completed, failed-fix, busy and persistence-warning outcomes; an individual failed fix is reported in the payload, not in the exit status. (3) S14 is verified by static call-graph inspection plus real-hook ref-delta assertions (see the S14 row); no spy/panic seams were added. (4) The five remaining guard-removal checks were performed in a throwaway `git worktree` at `39f78941` (shared `CARGO_TARGET_DIR`, removed afterwards); results are in the table below. (5) D2 now distinguishes `StateUnavailable`, `StateWriteFailed`, `AdvisedDurabilityUncertain`, `Advised`, `NoAction` and `Busy`, and documents the single immediate advisory for a newly recorded failure; D3, AC5, AC13, S20 and S36 already agreed and were not changed. - Guard-removal results for the five remaining guards (actual outcomes; every mutation lived only in the throwaway worktree and none is in the diff): @@ -415,7 +416,7 @@ Record actual outcomes in the validation report; compilation alone does not vali | S33 | `rm::coordination::ref_reconciliation_advisory_after_a_real_commit_is_busy_when_a_pass_wins_the_handoff` | Concurrency integration (real coordinator; hook parked between coordinate and advisory; pass on its own thread) | Pass takes the lock in the handoff window; advisory returns `Busy`; state and refs unchanged by the advisory; committed `Close` still reports success; only the orphan is deleted by the pass | Verified | | S34 | `ref_doctor::tests::recommendation_flags_unusable_and_invalid_state_without_rewriting` | Unit (doctor inspection) | Read-only doctor flags invalid timestamps and leaves bytes untouched | Verified | | S35 | `ref_advisory::tests::maintenance_advisory_reports_busy_and_leaves_invalid_state_byte_identical`; `rm::state_recovery::ref_reconciliation_busy_explicit_pass_leaves_invalid_state_byte_identical` | Unit; pass-level | Lock-busy advisory and explicit pass leave invalid state byte-identical | Verified | - | S36 | `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}`; `rm::state_recovery::ref_reconciliation_failed_state_write_during_normalization_has_two_distinct_legs` | Unit; pass-level | Pre-rename: previous bytes identical, `StateWriteFailed`/`NotApplied`. Post-rename: new state visible, `AdvisedDurabilityUncertain`/`DurabilityUncertain`, no durable-normalization claim; later invocation recovers | Verified | + | S36 | `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}`; `rm::state_recovery::ref_reconciliation_failed_state_write_during_normalization_has_two_distinct_legs` | Unit; pass-level | Pre-rename: previous bytes identical, `StateWriteFailed`/`NotApplied`. Post-rename: new state visible, `AdvisedDurabilityUncertain`/`DurabilityUncertain` when the normalization also recorded advice, or `StateWriteFailed`/`DurabilityUncertain` with no recommendation for an anchor or `NoAction` normalization (`ref_advisory::tests::{maintenance_anchor_post_rename_failure_is_state_write_failed_with_visible_anchor, maintenance_no_action_normalization_post_rename_failure_is_state_write_failed}`); no durable-normalization claim; later invocation recovers | Verified | | S37 | `ref_advisory::tests::maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately`; `rm::state_recovery::ref_reconciliation_successful_pass_normalizes_invalid_state_and_a_later_read_does_not_rewrite` | Unit; pass-level | Normalizing advisory and explicit pass persist `t ≤ now + tolerance`; near-future value preserved; repeated read does not rewrite | Verified | | S38 | `ref_advisory::tests::maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately` (fresh call re-reads the file); `rm::cross_process::ref_reconciliation_advisory_lock_serializes_independent_processes` | Unit; independent processes | A fresh call observes normalized state, performs no rewrite and does not re-fire | Verified | | S39 | `rm::state_recovery::ref_reconciliation_unnormalized_invalid_state_stays_eligible_until_a_successful_advisory` | Pass-level (injected clocks, held lock) | Repeated read-only evaluation and busy advisories never move eligibility or bytes; the next successful advisory fires exactly once and persists normalized state | Verified | From e96b9b3a22f6ea2f7aa8b05f8a05edb97cb76f11 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 08:55:55 +0200 Subject: [PATCH 13/16] tests+context: Verify missing maintenance DB opens without creation Add regression coverage proving that maintenance opening of a missing Agent Trace database returns the typed missing-database error without creating state-root entries. Mark the mutation-cursor reconciliation plan's acceptance criteria complete and record validation evidence for the completed implementation. Plan: mutation-cursor-ref-reconciliation-wiring Tasks: T05, T06 Co-authored-by: SCE --- cli/src/services/agent_trace_storage/mod.rs | 41 ++++++++++ ...tation-cursor-ref-reconciliation-wiring.md | 75 +++++++++++++++---- 2 files changed, 101 insertions(+), 15 deletions(-) diff --git a/cli/src/services/agent_trace_storage/mod.rs b/cli/src/services/agent_trace_storage/mod.rs index 79e0cc313..9d0a103c8 100644 --- a/cli/src/services/agent_trace_storage/mod.rs +++ b/cli/src/services/agent_trace_storage/mod.rs @@ -253,3 +253,44 @@ pub(crate) async fn resolve_existing_agent_trace_storage_for_maintenance_at_stat )?; open_existing_storage_for_maintenance(repository_identity, db_path).await } + +#[cfg(test)] +mod tests { + use super::{ + resolve_existing_agent_trace_storage_for_maintenance_at_state_root, + AgentTraceStorageContext, + }; + use crate::services::agent_trace_db::repository::ExistingRepositoryDbError; + + #[tokio::test(flavor = "multi_thread")] + async fn agent_trace_storage_maintenance_open_of_missing_db_creates_nothing() { + let repository = tempfile::tempdir().expect("repository dir"); + let state_root = tempfile::tempdir().expect("state root"); + let context = AgentTraceStorageContext { + repository_root: repository.path(), + explicit_repository_id: Some("maintenance-open-missing"), + repository_remote: "origin", + }; + + let Err(error) = resolve_existing_agent_trace_storage_for_maintenance_at_state_root( + &context, + state_root.path(), + ) + .await + else { + panic!("a missing database must not open"); + }; + + assert!(matches!( + error.downcast_ref::(), + Some(ExistingRepositoryDbError::Missing { .. }) + )); + let entries: Vec<_> = std::fs::read_dir(state_root.path()) + .expect("state root listing") + .collect(); + assert!( + entries.is_empty(), + "state root must stay empty: {entries:?}" + ); + } +} diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index af105b64e..2fb946b31 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -113,42 +113,42 @@ Facts the design relies on. T01 must re-read each cited location and either conf How this plan is proven complete. `/validate` runs these checks; no task performs final validation. -- [ ] AC1: Reconciliation has one real non-test **explicit** caller (`sce doctor --fix`) and no automatic reconciliation caller; the mutation-scope ingress has at most an advisory caller that cannot reach reconciliation. +- [x] AC1: Reconciliation has one real non-test **explicit** caller (`sce doctor --fix`) and no automatic reconciliation caller; the mutation-scope ingress has at most an advisory caller that cannot reach reconciliation. - Validate: `nix shell nixpkgs#ripgrep -c rg "reconcile|advise_if_due" cli/src --glob '!*test*'` shows the reconciliation entrypoint referenced only from `doctor/` (and its definition module), and `advise_if_due` (if present per the D2 decision rule) referenced from `hooks/mutation_scope.rs`; the hook module imports no reconciliation, `GitSnapshotService`, store, or DB-opener symbol for this purpose. -- [ ] AC2: `sce doctor --fix` deletes an orphan snapshot ref; `Close`/`Flush` never deletes. +- [x] AC2: `sce doctor --fix` deletes an orphan snapshot ref; `Close`/`Flush` never deletes. - Validate: T05 S1 drives the doctor fix path against a real Git repo and asserts the orphan `refs/sce/mutation-cursor//` is gone; S14 asserts the advisory path creates, updates, deletes, and inventories no refs (ref changes made by `coordinate()` itself, such as snapshot pins, are legitimate and are not attributed to maintenance). This plan does **not** claim automatic orphan reclamation. -- [ ] AC3: A successful pass establishes two distinct invariants, and claims nothing stronger. +- [x] AC3: A successful pass establishes two distinct invariants, and claims nothing stronger. - **Local consistency:** `durable_roots(W) ⊆ pinned_trees(W)` is checked before any deletion for the target worktree `W`. If it fails, the pass deletes nothing and returns `MissingRequiredPins`. - **Repository-wide deletion safety:** `delete(W, T) ⇒ T ∉ durable_roots(repository)`. A pin is removed only when its tree is outside every worktree's durable roots, so reconciliation never removes an existing SCE pin that protects another worktree's required tree. - Reconciliation does **not** repair, recreate, or verify missing pins belonging to other worktrees, and does not guarantee that every repository-wide durable root is pinned. - Validate: T05 S2, S3, S4, S5 and S13 pass. S2/S3 prove preservation; S4 proves local fail-closed; S13 proves another worktree's missing pin is neither repaired nor treated as a failure of `W`'s pass. -- [ ] AC4: Hook-path maintenance (advisory) cannot change the outcome of an already durably committed mutation boundary. +- [x] AC4: Hook-path maintenance (advisory) cannot change the outcome of an already durably committed mutation boundary. - Validate: T05 S17, S18a, S18b and S33 pass: every advisory outcome (`Anchored`, `NoAction`, `Advised`, `AdvisedDurabilityUncertain`, `Busy`, `StateWriteFailed`, `StateUnavailable`, corrupt/invalid state, seam error) and a failing diagnostic-output writer leave the hook result identical to the coordinate-only baseline. The advisory never creates, clears, or modifies the external-taint marker. After normal completion the marker state is whatever the existing coordinator leaves (cleared) and is unchanged by the advisory (S18a). `MarkerClearAfterCommit` remains a durably completed boundary with a successful hook result; the marker state resulting from the failed clear is preserved by the advisory, and the next real runtime invocation still takes the existing conservative inherited-taint recovery path (S18b). `MarkerClearAfterCommit` is never treated as equivalent to a successful clear. -- [ ] AC5: The mutation-hook completion path performs no reconciliation, ref listing, DB I/O, or new Git subprocess, and its maintenance-state writes follow a precise cadence. Time-based reminders: a stale state becomes eligible after `RECONCILIATION_ADVISORY_AFTER` (24 h) and further stale-state reminders are suppressed for 24 h after `last_advised`. Failure-triggered reminders: an outstanding failed explicit reconciliation is recommended immediately; a newly recorded failure (`last_attempt` newer than `last_advised`) may be advised even inside the previous 24 h window and is suppressed once recorded; repeated reminders for the same unresolved failure follow the normal 24 h cadence; a successful explicit reconciliation clears the failure-based recommendation. Anchoring and clock-skew normalization are the only other writes. Actionable reminders reach an explicit diagnostic channel (stderr) in addition to structured logging; delivery is best-effort and neither exactly-once nor guaranteed to be seen by a human (T03 remediation, context/cli/mutation-scope-ref-advisory.md). +- [x] AC5: The mutation-hook completion path performs no reconciliation, ref listing, DB I/O, or new Git subprocess, and its maintenance-state writes follow a precise cadence. Time-based reminders: a stale state becomes eligible after `RECONCILIATION_ADVISORY_AFTER` (24 h) and further stale-state reminders are suppressed for 24 h after `last_advised`. Failure-triggered reminders: an outstanding failed explicit reconciliation is recommended immediately; a newly recorded failure (`last_attempt` newer than `last_advised`) may be advised even inside the previous 24 h window and is suppressed once recorded; repeated reminders for the same unresolved failure follow the normal 24 h cadence; a successful explicit reconciliation clears the failure-based recommendation. Anchoring and clock-skew normalization are the only other writes. Actionable reminders reach an explicit diagnostic channel (stderr) in addition to structured logging; delivery is best-effort and neither exactly-once nor guaranteed to be seen by a human (T03 remediation, context/cli/mutation-scope-ref-advisory.md). - Validate: T05 S14, S15, S19 and S20 pass (S14: the advisory performs no Git-ref creation, update, deletion, or inventory, no reconciliation, and no DB I/O; coordinator ref effects are legitimate and distinguished from it); `rg` over the advisory module shows no import of `GitSnapshotService`, `MutationTraceStore`, DB openers, `resolve_git_dir`, `resolve_worktree_id`, `spawn_blocking`, or `Command`. -- [ ] AC6: Contention, cancellation, and failed maintenance never violate the pin-to-CAS safety boundary. +- [x] AC6: Contention, cancellation, and failed maintenance never violate the pin-to-CAS safety boundary. - Validate: T05 S6, S7 and S12 pass using deterministic synchronization (no sleep-based proof): concurrent `coordinate()` cannot race pin deletion; dropping the explicit-pass future during a ref-deletion worker does not release the lock until the worker finishes; each operation completes without self-deadlock. -- [ ] AC7: No new unbounded in-memory registry, dynamic dispatch, detached background worker, timeout-cancelled Git work, or nested runtime. +- [x] AC7: No new unbounded in-memory registry, dynamic dispatch, detached background worker, timeout-cancelled Git work, or nested runtime. - Validate: `git diff fix-logging...HEAD` inspection plus `nix shell nixpkgs#ripgrep -c rg "dyn Future|BoxFuture|tokio::spawn|Runtime::new|block_on|tokio::time::timeout|JoinHandle|abort\(|static .*(Mutex|RwLock|OnceLock|HashMap)" ` returns no new hits. -- [ ] AC8: Mutation-cursor and Quint model-based tests remain valid without protocol changes. +- [x] AC8: Mutation-cursor and Quint model-based tests remain valid without protocol changes. - Validate: `git diff fix-logging...HEAD --stat` shows no change to `protocol.rs` or `spec/mutation_cursor.qnt`; the existing `mbt` tests pass. -- [ ] AC9: `sce doctor` stays non-mutating — it never creates or logically modifies the Agent Trace DB (its schema inspection uses the non-creating read-only open of D4) and never writes maintenance state or refs; `sce doctor --fix` reports its actual cleanup result, exposes it as a typed `ref_reconciliation_fix` JSON object (outcome, counts, failure kind, state-warning kind/phase) alongside the unchanged `fix_results`, distinguishes `NotApplied` from `DurabilityUncertain` persistence warnings, and recovers from every failure, stale, and invalid-state condition. +- [x] AC9: `sce doctor` stays non-mutating — it never creates or logically modifies the Agent Trace DB (its schema inspection uses the non-creating read-only open of D4) and never writes maintenance state or refs; `sce doctor --fix` reports its actual cleanup result, exposes it as a typed `ref_reconciliation_fix` JSON object (outcome, counts, failure kind, state-warning kind/phase) alongside the unchanged `fix_results`, distinguishes `NotApplied` from `DurabilityUncertain` persistence warnings, and recovers from every failure, stale, and invalid-state condition. - Validate: T05 S9, S25, S26 and S27 pass: counts for deleted/retained/local-required; distinct output for completed, completed-with-state-persist-failure, failed (unavailable/invalid DB, malformed refs, missing pins), and busy; plain `sce doctor` writes no ref and no state; a recommendation is shown when stale, when the latest explicit pass failed (regardless of `last_advised`), or when state is invalid. The doctor recommendation is independent of whether a hook reminder was ever seen, which is why lossy hook delivery is acceptable. -- [ ] AC10: Context matches the implementation, the "retained and unwired" status is gone, and context states that automatic reclamation is deferred to a follow-up with its prerequisites. +- [x] AC10: Context matches the implementation, the "retained and unwired" status is gone, and context states that automatic reclamation is deferred to a follow-up with its prerequisites. - Validate: `nix shell nixpkgs#ripgrep -c rg -i "retained and unwired|unwired" context/cli` returns no stale claims; `rg -i "automatic.*(reclam|reconcil)" context` hits only the deferral/follow-up statements; documented APIs match code. -- [ ] AC11: Exactly one `WorktreeLock` acquisition per maintenance operation, with no self-deadlock. +- [x] AC11: Exactly one `WorktreeLock` acquisition per maintenance operation, with no self-deadlock. - Validate: T02 and T05 S12; `rg "acquire_inner_async|acquire_async|acquire_inner" cli/src/services/mutation_trace/runtime` shows each entrypoint acquires once and `reconcile_with_held_lock` never acquires. -- [ ] AC12: The hook completion path contains no unbounded maintenance work, and the documented concurrency contract for explicit reconciliation holds. +- [x] AC12: The hook completion path contains no unbounded maintenance work, and the documented concurrency contract for explicit reconciliation holds. - Validate: T05 S14, S15, S16 and S33 pass, as three distinct contention scenarios. S15 (direct advisory contention): with an explicit pass parked mid-inventory or mid-DB holding the lock, `advise_if_due` called directly returns `Busy` immediately and changes no state or refs. S16 (coordinator contention): with the lock held, `coordinate()` keeps its existing bounded lock acquisition; a short injected timeout fails with the existing lock-timeout error before durable completion, the advisory does not run after the failed coordinate, and the same boundary succeeds once the pass is released. S33 (post-coordinate handoff race): after a successful `coordinate()` released its lock, a competing explicit pass takes the lock before the paused advisory resumes; the advisory returns `Busy` and the committed boundary still returns success. The advisory takes no Git/DB parameters. No timeout, budget, or size cap is claimed to bound hook latency. -- [ ] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. +- [x] AC13: Maintenance state is durable, crash-safe, and cannot postpone eligibility indefinitely. - Two guarantee classes plus a distinct uncertain-write outcome (matching D3). **Read-only or unsuccessful operations** (plain `sce doctor`, lock-busy advisory/explicit pass, state that cannot be read or evaluated (`StateUnavailable`, which is not a failed write), a write that fails before the atomic rename (`NotApplied`), a write interrupted before atomic replacement) never mutate maintenance state, may observe invalid timestamps, evaluate them conservatively, never indefinitely postpone an advisory or explicit repair, and never report invalid state as normalized. **`NotApplied`** means the failure happened before the rename: the previous state remains intact and byte-identical, the advisory reports `StateWriteFailed`, and no successful advisory is claimed. **`DurabilityUncertain`** means the failure happened after the rename: the new state may already be visible, crash durability is unconfirmed, the advisory reports `AdvisedDurabilityUncertain` when the recommendation was being recorded (an actionable diagnostic remains available) and `StateWriteFailed` carrying `DurabilityUncertain` when only an anchor or normalization was being written (no recommendation is emitted), and a normalization written this way is never claimed to be durably persisted; a later invocation recovers safely in either case. Explicit reconciliation retains `CompletedStatePersistFailed` and the typed persistence-warning phase. **Successful state-writing operations** normalize invalid timestamps under `WorktreeLock`, persist atomically, and afterwards every stored timestamp satisfies `t ≤ now + FUTURE_SKEW_TOLERANCE`; valid near-future timestamps are preserved; repeated reads never move eligibility forward; later invocations observe the persisted normalization. - Validate: T05 S19–S25 and S34–S39 pass: single time-based advice per 24 h window with persistence before emission (a newly recorded failure may be advised inside that window, once, per AC5); concurrent processes cannot double-advise; state-persistence failure causes no advice and no scan; interrupted atomic write preserves the previous state; process restart preserves eligibility; timestamps within `FUTURE_SKEW_TOLERANCE` are valid (age 0, preserved, never advise by themselves) while timestamps beyond it and clock rollback beyond it are conservatively eligible and, on a successful write, normalized once, persisted, and recovered (S24 cases 1–7); the exact post-condition `t ≤ now + FUTURE_SKEW_TOLERANCE` is asserted only after successful state-writing operations (S24, S36, S37), while S34, S35 and S36's pre-rename leg assert byte-identical preexisting state instead, and S36's post-rename leg asserts only that the new state may be visible with durability reported as uncertain; corrupt/missing state is recovered. Deterministic regression tests (injected clocks and deterministic synchronization, no timing-based sleeps): S34 read-only doctor observing an invalid future timestamp; S35 lock-busy advisory leaving invalid state byte-identical; S36 failed state write during normalization, with both a pre-rename leg (previous bytes unchanged, `StateWriteFailed`) and a post-rename leg (visibility may have changed, `AdvisedDurabilityUncertain`, no durable-normalization claim), a later invocation recovering safely in both; S37 successful normalization persisting valid timestamps; S38 process restart after normalization; S39 eventual advisory eligibility after time progresses. -- [ ] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability (incidental WAL bookkeeping on existing databases is permitted and documented), and only the missing-DB case asserts no newly created database-related files or directories, including when the file is removed between a preliminary check and the open. Plain `sce doctor` shares the same non-creating read-only open. +- [x] AC14: Reconciliation opens only a verified existing DB through a non-creating read-only open that production callers cannot replace, and never initializes, repairs, or migrates it; failure cases assert logical database immutability (incidental WAL bookkeeping on existing databases is permitted and documented), and only the missing-DB case asserts no newly created database-related files or directories, including when the file is removed between a preliminary check and the open. Plain `sce doctor` shares the same non-creating read-only open. - **Missing DB:** the non-creating read-only Turso open fails without creating the DB file, its parent directory, or database sidecars (WAL/shared-memory); no Git ref is listed or deleted. A separate `sce doctor --fix` repair (global-state bootstrap) may create the parent directory independently of reconciliation; reconciliation itself creates nothing. - **Existing but invalid DB:** opening is permitted; incidental Turso WAL/shared-memory bookkeeping is permitted if unavoidable and documented; no schema migration, migration-metadata repair, repository-metadata initialization, or application-table INSERT/UPDATE/DELETE; no Git-ref deletion; verification fails with the matching typed error. Tests do not assert filesystem immutability for this case. - **Existing valid DB:** repository identity and schema readiness are verified before reconciliation; the existing multiprocess-WAL configuration is preserved; only the access reconciliation requires is performed; no initialization, repair, or migration. - Validate: T05 S28–S32 pass: missing DB (no new files or directories, non-creating open fails, refs not listed), incompatible schema, missing metadata, repository identity mismatch (refs untouched, logical DB contents byte-for-byte equivalent by table/row comparison, typed error), and normal existing-DB operation. -- [ ] AC15: Deferred automatic reclamation is explicit and not implied. +- [x] AC15: Deferred automatic reclamation is explicit and not implied. - Validate: the plan's D2 follow-up prerequisites are copied into the synchronized context by T06; no code path or doc claims automatic reclamation; T05 S14 proves hook completion never reconciles. ### Review-finding traceability @@ -489,3 +489,48 @@ Recorded by T01 (read-only audit, 2026-10-09). Line numbers refer to the tree at - Do you want a follow-up PR scoped now for automatic out-of-band reclamation (D2 prerequisites), or only the documented deferral? - `SkippedNoCheckoutIdentity` looks unreachable in the current code (the identity is Git-topology-derived and its failure is an `Err`). The plan assumes T06 removes it; confirm you do not want it kept for compatibility. - D4 adds a dedicated opener and one read-only verification method in `agent_trace_storage` / `agent_trace_db`. T01 may find that Turso cannot open an existing DB without incidental WAL bookkeeping; the plan treats that as an accepted limitation (logical contents unchanged) unless you want a stricter guarantee. + +## Validation Report + +**Status:** validated + +**Date:** 2026-10-09 + +### Commands run + +- `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml` -> exit 0 (498 passed, 0 failed) +- `nix develop -c ./scripts/run-cli-cargo.sh clippy --manifest-path cli/Cargo.toml --all-targets --all-features -- -D warnings` -> exit 0 (clean) +- `nix flake check` -> exit 0 (all checks passed; non-host systems omitted) +- `nix build .#ci-checks` -> exit 0 +- `nix run .#pkl-check-generated` -> not run (no generated config inputs touched) +- `git diff --check` -> exit 0 +- `git diff fix-logging...HEAD --stat` -> exit 0 (no `protocol.rs` or `.qnt` change) +- AC7 `rg` over changed files -> every added hit is in test code; no production hit +- focused `ref_reconciliation` 50, `maintenance` 88, `mutation_scope` 25, `doctor` 27, `agent_trace_storage` 1 passed (all non-zero) + +### Success-criteria verification + +- [x] AC1: only caller chain is `doctor/mod.rs` -> `ref_doctor` -> `reconcile_explicit`; the hook module references only `advise_after_completed_boundary` +- [x] AC2: doctor fix and advisory ref tests pass +- [x] AC3: worktree and fail-closed tests pass +- [x] AC4: advisory outcome and marker tests pass (`mutation_scope` 25 passed) +- [x] AC5: `ref_advisory.rs` non-test code imports no Git, store or DB symbol +- [x] AC6: coordination tests pass in the full suite +- [x] AC7: no production-code hits for the forbidden patterns +- [x] AC8: no `protocol.rs` or `spec/mutation_cursor.qnt` change; `mbt` tests pass +- [x] AC9: doctor tests pass +- [x] AC10: no "unwired" claims in `context/cli`; automatic-reclamation hits are deferral statements +- [x] AC11: each entrypoint acquires `WorktreeLock` once; `reconcile_with_held_lock` does not acquire +- [x] AC12: contention and handoff tests pass +- [x] AC13: maintenance-state and recovery tests pass +- [x] AC14: verified-open and logical-immutability tests pass, including the new `agent_trace_storage` missing-DB test +- [x] AC15: deferral and prerequisites recorded in `context/cli/mutation-trace-ref-reconciliation.md` + +### Failed checks and follow-ups + +- None. + +### Residual risks + +- Cross-process multiprocess-WAL coexistence is covered only in-process (T02 limitation 4). +- The new `agent_trace_storage` test is uncommitted in the working tree at validation time. From e6dccade0bab8fbd095fe7b59fd92ace661125d8 Mon Sep 17 00:00:00 2001 From: davidabram Date: Fri, 9 Oct 2026 09:20:20 +0200 Subject: [PATCH 14/16] tests: Use platform state root in doctor reconciliation sandbox The doctor reconciliation CLI test sandbox assumed a `state` directory under the temporary home, but on macOS the CLI resolves its state root to `Library/Application Support`. Resolve the sandbox state root per platform so the tests locate the same Agent Trace database and maintenance state as the CLI under test. Co-authored-by: SCE --- cli/src/app/doctor_reconciliation_cli_tests.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cli/src/app/doctor_reconciliation_cli_tests.rs b/cli/src/app/doctor_reconciliation_cli_tests.rs index 980393b29..b1f621a54 100644 --- a/cli/src/app/doctor_reconciliation_cli_tests.rs +++ b/cli/src/app/doctor_reconciliation_cli_tests.rs @@ -84,7 +84,11 @@ impl Sandbox { } fn state_root(&self) -> PathBuf { - self.dir.path().join("state") + if cfg!(target_os = "macos") { + self.dir.path().join("Library").join("Application Support") + } else { + self.dir.path().join("state") + } } fn db_path(&self) -> PathBuf { From 51df9e48489ae71c0611a968c80d0bd141ddb596 Mon Sep 17 00:00:00 2001 From: David Abram Date: Fri, 9 Oct 2026 10:30:49 +0200 Subject: [PATCH 15/16] tests: Consolidate ref reconciliation coverage Consolidate redundant reconciliation, advisory, doctor, and maintenance-state tests into table-driven cases while strengthening assertions for exit statuses, ref inventories, pin preservation, typed failures, and read-only database behavior. Remove obsolete test-only lock-acquisition wrappers and align the CLI/context coverage documentation with the retained scenarios. Plan: mutation-cursor-ref-reconciliation-wiring Tasks: T05 Co-authored-by: SCE --- .../app/doctor_reconciliation_cli_tests.rs | 41 +-- cli/src/services/agent_trace_db/lifecycle.rs | 4 +- cli/src/services/doctor/mod.rs | 170 +++++++------ cli/src/services/doctor/render.rs | 234 ++++++++++-------- cli/src/services/hooks/mutation_scope.rs | 11 - .../runtime/maintenance_state.rs | 115 ++++----- .../mutation_trace/runtime/ref_advisory.rs | 86 +++---- .../runtime/ref_maintenance/tests.rs | 161 ++++++------ .../ref_maintenance/tests/coordination.rs | 127 ++++------ .../runtime/ref_reconciliation.rs | 41 --- .../cli/mutation-trace-ref-reconciliation.md | 5 +- ...tation-cursor-ref-reconciliation-wiring.md | 27 +- 12 files changed, 475 insertions(+), 547 deletions(-) diff --git a/cli/src/app/doctor_reconciliation_cli_tests.rs b/cli/src/app/doctor_reconciliation_cli_tests.rs index b1f621a54..ed1dfff00 100644 --- a/cli/src/app/doctor_reconciliation_cli_tests.rs +++ b/cli/src/app/doctor_reconciliation_cli_tests.rs @@ -21,18 +21,9 @@ const EXIT_FAILURE: &str = "failure"; struct CliRun { stdout: String, stderr: String, - exit: String, } impl CliRun { - fn assert_exit_success(&self) { - assert_eq!( - self.exit, EXIT_SUCCESS, - "doctor exit status:\n{}\n{}", - self.stdout, self.stderr - ); - } - fn json(&self) -> Value { serde_json::from_str(&self.stdout) .unwrap_or_else(|error| panic!("doctor stdout is not JSON ({error}):\n{}", self.stdout)) @@ -168,11 +159,20 @@ impl Sandbox { String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) ); - CliRun { + let run = CliRun { stdout: std::fs::read_to_string(out.join("stdout")).expect("stdout"), stderr: std::fs::read_to_string(out.join("stderr")).expect("stderr"), - exit: std::fs::read_to_string(out.join("exit")).expect("exit"), - } + }; + let exit = std::fs::read_to_string(out.join("exit")).expect("exit"); + assert_eq!( + exit, + EXIT_SUCCESS, + "sce {} exit status:\n{}\n{}", + args.join(" "), + run.stdout, + run.stderr + ); + run } } @@ -241,7 +241,7 @@ async fn doctor_reconciliation_cli_child() { &mut stderr, )) .await; - let exit = if format!("{exit_code:?}") == format!("{:?}", ExitCode::SUCCESS) { + let exit = if exit_code == ExitCode::SUCCESS { EXIT_SUCCESS } else { EXIT_FAILURE @@ -260,7 +260,6 @@ async fn doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once() assert!(refs_before.contains(&tree)); let inspection = sandbox.run(&["doctor", "--format", "json"]); - inspection.assert_exit_success(); let report = inspection.json(); assert!( report["ref_reconciliation_fix"].is_null(), @@ -271,7 +270,6 @@ async fn doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once() assert_eq!(sandbox.state_bytes(), None); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); - fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!(reconciliation["outcome"], "completed", "{}", fix.stdout); @@ -307,7 +305,6 @@ async fn doctor_fix_reports_missing_database_without_creating_it_or_deleting_ref let db_path = sandbox.db_path(); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); - fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!(reconciliation["outcome"], "failed", "{}", fix.stdout); @@ -353,7 +350,6 @@ async fn doctor_fix_reports_repository_mismatch_without_touching_the_database_or let refs_before = sandbox.refs(); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); - fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!(reconciliation["outcome"], "failed", "{}", fix.stdout); @@ -398,7 +394,6 @@ async fn doctor_fix_reports_busy_when_the_worktree_lock_is_held_and_keeps_other_ held.try_lock().expect("hold the worktree lock externally"); let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); - fix.assert_exit_success(); let report = fix.json(); drop(held); @@ -448,7 +443,6 @@ impl Sandbox { } fn assert_unreadable_state_outcome(fix: &CliRun, sandbox: &Sandbox, label: &str) { - fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!( @@ -559,7 +553,6 @@ async fn doctor_fix_recovers_from_every_initial_maintenance_state_condition() { let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); - fix.assert_exit_success(); let report = fix.json(); let reconciliation = &report["ref_reconciliation_fix"]; assert_eq!( @@ -594,18 +587,10 @@ async fn doctor_fix_recovers_from_every_initial_maintenance_state_condition() { } let again = sandbox.run(&["doctor", "--fix", "--format", "json"]); - again.assert_exit_success(); assert_eq!( again.json()["ref_reconciliation_fix"]["outcome"], "completed", "{label}: the rewritten state is readable" ); } - - let sandbox = Sandbox::new("https://example.invalid/org/repo-a.git"); - sandbox.create_db(&sandbox.repository_id).await; - sandbox.orphan_pin(); - sandbox.make_state_unreadable(); - let fix = sandbox.run(&["doctor", "--fix", "--format", "json"]); - assert_unreadable_state_outcome(&fix, &sandbox, "unreadable_state_file"); } diff --git a/cli/src/services/agent_trace_db/lifecycle.rs b/cli/src/services/agent_trace_db/lifecycle.rs index 0a4f56569..b00acce28 100644 --- a/cli/src/services/agent_trace_db/lifecycle.rs +++ b/cli/src/services/agent_trace_db/lifecycle.rs @@ -396,9 +396,9 @@ mod tests { let problems = inspect_existing_db_schema(&path, || {}).await; assert_eq!(problems.len(), 1); - assert_ne!( + assert_eq!( problems[0].kind, - HealthProblemKind::AgentTraceDbSchemaNotReady + HealthProblemKind::AgentTraceDbConnectionFailed ); assert_eq!(std::fs::read(&path).expect("read"), before); assert_eq!(dir_entries(dir.path()), vec!["agent-trace.db".to_string()]); diff --git a/cli/src/services/doctor/mod.rs b/cli/src/services/doctor/mod.rs index 64f6aea97..b42906607 100644 --- a/cli/src/services/doctor/mod.rs +++ b/cli/src/services/doctor/mod.rs @@ -581,76 +581,106 @@ mod reconciliation_fix_tests { } #[test] - fn completed_pass_reports_counts_as_fixed() { - let record = reconciliation_fix_result(&ReconciliationFix::Completed(COUNTS)); - assert_eq!(record.outcome, FixResult::Fixed); - assert_eq!( - record.detail, - "Snapshot ref reconciliation completed: deleted 3, retained 2, locally required 1." - ); - } - - #[test] - fn completed_pass_with_pre_rename_failure_says_update_was_not_applied() { - let record = reconciliation_fix_result(&ReconciliationFix::CompletedStatePersistFailed { - counts: COUNTS, - warning: not_applied(), - }); - assert_eq!(record.outcome, FixResult::Fixed); - assert!(record.detail.contains("deleted 3")); - assert!(record - .detail - .contains("Maintenance-state update was not applied.")); - assert!(record.detail.contains("disk full")); - assert!(!record.detail.contains("were not recorded")); - } - - #[test] - fn completed_pass_with_post_rename_failure_says_durability_unconfirmed() { - let record = reconciliation_fix_result(&ReconciliationFix::CompletedStatePersistFailed { - counts: COUNTS, - warning: durability_uncertain(), - }); - assert_eq!(record.outcome, FixResult::Fixed); - assert!(record - .detail - .contains("Maintenance-state durability could not be confirmed.")); - assert!(!record.detail.contains("not applied")); - assert!(!record.detail.contains("were not recorded")); - } - - #[test] - fn unreadable_previous_state_does_not_claim_nothing_was_recorded() { - let record = reconciliation_fix_result(&ReconciliationFix::CompletedStatePersistFailed { - counts: COUNTS, - warning: StateWarning::PreviousStateUnreadable { - cause: "permission denied".to_string(), + fn reconciliation_fix_results_render_expected_text() { + struct Case { + label: &'static str, + fix: ReconciliationFix, + outcome: FixResult, + required: Vec<&'static str>, + forbidden: Vec<&'static str>, + } + let cases = [ + Case { + label: "completed", + fix: ReconciliationFix::Completed(COUNTS), + outcome: FixResult::Fixed, + required: vec![ + "Snapshot ref reconciliation completed: deleted 3, retained 2, locally required 1.", + ], + forbidden: vec!["Warning", "failed"], }, - }); - assert!(record.detail.contains("could not be read")); - assert!(record.detail.contains("permission denied")); - assert!(!record.detail.contains("were not recorded")); - } - - #[test] - fn failed_pass_is_never_reported_as_success() { - let record = reconciliation_fix_result(&ReconciliationFix::Failed { - kind: "agent_trace_db_missing", - message: "db missing".to_string(), - state_warning: Some(not_applied()), - }); - assert_eq!(record.outcome, FixResult::Failed); - assert!(record.detail.contains("agent_trace_db_missing")); - assert!(record.detail.contains("db missing")); - assert!(record - .detail - .contains("Maintenance-state update was not applied.")); - } - - #[test] - fn busy_pass_is_skipped_not_fixed() { - let record = reconciliation_fix_result(&ReconciliationFix::Busy); - assert_eq!(record.outcome, FixResult::Skipped); - assert!(record.detail.contains("skipped")); + Case { + label: "pre-rename persistence failure", + fix: ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: not_applied(), + }, + outcome: FixResult::Fixed, + required: vec![ + "deleted 3", + "Maintenance-state update was not applied.", + "disk full", + ], + forbidden: vec!["were not recorded", "failed"], + }, + Case { + label: "post-rename persistence failure", + fix: ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: durability_uncertain(), + }, + outcome: FixResult::Fixed, + required: vec![ + "deleted 3", + "Maintenance-state durability could not be confirmed.", + "io error", + ], + forbidden: vec!["not applied", "were not recorded", "failed"], + }, + Case { + label: "unreadable previous state", + fix: ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::PreviousStateUnreadable { + cause: "permission denied".to_string(), + }, + }, + outcome: FixResult::Fixed, + required: vec!["deleted 3", "could not be read", "permission denied"], + forbidden: vec!["were not recorded", "failed"], + }, + Case { + label: "failed pass", + fix: ReconciliationFix::Failed { + kind: "agent_trace_db_missing", + message: "db missing".to_string(), + state_warning: Some(not_applied()), + }, + outcome: FixResult::Failed, + required: vec![ + "agent_trace_db_missing", + "db missing", + "Maintenance-state update was not applied.", + ], + forbidden: vec!["completed"], + }, + Case { + label: "busy pass", + fix: ReconciliationFix::Busy, + outcome: FixResult::Skipped, + required: vec!["skipped"], + forbidden: vec!["completed", "deleted"], + }, + ]; + for case in cases { + let record = reconciliation_fix_result(&case.fix); + assert_eq!(record.outcome, case.outcome, "{}", case.label); + for fragment in case.required { + assert!( + record.detail.contains(fragment), + "{}: missing {fragment:?} in {:?}", + case.label, + record.detail + ); + } + for fragment in case.forbidden { + assert!( + !record.detail.contains(fragment), + "{}: unexpected {fragment:?} in {:?}", + case.label, + record.detail + ); + } + } } } diff --git a/cli/src/services/doctor/render.rs b/cli/src/services/doctor/render.rs index a4dc43e17..587eea855 100644 --- a/cli/src/services/doctor/render.rs +++ b/cli/src/services/doctor/render.rs @@ -1086,115 +1086,141 @@ mod reconciliation_render_tests { } #[test] - fn json_completed_outcome() { - assert_eq!( - ref_reconciliation_fix_json(&ReconciliationFix::Completed(COUNTS)), - json!({ - "outcome": "completed", - "counts": { "deleted": 3, "retained": 2, "local_required": 1 }, - "failure": null, - "state_warning": null, - }) - ); + fn json_ref_reconciliation_fix_results_match_contract() { + let counts = json!({ "deleted": 3, "retained": 2, "local_required": 1 }); + let cases = [ + ( + "completed", + ReconciliationFix::Completed(COUNTS), + json!({ + "outcome": "completed", + "counts": counts, + "failure": null, + "state_warning": null, + }), + ), + ( + "failed without warning", + ReconciliationFix::Failed { + kind: "agent_trace_db_missing", + message: "db missing".to_string(), + state_warning: None, + }, + json!({ + "outcome": "failed", + "counts": null, + "failure": { "kind": "agent_trace_db_missing", "message": "db missing" }, + "state_warning": null, + }), + ), + ( + "failed with warning", + ReconciliationFix::Failed { + kind: "missing_required_pins", + message: "pins".to_string(), + state_warning: Some(StateWarning::NotApplied { + phase: "write_staging", + cause: "full".to_string(), + }), + }, + json!({ + "outcome": "failed", + "counts": null, + "failure": { "kind": "missing_required_pins", "message": "pins" }, + "state_warning": { + "kind": "not_applied", + "phase": "write_staging", + "message": "Maintenance-state update was not applied.", + "cause": "full", + }, + }), + ), + ( + "busy", + ReconciliationFix::Busy, + json!({ + "outcome": "busy", + "counts": null, + "failure": null, + "state_warning": null, + }), + ), + ]; + for (label, fix, expected) in cases { + assert_eq!(ref_reconciliation_fix_json(&fix), expected, "{label}"); + } } #[test] - fn json_completed_state_persist_failed_distinguishes_warning_kinds() { - let not_applied = - ref_reconciliation_fix_json(&ReconciliationFix::CompletedStatePersistFailed { - counts: COUNTS, - warning: StateWarning::NotApplied { - phase: "rename", - cause: "denied".to_string(), - }, - }); - assert_eq!( - not_applied, - json!({ - "outcome": "completed_state_persist_failed", - "counts": { "deleted": 3, "retained": 2, "local_required": 1 }, - "failure": null, - "state_warning": { - "kind": "not_applied", - "phase": "rename", - "message": "Maintenance-state update was not applied.", - "cause": "denied", + fn json_completed_state_persist_failed_results_match_contract() { + let counts = json!({ "deleted": 3, "retained": 2, "local_required": 1 }); + let cases = [ + ( + "completed + not applied", + ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::NotApplied { + phase: "rename", + cause: "denied".to_string(), + }, }, - }) - ); - - let uncertain = - ref_reconciliation_fix_json(&ReconciliationFix::CompletedStatePersistFailed { - counts: COUNTS, - warning: StateWarning::DurabilityUncertain { - phase: "sync_parent_directory", - cause: "eio".to_string(), + json!({ + "outcome": "completed_state_persist_failed", + "counts": counts, + "failure": null, + "state_warning": { + "kind": "not_applied", + "phase": "rename", + "message": "Maintenance-state update was not applied.", + "cause": "denied", + }, + }), + ), + ( + "completed + durability uncertain", + ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::DurabilityUncertain { + phase: "sync_parent_directory", + cause: "eio".to_string(), + }, }, - }); - assert_eq!(uncertain["state_warning"]["kind"], "durability_uncertain"); - assert_eq!(uncertain["state_warning"]["phase"], "sync_parent_directory"); - assert_eq!( - uncertain["state_warning"]["message"], - "Maintenance-state durability could not be confirmed." - ); - - let unreadable = - ref_reconciliation_fix_json(&ReconciliationFix::CompletedStatePersistFailed { - counts: COUNTS, - warning: StateWarning::PreviousStateUnreadable { - cause: "eacces".to_string(), + json!({ + "outcome": "completed_state_persist_failed", + "counts": counts, + "failure": null, + "state_warning": { + "kind": "durability_uncertain", + "phase": "sync_parent_directory", + "message": "Maintenance-state durability could not be confirmed.", + "cause": "eio", + }, + }), + ), + ( + "completed + previous state unreadable", + ReconciliationFix::CompletedStatePersistFailed { + counts: COUNTS, + warning: StateWarning::PreviousStateUnreadable { + cause: "eacces".to_string(), + }, }, - }); - assert_eq!( - unreadable["state_warning"]["kind"], - "previous_state_unreadable" - ); - assert_eq!( - unreadable["state_warning"]["phase"], - serde_json::Value::Null - ); - } - - #[test] - fn json_failed_outcome_carries_failure_and_optional_warning() { - let without_warning = ref_reconciliation_fix_json(&ReconciliationFix::Failed { - kind: "agent_trace_db_missing", - message: "db missing".to_string(), - state_warning: None, - }); - assert_eq!( - without_warning, - json!({ - "outcome": "failed", - "counts": null, - "failure": { "kind": "agent_trace_db_missing", "message": "db missing" }, - "state_warning": null, - }) - ); - - let with_warning = ref_reconciliation_fix_json(&ReconciliationFix::Failed { - kind: "missing_required_pins", - message: "pins".to_string(), - state_warning: Some(StateWarning::NotApplied { - phase: "write_staging", - cause: "full".to_string(), - }), - }); - assert_eq!(with_warning["failure"]["kind"], "missing_required_pins"); - assert_eq!(with_warning["state_warning"]["kind"], "not_applied"); - assert_eq!(with_warning["state_warning"]["phase"], "write_staging"); - } - - #[test] - fn json_busy_outcome() { - assert_eq!( - ref_reconciliation_fix_json(&ReconciliationFix::Busy), - json!({ - "outcome": "busy", - "counts": null, - "failure": null, - "state_warning": null, - }) - ); + json!({ + "outcome": "completed_state_persist_failed", + "counts": counts, + "failure": null, + "state_warning": { + "kind": "previous_state_unreadable", + "phase": null, + "message": + "Maintenance state could not be read; the update was not attempted.", + "cause": "eacces", + }, + }), + ), + ]; + for (label, fix, expected) in cases { + assert_eq!(ref_reconciliation_fix_json(&fix), expected, "{label}"); + } } } diff --git a/cli/src/services/hooks/mutation_scope.rs b/cli/src/services/hooks/mutation_scope.rs index 6897bc29f..67ce7dfc6 100644 --- a/cli/src/services/hooks/mutation_scope.rs +++ b/cli/src/services/hooks/mutation_scope.rs @@ -933,17 +933,6 @@ mod advisory_trigger_tests { assert_eq!(calls.abandon.get(), 0); } - #[tokio::test] - async fn advisory_runs_after_marker_clear_after_commit_with_success_result() { - let calls = Calls::default(); - let error = CoordinateError::MarkerClearAfterCommit { - source: anyhow!("clear failed"), - committed: Box::new(outcome()), - }; - assert_eq!(drive(close(), Err(error), &calls).await.unwrap(), ""); - assert_eq!(calls.advise.get(), 1); - } - #[tokio::test] async fn advisory_never_runs_for_start_or_advance() { let calls = Calls::default(); diff --git a/cli/src/services/mutation_trace/runtime/maintenance_state.rs b/cli/src/services/mutation_trace/runtime/maintenance_state.rs index 9e664c711..e094a536b 100644 --- a/cli/src/services/mutation_trace/runtime/maintenance_state.rs +++ b/cli/src/services/mutation_trace/runtime/maintenance_state.rs @@ -626,15 +626,13 @@ mod tests { .count() } - fn assert_not_applied( - phase: PersistPhase, - previous_marker: Option, - ) -> (tempfile::TempDir, PathBuf) { + fn assert_not_applied(phase: PersistPhase, previous_marker: Option) { let dir = tempfile::tempdir().expect("temp dir"); let path = state_file(dir.path()); if let Some(marker) = previous_marker { write_state_atomically(&path, &state(marker)).expect("seed"); } + let previous_bytes = std::fs::read(&path).ok(); let fs = FaultInjectingFilesystem::failing_at(phase); let failure = @@ -649,9 +647,14 @@ mod tests { }; assert_eq!(reported, phase); assert!(staging_cleanup.is_none()); + assert_eq!(std::fs::read(&path).ok(), previous_bytes); + assert_eq!(path.exists(), previous_marker.is_some()); assert_eq!(stored_marker(&path), previous_marker); assert_eq!(staging_leftovers(dir.path()), 0); - (dir, path) + + write_state_atomically(&path, &state(100)).expect("recovery write"); + assert_eq!(stored_marker(&path), Some(100)); + assert_eq!(staging_leftovers(dir.path()), 0); } #[test] @@ -665,71 +668,49 @@ mod tests { } #[test] - fn maintenance_state_staging_creation_failure_preserves_previous_state() { - assert_not_applied(PersistPhase::CreateStaging, Some(1)); - } - - #[test] - fn maintenance_state_staging_write_failure_preserves_previous_state_and_cleans_up() { - assert_not_applied(PersistPhase::WriteStaging, Some(1)); - } - - #[test] - fn maintenance_state_staging_sync_failure_preserves_previous_state_and_cleans_up() { - assert_not_applied(PersistPhase::SyncStaging, Some(1)); - } - - #[test] - fn maintenance_state_rename_failure_preserves_previous_state_and_cleans_up() { - assert_not_applied(PersistPhase::Rename, Some(1)); - } - - #[test] - fn maintenance_state_rename_failure_without_previous_state_leaves_no_state() { - assert_not_applied(PersistPhase::Rename, None); - } - - #[test] - fn maintenance_state_parent_directory_open_failure_is_durability_uncertain() { - let dir = tempfile::tempdir().expect("temp dir"); - let path = state_file(dir.path()); - write_state_atomically(&path, &state(1)).expect("seed"); - - let fs = FaultInjectingFilesystem::failing_at(PersistPhase::OpenParentDirectory); - let failure = write_state_atomically_with(&fs, &path, &state(2)).expect_err("failure"); - assert!(matches!( - failure, - PersistFailure::DurabilityUncertain { - phase: PersistPhase::OpenParentDirectory, - .. - } - )); - assert!(failure.to_string().contains("may be visible")); - assert_eq!(stored_marker(&path), Some(2)); - assert_eq!(staging_leftovers(dir.path()), 0); + fn maintenance_state_pre_rename_failures_preserve_previous_state() { + for (phase, previous_marker) in [ + (PersistPhase::CreateStaging, Some(1)), + (PersistPhase::WriteStaging, Some(1)), + (PersistPhase::SyncStaging, Some(1)), + (PersistPhase::Rename, Some(1)), + (PersistPhase::Rename, None), + ] { + assert_not_applied(phase, previous_marker); + } } #[test] - fn maintenance_state_parent_directory_sync_failure_distinguishes_visibility_from_durability() { - let dir = tempfile::tempdir().expect("temp dir"); - let path = state_file(dir.path()); - write_state_atomically(&path, &state(1)).expect("seed"); - - let fs = FaultInjectingFilesystem::failing_at(PersistPhase::SyncParentDirectory); - let failure = write_state_atomically_with(&fs, &path, &state(2)).expect_err("failure"); - assert!(matches!( - failure, - PersistFailure::DurabilityUncertain { - phase: PersistPhase::SyncParentDirectory, - .. - } - )); - assert!(!failure.to_string().contains("previous state preserved")); - assert_eq!(stored_marker(&path), Some(2)); - assert_eq!(staging_leftovers(dir.path()), 0); - - write_state_atomically(&path, &state(3)).expect("recovery write"); - assert_eq!(stored_marker(&path), Some(3)); + fn maintenance_state_post_rename_failures_report_uncertain_durability() { + for phase in [ + PersistPhase::OpenParentDirectory, + PersistPhase::SyncParentDirectory, + ] { + let dir = tempfile::tempdir().expect("temp dir"); + let path = state_file(dir.path()); + write_state_atomically(&path, &state(1)).expect("seed"); + + let fs = FaultInjectingFilesystem::failing_at(phase); + let failure = write_state_atomically_with(&fs, &path, &state(2)).expect_err("failure"); + let PersistFailure::DurabilityUncertain { + phase: reported, .. + } = failure + else { + panic!("expected DurabilityUncertain for {phase:?}, got {failure:?}"); + }; + assert_eq!(reported, phase); + let rendered = failure.to_string(); + assert!(rendered.contains("may be visible"), "{phase:?}: {rendered}"); + assert!( + !rendered.contains("previous state preserved"), + "{phase:?}: {rendered}" + ); + assert_eq!(stored_marker(&path), Some(2), "{phase:?}"); + assert_eq!(staging_leftovers(dir.path()), 0, "{phase:?}"); + + write_state_atomically(&path, &state(3)).expect("recovery write"); + assert_eq!(stored_marker(&path), Some(3), "{phase:?}"); + } } #[test] diff --git a/cli/src/services/mutation_trace/runtime/ref_advisory.rs b/cli/src/services/mutation_trace/runtime/ref_advisory.rs index b8a9aef8d..b70a368ab 100644 --- a/cli/src/services/mutation_trace/runtime/ref_advisory.rs +++ b/cli/src/services/mutation_trace/runtime/ref_advisory.rs @@ -370,22 +370,6 @@ mod tests { )); } - #[test] - fn maintenance_advisory_preserves_timestamps_within_skew_tolerance() { - let dir = tempfile::tempdir().expect("temp dir"); - init_repo(dir.path()); - let near_future = MaintenanceState { - last_success: Some(NOW + FUTURE_SKEW_TOLERANCE_MS), - ..MaintenanceState::default() - }; - seed(dir.path(), &near_future); - let path = state_path(&dir.path().join(".git")); - let before = std::fs::read(&path).expect("state bytes"); - - assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); - assert_eq!(std::fs::read(&path).expect("state bytes"), before); - } - #[test] fn maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately() { let dir = tempfile::tempdir().expect("temp dir"); @@ -679,8 +663,8 @@ mod tests { } #[test] - fn maintenance_advisory_treats_timestamps_at_or_just_ahead_of_now_as_valid_age_zero() { - for ahead in [0, 60_000] { + fn maintenance_advisory_accepts_valid_timestamp_offsets_without_rewrite() { + for ahead in [0, 60_000, FUTURE_SKEW_TOLERANCE_MS] { let dir = tempfile::tempdir().expect("temp dir"); init_repo(dir.path()); let state = MaintenanceState { @@ -691,9 +675,15 @@ mod tests { let path = state_path(&dir.path().join(".git")); let before = std::fs::read(&path).expect("state bytes"); - assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction)); + assert!( + matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction), + "offset {ahead}" + ); assert_eq!(std::fs::read(&path).expect("state bytes"), before); - assert!(!evaluate_recommendation(&state, NOW).recommended); + assert!( + !evaluate_recommendation(&state, NOW).recommended, + "offset {ahead}" + ); } } @@ -722,22 +712,46 @@ mod tests { } #[test] - fn maintenance_advisory_recovers_wrong_version_and_oversized_state_by_anchoring() { - let oversized = vec![b' '; 5000]; - let wrong_version = br#"{"version":9999}"#.to_vec(); - for bytes in [oversized, wrong_version] { + fn maintenance_advisory_reanchors_unusable_state_variants() { + let cases: [(&str, Vec); 3] = [ + ("wrong schema version", br#"{"version":9999}"#.to_vec()), + ("oversized state file", vec![b' '; 5000]), + ("corrupt JSON", b"not json".to_vec()), + ]; + for (label, bytes) in cases { let dir = tempfile::tempdir().expect("temp dir"); init_repo(dir.path()); let path = state_path(&dir.path().join(".git")); std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); std::fs::write(&path, &bytes).expect("unusable state"); - assert!(matches!( - super::read_state(&path).expect("read"), - StateRead::Unusable - )); - assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored)); - assert_eq!(stored(dir.path()).anchor, Some(NOW)); + assert!( + matches!(read_state(&path).expect("read"), StateRead::Unusable), + "{label}" + ); + let outcome = advise(dir.path(), NOW); + assert!( + matches!(outcome, AdvisoryOutcome::Anchored), + "{label}: {outcome:?}" + ); + assert_eq!(outcome.recommendation(), None, "{label}"); + let anchored = stored(dir.path()); + assert_eq!(anchored.anchor, Some(NOW), "{label}"); + assert!( + !evaluate_recommendation(&anchored, NOW).recommended, + "{label}" + ); + + let anchored_bytes = std::fs::read(&path).expect("state bytes"); + assert!( + matches!(advise(dir.path(), NOW), AdvisoryOutcome::NoAction), + "{label}" + ); + assert_eq!( + std::fs::read(&path).expect("state bytes"), + anchored_bytes, + "{label}" + ); } } @@ -899,18 +913,6 @@ mod tests { assert_eq!(std::fs::read(&path).expect("state bytes"), bytes); } - #[test] - fn maintenance_advisory_recovers_corrupt_state_by_anchoring() { - let dir = tempfile::tempdir().expect("temp dir"); - init_repo(dir.path()); - let path = state_path(&dir.path().join(".git")); - std::fs::create_dir_all(path.parent().expect("parent")).expect("runtime dir"); - std::fs::write(&path, b"not json").expect("corrupt state"); - - assert!(matches!(advise(dir.path(), NOW), AdvisoryOutcome::Anchored)); - assert_eq!(stored(dir.path()).anchor, Some(NOW)); - } - #[test] fn maintenance_advisory_derives_linked_worktree_git_dir_from_subdirectory() { let dir = tempfile::tempdir().expect("temp dir"); diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs index 9aa167512..d19be1183 100644 --- a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs @@ -8,15 +8,14 @@ use super::super::maintenance_state::{ write_state_atomically_with, FaultInjectingFilesystem, MaintenanceState, PersistFailure, PersistPhase, StateRead, }; -use super::super::ref_advisory::{advise_if_due, AdvisoryOutcome}; use super::super::ref_reconciliation::{ - reconcile_with_held_lock, reconcile_worktree, ReconcileError, ReconcilePhase, - ReconciliationOutcome, ReconciliationReport, + reconcile_with_held_lock, ReconcileError, ReconcilePhase, ReconciliationOutcome, + ReconciliationReport, }; use super::super::worktree_lock::acquire_inner_async; use super::{ - open_authoritative_db_at_state_root, reconcile_explicit, reconcile_explicit_with, - ExplicitOutcome, SkipReason, StatePersistWarning, + open_authoritative_db_at_state_root, reconcile_explicit_with, ExplicitOutcome, SkipReason, + StatePersistWarning, }; use crate::services::agent_trace_db::repository::{ ExistingRepositoryDbError, RepositoryAgentTraceDb, @@ -135,6 +134,25 @@ async fn create_db(path: &Path, repository_id: &str) { .expect("initialize metadata"); } +fn ref_inventory(root: &Path) -> Vec { + let output = Command::new("git") + .args([ + "for-each-ref", + "--format=%(refname) %(objectname) %(symref)", + "refs/sce/", + ]) + .current_dir(root) + .output() + .expect("for-each-ref"); + assert!(output.status.success(), "for-each-ref failed"); + let mut lines: Vec = String::from_utf8_lossy(&output.stdout) + .lines() + .map(str::to_string) + .collect(); + lines.sort(); + lines +} + async fn pin_orphan(root: &Path) -> String { let snapshot = GitSnapshotService::new(root).await.expect("snapshot"); let tree = snapshot.capture_tree().await.expect("tree"); @@ -192,12 +210,6 @@ fn staging_leftovers(dir: &Path) -> usize { .count() } -#[test] -fn ref_reconciliation_production_entrypoint_accepts_only_a_repository_root() { - fn assert_signature AsyncFn(&'a Path) -> ExplicitOutcome>(_: F) {} - assert_signature(reconcile_explicit); -} - #[tokio::test(flavor = "multi_thread")] async fn ref_reconciliation_verified_database_reconciles_orphan_pin_and_records_success() { let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); @@ -534,16 +546,29 @@ async fn ref_reconciliation_unreadable_previous_state_is_reported_without_overwr let path = state_path(&git_dir); std::fs::create_dir_all(&path).expect("directory in place of state file"); + let pins_before = fixture.pins().await; + assert_eq!(pins_before.len(), 1); + let (outcome, _) = fixture.run().await; - let ExplicitOutcome::Failed { state_warning, .. } = outcome else { + let ExplicitOutcome::Failed { + error, + state_warning, + } = outcome + else { panic!("expected failure, got {outcome:?}"); }; + assert!(matches!( + unavailable_cause(&error), + ExistingRepositoryDbError::Missing { .. } + )); assert!(matches!( state_warning, Some(StatePersistWarning::PreviousStateUnreadable(_)) )); assert!(path.is_dir()); + assert_eq!(std::fs::read_dir(&path).expect("read state dir").count(), 0); + assert_eq!(fixture.pins().await, pins_before); } #[tokio::test(flavor = "multi_thread")] @@ -574,27 +599,6 @@ async fn ref_reconciliation_lock_held_implementation_never_reacquires_the_lock() assert_eq!(report.deleted, 1); } -#[tokio::test(flavor = "multi_thread")] -async fn ref_reconciliation_worktree_wrapper_acquires_once_and_releases() { - let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); - fixture.create_db().await; - pin_orphan(&fixture.root).await; - - let outcome = reconcile_worktree(&fixture.root, async || { - open_authoritative_db_at_state_root(&fixture.root, &fixture.state_root).await - }) - .await - .expect("reconcile"); - assert!(matches!(outcome, ReconciliationOutcome::Reconciled(_))); - - let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); - drop( - acquire_inner_async(&git_dir, Duration::ZERO, || {}) - .await - .expect("lock released after wrapper returns"), - ); -} - #[tokio::test(flavor = "multi_thread")] async fn ref_reconciliation_verified_open_never_creates_missing_database_or_directories() { let dir = tempfile::tempdir().expect("temp dir"); @@ -770,23 +774,45 @@ async fn ref_reconciliation_verified_open_never_recreates_a_database_removed_aft #[tokio::test(flavor = "multi_thread")] async fn ref_reconciliation_verified_open_is_read_only() { + const INSERT_WORKTREE: &str = "INSERT INTO mutation_trace_worktrees + (worktree_id, cursor_tree, revision, tainted, failure_kind, needs_rebaseline) + VALUES ('main', 'tree', x'0000000000000000', 0, 'healthy', 0)"; let dir = tempfile::tempdir().expect("temp dir"); let path = dir.path().join("agent-trace.db"); create_db(&path, "repo").await; + let writer = RepositoryAgentTraceDb::new_at(&path).await.expect("writer"); + writer.execute("BEGIN", ()).await.expect("begin"); + writer + .execute(INSERT_WORKTREE, ()) + .await + .expect("the statement is valid on a writable connection"); + writer.execute("ROLLBACK", ()).await.expect("rollback"); + drop(writer); + let (db, _) = RepositoryAgentTraceDb::open_verified_existing_at(&path, "repo") .await .expect("verified open"); + let error = db + .execute(INSERT_WORKTREE, ()) + .await + .expect_err("verified connection must reject writes"); + let rendered = format!("{error:#}"); + assert!(rendered.contains("database is readonly"), "{rendered}"); + drop(db); - assert!(db - .execute( - "INSERT INTO mutation_trace_worktrees - (worktree_id, cursor_tree, revision, tainted, failure_kind, needs_rebaseline) - VALUES ('main', 'tree', x'0000000000000000', 0, 'healthy', 0)", + let inspector = RepositoryAgentTraceDb::new_at(&path) + .await + .expect("inspector"); + let rows = inspector + .query_map( + "SELECT worktree_id FROM mutation_trace_worktrees", (), + |row| row.get::(0).map_err(Into::into), ) .await - .is_err()); + .expect("rows"); + assert!(rows.is_empty()); } #[tokio::test(flavor = "multi_thread")] @@ -890,64 +916,25 @@ async fn ref_reconciliation_malformed_namespace_ref_deletes_nothing_and_records_ "HEAD", ], ); + let inventory_before = ref_inventory(&fixture.root); + assert_eq!(inventory_before.len(), 2); let (outcome, _) = fixture.run().await; let ExplicitOutcome::Failed { error, .. } = outcome else { panic!("expected failure, got {outcome:?}"); }; - assert!(matches!(error, ReconcileError::MalformedPin { .. })); - let output = Command::new("git") - .args(["for-each-ref", "refs/sce/mutation-cursor/main/"]) - .current_dir(&fixture.root) - .output() - .expect("for-each-ref"); + let ReconcileError::MalformedPin { ref_name, .. } = &error else { + panic!("expected MalformedPin, got {error:?}"); + }; + assert_eq!(ref_name, "refs/sce/mutation-cursor/main/not-a-tree"); assert_eq!( - String::from_utf8_lossy(&output.stdout).lines().count(), - 2, - "no ref may be deleted" + ref_inventory(&fixture.root), + inventory_before, + "no ref may be deleted, replaced or retargeted" ); let StateRead::Valid(state) = fixture.stored_state().await else { panic!("expected recorded state"); }; assert_eq!(state.consecutive_failures, 1); } - -#[tokio::test(flavor = "multi_thread")] -async fn ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock() { - let fixture = Fixture::new("https://example.invalid/org/repo-a.git"); - fixture.create_db().await; - pin_orphan(&fixture.root).await; - let git_dir = resolve_git_dir(&fixture.root).await.expect("git dir"); - let path = state_path(&git_dir); - - let mut observed = Vec::new(); - let result = reconcile_explicit_with( - &fixture.root, - async || open_authoritative_db_at_state_root(&fixture.root, &fixture.state_root).await, - || NOW, - write_state_atomically, - |phase| { - observed.push((phase, advise_if_due(&fixture.root, || NOW), path.exists())); - }, - Duration::from_secs(10), - ) - .await; - - assert!(matches!(result, ExplicitOutcome::Completed(_))); - assert_eq!( - observed - .iter() - .map(|(phase, _, _)| *phase) - .collect::>(), - vec![ReconcilePhase::DbOpened, ReconcilePhase::PinsInventoried] - ); - assert!(observed - .iter() - .all(|(_, advice, present)| matches!(advice, AdvisoryOutcome::Busy) && !present)); - assert!(fixture.pins().await.is_empty()); - assert!(matches!( - advise_if_due(&fixture.root, || NOW), - AdvisoryOutcome::NoAction - )); -} diff --git a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs index 5eb35d3b8..e95d5c313 100644 --- a/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs +++ b/cli/src/services/mutation_trace/runtime/ref_maintenance/tests/coordination.rs @@ -174,15 +174,38 @@ async fn ref_reconciliation_real_coordinate_waits_for_a_pass_parked_after_pin_in assert_coordinate_blocked_until_pass_releases(ReconcilePhase::PinsInventoried).await; } -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries() -{ +#[derive(Clone, Copy)] +enum Boundary { + Close, + Flush, +} + +impl Boundary { + fn payload(self) -> MutationScopePayload { + match self { + Boundary::Close => close_payload(SCOPE, "close-1"), + Boundary::Flush => MutationScopePayload::Flush, + } + } + + fn scope_after_success(self) -> ScopeStatus { + match self { + Boundary::Close => ScopeStatus::Closed, + Boundary::Flush => ScopeStatus::Active, + } + } +} + +async fn assert_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries( + boundary: Boundary, +) { let fixture = started_fixture().await; let git_dir = git_dir_of(&fixture).await; let marker = ExternalTaintMarker::new(&git_dir); super::support::write_file(&fixture.root, "work.txt", "work\n"); let before = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; let refs_before = sce_refs(&fixture.root); + let pins_before = pinned_trees(&fixture.root, "main"); assert_eq!(before.scope, Some(ScopeStatus::Active)); let mut pass = spawn_parked_pass(&fixture, ReconcilePhase::DbOpened); @@ -190,10 +213,17 @@ async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_no let advisories = Arc::new(AtomicU32::new(0)); let contended = Arc::new(AtomicU32::new(0)); + let counting = |advisories: &Arc| { + let advisories = Arc::clone(advisories); + move |root: &Path| { + advisories.fetch_add(1, Ordering::SeqCst); + no_advice(root) + } + }; let failed = run_hook( &fixture.root, &fixture.db_path(), - close_payload(SCOPE, "close-1"), + boundary.payload(), Duration::from_millis(300), { let contended = Arc::clone(&contended); @@ -202,13 +232,7 @@ async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_no } }, || {}, - { - let advisories = Arc::clone(&advisories); - move |root| { - advisories.fetch_add(1, Ordering::SeqCst); - no_advice(root) - } - }, + counting(&advisories), ) .await .expect_err("the lock cannot be acquired while the pass holds it"); @@ -227,6 +251,7 @@ async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_no before ); assert_eq!(sce_refs(&fixture.root), refs_before); + assert_eq!(pinned_trees(&fixture.root, "main"), pins_before); assert!(!marker.exists().expect("marker")); assert_eq!(state_bytes(&git_dir), None); @@ -237,17 +262,11 @@ async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_no let output = run_hook( &fixture.root, &fixture.db_path(), - close_payload(SCOPE, "close-1"), + boundary.payload(), WORKTREE_LOCK_TIMEOUT, || {}, || {}, - { - let advisories = Arc::clone(&advisories); - move |root| { - advisories.fetch_add(1, Ordering::SeqCst); - no_advice(root) - } - }, + counting(&advisories), ) .await .expect("the same boundary succeeds once the maintenance pass released the lock"); @@ -256,7 +275,7 @@ async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_no assert_eq!(state_bytes(&git_dir), after_pass_state); let after = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; - assert_eq!(after.scope, Some(ScopeStatus::Closed)); + assert_eq!(after.scope, Some(boundary.scope_after_success())); assert_eq!(after.worktree.revision, before.worktree.revision + 1); assert_ne!(after.worktree.cursor_tree, before.worktree.cursor_tree); assert!(pinned_trees(&fixture.root, "main").contains(&after.worktree.cursor_tree.0)); @@ -265,66 +284,16 @@ async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_no } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn ref_reconciliation_flush_lock_timeout_under_maintenance_commits_nothing_and_retries() { - let fixture = started_fixture().await; - let git_dir = git_dir_of(&fixture).await; - let marker = ExternalTaintMarker::new(&git_dir); - super::support::write_file(&fixture.root, "work.txt", "work\n"); - let before = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; - let refs_before = sce_refs(&fixture.root); - - let mut pass = spawn_parked_pass(&fixture, ReconcilePhase::DbOpened); - await_signal(&mut pass.parked).await; - - let advisories = Arc::new(AtomicU32::new(0)); - let counting = |advisories: &Arc| { - let advisories = Arc::clone(advisories); - move |root: &Path| { - advisories.fetch_add(1, Ordering::SeqCst); - no_advice(root) - } - }; - let failed = run_hook( - &fixture.root, - &fixture.db_path(), - MutationScopePayload::Flush, - Duration::from_millis(300), - || {}, - || {}, - counting(&advisories), - ) - .await - .expect_err("the lock cannot be acquired while the pass holds it"); - assert!(format!("{failed:#}").contains("Timed out after")); - assert_eq!(advisories.load(Ordering::SeqCst), 0); - assert_eq!( - durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await, - before - ); - assert_eq!(sce_refs(&fixture.root), refs_before); - assert!(!marker.exists().expect("marker")); - assert_eq!(state_bytes(&git_dir), None); - - assert!(matches!(pass.finish().await, ExplicitOutcome::Completed(_))); +async fn ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries() +{ + assert_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries(Boundary::Close) + .await; +} - let output = run_hook( - &fixture.root, - &fixture.db_path(), - MutationScopePayload::Flush, - WORKTREE_LOCK_TIMEOUT, - || {}, - || {}, - counting(&advisories), - ) - .await - .expect("the same flush succeeds once the maintenance pass released the lock"); - assert_eq!(output, ""); - assert_eq!(advisories.load(Ordering::SeqCst), 1); - let after = durable_snapshot(&fixture.db_path(), &main_worktree(), SCOPE).await; - assert_eq!(after.worktree.revision, before.worktree.revision + 1); - assert_ne!(after.worktree.cursor_tree, before.worktree.cursor_tree); - assert!(pinned_trees(&fixture.root, "main").contains(&after.worktree.cursor_tree.0)); - assert!(!marker.exists().expect("marker")); +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ref_reconciliation_flush_lock_timeout_under_maintenance_commits_nothing_and_retries() { + assert_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries(Boundary::Flush) + .await; } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] diff --git a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs index 73a4583a0..50c7f63f7 100644 --- a/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs +++ b/cli/src/services/mutation_trace/runtime/ref_reconciliation.rs @@ -1,7 +1,5 @@ use std::collections::BTreeSet; use std::path::Path; -#[cfg(test)] -use std::time::Duration; use anyhow::Result; @@ -10,16 +8,9 @@ use crate::services::agent_trace_db::repository::RepositoryAgentTraceDb; use crate::services::mutation_trace::store::MutationTraceStore; use crate::services::mutation_trace::types::TreeId; -#[cfg(test)] -use super::git_snapshot::resolve_git_dir; use super::git_snapshot::{GitSnapshotService, PinInventoryError, PinnedRef}; -#[cfg(test)] -use super::worktree_lock::acquire_inner_async; use super::worktree_lock::{WorktreeLock, WorktreeLockError}; -#[cfg(test)] -const RECONCILIATION_LOCK_TIMEOUT: Duration = Duration::from_secs(10); - #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct ReconciliationReport { pub local_required: usize, @@ -76,38 +67,6 @@ impl std::fmt::Display for ReconcileError { impl std::error::Error for ReconcileError {} -#[cfg(test)] -pub async fn reconcile_worktree

( - repository_root: &Path, - open_db: P, -) -> std::result::Result -where - P: std::ops::AsyncFnOnce() -> Result, -{ - reconcile_worktree_inner(repository_root, open_db, || {}).await -} - -#[cfg(test)] -pub(super) async fn reconcile_worktree_inner( - repository_root: &Path, - open_db: P, - on_lock_contention: F, -) -> std::result::Result -where - P: std::ops::AsyncFnOnce() -> Result, - F: FnOnce() + Send + 'static, -{ - let git_dir = resolve_git_dir(repository_root) - .await - .map_err(ReconcileError::GitDir)?; - - let lock = acquire_inner_async(&git_dir, RECONCILIATION_LOCK_TIMEOUT, on_lock_contention) - .await - .map_err(ReconcileError::Lock)?; - - reconcile_with_held_lock(repository_root, &lock, open_db, |_| {}).await -} - #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(super) enum ReconcilePhase { DbOpened, diff --git a/context/cli/mutation-trace-ref-reconciliation.md b/context/cli/mutation-trace-ref-reconciliation.md index 831bfcca3..6c53a0809 100644 --- a/context/cli/mutation-trace-ref-reconciliation.md +++ b/context/cli/mutation-trace-ref-reconciliation.md @@ -67,9 +67,8 @@ each maintenance operation acquires it **exactly once**: no opener parameter; the injection seam is private to `ref_maintenance`. - `advise_if_due` takes one zero-wait `try_lock`, has no Git/DB/store parameters, and never reaches reconciliation. -- The lock-acquiring wrappers `reconcile_worktree` and `reconcile_worktree_inner` - (`acquire_inner_async` with `RECONCILIATION_LOCK_TIMEOUT`, then delegate) are - `#[cfg(test)]` only; production uses `reconcile_explicit`. +- `reconcile_with_held_lock` is the only reconciliation entrypoint; it never + acquires the lock. The former test-only lock-acquiring wrappers were removed. Mutating Git work receives `lock.lease()` and moves it into the `spawn_blocking` worker, so dropping the caller future cannot release the file lock while a ref diff --git a/context/plans/mutation-cursor-ref-reconciliation-wiring.md b/context/plans/mutation-cursor-ref-reconciliation-wiring.md index 2fb946b31..0483ce975 100644 --- a/context/plans/mutation-cursor-ref-reconciliation-wiring.md +++ b/context/plans/mutation-cursor-ref-reconciliation-wiring.md @@ -357,6 +357,7 @@ Record actual outcomes in the validation report; compilation alone does not vali - Reopened: 2026-10-09 (the earlier `done`/`synced` marking was unsupported: S6, S7, S16, S17, S18b and S33 were listed as "not added" or covered only through fake closures, helper tests or enum mappings). - Files changed (reopening pass): `cli/src/services/mutation_trace/runtime/ref_maintenance/tests.rs` and new `ref_maintenance/tests/{support,coordination,worktrees,cross_process,fail_closed,state_recovery}.rs`; `cli/src/app/doctor_reconciliation_cli_tests.rs` (new, registered in `cli/src/app.rs`); `ref_advisory.rs` (tests); `git_snapshot.rs` (deletion-cancellation test); production test seams only — `coordinator.rs` (`coordinate_inner` takes `lock_timeout`; `coordinate` passes `WORKTREE_LOCK_TIMEOUT`, so production behaviour is unchanged), `protected_worktree.rs` (`acquire_with_timeout` is `pub(super)`), `git_snapshot.rs` (`delete_pins_inner` forwards `on_worker_entered`; the dead `run_ref_mutation` wrapper was removed), `hooks/mutation_scope.rs` (`drive_mutation_scope` is `pub(crate)`). No protocol, Quint, dispatch, runtime or timeout changes. - Findings that shaped the tests: (1) a parked pass must run on its own thread and runtime — parking inside `tokio::spawn` blocks a worker and starves the process/IO driver the coordinator needs, which made the coordinator stall before it ever tried the lock; (2) `sce doctor --fix` runs a separate global-state repair that creates the Agent Trace DB parent directory before reconciliation, so the CLI-level missing-DB guarantee is "no DB file or sidecars", not "no parent directory"; (3) doctor failure kinds are `agent_trace_db_missing` and `agent_trace_db_repository_mismatch`, not a generic `agent_trace_db_unavailable`. + - Test-suite simplification (2026-10-09, PR #305 audit; no production behavior changed; scenario statuses unchanged): 19 registrations removed (498 → 479). Deleted: `ref_reconciliation_production_entrypoint_accepts_only_a_repository_root` (signature-only), `ref_reconciliation_worktree_wrapper_acquires_once_and_releases` (with the then-unused `#[cfg(test)]` `reconcile_worktree`, `reconcile_worktree_inner` and `RECONCILIATION_LOCK_TIMEOUT`), `ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock` (superseded by the two independent-task S15 tests), `advisory_runs_after_marker_clear_after_commit_with_success_result` (subset of `marker_clear_after_commit_logs_taint_warning_and_still_advises_once`). Consolidated into tables: five `maintenance_state` pre-rename tests → `maintenance_state_pre_rename_failures_preserve_previous_state`; two post-rename tests → `maintenance_state_post_rename_failures_report_uncertain_durability`; two timestamp tests → `maintenance_advisory_accepts_valid_timestamp_offsets_without_rewrite`; two unusable-state tests → `maintenance_advisory_reanchors_unusable_state_variants`; four doctor JSON tests → `json_ref_reconciliation_fix_results_match_contract` + `json_completed_state_persist_failed_results_match_contract` (whole-object equality); six doctor text tests → `reconciliation_fix_results_render_expected_text`. The duplicate `unreadable` leg was dropped from the S26 table. Close/Flush S16 tests share `assert_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries(Boundary)` and keep their exact names. Strengthened: malformed ref compares the full `refs/sce/` inventory (name, object id, symref) and the `MalformedPin` ref name; unreadable previous state asserts `AgentTraceDbUnavailable` → typed `ExistingRepositoryDbError::Missing`, untouched state directory and pins; the verified opener test proves the statement valid on a writable connection first and asserts `database is readonly`; `Sandbox::run` now asserts the real `ExitCode` (typed `==`) for every doctor invocation; the non-database lifecycle test asserts `AgentTraceDbConnectionFailed`; the pre-rename table also checks byte identity, absent-file and recovery. The guard-removal records below remain historical: test names they cite that no longer exist are covered by the consolidated equivalents named in the map above. Guard re-check: forcing `read_only(false)` in `db/mod.rs` makes `ref_reconciliation_verified_open_is_read_only` fail (restored). Verification: full CLI suite 479 passed, 0 failed; focused `maintenance_state` 6, `ref_advisory` 24, `ref_reconciliation` 48, `doctor` 19, `mutation_scope` 24, `lifecycle` 9 (all non-zero); 8 consecutive passes of the 62 `doctor_reconciliation_cli`/`ref_reconciliation`/cancellation/lock-timeout tests; clippy `-D warnings` clean; `nix flake check` passed; `git diff --check` clean. - Verification, remediation pass (2026-10-09): full CLI suite 495 passed, 0 failed (two new doctor CLI tests); focused `doctor_reconciliation_cli` 7, `ref_reconciliation` 50, `mutation_scope` 25, `maintenance` 85 (all non-zero); new doctor CLI tests 5 consecutive passes; every `ref_maintenance::tests` test and the cancellation tests 20 consecutive passes; `clippy --all-targets --all-features -D warnings` clean; `git diff --check` clean; `cargo fmt` made no change; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect). - Verification (earlier pass, 2026-10-09): full CLI suite 493 passed, 0 failed; focused `ref_reconciliation` 50, `mutation_scope` 25, `doctor` 25, `maintenance` 83 (all non-zero); `clippy --all-targets --all-features -D warnings` clean; `nix flake check` passed (clippy, fmt, CLI tests, Quint connect); `git diff --check` clean; 20 consecutive passes of every `ref_maintenance::tests::*` and cancellation test, 5 consecutive passes of the doctor CLI tests; no added comments, sleeps, `allow(dead_code)`, `dyn`, `BoxFuture`, detached tasks or extra production runtimes. - Guard-removal checks actually performed: repository-wide roots replaced by worktree-local roots → S3 and S13 tests fail; deletion worker drops its lease → the new deletion-cancellation test and the existing pin-cancellation test fail; advisory ignores lock contention → S15 (both phases) and S33 fail. The remaining guards (S6, S16, S17, S18b, S21) were checked in the remediation pass; see the results table below. @@ -384,18 +385,18 @@ Record actual outcomes in the validation report; compilation alone does not vali | S2 | `rm::worktrees::ref_reconciliation_historical_roots_of_a_real_coordinator_sequence_stay_protected` | Production integration (real coordinator, real Git) | A→B→C→D from real `Start`/`Flush` boundaries: all four trees stay pinned although only D is the cursor; one orphan is reclaimed | Verified | | S3 | `rm::worktrees::ref_reconciliation_pin_of_one_real_worktree_is_retained_while_another_needs_the_tree`; `rm::ref_reconciliation_retains_every_repository_wide_durable_root_and_never_repairs_other_worktrees` | Production integration (real linked worktree); synthetic roots | Main's pin of a tree that only the linked worktree durably requires is retained | Verified | | S4 | `rm::fail_closed::ref_reconciliation_missing_local_pin_fails_closed_while_other_orphans_exist` | Pass-level | Missing local required pin returns `MissingRequiredPins` and deletes nothing, even with an eligible orphan present | Verified | - | S5 | `rm::ref_reconciliation_malformed_namespace_ref_deletes_nothing_and_records_failure` | Pass-level | Symbolic ref in the namespace yields `MalformedPin`, zero deletion, failure recorded | Verified | + | S5 | `rm::ref_reconciliation_malformed_namespace_ref_deletes_nothing_and_records_failure` | Pass-level | Symbolic ref in the namespace yields `MalformedPin` naming that ref; the full `refs/sce/` inventory (ref name, object id, symref target) is identical before and after, so a delete-and-replace cannot pass; failure recorded | Verified | | S6 | `rm::coordination::ref_reconciliation_real_coordinate_waits_for_a_pass_parked_after_db_open`; `…_after_pin_inventory` | Production integration (real `coordinate_inner`, real lock) | Coordinator hits contention, opens no DB and arms no marker while the pass holds the lock; the pin created after release survives; only the orphan is deleted; new pin precedes the durable-root commit; lock free afterwards | Verified | | S7 | `services::mutation_trace::runtime::git_snapshot::tests::cancelled_delete_keeps_worktree_lock_until_git_ref_deletion_completes` | Cancellation integration (real `delete_pins`) | Dropping the caller and outer lock owner leaves the lock held by the parked worker; after release the pin is gone, no `.lock` files remain, a new pin proceeds. Existing `cancelled_pin_keeps_worktree_lock_until_git_ref_mutation_completes` covers the pin side | Verified | | S8 | `rm::ref_reconciliation_retains_every_repository_wide_durable_root_and_never_repairs_other_worktrees` (second pass); CLI test second `--fix` run | Pass-level; CLI | Second pass deletes nothing | Verified | - | S9 | `app::doctor_reconciliation_cli_tests::doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable` (persist-warning); `…::doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once` (plain doctor, completed); `…::doctor_fix_reports_missing_database_without_creating_it_or_deleting_refs` and `…::doctor_fix_reports_repository_mismatch_without_touching_the_database_or_refs` (failed); `…::doctor_fix_reports_busy_when_the_worktree_lock_is_held_and_keeps_other_repairs` (busy); `services::doctor::render::reconciliation_render_tests::json_completed_state_persist_failed_distinguishes_warning_kinds` and `services::doctor::reconciliation_fix_tests::*` (mapping tests for `NotApplied`/`DurabilityUncertain`); `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome`; `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}` | CLI end-to-end for completed, failed, busy and `completed_state_persist_failed` (kind `previous_state_unreadable`, via a directory at the state path: orphan deleted, counts 1/0/0, `failure` null, detail reports the warning without "failed", directory untouched, exit success); persistence layer and render/mapping unit tests for `NotApplied` and `DurabilityUncertain` | Plain doctor mutates neither refs nor state; `--fix` reports the true result and the CLI exit status is asserted. At CLI level only `previous_state_unreadable` is exercised for the persist-warning outcome; `not_applied` and `durability_uncertain` are covered at the persistence and rendering layers by design (no CLI fault seam) | Verified | + | S9 | `app::doctor_reconciliation_cli_tests::doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable` (persist-warning); `…::doctor_inspection_is_read_only_and_fix_reconciles_the_orphan_pin_once` (plain doctor, completed); `…::doctor_fix_reports_missing_database_without_creating_it_or_deleting_refs` and `…::doctor_fix_reports_repository_mismatch_without_touching_the_database_or_refs` (failed); `…::doctor_fix_reports_busy_when_the_worktree_lock_is_held_and_keeps_other_repairs` (busy); `services::doctor::render::reconciliation_render_tests::{json_ref_reconciliation_fix_results_match_contract (cases completed, failed without warning, failed with warning, busy), json_completed_state_persist_failed_results_match_contract (cases not_applied, durability_uncertain, previous_state_unreadable)}` (whole-object JSON equality per case) and `services::doctor::reconciliation_fix_tests::reconciliation_fix_results_render_expected_text` (table: completed, pre-rename failure, post-rename failure, unreadable previous state, failed, busy; required and forbidden text per case) (mapping tests for `NotApplied`/`DurabilityUncertain`); `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome`; `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}` | CLI end-to-end for completed, failed, busy and `completed_state_persist_failed` (kind `previous_state_unreadable`, via a directory at the state path: orphan deleted, counts 1/0/0, `failure` null, detail reports the warning without "failed", directory untouched, exit success); persistence layer and render/mapping unit tests for `NotApplied` and `DurabilityUncertain` | Plain doctor mutates neither refs nor state; `--fix` reports the true result and the CLI exit status is asserted. At CLI level only `previous_state_unreadable` is exercised for the persist-warning outcome; `not_applied` and `durability_uncertain` are covered at the persistence and rendering layers by design (no CLI fault seam) | Verified | | S10 | `rm::ref_reconciliation_failure_streak_counts_but_lock_contention_never_does`; `rm::fail_closed::ref_reconciliation_second_explicit_pass_is_busy_and_records_exactly_one_pass`; CLI busy test | Pass-level; CLI | Lock contention returns `Skipped(Busy)`, leaves state byte-identical and is not a failure | Verified | | S11 | `rm::fail_closed::ref_reconciliation_database_failure_never_arms_the_external_taint_marker`; `rm::ref_reconciliation_missing_database_creates_nothing_and_deletes_nothing` | Pass-level | DB failure leaves refs untouched, records a streak, never arms the marker | Verified | - | S12 | `rm::ref_reconciliation_lock_held_implementation_never_reacquires_the_lock`; `rm::ref_reconciliation_worktree_wrapper_acquires_once_and_releases`; `rm::ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock`; S6/S15 tests | Pass-level; production integration | Single acquisition per operation, shown by bounded completion under the existing self-contending lock (no acquisition counter) | Verified | + | S12 | `rm::ref_reconciliation_lock_held_implementation_never_reacquires_the_lock`; S6/S15 tests (the removed test-only `reconcile_worktree` wrapper test and synchronous advisory-busy test are superseded by these and by the production `reconcile_explicit` lock-once path exercised in S6/S15/S16/S21) | Pass-level; production integration | Single acquisition per operation, shown by bounded completion under the existing self-contending lock (no acquisition counter) | Verified | | S13 | `rm::worktrees::ref_reconciliation_missing_pin_in_another_real_worktree_neither_fails_nor_is_repaired` | Production integration (real linked worktree) | Reconciling main reclaims its orphan, keeps the pin protecting the linked tree, never recreates the linked worktree's missing pin, does not fail; the linked worktree's own pass fails closed | Verified | - | S14 | Behavioral: `services::hooks::mutation_scope::advisory_trigger_tests::{advisory_runs_once_after_completed_close_and_flush, advisory_runs_after_marker_clear_after_commit_with_success_result, advisory_never_runs_for_start_or_advance, advisory_never_runs_after_failed_coordinate, advisory_never_runs_for_abandon}` (when it runs); `rm::coordination::{ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries, ref_reconciliation_flush_lock_timeout_under_maintenance_commits_nothing_and_retries}` (real failed coordinate: advisory count 0); `rm::coordination::ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock` (real `Close`: refs identical before/after the advisory, marker snapshot identical, commit unchanged, zero-wait lock acquirable); `rm::coordination::ref_reconciliation_advisory_failures_after_a_real_commit_never_change_the_hook_result` (errors do not change the result); `rm::coordination::ref_reconciliation_marker_clear_failure_after_commit_keeps_taint_for_the_next_coordinate` (marker fingerprint identical across the advisory); `rm::coordination::ref_reconciliation_advisory_after_a_real_commit_is_busy_when_a_pass_wins_the_handoff` (refs and state untouched by the advisory). Structural: call graph below | Hook policy (fake coordinate); production integration (real coordinator, real Git) for ref delta; static inspection | Advisory fires only after a durably completed `Close`/`Flush` and never after a failed coordinate, `Start`, `Advance` or `Abandon`; it adds no ref delta, never modifies the external-taint marker, its errors never change a committed result, and the coordinator's lock is already released. Structural, from `advise_after_completed_boundary(&Path)`: it calls only `advise_if_due(repository_root, system_unix_time_ms)` → `derive_git_dir` (filesystem walk, `std::fs`, no Git subprocess), `read_state`/`plan_advisory`/`normalize`/`evaluate` (pure or `std::fs`/`std::io`, capped read), the synchronous `worktree_lock::acquire_inner` with `Duration::ZERO` (one `try_lock`; returns `TimedOut` before any sleep), `write_state_atomically` (staged file, rename, directory sync) and `outcome_after_write` (pure). Production code in `ref_advisory.rs` and `maintenance_state.rs` has no `crate::` import, no `async`/`await`/`tokio`/`spawn_blocking`, no `Command`, no Git invocation (the only `git` token is the `.git` path component in `derive_git_dir`), and no reference to `reconcile_explicit`, `reconcile_with_held_lock`, `GitSnapshotService`, `list_pins`, `delete_pins`, `RepositoryAgentTraceDb` or a store. The `async`/`spawn_blocking` hits in `worktree_lock.rs` are only `acquire_async`/`acquire_inner_async`, which the advisory does not call. The only production caller is the injected seam in `hooks/mutation_scope.rs`. Verified through static production call-graph inspection and real-hook ref-delta assertions. No spy/panic seams added. | Verified | - | S15 | `rm::coordination::ref_reconciliation_independent_advisory_task_is_busy_while_a_pass_is_parked_after_db_open`; `…_after_pin_inventory`; earlier synchronous `rm::ref_reconciliation_advisory_is_busy_while_an_explicit_pass_holds_the_lock` | Concurrency integration (independent blocking task) | Advisory returns `Busy` without waiting, state and refs unchanged, pass continues, later advisory follows the resulting state (`NoAction`, then `Advised` after the window). The synchronous test is retained but is not independent concurrency | Verified | - | S16 | `rm::coordination::ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Close`); `…_flush_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Flush`) | Production integration (real coordinator, injected 300 ms timeout) | Existing lock-timeout error; no durable transition, no pins, no marker, no advisory after the failure; the same boundary succeeds after release | Verified | + | S14 | Behavioral: `services::hooks::mutation_scope::advisory_trigger_tests::{advisory_runs_once_after_completed_close_and_flush, marker_clear_after_commit_logs_taint_warning_and_still_advises_once, advisory_never_runs_for_start_or_advance, advisory_never_runs_after_failed_coordinate, advisory_never_runs_for_abandon}` (when it runs); `rm::coordination::{ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries, ref_reconciliation_flush_lock_timeout_under_maintenance_commits_nothing_and_retries}` (real failed coordinate: advisory count 0); `rm::coordination::ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock` (real `Close`: refs identical before/after the advisory, marker snapshot identical, commit unchanged, zero-wait lock acquirable); `rm::coordination::ref_reconciliation_advisory_failures_after_a_real_commit_never_change_the_hook_result` (errors do not change the result); `rm::coordination::ref_reconciliation_marker_clear_failure_after_commit_keeps_taint_for_the_next_coordinate` (marker fingerprint identical across the advisory); `rm::coordination::ref_reconciliation_advisory_after_a_real_commit_is_busy_when_a_pass_wins_the_handoff` (refs and state untouched by the advisory). Structural: call graph below | Hook policy (fake coordinate); production integration (real coordinator, real Git) for ref delta; static inspection | Advisory fires only after a durably completed `Close`/`Flush` and never after a failed coordinate, `Start`, `Advance` or `Abandon`; it adds no ref delta, never modifies the external-taint marker, its errors never change a committed result, and the coordinator's lock is already released. Structural, from `advise_after_completed_boundary(&Path)`: it calls only `advise_if_due(repository_root, system_unix_time_ms)` → `derive_git_dir` (filesystem walk, `std::fs`, no Git subprocess), `read_state`/`plan_advisory`/`normalize`/`evaluate` (pure or `std::fs`/`std::io`, capped read), the synchronous `worktree_lock::acquire_inner` with `Duration::ZERO` (one `try_lock`; returns `TimedOut` before any sleep), `write_state_atomically` (staged file, rename, directory sync) and `outcome_after_write` (pure). Production code in `ref_advisory.rs` and `maintenance_state.rs` has no `crate::` import, no `async`/`await`/`tokio`/`spawn_blocking`, no `Command`, no Git invocation (the only `git` token is the `.git` path component in `derive_git_dir`), and no reference to `reconcile_explicit`, `reconcile_with_held_lock`, `GitSnapshotService`, `list_pins`, `delete_pins`, `RepositoryAgentTraceDb` or a store. The `async`/`spawn_blocking` hits in `worktree_lock.rs` are only `acquire_async`/`acquire_inner_async`, which the advisory does not call. The only production caller is the injected seam in `hooks/mutation_scope.rs`. Verified through static production call-graph inspection and real-hook ref-delta assertions. No spy/panic seams added. | Verified | + | S15 | `rm::coordination::ref_reconciliation_independent_advisory_task_is_busy_while_a_pass_is_parked_after_db_open`; `…_after_pin_inventory`; | Concurrency integration (independent blocking task) | Advisory returns `Busy` without waiting, state and refs unchanged, pass continues, later advisory follows the resulting state (`NoAction`, then `Advised` after the window). The earlier synchronous in-callback test was deleted: it asserted nothing these two tests do not (`Busy`, no state file while parked, refs intact, `NoAction` afterwards) and was not independent concurrency | Verified | + | S16 | `rm::coordination::ref_reconciliation_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Close`); `…_flush_lock_timeout_under_maintenance_commits_nothing_and_retries` (`Flush`); both are thin `#[tokio::test]` wrappers over `assert_coordinate_lock_timeout_under_maintenance_commits_nothing_and_retries(Boundary)`, so each stays separately runnable | Production integration (real coordinator, injected 300 ms timeout) | Existing lock-timeout error; no durable transition, no pins, no marker, no advisory after the failure; the same boundary succeeds after release | Verified | | S17 | `rm::coordination::ref_reconciliation_advisory_runs_after_the_real_coordinator_released_its_lock` | Hook/coordinator integration | Zero-wait `WorktreeLock` acquisition succeeds inside the advisory callback; the commit is already durable and unchanged afterwards; refs unchanged; empty stdout | Verified | | S18a | `rm::coordination::ref_reconciliation_advisory_failures_after_a_real_commit_never_change_the_hook_result`; `advisory_trigger_tests::{diagnostic_write_failure_does_not_change_mutation_result, state_failures_log_warn_and_write_diagnostic_without_claiming_advice, quiet_outcomes_log_debug_and_emit_no_diagnostic}` | Hook/coordinator integration; hook policy | Write-failed, corrupt and unreadable-state advisories leave the hook result, committed state, refs and marker unchanged | Verified | | S18b | `rm::coordination::ref_reconciliation_marker_clear_failure_after_commit_keeps_taint_for_the_next_coordinate` | Recovery integration (real coordinator; unlink fault injected inside the protected section) | Failed marker clear is a durable success; marker fingerprint identical across the advisory; after the fault is lifted the next real `coordinate()` takes the inherited-taint path (live scope `Abandoned`, health restored, revision advanced ≥2, marker cleared) whereas an untainted control flush left it `Active` | Verified | @@ -403,20 +404,20 @@ Record actual outcomes in the validation report; compilation alone does not vali | S20 | `ref_advisory::tests::{maintenance_advisory_advises_once_per_window_and_persists_before_returning, maintenance_failure_advisory_emits_once_and_doctor_still_recommends_repair, maintenance_failure_advisory_repeats_after_the_cadence_window, maintenance_failure_after_recent_success_is_recommended_and_advised_despite_older_reminder}` | Unit (advisory) | One time-based advice per window, persisted first; failure advice once then cadence | Verified | | S21 | `rm::cross_process::ref_reconciliation_advisory_lock_serializes_independent_processes`; `rm::fail_closed::ref_reconciliation_second_explicit_pass_is_busy_and_records_exactly_one_pass` | Independent OS processes (self re-exec, stdin/stdout rendezvous); in-process | Parked advisory in one process, `Busy` in a second, `NoAction` in a third; exactly one advice; no state write while parked. Explicit-vs-explicit is in-process only | Verified | | S22 | `ref_advisory::tests::maintenance_advisory_anchor_write_failure_reports_state_write_failed_without_advice`; `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome` | Unit; pass-level | Persistence failure: no advice; explicit pass reports `CompletedStatePersistFailed` with deletions intact | Verified | - | S23 | `maintenance_state::tests::maintenance_state_recovers_after_interrupted_pre_rename_write`; `ref_advisory::tests::maintenance_advisory_rename_failure_preserves_state_and_next_call_can_advise` | Unit | Interrupted write preserves previous state; fresh call advises at the right time | Verified | - | S24 | `ref_advisory::tests::{maintenance_advisory_treats_timestamps_at_or_just_ahead_of_now_as_valid_age_zero, maintenance_advisory_preserves_timestamps_within_skew_tolerance, maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately, maintenance_advisory_clock_rollback_beyond_tolerance_normalizes_while_within_it_does_not}`; `rm::state_recovery::ref_reconciliation_invalid_last_advised_does_not_suppress_a_due_advisory`; `maintenance_advisory_advises_once_per_window_and_persists_before_returning` | Unit (injected clocks) | Cases 1–7: equal and near-future valid; exactly +5 min valid; beyond tolerance (anchor, last_success, last_advised) normalized once; rollback beyond vs within tolerance; byte-identical repeats; window recurrence | Verified | - | S25 | `ref_advisory::tests::{maintenance_advisory_anchors_once_then_leaves_state_untouched, maintenance_advisory_recovers_corrupt_state_by_anchoring, maintenance_advisory_recovers_wrong_version_and_oversized_state_by_anchoring}`; `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition` | Unit; pass-level | Missing, corrupt, wrong-version and oversized state recover without failure and are rewritten valid | Verified | - | S26 | `app::doctor_reconciliation_cli_tests::doctor_fix_recovers_from_every_initial_maintenance_state_condition` (failed streak, expired advisory, invalid far-future timestamps, corrupt JSON, oversized file → real `sce doctor --fix`); `app::doctor_reconciliation_cli_tests::doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable` and the unreadable leg of the table test (shared assertion helper); pass-level: `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition`, `…::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`, `rm::ref_reconciliation_unreadable_previous_state_is_reported_without_overwriting`, `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome` | CLI end-to-end (real repo, real Agent Trace DB, real `sce doctor --fix --format json`); pass-level | From each readable condition the explicit pass runs regardless of advisory eligibility, reclaims the orphan pin, reports `completed` (deleted 1), clears the failure streak and failure fields, sets `last_success`, normalizes far-future timestamps (all `≤ now + FUTURE_SKEW_TOLERANCE`), leaves a state a second `--fix` reads as valid, and exits successfully. From an unreadable state path (a directory) the pass still deletes the orphan, reports `completed_state_persist_failed` with `previous_state_unreadable`, makes no false persistence claim, leaves the directory untouched, and exits successfully | Verified | + | S23 | `maintenance_state::tests::maintenance_state_recovers_after_interrupted_pre_rename_write`; `maintenance_state::tests::maintenance_state_pre_rename_failures_preserve_previous_state` (table: `CreateStaging`/`WriteStaging`/`SyncStaging`/`Rename` with previous state, `Rename` without; byte-identical or absent, no staging leftovers, later write recovers); `ref_advisory::tests::maintenance_advisory_rename_failure_preserves_state_and_next_call_can_advise` | Unit | Interrupted write preserves previous state; fresh call advises at the right time | Verified | + | S24 | `ref_advisory::tests::{maintenance_advisory_accepts_valid_timestamp_offsets_without_rewrite (table: offsets `0`, `+60 s`, `FUTURE_SKEW_TOLERANCE_MS`; `NoAction`, byte-identical state, no recommendation), maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately, maintenance_advisory_clock_rollback_beyond_tolerance_normalizes_while_within_it_does_not}`; `rm::state_recovery::ref_reconciliation_invalid_last_advised_does_not_suppress_a_due_advisory`; `maintenance_advisory_advises_once_per_window_and_persists_before_returning` | Unit (injected clocks) | Cases 1–7: equal and near-future valid; exactly +5 min valid; beyond tolerance (anchor, last_success, last_advised) normalized once; rollback beyond vs within tolerance; byte-identical repeats; window recurrence | Verified | + | S25 | `ref_advisory::tests::{maintenance_advisory_anchors_once_then_leaves_state_untouched, maintenance_advisory_reanchors_unusable_state_variants (table: wrong schema version, oversized file, corrupt JSON; `StateRead::Unusable`, `Anchored`, valid state with `anchor == NOW`, no recommendation, repeat call does not rewrite)}`; `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition` | Unit; pass-level | Missing, corrupt, wrong-version and oversized state recover without failure and are rewritten valid | Verified | + | S26 | `app::doctor_reconciliation_cli_tests::doctor_fix_recovers_from_every_initial_maintenance_state_condition` (failed streak, expired advisory, invalid far-future timestamps, corrupt JSON, oversized file → real `sce doctor --fix`); `app::doctor_reconciliation_cli_tests::doctor_fix_reports_state_persist_failure_when_maintenance_state_is_unreadable` (the table no longer repeats the unreadable case; this dedicated test is the single end-to-end check for `CompletedStatePersistFailed` / `previous_state_unreadable`); pass-level: `rm::state_recovery::ref_reconciliation_explicit_pass_runs_and_rewrites_valid_state_from_every_stored_condition`, `…::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`, `rm::ref_reconciliation_unreadable_previous_state_is_reported_without_overwriting` (primary error `AgentTraceDbUnavailable` with typed `ExistingRepositoryDbError::Missing`, `PreviousStateUnreadable` warning, state directory and pin inventory unchanged), `rm::ref_reconciliation_completed_pass_with_unwritten_state_reports_distinct_outcome` | CLI end-to-end (real repo, real Agent Trace DB, real `sce doctor --fix --format json`); pass-level | From each readable condition the explicit pass runs regardless of advisory eligibility, reclaims the orphan pin, reports `completed` (deleted 1), clears the failure streak and failure fields, sets `last_success`, normalizes far-future timestamps (all `≤ now + FUTURE_SKEW_TOLERANCE`), leaves a state a second `--fix` reads as valid, and exits successfully. From an unreadable state path (a directory) the pass still deletes the orphan, reports `completed_state_persist_failed` with `previous_state_unreadable`, makes no false persistence claim, leaves the directory untouched, and exits successfully | Verified | | S27 | `rm::state_recovery::ref_reconciliation_explicit_success_resets_the_streak_and_failure_keeps_last_success`; `rm::ref_reconciliation_failure_streak_counts_but_lock_contention_never_does` | Pass-level | Success resets streak and sets `last_success`; failure increments without touching `last_success`; busy changes nothing | Verified | | S28 | `rm::ref_reconciliation_missing_database_creates_nothing_and_deletes_nothing`; `rm::ref_reconciliation_verified_open_never_creates_missing_database_or_directories`; `rm::ref_reconciliation_verified_open_rejects_database_removed_before_open`; `app::doctor_reconciliation_cli_tests::doctor_fix_reports_missing_database_without_creating_it_or_deleting_refs` | Pass-level; opener; CLI | Typed `Missing`; no DB file, directory or sidecar created by the open; no refs listed or deleted. The CLI shows the parent directory comes from the separate global-state repair | Verified | | S29 | `rm::state_recovery::ref_reconciliation_verification_failures_leave_the_logical_database_unchanged` (case `incompatible`); `rm::ref_reconciliation_missing_metadata_or_schema_fails_before_pin_inventory` | Pass-level | `IncompatibleSchema`; full-row dump of every table equal before and after; no pin inventory; refs and marker untouched | Verified | | S30 | same test (case `missing_metadata`) | Pass-level | `MissingMetadata`; no `repository_metadata` row inserted (full-row dump equal) | Verified | | S31 | same test (case `mismatch`); `app::doctor_reconciliation_cli_tests::doctor_fix_reports_repository_mismatch_without_touching_the_database_or_refs` | Pass-level; CLI | `RepositoryMismatch`; foreign metadata row intact; refs untouched | Verified | - | S32 | `rm::ref_reconciliation_verified_database_reconciles_orphan_pin_and_records_success` (phase order `DbOpened` → `PinsInventoried`); `rm::ref_reconciliation_verified_open_coexists_with_a_live_multiprocess_wal_writer`; `rm::ref_reconciliation_verified_open_is_read_only` | Pass-level; opener | Identity and schema verified before any ref listing; multiprocess-WAL coexistence; the opened handle rejects writes | Verified | + | S32 | `rm::ref_reconciliation_verified_database_reconciles_orphan_pin_and_records_success` (phase order `DbOpened` → `PinsInventoried`); `rm::ref_reconciliation_verified_open_coexists_with_a_live_multiprocess_wal_writer`; `rm::ref_reconciliation_verified_open_is_read_only` | Pass-level; opener | Identity and schema verified before any ref listing; multiprocess-WAL coexistence; the opened handle rejects a statement first proven valid on a writable connection (rolled-back transaction) with `database is readonly`, and the rows stay empty. Breaking the guard (`read_only(false)` in `db/mod.rs`) makes this test fail | Verified | | S33 | `rm::coordination::ref_reconciliation_advisory_after_a_real_commit_is_busy_when_a_pass_wins_the_handoff` | Concurrency integration (real coordinator; hook parked between coordinate and advisory; pass on its own thread) | Pass takes the lock in the handoff window; advisory returns `Busy`; state and refs unchanged by the advisory; committed `Close` still reports success; only the orphan is deleted by the pass | Verified | | S34 | `ref_doctor::tests::recommendation_flags_unusable_and_invalid_state_without_rewriting` | Unit (doctor inspection) | Read-only doctor flags invalid timestamps and leaves bytes untouched | Verified | | S35 | `ref_advisory::tests::maintenance_advisory_reports_busy_and_leaves_invalid_state_byte_identical`; `rm::state_recovery::ref_reconciliation_busy_explicit_pass_leaves_invalid_state_byte_identical` | Unit; pass-level | Lock-busy advisory and explicit pass leave invalid state byte-identical | Verified | - | S36 | `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}`; `rm::state_recovery::ref_reconciliation_failed_state_write_during_normalization_has_two_distinct_legs` | Unit; pass-level | Pre-rename: previous bytes identical, `StateWriteFailed`/`NotApplied`. Post-rename: new state visible, `AdvisedDurabilityUncertain`/`DurabilityUncertain` when the normalization also recorded advice, or `StateWriteFailed`/`DurabilityUncertain` with no recommendation for an anchor or `NoAction` normalization (`ref_advisory::tests::{maintenance_anchor_post_rename_failure_is_state_write_failed_with_visible_anchor, maintenance_no_action_normalization_post_rename_failure_is_state_write_failed}`); no durable-normalization claim; later invocation recovers | Verified | + | S36 | `ref_advisory::tests::{maintenance_normalization_pre_rename_failure_keeps_previous_bytes_and_recovers_later, maintenance_normalization_post_rename_failure_is_visible_but_durability_is_uncertain}`; `maintenance_state::tests::{maintenance_state_pre_rename_failures_preserve_previous_state, maintenance_state_post_rename_failures_report_uncertain_durability (table: `OpenParentDirectory`, `SyncParentDirectory`; exact phase, `DurabilityUncertain`, new state visible, no `previous state preserved` claim, later write recovers)}`; `rm::state_recovery::ref_reconciliation_failed_state_write_during_normalization_has_two_distinct_legs` | Unit; pass-level | Pre-rename: previous bytes identical, `StateWriteFailed`/`NotApplied`. Post-rename: new state visible, `AdvisedDurabilityUncertain`/`DurabilityUncertain` when the normalization also recorded advice, or `StateWriteFailed`/`DurabilityUncertain` with no recommendation for an anchor or `NoAction` normalization (`ref_advisory::tests::{maintenance_anchor_post_rename_failure_is_state_write_failed_with_visible_anchor, maintenance_no_action_normalization_post_rename_failure_is_state_write_failed}`); no durable-normalization claim; later invocation recovers | Verified | | S37 | `ref_advisory::tests::maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately`; `rm::state_recovery::ref_reconciliation_successful_pass_normalizes_invalid_state_and_a_later_read_does_not_rewrite` | Unit; pass-level | Normalizing advisory and explicit pass persist `t ≤ now + tolerance`; near-future value preserved; repeated read does not rewrite | Verified | | S38 | `ref_advisory::tests::maintenance_advisory_normalizes_far_future_timestamps_and_advises_immediately` (fresh call re-reads the file); `rm::cross_process::ref_reconciliation_advisory_lock_serializes_independent_processes` | Unit; independent processes | A fresh call observes normalized state, performs no rewrite and does not re-fire | Verified | | S39 | `rm::state_recovery::ref_reconciliation_unnormalized_invalid_state_stays_eligible_until_a_successful_advisory` | Pass-level (injected clocks, held lock) | Repeated read-only evaluation and busy advisories never move eligibility or bytes; the next successful advisory fires exactly once and persists normalized state | Verified | @@ -498,7 +499,7 @@ Recorded by T01 (read-only audit, 2026-10-09). Line numbers refer to the tree at ### Commands run -- `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml` -> exit 0 (498 passed, 0 failed) +- `nix develop -c ./scripts/run-cli-cargo.sh test --manifest-path cli/Cargo.toml` -> exit 0 (498 passed, 0 failed before the test-suite simplification; 479 passed, 0 failed after it) - `nix develop -c ./scripts/run-cli-cargo.sh clippy --manifest-path cli/Cargo.toml --all-targets --all-features -- -D warnings` -> exit 0 (clean) - `nix flake check` -> exit 0 (all checks passed; non-host systems omitted) - `nix build .#ci-checks` -> exit 0 From 4d78c5e631a7958adcf2a249f1a7023d05632122 Mon Sep 17 00:00:00 2001 From: David Abram Date: Sat, 10 Oct 2026 16:08:31 +0200 Subject: [PATCH 16/16] runtime: Bound database write retries by contention deadline Allow a retry when its backoff can finish before the contention deadline without requiring the busy timeout to fit in the remaining budget. Add boundary tests covering sleep and immediate retry eligibility. Co-authored-by: SCE --- cli/src/services/db/mod.rs | 50 +++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/cli/src/services/db/mod.rs b/cli/src/services/db/mod.rs index 21ea96681..05f9d76d1 100644 --- a/cli/src/services/db/mod.rs +++ b/cli/src/services/db/mod.rs @@ -665,10 +665,7 @@ fn write_contention_retry_may_sleep( elapsed: std::time::Duration, backoff: std::time::Duration, ) -> bool { - let Some(remaining) = policy.contention_deadline.checked_sub(elapsed) else { - return false; - }; - !remaining.is_zero() && remaining >= backoff.saturating_add(policy.busy_timeout) + elapsed.saturating_add(backoff) < policy.contention_deadline } fn write_contention_retry_may_start_now( @@ -1468,3 +1465,48 @@ mod database_kind_tests { ); } } + +#[cfg(test)] +mod write_contention_eligibility_tests { + use std::time::Duration; + + use super::{ + write_contention_retry_may_sleep, write_contention_retry_may_start_now, + WriteContentionPolicy, + }; + + #[test] + fn retry_eligibility_depends_only_on_backoff_ending_before_deadline() { + let policy = WriteContentionPolicy { + db_name: "agent_trace_db", + max_attempts: 3, + backoff_cap: Duration::from_millis(100), + busy_timeout: Duration::from_secs(1), + contention_deadline: Duration::from_millis(2250), + }; + + for (elapsed_ms, backoff_ms, expected) in [ + (1185, 100, true), + (2200, 100, false), + (2250, 0, false), + (2249, 0, true), + ] { + let elapsed = Duration::from_millis(elapsed_ms); + let backoff = Duration::from_millis(backoff_ms); + assert_eq!( + write_contention_retry_may_sleep(policy, elapsed, backoff), + expected, + "elapsed={elapsed_ms}ms backoff={backoff_ms}ms" + ); + } + + assert!(write_contention_retry_may_start_now( + policy, + Duration::from_millis(2249) + )); + assert!(!write_contention_retry_may_start_now( + policy, + Duration::from_millis(2250) + )); + } +}