Agent Trace DB write contention - #299
Merged
davidabram merged 11 commits intoOct 5, 2026
Merged
davidabram merged 11 commits into
davidabram merged 11 commits into
Conversation
davidabram
added this pull request to stack #298
October 4, 2026 19:16
Document the staged plan for preventing event-ingestion loss during multiprocess-WAL writer contention. The plan combines Turso busy timeouts, a deadline-bounded jittered Agent Trace DB retry, read-only metadata opens, contention observability, and a measured acceptance suite while preserving other database retry behavior. Plan: agent-trace-db-write-contention Tasks: T01–T07 Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
agent-trace-db-write-contention
branch
from
October 4, 2026 19:24
d1d490d to
5f77cf2
Compare
Restore and wire the lock-contention suite to characterize SQLite retry-budget behavior under concurrent Rust API and real hook-process writes. Record the T01 baseline, including lock exhaustion, ingestion loss, latency percentiles, and the no-orphan/no-duplicate assertions, without changing production behavior. Plan: agent-trace-db-write-contention.md (T01) Co-authored-by: SCE <sce@crocoder.dev>
Strengthen the contention harness so strict runs reject lost events, non-zero hook exits, orphan rows, and over-persistence while preserving non-strict characterization behavior. Record the corrected acceptance and baseline contract in the write-contention plan. Plan: agent-trace-db-write-contention Tasks: T01 Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
agent-trace-db-write-contention
branch
from
October 4, 2026 20:10
b99112d to
ff3aaf5
Compare
Install a 500 ms Turso busy timeout on Agent Trace DB connections so transient writer contention waits inside the connection before the existing retry policy runs. Keep other database connection behavior unchanged and add coverage for resolver selection, raw lock acquisition, and production inserts. Plan: agent-trace-db-write-contention, task T02 Co-authored-by: SCE <sce@crocoder.dev>
Allow Agent Trace database busy-timeout and write-contention deadline overrides through the database_retry configuration without expanding local or auth database policy surfaces. Add dedicated schema and typed parsing, preserve existing retry policies, and render configured values in config output. Wire busy-timeout resolution to the Agent Trace database and provide the deadline resolver seam for the follow-up write-retry work. Plan: agent-trace-db-write-contention, task T03 Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
agent-trace-db-write-contention
branch
from
October 4, 2026 21:07
4be46e4 to
3449a79
Compare
Retry replay-safe Agent Trace transactions and idempotent writes with a typed Busy/BusySnapshot policy, bounded full-jitter backoff, and contention-deadline start rule while leaving reads and non-opted-in writes on generic query retry. Route the repository metadata and mutation-trace idempotent writes through the new seam, add focused policy and production contention coverage, and synchronize the implementation context. Plan: agent-trace-db-write-contention (T04) Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
agent-trace-db-write-contention
branch
3 times, most recently
from
October 5, 2026 07:59
0df6939 to
6259452
Compare
Make exhausted Agent Trace write-contention retries observable through a structured error carrying the database, operation, retry-budget, elapsed-time, and busy-cause fields, while emitting one structured tracing event at the exhaustion boundary. Preserve deterministic failures, successful retries, hook stdout, and fail-open behavior, and cover the contract with focused subscriber tests. Update the shared database context and mark T05 complete with its verification evidence. Plan: agent-trace-db-write-contention (T05) Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
agent-trace-db-write-contention
branch
from
October 5, 2026 08:03
6259452 to
1a3971c
Compare
Hook-runtime opens should not contend for the writer lock once repository metadata is already initialized. Read and validate the existing metadata row first, falling back to the existing claim path only for missing or incomplete metadata; add test-only write counting and coverage for the read-only fast path and mismatch handling. Plan: agent-trace-db-write-contention (T06) Co-authored-by: SCE <sce@crocoder.dev>
Align the lock-budget boundary and concurrent writer tests with the bounded busy-timeout and outer-retry contract, asserting attempt caps, exhaustion behavior, and persisted-row safety. Record the measured before-and-after behavior, host-stall failure mode, and supporting documentation for the shared Turso contention contract. This preserves strict contention assertions while treating N=8 as stress characterization and documenting the remaining deadline tradeoff. Ref: context/plans/agent-trace-db-write-contention.md T07 Co-authored-by: SCE <sce@crocoder.dev>
Document the measured Agent Trace DB write-contention contract, supported-load results, stress findings, and policy conclusion so the observed behavior is reproducible and distinct from the guarantees. Add test-only timeline instrumentation for retry attempts and backoff sleeps, and describe the instrumentation in the shared Turso DB contract. The instrumentation is confined to tests; production retry behavior is unchanged. Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
agent-trace-db-write-contention
branch
from
October 5, 2026 14:42
97c6f16 to
8fe6c65
Compare
Increase the Agent Trace busy timeout and write-contention deadline from 500/1250 ms to 1000/2250 ms after supported-load validation exposed bounded retry exhaustion and one lost distinct event under the old defaults. Extend the lock-budget boundary coverage and synchronize generated schema, tests, and durable context with the measured T08 policy decision. The policy remains capped at two attempts with the existing jitter/backoff behavior; exhausting writes now trade roughly one additional second of worst-case blocked-hook latency for coverage of the observed 1.6–1.7 s lock holders. Ref: context/plans/agent-trace-db-write-contention.md (T07, T08) Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
agent-trace-db-write-contention
branch
from
October 5, 2026 15:07
8fe6c65 to
08cc828
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.