Skip to content

Fix the #47 coalescing review: budgets, prefix publication, decode bound - #53

Merged
dbccpage merged 2 commits into
mainfrom
claude/coalesce-review-fixes
Nov 11, 2026
Merged

dbccpage merged 2 commits into
mainfrom
claude/coalesce-review-fixes

Conversation

@dbccpage

Copy link
Copy Markdown
Owner

Requested by Jeremy · project thread

Follow-up to the adversarial review of #47 (already merged). Stacked on #51; only the last commit is new.

Review item Change Regression test
P1-1 receipt budget Every transaction costs max(1, len(change)) event units in both peek_many and apply_many, because an empty transaction still yields an empty_transaction receipt per binding. test_empty_transactions_count_against_the_receipt_budget: two empty txs, max_events=1, max_outputs equal to the binding count. Fails on parent (invalid_generation_state); here it publishes the admissible prefix, then the rest.
P1-2 generation may not fit apply_many publishes the longest prefix that fits. It first cuts to the event budget, then halves on state_limit / state_bytes_limit / invalid_generation_state, before anything commits. It acks only that prefix and carries freshness only if the whole batch went out. A single transaction that does not fit is refused as before. test_an_oversized_generation_publishes_the_largest_prefix_that_fits, test_the_group_is_held_to_max_events (now a prefix, not a refusal), test_a_single_transaction_over_max_events_is_still_refused
P1-3 server-side decode bound A coalesced peek passes upto_lsn = min(frontier, durable_offset + max_bytes), so PostgreSQL decodes at most one max_bytes window of WAL per poll, not max_coalesce × max_bytes. If that window holds no commit, the next transaction is peeked alone up to the frontier (the pre-#47 behavior). Native test_production_coalesced_peek_decodes_a_bounded_wal_window: 3000 commits, max_bytes=128 KiB, max_coalesce=1024. Every coalesced peek's upto is within the window, the window binds, oversized generations fall back to prefixes, and the oracle matches.
P2-4 profiler Instruments peek_many / apply_many. Reports transactions_published, generations_published, transactions_per_s from published transactions, and connections per transaction. Contention evidence adds fraction_of_commits_observed. Phase profile run: 4 polls, 4 generations, 200 transactions.
P2-5 CLI Optional max_coalesce (1–1024, default 64; 1 = strict per-transaction), passed to PostgresWorker and printed by check. Documented in docs/production_profile.md. tests/test_production_cli_config.py
P2-6 overlap apply_many requires transactions[0][1] > previous_offset once duplicates are resolved. test_a_batch_overlapping_the_durable_offset_is_refused

Not done here: the full differential suite across extremum/TopK/joins/set operations, and testing under constrained PostgreSQL memory. Both are worth a separate PR.

Local evidence (PG 16.15 + wal2json + Redis 7): the 4 new runtime tests fail on the parent branch and pass here. Production unit tests: 170 passed. Native suite: 60 passed. black and mypy are clean.


Generated by Claude Code

- Each transaction costs max(1, changes) event units in peek_many and
  apply_many: an empty transaction still yields a receipt per binding.
- apply_many publishes the longest prefix that fits one generation: it
  cuts to the event budget up front, then halves on a size refusal
  (state_limit, state_bytes_limit, invalid_generation_state) before
  anything commits, and acknowledges only that prefix. Freshness is
  carried only when the whole batch is published.
- A coalesced peek decodes only commits within max_bytes of WAL past the
  durable offset; a transaction beyond that window is peeked alone.
- apply_many refuses a batch whose first transaction is not newer than
  the durable offset.
- The CLI admits optional max_coalesce (1 = strict per-transaction).
- The phase profiler instruments peek_many/apply_many and counts published
  transactions; the contention evidence reports the fraction of commits
  a hit observed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194kAmY4QqQGabdoB3xGFsr
@dbccpage dbccpage self-assigned this Nov 11, 2026
@dbccpage
dbccpage added this pull request to stack #54 November 11, 2026 10:39
@dbccpage
dbccpage marked this pull request as ready for review November 11, 2026 10:39
@dbccpage
dbccpage removed this pull request from stack #54 November 11, 2026 11:07
@dbccpage
dbccpage changed the base branch from claude/rd3-freshness-on-publish to main November 11, 2026 11:08
@dbccpage
dbccpage merged commit c448df4 into main Nov 11, 2026
5 checks passed
@dbccpage
dbccpage deleted the claude/coalesce-review-fixes branch November 11, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants