Skip to content

Advance freshness on bootstrap and on applies that drain the slot (RD-3) - #51

Merged
dbccpage merged 3 commits into
mainfrom
claude/rd3-freshness-on-publish
Nov 11, 2026
Merged

dbccpage merged 3 commits into
mainfrom
claude/rd3-freshness-on-publish

Conversation

@dbccpage

Copy link
Copy Markdown
Owner

Requested by Jeremy · project thread

Stacked on #49 (which is stacked on #48). Only the last commit is new here.

What

  • Authority.publish(..., observed_at=None): optional freshness in authority ms. It moves doc["updated"] forward (never back) and refuses a future observation as invalid_freshness_observation before commit.
  • Bootstrap passes the authority time read before CREATE_REPLICATION_SLOT: every commit before it is inside the exported snapshot.
  • Apply passes the poll's view time only when the peek showed this transaction was the last admitted one through the frontier. peek now reads two rows and returns (payload, offset, exhausted). A backlogged apply still does not reset freshness (test_backlog_publication_does_not_reset_source_freshness unchanged).
  • advance_idle passes it too.

Evidence (local PG 16.15 + wal2json + Redis 7)

New test_production_continuous_writes_keep_bounded_reads_fresh_past_max_lag: 35 s of writes at 4 tx/s with the idle path (advance_idle + caught_up) disabled, which models "a commit in every poll". Bounded hit rate in [30 s, 35 s]:

  • parent branch: 0% (661/661 head_miss; the head is withdrawn with source_lag_limit)
  • this PR: 100% (722/722)

Unit test test_publication_with_an_observation_advances_freshness. Full native suite (non-cluster, non-disruptive): 58 passed. test_production_read_contention_during_publication[bounded] failed once in 2 runs. It is the known drain-rate flake: about 25 tx/s drain against 20 writes/s, unchanged by this PR (phase profile 24.3 to 26.4 tx/s on both branches). The fix for that is the batching in #47.

Not in this PR


Generated by Claude Code

Authority.publish takes an optional observed_at (authority ms) and moves
doc['updated'] to it, refusing a future observation before commit.
Bootstrap passes the authority time read before CREATE_REPLICATION_SLOT;
an apply passes the poll's view time only when its peek showed the
transaction was the last one through the frontier (the peek now reads two
rows to tell); advance_idle passes it too. A backlogged apply still does
not reset freshness. Before this, continuous writes with no idle poll
froze 'updated' and every read missed with source_lag_limit after 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194kAmY4QqQGabdoB3xGFsr
@dbccpage dbccpage self-assigned this Nov 11, 2026

Copy link
Copy Markdown
Owner Author

Live Database Containers failed on one test only: test_production_read_contention_during_publication[bounded] ("native source did not catch up within 30 seconds"). Every other native test passed, including all the new ones in this stack.

This failure is not from this PR. It is the known drain-rate limit on main: the worker publishes one generation per WAL transaction, about 13–25 tx/s, while the test writes 20 tx/s. main has failed the same way (run 42984776547). The same job passed on #48 and #50. #47 fixes it (coalescing committed WAL transactions into one generation per poll). I have not copied that fix here, to keep these PRs small. I will re-run the job once, when the build jobs in this run finish.


Generated by Claude Code

@dbccpage
dbccpage added this pull request to stack #52 November 11, 2026 10:07
@dbccpage
dbccpage marked this pull request as ready for review November 11, 2026 10:07
peek_many now reports whether its batch drained the slot through the
frontier (every decoded row admitted and fewer than max_coalesce rows),
and the worker passes the poll's observation time to apply_many only then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194kAmY4QqQGabdoB3xGFsr
@dbccpage
dbccpage removed this pull request from stack #54 November 11, 2026 11:07
Base automatically changed from claude/rd2-signed-head-bootstrap-and-withdraw to main November 11, 2026 11:07
@dbccpage
dbccpage merged commit b6417e0 into main Nov 11, 2026
5 checks passed
@dbccpage
dbccpage deleted the claude/rd3-freshness-on-publish branch November 11, 2026 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants