Skip to content

Compress and share transaction set responses - #5459

Merged
SirTyson merged 23 commits into
pr/merge-ms-close-time-basefrom
pr/txset-compression
Nov 10, 2026
Merged

SirTyson merged 23 commits into
pr/merge-ms-close-time-basefrom
pr/txset-compression

Conversation

@SirTyson

@SirTyson SirTyson commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Update (2026-11-10): cheaper leader build, ballot-start signature prewarm, IPC size fix; perfnet passes on c6id

Head is now ebdd73635, 8 commits on top of c26b191fa. They fix the two things the 2026-11-09 perfnet run found, plus a liveness bug the next run found.

Leader build

On perfnet the leader's 1.1 s build was on the critical path every ledger.

  • ba4955347 validates only what selection can use. The leader still pulls 2x the capacity, but validates each phase in mempool order only until it holds 5/4 of the capacity in valid transactions. If some are invalid it refills: twice the shortfall, then the rest. The selected set is the same unless selection would have reached past the validated prefix.
  • a1bffae42 hashes the candidates on the validation workers, not the main thread. That was 18% of a cold build's instructions on hosts without SHA extensions, such as c5d.
  • 966d3bb41 precomputes the parallel builder's sort keys.
  • 95be616c7 decodes mempool pulls straight from the IPC payload, without copying.
  • 2a30c1dc0 adds phase timers herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check} and a hidden leader-build benchmark.
  • Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5 medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%). Main-thread instructions -41% (callgrind).

Signature prewarm from ballot start, on every validator (a91547bec)

The 2026-11-09 follower prewarm covered only about half of the leader's set. The fees are equal, so the mempool orders transactions by hash, but selection breaks ties randomly. The leader also had no prewarm.

  • Now, when slot N starts balloting, every validator verifies the signatures of exactly the window that the N+1 build validates first. This runs while N finishes balloting, externalizes and closes. The main thread only makes the overlay round trip; decoding and verification run on background tasks.
  • New metrics: herder.prewarm.{pull,duration}, herder.prewarm.{txs,verified,incomplete} and herder.txset.{build,validate}-sig-cache-{hit,miss}.
  • Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set validation takes 38 ms (cold 102). 0 cache misses.
  • 0cf227b94: Core's tx-set fetch timer is now scp.fetch.txset, with scp.fetch.txset-retry. overlay.fetch.txset is now only the Rust overlay's time; it had been fed by both.

Bug fix: oversized IPC payloads (ebdd73635)

Perfnet found this on a91547bec, on c5d.

  • What happened: 80 s into the load the whole network stopped at one ledger. 16 of 30 nodes logged Overlay IPC connection closed within 0.35 s of each other. The rest then lost quorum, and all lost sync.
  • Cause: the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700 SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats a rejected length as a corrupt stream: it stops reading and never recovers. Every validator starts the pull at the same ballot.
  • Fix:
    • The limit is now 64 MiB on both sides (ipc::MAX_PAYLOAD_BYTES and MAX_PAYLOAD_SIZE, which must match).
    • The overlay returns the longest prefix of the top transactions that fits, with a warning, so no mempool can produce an oversized response.
    • Neither side sends an over-limit payload: Rust refuses with PayloadTooLarge, and Core refuses before writing anything.
    • Core now logs the type and length when it rejects one.
    • This also covers a maximum-size peer tx set (MAX_TXSET_XDR_SIZE + 32 bytes), which could have hit the old limit too.
  • Test: a new [overlay-ipc] test with the real Rust overlay. A 20 MiB top-transactions response arrives whole; with the old code the pull returns nothing (0 == 20). An 80 MiB pull is truncated to the 63 highest-fee transactions, and the IPC keeps working.

Testing

  • a91547bec:
    • Laptop: new and changed tests, and the step-2 tags (149 test cases).
    • Devbox: full suite --all-versions PASS (selftest 604 test cases / 8,638,027 assertions, check-nondet, check-rust).
  • ebdd73635 (devbox):
    • New tests: the [overlay-ipc] test above, plus IPC tests for a 20 MB message, an over-limit length (rejected, with a receive timeout so a regression fails instead of hanging) and an over-limit send (nothing written). Rust tests cover truncation and the send guard.
    • [ipc] (7 test cases), [overlay-ipc] (20), partitioned [overlay]~[ipc-crossproc] and the Rust ipc:: tests (23): all pass.
    • Two pre-existing failures reproduce on a91547bec without the change:
      • The Rust test_tx_not_blocked_by_scp_flood fails its own validity check (an SCP flood of 6.7 ms is under its 10 ms floor).
      • The hidden IPC cross-process communication test needs an unbuilt overlay-stub binary.
    • Full suite --all-versions on ebdd73635 (devbox): PASS. selftest 605 test cases / 8,638,079 assertions across protocol versions, check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban host tests (103 test result: ok, 0 failed). The overlay crate's own tests run separately; see above.

Perfnet, head ebdd73635

30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s measured. One run per instance type, both on the same day: c5d.2xlarge (perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake, SHA-NI). Same 8 vCPU / 16 GiB.

c6id.2xlarge: PASS. All four windows on target, inclusion passed (2,880,000 txs), all checks passed.

Window avg p75 avg p99 p75 range over nodes
1 2,112 ms 2,334 ms 2,103-2,129 ms
2 2,099 ms 2,391 ms 2,091-2,113 ms
3 2,101 ms 2,324 ms 2,092-2,110 ms
4 2,099 ms 2,354 ms 2,089-2,110 ms

c5d.2xlarge: FAIL in window 1, but 19% faster than the previous head and #5458. The c5d network fell behind the offered load, so sets were full at 7,500 and the mempool backed up. That's exactly the case that stalled a91547bec, and here there were no IPC disconnects or sync losses. Correctness passed: error counters, in sync, no fork, still closing. Inclusion was skipped because the load stopped after window 1.

c5d, window 1 (5-min means) #5458 head #5459 c26b191fa #5459 ebdd73635
avg p75 / p99 3,250 / 3,891 ms 3,264 / 3,672 ms 2,637 / 2,920 ms
Ledgers per 5 min 96 96 119
Trigger to first ballot 2.52 s 2.55 s 1.94 s
Leader build 1.15 s 1.11 s 0.68 s (candidate validation 0.20 s)
Received-set validation 0.34 s 0.52 s 0.33 s
Nomination timeouts per node 5 106 34
Ledger close 0.229 s 0.228 s 0.229 s

Both instance types, same build:

c5d c6id
Ledger interval about 2.53 s about 2.03 s
Txs per ledger 7,500 (backlog) about 6,100 (kept up with 3,000 TPS)
Trigger to first ballot 1.94 s 1.22 s
Leader build 0.68 s 0.27 s
Received-set validation 0.33 s 0.20 s
Signature-cache coverage: build / validation 89.5% / 98.7% 87.9% / 97.8%
Prewarm: mean duration / main-thread pull / stopped early 0.21 s / 22 ms / 0.1% 0.10 s / 7 ms / 0%
Ledger close 0.229 s 0.164 s

Notes:

  • No A/B on c6id. Select loadgen source accounts in constant time #5458 has not been run on c6id, so the c6id PASS shows this head can hold the target on current-generation x86. It doesn't show how much of that comes from this PR. Every earlier run of this fixture was on c5d.
  • Two regimes. On c6id the network keeps up, so sets stay at about 6,100 transactions and triggers aren't all due at once; Merge ms close time #5454's early preparation runs, giving about 2 builds per ledger network-wide. On c5d it falls behind, sets are full, and every ledger is heavier. So per-ledger timings aren't like for like across the two.
  • What c5d still needs (about 0.24 s to pass):
    • Received-set validation still takes 0.33 s with 98.7% of its signatures cached, so the rest of validation is now the cost.
    • About 245 ms of the c5d build is outside the phase timers.
    • The remaining trigger-to-ballot time beyond build, fetch and validation is about 0.57 s of nomination messaging across 4 regions.
  • n = 1 per instance type.

Previous update (2026-11-09)

Genesis-upgrade fix, fetch robustness, follower signature prewarm.

Rebased onto the updated stack (#5453 now merges master @ 362dbc5: protocol 30, millisecond close times; #5454, #5457 and #5458 updated the same day). The PR's original four commits are kept (adapted to the new base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten commits are added on top.

Bug fix: networks starting at the genesis protocol stalled (465f6a4)

Root cause. With leader-only construction (5fff543), followers no longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's set is a legacy
TransactionSet, and the Rust overlay only carries GeneralizedTransactionSets (Core caches,
Rust decodes and Core receives only that type). So a follower could get the leader's set only by
being promoted to leader by nomination timeouts and building it itself, one slow round at a time.
Before this PR every validator built the identical empty legacy set (it depends only on the
previous ledger; upgrades live in the StellarValue), so it was always local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are unaffected.

Fix. When the LCL is before protocol 20, every validator constructs its nomination value at
the trigger again, as before; it is still voted only by round leaders. Leader-only construction
is unchanged from protocol 20 on.

Test. Rust overlay upgrades from genesis through legacy tx sets: 4 nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol 30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other changes below in place,
still fails (2/2), so they don't mask the bug.

Robustness on the same path

  • 193fee3: a locally built set completes a pending fetch of the same hash, so envelopes waiting
    for it are processed at once (the IPC path had lost upstream's ItemFetcher behaviour).
  • 942a6b3: a follower promoted to leader after losing sync can still propose; tracking is
    checked at the trigger, as before. This is what made the perfnet stall permanent.
  • ab4eb14, 5964ce6: unanswered tx-set requests are retried and reported. Core re-requests
    undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on every doubling of the
    request count and counts overlay.fetch.txset-retry; Rust sends a retry to a random other peer
    and prunes stale reservations by slot. Core also checks a delivered set's content hash.
  • 5c159ae: test for a leader whose own build completes a pending fetch and externalizes the
    slot inside SCP's nomination supplier.
  • Rebase onto the new base: the test-only nomination override from Tx set validity hardening #5481 now also hands its
    substituted set to the overlay (31e6147), since this PR caches the built set earlier.

Performance

  • 3903e4e, f2643ba: follower signature prewarm. The received-set validation timer rose (about
    158 -> 205 ms) because followers no longer validated their own mempool, which used to warm the
    signature-verification cache. Benchmark (6,000 one-signature txs, 4 tasks): 67 ms cold vs
    10.5 ms warm. Followers now pull their top mempool transactions once at the trigger and verify
    master-key signatures on background tasks (at most 20 verifications per tx; two worker threads
    left free; skipped if the node already voted; stopped when validation, a local build or
    externalization starts). They still build and compress nothing.
  • c26b191: no early tx-set preparation when the trigger is already due. Both timers fired at the
    same instant in unspecified order, so a predicted leader could build its set twice.
  • 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths (leader serializes in place,
    follower decodes in place and moves into the frame). Sub-millisecond for classic sets.

Testing

  • Laptop, on c26b191: new and changed tests (24, including the genesis repro and the trigger,
    early-nomination and Tx set validity hardening #5481 tests) and partitioned make check for
    [herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]
    (143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
  • Full suite --all-versions (devbox, at c26b191): selftest 598 test cases / 8.5M assertions across protocol versions, check-sorobans, check-rust and check-nondet pass.
  • Perfnet, head c26b191fa (30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, tx sets capped at 7,500; n = 1; run in parallel with the Select loadgen source accounts in constant time #5458 head on the same fixture):
    • The genesis-upgrade bug is fixed on a real network: all 30 nodes synced in 43 s from genesis through the legacy ledgers to protocol 30 (the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger 1).
    • Correct under load: no forks, all in sync, no invalid values/signatures, no restarts, no NOT IN CACHE/parse/cache-drop errors; 11 tx-set fetch retries, all resolved on the first re-request.
    • Leader-only construction works as designed (~1 build per ledger network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from ~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
    • Verdict FAIL at the same speed as the Select loadgen source accounts in constant time #5458 head (window-1 p75 3.26 s vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and follower validation of the received set was 0.52 s vs 0.34 s on Select loadgen source accounts in constant time #5458 (whose followers had warmed their signature cache by building), so the follower prewarm did not recover it at this load. Nomination round 1 (2 s timeout) times out on almost every ledger.
    • Follow-up: done in the 2026-11-10 update above (leader build, ballot-start prewarm, and the IPC size fix found by the next run).

Original description

The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet; the run of this PR's final head will be posted in the update above.

Description

This PR compresses transaction-set responses in the Rust overlay and shares the encoded bytes across peer sends. Every response uses the mandatory zstd protocol; there is no legacy/raw fallback or optional lazy encoding state. The leader hands the final XDR to Rust before Core's final validation, so compression can overlap that validation. Receivers retain the original compressed bytes, decompress and verify the set, then reuse the retained encoding when forwarding it.

A separate commit, 0ed3e540, removes private proposals on non-leader nodes. Followers enter nomination with a lazy value supplier and follow received proposals without pulling the mempool or constructing a set. First-round leaders can prepare early; a promoted leader constructs a value only when it needs one. The driver caches that value across retries, and externalization stops nomination even when no local value was built. Tests cover timeout promotion, current close times, following another leader, unavailable values, and reentrant nomination stop.

This PR builds on #5458. The comparison below evaluates these revisions against the original compressed implementation of this PR, using the same 10-tier-1, 30-validator fixture: 3,000 offered TPS, 6,000 transactions/block, 2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and 900-second measurement, with simulated apply.

3,000 offered TPS

Metric Original compression Updated leader-only path
Median block interval 2.018 s 2.120 s
Worst-node full-window p99 block interval 3.834 s 3.836 s
Tx-set fetch p99 0.617 s 0.601 s
Completed TPS 2,926.67 2,806.67
Harness verdict PASS FAIL (ob-0 only)

Median intervals and completed throughput use the canonical node; worst-node p99 uses the fixed measurement window. The harness uses a different histogram/window: ob-0 recorded 4.989 s against its 4-second limit, while the other 29 nodes and inclusion passed. Completed throughput fell 4.1% and median block interval rose 5.1%; this run does not establish that the revised path matches or improves the original compression baseline. These are single cloud runs with normal host placement.

Leader-only construction is verified: 886 builds/compressions across 421 closed ledgers, all correlated with asserted leadership, with no follower build or received-set recompression detected. Multiple nomination leaders can build in a ledger. Compared with the intermediate eager implementation, local compressions fell from 13,019 to 893 in their respective 900-second windows, roughly a 93% reduction per ledger. Payload savings remain 80.0%. The sampled received-set validation timer rose from about 158 ms to 205 ms versus that intermediate build; the precise cause is not isolated by this run.

Checklist

  • Reviewed the contributing document.
  • Kept leader-only proposal construction in its own commit on this PR's branch.
  • Formatting checked with make format and cargo fmt --check.
  • Local build, Jenkins package/image build, and exact-head Clang/GCC CI passed.
  • Rust suite: 194 passed, 3 ignored; the leader-only follow-up leaves those validated Rust sources unchanged.
  • Core validation: 41 focused SCP/nomination/tx-set cases, 17 overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent cases were rerun after configuring the local database fixture.
  • Performance comparison, runtime identity, resource checks and sealed-artifact audit completed.
  • Performance matches or improves the original compressed baseline.

@socket-security

socket-security Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​zstd@​0.13.310010093100100

View full report

@socket-security

socket-security Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Priority Alert  (click "▶" to expand/collapse) Action
Medium priority
License policy violation: cargo zstd-sys

Location: Package overview

From: Cargo.lock → cargo/zstd@0.13.3 → cargo/zstd-sys@2.1.0%2Bzstd.1.5.7

ℹ Read more on: This package | This alert | What is a license policy violation?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Find a package that does not violate your license policy or adjust your policy to allow this package's license.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore cargo/zstd-sys@2.1.0%2Bzstd.1.5.7. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn

View full report

@SirTyson
SirTyson force-pushed the pr/txset-compression branch from 3504e52 to 421fd60 Compare September 16, 2026 02:56
@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch 2 times, most recently from 03a55f7 to 195adf7 Compare September 16, 2026 03:23
@SirTyson
SirTyson force-pushed the pr/txset-compression branch from 421fd60 to 1b5c9dc Compare September 16, 2026 03:23
@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch from 195adf7 to e5ec7d4 Compare November 9, 2026 21:38
@SirTyson
SirTyson force-pushed the pr/txset-compression branch from 0ed3e54 to c26b191 Compare November 9, 2026 23:12
@SirTyson

SirTyson commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Updated (force-pushed, 2026-11-09): fixed the known bug where a network starting at the genesis protocol could not upgrade (overlay v2 can't carry pre-protocol-20 legacy tx sets, and with leader-only construction followers no longer built the identical empty set), added tx-set fetch robustness, a follower signature prewarm for the received-set validation slowdown, and rebased onto the updated #5458. Details and test results are in the "Update (2026-11-09)" section at the top of the description; a perfnet run of this head is in progress.

Commits on top of the base:

  • d0bacb851 Compress and share transaction set responses across peers
  • 31e6147b4 Prepare mandatory compressed transaction sets before validation
  • 88c65bb45 Clarify compressed transaction set delivery ownership
  • 5fff543bf Build local transaction sets only for nomination leaders
  • 465f6a40f Build legacy tx sets on every validator under leader-only nomination
  • 193fee3fe Complete pending tx-set fetches with locally built sets
  • 942a6b34e Let promoted nomination leaders propose after losing sync
  • ab4eb1422 Retry stalled tx-set fetches and report sets that never arrive
  • 2ac0af96d Avoid copying tx-set XDR on the overlay IPC paths
  • 3903e4e1b Pre-verify mempool signatures on followers at the trigger
  • 5964ce684 Back off tx-set fetch retries and retry only undecided slots
  • f2643baa0 Bound the follower signature prewarm
  • 5c159ae69 Test externalizing inside a leader's own tx-set build
  • c26b191fa Skip early tx-set preparation when the trigger is already due

Prefer the negotiated zstd tx-set response protocol and cache each set's encoded payload for fanout. Encode and decode off Tokio network workers, retain existing size and send-admission bounds, and preserve legacy peer compatibility.

Port 125a248 onto the batched-getdata branch. Rust tests: 191 passed, 3 pre-existing ignored; cargo fmt passed.
Leader-only construction left followers without a local tx set, so they
had to fetch the leader's. The Rust overlay only carries generalized tx
sets, so before protocol 20 (legacy TransactionSet) a follower could not
obtain the leader's set at all: it waited until nomination timeouts
promoted it to leader and it built the set itself. A network created at
the genesis protocol (perfnet) therefore stalled before reaching its
upgrade: 22 of 30 nodes reached ledger 2, 8 stayed at ledger 1.

For ledgers whose LCL is before protocol 20, every validator constructs
its nomination value at the trigger again, as before leader-only
construction; it is voted only if the node becomes a round leader. With
an empty mempool (genesis and upgrade-only ledgers) the legacy set
depends only on the previous ledger, so every validator holds the
leader's set. Leader-only construction is unchanged from protocol 20 on.

Add a Rust overlay simulation that starts at the genesis protocol,
closes legacy-set ledgers, upgrades straight to the latest protocol and
keeps closing ledgers. It fails without this change (a node stays at
ledger 2 while the others close).
PendingEnvelopes::addTxSet only cached the set. Envelopes waiting to
fetch that hash stayed in the fetching state until the overlay delivered
the set again or the sender re-sent the envelope. Upstream's ItemFetcher
released them on addTxSet; this branch's IPC fetch path lost that.

That matters for sets the overlay cannot deliver: a leader's legacy
(pre-protocol 20) set that a validator builds itself after the leader's
nomination arrived, or a promoted leader that builds the set it was
fetching. For generalized sets it saves the IPC round trip back from Rust.

addTxSet now completes a pending fetch the same way recvTxSet does: it
updates waiting value wrappers and reprocesses the waiting envelopes,
which can reenter SCP (the nomination supplier already handles that).
The waiting envelopes are detached before reprocessing, so reentrant
fetch changes cannot invalidate the iteration. Completing a fetch now
records the overlay.fetch.txset timer, which was never updated. A
redundant overlay delivery of a known set is logged at debug level.
With leader-only construction a follower builds its value only when a
nomination timeout promotes it to leader, possibly long after the
trigger. makeNominationValue rechecked herder tracking at that point, so
a node that lost sync while the slot was stalled (35 s without
externalizing) could never propose for that slot, even once promoted.
On perfnet that turned the legacy-tx-set stall into a permanent one:
nodes waiting for a set they could not fetch lost sync, and then could
not build it either.

Tracking is checked when the trigger starts nomination, as before; a
value built at the trigger kept being nominated after losing sync. The
supplier now only rechecks what makes building unsafe or stale (ledger
manager synced and not applying, LCL still the slot's predecessor, close
time valid).

Add a herder test: a follower in a stalled unanimous quorum loses sync
and must still propose a valid value once nomination timeouts promote
it. It fails without this change (the follower never builds).
Core requested each missing tx set from the Rust overlay once, assuming
the overlay handled timeouts and retries (stopAllOutsideRange said so),
while the overlay explicitly has neither: a request to a peer that is
silent or lacks the set (there is no negative response) stays reserved
until that peer disconnects. A follower then waited forever, silently.
Pending fetches and their envelopes were also never dropped when their
slots were purged.

Core: PendingEnvelopes keeps per-fetch state (start, last request, newest
waiting slot, request count). A timer repeats requests that have not been
answered after one second, every second, counts them in
overlay.fetch.txset-retry and logs a warning on every doubling of the
request count, naming the slot and how long it waited (and that legacy
pre-protocol 20 sets cannot be fetched over the Rust overlay, when the
LCL is legacy). Fetches whose waiting slots are purged are dropped.

Rust: a RequestTxSet for a hash Core is still waiting for is a retry. It
replaces the outstanding reservation and goes to another connected peer
(the SCP source only if it was not the previous target, otherwise a
random one), instead of being deduplicated against the stuck request.
LedgerClosed prunes reservations below the retained-slot window, which
also drops the prefetches of legacy sets that no peer can serve.

Tests: a herder test for retry timing, completion and slot purging, and
a dispatcher test for retry peer selection and pruning.
The leader copied its set's whole GeneralizedTransactionSet (toXDR) to
serialize it into a temporary vector, which CacheTxSet then copied into
the IPC message. Followers copied the received payload into a vector to
decode it, and makeFromWire copied the decoded set again into the frame.
Several frame constructors also copied freshly built local XDR.

- CacheTxSet serializes the frame's XDR directly into the IPC payload
  (TxSetXDRFrame::getGeneralizedXDR gives const access to it).
- TX_SET_AVAILABLE decodes in place from the payload and moves the set
  into the frame (new rvalue makeFromWire overloads), also used by
  toWireTxSetFrame, makeEmpty and makeFromHistoryTransactions.
- PendingEnvelopes::recvTxSet rejects a set whose recomputed content hash
  differs from the hash it was delivered for; Core keyed sets by the
  overlay's hash without checking.

Add a hidden benchmark ("received tx set processing costs") for the
dissemination path. For 6,000 classic payments (1.2 MB) on the laptop:
leader encode 0.71 -> 0.35 ms, follower frame 1.16 -> 1.04 ms (n=5
medians). The savings are small for classic sets and grow with Soroban
transactions' nested XDR. The benchmark also shows received-set
validation at 67 ms with a cold signature cache vs 10.5 ms warm
(4 tasks).
Before leader-only construction every validator built its own set at the
trigger, validating its mempool and so populating the signature
verification cache. When the leader's set arrived, its signatures were
mostly cache hits. Followers now validate the leader's set cold: the PR's
perfnet run saw the received-set validation timer rise from about 158 ms
to 205 ms. The "received tx set processing costs" benchmark reproduces
that: validating 6,000 transactions takes 67 ms with a cold signature
cache and 10.5 ms warm (4 tasks, the physical cores of a perfnet node).

At the trigger, a validator that is not a round-1 leader (and not on a
legacy tx-set protocol, where everyone builds) pulls its mempool's top
transactions once (up to a full set) and verifies their source, operation
source and fee source master-key signatures on background tasks, as many
as the batch executor would use. The verification goes through
SignatureUtils::verify, so it fills exactly the cache entries tx-set
validation looks up. Signatures by other signers need ledger state and
are left to validation. The prewarm stops as soon as validation of a
received set (a validity-cache miss) or a local build begins, which then
verify the rest themselves on all cores.

The follower's main thread does the pull while it waits for the leader's
nomination; it still builds and compresses nothing.

In-process simulations share the global signature cache across nodes, so
they cannot show the effect; it needs a perfnet comparison.
Follow-up to the fetch retries, from review:
- Retries back off (1, 2, 4, then every 8 s). A slow but live transfer
  of a multi-MB set, under exactly the load where links are slow, was
  duplicated to another peer every second.
- Only sets for slots after the LCL are re-requested. A fetch kept for
  the checkpoint slot (never purged) was re-requested every second until
  the checkpoint moved, and the overlay had evicted that set anyway.
- overlay.fetch.txset again only times sets delivered by the overlay,
  not fetches a local build completed.
- A Rust retry goes to a random connected peer other than the previous
  target. Preferring the SCP source again sent every other retry back to
  a source that may not answer.
Follow-up to the signature prewarm, from review:
- Skip it when this node already voted in the slot (it validated the
  leader's value cold, so the prewarm would only re-check cached
  signatures while balloting runs).
- Stop it when a slot externalizes and on herder shutdown, not only when
  validation or a local build starts.
- Leave two worker threads free for long background jobs (bucket
  merges): tasks = min(batch executor tasks, WORKER_THREADS - 2).
- Cap verifications per transaction at 20 and only try keys whose hint
  matches, so a crafted transaction with colliding hints cannot make the
  prewarm expensive.
- The test no longer asserts exact process-global verification counts
  (other code flushes them); it asserts that validation finds every
  signature cached.
Since locally built sets complete pending fetches, a leader that builds
the very set a peer's ballot is waiting for reprocesses that envelope
inside SCP's nomination value supplier. With the peer v-blocking, the
slot externalizes before the supplier returns. Cover that path: the
peer's CONFIRM waits for the (empty, hence identical) set, the local
leader's trigger builds it, the slot closes with the peer's value, and
nomination stops without voting or emitting a nomination.
setupTriggerNextLedger arms the early-preparation timer for a predicted
first-round leader at "now", next to the trigger timer. When the trigger
is already due, both fire at the same instant, and VirtualClock orders
equal-time events from different timers arbitrarily (event sequence
numbers are per timer). If preparation ran first, the leader built its
set once to prepare and again at the trigger (an underfilled prepared set
is refreshed), costing a full build on exactly the late ledgers where
the trigger is due immediately.

"early preparation respects manual and immediately due triggers" caught
it: with leader-only construction the test reschedules with both timers
due, and it failed in partitioned make check (base instance 400, seed
20766: two pulls instead of one) while passing with other instances.

Arm early preparation only when the trigger is still ahead; a due
trigger builds itself. The test now also checks that nothing was
scheduled to prepare, instead of depending on timer order.
Phase timers herder.txset.build-{pull,frames,trim,select,xdr,handoff,
roundtrip,final-check} around HerderImpl::buildTxSet and
makeTxSetFromTransactions, and a hidden benchmark ("leader tx set build
costs", tags [herder][.][benchmark]) that hands the herder 15,000 SAC
payments (ApplyLoad's generator) through an exact replay of the overlay
IPC decode and times each phase with a cold and a warm signature cache,
plus cold received-set validation for calibration against perfnet.
BENCH_REPS, BENCH_TASKS, BENCH_WARM and BENCH_PULL control it; under
callgrind with --instr-atstart=no only the cold builds are instrumented.
SirTyson and others added 8 commits November 10, 2026 13:22
The leader pulls twice the tx-set capacity from the mempool and validated
every candidate, although selection keeps at most the capacity. Signature
verification of the surplus was most of the build's cost (15,000 SAC
candidates for 7,500 slots on perfnet).

Keep the same pull, but validate each phase in mempool order only until it
holds 5/4 of its capacity in valid transactions of its generic surge
pricing lane (TxSetUtils::trimInvalidPrefix): the first 5/4 of the
capacity, then twice the shortfall if some were invalid, then everything
left, so at most three passes and each candidate validated at most once.
DEX-lane transactions don't count toward the target, since their lane
can't fill the phase. Candidates past the validated prefix are neither
selected nor reported invalid; they stay in the mempool.

The selected set is the same as when validating everything unless
selection would have reached past the prefix: a candidate that doesn't fit
the remaining resources leaving room that only later candidates fit, a
mempool order (total fee per operation) that differs from the inclusion
fee order of Soroban transactions, equal fees at the cutoff (broken
randomly either way), or a fee source paying for several candidates (its
fees accumulate over fewer of them).

Laptop benchmark (15,000 SAC candidates, 4 tasks, n=5 medians): cold
signature cache 261.0 -> 183.6 ms, warm 124.3 -> 98.7 ms.
Building a tx set hashed every candidate's full envelope on the main
thread (removing the invalid ones, then sorting the set by hash): SHA-256
of every candidate, 18% of a cold build's instructions on hosts without
SHA extensions such as c5d. Compute and cache each candidate's full hash
in the parallel validation task that already validates it (which also
computes its contents hash for the signature checks), and skip
removeTxs' lookups when nothing is invalid.

Same hashes, same order: only where they are computed changes. Each
frame is in exactly one task range and the batch completes before the
main thread reads the cached hash.

Laptop benchmark (has SHA-NI): warm build-trim 22.1 -> 18.0 ms.
The parallel Soroban phase builder sorts all candidates by inclusion fee
rate, and each comparison looked up both transactions' inclusion fee and
operation count through virtual calls (~1,450 instructions per comparison
on the main thread). Look them up once per transaction and compare the
cached values with the same comparator logic: TxFeeComparator gets an
overload that takes the fee bids and operation counts, and the plain
overload forwards to it, so both order identically (same tie-break).

Laptop benchmark (15,000 SAC candidates, 9,375 validated): build-select
17.1 -> 14.0 ms cold, 17.2 -> 13.6 ms warm.
OverlayIPC::getTopTransactions copied every transaction's XDR into a
temporary vector before decoding it. Decode well-formed entries straight
from the response payload (OverlayIPC::parseTopTransactions); entries
that are not 4-byte aligned, which a well-formed response never has, are
still decoded from an aligned copy, and entries that fail to decode are
still skipped. The leader build benchmark now decodes its replayed
response with the same function.

Laptop benchmark (15,000 SAC candidates): build-pull 19.4 -> 10.7 ms.
PendingEnvelopes timed its tx-set fetches (request to overlay delivery)
into overlay.fetch.txset, which RustOverlayManager also feeds with the
Rust overlay's own fetch times, synced as per-batch averages: one timer
mixing two measurements, with about two samples per ledger. Name Core's
timer scp.fetch.txset, next to scp.fetch.envelope from the same class,
and its retry counter scp.fetch.txset-retry; overlay.fetch.txset is left
to the overlay. docs/metrics.md lists both.
On perfnet (3k SAC TPS) followers still validated the leader's set with
a mostly cold signature cache, and the leader built with a cold one. The
follower prewarm verified each follower's own top 7,600 mempool txs at
the trigger, but with equal fees the mempool orders by hash while
selection breaks ties randomly, so the leader's 7,500 came from its top
15,200 and only about half were covered; the leader had no prewarm.

When slot N starts balloting, every tracking validator now starts a
prewarm for ledger N + 1 over exactly the candidates its build validates
first: per phase, the first validTxTargets() (5/4 of the capacity) in
mempool order, past the transactions of N's value, which stay in the
mempool until N externalizes. The leader's set comes from the same
window, so a finished prewarm covers its build and the followers'
validation. It runs while N finishes balloting, externalizes and closes:
externalizing N keeps it; the build or a received-set validation stops
it, as before. The follower's trigger-time prewarm stays as a top-up.

The main thread only does the overlay round trip (posted after the SCP
step that started the ballot; OverlayIPC::getTopTransactionsPayload
returns the response undecoded). Decoding, windowing and verification
run in at most min(batch executor tasks, WORKER_THREADS - 2) background
tasks, which also free what they allocated.

Metrics, so a perfnet run shows coverage: herder.prewarm.{pull,duration}
timers, herder.prewarm.{txs,verified,incomplete} meters, and signature
cache lookups during a build and during received-set validation,
herder.txset.{build,validate}-sig-cache-{hit,miss} (from the new
non-resetting PubKeyUtils::getVerifySigCacheCounts). docs/metrics.md
lists them, and the build phase timers.

The leader build benchmark gets a prewarmed mode. Laptop, 15,000 SAC
candidates, 4 tasks: build cold 169.2 / warm 84.1 / after a prewarm 83.0
ms; received-set validation 102.2 / 34.2 / 37.7 ms; both with no
signature verified after the prewarm, which itself takes 110.8 ms in
the background.
Both ends of the IPC rejected payloads over 16 MiB, and Core's reader
treats a rejected length as a corrupt stream: it stops reading and never
recovers, so the node can no longer send or receive SCP messages. The
ballot-start signature prewarm pulls 2 x the tx-set capacity plus the
balloting value's transactions: 22,700 SAC payments of ~752 bytes, about
17.2 MB, once the mempool holds that many. On perfnet (3k SAC TPS, 30
validators) every validator starts that pull at the same ballot, so 80 s
into the load 16 of the 30 nodes lost their overlay within 0.35 s, the
rest lost quorum, and the network stalled at one ledger.

- Raise the limit to 64 MiB on both sides, named once per side
  (ipc::MAX_PAYLOAD_BYTES in IPC.h, MAX_PAYLOAD_SIZE in messages.rs)
  with a note that they must match.
- The overlay sends at most the longest prefix of the top transactions
  that fits in the limit (the highest-priority ones), with the count
  matching, and warns when it truncates, so a large mempool can't produce
  an oversized response whatever the transaction sizes.
- Neither side sends a payload over the limit: CoreSender refuses it
  (IpcError::PayloadTooLarge), and Core's sendMessage refuses it before
  writing anything, keeping the connection.
- Core logs an error with the message type and length when it rejects a
  payload, instead of only "connection closed".
- Tests: a [overlay-ipc] test with the real Rust overlay where a 20 MiB
  top transactions response arrives whole (it fails with the old limit:
  the pull returns nothing) and an 80 MiB pull is truncated to the 63
  highest-fee transactions while the IPC keeps working; IPC tests for a
  20 MB message, an over-limit length (rejected, with a receive timeout
  so a regression fails instead of hanging) and an over-limit send
  (nothing written); Rust tests for the truncation and the send guard.
- docs/rust-overlay/ipc.md: the 64 MiB limit and how each side enforces
  it.
@SirTyson
SirTyson force-pushed the pr/txset-compression branch from dfd2e66 to 1a52bf6 Compare November 10, 2026 21:22
@SirTyson
SirTyson force-pushed the pr/loadgen-account-selection branch from a3d0e18 to 0fd7d0e Compare November 10, 2026 21:22
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-09)

This PR is rebased onto the reworked #5453, which merges current
`master` (362dbc5) into
`overlay-v2-shared`. That merge brings in the final #5423 (millisecond
close times, gated at protocol
30, with the protocol-30 bump). It also carries the overlay-v2
adaptations those need, including the
Rust overlay moving to `stellar-xdr` 30.0.0 to decode millisecond
`StellarValue`s.

**The old first commit ("Enable millisecond consensus close times at
protocol 28") is dropped.** The
master merge in #5453 supersedes it. It was an earlier draft of #5423
behind a compile-time
`MS_CLOSE_TIME` flag at protocol 28, with its own XDR pin and fixtures.
The overlay-v2-specific parts
needed for correctness are now in #5453.

This PR is now the trigger-timing work on top of millisecond close
times:

1. **Account for actual elapsed nomination time in the trigger drift
allowance** (ported unchanged
in substance). The next-ledger trigger is anchored on the previous
externalized close time C and
fires at `C + target`. It falls back to the local prepare-start anchor
when the time elapsed
since C exceeds the target plus an allowance for nomination. That
allowance used to be the sum of
nomination timeouts that had fired. Now it is the steady-clock time from
the local trigger to
entry into ballot. A slow round that never runs a timeout callback no
longer counts as clock
drift: slow tx-set fetch or validation, a busy main thread, or ballot
starting before the expired
timer runs. When the network is behind, the next trigger fires
immediately instead of waiting
out the nomination again. The extra wait a fallback can add is bounded
by our clock drift plus
   how late we triggered relative to the winning close time.
2. **Record the nomination trigger only when nomination proceeds**
(new). If a node skips
nomination because its clock is behind the last close time, it no longer
credits the wait for
   its peers to the allowance.
3. **Test the trigger anchor with sub-second close times** (new). With
close times 0, 250 and 999 ms
into the second, the next trigger is due exactly the target after the
close time, and the
   fallback boundary holds to the millisecond.
4. **Add trigger timing metrics** (new). Together they show whether a 2
s ledger is bound by apply,
   by nomination or by event-loop delay:
- `scp.timing.trigger-to-ballot` (timer): proposal construction plus
nomination, which is the
     drift allowance.
- `scp.trigger.lateness` (timer): how late the trigger callback ran
after its scheduled time.
- `scp.trigger.immediate` (meter): the next trigger was already due when
it was scheduled.

5. **Skip a delayed nomination emit that was already sent** (new,
test-only code path). With
`ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`, a deferred nomination
emit could fire after
`processEnvelope` had already emitted the same statement. SCP then
rejected the duplicate and
threw "moved to a bad state (nomination)", which made the hidden
"consensus close time trigger
timer" test flaky under parallel load. The deferred emit now only sends
a statement newer than
   our last one. A new SCP test replays the race deterministically.

None of these commits change values, validation or ledger output. They
only change local trigger
scheduling, metrics, logs and BUILD_TESTS-only code.

**Follow-up under consideration:** for proposals rebuilt at the trigger,
propose the scheduled
trigger time as the close time (bounded by `now` and LCL+1 ms), so a
leader's trigger lateness no
longer shifts the network's cadence. We'll do it if
`scp.trigger.lateness` p75 exceeds about 20 ms
on perfnet.

Testing, at `285aa42fc92d` on top of #5453 `04fbddb7d`:
- New and changed tests: `[trigger-construction]` 8/8 (the
drift-allowance tests and the 3 new
  ones) and `[early-nomination]` 6/6 pass.
- `[herder]` (partitioned `make check`): 77/77 test cases pass,
including the hidden real-clock
  trigger-timer simulations.
- At `4f577bd25`: the new SCP test passes. A copy of the test without
the fix fails with the
original exception, so the test reproduces the race. "consensus close
time trigger timer" passes
3/3 on its own and 2/2 inside a partitioned `[herder],[scp]` `make
check` (97/97 test cases, full
  check passes both times).
- Final branch: restacked onto the final #5453 head `14de3972a` (two
test-only commits on top of `04fbddb7d`); the five patches are identical
(`git range-diff`). The full suite passes on the #5458 head that
contains this PR (selftest 590 test cases, check-sorobans, check-rust,
check-nondet).


---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the runs of the final #5458 and #5459 heads contain
this PR and are reported on those PRs.

# Description

Merges in #5423.

This change adds ms closeTime. By itself, it doesn't improve TPS or
latency significantly. However, it reduces our long tail fo very slow
blocks, and also allows us to target intrasecond or subsecond block
times, which are not currently possible at whole second resolution.

In addition to taking the change from the PR, the 2nd commit makes some
minor adjustments to when we call the fallback timer. This was
intentionally conservative for main net launch assuming lots of drifting
clocks, but no reason we can't tighten it here.

Results, measured with 10 tier 1 topology (no watchers) with 2 second
block time target:

## 2000 TPS

| Metric | Original overlay-v2-shared |Changes |
|---|---:|---:|
| Median block interval | 2.090 s | 1.993 s |
| p99 block interval | 4.563 s |  4.298 s |
| Maximum block interval | 6.219 s | 5.090 s |
| Harness verdict | **FAIL** | **FAIL** |


Rebased on #5453 so opening
as a draft for now.

# Checklist
- [x] Reviewed the
[contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#opening-a-pull-request)
document
- [x] Rebased on top of master (no merge commits)
- [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio
extension)
- [x] Compiles
- [x] Ran all tests
- [ ] If change impacts performance, include supporting evidence per the
[performance
document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
SirTyson added a commit that referenced this pull request Nov 10, 2026
## Update (2026-11-09)

Rebased onto the updated #5454 (which now sits on #5453's merge of
`master` @ 362dbc5; the overlay crate moved to `stellar-xdr` 30.0.0
there). The original commit is unchanged apart from a test import
updated for stellar-xdr 30.0.0 (`stellar_xdr::curr::` to
`stellar_xdr::`). Eight commits on top, each with tests (an independent
review found no correctness, wire-format or deadlock issues in the
original commit; its low-severity findings are fixed below):

- **Stop serving a demand after a failed write.** A response job returns
at the first write that fails after the stream retries (usually a
disconnect), instead of looking up, writing and logging a failure for
every remaining batch. The requester re-demands unanswered hashes after
its response timeout.
- **Frame responses straight from the shared transaction bytes.** Each
frame's discriminant and envelope are copied directly into the batch,
with no per-transaction allocation; a transaction larger than the batch
budget is written as two borrowed parts with no copy. Each batch is
looked up under one TX-buffer lock, released before frames are copied or
written, so local submissions on the event loop never wait for a
response copy.
- **Buffer inbound TX stream reads (64 KiB).** One QUIC read takes in
whatever has arrived, often a whole response batch, instead of two reads
per frame. The reader never waits for more data than is available.
- **Queue INVs for all peers under one lock**, for both relayed and
locally submitted transactions, instead of one batcher lock per peer.
- **Handle each received INV batch with one seen-set lock and one
tracker lock**, instead of both locks per advertised hash. Request
decisions and their order are unchanged.
- **More tests:** batching edges (an oversized transaction first, two in
a row, frames exactly at the budget, carried hits, misses); a failed
batch write retried whole on a reopened stream; buffered reads; delivery
of INV batches completed by either path; INV request decisions and
order.

Wire bytes are unchanged. Testing on the rebased branch:
- 194 overlay crate tests pass (`cargo test` in `overlay/`, 3 ignored);
`cargo fmt --check` is clean.
- The standard build succeeds, and 25 C++ overlay-IPC test cases (136
assertions) pass: `[overlay-ipc]`, `[overlay][ipc]` and the 10-node
`[overlay-ipc-large]` consensus case.
- The full suite passes on the #5458 head that contains this PR
(selftest 590 test cases, check-sorobans, check-rust with these overlay
tests, check-nondet).

The follow-up commits have not been measured separately on perfnet; the
numbers above are for the original commit on the previous base. The
perfnet run of the final #5458 head (which contains this PR) is reported
on #5458.

**Next lever (not in this PR):** a node waits up to 100 ms (checked
every 50 ms) before announcing new transactions to a peer, so each
flooding hop adds 100-150 ms before the peer can request them. A perfnet
A/B of shorter delays (50 ms and 25 ms) is the planned follow-up,
comparing tx pull latency, tx-set fetch p99 and block intervals against
bandwidth and CPU.

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the runs of the final #5458 and #5459 heads contain
this PR and are reported on those PRs.

# Description

This PR batches GETDATA transaction responses in the Rust overlay.
Previously, a request for many transactions created a separate task and
stream write for every response. Under load, that produced unnecessary
scheduling and stream-lock overhead, and a slow peer could leave the
entire response demand queued in memory.

We now serve each request in one asynchronous job, combining complete
transaction frames into writes of up to 64 KiB. Each transaction keeps
its existing wire framing, and the writes use the existing stream
priorities and retry policy. Transactions larger than the batch budget
are sent individually.

Here, the control image is the PR this is based on
#5454. This is the 10 tier
1, 2 second target block latency test. Median block intervals are from
the canonical node; p99 is the worst node over the full measurement
window.

## 2,000 TPS

| Metric | Control | Batched GETDATA |
|---|---:|---:|
| Median block interval | 1.993 s | 2.000 s |
| Worst-node p99 block interval | 5.090 s | 2.508 s |
| Tx-set fetch p99 | 1.720 s | 0.778 s |
| Harness verdict | FAIL | PASS |

## 3,000 TPS

| Metric | Control  | Batched GETDATA|
|---|---:|---:|
| Median block interval | 2.434 s | 2.363 s |
| Worst-node p99 block interval | 4.122 s | 4.524 s |
| Tx-set fetch p99 | 1.757 s | 1.087 s |
| Harness verdict | FAIL | FAIL |

At 2,000 TPS, the largest improvement is in the tail: worst-node p99
falls by about half, and nearly all tx-set fetches longer than two
seconds disappear. This is consistent with reducing transaction-response
write overhead and contention affecting consensus transfers.

# Checklist

- [x] Reviewed the contributing document.
- [x] One commit on top of #5454, with no merge commits. (Now nine: the
original plus eight follow-ups, see the update above.)
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Release overlay build and Jenkins package build succeeded.
- [x] Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++
overlay-IPC cases / 82 assertions.
- [x] Performance comparison included above.
Base automatically changed from pr/loadgen-account-selection to pr/merge-ms-close-time-base November 10, 2026 22:04
@SirTyson
SirTyson marked this pull request as ready for review November 10, 2026 22:04
@SirTyson
SirTyson merged commit f65907b into pr/merge-ms-close-time-base Nov 10, 2026
5 checks passed
@SirTyson
SirTyson deleted the pr/txset-compression branch November 10, 2026 22:04
@SirTyson
SirTyson restored the pr/txset-compression branch November 10, 2026 22:06
SirTyson added a commit that referenced this pull request Nov 10, 2026
#5495)

**Prototype, not for merge.** This deletes the SCP nomination phase and
replaces it with a leader schedule that every node computes locally.
There is no flag, no protocol gate and no compatibility with nominating
nodes: it exists to measure what nomination costs on the overlay-v2
stack and what limits block time once it is gone.

Stacked on #5459 (`pr/txset-compression`, `ebdd73635`). The change is
one commit, `daa965f4c`. A second commit, `577f3d4f0`, only replaces the
branch's `README.md` with a description of the experiment, the results
below and the full test setup (topology, node configuration, network
settings, load, pass criteria); everything here was built, tested and
measured on `daa965f4c`.

## Result in one paragraph

On perfnet (30 validators, 4 AWS regions, c6id.2xlarge, 3,000 SAC TPS)
the prototype passes at a 2 s target with every correctness check green
and no wasted round, and takes about 0.19 s out of the time from trigger
to externalize (1.50 s to 1.31 s). Block time at 2 s is unchanged,
because the trigger sets the cadence when the network keeps up. At a 1 s
target it fails: the network settles at 1.33 s per ledger. The remaining
time is mostly cross-region round trips for the tx set and the ballot,
not nomination and not CPU. One run per setting.

## Design

- **Schedule.** For slot N, sort the transitive quorum by
`SHA256(LCL(N-1).hash || nodeID)`; `leader(N, r)` is entry `(r-1) mod
n`. Uniform, no weights, one sort per slot
(`herder/LeaderSchedule.{h,cpp}`, cached in `HerderSCPDriver`).
- **Rounds.** A round is a ballot counter. The round's leader builds its
value and bumps straight to `(r, value)`; the others adopt the leader's
PREPARE/CONFIRM at counter r once it passes proposal validation. A
timeout (the ballot timeout settings) moves a node to round r+1.
- **Late proposals.** A node that has not voted yet still votes for an
earlier round leader's ballot, at that ballot's own counter, when it
shows up late (up to 8 rounds back). Nomination had this property;
without it a slow first proposal is wasted and the next leader builds a
competing set.
- **Leader timer extension.** If the timer fires while the Herder
already holds the leader's statement (its tx set is being fetched or it
is queued), the timer is re-armed once.
- **Unchanged:** the ballot protocol's accept, confirm and commit rules,
`mValueOverride` precedence, the v-blocking join paths, value
validation, apply, ledger state and meta.
- NOMINATE envelopes are dropped.

## Why it should be safe (and where it is not a real design)

- Safety rests on the ballot protocol, which is safe for any choice of
ballot values. The change only alters which value a node picks for a new
ballot and when it bumps. Two independent reviews found no path to two
honest nodes externalizing different values, no vote the ballot protocol
forbids, and no new reachable assert or unbounded loop.
- Upgrade governance: adoption validates in the strict (former
nomination) mode, so a value needs a quorum of adopters that armed the
same upgrades. There is no upgrade stripping or merging.
- Known gaps, by design of a prototype:
- A follower follows a leader's empty-tx-set ballot under ballot
validation, so its upgrades are only checked for apply validity. With
parallel tx-set download on, one round leader could get an empty ledger
adopted network-wide.
- The schedule's node set is whatever transitive quorum each node
currently knows. It assumes a symmetric, static quorum (as on perfnet).
A real design needs a node set that is part of consensus state.
- Leader grinding through the header hash, and fairness by org or
quality, are out of scope.

## What was deleted

- `scp/NominationProtocol.{h,cpp}`, `scp/test/SCPUnitTests.cpp`, and the
nomination API of `SCP`, `Slot` and `SCPDriver` (`nominate`,
`combineCandidates`, `computeHashNode`, `getNodeWeight`, the nomination
timers and emit delay, and related helpers).
- `HerderSCPDriver::combineCandidates` / `compareTxSets` /
`getNodeWeight`, the metrics `scp.timing.nominated`,
`scp.timeout.nominate`, `scp.nomination.combinecandidates`, and
`ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`.
- Kept but unused, so existing configs still parse:
`FORCE_OLD_STYLE_LEADER_ELECTION`, `VALIDATOR_WEIGHT_CONFIG`,
`nominationtimeoutlimit` and the nomination timeout network settings.
- New metrics: `scp.timeout.leader` (leader timeouts per slot; an
extension counts) and `scp.leader.round` (the highest round a node
entered for the slot, so an upper bound on the round that externalized).

Net: 23 non-test files, +886 / -2,072.

## Testing (all on `daa965f4c`)

- New: `[scp][leaderschedule]` unit cases and a randomized multi-node
model in `scp/test/SCPTests.cpp`, `herder/test/LeaderScheduleTests.cpp`,
and network simulations in `herder/test/LeaderRoundsTests.cpp`
(including failover when the dead node leads round 1). The Herder,
quorum-tracker, PendingEnvelopes and overlay-IPC tests are ported;
nomination statistics, `combineCandidates` and weight tests are deleted
with the code.
- Full suite: PASS. 598 test cases / 2,379,273 assertions, libsodium 77,
xdrpp 7, check-rust and the soroban host tests (103 `test result: ok`, 0
failed).
- Full suite `--all-versions`: PASS. 598 test cases / 8,154,187
assertions. (#5459's head runs 605; the difference is the deleted
nomination tests.)
- Tag run
`[herder],[scp],[txset],[overlay-ipc],[overlay],[ipc],[upgrades],[simulation]`
(146 tests): passes except two `[acceptance]` tests, "hierarchical
topology scales 1 to 3" and "core nodes with outer nodes", which time
out the same way on `overlay-v2-shared` without this change and are not
in the full suite.
- `clang-format` 20.1.8: the 28 added or modified C++ files are clean.
- **Not passing:** a `--disable-tests` build stops in
`herder/HerderImpl.cpp`, on test headers reached through `#include
"simulation/LoadGenerator.h"` (line 37). That include is the same on
#5459's head and this commit does not touch those headers; the base was
not built with that flag to confirm, and because the build stops there
it is not known whether this commit's own files compile without tests.
- **Not run:** sanitizers; the tx-meta and LedgerCloseMeta determinism
checks (the change does not touch apply).
- **Test gaps** (from review): no SCP-level test for following an
empty-tx-set leader ballot, for the `UINT32_MAX` counter case, for the
`mValueOverride` restore, for the extension flag resetting across
rounds, or for the 8-round bound. The randomized model uses one shared
schedule and never exercises the pending-proposal path. Two hidden
trigger-timer tests that depended on the nomination emit delay were
deleted without a port.

## Perfnet

30 validators in 4 regions, 10 tier-1 orgs, c6id.2xlarge, SAC payments,
150 ms simulated apply per ledger, 900 s measured, all SCP timeouts 2 s
(perfnet's default). Baseline is #5459's head `ebdd73635` on the same
fixture and instance type, run the same day. One run per setting.

### 2 s target, 3,000 TPS: PASS

All four windows on target; inclusion (2,880,000 txs on every node),
error counters, in sync, no fork and still closing all passed.

| | #5459 `ebdd73635` | This PR `daa965f4c` |
|---|---:|---:|
| avg p75 per window | 2,112 / 2,099 / 2,101 / 2,099 ms | 2,109 / 2,066
/ 2,082 / 2,094 ms |
| avg p99 per window | 2,334 / 2,391 / 2,324 / 2,354 ms | 2,324 / 2,286
/ 2,296 / 2,303 ms |
| Ledgers per 5 min | 147-148 | 149-150 |
| Trigger to first ballot (mean) | 1.21 s | 0.88 s |
| Ballot to externalize (mean) | 0.29 s | 0.43 s |
| **Trigger to externalize** | **1.50 s** | **1.31 s** |
| Leader build / tx-set fetch / validation (mean) | 0.28 / 0.24 / 0.20 s
| 0.27 / 0.26 / 0.19 s |
| Ledger close | 0.164 s | 0.162 s |
| Nomination timeouts (network, one window) | 7 | n/a |
| Leader timeouts, prepare timeouts, rounds above 1, empty tx sets | n/a
| 0, 0, none, 0 |

- The time before the first ballot that is not build, fetch or
validation fell from about 0.48 s to about 0.15 s.
- Part of it came back in the ballot phase: nodes now start their ballot
when each has validated the leader's set, so the early ones wait for a
quorum of the others.
- `scp.timing.trigger-to-ballot` records 0 when a follower votes before
its own trigger, so it is not strictly comparable. Triggers were on time
in this run.

### 1 s target, 3,000 TPS: FAIL

Window 1 missed on all 30 nodes: avg p75 1,386 ms (limit 1,200), avg p99
1,599 ms. 225 ledgers per 5 minutes (1.33 s each), every tx set at the
3,750 cap, every trigger already due, mempool backed up. No leader or
prepare timeouts, every ledger decided in round 1, no empty tx sets;
error counters, in sync, no fork and still closing passed. No baseline
was run at this setting.

| Stage (5-min means) | 2 s target (6,000 txs) | 1 s target (3,750 txs)
|
|---|---:|---:|
| Leader build | 0.27 s | 0.25 s |
| Trigger to first ballot | 0.88 s | 0.70 s |
| Ballot to externalize | 0.43 s | 0.41 s |
| Ledger close (simulated apply 150 / 187 ms) | 0.16 s | 0.20 s |
| Sum | 1.47 s | 1.30 s |

## Where the time goes now

From one node's logs at the 2 s setting (456 ledgers as follower, 23 as
leader) and live logs from four regions at 1 s:

1. **Tx-set dissemination is 2.5 round trips.** A follower receives the
proposal, requests the set from the proposer, and the response (0.77 MB
zstd for 6,000 txs) takes two flights: fetch measured 27 ms in-region,
150 ms from us-east-1, 301 ms from eu-central-1 and 362 ms from
ap-southeast-1, as seen from us-west-2. Nothing is pushed, and full
transactions are sent although followers hold nearly all of them.
2. **Followers validate before voting, then the ballot needs four
one-way trips** (0.29 s on the baseline, where all nodes start
together).
3. **About 70 ms pass between "Built TX set" and the proposal reaching a
same-region peer.** `emitEnvelope` calls `persistSCPState` before
`broadcast`, which serializes and base64-encodes the tx set and writes
it to the database on the main thread. That this write is most of the 70
ms is inferred, not timed.
4. **The leader's build is on the critical path** (0.25-0.33 s). The
early build is discarded unless the set was capacity limited, and it
does not run at all when the trigger is already due.
5. Validation is 0.12-0.19 s with signatures already cached.

`EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` does not help here as it stands:
followers would confirm prepared before holding the set, and the ballot
protocol does not retry the commit vote at the same counter, so each
slot would wait for the ballot timeout.

## Open review findings (not fixed in this commit)

- The leader timer extension triggers on any pending envelope from the
round's leader for the slot, not only its proposal for that round. It
slows failover when a tx set is unfetchable; it does not occur on a
healthy network.
- A late proposal only helps nodes without a ballot, never the next
leader, so a proposal that arrives just after a timeout costs a second
fetch and validation.
- A leader re-proposes a value that is only structurally valid;
`bumpState` may replace it with the empty-tx-set value.
- Nodes that adopted a ballot but have not heard a quorum run no timer,
as in the plain ballot protocol, so recovery after a timeout depends on
the next leader being alive.
- Smaller items: a redundant `progressRound()` before `advanceSlot`,
repeated proposal validation while a leader has no ballot, leftover
diagnostics in one Herder test, stale "nominate" wording in a few
comments.

## Next steps

Not started. In order of expected effect on block time: push the tx set
with the proposal in one flight; take SCP state persistence off the
broadcast path (after timing it); let the commit vote resume when a late
tx set validates, so the prepare rounds can overlap with fetch and
validation; build the leader's set before the previous ledger closes,
which needs the leader to be known earlier than the previous header
hash. Then repeat the 1 s run, with a same-setting baseline and more
than one run per build.
SirTyson added a commit that referenced this pull request Nov 10, 2026
…der schedule (#5496)

Brings the stacked PRs that were merged into
`pr/merge-ms-close-time-base` into `overlay-v2-shared`:

- #5457 Batch GETDATA transaction responses
- #5458 Select loadgen source accounts in constant time
- #5459 Compress and share transaction set responses
- #5495 Prototype: replace SCP nomination with a deterministic leader
schedule
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant