Repository navigation
Compress and share transaction set responses - #5459
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
3504e52 to
421fd60
Compare
03a55f7 to
195adf7
Compare
421fd60 to
1b5c9dc
Compare
195adf7 to
e5ec7d4
Compare
0ed3e54 to
c26b191
Compare
|
Updated (force-pushed, 2026-11-09): fixed the known bug where a network starting at the genesis protocol could not upgrade (overlay v2 can't carry pre-protocol-20 legacy tx sets, and with leader-only construction followers no longer built the identical empty set), added tx-set fetch robustness, a follower signature prewarm for the received-set validation slowdown, and rebased onto the updated #5458. Details and test results are in the "Update (2026-11-09)" section at the top of the description; a perfnet run of this head is in progress. Commits on top of the base:
|
ebdd736 to
dfd2e66
Compare
e5ec7d4 to
a3d0e18
Compare
Prefer the negotiated zstd tx-set response protocol and cache each set's encoded payload for fanout. Encode and decode off Tokio network workers, retain existing size and send-admission bounds, and preserve legacy peer compatibility. Port 125a248 onto the batched-getdata branch. Rust tests: 191 passed, 3 pre-existing ignored; cargo fmt passed.
Leader-only construction left followers without a local tx set, so they had to fetch the leader's. The Rust overlay only carries generalized tx sets, so before protocol 20 (legacy TransactionSet) a follower could not obtain the leader's set at all: it waited until nomination timeouts promoted it to leader and it built the set itself. A network created at the genesis protocol (perfnet) therefore stalled before reaching its upgrade: 22 of 30 nodes reached ledger 2, 8 stayed at ledger 1. For ledgers whose LCL is before protocol 20, every validator constructs its nomination value at the trigger again, as before leader-only construction; it is voted only if the node becomes a round leader. With an empty mempool (genesis and upgrade-only ledgers) the legacy set depends only on the previous ledger, so every validator holds the leader's set. Leader-only construction is unchanged from protocol 20 on. Add a Rust overlay simulation that starts at the genesis protocol, closes legacy-set ledgers, upgrades straight to the latest protocol and keeps closing ledgers. It fails without this change (a node stays at ledger 2 while the others close).
PendingEnvelopes::addTxSet only cached the set. Envelopes waiting to fetch that hash stayed in the fetching state until the overlay delivered the set again or the sender re-sent the envelope. Upstream's ItemFetcher released them on addTxSet; this branch's IPC fetch path lost that. That matters for sets the overlay cannot deliver: a leader's legacy (pre-protocol 20) set that a validator builds itself after the leader's nomination arrived, or a promoted leader that builds the set it was fetching. For generalized sets it saves the IPC round trip back from Rust. addTxSet now completes a pending fetch the same way recvTxSet does: it updates waiting value wrappers and reprocesses the waiting envelopes, which can reenter SCP (the nomination supplier already handles that). The waiting envelopes are detached before reprocessing, so reentrant fetch changes cannot invalidate the iteration. Completing a fetch now records the overlay.fetch.txset timer, which was never updated. A redundant overlay delivery of a known set is logged at debug level.
With leader-only construction a follower builds its value only when a nomination timeout promotes it to leader, possibly long after the trigger. makeNominationValue rechecked herder tracking at that point, so a node that lost sync while the slot was stalled (35 s without externalizing) could never propose for that slot, even once promoted. On perfnet that turned the legacy-tx-set stall into a permanent one: nodes waiting for a set they could not fetch lost sync, and then could not build it either. Tracking is checked when the trigger starts nomination, as before; a value built at the trigger kept being nominated after losing sync. The supplier now only rechecks what makes building unsafe or stale (ledger manager synced and not applying, LCL still the slot's predecessor, close time valid). Add a herder test: a follower in a stalled unanimous quorum loses sync and must still propose a valid value once nomination timeouts promote it. It fails without this change (the follower never builds).
Core requested each missing tx set from the Rust overlay once, assuming the overlay handled timeouts and retries (stopAllOutsideRange said so), while the overlay explicitly has neither: a request to a peer that is silent or lacks the set (there is no negative response) stays reserved until that peer disconnects. A follower then waited forever, silently. Pending fetches and their envelopes were also never dropped when their slots were purged. Core: PendingEnvelopes keeps per-fetch state (start, last request, newest waiting slot, request count). A timer repeats requests that have not been answered after one second, every second, counts them in overlay.fetch.txset-retry and logs a warning on every doubling of the request count, naming the slot and how long it waited (and that legacy pre-protocol 20 sets cannot be fetched over the Rust overlay, when the LCL is legacy). Fetches whose waiting slots are purged are dropped. Rust: a RequestTxSet for a hash Core is still waiting for is a retry. It replaces the outstanding reservation and goes to another connected peer (the SCP source only if it was not the previous target, otherwise a random one), instead of being deduplicated against the stuck request. LedgerClosed prunes reservations below the retained-slot window, which also drops the prefetches of legacy sets that no peer can serve. Tests: a herder test for retry timing, completion and slot purging, and a dispatcher test for retry peer selection and pruning.
The leader copied its set's whole GeneralizedTransactionSet (toXDR) to
serialize it into a temporary vector, which CacheTxSet then copied into
the IPC message. Followers copied the received payload into a vector to
decode it, and makeFromWire copied the decoded set again into the frame.
Several frame constructors also copied freshly built local XDR.
- CacheTxSet serializes the frame's XDR directly into the IPC payload
(TxSetXDRFrame::getGeneralizedXDR gives const access to it).
- TX_SET_AVAILABLE decodes in place from the payload and moves the set
into the frame (new rvalue makeFromWire overloads), also used by
toWireTxSetFrame, makeEmpty and makeFromHistoryTransactions.
- PendingEnvelopes::recvTxSet rejects a set whose recomputed content hash
differs from the hash it was delivered for; Core keyed sets by the
overlay's hash without checking.
Add a hidden benchmark ("received tx set processing costs") for the
dissemination path. For 6,000 classic payments (1.2 MB) on the laptop:
leader encode 0.71 -> 0.35 ms, follower frame 1.16 -> 1.04 ms (n=5
medians). The savings are small for classic sets and grow with Soroban
transactions' nested XDR. The benchmark also shows received-set
validation at 67 ms with a cold signature cache vs 10.5 ms warm
(4 tasks).
Before leader-only construction every validator built its own set at the trigger, validating its mempool and so populating the signature verification cache. When the leader's set arrived, its signatures were mostly cache hits. Followers now validate the leader's set cold: the PR's perfnet run saw the received-set validation timer rise from about 158 ms to 205 ms. The "received tx set processing costs" benchmark reproduces that: validating 6,000 transactions takes 67 ms with a cold signature cache and 10.5 ms warm (4 tasks, the physical cores of a perfnet node). At the trigger, a validator that is not a round-1 leader (and not on a legacy tx-set protocol, where everyone builds) pulls its mempool's top transactions once (up to a full set) and verifies their source, operation source and fee source master-key signatures on background tasks, as many as the batch executor would use. The verification goes through SignatureUtils::verify, so it fills exactly the cache entries tx-set validation looks up. Signatures by other signers need ledger state and are left to validation. The prewarm stops as soon as validation of a received set (a validity-cache miss) or a local build begins, which then verify the rest themselves on all cores. The follower's main thread does the pull while it waits for the leader's nomination; it still builds and compresses nothing. In-process simulations share the global signature cache across nodes, so they cannot show the effect; it needs a perfnet comparison.
Follow-up to the fetch retries, from review: - Retries back off (1, 2, 4, then every 8 s). A slow but live transfer of a multi-MB set, under exactly the load where links are slow, was duplicated to another peer every second. - Only sets for slots after the LCL are re-requested. A fetch kept for the checkpoint slot (never purged) was re-requested every second until the checkpoint moved, and the overlay had evicted that set anyway. - overlay.fetch.txset again only times sets delivered by the overlay, not fetches a local build completed. - A Rust retry goes to a random connected peer other than the previous target. Preferring the SCP source again sent every other retry back to a source that may not answer.
Follow-up to the signature prewarm, from review: - Skip it when this node already voted in the slot (it validated the leader's value cold, so the prewarm would only re-check cached signatures while balloting runs). - Stop it when a slot externalizes and on herder shutdown, not only when validation or a local build starts. - Leave two worker threads free for long background jobs (bucket merges): tasks = min(batch executor tasks, WORKER_THREADS - 2). - Cap verifications per transaction at 20 and only try keys whose hint matches, so a crafted transaction with colliding hints cannot make the prewarm expensive. - The test no longer asserts exact process-global verification counts (other code flushes them); it asserts that validation finds every signature cached.
Since locally built sets complete pending fetches, a leader that builds the very set a peer's ballot is waiting for reprocesses that envelope inside SCP's nomination value supplier. With the peer v-blocking, the slot externalizes before the supplier returns. Cover that path: the peer's CONFIRM waits for the (empty, hence identical) set, the local leader's trigger builds it, the slot closes with the peer's value, and nomination stops without voting or emitting a nomination.
setupTriggerNextLedger arms the early-preparation timer for a predicted first-round leader at "now", next to the trigger timer. When the trigger is already due, both fire at the same instant, and VirtualClock orders equal-time events from different timers arbitrarily (event sequence numbers are per timer). If preparation ran first, the leader built its set once to prepare and again at the trigger (an underfilled prepared set is refreshed), costing a full build on exactly the late ledgers where the trigger is due immediately. "early preparation respects manual and immediately due triggers" caught it: with leader-only construction the test reschedules with both timers due, and it failed in partitioned make check (base instance 400, seed 20766: two pulls instead of one) while passing with other instances. Arm early preparation only when the trigger is still ahead; a due trigger builds itself. The test now also checks that nothing was scheduled to prepare, instead of depending on timer order.
Phase timers herder.txset.build-{pull,frames,trim,select,xdr,handoff,
roundtrip,final-check} around HerderImpl::buildTxSet and
makeTxSetFromTransactions, and a hidden benchmark ("leader tx set build
costs", tags [herder][.][benchmark]) that hands the herder 15,000 SAC
payments (ApplyLoad's generator) through an exact replay of the overlay
IPC decode and times each phase with a cold and a warm signature cache,
plus cold received-set validation for calibration against perfnet.
BENCH_REPS, BENCH_TASKS, BENCH_WARM and BENCH_PULL control it; under
callgrind with --instr-atstart=no only the cold builds are instrumented.
The leader pulls twice the tx-set capacity from the mempool and validated every candidate, although selection keeps at most the capacity. Signature verification of the surplus was most of the build's cost (15,000 SAC candidates for 7,500 slots on perfnet). Keep the same pull, but validate each phase in mempool order only until it holds 5/4 of its capacity in valid transactions of its generic surge pricing lane (TxSetUtils::trimInvalidPrefix): the first 5/4 of the capacity, then twice the shortfall if some were invalid, then everything left, so at most three passes and each candidate validated at most once. DEX-lane transactions don't count toward the target, since their lane can't fill the phase. Candidates past the validated prefix are neither selected nor reported invalid; they stay in the mempool. The selected set is the same as when validating everything unless selection would have reached past the prefix: a candidate that doesn't fit the remaining resources leaving room that only later candidates fit, a mempool order (total fee per operation) that differs from the inclusion fee order of Soroban transactions, equal fees at the cutoff (broken randomly either way), or a fee source paying for several candidates (its fees accumulate over fewer of them). Laptop benchmark (15,000 SAC candidates, 4 tasks, n=5 medians): cold signature cache 261.0 -> 183.6 ms, warm 124.3 -> 98.7 ms.
Building a tx set hashed every candidate's full envelope on the main thread (removing the invalid ones, then sorting the set by hash): SHA-256 of every candidate, 18% of a cold build's instructions on hosts without SHA extensions such as c5d. Compute and cache each candidate's full hash in the parallel validation task that already validates it (which also computes its contents hash for the signature checks), and skip removeTxs' lookups when nothing is invalid. Same hashes, same order: only where they are computed changes. Each frame is in exactly one task range and the batch completes before the main thread reads the cached hash. Laptop benchmark (has SHA-NI): warm build-trim 22.1 -> 18.0 ms.
The parallel Soroban phase builder sorts all candidates by inclusion fee rate, and each comparison looked up both transactions' inclusion fee and operation count through virtual calls (~1,450 instructions per comparison on the main thread). Look them up once per transaction and compare the cached values with the same comparator logic: TxFeeComparator gets an overload that takes the fee bids and operation counts, and the plain overload forwards to it, so both order identically (same tie-break). Laptop benchmark (15,000 SAC candidates, 9,375 validated): build-select 17.1 -> 14.0 ms cold, 17.2 -> 13.6 ms warm.
OverlayIPC::getTopTransactions copied every transaction's XDR into a temporary vector before decoding it. Decode well-formed entries straight from the response payload (OverlayIPC::parseTopTransactions); entries that are not 4-byte aligned, which a well-formed response never has, are still decoded from an aligned copy, and entries that fail to decode are still skipped. The leader build benchmark now decodes its replayed response with the same function. Laptop benchmark (15,000 SAC candidates): build-pull 19.4 -> 10.7 ms.
PendingEnvelopes timed its tx-set fetches (request to overlay delivery) into overlay.fetch.txset, which RustOverlayManager also feeds with the Rust overlay's own fetch times, synced as per-batch averages: one timer mixing two measurements, with about two samples per ledger. Name Core's timer scp.fetch.txset, next to scp.fetch.envelope from the same class, and its retry counter scp.fetch.txset-retry; overlay.fetch.txset is left to the overlay. docs/metrics.md lists both.
On perfnet (3k SAC TPS) followers still validated the leader's set with
a mostly cold signature cache, and the leader built with a cold one. The
follower prewarm verified each follower's own top 7,600 mempool txs at
the trigger, but with equal fees the mempool orders by hash while
selection breaks ties randomly, so the leader's 7,500 came from its top
15,200 and only about half were covered; the leader had no prewarm.
When slot N starts balloting, every tracking validator now starts a
prewarm for ledger N + 1 over exactly the candidates its build validates
first: per phase, the first validTxTargets() (5/4 of the capacity) in
mempool order, past the transactions of N's value, which stay in the
mempool until N externalizes. The leader's set comes from the same
window, so a finished prewarm covers its build and the followers'
validation. It runs while N finishes balloting, externalizes and closes:
externalizing N keeps it; the build or a received-set validation stops
it, as before. The follower's trigger-time prewarm stays as a top-up.
The main thread only does the overlay round trip (posted after the SCP
step that started the ballot; OverlayIPC::getTopTransactionsPayload
returns the response undecoded). Decoding, windowing and verification
run in at most min(batch executor tasks, WORKER_THREADS - 2) background
tasks, which also free what they allocated.
Metrics, so a perfnet run shows coverage: herder.prewarm.{pull,duration}
timers, herder.prewarm.{txs,verified,incomplete} meters, and signature
cache lookups during a build and during received-set validation,
herder.txset.{build,validate}-sig-cache-{hit,miss} (from the new
non-resetting PubKeyUtils::getVerifySigCacheCounts). docs/metrics.md
lists them, and the build phase timers.
The leader build benchmark gets a prewarmed mode. Laptop, 15,000 SAC
candidates, 4 tasks: build cold 169.2 / warm 84.1 / after a prewarm 83.0
ms; received-set validation 102.2 / 34.2 / 37.7 ms; both with no
signature verified after the prewarm, which itself takes 110.8 ms in
the background.
Both ends of the IPC rejected payloads over 16 MiB, and Core's reader treats a rejected length as a corrupt stream: it stops reading and never recovers, so the node can no longer send or receive SCP messages. The ballot-start signature prewarm pulls 2 x the tx-set capacity plus the balloting value's transactions: 22,700 SAC payments of ~752 bytes, about 17.2 MB, once the mempool holds that many. On perfnet (3k SAC TPS, 30 validators) every validator starts that pull at the same ballot, so 80 s into the load 16 of the 30 nodes lost their overlay within 0.35 s, the rest lost quorum, and the network stalled at one ledger. - Raise the limit to 64 MiB on both sides, named once per side (ipc::MAX_PAYLOAD_BYTES in IPC.h, MAX_PAYLOAD_SIZE in messages.rs) with a note that they must match. - The overlay sends at most the longest prefix of the top transactions that fits in the limit (the highest-priority ones), with the count matching, and warns when it truncates, so a large mempool can't produce an oversized response whatever the transaction sizes. - Neither side sends a payload over the limit: CoreSender refuses it (IpcError::PayloadTooLarge), and Core's sendMessage refuses it before writing anything, keeping the connection. - Core logs an error with the message type and length when it rejects a payload, instead of only "connection closed". - Tests: a [overlay-ipc] test with the real Rust overlay where a 20 MiB top transactions response arrives whole (it fails with the old limit: the pull returns nothing) and an 80 MiB pull is truncated to the 63 highest-fee transactions while the IPC keeps working; IPC tests for a 20 MB message, an over-limit length (rejected, with a receive timeout so a regression fails instead of hanging) and an over-limit send (nothing written); Rust tests for the truncation and the send guard. - docs/rust-overlay/ipc.md: the 64 MiB limit and how each side enforces it.
dfd2e66 to
1a52bf6
Compare
a3d0e18 to
0fd7d0e
Compare
## Update (2026-11-09) This PR is rebased onto the reworked #5453, which merges current `master` (362dbc5) into `overlay-v2-shared`. That merge brings in the final #5423 (millisecond close times, gated at protocol 30, with the protocol-30 bump). It also carries the overlay-v2 adaptations those need, including the Rust overlay moving to `stellar-xdr` 30.0.0 to decode millisecond `StellarValue`s. **The old first commit ("Enable millisecond consensus close times at protocol 28") is dropped.** The master merge in #5453 supersedes it. It was an earlier draft of #5423 behind a compile-time `MS_CLOSE_TIME` flag at protocol 28, with its own XDR pin and fixtures. The overlay-v2-specific parts needed for correctness are now in #5453. This PR is now the trigger-timing work on top of millisecond close times: 1. **Account for actual elapsed nomination time in the trigger drift allowance** (ported unchanged in substance). The next-ledger trigger is anchored on the previous externalized close time C and fires at `C + target`. It falls back to the local prepare-start anchor when the time elapsed since C exceeds the target plus an allowance for nomination. That allowance used to be the sum of nomination timeouts that had fired. Now it is the steady-clock time from the local trigger to entry into ballot. A slow round that never runs a timeout callback no longer counts as clock drift: slow tx-set fetch or validation, a busy main thread, or ballot starting before the expired timer runs. When the network is behind, the next trigger fires immediately instead of waiting out the nomination again. The extra wait a fallback can add is bounded by our clock drift plus how late we triggered relative to the winning close time. 2. **Record the nomination trigger only when nomination proceeds** (new). If a node skips nomination because its clock is behind the last close time, it no longer credits the wait for its peers to the allowance. 3. **Test the trigger anchor with sub-second close times** (new). With close times 0, 250 and 999 ms into the second, the next trigger is due exactly the target after the close time, and the fallback boundary holds to the millisecond. 4. **Add trigger timing metrics** (new). Together they show whether a 2 s ledger is bound by apply, by nomination or by event-loop delay: - `scp.timing.trigger-to-ballot` (timer): proposal construction plus nomination, which is the drift allowance. - `scp.trigger.lateness` (timer): how late the trigger callback ran after its scheduled time. - `scp.trigger.immediate` (meter): the next trigger was already due when it was scheduled. 5. **Skip a delayed nomination emit that was already sent** (new, test-only code path). With `ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`, a deferred nomination emit could fire after `processEnvelope` had already emitted the same statement. SCP then rejected the duplicate and threw "moved to a bad state (nomination)", which made the hidden "consensus close time trigger timer" test flaky under parallel load. The deferred emit now only sends a statement newer than our last one. A new SCP test replays the race deterministically. None of these commits change values, validation or ledger output. They only change local trigger scheduling, metrics, logs and BUILD_TESTS-only code. **Follow-up under consideration:** for proposals rebuilt at the trigger, propose the scheduled trigger time as the close time (bounded by `now` and LCL+1 ms), so a leader's trigger lateness no longer shifts the network's cadence. We'll do it if `scp.trigger.lateness` p75 exceeds about 20 ms on perfnet. Testing, at `285aa42fc92d` on top of #5453 `04fbddb7d`: - New and changed tests: `[trigger-construction]` 8/8 (the drift-allowance tests and the 3 new ones) and `[early-nomination]` 6/6 pass. - `[herder]` (partitioned `make check`): 77/77 test cases pass, including the hidden real-clock trigger-timer simulations. - At `4f577bd25`: the new SCP test passes. A copy of the test without the fix fails with the original exception, so the test reproduces the race. "consensus close time trigger timer" passes 3/3 on its own and 2/2 inside a partitioned `[herder],[scp]` `make check` (97/97 test cases, full check passes both times). - Final branch: restacked onto the final #5453 head `14de3972a` (two test-only commits on top of `04fbddb7d`); the five patches are identical (`git range-diff`). The full suite passes on the #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust, check-nondet). --- ## Original description > The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs. # Description Merges in #5423. This change adds ms closeTime. By itself, it doesn't improve TPS or latency significantly. However, it reduces our long tail fo very slow blocks, and also allows us to target intrasecond or subsecond block times, which are not currently possible at whole second resolution. In addition to taking the change from the PR, the 2nd commit makes some minor adjustments to when we call the fallback timer. This was intentionally conservative for main net launch assuming lots of drifting clocks, but no reason we can't tighten it here. Results, measured with 10 tier 1 topology (no watchers) with 2 second block time target: ## 2000 TPS | Metric | Original overlay-v2-shared |Changes | |---|---:|---:| | Median block interval | 2.090 s | 1.993 s | | p99 block interval | 4.563 s | 4.298 s | | Maximum block interval | 6.219 s | 5.090 s | | Harness verdict | **FAIL** | **FAIL** | Rebased on #5453 so opening as a draft for now. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#opening-a-pull-request) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
## Update (2026-11-09) Rebased onto the updated #5454 (which now sits on #5453's merge of `master` @ 362dbc5; the overlay crate moved to `stellar-xdr` 30.0.0 there). The original commit is unchanged apart from a test import updated for stellar-xdr 30.0.0 (`stellar_xdr::curr::` to `stellar_xdr::`). Eight commits on top, each with tests (an independent review found no correctness, wire-format or deadlock issues in the original commit; its low-severity findings are fixed below): - **Stop serving a demand after a failed write.** A response job returns at the first write that fails after the stream retries (usually a disconnect), instead of looking up, writing and logging a failure for every remaining batch. The requester re-demands unanswered hashes after its response timeout. - **Frame responses straight from the shared transaction bytes.** Each frame's discriminant and envelope are copied directly into the batch, with no per-transaction allocation; a transaction larger than the batch budget is written as two borrowed parts with no copy. Each batch is looked up under one TX-buffer lock, released before frames are copied or written, so local submissions on the event loop never wait for a response copy. - **Buffer inbound TX stream reads (64 KiB).** One QUIC read takes in whatever has arrived, often a whole response batch, instead of two reads per frame. The reader never waits for more data than is available. - **Queue INVs for all peers under one lock**, for both relayed and locally submitted transactions, instead of one batcher lock per peer. - **Handle each received INV batch with one seen-set lock and one tracker lock**, instead of both locks per advertised hash. Request decisions and their order are unchanged. - **More tests:** batching edges (an oversized transaction first, two in a row, frames exactly at the budget, carried hits, misses); a failed batch write retried whole on a reopened stream; buffered reads; delivery of INV batches completed by either path; INV request decisions and order. Wire bytes are unchanged. Testing on the rebased branch: - 194 overlay crate tests pass (`cargo test` in `overlay/`, 3 ignored); `cargo fmt --check` is clean. - The standard build succeeds, and 25 C++ overlay-IPC test cases (136 assertions) pass: `[overlay-ipc]`, `[overlay][ipc]` and the 10-node `[overlay-ipc-large]` consensus case. - The full suite passes on the #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust with these overlay tests, check-nondet). The follow-up commits have not been measured separately on perfnet; the numbers above are for the original commit on the previous base. The perfnet run of the final #5458 head (which contains this PR) is reported on #5458. **Next lever (not in this PR):** a node waits up to 100 ms (checked every 50 ms) before announcing new transactions to a peer, so each flooding hop adds 100-150 ms before the peer can request them. A perfnet A/B of shorter delays (50 ms and 25 ms) is the planned follow-up, comparing tx pull latency, tx-set fetch p99 and block intervals against bandwidth and CPU. --- ## Original description > The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs. # Description This PR batches GETDATA transaction responses in the Rust overlay. Previously, a request for many transactions created a separate task and stream write for every response. Under load, that produced unnecessary scheduling and stream-lock overhead, and a slow peer could leave the entire response demand queued in memory. We now serve each request in one asynchronous job, combining complete transaction frames into writes of up to 64 KiB. Each transaction keeps its existing wire framing, and the writes use the existing stream priorities and retry policy. Transactions larger than the batch budget are sent individually. Here, the control image is the PR this is based on #5454. This is the 10 tier 1, 2 second target block latency test. Median block intervals are from the canonical node; p99 is the worst node over the full measurement window. ## 2,000 TPS | Metric | Control | Batched GETDATA | |---|---:|---:| | Median block interval | 1.993 s | 2.000 s | | Worst-node p99 block interval | 5.090 s | 2.508 s | | Tx-set fetch p99 | 1.720 s | 0.778 s | | Harness verdict | FAIL | PASS | ## 3,000 TPS | Metric | Control | Batched GETDATA| |---|---:|---:| | Median block interval | 2.434 s | 2.363 s | | Worst-node p99 block interval | 4.122 s | 4.524 s | | Tx-set fetch p99 | 1.757 s | 1.087 s | | Harness verdict | FAIL | FAIL | At 2,000 TPS, the largest improvement is in the tail: worst-node p99 falls by about half, and nearly all tx-set fetches longer than two seconds disappear. This is consistent with reducing transaction-response write overhead and contention affecting consensus transfers. # Checklist - [x] Reviewed the contributing document. - [x] One commit on top of #5454, with no merge commits. (Now nine: the original plus eight follow-ups, see the update above.) - [x] Formatting checked with `make format` and `cargo fmt --check`. - [x] Release overlay build and Jenkins package build succeeded. - [x] Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++ overlay-IPC cases / 82 assertions. - [x] Performance comparison included above.
#5495) **Prototype, not for merge.** This deletes the SCP nomination phase and replaces it with a leader schedule that every node computes locally. There is no flag, no protocol gate and no compatibility with nominating nodes: it exists to measure what nomination costs on the overlay-v2 stack and what limits block time once it is gone. Stacked on #5459 (`pr/txset-compression`, `ebdd73635`). The change is one commit, `daa965f4c`. A second commit, `577f3d4f0`, only replaces the branch's `README.md` with a description of the experiment, the results below and the full test setup (topology, node configuration, network settings, load, pass criteria); everything here was built, tested and measured on `daa965f4c`. ## Result in one paragraph On perfnet (30 validators, 4 AWS regions, c6id.2xlarge, 3,000 SAC TPS) the prototype passes at a 2 s target with every correctness check green and no wasted round, and takes about 0.19 s out of the time from trigger to externalize (1.50 s to 1.31 s). Block time at 2 s is unchanged, because the trigger sets the cadence when the network keeps up. At a 1 s target it fails: the network settles at 1.33 s per ledger. The remaining time is mostly cross-region round trips for the tx set and the ballot, not nomination and not CPU. One run per setting. ## Design - **Schedule.** For slot N, sort the transitive quorum by `SHA256(LCL(N-1).hash || nodeID)`; `leader(N, r)` is entry `(r-1) mod n`. Uniform, no weights, one sort per slot (`herder/LeaderSchedule.{h,cpp}`, cached in `HerderSCPDriver`). - **Rounds.** A round is a ballot counter. The round's leader builds its value and bumps straight to `(r, value)`; the others adopt the leader's PREPARE/CONFIRM at counter r once it passes proposal validation. A timeout (the ballot timeout settings) moves a node to round r+1. - **Late proposals.** A node that has not voted yet still votes for an earlier round leader's ballot, at that ballot's own counter, when it shows up late (up to 8 rounds back). Nomination had this property; without it a slow first proposal is wasted and the next leader builds a competing set. - **Leader timer extension.** If the timer fires while the Herder already holds the leader's statement (its tx set is being fetched or it is queued), the timer is re-armed once. - **Unchanged:** the ballot protocol's accept, confirm and commit rules, `mValueOverride` precedence, the v-blocking join paths, value validation, apply, ledger state and meta. - NOMINATE envelopes are dropped. ## Why it should be safe (and where it is not a real design) - Safety rests on the ballot protocol, which is safe for any choice of ballot values. The change only alters which value a node picks for a new ballot and when it bumps. Two independent reviews found no path to two honest nodes externalizing different values, no vote the ballot protocol forbids, and no new reachable assert or unbounded loop. - Upgrade governance: adoption validates in the strict (former nomination) mode, so a value needs a quorum of adopters that armed the same upgrades. There is no upgrade stripping or merging. - Known gaps, by design of a prototype: - A follower follows a leader's empty-tx-set ballot under ballot validation, so its upgrades are only checked for apply validity. With parallel tx-set download on, one round leader could get an empty ledger adopted network-wide. - The schedule's node set is whatever transitive quorum each node currently knows. It assumes a symmetric, static quorum (as on perfnet). A real design needs a node set that is part of consensus state. - Leader grinding through the header hash, and fairness by org or quality, are out of scope. ## What was deleted - `scp/NominationProtocol.{h,cpp}`, `scp/test/SCPUnitTests.cpp`, and the nomination API of `SCP`, `Slot` and `SCPDriver` (`nominate`, `combineCandidates`, `computeHashNode`, `getNodeWeight`, the nomination timers and emit delay, and related helpers). - `HerderSCPDriver::combineCandidates` / `compareTxSets` / `getNodeWeight`, the metrics `scp.timing.nominated`, `scp.timeout.nominate`, `scp.nomination.combinecandidates`, and `ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`. - Kept but unused, so existing configs still parse: `FORCE_OLD_STYLE_LEADER_ELECTION`, `VALIDATOR_WEIGHT_CONFIG`, `nominationtimeoutlimit` and the nomination timeout network settings. - New metrics: `scp.timeout.leader` (leader timeouts per slot; an extension counts) and `scp.leader.round` (the highest round a node entered for the slot, so an upper bound on the round that externalized). Net: 23 non-test files, +886 / -2,072. ## Testing (all on `daa965f4c`) - New: `[scp][leaderschedule]` unit cases and a randomized multi-node model in `scp/test/SCPTests.cpp`, `herder/test/LeaderScheduleTests.cpp`, and network simulations in `herder/test/LeaderRoundsTests.cpp` (including failover when the dead node leads round 1). The Herder, quorum-tracker, PendingEnvelopes and overlay-IPC tests are ported; nomination statistics, `combineCandidates` and weight tests are deleted with the code. - Full suite: PASS. 598 test cases / 2,379,273 assertions, libsodium 77, xdrpp 7, check-rust and the soroban host tests (103 `test result: ok`, 0 failed). - Full suite `--all-versions`: PASS. 598 test cases / 8,154,187 assertions. (#5459's head runs 605; the difference is the deleted nomination tests.) - Tag run `[herder],[scp],[txset],[overlay-ipc],[overlay],[ipc],[upgrades],[simulation]` (146 tests): passes except two `[acceptance]` tests, "hierarchical topology scales 1 to 3" and "core nodes with outer nodes", which time out the same way on `overlay-v2-shared` without this change and are not in the full suite. - `clang-format` 20.1.8: the 28 added or modified C++ files are clean. - **Not passing:** a `--disable-tests` build stops in `herder/HerderImpl.cpp`, on test headers reached through `#include "simulation/LoadGenerator.h"` (line 37). That include is the same on #5459's head and this commit does not touch those headers; the base was not built with that flag to confirm, and because the build stops there it is not known whether this commit's own files compile without tests. - **Not run:** sanitizers; the tx-meta and LedgerCloseMeta determinism checks (the change does not touch apply). - **Test gaps** (from review): no SCP-level test for following an empty-tx-set leader ballot, for the `UINT32_MAX` counter case, for the `mValueOverride` restore, for the extension flag resetting across rounds, or for the 8-round bound. The randomized model uses one shared schedule and never exercises the pending-proposal path. Two hidden trigger-timer tests that depended on the nomination emit delay were deleted without a port. ## Perfnet 30 validators in 4 regions, 10 tier-1 orgs, c6id.2xlarge, SAC payments, 150 ms simulated apply per ledger, 900 s measured, all SCP timeouts 2 s (perfnet's default). Baseline is #5459's head `ebdd73635` on the same fixture and instance type, run the same day. One run per setting. ### 2 s target, 3,000 TPS: PASS All four windows on target; inclusion (2,880,000 txs on every node), error counters, in sync, no fork and still closing all passed. | | #5459 `ebdd73635` | This PR `daa965f4c` | |---|---:|---:| | avg p75 per window | 2,112 / 2,099 / 2,101 / 2,099 ms | 2,109 / 2,066 / 2,082 / 2,094 ms | | avg p99 per window | 2,334 / 2,391 / 2,324 / 2,354 ms | 2,324 / 2,286 / 2,296 / 2,303 ms | | Ledgers per 5 min | 147-148 | 149-150 | | Trigger to first ballot (mean) | 1.21 s | 0.88 s | | Ballot to externalize (mean) | 0.29 s | 0.43 s | | **Trigger to externalize** | **1.50 s** | **1.31 s** | | Leader build / tx-set fetch / validation (mean) | 0.28 / 0.24 / 0.20 s | 0.27 / 0.26 / 0.19 s | | Ledger close | 0.164 s | 0.162 s | | Nomination timeouts (network, one window) | 7 | n/a | | Leader timeouts, prepare timeouts, rounds above 1, empty tx sets | n/a | 0, 0, none, 0 | - The time before the first ballot that is not build, fetch or validation fell from about 0.48 s to about 0.15 s. - Part of it came back in the ballot phase: nodes now start their ballot when each has validated the leader's set, so the early ones wait for a quorum of the others. - `scp.timing.trigger-to-ballot` records 0 when a follower votes before its own trigger, so it is not strictly comparable. Triggers were on time in this run. ### 1 s target, 3,000 TPS: FAIL Window 1 missed on all 30 nodes: avg p75 1,386 ms (limit 1,200), avg p99 1,599 ms. 225 ledgers per 5 minutes (1.33 s each), every tx set at the 3,750 cap, every trigger already due, mempool backed up. No leader or prepare timeouts, every ledger decided in round 1, no empty tx sets; error counters, in sync, no fork and still closing passed. No baseline was run at this setting. | Stage (5-min means) | 2 s target (6,000 txs) | 1 s target (3,750 txs) | |---|---:|---:| | Leader build | 0.27 s | 0.25 s | | Trigger to first ballot | 0.88 s | 0.70 s | | Ballot to externalize | 0.43 s | 0.41 s | | Ledger close (simulated apply 150 / 187 ms) | 0.16 s | 0.20 s | | Sum | 1.47 s | 1.30 s | ## Where the time goes now From one node's logs at the 2 s setting (456 ledgers as follower, 23 as leader) and live logs from four regions at 1 s: 1. **Tx-set dissemination is 2.5 round trips.** A follower receives the proposal, requests the set from the proposer, and the response (0.77 MB zstd for 6,000 txs) takes two flights: fetch measured 27 ms in-region, 150 ms from us-east-1, 301 ms from eu-central-1 and 362 ms from ap-southeast-1, as seen from us-west-2. Nothing is pushed, and full transactions are sent although followers hold nearly all of them. 2. **Followers validate before voting, then the ballot needs four one-way trips** (0.29 s on the baseline, where all nodes start together). 3. **About 70 ms pass between "Built TX set" and the proposal reaching a same-region peer.** `emitEnvelope` calls `persistSCPState` before `broadcast`, which serializes and base64-encodes the tx set and writes it to the database on the main thread. That this write is most of the 70 ms is inferred, not timed. 4. **The leader's build is on the critical path** (0.25-0.33 s). The early build is discarded unless the set was capacity limited, and it does not run at all when the trigger is already due. 5. Validation is 0.12-0.19 s with signatures already cached. `EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` does not help here as it stands: followers would confirm prepared before holding the set, and the ballot protocol does not retry the commit vote at the same counter, so each slot would wait for the ballot timeout. ## Open review findings (not fixed in this commit) - The leader timer extension triggers on any pending envelope from the round's leader for the slot, not only its proposal for that round. It slows failover when a tx set is unfetchable; it does not occur on a healthy network. - A late proposal only helps nodes without a ballot, never the next leader, so a proposal that arrives just after a timeout costs a second fetch and validation. - A leader re-proposes a value that is only structurally valid; `bumpState` may replace it with the empty-tx-set value. - Nodes that adopted a ballot but have not heard a quorum run no timer, as in the plain ballot protocol, so recovery after a timeout depends on the next leader being alive. - Smaller items: a redundant `progressRound()` before `advanceSlot`, repeated proposal validation while a leader has no ballot, leftover diagnostics in one Herder test, stale "nominate" wording in a few comments. ## Next steps Not started. In order of expected effect on block time: push the tx set with the proposal in one flight; take SCP state persistence off the broadcast path (after timing it); let the commit vote resume when a late tx set validates, so the prepare rounds can overlap with fetch and validation; build the leader's set before the previous ledger closes, which needs the leader to be known earlier than the previous header hash. Then repeat the 1 s run, with a same-setting baseline and more than one run per build.
…der schedule (#5496) Brings the stacked PRs that were merged into `pr/merge-ms-close-time-base` into `overlay-v2-shared`: - #5457 Batch GETDATA transaction responses - #5458 Select loadgen source accounts in constant time - #5459 Compress and share transaction set responses - #5495 Prototype: replace SCP nomination with a deterministic leader schedule
Update (2026-11-10): cheaper leader build, ballot-start signature prewarm, IPC size fix; perfnet passes on c6id
Head is now
ebdd73635, 8 commits on top ofc26b191fa. They fix the two things the 2026-11-09 perfnet run found, plus a liveness bug the next run found.Leader build
On perfnet the leader's 1.1 s build was on the critical path every ledger.
ba4955347validates only what selection can use. The leader still pulls 2x the capacity, but validates each phase in mempool order only until it holds 5/4 of the capacity in valid transactions. If some are invalid it refills: twice the shortfall, then the rest. The selected set is the same unless selection would have reached past the validated prefix.a1bffae42hashes the candidates on the validation workers, not the main thread. That was 18% of a cold build's instructions on hosts without SHA extensions, such as c5d.966d3bb41precomputes the parallel builder's sort keys.95be616c7decodes mempool pulls straight from the IPC payload, without copying.2a30c1dc0adds phase timersherder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}and a hidden leader-build benchmark.Signature prewarm from ballot start, on every validator (
a91547bec)The 2026-11-09 follower prewarm covered only about half of the leader's set. The fees are equal, so the mempool orders transactions by hash, but selection breaks ties randomly. The leader also had no prewarm.
herder.prewarm.{pull,duration},herder.prewarm.{txs,verified,incomplete}andherder.txset.{build,validate}-sig-cache-{hit,miss}.0cf227b94: Core's tx-set fetch timer is nowscp.fetch.txset, withscp.fetch.txset-retry.overlay.fetch.txsetis now only the Rust overlay's time; it had been fed by both.Bug fix: oversized IPC payloads (
ebdd73635)Perfnet found this on
a91547bec, on c5d.Overlay IPC connection closedwithin 0.35 s of each other. The rest then lost quorum, and all lost sync.ipc::MAX_PAYLOAD_BYTESandMAX_PAYLOAD_SIZE, which must match).PayloadTooLarge, and Core refuses before writing anything.MAX_TXSET_XDR_SIZE+ 32 bytes), which could have hit the old limit too.[overlay-ipc]test with the real Rust overlay. A 20 MiB top-transactions response arrives whole; with the old code the pull returns nothing (0 == 20). An 80 MiB pull is truncated to the 63 highest-fee transactions, and the IPC keeps working.Testing
a91547bec:--all-versionsPASS (selftest 604 test cases / 8,638,027 assertions, check-nondet, check-rust).ebdd73635(devbox):[overlay-ipc]test above, plus IPC tests for a 20 MB message, an over-limit length (rejected, with a receive timeout so a regression fails instead of hanging) and an over-limit send (nothing written). Rust tests cover truncation and the send guard.[ipc](7 test cases),[overlay-ipc](20), partitioned[overlay]~[ipc-crossproc]and the Rustipc::tests (23): all pass.a91547becwithout the change:test_tx_not_blocked_by_scp_floodfails its own validity check (an SCP flood of 6.7 ms is under its 10 ms floor).IPC cross-process communicationtest needs an unbuiltoverlay-stubbinary.--all-versionsonebdd73635(devbox): PASS. selftest 605 test cases / 8,638,079 assertions across protocol versions, check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban host tests (103test result: ok, 0 failed). The overlay crate's own tests run separately; see above.Perfnet, head
ebdd7363530 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s measured. One run per instance type, both on the same day: c5d.2xlarge (perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake, SHA-NI). Same 8 vCPU / 16 GiB.
c6id.2xlarge: PASS. All four windows on target, inclusion passed (2,880,000 txs), all checks passed.
c5d.2xlarge: FAIL in window 1, but 19% faster than the previous head and #5458. The c5d network fell behind the offered load, so sets were full at 7,500 and the mempool backed up. That's exactly the case that stalled
a91547bec, and here there were no IPC disconnects or sync losses. Correctness passed: error counters, in sync, no fork, still closing. Inclusion was skipped because the load stopped after window 1.c26b191faebdd73635Both instance types, same build:
Notes:
Previous update (2026-11-09)
Genesis-upgrade fix, fetch robustness, follower signature prewarm.
Rebased onto the updated stack (#5453 now merges
master@ 362dbc5: protocol 30, millisecond close times; #5454, #5457 and #5458 updated the same day). The PR's original four commits are kept (adapted to the new base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten commits are added on top.Bug fix: networks starting at the genesis protocol stalled (465f6a4)
Root cause. With leader-only construction (5fff543), followers no longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's set is a legacy
TransactionSet, and the Rust overlay only carriesGeneralizedTransactionSets (Core caches,Rust decodes and Core receives only that type). So a follower could get the leader's set only by
being promoted to leader by nomination timeouts and building it itself, one slow round at a time.
Before this PR every validator built the identical empty legacy set (it depends only on the
previous ledger; upgrades live in the
StellarValue), so it was always local. On perfnet (30 nodes)22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are unaffected.
Fix. When the LCL is before protocol 20, every validator constructs its nomination value at
the trigger again, as before; it is still voted only by round leaders. Leader-only construction
is unchanged from protocol 20 on.
Test.
Rust overlay upgrades from genesis through legacy tx sets: 4 nodes, genesis protocol 0,four legacy ledgers at normal cadence, an upgrade straight to protocol 30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other changes below in place,
still fails (2/2), so they don't mask the bug.
Robustness on the same path
for it are processed at once (the IPC path had lost upstream's ItemFetcher behaviour).
checked at the trigger, as before. This is what made the perfnet stall permanent.
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on every doubling of the
request count and counts
overlay.fetch.txset-retry; Rust sends a retry to a random other peerand prunes stale reservations by slot. Core also checks a delivered set's content hash.
slot inside SCP's nomination supplier.
substituted set to the overlay (31e6147), since this PR caches the built set earlier.
Performance
158 -> 205 ms) because followers no longer validated their own mempool, which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4 tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at the trigger and verify
master-key signatures on background tasks (at most 20 verifications per tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a local build or
externalization starts). They still build and compress nothing.
same instant in unspecified order, so a predicted leader could build its set twice.
follower decodes in place and moves into the frame). Sub-millisecond for classic sets.
Testing
early-nomination and Tx set validity hardening #5481 tests) and partitioned
make checkfor[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.](143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
--all-versions(devbox, at c26b191): selftest 598 test cases / 8.5M assertions across protocol versions, check-sorobans, check-rust and check-nondet pass.c26b191fa(30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, tx sets capped at 7,500; n = 1; run in parallel with the Select loadgen source accounts in constant time #5458 head on the same fixture):NOT IN CACHE/parse/cache-drop errors; 11 tx-set fetch retries, all resolved on the first re-request.Original description
Description
This PR compresses transaction-set responses in the Rust overlay and shares the encoded bytes across peer sends. Every response uses the mandatory zstd protocol; there is no legacy/raw fallback or optional lazy encoding state. The leader hands the final XDR to Rust before Core's final validation, so compression can overlap that validation. Receivers retain the original compressed bytes, decompress and verify the set, then reuse the retained encoding when forwarding it.
A separate commit,
0ed3e540, removes private proposals on non-leader nodes. Followers enter nomination with a lazy value supplier and follow received proposals without pulling the mempool or constructing a set. First-round leaders can prepare early; a promoted leader constructs a value only when it needs one. The driver caches that value across retries, and externalization stops nomination even when no local value was built. Tests cover timeout promotion, current close times, following another leader, unavailable values, and reentrant nomination stop.This PR builds on #5458. The comparison below evaluates these revisions against the original compressed implementation of this PR, using the same 10-tier-1, 30-validator fixture: 3,000 offered TPS, 6,000 transactions/block, 2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and 900-second measurement, with simulated apply.
3,000 offered TPS
Median intervals and completed throughput use the canonical node; worst-node p99 uses the fixed measurement window. The harness uses a different histogram/window:
ob-0recorded 4.989 s against its 4-second limit, while the other 29 nodes and inclusion passed. Completed throughput fell 4.1% and median block interval rose 5.1%; this run does not establish that the revised path matches or improves the original compression baseline. These are single cloud runs with normal host placement.Leader-only construction is verified: 886 builds/compressions across 421 closed ledgers, all correlated with asserted leadership, with no follower build or received-set recompression detected. Multiple nomination leaders can build in a ledger. Compared with the intermediate eager implementation, local compressions fell from 13,019 to 893 in their respective 900-second windows, roughly a 93% reduction per ledger. Payload savings remain 80.0%. The sampled received-set validation timer rose from about 158 ms to 205 ms versus that intermediate build; the precise cause is not isolated by this run.
Checklist
make formatandcargo fmt --check.