Skip to content

Merge batched GETDATA, loadgen selection, tx-set compression, and leader schedule - #5496

Merged
SirTyson merged 43 commits into
overlay-v2-sharedfrom
pr/merge-ms-close-time-base
Nov 10, 2026
Merged

SirTyson merged 43 commits into
overlay-v2-sharedfrom
pr/merge-ms-close-time-base

Conversation

@SirTyson

Copy link
Copy Markdown
Contributor

Brings the stacked PRs that were merged into pr/merge-ms-close-time-base into overlay-v2-shared:

Port 8e926b3 onto the current timing and parallel-validation branch.
Serve each demand in one asynchronous job, combining complete transaction
frames into writes of up to 64 KiB. Preserve existing framing, priorities,
retries and standalone writes for larger transactions. Retain only the
next batch and current transaction while a peer is backpressured.

Cover multi-batch delivery, missing hashes, oversized transactions, exact
bytes and hashes, and SCP progress during a blocked TX write. Add an
ordered request barrier proving blocked responses do not retain the
entire demand; this regression fails before the change and passes after.
A batched response job kept looking up and writing the rest of a demand
after a write had already failed past send_peer_frames' retries. When a
peer disconnects, every remaining batch of every in-flight demand then
fails one by one, each with a warning, wasting buffer lookups and log
volume at the moment the node is busiest.

Move the job into serve_getdata and return at the first failed batch or
standalone write. The requester re-demands unanswered hashes from another
source after its response timeout, as it already must for a lost peer.
Abandoned hashes are logged once at debug level.

Add a test serving a multi-batch demand to a peer without streams: only
the first batch is attempted. It fails before this change.
Each served transaction was first framed into a new Vec by
to_flood_frame (allocation plus copy), then copied again into the
batch, and every hash took and released the TX buffer write lock. A
node that originates load serves each transaction to most of its
peers, so this runs at several times the network TPS.

Copy the 4-byte Transaction discriminant and the shared envelope bytes
directly into the batch (ValidatedTx::flood_frame_parts), and look up a
whole batch under one buffer lock, released before every write. A
transaction larger than the batch budget is written as two parts
straight from the shared bytes, with no copy. Wire bytes are unchanged.

Pin flood_frame_parts to the typed StellarMessage encoding, and assert
in the blocked-writer test that the buffer lock is free while a response
write waits (the test times out if the lock is held across the write).
GETDATA responses now arrive as batches of complete frames, but the
reader still pulled each frame out of QUIC with two reads (the length
prefix, then the body), and every quinn read takes the connection lock
that the connection driver and the SCP and TxSet streams also use. INV
and GETDATA frames on the same stream paid the same cost.

Wrap each inbound TX stream in a 64 KiB BufReader, so one read copies
whatever QUIC has received (often a whole response batch) and frames are
then parsed from memory. read_framed becomes generic over AsyncRead; a
frame larger than the buffer is read directly once the buffer drains.
Bytes delivered to the handlers are unchanged, and the buffer never
waits for more data than is available, so no latency is added.

Also drop the unused stream argument from handle_tx_stream_message and
handle_getdata (which no longer needs to be async).

Add a test reading a batch and oversized frames through a small
BufReader over a reader that returns short chunks and yields.
Every transaction received in a GETDATA response is announced to each
connected peer that has not advertised it. The relay took and released
the INV batcher's write lock once per peer, so on a 30-node mesh each
received transaction cost up to 28 write-lock round trips, contending
with every other peer's reader task and the housekeeping flush.

Add the entry for all peers under one lock, collect the batches that
filled up, and send them after releasing it. send_inv_batch only spawns
the write, so the batches still go out immediately. Which peers are
announced to, and the batching rules, are unchanged.

Add a test covering the relay targets (not the sender, not a peer that
already advertised the TX, every other peer) and the hand-off of a batch
that this TX completes. Export INV_BATCH_MAX_SIZE from flood for it.
Review follow-up. Since df7c3213f a response job held the TX buffer
write lock while it both looked up and copied up to 64 KiB of frames.
broadcast_tx takes the same lock inline in the event loop, so on a node
serving many peers a local submission (and any SCP command queued
behind it) could wait for several jobs' copies.

Serve each batch in two phases: under the lock, only look up the
transactions that fit the budget (a hit that does not fit is carried to
the next batch); then release it, copy the frames from the shared bytes
and write. A lone transaction above the budget is still written as two
borrowed parts. Also:
- reserve the batch exactly, so its capacity stays within the budget
  (it grew by doubling to about 117 KB before);
- count flood_fulfilled when a transaction is put into a write, so a
  failed batch no longer counts the lookup that closed it;
- share one TX_RESPONSE_BATCH_BYTES constant with the inbound read
  buffer (buffered_tx_stream);
- document lazy per-batch lookup, why a failed write ends the job, and
  that a retried batch can redeliver frames (dropped via tx_seen).

Add a test for the batching edges: an oversized TX as the first hit and
another straight after it, frames exactly at the budget, a carried
oversized TX at the end and interleaved misses, checking demand order,
exact bytes and metrics. It fails if the carried TX is dropped.
Review follow-up: two behaviours had no test.

- A batch whose write fails is rewritten whole on a reopened stream.
  Existing tests only reopened streams that were already empty. The new
  test finishes the sender's cached TX stream, serves a demand spanning
  several batches, and checks every transaction arrives once, in order,
  with no write errors and no duplicates at the receiver. It fails with
  send_peer_frames' retries disabled.
- The inbound TX stream buffer (buffered_tx_stream) takes in everything
  available in one inner read and returns each complete frame without
  waiting for more data; only a read past the available data waits.
  This pins the "adds no latency" property of 90bc974c4.
Review follow-up to 39f3d5570, which batched the INV batcher lock only
for transactions received from peers. broadcast_tx, which announces
transactions submitted by the local core, still took the write lock
once per peer, and it runs inline in the overlay event loop that also
carries SCP commands, so its lock round trips cost the most.

Move the logic into queue_inv and use it from both paths: add the entry
for every peer under one lock, then send the batches it completed after
releasing it.

The test added in 39f3d5570 only checked that a completed batch left the
batcher, so it would pass if that batch were then dropped. Add a test
over a real connection: fill the receiver's pending batch to one short
of full, complete it through the relay path and then through a local
broadcast, and wait for the receiver to demand all 1000 hashes each
time. The housekeeping flush needs 100 ms, so only the completed
batch's own send can deliver them; the test fails if that send is
removed. 30 repeated runs of the new tests: no failures.
Port the focused account-pool change from 75f3fbc onto batched-getdata.
Use uniform indexed selection and swap-removal instead of walking the
available-account hash set for every transaction. Preserve release only
after application, including Soroban setup transitions.

Cover repeated account recycling and consecutive runs using real ledger
application, with more submissions than available accounts and no rejected
transactions.
cleanupAccounts runs on the main thread in valueExternalized for every
ledger, on every node, whether or not that node generates load. It hashed
and allocated a set entry for the source of every transaction in the
externalized set (about 6,000 to 7,500 per ledger in perfnet's 3,000 TPS
runs), then walked every in-use account with a std::map lookup. When
ledgers fall behind the offered load, the in-use backlog grows by tens of
accounts per ledger per generator, so the walk grew with it on the path
between externalize and the next ledger.

Keep in-use accounts indexed by public key instead. Accounts selected
since the previous call are indexed at the start of the next one (their
keys are known once TxGenerator has generated a transaction for them),
and each externalized transaction's source is looked up in that index. The
cost is now one allocation-free lookup per transaction in the set plus one
index insert per newly selected account, independent of the backlog, with an
early return when no account is waiting. Which accounts are released, and
when, is unchanged.

Add a test for perfnet's block_time_check load: synthetic SAC payments in
overlay-only mode with more transactions than source accounts, which only
completes if accounts return to the pool after their ledgers close.
handle_inv_batch took the tx_seen read lock and the inv_tracker write
lock once per INV entry. Every node receives about N-1 advertisements
per transaction, so on a 30-node mesh at 3k TPS that is roughly 170k
lock round trips per second, contending with every peer's reader task,
the TX relay and the housekeeping retries.

Filter the batch against tx_seen under one read lock, then record
sources and pick the first-advertised hashes under one tracker write
lock. The two locks are never held together. Decisions are the same:
seen TXs are skipped and not recorded as sources, a hash is requested
only when this is its first advertisement, repeats in a batch add the
peer once, and the demand keeps batch order. Tracker calls happen in
the same order, so its LRU state is the same. Also fix a comment that
claimed seen TXs were recorded as sources.

Add a test over a real connection: one INV batch mixing a new hash, a
seen hash, a hash first advertised by another peer, a repeat, and an
unsorted order. It checks the demand count, tracked sources and the
order in which the demanded transactions arrive. It passes before and
after this change, and fails if the seen filter or the first-advert
check is removed or the demand order is reversed. 30 repeated runs: no
failures.
…tion

LoadGenerator::execute built a fresh TxMetrics for every submitted
transaction, which resolved all ten loadgen meters through
MetricsRegistry::NewMeter: a MetricName allocation, a mutex, and two
searches of the registry's std::map each. It also copied the whole
envelope into a new StellarMessage (twice for classic payments) only to
measure its size.

Use the TxMetrics member resolved once at construction, which refers to
the same registry meters, and compute the message size as the 4-byte
message type discriminant plus the envelope's XDR size. The recorded
values are unchanged.

Extend the overlay-only synthetic SAC test to check the per-transaction
meters, including that loadgen.txn.bytes still equals the wire size of
each submitted StellarMessage.
The section asserted that no transaction was rejected, but on this branch
HerderImpl::recvTransaction hands every transaction to the overlay and
always returns TX_STATUS_PENDING, so loadgen.txn.rejected cannot move. The
checks that matter are that every transaction applies successfully and that
the run completes, which requires each account's cached sequence number to
match the ledger. Say so, drop the rejected assertion, and stop cranking as
soon as a run fails instead of waiting for the timeout.

Submit 2x the accounts at 200 tx/s instead of 3x at 100 tx/s. That still
forces every account through the pool more than once and stays within the
unique-account check (3 x 200 txs per one-second ledger = 600 <= 1,000
accounts), and each run takes about 10 s of submission instead of 30 s.
Prefer the negotiated zstd tx-set response protocol and cache each set's encoded payload for fanout. Encode and decode off Tokio network workers, retain existing size and send-admission bounds, and preserve legacy peer compatibility.

Port 125a248 onto the batched-getdata branch. Rust tests: 191 passed, 3 pre-existing ignored; cargo fmt passed.
HerderImpl::processExternalized needs two things from every transaction
in the externalized set on the main thread: its full hash, so the Rust
overlay can drop it from the mempool, and (in BUILD_TESTS) its source
account, so loadgen can release it. It got both by calling
createTransactionFrames, which copies every envelope into a new
TransactionFrame (with an OperationFrame per operation) only to hash it
and read its source, then frees them all. That is thousands of frames per
ledger at perfnet's load.

Add TxSetXDRFrame::forEachTransactionEnvelope, which visits the set's
envelopes in createTransactionFrames' order, and getTransactionFullHashes,
which hashes each envelope's XDR in place: the same SHA-256 that
TransactionFrame::getFullHash and FeeBumpTransactionFrame::getFullHash
compute. LoadGenerator::cleanupAccounts takes the tx set and reads each
source from the envelope with getEnvelopeSourceID, which mirrors
getSourceID: the unmuxed source, or a fee bump's inner source. The overlay
receives the same hashes in the same order, and loadgen releases the same
accounts.

Add a test that builds legacy and generalized sets (sequential and
parallel phases; v0, v1, muxed and fee-bump envelopes) from XDR and checks
the visited envelopes, full hashes and sources against the frames
createTransactionFrames builds from the same set.
Leader-only construction left followers without a local tx set, so they
had to fetch the leader's. The Rust overlay only carries generalized tx
sets, so before protocol 20 (legacy TransactionSet) a follower could not
obtain the leader's set at all: it waited until nomination timeouts
promoted it to leader and it built the set itself. A network created at
the genesis protocol (perfnet) therefore stalled before reaching its
upgrade: 22 of 30 nodes reached ledger 2, 8 stayed at ledger 1.

For ledgers whose LCL is before protocol 20, every validator constructs
its nomination value at the trigger again, as before leader-only
construction; it is voted only if the node becomes a round leader. With
an empty mempool (genesis and upgrade-only ledgers) the legacy set
depends only on the previous ledger, so every validator holds the
leader's set. Leader-only construction is unchanged from protocol 20 on.

Add a Rust overlay simulation that starts at the genesis protocol,
closes legacy-set ledgers, upgrades straight to the latest protocol and
keeps closing ledgers. It fails without this change (a node stays at
ledger 2 while the others close).
PendingEnvelopes::addTxSet only cached the set. Envelopes waiting to
fetch that hash stayed in the fetching state until the overlay delivered
the set again or the sender re-sent the envelope. Upstream's ItemFetcher
released them on addTxSet; this branch's IPC fetch path lost that.

That matters for sets the overlay cannot deliver: a leader's legacy
(pre-protocol 20) set that a validator builds itself after the leader's
nomination arrived, or a promoted leader that builds the set it was
fetching. For generalized sets it saves the IPC round trip back from Rust.

addTxSet now completes a pending fetch the same way recvTxSet does: it
updates waiting value wrappers and reprocesses the waiting envelopes,
which can reenter SCP (the nomination supplier already handles that).
The waiting envelopes are detached before reprocessing, so reentrant
fetch changes cannot invalidate the iteration. Completing a fetch now
records the overlay.fetch.txset timer, which was never updated. A
redundant overlay delivery of a known set is logged at debug level.
With leader-only construction a follower builds its value only when a
nomination timeout promotes it to leader, possibly long after the
trigger. makeNominationValue rechecked herder tracking at that point, so
a node that lost sync while the slot was stalled (35 s without
externalizing) could never propose for that slot, even once promoted.
On perfnet that turned the legacy-tx-set stall into a permanent one:
nodes waiting for a set they could not fetch lost sync, and then could
not build it either.

Tracking is checked when the trigger starts nomination, as before; a
value built at the trigger kept being nominated after losing sync. The
supplier now only rechecks what makes building unsafe or stale (ledger
manager synced and not applying, LCL still the slot's predecessor, close
time valid).

Add a herder test: a follower in a stalled unanimous quorum loses sync
and must still propose a valid value once nomination timeouts promote
it. It fails without this change (the follower never builds).
Core requested each missing tx set from the Rust overlay once, assuming
the overlay handled timeouts and retries (stopAllOutsideRange said so),
while the overlay explicitly has neither: a request to a peer that is
silent or lacks the set (there is no negative response) stays reserved
until that peer disconnects. A follower then waited forever, silently.
Pending fetches and their envelopes were also never dropped when their
slots were purged.

Core: PendingEnvelopes keeps per-fetch state (start, last request, newest
waiting slot, request count). A timer repeats requests that have not been
answered after one second, every second, counts them in
overlay.fetch.txset-retry and logs a warning on every doubling of the
request count, naming the slot and how long it waited (and that legacy
pre-protocol 20 sets cannot be fetched over the Rust overlay, when the
LCL is legacy). Fetches whose waiting slots are purged are dropped.

Rust: a RequestTxSet for a hash Core is still waiting for is a retry. It
replaces the outstanding reservation and goes to another connected peer
(the SCP source only if it was not the previous target, otherwise a
random one), instead of being deduplicated against the stuck request.
LedgerClosed prunes reservations below the retained-slot window, which
also drops the prefetches of legacy sets that no peer can serve.

Tests: a herder test for retry timing, completion and slot purging, and
a dispatcher test for retry peer selection and pruning.
The leader copied its set's whole GeneralizedTransactionSet (toXDR) to
serialize it into a temporary vector, which CacheTxSet then copied into
the IPC message. Followers copied the received payload into a vector to
decode it, and makeFromWire copied the decoded set again into the frame.
Several frame constructors also copied freshly built local XDR.

- CacheTxSet serializes the frame's XDR directly into the IPC payload
  (TxSetXDRFrame::getGeneralizedXDR gives const access to it).
- TX_SET_AVAILABLE decodes in place from the payload and moves the set
  into the frame (new rvalue makeFromWire overloads), also used by
  toWireTxSetFrame, makeEmpty and makeFromHistoryTransactions.
- PendingEnvelopes::recvTxSet rejects a set whose recomputed content hash
  differs from the hash it was delivered for; Core keyed sets by the
  overlay's hash without checking.

Add a hidden benchmark ("received tx set processing costs") for the
dissemination path. For 6,000 classic payments (1.2 MB) on the laptop:
leader encode 0.71 -> 0.35 ms, follower frame 1.16 -> 1.04 ms (n=5
medians). The savings are small for classic sets and grow with Soroban
transactions' nested XDR. The benchmark also shows received-set
validation at 67 ms with a cold signature cache vs 10.5 ms warm
(4 tasks).
Before leader-only construction every validator built its own set at the
trigger, validating its mempool and so populating the signature
verification cache. When the leader's set arrived, its signatures were
mostly cache hits. Followers now validate the leader's set cold: the PR's
perfnet run saw the received-set validation timer rise from about 158 ms
to 205 ms. The "received tx set processing costs" benchmark reproduces
that: validating 6,000 transactions takes 67 ms with a cold signature
cache and 10.5 ms warm (4 tasks, the physical cores of a perfnet node).

At the trigger, a validator that is not a round-1 leader (and not on a
legacy tx-set protocol, where everyone builds) pulls its mempool's top
transactions once (up to a full set) and verifies their source, operation
source and fee source master-key signatures on background tasks, as many
as the batch executor would use. The verification goes through
SignatureUtils::verify, so it fills exactly the cache entries tx-set
validation looks up. Signatures by other signers need ledger state and
are left to validation. The prewarm stops as soon as validation of a
received set (a validity-cache miss) or a local build begins, which then
verify the rest themselves on all cores.

The follower's main thread does the pull while it waits for the leader's
nomination; it still builds and compresses nothing.

In-process simulations share the global signature cache across nodes, so
they cannot show the effect; it needs a perfnet comparison.
Follow-up to the fetch retries, from review:
- Retries back off (1, 2, 4, then every 8 s). A slow but live transfer
  of a multi-MB set, under exactly the load where links are slow, was
  duplicated to another peer every second.
- Only sets for slots after the LCL are re-requested. A fetch kept for
  the checkpoint slot (never purged) was re-requested every second until
  the checkpoint moved, and the overlay had evicted that set anyway.
- overlay.fetch.txset again only times sets delivered by the overlay,
  not fetches a local build completed.
- A Rust retry goes to a random connected peer other than the previous
  target. Preferring the SCP source again sent every other retry back to
  a source that may not answer.
Follow-up to the signature prewarm, from review:
- Skip it when this node already voted in the slot (it validated the
  leader's value cold, so the prewarm would only re-check cached
  signatures while balloting runs).
- Stop it when a slot externalizes and on herder shutdown, not only when
  validation or a local build starts.
- Leave two worker threads free for long background jobs (bucket
  merges): tasks = min(batch executor tasks, WORKER_THREADS - 2).
- Cap verifications per transaction at 20 and only try keys whose hint
  matches, so a crafted transaction with colliding hints cannot make the
  prewarm expensive.
- The test no longer asserts exact process-global verification counts
  (other code flushes them); it asserts that validation finds every
  signature cached.
Since locally built sets complete pending fetches, a leader that builds
the very set a peer's ballot is waiting for reprocesses that envelope
inside SCP's nomination value supplier. With the peer v-blocking, the
slot externalizes before the supplier returns. Cover that path: the
peer's CONFIRM waits for the (empty, hence identical) set, the local
leader's trigger builds it, the slot closes with the peer's value, and
nomination stops without voting or emitting a nomination.
setupTriggerNextLedger arms the early-preparation timer for a predicted
first-round leader at "now", next to the trigger timer. When the trigger
is already due, both fire at the same instant, and VirtualClock orders
equal-time events from different timers arbitrarily (event sequence
numbers are per timer). If preparation ran first, the leader built its
set once to prepare and again at the trigger (an underfilled prepared set
is refreshed), costing a full build on exactly the late ledgers where
the trigger is due immediately.

"early preparation respects manual and immediately due triggers" caught
it: with leader-only construction the test reschedules with both timers
due, and it failed in partitioned make check (base instance 400, seed
20766: two pulls instead of one) while passing with other instances.

Arm early preparation only when the trigger is still ahead; a due
trigger builds itself. The test now also checks that nothing was
scheduled to prepare, instead of depending on timer order.
Phase timers herder.txset.build-{pull,frames,trim,select,xdr,handoff,
roundtrip,final-check} around HerderImpl::buildTxSet and
makeTxSetFromTransactions, and a hidden benchmark ("leader tx set build
costs", tags [herder][.][benchmark]) that hands the herder 15,000 SAC
payments (ApplyLoad's generator) through an exact replay of the overlay
IPC decode and times each phase with a cold and a warm signature cache,
plus cold received-set validation for calibration against perfnet.
BENCH_REPS, BENCH_TASKS, BENCH_WARM and BENCH_PULL control it; under
callgrind with --instr-atstart=no only the cold builds are instrumented.
The leader pulls twice the tx-set capacity from the mempool and validated
every candidate, although selection keeps at most the capacity. Signature
verification of the surplus was most of the build's cost (15,000 SAC
candidates for 7,500 slots on perfnet).

Keep the same pull, but validate each phase in mempool order only until it
holds 5/4 of its capacity in valid transactions of its generic surge
pricing lane (TxSetUtils::trimInvalidPrefix): the first 5/4 of the
capacity, then twice the shortfall if some were invalid, then everything
left, so at most three passes and each candidate validated at most once.
DEX-lane transactions don't count toward the target, since their lane
can't fill the phase. Candidates past the validated prefix are neither
selected nor reported invalid; they stay in the mempool.

The selected set is the same as when validating everything unless
selection would have reached past the prefix: a candidate that doesn't fit
the remaining resources leaving room that only later candidates fit, a
mempool order (total fee per operation) that differs from the inclusion
fee order of Soroban transactions, equal fees at the cutoff (broken
randomly either way), or a fee source paying for several candidates (its
fees accumulate over fewer of them).

Laptop benchmark (15,000 SAC candidates, 4 tasks, n=5 medians): cold
signature cache 261.0 -> 183.6 ms, warm 124.3 -> 98.7 ms.
SirTyson and others added 13 commits November 10, 2026 13:22
Building a tx set hashed every candidate's full envelope on the main
thread (removing the invalid ones, then sorting the set by hash): SHA-256
of every candidate, 18% of a cold build's instructions on hosts without
SHA extensions such as c5d. Compute and cache each candidate's full hash
in the parallel validation task that already validates it (which also
computes its contents hash for the signature checks), and skip
removeTxs' lookups when nothing is invalid.

Same hashes, same order: only where they are computed changes. Each
frame is in exactly one task range and the batch completes before the
main thread reads the cached hash.

Laptop benchmark (has SHA-NI): warm build-trim 22.1 -> 18.0 ms.
The parallel Soroban phase builder sorts all candidates by inclusion fee
rate, and each comparison looked up both transactions' inclusion fee and
operation count through virtual calls (~1,450 instructions per comparison
on the main thread). Look them up once per transaction and compare the
cached values with the same comparator logic: TxFeeComparator gets an
overload that takes the fee bids and operation counts, and the plain
overload forwards to it, so both order identically (same tie-break).

Laptop benchmark (15,000 SAC candidates, 9,375 validated): build-select
17.1 -> 14.0 ms cold, 17.2 -> 13.6 ms warm.
OverlayIPC::getTopTransactions copied every transaction's XDR into a
temporary vector before decoding it. Decode well-formed entries straight
from the response payload (OverlayIPC::parseTopTransactions); entries
that are not 4-byte aligned, which a well-formed response never has, are
still decoded from an aligned copy, and entries that fail to decode are
still skipped. The leader build benchmark now decodes its replayed
response with the same function.

Laptop benchmark (15,000 SAC candidates): build-pull 19.4 -> 10.7 ms.
PendingEnvelopes timed its tx-set fetches (request to overlay delivery)
into overlay.fetch.txset, which RustOverlayManager also feeds with the
Rust overlay's own fetch times, synced as per-batch averages: one timer
mixing two measurements, with about two samples per ledger. Name Core's
timer scp.fetch.txset, next to scp.fetch.envelope from the same class,
and its retry counter scp.fetch.txset-retry; overlay.fetch.txset is left
to the overlay. docs/metrics.md lists both.
On perfnet (3k SAC TPS) followers still validated the leader's set with
a mostly cold signature cache, and the leader built with a cold one. The
follower prewarm verified each follower's own top 7,600 mempool txs at
the trigger, but with equal fees the mempool orders by hash while
selection breaks ties randomly, so the leader's 7,500 came from its top
15,200 and only about half were covered; the leader had no prewarm.

When slot N starts balloting, every tracking validator now starts a
prewarm for ledger N + 1 over exactly the candidates its build validates
first: per phase, the first validTxTargets() (5/4 of the capacity) in
mempool order, past the transactions of N's value, which stay in the
mempool until N externalizes. The leader's set comes from the same
window, so a finished prewarm covers its build and the followers'
validation. It runs while N finishes balloting, externalizes and closes:
externalizing N keeps it; the build or a received-set validation stops
it, as before. The follower's trigger-time prewarm stays as a top-up.

The main thread only does the overlay round trip (posted after the SCP
step that started the ballot; OverlayIPC::getTopTransactionsPayload
returns the response undecoded). Decoding, windowing and verification
run in at most min(batch executor tasks, WORKER_THREADS - 2) background
tasks, which also free what they allocated.

Metrics, so a perfnet run shows coverage: herder.prewarm.{pull,duration}
timers, herder.prewarm.{txs,verified,incomplete} meters, and signature
cache lookups during a build and during received-set validation,
herder.txset.{build,validate}-sig-cache-{hit,miss} (from the new
non-resetting PubKeyUtils::getVerifySigCacheCounts). docs/metrics.md
lists them, and the build phase timers.

The leader build benchmark gets a prewarmed mode. Laptop, 15,000 SAC
candidates, 4 tasks: build cold 169.2 / warm 84.1 / after a prewarm 83.0
ms; received-set validation 102.2 / 34.2 / 37.7 ms; both with no
signature verified after the prewarm, which itself takes 110.8 ms in
the background.
Both ends of the IPC rejected payloads over 16 MiB, and Core's reader
treats a rejected length as a corrupt stream: it stops reading and never
recovers, so the node can no longer send or receive SCP messages. The
ballot-start signature prewarm pulls 2 x the tx-set capacity plus the
balloting value's transactions: 22,700 SAC payments of ~752 bytes, about
17.2 MB, once the mempool holds that many. On perfnet (3k SAC TPS, 30
validators) every validator starts that pull at the same ballot, so 80 s
into the load 16 of the 30 nodes lost their overlay within 0.35 s, the
rest lost quorum, and the network stalled at one ledger.

- Raise the limit to 64 MiB on both sides, named once per side
  (ipc::MAX_PAYLOAD_BYTES in IPC.h, MAX_PAYLOAD_SIZE in messages.rs)
  with a note that they must match.
- The overlay sends at most the longest prefix of the top transactions
  that fits in the limit (the highest-priority ones), with the count
  matching, and warns when it truncates, so a large mempool can't produce
  an oversized response whatever the transaction sizes.
- Neither side sends a payload over the limit: CoreSender refuses it
  (IpcError::PayloadTooLarge), and Core's sendMessage refuses it before
  writing anything, keeping the connection.
- Core logs an error with the message type and length when it rejects a
  payload, instead of only "connection closed".
- Tests: a [overlay-ipc] test with the real Rust overlay where a 20 MiB
  top transactions response arrives whole (it fails with the old limit:
  the pull returns nothing) and an 80 MiB pull is truncated to the 63
  highest-fee transactions while the IPC keeps working; IPC tests for a
  20 MB message, an over-limit length (rejected, with a receive timeout
  so a regression fails instead of hanging) and an over-limit send
  (nothing written); Rust tests for the truncation and the send guard.
- docs/rust-overlay/ipc.md: the 64 MiB limit and how each side enforces
  it.
Experimental. Not protocol-gated and not compatible with nominating nodes.

Nomination is removed. For slot N every node orders the transitive quorum
by SHA256(hash of ledger header N-1 || node id); the leader of round r (a
round is a ballot counter, starting at 1) is entry (r - 1) mod n. The round
leader builds a value and proposes it straight into the ballot protocol as
PREPARE at counter r. Other nodes vote for the leader's ballot at that
counter if it passes proposal validation (the former nomination-time
checks, e.g. only upgrades the node armed). A leader timer (while the node
has no ballot for its round) or the ballot timer moves a node to the next
round. Accept, confirm and commit are unchanged, and every new vote still
goes through bumpState (value override, updateCurrentValue).

Details:
- Followers vote for the round 1 leader as soon as they see its ballot,
  without waiting for their own trigger.
- A node that has not voted yet votes for an earlier round's leader ballot
  at that ballot's counter when it arrives late, and keeps waiting for its
  current round's leader.
- The leader timer gets one extension when the Herder already holds the
  leader's statement (e.g. it is fetching its transaction set).
- A leader's empty-tx-set ballot is followed under ballot validation.
- No leader and no leader timer at the UINT32_MAX counter.
- State restore sets the value override and the round.
- Herder: the trigger starts the slot; the early tx-set build and the
  follower signature prewarm key off the round leader; NOMINATE envelopes
  are discarded. New metrics scp.timeout.leader and scp.leader.round;
  scp.timing.nominated, scp.timeout.nominate and
  scp.nomination.combinecandidates are gone.

Removed: NominationProtocol, the nomination API of SCP/Slot/SCPDriver,
combineCandidates, node weights, upgrade stripping, the nomination emit
delay, and the nomination-only tests (SCPUnitTests.cpp, nomination cases
of SCPTests.cpp, weight / win-rate / timeout statistics in HerderTests.cpp).
FORCE_OLD_STYLE_LEADER_ELECTION, VALIDATOR_WEIGHT_CONFIG and the nomination
timeout settings still parse but are unused.

Tests: leader round unit cases and a randomized multi-node model in
SCPTests.cpp, LeaderScheduleTests.cpp, LeaderRoundsTests.cpp (network
simulations, including failover with a validator down); Herder, quorum
tracker, PendingEnvelopes and overlay IPC tests ported to leader rounds.

Known limitations: symmetric, static quorums are assumed (the schedule's
node set is each node's view of the transitive quorum); uniform leader
weights; the header hash can be ground by a leader; no upgrade stripping
or merging; a leader re-proposes a value that is only structurally valid.
… README

Replace the stock README on this experimental branch with a description of
the change, the 2 s (pass) and 1 s (fail) perfnet results measured on
daa965f, and how they were run: topology, hardware, node configuration,
network settings, load and pass criteria.
## Update (2026-11-09)

Rebased onto the updated #5454 (which now sits on #5453's merge of
`master` @ 362dbc5; the overlay crate moved to `stellar-xdr` 30.0.0
there). The original commit is unchanged apart from a test import
updated for stellar-xdr 30.0.0 (`stellar_xdr::curr::` to
`stellar_xdr::`). Eight commits on top, each with tests (an independent
review found no correctness, wire-format or deadlock issues in the
original commit; its low-severity findings are fixed below):

- **Stop serving a demand after a failed write.** A response job returns
at the first write that fails after the stream retries (usually a
disconnect), instead of looking up, writing and logging a failure for
every remaining batch. The requester re-demands unanswered hashes after
its response timeout.
- **Frame responses straight from the shared transaction bytes.** Each
frame's discriminant and envelope are copied directly into the batch,
with no per-transaction allocation; a transaction larger than the batch
budget is written as two borrowed parts with no copy. Each batch is
looked up under one TX-buffer lock, released before frames are copied or
written, so local submissions on the event loop never wait for a
response copy.
- **Buffer inbound TX stream reads (64 KiB).** One QUIC read takes in
whatever has arrived, often a whole response batch, instead of two reads
per frame. The reader never waits for more data than is available.
- **Queue INVs for all peers under one lock**, for both relayed and
locally submitted transactions, instead of one batcher lock per peer.
- **Handle each received INV batch with one seen-set lock and one
tracker lock**, instead of both locks per advertised hash. Request
decisions and their order are unchanged.
- **More tests:** batching edges (an oversized transaction first, two in
a row, frames exactly at the budget, carried hits, misses); a failed
batch write retried whole on a reopened stream; buffered reads; delivery
of INV batches completed by either path; INV request decisions and
order.

Wire bytes are unchanged. Testing on the rebased branch:
- 194 overlay crate tests pass (`cargo test` in `overlay/`, 3 ignored);
`cargo fmt --check` is clean.
- The standard build succeeds, and 25 C++ overlay-IPC test cases (136
assertions) pass: `[overlay-ipc]`, `[overlay][ipc]` and the 10-node
`[overlay-ipc-large]` consensus case.
- The full suite passes on the #5458 head that contains this PR
(selftest 590 test cases, check-sorobans, check-rust with these overlay
tests, check-nondet).

The follow-up commits have not been measured separately on perfnet; the
numbers above are for the original commit on the previous base. The
perfnet run of the final #5458 head (which contains this PR) is reported
on #5458.

**Next lever (not in this PR):** a node waits up to 100 ms (checked
every 50 ms) before announcing new transactions to a peer, so each
flooding hop adds 100-150 ms before the peer can request them. A perfnet
A/B of shorter delays (50 ms and 25 ms) is the planned follow-up,
comparing tx pull latency, tx-set fetch p99 and block intervals against
bandwidth and CPU.

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the runs of the final #5458 and #5459 heads contain
this PR and are reported on those PRs.

# Description

This PR batches GETDATA transaction responses in the Rust overlay.
Previously, a request for many transactions created a separate task and
stream write for every response. Under load, that produced unnecessary
scheduling and stream-lock overhead, and a slow peer could leave the
entire response demand queued in memory.

We now serve each request in one asynchronous job, combining complete
transaction frames into writes of up to 64 KiB. Each transaction keeps
its existing wire framing, and the writes use the existing stream
priorities and retry policy. Transactions larger than the batch budget
are sent individually.

Here, the control image is the PR this is based on
#5454. This is the 10 tier
1, 2 second target block latency test. Median block intervals are from
the canonical node; p99 is the worst node over the full measurement
window.

## 2,000 TPS

| Metric | Control | Batched GETDATA |
|---|---:|---:|
| Median block interval | 1.993 s | 2.000 s |
| Worst-node p99 block interval | 5.090 s | 2.508 s |
| Tx-set fetch p99 | 1.720 s | 0.778 s |
| Harness verdict | FAIL | PASS |

## 3,000 TPS

| Metric | Control  | Batched GETDATA|
|---|---:|---:|
| Median block interval | 2.434 s | 2.363 s |
| Worst-node p99 block interval | 4.122 s | 4.524 s |
| Tx-set fetch p99 | 1.757 s | 1.087 s |
| Harness verdict | FAIL | FAIL |

At 2,000 TPS, the largest improvement is in the tail: worst-node p99
falls by about half, and nearly all tx-set fetches longer than two
seconds disappear. This is consistent with reducing transaction-response
write overhead and contention affecting consensus transfers.

# Checklist

- [x] Reviewed the contributing document.
- [x] One commit on top of #5454, with no merge commits. (Now nine: the
original plus eight follow-ups, see the update above.)
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Release overlay build and Jenkins package build succeeded.
- [x] Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++
overlay-IPC cases / 82 assertions.
- [x] Performance comparison included above.
## Update (2026-11-09)

Rebased onto the updated #5457 (the stack now includes #5453's merge of
`master` @ 362dbc5). Four commits now sit on top of the original
change.

- **Match externalized accounts in time proportional to the tx set.**
`cleanupAccounts` runs in `valueExternalized` on every node for every
ledger. It used to allocate a set entry for the source of every
transaction in the externalized set, then walk every in-use account with
a map lookup. When ledgers fall behind the offered load, the in-use
backlog grows, and that walk grew with it on the externalize path.
In-use accounts are now indexed by public key, and each externalized
transaction's source is looked up there. The cost no longer depends on
the backlog, and the lookups allocate nothing. Which accounts are
released, and when, is unchanged.
- **Stop re-registering loadgen metrics and copying envelopes per
transaction.** `execute()` used to resolve ten meters through the
metrics registry for every transaction, and copied the envelope into a
`StellarMessage` just to size it. It now uses the meters resolved at
construction and sizes the message from the envelope. The recorded
values are identical.
- **Hash externalized transactions without rebuilding their frames.** To
give the Rust overlay the full hashes of the externalized transactions,
and loadgen their source accounts, `processExternalized` built a
`TransactionFrame` for every transaction in the set on the main thread
(about 7k per ledger at 3k TPS). That copied each envelope and built one
operation frame per operation. The set's envelopes are now hashed in
place (`TxSetXDRFrame::getTransactionFullHashes`), and loadgen reads
sources straight from the envelopes. The overlay receives the same
hashes in the same order, and loadgen releases the same accounts.
- **Tests.** On this branch submission never rejects, so the recycling
test's "no rejected transactions" assertion could not fail. It now
relies on every transaction applying and on runs completing with synced
sequence numbers. It also fails fast and runs in about a third of the
time. New tests:
- "synthetic SAC load recycles source accounts in overlay-only mode"
runs perfnet's load (synthetic SAC payments, overlay-only, more
transactions than accounts). It checks completion and the
per-transaction meters, including `loadgen.txn.bytes`.
- "externalized tx set hashes and sources match transaction frames" pins
the hashes and sources from legacy and generalized sets (sequential and
parallel phases; v0, v1, muxed and fee-bump envelopes) against frames
built from the same set.

Known limit: in the `MIXED_PREGEN_*` modes, the Soroban stream fails the
run when every source account has a transaction in flight, whereas the
other modes wait. At perfnet's 33k accounts per generator, this needs a
backlog of hundreds of slow ledgers, which is longer than a
block_time_check run.

Testing:
- New and changed tests by name: the six loadgen tests (including both
new ones and the sped-up recycling test) pass.
- `[loadgen],[herder],[txset]` (partitioned `make check`): 94/94 test
cases pass.
- Full suite at the final head `e5ec7d4cb` (which contains the whole
stack): selftest 590 test cases / 2.85M assertions, check-sorobans,
check-rust, check-nondet all pass.
- An independent review found no correctness, determinism or safety
issues in the original commit or the follow-ups.

The performance tables above are for the original commit on the previous
base. Perfnet run of the final head (2026-11-09):

**Perfnet, final head `e5ec7d4cb`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered from all 30 nodes, 2 s target, 150 ms
simulated apply, tx sets capped at 7,500; n = 1):
- Correct: protocol 30 with millisecond close times on all nodes; no
forks, all in sync, no invalid values/signatures, no restarts, no
overlay errors during the load.
- Verdict FAIL (window 1 p75 3.25 s on every node, p99 3.89 s),
essentially the same speed as the previous image of this PR on the same
fixture (p75 3.32 s, p99 3.92 s): about 3.1 s per ledger with full
7,500-tx ledgers and a growing backlog.
- Per ledger: ~0.3 s externalize + close (incl. 150 ms apply), **~1.05 s
tx-set build** before nomination, ~1.34 s nomination (of which ~0.33 s
tx-set fetch, ~0.34 s validation), ~0.28 s ballot. Every trigger was
already due (backlog), so the leader's early tx-set preparation never
ran and the build sits on the critical path. The new trigger metrics
(#5454) show lateness p75 0.25 ms and no prepare-start fallbacks.
- This PR's changes: loadgen submit 7-10% cheaper (mean 2.71 vs 2.91
ms); demand timeouts -21% and duplicate flood bytes -21% (both only
during the first minutes of load); neither is on the ledger critical
path in this fixture.

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet (30 validators in
4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated
apply, 900 s load); the run of this PR's final head will be posted here.

# Description

This PR removes the linear account-selection operation from loadgen.
Previously, each transaction selected a random position in an unordered
set and walked to it. With large account pools, that traversal stalled
submission callbacks on the main thread.

Available accounts now live in a vector. Selection uses a uniform random
index, and removal swaps the selected account with the last entry before
popping it. Both operations take constant time. Accounts return to the
available pool after their ledger has applied; the test covers account
recycling and consecutive loadgen runs.

Here, the control image is the PR this is based on,
#5457. This is the 10 tier
1, 2 second target block latency test. Median block intervals are from
the canonical node; p99 is the worst node over the full measurement
window.

## 2,000 TPS

| Metric | Control | Loadgen account selection |
|---|---:|---:|
| Median block interval | 2.000 s | 1.956 s |
| Worst-node p99 block interval | 2.508 s | 2.595 s |
| Mean loadgen submission batch | 48.28 ms | 2.68 ms |
| Harness verdict | PASS | PASS |

## 3,000 TPS

| Metric | Control | Loadgen account selection |
|---|---:|---:|
| Median block interval | 2.363 s | 2.341 s |
| Worst-node p99 block interval | 4.524 s | 3.105 s |
| Mean loadgen submission batch | 88.31 ms | 3.62 ms |
| Harness verdict | FAIL | FAIL |

Submission overhead falls substantially at both loads. At 3,000 TPS,
worst-node p99 improves, but both runs still fail the harness p75
threshold. Submission-batch means use timer count/sum differences across
the ten load generators.

# Checklist

- [x] Reviewed the contributing document.
- [x] One commit on top of #5457, with no merge commits.
- [x] Formatting checked with `make format`.
- [x] Local build and Jenkins package build succeeded.
- [x] Relevant loadgen tests passed: 8 cases / 947 assertions, including
real-apply account recycling and consecutive runs.
- [x] Performance comparison included above.
## Update (2026-11-10): cheaper leader build, ballot-start signature
prewarm, IPC size fix; perfnet passes on c6id

Head is now `ebdd73635`, 8 commits on top of `c26b191fa`. They fix the
two things the 2026-11-09 perfnet run found, plus a liveness bug the
next run found.

### Leader build
On perfnet the leader's 1.1 s build was on the critical path every
ledger.
- `ba4955347` **validates only what selection can use.** The leader
still pulls 2x the capacity, but validates each phase in mempool order
only until it holds 5/4 of the capacity in valid transactions. If some
are invalid it refills: twice the shortfall, then the rest. The selected
set is the same unless selection would have reached past the validated
prefix.
- `a1bffae42` hashes the candidates on the validation workers, not the
main thread. That was 18% of a cold build's instructions on hosts
without SHA extensions, such as c5d.
- `966d3bb41` precomputes the parallel builder's sort keys.
- `95be616c7` decodes mempool pulls straight from the IPC payload,
without copying.
- `2a30c1dc0` adds phase timers
`herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}`
and a hidden leader-build benchmark.
- Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5
medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%).
Main-thread instructions -41% (callgrind).

### Signature prewarm from ballot start, on every validator
(`a91547bec`)
The 2026-11-09 follower prewarm covered only about half of the leader's
set. The fees are equal, so the mempool orders transactions by hash, but
selection breaks ties randomly. The leader also had no prewarm.
- Now, when slot N starts balloting, every validator verifies the
signatures of exactly the window that the N+1 build validates first.
This runs while N finishes balloting, externalizes and closes. The main
thread only makes the overlay round trip; decoding and verification run
on background tasks.
- New metrics: `herder.prewarm.{pull,duration}`,
`herder.prewarm.{txs,verified,incomplete}` and
`herder.txset.{build,validate}-sig-cache-{hit,miss}`.
- Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set
validation takes 38 ms (cold 102). 0 cache misses.
- `0cf227b94`: Core's tx-set fetch timer is now `scp.fetch.txset`, with
`scp.fetch.txset-retry`. `overlay.fetch.txset` is now only the Rust
overlay's time; it had been fed by both.

### Bug fix: oversized IPC payloads (`ebdd73635`)
Perfnet found this on `a91547bec`, on c5d.
- **What happened:** 80 s into the load the whole network stopped at one
ledger. 16 of 30 nodes logged `Overlay IPC connection closed` within
0.35 s of each other. The rest then lost quorum, and all lost sync.
- **Cause:** the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700
SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the
Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats
a rejected length as a corrupt stream: it stops reading and never
recovers. Every validator starts the pull at the same ballot.
- **Fix:**
- The limit is now 64 MiB on both sides (`ipc::MAX_PAYLOAD_BYTES` and
`MAX_PAYLOAD_SIZE`, which must match).
- The overlay returns the longest prefix of the top transactions that
fits, with a warning, so no mempool can produce an oversized response.
- Neither side sends an over-limit payload: Rust refuses with
`PayloadTooLarge`, and Core refuses before writing anything.
  - Core now logs the type and length when it rejects one.
- This also covers a maximum-size peer tx set (`MAX_TXSET_XDR_SIZE` + 32
bytes), which could have hit the old limit too.
- **Test:** a new `[overlay-ipc]` test with the real Rust overlay. A 20
MiB top-transactions response arrives whole; with the old code the pull
returns nothing (`0 == 20`). An 80 MiB pull is truncated to the 63
highest-fee transactions, and the IPC keeps working.

### Testing
- **`a91547bec`:**
  - Laptop: new and changed tests, and the step-2 tags (149 test cases).
- Devbox: full suite `--all-versions` PASS (selftest 604 test cases /
8,638,027 assertions, check-nondet, check-rust).
- **`ebdd73635`** (devbox):
- New tests: the `[overlay-ipc]` test above, plus IPC tests for a 20 MB
message, an over-limit length (rejected, with a receive timeout so a
regression fails instead of hanging) and an over-limit send (nothing
written). Rust tests cover truncation and the send guard.
- `[ipc]` (7 test cases), `[overlay-ipc]` (20), partitioned
`[overlay]~[ipc-crossproc]` and the Rust `ipc::` tests (23): all pass.
- Two pre-existing failures reproduce on `a91547bec` without the change:
- The Rust `test_tx_not_blocked_by_scp_flood` fails its own validity
check (an SCP flood of 6.7 ms is under its 10 ms floor).
- The hidden `IPC cross-process communication` test needs an unbuilt
`overlay-stub` binary.
- Full suite `--all-versions` on `ebdd73635` (devbox): PASS. selftest
605 test cases / 8,638,079 assertions across protocol versions,
check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban
host tests (103 `test result: ok`, 0 failed). The overlay crate's own
tests run separately; see above.

### Perfnet, head `ebdd73635`
30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2
s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s
measured. One run per instance type, both on the same day: c5d.2xlarge
(perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake,
SHA-NI). Same 8 vCPU / 16 GiB.

**c6id.2xlarge: PASS.** All four windows on target, inclusion passed
(2,880,000 txs), all checks passed.

| Window | avg p75 | avg p99 | p75 range over nodes |
|---|---:|---:|---|
| 1 | 2,112 ms | 2,334 ms | 2,103-2,129 ms |
| 2 | 2,099 ms | 2,391 ms | 2,091-2,113 ms |
| 3 | 2,101 ms | 2,324 ms | 2,092-2,110 ms |
| 4 | 2,099 ms | 2,354 ms | 2,089-2,110 ms |

**c5d.2xlarge: FAIL** in window 1, but 19% faster than the previous head
and #5458. The c5d network fell behind the offered load, so sets were
full at 7,500 and the mempool backed up. That's exactly the case that
stalled `a91547bec`, and here there were no IPC disconnects or sync
losses. Correctness passed: error counters, in sync, no fork, still
closing. Inclusion was skipped because the load stopped after window 1.

| c5d, window 1 (5-min means) | #5458 head | #5459 `c26b191fa` | **#5459
`ebdd73635`** |
|---|---:|---:|---:|
| avg p75 / p99 | 3,250 / 3,891 ms | 3,264 / 3,672 ms | **2,637 / 2,920
ms** |
| Ledgers per 5 min | 96 | 96 | 119 |
| Trigger to first ballot | 2.52 s | 2.55 s | 1.94 s |
| Leader build | 1.15 s | 1.11 s | 0.68 s (candidate validation 0.20 s)
|
| Received-set validation | 0.34 s | 0.52 s | 0.33 s |
| Nomination timeouts per node | 5 | 106 | 34 |
| Ledger close | 0.229 s | 0.228 s | 0.229 s |

Both instance types, same build:

| | c5d | c6id |
|---|---:|---:|
| Ledger interval | about 2.53 s | about 2.03 s |
| Txs per ledger | 7,500 (backlog) | about 6,100 (kept up with 3,000
TPS) |
| Trigger to first ballot | 1.94 s | 1.22 s |
| Leader build | 0.68 s | 0.27 s |
| Received-set validation | 0.33 s | 0.20 s |
| Signature-cache coverage: build / validation | 89.5% / 98.7% | 87.9% /
97.8% |
| Prewarm: mean duration / main-thread pull / stopped early | 0.21 s /
22 ms / 0.1% | 0.10 s / 7 ms / 0% |
| Ledger close | 0.229 s | 0.164 s |

Notes:
- **No A/B on c6id.** #5458 has not been run on c6id, so the c6id PASS
shows this head can hold the target on current-generation x86. It
doesn't show how much of that comes from this PR. Every earlier run of
this fixture was on c5d.
- **Two regimes.** On c6id the network keeps up, so sets stay at about
6,100 transactions and triggers aren't all due at once; #5454's early
preparation runs, giving about 2 builds per ledger network-wide. On c5d
it falls behind, sets are full, and every ledger is heavier. So
per-ledger timings aren't like for like across the two.
- **What c5d still needs** (about 0.24 s to pass):
- Received-set validation still takes 0.33 s with 98.7% of its
signatures cached, so the rest of validation is now the cost.
  - About 245 ms of the c5d build is outside the phase timers.
- The remaining trigger-to-ballot time beyond build, fetch and
validation is about 0.57 s of nomination messaging across 4 regions.
- n = 1 per instance type.

---

## Previous update (2026-11-09)

_Genesis-upgrade fix, fetch robustness, follower signature prewarm._

Rebased onto the updated stack (#5453 now merges `master` @ 362dbc5:
protocol 30, millisecond close times; #5454, #5457 and #5458 updated the
same day). The PR's original four commits are kept (adapted to the new
base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten
commits are added on top.

### Bug fix: networks starting at the genesis protocol stalled
(465f6a4)

**Root cause.** With leader-only construction (5fff543), followers no
longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's
set is a legacy
`TransactionSet`, and the Rust overlay only carries
`GeneralizedTransactionSet`s (Core caches,
Rust decodes and Core receives only that type). So a follower could get
the leader's set only by
being promoted to leader by nomination timeouts and building it itself,
one slow round at a time.
Before this PR every validator built the identical empty legacy set (it
depends only on the
previous ledger; upgrades live in the `StellarValue`), so it was always
local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade
ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are
unaffected.

**Fix.** When the LCL is before protocol 20, every validator constructs
its nomination value at
the trigger again, as before; it is still voted only by round leaders.
Leader-only construction
is unchanged from protocol 20 on.

**Test.** `Rust overlay upgrades from genesis through legacy tx sets`: 4
nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol
30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at
ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other
changes below in place,
still fails (2/2), so they don't mask the bug.

### Robustness on the same path
- 193fee3: a locally built set completes a pending fetch of the same
hash, so envelopes waiting
for it are processed at once (the IPC path had lost upstream's
ItemFetcher behaviour).
- 942a6b3: a follower promoted to leader after losing sync can still
propose; tracking is
checked at the trigger, as before. This is what made the perfnet stall
permanent.
- ab4eb14, 5964ce6: unanswered tx-set requests are retried and
reported. Core re-requests
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on
every doubling of the
request count and counts `overlay.fetch.txset-retry`; Rust sends a retry
to a random other peer
and prunes stale reservations by slot. Core also checks a delivered
set's content hash.
- 5c159ae: test for a leader whose own build completes a pending fetch
and externalizes the
  slot inside SCP's nomination supplier.
- Rebase onto the new base: the test-only nomination override from #5481
now also hands its
substituted set to the overlay (31e6147), since this PR caches the
built set earlier.

### Performance
- 3903e4e, f2643ba: follower signature prewarm. The received-set
validation timer rose (about
158 -> 205 ms) because followers no longer validated their own mempool,
which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4
tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at
the trigger and verify
master-key signatures on background tasks (at most 20 verifications per
tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a
local build or
  externalization starts). They still build and compress nothing.
- c26b191: no early tx-set preparation when the trigger is already
due. Both timers fired at the
same instant in unspecified order, so a predicted leader could build its
set twice.
- 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths
(leader serializes in place,
follower decodes in place and moves into the frame). Sub-millisecond for
classic sets.

### Testing
- Laptop, on c26b191: new and changed tests (24, including the genesis
repro and the trigger,
  early-nomination and #5481 tests) and partitioned `make check` for

`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]`
  (143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
- Full suite `--all-versions` (devbox, at c26b191): selftest 598 test
cases / 8.5M assertions across protocol versions, check-sorobans,
check-rust and check-nondet pass.
- **Perfnet, head `c26b191fa`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply,
tx sets capped at 7,500; n = 1; run in parallel with the #5458 head on
the same fixture):
- **The genesis-upgrade bug is fixed on a real network:** all 30 nodes
synced in 43 s from genesis through the legacy ledgers to protocol 30
(the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger
1).
- Correct under load: no forks, all in sync, no invalid
values/signatures, no restarts, no `NOT IN CACHE`/parse/cache-drop
errors; 11 tx-set fetch retries, all resolved on the first re-request.
- Leader-only construction works as designed (~1 build per ledger
network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from
~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
- Verdict FAIL at the same speed as the #5458 head (window-1 p75 3.26 s
vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and
follower validation of the received set was 0.52 s vs 0.34 s on #5458
(whose followers had warmed their signature cache by building), so the
follower prewarm did not recover it at this load. Nomination round 1 (2
s timeout) times out on almost every ledger.
- Follow-up: done in the 2026-11-10 update above (leader build,
ballot-start prewarm, and the IPC size fix found by the next run).

---

## Original description

> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet; the run of this
PR's final head will be posted in the update above.

# Description

This PR compresses transaction-set responses in the Rust overlay and
shares the encoded bytes across peer sends. Every response uses the
mandatory zstd protocol; there is no legacy/raw fallback or optional
lazy encoding state. The leader hands the final XDR to Rust before
Core's final validation, so compression can overlap that validation.
Receivers retain the original compressed bytes, decompress and verify
the set, then reuse the retained encoding when forwarding it.

A separate commit, `0ed3e540`, removes private proposals on non-leader
nodes. Followers enter nomination with a lazy value supplier and follow
received proposals without pulling the mempool or constructing a set.
First-round leaders can prepare early; a promoted leader constructs a
value only when it needs one. The driver caches that value across
retries, and externalization stops nomination even when no local value
was built. Tests cover timeout promotion, current close times, following
another leader, unavailable values, and reentrant nomination stop.

This PR builds on #5458. The
comparison below evaluates these revisions against the **original
compressed implementation of this PR**, using the same 10-tier-1,
30-validator fixture: 3,000 offered TPS, 6,000 transactions/block,
2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and
900-second measurement, with simulated apply.

## 3,000 offered TPS

| Metric | Original compression | Updated leader-only path |
|---|---:|---:|
| Median block interval | 2.018 s | 2.120 s |
| Worst-node full-window p99 block interval | 3.834 s | 3.836 s |
| Tx-set fetch p99 | 0.617 s | 0.601 s |
| Completed TPS | 2,926.67 | 2,806.67 |
| Harness verdict | PASS | FAIL (ob-0 only) |

Median intervals and completed throughput use the canonical node;
worst-node p99 uses the fixed measurement window. The harness uses a
different histogram/window: `ob-0` recorded 4.989 s against its 4-second
limit, while the other 29 nodes and inclusion passed. Completed
throughput fell 4.1% and median block interval rose 5.1%; this run does
**not** establish that the revised path matches or improves the original
compression baseline. These are single cloud runs with normal host
placement.

Leader-only construction is verified: 886 builds/compressions across 421
closed ledgers, all correlated with asserted leadership, with no
follower build or received-set recompression detected. Multiple
nomination leaders can build in a ledger. Compared with the intermediate
eager implementation, local compressions fell from 13,019 to 893 in
their respective 900-second windows, roughly a 93% reduction per ledger.
Payload savings remain 80.0%. The sampled received-set validation timer
rose from about 158 ms to 205 ms versus that intermediate build; the
precise cause is not isolated by this run.

# Checklist

- [x] Reviewed the contributing document.
- [x] Kept leader-only proposal construction in its own commit on this
PR's branch.
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Local build, Jenkins package/image build, and exact-head Clang/GCC
CI passed.
- [x] Rust suite: 194 passed, 3 ignored; the leader-only follow-up
leaves those validated Rust sources unchanged.
- [x] Core validation: 41 focused SCP/nomination/tx-set cases, 17
overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent
cases were rerun after configuring the local database fixture.
- [x] Performance comparison, runtime identity, resource checks and
sealed-artifact audit completed.
- [ ] Performance matches or improves the original compressed baseline.
#5495)

**Prototype, not for merge.** This deletes the SCP nomination phase and
replaces it with a leader schedule that every node computes locally.
There is no flag, no protocol gate and no compatibility with nominating
nodes: it exists to measure what nomination costs on the overlay-v2
stack and what limits block time once it is gone.

Stacked on #5459 (`pr/txset-compression`, `ebdd73635`). The change is
one commit, `daa965f4c`. A second commit, `577f3d4f0`, only replaces the
branch's `README.md` with a description of the experiment, the results
below and the full test setup (topology, node configuration, network
settings, load, pass criteria); everything here was built, tested and
measured on `daa965f4c`.

## Result in one paragraph

On perfnet (30 validators, 4 AWS regions, c6id.2xlarge, 3,000 SAC TPS)
the prototype passes at a 2 s target with every correctness check green
and no wasted round, and takes about 0.19 s out of the time from trigger
to externalize (1.50 s to 1.31 s). Block time at 2 s is unchanged,
because the trigger sets the cadence when the network keeps up. At a 1 s
target it fails: the network settles at 1.33 s per ledger. The remaining
time is mostly cross-region round trips for the tx set and the ballot,
not nomination and not CPU. One run per setting.

## Design

- **Schedule.** For slot N, sort the transitive quorum by
`SHA256(LCL(N-1).hash || nodeID)`; `leader(N, r)` is entry `(r-1) mod
n`. Uniform, no weights, one sort per slot
(`herder/LeaderSchedule.{h,cpp}`, cached in `HerderSCPDriver`).
- **Rounds.** A round is a ballot counter. The round's leader builds its
value and bumps straight to `(r, value)`; the others adopt the leader's
PREPARE/CONFIRM at counter r once it passes proposal validation. A
timeout (the ballot timeout settings) moves a node to round r+1.
- **Late proposals.** A node that has not voted yet still votes for an
earlier round leader's ballot, at that ballot's own counter, when it
shows up late (up to 8 rounds back). Nomination had this property;
without it a slow first proposal is wasted and the next leader builds a
competing set.
- **Leader timer extension.** If the timer fires while the Herder
already holds the leader's statement (its tx set is being fetched or it
is queued), the timer is re-armed once.
- **Unchanged:** the ballot protocol's accept, confirm and commit rules,
`mValueOverride` precedence, the v-blocking join paths, value
validation, apply, ledger state and meta.
- NOMINATE envelopes are dropped.

## Why it should be safe (and where it is not a real design)

- Safety rests on the ballot protocol, which is safe for any choice of
ballot values. The change only alters which value a node picks for a new
ballot and when it bumps. Two independent reviews found no path to two
honest nodes externalizing different values, no vote the ballot protocol
forbids, and no new reachable assert or unbounded loop.
- Upgrade governance: adoption validates in the strict (former
nomination) mode, so a value needs a quorum of adopters that armed the
same upgrades. There is no upgrade stripping or merging.
- Known gaps, by design of a prototype:
- A follower follows a leader's empty-tx-set ballot under ballot
validation, so its upgrades are only checked for apply validity. With
parallel tx-set download on, one round leader could get an empty ledger
adopted network-wide.
- The schedule's node set is whatever transitive quorum each node
currently knows. It assumes a symmetric, static quorum (as on perfnet).
A real design needs a node set that is part of consensus state.
- Leader grinding through the header hash, and fairness by org or
quality, are out of scope.

## What was deleted

- `scp/NominationProtocol.{h,cpp}`, `scp/test/SCPUnitTests.cpp`, and the
nomination API of `SCP`, `Slot` and `SCPDriver` (`nominate`,
`combineCandidates`, `computeHashNode`, `getNodeWeight`, the nomination
timers and emit delay, and related helpers).
- `HerderSCPDriver::combineCandidates` / `compareTxSets` /
`getNodeWeight`, the metrics `scp.timing.nominated`,
`scp.timeout.nominate`, `scp.nomination.combinecandidates`, and
`ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`.
- Kept but unused, so existing configs still parse:
`FORCE_OLD_STYLE_LEADER_ELECTION`, `VALIDATOR_WEIGHT_CONFIG`,
`nominationtimeoutlimit` and the nomination timeout network settings.
- New metrics: `scp.timeout.leader` (leader timeouts per slot; an
extension counts) and `scp.leader.round` (the highest round a node
entered for the slot, so an upper bound on the round that externalized).

Net: 23 non-test files, +886 / -2,072.

## Testing (all on `daa965f4c`)

- New: `[scp][leaderschedule]` unit cases and a randomized multi-node
model in `scp/test/SCPTests.cpp`, `herder/test/LeaderScheduleTests.cpp`,
and network simulations in `herder/test/LeaderRoundsTests.cpp`
(including failover when the dead node leads round 1). The Herder,
quorum-tracker, PendingEnvelopes and overlay-IPC tests are ported;
nomination statistics, `combineCandidates` and weight tests are deleted
with the code.
- Full suite: PASS. 598 test cases / 2,379,273 assertions, libsodium 77,
xdrpp 7, check-rust and the soroban host tests (103 `test result: ok`, 0
failed).
- Full suite `--all-versions`: PASS. 598 test cases / 8,154,187
assertions. (#5459's head runs 605; the difference is the deleted
nomination tests.)
- Tag run
`[herder],[scp],[txset],[overlay-ipc],[overlay],[ipc],[upgrades],[simulation]`
(146 tests): passes except two `[acceptance]` tests, "hierarchical
topology scales 1 to 3" and "core nodes with outer nodes", which time
out the same way on `overlay-v2-shared` without this change and are not
in the full suite.
- `clang-format` 20.1.8: the 28 added or modified C++ files are clean.
- **Not passing:** a `--disable-tests` build stops in
`herder/HerderImpl.cpp`, on test headers reached through `#include
"simulation/LoadGenerator.h"` (line 37). That include is the same on
#5459's head and this commit does not touch those headers; the base was
not built with that flag to confirm, and because the build stops there
it is not known whether this commit's own files compile without tests.
- **Not run:** sanitizers; the tx-meta and LedgerCloseMeta determinism
checks (the change does not touch apply).
- **Test gaps** (from review): no SCP-level test for following an
empty-tx-set leader ballot, for the `UINT32_MAX` counter case, for the
`mValueOverride` restore, for the extension flag resetting across
rounds, or for the 8-round bound. The randomized model uses one shared
schedule and never exercises the pending-proposal path. Two hidden
trigger-timer tests that depended on the nomination emit delay were
deleted without a port.

## Perfnet

30 validators in 4 regions, 10 tier-1 orgs, c6id.2xlarge, SAC payments,
150 ms simulated apply per ledger, 900 s measured, all SCP timeouts 2 s
(perfnet's default). Baseline is #5459's head `ebdd73635` on the same
fixture and instance type, run the same day. One run per setting.

### 2 s target, 3,000 TPS: PASS

All four windows on target; inclusion (2,880,000 txs on every node),
error counters, in sync, no fork and still closing all passed.

| | #5459 `ebdd73635` | This PR `daa965f4c` |
|---|---:|---:|
| avg p75 per window | 2,112 / 2,099 / 2,101 / 2,099 ms | 2,109 / 2,066
/ 2,082 / 2,094 ms |
| avg p99 per window | 2,334 / 2,391 / 2,324 / 2,354 ms | 2,324 / 2,286
/ 2,296 / 2,303 ms |
| Ledgers per 5 min | 147-148 | 149-150 |
| Trigger to first ballot (mean) | 1.21 s | 0.88 s |
| Ballot to externalize (mean) | 0.29 s | 0.43 s |
| **Trigger to externalize** | **1.50 s** | **1.31 s** |
| Leader build / tx-set fetch / validation (mean) | 0.28 / 0.24 / 0.20 s
| 0.27 / 0.26 / 0.19 s |
| Ledger close | 0.164 s | 0.162 s |
| Nomination timeouts (network, one window) | 7 | n/a |
| Leader timeouts, prepare timeouts, rounds above 1, empty tx sets | n/a
| 0, 0, none, 0 |

- The time before the first ballot that is not build, fetch or
validation fell from about 0.48 s to about 0.15 s.
- Part of it came back in the ballot phase: nodes now start their ballot
when each has validated the leader's set, so the early ones wait for a
quorum of the others.
- `scp.timing.trigger-to-ballot` records 0 when a follower votes before
its own trigger, so it is not strictly comparable. Triggers were on time
in this run.

### 1 s target, 3,000 TPS: FAIL

Window 1 missed on all 30 nodes: avg p75 1,386 ms (limit 1,200), avg p99
1,599 ms. 225 ledgers per 5 minutes (1.33 s each), every tx set at the
3,750 cap, every trigger already due, mempool backed up. No leader or
prepare timeouts, every ledger decided in round 1, no empty tx sets;
error counters, in sync, no fork and still closing passed. No baseline
was run at this setting.

| Stage (5-min means) | 2 s target (6,000 txs) | 1 s target (3,750 txs)
|
|---|---:|---:|
| Leader build | 0.27 s | 0.25 s |
| Trigger to first ballot | 0.88 s | 0.70 s |
| Ballot to externalize | 0.43 s | 0.41 s |
| Ledger close (simulated apply 150 / 187 ms) | 0.16 s | 0.20 s |
| Sum | 1.47 s | 1.30 s |

## Where the time goes now

From one node's logs at the 2 s setting (456 ledgers as follower, 23 as
leader) and live logs from four regions at 1 s:

1. **Tx-set dissemination is 2.5 round trips.** A follower receives the
proposal, requests the set from the proposer, and the response (0.77 MB
zstd for 6,000 txs) takes two flights: fetch measured 27 ms in-region,
150 ms from us-east-1, 301 ms from eu-central-1 and 362 ms from
ap-southeast-1, as seen from us-west-2. Nothing is pushed, and full
transactions are sent although followers hold nearly all of them.
2. **Followers validate before voting, then the ballot needs four
one-way trips** (0.29 s on the baseline, where all nodes start
together).
3. **About 70 ms pass between "Built TX set" and the proposal reaching a
same-region peer.** `emitEnvelope` calls `persistSCPState` before
`broadcast`, which serializes and base64-encodes the tx set and writes
it to the database on the main thread. That this write is most of the 70
ms is inferred, not timed.
4. **The leader's build is on the critical path** (0.25-0.33 s). The
early build is discarded unless the set was capacity limited, and it
does not run at all when the trigger is already due.
5. Validation is 0.12-0.19 s with signatures already cached.

`EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` does not help here as it stands:
followers would confirm prepared before holding the set, and the ballot
protocol does not retry the commit vote at the same counter, so each
slot would wait for the ballot timeout.

## Open review findings (not fixed in this commit)

- The leader timer extension triggers on any pending envelope from the
round's leader for the slot, not only its proposal for that round. It
slows failover when a tx set is unfetchable; it does not occur on a
healthy network.
- A late proposal only helps nodes without a ballot, never the next
leader, so a proposal that arrives just after a timeout costs a second
fetch and validation.
- A leader re-proposes a value that is only structurally valid;
`bumpState` may replace it with the empty-tx-set value.
- Nodes that adopted a ballot but have not heard a quorum run no timer,
as in the plain ballot protocol, so recovery after a timeout depends on
the next leader being alive.
- Smaller items: a redundant `progressRound()` before `advanceSlot`,
repeated proposal validation while a leader has no ballot, leftover
diagnostics in one Herder test, stale "nominate" wording in a few
comments.

## Next steps

Not started. In order of expected effect on block time: push the tx set
with the proposal in one flight; take SCP state persistence off the
broadcast path (after timing it); let the commit vote resume when a late
tx set validates, so the prepare rounds can overlap with fetch and
validation; build the leader's set before the previous ledger closes,
which needs the leader to be known earlier than the previous header
hash. Then repeat the 1 s run, with a same-setting baseline and more
than one run per build.
@SirTyson
SirTyson merged commit cc51aba into overlay-v2-shared Nov 10, 2026
@SirTyson
SirTyson deleted the pr/merge-ms-close-time-base branch November 10, 2026 22:10
SirTyson added a commit that referenced this pull request Nov 11, 2026
**Prototype, not for merge as is.** Like #5495 it has no flag and no
protocol gate. It moves tx-set dissemination from a request model to a
push model: the round's leader sends the tx set it is about to propose
to its peers as soon as the set is built, instead of waiting for each
follower to ask for it.

Stacked on #5495. The branch sits on `b524ef962`, the head #5495 had
when it was merged; `overlay-v2-shared` (`cc51aba48`, #5496) has the
same tree, so this PR shows only its own four commits:

- `0f9255d47` Start QUIC connections with an 8 MiB congestion window
- `8f7768be7` Push the leader's tx set to peers when it is built
(prototype)
- `d0fe62652` Document the remaining gaps of the tx-set push (docs and
one `.gitattributes` line, no code)
- `2f9baef24` Describe the tx-set push experiment and its perfnet
results in the README (the branch `README.md` now covers this
experiment, with #5495's values as "before"; README only)

Everything below was built, tested and measured on `8f7768be7`.

## Result in one paragraph

On perfnet (30 validators, 4 AWS regions, c6id.2xlarge, 3,000 SAC TPS)
at the 1 s target that #5495 failed, all four windows are on target on
every node: avg p75 1,091 / 1,096 / 1,109 / 1,115 ms against 1,386 ms,
and 1.05 s per ledger against 1.33 s. A follower's Core gets the
leader's tx set 16 ms after it asks for it, where it took 217 ms. One
caveat: every node's ledgers hold 60 transactions more than the
2,880,000 that were offered. The counters point to about 60 transactions
being included in two ledgers, which only the fixture's overlay-only
mode allows; see "The inclusion check" below. The run was judged with
the exact check and reported FAIL for that check alone. It is accepted
as an artifact of that mode, and the check is being relaxed to allow
such duplicates (stellar-experimental/performance-testing#4). One run,
against one run of the base from the same day.

## Design

- **Trigger.** A new Core-to-overlay IPC message `PUSH_TX_SET
[hash][slot]`. `HerderImpl` sends it right after `CACHE_TX_SET` from
inside the build when the node leads the current round, after the build
for a first-round leader's capacity-limited prepared set (the one case
where the prepared set is proposed unchanged), and again when a prepared
set is taken at the trigger. Later-round leaders go through the same
build hook. `CACHE_TX_SET` alone does not push: it also carries sets
that are never proposed.
- **Sender.** The overlay writes the cached, already compressed set to
the tx-set stream of every connected peer that has not been sent it. One
push in flight per peer; a peer that does not read is not sent further
sets and requests them.
- **Wire.** The pushed frame is the existing tx-set frame, sent without
a request. No new wire message, no claimed hash, no slot.
- **Receiver.** The overlay decompresses, strict-parses and hashes the
frame as it does a response and caches it under the hash it computed.
Core is not told. When an SCP statement names that hash, Core's
`RequestTxSet` is answered from the cache at once.
- **Push against request.** If the proposal reaches a follower before
the push is complete, the follower requests the set as before. The
leader does not answer the first request from a peer for a set that is
still being written to it or was written less than 500 ms ago: the bytes
are already on the same ordered stream. Every other request is answered.
- **Fallback.** The request path is unchanged. A node that is not
connected to the leader, connected later, or was busy with an earlier
push has no cached set and requests it; Core's 1 s retry to another peer
is still behind that. The push is not relayed: a node that is not
connected to the leader asks the peer that relayed the proposal, which
holds the pushed copy.
- **One flight.** quinn paces one congestion window over 0.8 round trips
and does not grow the window while the sender is application limited.
Measured locally (two overlay nodes over a loopback with a netem delay,
150 ms round trip, 640 KB frame, one run per setting): with quinn's
default window the frame arrives 229-253 ms after the send on a warm
connection (75 ms of that is the one-way delay) and 965 ms on a cold
one; with an initial window of 8 MiB it arrives after 94 ms. Without
that, a push would lose the race against the proposal (sent about 70 ms
later) on every cross-region link. Hence the first commit. The window
applies to every stream of a connection; it is a prototype setting.
- **Considered and not done:** a compact wire form (transaction hashes,
rebuilt from the follower's mempool; a follower that lacks one
transaction needs another round trip, and at 3,000 TPS the newest
transactions are still flooding); an announcement before the body (a new
way to stall a follower); relaying (in a full mesh every set would
arrive 29 times without an inventory exchange).

## Why it should be safe

- No change to tx-set validation, the voting rules, the ballot protocol
or apply. The C++ diff adds IPC sends on the main thread and metrics.
- A follower only uses a set whose hash is in a signed SCP statement:
the overlay files a pushed set under the hash of its own content, Core
asks only for hashes named by statements it is processing, accepts a set
only while it is fetching that hash, and re-checks the content hash. A
pushed set that does not match the proposal is never looked up; the
proposed set is requested. This is tested at three levels (below).
- What a leader can do with the push: push nothing or late (followers
request, as before); push a set it does not propose (it sits in a
bounded cache and is never used); push to some peers only (the others
request); push garbage or a truncated frame (dropped at decode; a
stalled frame blocks only that peer's own tx-set stream, as a stalled
response does today); push many or large sets (bounded, next point).
- Unrequested tx-set frames were accepted without limits before this
change. They now have a per-peer token bucket applied before
decompression (burst 8, 4 per second; answers to the node's own requests
do not use it) and limits on cached sets from peers that Core has not
asked for (per peer 4 pushed and 16 requested, 128 MiB in total, largest
holder evicted first).
- Known gaps, stated in `docs/rust-overlay/txset-fetching.md`:
- A follower that drops a pushed set on arrival (its own limits) asks
inside the 500 ms in which the leader skips the request, so it is only
served by Core's 1 s retry. Honest leaders do not trigger those limits.
- The overlay prefetches any hash named in an SCP envelope before Core
checks the signature, so a peer can buy one frame over the rate per
envelope; a reconnect resets a peer's allowance; many colluding peers
can even out the shared byte budget; the 100-entry cache itself is first
in, first out over all sets.
- Hardening of the overlay's XDR parsing limits is not addressed here;
it is the same on the base for every message type.

## Testing (all on `8f7768be7`)

- New, C++ (`herder/test/TxSetPushTests.cpp`, 4 network simulations with
real overlay processes): the leader's push is used (no follower waits
for the retry); consensus through the request path when no push is sent;
a network where half the leaders push; every leader pushes one set and
proposes another (the pushed set is never used, every ledger
externalizes the proposed set). One IPC section in
`overlay/test/IPCTests.cpp`.
- New, Rust: 16 unit tests for the push bookkeeping
(`flood/txset_push.rs`), 7 two- and three-node tests
(`libp2p_overlay/txset_push_tests.rs`: push reaches every peer once, a
crossing request is skipped once, a late request is answered, a peer
that missed the push is served, the record is cleared on a closed
connection, the rate limit, a malformed frame), 7 App-level tests
(`main.rs`: a pushed set reaches Core only on request, a set with
another hash does not answer a request, the limits), 4 end-to-end tests
with two overlay binaries (`tests/e2e_binary.rs`: push before the
request, no push, pushed set with another hash, a dropped push recovered
by a request).
- Full suite: PASS. 602 test cases / 2,383,553 assertions in 121
batches, libsodium 77, xdrpp 7, check-rust and the soroban host tests
(103 `test result: ok`, 0 failed).
- Full suite `--all-versions`: PASS. 602 test cases / 8,155,400
assertions.
- Tag run
`[herder],[scp],[txset],[overlay-ipc],[overlay],[ipc],[upgrades],[simulation]`
(150 tests, on the PR head `d0fe62652`): passes except three
`[acceptance]` tests that are not in the full suite and fail without
this change: "hierarchical topology scales 1 to 3" and "core nodes with
outer nodes" (as reported in #5495), and "SCP Driver / validateValue
txSet cached" (`HerderTests.cpp:2964`, `0 == 9`), which fails on #5495's
source too whenever the test runs with `--base-instance` 50 or 400, i.e.
in partition 2 or 9 of a partitioned run, and passes with 0 or 450.
- `clang-format` 20.1.8 on the changed C++ files and `cargo fmt --check`
on the overlay crate: clean.
- **Not passing:** a `--disable-tests` build stops in
`herder/HerderImpl.cpp` on the same test headers as on #5495. The three
other non-test objects this change touches compile without
`BUILD_TESTS`.
- **Not run:** sanitizers; the tx-meta and LedgerCloseMeta determinism
checks (the change does not touch apply).
- **Test gaps:** no test with a peer that really stops reading its
tx-set stream; none that drives a failed push write end to end; the C++
simulations run with empty mempools, so the prepared-set push and
realistic set sizes are only exercised on perfnet; with three or more
nodes, the path "request skipped, then Core's retry to another peer" is
not tested; the network tests use fixed ports and 1 s windows.
- Three independent review passes of the diff; the last one found no
correctness, determinism or safety problem. Its minor findings are under
"Open review findings".

## Perfnet

30 validators in 4 regions, 10 tier-1 orgs, c6id.2xlarge, SAC payments,
150 ms simulated apply per ledger (50 us per op at this setting), 900 s
measured, all SCP timeouts 2 s. Target 1,000 ms, 3,000 TPS. Base:
#5495's commit `daa965f4c` (before its rebases, which changed
formatting, the README and a CI file), build #3674, the same fixture and
instance type on the same day; its load was stopped after the first
window, as block_time_check does on a miss. This PR: build #3675 of
`8f7768be7`. **One run each.**

| | #5495 `daa965f4c` | This PR `8f7768be7` |
|---|---:|---:|
| Windows | 1 of 1 missed (all 30 nodes) | **4 of 4 on target (all 30
nodes)** |
| avg p75 per window | 1,386 ms | 1,091 / 1,096 / 1,109 / 1,115 ms |
| avg p99 per window | 1,599 ms | 1,227 / 1,243 / 1,253 / 1,290 ms |
| Ledgers per 5 min | 225-226 (1.33 s each) | 285-286 / 283-284 /
281-282 (1.05-1.07 s) |
| Txs per ledger | 3,750 (the cap) | 3,155 (below the cap) |
| Tx-set fetch `scp.fetch.txset`: mean (p75 / p99) | 0.217 s (0.33 /
n/r) | **0.016 s (0.017 / 0.024)** |
| Sets a follower had to await from the network | every set (0.198 s
mean) | none in window 1 |
| Trigger to first ballot: mean (p75 / p99) | 0.70 s | 0.42 s (0.47 /
0.57) |
| Ballot to externalize: mean (p75 / p99) | 0.41 s (0.53 / 0.76) | 0.33
s (0.34 / 0.50) |
| Leader build / follower validation / ledger close (mean) | 0.25 / 0.12
/ 0.20 s | 0.15 / 0.11 / 0.16 s |
| **Trigger to close (sum)** | **1.30 s** | **0.91 s** |
| Leader timeouts, prepare timeouts, rounds above 1, empty tx sets | 0,
0, none, 0 | 0, 0, none, 0 |
| Error counters, in sync, no fork, still closing | passed | passed |
| Inclusion | skipped (early stop) | **failed: 2,880,060 on every node,
2,880,000 offered** |
| Verdict as judged | FAIL (block time) | FAIL (inclusion only, with the
exact check; passes the relaxed one) |

Stage values are 5-minute means at the end of window 1, median over the
30 nodes.

- Push counters over the whole load (network totals): 28,507 pushes
sent, 28,456 received, 28,181 already in place when Core asked, 64
requests that crossed a push (not answered twice), 0 dropped, 0 evicted,
0 retries, 0 write errors.
- From live logs of one node per region: a follower's Core has the set a
median 126 ms after the leader logs "Built TX set" (76-169 ms by region
pair). On #5495 that took about 480 ms on a 147 ms link. A follower
externalizes 0.45-0.55 s after the leader's set is built (0.97 s on
#5495 at the 2 s setting).
- The two runs are in different regimes. #5495 could not close a ledger
in under 1.33 s, so every set was at the cap and the mempool backed up.
Here the cycle fits into about 0.91 s and ledgers hold what was offered.
That also makes stages this change does not touch cheaper (the leader
pulls 3,200 candidates instead of 7,700; sets are smaller).
- Block time is about 1.05 s although the stages sum to 0.91 s; not
looked into.
- A slow drift over the 16 minutes: avg p75 1,091 to 1,115 ms, trigger
to first ballot 0.42 to 0.45 s. Cause not identified.
- **Not separable in one run:** the push and the 8 MiB window. The
window also applies to SCP and transaction flooding.

### The inclusion check: 60 transactions too many

block_time_check compares the transactions in every node's ledgers with
the number offered. All 30 nodes hold exactly 60 more (same value
everywhere, no fork). The generators submitted exactly 2,880,000 (96,000
each, none rejected) and every node pulled every foreign transaction
exactly once, so about 60 transactions (0.002%) were included in two
ledgers. #5495's completed run at 2 s had the exact count; its 1 s run
stopped before this check.

How that can happen here (the code path is the stack's documented
behaviour; that it explains these 60 is inferred from the counters, the
logs carry no transaction hashes):

1. The fixture runs in overlay-only mode. Apply is skipped, sequence
numbers do not advance, and the one-transaction-per-source-account rule
is off, so nothing in Core rejects a transaction that an earlier ledger
already held. With real apply such a copy fails validation when a leader
builds its set.
2. The overlay's mempool removes a ledger's transactions at
externalization and deliberately keeps no memory of them: the test
`externalization_removes_queued_admission_without_remembering_hashes`
says "Truly later admissions are deliberately allowed; no finalized hash
history remains. Ledger-state validity is still checked by Core." A copy
that reaches a node for the first time after the removal is admitted as
new and included when that node next leads.
3. A leader that is not backed up takes everything in its mempool,
including transactions that are still flooding to the far nodes (an
inventory batch plus one and a half round trips: up to about 0.5 s).
4. With the push, a follower externalizes 0.45-0.55 s after the leader's
set is built, which is now as fast as the slowest flooding. On #5495 it
was about a second, so the two never met.

So the design relies on Core's ledger-state validation, which this
fixture switches off, and the push's speed-up is what makes the two
meet. This PR does not change that, and the overlay keeps its behaviour
(remembering finalized hashes there would change production behaviour
for a test-mode artifact). The duplicates are accepted for this fixture:
block_time_check's inclusion check is relaxed to allow up to 1 in 10,000
transactions more than offered, with the caveat written next to it
(stellar-experimental/performance-testing#4). Fewer than offered, more
than that, and nodes that disagree still fail. No run has been made with
the relaxed check; applied to this run's recorded counts it passes with
60 over.

## Open review findings (not fixed in `8f7768be7`)

- The in-flight count of a push is only released when the write task
ends. A panic in that task, or a stream open that never completes while
the peer is disconnected, would leave the peer marked busy (it would
then always request).
- A push that is still being written when a connection to that peer
closes is no longer recorded: a crossing request gets a second copy, and
a request skipped before the close is not answered if the write then
fails (Core's retry covers it).
- Per-peer unclaimed entries can add up to 20 while the cache holds 100
sets of any kind.
- A set is accounted to the peer that delivered it first, so a peer that
delivers the leader's set first and then pushes four more evicts it.
- The token bucket's parameters assume at most two sets per led slot at
500 ms or more per slot.

## Next steps

Not started: a run of the base with only the congestion window, and of
the push on quinn's default window, to separate the two effects; repeats
(n = 1 here); a longer run to see whether the drift continues; then 800
ms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant