Repository navigation
Prototype: send transactions to the upcoming round leaders only, with an admission check - #5501
Merged
SirTyson merged 9 commits intoNov 12, 2026
Conversation
…allot, one build before the trigger (prototype)
Takes the round leader's transaction set build off the consensus critical
path. Prototype on overlay-v2-shared, not protocol gated (like the leader
schedule it changes).
Schedule
- The node whose value closed a ledger leads the next one, for up to
LEADER_TERM_LEDGERS (5) ledgers; terms end on ledger numbers divisible by
5, so the schedule of a slot is still a function of the previous ledger's
header and the node set.
- The order of a slot is seeded with the hash of the ledger before the last
closed one (the header's previousLedgerHash), so it is known one ledger
earlier.
- A leader loses its term at once, and goes to the end of the next order,
when its ledger closed with a consensus-level empty-tx-set value or
applied no transaction (a set found invalid at apply). A leader whose
round times out loses it too: the leader of a later round puts forward
the value it already votes for under its own signature
(SCPDriver::reproposeValue), and the node that signed the value that
closes a ledger is the next leader.
Pre-validation
- LedgerManager::startSpeculativeApply can hand back the state a
speculative execution left (ExecutedLedgerState: the previous ledger's
snapshot plus the account entries the execution changed, read from the
parked LedgerTxn without sealing it).
- A node that expects to lead the next ledger in round 1 or 2 validates
the mempool's top candidates against that state on background threads
while the executed ledger is still in its ballot. The results are used
only if the ledger closes with the very value that was executed, and
never for transactions whose validity depends on the close time.
- All builds on one last closed ledger share a validation result cache
(TxValidationCache), seeded by the pre-validation: a build only validates
what it has not seen.
One build per ledger
- The first-round leader builds once, a quarter of the target close time
(at most 250 ms) before the trigger, pushes the set to its peers as soon
as its XDR exists, and proposes it unchanged at the trigger. It used to
build right after the close and again at the trigger.
New metrics: herder.prevalidate.{started,used,discarded,incomplete,txs,
pull,duration}, herder.txset.validation-cache-{hit,miss},
ledger.apply.capture-state.
…next order Test-only. The helper that makes the local node the first leader moves to the start of a term when test setup left the node with a forfeited term, and the healthy-network test expects the previous proposer of an empty ledger at the end of the order.
…at a term boundary Second review's findings. - A later round's leader that has accepted the value it votes for as prepared keeps it as it is: other nodes may have confirmed it, and a replacement under another signature could only split the round. The replacement is only offered before that (SCPDriver::reproposeValue), for a set the node holds, and counted (scp.leader.resigned). - A ledger that applied nothing sends its proposer to the end of the next order also when the next ledger starts a term. - The pre-validation job reports itself complete only if no task was stopped, and its candidate decode cannot throw out of call_once.
The first-round leader's lead time before the trigger was a quarter of the target close time. At 48 validators, 800 ms and 5,000 TPS the build took 180 ms of those 200 ms, so the set left 20 ms before the trigger and reached the followers about 100 ms after the proposal. The lead time now also covers the node's last build (with a quarter to spare) plus 100 ms for the push, up to half of the target close time.
…'s key sorts Test-only. The test configuration's node key is derived from the instance number and the day's seed. For a key that sorts behind nearly every other one the helpers could not find a peer that comes after it within their tries (PendingEnvelopes tests: 'could not pick a peer that does not lead round 1'; seen with seed 20769 at base instance 1450, and on the base branch with seed 20768 at 850). createAppWithFollowerPeer now also varies the local key; selectLocalFirstLeader tries more absent peers.
SirTyson
force-pushed
the
perf/tx-flood-to-leaders
branch
from
November 12, 2026 07:37
664a35f to
6b3c7ae
Compare
…or this node's last build At 90 validators, 800 ms and 10,000 TPS the first ledger of every leader term came late (929 ms against 768-814 ms for the term's other four): a node that starts a term has not built since before the load rose, so a lead time derived from its last build (about 1 ms, an empty set) fell back to the 200 ms minimum, the 190-220 ms build ended at the trigger and the pushed set reached the followers 0.17 s after the proposal. One slow term then tipped the network into a backlog it did not leave. The expected build is now the longer of the node's last build and what a set of the last closed ledger's size costs at the node's measured cost per transaction (40 us until it has built a set of 100 transactions or more).
SirTyson
force-pushed
the
perf/tx-flood-to-leaders
branch
from
November 12, 2026 09:44
6b3c7ae to
8cee915
Compare
…on check (prototype) A node that accepts a transaction from a client checks it against its last closed ledger, keeps it, and has its overlay send the body straight to the leaders that will build the next ledgers instead of advertising it to every peer. Overlays prove which validator they belong to with a binding signed by the node key. A target that is not a directly connected, identified peer makes the node fall back to the advert flood. The targets (HerderImpl::computeTxFloodTargets) are the leaders of the node's round of the next slot and of the round after it. With leader terms the node that closes a ledger leads the next one too; one slot before a new term starts, that term's expected first leader is a target as well, so that it holds the transactions when it validates and builds its first set. When the targets change, a kept transaction is sent to a new target only while it is fresh or once it missed three ledgers: anything else is most likely in the set in consensus, and a copy sent by a node that runs late would reach the new target after that set externalized there. Options for measurements: TX_FLOOD_TARGET_LEADERS (0 = every validator), TX_FLOOD_LEGACY_ADVERTS, BATCH_EXECUTOR_TASKS. New meter overlay.flood.tx-late-copy: transactions admitted although one of the last few externalized ledgers already has them.
SirTyson
force-pushed
the
perf/tx-flood-to-leaders
branch
from
November 12, 2026 10:08
8cee915 to
6ab837c
Compare
added 2 commits
November 12, 2026 16:47
…only at 800 ms on 16-core instances; the goal fixture; where a ledger's time goes there
SirTyson
marked this pull request as ready for review
November 12, 2026 17:07
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, not for merge. Stacked on #5502 (leader terms, pre-validation during the previous ballot, one build before the trigger): this PR's own changes are the last two commits,
6ab837c27(the code) and two README commits (the results of the whole stack); the commits before them are #5502's (head8aa7f6bf0) and leave this diff when #5502 merges.A node that accepts a transaction from a client checks it against its last closed ledger, keeps it, and has its overlay send the body straight to the leaders of the current and the next round of the next ledger, instead of advertising it to every peer (INV/GETDATA). Since protocol 30 a follower votes without validating, so it does not need a transaction before the leader's set arrives.
Results in short
One run per configuration everywhere; perfnet
block_time_check, SAC payments, 150 ms simulated apply, overlay-only mode.On top of #5502, 16 tier-1 orgs (48 validators in 4 regions), 800 ms target, 8,000 TPS, c6id.8xlarge (16 cores):
On top of #5502, the goal fixture: 30 orgs (90 validators), 800 ms, 10,000 TPS, c6id.8xlarge (this PR's head):
Before the rebase (on
overlay-v2-sharedat26bea1150, without #5502), 16 orgs, 1 s target, 5,000 TPS:What changes
HerderImpl::computeTxFloodTargets(). With the leader schedule: the leaders of the node's current round and of the next one for the slot after the LCL (two nodes). With Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502's leader terms the node that closes a ledger leads the next one too; one slot before a new term starts (every fifth ledger), that term's expected first leader is a third target, so that it holds the transactions when it pre-validates and builds its first set. It is the first node of the order seeded with the LCL's hash, so it is known a full ledger ahead. "Everyone" (the old flood) when the node is not tracking or more than one ledger behind, when there is no schedule, and for a pre-protocol-20 LCL. It is re-evaluated at a ledger close, at a round change, at an externalize and at loss of sync, and sent to the overlay when it changes (IPCSET_TX_TARGETS). A changed leader schedule only has to change this function.flood/tx_router.rs, at most 100,000 transactions and 300 s) until it is externalized, removed by Core or expires. When the targets change, it goes to a newly added target only if it may still be wanted there: it is younger than 200 ms, or three ledgers externalized on this node without it since it was accepted, or it is older than 3 s (no ledger closes). Anything else is most likely in the set in consensus. Nothing is sent again to a peer that got it less than 60 s ago on the same connection; a failed send or a reconnected peer is served regardless. A node that received a transaction as a target keeps it in its mempool and does not relay it.HerderImpl::checkTxAdmission, inrecvTransaction):checkValidagainst the LCL, the check the leader's build runs on every candidate, and one pending transaction per source account per accepting node (TRY_AGAIN_LATER;DUPLICATEfor the same hash). On the baserecvTransactionchecked nothing. The per-account record goes when a set that uses up the sequence number closes, when the LCL shows the sequence number used some other way, or after 300 s.SHA256(networkID || "stellar-overlay-peer-id" || peerId)with the node key; the overlay shows[nodeID][signature]to each peer on a new stream (/stellar/node-id/1.0.0, acknowledged, retried), and the receiver checks it against the peer id QUIC authenticated. The node key stays in Core.TX_FLOOD_TARGET_LEADERS(default 2; 0 = every validator of the schedule, still direct and without adverts),TX_FLOOD_LEGACY_ADVERTS(the base's flood from the same binary),BATCH_EXECUTOR_TASKS(number of validation tasks; above the physical core count they run on hyper-thread siblings).overlay.flood.tx-direct-sent,.tx-direct-sent-bytes,.tx-direct-recv,.tx-local-flooded,.tx-late-copy(a transaction admitted although one of the last few externalized ledgers already has it),overlay.node-id.identified,.rejected,herder.tx-admission.check,.invalid,.account-busy.Why ledger outputs cannot change, and what does change
{"status":"ERROR"}without the result code.Perfnet
On top of #5502: 16 orgs, 800 ms, 8,000 TPS, c6id.8xlarge
block_time_check(perfnetbtc2-slop),--org-count 16 --block-time-ms 800 --soroban-tps 8000 --soroban-type sac --classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900 --instance-type c6id.8xlarge, 48 validators in four regions, no log saving. The #5502 column is that PR's own run of the same day at the same fixture. The stack column is build #3685 of6b3c7ae0e, which is this PR's head without the changed re-send rule, without the late-copy meter and without #5502's last commit.overlay.byte.readper nodeoverlay.message.readper nodeherder.txset.build; builds per ledgerherder.txset.validateoverlay-v2-sharedat this fixture on the same instances (reported in Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502): window 1 on target at 846 ms, window 2 missed, trigger to close 0.67 to 0.69 s.overlay.flood.tx-late-copy). The 16-org fixture was not rerun with the head.On top of #5502: the goal fixture, 30 orgs, 800 ms, 10,000 TPS, c6id.8xlarge
The same command with
--org-count 30 --soroban-tps 10000; 90 validators, 10,080 TPS offered. The stack column is build #3688 of this PR's head6ab837c27. The #5502 column is that PR's own run of its head8aa7f6bf0two hours earlier (some rows are from its first run at this fixture, marked G).herder.txset.validateoverlay.byte.readper nodeoverlay.flood.tx-late-copyNot measured: the 16-org fixture again with the head's re-send rule; the base
overlay-v2-sharedat the goal fixture; the 4-core instances with #5502 underneath;TX_FLOOD_TARGET_LEADERS=0; real apply; more than one run of anything.Before the rebase: 16 orgs, 1 s, 5,000 TPS (on
overlay-v2-sharedat26bea1150, without #5502)All runs:
block_time_check(perfnetbtc2-slop @ 6dd2064),--block-time-ms 1000 --soroban-tps 5000 --soroban-type sac --classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900, validators in us-east-1, us-west-2, eu-central-1 and ap-southeast-1, every validator generates load, overlay-only mode (Core does not apply; execution is a sleep). 5-minute means at the end of window 1, median node.0c9a02e56(tree of26bea1150)dc91c8964664a35fdcdc91c8964BATCH_EXECUTOR_TASKS=7)overlay.byte.readper nodeoverlay.message.readper nodeherder.txset.buildherder.txset.validateNot measured before the rebase: this change at 30 orgs; the base with 7 tasks; real apply (with real execution the check and the execution compete for the same cores).
Testing
On
6ab837c27398(this PR's head, on #5502 at8aa7f6bf0), devbox, 32 partitions:--all-versions: PASS (exit 0, 735 s).[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[simulation]~[.],[loadgen]~[.], which includes Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502's tests): only the three failures recorded for the base ("SCP Driver" validity-cache hits, "hierarchical topology scales 1 to 3", "core nodes with outer nodes").6b3c7ae0e) onemake check-rustrun failed 1 of 219 library tests and the output was not kept, so the test is not known. About twenty later runs of the same suite passed (apart from failures of tests I was changing at the time). Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502 does not touchoverlay/.New tests:
herder/test/TxFloodTargetsTests.cpp(6 cases):TX_FLOOD_TARGET_LEADERS = 0each transaction goes to the three other nodes exactly once.flood/tx_router.rs(20 unit tests): targets only and following the targets; a kept transaction follows a target change only while fresh, after three ledgers without it, or after 3 s (mutation-checked: with the rule disabled five tests fail); a removal by Core is not a ledger; flood without reachable targets; a directly received transaction joins a later flood and serves the body.libp2p_overlay/tx_direct_tests.rs(4 network tests on loopback, 1 unit test): the same over real connections, including a late copy being counted; forged, replayed and wrong-network identity claims are rejected, and claims are bounded per connection.main.rs(3): the IPC handlers.Existing tests changed:
overlay/.../dispatch_tests.rs(2): they fed a body without a demand and expected a relay; they now register the demand, and one asserts that an undemanded body is not relayed.main/test/CommandHandlerTests.cpp"transaction envelope bridge": V1 and fee-bump envelopes are refused again below protocol 13, and the too-early / too-late cases are back.Review: four passes of an independent review (two on the old base, one of the rebase, one of the re-send rule); no consensus or determinism finding in any. Open from the last two: the flush after an externalize plans against the outgoing targets (one stray body for transactions that waited three ledgers); a lost send re-delivers to every held-back target, not only the lost peer; after a leader failure the round-2 leader lacks transactions older than 200 ms until the ledger closes; the 200 ms bound is a constant, not derived from the block time; no test covers a watcher, a wrong term prediction, a lagging sender or the
main.rswiring of the ledger count.Not passing or not run:
--disable-testsbuild was not tried; Prototype: replace SCP nomination with a deterministic leader schedule #5495 records that the base does not build that way.Commits
Three commits on top of #5502: the code, and two for the README (which now describes and reports the whole stack: #5502's leader terms and pre-validation, and this PR). Builds that perfnet ran: #3688 of the head
6ab837c27(the goal-fixture run), #3685 of6b3c7ae0e(the 16-org run; that commit is no longer on the branch) and, before the rebase, #3680 ofdc91c8964and #3683 of664a35fdc.