Repository navigation
Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger - #5502
Merged
SirTyson merged 6 commits intoNov 12, 2026
Conversation
…allot, one build before the trigger (prototype)
Takes the round leader's transaction set build off the consensus critical
path. Prototype on overlay-v2-shared, not protocol gated (like the leader
schedule it changes).
Schedule
- The node whose value closed a ledger leads the next one, for up to
LEADER_TERM_LEDGERS (5) ledgers; terms end on ledger numbers divisible by
5, so the schedule of a slot is still a function of the previous ledger's
header and the node set.
- The order of a slot is seeded with the hash of the ledger before the last
closed one (the header's previousLedgerHash), so it is known one ledger
earlier.
- A leader loses its term at once, and goes to the end of the next order,
when its ledger closed with a consensus-level empty-tx-set value or
applied no transaction (a set found invalid at apply). A leader whose
round times out loses it too: the leader of a later round puts forward
the value it already votes for under its own signature
(SCPDriver::reproposeValue), and the node that signed the value that
closes a ledger is the next leader.
Pre-validation
- LedgerManager::startSpeculativeApply can hand back the state a
speculative execution left (ExecutedLedgerState: the previous ledger's
snapshot plus the account entries the execution changed, read from the
parked LedgerTxn without sealing it).
- A node that expects to lead the next ledger in round 1 or 2 validates
the mempool's top candidates against that state on background threads
while the executed ledger is still in its ballot. The results are used
only if the ledger closes with the very value that was executed, and
never for transactions whose validity depends on the close time.
- All builds on one last closed ledger share a validation result cache
(TxValidationCache), seeded by the pre-validation: a build only validates
what it has not seen.
One build per ledger
- The first-round leader builds once, a quarter of the target close time
(at most 250 ms) before the trigger, pushes the set to its peers as soon
as its XDR exists, and proposes it unchanged at the trigger. It used to
build right after the close and again at the trigger.
New metrics: herder.prevalidate.{started,used,discarded,incomplete,txs,
pull,duration}, herder.txset.validation-cache-{hit,miss},
ledger.apply.capture-state.
…next order Test-only. The helper that makes the local node the first leader moves to the start of a term when test setup left the node with a forfeited term, and the healthy-network test expects the previous proposer of an empty ledger at the end of the order.
…at a term boundary Second review's findings. - A later round's leader that has accepted the value it votes for as prepared keeps it as it is: other nodes may have confirmed it, and a replacement under another signature could only split the round. The replacement is only offered before that (SCPDriver::reproposeValue), for a set the node holds, and counted (scp.leader.resigned). - A ledger that applied nothing sends its proposer to the end of the next order also when the next ledger starts a term. - The pre-validation job reports itself complete only if no task was stopped, and its candidate decode cannot throw out of call_once.
The first-round leader's lead time before the trigger was a quarter of the target close time. At 48 validators, 800 ms and 5,000 TPS the build took 180 ms of those 200 ms, so the set left 20 ms before the trigger and reached the followers about 100 ms after the proposal. The lead time now also covers the node's last build (with a quarter to spare) plus 100 ms for the push, up to half of the target close time.
…'s key sorts Test-only. The test configuration's node key is derived from the instance number and the day's seed. For a key that sorts behind nearly every other one the helpers could not find a peer that comes after it within their tries (PendingEnvelopes tests: 'could not pick a peer that does not lead round 1'; seen with seed 20769 at base instance 1450, and on the base branch with seed 20768 at 850). createAppWithFollowerPeer now also varies the local key; selectLocalFirstLeader tries more absent peers.
…or this node's last build At 90 validators, 800 ms and 10,000 TPS the first ledger of every leader term came late (929 ms against 768-814 ms for the term's other four): a node that starts a term has not built since before the load rose, so a lead time derived from its last build (about 1 ms, an empty set) fell back to the 200 ms minimum, the 190-220 ms build ended at the trigger and the pushed set reached the followers 0.17 s after the proposal. One slow term then tipped the network into a backlog it did not leave. The expected build is now the longer of the node's last build and what a set of the last closed ledger's size costs at the node's measured cost per transaction (40 us until it has built a set of 100 transactions or more).
SirTyson
added a commit
that referenced
this pull request
Nov 12, 2026
… an admission check (#5501) **Prototype, not for merge. Stacked on #5502** (leader terms, pre-validation during the previous ballot, one build before the trigger): this PR's own changes are the last two commits, `6ab837c27` (the code) and two README commits (the results of the whole stack); the commits before them are #5502's (head `8aa7f6bf0`) and leave this diff when #5502 merges. A node that accepts a transaction from a client checks it against its last closed ledger, keeps it, and has its overlay send the body straight to the leaders of the current and the next round of the next ledger, instead of advertising it to every peer (INV/GETDATA). Since protocol 30 a follower votes without validating, so it does not need a transaction before the leader's set arrives. ## Results in short One run per configuration everywhere; perfnet `block_time_check`, SAC payments, 150 ms simulated apply, overlay-only mode. **On top of #5502, 16 tier-1 orgs (48 validators in 4 regions), 800 ms target, 8,000 TPS, c6id.8xlarge (16 cores):** - **Block time is the same as #5502 alone, on target in all four windows:** avg p75 804 to 805 ms against 805 to 807 ms; trigger to close 0.43 to 0.44 s against 0.44 to 0.45 s. Both runs sit at the fixture's floor, so this run shows neither a gain nor a loss in block time. - **Overlay traffic per node falls from 21.2 to 6.2 MB/s each way (-71%).** No adverts or demands are left; 3.3 transaction bodies are sent per offered transaction. - The leader's side is unchanged: one build per ledger before the trigger, the pre-validation covers 66.7% of the candidates (69% on #5502 alone), leader terms of exactly 5 ledgers, no timeout. - Followers verify signatures inline: the apply-time check takes 0.099 s against 0.081 s, still inside the ballot on 16 cores. - The verdict is FAIL on the inclusion count only, as for #5502 alone (the check is exact; see "Inclusion" under Perfnet). **On top of #5502, the goal fixture: 30 orgs (90 validators), 800 ms, 10,000 TPS, c6id.8xlarge (this PR's head):** - **FAIL: window 1 at avg p75 980 ms (limit 960), p99 1,441 ms (inside its 1,600 limit); the load was stopped after it.** #5502 alone at this fixture: 1,032 ms and p99 1,887 ms. - A ledger every 0.90 s against 1.02 s; 39% of the sets at the 10,000 cap against 93%; three of four leader builds before the trigger against none; trigger to close 0.77 s against 0.97 s. The gain is in trigger to first ballot (0.39 s against 0.58 s): the leader's mempool holds about one ledger of candidates instead of a 20,200 backlog. - **Overlay traffic per node: 15.7 MB/s against 43.** At 90 validators most of what is left is SCP (35,700 messages per second per node). - What still does not fit into 800 ms is the ballot (0.32 s), the set reaching followers 0.17 s after the proposal, and the builds that still run at the trigger. None of these is transaction dissemination. - **Where the remaining time goes** (twelve nodes' logs): the leader's build starts when the previous ledger closes and takes 0.29 s to the proposal; the set then needs 0.19 s to reach a follower (the leader sends 5.7 MB of XDR, compressed, to each of 89 peers); the ballot takes 0.34 to 0.36 s; the commit 0.07 s. Sum 0.89 s on the leader. The README has the table. **Before the rebase (on `overlay-v2-shared` at `26bea1150`, without #5502), 16 orgs, 1 s target, 5,000 TPS:** - Overlay traffic per node fell from 14.0 to 4.7 MB/s each way (-66%) in every run. - Whether the block time held depended on validation CPU: FAIL with the default 3 validation tasks on 4-core c6id.2xlarge (avg p75 1,367 ms; base 1,081 ms), PASS with 7 tasks on the same instances (1,030 ms, every check including an exact inclusion count), all windows on target on 16-core instances (1,019 to 1,021 ms; inclusion +250). - At 30 orgs the base failed 1 s / 5,000 TPS (1,298 ms); this change was not run there. ## What changes - **One function decides who gets transactions**: `HerderImpl::computeTxFloodTargets()`. With the leader schedule: the leaders of the node's current round and of the next one for the slot after the LCL (two nodes). With #5502's leader terms the node that closes a ledger leads the next one too; one slot before a new term starts (every fifth ledger), that term's expected first leader is a third target, so that it holds the transactions when it pre-validates and builds its first set. It is the first node of the order seeded with the LCL's hash, so it is known a full ledger ahead. "Everyone" (the old flood) when the node is not tracking or more than one ledger behind, when there is no schedule, and for a pre-protocol-20 LCL. It is re-evaluated at a ledger close, at a round change, at an externalize and at loss of sync, and sent to the overlay when it changes (IPC `SET_TX_TARGETS`). A changed leader schedule only has to change this function. - **The accepting node keeps the transaction** (overlay `flood/tx_router.rs`, at most 100,000 transactions and 300 s) until it is externalized, removed by Core or expires. When the targets change, it goes to a newly added target only if it may still be wanted there: it is younger than 200 ms, or three ledgers externalized on this node without it since it was accepted, or it is older than 3 s (no ledger closes). Anything else is most likely in the set in consensus. Nothing is sent again to a peer that got it less than 60 s ago on the same connection; a failed send or a reconnected peer is served regardless. A node that received a transaction as a target keeps it in its mempool and does not relay it. - **Fallback to the flood.** If a target is not a directly connected, identified peer, the node floods its transactions with adverts as before, so they reach every mempool whatever the topology. A node that got a transaction directly joins a later flood of it and can serve the body. - **Admission check** (`HerderImpl::checkTxAdmission`, in `recvTransaction`): `checkValid` against the LCL, the check the leader's build runs on every candidate, and one pending transaction per source account per accepting node (`TRY_AGAIN_LATER`; `DUPLICATE` for the same hash). On the base `recvTransaction` checked nothing. The per-account record goes when a set that uses up the sequence number closes, when the LCL shows the sequence number used some other way, or after 300 s. - **Overlays prove which validator they belong to.** The overlay's libp2p identity is a random key. At startup it sends its peer id to Core; Core signs `SHA256(networkID || "stellar-overlay-peer-id" || peerId)` with the node key; the overlay shows `[nodeID][signature]` to each peer on a new stream (`/stellar/node-id/1.0.0`, acknowledged, retried), and the receiver checks it against the peer id QUIC authenticated. The node key stays in Core. - **No leader prewarm of its own any more.** The version on the old base verified arriving signatures between the close and the trigger. #5502's leader builds once before the trigger and pre-validates during the previous ballot; bodies that arrive after those pulls are verified in the build, as on #5502 alone. - **Options, for measurements:** `TX_FLOOD_TARGET_LEADERS` (default 2; 0 = every validator of the schedule, still direct and without adverts), `TX_FLOOD_LEGACY_ADVERTS` (the base's flood from the same binary), `BATCH_EXECUTOR_TASKS` (number of validation tasks; above the physical core count they run on hyper-thread siblings). - New metrics: `overlay.flood.tx-direct-sent`, `.tx-direct-sent-bytes`, `.tx-direct-recv`, `.tx-local-flooded`, `.tx-late-copy` (a transaction admitted although one of the last few externalized ledgers already has it), `overlay.node-id.identified`, `.rejected`, `herder.tx-admission.check`, `.invalid`, `.account-busy`. ## Why ledger outputs cannot change, and what does change - The change decides which transactions reach a leader's mempool, when, and which submissions are refused. Building a set from a mempool, validating it, voting and applying are untouched: a given tx set produces the same ledgers and meta. - The admission check reads the LCL on the main thread and writes nothing. - The identity binding is domain-separated from everything else the node key signs. - **What gets worse, by design:** - Followers no longer hold transactions before the set arrives, so the background signature verification finds nothing and the apply-time check verifies signatures inline (see the results). - A term's first leader is predicted one ledger ahead. If the ledger in between costs that node its turn (#5502's forfeit), another node leads and gets the transactions only at the close. #5502 also lets the second node of the expected schedule pre-validate; it gets the transactions when the ledger before closes, not earlier. - A predicted target that is not a connected, identified peer makes the node flood for the whole ledger before the term. - A restarted node forgets the transactions it accepted and had not seen included; on the base every node held them. - A transaction submitted through a node that cannot reach both targets directly costs what it costs today. - **Left open:** - A peer can still fill a target's mempool with invalid transactions that outbid valid ones (the base's exposure, minus the relay). The leader validates when it builds and drops them then. - A refused submission returns `{"status":"ERROR"}` without the result code. - A pending transaction that dies for a reason other than its sequence number (expired time bound) holds its account on that node for up to 300 s. - Fee bumps are accepted and then dropped by the overlay, as on the base. - No protocol gating and no version negotiation: a node of the base ignores the identity stream, and nodes with this change flood toward it. ## Perfnet ### On top of #5502: 16 orgs, 800 ms, 8,000 TPS, c6id.8xlarge `block_time_check` (perfnet `btc2-slop`), `--org-count 16 --block-time-ms 800 --soroban-tps 8000 --soroban-type sac --classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900 --instance-type c6id.8xlarge`, 48 validators in four regions, no log saving. The #5502 column is that PR's own run of the same day at the same fixture. The stack column is build #3685 of `6b3c7ae0e`, which is this PR's head **without** the changed re-send rule, without the late-copy meter and without #5502's last commit. | Windows 1 / 2 / 3 / 4 | #5502 alone (build #3684) | This PR on #5502 (build #3685) | |---|---:|---:| | Windows on target | 4 of 4 | **4 of 4** | | Avg p75 block time, ms | 806.9 / 805.6 / 805.3 / 805.3 | **804.9 / 804.1 / 803.8 / 803.9** | | Avg p99, ms (limit 1,600) | 1,153 / 1,188 / 1,100 / 1,249 | 1,170 / 1,209 / 1,039 / 1,172 | | Txs per ledger; sets at the 8,000 cap | 6,450; 3.8% | 6,440; 3.4% | | **`overlay.byte.read` per node** | **21.2 MB/s** | **6.2 MB/s (-71%)** | | `overlay.message.read` per node | 20,500 /s | 12,550 /s | | Adverts per node | 7,848 /s | 0 | | Direct sends per offered transaction | none | 3.3 | | Leader build `herder.txset.build`; builds per ledger | 0.16 s; 1 | 0.15 s; 1 | | Trigger to first ballot | 0.108 to 0.115 s | 0.100 to 0.118 s | | Ballot to externalize | 0.29 s | 0.29 s | | Externalize to close (core logs, mean) | 0.045 to 0.048 s | 0.041 s | | **Trigger to close** | 0.44 to 0.45 s | **0.43 to 0.44 s** | | Apply-time check `herder.txset.validate` | 0.080 s | 0.099 s | | Signature-cache misses in that check, per node | (followers hold the transactions) | 7,490 /s (44% of the lookups) | | Validation-cache hit share in the leader's build | 69% | 66.7% | | Leader terms | exactly 5 ledgers | exactly 5 ledgers, 225 terms, 48 leaders | | Leader and prepare timeouts, empty sets | 0 | 0 | | Inclusion, over the 7,680,000 offered | +80 | **+6,610** | | Error counters, in sync, no fork, still closing | passed | passed | | Verdict | FAIL on inclusion only | FAIL on inclusion only | - The base `overlay-v2-shared` at this fixture on the same instances (reported in #5502): window 1 on target at 846 ms, window 2 missed, trigger to close 0.67 to 0.69 s. - **Inclusion.** The check requires exactly the offered count. #5502 alone held 80 more, this run 6,610 more (0.086%): copies of transactions that reach a node after their ledger closed and are included again, which only overlay-only mode permits. This run's build re-sent every pending transaction to newly added targets; the head sends only fresh ones or ones that missed three ledgers, and the goal-fixture run with it had 81 on 3,870,249 (0.002%) and counted 688 late copies (`overlay.flood.tx-late-copy`). The 16-org fixture was not rerun with the head. - Every node identified all 47 peers, nothing was flooded, no send failed, no submission was refused; the admission check takes 0.094 ms per transaction on the main thread. ### On top of #5502: the goal fixture, 30 orgs, 800 ms, 10,000 TPS, c6id.8xlarge The same command with `--org-count 30 --soroban-tps 10000`; 90 validators, 10,080 TPS offered. The stack column is build #3688 of this PR's head `6ab837c27`. The #5502 column is that PR's own run of its head `8aa7f6bf0` two hours earlier (some rows are from its first run at this fixture, marked G). | Window 1 (the load was stopped after it in both runs) | #5502 alone (build #3686) | This PR on #5502 (build #3688) | |---|---:|---:| | Avg p75 block time, ms (limit 960) | 1,031.6 | **979.6** | | Avg p99, ms (limit 1,600) | 1,887 | **1,441** | | Time between ledgers | 1.02 s | 0.90 s (median 0.88 s) | | The first minute of the load | 802 ms for 30 s, backlog from 47 s | 805 ms over 60 s | | Sets at the 10,000 cap | 93% | 38.7% | | Leader builds before the trigger | 0 of 40 | 40 of 55 | | Leader build; candidates pulled | 0.33 s; 20,200 | 0.24 s; median 8,350 (max 20,200) | | Trigger to first ballot | 0.58 s | **0.39 s** | | Ballot to externalize | 0.32 s | 0.32 s | | Externalize to close | 0.07 s | 0.065 s | | **Trigger to close** | 0.97 s | **0.77 s** | | Set at the followers after the proposal | +230 ms (G) | +174 ms | | Apply-time check `herder.txset.validate` | (see #5502) | 0.144 s | | Validation-cache hit share in the leader's build | 66.3% (G) | 66.7% | | **`overlay.byte.read` per node** | **43 MB/s** (G) | **15.7 MB/s** | | Direct sends per submitted transaction | none | 2.5 to 2.6 while keeping up, 3.7 in the backlog | | Leader terms; timeouts | 5 ledgers; none | 5 ledgers; none | | Submitted / included at the end | 49,855 short | 81 over (backlog drained) | | `overlay.flood.tx-late-copy` | n/a | 688, on 14 of 90 nodes | | Error counters, in sync, no fork, still closing | passed | passed | | Verdict | FAIL, window 1 (p75 and p99) | FAIL, window 1 (p75 only) | **Not measured:** the 16-org fixture again with the head's re-send rule; the base `overlay-v2-shared` at the goal fixture; the 4-core instances with #5502 underneath; `TX_FLOOD_TARGET_LEADERS=0`; real apply; more than one run of anything. ### Before the rebase: 16 orgs, 1 s, 5,000 TPS (on `overlay-v2-shared` at `26bea1150`, without #5502) All runs: `block_time_check` (perfnet `btc2-slop @ 6dd2064`), `--block-time-ms 1000 --soroban-tps 5000 --soroban-type sac --classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900`, validators in us-east-1, us-west-2, eu-central-1 and ap-southeast-1, every validator generates load, overlay-only mode (Core does not apply; execution is a sleep). 5-minute means at the end of window 1, median node. | | Base | This PR, 3 tasks | This PR, 7 tasks | This PR, 16 cores | Base, 30 orgs | |---|---:|---:|---:|---:|---:| | Build (Jenkins; commit) | #3678; `0c9a02e56` (tree of `26bea1150`) | #3680; `dc91c8964` | #3683; `664a35fdc` | #3680; `dc91c8964` | #3678 | | Validators x instance type | 48 x c6id.2xlarge | 48 x c6id.2xlarge | 48 x c6id.2xlarge | 48 x c6id.8xlarge | 90 x c6id.2xlarge | | Validation tasks | 3 | 3 | 7 (`BATCH_EXECUTOR_TASKS=7`) | 15 | 3 | | Verdict | FAIL on inclusion only (+60) | **FAIL**, window 1 | **PASS** | FAIL on inclusion only (+250) | **FAIL**, window 1 | | Windows on target | 4 of 4 | 0 of 1 (load stopped) | 4 of 4 | 4 of 4 | 0 of 1 (load stopped) | | Avg p75 per window, ms | 1,081 / 1,132 / 1,118 / 1,142 | 1,367 | 1,030 / 1,030 / 1,031 / 1,031 | 1,019 / 1,021 / 1,020 / 1,021 | 1,298 | | `overlay.byte.read` per node | 13.97 MB/s | 4.77 MB/s | 4.71 MB/s | 4.77 MB/s | 26.69 MB/s | | `overlay.message.read` per node | 14,804 /s | 9,374 /s | 9,914 /s | 10,085 /s | 33,523 /s | | Direct sends per offered transaction | none | 9.0 | 3.4 | 3.1 | none | | Leader build `herder.txset.build` | 0.235 s | 0.374 s | 0.099 s | 0.063 s | 0.314 s | | Trigger to first ballot | 0.413 s | 0.539 s | 0.300 s | 0.232 s | 0.613 s | | Ballot to externalize | 0.296 s | 0.290 s | 0.293 s | 0.284 s | 0.387 s | | Apply-time check `herder.txset.validate` | 0.179 s | 0.277 s | 0.168 s | 0.078 s | 0.247 s | | Signature-cache misses in that check | 2,089 /s | 4,550 /s | 4,654 /s | 4,647 /s | 2,841 /s | | Externalize to close (core logs, median) | 58 ms | 145 to 210 ms | 44 ms | 28 ms | 61 to 102 ms | | Ledgers per 5 minutes; txs per ledger | 291; 5,152 | 266; 5,637 | 300; 5,001 | 300; 5,000 | 262; 5,742 | | Error counters, in sync, no fork, still closing | passed | passed | passed | passed | passed | - The base column is run 3 of #5498's record (same fixture, same box, the evening before); it was not rerun. - **3 tasks:** 75% of the sets were at the 6,250 cap (1.27 s per ledger) and 16% were starved (as few as 938 transactions): after a slow ledger the next trigger is already due and the new leader builds before the pending transactions reach it. - **7 tasks and 16 cores:** no starved sets, no backlog; the leader pulls about 5,000 candidates at its trigger. - Every run of this PR: every node identified all 47 peers as validators (0 rejected claims), nothing was flooded, no message was dropped, no submission was refused; the admission check takes about 0.12 ms per transaction on the main thread. - **Inclusion.** The check requires every node to hold exactly the offered transactions. Base: +60; 16-core run: +250; 7-task run: exact. In overlay-only mode Core's sequence-number check is off, so a copy of a finalized transaction that a node admits again is included when that node next leads (performance-testing#4, not merged, would allow up to 480 here). Which transactions were duplicated in the 16-core run, and why more than on the base, is **not established**. **Not measured before the rebase:** this change at 30 orgs; the base with 7 tasks; real apply (with real execution the check and the execution compete for the same cores). ## Testing On `6ab837c27398` (this PR's head, on #5502 at `8aa7f6bf0`), devbox, 32 partitions: - **Full suite: PASS** (exit 0, 718 s), with libsodium, xdrpp and the overlay's Rust tests (224 library, 47 binary, 7 end-to-end; 4 ignored). - **Full suite `--all-versions`: PASS** (exit 0, 735 s). - Tag run (`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[simulation]~[.],[loadgen]~[.]`, which includes #5502's tests): only the three failures recorded for the base ("SCP Driver" validity-cache hits, "hierarchical topology scales 1 to 3", "core nodes with outer nodes"). - **One unexplained Rust failure:** on an earlier head (`6b3c7ae0e`) one `make check-rust` run failed 1 of 219 library tests and the output was not kept, so the test is not known. About twenty later runs of the same suite passed (apart from failures of tests I was changing at the time). #5502 does not touch `overlay/`. New tests: - `herder/test/TxFloodTargetsTests.cpp` (6 cases): - In a 4-validator mesh every submitted transaction is applied on every node with no advert and no demand, and every target list each node sent to its overlay equals what the schedule functions give for the LCL and round it was sent at: two round leaders, and a third node only in the ledger before a new term. A list sent one ledger before a term start names the node that then leads the term, unless the ledger in between forfeited. **Mutation-checked** three ways: the router forced to flood (96 adverts for 32 transactions), the term lookahead removed, and the lookahead seeded with the wrong hash; each fails the test. - With `TX_FLOOD_TARGET_LEADERS = 0` each transaction goes to the three other nodes exactly once. - With one of three validators down every transaction is still included and every target list matches the schedule at the node's round, including rounds after a timeout. - In overlay-only mode with load from all four nodes, no transaction hash appears in two ledgers (2,400 transactions; it does not reproduce the perfnet duplicates, which need network latency). - The admission check (accepted once per account, duplicate, bad signature, used and future sequence number, low fee, missing account, freed by the including ledger, by another transaction of the account, by a sequence bump, by timeout; overlay-only mode; pregenerated load); the experiment options. - Rust `flood/tx_router.rs` (20 unit tests): targets only and following the targets; a kept transaction follows a target change only while fresh, after three ledgers without it, or after 3 s (**mutation-checked**: with the rule disabled five tests fail); a removal by Core is not a ledger; flood without reachable targets; a directly received transaction joins a later flood and serves the body. - Rust `libp2p_overlay/tx_direct_tests.rs` (4 network tests on loopback, 1 unit test): the same over real connections, including a late copy being counted; forged, replayed and wrong-network identity claims are rejected, and claims are bounded per connection. `main.rs` (3): the IPC handlers. Existing tests changed: - `overlay/.../dispatch_tests.rs` (2): they fed a body without a demand and expected a relay; they now register the demand, and one asserts that an undemanded body is not relayed. - `main/test/CommandHandlerTests.cpp` "transaction envelope bridge": V1 and fee-bump envelopes are refused again below protocol 13, and the too-early / too-late cases are back. Review: four passes of an independent review (two on the old base, one of the rebase, one of the re-send rule); no consensus or determinism finding in any. Open from the last two: the flush after an externalize plans against the outgoing targets (one stray body for transactions that waited three ledgers); a lost send re-delivers to every held-back target, not only the lost peer; after a leader failure the round-2 leader lacks transactions older than 200 ms until the ledger closes; the 200 ms bound is a constant, not derived from the block time; no test covers a watcher, a wrong term prediction, a lagging sender or the `main.rs` wiring of the ledger count. Not passing or not run: - Tests outside the regular suite that fail the same way on the base: "resilience tests", "cycle4 topology", "asymmetric topology report cost", and the three above. - A `--disable-tests` build was not tried; #5495 records that the base does not build that way. - No sanitizer run. No tx-meta baseline run (the diff does not touch apply). ## Commits Three commits on top of #5502: the code, and two for the README (which now describes and reports the whole stack: #5502's leader terms and pre-validation, and this PR). Builds that perfnet ran: #3688 of the head `6ab837c27` (the goal-fixture run), #3685 of `6b3c7ae0e` (the 16-org run; that commit is no longer on the branch) and, before the rebase, #3680 of `dc91c8964` and #3683 of `664a35fdc`.
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, not for merge as is. On
overlay-v2-shared(after #5497 and #5498). It changes the leader schedule (consensus behaviour) without a protocol gate, like the schedule it builds on (#5495), and exists to measure what taking the leader's transaction-set build off the critical path is worth.What it does
Today the round leader of ledger N+1 is only known once ledger N has closed, and it builds its transaction set after that: right after the close and, usually, again at the trigger. This PR does three things.
LEADER_TERM_LEDGERS); terms end on ledger numbers divisible by 5, so a slot's schedule is still a function of the previous ledger's header and the node set (restart and catchup need no history). The order of a slot is seeded with the hash of the ledger before the last closed one, so it is known one ledger earlier. A leader loses its term at once, and goes to the end of the next order, when its ledger closed with a consensus-level empty-tx-set value or applied no transaction (a set found invalid at apply, Prototype: vote without validating the tx set; validate and execute speculatively, skip an invalid set at apply #5498). A leader whose round times out loses it too: the next round's leader builds its own value or, if it already votes for the old leader's value and is not yet tied to it, proposes it under its own signature (SCPDriver::reproposeValue).LedgerManager::startSpeculativeApplycan hand back the state a speculative execution left (ExecutedLedgerState: the previous ledger's snapshot plus the account entries the execution changed, read from the parkedLedgerTxnwithout sealing it). The nodes that expect to lead the next ledger in round 1 or round 2 runcheckValidon the mempool's top candidates against that state, on background threads, while the executed ledger is still in its ballot. The results are used only if the ledger closes with the very value that was executed, never for transactions whose validity depends on the close time, and all builds on one last closed ledger share them (TxValidationCache).Non-leaders already do not build sets on this branch; the second leader in line pre-validates but does not build or push unless a timeout promotes it.
Results
Perfnet (
block_time_check), 16 tier-1 orgs = 48 validators in 4 regions, c6id.2xlarge, SAC payments, simulated apply budget 150 ms, 800 ms target, one run per build. Base =overlay-v2-sharedat26bea1150(build of the identical tree0c9a02e56).8fd536b65, the first commit before the rebase)deec2d87c, the head before the rebase and the test-only commit)At 8,000 TPS on these 4-core instances both builds are over capacity and this PR changes little (5% more ledgers). The mechanism has no room there: the apply-time check plus the simulated apply (0.48 s) outlast the ballot (0.31 s), so the leader finishes executing ledger N only about 40 ms before N closes, the trigger for N+1 is already due, and the build starts about 60 ms after the pre-validation did. What limits that fixture is per-transaction validation on 3 validation tasks, not the order of the stages.
On 16-core instances (c6id.8xlarge, 15 validation tasks) at the same load, the base drifts and misses; this PR holds 800 ms.
On these instances the apply-time check takes 0.08 s instead of 0.29 s, so a ledger's execution ends inside its ballot and the PR's parts have room to run. The base's leader needs about 0.82 s of work per ledger, more than the 800 ms target, so its ledgers grow and it falls behind; with this PR the leader's work fits before the trigger and the trigger sets the pace, with 0.45 s of work in an 0.81 s cycle. The base's run also ended with 434 inclusions over the offered count (not judged, its load was stopped), the size of this PR's +460 at 5,000 TPS: that excess is not specific to this change. One sample of per-thread CPU during this PR's run showed one Core thread at 99.9% with the machine 90% idle: the next limit looks like Core's main thread.
The goal fixture (30 orgs = 90 validators, 800 ms, 10,000 TPS, c6id.8xlarge), this PR alone, two runs: narrow misses.
deec2d87c)8aa7f6bf0)Both runs start on target and then fall into a backlog they do not leave (93-96% of the sets at the 10,000 cap, 1.02-1.03 s per ledger). The node count itself cost little: ballot 0.32 s (0.29 s at 48 nodes), main thread at a quarter of a core, 43 MB/s of overlay traffic per node without loss, leader terms of exactly 5 throughout. The first run showed a flaw of this PR (a node starting a term sized its build's lead time from a build made before the load, so the first ledger of every term was late); the last commit fixes that, and the second run shows the fix working and not being enough. What is left: at this load the cycle has no slack. Before the backlog a ledger led from us-east-1 takes 741-802 ms, from eu-central-1 790-841 ms, from ap-southeast-1 850-922 ms; there are 0.2-0.29 s between a close and the next trigger, and the build (0.2-0.25 s) cannot start before the previous ledger closes, because the set names that ledger's hash. This PR validates during the previous ballot; it does not build during it. There is no base run at this fixture.
What these runs cannot show: the fixture runs in overlay-only mode, so ledger state never changes and no round timed out. Validation against the executed state, the forfeit rules and the re-sign are covered by unit tests only, and the simulated apply costs no CPU, so the pre-validation did not compete with a real execution. n = 1 per build; no ablation of the three parts.
Correctness
--check-test-tx-meta, all versions) and the LedgerCloseMeta golden data are identical.previousLedgerHash,ledgerSeq,scpValue,txSetResultHash) plus the node set. Safety rests on the ballot protocol for any choice of leaders. The re-sign is only offered while the new leader has not accepted the old value as prepared, the same state in which it could build a new value.Known gaps and behaviour changes
Testing
src/herder/test/ProposalPrevalidationTests.cpp(9 test cases: the captured state, results used / dropped / not kept, upgrades, time-bounded transactions, a with-and-without comparison over chained sequence numbers, a drained fee source, a merged source and a fee bump, job lifetime, lead time, and a 4-node network that keeps terms and builds from pre-validated candidates),LeaderScheduleTests.cpp(term outcome, next schedule, terms over a chain of ledgers), "leader rounds core5" (the re-propose hook, tied and not tied), "the leader of a later round signs the value it proposes again" (real timers, both cases). Two[early-proposal]tests that encoded "rebuild at the trigger" are rewritten.8aa7f6bf0).ALL_VERSIONSon it: one run with a single failure in an existing network test ("network externalizes an invalid tx set and applies it as empty": the faulty leader's set never externalized in the 20 ledgers watched; not reproduced in three runs of that test alone with the same arguments, cause not established), and one clean run. Both pass on the commit before (6716b4e2e), which is test-only: two leader-schedule test helpers failed for some instance numbers with the day's test seed (a flaw the base has too); they no longer depend on where the test node's key sorts. tx-meta baseline and LCM golden data unchanged, checked on3257ef3ddbefore the rebase; the commits since only change when the leader starts its build, and tests.--disable-tests: every object this PR touches compiles exceptherder/HerderImpl.o, which fails with the six errors the base already has (test headers reached through itssimulation/LoadGenerator.hinclude).clang-format(v20, the repository's.clang-format) on every changed file.