Skip to content

Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger - #5502

Merged
SirTyson merged 6 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/leader-prebuild-pipelining
Nov 12, 2026
Merged

SirTyson merged 6 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/leader-prebuild-pipelining

Conversation

@SirTyson

@SirTyson SirTyson commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Prototype, not for merge as is. On overlay-v2-shared (after #5497 and #5498). It changes the leader schedule (consensus behaviour) without a protocol gate, like the schedule it builds on (#5495), and exists to measure what taking the leader's transaction-set build off the critical path is worth.

What it does

Today the round leader of ledger N+1 is only known once ledger N has closed, and it builds its transaction set after that: right after the close and, usually, again at the trigger. This PR does three things.

  1. Leader terms. The node whose value closed a ledger leads the next one, for up to 5 ledgers (LEADER_TERM_LEDGERS); terms end on ledger numbers divisible by 5, so a slot's schedule is still a function of the previous ledger's header and the node set (restart and catchup need no history). The order of a slot is seeded with the hash of the ledger before the last closed one, so it is known one ledger earlier. A leader loses its term at once, and goes to the end of the next order, when its ledger closed with a consensus-level empty-tx-set value or applied no transaction (a set found invalid at apply, Prototype: vote without validating the tx set; validate and execute speculatively, skip an invalid set at apply #5498). A leader whose round times out loses it too: the next round's leader builds its own value or, if it already votes for the old leader's value and is not yet tied to it, proposes it under its own signature (SCPDriver::reproposeValue).
  2. Pre-validation during the previous ballot. LedgerManager::startSpeculativeApply can hand back the state a speculative execution left (ExecutedLedgerState: the previous ledger's snapshot plus the account entries the execution changed, read from the parked LedgerTxn without sealing it). The nodes that expect to lead the next ledger in round 1 or round 2 run checkValid on the mempool's top candidates against that state, on background threads, while the executed ledger is still in its ballot. The results are used only if the ledger closes with the very value that was executed, never for transactions whose validity depends on the close time, and all builds on one last closed ledger share them (TxValidationCache).
  3. One build per ledger, before the trigger. The first-round leader builds once, a lead time before the trigger, pushes the set to its peers as soon as its XDR exists (Prototype: push the leader's tx set to peers when it is built #5497's push), and proposes it unchanged at the trigger. The lead time is a quarter of the target close time, raised to cover the node's last build plus the push, up to half of the target. Consensus messages still start at the trigger.

Non-leaders already do not build sets on this branch; the second leader in line pre-validates but does not build or push unless a timeout promotes it.

Results

Perfnet (block_time_check), 16 tier-1 orgs = 48 validators in 4 regions, c6id.2xlarge, SAC payments, simulated apply budget 150 ms, 800 ms target, one run per build. Base = overlay-v2-shared at 26bea1150 (build of the identical tree 0c9a02e56).

5,000 TPS Base This PR (8fd536b65, the first commit before the rebase)
Windows on target 0 of 1 (load stopped) 4 of 4, on all 48 nodes
avg p75 block time (limit < 960 ms) 1,012 ms 810 ms (809-811 over the windows)
avg p99 1,189 ms 875-941 ms
Seconds per ledger 0.92 0.80
Trigger to first ballot 0.465 s 0.10 s
Ballot to externalize 0.298 s 0.291 s
Externalize to ledger closed 0.062 s 0.045-0.050 s
Trigger to close 0.83 s 0.44 s
Leader builds per ledger; build time 1.78; 0.225 s 0.99; 0.17 s (validation stage 0.14 -> 0.07 s)
Pushed set in place when Core asks 4,141 of 20,933 43,634 of 59,999
  • Leader terms: 219 terms of exactly 5 ledgers on the fixed boundaries, 2 double terms (the seeded order put the same node first again), none cut short; no timeout, no re-proposal.
  • Pre-validation: 2,404 started, 2,404 used, 49 cut short by the build, none discarded; 0.15 s each off the main thread (mempool pull 5 ms on it, state capture 1 ms on the apply thread); a build finds 64-66% of its candidates already validated.
  • Most of the gain is the removed second build (in the base the early build ran into the trigger on 69 of 84 sampled ledgers and was then repeated); the pre-validation is the smaller part.
  • The verdict of the change's run is still FAIL, on the inclusion count only: every node's ledgers hold 4,800,460 transactions for 4,800,000 offered. That is the shape of the known duplicate inclusions of perfnet's overlay-only mode (Core does not apply, so a late copy of a finalized transaction is not rejected), but the excess was 60 in earlier runs and this run cannot say why it is 460 here. Open.
  • In that run the build used 180 of its 200 ms lead time, so the pushed set still reached followers about 100 ms after the proposal. The last commit lengthens the lead time; it is measured at 8,000 TPS below, not at 5,000.
8,000 TPS Base This PR (deec2d87c, the head before the rebase and the test-only commit)
Verdict FAIL, window 1 missed on 48/48 FAIL, window 1 missed on 48/48
avg p75 block time 1,337 ms 1,254 ms
Seconds per ledger 1.27 1.20
Included transactions per second (8,000 offered) 6,320 6,660
Trigger to first ballot / to close 0.729 s / 1.22 s 0.666 s / 1.15 s
Leader build 0.555 s 0.584 s (16,200 candidates from a backed-up mempool in both, every set at the 8,000 cap)
Apply side execution 0.478 s (apply-time check 0.293 s + sleep) against a 0.305 s ballot 0.475 s against 0.305 s
Pre-validation n/a half cut short by the build; 3.7% of a build's candidates already validated (5,000 TPS: 64%)

At 8,000 TPS on these 4-core instances both builds are over capacity and this PR changes little (5% more ledgers). The mechanism has no room there: the apply-time check plus the simulated apply (0.48 s) outlast the ballot (0.31 s), so the leader finishes executing ledger N only about 40 ms before N closes, the trigger for N+1 is already due, and the build starts about 60 ms after the pre-validation did. What limits that fixture is per-transaction validation on 3 validation tasks, not the order of the stages.

On 16-core instances (c6id.8xlarge, 15 validation tasks) at the same load, the base drifts and misses; this PR holds 800 ms.

8,000 TPS, 800 ms, 48 validators, c6id.8xlarge Base This PR
Windows window 1 on target, window 2 missed (one node's p99 1,647 ms; load stopped after 10 minutes) all four on target on 48/48, the full 15 minutes
avg p75 block time (limit < 960 ms) 846, then 869 ms (rising) 805-807 ms (flat)
avg p99 (limit 1,600 ms) 1,368, then 1,480 ms 1,100-1,250 ms
Seconds per ledger 0.84-0.85 0.81
Leader builds per ledger, and when 1.9: one at the previous close that is never proposed, a second at the trigger 1.0: one build of 0.18 s, done 114 ms before the trigger, proposed unchanged
Candidates already validated when the build starts n/a 69% (pre-validation 0.10 s, 99.6% complete)
Followers whose overlay had the set before the proposal 0 of 1,914 72% (2,435 of 3,399)
Trigger to first ballot 0.34-0.36 s 0.11 s
Ballot to externalize / externalize to close 0.29 s / 0.04 s 0.29 s / 0.045 s
Trigger to close 0.67-0.69 s 0.45 s
Verdict FAIL (window 2) FAIL on the inclusion count only (+80 of 7,680,000)

On these instances the apply-time check takes 0.08 s instead of 0.29 s, so a ledger's execution ends inside its ballot and the PR's parts have room to run. The base's leader needs about 0.82 s of work per ledger, more than the 800 ms target, so its ledgers grow and it falls behind; with this PR the leader's work fits before the trigger and the trigger sets the pace, with 0.45 s of work in an 0.81 s cycle. The base's run also ended with 434 inclusions over the offered count (not judged, its load was stopped), the size of this PR's +460 at 5,000 TPS: that excess is not specific to this change. One sample of per-thread CPU during this PR's run showed one Core thread at 99.9% with the machine 90% idle: the next limit looks like Core's main thread.

The goal fixture (30 orgs = 90 validators, 800 ms, 10,000 TPS, c6id.8xlarge), this PR alone, two runs: narrow misses.

Commit before the last (deec2d87c) PR head's code (8aa7f6bf0)
Verdict FAIL, window 1 missed on 90/90 FAIL, window 1 missed on 90/90
avg p75 block time (limit < 960 ms) 1,046 ms 1,032 ms
Load held at 800 ms per ledger for the first 38 s the first 47 s
First ledger of a leader term / the other four 929 ms / 768-814 ms 805 ms / about the same
Timeouts, forks, sync losses, skipped sets none none

Both runs start on target and then fall into a backlog they do not leave (93-96% of the sets at the 10,000 cap, 1.02-1.03 s per ledger). The node count itself cost little: ballot 0.32 s (0.29 s at 48 nodes), main thread at a quarter of a core, 43 MB/s of overlay traffic per node without loss, leader terms of exactly 5 throughout. The first run showed a flaw of this PR (a node starting a term sized its build's lead time from a build made before the load, so the first ledger of every term was late); the last commit fixes that, and the second run shows the fix working and not being enough. What is left: at this load the cycle has no slack. Before the backlog a ledger led from us-east-1 takes 741-802 ms, from eu-central-1 790-841 ms, from ap-southeast-1 850-922 ms; there are 0.2-0.29 s between a close and the next trigger, and the build (0.2-0.25 s) cannot start before the previous ledger closes, because the set names that ledger's hash. This PR validates during the previous ballot; it does not build during it. There is no base run at this fixture.

What these runs cannot show: the fixture runs in overlay-only mode, so ledger state never changes and no round timed out. Validation against the executed state, the forfeit rules and the re-sign are covered by unit tests only, and the simulated apply costs no CPU, so the pre-validation did not compete with a real execution. n = 1 per build; no ablation of the three parts.

Correctness

  • Ledger outputs and meta are unchanged. A set built from pre-validated results goes through the same selection, XDR round trip and final check, and every node still validates it at apply; apply never sees the result cache. tx-meta baseline (--check-test-tx-meta, all versions) and the LedgerCloseMeta golden data are identical.
  • A wrong pre-validation result cannot fork; it would make the leader propose a set that every node skips alike at apply (and costs that leader its term).
  • Schedule: every input is in the last closed header (previousLedgerHash, ledgerSeq, scpValue, txSetResultHash) plus the node set. Safety rests on the ballot protocol for any choice of leaders. The re-sign is only offered while the new leader has not accepted the old value as prepared, the same state in which it could build a new value.
  • Two independent reviews (reading only), no critical finding; their major findings are fixed and listed with resolutions in the work record.

Known gaps and behaviour changes

  • One leader holds up to 5 consecutive ledgers and is known several ledgers ahead; the proposer of the ledger two before a term picks that term's seed (grinding buys 5 ledgers instead of 1).
  • A later round's leader that has already accepted the old leader's value as prepared keeps it, so that leader keeps its term although a round timed out. Closing this needs the externalizing round in the ledger (XDR).
  • Transactions that arrive during the lead time before the trigger wait one more ledger.
  • The test-only overlay-only flag is an input of the schedule that is not in the ledger.
  • The state capture copies the changed account entries on the apply thread before the commit; 1 ms in the runs below, where overlay-only mode changes no entries. Its cost with real apply is not measured.
  • Not covered by tests: Soroban candidates in the pre-validation, restart or catchup onto a mid-term ledger, a multi-node test of the re-sign with a delayed link, a ThreadSanitizer run.

Testing

  • New: src/herder/test/ProposalPrevalidationTests.cpp (9 test cases: the captured state, results used / dropped / not kept, upgrades, time-bounded transactions, a with-and-without comparison over chained sequence numbers, a drained fee source, a merged source and a fee bump, job lifetime, lead time, and a 4-node network that keeps terms and builds from pre-validated candidates), LeaderScheduleTests.cpp (term outcome, next schedule, terms over a chain of ledgers), "leader rounds core5" (the re-propose hook, tied and not tied), "the leader of a later round signs the value it proposes again" (real timers, both cases). Two [early-proposal] tests that encoded "rebuild at the trigger" are rewritten.
  • Mutation check: 8 mutants (stale results accepted, the view ignoring the execution, the closing ledger's transactions pre-validated, time-bounded results cached, forfeiter not moved, no re-sign, rebuild at the trigger, incumbent not kept), each caught by a test.
  • Full suite passes on the PR head (8aa7f6bf0). ALL_VERSIONS on it: one run with a single failure in an existing network test ("network externalizes an invalid tx set and applies it as empty": the faulty leader's set never externalized in the 20 ledgers watched; not reproduced in three runs of that test alone with the same arguments, cause not established), and one clean run. Both pass on the commit before (6716b4e2e), which is test-only: two leader-schedule test helpers failed for some instance numbers with the day's test seed (a flaw the base has too); they no longer depend on where the test node's key sorts. tx-meta baseline and LCM golden data unchanged, checked on 3257ef3dd before the rebase; the commits since only change when the leader starts its build, and tests.
  • --disable-tests: every object this PR touches compiles except herder/HerderImpl.o, which fails with the six errors the base already has (test headers reached through its simulation/LoadGenerator.h include).
  • clang-format (v20, the repository's .clang-format) on every changed file.

…allot, one build before the trigger (prototype)

Takes the round leader's transaction set build off the consensus critical
path. Prototype on overlay-v2-shared, not protocol gated (like the leader
schedule it changes).

Schedule
- The node whose value closed a ledger leads the next one, for up to
  LEADER_TERM_LEDGERS (5) ledgers; terms end on ledger numbers divisible by
  5, so the schedule of a slot is still a function of the previous ledger's
  header and the node set.
- The order of a slot is seeded with the hash of the ledger before the last
  closed one (the header's previousLedgerHash), so it is known one ledger
  earlier.
- A leader loses its term at once, and goes to the end of the next order,
  when its ledger closed with a consensus-level empty-tx-set value or
  applied no transaction (a set found invalid at apply). A leader whose
  round times out loses it too: the leader of a later round puts forward
  the value it already votes for under its own signature
  (SCPDriver::reproposeValue), and the node that signed the value that
  closes a ledger is the next leader.

Pre-validation
- LedgerManager::startSpeculativeApply can hand back the state a
  speculative execution left (ExecutedLedgerState: the previous ledger's
  snapshot plus the account entries the execution changed, read from the
  parked LedgerTxn without sealing it).
- A node that expects to lead the next ledger in round 1 or 2 validates
  the mempool's top candidates against that state on background threads
  while the executed ledger is still in its ballot. The results are used
  only if the ledger closes with the very value that was executed, and
  never for transactions whose validity depends on the close time.
- All builds on one last closed ledger share a validation result cache
  (TxValidationCache), seeded by the pre-validation: a build only validates
  what it has not seen.

One build per ledger
- The first-round leader builds once, a quarter of the target close time
  (at most 250 ms) before the trigger, pushes the set to its peers as soon
  as its XDR exists, and proposes it unchanged at the trigger. It used to
  build right after the close and again at the trigger.

New metrics: herder.prevalidate.{started,used,discarded,incomplete,txs,
pull,duration}, herder.txset.validation-cache-{hit,miss},
ledger.apply.capture-state.
…next order

Test-only. The helper that makes the local node the first leader moves to the
start of a term when test setup left the node with a forfeited term, and the
healthy-network test expects the previous proposer of an empty ledger at the
end of the order.
…at a term boundary

Second review's findings.
- A later round's leader that has accepted the value it votes for as prepared
  keeps it as it is: other nodes may have confirmed it, and a replacement
  under another signature could only split the round. The replacement is
  only offered before that (SCPDriver::reproposeValue), for a set the node
  holds, and counted (scp.leader.resigned).
- A ledger that applied nothing sends its proposer to the end of the next
  order also when the next ledger starts a term.
- The pre-validation job reports itself complete only if no task was
  stopped, and its candidate decode cannot throw out of call_once.
The first-round leader's lead time before the trigger was a quarter of the
target close time. At 48 validators, 800 ms and 5,000 TPS the build took
180 ms of those 200 ms, so the set left 20 ms before the trigger and reached
the followers about 100 ms after the proposal. The lead time now also covers
the node's last build (with a quarter to spare) plus 100 ms for the push, up
to half of the target close time.
…'s key sorts

Test-only. The test configuration's node key is derived from the instance
number and the day's seed. For a key that sorts behind nearly every other one
the helpers could not find a peer that comes after it within their tries
(PendingEnvelopes tests: 'could not pick a peer that does not lead round 1';
seen with seed 20769 at base instance 1450, and on the base branch with seed
20768 at 850). createAppWithFollowerPeer now also varies the local key;
selectLocalFirstLeader tries more absent peers.
…or this node's last build

At 90 validators, 800 ms and 10,000 TPS the first ledger of every leader term
came late (929 ms against 768-814 ms for the term's other four): a node that
starts a term has not built since before the load rose, so a lead time derived
from its last build (about 1 ms, an empty set) fell back to the 200 ms
minimum, the 190-220 ms build ended at the trigger and the pushed set reached
the followers 0.17 s after the proposal. One slow term then tipped the network
into a backlog it did not leave.

The expected build is now the longer of the node's last build and what a set
of the last closed ledger's size costs at the node's measured cost per
transaction (40 us until it has built a set of 100 transactions or more).
SirTyson added a commit that referenced this pull request Nov 12, 2026
… an admission check (#5501)

**Prototype, not for merge. Stacked on #5502** (leader terms,
pre-validation during the previous ballot, one build before the
trigger): this PR's own changes are the last two commits, `6ab837c27`
(the code) and two README commits (the results of the whole stack); the
commits before them are #5502's (head `8aa7f6bf0`) and leave this diff
when #5502 merges.

A node that accepts a transaction from a client checks it against its
last closed ledger, keeps it, and has its overlay send the body straight
to the leaders of the current and the next round of the next ledger,
instead of advertising it to every peer (INV/GETDATA). Since protocol 30
a follower votes without validating, so it does not need a transaction
before the leader's set arrives.

## Results in short

One run per configuration everywhere; perfnet `block_time_check`, SAC
payments, 150 ms simulated apply, overlay-only mode.

**On top of #5502, 16 tier-1 orgs (48 validators in 4 regions), 800 ms
target, 8,000 TPS, c6id.8xlarge (16 cores):**

- **Block time is the same as #5502 alone, on target in all four
windows:** avg p75 804 to 805 ms against 805 to 807 ms; trigger to close
0.43 to 0.44 s against 0.44 to 0.45 s. Both runs sit at the fixture's
floor, so this run shows neither a gain nor a loss in block time.
- **Overlay traffic per node falls from 21.2 to 6.2 MB/s each way
(-71%).** No adverts or demands are left; 3.3 transaction bodies are
sent per offered transaction.
- The leader's side is unchanged: one build per ledger before the
trigger, the pre-validation covers 66.7% of the candidates (69% on #5502
alone), leader terms of exactly 5 ledgers, no timeout.
- Followers verify signatures inline: the apply-time check takes 0.099 s
against 0.081 s, still inside the ballot on 16 cores.
- The verdict is FAIL on the inclusion count only, as for #5502 alone
(the check is exact; see "Inclusion" under Perfnet).

**On top of #5502, the goal fixture: 30 orgs (90 validators), 800 ms,
10,000 TPS, c6id.8xlarge (this PR's head):**

- **FAIL: window 1 at avg p75 980 ms (limit 960), p99 1,441 ms (inside
its 1,600 limit); the load was stopped after it.** #5502 alone at this
fixture: 1,032 ms and p99 1,887 ms.
- A ledger every 0.90 s against 1.02 s; 39% of the sets at the 10,000
cap against 93%; three of four leader builds before the trigger against
none; trigger to close 0.77 s against 0.97 s. The gain is in trigger to
first ballot (0.39 s against 0.58 s): the leader's mempool holds about
one ledger of candidates instead of a 20,200 backlog.
- **Overlay traffic per node: 15.7 MB/s against 43.** At 90 validators
most of what is left is SCP (35,700 messages per second per node).
- What still does not fit into 800 ms is the ballot (0.32 s), the set
reaching followers 0.17 s after the proposal, and the builds that still
run at the trigger. None of these is transaction dissemination.
- **Where the remaining time goes** (twelve nodes' logs): the leader's
build starts when the previous ledger closes and takes 0.29 s to the
proposal; the set then needs 0.19 s to reach a follower (the leader
sends 5.7 MB of XDR, compressed, to each of 89 peers); the ballot takes
0.34 to 0.36 s; the commit 0.07 s. Sum 0.89 s on the leader. The README
has the table.

**Before the rebase (on `overlay-v2-shared` at `26bea1150`, without
#5502), 16 orgs, 1 s target, 5,000 TPS:**

- Overlay traffic per node fell from 14.0 to 4.7 MB/s each way (-66%) in
every run.
- Whether the block time held depended on validation CPU: FAIL with the
default 3 validation tasks on 4-core c6id.2xlarge (avg p75 1,367 ms;
base 1,081 ms), PASS with 7 tasks on the same instances (1,030 ms, every
check including an exact inclusion count), all windows on target on
16-core instances (1,019 to 1,021 ms; inclusion +250).
- At 30 orgs the base failed 1 s / 5,000 TPS (1,298 ms); this change was
not run there.

## What changes

- **One function decides who gets transactions**:
`HerderImpl::computeTxFloodTargets()`. With the leader schedule: the
leaders of the node's current round and of the next one for the slot
after the LCL (two nodes). With #5502's leader terms the node that
closes a ledger leads the next one too; one slot before a new term
starts (every fifth ledger), that term's expected first leader is a
third target, so that it holds the transactions when it pre-validates
and builds its first set. It is the first node of the order seeded with
the LCL's hash, so it is known a full ledger ahead. "Everyone" (the old
flood) when the node is not tracking or more than one ledger behind,
when there is no schedule, and for a pre-protocol-20 LCL. It is
re-evaluated at a ledger close, at a round change, at an externalize and
at loss of sync, and sent to the overlay when it changes (IPC
`SET_TX_TARGETS`). A changed leader schedule only has to change this
function.
- **The accepting node keeps the transaction** (overlay
`flood/tx_router.rs`, at most 100,000 transactions and 300 s) until it
is externalized, removed by Core or expires. When the targets change, it
goes to a newly added target only if it may still be wanted there: it is
younger than 200 ms, or three ledgers externalized on this node without
it since it was accepted, or it is older than 3 s (no ledger closes).
Anything else is most likely in the set in consensus. Nothing is sent
again to a peer that got it less than 60 s ago on the same connection; a
failed send or a reconnected peer is served regardless. A node that
received a transaction as a target keeps it in its mempool and does not
relay it.
- **Fallback to the flood.** If a target is not a directly connected,
identified peer, the node floods its transactions with adverts as
before, so they reach every mempool whatever the topology. A node that
got a transaction directly joins a later flood of it and can serve the
body.
- **Admission check** (`HerderImpl::checkTxAdmission`, in
`recvTransaction`): `checkValid` against the LCL, the check the leader's
build runs on every candidate, and one pending transaction per source
account per accepting node (`TRY_AGAIN_LATER`; `DUPLICATE` for the same
hash). On the base `recvTransaction` checked nothing. The per-account
record goes when a set that uses up the sequence number closes, when the
LCL shows the sequence number used some other way, or after 300 s.
- **Overlays prove which validator they belong to.** The overlay's
libp2p identity is a random key. At startup it sends its peer id to
Core; Core signs `SHA256(networkID || "stellar-overlay-peer-id" ||
peerId)` with the node key; the overlay shows `[nodeID][signature]` to
each peer on a new stream (`/stellar/node-id/1.0.0`, acknowledged,
retried), and the receiver checks it against the peer id QUIC
authenticated. The node key stays in Core.
- **No leader prewarm of its own any more.** The version on the old base
verified arriving signatures between the close and the trigger. #5502's
leader builds once before the trigger and pre-validates during the
previous ballot; bodies that arrive after those pulls are verified in
the build, as on #5502 alone.
- **Options, for measurements:** `TX_FLOOD_TARGET_LEADERS` (default 2; 0
= every validator of the schedule, still direct and without adverts),
`TX_FLOOD_LEGACY_ADVERTS` (the base's flood from the same binary),
`BATCH_EXECUTOR_TASKS` (number of validation tasks; above the physical
core count they run on hyper-thread siblings).
- New metrics: `overlay.flood.tx-direct-sent`, `.tx-direct-sent-bytes`,
`.tx-direct-recv`, `.tx-local-flooded`, `.tx-late-copy` (a transaction
admitted although one of the last few externalized ledgers already has
it), `overlay.node-id.identified`, `.rejected`,
`herder.tx-admission.check`, `.invalid`, `.account-busy`.

## Why ledger outputs cannot change, and what does change

- The change decides which transactions reach a leader's mempool, when,
and which submissions are refused. Building a set from a mempool,
validating it, voting and applying are untouched: a given tx set
produces the same ledgers and meta.
- The admission check reads the LCL on the main thread and writes
nothing.
- The identity binding is domain-separated from everything else the node
key signs.
- **What gets worse, by design:**
- Followers no longer hold transactions before the set arrives, so the
background signature verification finds nothing and the apply-time check
verifies signatures inline (see the results).
- A term's first leader is predicted one ledger ahead. If the ledger in
between costs that node its turn (#5502's forfeit), another node leads
and gets the transactions only at the close. #5502 also lets the second
node of the expected schedule pre-validate; it gets the transactions
when the ledger before closes, not earlier.
- A predicted target that is not a connected, identified peer makes the
node flood for the whole ledger before the term.
- A restarted node forgets the transactions it accepted and had not seen
included; on the base every node held them.
- A transaction submitted through a node that cannot reach both targets
directly costs what it costs today.
- **Left open:**
- A peer can still fill a target's mempool with invalid transactions
that outbid valid ones (the base's exposure, minus the relay). The
leader validates when it builds and drops them then.
- A refused submission returns `{"status":"ERROR"}` without the result
code.
- A pending transaction that dies for a reason other than its sequence
number (expired time bound) holds its account on that node for up to 300
s.
- Fee bumps are accepted and then dropped by the overlay, as on the
base.
- No protocol gating and no version negotiation: a node of the base
ignores the identity stream, and nodes with this change flood toward it.

## Perfnet

### On top of #5502: 16 orgs, 800 ms, 8,000 TPS, c6id.8xlarge

`block_time_check` (perfnet `btc2-slop`), `--org-count 16
--block-time-ms 800 --soroban-tps 8000 --soroban-type sac --classic-tps
0 --simulate-apply-budget-ms 150 --load-duration-s 900 --instance-type
c6id.8xlarge`, 48 validators in four regions, no log saving. The #5502
column is that PR's own run of the same day at the same fixture. The
stack column is build #3685 of `6b3c7ae0e`, which is this PR's head
**without** the changed re-send rule, without the late-copy meter and
without #5502's last commit.

| Windows 1 / 2 / 3 / 4 | #5502 alone (build #3684) | This PR on #5502
(build #3685) |
|---|---:|---:|
| Windows on target | 4 of 4 | **4 of 4** |
| Avg p75 block time, ms | 806.9 / 805.6 / 805.3 / 805.3 | **804.9 /
804.1 / 803.8 / 803.9** |
| Avg p99, ms (limit 1,600) | 1,153 / 1,188 / 1,100 / 1,249 | 1,170 /
1,209 / 1,039 / 1,172 |
| Txs per ledger; sets at the 8,000 cap | 6,450; 3.8% | 6,440; 3.4% |
| **`overlay.byte.read` per node** | **21.2 MB/s** | **6.2 MB/s (-71%)**
|
| `overlay.message.read` per node | 20,500 /s | 12,550 /s |
| Adverts per node | 7,848 /s | 0 |
| Direct sends per offered transaction | none | 3.3 |
| Leader build `herder.txset.build`; builds per ledger | 0.16 s; 1 |
0.15 s; 1 |
| Trigger to first ballot | 0.108 to 0.115 s | 0.100 to 0.118 s |
| Ballot to externalize | 0.29 s | 0.29 s |
| Externalize to close (core logs, mean) | 0.045 to 0.048 s | 0.041 s |
| **Trigger to close** | 0.44 to 0.45 s | **0.43 to 0.44 s** |
| Apply-time check `herder.txset.validate` | 0.080 s | 0.099 s |
| Signature-cache misses in that check, per node | (followers hold the
transactions) | 7,490 /s (44% of the lookups) |
| Validation-cache hit share in the leader's build | 69% | 66.7% |
| Leader terms | exactly 5 ledgers | exactly 5 ledgers, 225 terms, 48
leaders |
| Leader and prepare timeouts, empty sets | 0 | 0 |
| Inclusion, over the 7,680,000 offered | +80 | **+6,610** |
| Error counters, in sync, no fork, still closing | passed | passed |
| Verdict | FAIL on inclusion only | FAIL on inclusion only |

- The base `overlay-v2-shared` at this fixture on the same instances
(reported in #5502): window 1 on target at 846 ms, window 2 missed,
trigger to close 0.67 to 0.69 s.
- **Inclusion.** The check requires exactly the offered count. #5502
alone held 80 more, this run 6,610 more (0.086%): copies of transactions
that reach a node after their ledger closed and are included again,
which only overlay-only mode permits. This run's build re-sent every
pending transaction to newly added targets; the head sends only fresh
ones or ones that missed three ledgers, and the goal-fixture run with it
had 81 on 3,870,249 (0.002%) and counted 688 late copies
(`overlay.flood.tx-late-copy`). The 16-org fixture was not rerun with
the head.
- Every node identified all 47 peers, nothing was flooded, no send
failed, no submission was refused; the admission check takes 0.094 ms
per transaction on the main thread.

### On top of #5502: the goal fixture, 30 orgs, 800 ms, 10,000 TPS,
c6id.8xlarge

The same command with `--org-count 30 --soroban-tps 10000`; 90
validators, 10,080 TPS offered. The stack column is build #3688 of this
PR's head `6ab837c27`. The #5502 column is that PR's own run of its head
`8aa7f6bf0` two hours earlier (some rows are from its first run at this
fixture, marked G).

| Window 1 (the load was stopped after it in both runs) | #5502 alone
(build #3686) | This PR on #5502 (build #3688) |
|---|---:|---:|
| Avg p75 block time, ms (limit 960) | 1,031.6 | **979.6** |
| Avg p99, ms (limit 1,600) | 1,887 | **1,441** |
| Time between ledgers | 1.02 s | 0.90 s (median 0.88 s) |
| The first minute of the load | 802 ms for 30 s, backlog from 47 s |
805 ms over 60 s |
| Sets at the 10,000 cap | 93% | 38.7% |
| Leader builds before the trigger | 0 of 40 | 40 of 55 |
| Leader build; candidates pulled | 0.33 s; 20,200 | 0.24 s; median
8,350 (max 20,200) |
| Trigger to first ballot | 0.58 s | **0.39 s** |
| Ballot to externalize | 0.32 s | 0.32 s |
| Externalize to close | 0.07 s | 0.065 s |
| **Trigger to close** | 0.97 s | **0.77 s** |
| Set at the followers after the proposal | +230 ms (G) | +174 ms |
| Apply-time check `herder.txset.validate` | (see #5502) | 0.144 s |
| Validation-cache hit share in the leader's build | 66.3% (G) | 66.7% |
| **`overlay.byte.read` per node** | **43 MB/s** (G) | **15.7 MB/s** |
| Direct sends per submitted transaction | none | 2.5 to 2.6 while
keeping up, 3.7 in the backlog |
| Leader terms; timeouts | 5 ledgers; none | 5 ledgers; none |
| Submitted / included at the end | 49,855 short | 81 over (backlog
drained) |
| `overlay.flood.tx-late-copy` | n/a | 688, on 14 of 90 nodes |
| Error counters, in sync, no fork, still closing | passed | passed |
| Verdict | FAIL, window 1 (p75 and p99) | FAIL, window 1 (p75 only) |

**Not measured:** the 16-org fixture again with the head's re-send rule;
the base `overlay-v2-shared` at the goal fixture; the 4-core instances
with #5502 underneath; `TX_FLOOD_TARGET_LEADERS=0`; real apply; more
than one run of anything.

### Before the rebase: 16 orgs, 1 s, 5,000 TPS (on `overlay-v2-shared`
at `26bea1150`, without #5502)

All runs: `block_time_check` (perfnet `btc2-slop @ 6dd2064`),
`--block-time-ms 1000 --soroban-tps 5000 --soroban-type sac
--classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900`,
validators in us-east-1, us-west-2, eu-central-1 and ap-southeast-1,
every validator generates load, overlay-only mode (Core does not apply;
execution is a sleep). 5-minute means at the end of window 1, median
node.

| | Base | This PR, 3 tasks | This PR, 7 tasks | This PR, 16 cores |
Base, 30 orgs |
|---|---:|---:|---:|---:|---:|
| Build (Jenkins; commit) | #3678; `0c9a02e56` (tree of `26bea1150`) |
#3680; `dc91c8964` | #3683; `664a35fdc` | #3680; `dc91c8964` | #3678 |
| Validators x instance type | 48 x c6id.2xlarge | 48 x c6id.2xlarge |
48 x c6id.2xlarge | 48 x c6id.8xlarge | 90 x c6id.2xlarge |
| Validation tasks | 3 | 3 | 7 (`BATCH_EXECUTOR_TASKS=7`) | 15 | 3 |
| Verdict | FAIL on inclusion only (+60) | **FAIL**, window 1 | **PASS**
| FAIL on inclusion only (+250) | **FAIL**, window 1 |
| Windows on target | 4 of 4 | 0 of 1 (load stopped) | 4 of 4 | 4 of 4 |
0 of 1 (load stopped) |
| Avg p75 per window, ms | 1,081 / 1,132 / 1,118 / 1,142 | 1,367 | 1,030
/ 1,030 / 1,031 / 1,031 | 1,019 / 1,021 / 1,020 / 1,021 | 1,298 |
| `overlay.byte.read` per node | 13.97 MB/s | 4.77 MB/s | 4.71 MB/s |
4.77 MB/s | 26.69 MB/s |
| `overlay.message.read` per node | 14,804 /s | 9,374 /s | 9,914 /s |
10,085 /s | 33,523 /s |
| Direct sends per offered transaction | none | 9.0 | 3.4 | 3.1 | none |
| Leader build `herder.txset.build` | 0.235 s | 0.374 s | 0.099 s |
0.063 s | 0.314 s |
| Trigger to first ballot | 0.413 s | 0.539 s | 0.300 s | 0.232 s |
0.613 s |
| Ballot to externalize | 0.296 s | 0.290 s | 0.293 s | 0.284 s | 0.387
s |
| Apply-time check `herder.txset.validate` | 0.179 s | 0.277 s | 0.168 s
| 0.078 s | 0.247 s |
| Signature-cache misses in that check | 2,089 /s | 4,550 /s | 4,654 /s
| 4,647 /s | 2,841 /s |
| Externalize to close (core logs, median) | 58 ms | 145 to 210 ms | 44
ms | 28 ms | 61 to 102 ms |
| Ledgers per 5 minutes; txs per ledger | 291; 5,152 | 266; 5,637 | 300;
5,001 | 300; 5,000 | 262; 5,742 |
| Error counters, in sync, no fork, still closing | passed | passed |
passed | passed | passed |

- The base column is run 3 of #5498's record (same fixture, same box,
the evening before); it was not rerun.
- **3 tasks:** 75% of the sets were at the 6,250 cap (1.27 s per ledger)
and 16% were starved (as few as 938 transactions): after a slow ledger
the next trigger is already due and the new leader builds before the
pending transactions reach it.
- **7 tasks and 16 cores:** no starved sets, no backlog; the leader
pulls about 5,000 candidates at its trigger.
- Every run of this PR: every node identified all 47 peers as validators
(0 rejected claims), nothing was flooded, no message was dropped, no
submission was refused; the admission check takes about 0.12 ms per
transaction on the main thread.
- **Inclusion.** The check requires every node to hold exactly the
offered transactions. Base: +60; 16-core run: +250; 7-task run: exact.
In overlay-only mode Core's sequence-number check is off, so a copy of a
finalized transaction that a node admits again is included when that
node next leads (performance-testing#4, not merged, would allow up to
480 here). Which transactions were duplicated in the 16-core run, and
why more than on the base, is **not established**.

**Not measured before the rebase:** this change at 30 orgs; the base
with 7 tasks; real apply (with real execution the check and the
execution compete for the same cores).

## Testing

On `6ab837c27398` (this PR's head, on #5502 at `8aa7f6bf0`), devbox, 32
partitions:
- **Full suite: PASS** (exit 0, 718 s), with libsodium, xdrpp and the
overlay's Rust tests (224 library, 47 binary, 7 end-to-end; 4 ignored).
- **Full suite `--all-versions`: PASS** (exit 0, 735 s).
- Tag run
(`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[simulation]~[.],[loadgen]~[.]`,
which includes #5502's tests): only the three failures recorded for the
base ("SCP Driver" validity-cache hits, "hierarchical topology scales 1
to 3", "core nodes with outer nodes").
- **One unexplained Rust failure:** on an earlier head (`6b3c7ae0e`) one
`make check-rust` run failed 1 of 219 library tests and the output was
not kept, so the test is not known. About twenty later runs of the same
suite passed (apart from failures of tests I was changing at the time).
#5502 does not touch `overlay/`.

New tests:
- `herder/test/TxFloodTargetsTests.cpp` (6 cases):
- In a 4-validator mesh every submitted transaction is applied on every
node with no advert and no demand, and every target list each node sent
to its overlay equals what the schedule functions give for the LCL and
round it was sent at: two round leaders, and a third node only in the
ledger before a new term. A list sent one ledger before a term start
names the node that then leads the term, unless the ledger in between
forfeited. **Mutation-checked** three ways: the router forced to flood
(96 adverts for 32 transactions), the term lookahead removed, and the
lookahead seeded with the wrong hash; each fails the test.
- With `TX_FLOOD_TARGET_LEADERS = 0` each transaction goes to the three
other nodes exactly once.
- With one of three validators down every transaction is still included
and every target list matches the schedule at the node's round,
including rounds after a timeout.
- In overlay-only mode with load from all four nodes, no transaction
hash appears in two ledgers (2,400 transactions; it does not reproduce
the perfnet duplicates, which need network latency).
- The admission check (accepted once per account, duplicate, bad
signature, used and future sequence number, low fee, missing account,
freed by the including ledger, by another transaction of the account, by
a sequence bump, by timeout; overlay-only mode; pregenerated load); the
experiment options.
- Rust `flood/tx_router.rs` (20 unit tests): targets only and following
the targets; a kept transaction follows a target change only while
fresh, after three ledgers without it, or after 3 s
(**mutation-checked**: with the rule disabled five tests fail); a
removal by Core is not a ledger; flood without reachable targets; a
directly received transaction joins a later flood and serves the body.
- Rust `libp2p_overlay/tx_direct_tests.rs` (4 network tests on loopback,
1 unit test): the same over real connections, including a late copy
being counted; forged, replayed and wrong-network identity claims are
rejected, and claims are bounded per connection. `main.rs` (3): the IPC
handlers.

Existing tests changed:
- `overlay/.../dispatch_tests.rs` (2): they fed a body without a demand
and expected a relay; they now register the demand, and one asserts that
an undemanded body is not relayed.
- `main/test/CommandHandlerTests.cpp` "transaction envelope bridge": V1
and fee-bump envelopes are refused again below protocol 13, and the
too-early / too-late cases are back.

Review: four passes of an independent review (two on the old base, one
of the rebase, one of the re-send rule); no consensus or determinism
finding in any. Open from the last two: the flush after an externalize
plans against the outgoing targets (one stray body for transactions that
waited three ledgers); a lost send re-delivers to every held-back
target, not only the lost peer; after a leader failure the round-2
leader lacks transactions older than 200 ms until the ledger closes; the
200 ms bound is a constant, not derived from the block time; no test
covers a watcher, a wrong term prediction, a lagging sender or the
`main.rs` wiring of the ledger count.

Not passing or not run:
- Tests outside the regular suite that fail the same way on the base:
"resilience tests", "cycle4 topology", "asymmetric topology report
cost", and the three above.
- A `--disable-tests` build was not tried; #5495 records that the base
does not build that way.
- No sanitizer run. No tx-meta baseline run (the diff does not touch
apply).

## Commits

Three commits on top of #5502: the code, and two for the README (which
now describes and reports the whole stack: #5502's leader terms and
pre-validation, and this PR). Builds that perfnet ran: #3688 of the head
`6ab837c27` (the goal-fixture run), #3685 of `6b3c7ae0e` (the 16-org
run; that commit is no longer on the branch) and, before the rebase,
#3680 of `dc91c8964` and #3683 of `664a35fdc`.
@SirTyson
SirTyson merged commit 8aa7f6b into stellar:overlay-v2-shared Nov 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant