Skip to content

Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (slower than the base in its one goal-fixture run) - #5503

Closed
SirTyson wants to merge 4 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/leader-build-during-ballot
Closed

SirTyson wants to merge 4 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/leader-build-during-ballot

Conversation

@SirTyson

@SirTyson SirTyson commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Prototype, not for merge as is: its one run at the goal fixture is worse than the base's (1,068 ms against 980 ms, on another instance type; details under Results). On overlay-v2-shared at 5d24d5494 (after #5502 and #5501). It changes when a round leader chooses and executes its transaction set; it does not change apply, commit, tx-set validation, the schedule or anything on the wire.

What it does

A transaction set names the hash of the ledger it goes on top of, so until now the first-round leader could not start building the set for ledger N+1 before N had closed. At 90 validators and 10,000 TPS that build (0.24 s) is the first 0.29 s of every ledger, and the cycle misses 800 ms (#5501's goal-fixture run: 0.89 s on the leader).

  1. The leader selects its next set during the previous ballot. The node that expects to lead round 1 of N+1 already validates its candidates on the state its speculative execution of N left (Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502's pre-validation). That job now goes on, on its last task's thread: one transaction per source account, validity trim, surge pricing, and the set's XDR without a parent (HerderImpl::selectEarlyTxSet, TxSetBuilder::selectPhases / makeUnstamped). When N has closed, the leader only names N as the parent, hashes the set, hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet, TxSetBuilder::stamp). The trigger proposes it unchanged, as it does for a set built before the trigger.
    • It is used only if all of these hold: N closed with exactly the executed value; this node leads round 1 of N+1; the limits the selection assumed (header and Soroban ledger limits) are those of the closed ledger; its time-dependent transactions are valid for the close time the set gets (they are validated again on the closed ledger if the close time's second moved); a build would have to start at once (the trigger is due or less than a build's lead time away); and a build of that size would take time (at least 20 ms at the node's measured cost per transaction; an empty selection is never proposed).
    • Otherwise the ledger gets one build as before, which starts from the job's validation results.
  2. The leader executes its own set as soon as it is prepared, not when it emits its first ballot statement at the trigger (HerderImpl::executePreparedTxSet). The value the trigger proposes is known when the set is. If that execution finds the set invalid before the proposal, the set is dropped and the trigger builds again.
  3. The check of the written set runs in the background (TxSetSelfCheck): the XDR round trip and the validation of the re-parsed set, for sets the herder builds as a leader on protocols that validate sets at apply. The structural validation of the selection and the shape of the XDR are still checked before the set leaves the builder, and throw as before.

makeTxSetFromTransactions is split for this: selection takes the limits (TxSetBuildLimits) and the validation context explicitly instead of reading the LCL through the application, so it can run for a ledger that is not the LCL and off the main thread. Existing callers get the same checks, exceptions and random draws in the same order.

Not done: the set's parent hash before the previous ledger closes. The header hash needs the bucket-list hash after the ledger's batch, which the commit computes in sealLedgerTxnAndStoreInBucketsAndDB (eviction resolution, hot-archive and live batches added in place, in-memory Soroban state, module cache). None of that can be undone if another value externalizes, so computing the hash early means running those steps on copies and installing them at the commit. That is a restructuring of the commit and is left out. With it the set could be pushed during the previous ballot; without it the push starts at the close.

Two faults of the base are fixed on the way:

Results

One run at the goal fixture, and it is negative. Perfnet block_time_check, 30 tier-1 orgs = 90 validators in 4 regions, 800 ms target, 10,080 SAC TPS offered, simulated apply budget 150 ms, 900 s load, n=1 per build. The base's run (S30, #5501's PR head 6ab837c27, whose tree is this PR's base plus README commits) was on 90 x c6id.8xlarge; this PR's run (B1, 18a3a0238) was on 90 x m6id.8xlarge eleven hours later, because c6id.8xlarge could not be had (five launch attempts ended in InsufficientInstanceCapacity). Same CPU model (Xeon 8375C) and core count, 128 GiB instead of 64, other zones: the two runs differ in the build and in the instance type.

Window 1 Base (S30, c6id.8xlarge) This PR (B1, m6id.8xlarge)
Verdict FAIL, window 1 missed on 90/90 (p75) FAIL, window 1 missed on 90/90 (p75 and p99)
avg p75 block time (limit < 960 ms) 980 ms 1,068 ms
avg p99 (limit 1,600 ms) 1,441 ms 1,664 ms
Seconds per ledger 0.90 1.03
First minute of the load 805 ms; 2 of 75 sets at the 10,000 cap 834 ms (median 806); 23 of 72 at the cap, 12 below 6,000
Sets at the cap in the window; candidates per build (median) 39%; 8,350 91%; 20,200
Ledgers whose leader used a set selected early n/a 75 of about 433 (17%); 269 times not ready at the close; 0 discarded
Previous close to set pushed, with an early set n/a at the close (24 ms to complete it; a build takes 0.30 s)
Leader's cycle, previous close to close (sampled leaders) 0.889 s (56 ledgers) 0.928 s (20): 0.744 s with an early set (6), 1.006 s with a build (14)
Background job after the leader's execution pre-validation 0.05 s pre-validation 0.18 s + selection 0.22 s
Builder checks on the build's path round trip 22 ms + final check 28 ms selection check 37 ms (round trip and re-check in the background)
Sets found invalid at apply; self-check failures; timeouts; forks 0; n/a; 0; 0 0; 0 of 432; 0; 0
Overlay traffic per node 15.7 MB/s 14.2 MB/s

What the run shows:

  • The selection is rarely ready. It can only start when the leader has executed the previous ledger, and it takes 0.4 s (I had estimated 0.15 to 0.3 s), about what is left of that ledger's ballot.
  • Where it is ready, the mechanism works: the set is pushed when the previous ledger closes, and those ledgers took 0.74 s against 1.01 s (6 and 14 sampled ledgers).
  • Using it leaves the ledgers unevenly filled. A set selected early after a ledger that was built at its close holds only what arrived in the 0.3 to 0.5 s between the two pulls (4,620, 4,875, 4,787, 5,022 transactions on the sampled leaders; a ledger has to carry about 8,060). What it leaves behind fills the next built set to the cap, a capped set makes its ledger late, and the network was in a backlog within the first minute. I had named the later pull as a cost but reasoned about a steady state in which every pull is one cycle after the last; with early and built sets alternating there is none.
  • Moving the round trip off the path gained little on built sets: the structural check that stays in front of the push takes 37 ms at these sizes, the two checks that moved took 50 ms in the base.
  • How much of the 88 ms between the two runs is the instance type, this cannot say: there is no run of the base on m6id.8xlarge (one more 90-node launch, about 60-75 USD).

Not measured: the 48-validator, 8,000 TPS fixture that the base holds at 804-805 ms (the planned regression run was not launched after this result); anything with a state that changes, time-bounded transactions or a real apply (see below).

What I would change before measuring again (not in this PR): use a set selected early only when it is full (at the phase's capacity or at least as large as the ledger that just closed), which is the backlog, where the build is on the critical path, and never where a build fits before the trigger; and make the job fit into the ballot (decode only what the selection can need; the background thread spends 0.16 s in surge pricing that takes 0.06-0.08 s on the main thread).

Correctness

  • For a given transaction set, ledger state and meta cannot change: nothing in apply, commit or validation is touched, and a speculative execution is still committed only for the exact externalized value.
  • A set selected early is validated by the same code as a built one (checkValid, fee-source totals, source uniqueness, surge pricing), against the state the execution of the previous ledger's value left. Execution is deterministic and the selection is only used if that value closed the ledger; the commit adds upgrades (a value with upgrades is never used), eviction and header hashes, none of which validation reads. The limits are compared with the closed ledger's before the set is used.
  • A wrong set cannot fork: every node validates it at apply and closes the ledger without applying it.
  • Threading: the selection runs on one background thread on immutable inputs and hands its result over once under the job's mutex; a stopped job hands over nothing. Everything that touches herder state runs on the main thread.

Tests

  • 22 new test cases in src/herder/test/ProposalPrevalidationTests.cpp (tag [early-txset]): the set selected during the previous ballot is proposed unchanged and applies; it holds what a build on the closed ledger holds (chained sequence numbers, a drained fee source, a merged account, a fee bump, a Soroban upload, surge pricing, candidates past the pre-validation window, one account in both phases); it is not used when another value or another close time closed the ledger, when the executed set was invalid or carried upgrades, when the limits differ, when a time-bounded transaction is not valid for the final close time, when it is empty or small, or when a build still fits before the trigger; the leader's early execution is committed when its value externalizes, dropped otherwise, restarted when the node votes for another value, and carries the upgrades due; an invalid prepared set is dropped by the leader's own execution, and only a set this node built for that close time counts; the self-check passes, fails on broken sets, drops a set that is not proposed yet, and is recorded for one that is; shutdown during a selection or a check; a 4-node network in which leaders propose sets selected during the previous ballot and every ledger carries exactly the one payment that is valid on top of the previous one.
  • Three existing tests adapted: two counted mempool pulls as builds (the leader's early execution pulls candidates for the ledger after it); "SCP Driver / validateValue txSet cached" depended on which of its two validators leads and failed on the base at --base-instance 1250.
  • Mutation check: 22 mutations of the change, each against the test meant to notice it: 21 killed; the survivor is equivalent (the cluster count dropped from the limits comparison; the compared resource limits already contain it). Not covered by any test: reverting the network-ID fix (ASan only) and the tie-break seed (unused in test builds).
  • ASan: the change's tests and Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502's pre-validation and speculative-apply tests pass with no report (37 test cases).
  • Full suite (657 test cases) and --all-versions: pass, on the commit before the README change.
  • CI on this PR stops in its format check, before compiling: it flags 10 files, all of the base and none touched here (5 under overlay/src, src/overlay/OverlayIPC.cpp, src/overlay/RustOverlayManager.cpp, src/main/Config.cpp, two test files), as it did on Prototype: send transactions to the upcoming round leaders only, with an admission check #5501. No file of this PR is flagged. The builds and test runs above are local (clang-20).
  • A --disable-tests compile fails in herder/HerderImpl.cpp exactly as the base does (six errors in test headers reached through simulation/LoadGenerator.h); the other four touched sources compile without BUILD_TESTS.
  • Two independent reviews (correctness, threading, consensus safety); their findings are fixed; what they left open is in the two bullets on mutation coverage above.

What is prototype-grade and what was not measured

  • Measured once, at one fixture, on another instance type than the base (see Results). The first ledger of a term never had an early set in the sampled data (0 of 4).
  • Not measurable in the perfnet fixture at all (overlay-only mode, simulated apply, no time bounds): validity against a state that changes (unit tests only); contention between the selection thread and a real apply; the re-validation of time-bounded transactions at the close; submit-to-apply latency, which a set selected early raises for transactions that arrive during the rest of the previous ballot.
  • The parent hash is not available before the previous ledger closes, so the push still starts at the close.
  • The check of the written set no longer precedes the push and the proposal. What only the XDR round trip would catch now costs a ledger and the leader's term instead of stopping the builder.
  • The first ledger of a term gains little by design: its leader executed the previous ledger as a follower.
  • No protocol gate is needed (nothing other nodes see changes), but the branch as a whole is a prototype (Prototype: replace SCP nomination with a deterministic leader schedule #5495, Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502).

Garand Tyson added 4 commits November 12, 2026 19:21
…lot and executes its prepared set before the trigger

The first-round leader's build of a transaction set sat between the close of
the previous ledger and its proposal: a set names its parent's header hash,
so the build could not start before that ledger had closed.

1. Selection during the previous ballot. The node that expects to lead round
   1 of ledger N+1 selects its set for N+1 while N is still in its ballot:
   the pre-validation job (which validates candidates on the state the
   node's speculative execution of N left) goes on, on its last task's
   thread, to trim and surge-price what it pulled and to write the set's XDR
   without a parent. When N has closed with the executed value and a build
   would have to start at once (the trigger is due, or less than a build's
   lead time away), the node only names N as the parent, hashes the set,
   hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet).
   The selection is used only if N closed with exactly the executed value,
   the limits it assumed are those of the closed ledger, its time-dependent
   transactions are valid for the close time the set gets, and a build of
   its size would take time (an empty selection is never proposed).
   Otherwise the ledger gets one build as before.

2. The leader executes its own set as soon as it is prepared. The value the
   trigger will propose is known when the set is (HerderImpl::
   executePreparedTxSet), so the ledger goes to the apply thread then, not
   when the first ballot statement is emitted. If that execution finds the
   set invalid before the proposal, the set is dropped.

3. The builder's check of the written set (the XDR round trip and the
   validation of the re-parsed set) runs on a background thread for sets the
   herder builds as a leader on protocols that validate tx sets at apply
   (TxSetSelfCheck). The structure of the selection and the shape of the
   XDR are still checked before the set leaves the builder.

makeTxSetFromTransactions is split for this: selection takes the limits
(TxSetBuildLimits) and the validation context explicitly instead of reading
the LCL through the application, so that it can run for a ledger that is not
the LCL and off the main thread.

Two faults of the base are fixed on the way: the pre-validation's transaction
frames referred to a network ID owned by a task's closure, and a build kept
one transaction per source account within each phase but not across the two
phases (the builder's own check then threw).

No change to apply, commit, tx-set validation or the wire format.
…d the deferred self-check

New test cases (tag [early-txset]) in ProposalPrevalidationTests.cpp. Two
leader-build tests in HerderTests.cpp count builds instead of mempool pulls
(the leader's early execution pulls candidates for the ledger after it), and
"SCP Driver / validateValue txSet cached" no longer depends on which of its
two validators leads (it failed on the base at --base-instance 1250).
…he goal fixture did not help

Describes the sixth experiment and its limits, and reports its run at 90
validators, 800 ms and 10,000 TPS (on m6id.8xlarge; the earlier runs of this
fixture were on c6id.8xlarge): window 1 at 1,068 ms against 980 ms without
it. The set was ready for 17% of the ledgers, and sets selected early after
a built one were short, which filled the following sets to the cap. Not
measured at 48 validators.
@SirTyson
SirTyson force-pushed the perf/leader-build-during-ballot branch from 664bfe8 to 73044be Compare November 12, 2026 22:48
@SirTyson SirTyson changed the title Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (no gain at the goal fixture yet) Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (slower than the base in its one goal-fixture run) Nov 12, 2026
@SirTyson SirTyson closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant