Repository navigation
Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (slower than the base in its one goal-fixture run) - #5503
Closed
SirTyson wants to merge 4 commits into
Conversation
added 4 commits
November 12, 2026 19:21
…lot and executes its prepared set before the trigger The first-round leader's build of a transaction set sat between the close of the previous ledger and its proposal: a set names its parent's header hash, so the build could not start before that ledger had closed. 1. Selection during the previous ballot. The node that expects to lead round 1 of ledger N+1 selects its set for N+1 while N is still in its ballot: the pre-validation job (which validates candidates on the state the node's speculative execution of N left) goes on, on its last task's thread, to trim and surge-price what it pulled and to write the set's XDR without a parent. When N has closed with the executed value and a build would have to start at once (the trigger is due, or less than a build's lead time away), the node only names N as the parent, hashes the set, hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet). The selection is used only if N closed with exactly the executed value, the limits it assumed are those of the closed ledger, its time-dependent transactions are valid for the close time the set gets, and a build of its size would take time (an empty selection is never proposed). Otherwise the ledger gets one build as before. 2. The leader executes its own set as soon as it is prepared. The value the trigger will propose is known when the set is (HerderImpl:: executePreparedTxSet), so the ledger goes to the apply thread then, not when the first ballot statement is emitted. If that execution finds the set invalid before the proposal, the set is dropped. 3. The builder's check of the written set (the XDR round trip and the validation of the re-parsed set) runs on a background thread for sets the herder builds as a leader on protocols that validate tx sets at apply (TxSetSelfCheck). The structure of the selection and the shape of the XDR are still checked before the set leaves the builder. makeTxSetFromTransactions is split for this: selection takes the limits (TxSetBuildLimits) and the validation context explicitly instead of reading the LCL through the application, so that it can run for a ledger that is not the LCL and off the main thread. Two faults of the base are fixed on the way: the pre-validation's transaction frames referred to a network ID owned by a task's closure, and a build kept one transaction per source account within each phase but not across the two phases (the builder's own check then threw). No change to apply, commit, tx-set validation or the wire format.
…d the deferred self-check New test cases (tag [early-txset]) in ProposalPrevalidationTests.cpp. Two leader-build tests in HerderTests.cpp count builds instead of mempool pulls (the leader's early execution pulls candidates for the ledger after it), and "SCP Driver / validateValue txSet cached" no longer depends on which of its two validators leads (it failed on the base at --base-instance 1250).
…e validated at apply
…he goal fixture did not help Describes the sixth experiment and its limits, and reports its run at 90 validators, 800 ms and 10,000 TPS (on m6id.8xlarge; the earlier runs of this fixture were on c6id.8xlarge): window 1 at 1,068 ms against 980 ms without it. The set was ready for 17% of the ledgers, and sets selected early after a built one were short, which filled the following sets to the cap. Not measured at 48 validators.
SirTyson
force-pushed
the
perf/leader-build-during-ballot
branch
from
November 12, 2026 22:48
664bfe8 to
73044be
Compare
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, not for merge as is: its one run at the goal fixture is worse than the base's (1,068 ms against 980 ms, on another instance type; details under Results). On
overlay-v2-sharedat5d24d5494(after #5502 and #5501). It changes when a round leader chooses and executes its transaction set; it does not change apply, commit, tx-set validation, the schedule or anything on the wire.What it does
A transaction set names the hash of the ledger it goes on top of, so until now the first-round leader could not start building the set for ledger N+1 before N had closed. At 90 validators and 10,000 TPS that build (0.24 s) is the first 0.29 s of every ledger, and the cycle misses 800 ms (#5501's goal-fixture run: 0.89 s on the leader).
HerderImpl::selectEarlyTxSet,TxSetBuilder::selectPhases/makeUnstamped). When N has closed, the leader only names N as the parent, hashes the set, hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet,TxSetBuilder::stamp). The trigger proposes it unchanged, as it does for a set built before the trigger.HerderImpl::executePreparedTxSet). The value the trigger proposes is known when the set is. If that execution finds the set invalid before the proposal, the set is dropped and the trigger builds again.TxSetSelfCheck): the XDR round trip and the validation of the re-parsed set, for sets the herder builds as a leader on protocols that validate sets at apply. The structural validation of the selection and the shape of the XDR are still checked before the set leaves the builder, and throw as before.makeTxSetFromTransactionsis split for this: selection takes the limits (TxSetBuildLimits) and the validation context explicitly instead of reading the LCL through the application, so it can run for a ledger that is not the LCL and off the main thread. Existing callers get the same checks, exceptions and random draws in the same order.Not done: the set's parent hash before the previous ledger closes. The header hash needs the bucket-list hash after the ledger's batch, which the commit computes in
sealLedgerTxnAndStoreInBucketsAndDB(eviction resolution, hot-archive and live batches added in place, in-memory Soroban state, module cache). None of that can be undone if another value externalizes, so computing the hash early means running those steps on copies and installing them at the commit. That is a restructuring of the commit and is left out. With it the set could be pushed during the previous ballot; without it the push starts at the close.Two faults of the base are fixed on the way:
Results
One run at the goal fixture, and it is negative. Perfnet
block_time_check, 30 tier-1 orgs = 90 validators in 4 regions, 800 ms target, 10,080 SAC TPS offered, simulated apply budget 150 ms, 900 s load, n=1 per build. The base's run (S30, #5501's PR head6ab837c27, whose tree is this PR's base plus README commits) was on 90 x c6id.8xlarge; this PR's run (B1,18a3a0238) was on 90 x m6id.8xlarge eleven hours later, because c6id.8xlarge could not be had (five launch attempts ended inInsufficientInstanceCapacity). Same CPU model (Xeon 8375C) and core count, 128 GiB instead of 64, other zones: the two runs differ in the build and in the instance type.What the run shows:
Not measured: the 48-validator, 8,000 TPS fixture that the base holds at 804-805 ms (the planned regression run was not launched after this result); anything with a state that changes, time-bounded transactions or a real apply (see below).
What I would change before measuring again (not in this PR): use a set selected early only when it is full (at the phase's capacity or at least as large as the ledger that just closed), which is the backlog, where the build is on the critical path, and never where a build fits before the trigger; and make the job fit into the ballot (decode only what the selection can need; the background thread spends 0.16 s in surge pricing that takes 0.06-0.08 s on the main thread).
Correctness
checkValid, fee-source totals, source uniqueness, surge pricing), against the state the execution of the previous ledger's value left. Execution is deterministic and the selection is only used if that value closed the ledger; the commit adds upgrades (a value with upgrades is never used), eviction and header hashes, none of which validation reads. The limits are compared with the closed ledger's before the set is used.Tests
src/herder/test/ProposalPrevalidationTests.cpp(tag[early-txset]): the set selected during the previous ballot is proposed unchanged and applies; it holds what a build on the closed ledger holds (chained sequence numbers, a drained fee source, a merged account, a fee bump, a Soroban upload, surge pricing, candidates past the pre-validation window, one account in both phases); it is not used when another value or another close time closed the ledger, when the executed set was invalid or carried upgrades, when the limits differ, when a time-bounded transaction is not valid for the final close time, when it is empty or small, or when a build still fits before the trigger; the leader's early execution is committed when its value externalizes, dropped otherwise, restarted when the node votes for another value, and carries the upgrades due; an invalid prepared set is dropped by the leader's own execution, and only a set this node built for that close time counts; the self-check passes, fails on broken sets, drops a set that is not proposed yet, and is recorded for one that is; shutdown during a selection or a check; a 4-node network in which leaders propose sets selected during the previous ballot and every ledger carries exactly the one payment that is valid on top of the previous one.--base-instance 1250.--all-versions: pass, on the commit before the README change.overlay/src,src/overlay/OverlayIPC.cpp,src/overlay/RustOverlayManager.cpp,src/main/Config.cpp, two test files), as it did on Prototype: send transactions to the upcoming round leaders only, with an admission check #5501. No file of this PR is flagged. The builds and test runs above are local (clang-20).--disable-testscompile fails inherder/HerderImpl.cppexactly as the base does (six errors in test headers reached throughsimulation/LoadGenerator.h); the other four touched sources compile withoutBUILD_TESTS.What is prototype-grade and what was not measured