Skip to content

Prototype: the leader selects its next tx set from the current one before it closes, and pulls less for a build (real-apply results, README) - #5505

Draft
SirTyson wants to merge 12 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/txset-build-off-critical-path
Draft

SirTyson wants to merge 12 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/txset-build-off-critical-path

Conversation

@SirTyson

@SirTyson SirTyson commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Prototype, draft. Both steps work and no set was invalid or short, but the block time gain is not established: at 90 validators and 8,000 TPS with real apply the leader's tx set build went from 176 to 121 ms (-31%) and the time per ledger from 873 to 858 ms (-1.7%, one run each, inside the spread of two runs of one point). On overlay-v2-shared at 5d24d5494. It changes which candidates a round leader looks at and when it selects them; it does not change apply, commit, the validation of a tx set, the schedule or anything on the wire.

Rebased on 2026-11-14 onto overlay-v2-shared at d9ae5fcec, after #5504 was merged. The copy of #5504's README commit is gone; the 12 commits that remain are unchanged in content, and the head's tree is identical to the old head's (c8417bc90). The commit ids in the text below are the ones before the rebase, under which the builds, test runs and reviews were made; they stay reachable as branch perf/txset-build-off-critical-path-measured of SirTyson/stellar-core. Old to new: 761d3727a -> cbd2a5c87, 1782051a3 -> 39ea25fa3, 49ae8e4ab -> 9724d343b, c426295d2 -> 650811175, 54260eafb -> e0101b90d, e5eaa8e1c -> 754a7ba0a, 64e8f5944 -> fc949267b, 16d0f7bc5 -> 03d8857c0, 3973b9321 -> 030f4dbd5, d33af26e3 -> 1825d266f, f19140f89 -> 09cdc0aae, README c8417bc90 -> aeee8a5ca.

What this diff holds (12 commits):

  • 761d3727a, 1782051a3, 49ae8e4ab, c426295d2: test-only load generator modes for real apply (sac_setup, sac_payment: payments of the native asset through its contract, each to a fresh address), and a 150-ledger wait for a sac_payment run's last payments. The first three are the branch perf/loadgen-real-sac that the README's real-apply runs were made with.
  • 54260eafb: a hit in the bucket list's account cache is read under the shared lock.
  • e5eaa8e1c, 64e8f5944, 16d0f7bc5: the three commits of Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (slower than the base in its one goal-fixture run) #5503 (closed), unchanged. This PR replaces that attempt's selection and keeps its builder split and the leader's early execution of its prepared set.
  • 3973b9321: the change, and the commit every network run below was made on.
  • d33af26e3: two rules for when a short phase pulls deeper, corrected after the last review (see Review). They did not come into play in the measured load.
  • f19140f89: clang-format on two lines of the load generator commits.
  • The last commit is this PR's README update. (README: real apply at 90 validators and 800 ms, and what limits it #5504, which this PR was stacked on, is merged.)

What it does

A leader builds its tx set on the main thread when the previous ledger has closed. With real apply at 90 validators and 8,000 TPS that build is 0.26 s of a 1.16 s cycle (README, "What limits real apply").

  1. A smaller build. A build pulls the set's capacity plus 10% from the mempool (EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT), not twice the capacity. A phase that lost candidates (invalid, or a second transaction of an account) and is short pulls again, twice as deep, as long as a pull brings it something new, up to three pulls. The structural check of the selection runs beside the XDR write and ends before the set is handed to the overlay. A leader that executes its own vote does not read the written set back (that execution validates the same bytes); if the execution does not start, the read-back runs in the background.

  2. The next set is selected before the current ledger closes. The node first in line to lead ledger N+1 starts a background job at its vote for ledger N (src/herder/EarlyTxSetSelection.{h,cpp}). The job takes N's tx set as applied (sequence numbers, fees), validates the mempool's top candidates on that view in parallel and selects the phases. When N has closed with that value, TxSetBuilder::completeSelection on the main thread:

    • validates every selected transaction on the closed ledger, for the set's real close time, with the function a follower's validation uses, and removes what fails;
    • tops the set up: the job's leftovers, then what the mempool received since, validated the same way with the same fee-source totals, one per source account; the classic phase is selected again, the parallel Soroban phase is extended in place (ParallelPhaseExtender);
    • computes the base fees by a build's rule, runs the structural check and writes the set.

    A selection for another value, with other limits, after an upgrade, or one that is not ready, is dropped and the set is built from scratch. EXPERIMENTAL_EARLY_TX_SET_SELECTION (default true) turns it off.

The two requirements this was built to:

  • (a) No invalid set, whatever the overlap between ledger N's transactions and the candidates for N+1. Nothing of the prediction reaches a proposal unverified: the validity pass and the top-up validate on the closed ledger, and the structural check (checkSelection, what ApplicableTxSetFrame::checkValid checks) is awaited before the set leaves the builder.
  • (b) Sets as full as a build would make them. A phase is full only at its capacity; every completion with room looks at the mempool's top; a phase that lost candidates looks deeper.

Results

Perfnet block_time_check, real apply (SAC payments, no conflicts), 800 ms target, 16 clusters, m6id.8xlarge with local NVMe, 300 s judged load, one run per point. Every run has the transaction meta off (METADATA_DEBUG_LEDGERS=0, DISABLE_TX_META_FOR_TESTING=true), a 4 GB account cache and 256-byte bucket index pages (BUCKETLIST_DB_MEMORY_FOR_CACHING=4096, BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT=8). Base: 54260eafb (this branch below the #5503 commits). Change: 3973b9321.

90 validators, 8,000 TPS Base This PR
Verdict FAIL: 147 of 2,880,000 payments never included (4 generators report it); block time inside the limits PASS, all 2,880,000 included
Time per ledger: mean / p75 / p99 873 / 928 / 1,100 ms 858 / 908 / 1,107 ms
Block time p75 / p99, median node, two windows 946 / 1,143 and 888 / 1,125 ms 937 / 1,151 and 939 / 1,151 ms
Leader's tx set build 176 ms 121 ms (-31%)
Sets completed from an early selection 177 of 190 (93%); 9 not ready, 4 not started
Completion: validity pass / top-up / XDR write with the check beside it 16 / 60 / 25 to 27 ms
Selection ready before it is needed; time it takes 313 ms; 142 ms
Leader: set built to consensus 468 ms 514 ms
Follower: tx set fetch 98 ms 155 ms
Consensus to close, leader / follower 196 / 220 ms 198 / 210 ms
Signature checks inside builds 452,437 32,107
Selected transactions removed at the close; sets short with more in the mempool; refill pulls 0; 0; 0
Sets found invalid at apply; executions dropped; payments failed 0; 0; 0 0; 0; 0

On 6 validators at 8,000 TPS (both on target, verdict PASS for both): build 144 -> 94 ms, 88% of the sets from an early selection, time per ledger 802 and 801 ms.

The base at 6,000 TPS and 90 validators (same settings): 801 ms per ledger, p75 805 to 808 ms, on target with room; so the comparison was made at 8,000 TPS.

What the runs show:

  • No invalid set and no short set, on 6 and on 90 validators. The load has no overlap between consecutive sets (no selected transaction was removed), so the removals and the refill are covered by unit tests only.
  • The build's 55 ms do not arrive in the block time, and the reason is in the base, not in the change. The base pushes its set to the peers as soon as the XDR is written and only then reads it back and runs its final check (51 ms): that work passes while the set is already on its way. The change has nothing left to do after the push, so its followers wait 57 ms longer for the set, counted from the proposal. What counts is the time from the close of N until the XDR of N+1's set exists: about 125 ms in the base, about 110 ms for a completion.
  • The selection is made too early. It is pulled about 0.45 s before the close, so a third of the final set arrives after it and is pulled, decoded, validated and added on the main thread at the close (60 ms). The start is timed by the time between a node's vote and its own build, which a node measures only when it builds: 5 times in 450 ledgers at 90 validators.
  • 15 ms is about what two runs of one point differed by in the earlier real-apply series (4 and 14 ms).
  • Both base runs lost payments (59 at 6,000 TPS, 147 at 8,000 TPS); this PR's run lost none. One run, and nothing in the change is meant to affect it.

Settings (6 validators, 8,000 TPS, per payment; README for the table): every live bucket in memory is the fastest apply (execution 33.7 us against 44.3 us with the meta off alone) but the close stalls for up to 1.7 s at bucket spills; 256-byte index pages give 38.0 us without stalls; the account cache with its hits under the shared lock no longer slows the tx set validation (101 -> 73 ms) and adds nothing measurable on top of the small pages (37.3 us).

What I would do next (not in this PR): time the selection's start on every node at every ledger so that it pulls about 0.2 s before the close; a second, small pull by the job just before it selects; and, the largest piece, push the selected transactions to the peers during N's ballot and send only the difference at the close (the 0.15 s of delivery are now the largest part of the cycle that is neither ballot nor apply).

The README has the full tables, the method and the limits ("Real apply with the meta off, and the tx set build off the critical path").

Correctness

  • For a given tx set, ledger state and meta cannot change: nothing in apply, commit or set validation is touched. The cache change reads immutable bucket entries under the lock that guards the map; only the eviction order differs.
  • A set proposed by a completion has passed, on the closed ledger: getInvalidTxListWithErrors for every transaction it kept, trimInvalid with the same fee-source totals for every one it added, the one-per-account rule across both phases, and checkSelection on the final phases. A throw in any of it falls back to a build.
  • A wrong set cannot fork: every node validates the set at apply and closes an empty ledger if it is invalid (experiment 3).
  • Threading: the job's tasks read immutable inputs (the pulled bytes, the snapshot of the last closed ledger, N's tx set) and write only to the job, under its mutex; a stopped job hands nothing over. Completion and everything that touches herder state run on the main thread. The structural check beside the XDR write is joined on every exit.

Tests

  • New: src/herder/test/EarlyTxSetTests.cpp (the overlap cases: next sequence numbers, a drained or merged account, changed signers, fee bumps, a failed transaction of N, a randomized overlap compared with a build from scratch; each with a section that skips the validity pass and shows the set would be invalid; top-up, refill, fee sources, base fees, the Soroban extender, close-time-dependent candidates, another value or other limits closing the ledger, a job that is late, shutdown, a network test). ProposalPrevalidationTests.cpp and HerderTests.cpp adapted; 10 test cases of the replaced mechanism removed.
  • Full suite (32 partitions, lints included, clang 20): passes on 3973b9321 (668 test cases) and, with --all-versions, on 3973b9321, on d33af26e3 and on the source tree of this PR's head (669 test cases).
  • CI has not compiled this PR. Both build-linux jobs stop in the format check, on 9 files of the base that this PR does not touch (5 of the Rust overlay, TxFloodTargetsTests.cpp, CommandHandlerTests.cpp, OverlayIPC.cpp, RustOverlayManager.cpp), as they did for Prototype: send transactions to the upcoming round leaders only, with an admission check #5501 and Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (slower than the base in its one goal-fixture run) #5503. So there is no gcc build of this change.
  • Address sanitizer: the three tags of this change pass (on 3973b9321). No thread sanitizer run: the build box cannot build one (the nightly toolchain has no rust-src).
  • Mutation run on 3973b9321: 18 of 19 mutations of the safety and fullness rules are caught (validity pass removed, wrong close time, wrong value or limits, fee-source totals, one-per-account, extender limits, refill); the survivor is equivalent.
  • A --disable-tests build fails as the base's does (HerderImpl.cpp reaches test headers through simulation/LoadGenerator.h); the other objects this PR touches compile without tests.
  • clang-format 20 on the 27 source files this PR changes: clean.

Review

Three independent reviews (of work-in-progress, of the first squashed head and of 3973b9321): no finding on consensus safety, determinism, data races or lifetimes. Their fullness findings are fixed; the last two, both from rules added for cost after the second review, are fixed in d33af26e3 with tests that fail on the old rules:

  • eight or more valid leftovers that could not be added closed a phase, also when they only had nowhere to go;
  • a phase that lost candidates looked deeper only if the mempool's top held a phase's capacity of its kind.

Known limits

  • The mempool has one fee order for both phases. A phase with no candidate in the top (capacity plus the margin) gets none, where a build looked twice as deep before. Needs a pull per phase.
  • A completed Soroban phase is extended transaction by transaction; clusters are never merged and no stage is added, so it can be packed less tightly than a built one.
  • A completed phase at its capacity that lost nothing keeps the selection's surge pricing verdict; if everything it left out is invalid on the closed ledger, the set charges the lowest included fee where a build would charge the base fee. The set is valid either way.
  • A phase that lost a candidate and is short pulls once more even where that is futile.
  • Untested: the base-fee flags across two re-selections, the fallback when no thread can be started for the structural check, concurrent readers and writers of the account cache. The network test accepts zero completions (its jobs run in real time, the network in virtual time).
  • Not measured: classic or mixed loads, conflicting Soroban transactions, 10,000 TPS, the change at 6,000 TPS, and the last commit on a network.

Checklist

  • Reviewed the contributing document
  • Rebased on top of overlay-v2-shared at d9ae5fcec (no merge commits)
  • Ran clang-format (20.1.8 on the changed files; the build box has no v8)
  • Compiles
  • Ran all tests
  • If change impacts performance, include supporting evidence per the performance document

SirTyson and others added 12 commits November 14, 2026 20:47
…t is settable (completiontimeout)

A generator of the real-apply load reported failure although every one of
its payments was applied: after its last submission loadgen waits 20
ledgers for its accounts, and on a 90-validator network 1 to 9 of 90
generators had a last batch that was included 25 to 70 s later (payments
go to the round leaders only, without an acknowledgement). The run then
got no verdict on inclusion.

SAC_PAYMENT now waits 150 ledgers by default, and generateload takes
completiontimeout=<ledgers> for any mode. Other modes keep 20.
A lookup in the per-bucket account cache (BUCKETLIST_DB_MEMORY_FOR_CACHING)
took the cache's lock exclusively even for a hit, because
RandomEvictionCache::maybeGet updates the entry's access time and the
hit/miss counters. Ledger apply loads accounts from many worker threads at
once, and every load asks the cache of each disk-indexed bucket on its way
down the bucket list, so with the cache on the workers' account loads
queued up on that lock (measured on a 6-validator network at 8,000 SAC
payments per second: 29 us per load with the cache against 14 us without).

RandomEvictionCache gets peek(), a const lookup that records nothing, and
LiveBucketIndex::getCachedEntry uses it under the shared lock. Inserting an
entry still takes the lock exclusively. A hit no longer makes its entry
younger for eviction: when a cache is smaller than its bucket's accounts,
eviction goes by insertion time. A cache sized to hold all accounts (the
case this is for) never evicts.
…lot and executes its prepared set before the trigger

The first-round leader's build of a transaction set sat between the close of
the previous ledger and its proposal: a set names its parent's header hash,
so the build could not start before that ledger had closed.

1. Selection during the previous ballot. The node that expects to lead round
   1 of ledger N+1 selects its set for N+1 while N is still in its ballot:
   the pre-validation job (which validates candidates on the state the
   node's speculative execution of N left) goes on, on its last task's
   thread, to trim and surge-price what it pulled and to write the set's XDR
   without a parent. When N has closed with the executed value and a build
   would have to start at once (the trigger is due, or less than a build's
   lead time away), the node only names N as the parent, hashes the set,
   hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet).
   The selection is used only if N closed with exactly the executed value,
   the limits it assumed are those of the closed ledger, its time-dependent
   transactions are valid for the close time the set gets, and a build of
   its size would take time (an empty selection is never proposed).
   Otherwise the ledger gets one build as before.

2. The leader executes its own set as soon as it is prepared. The value the
   trigger will propose is known when the set is (HerderImpl::
   executePreparedTxSet), so the ledger goes to the apply thread then, not
   when the first ballot statement is emitted. If that execution finds the
   set invalid before the proposal, the set is dropped.

3. The builder's check of the written set (the XDR round trip and the
   validation of the re-parsed set) runs on a background thread for sets the
   herder builds as a leader on protocols that validate tx sets at apply
   (TxSetSelfCheck). The structure of the selection and the shape of the
   XDR are still checked before the set leaves the builder.

makeTxSetFromTransactions is split for this: selection takes the limits
(TxSetBuildLimits) and the validation context explicitly instead of reading
the LCL through the application, so that it can run for a ledger that is not
the LCL and off the main thread.

Two faults of the base are fixed on the way: the pre-validation's transaction
frames referred to a network ID owned by a task's closure, and a build kept
one transaction per source account within each phase but not across the two
phases (the builder's own check then threw).

No change to apply, commit, tx-set validation or the wire format.
…d the deferred self-check

New test cases (tag [early-txset]) in ProposalPrevalidationTests.cpp. Two
leader-build tests in HerderTests.cpp count builds instead of mempool pulls
(the leader's early execution pulls candidates for the ledger after it), and
"SCP Driver / validateValue txSet cached" no longer depends on which of its
two validators leads (it failed on the base at --base-instance 1250).
…fore it closes, and pulls less for a build

Two changes to how a leader gets its transaction set, on top of the
reused parts of the closed attempt (builder split, deferred check of the
written set, early execution of a prepared set).

1. A smaller build. A build pulls the capacity of a set plus a margin
   (EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT, default 10) from the mempool
   instead of twice the capacity. If a phase loses candidates (invalid, or
   a second transaction of an account) and ends below its capacity while
   the mempool has more, the build pulls again, twice as much, at most
   three times. The structural check of the selection runs on a second
   thread beside the XDR write and is awaited before the set is handed to
   anyone. The re-read of the written set is left to the node's own
   execution of its value where it executes its votes, and runs in the
   background otherwise.

2. Early selection (EXPERIMENTAL_EARLY_TX_SET_SELECTION, default on). When
   a node votes for a value for ledger N and expects to lead round 1 of
   N+1, it plans a background job that selects the set for N+1 while N is
   in its ballot: on the last closed ledger (N-1) and a prediction of what
   N's set does to it (sequence numbers consumed, full fees charged). The
   job starts as late as it can and still finish before the set is needed.

   The prediction is never trusted. At the build for N+1, if N closed with
   exactly the value the job was for, the selection is completed: every
   one of its transactions is validated against N as it closed, with the
   set's close time, by the function that validates a received set; what
   fails is removed; while a phase has room, further candidates (what the
   job did not select, then what the mempool holds that the job did not
   see) are validated the same way and added, to a parallel Soroban phase
   one by one (ParallelPhaseExtender), to a sequential phase by selecting
   it again; base fees are set by the rule of a build; and the set goes
   through the same checks and hand-off as a built one. Anything else (N
   closed with another value, with upgrades, the job not finished, a
   failed check) is a build.

Tests: herder/test/EarlyTxSetTests.cpp. A completed set is validated in
full on the closed ledger, read back from its bytes, for: the candidates
being exactly N's transactions; the same accounts with their next
sequence numbers; random overlaps; a transaction of N failing; an account
drained, merged away or re-keyed by N; fee bumps; N's set skipped at
apply; N closing with another value, close time or an upgrade; a job
that is late, still running, or overlapping the next one; time bounds.
Three of them show the set invalid when the validity pass is switched
off. Fullness: top-up with late arrivals, refill after removals and for
builds with 0 to 90% invalid candidates, Soroban top-up and conflicts,
base fees after removal and refill.

Nothing in apply, commit, set validation or the wire format changes.
…ne that lost candidates looks deeper whatever the mempool's top holds of its kind

The stop rules of the completion's top-up and of a build's refill had two
shortcuts that could leave a phase short while the mempool held valid
transactions for it:
- eight or more valid left-overs that could not be added closed the phase,
  also when they only had nowhere to go (a full cluster of one entry, a lane
  of its own) and later arrivals would have fit;
- a phase that lost candidates pulled deeper only if the mempool's top held
  at least the phase's capacity of its kind (compared in transactions against
  operations), which a mempool top of the other phase prevents.

Now, as in a build: a phase stops at its capacity; after one look at the
mempool's top if it lost nothing (lost: removed by the validity pass, invalid,
or of an account already in the set; a valid candidate that did not fit is
not a loss); and otherwise when a deeper pull holds nothing new of its kind.

Tests: many left-overs with nowhere to go; a completion and a build with a
mempool top of the other phase.
Two lines of the load generator commits that clang-format 20 (CI's version)
breaks differently. No other change.
…ixth experiment (the next tx set selected before the close)

- What this branch is: experiment 6 (a smaller build; the leader of ledger
  N+1 selects its set from ledger N's set during N's ballot and completes it
  on the closed ledger), with its two supporting changes.
- Results: the bucket list settings on 6 validators; the base with the meta
  off at 90 validators (801 ms per ledger at 6,000 TPS, 873 ms at 8,000 TPS);
  experiment 6 at 8,000 TPS (build 176 -> 121 ms, 858 ms per ledger, no
  invalid or short set), and why the shorter build does not show in the
  block time.
- How it was measured, limits of the numbers, known limitations, test status.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant