Repository navigation
Prototype: the leader selects its next tx set from the current one before it closes, and pulls less for a build (real-apply results, README) - #5505
Draft
SirTyson wants to merge 12 commits into
Conversation
SirTyson
force-pushed
the
perf/txset-build-off-critical-path
branch
2 times, most recently
from
November 13, 2026 23:57
5683e7d to
c8417bc
Compare
6 tasks done
…t is settable (completiontimeout) A generator of the real-apply load reported failure although every one of its payments was applied: after its last submission loadgen waits 20 ledgers for its accounts, and on a 90-validator network 1 to 9 of 90 generators had a last batch that was included 25 to 70 s later (payments go to the round leaders only, without an acknowledgement). The run then got no verdict on inclusion. SAC_PAYMENT now waits 150 ledgers by default, and generateload takes completiontimeout=<ledgers> for any mode. Other modes keep 20.
A lookup in the per-bucket account cache (BUCKETLIST_DB_MEMORY_FOR_CACHING) took the cache's lock exclusively even for a hit, because RandomEvictionCache::maybeGet updates the entry's access time and the hit/miss counters. Ledger apply loads accounts from many worker threads at once, and every load asks the cache of each disk-indexed bucket on its way down the bucket list, so with the cache on the workers' account loads queued up on that lock (measured on a 6-validator network at 8,000 SAC payments per second: 29 us per load with the cache against 14 us without). RandomEvictionCache gets peek(), a const lookup that records nothing, and LiveBucketIndex::getCachedEntry uses it under the shared lock. Inserting an entry still takes the lock exclusively. A hit no longer makes its entry younger for eviction: when a cache is smaller than its bucket's accounts, eviction goes by insertion time. A cache sized to hold all accounts (the case this is for) never evicts.
…lot and executes its prepared set before the trigger The first-round leader's build of a transaction set sat between the close of the previous ledger and its proposal: a set names its parent's header hash, so the build could not start before that ledger had closed. 1. Selection during the previous ballot. The node that expects to lead round 1 of ledger N+1 selects its set for N+1 while N is still in its ballot: the pre-validation job (which validates candidates on the state the node's speculative execution of N left) goes on, on its last task's thread, to trim and surge-price what it pulled and to write the set's XDR without a parent. When N has closed with the executed value and a build would have to start at once (the trigger is due, or less than a build's lead time away), the node only names N as the parent, hashes the set, hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet). The selection is used only if N closed with exactly the executed value, the limits it assumed are those of the closed ledger, its time-dependent transactions are valid for the close time the set gets, and a build of its size would take time (an empty selection is never proposed). Otherwise the ledger gets one build as before. 2. The leader executes its own set as soon as it is prepared. The value the trigger will propose is known when the set is (HerderImpl:: executePreparedTxSet), so the ledger goes to the apply thread then, not when the first ballot statement is emitted. If that execution finds the set invalid before the proposal, the set is dropped. 3. The builder's check of the written set (the XDR round trip and the validation of the re-parsed set) runs on a background thread for sets the herder builds as a leader on protocols that validate tx sets at apply (TxSetSelfCheck). The structure of the selection and the shape of the XDR are still checked before the set leaves the builder. makeTxSetFromTransactions is split for this: selection takes the limits (TxSetBuildLimits) and the validation context explicitly instead of reading the LCL through the application, so that it can run for a ledger that is not the LCL and off the main thread. Two faults of the base are fixed on the way: the pre-validation's transaction frames referred to a network ID owned by a task's closure, and a build kept one transaction per source account within each phase but not across the two phases (the builder's own check then threw). No change to apply, commit, tx-set validation or the wire format.
…d the deferred self-check New test cases (tag [early-txset]) in ProposalPrevalidationTests.cpp. Two leader-build tests in HerderTests.cpp count builds instead of mempool pulls (the leader's early execution pulls candidates for the ledger after it), and "SCP Driver / validateValue txSet cached" no longer depends on which of its two validators leads (it failed on the base at --base-instance 1250).
…e validated at apply
…fore it closes, and pulls less for a build Two changes to how a leader gets its transaction set, on top of the reused parts of the closed attempt (builder split, deferred check of the written set, early execution of a prepared set). 1. A smaller build. A build pulls the capacity of a set plus a margin (EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT, default 10) from the mempool instead of twice the capacity. If a phase loses candidates (invalid, or a second transaction of an account) and ends below its capacity while the mempool has more, the build pulls again, twice as much, at most three times. The structural check of the selection runs on a second thread beside the XDR write and is awaited before the set is handed to anyone. The re-read of the written set is left to the node's own execution of its value where it executes its votes, and runs in the background otherwise. 2. Early selection (EXPERIMENTAL_EARLY_TX_SET_SELECTION, default on). When a node votes for a value for ledger N and expects to lead round 1 of N+1, it plans a background job that selects the set for N+1 while N is in its ballot: on the last closed ledger (N-1) and a prediction of what N's set does to it (sequence numbers consumed, full fees charged). The job starts as late as it can and still finish before the set is needed. The prediction is never trusted. At the build for N+1, if N closed with exactly the value the job was for, the selection is completed: every one of its transactions is validated against N as it closed, with the set's close time, by the function that validates a received set; what fails is removed; while a phase has room, further candidates (what the job did not select, then what the mempool holds that the job did not see) are validated the same way and added, to a parallel Soroban phase one by one (ParallelPhaseExtender), to a sequential phase by selecting it again; base fees are set by the rule of a build; and the set goes through the same checks and hand-off as a built one. Anything else (N closed with another value, with upgrades, the job not finished, a failed check) is a build. Tests: herder/test/EarlyTxSetTests.cpp. A completed set is validated in full on the closed ledger, read back from its bytes, for: the candidates being exactly N's transactions; the same accounts with their next sequence numbers; random overlaps; a transaction of N failing; an account drained, merged away or re-keyed by N; fee bumps; N's set skipped at apply; N closing with another value, close time or an upgrade; a job that is late, still running, or overlapping the next one; time bounds. Three of them show the set invalid when the validity pass is switched off. Fullness: top-up with late arrivals, refill after removals and for builds with 0 to 90% invalid candidates, Soroban top-up and conflicts, base fees after removal and refill. Nothing in apply, commit, set validation or the wire format changes.
…ne that lost candidates looks deeper whatever the mempool's top holds of its kind The stop rules of the completion's top-up and of a build's refill had two shortcuts that could leave a phase short while the mempool held valid transactions for it: - eight or more valid left-overs that could not be added closed the phase, also when they only had nowhere to go (a full cluster of one entry, a lane of its own) and later arrivals would have fit; - a phase that lost candidates pulled deeper only if the mempool's top held at least the phase's capacity of its kind (compared in transactions against operations), which a mempool top of the other phase prevents. Now, as in a build: a phase stops at its capacity; after one look at the mempool's top if it lost nothing (lost: removed by the validity pass, invalid, or of an account already in the set; a valid candidate that did not fit is not a loss); and otherwise when a deeper pull holds nothing new of its kind. Tests: many left-overs with nowhere to go; a completion and a build with a mempool top of the other phase.
Two lines of the load generator commits that clang-format 20 (CI's version) breaks differently. No other change.
…ixth experiment (the next tx set selected before the close) - What this branch is: experiment 6 (a smaller build; the leader of ledger N+1 selects its set from ledger N's set during N's ballot and completes it on the closed ledger), with its two supporting changes. - Results: the bucket list settings on 6 validators; the base with the meta off at 90 validators (801 ms per ledger at 6,000 TPS, 873 ms at 8,000 TPS); experiment 6 at 8,000 TPS (build 176 -> 121 ms, 858 ms per ledger, no invalid or short set), and why the shorter build does not show in the block time. - How it was measured, limits of the numbers, known limitations, test status.
SirTyson
force-pushed
the
perf/txset-build-off-critical-path
branch
from
November 14, 2026 20:47
c8417bc to
aeee8a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, draft. Both steps work and no set was invalid or short, but the block time gain is not established: at 90 validators and 8,000 TPS with real apply the leader's tx set build went from 176 to 121 ms (-31%) and the time per ledger from 873 to 858 ms (-1.7%, one run each, inside the spread of two runs of one point). On
overlay-v2-sharedat5d24d5494. It changes which candidates a round leader looks at and when it selects them; it does not change apply, commit, the validation of a tx set, the schedule or anything on the wire.Rebased on 2026-11-14 onto
overlay-v2-sharedatd9ae5fcec, after #5504 was merged. The copy of #5504's README commit is gone; the 12 commits that remain are unchanged in content, and the head's tree is identical to the old head's (c8417bc90). The commit ids in the text below are the ones before the rebase, under which the builds, test runs and reviews were made; they stay reachable as branchperf/txset-build-off-critical-path-measuredofSirTyson/stellar-core. Old to new:761d3727a->cbd2a5c87,1782051a3->39ea25fa3,49ae8e4ab->9724d343b,c426295d2->650811175,54260eafb->e0101b90d,e5eaa8e1c->754a7ba0a,64e8f5944->fc949267b,16d0f7bc5->03d8857c0,3973b9321->030f4dbd5,d33af26e3->1825d266f,f19140f89->09cdc0aae, READMEc8417bc90->aeee8a5ca.What this diff holds (12 commits):
761d3727a,1782051a3,49ae8e4ab,c426295d2: test-only load generator modes for real apply (sac_setup,sac_payment: payments of the native asset through its contract, each to a fresh address), and a 150-ledger wait for asac_paymentrun's last payments. The first three are the branchperf/loadgen-real-sacthat the README's real-apply runs were made with.54260eafb: a hit in the bucket list's account cache is read under the shared lock.e5eaa8e1c,64e8f5944,16d0f7bc5: the three commits of Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (slower than the base in its one goal-fixture run) #5503 (closed), unchanged. This PR replaces that attempt's selection and keeps its builder split and the leader's early execution of its prepared set.3973b9321: the change, and the commit every network run below was made on.d33af26e3: two rules for when a short phase pulls deeper, corrected after the last review (see Review). They did not come into play in the measured load.f19140f89:clang-formaton two lines of the load generator commits.What it does
A leader builds its tx set on the main thread when the previous ledger has closed. With real apply at 90 validators and 8,000 TPS that build is 0.26 s of a 1.16 s cycle (README, "What limits real apply").
A smaller build. A build pulls the set's capacity plus 10% from the mempool (
EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT), not twice the capacity. A phase that lost candidates (invalid, or a second transaction of an account) and is short pulls again, twice as deep, as long as a pull brings it something new, up to three pulls. The structural check of the selection runs beside the XDR write and ends before the set is handed to the overlay. A leader that executes its own vote does not read the written set back (that execution validates the same bytes); if the execution does not start, the read-back runs in the background.The next set is selected before the current ledger closes. The node first in line to lead ledger N+1 starts a background job at its vote for ledger N (
src/herder/EarlyTxSetSelection.{h,cpp}). The job takes N's tx set as applied (sequence numbers, fees), validates the mempool's top candidates on that view in parallel and selects the phases. When N has closed with that value,TxSetBuilder::completeSelectionon the main thread:ParallelPhaseExtender);A selection for another value, with other limits, after an upgrade, or one that is not ready, is dropped and the set is built from scratch.
EXPERIMENTAL_EARLY_TX_SET_SELECTION(default true) turns it off.The two requirements this was built to:
checkSelection, whatApplicableTxSetFrame::checkValidchecks) is awaited before the set leaves the builder.Results
Perfnet
block_time_check, real apply (SAC payments, no conflicts), 800 ms target, 16 clusters, m6id.8xlarge with local NVMe, 300 s judged load, one run per point. Every run has the transaction meta off (METADATA_DEBUG_LEDGERS=0,DISABLE_TX_META_FOR_TESTING=true), a 4 GB account cache and 256-byte bucket index pages (BUCKETLIST_DB_MEMORY_FOR_CACHING=4096,BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT=8). Base:54260eafb(this branch below the #5503 commits). Change:3973b9321.On 6 validators at 8,000 TPS (both on target, verdict PASS for both): build 144 -> 94 ms, 88% of the sets from an early selection, time per ledger 802 and 801 ms.
The base at 6,000 TPS and 90 validators (same settings): 801 ms per ledger, p75 805 to 808 ms, on target with room; so the comparison was made at 8,000 TPS.
What the runs show:
Settings (6 validators, 8,000 TPS, per payment; README for the table): every live bucket in memory is the fastest apply (execution 33.7 us against 44.3 us with the meta off alone) but the close stalls for up to 1.7 s at bucket spills; 256-byte index pages give 38.0 us without stalls; the account cache with its hits under the shared lock no longer slows the tx set validation (101 -> 73 ms) and adds nothing measurable on top of the small pages (37.3 us).
What I would do next (not in this PR): time the selection's start on every node at every ledger so that it pulls about 0.2 s before the close; a second, small pull by the job just before it selects; and, the largest piece, push the selected transactions to the peers during N's ballot and send only the difference at the close (the 0.15 s of delivery are now the largest part of the cycle that is neither ballot nor apply).
The README has the full tables, the method and the limits ("Real apply with the meta off, and the tx set build off the critical path").
Correctness
getInvalidTxListWithErrorsfor every transaction it kept,trimInvalidwith the same fee-source totals for every one it added, the one-per-account rule across both phases, andcheckSelectionon the final phases. A throw in any of it falls back to a build.Tests
src/herder/test/EarlyTxSetTests.cpp(the overlap cases: next sequence numbers, a drained or merged account, changed signers, fee bumps, a failed transaction of N, a randomized overlap compared with a build from scratch; each with a section that skips the validity pass and shows the set would be invalid; top-up, refill, fee sources, base fees, the Soroban extender, close-time-dependent candidates, another value or other limits closing the ledger, a job that is late, shutdown, a network test).ProposalPrevalidationTests.cppandHerderTests.cppadapted; 10 test cases of the replaced mechanism removed.3973b9321(668 test cases) and, with--all-versions, on3973b9321, ond33af26e3and on the source tree of this PR's head (669 test cases).build-linuxjobs stop in the format check, on 9 files of the base that this PR does not touch (5 of the Rust overlay,TxFloodTargetsTests.cpp,CommandHandlerTests.cpp,OverlayIPC.cpp,RustOverlayManager.cpp), as they did for Prototype: send transactions to the upcoming round leaders only, with an admission check #5501 and Prototype: the leader selects its next tx set during the previous ballot and executes its own set before the trigger (slower than the base in its one goal-fixture run) #5503. So there is no gcc build of this change.3973b9321). No thread sanitizer run: the build box cannot build one (the nightly toolchain has norust-src).3973b9321: 18 of 19 mutations of the safety and fullness rules are caught (validity pass removed, wrong close time, wrong value or limits, fee-source totals, one-per-account, extender limits, refill); the survivor is equivalent.--disable-testsbuild fails as the base's does (HerderImpl.cppreaches test headers throughsimulation/LoadGenerator.h); the other objects this PR touches compile without tests.clang-format20 on the 27 source files this PR changes: clean.Review
Three independent reviews (of work-in-progress, of the first squashed head and of
3973b9321): no finding on consensus safety, determinism, data races or lifetimes. Their fullness findings are fixed; the last two, both from rules added for cost after the second review, are fixed ind33af26e3with tests that fail on the old rules:Known limits
Checklist
overlay-v2-sharedatd9ae5fcec(no merge commits)clang-format(20.1.8 on the changed files; the build box has no v8)