Skip to content

The next tx set completed during the commit, and a cheaper apply: 10,000 TPS holds 800 ms (experiment 9, prototype) - #5511

Draft
SirTyson wants to merge 34 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/apply-serial-overhead
Draft

SirTyson wants to merge 34 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/apply-serial-overhead

Conversation

@SirTyson

@SirTyson SirTyson commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Prototype, draft. With the leader's next tx set completed while the ledger before is committed, and two serial steps of apply made cheap, 90 validators hold the 800 ms target at 10,000 TPS with real apply: verdict PASS, a ledger every 833 ms, window p75 876 ms. Before this PR the same network is saturated at a ledger a second (p75 1,035 ms, FAIL); with the two apply changes alone it is at 924 ms (p75 966 ms, FAIL by 6 ms). One run each.

On overlay-v2-shared, stacked on #5510 (open draft, 1a69a9d51): the diff also shows its commits and those of #5505 and #5506.

What this diff holds beyond #5510 (6 commits):

  • 35a71abd2: the tx set rebuilt from wire in parallel, and no key set for an eviction scan without candidates. Always on.
  • 08e3e66dc: the leader's next tx set completed during the commit, behind EXPERIMENTAL_COMPLETE_TX_SET_DURING_COMMIT (off by default).
  • e691422bb: fixes from the review of that commit. Both network runs below were made on this commit.
  • 960ee9e9b: README (experiment 9).
  • 572d4e08d: fixes from a second review, off the path the runs took (below). Not on a network.
  • b2585f23b: README, the results of experiments 8 and 9 in its introduction. Text only.

How this came about

The question was where execution's time goes. apply-load on a development machine (10,000 native-asset contract payments per ledger, one per transaction, 16 clusters; a sampling profile and wall-clock scopes around the phases) says that the contract executions are an eighth of a ledger's apply:

Per ledger, wall time (413.5 ms in all) ms
Tx set validation at apply 111 (rebuilding the set from wire: 21; the checks: 89)
Fees and sequence numbers 33
The contract executions, on 16 threads 52
Serial steps around them (pre-apply 36, writing the changes into the ledger transaction 25, refunds 17, staging 5) 83
Commit 120 (level-0 add 71, or 18 with #5510's shards; key set for the eviction scan 23; the ledger transaction's commit 20)

And on the network the path from one set to the next was: execute 490 ms, commit 162 ms, the next leader's completion of its set 178 ms, delivery 108 ms. This PR takes the two steps of apply that are overhead for this load, and moves the leader's completion into the commit.

What it does

1. The tx set from wire, in parallel (TxSetPhaseFrame::makeFromWire). A node rebuilt a received set one transaction at a time: a frame and a full hash (XDR and SHA-256) each, and the sortedness of every cluster. The clusters of the Soroban phase are independent; they are now built by tasks of the batch executor. prepareForApply takes a task bound, 1 by default; only validateTxSetForApply, on the apply thread, passes the bound it already gives the validation that follows it.

2. No key set for an eviction scan without candidates (BucketManager::resolveBackgroundEvictionScan). The commit built a sorted set of every key the ledger modified (30,000 here, compared as XDR) to hold eviction candidates against, before it was known whether the scan had any. It now takes a provider and calls it only when there are candidates.

3. The next tx set completed during the commit (EXPERIMENTAL_COMPLETE_TX_SET_DURING_COMMIT). A node that selected its next set early (#5505) completed it when the ledger before (N) had closed: a validity pass on the closed ledger, a top-up with what arrived since, a check of the selection's structure, the set's XDR.

  • Of the closed ledger that work needs the state, which is final when N has been executed, and the hash of N's header, which comes out of the commit.
  • With the setting, once N has been executed and has externalized with the executed value, the node runs the completion on the state the execution left (ExecutedLedgerState, already captured for Prototype: leader terms, pre-validate the next tx set during the previous ballot, build once before the trigger #5502's pre-validation), on its main thread while the apply thread commits: HerderImpl::prepareEarlyCompletion, TxSetBuilder::prepareSelection (the structure check, and the XDR with an empty previous ledger hash).
  • When N has closed with that value and a set is still selected within the same limits, TxSetBuilder::writePreparedSelection puts the hash in, the set gets its contents hash and is handed to consensus and the overlay; its self check runs in the background as for any set.
  • Otherwise the early selection is completed at the close as before: it stays with its job.
  • Transactions that arrive during the commit wait for the set after.
  • Nothing is prepared when the selection or a candidate has time bounds or a minimum sequence age: such a transaction cannot be judged before the close time is known. Loadgen's transactions have none.

Results

Fixture: #5510's run with shards. 90 validators (m6id.8xlarge) in four regions, 800 ms target, 10,000 native-asset contract payments per second executed for real, meta off, 16 clusters, the tx set as coded pieces, no faulty nodes, level 0 as 8 shards per group. Package 30.0.1-3700.e691422bbb.noble~perf.apply.serial.overhead~perftests for the two runs of this PR, 22 minutes apart, differing in the setting only; the first column is #5510's run, two days earlier, on this branch's base. Timers are means per ledger pooled over the 90 nodes between 150 s and 315 s of the load.

Before (#5510) Parts 1 and 2 Parts 1, 2 and 3
Verdict (p75 in [640, 960) ms, p99 at most 1,600 ms) FAIL, 0 of 90 nodes FAIL, 12 to 14 of 90 nodes PASS, 90 of 90 nodes
Window p75 / p99, median over nodes 1,035 / 1,356 ms 966 / 1,190 ms 876 / 1,060 ms
Payments included and applied, of 3,600,000 3,600,000, 0 failed 3,600,000, 0 failed 3,600,000, 0 failed
Time between ledgers 1,002.6 ms 924.1 ms 833.2 ms
Payments per ledger 9,988 (the cap) 9,247 8,338
Tx set validation at apply 162.8 ms, 16.3 us per payment 124.9 ms, 13.5 us 114.1 ms, 13.7 us
Execution in all (validation included) 490.5 ms, 49.1 us 434.6 ms, 47.0 us 398.6 ms, 47.8 us
Commit 161.5 ms, 16.2 us 123.7 ms, 13.4 us 113.2 ms, 13.6 us
The leader's work at the close 178.0 ms 171.3 ms 40.2 ms
The completion, during the commit 172.3 ms
Early selections prepared and used, of those completed 0 all

Every check other than the windows passed in all three runs: inclusion on every node, in sync, no fork, error counters, no restarts.

  • Parts 1 and 2 make a payment a sixth cheaper to validate and to commit: the network goes from saturated to almost keeping up.
  • Part 3 takes 131 ms of serial work off the leader's path after the close. The network settles at 833 ms with ledgers that hold the load of 833 ms; a shorter cycle makes every ledger smaller, and so cheaper again.
  • With part 3 the leader's completion (172 ms on its main thread, from the moment the ledger externalizes) is 60 ms longer than the commit beside it (113 ms): the leader takes a ledger's close 50 ms later than without the setting, followers 28 ms earlier. Two thirds of the completion is the top-up, because the selection now starts when the node votes and sees less.

Local measurements (a development machine with the CPU of the network's nodes):

Before After
apply-load, ledger close, parts 1 and 2 (base and change in turn, 3 rounds of 40 ledgers) 416.6 / 416.2 / 412.6 ms 376.8 / 367.3 / 363.2 ms (-11%)
the same with level 0 as 8 shards per group 390.8 / 385.0 / 382.7 ms 350.8 / 350.8 / 341.3 ms (-10%)
"early tx set completion costs" (hidden benchmark), the leader's work at the close, part 3 73.9 ms 12.8 ms; the completion during the commit: 99.5 ms

What I predicted for part 3 before it was built: a full ledger every 880 to 920 ms, still failing. It came out at 833 ms and passes.

Correctness

  • No consensus value changes. Part 1 builds the same frames in the same order and refuses what was refused. Part 2 sees the same key set whenever it looks at it. Part 3 changes when a leader does its work and which of the waiting transactions a set holds, not what a valid set is.
  • The executed state is the closed ledger's for everything validation reads: fees, sequence numbers and refunds are applied by the execution, the commit touches no account, the value has no upgrades, and the state's view refuses any other kind of entry instead of answering with an older version. It is used only for the ledger and the value it was executed with, after that value has externalized.
  • The set is checked as any set: its structure before anything is written, the written set in the background after it is handed over, and by every node, the leader included, when it is applied. A wrongly prepared set would be an invalid set that every node skips alike: a lost ledger, not a divergence.
  • Threads. Part 1 uses the batch executor from the apply thread, immediately before a validation that already does, within the same bound. Part 3's validation runs on an executor of its own, because the application's runs one batch at a time and the apply thread may be using it.
  • With the setting off, part 3 adds one assignment per externalized ledger and a log line.

Tests

  • New: a parallel phase rebuilt from wire by 1 to 16 tasks holds the same frames, and a malformed one is refused by all (TxSetTests.cpp); the key provider is not called by a scan without candidates (BucketManagerTests.cpp); the completion during the commit (ProposalPrevalidationTests.cpp): executed first and then externalized, and the other way round; what comes after the externalization is not in the set; the set is valid on the closed ledger, reads back from its XDR, passes its self check and applies; another value externalizes; a close-time-dependent transaction in the selection and among the later arrivals; without the setting.
  • Full suite (32 partitions, clang 20) and the suite with --all-versions: pass at e691422bb and at the head.
  • clang-format 20 on the changed C++ files.
  • Without BUILD_TESTS: TxSetFrame.o, TxSetUtils.o, BucketManager.o, LedgerManagerImpl.o and Config.o compile; HerderImpl.o fails with exactly the six errors of the base stack (test headers reached through simulation/LoadGenerator.h), none in this PR's code.
  • Not run: sanitizers. Not tested: limits that change between the completion and the close, a failing completion, two speculative executions outstanding, the early selection not ready when the ledger externalizes.
  • CI: as for the PRs below it, the build-linux jobs stop in the format check on files of the base that this PR does not touch.

Review

Two independent reviews. The first, of 08e3e66dc: parts 1 and 2 sound (the same sets accepted and refused, no race, the same key set whenever it is read); for part 3, the executed state equivalent to the closed ledger for everything validation reads, no way to use a prepared selection on the wrong ledger, and writePreparedSelection equal line by line to the path of writeSelection it replaces. Its findings, all in part 3 and fixed in e691422bb:

  • a crash path: the completion could run a batch on the application's executor while the apply thread ran one (with a second speculative execution outstanding, or a ledger queued behind the one being committed). It now has an executor of its own;
  • the completion could run before the ledger had externalized, holding up the main thread during the ledger's own consensus. It now waits for the externalization;
  • the early selection was consumed whether or not the completion could be used; a transaction not yet valid at a guessed close time could have been removed from the mempool. The selection now stays with its job, and nothing is prepared when a transaction depends on the close time.

The second, of e691422bb: every fix holds, no blocker, no divergence, crash or race found. Two latency points behind the setting, fixed in 572d4e08d: when a ledger externalizes after it was executed, the completion ran inside SCP's externalize call, before this node's own externalize statement was sent (it is posted to the main thread now; in the network runs ledgers externalized before their execution ended, so they took the other path); and the early selection started at the vote also on ledgers where nothing ends up prepared (it now does so only while completions are being prepared). Left as they are:

Known limits

  • One network run per column, and the first column is another day's run on another package. With parts 1 and 2 alone the p75 limit is missed by 6 ms on most nodes: a second run could fall on either side.
  • Part 3 does not engage on traffic with time bounds (above). Preparing for a range of close times would lift this.
  • The completion blocks the leader's main thread for its whole length (172 ms at 10,000 TPS).
  • The serial per-transaction work of apply (fees, pre-apply, the write-back, refunds: 111 ms locally) is untouched; so are the in-memory Soroban state update the commit waits for (28 ms with shards) and the freeing of the ledger transaction's entries (about 23 ms).

Checklist

SirTyson and others added 30 commits November 14, 2026 20:47
…t is settable (completiontimeout)

A generator of the real-apply load reported failure although every one of
its payments was applied: after its last submission loadgen waits 20
ledgers for its accounts, and on a 90-validator network 1 to 9 of 90
generators had a last batch that was included 25 to 70 s later (payments
go to the round leaders only, without an acknowledgement). The run then
got no verdict on inclusion.

SAC_PAYMENT now waits 150 ledgers by default, and generateload takes
completiontimeout=<ledgers> for any mode. Other modes keep 20.
A lookup in the per-bucket account cache (BUCKETLIST_DB_MEMORY_FOR_CACHING)
took the cache's lock exclusively even for a hit, because
RandomEvictionCache::maybeGet updates the entry's access time and the
hit/miss counters. Ledger apply loads accounts from many worker threads at
once, and every load asks the cache of each disk-indexed bucket on its way
down the bucket list, so with the cache on the workers' account loads
queued up on that lock (measured on a 6-validator network at 8,000 SAC
payments per second: 29 us per load with the cache against 14 us without).

RandomEvictionCache gets peek(), a const lookup that records nothing, and
LiveBucketIndex::getCachedEntry uses it under the shared lock. Inserting an
entry still takes the lock exclusively. A hit no longer makes its entry
younger for eviction: when a cache is smaller than its bucket's accounts,
eviction goes by insertion time. A cache sized to hold all accounts (the
case this is for) never evicts.
…lot and executes its prepared set before the trigger

The first-round leader's build of a transaction set sat between the close of
the previous ledger and its proposal: a set names its parent's header hash,
so the build could not start before that ledger had closed.

1. Selection during the previous ballot. The node that expects to lead round
   1 of ledger N+1 selects its set for N+1 while N is still in its ballot:
   the pre-validation job (which validates candidates on the state the
   node's speculative execution of N left) goes on, on its last task's
   thread, to trim and surge-price what it pulled and to write the set's XDR
   without a parent. When N has closed with the executed value and a build
   would have to start at once (the trigger is due, or less than a build's
   lead time away), the node only names N as the parent, hashes the set,
   hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet).
   The selection is used only if N closed with exactly the executed value,
   the limits it assumed are those of the closed ledger, its time-dependent
   transactions are valid for the close time the set gets, and a build of
   its size would take time (an empty selection is never proposed).
   Otherwise the ledger gets one build as before.

2. The leader executes its own set as soon as it is prepared. The value the
   trigger will propose is known when the set is (HerderImpl::
   executePreparedTxSet), so the ledger goes to the apply thread then, not
   when the first ballot statement is emitted. If that execution finds the
   set invalid before the proposal, the set is dropped.

3. The builder's check of the written set (the XDR round trip and the
   validation of the re-parsed set) runs on a background thread for sets the
   herder builds as a leader on protocols that validate tx sets at apply
   (TxSetSelfCheck). The structure of the selection and the shape of the
   XDR are still checked before the set leaves the builder.

makeTxSetFromTransactions is split for this: selection takes the limits
(TxSetBuildLimits) and the validation context explicitly instead of reading
the LCL through the application, so that it can run for a ledger that is not
the LCL and off the main thread.

Two faults of the base are fixed on the way: the pre-validation's transaction
frames referred to a network ID owned by a task's closure, and a build kept
one transaction per source account within each phase but not across the two
phases (the builder's own check then threw).

No change to apply, commit, tx-set validation or the wire format.
…d the deferred self-check

New test cases (tag [early-txset]) in ProposalPrevalidationTests.cpp. Two
leader-build tests in HerderTests.cpp count builds instead of mempool pulls
(the leader's early execution pulls candidates for the ledger after it), and
"SCP Driver / validateValue txSet cached" no longer depends on which of its
two validators leads (it failed on the base at --base-instance 1250).
…fore it closes, and pulls less for a build

Two changes to how a leader gets its transaction set, on top of the
reused parts of the closed attempt (builder split, deferred check of the
written set, early execution of a prepared set).

1. A smaller build. A build pulls the capacity of a set plus a margin
   (EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT, default 10) from the mempool
   instead of twice the capacity. If a phase loses candidates (invalid, or
   a second transaction of an account) and ends below its capacity while
   the mempool has more, the build pulls again, twice as much, at most
   three times. The structural check of the selection runs on a second
   thread beside the XDR write and is awaited before the set is handed to
   anyone. The re-read of the written set is left to the node's own
   execution of its value where it executes its votes, and runs in the
   background otherwise.

2. Early selection (EXPERIMENTAL_EARLY_TX_SET_SELECTION, default on). When
   a node votes for a value for ledger N and expects to lead round 1 of
   N+1, it plans a background job that selects the set for N+1 while N is
   in its ballot: on the last closed ledger (N-1) and a prediction of what
   N's set does to it (sequence numbers consumed, full fees charged). The
   job starts as late as it can and still finish before the set is needed.

   The prediction is never trusted. At the build for N+1, if N closed with
   exactly the value the job was for, the selection is completed: every
   one of its transactions is validated against N as it closed, with the
   set's close time, by the function that validates a received set; what
   fails is removed; while a phase has room, further candidates (what the
   job did not select, then what the mempool holds that the job did not
   see) are validated the same way and added, to a parallel Soroban phase
   one by one (ParallelPhaseExtender), to a sequential phase by selecting
   it again; base fees are set by the rule of a build; and the set goes
   through the same checks and hand-off as a built one. Anything else (N
   closed with another value, with upgrades, the job not finished, a
   failed check) is a build.

Tests: herder/test/EarlyTxSetTests.cpp. A completed set is validated in
full on the closed ledger, read back from its bytes, for: the candidates
being exactly N's transactions; the same accounts with their next
sequence numbers; random overlaps; a transaction of N failing; an account
drained, merged away or re-keyed by N; fee bumps; N's set skipped at
apply; N closing with another value, close time or an upgrade; a job
that is late, still running, or overlapping the next one; time bounds.
Three of them show the set invalid when the validity pass is switched
off. Fullness: top-up with late arrivals, refill after removals and for
builds with 0 to 90% invalid candidates, Soroban top-up and conflicts,
base fees after removal and refill.

Nothing in apply, commit, set validation or the wire format changes.
…ne that lost candidates looks deeper whatever the mempool's top holds of its kind

The stop rules of the completion's top-up and of a build's refill had two
shortcuts that could leave a phase short while the mempool held valid
transactions for it:
- eight or more valid left-overs that could not be added closed the phase,
  also when they only had nowhere to go (a full cluster of one entry, a lane
  of its own) and later arrivals would have fit;
- a phase that lost candidates pulled deeper only if the mempool's top held
  at least the phase's capacity of its kind (compared in transactions against
  operations), which a mempool top of the other phase prevents.

Now, as in a build: a phase stops at its capacity; after one look at the
mempool's top if it lost nothing (lost: removed by the validity pass, invalid,
or of an account already in the set; a valid candidate that did not fit is
not a loss); and otherwise when a deeper pull holds nothing new of its kind.

Tests: many left-overs with nowhere to go; a completion and a build with a
mempool top of the other phase.
Two lines of the load generator commits that clang-format 20 (CI's version)
breaks differently. No other change.
…ixth experiment (the next tx set selected before the close)

- What this branch is: experiment 6 (a smaller build; the leader of ledger
  N+1 selects its set from ledger N's set during N's ballot and completes it
  on the closed ledger), with its two supporting changes.
- Results: the bucket list settings on 6 validators; the base with the meta
  off at 90 validators (801 ms per ledger at 6,000 TPS, 873 ms at 8,000 TPS);
  experiment 6 at 8,000 TPS (build 176 -> 121 ms, 858 ms per ledger, no
  invalid or short set), and why the shorter build does not show in the
  block time.
- How it was measured, limits of the numbers, known limitations, test status.
…rom the schedule's nodes and the slot's seed

EXPERIMENTAL_TX_SET_TREE = off | star | hash | rtt (default off).

Until now the round's leader writes its whole (compressed) tx set to every
peer: 89 copies at 90 validators. With a tree it writes it to its children
only, and every node passes the bytes on to its own children.

- The tree (overlay/src/flood/txset_tree.rs). For a slot and a leader: the
  leader is the root, the other nodes of the leader schedule follow in the
  order of SHA256(seed || leader || node), and position i's children are the
  positions F*i+1 .. F*i+F (the complete F-ary tree, the least depth a
  fanout of F allows). The seed is the seed of the slot's leader schedule.
  F is EXPERIMENTAL_TX_SET_TREE_FANOUT; 0 = the smallest F that reaches
  every node in two hops (9 for 90 nodes). Every node derives the same tree
  from what it already agrees on.
- Core tells its overlay the nodes and the seeds of the next two slots at
  every ledger close (SetTxSetTree, HerderImpl::updateTxSetTree).
- A pushed frame carries a header in front of the ordinary tx-set frame:
  kind, slot, leader, hop count and the leader's send time (and, in rtt
  mode, the order of the nodes).
- A node relays a tagged frame to its children before it decompresses or
  checks it, if it knows the tree of that slot, the sender is its parent in
  it (by the validator identity of the connection) and it has not relayed
  that frame. It then verifies and files the set by the hash of its content
  as before.
- A node that expects a set through the tree waits 300 ms from the first
  mention of the set's hash before it asks a peer for it; otherwise every
  node the leader did not push to would ask it when the proposal arrives.
- star: to every peer as before, with the header (the baseline for
  measurements). A leader that cannot reach one of its children, or does
  not know the slot's tree, pushes a star.
- rtt: the leader measures the round trip to every peer (a 13-byte probe
  on the tx-set stream), groups its peers by the gaps in the sorted times,
  gives every group relays in proportion to its size and fills a relay's
  children from its own group. The order travels in the header.
- Metrics overlay.txset-tree.* (copies written as root and as relay,
  received, refused, duplicates, unreachable children, requests put off and
  those sent after the wait, delay since the leader's send in total, per
  hop count and in buckets) and overlay.txset-push.sent-bytes.

Nothing in SCP, the schedule, apply or a tx set's content changes: a set is
still identified by the hash of its content, computed by the receiver, and
reaches Core only when an SCP statement names that hash.

Tests: the tree's logic (order, least depth, header, relay decision, wait,
rtt order), seven overlay nodes on loopback (delivery down the tree, a relay
that does not relay, a frame from a node that is not the parent, star, an
explicit order, probes), and 7-validator networks under consensus in each
mode.
…odes below a busy child

After an independent review of the first commit:
- Only a node that this node's Core expects to lead the slot can root a
  tree: Core sends the leaders of the first three rounds of the next slot,
  and for the slot after it those plus the first three of a new order
  (SetTxSetTree carries them per slot). A frame that names another leader is
  filed and not relayed.
- An order chosen by the sender (rtt mode) is followed only by nodes in rtt
  mode, only for a tree of two levels, and only if it refers to the same
  list of nodes (a digest of the list travels with the order).
- A node relays at most two different frames per slot and leader (it told
  copies apart by their first and last bytes before, which a sender can
  vary), and has at most one relay in flight per child.
- A frame whose body decode would refuse for its size is not relayed.
- A probe answer is a measurement only if it answers the probe this node
  sent that validator last; one answer to a peer's probes is in flight at a
  time.
- If the leader cannot send a child the set (an earlier push to it is still
  in flight), the nodes below that child get it from the leader directly, as
  a star. Before, they waited for a relay that had nothing to relay.
- A node that is not one of the tree's nodes does not wait before it asks.
- The layout by round-trip times is computed outside the tree's lock.
- Counters: overlay.txset-tree.child-busy, relay-busy, relay-limited. The
  delay-le-* buckets are disjoint ranges; their descriptions say so.

Tests for each of these (tree module, seven overlay nodes on loopback, the
IPC payload's layout).
…elay in the middle

The leader grouped its peers by gaps in the sorted round-trip times, where a
gap was 8 ms or a fifth of the larger time, and gave a group's relays the
nearest nodes of the group. Two regions at similar distances from the leader
(us-west-2 and ap-southeast-1 seen from eu-central-1, for one) became one
group whose relays were all in the nearer region: the nodes of the other one
got the set over two long hops.

Now a group ends at a gap of more than 8 ms only, and a group's nodes, sorted
by round-trip time, are cut into consecutive runs, one per relay, with the
run's middle node as its relay. Regions at the same distance still differ by
a little, so the runs mostly fall apart along them. With more groups than
relays all nodes are one group, laid out the same way (before: nearest
first, which is not a two-level tree a receiver follows).

In a model with typical round trips between four AWS regions (the item's
results/dissemination-model.py) the mean delay of a set goes from 95 to 84 ms
and the 90th percentile from 178 to 134 ms, which is what a leader could do
if it knew every distance.
The test sent three frames back to back and expected two of them to reach a
child; with one relay in flight per child the second could be skipped.
…re-coded pieces

EXPERIMENTAL_TX_SET_TREE="coded": the leader cuts the frame it would push
into k pieces, adds m-k Reed-Solomon parity pieces (16 pieces of which any 8
rebuild the frame at 90 validators), signs the root of their Merkle tree with
its overlay key and writes every piece to that piece's relays; a relay passes
its piece on. A node gets its m pieces from m different senders, so up to m-k
of them can be down, slow or hostile without the node having to wait and ask.

A piece is kept and passed on only if it comes from the node that owes it in
the layout, names a leader Core expects for the slot, carries the leader's
signatures and a Merkle path to the signed root. A rebuilt frame goes the way
of every received frame (decompress, strict parse, content hash).

EXPERIMENTAL_TX_SET_TREE_WITHHOLDERS_FOR_TESTING=N makes the N validators
with the smallest node IDs pass nothing on, in the tree modes and in coded
mode (tests and fault runs).
…'s seed, and do not code as a leader nobody expects

From the review of the coded mode:
- A piece frame is counted against its sender (six per slot and leader)
  before the layout is consulted, whatever the frame turns out to be, and
  the layout is worked out once per slot and leader.
- The signed message holds the seed of the slot's layout, so a root signed
  in another history of a network with the same name and keys is refused.
- A node that Core does not name as a leader of the slot pushes its set to
  everyone instead of cutting pieces (or rooting a tree) that every node
  would refuse.
- A rebuilt set counts as received from its leader, not from the sender of
  the piece that completed it.
- EXPERIMENTAL_TX_SET_TREE_WITHHOLDERS_FOR_TESTING is a testing-only option.
… signed its root

From the second review of the coded mode:
- The overlay key that signed a root is kept with the root, and a set
  rebuilt from pieces counts as received from that overlay, connected or
  not. Before, a node not connected to the leader charged the sender of
  the piece that completed the set, which the leader could choose.
- A root gets its place only from a frame whose signatures were checked.
- The leader named in a piece must be one of the layout's nodes before
  the frame is counted.
- A node that is not a named leader of the slot logs that it pushes to
  every peer (TXSET_CODED_STAR, TXSET_TREE_STAR).
- Tests: the unexpected leader in hash mode, the attribution, a settle
  before a counter is read. Docs and comments that still described the
  old behaviour of an unnamed leader.
…eal apply at 8,000 and 10,000 TPS, against the spanning tree and the push, with validators withholding, and what limits 10,000 TPS
…ne config line put back as clang-format has it
1024 16-bit elements per member, from a ChaCha20 keystream keyed by the
SHA-256 of a domain tag and the member; members are added and removed
element-wise, so the hash of a set does not depend on the order of its
members or on how the set is split into parts.
EXPERIMENTAL_SHARDED_TOP_BUCKET = N (0, the default, is off). The hash of a
level-0 bucket becomes the LtHash16 of its records instead of the SHA-256 of
its file, so a ledger's entries can be hashed and written in parts: they are
cut into shards by key (N for contract data and code, N for TTL entries, N
for everything else), and every shard is sorted, written and synced, merged
with the shard of the ledger before, hashed and indexed on its own thread.
The bucket's file, byte for byte the file level 0 has without the setting, is
written afterwards by a thread of its own; whoever needs it waits for it. It
is what spills into level 1 and what is published, so levels 1 and deeper
keep SHA-256 and their buckets.

The setting changes the bucket list hash of the ledger header: a network has
it on at every node or at none. The number of shards is each node's choice.

A node that stops between a ledger's commit and the end of the bucket's file
rebuilds the file at startup from the ledger's shard files and the bucket
they were added to. Catchup takes a bucket named by either hash when the
setting is on.

STELLAR_TEST_SHARDED_TOP_BUCKET=N runs the tests that take their config from
getTestConfig with the setting.
… review fixes

- A bucket file is taken under the commutative hash only if it is the
  canonical file of its records: the metadata entry first, entries in the
  strict order of their keys, canonical XDR, nothing after the last record.
  The hash alone does not see the order, and catchup would have adopted a
  file that the next merge aborts on.
- The file's writer uses nothing of the bucket manager, and the bucket
  manager waits for it before it removes its directories.
- The shards' worker pool is not pinned (BatchExecutor(pinWorkers = false)):
  its batches can overlap those of the application's executor. It takes
  BATCH_EXECUTOR_TASKS.
- The setting is refused on the public network, and a level 0 written
  without it is reported at startup instead of in a ledger close.
- The shard of a key is mixed with a value only the process knows.
- The digest of an LtHash16 state has its own tag.
- A bucket without a metadata entry has version 0, as its file says.
- bucket.top-shard.gc-wait times the wait of the bucket garbage collection.

Tests: the canonical file check; the equivalence test also at the last
protocol without and the first with INIT and metadata entries.
…of the state update

- The canonical file check reads the file record by record instead of
  whole: with the setting on, catchup hands it every live bucket that is not
  named by its SHA-256, of any size.
- A file that is not canonical is logged by catchup like any other bucket
  that fails verification.
- The bucket manager, before it removes its directories, waits for the file
  of every bucket made ahead of its file, not only for level 0's current one
  (a bucket list may have been assumed over it).
- bucket.top-shard.gc-wait is registered only with the setting.

Tests: two more non-canonical files; the setting over state written without
it is refused at startup. A hidden benchmark of the in-memory Soroban state
update, which the ledger commit runs beside the bucket list add and waits
for.
… (experiment 8): real apply at 10,000 TPS with the setting on and off, and what limits it after
…on scan without candidates

Two serial steps of a ledger's apply that are overhead at 10,000 payments
per ledger (apply-load, 16 clusters: 21 ms and 23 ms of a 412 ms close).

- TxSetPhaseFrame::makeFromWire builds the clusters of a parallel Soroban
  phase (a frame and its full hash per transaction, the checks of its fee
  fields and of the cluster's order) by tasks of the batch executor when the
  caller allows it. prepareForApply takes a task bound, 1 by default; only
  validateTxSetForApply, on the apply thread, passes the bound it gives the
  validation that follows. The same set is built; what was refused is
  refused.
- BucketManager::resolveBackgroundEvictionScan takes a provider of the keys
  the ledger modified and calls it only if the scan has candidates. The set
  (LedgerTxn::getAllKeysWithoutSealing: sorted, every modified key) was built
  on every ledger before it was known whether anything could be evicted.

Tests: a phase parsed by 1 and by many tasks holds the same frames, and a
malformed one is refused by all; the provider is not called by a scan
without candidates.
…itted

EXPERIMENTAL_COMPLETE_TX_SET_DURING_COMMIT (off by default). A node that
selected its next set early completed it when the ledger before had closed:
the validity pass on the closed ledger, the top-up, the check of the
selection's structure and the set's XDR, on the path of the next ledger
(178 ms at 10,000 payments per ledger on a 90-validator network).

Of the closed ledger that work needs the state, which is final when the
ledger has been executed, and the hash of its header, which comes out of
the commit. With the setting, when the ledger has been executed with the
value the selection was made for, the node completes the selection on the
state that execution left (ExecutedLedgerState), checks it and writes its
XDR with an empty previous ledger hash, on the main thread while the apply
thread commits (TxSetBuilder::prepareSelection). When the ledger has closed
with that value, the hash is put in and the set is handed over
(writePreparedSelection); the written set is checked in the background as
before. If the ledger closes with another value, the limits of a set
changed, or the completion holds a transaction whose validity depends on
the close time, the set is built as without an early selection.

Transactions that arrive during the commit wait for the set after.

The validation uses the batch executor only when the apply thread cannot:
it has finished executing, and has only the commit of that execution left,
or nothing until this thread hands it something.

Tests: the set completed during the commit is the selection plus what had
arrived by the end of the execution, valid on the ledger that closed, and
applies; it is dropped if the ledger closes with another value; nothing
changes without the setting. The completion benchmark (hidden) can run the
new path (BENCH_DURING_COMMIT=1).
Garand Tyson added 3 commits November 16, 2026 23:24
- The preparation's validation runs on an executor of its own (unpinned;
  TxSetValidationContext::mExecutor) instead of the application's, which
  runs one batch at a time and which the apply thread may be using: for a
  ledger queued behind the one it commits, or for another speculative
  execution.
- The completion is prepared when the ledger has externalized with the
  executed value, whichever of execution and externalization comes last,
  not before: until then the main thread has the ledger's consensus to run.
- The early selection stays with its job: if nothing comes of the
  preparation it is completed at the close as without the setting.
- Nothing is prepared if the selection, its leftovers or a candidate of the
  top-up has a validity that depends on the close time: such a transaction
  can be judged neither valid nor invalid before the close time is known.
- A prepared selection that fails its structure check is treated like a set
  that fails it at the close.
- With the setting the early selection starts when the node votes: it is
  needed a commit earlier than the build its start was aimed at.
- makeFromWire hands the clusters to its tasks one by one.
- Meters early-prepare-not-ready, early-prepare-skipped; a warning when the
  setting cannot take effect.

Tests: executed first then externalized, and the other way round; another
value externalizes; a close-time-dependent transaction in the selection and
among the later arrivals.
…ply (experiment 9): real apply at 10,000 TPS holds the 800 ms target
- When a ledger externalizes after it was executed, the completion is
  posted to the main thread instead of run from valueExternalized: that is
  called from inside SCP, before the statement that says this node
  externalized has been sent.
- The early selection starts when the node votes only while completions are
  being prepared (and speculative apply is on). After an attempt that
  prepared nothing (a transaction whose validity depends on the close time,
  a selection that was not ready) it starts as late as without the setting:
  a selection completed at the close is better made late.
- The warning names both settings the setting needs.

README: what was and was not on a network.
@socket-security

socket-security Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​reed-solomon-simd@​3.0.1811009310080

View full report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant