Skip to content

Prototype: the leader's tx set travels as erasure-coded pieces, or down a spanning tree (real-apply results, README) - #5506

Draft
SirTyson wants to merge 23 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/txset-spanning-tree
Draft

SirTyson wants to merge 23 commits into
stellar:overlay-v2-sharedfrom
SirTyson:perf/txset-spanning-tree

Conversation

@SirTyson

@SirTyson SirTyson commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Prototype, draft, config-gated and off by default. With the leader's tx set delivered as erasure-coded pieces, 90 validators hold the 800 ms target at 8,000 TPS with real apply (a ledger every 804 ms, verdict PASS), where the push of the whole set to every peer takes 869 ms and a spanning tree 832 ms; the leader writes 9.6 MB per set instead of 95.7 MB. With 29 of the 90 validators withholding every piece they should relay it takes 837 ms and 1.7% of the deliveries fall back to a request. At 10,000 TPS the ledgers are full and take 1.0 s, and what limits it is the serial apply path, not the delivery. One run per point.

On overlay-v2-shared, stacked on #5505 (open draft, aeee8a5ca): the diff also shows its commits.

Rebased on 2026-11-14, after #5504 was merged and #5505 was rebased onto it. The commits are unchanged in content (the head's tree is identical to the one before). The commit ids in the text below the list are the ones before the rebase, under which the package, the network runs, the test runs and the reviews were made; they stay reachable as branch perf/txset-spanning-tree-measured of SirTyson/stellar-core. Old to new: 8015877d9 (the measured commit) -> 330d185e5, d3323573d -> ea966452d, 508aa8107 -> a1cb73f28.

What this diff holds beyond #5505 (11 commits):

  • a8b5b7885, aad1eece9, f143b4b2d, aa8c4d5f6: the spanning tree for the leader's push (EXPERIMENTAL_TX_SET_TREE = star, hash, rtt), with the bounds a first review asked for, and the layout by round-trip times.
  • 712f6933a: docs/rust-overlay/txset-fetching.md.
  • 37fc98cf6: the coded mode (coded).
  • 330d185e5 (before the rebase: 8015877d9): fixes from the review of the coded mode. Every network run below was made on this commit.
  • ea966452d: fixes from the review of that commit (which peer a rebuilt set is counted against, a log line, tests, docs). Not on a network.
  • 66bc6e36f and the last commit: README. a1cb73f28: formatting of this PR's lines.

What it does

The leader of this branch writes its whole tx set to every peer when it has built it (#5497): 89 copies at 90 validators, 96 MB per ledger at 8,000 TPS. A follower cannot vote before it holds the set.

EXPERIMENTAL_TX_SET_TREE (default off) changes who sends the set's bytes to whom. The layout is derived the way the leader schedule is, from the nodes of the transitive quorum and the slot's seed: for a slot and a leader, the other nodes are ordered by SHA256(seed || leader || node). Every node computes it; nothing is negotiated. Core tells its overlay the nodes, the seeds of the next two slots and who may lead them at every ledger close (SetTxSetTree).

  • star: the push as it is, with a header that carries the leader's send time. The baseline of the measurements.
  • hash: a spanning tree. The order is laid out as a complete tree of the smallest fanout that reaches every node in two hops (9 at 90 validators). The leader writes the set to its 9 children; every node passes the bytes on to its own children, if the frame comes from its parent in the tree of a leader its Core expects.
  • rtt: the same tree, laid out by the leader from round-trip times it measures to its peers; the order travels with the set.
  • coded: the leader cuts the compressed set into 16 pieces of which any 8 rebuild it (Reed-Solomon, reed-solomon-simd; fewer pieces in a small network) and signs the root of the pieces' Merkle tree with its overlay key. Every piece has 5 relays, to which the leader writes it; a relay passes its piece on to a fifth of the other nodes. A node gets its 16 pieces from 16 different senders, no piece makes more than two hops, and a node has the set when the 8 fastest of its 16 pieces have arrived. Up to 8 of a node's 16 senders can be down, slow or lying without the node having to wait and ask.

A piece is kept, and passed on, only if it comes from the node that owes it in the layout, names a leader Core expects for the slot, and carries the leader's signatures (the validator's over its overlay's key, that key's over the slot, its seed, the shape of the code, the frame's length and the root) and a Merkle path to the signed root. A rebuilt frame goes the way of every received frame: decompressed, parsed strictly, filed under the hash of its content.

A node that expects a set through relays waits 300 ms before it asks a peer for it. A node its Core does not name as a leader of the slot pushes to everyone instead.

Wire formats, checks and limits: docs/rust-overlay/txset-fetching.md ("Spanning tree for the push", "Coded pieces").

Results

Perfnet block_time_check, real apply (native SAC payments, no conflicts), 90 validators (30 organizations, 4 regions), 800 ms target, 16 clusters, m6id.8xlarge with local NVMe, transaction meta off, a 4 GB account cache and 256-byte bucket index pages (the settings of #5505's runs), 360 s of load. One build for every run (8015877d9); the only difference between the modes is EXPERIMENTAL_TX_SET_TREE. One run per point.

Best results: coded pieces at 8,000 and 10,000 TPS

Coded pieces 8,000 TPS 10,000 TPS
Verdict PASS FAIL (block time p75 over its limit of 960 ms)
Time per ledger: mean / p50 / p75 / p99 804 / 800 / 817 / 953 ms 978 / 964 / 1,043 / 1,359 ms
Block time p75 / p99, median node, two windows 837 / 994 and 836 / 994 ms 1,065 / 1,367 and 1,068 / 1,368 ms
Payments per ledger 6,400 (cap 8,000) 9,674; the cap of 10,000 in every ledger once the load is steady
Payments included, of those offered; failed 2,880,000 of 2,880,000; 0 3,600,000 of 3,600,000; 0
Leader's upload per set 9.6 MB 15.0 MB
Leader's send to the set being in a follower's overlay 104 ms 107 ms
Sets a node had to ask for; pieces invalid, refused or not written; rebuilds failed 0; 0; 0 0; 0; 0
Tx set validation at apply / execution / commit, per ledger 103 / 323 / 133 ms 164 / 495 / 205 ms

At 8,000 TPS the network is on its target for the first time in this fixture (873, 858 and 869 ms per ledger in the three runs without the experiment). At 10,000 TPS it carries the load at one full ledger per second and loses nothing, but does not hold 800 ms.

Coded pieces against the spanning tree and the push to every peer, 8,000 TPS

Same build Star (push to every peer) Hash tree (fanout 9) Coded pieces
Verdict FAIL (p75 over 960 ms on 42 and 83 of 90 nodes) PASS PASS
Time per ledger: mean / p50 / p75 / p99 869 / 853 / 935 / 1,104 ms 832 / 817 / 868 / 1,055 ms 804 / 800 / 817 / 953 ms
Block time p75 / p99, median node, two windows 960 / 1,139 and 965 / 1,140 ms 901 / 1,136 and 900 / 1,136 ms 837 / 994 and 836 / 994 ms
Leader's upload per set 95.7 MB (89 copies) 9.1 MB (9 copies) 9.6 MB (80 pieces)
Written by all relays per set 0 81 MB (a relay 9 MB) 163 MB (a relay 2.0 MB)
Leader's send to the set being in a follower's overlay 173 ms 134 ms 104 ms
Sets in hand within 100 / 150 / 200 / 300 ms of the send 4% / 36% / 65% / 100% 31% / 61% / 81% / 100% 47% / 86% / 100% / 100%
Core's wait for a set it was told of (scp.fetch.txset) 157 ms 128 ms 66 ms
Leader's trigger to the first ballot 289 ms 263 ms 137 ms
Leader: set built to consensus 516 ms 493 ms 431 ms
Sets a node asked for after its 300 ms wait no wait in star 149 of 17,176 0 of 12,394
All overlay bytes written, per node 15.7 MB/s 16.5 MB/s 18.7 MB/s
Payments included, of 2,880,000; failed all; 0 all; 0 all; 0
  • Coded pieces: 65 ms less per ledger than the star (-7.5%), the tree 37 ms. The coded run sits on the target, so 65 ms is the least the coding gains at this load.
  • Both take 90% of the leader's upload away. The tree gets a third of the way because a node still has one path to the set. In the coded layout a node needs the 8 fastest of 16 paths and there is no slow tail: every set was in hand within 200 ms, where the tree had 19% later than that and 149 sets that had to be asked for. That tail is the tree's block time p99 (1,136 ms, as the star's); the coding lowers it to 994 ms.
  • The price: the network writes 1.8 times the tx-set bytes of the star (172 against 96 MB per set), 6% more of everything the overlay writes per ledger. Rebuilding a set takes 1.3 ms.
  • Under simulated apply (10,080 TPS, three earlier runs of the tree modes) the block time did not move: that fixture was on target with slack. The delivery went from 168 ms to 131 ms (hash tree) and 84 ms (tree by round-trip times). Neither the tree by round-trip times nor a coded layout that uses round-trip times was run with real apply.

Coded pieces under Byzantine conditions: 29 of 90 validators withholding

EXPERIMENTAL_TX_SET_TREE_WITHHOLDERS_FOR_TESTING=29: the 29 validators with the smallest node IDs, just under a third, pass no piece on. They still lead, receive and vote: this is the withholding part of a Byzantine fault, not all of it. The prediction was written before the run, from a model of the layout fitted to the star run.

Coded pieces, 8,000 TPS Nobody withholding 29 of 90 withholding Predicted Star, nobody withholding
Verdict PASS PASS PASS FAIL
Time per ledger: mean / p75 / p99 804 / 817 / 953 ms 837 / 879 / 1,042 ms 815 to 850 ms 869 / 935 / 1,104 ms
Block time p75 / p99, median node 837 / 994 ms 900 / 1,131 ms 960 / 1,139 ms
Leader's send to the set being in hand 104 ms 141 ms 130 to 150 ms 173 ms
Sets in hand within 150 / 200 / 300 ms 86% / 100% / 100% 54% / 92% / 100% 36% / 65% / 100%
Deliveries that ended with a request after the 300 ms wait 0 543 of 31,506 (1.7%) 1 to 3% no wait in star
Core's one-second retries 0 0 1
Pieces invalid or refused; rebuilds failed 0; 0 0; 0 0
Payments included, of 2,880,000; failed all; 0 all; 0 all; 0

With a third of the relays silent the coded network closes every ledger without Core's retry, faster than the healthy star and about as fast as the healthy tree. The same fault in the tree was not run on a network (the model gives it 28% of deliveries waiting and asking). Wrong pieces, a root signed by another validator or for another history, a validator that may not lead, a flooding sender and a leader whose pieces do not fit together are covered by tests with 9 overlay nodes, not by a network run.

Where a ledger's time goes at 10,000 TPS (coded pieces, the steady part of the load)

Step, in the order of the leader's cycle Per 10,000-payment ledger Share of the 1,007 ms cycle At 8,000 TPS (6,400 payments)
The leader completes its next tx set once its ledger has closed 146 ms 14% 103 ms
Tx set validation at apply 164 ms 16% 103 ms
Execution 495 ms 49% 323 ms
Commit 205 ms 20% 133 ms
Sum 1,010 ms (measured: 1,007 ms) 661 ms (on the 800 ms target)
Inside the execution, not limiting any more: the set's way to the followers 107 ms 104 ms
Followers: set in hand to consensus 313 ms 307 ms
Consensus to ledger closed: leader / follower 285 / 412 ms 193 / 197 ms
  • The bottleneck at 10,000 TPS is the serial apply path, not consensus and not the network. The four steps add up to the cycle within 3 ms, the execution is still running when consensus arrives, and the machines are a fifth busy (6.5 of 32 CPUs).
  • Per payment the path costs the same at both loads: 16 us of validation, 50 us of execution, 21 us of commit. A ledger of n payments needs about 26 ms + 98 us x n from one close to the next.
  • 10,000 TPS at 800 ms is missed by about 2%: a ledger then holds 8,000 payments, which need 813 ms by that line. Once a ledger is longer than the target the next one holds more payments, up to the cap, which is why the run sits at full ledgers and 1.0 s. The same line puts this build's limit at about 9,800 TPS (two runs, one line: an estimate).

Next steps (not in this PR; sizes are estimates from the table above):

  1. Take the tx set validation off the apply path (164 ms per full ledger): validate a set while the previous ledger commits. A part of it is enough for 10,000 TPS at 800 ms by the line above.
  2. Commit ledger N while ledger N+1 executes (205 ms).
  3. The serial part of the execution (495 ms): the earlier profile of the apply thread shows 39% XDR copies and 18% malloc and free, with the 16 workers a tenth busy.
  4. Complete the next tx set earlier (146 ms): time Prototype: the leader selects its next tx set from the current one before it closes, and pulls less for a build (real-apply results, README) #5505's early selection so that less is topped up at the close.
  5. A second run of the star and of the coded mode, and runs at 9,000 and 9,500 TPS: every point here is one run.
  6. The coded mode itself: a layout that uses round-trip times; a bound from Core on how large a set a leader may cut into pieces; whom a node asks after its wait.
  7. More of the Byzantine picture on a network: the tree under the same fault, relays that send wrong pieces, a leader that withholds from a part of the network.

The README has these tables with the method and the limits ("The leader's tx set as erasure-coded pieces").

Correctness

  • Consensus does not depend on the dissemination. Nothing in SCP, the schedule, apply or a set's content changes. A set is identified by the hash of its content, computed by the receiver, and reaches Core only when an SCP statement names that hash, as before. A wrong piece, root or rebuild gives bytes that fail the decompression or the parse, or a set with another hash that no statement names.
  • A rebuilt frame takes the steps of a received one: txset_encoding::decode, a strict GeneralizedTransactionSet parse, deliver_txset.
  • With off (the default) the paths reduce to the base's, apart from answering round-trip probes and parsing tagged frames, which the base would drop as malformed.
  • What a peer can make a node keep or do is bounded by the layout (one piece index per sender, slot and leader), six frames per sender, slot and leader (counted before any other work), two signed roots per slot and leader, the slots and leaders Core names, and four pieces in flight per peer. Nothing is decompressed before 8 of a node's senders agree on a root its leader signed.
  • Threading: three overlay locks, never nested, none held across an await; coding and rebuilding run on blocking tasks.

Tests

  • New in the overlay: flood/txset_tree.rs and flood/txset_coded.rs (layout, the code, Merkle paths, the piece header, the limits), libp2p_overlay/txset_tree_tests.rs (13 tests with 3 to 7 nodes on loopback) and txset_coded_tests.rs (8 tests with 9 nodes: delivery, withholding relays with and without enough pieces left, wrong pieces of six kinds, pieces that do not fit together, a leader nobody expects, the attribution of a rebuilt set, the signatures). The overlay's tests: 269 + 48 + 7 pass.
  • New in Core: src/herder/test/TxSetTreeTests.cpp, 8 test cases with 7 validators under consensus (each mode; a node that does not relay; withholding in the tree and in coded mode).
  • Full suite (32 partitions, lints included, clang 20): passes on 8015877d9 (677 test cases), and with --all-versions on 8015877d9, on d3323573d and on 508aa8107, whose source tree is this PR's head's (677 test cases each).
  • clang-format 20 on the 12 C++ files this PR changes: no difference that the base does not have (OverlayIPC.cpp and RustOverlayManager.cpp are among the base's unformatted files). rustfmt on the new Rust files and on the lines this PR adds to the others; the base's lines in libp2p_overlay.rs, main.rs and tx_router.rs are left as they are.
  • Without BUILD_TESTS: Config.o, OverlayIPC.o, OverlayMetrics.o and RustOverlayManager.o compile; HerderImpl.o fails with exactly the six errors of the base stack (test headers reached through simulation/LoadGenerator.h), none in this PR's code. So a --disable-tests build fails as the base's does.
  • Not run: sanitizers.
  • CI: the build-linux jobs of Prototype: the leader selects its next tx set from the current one before it closes, and pulls less for a build (real-apply results, README) #5505 stop in the format check on files of the base that neither PR touches, so CI has not compiled this stack.

Review

Three independent reviews (of the tree before any run; of the coded mode before any run; of the fixes before this PR): no path was found by which the change alters what a node proposes, votes for or applies. The abuse bounds of the tree (who may root one, what a relay follows, relays in flight) and of the coded mode (work before a limit, replay of a signed root in another history, a leader nobody expects, who a rebuilt set is charged to) come from them and are tested. Open from the reviews, and in the README's limits:

  • a hostile leader that Core names for the slot can sign frames of up to the general limit (16 MiB): it writes 10 frame sizes and the network about 178, two roots per slot;
  • a node connected to fewer than 8 of its 16 senders, or with another view of the nodes, waits 300 ms for every set; after the wait the request may go to a peer that does not hold the set;
  • the send time and the hop count in a frame are not signed (a relay can skew the delivery metric).

Known limits

  • One run per point; no mode was run twice. The 65 ms between the star and the coded mode are four times the spread of the three earlier runs of the fixture.
  • The fault run tests silent relays only, by a setting on nodes that otherwise behave.
  • No version negotiation: every node of a network needs the same mode.
  • The layout is by hash; the coded mode does not use round-trip times.
  • The bottleneck table reads timers of one run; the steps were not traced.
  • The last source commit (d3323573d) was not on a network.
  • A new dependency: reed-solomon-simd 3.0.1 (MIT and BSD-3-Clause; pinned to 3.0 so that the rest of the lock file does not move).

Checklist

SirTyson pushed a commit to SirTyson/stellar-core that referenced this pull request Nov 14, 2026
SirTyson and others added 23 commits November 14, 2026 20:47
…t is settable (completiontimeout)

A generator of the real-apply load reported failure although every one of
its payments was applied: after its last submission loadgen waits 20
ledgers for its accounts, and on a 90-validator network 1 to 9 of 90
generators had a last batch that was included 25 to 70 s later (payments
go to the round leaders only, without an acknowledgement). The run then
got no verdict on inclusion.

SAC_PAYMENT now waits 150 ledgers by default, and generateload takes
completiontimeout=<ledgers> for any mode. Other modes keep 20.
A lookup in the per-bucket account cache (BUCKETLIST_DB_MEMORY_FOR_CACHING)
took the cache's lock exclusively even for a hit, because
RandomEvictionCache::maybeGet updates the entry's access time and the
hit/miss counters. Ledger apply loads accounts from many worker threads at
once, and every load asks the cache of each disk-indexed bucket on its way
down the bucket list, so with the cache on the workers' account loads
queued up on that lock (measured on a 6-validator network at 8,000 SAC
payments per second: 29 us per load with the cache against 14 us without).

RandomEvictionCache gets peek(), a const lookup that records nothing, and
LiveBucketIndex::getCachedEntry uses it under the shared lock. Inserting an
entry still takes the lock exclusively. A hit no longer makes its entry
younger for eviction: when a cache is smaller than its bucket's accounts,
eviction goes by insertion time. A cache sized to hold all accounts (the
case this is for) never evicts.
…lot and executes its prepared set before the trigger

The first-round leader's build of a transaction set sat between the close of
the previous ledger and its proposal: a set names its parent's header hash,
so the build could not start before that ledger had closed.

1. Selection during the previous ballot. The node that expects to lead round
   1 of ledger N+1 selects its set for N+1 while N is still in its ballot:
   the pre-validation job (which validates candidates on the state the
   node's speculative execution of N left) goes on, on its last task's
   thread, to trim and surge-price what it pulled and to write the set's XDR
   without a parent. When N has closed with the executed value and a build
   would have to start at once (the trigger is due, or less than a build's
   lead time away), the node only names N as the parent, hashes the set,
   hands it to the overlay and pushes it (HerderImpl::prepareEarlyTxSet).
   The selection is used only if N closed with exactly the executed value,
   the limits it assumed are those of the closed ledger, its time-dependent
   transactions are valid for the close time the set gets, and a build of
   its size would take time (an empty selection is never proposed).
   Otherwise the ledger gets one build as before.

2. The leader executes its own set as soon as it is prepared. The value the
   trigger will propose is known when the set is (HerderImpl::
   executePreparedTxSet), so the ledger goes to the apply thread then, not
   when the first ballot statement is emitted. If that execution finds the
   set invalid before the proposal, the set is dropped.

3. The builder's check of the written set (the XDR round trip and the
   validation of the re-parsed set) runs on a background thread for sets the
   herder builds as a leader on protocols that validate tx sets at apply
   (TxSetSelfCheck). The structure of the selection and the shape of the
   XDR are still checked before the set leaves the builder.

makeTxSetFromTransactions is split for this: selection takes the limits
(TxSetBuildLimits) and the validation context explicitly instead of reading
the LCL through the application, so that it can run for a ledger that is not
the LCL and off the main thread.

Two faults of the base are fixed on the way: the pre-validation's transaction
frames referred to a network ID owned by a task's closure, and a build kept
one transaction per source account within each phase but not across the two
phases (the builder's own check then threw).

No change to apply, commit, tx-set validation or the wire format.
…d the deferred self-check

New test cases (tag [early-txset]) in ProposalPrevalidationTests.cpp. Two
leader-build tests in HerderTests.cpp count builds instead of mempool pulls
(the leader's early execution pulls candidates for the ledger after it), and
"SCP Driver / validateValue txSet cached" no longer depends on which of its
two validators leads (it failed on the base at --base-instance 1250).
…fore it closes, and pulls less for a build

Two changes to how a leader gets its transaction set, on top of the
reused parts of the closed attempt (builder split, deferred check of the
written set, early execution of a prepared set).

1. A smaller build. A build pulls the capacity of a set plus a margin
   (EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT, default 10) from the mempool
   instead of twice the capacity. If a phase loses candidates (invalid, or
   a second transaction of an account) and ends below its capacity while
   the mempool has more, the build pulls again, twice as much, at most
   three times. The structural check of the selection runs on a second
   thread beside the XDR write and is awaited before the set is handed to
   anyone. The re-read of the written set is left to the node's own
   execution of its value where it executes its votes, and runs in the
   background otherwise.

2. Early selection (EXPERIMENTAL_EARLY_TX_SET_SELECTION, default on). When
   a node votes for a value for ledger N and expects to lead round 1 of
   N+1, it plans a background job that selects the set for N+1 while N is
   in its ballot: on the last closed ledger (N-1) and a prediction of what
   N's set does to it (sequence numbers consumed, full fees charged). The
   job starts as late as it can and still finish before the set is needed.

   The prediction is never trusted. At the build for N+1, if N closed with
   exactly the value the job was for, the selection is completed: every
   one of its transactions is validated against N as it closed, with the
   set's close time, by the function that validates a received set; what
   fails is removed; while a phase has room, further candidates (what the
   job did not select, then what the mempool holds that the job did not
   see) are validated the same way and added, to a parallel Soroban phase
   one by one (ParallelPhaseExtender), to a sequential phase by selecting
   it again; base fees are set by the rule of a build; and the set goes
   through the same checks and hand-off as a built one. Anything else (N
   closed with another value, with upgrades, the job not finished, a
   failed check) is a build.

Tests: herder/test/EarlyTxSetTests.cpp. A completed set is validated in
full on the closed ledger, read back from its bytes, for: the candidates
being exactly N's transactions; the same accounts with their next
sequence numbers; random overlaps; a transaction of N failing; an account
drained, merged away or re-keyed by N; fee bumps; N's set skipped at
apply; N closing with another value, close time or an upgrade; a job
that is late, still running, or overlapping the next one; time bounds.
Three of them show the set invalid when the validity pass is switched
off. Fullness: top-up with late arrivals, refill after removals and for
builds with 0 to 90% invalid candidates, Soroban top-up and conflicts,
base fees after removal and refill.

Nothing in apply, commit, set validation or the wire format changes.
…ne that lost candidates looks deeper whatever the mempool's top holds of its kind

The stop rules of the completion's top-up and of a build's refill had two
shortcuts that could leave a phase short while the mempool held valid
transactions for it:
- eight or more valid left-overs that could not be added closed the phase,
  also when they only had nowhere to go (a full cluster of one entry, a lane
  of its own) and later arrivals would have fit;
- a phase that lost candidates pulled deeper only if the mempool's top held
  at least the phase's capacity of its kind (compared in transactions against
  operations), which a mempool top of the other phase prevents.

Now, as in a build: a phase stops at its capacity; after one look at the
mempool's top if it lost nothing (lost: removed by the validity pass, invalid,
or of an account already in the set; a valid candidate that did not fit is
not a loss); and otherwise when a deeper pull holds nothing new of its kind.

Tests: many left-overs with nowhere to go; a completion and a build with a
mempool top of the other phase.
Two lines of the load generator commits that clang-format 20 (CI's version)
breaks differently. No other change.
…ixth experiment (the next tx set selected before the close)

- What this branch is: experiment 6 (a smaller build; the leader of ledger
  N+1 selects its set from ledger N's set during N's ballot and completes it
  on the closed ledger), with its two supporting changes.
- Results: the bucket list settings on 6 validators; the base with the meta
  off at 90 validators (801 ms per ledger at 6,000 TPS, 873 ms at 8,000 TPS);
  experiment 6 at 8,000 TPS (build 176 -> 121 ms, 858 ms per ledger, no
  invalid or short set), and why the shorter build does not show in the
  block time.
- How it was measured, limits of the numbers, known limitations, test status.
…rom the schedule's nodes and the slot's seed

EXPERIMENTAL_TX_SET_TREE = off | star | hash | rtt (default off).

Until now the round's leader writes its whole (compressed) tx set to every
peer: 89 copies at 90 validators. With a tree it writes it to its children
only, and every node passes the bytes on to its own children.

- The tree (overlay/src/flood/txset_tree.rs). For a slot and a leader: the
  leader is the root, the other nodes of the leader schedule follow in the
  order of SHA256(seed || leader || node), and position i's children are the
  positions F*i+1 .. F*i+F (the complete F-ary tree, the least depth a
  fanout of F allows). The seed is the seed of the slot's leader schedule.
  F is EXPERIMENTAL_TX_SET_TREE_FANOUT; 0 = the smallest F that reaches
  every node in two hops (9 for 90 nodes). Every node derives the same tree
  from what it already agrees on.
- Core tells its overlay the nodes and the seeds of the next two slots at
  every ledger close (SetTxSetTree, HerderImpl::updateTxSetTree).
- A pushed frame carries a header in front of the ordinary tx-set frame:
  kind, slot, leader, hop count and the leader's send time (and, in rtt
  mode, the order of the nodes).
- A node relays a tagged frame to its children before it decompresses or
  checks it, if it knows the tree of that slot, the sender is its parent in
  it (by the validator identity of the connection) and it has not relayed
  that frame. It then verifies and files the set by the hash of its content
  as before.
- A node that expects a set through the tree waits 300 ms from the first
  mention of the set's hash before it asks a peer for it; otherwise every
  node the leader did not push to would ask it when the proposal arrives.
- star: to every peer as before, with the header (the baseline for
  measurements). A leader that cannot reach one of its children, or does
  not know the slot's tree, pushes a star.
- rtt: the leader measures the round trip to every peer (a 13-byte probe
  on the tx-set stream), groups its peers by the gaps in the sorted times,
  gives every group relays in proportion to its size and fills a relay's
  children from its own group. The order travels in the header.
- Metrics overlay.txset-tree.* (copies written as root and as relay,
  received, refused, duplicates, unreachable children, requests put off and
  those sent after the wait, delay since the leader's send in total, per
  hop count and in buckets) and overlay.txset-push.sent-bytes.

Nothing in SCP, the schedule, apply or a tx set's content changes: a set is
still identified by the hash of its content, computed by the receiver, and
reaches Core only when an SCP statement names that hash.

Tests: the tree's logic (order, least depth, header, relay decision, wait,
rtt order), seven overlay nodes on loopback (delivery down the tree, a relay
that does not relay, a frame from a node that is not the parent, star, an
explicit order, probes), and 7-validator networks under consensus in each
mode.
…odes below a busy child

After an independent review of the first commit:
- Only a node that this node's Core expects to lead the slot can root a
  tree: Core sends the leaders of the first three rounds of the next slot,
  and for the slot after it those plus the first three of a new order
  (SetTxSetTree carries them per slot). A frame that names another leader is
  filed and not relayed.
- An order chosen by the sender (rtt mode) is followed only by nodes in rtt
  mode, only for a tree of two levels, and only if it refers to the same
  list of nodes (a digest of the list travels with the order).
- A node relays at most two different frames per slot and leader (it told
  copies apart by their first and last bytes before, which a sender can
  vary), and has at most one relay in flight per child.
- A frame whose body decode would refuse for its size is not relayed.
- A probe answer is a measurement only if it answers the probe this node
  sent that validator last; one answer to a peer's probes is in flight at a
  time.
- If the leader cannot send a child the set (an earlier push to it is still
  in flight), the nodes below that child get it from the leader directly, as
  a star. Before, they waited for a relay that had nothing to relay.
- A node that is not one of the tree's nodes does not wait before it asks.
- The layout by round-trip times is computed outside the tree's lock.
- Counters: overlay.txset-tree.child-busy, relay-busy, relay-limited. The
  delay-le-* buckets are disjoint ranges; their descriptions say so.

Tests for each of these (tree module, seven overlay nodes on loopback, the
IPC payload's layout).
…elay in the middle

The leader grouped its peers by gaps in the sorted round-trip times, where a
gap was 8 ms or a fifth of the larger time, and gave a group's relays the
nearest nodes of the group. Two regions at similar distances from the leader
(us-west-2 and ap-southeast-1 seen from eu-central-1, for one) became one
group whose relays were all in the nearer region: the nodes of the other one
got the set over two long hops.

Now a group ends at a gap of more than 8 ms only, and a group's nodes, sorted
by round-trip time, are cut into consecutive runs, one per relay, with the
run's middle node as its relay. Regions at the same distance still differ by
a little, so the runs mostly fall apart along them. With more groups than
relays all nodes are one group, laid out the same way (before: nearest
first, which is not a two-level tree a receiver follows).

In a model with typical round trips between four AWS regions (the item's
results/dissemination-model.py) the mean delay of a set goes from 95 to 84 ms
and the 90th percentile from 178 to 134 ms, which is what a leader could do
if it knew every distance.
The test sent three frames back to back and expected two of them to reach a
child; with one relay in flight per child the second could be skipped.
…re-coded pieces

EXPERIMENTAL_TX_SET_TREE="coded": the leader cuts the frame it would push
into k pieces, adds m-k Reed-Solomon parity pieces (16 pieces of which any 8
rebuild the frame at 90 validators), signs the root of their Merkle tree with
its overlay key and writes every piece to that piece's relays; a relay passes
its piece on. A node gets its m pieces from m different senders, so up to m-k
of them can be down, slow or hostile without the node having to wait and ask.

A piece is kept and passed on only if it comes from the node that owes it in
the layout, names a leader Core expects for the slot, carries the leader's
signatures and a Merkle path to the signed root. A rebuilt frame goes the way
of every received frame (decompress, strict parse, content hash).

EXPERIMENTAL_TX_SET_TREE_WITHHOLDERS_FOR_TESTING=N makes the N validators
with the smallest node IDs pass nothing on, in the tree modes and in coded
mode (tests and fault runs).
…'s seed, and do not code as a leader nobody expects

From the review of the coded mode:
- A piece frame is counted against its sender (six per slot and leader)
  before the layout is consulted, whatever the frame turns out to be, and
  the layout is worked out once per slot and leader.
- The signed message holds the seed of the slot's layout, so a root signed
  in another history of a network with the same name and keys is refused.
- A node that Core does not name as a leader of the slot pushes its set to
  everyone instead of cutting pieces (or rooting a tree) that every node
  would refuse.
- A rebuilt set counts as received from its leader, not from the sender of
  the piece that completed it.
- EXPERIMENTAL_TX_SET_TREE_WITHHOLDERS_FOR_TESTING is a testing-only option.
… signed its root

From the second review of the coded mode:
- The overlay key that signed a root is kept with the root, and a set
  rebuilt from pieces counts as received from that overlay, connected or
  not. Before, a node not connected to the leader charged the sender of
  the piece that completed the set, which the leader could choose.
- A root gets its place only from a frame whose signatures were checked.
- The leader named in a piece must be one of the layout's nodes before
  the frame is counted.
- A node that is not a named leader of the slot logs that it pushes to
  every peer (TXSET_CODED_STAR, TXSET_TREE_STAR).
- Tests: the unexpected leader in hash mode, the attribution, a settle
  before a counter is read. Docs and comments that still described the
  old behaviour of an unnamed leader.
…eal apply at 8,000 and 10,000 TPS, against the spanning tree and the push, with validators withholding, and what limits 10,000 TPS
…ne config line put back as clang-format has it
@SirTyson
SirTyson force-pushed the perf/txset-spanning-tree branch from cf9b50d to 3714745 Compare November 14, 2026 20:48
@socket-security

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​reed-solomon-simd@​3.0.1811009310080

View full report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant