Repository navigation
Select loadgen source accounts in constant time - #5458
Merged
SirTyson merged 5 commits intoNov 10, 2026
Merged
Conversation
This was referenced Sep 16, 2026
SirTyson
force-pushed
the
pr/batched-getdata
branch
from
September 16, 2026 02:56
af33e37 to
cf22578
Compare
SirTyson
force-pushed
the
pr/loadgen-account-selection
branch
from
September 16, 2026 02:56
26e0f29 to
03a55f7
Compare
SirTyson
force-pushed
the
pr/batched-getdata
branch
from
September 16, 2026 03:23
cf22578 to
f6ffa05
Compare
SirTyson
force-pushed
the
pr/loadgen-account-selection
branch
from
September 16, 2026 03:23
03a55f7 to
195adf7
Compare
SirTyson
force-pushed
the
pr/batched-getdata
branch
from
November 9, 2026 21:38
f6ffa05 to
ecfcdf7
Compare
SirTyson
force-pushed
the
pr/loadgen-account-selection
branch
from
November 9, 2026 21:38
195adf7 to
e5ec7d4
Compare
This was referenced Nov 9, 2026
Merged
Contributor
Author
|
Updated (force-pushed, 2026-11-09): rebased onto the updated #5457 (stack now includes the master merge), plus four follow-up commits: per-ledger account matching proportional to the tx set, no per-tx metric re-registration/envelope copies, hashing externalized txs without rebuilding frames, and test fixes. Full suite passes at this head; a perfnet run of it is in progress and will be posted here. Commits on top of the base:
|
SirTyson
force-pushed
the
pr/batched-getdata
branch
from
November 10, 2026 21:05
ecfcdf7 to
2e5b700
Compare
SirTyson
force-pushed
the
pr/loadgen-account-selection
branch
from
November 10, 2026 21:05
e5ec7d4 to
a3d0e18
Compare
Port the focused account-pool change from 75f3fbc onto batched-getdata. Use uniform indexed selection and swap-removal instead of walking the available-account hash set for every transaction. Preserve release only after application, including Soroban setup transitions. Cover repeated account recycling and consecutive runs using real ledger application, with more submissions than available accounts and no rejected transactions.
cleanupAccounts runs on the main thread in valueExternalized for every ledger, on every node, whether or not that node generates load. It hashed and allocated a set entry for the source of every transaction in the externalized set (about 6,000 to 7,500 per ledger in perfnet's 3,000 TPS runs), then walked every in-use account with a std::map lookup. When ledgers fall behind the offered load, the in-use backlog grows by tens of accounts per ledger per generator, so the walk grew with it on the path between externalize and the next ledger. Keep in-use accounts indexed by public key instead. Accounts selected since the previous call are indexed at the start of the next one (their keys are known once TxGenerator has generated a transaction for them), and each externalized transaction's source is looked up in that index. The cost is now one allocation-free lookup per transaction in the set plus one index insert per newly selected account, independent of the backlog, with an early return when no account is waiting. Which accounts are released, and when, is unchanged. Add a test for perfnet's block_time_check load: synthetic SAC payments in overlay-only mode with more transactions than source accounts, which only completes if accounts return to the pool after their ledgers close.
…tion LoadGenerator::execute built a fresh TxMetrics for every submitted transaction, which resolved all ten loadgen meters through MetricsRegistry::NewMeter: a MetricName allocation, a mutex, and two searches of the registry's std::map each. It also copied the whole envelope into a new StellarMessage (twice for classic payments) only to measure its size. Use the TxMetrics member resolved once at construction, which refers to the same registry meters, and compute the message size as the 4-byte message type discriminant plus the envelope's XDR size. The recorded values are unchanged. Extend the overlay-only synthetic SAC test to check the per-transaction meters, including that loadgen.txn.bytes still equals the wire size of each submitted StellarMessage.
The section asserted that no transaction was rejected, but on this branch HerderImpl::recvTransaction hands every transaction to the overlay and always returns TX_STATUS_PENDING, so loadgen.txn.rejected cannot move. The checks that matter are that every transaction applies successfully and that the run completes, which requires each account's cached sequence number to match the ledger. Say so, drop the rejected assertion, and stop cranking as soon as a run fails instead of waiting for the timeout. Submit 2x the accounts at 200 tx/s instead of 3x at 100 tx/s. That still forces every account through the pool more than once and stays within the unique-account check (3 x 200 txs per one-second ledger = 600 <= 1,000 accounts), and each run takes about 10 s of submission instead of 30 s.
HerderImpl::processExternalized needs two things from every transaction in the externalized set on the main thread: its full hash, so the Rust overlay can drop it from the mempool, and (in BUILD_TESTS) its source account, so loadgen can release it. It got both by calling createTransactionFrames, which copies every envelope into a new TransactionFrame (with an OperationFrame per operation) only to hash it and read its source, then frees them all. That is thousands of frames per ledger at perfnet's load. Add TxSetXDRFrame::forEachTransactionEnvelope, which visits the set's envelopes in createTransactionFrames' order, and getTransactionFullHashes, which hashes each envelope's XDR in place: the same SHA-256 that TransactionFrame::getFullHash and FeeBumpTransactionFrame::getFullHash compute. LoadGenerator::cleanupAccounts takes the tx set and reads each source from the envelope with getEnvelopeSourceID, which mirrors getSourceID: the unmuxed source, or a fee bump's inner source. The overlay receives the same hashes in the same order, and loadgen releases the same accounts. Add a test that builds legacy and generalized sets (sequential and parallel phases; v0, v1, muxed and fee-bump envelopes) from XDR and checks the visited envelopes, full hashes and sources against the frames createTransactionFrames builds from the same set.
SirTyson
force-pushed
the
pr/batched-getdata
branch
from
November 10, 2026 21:22
2e5b700 to
7a6914d
Compare
SirTyson
force-pushed
the
pr/loadgen-account-selection
branch
from
November 10, 2026 21:22
a3d0e18 to
0fd7d0e
Compare
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
## Update (2026-11-09) This PR is rebased onto the reworked #5453, which merges current `master` (362dbc5) into `overlay-v2-shared`. That merge brings in the final #5423 (millisecond close times, gated at protocol 30, with the protocol-30 bump). It also carries the overlay-v2 adaptations those need, including the Rust overlay moving to `stellar-xdr` 30.0.0 to decode millisecond `StellarValue`s. **The old first commit ("Enable millisecond consensus close times at protocol 28") is dropped.** The master merge in #5453 supersedes it. It was an earlier draft of #5423 behind a compile-time `MS_CLOSE_TIME` flag at protocol 28, with its own XDR pin and fixtures. The overlay-v2-specific parts needed for correctness are now in #5453. This PR is now the trigger-timing work on top of millisecond close times: 1. **Account for actual elapsed nomination time in the trigger drift allowance** (ported unchanged in substance). The next-ledger trigger is anchored on the previous externalized close time C and fires at `C + target`. It falls back to the local prepare-start anchor when the time elapsed since C exceeds the target plus an allowance for nomination. That allowance used to be the sum of nomination timeouts that had fired. Now it is the steady-clock time from the local trigger to entry into ballot. A slow round that never runs a timeout callback no longer counts as clock drift: slow tx-set fetch or validation, a busy main thread, or ballot starting before the expired timer runs. When the network is behind, the next trigger fires immediately instead of waiting out the nomination again. The extra wait a fallback can add is bounded by our clock drift plus how late we triggered relative to the winning close time. 2. **Record the nomination trigger only when nomination proceeds** (new). If a node skips nomination because its clock is behind the last close time, it no longer credits the wait for its peers to the allowance. 3. **Test the trigger anchor with sub-second close times** (new). With close times 0, 250 and 999 ms into the second, the next trigger is due exactly the target after the close time, and the fallback boundary holds to the millisecond. 4. **Add trigger timing metrics** (new). Together they show whether a 2 s ledger is bound by apply, by nomination or by event-loop delay: - `scp.timing.trigger-to-ballot` (timer): proposal construction plus nomination, which is the drift allowance. - `scp.trigger.lateness` (timer): how late the trigger callback ran after its scheduled time. - `scp.trigger.immediate` (meter): the next trigger was already due when it was scheduled. 5. **Skip a delayed nomination emit that was already sent** (new, test-only code path). With `ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`, a deferred nomination emit could fire after `processEnvelope` had already emitted the same statement. SCP then rejected the duplicate and threw "moved to a bad state (nomination)", which made the hidden "consensus close time trigger timer" test flaky under parallel load. The deferred emit now only sends a statement newer than our last one. A new SCP test replays the race deterministically. None of these commits change values, validation or ledger output. They only change local trigger scheduling, metrics, logs and BUILD_TESTS-only code. **Follow-up under consideration:** for proposals rebuilt at the trigger, propose the scheduled trigger time as the close time (bounded by `now` and LCL+1 ms), so a leader's trigger lateness no longer shifts the network's cadence. We'll do it if `scp.trigger.lateness` p75 exceeds about 20 ms on perfnet. Testing, at `285aa42fc92d` on top of #5453 `04fbddb7d`: - New and changed tests: `[trigger-construction]` 8/8 (the drift-allowance tests and the 3 new ones) and `[early-nomination]` 6/6 pass. - `[herder]` (partitioned `make check`): 77/77 test cases pass, including the hidden real-clock trigger-timer simulations. - At `4f577bd25`: the new SCP test passes. A copy of the test without the fix fails with the original exception, so the test reproduces the race. "consensus close time trigger timer" passes 3/3 on its own and 2/2 inside a partitioned `[herder],[scp]` `make check` (97/97 test cases, full check passes both times). - Final branch: restacked onto the final #5453 head `14de3972a` (two test-only commits on top of `04fbddb7d`); the five patches are identical (`git range-diff`). The full suite passes on the #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust, check-nondet). --- ## Original description > The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs. # Description Merges in #5423. This change adds ms closeTime. By itself, it doesn't improve TPS or latency significantly. However, it reduces our long tail fo very slow blocks, and also allows us to target intrasecond or subsecond block times, which are not currently possible at whole second resolution. In addition to taking the change from the PR, the 2nd commit makes some minor adjustments to when we call the fallback timer. This was intentionally conservative for main net launch assuming lots of drifting clocks, but no reason we can't tighten it here. Results, measured with 10 tier 1 topology (no watchers) with 2 second block time target: ## 2000 TPS | Metric | Original overlay-v2-shared |Changes | |---|---:|---:| | Median block interval | 2.090 s | 1.993 s | | p99 block interval | 4.563 s | 4.298 s | | Maximum block interval | 6.219 s | 5.090 s | | Harness verdict | **FAIL** | **FAIL** | Rebased on #5453 so opening as a draft for now. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#opening-a-pull-request) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
## Update (2026-11-09) Rebased onto the updated #5454 (which now sits on #5453's merge of `master` @ 362dbc5; the overlay crate moved to `stellar-xdr` 30.0.0 there). The original commit is unchanged apart from a test import updated for stellar-xdr 30.0.0 (`stellar_xdr::curr::` to `stellar_xdr::`). Eight commits on top, each with tests (an independent review found no correctness, wire-format or deadlock issues in the original commit; its low-severity findings are fixed below): - **Stop serving a demand after a failed write.** A response job returns at the first write that fails after the stream retries (usually a disconnect), instead of looking up, writing and logging a failure for every remaining batch. The requester re-demands unanswered hashes after its response timeout. - **Frame responses straight from the shared transaction bytes.** Each frame's discriminant and envelope are copied directly into the batch, with no per-transaction allocation; a transaction larger than the batch budget is written as two borrowed parts with no copy. Each batch is looked up under one TX-buffer lock, released before frames are copied or written, so local submissions on the event loop never wait for a response copy. - **Buffer inbound TX stream reads (64 KiB).** One QUIC read takes in whatever has arrived, often a whole response batch, instead of two reads per frame. The reader never waits for more data than is available. - **Queue INVs for all peers under one lock**, for both relayed and locally submitted transactions, instead of one batcher lock per peer. - **Handle each received INV batch with one seen-set lock and one tracker lock**, instead of both locks per advertised hash. Request decisions and their order are unchanged. - **More tests:** batching edges (an oversized transaction first, two in a row, frames exactly at the budget, carried hits, misses); a failed batch write retried whole on a reopened stream; buffered reads; delivery of INV batches completed by either path; INV request decisions and order. Wire bytes are unchanged. Testing on the rebased branch: - 194 overlay crate tests pass (`cargo test` in `overlay/`, 3 ignored); `cargo fmt --check` is clean. - The standard build succeeds, and 25 C++ overlay-IPC test cases (136 assertions) pass: `[overlay-ipc]`, `[overlay][ipc]` and the 10-node `[overlay-ipc-large]` consensus case. - The full suite passes on the #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust with these overlay tests, check-nondet). The follow-up commits have not been measured separately on perfnet; the numbers above are for the original commit on the previous base. The perfnet run of the final #5458 head (which contains this PR) is reported on #5458. **Next lever (not in this PR):** a node waits up to 100 ms (checked every 50 ms) before announcing new transactions to a peer, so each flooding hop adds 100-150 ms before the peer can request them. A perfnet A/B of shorter delays (50 ms and 25 ms) is the planned follow-up, comparing tx pull latency, tx-set fetch p99 and block intervals against bandwidth and CPU. --- ## Original description > The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs. # Description This PR batches GETDATA transaction responses in the Rust overlay. Previously, a request for many transactions created a separate task and stream write for every response. Under load, that produced unnecessary scheduling and stream-lock overhead, and a slow peer could leave the entire response demand queued in memory. We now serve each request in one asynchronous job, combining complete transaction frames into writes of up to 64 KiB. Each transaction keeps its existing wire framing, and the writes use the existing stream priorities and retry policy. Transactions larger than the batch budget are sent individually. Here, the control image is the PR this is based on #5454. This is the 10 tier 1, 2 second target block latency test. Median block intervals are from the canonical node; p99 is the worst node over the full measurement window. ## 2,000 TPS | Metric | Control | Batched GETDATA | |---|---:|---:| | Median block interval | 1.993 s | 2.000 s | | Worst-node p99 block interval | 5.090 s | 2.508 s | | Tx-set fetch p99 | 1.720 s | 0.778 s | | Harness verdict | FAIL | PASS | ## 3,000 TPS | Metric | Control | Batched GETDATA| |---|---:|---:| | Median block interval | 2.434 s | 2.363 s | | Worst-node p99 block interval | 4.122 s | 4.524 s | | Tx-set fetch p99 | 1.757 s | 1.087 s | | Harness verdict | FAIL | FAIL | At 2,000 TPS, the largest improvement is in the tail: worst-node p99 falls by about half, and nearly all tx-set fetches longer than two seconds disappear. This is consistent with reducing transaction-response write overhead and contention affecting consensus transfers. # Checklist - [x] Reviewed the contributing document. - [x] One commit on top of #5454, with no merge commits. (Now nine: the original plus eight follow-ups, see the update above.) - [x] Formatting checked with `make format` and `cargo fmt --check`. - [x] Release overlay build and Jenkins package build succeeded. - [x] Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++ overlay-IPC cases / 82 assertions. - [x] Performance comparison included above.
Base automatically changed from
pr/batched-getdata
to
pr/merge-ms-close-time-base
November 10, 2026 22:02
SirTyson
marked this pull request as ready for review
November 10, 2026 22:04
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
## Update (2026-11-10): cheaper leader build, ballot-start signature
prewarm, IPC size fix; perfnet passes on c6id
Head is now `ebdd73635`, 8 commits on top of `c26b191fa`. They fix the
two things the 2026-11-09 perfnet run found, plus a liveness bug the
next run found.
### Leader build
On perfnet the leader's 1.1 s build was on the critical path every
ledger.
- `ba4955347` **validates only what selection can use.** The leader
still pulls 2x the capacity, but validates each phase in mempool order
only until it holds 5/4 of the capacity in valid transactions. If some
are invalid it refills: twice the shortfall, then the rest. The selected
set is the same unless selection would have reached past the validated
prefix.
- `a1bffae42` hashes the candidates on the validation workers, not the
main thread. That was 18% of a cold build's instructions on hosts
without SHA extensions, such as c5d.
- `966d3bb41` precomputes the parallel builder's sort keys.
- `95be616c7` decodes mempool pulls straight from the IPC payload,
without copying.
- `2a30c1dc0` adds phase timers
`herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}`
and a hidden leader-build benchmark.
- Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5
medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%).
Main-thread instructions -41% (callgrind).
### Signature prewarm from ballot start, on every validator
(`a91547bec`)
The 2026-11-09 follower prewarm covered only about half of the leader's
set. The fees are equal, so the mempool orders transactions by hash, but
selection breaks ties randomly. The leader also had no prewarm.
- Now, when slot N starts balloting, every validator verifies the
signatures of exactly the window that the N+1 build validates first.
This runs while N finishes balloting, externalizes and closes. The main
thread only makes the overlay round trip; decoding and verification run
on background tasks.
- New metrics: `herder.prewarm.{pull,duration}`,
`herder.prewarm.{txs,verified,incomplete}` and
`herder.txset.{build,validate}-sig-cache-{hit,miss}`.
- Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set
validation takes 38 ms (cold 102). 0 cache misses.
- `0cf227b94`: Core's tx-set fetch timer is now `scp.fetch.txset`, with
`scp.fetch.txset-retry`. `overlay.fetch.txset` is now only the Rust
overlay's time; it had been fed by both.
### Bug fix: oversized IPC payloads (`ebdd73635`)
Perfnet found this on `a91547bec`, on c5d.
- **What happened:** 80 s into the load the whole network stopped at one
ledger. 16 of 30 nodes logged `Overlay IPC connection closed` within
0.35 s of each other. The rest then lost quorum, and all lost sync.
- **Cause:** the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700
SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the
Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats
a rejected length as a corrupt stream: it stops reading and never
recovers. Every validator starts the pull at the same ballot.
- **Fix:**
- The limit is now 64 MiB on both sides (`ipc::MAX_PAYLOAD_BYTES` and
`MAX_PAYLOAD_SIZE`, which must match).
- The overlay returns the longest prefix of the top transactions that
fits, with a warning, so no mempool can produce an oversized response.
- Neither side sends an over-limit payload: Rust refuses with
`PayloadTooLarge`, and Core refuses before writing anything.
- Core now logs the type and length when it rejects one.
- This also covers a maximum-size peer tx set (`MAX_TXSET_XDR_SIZE` + 32
bytes), which could have hit the old limit too.
- **Test:** a new `[overlay-ipc]` test with the real Rust overlay. A 20
MiB top-transactions response arrives whole; with the old code the pull
returns nothing (`0 == 20`). An 80 MiB pull is truncated to the 63
highest-fee transactions, and the IPC keeps working.
### Testing
- **`a91547bec`:**
- Laptop: new and changed tests, and the step-2 tags (149 test cases).
- Devbox: full suite `--all-versions` PASS (selftest 604 test cases /
8,638,027 assertions, check-nondet, check-rust).
- **`ebdd73635`** (devbox):
- New tests: the `[overlay-ipc]` test above, plus IPC tests for a 20 MB
message, an over-limit length (rejected, with a receive timeout so a
regression fails instead of hanging) and an over-limit send (nothing
written). Rust tests cover truncation and the send guard.
- `[ipc]` (7 test cases), `[overlay-ipc]` (20), partitioned
`[overlay]~[ipc-crossproc]` and the Rust `ipc::` tests (23): all pass.
- Two pre-existing failures reproduce on `a91547bec` without the change:
- The Rust `test_tx_not_blocked_by_scp_flood` fails its own validity
check (an SCP flood of 6.7 ms is under its 10 ms floor).
- The hidden `IPC cross-process communication` test needs an unbuilt
`overlay-stub` binary.
- Full suite `--all-versions` on `ebdd73635` (devbox): PASS. selftest
605 test cases / 8,638,079 assertions across protocol versions,
check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban
host tests (103 `test result: ok`, 0 failed). The overlay crate's own
tests run separately; see above.
### Perfnet, head `ebdd73635`
30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2
s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s
measured. One run per instance type, both on the same day: c5d.2xlarge
(perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake,
SHA-NI). Same 8 vCPU / 16 GiB.
**c6id.2xlarge: PASS.** All four windows on target, inclusion passed
(2,880,000 txs), all checks passed.
| Window | avg p75 | avg p99 | p75 range over nodes |
|---|---:|---:|---|
| 1 | 2,112 ms | 2,334 ms | 2,103-2,129 ms |
| 2 | 2,099 ms | 2,391 ms | 2,091-2,113 ms |
| 3 | 2,101 ms | 2,324 ms | 2,092-2,110 ms |
| 4 | 2,099 ms | 2,354 ms | 2,089-2,110 ms |
**c5d.2xlarge: FAIL** in window 1, but 19% faster than the previous head
and #5458. The c5d network fell behind the offered load, so sets were
full at 7,500 and the mempool backed up. That's exactly the case that
stalled `a91547bec`, and here there were no IPC disconnects or sync
losses. Correctness passed: error counters, in sync, no fork, still
closing. Inclusion was skipped because the load stopped after window 1.
| c5d, window 1 (5-min means) | #5458 head | #5459 `c26b191fa` | **#5459
`ebdd73635`** |
|---|---:|---:|---:|
| avg p75 / p99 | 3,250 / 3,891 ms | 3,264 / 3,672 ms | **2,637 / 2,920
ms** |
| Ledgers per 5 min | 96 | 96 | 119 |
| Trigger to first ballot | 2.52 s | 2.55 s | 1.94 s |
| Leader build | 1.15 s | 1.11 s | 0.68 s (candidate validation 0.20 s)
|
| Received-set validation | 0.34 s | 0.52 s | 0.33 s |
| Nomination timeouts per node | 5 | 106 | 34 |
| Ledger close | 0.229 s | 0.228 s | 0.229 s |
Both instance types, same build:
| | c5d | c6id |
|---|---:|---:|
| Ledger interval | about 2.53 s | about 2.03 s |
| Txs per ledger | 7,500 (backlog) | about 6,100 (kept up with 3,000
TPS) |
| Trigger to first ballot | 1.94 s | 1.22 s |
| Leader build | 0.68 s | 0.27 s |
| Received-set validation | 0.33 s | 0.20 s |
| Signature-cache coverage: build / validation | 89.5% / 98.7% | 87.9% /
97.8% |
| Prewarm: mean duration / main-thread pull / stopped early | 0.21 s /
22 ms / 0.1% | 0.10 s / 7 ms / 0% |
| Ledger close | 0.229 s | 0.164 s |
Notes:
- **No A/B on c6id.** #5458 has not been run on c6id, so the c6id PASS
shows this head can hold the target on current-generation x86. It
doesn't show how much of that comes from this PR. Every earlier run of
this fixture was on c5d.
- **Two regimes.** On c6id the network keeps up, so sets stay at about
6,100 transactions and triggers aren't all due at once; #5454's early
preparation runs, giving about 2 builds per ledger network-wide. On c5d
it falls behind, sets are full, and every ledger is heavier. So
per-ledger timings aren't like for like across the two.
- **What c5d still needs** (about 0.24 s to pass):
- Received-set validation still takes 0.33 s with 98.7% of its
signatures cached, so the rest of validation is now the cost.
- About 245 ms of the c5d build is outside the phase timers.
- The remaining trigger-to-ballot time beyond build, fetch and
validation is about 0.57 s of nomination messaging across 4 regions.
- n = 1 per instance type.
---
## Previous update (2026-11-09)
_Genesis-upgrade fix, fetch robustness, follower signature prewarm._
Rebased onto the updated stack (#5453 now merges `master` @ 362dbc5:
protocol 30, millisecond close times; #5454, #5457 and #5458 updated the
same day). The PR's original four commits are kept (adapted to the new
base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten
commits are added on top.
### Bug fix: networks starting at the genesis protocol stalled
(465f6a4)
**Root cause.** With leader-only construction (5fff543), followers no
longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's
set is a legacy
`TransactionSet`, and the Rust overlay only carries
`GeneralizedTransactionSet`s (Core caches,
Rust decodes and Core receives only that type). So a follower could get
the leader's set only by
being promoted to leader by nomination timeouts and building it itself,
one slow round at a time.
Before this PR every validator built the identical empty legacy set (it
depends only on the
previous ledger; upgrades live in the `StellarValue`), so it was always
local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade
ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are
unaffected.
**Fix.** When the LCL is before protocol 20, every validator constructs
its nomination value at
the trigger again, as before; it is still voted only by round leaders.
Leader-only construction
is unchanged from protocol 20 on.
**Test.** `Rust overlay upgrades from genesis through legacy tx sets`: 4
nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol
30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at
ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other
changes below in place,
still fails (2/2), so they don't mask the bug.
### Robustness on the same path
- 193fee3: a locally built set completes a pending fetch of the same
hash, so envelopes waiting
for it are processed at once (the IPC path had lost upstream's
ItemFetcher behaviour).
- 942a6b3: a follower promoted to leader after losing sync can still
propose; tracking is
checked at the trigger, as before. This is what made the perfnet stall
permanent.
- ab4eb14, 5964ce6: unanswered tx-set requests are retried and
reported. Core re-requests
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on
every doubling of the
request count and counts `overlay.fetch.txset-retry`; Rust sends a retry
to a random other peer
and prunes stale reservations by slot. Core also checks a delivered
set's content hash.
- 5c159ae: test for a leader whose own build completes a pending fetch
and externalizes the
slot inside SCP's nomination supplier.
- Rebase onto the new base: the test-only nomination override from #5481
now also hands its
substituted set to the overlay (31e6147), since this PR caches the
built set earlier.
### Performance
- 3903e4e, f2643ba: follower signature prewarm. The received-set
validation timer rose (about
158 -> 205 ms) because followers no longer validated their own mempool,
which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4
tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at
the trigger and verify
master-key signatures on background tasks (at most 20 verifications per
tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a
local build or
externalization starts). They still build and compress nothing.
- c26b191: no early tx-set preparation when the trigger is already
due. Both timers fired at the
same instant in unspecified order, so a predicted leader could build its
set twice.
- 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths
(leader serializes in place,
follower decodes in place and moves into the frame). Sub-millisecond for
classic sets.
### Testing
- Laptop, on c26b191: new and changed tests (24, including the genesis
repro and the trigger,
early-nomination and #5481 tests) and partitioned `make check` for
`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]`
(143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
- Full suite `--all-versions` (devbox, at c26b191): selftest 598 test
cases / 8.5M assertions across protocol versions, check-sorobans,
check-rust and check-nondet pass.
- **Perfnet, head `c26b191fa`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply,
tx sets capped at 7,500; n = 1; run in parallel with the #5458 head on
the same fixture):
- **The genesis-upgrade bug is fixed on a real network:** all 30 nodes
synced in 43 s from genesis through the legacy ledgers to protocol 30
(the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger
1).
- Correct under load: no forks, all in sync, no invalid
values/signatures, no restarts, no `NOT IN CACHE`/parse/cache-drop
errors; 11 tx-set fetch retries, all resolved on the first re-request.
- Leader-only construction works as designed (~1 build per ledger
network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from
~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
- Verdict FAIL at the same speed as the #5458 head (window-1 p75 3.26 s
vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and
follower validation of the received set was 0.52 s vs 0.34 s on #5458
(whose followers had warmed their signature cache by building), so the
follower prewarm did not recover it at this load. Nomination round 1 (2
s timeout) times out on almost every ledger.
- Follow-up: done in the 2026-11-10 update above (leader build,
ballot-start prewarm, and the IPC size fix found by the next run).
---
## Original description
> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet; the run of this
PR's final head will be posted in the update above.
# Description
This PR compresses transaction-set responses in the Rust overlay and
shares the encoded bytes across peer sends. Every response uses the
mandatory zstd protocol; there is no legacy/raw fallback or optional
lazy encoding state. The leader hands the final XDR to Rust before
Core's final validation, so compression can overlap that validation.
Receivers retain the original compressed bytes, decompress and verify
the set, then reuse the retained encoding when forwarding it.
A separate commit, `0ed3e540`, removes private proposals on non-leader
nodes. Followers enter nomination with a lazy value supplier and follow
received proposals without pulling the mempool or constructing a set.
First-round leaders can prepare early; a promoted leader constructs a
value only when it needs one. The driver caches that value across
retries, and externalization stops nomination even when no local value
was built. Tests cover timeout promotion, current close times, following
another leader, unavailable values, and reentrant nomination stop.
This PR builds on #5458. The
comparison below evaluates these revisions against the **original
compressed implementation of this PR**, using the same 10-tier-1,
30-validator fixture: 3,000 offered TPS, 6,000 transactions/block,
2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and
900-second measurement, with simulated apply.
## 3,000 offered TPS
| Metric | Original compression | Updated leader-only path |
|---|---:|---:|
| Median block interval | 2.018 s | 2.120 s |
| Worst-node full-window p99 block interval | 3.834 s | 3.836 s |
| Tx-set fetch p99 | 0.617 s | 0.601 s |
| Completed TPS | 2,926.67 | 2,806.67 |
| Harness verdict | PASS | FAIL (ob-0 only) |
Median intervals and completed throughput use the canonical node;
worst-node p99 uses the fixed measurement window. The harness uses a
different histogram/window: `ob-0` recorded 4.989 s against its 4-second
limit, while the other 29 nodes and inclusion passed. Completed
throughput fell 4.1% and median block interval rose 5.1%; this run does
**not** establish that the revised path matches or improves the original
compression baseline. These are single cloud runs with normal host
placement.
Leader-only construction is verified: 886 builds/compressions across 421
closed ledgers, all correlated with asserted leadership, with no
follower build or received-set recompression detected. Multiple
nomination leaders can build in a ledger. Compared with the intermediate
eager implementation, local compressions fell from 13,019 to 893 in
their respective 900-second windows, roughly a 93% reduction per ledger.
Payload savings remain 80.0%. The sampled received-set validation timer
rose from about 158 ms to 205 ms versus that intermediate build; the
precise cause is not isolated by this run.
# Checklist
- [x] Reviewed the contributing document.
- [x] Kept leader-only proposal construction in its own commit on this
PR's branch.
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Local build, Jenkins package/image build, and exact-head Clang/GCC
CI passed.
- [x] Rust suite: 194 passed, 3 ignored; the leader-only follow-up
leaves those validated Rust sources unchanged.
- [x] Core validation: 41 focused SCP/nomination/tx-set cases, 17
overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent
cases were rerun after configuring the local database fixture.
- [x] Performance comparison, runtime identity, resource checks and
sealed-artifact audit completed.
- [ ] Performance matches or improves the original compressed baseline.
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
…der schedule (#5496) Brings the stacked PRs that were merged into `pr/merge-ms-close-time-base` into `overlay-v2-shared`: - #5457 Batch GETDATA transaction responses - #5458 Select loadgen source accounts in constant time - #5459 Compress and share transaction set responses - #5495 Prototype: replace SCP nomination with a deterministic leader schedule
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update (2026-11-09)
Rebased onto the updated #5457 (the stack now includes #5453's merge of
master@ 362dbc5). Four commits now sit on top of the original change.cleanupAccountsruns invalueExternalizedon every node for every ledger. It used to allocate a set entry for the source of every transaction in the externalized set, then walk every in-use account with a map lookup. When ledgers fall behind the offered load, the in-use backlog grows, and that walk grew with it on the externalize path. In-use accounts are now indexed by public key, and each externalized transaction's source is looked up there. The cost no longer depends on the backlog, and the lookups allocate nothing. Which accounts are released, and when, is unchanged.execute()used to resolve ten meters through the metrics registry for every transaction, and copied the envelope into aStellarMessagejust to size it. It now uses the meters resolved at construction and sizes the message from the envelope. The recorded values are identical.processExternalizedbuilt aTransactionFramefor every transaction in the set on the main thread (about 7k per ledger at 3k TPS). That copied each envelope and built one operation frame per operation. The set's envelopes are now hashed in place (TxSetXDRFrame::getTransactionFullHashes), and loadgen reads sources straight from the envelopes. The overlay receives the same hashes in the same order, and loadgen releases the same accounts.loadgen.txn.bytes.Known limit: in the
MIXED_PREGEN_*modes, the Soroban stream fails the run when every source account has a transaction in flight, whereas the other modes wait. At perfnet's 33k accounts per generator, this needs a backlog of hundreds of slow ledgers, which is longer than a block_time_check run.Testing:
[loadgen],[herder],[txset](partitionedmake check): 94/94 test cases pass.e5ec7d4cb(which contains the whole stack): selftest 590 test cases / 2.85M assertions, check-sorobans, check-rust, check-nondet all pass.The performance tables above are for the original commit on the previous base. Perfnet run of the final head (2026-11-09):
Perfnet, final head
e5ec7d4cb(30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered from all 30 nodes, 2 s target, 150 ms simulated apply, tx sets capped at 7,500; n = 1):Original description
Description
This PR removes the linear account-selection operation from loadgen. Previously, each transaction selected a random position in an unordered set and walked to it. With large account pools, that traversal stalled submission callbacks on the main thread.
Available accounts now live in a vector. Selection uses a uniform random index, and removal swaps the selected account with the last entry before popping it. Both operations take constant time. Accounts return to the available pool after their ledger has applied; the test covers account recycling and consecutive loadgen runs.
Here, the control image is the PR this is based on, #5457. This is the 10 tier 1, 2 second target block latency test. Median block intervals are from the canonical node; p99 is the worst node over the full measurement window.
2,000 TPS
3,000 TPS
Submission overhead falls substantially at both loads. At 3,000 TPS, worst-node p99 improves, but both runs still fail the harness p75 threshold. Submission-batch means use timer count/sum differences across the ten load generators.
Checklist
make format.