Repository navigation
Merge parallel txset changes - #5453
Merged
SirTyson merged 168 commits intoNov 10, 2026
Merged
Conversation
…#5340) Adds a non-blocking CI job to run Horizon integration tests on pull requests and merges to master by compiling Core from source.
just a little more peer performance tuning.
Resolves stellar#5363 Test had a race between network externalizing new ledgers and new node joining with a gap larger than default MAX_SLOTS_TO_REMEMBER=12. Solution is to bump the config.
# Description This adds a new mode to apply load, allowing us for more "end to end" simulations. In the new mode, we time and simulate ballot phase through the end of apply, but in a single node network such that SCP state transitions occur instantly. This is useful, as it allows us to benchmark the processing overhead associated with consensus, as well as measure cache performance and dedup similar work done during tx set validation and the later application of this transaction set. The motivation for this test was from my recent block latency experiments. I noticed how optimizations in network calls, reducing bandwidth, etc were not moving the needle on actual block latency. It turned out that consensus latency was greatly affected by CPU based bottlenecks, not just network calls. On supercluster, it's challenging to benchmark CPU costs of concensus, as you can't easily run tracy or profilers and have limited visibility to single node performance. This test allows us to much more easily measure and improve the processing heavy aspects of concensus. In overlay-v2-shared for a block with 6K SAC transfers, we saw about 300 ms of non-network "ingestion" time when receiving a tx set and 500 ms of apply. Imo this was a significant blind spot in our single node apply load tests previously. Note that the actual number produced by this test is meaningless, as consensus without a network tells us very little about overall performance. However, the reported phase timings are very useful in identifying bottlenecks and comparing solutions. This simulates the processing done by a non-leader node. In the future, it may also be helpful to add a mode for block construction as well. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
The state data structures are rather large and nested (due to containing XDR), so when they get large de-allocation may take up to a few ms. We don't really need to block the apply thread on this de-allocation, so we can hide it in the background thread.
Currently `getAllEntries` call seals the LTX and unnecessarily copies every entry, which adds up to a few ms if thousands of entries are involved. With this change we just return the references to all the entries. Since this call is used just once in the 'production' path, and the results are only used in a couple of read-only calls, it shouldn't be hard to maintain the lifetime correctness for this method.
Resolves stellar#704. This PR clamps the results of `ApplicableTxSetFrame::getTotalFees` and `ApplicableTxSetGrame::getTotalInclusionFees` to `INT64_MAX` so that these functions do not overflow. The only caller of these functions is `compareTxSets`, and the only caller of that function is `HerderSCPDriver::combineCandidates`. I think this is why stellar#704 ascribes the bug to `combineCandidates`, but I'd argue it's lower down in these fee summing functions. I've verified that this fixes the PoC in stellar#704. The unit test I added is much more targeted; it focuses on the underlying functions themselves.
Deduplicate query messages
Skip background tx sig verification for unauthenticated peers
Clamp additions in tx set fee summing functions
- Make the Node 24 env-var comment date-agnostic (2026-06-16 has passed) - Remove the unused Corepack step; this repo has no package.json, so it only ever shimmed yarn/pnpm that are never invoked - Replace dtolnay/rust-toolchain with `rustup update`, matching the convention in this repo's other Rust workflows and dropping a third-party action from a security workflow Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This includes several micro-optimizations that have a significant impact on the Soroban LTX commit stage performance (~-40ms on large TPL benchmarks; surprisingly much more impact than parallelization change this is based on): - Replace several hot path copies with moves - Avoid unnecessary LTX loads that were only needed to distinguish between INIT and LIVE entries; propagate a flag that distinguishes these instead - Get rid of an unnecessary inner LTX in commitChangesToLedgerTxn
# Description Optimizations for Soroban LTX commit path. This includes several micro-optimizations that have a significant impact on the Soroban LTX commit stage performance (~-40ms on large TPL benchmarks; surprisingly much more impact than parallelization change this is based on): - Replace several hot path copies with moves - Avoid unnecessary LTX loads that were only needed to distinguish between INIT and LIVE entries; propagate a flag that distinguishes these instead - Get rid of an unnecessary inner LTX in commitChangesToLedgerTxn # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Test has been failing recently with ``` stellar-core/src/overlay/test/TCPPeerTests.cpp:617: FAILED: REQUIRE( msgWrite.count() == prevMsgWrite + 2 ) with expansion: 6 == 5 ``` this fixes two potential causes (SCP firing and pings firing). confirmed with a bunch of running-in-a-loop with different prng seeds and varying forced delays, which is about the best one can do.
Replace the single signature cache global mutex + RandomEvictionCache with 64 sharded caches (each with its own mutex) so concurrent signature verification threads no longer block on a shared lock. In addition, replace another global mutex for metrics with atomic counters. Also use the single-lookup maybeGet() in place of the exists()/get() double-lookup and use Rust verification unconditionally.
# Description Replace the single global mutex + RandomEvictionCache with 16 sharded caches (each with its own mutex) so concurrent signature verification threads no longer block on a shared lock. Also use the single-lookup maybeGet() in place of the exists()/get() double-lookup and use Rust verification unconditionally (an overdue cleanup). # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Just adds a metric that monitors the size of the tcmalloc heap
Derive the protocol from MAX_SUPPORTED_PROTOCOL_VERSION + 1 in src/util/ProtocolVersion.h instead of hardcoding it, since the test build enables next. Skip Horizon's stellar-rpc tests, as no released RPC image supports the next protocol.
…ellar#5477) The RPC and Horizon integration tests build core with next enabled, so derive their protocol from `MAX_SUPPORTED_PROTOCOL_VERSION + 1` instead of hardcoding it (the hardcoded RPC `'27'` is failing). Horizon's stellar-rpc tests are skipped since no released RPC image supports the next protocol; Horizon caps the tested protocol at its own max.
Brings overlay-v2-shared up to date with master. The previous master merge (b90d48e, stellar#5428) was recorded with master's tip 100cc38 missing from its parents, so this merge used 100cc38 as its base (temporary graft of b90d48e, removed afterwards). Notable master changes brought in: - Protocol 30: CURRENT_LEDGER_PROTOCOL_VERSION follows MAX_SUPPORTED_PROTOCOL_VERSION; CAP-0084 and CAP-0087 ungated, p29 and p30 Soroban hosts (stellar#5472, stellar#5478, stellar#5484, stellar#5483). - Millisecond-resolution close times at protocol 30 (stellar#5423): ConsensusTime/ApplyTime types, STELLAR_VALUE_SIGNED_MS and STELLAR_VALUE_EMPTY_TX_SET_MS. - Tx set validity hardening (stellar#5481), lower max message size in protocol 29 (stellar#733), sub-second ledger edge-case fixes. - Simplified ledger view hierarchy (stellar#5461); the overlay-only-mode sequence number skip now lives in TransactionFrame::checkValid. - Banned accounts deprecated (CAP-0077 is active). - Apply-path performance work (stellar#5402, stellar#5411, stellar#5414, stellar#5415, stellar#5418, stellar#5421, stellar#5436, stellar#5286), apply-load tx set construction mode (stellar#5448), --check-lcm (stellar#5435), v29.0.0 release merge. Conflict resolution: - Files overlay-v2-shared deleted and master modified (the C++ overlay, TransactionQueue, TransactionQueueTests, UpgradesTests, private/Horizon/RPC workflows): kept deleted. Master's changes to them are C++ overlay peer/flow-control logic, tx queue admission and tests for those. - Herder: kept the Rust-overlay submission path (recvTransaction returns TxSubmitStatus) and dropped the deprecated `force` parameter as master did. triggerNextLedger keeps the prepared/built tx set path from the Rust mempool, now with ConsensusTime close times and master's apply-time close-time offset. The early tx set preparation path (prepareTxSet, buildTxSet, setupTriggerNextLedger) takes ConsensusTime. Kept master's mNominatedTxSetOverride test hook (applied before the tx set is cached in the Rust overlay) and the herder.txset.build timer (now timing buildTxSet). Dropped the banned-accounts setup and tx queue maintenance. - AppConnector: dropped both ov2's maybeUpdateImmutableLedgerView and master's C++-overlay getOverlayThreadSnapshot. - LoadGenerator: master's CachedTestAccountPtr with ov2's TxSubmitStatus. - Tests: kept ov2's removal of tests that need the C++ overlay, standalone mode or the tx queue (standalone, manualclose, flooding, tx queue limits, far-future slot cleanup, herderExternalizesValuesWithProtocol, MAX_SAC_TPS, custom token, soroswap and noisy binary search apply-load tests). Added master's ms close time HerderTests and the non-canonical empty tx set test on ov2's Rust-overlay topologies. The tx set size test checks against overlay-v2's own tx set allowance. The sub-second getUpperBoundCloseTimeOffset test checks the offset directly, since there is no C++ tx queue admission. Tests that used CheckValidLedgerViewWrapper now use getLCLView(). - Cargo.lock: master's lockfile plus the overlay crate's packages, regenerated by cargo; semver-compatible duplicates take the newer version. - Build: Makefile.am keeps the Rust overlay rules and adds tracy import-chrome; ci-build.sh keeps ASAN off and takes master's -g1. - Needed to compile: PendingEnvelopes.cpp includes LedgerManager.h (master got it through the C++ OverlayManager.h); ov2's early-nomination and checkpoint-slot tests use ConsensusTime/ApplyTimeOffset; OverlayIPC tests take ledger views from the LedgerManager.
The Rust overlay pinned stellar-xdr 26.0.0, which predates protocol 30's XDR. It strict-decodes every StellarMessage it receives, so at protocol 30 it would have: - failed to decode millisecond close-time StellarValues (STELLAR_VALUE_SIGNED_MS), so tx set hashes were no longer extracted from SCP envelopes: no proactive tx set fetch and no tx set source tracking. Core still fetched on demand, but later. - dropped transactions and tx sets containing protocol 30 XDR arms (CAP-0084 muxed contract addresses). stellar-xdr 30.0.0 is generated from exactly the protocol XDR this branch builds against (4f524bbac), and Cargo.lock already carries it for the p30 Soroban host. The crate no longer has a `curr` module, so paths change from stellar_xdr::curr:: to stellar_xdr::. collect_stellar_value_hash now skips empty-tx-set values (both close-time formats). Their tx set hash is Core's synthetic empty set hash, which Core builds locally, so fetching it from the network can never succeed. Tests cover every StellarValue format, an ms value in a ballot, and truncated ms values. Message size limits stay overlay v2's own: a 16 MiB message cap in the Rust overlay and the IPC channel, and a 10 MiB tx set allowance (overlay/NetworkConstants.h). Master's protocol 29 cut to a 5 MiB message cap and a 4 MiB tx set allowance (stellar#733) lives in the C++ overlay's Peer.h/FlowControlCapacity.h. It protects that overlay from externalizing a set upgraded peers can't download, which doesn't apply here, and it would cap perf runs (3k SAC TPS at 2s ledgers needs about 6 MB of Soroban bytes per ledger). The merge adapts master's tx set size test to these constants.
Master's stellar#5423 made ARTIFICIALLY_SET_CLOSE_TIME_FOR_TESTING a millisecond value. Two overlay v2-only tests still set it in seconds, which would now mean 2ms and 5ms ledgers: "SCP checkpoint" (2 -> 2000) and "SCP message capture from previous ledger" (5 -> 5000). Master tests the upgrade to millisecond close times in UpgradesTests, which overlay-v2-shared does not build (it needs the C++ overlay). Add overlay v2 versions of the two tests to HerderTests: - "millisecond close time upgrade boundary": a 4-node Rust-overlay network starts at protocol 29, upgrades to protocol 30 and must keep closing ledgers, switching to ms values exactly at the boundary. A fourth node is stopped before the upgrade and restarted from its database afterwards; it must resync across the boundary from relayed SCP state without catchup. The simulation runs in real time, so the upgrade is scheduled relative to now, and the node is removed and re-added since overlay v2 can't partition a single connection. - "upgrade scheduling under sub-second ledgers": upgrade scheduling reads only the whole-second close time. Also start "prepare nomination before trigger" 375ms into a second, so the early-prepared close time carries milliseconds at protocol 30.
This affects both nominating a new transaction set, and validating the incoming transaction set. Thanks to the fact that most of the time we're either applying the ledger, or validating a transaction set, we can use the efficient CPU-pinned batch executor for this. This speeds up the invalid transaction trimming step by ~20ms on large benchmarks, and in general should increase the transaction validation step proportionally to the number of cores.
Master's --capture-lcm/--check-lcm mode (stellar#5435, run by ci-build.sh over [tx]) makes getTestConfig() enable ENABLE_SOROBAN_DIAGNOSTIC_EVENTS, and Application rejects that on networked validators. Master's loopback simulations stay RUN_STANDALONE, but overlay v2's Simulation::addNode always clears RUN_STANDALONE (its nodes talk through Rust overlay processes). So every simulation-based [tx] test threw at startup under --check-lcm, e.g. "network externalizes empty-tx-set on missing value". Simulations never capture LedgerCloseMeta (the constructor calls disableLcmCapture()), so their nodes don't need diagnostic events. Turn them off in addNode. Runs without LCM tracking are unaffected, since getTestConfig() only enables the events while tracking LCM.
ci-build.sh checks [tx] against the LedgerCloseMeta golden data (`--rng-seed 12345 --check-lcm`). With master's golden files, 32 of them mismatch on overlay v2, while all 144 [tx] test cases pass: - 29 Soroban files: InvokeHostFunctionTests "parallel txs|multi RO extensions ..." runs 0-24, "read-only bumps across threads" and two "restore with nonexistent key" sections, plus FrozenLedgerKeysTests "frozen ledger keys in Soroban footprint". These tests draw inputs from the global test PRNG after closing ledgers. On master, the C++ TransactionQueue draws from the same PRNG at construction and in shift() on every ledger close. Overlay v2 has no C++ queue, so later draws differ. In "read-only bumps across threads", ledgers 1-23 are byte-identical and ledger 24 applies the same accounts, sequence numbers and fees, but with different random extendTo arguments. The apply path and src/rust are identical to master. - 3 TxEnvelopeTests files for "getUpperBoundCloseTimeOffset under sub-second ledgers", which overlay v2 rewrote in the master merge (no C++ queue admission to exercise). Regenerated from the source tree root with `stellar-core test '[tx]' --rng-seed 12345 --capture-lcm` at the current protocol, the only one overlay v2's CI builds. A full capture changes exactly these 32 files: no additions, deletions or index changes.
SirTyson
force-pushed
the
merge-parallel-txset-changes
branch
from
November 9, 2026 21:38
368070a to
14de397
Compare
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This was referenced Nov 9, 2026
Contributor
Author
|
Updated (force-pushed, 2026-11-09): merged current Commits on top of the base:
|
7 of 8 tasks
The overlay-v2 experiment branches don't track master's protocol/overlay version pairing, and the check fails against overlay-v2-shared's ProtocolVersion.h.
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
## Update (2026-11-09) This PR is rebased onto the reworked #5453, which merges current `master` (362dbc5) into `overlay-v2-shared`. That merge brings in the final #5423 (millisecond close times, gated at protocol 30, with the protocol-30 bump). It also carries the overlay-v2 adaptations those need, including the Rust overlay moving to `stellar-xdr` 30.0.0 to decode millisecond `StellarValue`s. **The old first commit ("Enable millisecond consensus close times at protocol 28") is dropped.** The master merge in #5453 supersedes it. It was an earlier draft of #5423 behind a compile-time `MS_CLOSE_TIME` flag at protocol 28, with its own XDR pin and fixtures. The overlay-v2-specific parts needed for correctness are now in #5453. This PR is now the trigger-timing work on top of millisecond close times: 1. **Account for actual elapsed nomination time in the trigger drift allowance** (ported unchanged in substance). The next-ledger trigger is anchored on the previous externalized close time C and fires at `C + target`. It falls back to the local prepare-start anchor when the time elapsed since C exceeds the target plus an allowance for nomination. That allowance used to be the sum of nomination timeouts that had fired. Now it is the steady-clock time from the local trigger to entry into ballot. A slow round that never runs a timeout callback no longer counts as clock drift: slow tx-set fetch or validation, a busy main thread, or ballot starting before the expired timer runs. When the network is behind, the next trigger fires immediately instead of waiting out the nomination again. The extra wait a fallback can add is bounded by our clock drift plus how late we triggered relative to the winning close time. 2. **Record the nomination trigger only when nomination proceeds** (new). If a node skips nomination because its clock is behind the last close time, it no longer credits the wait for its peers to the allowance. 3. **Test the trigger anchor with sub-second close times** (new). With close times 0, 250 and 999 ms into the second, the next trigger is due exactly the target after the close time, and the fallback boundary holds to the millisecond. 4. **Add trigger timing metrics** (new). Together they show whether a 2 s ledger is bound by apply, by nomination or by event-loop delay: - `scp.timing.trigger-to-ballot` (timer): proposal construction plus nomination, which is the drift allowance. - `scp.trigger.lateness` (timer): how late the trigger callback ran after its scheduled time. - `scp.trigger.immediate` (meter): the next trigger was already due when it was scheduled. 5. **Skip a delayed nomination emit that was already sent** (new, test-only code path). With `ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING`, a deferred nomination emit could fire after `processEnvelope` had already emitted the same statement. SCP then rejected the duplicate and threw "moved to a bad state (nomination)", which made the hidden "consensus close time trigger timer" test flaky under parallel load. The deferred emit now only sends a statement newer than our last one. A new SCP test replays the race deterministically. None of these commits change values, validation or ledger output. They only change local trigger scheduling, metrics, logs and BUILD_TESTS-only code. **Follow-up under consideration:** for proposals rebuilt at the trigger, propose the scheduled trigger time as the close time (bounded by `now` and LCL+1 ms), so a leader's trigger lateness no longer shifts the network's cadence. We'll do it if `scp.trigger.lateness` p75 exceeds about 20 ms on perfnet. Testing, at `285aa42fc92d` on top of #5453 `04fbddb7d`: - New and changed tests: `[trigger-construction]` 8/8 (the drift-allowance tests and the 3 new ones) and `[early-nomination]` 6/6 pass. - `[herder]` (partitioned `make check`): 77/77 test cases pass, including the hidden real-clock trigger-timer simulations. - At `4f577bd25`: the new SCP test passes. A copy of the test without the fix fails with the original exception, so the test reproduces the race. "consensus close time trigger timer" passes 3/3 on its own and 2/2 inside a partitioned `[herder],[scp]` `make check` (97/97 test cases, full check passes both times). - Final branch: restacked onto the final #5453 head `14de3972a` (two test-only commits on top of `04fbddb7d`); the five patches are identical (`git range-diff`). The full suite passes on the #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust, check-nondet). --- ## Original description > The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs. # Description Merges in #5423. This change adds ms closeTime. By itself, it doesn't improve TPS or latency significantly. However, it reduces our long tail fo very slow blocks, and also allows us to target intrasecond or subsecond block times, which are not currently possible at whole second resolution. In addition to taking the change from the PR, the 2nd commit makes some minor adjustments to when we call the fallback timer. This was intentionally conservative for main net launch assuming lots of drifting clocks, but no reason we can't tighten it here. Results, measured with 10 tier 1 topology (no watchers) with 2 second block time target: ## 2000 TPS | Metric | Original overlay-v2-shared |Changes | |---|---:|---:| | Median block interval | 2.090 s | 1.993 s | | p99 block interval | 4.563 s | 4.298 s | | Maximum block interval | 6.219 s | 5.090 s | | Harness verdict | **FAIL** | **FAIL** | Rebased on #5453 so opening as a draft for now. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#opening-a-pull-request) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
## Update (2026-11-09) Rebased onto the updated #5454 (which now sits on #5453's merge of `master` @ 362dbc5; the overlay crate moved to `stellar-xdr` 30.0.0 there). The original commit is unchanged apart from a test import updated for stellar-xdr 30.0.0 (`stellar_xdr::curr::` to `stellar_xdr::`). Eight commits on top, each with tests (an independent review found no correctness, wire-format or deadlock issues in the original commit; its low-severity findings are fixed below): - **Stop serving a demand after a failed write.** A response job returns at the first write that fails after the stream retries (usually a disconnect), instead of looking up, writing and logging a failure for every remaining batch. The requester re-demands unanswered hashes after its response timeout. - **Frame responses straight from the shared transaction bytes.** Each frame's discriminant and envelope are copied directly into the batch, with no per-transaction allocation; a transaction larger than the batch budget is written as two borrowed parts with no copy. Each batch is looked up under one TX-buffer lock, released before frames are copied or written, so local submissions on the event loop never wait for a response copy. - **Buffer inbound TX stream reads (64 KiB).** One QUIC read takes in whatever has arrived, often a whole response batch, instead of two reads per frame. The reader never waits for more data than is available. - **Queue INVs for all peers under one lock**, for both relayed and locally submitted transactions, instead of one batcher lock per peer. - **Handle each received INV batch with one seen-set lock and one tracker lock**, instead of both locks per advertised hash. Request decisions and their order are unchanged. - **More tests:** batching edges (an oversized transaction first, two in a row, frames exactly at the budget, carried hits, misses); a failed batch write retried whole on a reopened stream; buffered reads; delivery of INV batches completed by either path; INV request decisions and order. Wire bytes are unchanged. Testing on the rebased branch: - 194 overlay crate tests pass (`cargo test` in `overlay/`, 3 ignored); `cargo fmt --check` is clean. - The standard build succeeds, and 25 C++ overlay-IPC test cases (136 assertions) pass: `[overlay-ipc]`, `[overlay][ipc]` and the 10-node `[overlay-ipc-large]` consensus case. - The full suite passes on the #5458 head that contains this PR (selftest 590 test cases, check-sorobans, check-rust with these overlay tests, check-nondet). The follow-up commits have not been measured separately on perfnet; the numbers above are for the original commit on the previous base. The perfnet run of the final #5458 head (which contains this PR) is reported on #5458. **Next lever (not in this PR):** a node waits up to 100 ms (checked every 50 ms) before announcing new transactions to a peer, so each flooding hop adds 100-150 ms before the peer can request them. A perfnet A/B of shorter delays (50 ms and 25 ms) is the planned follow-up, comparing tx pull latency, tx-set fetch p99 and block intervals against bandwidth and CPU. --- ## Original description > The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the runs of the final #5458 and #5459 heads contain this PR and are reported on those PRs. # Description This PR batches GETDATA transaction responses in the Rust overlay. Previously, a request for many transactions created a separate task and stream write for every response. Under load, that produced unnecessary scheduling and stream-lock overhead, and a slow peer could leave the entire response demand queued in memory. We now serve each request in one asynchronous job, combining complete transaction frames into writes of up to 64 KiB. Each transaction keeps its existing wire framing, and the writes use the existing stream priorities and retry policy. Transactions larger than the batch budget are sent individually. Here, the control image is the PR this is based on #5454. This is the 10 tier 1, 2 second target block latency test. Median block intervals are from the canonical node; p99 is the worst node over the full measurement window. ## 2,000 TPS | Metric | Control | Batched GETDATA | |---|---:|---:| | Median block interval | 1.993 s | 2.000 s | | Worst-node p99 block interval | 5.090 s | 2.508 s | | Tx-set fetch p99 | 1.720 s | 0.778 s | | Harness verdict | FAIL | PASS | ## 3,000 TPS | Metric | Control | Batched GETDATA| |---|---:|---:| | Median block interval | 2.434 s | 2.363 s | | Worst-node p99 block interval | 4.122 s | 4.524 s | | Tx-set fetch p99 | 1.757 s | 1.087 s | | Harness verdict | FAIL | FAIL | At 2,000 TPS, the largest improvement is in the tail: worst-node p99 falls by about half, and nearly all tx-set fetches longer than two seconds disappear. This is consistent with reducing transaction-response write overhead and contention affecting consensus transfers. # Checklist - [x] Reviewed the contributing document. - [x] One commit on top of #5454, with no merge commits. (Now nine: the original plus eight follow-ups, see the update above.) - [x] Formatting checked with `make format` and `cargo fmt --check`. - [x] Release overlay build and Jenkins package build succeeded. - [x] Relevant tests passed: 185 Rust tests (3 ignored), plus 17 C++ overlay-IPC cases / 82 assertions. - [x] Performance comparison included above.
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
## Update (2026-11-09) Rebased onto the updated #5457 (the stack now includes #5453's merge of `master` @ 362dbc5). Four commits now sit on top of the original change. - **Match externalized accounts in time proportional to the tx set.** `cleanupAccounts` runs in `valueExternalized` on every node for every ledger. It used to allocate a set entry for the source of every transaction in the externalized set, then walk every in-use account with a map lookup. When ledgers fall behind the offered load, the in-use backlog grows, and that walk grew with it on the externalize path. In-use accounts are now indexed by public key, and each externalized transaction's source is looked up there. The cost no longer depends on the backlog, and the lookups allocate nothing. Which accounts are released, and when, is unchanged. - **Stop re-registering loadgen metrics and copying envelopes per transaction.** `execute()` used to resolve ten meters through the metrics registry for every transaction, and copied the envelope into a `StellarMessage` just to size it. It now uses the meters resolved at construction and sizes the message from the envelope. The recorded values are identical. - **Hash externalized transactions without rebuilding their frames.** To give the Rust overlay the full hashes of the externalized transactions, and loadgen their source accounts, `processExternalized` built a `TransactionFrame` for every transaction in the set on the main thread (about 7k per ledger at 3k TPS). That copied each envelope and built one operation frame per operation. The set's envelopes are now hashed in place (`TxSetXDRFrame::getTransactionFullHashes`), and loadgen reads sources straight from the envelopes. The overlay receives the same hashes in the same order, and loadgen releases the same accounts. - **Tests.** On this branch submission never rejects, so the recycling test's "no rejected transactions" assertion could not fail. It now relies on every transaction applying and on runs completing with synced sequence numbers. It also fails fast and runs in about a third of the time. New tests: - "synthetic SAC load recycles source accounts in overlay-only mode" runs perfnet's load (synthetic SAC payments, overlay-only, more transactions than accounts). It checks completion and the per-transaction meters, including `loadgen.txn.bytes`. - "externalized tx set hashes and sources match transaction frames" pins the hashes and sources from legacy and generalized sets (sequential and parallel phases; v0, v1, muxed and fee-bump envelopes) against frames built from the same set. Known limit: in the `MIXED_PREGEN_*` modes, the Soroban stream fails the run when every source account has a transaction in flight, whereas the other modes wait. At perfnet's 33k accounts per generator, this needs a backlog of hundreds of slow ledgers, which is longer than a block_time_check run. Testing: - New and changed tests by name: the six loadgen tests (including both new ones and the sped-up recycling test) pass. - `[loadgen],[herder],[txset]` (partitioned `make check`): 94/94 test cases pass. - Full suite at the final head `e5ec7d4cb` (which contains the whole stack): selftest 590 test cases / 2.85M assertions, check-sorobans, check-rust, check-nondet all pass. - An independent review found no correctness, determinism or safety issues in the original commit or the follow-ups. The performance tables above are for the original commit on the previous base. Perfnet run of the final head (2026-11-09): **Perfnet, final head `e5ec7d4cb`** (30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered from all 30 nodes, 2 s target, 150 ms simulated apply, tx sets capped at 7,500; n = 1): - Correct: protocol 30 with millisecond close times on all nodes; no forks, all in sync, no invalid values/signatures, no restarts, no overlay errors during the load. - Verdict FAIL (window 1 p75 3.25 s on every node, p99 3.89 s), essentially the same speed as the previous image of this PR on the same fixture (p75 3.32 s, p99 3.92 s): about 3.1 s per ledger with full 7,500-tx ledgers and a growing backlog. - Per ledger: ~0.3 s externalize + close (incl. 150 ms apply), **~1.05 s tx-set build** before nomination, ~1.34 s nomination (of which ~0.33 s tx-set fetch, ~0.34 s validation), ~0.28 s ballot. Every trigger was already due (backlog), so the leader's early tx-set preparation never ran and the build sits on the critical path. The new trigger metrics (#5454) show lateness p75 0.25 ms and no prepare-start fallbacks. - This PR's changes: loadgen submit 7-10% cheaper (mean 2.71 vs 2.91 ms); demand timeouts -21% and duplicate flood bytes -21% (both only during the first minutes of load); neither is on the ledger critical path in this fixture. --- ## Original description > The numbers below come from an earlier simulation environment and are kept for history. The stack is now measured on perfnet (30 validators in 4 AWS regions, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply, 900 s load); the run of this PR's final head will be posted here. # Description This PR removes the linear account-selection operation from loadgen. Previously, each transaction selected a random position in an unordered set and walked to it. With large account pools, that traversal stalled submission callbacks on the main thread. Available accounts now live in a vector. Selection uses a uniform random index, and removal swaps the selected account with the last entry before popping it. Both operations take constant time. Accounts return to the available pool after their ledger has applied; the test covers account recycling and consecutive loadgen runs. Here, the control image is the PR this is based on, #5457. This is the 10 tier 1, 2 second target block latency test. Median block intervals are from the canonical node; p99 is the worst node over the full measurement window. ## 2,000 TPS | Metric | Control | Loadgen account selection | |---|---:|---:| | Median block interval | 2.000 s | 1.956 s | | Worst-node p99 block interval | 2.508 s | 2.595 s | | Mean loadgen submission batch | 48.28 ms | 2.68 ms | | Harness verdict | PASS | PASS | ## 3,000 TPS | Metric | Control | Loadgen account selection | |---|---:|---:| | Median block interval | 2.363 s | 2.341 s | | Worst-node p99 block interval | 4.524 s | 3.105 s | | Mean loadgen submission batch | 88.31 ms | 3.62 ms | | Harness verdict | FAIL | FAIL | Submission overhead falls substantially at both loads. At 3,000 TPS, worst-node p99 improves, but both runs still fail the harness p75 threshold. Submission-batch means use timer count/sum differences across the ten load generators. # Checklist - [x] Reviewed the contributing document. - [x] One commit on top of #5457, with no merge commits. - [x] Formatting checked with `make format`. - [x] Local build and Jenkins package build succeeded. - [x] Relevant loadgen tests passed: 8 cases / 947 assertions, including real-apply account recycling and consecutive runs. - [x] Performance comparison included above.
SirTyson
added a commit
that referenced
this pull request
Nov 10, 2026
## Update (2026-11-10): cheaper leader build, ballot-start signature
prewarm, IPC size fix; perfnet passes on c6id
Head is now `ebdd73635`, 8 commits on top of `c26b191fa`. They fix the
two things the 2026-11-09 perfnet run found, plus a liveness bug the
next run found.
### Leader build
On perfnet the leader's 1.1 s build was on the critical path every
ledger.
- `ba4955347` **validates only what selection can use.** The leader
still pulls 2x the capacity, but validates each phase in mempool order
only until it holds 5/4 of the capacity in valid transactions. If some
are invalid it refills: twice the shortfall, then the rest. The selected
set is the same unless selection would have reached past the validated
prefix.
- `a1bffae42` hashes the candidates on the validation workers, not the
main thread. That was 18% of a cold build's instructions on hosts
without SHA extensions, such as c5d.
- `966d3bb41` precomputes the parallel builder's sort keys.
- `95be616c7` decodes mempool pulls straight from the IPC payload,
without copying.
- `2a30c1dc0` adds phase timers
`herder.txset.build-{pull,frames,trim,select,xdr,handoff,roundtrip,final-check}`
and a hidden leader-build benchmark.
- Laptop benchmark (15,000 SAC candidates, cap 7,500, 4 tasks, n=5
medians): cold build 261 -> 168 ms (-36%), warm 124 -> 84 ms (-32%).
Main-thread instructions -41% (callgrind).
### Signature prewarm from ballot start, on every validator
(`a91547bec`)
The 2026-11-09 follower prewarm covered only about half of the leader's
set. The fees are equal, so the mempool orders transactions by hash, but
selection breaks ties randomly. The leader also had no prewarm.
- Now, when slot N starts balloting, every validator verifies the
signatures of exactly the window that the N+1 build validates first.
This runs while N finishes balloting, externalizes and closes. The main
thread only makes the overlay round trip; decoding and verification run
on background tasks.
- New metrics: `herder.prewarm.{pull,duration}`,
`herder.prewarm.{txs,verified,incomplete}` and
`herder.txset.{build,validate}-sig-cache-{hit,miss}`.
- Laptop: a build after a prewarm takes 83 ms (cold 169). Received-set
validation takes 38 ms (cold 102). 0 cache misses.
- `0cf227b94`: Core's tx-set fetch timer is now `scp.fetch.txset`, with
`scp.fetch.txset-retry`. `overlay.fetch.txset` is now only the Rust
overlay's time; it had been fed by both.
### Bug fix: oversized IPC payloads (`ebdd73635`)
Perfnet found this on `a91547bec`, on c5d.
- **What happened:** 80 s into the load the whole network stopped at one
ledger. 16 of 30 nodes logged `Overlay IPC connection closed` within
0.35 s of each other. The rest then lost quorum, and all lost sync.
- **Cause:** the prewarm pulls 2 x 7,600 + 7,500 transactions (22,700
SAC payments, about 17.2 MB) once the mempool backs up. Both ends of the
Core-overlay IPC rejected payloads over 16 MiB, and Core's reader treats
a rejected length as a corrupt stream: it stops reading and never
recovers. Every validator starts the pull at the same ballot.
- **Fix:**
- The limit is now 64 MiB on both sides (`ipc::MAX_PAYLOAD_BYTES` and
`MAX_PAYLOAD_SIZE`, which must match).
- The overlay returns the longest prefix of the top transactions that
fits, with a warning, so no mempool can produce an oversized response.
- Neither side sends an over-limit payload: Rust refuses with
`PayloadTooLarge`, and Core refuses before writing anything.
- Core now logs the type and length when it rejects one.
- This also covers a maximum-size peer tx set (`MAX_TXSET_XDR_SIZE` + 32
bytes), which could have hit the old limit too.
- **Test:** a new `[overlay-ipc]` test with the real Rust overlay. A 20
MiB top-transactions response arrives whole; with the old code the pull
returns nothing (`0 == 20`). An 80 MiB pull is truncated to the 63
highest-fee transactions, and the IPC keeps working.
### Testing
- **`a91547bec`:**
- Laptop: new and changed tests, and the step-2 tags (149 test cases).
- Devbox: full suite `--all-versions` PASS (selftest 604 test cases /
8,638,027 assertions, check-nondet, check-rust).
- **`ebdd73635`** (devbox):
- New tests: the `[overlay-ipc]` test above, plus IPC tests for a 20 MB
message, an over-limit length (rejected, with a receive timeout so a
regression fails instead of hanging) and an over-limit send (nothing
written). Rust tests cover truncation and the send guard.
- `[ipc]` (7 test cases), `[overlay-ipc]` (20), partitioned
`[overlay]~[ipc-crossproc]` and the Rust `ipc::` tests (23): all pass.
- Two pre-existing failures reproduce on `a91547bec` without the change:
- The Rust `test_tx_not_blocked_by_scp_flood` fails its own validity
check (an SCP flood of 6.7 ms is under its 10 ms floor).
- The hidden `IPC cross-process communication` test needs an unbuilt
`overlay-stub` binary.
- Full suite `--all-versions` on `ebdd73635` (devbox): PASS. selftest
605 test cases / 8,638,079 assertions across protocol versions,
check-nondet, libsodium (77) and xdrpp (7), check-rust and the soroban
host tests (103 `test result: ok`, 0 failed). The overlay crate's own
tests run separately; see above.
### Perfnet, head `ebdd73635`
30 validators in 4 AWS regions, 10 tier-1 orgs, 3,000 SAC TPS offered, 2
s target, 150 ms simulated apply, tx sets capped at 7,500, 900 s
measured. One run per instance type, both on the same day: c5d.2xlarge
(perfnet's default, Cascade Lake, no SHA-NI) and c6id.2xlarge (Ice Lake,
SHA-NI). Same 8 vCPU / 16 GiB.
**c6id.2xlarge: PASS.** All four windows on target, inclusion passed
(2,880,000 txs), all checks passed.
| Window | avg p75 | avg p99 | p75 range over nodes |
|---|---:|---:|---|
| 1 | 2,112 ms | 2,334 ms | 2,103-2,129 ms |
| 2 | 2,099 ms | 2,391 ms | 2,091-2,113 ms |
| 3 | 2,101 ms | 2,324 ms | 2,092-2,110 ms |
| 4 | 2,099 ms | 2,354 ms | 2,089-2,110 ms |
**c5d.2xlarge: FAIL** in window 1, but 19% faster than the previous head
and #5458. The c5d network fell behind the offered load, so sets were
full at 7,500 and the mempool backed up. That's exactly the case that
stalled `a91547bec`, and here there were no IPC disconnects or sync
losses. Correctness passed: error counters, in sync, no fork, still
closing. Inclusion was skipped because the load stopped after window 1.
| c5d, window 1 (5-min means) | #5458 head | #5459 `c26b191fa` | **#5459
`ebdd73635`** |
|---|---:|---:|---:|
| avg p75 / p99 | 3,250 / 3,891 ms | 3,264 / 3,672 ms | **2,637 / 2,920
ms** |
| Ledgers per 5 min | 96 | 96 | 119 |
| Trigger to first ballot | 2.52 s | 2.55 s | 1.94 s |
| Leader build | 1.15 s | 1.11 s | 0.68 s (candidate validation 0.20 s)
|
| Received-set validation | 0.34 s | 0.52 s | 0.33 s |
| Nomination timeouts per node | 5 | 106 | 34 |
| Ledger close | 0.229 s | 0.228 s | 0.229 s |
Both instance types, same build:
| | c5d | c6id |
|---|---:|---:|
| Ledger interval | about 2.53 s | about 2.03 s |
| Txs per ledger | 7,500 (backlog) | about 6,100 (kept up with 3,000
TPS) |
| Trigger to first ballot | 1.94 s | 1.22 s |
| Leader build | 0.68 s | 0.27 s |
| Received-set validation | 0.33 s | 0.20 s |
| Signature-cache coverage: build / validation | 89.5% / 98.7% | 87.9% /
97.8% |
| Prewarm: mean duration / main-thread pull / stopped early | 0.21 s /
22 ms / 0.1% | 0.10 s / 7 ms / 0% |
| Ledger close | 0.229 s | 0.164 s |
Notes:
- **No A/B on c6id.** #5458 has not been run on c6id, so the c6id PASS
shows this head can hold the target on current-generation x86. It
doesn't show how much of that comes from this PR. Every earlier run of
this fixture was on c5d.
- **Two regimes.** On c6id the network keeps up, so sets stay at about
6,100 transactions and triggers aren't all due at once; #5454's early
preparation runs, giving about 2 builds per ledger network-wide. On c5d
it falls behind, sets are full, and every ledger is heavier. So
per-ledger timings aren't like for like across the two.
- **What c5d still needs** (about 0.24 s to pass):
- Received-set validation still takes 0.33 s with 98.7% of its
signatures cached, so the rest of validation is now the cost.
- About 245 ms of the c5d build is outside the phase timers.
- The remaining trigger-to-ballot time beyond build, fetch and
validation is about 0.57 s of nomination messaging across 4 regions.
- n = 1 per instance type.
---
## Previous update (2026-11-09)
_Genesis-upgrade fix, fetch robustness, follower signature prewarm._
Rebased onto the updated stack (#5453 now merges `master` @ 362dbc5:
protocol 30, millisecond close times; #5454, #5457 and #5458 updated the
same day). The PR's original four commits are kept (adapted to the new
base: stellar-xdr 30.0.0 paths, #5457's batched frame writes); ten
commits are added on top.
### Bug fix: networks starting at the genesis protocol stalled
(465f6a4)
**Root cause.** With leader-only construction (5fff543), followers no
longer build a tx set at
the trigger and must fetch the leader's. Before protocol 20 a ledger's
set is a legacy
`TransactionSet`, and the Rust overlay only carries
`GeneralizedTransactionSet`s (Core caches,
Rust decodes and Core receives only that type). So a follower could get
the leader's set only by
being promoted to leader by nomination timeouts and building it itself,
one slow round at a time.
Before this PR every validator built the identical empty legacy set (it
depends only on the
previous ledger; upgrades live in the `StellarValue`), so it was always
local. On perfnet (30 nodes)
22 nodes reached ledger 2 and stalled, 8 stayed at ledger 1. The upgrade
ledger itself
(LCL < 20) has the same problem; ledgers from protocol 20 on are
unaffected.
**Fix.** When the LCL is before protocol 20, every validator constructs
its nomination value at
the trigger again, as before; it is still voted only by round leaders.
Leader-only construction
is unchanged from protocol 20 on.
**Test.** `Rust overlay upgrades from genesis through legacy tx sets`: 4
nodes, genesis protocol 0,
four legacy ledgers at normal cadence, an upgrade straight to protocol
30, three more ledgers with
real (generalized) tx sets. It fails without the fix (a node stuck at
ledger 2) and passes with it.
Ablation on the final stack: disabling only this fix, with all the other
changes below in place,
still fails (2/2), so they don't mask the bug.
### Robustness on the same path
- 193fee3: a locally built set completes a pending fetch of the same
hash, so envelopes waiting
for it are processed at once (the IPC path had lost upstream's
ItemFetcher behaviour).
- 942a6b3: a follower promoted to leader after losing sync can still
propose; tracking is
checked at the trigger, as before. This is what made the perfnet stall
permanent.
- ab4eb14, 5964ce6: unanswered tx-set requests are retried and
reported. Core re-requests
undecided slots' sets with backoff (1, 2, 4, then every 8 s), warns on
every doubling of the
request count and counts `overlay.fetch.txset-retry`; Rust sends a retry
to a random other peer
and prunes stale reservations by slot. Core also checks a delivered
set's content hash.
- 5c159ae: test for a leader whose own build completes a pending fetch
and externalizes the
slot inside SCP's nomination supplier.
- Rebase onto the new base: the test-only nomination override from #5481
now also hands its
substituted set to the overlay (31e6147), since this PR caches the
built set earlier.
### Performance
- 3903e4e, f2643ba: follower signature prewarm. The received-set
validation timer rose (about
158 -> 205 ms) because followers no longer validated their own mempool,
which used to warm the
signature-verification cache. Benchmark (6,000 one-signature txs, 4
tasks): 67 ms cold vs
10.5 ms warm. Followers now pull their top mempool transactions once at
the trigger and verify
master-key signatures on background tasks (at most 20 verifications per
tx; two worker threads
left free; skipped if the node already voted; stopped when validation, a
local build or
externalization starts). They still build and compress nothing.
- c26b191: no early tx-set preparation when the trigger is already
due. Both timers fired at the
same instant in unspecified order, so a predicted leader could build its
set twice.
- 2ac0af9: no deep copies of tx-set XDR on the overlay IPC paths
(leader serializes in place,
follower decodes in place and moves into the frame). Sub-millisecond for
classic sets.
### Testing
- Laptop, on c26b191: new and changed tests (24, including the genesis
repro and the trigger,
early-nomination and #5481 tests) and partitioned `make check` for
`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[nominationprotocol]~[.]`
(143 test cases) plus the overlay's Rust tests (169 + 32 + 3) pass.
- Full suite `--all-versions` (devbox, at c26b191): selftest 598 test
cases / 8.5M assertions across protocol versions, check-sorobans,
check-rust and check-nondet pass.
- **Perfnet, head `c26b191fa`** (30 validators in 4 AWS regions, 10
tier-1 orgs, 3,000 SAC TPS offered, 2 s target, 150 ms simulated apply,
tx sets capped at 7,500; n = 1; run in parallel with the #5458 head on
the same fixture):
- **The genesis-upgrade bug is fixed on a real network:** all 30 nodes
synced in 43 s from genesis through the legacy ledgers to protocol 30
(the previous image of this PR left 22 nodes at ledger 2 and 8 at ledger
1).
- Correct under load: no forks, all in sync, no invalid
values/signatures, no restarts, no `NOT IN CACHE`/parse/cache-drop
errors; 11 tx-set fetch retries, all resolved on the first re-request.
- Leader-only construction works as designed (~1 build per ledger
network-wide instead of 30), and zstd shrinks each 7,500-tx SAC set from
~5.6 MB to ~1.0 MB (5.7x, ~7.5 ms to encode).
- Verdict FAIL at the same speed as the #5458 head (window-1 p75 3.26 s
vs 3.25 s): the leader's ~1.1 s build stays on the critical path, and
follower validation of the received set was 0.52 s vs 0.34 s on #5458
(whose followers had warmed their signature cache by building), so the
follower prewarm did not recover it at this load. Nomination round 1 (2
s timeout) times out on almost every ledger.
- Follow-up: done in the 2026-11-10 update above (leader build,
ballot-start prewarm, and the IPC size fix found by the next run).
---
## Original description
> The numbers below come from an earlier simulation environment and are
kept for history. The stack is now measured on perfnet; the run of this
PR's final head will be posted in the update above.
# Description
This PR compresses transaction-set responses in the Rust overlay and
shares the encoded bytes across peer sends. Every response uses the
mandatory zstd protocol; there is no legacy/raw fallback or optional
lazy encoding state. The leader hands the final XDR to Rust before
Core's final validation, so compression can overlap that validation.
Receivers retain the original compressed bytes, decompress and verify
the set, then reuse the retained encoding when forwarding it.
A separate commit, `0ed3e540`, removes private proposals on non-leader
nodes. Followers enter nomination with a lazy value supplier and follow
received proposals without pulling the mempool or constructing a set.
First-round leaders can prepare early; a promoted leader constructs a
value only when it needs one. The driver caches that value across
retries, and externalization stops nomination even when no local value
was built. Tests cover timeout promotion, current close times, following
another leader, unavailable values, and reentrant nomination stop.
This PR builds on #5458. The
comparison below evaluates these revisions against the **original
compressed implementation of this PR**, using the same 10-tier-1,
30-validator fixture: 3,000 offered TPS, 6,000 transactions/block,
2-second target, 8 CPUs/16 GiB per validator, 60-second warmup and
900-second measurement, with simulated apply.
## 3,000 offered TPS
| Metric | Original compression | Updated leader-only path |
|---|---:|---:|
| Median block interval | 2.018 s | 2.120 s |
| Worst-node full-window p99 block interval | 3.834 s | 3.836 s |
| Tx-set fetch p99 | 0.617 s | 0.601 s |
| Completed TPS | 2,926.67 | 2,806.67 |
| Harness verdict | PASS | FAIL (ob-0 only) |
Median intervals and completed throughput use the canonical node;
worst-node p99 uses the fixed measurement window. The harness uses a
different histogram/window: `ob-0` recorded 4.989 s against its 4-second
limit, while the other 29 nodes and inclusion passed. Completed
throughput fell 4.1% and median block interval rose 5.1%; this run does
**not** establish that the revised path matches or improves the original
compression baseline. These are single cloud runs with normal host
placement.
Leader-only construction is verified: 886 builds/compressions across 421
closed ledgers, all correlated with asserted leadership, with no
follower build or received-set recompression detected. Multiple
nomination leaders can build in a ledger. Compared with the intermediate
eager implementation, local compressions fell from 13,019 to 893 in
their respective 900-second windows, roughly a 93% reduction per ledger.
Payload savings remain 80.0%. The sampled received-set validation timer
rose from about 158 ms to 205 ms versus that intermediate build; the
precise cause is not isolated by this run.
# Checklist
- [x] Reviewed the contributing document.
- [x] Kept leader-only proposal construction in its own commit on this
PR's branch.
- [x] Formatting checked with `make format` and `cargo fmt --check`.
- [x] Local build, Jenkins package/image build, and exact-head Clang/GCC
CI passed.
- [x] Rust suite: 194 passed, 3 ignored; the leader-only follow-up
leaves those validated Rust sources unchanged.
- [x] Core validation: 41 focused SCP/nomination/tx-set cases, 17
overlay IPC cases, and all 72 Herder cases covered. PostgreSQL-dependent
cases were rerun after configuring the local database fixture.
- [x] Performance comparison, runtime identity, resource checks and
sealed-artifact audit completed.
- [ ] Performance matches or improves the original compressed baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update (2026-11-09): what changed
This PR now brings overlay-v2-shared up to date with current master (362dbc5) and takes the latest #5449.
Commits
af2c821daMerge master (362dbc5) into overlay-v2-shared. This brings in protocol 30 (CAP-0084, CAP-0087, p29/p30 Soroban hosts), millisecond close times (Millisecond-resolution close times (MS_CLOSE_TIME, vnext) #5423), tx set validity hardening (Tx set validity hardening #5481), the ledger view simplification (Simplify the ledger view hierarchy and interfaces. #5461), and master's apply-path performance work. The previous master merge (Merge master commit 100cc3816c59357df488b17972aa5e2846ead831 into overlay V2 #5428) left master's tip out of its parents, so this merge used 100cc38 as its base. Resolutions:ConsensusTime/ApplyTimeclose times. This includes early nomination preparation.recvTransactiondrops the deprecatedforceflag.3b496b738overlay: decode protocol 30 XDR with stellar-xdr 30.0.0. The Rust overlay pinned stellar-xdr 26, which can't decode protocol 30's millisecondStellarValues. At protocol 30 it would have stopped extracting tx set hashes from SCP messages: no proactive tx set fetch and no source tracking. It would also have dropped txs and tx sets with CAP-0084 arms. stellar-xdr 30.0.0 matches our protocol XDR exactly (4f524bbac). Empty-tx-set values no longer trigger a fetch of the synthetic empty set hash.fe5eefb32Adapt overlay v2 herder tests to millisecond close times.ARTIFICIALLY_SET_CLOSE_TIME_FOR_TESTINGis now in milliseconds, and overlay v2-only tests are updated to match.04fbddb7dParallelize transaction validation for transaction sets (Parallelize transaction validation for transaction sets. #5449, latest version, applied as-is).f006455fcDisable Soroban diagnostic events on overlay v2 simulation nodes. In--check-lcmmode (the CI step master added), master's test configs turn on diagnostic events, which networked validators reject. Overlay v2's simulation nodes are never standalone, so every simulation-based[tx]test threw at startup.14de3972aRecapture LCM golden data for overlay v2's test PRNG stream. 32 golden files differ from master's.TransactionQueuealso draws from that PRNG on every ledger close; overlay v2 has no such queue, so the streams diverge. The apply path itself is identical to master's.getUpperBoundCloseTimeOffsettest.Dropped: "Parallelize Soroban pre-apply step" (it is in master as #5402) and the older #5449 snapshot.
Notes
TransactionFrame::checkValid(Simplify the ledger view hierarchy and interfaces. #5461), so it also covers Parallelize transaction validation for transaction sets. #5449's parallel validation.Testing (devbox; ladder at 04fbddb,
[tx] --check-lcmat 14de397):cargo test: pass. This includes the new tests covering everyStellarValueformat.[herder],[txset],[overlay],[overlay-ipc],[loadgen],[upgrades],[ledger],[history],[catchup],[scp],[simulation],[parallel]): 168 test cases pass.~[acceptance]~[.]plus Soroban host and overlay tests): 580 test cases pass.[tx] --rng-seed 12345 --check-lcm(ci-build.sh): 144 test cases pass with 0 golden mismatches.The stack above this PR (#5454, #5457, #5458, #5459) was rebased onto it and updated the same day.
Original description
Description
Merges changes from #5449. This improves out tx set validation speed by makiing it parallel, which is important for consensus latency.
Improvements are as follows. Measured with 10 tier 1 topology (no watchers) with 2 second block time target:
2000 TPS
This still fails at 2000 TPS, but is much, much closer. It appears that whole second rounding in close times can cause poor trigger times, leading to long p99 blocks. This should be addressed with #5423.
Checklist
clang-formatv8.0.0 (viamake formator the Visual Studio extension)