Repository navigation
Prototype: push the leader's tx set to peers when it is built - #5497
Merged
SirTyson merged 4 commits intoNov 11, 2026
Merged
Conversation
added 4 commits
November 10, 2026 22:55
quinn paces one congestion window over 0.8 round trips and does not grow the window while the sender is application limited. A connection that mostly idles therefore sends a burst of S bytes in about 0.8 * S / window round trips: a 640 KB tx set took 1 to 1.3 round trips on top of the one-way delay on a warm connection, and 6 to 7 on a cold one (two nodes over a loopback with a 150 ms round trip). With an initial window of 8 MiB it arrives about 20 ms after the one-way delay. Add an initial_congestion_window option to the vendored libp2p-quic config (CUBIC's initial window; recorded in congestion-window.patch) and set it to 8 MiB for every peer connection. CUBIC still shrinks the window on loss. STELLAR_OVERLAY_QUIC_INITIAL_CWND overrides it in bytes (0 = quinn's default); the effective window is logged at startup. This is a prototype setting for validators on good links: it applies to every stream of a connection, and a fresh connection may send 8 MiB without probing the path. The ignored test txset_transfer_latency_experiment is the harness behind the numbers; it needs a link with real delay (netem).
Tx-set dissemination was a request model: a follower saw the leader's proposal, asked for the tx set, and received it about two round trips later. The round's leader now sends the set it is about to propose to every connected peer as soon as the set is built, so a follower normally holds it when the proposal arrives. The request path is unchanged and remains the fallback for a node that did not get the push. Core: a new IPC message PUSH_TX_SET [hash][slot]. HerderImpl sends it right after CACHE_TX_SET from inside the build when the node leads the current round, after the build for a first-round leader's capacity-limited prepared set, and again when a prepared set is taken at the trigger. Overlay, sender: the dispatcher writes the cached, already compressed set to the tx-set stream of every connected peer that has not been sent it, one push in flight per peer. The first request from a peer for a set that is being written to it, or was written less than 500 ms ago, is not answered: the bytes are already on the same ordered stream. Every other request is answered, a failed push answers a skipped request after all, and a closed connection clears the record for that peer. Overlay, receiver: a pushed set is an ordinary tx-set frame nobody asked for. It is decompressed, strict-parsed and filed under the hash the receiver computed; it carries no claimed hash. Core is not told: it gets a set only through RequestTxSet for a hash named by an SCP statement, and re-checks the content hash. So a pushed set that does not match the leader's statement is never used, and the proposed set is requested. Since unrequested frames are now a normal event they are bounded: a per-peer token bucket applied before decompression (answers to this node's requests are free), and limits on cached sets from peers that Core has not asked for (per peer, and a byte budget over all peers). No change to tx-set validation, voting or apply. Six meters overlay.txset-push.* report what happened; see docs/rust-overlay/txset-fetching.md.
Corrections to docs/rust-overlay/txset-fetching.md from review: what happens to a push that is in flight when a connection closes, that the one-push-in-flight bound covers pushes only, which peer gets a token back, and that the 100-entry cache is not protected by the limits on unclaimed sets. Also mark the congestion-window patch record as whitespace-exact, like the other vendor patch. No code change.
…ADME The branch README now covers the latest experiment (the leader pushes its tx set, measured on 8f7768b) with the previous experiment's values (the leader schedule alone, daa965f) as "before": 1,000 ms target at 3,000 TPS, block time windows, stage timings, the push counters, the duplicate inclusions that overlay-only mode allows and why they are accepted, and the test setup. README only.
SirTyson
marked this pull request as ready for review
November 11, 2026 20:43
SirTyson
added a commit
that referenced
this pull request
Nov 11, 2026
…peculatively, skip an invalid set at apply (#5498) **Prototype, not for merge.** Stacked on #5497 (push the leader's tx set). This PR adds two commits to #5497's four: `a33da62c4`, the change, and `0c9a02e56`, a fix for a bug in the Core-overlay IPC reader that the perfnet run below hit (it is in the stack below this PR). From protocol 30, a validator votes for the round leader's value without validating its transaction set. The validity check moves to ledger apply, where an invalid set closes an empty ledger, and the ledger is executed speculatively while consensus is still in flight. The transaction set is made durable before the vote by the overlay, in its compressed form. There is no compatibility work beyond the protocol gate; the PR exists to measure what validation, persistence and apply cost on the leader-schedule stack. ## Result in one paragraph At the new target fixture (16 tier-1 orgs = 48 validators in 4 regions, c6id.2xlarge, 1 s target, 5,000 SAC TPS, one run per build) #5497 alone misses: 1.42 s per ledger, avg p75 1,461 ms, every tx set at the cap. With this PR on top, all four windows are on target on all 48 nodes: avg p75 1,094 to 1,137 ms, 1.04 to 1.07 s per ledger, and the time from trigger to ledger close falls from 1.37 s to about 0.8 s. Every ledger was executed speculatively and committed as executed (hit rate 100%, no execution dropped, no set skipped). The run's verdict is still **FAIL**: one node dropped out of consensus 15 s before the load ended, because Core's IPC reader thread took an interrupted `recv` for a closed connection. That bug is in `src/overlay/IPC.cpp`, below this PR; the second commit fixes it, and a rerun with the fix is in progress. The comparison is between two regimes (over capacity against keeping up), and the speculation gain on this fixture is an upper bound (see Perfnet). ## What changes (all gated on protocol 30; protocol 29 and older are unchanged) - **Vote without validating.** `HerderSCPDriver::validateValueAgainstLocalState` returns `kFullyValidatedValue` for a value whose tx set the node holds, without running the set's validity check. Still checked before the vote: the StellarValue itself (signature, close time, upgrades), that the node holds the set, and one O(1) property of the set, that it was built on the LCL. - **Validate at apply; an invalid set applies as an empty ledger.** `LedgerManagerImpl::validateTxSetForApply` runs `prepareForApply` and `checkValid` against the previous ledger with the value's close time. If either fails, nothing of the set is applied: no fee, no sequence bump, no result. - The header keeps the externalized value (the invalid set's hash); `txSetResultHash` is the hash of an empty result set. - Upgrades in the value still apply. - History keeps the set although the ledger has no results, and LedgerCloseMeta carries it with no transaction meta. A node replaying the ledger needs the set to match the header, and reaches the same verdict through the same code. - This is not the existing empty-tx-set value: consensus runs on, and externalizes, the invalid value. - **Speculative execution.** `applyLedger` is split into `executeLedger` (validation, fees, transaction execution; everything it does lives in an object that rolls back when dropped) and `commitLedger` (history, upgrades, seal into the BucketList and database, meta, LCL). When a validator emits a PREPARE or CONFIRM for a value for LCL+1, the ledger apply thread executes that ledger. When a value externalizes, the executed ledger is committed if it is for the same ledger, value and set; otherwise it is dropped and the ledger is applied from scratch. `EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` (default true) turns it off. - **Persist the compressed set, in the overlay.** The Rust overlay writes the zstd frame of every tx set Core holds to `<BUCKET_DIR_PATH>/txsets` (temp file, `fdatasync`, rename) before it delivers a set to Core, and for a set Core built before it handles Core's next IPC message (the push and the proposal). Core no longer serializes, base64-encodes and commits those sets to SQLite in front of its first vote; after a restart it requests the sets its restored SCP state names, and the overlay reads them back from disk. Only sets Core holds are stored, and they are deleted by slot age, so peers filling the overlay's cache cannot evict them. If a set Core built cannot be stored, the overlay exits instead of sending anything that refers to it. - The validity check no longer reads the LCL through the application: it takes a `TxSetValidationContext` (previous header, Soroban config, a factory for read-only ledger views), so it can run on the apply thread against apply's own snapshot. - New metrics: `ledger.apply.{txset-skipped, execute, commit, speculation-started, speculation-hit, speculation-miss, speculation-idle}`, `herder.scp.persist`. `herder.txset.validate` now times the apply-time check. ## Why it should be safe, and where it is not - **Consensus safety is unchanged**: nodes vote for more values than before, never for two. - **The apply-time verdict is a function of the previous ledger, the set and the value's close time**, read from an immutable snapshot: the same inputs the pre-vote check used. Live apply and replay run the same function. - **Speculation is not observable.** The same two functions run in the same order whether execution was early or not. A test compares ledger hashes for no speculation, speculation on the externalized value and speculation on another value. - **What gets worse, by design of the experiment:** - Any nondeterminism in `checkValid` used to cost liveness at most. It now forks the ledger. Its node-local inputs are the binary's protocol version, the network passphrase and, in test builds, loadgen flags (`ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING`, `LOADGEN_BYTE_COUNT_*`): validators that differ in those would disagree on validity. - A leader can get an invalid set externalized. The ledger is empty, and the set's transactions are still removed from every mempool at externalize, so a faulty leader can drop pending transactions at no cost whenever it leads. - Downstream consumers see new shapes: a LedgerCloseMeta with a non-empty tx set and no transaction meta, and a history `transactions` entry without a `results` entry. ## Testing On `a33da62c4` (the change), devbox, 32 partitions: - **Full suite: PASS.** 614 test cases, 2,384,306 assertions, plus libsodium, xdrpp, check-rust and the Soroban host tests. 12 of them are this PR's new cases and 4 are #5497's. - **Overlay Rust tests: PASS.** 199 library, 44 binary, 7 end-to-end. - **Full suite `--all-versions`: PASS.** 614 test cases, 8,126,148 assertions. On the same change before the rebase (`25f48abb6`, tree of `3b8a8356c`): - **tx-meta baseline** (`[tx]`, all versions, seed 12345): unchanged, 144 test cases. - **LedgerCloseMeta golden data** (`--capture-lcm`): no file changed. New tests: - `herder/test/ApplyTimeTxSetValidationTests.cpp` (6 cases): an invalid or malformed set closes an empty ledger and the next ledger applies normally; upgrades of a skipped value apply; the same set applies or is skipped depending on the value's close time; protocol 29 applies the set as before; `validateValue` levels on both sides of the gate; a 4-node network externalizes a leader's invalid sets and every node closes them empty with the same hash; a node restarts and gets its tx set back from the overlay's store, with no tx-set row in SQLite. - `ledger/test/SpeculativeApplyTests.cpp` (5 cases): an executed ledger is committed when its value externalizes; it is dropped for another set, another close time, a newer speculation, an explicit abort, and at shutdown; an invalid set under speculation; identical ledger hashes with and without speculation, including upgrades and skipped sets; a 4-node network where every node commits speculatively executed ledgers. - `history/test/HistoryTests.cpp` (1 case): a node replays skipped ledgers from an archive and closes the same hashes. It fails if history does not keep the skipped set. - Rust: `flood::txset_store` (4) and App tests (5): on disk before Core gets the set; an unstorable set is not delivered, and stops the overlay if Core built it; served to Core after a restart; deleted by slot age; peers cannot evict a stored set. Existing tests changed: four "SCP Driver" assertions and the signature-prewarm test that encoded validation before voting; "network does not externalize non-canonical empty tx set" is now "network externalizes non-canonical empty tx set" (it externalizes and closes an empty ledger). On `0c9a02e56` (the IPC fix on top, this PR's head): **full suite PASS**, 615 test cases, 2,385,530 assertions. The new test "IPC read survives a signal" fails without the fix. `--all-versions` on this head is running. Not passing or not run: - Tests outside the regular suite that fail the same way without this change: "Catchup recent" and "resilience tests" (checked on `overlay-v2-shared` `87201aa66`), "hierarchical topology scales 1 to 3" (as recorded in #5495). - One "SCP Driver" assertion (protocol 29, validity-cache hits) failed once in a loaded partitioned run of the pre-rebase change and passed in three runs alone. Not explained. - A `--disable-tests` build was not tried; #5495 records that the base does not build that way. - No sanitizer run. ## Perfnet 16 orgs (48 validators), 4 regions, c6id.2xlarge, 1,000 ms target, 5,000 SAC TPS, 150 ms simulated apply per ledger (30 us per op), tx-set cap 6,250, 900 s measured, SCP timeouts 2 s. Same day, back to back, **n = 1 per build**. This fixture has no earlier comparison point. | | #5497 alone (build #3675, `8f7768be7`) | This PR without the IPC fix (build #3677, `a33da62c4`) | |---|---:|---:| | Verdict | **FAIL**: window 1 missed on 48/48 nodes; load stopped after it | **FAIL**: node-45 out of sync and its load generator failed; all four windows on target on 48/48 nodes | | avg p75 per window | 1,461 ms | 1,094 / 1,116 / 1,130 / 1,137 ms | | avg p99 per window | 1,638 ms | 1,313 / 1,453 / 1,408 / 1,435 ms | | Ledgers per 5 min (per node) | 211-212 (1.42 s each) | 288-290, then 281-283 (1.04-1.07 s each) | | Txs per ledger; sets at the 6,250 cap | 6,250; every set | 5,190-5,340; 5% of ledgers | | Leader build (mean) | 0.411 s | 0.232-0.249 s | | Trigger to first ballot (mean) | 0.801 s | 0.421-0.468 s | | Ballot to externalize (mean) | 0.367 s | 0.296-0.298 s | | Externalize to ledger close complete (core logs, mean) | 0.208 s | 0.071 s | | **Trigger to close** | **1.37 s** | **0.79-0.84 s** | | `herder.txset.validate` (mean) | 0.195 s, before the vote, main thread | 0.178-0.190 s, at apply, apply thread, during the ballot | | `ledger.apply.execute` / `ledger.apply.commit` (mean) | no such timers | 0.33-0.35 s / 0.037-0.041 s | | Speculation: started / hit / miss / sets skipped (network, whole load) | n/a | 46,847 / 46,853 / 0 / 0 (hit rate 1.000 on every node) | | Finished execution waiting for externalize (mean, median node) | n/a | 8-10 ms | | `herder.scp.persist` (mean; p99), in front of every SCP broadcast | no timer in this build | 2.2-2.6 ms; 5-6 ms | | Leader and prepare timeouts, rounds above 1, empty tx sets | 0, 0, none, 0 | 0, 0, none, 0 | | Error counters / in sync / no fork / still closing | passed / passed / passed / passed | passed / **node-45 not in sync** / passed / passed | | Inclusion | check skipped (early stop); 6.8% short, over capacity | check skipped (a generator failed); 4,798,690 of 4,800,000 on the 47 healthy nodes | Reading it: - **Two regimes.** #5497 alone is over capacity at this setting: 5,000 TPS need 7,075 transactions per 1.42 s ledger against a cap of 6,250, so every set is full, the mempool backs up and every trigger is already due. With this PR the network keeps up, sets are not full and the leader's build is shorter for that reason alone (0.41 s to 0.24 s). The stage differences are not all this PR's code. - **What this PR's code accounts for:** followers no longer validate (0.195 s) or write the set to SQLite before their first vote, and apply after externalize shrinks from 0.21 s to the commit (0.04 s) plus whatever of the execution is still running (0.07 s in total from the logs). - **The speculation gain is an upper bound.** The simulated apply is a sleep, so running it during the ballot costs no CPU. Real execution would compete with SCP processing and the signature prewarm. - **The sleep is not placed the same way in the two builds.** In the base it covers everything from the start of apply (commit included). Here it sits in the execute phase, and validation and commit come on top. - **Executions finish with little to spare.** The validation plus the simulated execution take 0.33-0.35 s against about 0.3 s of ballot; a finished execution waited 8-10 ms on average for its value to externalize. - **The push from #5497 loses its race here.** In the base run 13,914 of 14,352 pushed sets were in place when Core asked. In this run Core's request reached the leader before the push had arrived in 38,668 cases against 7,032, because the proposal now leaves earlier relative to the push (no SQLite write in front of it). It cost about 0.02 s per ledger (`scp.fetch.txset` 0.052 s against 0.033 s); no retry was needed. **The failure.** node-45 stopped following consensus at ledger 1024, 15 s before the generators finished; node-23 and node-32 did the same two minutes after the load. On each, Core logged `Overlay IPC connection closed` a few milliseconds into a checkpoint publish while both processes stayed up and the socket stayed open, with the overlay's send queue full. `readExact` in `src/overlay/IPC.cpp` returns "closed" for any `recv` result <= 0, `EINTR` included, and stellar-core's signal handlers do not restart system calls; the inference (no errno was logged) is a SIGCHLD from the publish's child processes delivered to the reader thread. The base run had no such event in 30 minutes, this run three in 40: one run each cannot say whether this PR makes it more likely. None of this PR's own error paths fired on any node. Commit `0c9a02e56` retries on `EINTR` and adds a test that interrupts a blocked reader. A rerun of this PR's head with that fix (build #3678) is in progress; the table gets a third column from it. ## Review findings not fixed An independent review of the pre-rebase change found no case where the speculative path commits a different ledger than the regular one, or where live apply and replay decide differently. Not fixed: - The two "gets worse" items above (node-local test flags in validity; mempool purge of a skipped set). - A `CacheTxSet` that the overlay drops for a hash mismatch or an encode error (both Core bugs) leaves Core believing the set is stored. - A value is not speculated again after its execution was dropped; `ledger.ledger.close` leaves out the time a finished execution waits and executions that were dropped; in test builds `getLastClosedLedgerTxMeta()` can show a ledger that is still speculative. - Test gaps: the hash-equality test covers classic payments and upgrades only (no Soroban parallel apply, no DEX offers, no meta comparison); speculation tests always let execution finish before externalizing or aborting; no kill-restart test of the store. - Open question: a parked execution keeps a SQL transaction open for the ballot window. Not confirmed that no other writer on the main database can make its commit fail. - The merge with #5497 (a pushed set is stored before Core gets it, like any other) was not reviewed separately. ## Next steps Not started. The commit and the next leader's build still wait for externalize. Letting the next leader build on the speculatively executed state is the next 0.2 s or so, and needs either every node pre-building or a schedule seed that is known one ledger earlier.
SirTyson
added a commit
that referenced
this pull request
Nov 11, 2026
…time tx-set validation The benchmark fixture is now 16 organizations (48 validators) in 4 regions at 5,000 SAC TPS and a 1,000 ms target. Report the two runs at that fixture: the leader tx-set push alone (#5497) misses the first window (p75 1,461 ms), apply-time validation with speculative execution (#5498) is on target in all four windows (p75 1,094-1,137 ms), with a FAIL verdict for a node that lost its IPC reader (fixed in 0c9a02e, not rerun). Describe the third experiment, update the measurement setup, limits and known limitations, and condense the 30-validator results.
SirTyson
added a commit
that referenced
this pull request
Nov 11, 2026
… validation results (#5499) README only, no code change. The benchmark for this branch is now **48 validators (16 organizations, 4 regions), 5,000 SAC TPS, 1,000 ms target**. The README reported the 30-validator / 3,000 TPS runs of #5495 and #5497; this updates it to the runs at the new fixture and adds #5498. ## What the README now says | | Tx-set push alone (#5497, `8f7768be7`) | With apply-time validation (#5498, `0c9a02e56`) | |---|---|---| | Block time windows | missed the first window on all 48 nodes | on target in all four windows, on all 48 nodes | | p75, average over 48 nodes | 1,461 ms (limit 1,200) | 1,081 / 1,132 / 1,118 / 1,142 ms | | Time per ledger | 1.42 s | 1.03-1.08 s | | Trigger to ledger closed | 1.37 s | 0.76-0.84 s | | Run verdict | FAIL (block time) | FAIL (inclusion check only) | - **The run with the change fails the inclusion check, and only that check.** Every node holds 4,800,060 transactions against 4,800,000 offered. The check is an exact comparison (performance-testing#4, which would allow this, is not merged). - **The 60 extra transactions are not explained.** The 30-validator run of #5497 had exactly 60 extra too. The explanation given then (late copies included twice in overlay-only mode) was an inference, and it does not predict the same number at a different rate and network size. The README now says so. - **The IPC bug.** The change was first run without its last commit (`a33da62c4`): same block times, but one node stopped reading its overlay IPC socket 15 s before the end of the load. The fix (`0c9a02e56`) is in the reported run, where fourteen sweeps of all 48 nodes (44 checkpoint publishes per node) found no affected node. One clean run does not prove the fix; the "before" run had no such event either. - One run per build. The gain from speculative execution is an upper bound, because apply is simulated by a sleep. ## Changes - Title and intro: three experiments; the benchmark fixture. - "What this branch is": adds experiment 3 (vote without validating, validate at apply, empty ledger for an invalid set, speculative execution, tx set stored by the overlay; protocol 30 gate). - "Performance results": the 48-validator runs, windows, stage table, speculation and tx-set store numbers, the inclusion check, the IPC bug, where a ledger's time goes now. - The 30-validator / 3,000 TPS results are condensed into one section that links to the README as of #5497 for the details. - "How it was measured": 16 organizations, 12 nodes per region, full mesh of 47 peers, tx-set cap 6,250, 30 us simulated apply per operation, the command with `--org-count 16`. - "Known limitations": adds the open items of #5498. - Test status of the branch head, including one `--all-versions` failure that comes from a test helper of #5495 (it fails for one combination of the day's test seed and a partition's instance number, also on the binary of #5495 alone). ## Corrections to the previous README - It said the perfnet inclusion check "has since been relaxed". That change is not merged. - It presented the cause of the 60 duplicate inclusions as understood. It is not. ## Not verified - The quorum threshold (11 of 16 organizations) is computed from stellar-core's rule for generated quorum sets, not read back from a node. - Round-trip times between regions are the ones measured on the 30-validator fixture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, not for merge as is. Like #5495 it has no flag and no protocol gate. It moves tx-set dissemination from a request model to a push model: the round's leader sends the tx set it is about to propose to its peers as soon as the set is built, instead of waiting for each follower to ask for it.
Stacked on #5495. The branch sits on
b524ef962, the head #5495 had when it was merged;overlay-v2-shared(cc51aba48, #5496) has the same tree, so this PR shows only its own four commits:0f9255d47Start QUIC connections with an 8 MiB congestion window8f7768be7Push the leader's tx set to peers when it is built (prototype)d0fe62652Document the remaining gaps of the tx-set push (docs and one.gitattributesline, no code)2f9baef24Describe the tx-set push experiment and its perfnet results in the README (the branchREADME.mdnow covers this experiment, with Prototype: replace SCP nomination with a deterministic leader schedule #5495's values as "before"; README only)Everything below was built, tested and measured on
8f7768be7.Result in one paragraph
On perfnet (30 validators, 4 AWS regions, c6id.2xlarge, 3,000 SAC TPS) at the 1 s target that #5495 failed, all four windows are on target on every node: avg p75 1,091 / 1,096 / 1,109 / 1,115 ms against 1,386 ms, and 1.05 s per ledger against 1.33 s. A follower's Core gets the leader's tx set 16 ms after it asks for it, where it took 217 ms. One caveat: every node's ledgers hold 60 transactions more than the 2,880,000 that were offered. The counters point to about 60 transactions being included in two ledgers, which only the fixture's overlay-only mode allows; see "The inclusion check" below. The run was judged with the exact check and reported FAIL for that check alone. It is accepted as an artifact of that mode, and the check is being relaxed to allow such duplicates (stellar-experimental/performance-testing#4). One run, against one run of the base from the same day.
Design
PUSH_TX_SET [hash][slot].HerderImplsends it right afterCACHE_TX_SETfrom inside the build when the node leads the current round, after the build for a first-round leader's capacity-limited prepared set (the one case where the prepared set is proposed unchanged), and again when a prepared set is taken at the trigger. Later-round leaders go through the same build hook.CACHE_TX_SETalone does not push: it also carries sets that are never proposed.RequestTxSetis answered from the cache at once.Why it should be safe
docs/rust-overlay/txset-fetching.md:Testing (all on
8f7768be7)herder/test/TxSetPushTests.cpp, 4 network simulations with real overlay processes): the leader's push is used (no follower waits for the retry); consensus through the request path when no push is sent; a network where half the leaders push; every leader pushes one set and proposes another (the pushed set is never used, every ledger externalizes the proposed set). One IPC section inoverlay/test/IPCTests.cpp.flood/txset_push.rs), 7 two- and three-node tests (libp2p_overlay/txset_push_tests.rs: push reaches every peer once, a crossing request is skipped once, a late request is answered, a peer that missed the push is served, the record is cleared on a closed connection, the rate limit, a malformed frame), 7 App-level tests (main.rs: a pushed set reaches Core only on request, a set with another hash does not answer a request, the limits), 4 end-to-end tests with two overlay binaries (tests/e2e_binary.rs: push before the request, no push, pushed set with another hash, a dropped push recovered by a request).test result: ok, 0 failed).--all-versions: PASS. 602 test cases / 8,155,400 assertions.[herder],[scp],[txset],[overlay-ipc],[overlay],[ipc],[upgrades],[simulation](150 tests, on the PR headd0fe62652): passes except three[acceptance]tests that are not in the full suite and fail without this change: "hierarchical topology scales 1 to 3" and "core nodes with outer nodes" (as reported in Prototype: replace SCP nomination with a deterministic leader schedule #5495), and "SCP Driver / validateValue txSet cached" (HerderTests.cpp:2964,0 == 9), which fails on Prototype: replace SCP nomination with a deterministic leader schedule #5495's source too whenever the test runs with--base-instance50 or 400, i.e. in partition 2 or 9 of a partitioned run, and passes with 0 or 450.clang-format20.1.8 on the changed C++ files andcargo fmt --checkon the overlay crate: clean.--disable-testsbuild stops inherder/HerderImpl.cppon the same test headers as on Prototype: replace SCP nomination with a deterministic leader schedule #5495. The three other non-test objects this change touches compile withoutBUILD_TESTS.Perfnet
30 validators in 4 regions, 10 tier-1 orgs, c6id.2xlarge, SAC payments, 150 ms simulated apply per ledger (50 us per op at this setting), 900 s measured, all SCP timeouts 2 s. Target 1,000 ms, 3,000 TPS. Base: #5495's commit
daa965f4c(before its rebases, which changed formatting, the README and a CI file), build #3674, the same fixture and instance type on the same day; its load was stopped after the first window, as block_time_check does on a miss. This PR: build #3675 of8f7768be7. One run each.daa965f4c8f7768be7scp.fetch.txset: mean (p75 / p99)Stage values are 5-minute means at the end of window 1, median over the 30 nodes.
The inclusion check: 60 transactions too many
block_time_check compares the transactions in every node's ledgers with the number offered. All 30 nodes hold exactly 60 more (same value everywhere, no fork). The generators submitted exactly 2,880,000 (96,000 each, none rejected) and every node pulled every foreign transaction exactly once, so about 60 transactions (0.002%) were included in two ledgers. #5495's completed run at 2 s had the exact count; its 1 s run stopped before this check.
How that can happen here (the code path is the stack's documented behaviour; that it explains these 60 is inferred from the counters, the logs carry no transaction hashes):
externalization_removes_queued_admission_without_remembering_hashessays "Truly later admissions are deliberately allowed; no finalized hash history remains. Ledger-state validity is still checked by Core." A copy that reaches a node for the first time after the removal is admitted as new and included when that node next leads.So the design relies on Core's ledger-state validation, which this fixture switches off, and the push's speed-up is what makes the two meet. This PR does not change that, and the overlay keeps its behaviour (remembering finalized hashes there would change production behaviour for a test-mode artifact). The duplicates are accepted for this fixture: block_time_check's inclusion check is relaxed to allow up to 1 in 10,000 transactions more than offered, with the caveat written next to it (stellar-experimental/performance-testing#4). Fewer than offered, more than that, and nodes that disagree still fail. No run has been made with the relaxed check; applied to this run's recorded counts it passes with 60 over.
Open review findings (not fixed in
8f7768be7)Next steps
Not started: a run of the base with only the congestion window, and of the push on quinn's default window, to separate the two effects; repeats (n = 1 here); a longer run to see whether the drift continues; then 800 ms.