Repository navigation
Prototype: vote without validating the tx set; validate and execute speculatively, skip an invalid set at apply - #5498
Merged
SirTyson merged 6 commits intoNov 11, 2026
Conversation
quinn paces one congestion window over 0.8 round trips and does not grow the window while the sender is application limited. A connection that mostly idles therefore sends a burst of S bytes in about 0.8 * S / window round trips: a 640 KB tx set took 1 to 1.3 round trips on top of the one-way delay on a warm connection, and 6 to 7 on a cold one (two nodes over a loopback with a 150 ms round trip). With an initial window of 8 MiB it arrives about 20 ms after the one-way delay. Add an initial_congestion_window option to the vendored libp2p-quic config (CUBIC's initial window; recorded in congestion-window.patch) and set it to 8 MiB for every peer connection. CUBIC still shrinks the window on loss. STELLAR_OVERLAY_QUIC_INITIAL_CWND overrides it in bytes (0 = quinn's default); the effective window is logged at startup. This is a prototype setting for validators on good links: it applies to every stream of a connection, and a fresh connection may send 8 MiB without probing the path. The ignored test txset_transfer_latency_experiment is the harness behind the numbers; it needs a link with real delay (netem).
Tx-set dissemination was a request model: a follower saw the leader's proposal, asked for the tx set, and received it about two round trips later. The round's leader now sends the set it is about to propose to every connected peer as soon as the set is built, so a follower normally holds it when the proposal arrives. The request path is unchanged and remains the fallback for a node that did not get the push. Core: a new IPC message PUSH_TX_SET [hash][slot]. HerderImpl sends it right after CACHE_TX_SET from inside the build when the node leads the current round, after the build for a first-round leader's capacity-limited prepared set, and again when a prepared set is taken at the trigger. Overlay, sender: the dispatcher writes the cached, already compressed set to the tx-set stream of every connected peer that has not been sent it, one push in flight per peer. The first request from a peer for a set that is being written to it, or was written less than 500 ms ago, is not answered: the bytes are already on the same ordered stream. Every other request is answered, a failed push answers a skipped request after all, and a closed connection clears the record for that peer. Overlay, receiver: a pushed set is an ordinary tx-set frame nobody asked for. It is decompressed, strict-parsed and filed under the hash the receiver computed; it carries no claimed hash. Core is not told: it gets a set only through RequestTxSet for a hash named by an SCP statement, and re-checks the content hash. So a pushed set that does not match the leader's statement is never used, and the proposed set is requested. Since unrequested frames are now a normal event they are bounded: a per-peer token bucket applied before decompression (answers to this node's requests are free), and limits on cached sets from peers that Core has not asked for (per peer, and a byte budget over all peers). No change to tx-set validation, voting or apply. Six meters overlay.txset-push.* report what happened; see docs/rust-overlay/txset-fetching.md.
Corrections to docs/rust-overlay/txset-fetching.md from review: what happens to a push that is in flight when a connection closes, that the one-push-in-flight bound covers pushes only, which peer gets a token back, and that the 100-entry cache is not protected by the limits on unclaimed sets. Also mark the congestion-window patch record as whitespace-exact, like the other vendor patch. No code change.
…ADME The branch README now covers the latest experiment (the leader pushes its tx set, measured on 8f7768b) with the previous experiment's values (the leader schedule alone, daa965f) as "before": 1,000 ms target at 3,000 TPS, block time windows, stage timings, the push counters, the duplicate inclusions that overlay-only mode allows and why they are accepted, and the test setup. README only.
…y, skip an invalid set at apply (prototype)
From protocol 30 (APPLY_TIME_TX_SET_VALIDATION_PROTOCOL_VERSION):
- A validator votes for a value as soon as it holds the value's transaction
set. The set is no longer validated before the vote; the only thing still
checked is that it was built on the last closed ledger.
- The validity check runs when the ledger is applied, against the previous
ledger and with the value's close time. A set that fails it is not applied:
the ledger closes as if the set were empty. The header keeps the
externalized value, upgrades in the value still apply, and history and
LedgerCloseMeta keep the set, so that a replaying node reaches the same
verdict.
- Ledger apply is split into an execute phase (validation, fees, transaction
execution; no effect outside of the returned object) and a commit phase.
When a validator votes for a value for the next ledger, the ledger apply
thread executes that ledger while consensus is in flight; when a value
externalizes, the executed ledger is committed if it is for the same value
and dropped otherwise. EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY (default true)
turns this off.
- The Rust overlay writes the compressed form of every transaction set Core
holds to disk before it delivers the set to Core, or (for a set Core built)
before it handles Core's next message, the proposal. Core no longer writes
those sets to its database with the SCP state, and gets them back from the
overlay after a restart.
Protocol 29 and older are unchanged.
The validity check takes a TxSetValidationContext instead of reading the
last closed ledger through the application, so that it can run on the ledger
apply thread against apply's own snapshot.
New metrics: ledger.apply.{txset-skipped, execute, commit,
speculation-started, speculation-hit, speculation-miss, speculation-idle},
herder.scp.persist. herder.txset.validate now times the apply-time check.
…et for a closed connection readExact and writeExact treated any recv/send result <= 0 as the end of the stream, EINTR included. stellar-core's signal handlers do not restart system calls, and a process-directed signal such as SIGCHLD (child processes of a history publish) can be delivered to the IPC reader thread while it is blocked in recv. The reader then logged 'Overlay IPC connection closed' and exited with the socket still open: Core heard nothing more from its overlay, the overlay blocked on the full socket, and the node dropped out of consensus. Seen on perfnet on 3 of 48 nodes in one 40-minute run (16 orgs, 1 s target, 5,000 TPS), each time a few milliseconds into a checkpoint publish. Retry on EINTR. The wire framing (IPCChannel::receive) is covered by a test that interrupts a blocked reader before the header and between header and payload.
SirTyson
marked this pull request as ready for review
November 11, 2026 20:43
SirTyson
added a commit
that referenced
this pull request
Nov 11, 2026
…time tx-set validation The benchmark fixture is now 16 organizations (48 validators) in 4 regions at 5,000 SAC TPS and a 1,000 ms target. Report the two runs at that fixture: the leader tx-set push alone (#5497) misses the first window (p75 1,461 ms), apply-time validation with speculative execution (#5498) is on target in all four windows (p75 1,094-1,137 ms), with a FAIL verdict for a node that lost its IPC reader (fixed in 0c9a02e, not rerun). Describe the third experiment, update the measurement setup, limits and known limitations, and condense the 30-validator results.
SirTyson
added a commit
that referenced
this pull request
Nov 11, 2026
… validation results (#5499) README only, no code change. The benchmark for this branch is now **48 validators (16 organizations, 4 regions), 5,000 SAC TPS, 1,000 ms target**. The README reported the 30-validator / 3,000 TPS runs of #5495 and #5497; this updates it to the runs at the new fixture and adds #5498. ## What the README now says | | Tx-set push alone (#5497, `8f7768be7`) | With apply-time validation (#5498, `0c9a02e56`) | |---|---|---| | Block time windows | missed the first window on all 48 nodes | on target in all four windows, on all 48 nodes | | p75, average over 48 nodes | 1,461 ms (limit 1,200) | 1,081 / 1,132 / 1,118 / 1,142 ms | | Time per ledger | 1.42 s | 1.03-1.08 s | | Trigger to ledger closed | 1.37 s | 0.76-0.84 s | | Run verdict | FAIL (block time) | FAIL (inclusion check only) | - **The run with the change fails the inclusion check, and only that check.** Every node holds 4,800,060 transactions against 4,800,000 offered. The check is an exact comparison (performance-testing#4, which would allow this, is not merged). - **The 60 extra transactions are not explained.** The 30-validator run of #5497 had exactly 60 extra too. The explanation given then (late copies included twice in overlay-only mode) was an inference, and it does not predict the same number at a different rate and network size. The README now says so. - **The IPC bug.** The change was first run without its last commit (`a33da62c4`): same block times, but one node stopped reading its overlay IPC socket 15 s before the end of the load. The fix (`0c9a02e56`) is in the reported run, where fourteen sweeps of all 48 nodes (44 checkpoint publishes per node) found no affected node. One clean run does not prove the fix; the "before" run had no such event either. - One run per build. The gain from speculative execution is an upper bound, because apply is simulated by a sleep. ## Changes - Title and intro: three experiments; the benchmark fixture. - "What this branch is": adds experiment 3 (vote without validating, validate at apply, empty ledger for an invalid set, speculative execution, tx set stored by the overlay; protocol 30 gate). - "Performance results": the 48-validator runs, windows, stage table, speculation and tx-set store numbers, the inclusion check, the IPC bug, where a ledger's time goes now. - The 30-validator / 3,000 TPS results are condensed into one section that links to the README as of #5497 for the details. - "How it was measured": 16 organizations, 12 nodes per region, full mesh of 47 peers, tx-set cap 6,250, 30 us simulated apply per operation, the command with `--org-count 16`. - "Known limitations": adds the open items of #5498. - Test status of the branch head, including one `--all-versions` failure that comes from a test helper of #5495 (it fails for one combination of the day's test seed and a partition's instance number, also on the binary of #5495 alone). ## Corrections to the previous README - It said the perfnet inclusion check "has since been relaxed". That change is not merged. - It presented the cause of the 60 duplicate inclusions as understood. It is not. ## Not verified - The quorum threshold (11 of 16 organizations) is computed from stellar-core's rule for generated quorum sets, not read back from a node. - Round-trip times between regions are the ones measured on the 30-validator fixture.
SirTyson
added a commit
that referenced
this pull request
Nov 12, 2026
… an admission check (#5501) **Prototype, not for merge. Stacked on #5502** (leader terms, pre-validation during the previous ballot, one build before the trigger): this PR's own changes are the last two commits, `6ab837c27` (the code) and two README commits (the results of the whole stack); the commits before them are #5502's (head `8aa7f6bf0`) and leave this diff when #5502 merges. A node that accepts a transaction from a client checks it against its last closed ledger, keeps it, and has its overlay send the body straight to the leaders of the current and the next round of the next ledger, instead of advertising it to every peer (INV/GETDATA). Since protocol 30 a follower votes without validating, so it does not need a transaction before the leader's set arrives. ## Results in short One run per configuration everywhere; perfnet `block_time_check`, SAC payments, 150 ms simulated apply, overlay-only mode. **On top of #5502, 16 tier-1 orgs (48 validators in 4 regions), 800 ms target, 8,000 TPS, c6id.8xlarge (16 cores):** - **Block time is the same as #5502 alone, on target in all four windows:** avg p75 804 to 805 ms against 805 to 807 ms; trigger to close 0.43 to 0.44 s against 0.44 to 0.45 s. Both runs sit at the fixture's floor, so this run shows neither a gain nor a loss in block time. - **Overlay traffic per node falls from 21.2 to 6.2 MB/s each way (-71%).** No adverts or demands are left; 3.3 transaction bodies are sent per offered transaction. - The leader's side is unchanged: one build per ledger before the trigger, the pre-validation covers 66.7% of the candidates (69% on #5502 alone), leader terms of exactly 5 ledgers, no timeout. - Followers verify signatures inline: the apply-time check takes 0.099 s against 0.081 s, still inside the ballot on 16 cores. - The verdict is FAIL on the inclusion count only, as for #5502 alone (the check is exact; see "Inclusion" under Perfnet). **On top of #5502, the goal fixture: 30 orgs (90 validators), 800 ms, 10,000 TPS, c6id.8xlarge (this PR's head):** - **FAIL: window 1 at avg p75 980 ms (limit 960), p99 1,441 ms (inside its 1,600 limit); the load was stopped after it.** #5502 alone at this fixture: 1,032 ms and p99 1,887 ms. - A ledger every 0.90 s against 1.02 s; 39% of the sets at the 10,000 cap against 93%; three of four leader builds before the trigger against none; trigger to close 0.77 s against 0.97 s. The gain is in trigger to first ballot (0.39 s against 0.58 s): the leader's mempool holds about one ledger of candidates instead of a 20,200 backlog. - **Overlay traffic per node: 15.7 MB/s against 43.** At 90 validators most of what is left is SCP (35,700 messages per second per node). - What still does not fit into 800 ms is the ballot (0.32 s), the set reaching followers 0.17 s after the proposal, and the builds that still run at the trigger. None of these is transaction dissemination. - **Where the remaining time goes** (twelve nodes' logs): the leader's build starts when the previous ledger closes and takes 0.29 s to the proposal; the set then needs 0.19 s to reach a follower (the leader sends 5.7 MB of XDR, compressed, to each of 89 peers); the ballot takes 0.34 to 0.36 s; the commit 0.07 s. Sum 0.89 s on the leader. The README has the table. **Before the rebase (on `overlay-v2-shared` at `26bea1150`, without #5502), 16 orgs, 1 s target, 5,000 TPS:** - Overlay traffic per node fell from 14.0 to 4.7 MB/s each way (-66%) in every run. - Whether the block time held depended on validation CPU: FAIL with the default 3 validation tasks on 4-core c6id.2xlarge (avg p75 1,367 ms; base 1,081 ms), PASS with 7 tasks on the same instances (1,030 ms, every check including an exact inclusion count), all windows on target on 16-core instances (1,019 to 1,021 ms; inclusion +250). - At 30 orgs the base failed 1 s / 5,000 TPS (1,298 ms); this change was not run there. ## What changes - **One function decides who gets transactions**: `HerderImpl::computeTxFloodTargets()`. With the leader schedule: the leaders of the node's current round and of the next one for the slot after the LCL (two nodes). With #5502's leader terms the node that closes a ledger leads the next one too; one slot before a new term starts (every fifth ledger), that term's expected first leader is a third target, so that it holds the transactions when it pre-validates and builds its first set. It is the first node of the order seeded with the LCL's hash, so it is known a full ledger ahead. "Everyone" (the old flood) when the node is not tracking or more than one ledger behind, when there is no schedule, and for a pre-protocol-20 LCL. It is re-evaluated at a ledger close, at a round change, at an externalize and at loss of sync, and sent to the overlay when it changes (IPC `SET_TX_TARGETS`). A changed leader schedule only has to change this function. - **The accepting node keeps the transaction** (overlay `flood/tx_router.rs`, at most 100,000 transactions and 300 s) until it is externalized, removed by Core or expires. When the targets change, it goes to a newly added target only if it may still be wanted there: it is younger than 200 ms, or three ledgers externalized on this node without it since it was accepted, or it is older than 3 s (no ledger closes). Anything else is most likely in the set in consensus. Nothing is sent again to a peer that got it less than 60 s ago on the same connection; a failed send or a reconnected peer is served regardless. A node that received a transaction as a target keeps it in its mempool and does not relay it. - **Fallback to the flood.** If a target is not a directly connected, identified peer, the node floods its transactions with adverts as before, so they reach every mempool whatever the topology. A node that got a transaction directly joins a later flood of it and can serve the body. - **Admission check** (`HerderImpl::checkTxAdmission`, in `recvTransaction`): `checkValid` against the LCL, the check the leader's build runs on every candidate, and one pending transaction per source account per accepting node (`TRY_AGAIN_LATER`; `DUPLICATE` for the same hash). On the base `recvTransaction` checked nothing. The per-account record goes when a set that uses up the sequence number closes, when the LCL shows the sequence number used some other way, or after 300 s. - **Overlays prove which validator they belong to.** The overlay's libp2p identity is a random key. At startup it sends its peer id to Core; Core signs `SHA256(networkID || "stellar-overlay-peer-id" || peerId)` with the node key; the overlay shows `[nodeID][signature]` to each peer on a new stream (`/stellar/node-id/1.0.0`, acknowledged, retried), and the receiver checks it against the peer id QUIC authenticated. The node key stays in Core. - **No leader prewarm of its own any more.** The version on the old base verified arriving signatures between the close and the trigger. #5502's leader builds once before the trigger and pre-validates during the previous ballot; bodies that arrive after those pulls are verified in the build, as on #5502 alone. - **Options, for measurements:** `TX_FLOOD_TARGET_LEADERS` (default 2; 0 = every validator of the schedule, still direct and without adverts), `TX_FLOOD_LEGACY_ADVERTS` (the base's flood from the same binary), `BATCH_EXECUTOR_TASKS` (number of validation tasks; above the physical core count they run on hyper-thread siblings). - New metrics: `overlay.flood.tx-direct-sent`, `.tx-direct-sent-bytes`, `.tx-direct-recv`, `.tx-local-flooded`, `.tx-late-copy` (a transaction admitted although one of the last few externalized ledgers already has it), `overlay.node-id.identified`, `.rejected`, `herder.tx-admission.check`, `.invalid`, `.account-busy`. ## Why ledger outputs cannot change, and what does change - The change decides which transactions reach a leader's mempool, when, and which submissions are refused. Building a set from a mempool, validating it, voting and applying are untouched: a given tx set produces the same ledgers and meta. - The admission check reads the LCL on the main thread and writes nothing. - The identity binding is domain-separated from everything else the node key signs. - **What gets worse, by design:** - Followers no longer hold transactions before the set arrives, so the background signature verification finds nothing and the apply-time check verifies signatures inline (see the results). - A term's first leader is predicted one ledger ahead. If the ledger in between costs that node its turn (#5502's forfeit), another node leads and gets the transactions only at the close. #5502 also lets the second node of the expected schedule pre-validate; it gets the transactions when the ledger before closes, not earlier. - A predicted target that is not a connected, identified peer makes the node flood for the whole ledger before the term. - A restarted node forgets the transactions it accepted and had not seen included; on the base every node held them. - A transaction submitted through a node that cannot reach both targets directly costs what it costs today. - **Left open:** - A peer can still fill a target's mempool with invalid transactions that outbid valid ones (the base's exposure, minus the relay). The leader validates when it builds and drops them then. - A refused submission returns `{"status":"ERROR"}` without the result code. - A pending transaction that dies for a reason other than its sequence number (expired time bound) holds its account on that node for up to 300 s. - Fee bumps are accepted and then dropped by the overlay, as on the base. - No protocol gating and no version negotiation: a node of the base ignores the identity stream, and nodes with this change flood toward it. ## Perfnet ### On top of #5502: 16 orgs, 800 ms, 8,000 TPS, c6id.8xlarge `block_time_check` (perfnet `btc2-slop`), `--org-count 16 --block-time-ms 800 --soroban-tps 8000 --soroban-type sac --classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900 --instance-type c6id.8xlarge`, 48 validators in four regions, no log saving. The #5502 column is that PR's own run of the same day at the same fixture. The stack column is build #3685 of `6b3c7ae0e`, which is this PR's head **without** the changed re-send rule, without the late-copy meter and without #5502's last commit. | Windows 1 / 2 / 3 / 4 | #5502 alone (build #3684) | This PR on #5502 (build #3685) | |---|---:|---:| | Windows on target | 4 of 4 | **4 of 4** | | Avg p75 block time, ms | 806.9 / 805.6 / 805.3 / 805.3 | **804.9 / 804.1 / 803.8 / 803.9** | | Avg p99, ms (limit 1,600) | 1,153 / 1,188 / 1,100 / 1,249 | 1,170 / 1,209 / 1,039 / 1,172 | | Txs per ledger; sets at the 8,000 cap | 6,450; 3.8% | 6,440; 3.4% | | **`overlay.byte.read` per node** | **21.2 MB/s** | **6.2 MB/s (-71%)** | | `overlay.message.read` per node | 20,500 /s | 12,550 /s | | Adverts per node | 7,848 /s | 0 | | Direct sends per offered transaction | none | 3.3 | | Leader build `herder.txset.build`; builds per ledger | 0.16 s; 1 | 0.15 s; 1 | | Trigger to first ballot | 0.108 to 0.115 s | 0.100 to 0.118 s | | Ballot to externalize | 0.29 s | 0.29 s | | Externalize to close (core logs, mean) | 0.045 to 0.048 s | 0.041 s | | **Trigger to close** | 0.44 to 0.45 s | **0.43 to 0.44 s** | | Apply-time check `herder.txset.validate` | 0.080 s | 0.099 s | | Signature-cache misses in that check, per node | (followers hold the transactions) | 7,490 /s (44% of the lookups) | | Validation-cache hit share in the leader's build | 69% | 66.7% | | Leader terms | exactly 5 ledgers | exactly 5 ledgers, 225 terms, 48 leaders | | Leader and prepare timeouts, empty sets | 0 | 0 | | Inclusion, over the 7,680,000 offered | +80 | **+6,610** | | Error counters, in sync, no fork, still closing | passed | passed | | Verdict | FAIL on inclusion only | FAIL on inclusion only | - The base `overlay-v2-shared` at this fixture on the same instances (reported in #5502): window 1 on target at 846 ms, window 2 missed, trigger to close 0.67 to 0.69 s. - **Inclusion.** The check requires exactly the offered count. #5502 alone held 80 more, this run 6,610 more (0.086%): copies of transactions that reach a node after their ledger closed and are included again, which only overlay-only mode permits. This run's build re-sent every pending transaction to newly added targets; the head sends only fresh ones or ones that missed three ledgers, and the goal-fixture run with it had 81 on 3,870,249 (0.002%) and counted 688 late copies (`overlay.flood.tx-late-copy`). The 16-org fixture was not rerun with the head. - Every node identified all 47 peers, nothing was flooded, no send failed, no submission was refused; the admission check takes 0.094 ms per transaction on the main thread. ### On top of #5502: the goal fixture, 30 orgs, 800 ms, 10,000 TPS, c6id.8xlarge The same command with `--org-count 30 --soroban-tps 10000`; 90 validators, 10,080 TPS offered. The stack column is build #3688 of this PR's head `6ab837c27`. The #5502 column is that PR's own run of its head `8aa7f6bf0` two hours earlier (some rows are from its first run at this fixture, marked G). | Window 1 (the load was stopped after it in both runs) | #5502 alone (build #3686) | This PR on #5502 (build #3688) | |---|---:|---:| | Avg p75 block time, ms (limit 960) | 1,031.6 | **979.6** | | Avg p99, ms (limit 1,600) | 1,887 | **1,441** | | Time between ledgers | 1.02 s | 0.90 s (median 0.88 s) | | The first minute of the load | 802 ms for 30 s, backlog from 47 s | 805 ms over 60 s | | Sets at the 10,000 cap | 93% | 38.7% | | Leader builds before the trigger | 0 of 40 | 40 of 55 | | Leader build; candidates pulled | 0.33 s; 20,200 | 0.24 s; median 8,350 (max 20,200) | | Trigger to first ballot | 0.58 s | **0.39 s** | | Ballot to externalize | 0.32 s | 0.32 s | | Externalize to close | 0.07 s | 0.065 s | | **Trigger to close** | 0.97 s | **0.77 s** | | Set at the followers after the proposal | +230 ms (G) | +174 ms | | Apply-time check `herder.txset.validate` | (see #5502) | 0.144 s | | Validation-cache hit share in the leader's build | 66.3% (G) | 66.7% | | **`overlay.byte.read` per node** | **43 MB/s** (G) | **15.7 MB/s** | | Direct sends per submitted transaction | none | 2.5 to 2.6 while keeping up, 3.7 in the backlog | | Leader terms; timeouts | 5 ledgers; none | 5 ledgers; none | | Submitted / included at the end | 49,855 short | 81 over (backlog drained) | | `overlay.flood.tx-late-copy` | n/a | 688, on 14 of 90 nodes | | Error counters, in sync, no fork, still closing | passed | passed | | Verdict | FAIL, window 1 (p75 and p99) | FAIL, window 1 (p75 only) | **Not measured:** the 16-org fixture again with the head's re-send rule; the base `overlay-v2-shared` at the goal fixture; the 4-core instances with #5502 underneath; `TX_FLOOD_TARGET_LEADERS=0`; real apply; more than one run of anything. ### Before the rebase: 16 orgs, 1 s, 5,000 TPS (on `overlay-v2-shared` at `26bea1150`, without #5502) All runs: `block_time_check` (perfnet `btc2-slop @ 6dd2064`), `--block-time-ms 1000 --soroban-tps 5000 --soroban-type sac --classic-tps 0 --simulate-apply-budget-ms 150 --load-duration-s 900`, validators in us-east-1, us-west-2, eu-central-1 and ap-southeast-1, every validator generates load, overlay-only mode (Core does not apply; execution is a sleep). 5-minute means at the end of window 1, median node. | | Base | This PR, 3 tasks | This PR, 7 tasks | This PR, 16 cores | Base, 30 orgs | |---|---:|---:|---:|---:|---:| | Build (Jenkins; commit) | #3678; `0c9a02e56` (tree of `26bea1150`) | #3680; `dc91c8964` | #3683; `664a35fdc` | #3680; `dc91c8964` | #3678 | | Validators x instance type | 48 x c6id.2xlarge | 48 x c6id.2xlarge | 48 x c6id.2xlarge | 48 x c6id.8xlarge | 90 x c6id.2xlarge | | Validation tasks | 3 | 3 | 7 (`BATCH_EXECUTOR_TASKS=7`) | 15 | 3 | | Verdict | FAIL on inclusion only (+60) | **FAIL**, window 1 | **PASS** | FAIL on inclusion only (+250) | **FAIL**, window 1 | | Windows on target | 4 of 4 | 0 of 1 (load stopped) | 4 of 4 | 4 of 4 | 0 of 1 (load stopped) | | Avg p75 per window, ms | 1,081 / 1,132 / 1,118 / 1,142 | 1,367 | 1,030 / 1,030 / 1,031 / 1,031 | 1,019 / 1,021 / 1,020 / 1,021 | 1,298 | | `overlay.byte.read` per node | 13.97 MB/s | 4.77 MB/s | 4.71 MB/s | 4.77 MB/s | 26.69 MB/s | | `overlay.message.read` per node | 14,804 /s | 9,374 /s | 9,914 /s | 10,085 /s | 33,523 /s | | Direct sends per offered transaction | none | 9.0 | 3.4 | 3.1 | none | | Leader build `herder.txset.build` | 0.235 s | 0.374 s | 0.099 s | 0.063 s | 0.314 s | | Trigger to first ballot | 0.413 s | 0.539 s | 0.300 s | 0.232 s | 0.613 s | | Ballot to externalize | 0.296 s | 0.290 s | 0.293 s | 0.284 s | 0.387 s | | Apply-time check `herder.txset.validate` | 0.179 s | 0.277 s | 0.168 s | 0.078 s | 0.247 s | | Signature-cache misses in that check | 2,089 /s | 4,550 /s | 4,654 /s | 4,647 /s | 2,841 /s | | Externalize to close (core logs, median) | 58 ms | 145 to 210 ms | 44 ms | 28 ms | 61 to 102 ms | | Ledgers per 5 minutes; txs per ledger | 291; 5,152 | 266; 5,637 | 300; 5,001 | 300; 5,000 | 262; 5,742 | | Error counters, in sync, no fork, still closing | passed | passed | passed | passed | passed | - The base column is run 3 of #5498's record (same fixture, same box, the evening before); it was not rerun. - **3 tasks:** 75% of the sets were at the 6,250 cap (1.27 s per ledger) and 16% were starved (as few as 938 transactions): after a slow ledger the next trigger is already due and the new leader builds before the pending transactions reach it. - **7 tasks and 16 cores:** no starved sets, no backlog; the leader pulls about 5,000 candidates at its trigger. - Every run of this PR: every node identified all 47 peers as validators (0 rejected claims), nothing was flooded, no message was dropped, no submission was refused; the admission check takes about 0.12 ms per transaction on the main thread. - **Inclusion.** The check requires every node to hold exactly the offered transactions. Base: +60; 16-core run: +250; 7-task run: exact. In overlay-only mode Core's sequence-number check is off, so a copy of a finalized transaction that a node admits again is included when that node next leads (performance-testing#4, not merged, would allow up to 480 here). Which transactions were duplicated in the 16-core run, and why more than on the base, is **not established**. **Not measured before the rebase:** this change at 30 orgs; the base with 7 tasks; real apply (with real execution the check and the execution compete for the same cores). ## Testing On `6ab837c27398` (this PR's head, on #5502 at `8aa7f6bf0`), devbox, 32 partitions: - **Full suite: PASS** (exit 0, 718 s), with libsodium, xdrpp and the overlay's Rust tests (224 library, 47 binary, 7 end-to-end; 4 ignored). - **Full suite `--all-versions`: PASS** (exit 0, 735 s). - Tag run (`[herder]~[.],[scp]~[.],[txset]~[.],[overlay-ipc],[overlay]~[.],[ipc]~[.],[upgrades]~[.],[simulation]~[.],[loadgen]~[.]`, which includes #5502's tests): only the three failures recorded for the base ("SCP Driver" validity-cache hits, "hierarchical topology scales 1 to 3", "core nodes with outer nodes"). - **One unexplained Rust failure:** on an earlier head (`6b3c7ae0e`) one `make check-rust` run failed 1 of 219 library tests and the output was not kept, so the test is not known. About twenty later runs of the same suite passed (apart from failures of tests I was changing at the time). #5502 does not touch `overlay/`. New tests: - `herder/test/TxFloodTargetsTests.cpp` (6 cases): - In a 4-validator mesh every submitted transaction is applied on every node with no advert and no demand, and every target list each node sent to its overlay equals what the schedule functions give for the LCL and round it was sent at: two round leaders, and a third node only in the ledger before a new term. A list sent one ledger before a term start names the node that then leads the term, unless the ledger in between forfeited. **Mutation-checked** three ways: the router forced to flood (96 adverts for 32 transactions), the term lookahead removed, and the lookahead seeded with the wrong hash; each fails the test. - With `TX_FLOOD_TARGET_LEADERS = 0` each transaction goes to the three other nodes exactly once. - With one of three validators down every transaction is still included and every target list matches the schedule at the node's round, including rounds after a timeout. - In overlay-only mode with load from all four nodes, no transaction hash appears in two ledgers (2,400 transactions; it does not reproduce the perfnet duplicates, which need network latency). - The admission check (accepted once per account, duplicate, bad signature, used and future sequence number, low fee, missing account, freed by the including ledger, by another transaction of the account, by a sequence bump, by timeout; overlay-only mode; pregenerated load); the experiment options. - Rust `flood/tx_router.rs` (20 unit tests): targets only and following the targets; a kept transaction follows a target change only while fresh, after three ledgers without it, or after 3 s (**mutation-checked**: with the rule disabled five tests fail); a removal by Core is not a ledger; flood without reachable targets; a directly received transaction joins a later flood and serves the body. - Rust `libp2p_overlay/tx_direct_tests.rs` (4 network tests on loopback, 1 unit test): the same over real connections, including a late copy being counted; forged, replayed and wrong-network identity claims are rejected, and claims are bounded per connection. `main.rs` (3): the IPC handlers. Existing tests changed: - `overlay/.../dispatch_tests.rs` (2): they fed a body without a demand and expected a relay; they now register the demand, and one asserts that an undemanded body is not relayed. - `main/test/CommandHandlerTests.cpp` "transaction envelope bridge": V1 and fee-bump envelopes are refused again below protocol 13, and the too-early / too-late cases are back. Review: four passes of an independent review (two on the old base, one of the rebase, one of the re-send rule); no consensus or determinism finding in any. Open from the last two: the flush after an externalize plans against the outgoing targets (one stray body for transactions that waited three ledgers); a lost send re-delivers to every held-back target, not only the lost peer; after a leader failure the round-2 leader lacks transactions older than 200 ms until the ledger closes; the 200 ms bound is a constant, not derived from the block time; no test covers a watcher, a wrong term prediction, a lagging sender or the `main.rs` wiring of the ledger count. Not passing or not run: - Tests outside the regular suite that fail the same way on the base: "resilience tests", "cycle4 topology", "asymmetric topology report cost", and the three above. - A `--disable-tests` build was not tried; #5495 records that the base does not build that way. - No sanitizer run. No tx-meta baseline run (the diff does not touch apply). ## Commits Three commits on top of #5502: the code, and two for the README (which now describes and reports the whole stack: #5502's leader terms and pre-validation, and this PR). Builds that perfnet ran: #3688 of the head `6ab837c27` (the goal-fixture run), #3685 of `6b3c7ae0e` (the 16-org run; that commit is no longer on the branch) and, before the rebase, #3680 of `dc91c8964` and #3683 of `664a35fdc`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, not for merge. Stacked on #5497 (push the leader's tx set). This PR adds two commits to #5497's four:
a33da62c4, the change, and0c9a02e56, a fix for a bug in the Core-overlay IPC reader that the perfnet run below hit (it is in the stack below this PR).From protocol 30, a validator votes for the round leader's value without validating its transaction set. The validity check moves to ledger apply, where an invalid set closes an empty ledger, and the ledger is executed speculatively while consensus is still in flight. The transaction set is made durable before the vote by the overlay, in its compressed form. There is no compatibility work beyond the protocol gate; the PR exists to measure what validation, persistence and apply cost on the leader-schedule stack.
Result in one paragraph
At the new target fixture (16 tier-1 orgs = 48 validators in 4 regions, c6id.2xlarge, 1 s target, 5,000 SAC TPS, one run per build) #5497 alone misses: 1.42 s per ledger, avg p75 1,461 ms, every tx set at the cap. With this PR on top, all four windows are on target on all 48 nodes: avg p75 1,094 to 1,137 ms, 1.04 to 1.07 s per ledger, and the time from trigger to ledger close falls from 1.37 s to about 0.8 s. Every ledger was executed speculatively and committed as executed (hit rate 100%, no execution dropped, no set skipped). The run's verdict is still FAIL: one node dropped out of consensus 15 s before the load ended, because Core's IPC reader thread took an interrupted
recvfor a closed connection. That bug is insrc/overlay/IPC.cpp, below this PR; the second commit fixes it, and a rerun with the fix is in progress. The comparison is between two regimes (over capacity against keeping up), and the speculation gain on this fixture is an upper bound (see Perfnet).What changes (all gated on protocol 30; protocol 29 and older are unchanged)
HerderSCPDriver::validateValueAgainstLocalStatereturnskFullyValidatedValuefor a value whose tx set the node holds, without running the set's validity check. Still checked before the vote: the StellarValue itself (signature, close time, upgrades), that the node holds the set, and one O(1) property of the set, that it was built on the LCL.LedgerManagerImpl::validateTxSetForApplyrunsprepareForApplyandcheckValidagainst the previous ledger with the value's close time. If either fails, nothing of the set is applied: no fee, no sequence bump, no result.txSetResultHashis the hash of an empty result set.applyLedgeris split intoexecuteLedger(validation, fees, transaction execution; everything it does lives in an object that rolls back when dropped) andcommitLedger(history, upgrades, seal into the BucketList and database, meta, LCL). When a validator emits a PREPARE or CONFIRM for a value for LCL+1, the ledger apply thread executes that ledger. When a value externalizes, the executed ledger is committed if it is for the same ledger, value and set; otherwise it is dropped and the ledger is applied from scratch.EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY(default true) turns it off.<BUCKET_DIR_PATH>/txsets(temp file,fdatasync, rename) before it delivers a set to Core, and for a set Core built before it handles Core's next IPC message (the push and the proposal). Core no longer serializes, base64-encodes and commits those sets to SQLite in front of its first vote; after a restart it requests the sets its restored SCP state names, and the overlay reads them back from disk. Only sets Core holds are stored, and they are deleted by slot age, so peers filling the overlay's cache cannot evict them. If a set Core built cannot be stored, the overlay exits instead of sending anything that refers to it.TxSetValidationContext(previous header, Soroban config, a factory for read-only ledger views), so it can run on the apply thread against apply's own snapshot.ledger.apply.{txset-skipped, execute, commit, speculation-started, speculation-hit, speculation-miss, speculation-idle},herder.scp.persist.herder.txset.validatenow times the apply-time check.Why it should be safe, and where it is not
checkValidused to cost liveness at most. It now forks the ledger. Its node-local inputs are the binary's protocol version, the network passphrase and, in test builds, loadgen flags (ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING,LOADGEN_BYTE_COUNT_*): validators that differ in those would disagree on validity.transactionsentry without aresultsentry.Testing
On
a33da62c4(the change), devbox, 32 partitions:--all-versions: PASS. 614 test cases, 8,126,148 assertions.On the same change before the rebase (
25f48abb6, tree of3b8a8356c):[tx], all versions, seed 12345): unchanged, 144 test cases.--capture-lcm): no file changed.New tests:
herder/test/ApplyTimeTxSetValidationTests.cpp(6 cases): an invalid or malformed set closes an empty ledger and the next ledger applies normally; upgrades of a skipped value apply; the same set applies or is skipped depending on the value's close time; protocol 29 applies the set as before;validateValuelevels on both sides of the gate; a 4-node network externalizes a leader's invalid sets and every node closes them empty with the same hash; a node restarts and gets its tx set back from the overlay's store, with no tx-set row in SQLite.ledger/test/SpeculativeApplyTests.cpp(5 cases): an executed ledger is committed when its value externalizes; it is dropped for another set, another close time, a newer speculation, an explicit abort, and at shutdown; an invalid set under speculation; identical ledger hashes with and without speculation, including upgrades and skipped sets; a 4-node network where every node commits speculatively executed ledgers.history/test/HistoryTests.cpp(1 case): a node replays skipped ledgers from an archive and closes the same hashes. It fails if history does not keep the skipped set.flood::txset_store(4) and App tests (5): on disk before Core gets the set; an unstorable set is not delivered, and stops the overlay if Core built it; served to Core after a restart; deleted by slot age; peers cannot evict a stored set.Existing tests changed: four "SCP Driver" assertions and the signature-prewarm test that encoded validation before voting; "network does not externalize non-canonical empty tx set" is now "network externalizes non-canonical empty tx set" (it externalizes and closes an empty ledger).
On
0c9a02e56(the IPC fix on top, this PR's head): full suite PASS, 615 test cases, 2,385,530 assertions. The new test "IPC read survives a signal" fails without the fix.--all-versionson this head is running.Not passing or not run:
overlay-v2-shared87201aa66), "hierarchical topology scales 1 to 3" (as recorded in Prototype: replace SCP nomination with a deterministic leader schedule #5495).--disable-testsbuild was not tried; Prototype: replace SCP nomination with a deterministic leader schedule #5495 records that the base does not build that way.Perfnet
16 orgs (48 validators), 4 regions, c6id.2xlarge, 1,000 ms target, 5,000 SAC TPS, 150 ms simulated apply per ledger (30 us per op), tx-set cap 6,250, 900 s measured, SCP timeouts 2 s. Same day, back to back, n = 1 per build. This fixture has no earlier comparison point.
8f7768be7)a33da62c4)herder.txset.validate(mean)ledger.apply.execute/ledger.apply.commit(mean)herder.scp.persist(mean; p99), in front of every SCP broadcastReading it:
scp.fetch.txset0.052 s against 0.033 s); no retry was needed.The failure. node-45 stopped following consensus at ledger 1024, 15 s before the generators finished; node-23 and node-32 did the same two minutes after the load. On each, Core logged
Overlay IPC connection closeda few milliseconds into a checkpoint publish while both processes stayed up and the socket stayed open, with the overlay's send queue full.readExactinsrc/overlay/IPC.cppreturns "closed" for anyrecvresult <= 0,EINTRincluded, and stellar-core's signal handlers do not restart system calls; the inference (no errno was logged) is a SIGCHLD from the publish's child processes delivered to the reader thread. The base run had no such event in 30 minutes, this run three in 40: one run each cannot say whether this PR makes it more likely. None of this PR's own error paths fired on any node. Commit0c9a02e56retries onEINTRand adds a test that interrupts a blocked reader.A rerun of this PR's head with that fix (build #3678) is in progress; the table gets a third column from it.
Review findings not fixed
An independent review of the pre-rebase change found no case where the speculative path commits a different ledger than the regular one, or where live apply and replay decide differently. Not fixed:
CacheTxSetthat the overlay drops for a hash mismatch or an encode error (both Core bugs) leaves Core believing the set is stored.ledger.ledger.closeleaves out the time a finished execution waits and executions that were dropped; in test buildsgetLastClosedLedgerTxMeta()can show a ledger that is still speculative.Next steps
Not started. The commit and the next leader's build still wait for externalize. Letting the next leader build on the speculatively executed state is the next 0.2 s or so, and needs either every node pre-building or a schedule seed that is known one ledger earlier.