From 39e30bd2f57bf625d0dc2ed88527d62faa72c6c8 Mon Sep 17 00:00:00 2001 From: Garand Tyson Date: Wed, 11 Nov 2026 12:51:49 -0800 Subject: [PATCH 1/5] README: benchmark at 48 validators, 5,000 TPS, 1 s; results of apply-time tx-set validation The benchmark fixture is now 16 organizations (48 validators) in 4 regions at 5,000 SAC TPS and a 1,000 ms target. Report the two runs at that fixture: the leader tx-set push alone (#5497) misses the first window (p75 1,461 ms), apply-time validation with speculative execution (#5498) is on target in all four windows (p75 1,094-1,137 ms), with a FAIL verdict for a node that lost its IPC reader (fixed in 0c9a02e56, not rerun). Describe the third experiment, update the measurement setup, limits and known limitations, and condense the 30-validator results. --- README.md | 425 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 254 insertions(+), 171 deletions(-) diff --git a/README.md b/README.md index 9a5a5b1a5..1b9c83584 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,23 @@ -# stellar-core: leader schedule and leader tx-set push (experimental branch) +# stellar-core: leader schedule, leader tx-set push and apply-time tx-set validation (experimental branch) **This branch is an experiment. It is not production code and must not run on a real network.** -It has no feature flag, no protocol gate, and it is not compatible with nodes that nominate. +The leader schedule and the tx-set push have no feature flag and no protocol gate, and the branch +is not compatible with nodes that nominate. -The latest experiment (the leader pushes its tx set) was measured on commit `8f7768be7` on -2026-11-11. "Before" is the previous experiment on this branch (the leader schedule alone), commit -`daa965f4c`, measured on 2026-11-10 on the same fixture. One run per setting on each side. +The benchmark is now **48 validators (16 organizations), 5,000 TPS, 1,000 ms target block time**. +The latest experiment (vote without validating the tx set, validate and execute at apply) was +measured there on commit `a33da62c4` on 2026-11-11. "Before" is the previous experiment on this +branch (the leader's tx-set push), commit `8f7768be7`, measured on the same day on the same +fixture. One run per build. ## What this branch is It is the overlay-v2 stack up to [#5459](https://github.com/stellar/stellar-core/pull/5459) (Rust overlay over QUIC, millisecond close times, compressed tx-set responses, leader-only tx-set -construction) plus two experiments: +construction) plus three experiments: -1. **No nomination** ([#5495](https://github.com/stellar/stellar-core/pull/5495), the "before" - below). The SCP nomination phase is deleted and replaced by a leader schedule that every node - computes locally. +1. **No nomination** ([#5495](https://github.com/stellar/stellar-core/pull/5495)). The SCP + nomination phase is deleted and replaced by a leader schedule that every node computes locally. - **Schedule.** For slot N, the nodes of the transitive quorum are sorted by `SHA256(hash of ledger header N-1 || node id)`. The leader of round r is entry `(r-1) mod n`. - **Rounds.** A round is a ballot counter. The round's leader builds its value and goes straight @@ -23,11 +25,10 @@ construction) plus two experiments: passes proposal validation. A timeout (the ballot timeout settings) hands the slot to the next leader. 2. **The leader pushes its tx set** ([#5497](https://github.com/stellar/stellar-core/pull/5497), - the latest experiment). Before, a follower saw the leader's proposal, asked for the tx set and - received it about two round trips later. Now: + the "before" below). Before it, a follower saw the leader's proposal, asked for the tx set and + received it about two round trips later. With it: - **Push.** The round's leader sends the set it is about to propose to every connected peer as - soon as the set is built, without waiting for a request. The proposal follows about 70 ms - later. + soon as the set is built, without waiting for a request. - **Use.** A follower's overlay verifies a pushed set and keeps it under the hash of its own content. Core only gets it when an SCP statement names that hash, so a pushed set that does not match the proposal is never used. @@ -35,148 +36,197 @@ construction) plus two experiments: - **One flight.** Every QUIC connection starts with an 8 MiB congestion window, so that a set leaves in one flight instead of being paced over a round trip. - Details and limits: [docs/rust-overlay/txset-fetching.md](docs/rust-overlay/txset-fetching.md). - -Unchanged by both: the ballot protocol's accept, confirm and commit rules, value validation, apply, -ledger state and meta. +3. **Vote without validating the tx set; validate and execute at apply** + ([#5498](https://github.com/stellar/stellar-core/pull/5498), the latest experiment). Gated on + protocol 30; protocol 29 and older behave as before. + - **Vote.** A node votes for the leader's value as soon as it holds the tx set, without running + the set's validity check. Still checked before the vote: the value itself (signature, close + time, upgrades) and that the set was built on the last closed ledger. + - **Validate at apply.** The validity check runs when the ledger is applied. An invalid set + closes an **empty ledger**: no fee, no sequence bump, no result. The header keeps the + externalized value, upgrades in the value still apply, and history keeps the set so that a + replaying node reaches the same verdict. + - **Speculative execution.** When a node emits a PREPARE or CONFIRM for a value for the next + ledger, its apply thread validates and executes that ledger while the ballot is still running. + When the value externalizes, the executed ledger is committed; if another value externalizes, + it is dropped and the ledger is applied from scratch. `EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` + (default true) turns it off. + - **Durable tx set, written by the overlay.** The Rust overlay writes the compressed (zstd) set + to `/txsets` before it hands a set to Core, and before it sends anything that + refers to a set Core built. Core no longer writes the set to SQLite in front of its first + vote, and gets its sets back from the overlay after a restart. + - Details, safety argument and review findings: + [#5498](https://github.com/stellar/stellar-core/pull/5498). + +Unchanged by all three: the ballot protocol's accept, confirm and commit rules. For a valid tx +set, ledger state and meta are also unchanged; experiment 3 changes when a set is validated and +executed, and what an invalid set does to the ledger. ## Performance results -Target block time 1,000 ms, 3,000 TPS, 30 validators in 4 regions. +Target block time 1,000 ms, 5,000 TPS, 48 validators in 4 regions. -| | Before (`daa965f4c`) | With the tx-set push (`8f7768be7`) | +| | Before: tx-set push (`8f7768be7`) | With apply-time validation (`a33da62c4`) | |---|---|---| -| Block time windows | **missed** the first window; the load stops there | **on target in all four windows**, on all 30 nodes | -| p75, average over 30 nodes | 1,386 ms (limit 1,200) | 1,091 / 1,096 / 1,109 / 1,115 ms | -| p99, average over 30 nodes | 1,599 ms (limit 2,000) | 1,227 / 1,243 / 1,253 / 1,290 ms | -| Time per ledger | 1.33 s | 1.05-1.07 s | -| Tx-set fetch as seen by a follower's Core | 217 ms | 16 ms | - -In short: before, the stages of one ledger added up to more than a second and the network backed -up. With the push the tx set is already at the follower when the proposal arrives, a ledger's -stages fit into about 0.9 s, and the network keeps up with the load at a 1 s target. - -**One caveat on correctness checks.** Every node's ledgers held 60 transactions more than the -2,880,000 offered: about 60 transactions were included in two ledgers. This is an artifact of the -test mode, in which ledgers are not applied (see [The inclusion check](#the-inclusion-check)). The -run was judged with the exact check and reported FAIL for that check alone; the check has since -been relaxed to allow such duplicates -([performance-testing#4](https://github.com/stellar-experimental/performance-testing/pull/4)). -All other checks passed. - -### 1,000 ms target, with the tx-set push +| Block time windows | **missed** the first window on all 48 nodes; the load stops there | **on target in all four windows**, on all 48 nodes | +| p75, average over 48 nodes | 1,461 ms (limit 1,200) | 1,094 / 1,116 / 1,130 / 1,137 ms | +| p99, average over 48 nodes | 1,638 ms (limit 2,000) | 1,313 / 1,453 / 1,408 / 1,435 ms | +| Time per ledger | 1.42 s | 1.04-1.07 s | +| Trigger to ledger closed (sum of the stages) | 1.37 s | 0.79-0.84 s | +| Run verdict | FAIL (block time) | FAIL (one node dropped out of consensus, see the caveat) | + +In short: before, the stages of one ledger added up to 1.37 s, every tx set was full and the +network backed up. With validation moved out of the vote and the ledger executed while the ballot +runs, a ledger's stages fit into about 0.8 s and the network keeps up with 5,000 TPS at a 1 s +target. + +**Caveat: the run with the change is a FAIL, although every block time window was on target.** +15 s before the load ended, one node (of 48) stopped reading from its overlay process and fell out +of consensus; two more did the same two minutes after the load. The cause is a bug in the +Core-overlay IPC socket code that this experiment did not introduce (see +[The node that dropped out](#the-node-that-dropped-out)). The fix is on this branch (`0c9a02e56`); +it has not been measured on a network yet. Because the load counted as ended early, the inclusion +check was not judged. + +### 1,000 ms target, with apply-time validation Load ran its full 900 s. Every node was on target in every window. | Window | p75, average (range over nodes) | p99, average (range over nodes) | |---|---|---| -| 1 | 1,091 ms (1,085-1,096) | 1,227 ms (1,205-1,273) | -| 2 | 1,096 ms (1,090-1,105) | 1,243 ms (1,227-1,272) | -| 3 | 1,109 ms (1,104-1,116) | 1,253 ms (1,231-1,360) | -| 4 (2 minutes) | 1,115 ms (1,109-1,125) | 1,290 ms (1,245-1,396) | +| 1 | 1,094 ms (1,083-1,110) | 1,313 ms (1,260-1,394) | +| 2 | 1,116 ms (1,107-1,126) | 1,453 ms (1,393-1,522) | +| 3 | 1,130 ms (1,116-1,145) | 1,408 ms (1,357-1,499) | +| 4 (2.4 minutes) | 1,137 ms (1,121-1,160) | 1,435 ms (1,381-1,486) | Limits: p75 below 1,200 ms, p99 at most 2,000 ms. Checks: -- **No fork**, all 30 nodes in sync, still closing ledgers after the load: passed. -- **Error counters zero** (invalid signatures, invariant failures, publish failures, invalid values, - lost sync), no node restarts: passed. -- **No wasted rounds:** every ledger was decided by the first leader. 0 leader timeouts, 0 ballot - timeouts, 0 empty transaction sets. -- **Inclusion:** 2,880,060 transactions in every node's ledgers against 2,880,000 offered. See - below. +- **No fork**, still closing ledgers after the load: passed. +- **Error counters zero** (invalid signatures, invariant failures, publish failures, invalid + values), no node restarts: passed. +- **All nodes in sync: failed.** One node was out of sync at the end of the load. +- **No wasted rounds during the load:** every ledger was decided by the first leader. 0 leader + timeouts, 0 ballot timeouts, 0 empty transaction sets. No tx set was found invalid at apply. +- **Inclusion: not judged.** Every one of the 47 healthy nodes holds 4,798,690 transactions + against 4,800,000 offered, 1,310 fewer (0.027%). The node that dropped out kept generating for + 15 s (about 104 transactions per second) while nothing left it, which accounts for a shortfall + of that size. That these are the missing ones is inferred, not shown. ### Before and after, stage by stage -5-minute means at the end of the first window, median over the 30 nodes. +5-minute means at the end of each run's first window, median over the 48 nodes. -| | Before (`daa965f4c`) | With the tx-set push (`8f7768be7`) | +| | Before (`8f7768be7`) | With apply-time validation (`a33da62c4`) | |---|---:|---:| -| Ledgers per 5 minutes | 225 | 285 | -| Transactions per ledger | 3,750, the cap, on every ledger | 3,155 (what was offered; below the cap) | -| Leader tx-set build | 0.25 s | 0.15 s | -| Tx-set fetch (follower's Core, request to delivery) | 0.217 s | **0.016 s** | -| Tx sets a follower had to wait for from the network | every one | none in this window | -| Tx-set validation (follower) | 0.12 s | 0.11 s | -| Trigger to first ballot (includes the build, the fetch and validation) | 0.70 s | **0.42 s** | -| First ballot to externalize | 0.41 s | 0.33 s | -| Ledger close (simulated apply, 50 us per operation) | 0.20 s | 0.16 s | -| **Sum: trigger to close** | **1.30 s** | **0.91 s** | +| Ledgers per 5 minutes | 211-212 | 288-290 | +| Transactions per ledger | 6,250, the cap, on every ledger | 5,189 (what was offered; over the whole load 5% of the sampled ledgers were at the cap) | +| Leader tx-set build | 0.41 s | 0.23 s | +| Tx-set validation | 0.195 s, before the vote | 0.180 s, at apply, while the ballot runs | +| Trigger to first ballot (includes the build and the fetch; before, also validation) | 0.80 s | **0.42 s** | +| First ballot to externalize | 0.37 s | 0.30 s | +| Externalize to ledger closed (from node logs) | 0.21 s | **0.07 s** | +| **Sum: trigger to ledger closed** | **1.37 s** | **0.79 s** | +| Tx-set fetch (follower's Core, request to delivery) | 0.033 s | 0.052 s | | Leader timeouts, ballot timeouts, empty tx sets | 0, 0, 0 | 0, 0, 0 | -What the push did, over the whole load (totals of the 30 nodes): +Over the later windows of the run with the change, the sum is 0.81-0.84 s. + +What the new parts did, over the whole load of the run with the change: | | | |---|---:| -| Tx sets pushed (one per peer) | 28,507 | -| Deliveries that were already at the follower when its Core asked | 28,181 | -| Requests that crossed a push (the leader did not send a second copy) | 64 | -| Pushes dropped or evicted by a follower's limits, write errors | 0 | -| Requests Core had to repeat after its 1 s retry interval | 0 | - -- **The tx-set round trips are gone.** On one follower whose logs were saved (us-west-2), all 817 - sets it received during the load arrived by push before its Core asked, a median 38 ms ahead of - the request (minimum 18 ms). Measured from four nodes in four regions, a follower's Core held the - set a median 126 ms after the leader finished building it (76-169 ms by region pair); before, that - took about 480 ms on a 147 ms link. -- **The two runs are in different regimes.** Before, a ledger could not close in less than 1.33 s, - so 3,000 TPS filled every set to the cap and the mempool backed up. Now ledgers hold what was - offered. That also makes stages the push does not touch cheaper: the leader pulls 3,200 - candidates instead of 7,700, and sets are smaller. -- **Two changes were measured together:** the push and the 8 MiB congestion window. One run cannot - say how much each contributes. -- **Block time is about 1.05 s although the stages sum to 0.91 s**, and the windows drifted up by - about 25 ms over the 16 minutes. Neither was investigated. - -### The inclusion check - -The test compares the transactions in every node's ledgers with the number offered. All 30 nodes -held exactly 60 more (the same on every node, so no fork). The generators had submitted exactly -2,880,000 and every node had pulled each transaction once, so about 60 transactions (0.002%) were -included in two ledgers. - -Why this happens in the test and not on a network that applies ledgers: - -1. A leader that is not backed up proposes everything in its mempool, including a transaction - that was submitted a moment ago and is still being flooded to the far nodes (up to about 0.5 s). -2. With the push, a follower closes the ledger about 0.5 s after the leader built its set, so now - and then a far node closes the ledger before its own copy of that transaction has arrived. -3. The overlay's mempool removes a ledger's transactions when the ledger closes and keeps no memory - of them, on purpose. The late copy is admitted as a new transaction. -4. On a real network Core rejects that copy when the node next builds a set, because its sequence - number was already used. The test runs in "overlay-only" mode: ledgers are not applied, sequence - numbers never advance, and that check is switched off. So the node includes the transaction a - second time. - -That this path produced the 60 is inferred from the counters; the logs carry no transaction hashes. -Before, a follower closed the ledger about a second after the leader's build, well after flooding -had finished, so the two never met: the completed "before" run at 2,000 ms held exactly 2,880,000. - -It is accepted as an artifact of overlay-only mode. The check now allows up to 1 in 10,000 -transactions more than offered and reports how many there were; fewer than offered, more than -that, and nodes that disagree still fail. - -### Not rerun: the 2,000 ms target - -The 2,000 ms target was measured for the "before" commit only: PASS in all four windows (p75 -2,066-2,109 ms, p99 2,286-2,324 ms), with exactly 2,880,000 transactions on every node, and 0.19 s -less from trigger to externalize than with nomination (1.31 s against 1.50 s on #5459). The details -are in [#5495](https://github.com/stellar/stellar-core/pull/5495). It was not repeated with the -push. +| Ledgers executed speculatively (total of the 48 nodes) | 46,847 | +| Speculatively executed ledgers that were committed | 46,853 (46,852 ledgers closed; the counters are scraped at slightly different times) | +| Executions dropped because another value externalized | 0 | +| Tx sets found invalid at apply (empty ledger) | 0 | +| Execution (validation plus simulated apply), mean | 0.33-0.35 s | +| Commit after externalize, mean | 0.037-0.041 s | +| Overlay's write of a tx set to disk (8 nodes, 6,957 writes) | median 3.6 ms, p99 8.9 ms, maximum 237 ms | +| Core's write of SCP state before a vote, now without the tx set | 2.2-2.6 ms mean | + +- **Every ledger was executed before it externalized or while it did.** No execution was wasted: + with one leader per round and no timeouts, the value a node votes for is the value that + externalizes. +- **The execution is slightly longer than the ballot.** It takes 0.34-0.35 s (0.18-0.19 s of + validation plus the simulated apply), the ballot 0.30 s. In the logs of eight nodes, the ledger + closed more than 50 ms after externalize on 58% of the ledgers and more than 100 ms after on 21% + (a commit alone takes about 40 ms). Before, it was about 207 ms on every ledger. +- **The gain from speculation is an upper bound.** Apply is simulated by a sleep that uses no CPU, + so running it beside the ballot is free here. Real execution would compete with SCP processing + and signature verification. +- **The two builds do not simulate the same work per ledger.** Before, the sleep starts with the + apply and absorbs the commit. With the change it sits in the execution, and validation and the + commit come on top of it. +- **The two runs are in different regimes**, as in the earlier comparison. Before, a ledger could + not close in less than 1.42 s, so 5,000 TPS filled every set to the cap and the mempool backed + up. Now ledgers hold what was offered. That alone shortens the leader's build and the sets. One + run on each side cannot separate it from the effect of the change. +- **The push mostly loses its race now.** The leader's proposal leaves earlier relative to the + push, so a follower's request usually reaches the leader while the push is still in flight + (38,668 such requests against 7,032 sets that were already in place; before: 17 against 13,914). + The leader does not send a second copy, and the fetch costs about 0.02 s more per ledger. Why + the proposal overtakes the push is inferred from the removed database write, not timed. +- **Block time is 1.04-1.07 s although the stages sum to about 0.8 s**, and the windows drifted up + by about 45 ms over the 16 minutes. Neither was investigated. + +### The node that dropped out + +- **What happened.** 15 s before the end of the load, Core on one node logged "Overlay IPC + connection closed" and stopped reading from its overlay process. Both processes kept running and + the socket between them was still open, with unread data on Core's side. The node heard nothing + further from the network and lost sync. Two more nodes did the same two minutes after the load. +- **When.** All three times within 26 ms of a step of a history checkpoint publish. +- **Cause (inferred; the logs carry no error code).** Core's IPC reader treated any failed `recv` + as a closed connection, including one interrupted by a signal (`EINTR`). stellar-core handles + `SIGCHLD`, and a checkpoint publish runs child processes. +- **Fix.** `0c9a02e56` on this branch: an interrupted read or write on the IPC socket is retried. + A unit test interrupts a blocked read with a signal and fails without the fix. The fix has not + been run on a network yet. +- **Not caused by this experiment, as far as the code shows.** The reader is the same code in the + "before" build. That run had no such event in 30 minutes and this one had three in 40; one run + each cannot say whether the change makes it more likely. ### Where a ledger's time goes now -With the tx-set round trips gone, what is left of the 0.91 s from trigger to close at 1,000 ms: +What is left of the 0.79-0.84 s from trigger to ledger closed: + +1. **Trigger to first ballot: 0.42-0.47 s.** Most of it is the leader's build (0.23-0.25 s), on + the critical path. The first-round leader also builds once before the trigger and discards + that set unless it was capacity limited (542-577 builds for 281-290 ledgers). The tx-set fetch adds 0.05 s. +2. **The ballot: 0.30 s.** Nothing is validated in it any more; it is message delays between four + regions. +3. **Externalize to ledger closed: 0.07 s.** The commit (0.04 s), plus the end of the execution + on about half of the ledgers. + +The execution itself (0.34-0.35 s) runs beside the ballot. -1. **The ballot: 0.33 s.** Followers validate before they vote, and the ballot then needs four - one-way trips. -2. **The leader's build: 0.15 s**, on the critical path. With the trigger mostly on time, the - first-round leader also builds once before the trigger and discards that set unless it was - capacity limited (562 builds for 285 ledgers). -3. **Validation: 0.11 s.** -4. **About 70 ms between the leader finishing its set and its proposal leaving.** SCP state, - including the tx set, is written to the database before each broadcast. That this write accounts - for most of the 70 ms is inferred, not timed. The push already leaves during this time. -5. **Ledger close: 0.16 s**, almost all of it the simulated apply. +### Earlier fixture: 30 validators, 3,000 TPS + +The first two experiments were measured on a smaller fixture (10 organizations, 30 validators, +3,000 TPS, 1,000 ms target, 2026-11-10 and 2026-11-11). It is no longer the benchmark. + +| | Leader schedule alone (`daa965f4c`) | With the tx-set push (`8f7768be7`) | +|---|---|---| +| Block time windows | missed the first window | on target in all four windows, on all 30 nodes | +| p75, average over 30 nodes | 1,386 ms | 1,091 / 1,096 / 1,109 / 1,115 ms | +| Time per ledger | 1.33 s | 1.05-1.07 s | +| Tx-set fetch as seen by a follower's Core | 217 ms | 16 ms | +| Trigger to ledger closed (sum of the stages) | 1.30 s | 0.91 s | + +- With the push, every node's ledgers held 60 transactions more than the 2,880,000 offered: about + 60 transactions were included in two ledgers. This is an artifact of the test mode. Ledgers are + not applied, so sequence numbers never advance, and a copy of a transaction that reaches a far + node after its ledger closed is admitted and included again. The run was judged with an exact + comparison and reported FAIL for that check alone. A change that lets the check allow such + duplicates is proposed + ([performance-testing#4](https://github.com/stellar-experimental/performance-testing/pull/4)), + not merged. +- A 2,000 ms target was measured for the leader schedule alone: PASS in all four windows (p75 + 2,066-2,109 ms). +- The full results, stage tables and the measurement setup of that fixture are in + [the README as of #5497](https://github.com/stellar/stellar-core/blob/6d5fc9e814abee2b7e93553b3903b56929bedf91/README.md) + and in [#5495](https://github.com/stellar/stellar-core/pull/5495). ## How it was measured @@ -188,22 +238,22 @@ runs the load, judges it, and terminates the network. | | | |---|---| -| Validators | 30, all validating, no watchers | -| Organizations | 10 home domains of 3 validators each, all `HIGH` quality | -| Quorum set | Generated by stellar-core from the home domains: 7 of 10 organizations, 2 of 3 validators within each | -| Regions | 4 AWS regions: eu-central-1 (8 nodes), ap-southeast-1 (8), us-east-1 (7), us-west-2 (7) | -| Placement | 9 organizations have their three validators in three different regions; one has two in us-east-1 and one in ap-southeast-1. The layout was identical in both runs | -| Connectivity | Full mesh: every node is configured with the other 29 as its only peers | -| Round-trip times seen from us-west-2 | us-east-1 about 60 ms, eu-central-1 about 147 ms, ap-southeast-1 about 180 ms | +| Validators | 48, all validating, no watchers | +| Organizations | 16 home domains of 3 validators each, all `HIGH` quality | +| Quorum set | Generated by stellar-core from the home domains. Its rule gives 11 of 16 organizations, 2 of 3 validators within each (computed from the rule; not read back from a node) | +| Regions | 4 AWS regions, 12 nodes each: eu-central-1, us-west-2, ap-southeast-1, us-east-1 | +| Placement | 14 organizations have their three validators in three different regions; 2 have them in two regions. Both runs used the same layout seed | +| Connectivity | Full mesh: every node is configured with the other 47 as its only peers | +| Round-trip times seen from us-west-2 (measured on the earlier fixture, same regions) | us-east-1 about 60 ms, eu-central-1 about 147 ms, ap-southeast-1 about 180 ms | ### Hardware and software | | | |---|---| | Instance type | c6id.2xlarge: 8 vCPU (Intel Ice Lake, with SHA extensions), 16 GiB RAM, local NVMe | -| Build, with the push | Commit `8f7768be7`, the perftests package `30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests`. All 30 nodes reported that commit in their build label | -| Build, before | Commit `daa965f4c`, the perftests package `30.0.1-3674.daa965f4ce.noble~perf.leader.schedule.no.nomination~perftests` | -| Overlay | The Rust overlay of this stack: QUIC, one connection per peer. With the push, every connection starts with an 8 MiB congestion window (quinn's default before) | +| Build, with apply-time validation | Commit `a33da62c4`, the perftests package `30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests`. All 48 nodes reported that commit in their build label | +| Build, before | Commit `8f7768be7`, the perftests package `30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests`. All 48 nodes reported that commit in their build label | +| Overlay | The Rust overlay of this stack: QUIC, one connection per peer, 8 MiB initial congestion window. With the change it also stores tx sets on the local NVMe | | Ledger protocol | 30, upgraded from genesis during preparation | | Database | SQLite | @@ -216,8 +266,8 @@ ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING=true UNSAFE_QUORUM=true FAILURE_SAFETY=0 GENESIS_TEST_ACCOUNT_COUNT=1000000 -TARGET_PEER_CONNECTIONS=29 -MAX_ADDITIONAL_PEER_CONNECTIONS=87 +TARGET_PEER_CONNECTIONS=47 +MAX_ADDITIONAL_PEER_CONNECTIONS=141 PREFERRED_PEERS_ONLY=true FLOOD_DEMAND_BACKOFF_DELAY_MS=1000 INVARIANT_CHECKS=[] @@ -225,48 +275,49 @@ TESTING_IGNORE_LEDGER_TIME_UPGRADE_BOUNDS=true TESTING_MAX_CLASSIC_BYTE_ALLOWANCE=1048576 TESTING_MAX_SOROBAN_BYTE_ALLOWANCE=9437184 LOADGEN_PREGENERATED_TRANSACTIONS_FILE="/var/lib/stellar-core/loadgen-txs.xdr" -OP_APPLY_SLEEP_TIME_DURATION_FOR_TESTING=[50] # [25] in the 2,000 ms run +OP_APPLY_SLEEP_TIME_DURATION_FOR_TESTING=[30] OP_APPLY_SLEEP_TIME_WEIGHT_FOR_TESTING=[100] ``` +`EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` was on (the default) in the run with the change. `EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` was off (the default). Catchup and automatic restart are off, so a node that crashes or falls behind stays that way and fails the run. ### Network settings (applied by upgrade before the load) -| Setting | 1,000 ms runs (before and after) | 2,000 ms run (before only) | -|---|---:|---:| -| Target ledger close time | 1,000 ms | 2,000 ms | -| SCP timeouts (ballot and nomination, initial and increment) | 2,000 ms | 2,000 ms | -| Maximum transactions per ledger (125% of the offered load per block) | 3,750 | 7,500 | -| Maximum tx-set size in bytes | 10 MiB (1 MiB classic + 9 MiB Soroban) | same | -| Soroban limits | raised to fit the load | raised to fit the load | +| Setting | Both runs | +|---|---:| +| Target ledger close time | 1,000 ms | +| SCP timeouts (ballot and nomination, initial and increment) | 2,000 ms | +| Maximum transactions per ledger (125% of the offered load per block) | 6,250 | +| Maximum tx-set size in bytes | 10 MiB (1 MiB classic + 9 MiB Soroban) | +| Soroban limits | raised to fit the load | The leader timer of this branch uses the ballot timeout, so a round's leader had 2 s (4 s with its -one extension) in every run. +one extension). ### Load | | | |---|---| | Transactions | Stellar Asset Contract (SAC) payments, pregenerated; no classic transactions | -| Rate | 3,000 TPS offered: every validator generates 100 TPS | -| Accounts | 1,000,000 genesis accounts, 33,333 per generator | +| Rate | 5,000 TPS offered: every validator generates about 104 TPS | +| Accounts | 1,000,000 genesis accounts, 20,833 per generator | | Duration | 60 s warm-up, then 900 s measured | -| Apply | Skipped ("overlay-only" mode) and replaced by a sleep per operation, sized as a 150 ms budget per ledger at the offered load: 50 us per operation at 1,000 ms, 25 us at 2,000 ms | +| Apply | Skipped ("overlay-only" mode) and replaced by a sleep per operation, sized as a 150 ms budget per ledger at the offered load: 30 us per operation | Command (before and after differ only in the build): ``` block_time_check.py \ - --stellar-core-version 30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests \ - --block-time-ms 1000 \ - --soroban-tps 3000 --soroban-type sac --classic-tps 0 \ + --stellar-core-version 30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests \ + --org-count 16 --block-time-ms 1000 \ + --soroban-tps 5000 --soroban-type sac --classic-tps 0 \ --simulate-apply-budget-ms 150 --load-duration-s 900 \ --instance-type c6id.2xlarge --save-logs always ``` -Defaults that matter: 10 organizations, validators as generators, `--scp-timeout-ms 2000`, +Defaults that matter: validators as generators, `--scp-timeout-ms 2000`, `--max-tx-set-bytes 10485760`, 1,000,000 genesis accounts, seed 0. ### How a run is judged @@ -275,26 +326,50 @@ Defaults that matter: 10 organizations, validators as generators, `--scp-timeout 5-minute windows after the warm-up. - **PASS** needs, on every node in every window: p75 within [0.8 T, 1.2 T), p99 at most 2 T, and at least one close per 2 T on average, where T is the target block time. It also needs every node's - ledgers to hold the offered transactions, zero error counters, all nodes in sync, no fork, and - the network still closing ledgers afterwards. -- "Hold the offered transactions" was an exact comparison when these runs were judged. It now - allows up to 1 in 10,000 more, for the duplicates overlay-only mode lets through (see - [The inclusion check](#the-inclusion-check)). + ledgers to hold exactly the offered transactions, zero error counters, all nodes in sync, no + fork, and the network still closing ledgers afterwards. - The load stops at the first window that misses, so a FAIL can rest on one window (as in the - "before" run). + "before" run). The inclusion check is skipped when the load ends early or a generator fails + (both runs). ### Limits of these numbers -- One run on each side. The "before" numbers at 1,000 ms come from a single 5-minute window, and - from a commit that differs from this branch's base in formatting only. -- The push and the 8 MiB congestion window were not measured separately. -- Apply is simulated, so these are consensus and overlay numbers, not end-to-end ledger close. +- One run on each side. The "before" numbers come from a single 5-minute window. +- The run with the change is a FAIL for a node that lost its IPC reader. The block time windows + include that node up to the moment it stopped; the stage numbers are medians over the 48 nodes. + The fix has not been measured. +- Apply is simulated by a sleep, so these are consensus and overlay numbers, not end-to-end ledger + close. The gain from speculative execution is an upper bound, and the two builds do not sleep + for the same part of the apply (above). +- The inclusion check was judged in neither run. +- `herder.txset.validate` times the pre-vote check on the main thread in the "before" build and the + apply-time check on the apply thread in the build with the change. - `scp.timing.trigger-to-ballot` records 0 when a follower votes before its own trigger fires. -- The timings between a leader and its followers come from live logs of four nodes (one per - region) and the saved logs of one node. +- Externalize-to-close and the tx-set store timings come from the logs of eight nodes (two per + region) in the run with the change, and of four nodes in the "before" run. +- No run has measured an invalid tx set, a dropped speculative execution or a restart from the + tx-set store on a network; those paths are covered by unit tests only. ## Known limitations of the prototype +Apply-time validation and speculative execution: +- A nondeterministic validity check used to cost liveness at most. It now forks the ledger. The + check's node-local inputs are the binary's protocol version, the network passphrase and, in test + builds, the load generation flags; validators that differ in those would disagree. +- A leader can get an invalid set externalized. The ledger is empty, and the set's transactions + are still removed from every mempool, so a faulty leader can drop pending transactions at no + cost whenever it leads. +- Downstream consumers see new shapes: a LedgerCloseMeta with a non-empty tx set and no + transaction meta, and a history `transactions` entry without a `results` entry. +- If the overlay drops a set Core built (hash mismatch or encode error, both Core bugs), Core + still believes the set is stored. +- A value whose execution was dropped is not executed speculatively again. `ledger.ledger.close` + leaves out the time a finished execution waits for externalize. +- A parked execution keeps a SQL transaction open for the length of the ballot. It is not + confirmed that no other writer on the main database can make its commit fail. +- Speculative execution was never measured with real transaction execution. +- There is no compatibility work beyond the protocol gate. + Tx-set push: - The push is not relayed. A node that is not connected to the leader requests the set from the peer that relayed the proposal. @@ -303,7 +378,7 @@ Tx-set push: - The limits on unrequested sets can be stepped around by a peer that sends SCP envelopes naming unknown tx sets, and the 8 MiB congestion window applies to every stream of a connection. Both are prototype settings. -- Transactions can be included twice in overlay-only mode (above). +- Transactions can be included twice in overlay-only mode (see the earlier fixture). Leader schedule: - The leader schedule's node set is whatever transitive quorum each node currently knows. It @@ -313,10 +388,18 @@ Leader schedule: - Leader grinding through the header hash and fairness between organizations are out of scope. - The leader timer extension is broader than intended, and a proposal that arrives just after a timeout is wasted. Neither occurred in these runs. -- A build with `--disable-tests` does not compile at this commit (test headers reached from - `herder/HerderImpl.cpp`). +- A build with `--disable-tests` did not compile with the leader schedule (test headers reached + from `herder/HerderImpl.cpp`) and was not tried since. ## Building and the rest of the documentation Unchanged from stellar-core: see [INSTALL.md](./INSTALL.md), [CONTRIBUTING.md](./CONTRIBUTING.md) -and [`./docs`](./docs). The full test suite passes on `8f7768be7`, also with `--all-versions`. +and [`./docs`](./docs). + +Test status of this branch's code (`0c9a02e56`, the same tree as the merge of #5498): +- The full test suite passes (615 test cases). +- With `--all-versions` the suite passes on `a33da62c4`, the commit before the IPC fix. On + `0c9a02e56` one test case that the fix does not touch ("PendingEnvelopes recvSCPEnvelope", at + protocol version 4) failed in one run. It passes alone and fails when it runs after three other + test cases in the same process; whether it does the same without this branch's changes is not + established. From 700be535db3ab5cd0409bc2fc7a395412427f692 Mon Sep 17 00:00:00 2001 From: Garand Tyson Date: Wed, 11 Nov 2026 13:10:22 -0800 Subject: [PATCH 2/5] README: the one --all-versions failure is a flaw of a leader-schedule test helper It fails for one combination of the day's test seed and a partition's instance number, also on the binary of #5495 alone. --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 1b9c83584..57e5098ee 100644 --- a/README.md +++ b/README.md @@ -399,7 +399,9 @@ and [`./docs`](./docs). Test status of this branch's code (`0c9a02e56`, the same tree as the merge of #5498): - The full test suite passes (615 test cases). - With `--all-versions` the suite passes on `a33da62c4`, the commit before the IPC fix. On - `0c9a02e56` one test case that the fix does not touch ("PendingEnvelopes recvSCPEnvelope", at - protocol version 4) failed in one run. It passes alone and fails when it runs after three other - test cases in the same process; whether it does the same without this branch's changes is not - established. + `0c9a02e56` one test case of the leader-schedule change failed ("PendingEnvelopes + recvSCPEnvelope", at protocol version 4): its helper tries 100 random peers and needs one that + does not lead round 1, and for that day's test seed and that partition's node key the local + node sorts last in the schedule, so every peer leads. The binary of #5495 alone fails the same + way with the same seed and instance number, and the test passes with any other instance number + tried. It is a flaw of that test helper, not of the later changes. Not fixed. From 5fbee22fff1e04e1c6d4290007b5a4b3298fb3f5 Mon Sep 17 00:00:00 2001 From: Garand Tyson Date: Wed, 11 Nov 2026 13:10:36 -0800 Subject: [PATCH 3/5] README: say which instance number the test passes with --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 57e5098ee..a2f63e629 100644 --- a/README.md +++ b/README.md @@ -403,5 +403,5 @@ Test status of this branch's code (`0c9a02e56`, the same tree as the merge of #5 recvSCPEnvelope", at protocol version 4): its helper tries 100 random peers and needs one that does not lead round 1, and for that day's test seed and that partition's node key the local node sorts last in the schedule, so every peer leads. The binary of #5495 alone fails the same - way with the same seed and instance number, and the test passes with any other instance number - tried. It is a flaw of that test helper, not of the later changes. Not fixed. + way with the same seed and instance number, and the test passes with instance number 0. It is + a flaw of that test helper, not of the later changes. Not fixed. From 8fb702d9498eff173c5a09d4dd56d8acd71f9393 Mon Sep 17 00:00:00 2001 From: Garand Tyson Date: Wed, 11 Nov 2026 13:26:39 -0800 Subject: [PATCH 4/5] README: report the run with the IPC fix (0c9a02e56) as the result of apply-time validation All four windows on target on 48 nodes (p75 1,081-1,142 ms), no node lost its IPC connection. The verdict is FAIL on the exact inclusion check alone: 60 transactions more than offered on every node, the same excess as in the 30-validator run with the push; say that it is not explained. The run without the fix becomes the earlier run that found the IPC bug. --- README.md | 200 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 113 insertions(+), 87 deletions(-) diff --git a/README.md b/README.md index a2f63e629..3ecd0c1be 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ is not compatible with nodes that nominate. The benchmark is now **48 validators (16 organizations), 5,000 TPS, 1,000 ms target block time**. The latest experiment (vote without validating the tx set, validate and execute at apply) was -measured there on commit `a33da62c4` on 2026-11-11. "Before" is the previous experiment on this -branch (the leader's tx-set push), commit `8f7768be7`, measured on the same day on the same -fixture. One run per build. +measured there on commit `0c9a02e56`, the code of this branch, on 2026-11-11. "Before" is the +previous experiment on this branch (the leader's tx-set push), commit `8f7768be7`, measured on the +same day on the same fixture. One run per build. ## What this branch is @@ -66,27 +66,30 @@ executed, and what an invalid set does to the ledger. Target block time 1,000 ms, 5,000 TPS, 48 validators in 4 regions. -| | Before: tx-set push (`8f7768be7`) | With apply-time validation (`a33da62c4`) | +| | Before: tx-set push (`8f7768be7`) | With apply-time validation (`0c9a02e56`) | |---|---|---| | Block time windows | **missed** the first window on all 48 nodes; the load stops there | **on target in all four windows**, on all 48 nodes | -| p75, average over 48 nodes | 1,461 ms (limit 1,200) | 1,094 / 1,116 / 1,130 / 1,137 ms | -| p99, average over 48 nodes | 1,638 ms (limit 2,000) | 1,313 / 1,453 / 1,408 / 1,435 ms | -| Time per ledger | 1.42 s | 1.04-1.07 s | -| Trigger to ledger closed (sum of the stages) | 1.37 s | 0.79-0.84 s | -| Run verdict | FAIL (block time) | FAIL (one node dropped out of consensus, see the caveat) | +| p75, average over 48 nodes | 1,461 ms (limit 1,200) | 1,081 / 1,132 / 1,118 / 1,142 ms | +| p99, average over 48 nodes | 1,638 ms (limit 2,000) | 1,292 / 1,413 / 1,598 / 1,595 ms | +| Time per ledger | 1.42 s | 1.03-1.08 s | +| Trigger to ledger closed (sum of the stages) | 1.37 s | 0.76-0.84 s | +| Run verdict | FAIL (block time) | FAIL (inclusion check only, see the caveat) | In short: before, the stages of one ledger added up to 1.37 s, every tx set was full and the network backed up. With validation moved out of the vote and the ledger executed while the ballot runs, a ledger's stages fit into about 0.8 s and the network keeps up with 5,000 TPS at a 1 s target. -**Caveat: the run with the change is a FAIL, although every block time window was on target.** -15 s before the load ended, one node (of 48) stopped reading from its overlay process and fell out -of consensus; two more did the same two minutes after the load. The cause is a bug in the -Core-overlay IPC socket code that this experiment did not introduce (see -[The node that dropped out](#the-node-that-dropped-out)). The fix is on this branch (`0c9a02e56`); -it has not been measured on a network yet. Because the load counted as ended early, the inclusion -check was not judged. +**Caveat: the run with the change is a FAIL on the inclusion check, and on that check alone.** +Every node's ledgers held 4,800,060 transactions against 4,800,000 offered: 60 more (0.00125%), +the same on all 48 nodes. The check is an exact comparison. All other checks passed. The 60 are +not explained (see [The inclusion check](#the-inclusion-check)). + +**An earlier run of the same change found a bug.** The change was first measured without its last +commit (`a33da62c4`). The block times were about the same (p75 1,094 / 1,116 / 1,130 / 1,137 ms, all +windows on target), but one node fell out of consensus 15 s before the end of the load. The cause +was in the Core-overlay IPC code and is fixed in `0c9a02e56`; in the run reported here no node was +affected (see [The IPC bug](#the-ipc-bug-found-by-the-first-run)). ### 1,000 ms target, with apply-time validation @@ -94,63 +97,62 @@ Load ran its full 900 s. Every node was on target in every window. | Window | p75, average (range over nodes) | p99, average (range over nodes) | |---|---|---| -| 1 | 1,094 ms (1,083-1,110) | 1,313 ms (1,260-1,394) | -| 2 | 1,116 ms (1,107-1,126) | 1,453 ms (1,393-1,522) | -| 3 | 1,130 ms (1,116-1,145) | 1,408 ms (1,357-1,499) | -| 4 (2.4 minutes) | 1,137 ms (1,121-1,160) | 1,435 ms (1,381-1,486) | +| 1 | 1,081 ms (1,078-1,087) | 1,292 ms (1,223-1,358) | +| 2 | 1,132 ms (1,118-1,150) | 1,413 ms (1,355-1,511) | +| 3 | 1,118 ms (1,109-1,126) | 1,598 ms (1,433-1,710) | +| 4 (2.2 minutes) | 1,142 ms (1,127-1,170) | 1,595 ms (1,413-1,712) | Limits: p75 below 1,200 ms, p99 at most 2,000 ms. Checks: -- **No fork**, still closing ledgers after the load: passed. -- **Error counters zero** (invalid signatures, invariant failures, publish failures, invalid - values), no node restarts: passed. -- **All nodes in sync: failed.** One node was out of sync at the end of the load. -- **No wasted rounds during the load:** every ledger was decided by the first leader. 0 leader - timeouts, 0 ballot timeouts, 0 empty transaction sets. No tx set was found invalid at apply. -- **Inclusion: not judged.** Every one of the 47 healthy nodes holds 4,798,690 transactions - against 4,800,000 offered, 1,310 fewer (0.027%). The node that dropped out kept generating for - 15 s (about 104 transactions per second) while nothing left it, which accounts for a shortfall - of that size. That these are the missing ones is inferred, not shown. +- **No fork**, all 48 nodes in sync, still closing ledgers after the load: passed. +- **Error counters zero** (invalid signatures, invariant failures, publish failures), no node lost + sync: passed. +- **No wasted rounds:** every ledger was decided by the first leader. 0 leader timeouts, 0 ballot + timeouts, 0 empty transaction sets. No tx set was found invalid at apply. +- **Inclusion: failed.** 4,800,060 transactions in every node's ledgers against 4,800,000 offered. + See below. ### Before and after, stage by stage 5-minute means at the end of each run's first window, median over the 48 nodes. -| | Before (`8f7768be7`) | With apply-time validation (`a33da62c4`) | +| | Before (`8f7768be7`) | With apply-time validation (`0c9a02e56`) | |---|---:|---:| -| Ledgers per 5 minutes | 211-212 | 288-290 | -| Transactions per ledger | 6,250, the cap, on every ledger | 5,189 (what was offered; over the whole load 5% of the sampled ledgers were at the cap) | -| Leader tx-set build | 0.41 s | 0.23 s | -| Tx-set validation | 0.195 s, before the vote | 0.180 s, at apply, while the ballot runs | -| Trigger to first ballot (includes the build and the fetch; before, also validation) | 0.80 s | **0.42 s** | +| Ledgers per 5 minutes | 211-212 | 291-292 | +| Transactions per ledger | 6,250, the cap, on every ledger | 5,152 (what was offered; below the cap) | +| Leader tx-set build | 0.41 s | 0.24 s | +| Tx-set validation | 0.195 s, before the vote | 0.179 s, at apply, while the ballot runs | +| Trigger to first ballot (includes the build and the fetch; before, also validation) | 0.80 s | **0.41 s** | | First ballot to externalize | 0.37 s | 0.30 s | -| Externalize to ledger closed (from node logs) | 0.21 s | **0.07 s** | -| **Sum: trigger to ledger closed** | **1.37 s** | **0.79 s** | -| Tx-set fetch (follower's Core, request to delivery) | 0.033 s | 0.052 s | +| Externalize to ledger closed (from node logs) | 0.21 s | **0.05-0.07 s** | +| **Sum: trigger to ledger closed** | **1.37 s** | **0.76-0.78 s** | +| Tx-set fetch (follower's Core, request to delivery) | 0.033 s | 0.051 s | | Leader timeouts, ballot timeouts, empty tx sets | 0, 0, 0 | 0, 0, 0 | -Over the later windows of the run with the change, the sum is 0.81-0.84 s. +In the later windows of the run with the change, trigger to first ballot is 0.47 s and the sum +0.82-0.84 s. Externalize to ledger closed is the range of the per-node medians of eight nodes over +the whole load. What the new parts did, over the whole load of the run with the change: | | | |---|---:| -| Ledgers executed speculatively (total of the 48 nodes) | 46,847 | -| Speculatively executed ledgers that were committed | 46,853 (46,852 ledgers closed; the counters are scraped at slightly different times) | +| Ledgers executed speculatively (total of the 48 nodes) | 46,194 | +| Speculatively executed ledgers that were committed | 46,206 (46,204 ledgers closed; the counters are scraped at slightly different times) | | Executions dropped because another value externalized | 0 | | Tx sets found invalid at apply (empty ledger) | 0 | | Execution (validation plus simulated apply), mean | 0.33-0.35 s | | Commit after externalize, mean | 0.037-0.041 s | -| Overlay's write of a tx set to disk (8 nodes, 6,957 writes) | median 3.6 ms, p99 8.9 ms, maximum 237 ms | +| Overlay's write of a tx set to disk (8 nodes, 6,941 writes) | median 4.3 ms, p99 10.8 ms, maximum 356 ms | | Core's write of SCP state before a vote, now without the tx set | 2.2-2.6 ms mean | - **Every ledger was executed before it externalized or while it did.** No execution was wasted: with one leader per round and no timeouts, the value a node votes for is the value that externalizes. -- **The execution is slightly longer than the ballot.** It takes 0.34-0.35 s (0.18-0.19 s of +- **The execution is slightly longer than the ballot.** It takes 0.33-0.35 s (0.18-0.19 s of validation plus the simulated apply), the ballot 0.30 s. In the logs of eight nodes, the ledger - closed more than 50 ms after externalize on 58% of the ledgers and more than 100 ms after on 21% + closed more than 50 ms after externalize on 60% of the ledgers and more than 100 ms after on 21% (a commit alone takes about 40 ms). Before, it was about 207 ms on every ledger. - **The gain from speculation is an upper bound.** Apply is simulated by a sleep that uses no CPU, so running it beside the ballot is free here. Real execution would compete with SCP processing @@ -164,42 +166,71 @@ What the new parts did, over the whole load of the run with the change: run on each side cannot separate it from the effect of the change. - **The push mostly loses its race now.** The leader's proposal leaves earlier relative to the push, so a follower's request usually reaches the leader while the push is still in flight - (38,668 such requests against 7,032 sets that were already in place; before: 17 against 13,914). + (38,883 such requests against 6,240 sets that were already in place; before: 17 against 13,914). The leader does not send a second copy, and the fetch costs about 0.02 s more per ledger. Why the proposal overtakes the push is inferred from the removed database write, not timed. -- **Block time is 1.04-1.07 s although the stages sum to about 0.8 s**, and the windows drifted up - by about 45 ms over the 16 minutes. Neither was investigated. +- **Block time is 1.03-1.08 s although the stages sum to about 0.8 s.** The windows drifted up + over the 16 minutes: p75 by about 60 ms, p99 from 1,292 to about 1,600 ms. Neither was + investigated. +- **The two runs of the change agree.** Without the IPC fix (`a33da62c4`): p75 1,094-1,137 ms, + 288-290 ledgers in the first 5 minutes, trigger to first ballot 0.42 s, first ballot to + externalize 0.30 s, every ledger executed speculatively and committed. + +### The inclusion check + +The test compares the transactions in every node's ledgers with the number offered. In the run +with the change all 48 nodes held exactly 60 more (the same on every node, so no fork). The +generators had submitted exactly 4,800,000 and none was rejected. + +- **The same excess, exactly 60, appeared on the earlier fixture** in the run with the tx-set push + (30 validators, 3,000 TPS: 2,880,060 against 2,880,000). +- **The explanation given then was an inference**: the test runs in "overlay-only" mode, where + ledgers are not applied and sequence numbers never advance, so a copy of a transaction that + reaches a far node after its ledger closed is admitted again and included a second time. The + logs carry no transaction hashes, so this was never shown. +- **It does not predict the same number** at a different rate and network size. Which + transactions are counted twice, or whether something else adds 60, is open. +- The "before" run and the first run of the change did not get to this check (the load ended + early or a generator failed). In the first run of the change the 47 healthy nodes held + 4,798,690, fewer than offered; the node that dropped out kept generating for 15 s while nothing + left it. +- A change that lets the check allow up to 1 in 10,000 more than offered is proposed + ([performance-testing#4](https://github.com/stellar-experimental/performance-testing/pull/4)), + not merged. Under it this run would pass. -### The node that dropped out +### The IPC bug found by the first run -- **What happened.** 15 s before the end of the load, Core on one node logged "Overlay IPC - connection closed" and stopped reading from its overlay process. Both processes kept running and - the socket between them was still open, with unread data on Core's side. The node heard nothing - further from the network and lost sync. Two more nodes did the same two minutes after the load. +- **What happened.** In the first run of the change (`a33da62c4`), 15 s before the end of the + load, Core on one node logged "Overlay IPC connection closed" and stopped reading from its + overlay process. Both processes kept running and the socket between them was still open, with + unread data on Core's side. The node heard nothing further from the network and lost sync. Two + more nodes did the same two minutes after the load. - **When.** All three times within 26 ms of a step of a history checkpoint publish. - **Cause (inferred; the logs carry no error code).** Core's IPC reader treated any failed `recv` as a closed connection, including one interrupted by a signal (`EINTR`). stellar-core handles `SIGCHLD`, and a checkpoint publish runs child processes. -- **Fix.** `0c9a02e56` on this branch: an interrupted read or write on the IPC socket is retried. - A unit test interrupts a blocked read with a signal and fails without the fix. The fix has not - been run on a network yet. -- **Not caused by this experiment, as far as the code shows.** The reader is the same code in the - "before" build. That run had no such event in 30 minutes and this one had three in 40; one run - each cannot say whether the change makes it more likely. +- **Fix.** `0c9a02e56`: an interrupted read or write on the IPC socket is retried. A unit test + interrupts a blocked read with a signal and fails without the fix. +- **In the run with the fix** all 48 nodes were checked seven times, before, during and after the + load: none logged the warning and none had a stuck socket. +- **One clean run does not prove the fix.** The reader was the same code in the "before" build, + and that run had no such event in 30 minutes either. The first run of the change had three in + 40 minutes; whether the change makes the event more likely is not known. ### Where a ledger's time goes now -What is left of the 0.79-0.84 s from trigger to ledger closed: +What is left of the 0.76-0.84 s from trigger to ledger closed: -1. **Trigger to first ballot: 0.42-0.47 s.** Most of it is the leader's build (0.23-0.25 s), on +1. **Trigger to first ballot: 0.41-0.47 s.** Most of it is the leader's build (0.24-0.25 s), on the critical path. The first-round leader also builds once before the trigger and discards - that set unless it was capacity limited (542-577 builds for 281-290 ledgers). The tx-set fetch adds 0.05 s. + that set unless it was capacity limited (540-568 builds for 279-292 ledgers). The tx-set fetch + adds 0.05 s. 2. **The ballot: 0.30 s.** Nothing is validated in it any more; it is message delays between four regions. -3. **Externalize to ledger closed: 0.07 s.** The commit (0.04 s), plus the end of the execution - on about half of the ledgers. +3. **Externalize to ledger closed: 0.05-0.07 s.** The commit (0.04 s), plus the end of the + execution on more than half of the ledgers. -The execution itself (0.34-0.35 s) runs beside the ballot. +The execution itself (0.33-0.35 s) runs beside the ballot. ### Earlier fixture: 30 validators, 3,000 TPS @@ -214,14 +245,9 @@ The first two experiments were measured on a smaller fixture (10 organizations, | Tx-set fetch as seen by a follower's Core | 217 ms | 16 ms | | Trigger to ledger closed (sum of the stages) | 1.30 s | 0.91 s | -- With the push, every node's ledgers held 60 transactions more than the 2,880,000 offered: about - 60 transactions were included in two ledgers. This is an artifact of the test mode. Ledgers are - not applied, so sequence numbers never advance, and a copy of a transaction that reaches a far - node after its ledger closed is admitted and included again. The run was judged with an exact - comparison and reported FAIL for that check alone. A change that lets the check allow such - duplicates is proposed - ([performance-testing#4](https://github.com/stellar-experimental/performance-testing/pull/4)), - not merged. +- With the push, every node's ledgers held 60 transactions more than the 2,880,000 offered. The + run was judged with an exact comparison and reported FAIL for that check alone (see + [The inclusion check](#the-inclusion-check)). - A 2,000 ms target was measured for the leader schedule alone: PASS in all four windows (p75 2,066-2,109 ms). - The full results, stage tables and the measurement setup of that fixture are in @@ -230,7 +256,7 @@ The first two experiments were measured on a smaller fixture (10 organizations, ## How it was measured -Both runs (before and after) use the `block_time_check` mission of perfnet, SDF's tooling for +All runs use the `block_time_check` mission of perfnet, SDF's tooling for performance networks on AWS. Each run launches a fresh network, brings it to the target settings, runs the load, judges it, and terminates the network. @@ -242,7 +268,7 @@ runs the load, judges it, and terminates the network. | Organizations | 16 home domains of 3 validators each, all `HIGH` quality | | Quorum set | Generated by stellar-core from the home domains. Its rule gives 11 of 16 organizations, 2 of 3 validators within each (computed from the rule; not read back from a node) | | Regions | 4 AWS regions, 12 nodes each: eu-central-1, us-west-2, ap-southeast-1, us-east-1 | -| Placement | 14 organizations have their three validators in three different regions; 2 have them in two regions. Both runs used the same layout seed | +| Placement | 14 organizations have their three validators in three different regions; 2 have them in two regions. All runs used the same layout seed | | Connectivity | Full mesh: every node is configured with the other 47 as its only peers | | Round-trip times seen from us-west-2 (measured on the earlier fixture, same regions) | us-east-1 about 60 ms, eu-central-1 about 147 ms, ap-southeast-1 about 180 ms | @@ -251,7 +277,7 @@ runs the load, judges it, and terminates the network. | | | |---|---| | Instance type | c6id.2xlarge: 8 vCPU (Intel Ice Lake, with SHA extensions), 16 GiB RAM, local NVMe | -| Build, with apply-time validation | Commit `a33da62c4`, the perftests package `30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests`. All 48 nodes reported that commit in their build label | +| Build, with apply-time validation | Commit `0c9a02e56`, the perftests package `30.0.1-3678.0c9a02e566.noble~perf.apply.time.txset.validation~perftests` (the first run of the change: `a33da62c4`, `30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests`) | | Build, before | Commit `8f7768be7`, the perftests package `30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests`. All 48 nodes reported that commit in their build label | | Overlay | The Rust overlay of this stack: QUIC, one connection per peer, 8 MiB initial congestion window. With the change it also stores tx sets on the local NVMe | | Ledger protocol | 30, upgraded from genesis during preparation | @@ -279,13 +305,13 @@ OP_APPLY_SLEEP_TIME_DURATION_FOR_TESTING=[30] OP_APPLY_SLEEP_TIME_WEIGHT_FOR_TESTING=[100] ``` -`EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` was on (the default) in the run with the change. +`EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` was on (the default) in the runs with the change. `EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` was off (the default). Catchup and automatic restart are off, so a node that crashes or falls behind stays that way and fails the run. ### Network settings (applied by upgrade before the load) -| Setting | Both runs | +| Setting | All runs | |---|---:| | Target ledger close time | 1,000 ms | | SCP timeouts (ballot and nomination, initial and increment) | 2,000 ms | @@ -306,11 +332,11 @@ one extension). | Duration | 60 s warm-up, then 900 s measured | | Apply | Skipped ("overlay-only" mode) and replaced by a sleep per operation, sized as a 150 ms budget per ledger at the offered load: 30 us per operation | -Command (before and after differ only in the build): +Command (the runs differ only in the build): ``` block_time_check.py \ - --stellar-core-version 30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests \ + --stellar-core-version 30.0.1-3678.0c9a02e566.noble~perf.apply.time.txset.validation~perftests \ --org-count 16 --block-time-ms 1000 \ --soroban-tps 5000 --soroban-type sac --classic-tps 0 \ --simulate-apply-budget-ms 150 --load-duration-s 900 \ @@ -330,18 +356,17 @@ Defaults that matter: validators as generators, `--scp-timeout-ms 2000`, fork, and the network still closing ledgers afterwards. - The load stops at the first window that misses, so a FAIL can rest on one window (as in the "before" run). The inclusion check is skipped when the load ends early or a generator fails - (both runs). + (the "before" run and the first run of the change). ### Limits of these numbers -- One run on each side. The "before" numbers come from a single 5-minute window. -- The run with the change is a FAIL for a node that lost its IPC reader. The block time windows - include that node up to the moment it stopped; the stage numbers are medians over the 48 nodes. - The fix has not been measured. +- One run on each side, plus the first run of the change without the IPC fix. The "before" + numbers come from a single 5-minute window. +- The run with the change is a FAIL on the inclusion check, and the 60 extra transactions are not + explained. - Apply is simulated by a sleep, so these are consensus and overlay numbers, not end-to-end ledger close. The gain from speculative execution is an upper bound, and the two builds do not sleep for the same part of the apply (above). -- The inclusion check was judged in neither run. - `herder.txset.validate` times the pre-vote check on the main thread in the "before" build and the apply-time check on the apply thread in the build with the change. - `scp.timing.trigger-to-ballot` records 0 when a follower votes before its own trigger fires. @@ -378,7 +403,8 @@ Tx-set push: - The limits on unrequested sets can be stepped around by a peer that sends SCP envelopes naming unknown tx sets, and the 8 MiB congestion window applies to every stream of a connection. Both are prototype settings. -- Transactions can be included twice in overlay-only mode (see the earlier fixture). +- Both runs that reached the inclusion check with the push held 60 transactions more than offered + (see [The inclusion check](#the-inclusion-check)). Not explained. Leader schedule: - The leader schedule's node set is whatever transitive quorum each node currently knows. It From cb9ed0062d7f6fc28e27731cbb0f5d292db9879e Mon Sep 17 00:00:00 2001 From: Garand Tyson Date: Wed, 11 Nov 2026 13:54:34 -0800 Subject: [PATCH 5/5] README: fourteen IPC sweeps over 44 checkpoints per node in the run with the fix --- README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 3ecd0c1be..d6f248438 100644 --- a/README.md +++ b/README.md @@ -211,11 +211,13 @@ generators had submitted exactly 4,800,000 and none was rejected. `SIGCHLD`, and a checkpoint publish runs child processes. - **Fix.** `0c9a02e56`: an interrupted read or write on the IPC socket is retried. A unit test interrupts a blocked read with a signal and fails without the fix. -- **In the run with the fix** all 48 nodes were checked seven times, before, during and after the - load: none logged the warning and none had a stuck socket. -- **One clean run does not prove the fix.** The reader was the same code in the "before" build, - and that run had no such event in 30 minutes either. The first run of the change had three in - 40 minutes; whether the change makes the event more likely is not known. +- **In the run with the fix** all 48 nodes were checked fourteen times, from before the load until + 17 s before the network was terminated: none logged the warning and none had a stuck socket. + Every node published 44 checkpoints in that time (about 2,100 in the network); the three events + of the first run fell within its first 19 checkpoints per node (about 900). +- **This is not a proof.** The fix logs nothing when it retries, so there is no count of + interrupted reads it survived, and the "before" build, with the unfixed reader, had no such + event in its 30 minutes either. Whether the change makes the event more likely is not known. ### Where a ledger's time goes now @@ -277,7 +279,7 @@ runs the load, judges it, and terminates the network. | | | |---|---| | Instance type | c6id.2xlarge: 8 vCPU (Intel Ice Lake, with SHA extensions), 16 GiB RAM, local NVMe | -| Build, with apply-time validation | Commit `0c9a02e56`, the perftests package `30.0.1-3678.0c9a02e566.noble~perf.apply.time.txset.validation~perftests` (the first run of the change: `a33da62c4`, `30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests`) | +| Build, with apply-time validation | Commit `0c9a02e56`, the perftests package `30.0.1-3678.0c9a02e566.noble~perf.apply.time.txset.validation~perftests` (the first run of the change: `a33da62c4`, `30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests`). All 48 nodes reported the commit in their build label | | Build, before | Commit `8f7768be7`, the perftests package `30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests`. All 48 nodes reported that commit in their build label | | Overlay | The Rust overlay of this stack: QUIC, one connection per peer, 8 MiB initial congestion window. With the change it also stores tx sets on the local NVMe | | Ledger protocol | 30, upgraded from genesis during preparation |