diff --git a/README.md b/README.md index 9a5a5b1a5..d6f248438 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,23 @@ -# stellar-core: leader schedule and leader tx-set push (experimental branch) +# stellar-core: leader schedule, leader tx-set push and apply-time tx-set validation (experimental branch) **This branch is an experiment. It is not production code and must not run on a real network.** -It has no feature flag, no protocol gate, and it is not compatible with nodes that nominate. +The leader schedule and the tx-set push have no feature flag and no protocol gate, and the branch +is not compatible with nodes that nominate. -The latest experiment (the leader pushes its tx set) was measured on commit `8f7768be7` on -2026-11-11. "Before" is the previous experiment on this branch (the leader schedule alone), commit -`daa965f4c`, measured on 2026-11-10 on the same fixture. One run per setting on each side. +The benchmark is now **48 validators (16 organizations), 5,000 TPS, 1,000 ms target block time**. +The latest experiment (vote without validating the tx set, validate and execute at apply) was +measured there on commit `0c9a02e56`, the code of this branch, on 2026-11-11. "Before" is the +previous experiment on this branch (the leader's tx-set push), commit `8f7768be7`, measured on the +same day on the same fixture. One run per build. ## What this branch is It is the overlay-v2 stack up to [#5459](https://github.com/stellar/stellar-core/pull/5459) (Rust overlay over QUIC, millisecond close times, compressed tx-set responses, leader-only tx-set -construction) plus two experiments: +construction) plus three experiments: -1. **No nomination** ([#5495](https://github.com/stellar/stellar-core/pull/5495), the "before" - below). The SCP nomination phase is deleted and replaced by a leader schedule that every node - computes locally. +1. **No nomination** ([#5495](https://github.com/stellar/stellar-core/pull/5495)). The SCP + nomination phase is deleted and replaced by a leader schedule that every node computes locally. - **Schedule.** For slot N, the nodes of the transitive quorum are sorted by `SHA256(hash of ledger header N-1 || node id)`. The leader of round r is entry `(r-1) mod n`. - **Rounds.** A round is a ballot counter. The round's leader builds its value and goes straight @@ -23,11 +25,10 @@ construction) plus two experiments: passes proposal validation. A timeout (the ballot timeout settings) hands the slot to the next leader. 2. **The leader pushes its tx set** ([#5497](https://github.com/stellar/stellar-core/pull/5497), - the latest experiment). Before, a follower saw the leader's proposal, asked for the tx set and - received it about two round trips later. Now: + the "before" below). Before it, a follower saw the leader's proposal, asked for the tx set and + received it about two round trips later. With it: - **Push.** The round's leader sends the set it is about to propose to every connected peer as - soon as the set is built, without waiting for a request. The proposal follows about 70 ms - later. + soon as the set is built, without waiting for a request. - **Use.** A follower's overlay verifies a pushed set and keeps it under the hash of its own content. Core only gets it when an SCP statement names that hash, so a pushed set that does not match the proposal is never used. @@ -35,152 +36,229 @@ construction) plus two experiments: - **One flight.** Every QUIC connection starts with an 8 MiB congestion window, so that a set leaves in one flight instead of being paced over a round trip. - Details and limits: [docs/rust-overlay/txset-fetching.md](docs/rust-overlay/txset-fetching.md). - -Unchanged by both: the ballot protocol's accept, confirm and commit rules, value validation, apply, -ledger state and meta. +3. **Vote without validating the tx set; validate and execute at apply** + ([#5498](https://github.com/stellar/stellar-core/pull/5498), the latest experiment). Gated on + protocol 30; protocol 29 and older behave as before. + - **Vote.** A node votes for the leader's value as soon as it holds the tx set, without running + the set's validity check. Still checked before the vote: the value itself (signature, close + time, upgrades) and that the set was built on the last closed ledger. + - **Validate at apply.** The validity check runs when the ledger is applied. An invalid set + closes an **empty ledger**: no fee, no sequence bump, no result. The header keeps the + externalized value, upgrades in the value still apply, and history keeps the set so that a + replaying node reaches the same verdict. + - **Speculative execution.** When a node emits a PREPARE or CONFIRM for a value for the next + ledger, its apply thread validates and executes that ledger while the ballot is still running. + When the value externalizes, the executed ledger is committed; if another value externalizes, + it is dropped and the ledger is applied from scratch. `EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` + (default true) turns it off. + - **Durable tx set, written by the overlay.** The Rust overlay writes the compressed (zstd) set + to `/txsets` before it hands a set to Core, and before it sends anything that + refers to a set Core built. Core no longer writes the set to SQLite in front of its first + vote, and gets its sets back from the overlay after a restart. + - Details, safety argument and review findings: + [#5498](https://github.com/stellar/stellar-core/pull/5498). + +Unchanged by all three: the ballot protocol's accept, confirm and commit rules. For a valid tx +set, ledger state and meta are also unchanged; experiment 3 changes when a set is validated and +executed, and what an invalid set does to the ledger. ## Performance results -Target block time 1,000 ms, 3,000 TPS, 30 validators in 4 regions. +Target block time 1,000 ms, 5,000 TPS, 48 validators in 4 regions. -| | Before (`daa965f4c`) | With the tx-set push (`8f7768be7`) | +| | Before: tx-set push (`8f7768be7`) | With apply-time validation (`0c9a02e56`) | |---|---|---| -| Block time windows | **missed** the first window; the load stops there | **on target in all four windows**, on all 30 nodes | -| p75, average over 30 nodes | 1,386 ms (limit 1,200) | 1,091 / 1,096 / 1,109 / 1,115 ms | -| p99, average over 30 nodes | 1,599 ms (limit 2,000) | 1,227 / 1,243 / 1,253 / 1,290 ms | -| Time per ledger | 1.33 s | 1.05-1.07 s | -| Tx-set fetch as seen by a follower's Core | 217 ms | 16 ms | - -In short: before, the stages of one ledger added up to more than a second and the network backed -up. With the push the tx set is already at the follower when the proposal arrives, a ledger's -stages fit into about 0.9 s, and the network keeps up with the load at a 1 s target. - -**One caveat on correctness checks.** Every node's ledgers held 60 transactions more than the -2,880,000 offered: about 60 transactions were included in two ledgers. This is an artifact of the -test mode, in which ledgers are not applied (see [The inclusion check](#the-inclusion-check)). The -run was judged with the exact check and reported FAIL for that check alone; the check has since -been relaxed to allow such duplicates -([performance-testing#4](https://github.com/stellar-experimental/performance-testing/pull/4)). -All other checks passed. - -### 1,000 ms target, with the tx-set push +| Block time windows | **missed** the first window on all 48 nodes; the load stops there | **on target in all four windows**, on all 48 nodes | +| p75, average over 48 nodes | 1,461 ms (limit 1,200) | 1,081 / 1,132 / 1,118 / 1,142 ms | +| p99, average over 48 nodes | 1,638 ms (limit 2,000) | 1,292 / 1,413 / 1,598 / 1,595 ms | +| Time per ledger | 1.42 s | 1.03-1.08 s | +| Trigger to ledger closed (sum of the stages) | 1.37 s | 0.76-0.84 s | +| Run verdict | FAIL (block time) | FAIL (inclusion check only, see the caveat) | + +In short: before, the stages of one ledger added up to 1.37 s, every tx set was full and the +network backed up. With validation moved out of the vote and the ledger executed while the ballot +runs, a ledger's stages fit into about 0.8 s and the network keeps up with 5,000 TPS at a 1 s +target. + +**Caveat: the run with the change is a FAIL on the inclusion check, and on that check alone.** +Every node's ledgers held 4,800,060 transactions against 4,800,000 offered: 60 more (0.00125%), +the same on all 48 nodes. The check is an exact comparison. All other checks passed. The 60 are +not explained (see [The inclusion check](#the-inclusion-check)). + +**An earlier run of the same change found a bug.** The change was first measured without its last +commit (`a33da62c4`). The block times were about the same (p75 1,094 / 1,116 / 1,130 / 1,137 ms, all +windows on target), but one node fell out of consensus 15 s before the end of the load. The cause +was in the Core-overlay IPC code and is fixed in `0c9a02e56`; in the run reported here no node was +affected (see [The IPC bug](#the-ipc-bug-found-by-the-first-run)). + +### 1,000 ms target, with apply-time validation Load ran its full 900 s. Every node was on target in every window. | Window | p75, average (range over nodes) | p99, average (range over nodes) | |---|---|---| -| 1 | 1,091 ms (1,085-1,096) | 1,227 ms (1,205-1,273) | -| 2 | 1,096 ms (1,090-1,105) | 1,243 ms (1,227-1,272) | -| 3 | 1,109 ms (1,104-1,116) | 1,253 ms (1,231-1,360) | -| 4 (2 minutes) | 1,115 ms (1,109-1,125) | 1,290 ms (1,245-1,396) | +| 1 | 1,081 ms (1,078-1,087) | 1,292 ms (1,223-1,358) | +| 2 | 1,132 ms (1,118-1,150) | 1,413 ms (1,355-1,511) | +| 3 | 1,118 ms (1,109-1,126) | 1,598 ms (1,433-1,710) | +| 4 (2.2 minutes) | 1,142 ms (1,127-1,170) | 1,595 ms (1,413-1,712) | Limits: p75 below 1,200 ms, p99 at most 2,000 ms. Checks: -- **No fork**, all 30 nodes in sync, still closing ledgers after the load: passed. -- **Error counters zero** (invalid signatures, invariant failures, publish failures, invalid values, - lost sync), no node restarts: passed. +- **No fork**, all 48 nodes in sync, still closing ledgers after the load: passed. +- **Error counters zero** (invalid signatures, invariant failures, publish failures), no node lost + sync: passed. - **No wasted rounds:** every ledger was decided by the first leader. 0 leader timeouts, 0 ballot - timeouts, 0 empty transaction sets. -- **Inclusion:** 2,880,060 transactions in every node's ledgers against 2,880,000 offered. See - below. + timeouts, 0 empty transaction sets. No tx set was found invalid at apply. +- **Inclusion: failed.** 4,800,060 transactions in every node's ledgers against 4,800,000 offered. + See below. ### Before and after, stage by stage -5-minute means at the end of the first window, median over the 30 nodes. +5-minute means at the end of each run's first window, median over the 48 nodes. -| | Before (`daa965f4c`) | With the tx-set push (`8f7768be7`) | +| | Before (`8f7768be7`) | With apply-time validation (`0c9a02e56`) | |---|---:|---:| -| Ledgers per 5 minutes | 225 | 285 | -| Transactions per ledger | 3,750, the cap, on every ledger | 3,155 (what was offered; below the cap) | -| Leader tx-set build | 0.25 s | 0.15 s | -| Tx-set fetch (follower's Core, request to delivery) | 0.217 s | **0.016 s** | -| Tx sets a follower had to wait for from the network | every one | none in this window | -| Tx-set validation (follower) | 0.12 s | 0.11 s | -| Trigger to first ballot (includes the build, the fetch and validation) | 0.70 s | **0.42 s** | -| First ballot to externalize | 0.41 s | 0.33 s | -| Ledger close (simulated apply, 50 us per operation) | 0.20 s | 0.16 s | -| **Sum: trigger to close** | **1.30 s** | **0.91 s** | +| Ledgers per 5 minutes | 211-212 | 291-292 | +| Transactions per ledger | 6,250, the cap, on every ledger | 5,152 (what was offered; below the cap) | +| Leader tx-set build | 0.41 s | 0.24 s | +| Tx-set validation | 0.195 s, before the vote | 0.179 s, at apply, while the ballot runs | +| Trigger to first ballot (includes the build and the fetch; before, also validation) | 0.80 s | **0.41 s** | +| First ballot to externalize | 0.37 s | 0.30 s | +| Externalize to ledger closed (from node logs) | 0.21 s | **0.05-0.07 s** | +| **Sum: trigger to ledger closed** | **1.37 s** | **0.76-0.78 s** | +| Tx-set fetch (follower's Core, request to delivery) | 0.033 s | 0.051 s | | Leader timeouts, ballot timeouts, empty tx sets | 0, 0, 0 | 0, 0, 0 | -What the push did, over the whole load (totals of the 30 nodes): +In the later windows of the run with the change, trigger to first ballot is 0.47 s and the sum +0.82-0.84 s. Externalize to ledger closed is the range of the per-node medians of eight nodes over +the whole load. + +What the new parts did, over the whole load of the run with the change: | | | |---|---:| -| Tx sets pushed (one per peer) | 28,507 | -| Deliveries that were already at the follower when its Core asked | 28,181 | -| Requests that crossed a push (the leader did not send a second copy) | 64 | -| Pushes dropped or evicted by a follower's limits, write errors | 0 | -| Requests Core had to repeat after its 1 s retry interval | 0 | - -- **The tx-set round trips are gone.** On one follower whose logs were saved (us-west-2), all 817 - sets it received during the load arrived by push before its Core asked, a median 38 ms ahead of - the request (minimum 18 ms). Measured from four nodes in four regions, a follower's Core held the - set a median 126 ms after the leader finished building it (76-169 ms by region pair); before, that - took about 480 ms on a 147 ms link. -- **The two runs are in different regimes.** Before, a ledger could not close in less than 1.33 s, - so 3,000 TPS filled every set to the cap and the mempool backed up. Now ledgers hold what was - offered. That also makes stages the push does not touch cheaper: the leader pulls 3,200 - candidates instead of 7,700, and sets are smaller. -- **Two changes were measured together:** the push and the 8 MiB congestion window. One run cannot - say how much each contributes. -- **Block time is about 1.05 s although the stages sum to 0.91 s**, and the windows drifted up by - about 25 ms over the 16 minutes. Neither was investigated. +| Ledgers executed speculatively (total of the 48 nodes) | 46,194 | +| Speculatively executed ledgers that were committed | 46,206 (46,204 ledgers closed; the counters are scraped at slightly different times) | +| Executions dropped because another value externalized | 0 | +| Tx sets found invalid at apply (empty ledger) | 0 | +| Execution (validation plus simulated apply), mean | 0.33-0.35 s | +| Commit after externalize, mean | 0.037-0.041 s | +| Overlay's write of a tx set to disk (8 nodes, 6,941 writes) | median 4.3 ms, p99 10.8 ms, maximum 356 ms | +| Core's write of SCP state before a vote, now without the tx set | 2.2-2.6 ms mean | + +- **Every ledger was executed before it externalized or while it did.** No execution was wasted: + with one leader per round and no timeouts, the value a node votes for is the value that + externalizes. +- **The execution is slightly longer than the ballot.** It takes 0.33-0.35 s (0.18-0.19 s of + validation plus the simulated apply), the ballot 0.30 s. In the logs of eight nodes, the ledger + closed more than 50 ms after externalize on 60% of the ledgers and more than 100 ms after on 21% + (a commit alone takes about 40 ms). Before, it was about 207 ms on every ledger. +- **The gain from speculation is an upper bound.** Apply is simulated by a sleep that uses no CPU, + so running it beside the ballot is free here. Real execution would compete with SCP processing + and signature verification. +- **The two builds do not simulate the same work per ledger.** Before, the sleep starts with the + apply and absorbs the commit. With the change it sits in the execution, and validation and the + commit come on top of it. +- **The two runs are in different regimes**, as in the earlier comparison. Before, a ledger could + not close in less than 1.42 s, so 5,000 TPS filled every set to the cap and the mempool backed + up. Now ledgers hold what was offered. That alone shortens the leader's build and the sets. One + run on each side cannot separate it from the effect of the change. +- **The push mostly loses its race now.** The leader's proposal leaves earlier relative to the + push, so a follower's request usually reaches the leader while the push is still in flight + (38,883 such requests against 6,240 sets that were already in place; before: 17 against 13,914). + The leader does not send a second copy, and the fetch costs about 0.02 s more per ledger. Why + the proposal overtakes the push is inferred from the removed database write, not timed. +- **Block time is 1.03-1.08 s although the stages sum to about 0.8 s.** The windows drifted up + over the 16 minutes: p75 by about 60 ms, p99 from 1,292 to about 1,600 ms. Neither was + investigated. +- **The two runs of the change agree.** Without the IPC fix (`a33da62c4`): p75 1,094-1,137 ms, + 288-290 ledgers in the first 5 minutes, trigger to first ballot 0.42 s, first ballot to + externalize 0.30 s, every ledger executed speculatively and committed. ### The inclusion check -The test compares the transactions in every node's ledgers with the number offered. All 30 nodes -held exactly 60 more (the same on every node, so no fork). The generators had submitted exactly -2,880,000 and every node had pulled each transaction once, so about 60 transactions (0.002%) were -included in two ledgers. +The test compares the transactions in every node's ledgers with the number offered. In the run +with the change all 48 nodes held exactly 60 more (the same on every node, so no fork). The +generators had submitted exactly 4,800,000 and none was rejected. + +- **The same excess, exactly 60, appeared on the earlier fixture** in the run with the tx-set push + (30 validators, 3,000 TPS: 2,880,060 against 2,880,000). +- **The explanation given then was an inference**: the test runs in "overlay-only" mode, where + ledgers are not applied and sequence numbers never advance, so a copy of a transaction that + reaches a far node after its ledger closed is admitted again and included a second time. The + logs carry no transaction hashes, so this was never shown. +- **It does not predict the same number** at a different rate and network size. Which + transactions are counted twice, or whether something else adds 60, is open. +- The "before" run and the first run of the change did not get to this check (the load ended + early or a generator failed). In the first run of the change the 47 healthy nodes held + 4,798,690, fewer than offered; the node that dropped out kept generating for 15 s while nothing + left it. +- A change that lets the check allow up to 1 in 10,000 more than offered is proposed + ([performance-testing#4](https://github.com/stellar-experimental/performance-testing/pull/4)), + not merged. Under it this run would pass. + +### The IPC bug found by the first run + +- **What happened.** In the first run of the change (`a33da62c4`), 15 s before the end of the + load, Core on one node logged "Overlay IPC connection closed" and stopped reading from its + overlay process. Both processes kept running and the socket between them was still open, with + unread data on Core's side. The node heard nothing further from the network and lost sync. Two + more nodes did the same two minutes after the load. +- **When.** All three times within 26 ms of a step of a history checkpoint publish. +- **Cause (inferred; the logs carry no error code).** Core's IPC reader treated any failed `recv` + as a closed connection, including one interrupted by a signal (`EINTR`). stellar-core handles + `SIGCHLD`, and a checkpoint publish runs child processes. +- **Fix.** `0c9a02e56`: an interrupted read or write on the IPC socket is retried. A unit test + interrupts a blocked read with a signal and fails without the fix. +- **In the run with the fix** all 48 nodes were checked fourteen times, from before the load until + 17 s before the network was terminated: none logged the warning and none had a stuck socket. + Every node published 44 checkpoints in that time (about 2,100 in the network); the three events + of the first run fell within its first 19 checkpoints per node (about 900). +- **This is not a proof.** The fix logs nothing when it retries, so there is no count of + interrupted reads it survived, and the "before" build, with the unfixed reader, had no such + event in its 30 minutes either. Whether the change makes the event more likely is not known. -Why this happens in the test and not on a network that applies ledgers: - -1. A leader that is not backed up proposes everything in its mempool, including a transaction - that was submitted a moment ago and is still being flooded to the far nodes (up to about 0.5 s). -2. With the push, a follower closes the ledger about 0.5 s after the leader built its set, so now - and then a far node closes the ledger before its own copy of that transaction has arrived. -3. The overlay's mempool removes a ledger's transactions when the ledger closes and keeps no memory - of them, on purpose. The late copy is admitted as a new transaction. -4. On a real network Core rejects that copy when the node next builds a set, because its sequence - number was already used. The test runs in "overlay-only" mode: ledgers are not applied, sequence - numbers never advance, and that check is switched off. So the node includes the transaction a - second time. +### Where a ledger's time goes now -That this path produced the 60 is inferred from the counters; the logs carry no transaction hashes. -Before, a follower closed the ledger about a second after the leader's build, well after flooding -had finished, so the two never met: the completed "before" run at 2,000 ms held exactly 2,880,000. +What is left of the 0.76-0.84 s from trigger to ledger closed: -It is accepted as an artifact of overlay-only mode. The check now allows up to 1 in 10,000 -transactions more than offered and reports how many there were; fewer than offered, more than -that, and nodes that disagree still fail. +1. **Trigger to first ballot: 0.41-0.47 s.** Most of it is the leader's build (0.24-0.25 s), on + the critical path. The first-round leader also builds once before the trigger and discards + that set unless it was capacity limited (540-568 builds for 279-292 ledgers). The tx-set fetch + adds 0.05 s. +2. **The ballot: 0.30 s.** Nothing is validated in it any more; it is message delays between four + regions. +3. **Externalize to ledger closed: 0.05-0.07 s.** The commit (0.04 s), plus the end of the + execution on more than half of the ledgers. -### Not rerun: the 2,000 ms target +The execution itself (0.33-0.35 s) runs beside the ballot. -The 2,000 ms target was measured for the "before" commit only: PASS in all four windows (p75 -2,066-2,109 ms, p99 2,286-2,324 ms), with exactly 2,880,000 transactions on every node, and 0.19 s -less from trigger to externalize than with nomination (1.31 s against 1.50 s on #5459). The details -are in [#5495](https://github.com/stellar/stellar-core/pull/5495). It was not repeated with the -push. +### Earlier fixture: 30 validators, 3,000 TPS -### Where a ledger's time goes now +The first two experiments were measured on a smaller fixture (10 organizations, 30 validators, +3,000 TPS, 1,000 ms target, 2026-11-10 and 2026-11-11). It is no longer the benchmark. -With the tx-set round trips gone, what is left of the 0.91 s from trigger to close at 1,000 ms: +| | Leader schedule alone (`daa965f4c`) | With the tx-set push (`8f7768be7`) | +|---|---|---| +| Block time windows | missed the first window | on target in all four windows, on all 30 nodes | +| p75, average over 30 nodes | 1,386 ms | 1,091 / 1,096 / 1,109 / 1,115 ms | +| Time per ledger | 1.33 s | 1.05-1.07 s | +| Tx-set fetch as seen by a follower's Core | 217 ms | 16 ms | +| Trigger to ledger closed (sum of the stages) | 1.30 s | 0.91 s | -1. **The ballot: 0.33 s.** Followers validate before they vote, and the ballot then needs four - one-way trips. -2. **The leader's build: 0.15 s**, on the critical path. With the trigger mostly on time, the - first-round leader also builds once before the trigger and discards that set unless it was - capacity limited (562 builds for 285 ledgers). -3. **Validation: 0.11 s.** -4. **About 70 ms between the leader finishing its set and its proposal leaving.** SCP state, - including the tx set, is written to the database before each broadcast. That this write accounts - for most of the 70 ms is inferred, not timed. The push already leaves during this time. -5. **Ledger close: 0.16 s**, almost all of it the simulated apply. +- With the push, every node's ledgers held 60 transactions more than the 2,880,000 offered. The + run was judged with an exact comparison and reported FAIL for that check alone (see + [The inclusion check](#the-inclusion-check)). +- A 2,000 ms target was measured for the leader schedule alone: PASS in all four windows (p75 + 2,066-2,109 ms). +- The full results, stage tables and the measurement setup of that fixture are in + [the README as of #5497](https://github.com/stellar/stellar-core/blob/6d5fc9e814abee2b7e93553b3903b56929bedf91/README.md) + and in [#5495](https://github.com/stellar/stellar-core/pull/5495). ## How it was measured -Both runs (before and after) use the `block_time_check` mission of perfnet, SDF's tooling for +All runs use the `block_time_check` mission of perfnet, SDF's tooling for performance networks on AWS. Each run launches a fresh network, brings it to the target settings, runs the load, judges it, and terminates the network. @@ -188,22 +266,22 @@ runs the load, judges it, and terminates the network. | | | |---|---| -| Validators | 30, all validating, no watchers | -| Organizations | 10 home domains of 3 validators each, all `HIGH` quality | -| Quorum set | Generated by stellar-core from the home domains: 7 of 10 organizations, 2 of 3 validators within each | -| Regions | 4 AWS regions: eu-central-1 (8 nodes), ap-southeast-1 (8), us-east-1 (7), us-west-2 (7) | -| Placement | 9 organizations have their three validators in three different regions; one has two in us-east-1 and one in ap-southeast-1. The layout was identical in both runs | -| Connectivity | Full mesh: every node is configured with the other 29 as its only peers | -| Round-trip times seen from us-west-2 | us-east-1 about 60 ms, eu-central-1 about 147 ms, ap-southeast-1 about 180 ms | +| Validators | 48, all validating, no watchers | +| Organizations | 16 home domains of 3 validators each, all `HIGH` quality | +| Quorum set | Generated by stellar-core from the home domains. Its rule gives 11 of 16 organizations, 2 of 3 validators within each (computed from the rule; not read back from a node) | +| Regions | 4 AWS regions, 12 nodes each: eu-central-1, us-west-2, ap-southeast-1, us-east-1 | +| Placement | 14 organizations have their three validators in three different regions; 2 have them in two regions. All runs used the same layout seed | +| Connectivity | Full mesh: every node is configured with the other 47 as its only peers | +| Round-trip times seen from us-west-2 (measured on the earlier fixture, same regions) | us-east-1 about 60 ms, eu-central-1 about 147 ms, ap-southeast-1 about 180 ms | ### Hardware and software | | | |---|---| | Instance type | c6id.2xlarge: 8 vCPU (Intel Ice Lake, with SHA extensions), 16 GiB RAM, local NVMe | -| Build, with the push | Commit `8f7768be7`, the perftests package `30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests`. All 30 nodes reported that commit in their build label | -| Build, before | Commit `daa965f4c`, the perftests package `30.0.1-3674.daa965f4ce.noble~perf.leader.schedule.no.nomination~perftests` | -| Overlay | The Rust overlay of this stack: QUIC, one connection per peer. With the push, every connection starts with an 8 MiB congestion window (quinn's default before) | +| Build, with apply-time validation | Commit `0c9a02e56`, the perftests package `30.0.1-3678.0c9a02e566.noble~perf.apply.time.txset.validation~perftests` (the first run of the change: `a33da62c4`, `30.0.1-3677.a33da62c49.noble~perf.apply.time.txset.validation~perftests`). All 48 nodes reported the commit in their build label | +| Build, before | Commit `8f7768be7`, the perftests package `30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests`. All 48 nodes reported that commit in their build label | +| Overlay | The Rust overlay of this stack: QUIC, one connection per peer, 8 MiB initial congestion window. With the change it also stores tx sets on the local NVMe | | Ledger protocol | 30, upgraded from genesis during preparation | | Database | SQLite | @@ -216,8 +294,8 @@ ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING=true UNSAFE_QUORUM=true FAILURE_SAFETY=0 GENESIS_TEST_ACCOUNT_COUNT=1000000 -TARGET_PEER_CONNECTIONS=29 -MAX_ADDITIONAL_PEER_CONNECTIONS=87 +TARGET_PEER_CONNECTIONS=47 +MAX_ADDITIONAL_PEER_CONNECTIONS=141 PREFERRED_PEERS_ONLY=true FLOOD_DEMAND_BACKOFF_DELAY_MS=1000 INVARIANT_CHECKS=[] @@ -225,48 +303,49 @@ TESTING_IGNORE_LEDGER_TIME_UPGRADE_BOUNDS=true TESTING_MAX_CLASSIC_BYTE_ALLOWANCE=1048576 TESTING_MAX_SOROBAN_BYTE_ALLOWANCE=9437184 LOADGEN_PREGENERATED_TRANSACTIONS_FILE="/var/lib/stellar-core/loadgen-txs.xdr" -OP_APPLY_SLEEP_TIME_DURATION_FOR_TESTING=[50] # [25] in the 2,000 ms run +OP_APPLY_SLEEP_TIME_DURATION_FOR_TESTING=[30] OP_APPLY_SLEEP_TIME_WEIGHT_FOR_TESTING=[100] ``` +`EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY` was on (the default) in the runs with the change. `EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` was off (the default). Catchup and automatic restart are off, so a node that crashes or falls behind stays that way and fails the run. ### Network settings (applied by upgrade before the load) -| Setting | 1,000 ms runs (before and after) | 2,000 ms run (before only) | -|---|---:|---:| -| Target ledger close time | 1,000 ms | 2,000 ms | -| SCP timeouts (ballot and nomination, initial and increment) | 2,000 ms | 2,000 ms | -| Maximum transactions per ledger (125% of the offered load per block) | 3,750 | 7,500 | -| Maximum tx-set size in bytes | 10 MiB (1 MiB classic + 9 MiB Soroban) | same | -| Soroban limits | raised to fit the load | raised to fit the load | +| Setting | All runs | +|---|---:| +| Target ledger close time | 1,000 ms | +| SCP timeouts (ballot and nomination, initial and increment) | 2,000 ms | +| Maximum transactions per ledger (125% of the offered load per block) | 6,250 | +| Maximum tx-set size in bytes | 10 MiB (1 MiB classic + 9 MiB Soroban) | +| Soroban limits | raised to fit the load | The leader timer of this branch uses the ballot timeout, so a round's leader had 2 s (4 s with its -one extension) in every run. +one extension). ### Load | | | |---|---| | Transactions | Stellar Asset Contract (SAC) payments, pregenerated; no classic transactions | -| Rate | 3,000 TPS offered: every validator generates 100 TPS | -| Accounts | 1,000,000 genesis accounts, 33,333 per generator | +| Rate | 5,000 TPS offered: every validator generates about 104 TPS | +| Accounts | 1,000,000 genesis accounts, 20,833 per generator | | Duration | 60 s warm-up, then 900 s measured | -| Apply | Skipped ("overlay-only" mode) and replaced by a sleep per operation, sized as a 150 ms budget per ledger at the offered load: 50 us per operation at 1,000 ms, 25 us at 2,000 ms | +| Apply | Skipped ("overlay-only" mode) and replaced by a sleep per operation, sized as a 150 ms budget per ledger at the offered load: 30 us per operation | -Command (before and after differ only in the build): +Command (the runs differ only in the build): ``` block_time_check.py \ - --stellar-core-version 30.0.1-3675.8f7768be7a.noble~perf.leader.txset.push~perftests \ - --block-time-ms 1000 \ - --soroban-tps 3000 --soroban-type sac --classic-tps 0 \ + --stellar-core-version 30.0.1-3678.0c9a02e566.noble~perf.apply.time.txset.validation~perftests \ + --org-count 16 --block-time-ms 1000 \ + --soroban-tps 5000 --soroban-type sac --classic-tps 0 \ --simulate-apply-budget-ms 150 --load-duration-s 900 \ --instance-type c6id.2xlarge --save-logs always ``` -Defaults that matter: 10 organizations, validators as generators, `--scp-timeout-ms 2000`, +Defaults that matter: validators as generators, `--scp-timeout-ms 2000`, `--max-tx-set-bytes 10485760`, 1,000,000 genesis accounts, seed 0. ### How a run is judged @@ -275,26 +354,49 @@ Defaults that matter: 10 organizations, validators as generators, `--scp-timeout 5-minute windows after the warm-up. - **PASS** needs, on every node in every window: p75 within [0.8 T, 1.2 T), p99 at most 2 T, and at least one close per 2 T on average, where T is the target block time. It also needs every node's - ledgers to hold the offered transactions, zero error counters, all nodes in sync, no fork, and - the network still closing ledgers afterwards. -- "Hold the offered transactions" was an exact comparison when these runs were judged. It now - allows up to 1 in 10,000 more, for the duplicates overlay-only mode lets through (see - [The inclusion check](#the-inclusion-check)). + ledgers to hold exactly the offered transactions, zero error counters, all nodes in sync, no + fork, and the network still closing ledgers afterwards. - The load stops at the first window that misses, so a FAIL can rest on one window (as in the - "before" run). + "before" run). The inclusion check is skipped when the load ends early or a generator fails + (the "before" run and the first run of the change). ### Limits of these numbers -- One run on each side. The "before" numbers at 1,000 ms come from a single 5-minute window, and - from a commit that differs from this branch's base in formatting only. -- The push and the 8 MiB congestion window were not measured separately. -- Apply is simulated, so these are consensus and overlay numbers, not end-to-end ledger close. +- One run on each side, plus the first run of the change without the IPC fix. The "before" + numbers come from a single 5-minute window. +- The run with the change is a FAIL on the inclusion check, and the 60 extra transactions are not + explained. +- Apply is simulated by a sleep, so these are consensus and overlay numbers, not end-to-end ledger + close. The gain from speculative execution is an upper bound, and the two builds do not sleep + for the same part of the apply (above). +- `herder.txset.validate` times the pre-vote check on the main thread in the "before" build and the + apply-time check on the apply thread in the build with the change. - `scp.timing.trigger-to-ballot` records 0 when a follower votes before its own trigger fires. -- The timings between a leader and its followers come from live logs of four nodes (one per - region) and the saved logs of one node. +- Externalize-to-close and the tx-set store timings come from the logs of eight nodes (two per + region) in the run with the change, and of four nodes in the "before" run. +- No run has measured an invalid tx set, a dropped speculative execution or a restart from the + tx-set store on a network; those paths are covered by unit tests only. ## Known limitations of the prototype +Apply-time validation and speculative execution: +- A nondeterministic validity check used to cost liveness at most. It now forks the ledger. The + check's node-local inputs are the binary's protocol version, the network passphrase and, in test + builds, the load generation flags; validators that differ in those would disagree. +- A leader can get an invalid set externalized. The ledger is empty, and the set's transactions + are still removed from every mempool, so a faulty leader can drop pending transactions at no + cost whenever it leads. +- Downstream consumers see new shapes: a LedgerCloseMeta with a non-empty tx set and no + transaction meta, and a history `transactions` entry without a `results` entry. +- If the overlay drops a set Core built (hash mismatch or encode error, both Core bugs), Core + still believes the set is stored. +- A value whose execution was dropped is not executed speculatively again. `ledger.ledger.close` + leaves out the time a finished execution waits for externalize. +- A parked execution keeps a SQL transaction open for the length of the ballot. It is not + confirmed that no other writer on the main database can make its commit fail. +- Speculative execution was never measured with real transaction execution. +- There is no compatibility work beyond the protocol gate. + Tx-set push: - The push is not relayed. A node that is not connected to the leader requests the set from the peer that relayed the proposal. @@ -303,7 +405,8 @@ Tx-set push: - The limits on unrequested sets can be stepped around by a peer that sends SCP envelopes naming unknown tx sets, and the 8 MiB congestion window applies to every stream of a connection. Both are prototype settings. -- Transactions can be included twice in overlay-only mode (above). +- Both runs that reached the inclusion check with the push held 60 transactions more than offered + (see [The inclusion check](#the-inclusion-check)). Not explained. Leader schedule: - The leader schedule's node set is whatever transitive quorum each node currently knows. It @@ -313,10 +416,20 @@ Leader schedule: - Leader grinding through the header hash and fairness between organizations are out of scope. - The leader timer extension is broader than intended, and a proposal that arrives just after a timeout is wasted. Neither occurred in these runs. -- A build with `--disable-tests` does not compile at this commit (test headers reached from - `herder/HerderImpl.cpp`). +- A build with `--disable-tests` did not compile with the leader schedule (test headers reached + from `herder/HerderImpl.cpp`) and was not tried since. ## Building and the rest of the documentation Unchanged from stellar-core: see [INSTALL.md](./INSTALL.md), [CONTRIBUTING.md](./CONTRIBUTING.md) -and [`./docs`](./docs). The full test suite passes on `8f7768be7`, also with `--all-versions`. +and [`./docs`](./docs). + +Test status of this branch's code (`0c9a02e56`, the same tree as the merge of #5498): +- The full test suite passes (615 test cases). +- With `--all-versions` the suite passes on `a33da62c4`, the commit before the IPC fix. On + `0c9a02e56` one test case of the leader-schedule change failed ("PendingEnvelopes + recvSCPEnvelope", at protocol version 4): its helper tries 100 random peers and needs one that + does not lead round 1, and for that day's test seed and that partition's node key the local + node sorts last in the schedule, so every peer leads. The binary of #5495 alone fails the same + way with the same seed and instance number, and the test passes with instance number 0. It is + a flaw of that test helper, not of the later changes. Not fixed.