diff --git a/README.md b/README.md index 433bc284c..dc9994125 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# stellar-core: leader schedule with terms, leader tx-set push, apply-time tx-set validation, transactions sent to the leaders only (experimental branch) +# stellar-core: leader schedule with terms, leader tx-set push, apply-time tx-set validation, transactions sent to the leaders only, next tx set selected during the previous ballot (experimental branch) **This branch is an experiment. It is not production code and must not run on a real network.** The leader schedule and the tx-set push have no feature flag and no protocol gate, and the branch @@ -22,13 +22,22 @@ sleeps instead), the same 90 validators at 800 ms hold the target at 4,000 TPS ( 8,000 TPS. One run per point. Results and the ranked bottlenecks: [Real apply at 90 validators and 800 ms](#real-apply-at-90-validators-and-800-ms). +**With the transaction meta off, a bucket list account cache and 256-byte bucket index pages** +(2026-11-13, real execution, one run per point), the 90 validators hold 800 ms at 6,000 TPS +(block time p75 805 to 808 ms, p99 872 to 880 ms) and take 0.87 s per ledger at 8,000 TPS. The +sixth experiment (the leader selects its next tx set while the current ledger is in its ballot, +and pulls less for a build) shortens the leader's tx set build at 8,000 TPS from 176 to 121 ms. +The time per ledger goes from 873 to 858 ms, which one run on each side does not establish. No +tx set was invalid or short. Results, and why the shorter build does not show in the block time: +[Real apply with the meta off, and the tx set build off the critical path](#real-apply-with-the-meta-off-and-the-tx-set-build-off-the-critical-path). + The earlier benchmark (48 validators, 5,000 TPS, 1,000 ms, 4-core instances) is kept below. ## What this branch is It is the overlay-v2 stack up to [#5459](https://github.com/stellar/stellar-core/pull/5459) (Rust overlay over QUIC, millisecond close times, compressed tx-set responses, leader-only tx-set -construction) plus five experiments: +construction) plus six experiments: 1. **No nomination** ([#5495](https://github.com/stellar/stellar-core/pull/5495)). The SCP nomination phase is deleted and replaced by a leader schedule that every node computes locally. @@ -86,8 +95,8 @@ construction) plus five experiments: - **One build.** The first-round leader builds once, a lead time before its trigger, pushes the set as soon as it exists and proposes it unchanged at the trigger. 5. **Transactions go to the upcoming leaders only** - ([#5501](https://github.com/stellar/stellar-core/pull/5501), the latest experiment). No - protocol gate and no version negotiation. + ([#5501](https://github.com/stellar/stellar-core/pull/5501)). No protocol gate and no version + negotiation. - **Targets.** A node that accepts a transaction no longer advertises it to every peer. Its overlay sends the body straight to the leaders of the node's current round and of the next one for the next ledger and, one ledger before a new leader term starts, to that term's @@ -103,11 +112,41 @@ construction) plus five experiments: connection key, signed by the node key. - **Cost.** Followers no longer hold the transactions before the leader's set arrives, so they verify signatures when they check the set at apply instead of in the background. - -Unchanged by all five: the ballot protocol's accept, confirm and commit rules. For a valid tx +6. **The leader's tx set build off the critical path** + ([#5505](https://github.com/stellar/stellar-core/pull/5505), the latest experiment). No protocol change: the sets are ordinary + tx sets, and every node validates them at apply as before. + - **A smaller build.** A build pulls the set's capacity plus 10% from the mempool + (`EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT`); it pulled twice the capacity. A phase that lost + candidates (invalid ones, or a second transaction of an account) and is short pulls again, + twice as deep, as long as a pull brings it something new, up to three pulls. The structural + check of the selection runs beside the XDR write and ends before the set goes to the + overlay. A leader that executes its own vote no longer reads the written set back: that + execution validates it. If the execution does not start, the read-back runs in the + background. + - **Early selection.** The node first in line to lead ledger N+1 selects N+1's set on + background threads while ledger N is in its ballot. It starts from N's tx set, which it + holds at its first vote: it assumes N's transactions applied (sequence numbers and fees), + validates its mempool's top candidates on that view and selects the phases. + - **Completion at the close.** When N has closed with the value the selection was made for, + the main thread validates every selected transaction on the closed ledger, for the set's + real close time, and removes what does not hold. It then tops the set up with what the + mempool received since, validated the same way (the Soroban phase is extended in place, the + classic phase is selected again), computes the base fees, runs the structural check and + writes the set. A selection for another value or other limits, after an upgrade, or one that + is not ready is dropped, and the set is built from scratch. + - `EXPERIMENTAL_EARLY_TX_SET_SELECTION` (default true) turns the early selection off. + - It replaces the selection of [#5503](https://github.com/stellar/stellar-core/pull/5503) + (closed), which waited for the leader's own execution of ledger N, and keeps that attempt's + early execution: a leader starts executing the set it prepared as soon as it is built. + - Two supporting changes came with it: a hit in the bucket list's account cache is read under + the shared lock, and the load generator has the two test-only modes of the real-apply runs + (`sac_setup`, `sac_payment`). + +Unchanged by all six: the ballot protocol's accept, confirm and commit rules. For a valid tx set, ledger state and meta are also unchanged; experiment 3 changes when a set is validated and executed, and what an invalid set does to the ledger; experiment 4 changes who leads; experiment -5 changes which transactions reach a leader, and which submissions a node refuses. +5 changes which transactions reach a leader, and which submissions a node refuses; experiment 6 +changes which candidates a leader looks at and when it selects them, not the checks a set passes. ## Performance results @@ -302,7 +341,7 @@ Ranked by their share of the leader's cycle (90 validators, 16 clusters, local N | # | Step | At 8,000 TPS (1.16 s) | At 6,000 TPS (0.86 s) | Grows with the load | Most plausible fix | |---|---|---:|---:|---|---| | 1 | Ballot, first ballot to externalize | 310 ms, 27% | 307 ms, 36% | no | None found here. It is the floor of the cycle, and the window that speculative execution hides in | -| 2 | Tx set build on the leader's main thread. It needs the previous ledger closed, and once the cycle has no slack the trigger is already due when it ends | 261 ms, 23% | 137 ms, 16% | yes; between 6,000 and 8,000 TPS faster than the load, because of the backlog | Build the next set while the current ledger is in its ballot. Tried once as [#5503](https://github.com/stellar/stellar-core/pull/5503) with simulated apply (slower than the base in its one run, closed); not measured with real apply | +| 2 | Tx set build on the leader's main thread. It needs the previous ledger closed, and once the cycle has no slack the trigger is already due when it ends | 261 ms, 23% | 137 ms, 16% | yes; between 6,000 and 8,000 TPS faster than the load, because of the backlog | Build the next set while the current ledger is in its ballot. Tried once as [#5503](https://github.com/stellar/stellar-core/pull/5503) with simulated apply (slower than the base in its one run, closed). Experiment 6 does it from the current ledger's tx set: measured with real apply in the next section (build 31% shorter, block time 2%) | | 3 | Commit, always after externalize | 244 ms, 21% | 177 ms, 21% | linear, 31 to 35 us per payment | No debug meta on a validator: **measured, 43% less commit per payment** (below) | | 4 | Between the proposal and the followers' vote, beyond the ballot: the set's compression, storage, push, fetch, decoding and hashing, and the main threads' queues | 201 ms, 17% | 142 ms, 17% | yes | Not known. Only the fetch has a timer (88 and 58 ms of it); the rest needs timers first | | 5 | Execution beyond the ballot (from about 4,700 payments per ledger on) | 81 ms, 7%; on a follower 167 ms | 46 ms, 5% | linear, 65 us per payment | Meta off and the account cache: measured (below). Then the serial account prefetch and the tx set validation | @@ -363,14 +402,125 @@ type at 8,000 TPS; apply cost per node does not depend on the network size): prefetch falls by 70%, but an account load on a worker takes twice as long, because a cache hit takes an exclusive lock, and the tx set validation gives back what it gained. Execution is 9% shorter and the block time does not move. -- **Not measured: either setting at 90 validators.** Scaled per payment, meta off would take - about 105 ms off the commit at 8,000 TPS and 70 ms at 6,000 TPS (about 30 and 20 ms of that is - the test build's copy), which puts 6,000 TPS near 800 ms and leaves 8,000 TPS above 1 s. +- **Either setting at 90 validators** was measured after this was written: see the next + section. The estimate made before, scaled per payment: meta off would take about 105 ms off + the commit at 8,000 TPS and 70 ms at 6,000 TPS (about 30 and 20 ms of that is the test build's + copy), which puts 6,000 TPS near 800 ms and leaves 8,000 TPS above 1 s. Measured: 801 ms at + 6,000 TPS and 873 ms at 8,000 TPS. - **At 90 validators one low-priority core thread is busy 100% of a core** in every run, independent of the load (3% at 24 validators). Most likely the quorum intersection checker, which is on by default; inferred from the code, not profiled. It does not sit in the block time. +### Real apply with the meta off, and the tx set build off the critical path + +Runs of 2026-11-13. Every run of this section executes the load, with the transaction meta off +(`METADATA_DEBUG_LEDGERS=0`; in these test builds also `DISABLE_TX_META_FOR_TESTING=true`), +16 clusters, an 800 ms target, m6id.8xlarge with core's data on the local NVMe disk, and the +load and limits of the real-apply runs above. One run per point. + +**Bucket list settings**, on 6 validators at 8,000 TPS offered (apply cost per node does not +depend on the network size). Means over the whole load, medians over the 6 nodes. The table of +the two settings above averages over the load's last five minutes, so compare inside this table +only; its first two rows are the same two runs. + +| Settings beyond meta off | Execution / commit / whole close, per payment | Tx set validation at apply | Time per ledger: mean / p99 / longest | Outcome | +|---|---:|---:|---:|---| +| none | 44.3 / 15.8 / 60.1 us | 77 ms | 802 / 938 / 1,042 ms | all 2,880,000 payments included | +| 4 GB account cache, a hit takes an exclusive lock | 40.2 / 16.4 / 56.5 us | 101 ms | 814 / 1,061 / 1,231 ms | all included (two generators timed out on their last payments) | +| Every live bucket in memory (`BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT=0`) | **33.7** / 17.6 / 51.3 us | 63 ms | 829 / **1,637** / 2,195 ms | all included; **the close stalls** | +| 256-byte index pages (`BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT=8`; the default is 14, 16 KiB) | 38.0 / 16.0 / 54.1 us | 66 ms | 816 / 1,047 / 2,792 ms | all included; one ledger took 2.0 s on every node | +| 4 GB account cache, a hit is read under the shared lock (this branch), and 256-byte index pages | 37.3 / 17.0 / 54.3 us | 73 ms | **802 / 918 / 1,194 ms** | all included, **verdict PASS** | + +- **Every live bucket in memory is the fastest apply and not usable as it is.** Execution per + payment is 24% lower than with the meta off alone, but the time from consensus to the close is + 0.4 to 0.5 s at every 8th ledger and 0.7 to 1.7 s at every 32nd, and grows during the run + (median 186 ms). Inferred from the code, not profiled: with this setting a merged bucket is + loaded into memory entry by entry when it is indexed, and the deeper buckets grow with the run. +- **256-byte index pages: execution 14% lower, the whole close 10% lower, no periodic stalls.** +- **The account cache no longer slows the tx set validation** once a hit is read under the + shared lock (101 ms with the exclusive lock, 73 ms with the shared one, 66 to 77 ms without a + cache). On top of the small index pages it adds nothing these runs can show (37.3 against + 38.0 us of execution per payment). +- The 90-validator runs below use the last row's settings. + +**90 validators**, 30 organizations, as in the real-apply runs above. "Base" is this branch +before experiment 6, with the load generator modes and the cache's shared lock (`54260eafb`). +The meta-on columns are the 16-cluster runs of the table above: another build, and core's +default bucket settings. + +| | Meta on, 6,000 TPS | **Base, 6,000 TPS** | Meta on, 8,000 TPS | **Base, 8,000 TPS** | **Experiment 6, 8,000 TPS** | +|---|---:|---:|---:|---:|---:| +| Payments per ledger | 5,150 | 4,800 | 8,000 (the cap) | 7,085 | 7,076 | +| **Time per ledger: mean / p75 / p99** | 0.86 s | **801 / 806 / 868 ms** | 1.16 s | **873 / 928 / 1,100 ms** | **858 / 908 / 1,107 ms** | +| Block time p75 / p99 of the median node (limits 960 / 1,600 ms), first and second window | 927 / 1,164 ms | 805 / 872 and 808 / 880 ms | 1,236 / 1,482 ms | 946 / 1,143 and 888 / 1,125 ms | 937 / 1,151 and 939 / 1,151 ms | +| **Leader: tx set build** | 137 ms | 122 ms | 261 ms | **176 ms** | **121 ms** | +| Leader: set built to consensus | 449 ms | 392 ms | 511 ms | **468 ms** | **514 ms** | +| Follower: tx set fetch | 58 ms | 27 ms | 88 ms | **98 ms** | **155 ms** | +| Consensus to ledger closed: leader / follower | 223 / 242 ms | 140 / 136 ms | 325 / 453 ms | 196 / 220 ms | 198 / 210 ms | +| Execution / commit, per payment | 65 / 34 us | 40.0 / 16.4 us | 65 / 31 us | 40.0 / 16.1 us | 39.2 / 16.3 us | +| Of the ballot, left after execution | 14 ms | 78 ms | 0 | 9 ms | 13 ms | +| Payments included, of those offered | (inclusion check not run) | 2,159,941 of 2,160,000 | about 6,860 TPS | 2,879,853 of 2,880,000 | **2,880,000 of 2,880,000** | +| Outcome | inside the limits, 16% over the target and rising | **on target on all 90 nodes**; verdict FAIL for the 59 missing payments | window missed on all 90 nodes | inside the limits, **9% over the target**; verdict FAIL for the 147 missing payments | inside the limits, 7% over the target; **verdict PASS** | + +- **With the meta off and these settings the base holds 800 ms at 6,000 TPS with room** (a + quarter of the ballot is left after execution) **and takes 0.87 s per ledger at 8,000 TPS**, + where it took 1.16 s. Per payment, execution is 40 us instead of 65 and the commit 16 us + instead of 31 to 35. +- **At 8,000 TPS the base has no slack left**: 9 ms of the ballot remain after execution, and + the build starts when the previous ledger has closed (trigger to first ballot 0.30 s; 0.09 s at + 6,000 TPS). The leader's cycle is build 176 ms, set built to consensus 468 ms, consensus to + close 196 ms. +- **Both base runs lost payments**: 59 and 147 were never included, and the 3 and 4 generators + that had sent them report a failure, which is the whole reason for the verdict. No node had a + pending transaction of its own left. Not looked into; the base sends a transaction to the + upcoming leaders only (experiment 5). The generators' timeouts of the earlier series are gone: + a generator now waits 150 ledgers for its last payments. + +**Experiment 6: the next tx set selected before the close.** Base and experiment at the same +load and settings; on 6 validators both networks are on target, so only the build can differ. + +| | 6 validators, base | 6 validators, experiment 6 | 90 validators, base | 90 validators, experiment 6 | +|---|---:|---:|---:|---:| +| Leader's tx set build (main thread) | 144 ms | **94 ms** | 176 ms | **121 ms (-31%)** | +| Sets completed from an early selection | | 88% | | 93% | +| Completion: validity pass on the closed ledger | | 14 ms | | 16 ms | +| Completion: top-up from the mempool (share of the final set it adds) | | 39 ms (27%) | | 60 ms (34%) | +| Completion: XDR write, with the structural check beside it | | 23 to 24 ms | | 25 to 27 ms | +| The selection: ready before it is needed / time it takes in the background | | 189 / 108 ms | | 313 / 142 ms | +| Base build: after the set is pushed to the peers (read-back and final check) | 40 ms | | 51 ms | | +| Signature checks inside builds | 411,060 | 48,436 | 452,437 | 32,107 | +| Selected transactions removed at the close; sets short of the cap with more in the mempool; refill pulls | | 0; 0; 0 | | 0; 0; 0 | +| Sets found invalid at apply; executions dropped; payments failed | 0; 0; 0 | 0; 0; 0 | 0; 0; 0 | 0; 0; 0 | +| Time per ledger | 802 ms | 801 ms | 873 ms | 858 ms | + +- **No set of the experiment was invalid, and none was short.** That holds on 6 and on 90 + validators, with every offered payment included. This load has no overlap between consecutive + sets (no selected transaction had to be removed), so the removals and the refill were not + exercised on a network. +- **The leader's build is 55 ms shorter at 90 validators and the time per ledger 15 ms.** Two + runs of one point differed by 4 and by 14 ms in the earlier series, so the block time gain is + not established; it is small at most. +- **Why the build's 55 ms do not arrive in the block time.** The followers' fetch of the set is + 57 ms longer and the leader's way to consensus 46 ms. The base pushes its set to the peers as + soon as the XDR is written and only then reads it back and runs the final check: those 51 ms + pass while the set is already travelling. The experiment has nothing left to do after the + push. What counts is the time from the close of ledger N until the XDR of the next set exists, + and that is about the same: 125 ms in the base (176 less 51), about 110 ms for a completion. + After it the set needs about 0.15 s to reach the followers in both builds. +- **The selection is made too early.** At 90 validators it is ready 0.31 s before it is needed + and takes 0.14 s, so its candidates are pulled about 0.45 s before the close. A third of the + final set arrives after that and is pulled, decoded, validated and added on the main thread at + the close: 60 ms, close to what the base spends on its pull, decoding and trimming together + (68 ms). The + start is timed by the time between a node's vote and its own build, which a node measures only + when it builds: 5 times in 450 ledgers at 90 validators. +- **Next, by what it should buy** (not measured): time the selection's start on every node at + every ledger, so that it pulls about 0.2 s before the close; let the selection take a second, + small pull just before it selects; and the largest piece, push the selected transactions to + the peers while ledger N is in its ballot and send only the difference at the close. The + 0.15 s of delivery are now the largest part of the cycle that is neither ballot nor apply. + ### Earlier benchmark: 1,000 ms target, 5,000 TPS, 48 validators, 4-core instances The measurement of experiment 3 (vote without validating the tx set, validate and execute at @@ -721,10 +871,11 @@ block_time_check.py \ To reproduce: -- **The load is not in this branch.** The two test-only load generator modes, `sac_setup` (creates - the contract instance) and `sac_payment` (the payments, each to a fresh contract address), are - on the branch `perf/loadgen-real-sac` of `SirTyson/stellar-core` (`e168408bb`, three commits on - `5d24d5494`, in `src/simulation/LoadGenerator.cpp`). +- **The load** is two test-only load generator modes in `src/simulation/LoadGenerator.cpp`: + `sac_setup` (creates the contract instance) and `sac_payment` (the payments, each to a fresh + contract address). They came into this branch with experiment 6; the runs of 2026-11-12 used + the same code as the branch `perf/loadgen-real-sac` of `SirTyson/stellar-core` (`e168408bb`, + three commits on `5d24d5494`). - **The harness flags** (`--real-apply`, `--soroban-clusters`, a list of instance types, the data directory on the local NVMe disk, the cost parameter setting) are in pull request #5 of perfnet's repository (`stellar-experimental/performance-testing`), not merged yet. @@ -735,6 +886,18 @@ To reproduce: and call graphs at 199 Hz with DWARF unwinding and the package's debug symbols (the packaged binary has no frame pointers). +**The runs with the meta off** (2026-11-13) differ from the real-apply runs above in these +points: + +| | Meta off: 6 and 90 validators | +|---|---| +| Node settings | `--cfg METADATA_DEBUG_LEDGERS=0 --cfg DISABLE_TX_META_FOR_TESTING=true` in every run; from the third settings run on also `--cfg BUCKETLIST_DB_MEMORY_FOR_CACHING=4096 --cfg BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT=8` | +| Builds | The measured commit of experiment 6 is `3973b9321`; the branch has one source change after it (`d33af26e3`: when a short phase pulls deeper; and a formatting commit), whose rules did not come into play in these runs. Base: `54260eafb`, package `30.0.1-3692.54260eafba.noble~perf.txset.build.settings.base~perftests` (this branch at `5d24d5494`, the load generator modes, and the account cache's hits under the shared lock). Experiment 6: `3973b9321`, package `30.0.1-3693.3973b9321a.noble~perf.txset.build.off.critical.path~perftests`. The first two settings runs: `e168408bb` | +| Load generator | A `sac_payment` run waits 150 ledgers for its last payments (20 before). The harness's deadline for the load allows for that wait; that harness change is not published | +| 6 validators | 2 organizations, 8,000 TPS offered, tx set cap 8,000 | +| Timers | Read from every node's metrics while the network was up, 315 s into the load: means since the load's start, medians over the nodes. The leader's build and the timers of experiment 6: the difference between the readings at 150 s and at 315 s, over all builds of all nodes | +| Log times, CPU | From all nodes' logs and per-thread CPU counters, as above | + ### Limits of these numbers - One run on each side, plus the first run of the change without the IPC fix. The "before" @@ -770,6 +933,23 @@ Real apply: - After the one dropped speculative execution, `ledger.transaction.count` and the `ledger.apply` success counters had counted that ledger twice. +Meta off, the bucket list settings and experiment 6: +- One run per point. The block time difference between the base and experiment 6 (15 ms) is + inside the spread of two runs of one point in the earlier series (4 and 14 ms). +- The meta-on runs they are set beside used another build and core's default bucket settings, + and meta, cache and index pages were not separated at 90 validators. +- The load has no overlap between consecutive tx sets: no selected transaction was ever removed + at the close and no refill was needed. Those paths, and every case of requirement "no invalid + set", are covered by unit tests only. +- Both base runs lost payments (59 and 147) and have the verdict FAIL for that. Not explained. + The run of experiment 6 lost none; one run, and nothing in the change is meant to affect it. +- The two-step change was measured as one build. What each step contributes is read from the + timers, not from separate runs. +- The apply numbers of the settings runs are from 6 validators. +- The 6-validator run of experiment 6 and the base's 8,000 TPS run were launched in the same + minute and carry the same network passphrase. They were separate networks, and everything + reported here was read per instance. + ## Known limitations of the prototype Transactions to the leaders only: @@ -791,6 +971,30 @@ Transactions to the leaders only: - Copies of a transaction that reach a node after its ledger closed are counted (`overlay.flood.tx-late-copy`), not dropped. +The next tx set selected before the close, and the smaller build: +- The mempool has one fee order for both phases. A phase with no candidate in the mempool's top + (capacity plus the margin) gets none, where a build looked twice as deep before. The fix is a + pull per phase. +- The start of the selection is timed by an estimate that a node only updates when it builds a + set itself. At 90 validators the selection is ready about 0.3 s before it is needed, and a + third of the set is added on the main thread at the close. +- A Soroban phase that was completed is extended transaction by transaction: clusters are never + merged and no stage is added, so it can be packed less tightly than a built one. With the + conflict-free load of these runs it used 16 clusters where a build used 13 or 14. +- If ledger N's set is not the one that closes, the selection made from it is dropped and the + set is built from scratch. +- The read-back that is left to the leader's execution is matched by tx set hash only. +- A phase that lost a candidate and is short pulls once more even where the mempool holds + nothing else of its kind (one stale classic transaction under a Soroban backlog costs a pull + of twice the size). +- A completed phase that is at its capacity and lost nothing keeps the surge pricing verdict of + the selection, which was made on the predicted state. If everything it left out is invalid on + the closed ledger, the set charges the lowest included fee where a build would charge the + ledger's base fee. The set is valid either way. +- No thread sanitizer run exists for this change (the build box cannot build one); its threads + were reviewed and run under the address sanitizer and the unit tests. +- Not tried with classic transactions or mixed loads on a network. + Leader terms and pre-validation: see [#5502](https://github.com/stellar/stellar-core/pull/5502). The schedule change has no protocol gate, and a term gives one node five ledgers in a row. @@ -841,6 +1045,11 @@ Leader schedule: Unchanged from stellar-core: see [INSTALL.md](./INSTALL.md), [CONTRIBUTING.md](./CONTRIBUTING.md) and [`./docs`](./docs). +Test status with experiment 6 (`f19140f89`, the branch's last source commit): the full test suite +passes, also with `--all-versions` (669 test cases). The address sanitizer ran on the three test +tags of the experiment (at `3973b9321`); there is no thread sanitizer run. A `--disable-tests` +build still fails in `herder/HerderImpl.cpp`, in the code of the leader schedule. + Test status with experiments 4 and 5 (`6ab837c27`, #5501 on top of #5502): the full test suite passes, also with `--all-versions`, including the Rust overlay's tests. diff --git a/docs/software/commands.md b/docs/software/commands.md index 052437d90..a55411279 100644 --- a/docs/software/commands.md +++ b/docs/software/commands.md @@ -537,7 +537,7 @@ this survey mechanism, just set `SURVEYOR_KEYS` to `$self` or a bogus key ### The following HTTP commands are exposed on test instances * **generateload** `generateload[?mode= - (pay|soroban_upload|soroban_invoke_setup|soroban_invoke|upgrade_setup|create_upgrade|mixed_classic_soroban|pay_pregenerated|stop)&accounts=N&offset=K&txs=M&txrate=R&spikesize=S&spikeinterval=I&maxfeerate=F&skiplowfeetxs=(0|1)&minpercentsuccess=S&instances=Y&wasms=Z&payweight=P&sorobanuploadweight=Q&sorobaninvokeweight=R&file=F]` + (pay|soroban_upload|soroban_invoke_setup|soroban_invoke|upgrade_setup|create_upgrade|mixed_classic_soroban|pay_pregenerated|stop)&accounts=N&offset=K&txs=M&txrate=R&spikesize=S&spikeinterval=I&maxfeerate=F&skiplowfeetxs=(0|1)&completiontimeout=L&minpercentsuccess=S&instances=Y&wasms=Z&payweight=P&sorobanuploadweight=Q&sorobaninvokeweight=R&file=F]` Artificially generate load for testing; must be used with `ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING` set to true. @@ -608,6 +608,9 @@ this survey mechanism, just set `SURVEYOR_KEYS` to `$self` or a bogus key * when `skiplowfeetxs` is set to `true` the transactions that are not accepted by the node due to having too low fee to pass the rate limiting are silently skipped. Otherwise (by default), such transactions would cause load generation to fail. + * `completiontimeout` is the number of ledgers a run waits, after it has + submitted its last transaction, for its transactions to be applied before + it reports failure. The default is 20 ledgers, and 150 for `sac_payment`. Soroban load generation also makes use of the `minpercentsuccess` parameter, which determines the minimum percentage of Soroban transactions that must diff --git a/src/bucket/LiveBucketIndex.cpp b/src/bucket/LiveBucketIndex.cpp index e0122b234..389ae7972 100644 --- a/src/bucket/LiveBucketIndex.cpp +++ b/src/bucket/LiveBucketIndex.cpp @@ -200,13 +200,27 @@ LiveBucketIndex::markBloomMiss() const std::shared_ptr LiveBucketIndex::getCachedEntry(LedgerKey const& k) const { - if (shouldUseCache() && isCachedType(k)) + if (mDiskIndex && isCachedType(k)) { - SharedLockExclusive lock(mCacheMutex); - auto cachePtr = mCache->maybeGet(k); + // Lookups only read the cache (`peek` records nothing), so they run + // under the shared lock, side by side: ledger apply loads accounts + // from many threads at once, and each load asks the cache of every + // bucket on its way down the bucket list. Only inserting an entry + // takes the lock exclusively (maybeAddToCache). What a lookup gives + // up is the access time of the entry it hits: when the cache is + // smaller than the bucket's accounts, eviction goes by when an entry + // was put in, not by when it was last read. + SharedLockShared lock(mCacheMutex); + if (!mCache) + { + return nullptr; + } + auto cachePtr = mCache->peek(k); if (cachePtr) { mCacheHitMeter.Mark(); + // The entry is shared; the copy of the pointer is taken under + // the lock. return *cachePtr; } diff --git a/src/herder/EarlyTxSetSelection.cpp b/src/herder/EarlyTxSetSelection.cpp new file mode 100644 index 000000000..cd453225b --- /dev/null +++ b/src/herder/EarlyTxSetSelection.cpp @@ -0,0 +1,501 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#include "herder/EarlyTxSetSelection.h" +#include "crypto/ByteSlice.h" +#include "crypto/ShortHash.h" +#include "herder/TxSetUtils.h" +#include "main/AppConnector.h" +#include "main/Application.h" +#include "transactions/EventManager.h" +#include "transactions/MutableTransactionResult.h" +#include "transactions/TransactionFrameBase.h" +#include "transactions/TransactionUtils.h" +#include "util/GlobalChecks.h" +#include "util/Logging.h" +#include "util/MetricsRegistry.h" +#include "util/ProtocolVersion.h" +#include "xdrpp/marshal.h" + +#include +#include +#include +#include +#include + +namespace stellar +{ +namespace +{ +class PredictedLedgerView : public AbstractLedgerView +{ + public: + PredictedLedgerView(ImmutableLedgerDataPtr previous, + std::shared_ptr header, + std::shared_ptr changes, + MetricsRegistry& metrics) + : mPrevious(std::move(previous), metrics) + , mHeader(std::move(header)) + , mChanges(std::move(changes)) + { + releaseAssert(mHeader && mChanges); + } + + LedgerHeaderWrapper + getLedgerHeader() const override + { + return LedgerHeaderWrapper(mHeader); + } + + SorobanNetworkConfig const* + getSorobanNetworkConfig() const override + { + return mPrevious.getSorobanNetworkConfig(); + } + + LedgerEntryWrapper + getAccount(AccountID const& account) const override + { + return load(accountKey(account)); + } + + LedgerEntryWrapper + getAccount(LedgerHeaderWrapper const& header, + TransactionFrame const& tx) const override + { + return getAccount(tx.getSourceID()); + } + + LedgerEntryWrapper + getAccount(LedgerHeaderWrapper const& header, TransactionFrame const& tx, + AccountID const& accountID) const override + { + return getAccount(accountID); + } + + LedgerEntryWrapper + load(LedgerKey const& key) const override + { + if (key.type() != ACCOUNT) + { + // Nothing is predicted about other entries: the previous + // ledger's version could be stale. + throw std::logic_error( + "the predicted state of a ledger only holds account entries"); + } + auto entry = mPrevious.loadLiveEntry(key); + if (!entry) + { + return LedgerEntryWrapper(entry); + } + auto it = mChanges->find(key.account().accountID); + if (it == mChanges->end()) + { + return LedgerEntryWrapper(entry); + } + auto predicted = std::make_shared(*entry); + auto& account = predicted->data.account(); + account.seqNum = std::max(account.seqNum, it->second.mSeqNum); + account.balance -= std::min(account.balance, it->second.mFees); + return LedgerEntryWrapper( + std::shared_ptr(std::move(predicted))); + } + + private: + ApplyLedgerView mPrevious; + std::shared_ptr mHeader; + std::shared_ptr mChanges; +}; +} + +void +predictAccountChanges(TransactionFrameBase const& tx, + PredictedAccountChanges& changes) +{ + auto& source = changes[tx.getSourceID()]; + source.mSeqNum = std::max(source.mSeqNum, tx.getSeqNum()); + auto& feeSource = changes[tx.getFeeSourceID()]; + auto const fee = std::max(0, tx.getFullFee()); + feeSource.mFees = + feeSource.mFees > std::numeric_limits::max() - fee + ? std::numeric_limits::max() + : feeSource.mFees + fee; +} + +std::unique_ptr +makePredictedLedgerView(ImmutableLedgerDataPtr previous, + std::shared_ptr header, + std::shared_ptr changes, + MetricsRegistry& metrics) +{ + return std::make_unique( + std::move(previous), std::move(header), std::move(changes), metrics); +} + +void +forEachPulledTx(std::vector const& payload, + std::function const& f) +{ + if (payload.size() < 4) + { + return; + } + uint32_t count = 0; + std::memcpy(&count, payload.data(), 4); + size_t offset = 4; + for (uint32_t i = 0; i < count && offset + 4 <= payload.size(); ++i) + { + uint32_t length = 0; + std::memcpy(&length, payload.data() + offset, 4); + offset += 4; + if (length > payload.size() - offset) + { + break; + } + f(payload.data() + offset, length); + offset += length; + } +} + +uint64_t +hashPulledTx(uint8_t const* data, size_t length) +{ + // Keyed per process: nobody outside can make one transaction's bytes + // hash like another's. + return shortHash::computeHash(ByteSlice(data, length)); +} + +bool +decodePulledTx(uint8_t const* data, size_t length, TransactionEnvelope& tx) +{ + try + { + // XDR is read in 4-byte words from 4-byte aligned memory. Well + // formed entries are (XDR lengths are multiples of 4), so decode + // them in place; copy anything else to aligned memory first. + if ((reinterpret_cast(data) & 3) == 0) + { + xdr::xdr_get g(data, data + length); + xdr::xdr_argpack_archive(g, tx); + g.done(); + } + else + { + std::vector const copy(data, data + length); + xdr::xdr_from_opaque(copy, tx); + } + return true; + } + catch (std::exception const&) + { + return false; + } +} + +void +EarlyTxSetJob::prepare(Application& app) +{ + ZoneScoped; + releaseAssert(mPayload && mClosingTxSet && mPreviousLedger); + auto const* base = mPayload->data(); + forEachPulledTx(*mPayload, [&](uint8_t const* data, size_t length) { + mPulled.emplace_back(static_cast(data - base), length); + mSeen.insert(hashPulledTx(data, length)); + }); + + // What N does, as far as its transaction set says. + auto changes = std::make_shared(); + mClosingTxHashes.reserve(mClosingTxSet->sizeTxTotal()); + for (auto const& phase : + mClosingTxSet->createTransactionFrames(app.getNetworkID())) + { + for (auto const& tx : phase) + { + mClosingTxHashes.insert(tx->getFullHash()); + predictAccountChanges(*tx, *changes); + } + } + mPredicted = std::move(changes); + + // N's header as far as validation reads it: number, value (close time), + // and what only an upgrade changes (N has none). + auto const& previous = mPreviousLedger->getLastClosedLedgerHeader(); + auto header = std::make_shared(previous.header); + header->ledgerSeq = mClosingLedgerSeq; + header->previousLedgerHash = previous.hash; + header->scpValue = mClosingValue; + mPredictedHeader = std::move(header); +} + +std::shared_ptr +EarlyTxSetJob::select(Application& app) +{ + ZoneScoped; + auto& metrics = app.getMetrics(); + auto timer = + metrics.NewTimer({"herder", "txset", "early-job-select"}).TimeScope(); + + // Every candidate in mempool order, and what the tasks found. + std::vector candidates; + { + std::lock_guard lock(mMutex); + if (mStop.load()) + { + return nullptr; + } + size_t total = 0; + for (auto const& results : mTaskResults) + { + total += results.size(); + } + candidates.reserve(total); + for (auto& results : mTaskResults) + { + candidates.insert(candidates.end(), + std::make_move_iterator(results.begin()), + std::make_move_iterator(results.end())); + results.clear(); + } + } + std::sort(candidates.begin(), candidates.end(), + [](Candidate const& a, Candidate const& b) { + return a.mIndex < b.mIndex; + }); + // The results hold for the predicted state and for the close time this + // job assumes: they are used by this selection and by nothing else. + TxValidationCache results; + results.reserve(candidates.size()); + TxFrameList classicTxs; + TxFrameList sorobanTxs; + for (auto const& candidate : candidates) + { + results.emplace(candidate.mTx->getFullHash(), candidate.mResult); + (candidate.mTx->isSoroban() ? sorobanTxs : classicTxs) + .push_back(candidate.mTx); + } + TxSetUtils::keepOnePerSourceAccount(classicTxs, sorobanTxs); + + auto selection = std::make_shared(); + selection->mLedgerSeq = mClosingLedgerSeq + 1; + selection->mPreviousLedger = mPreviousLedger; + selection->mSelectionSeed = mSelectionSeed; + // The limits of the ledger before N. N carries no upgrades, so they are + // N's as well; the node compares them with the closed ledger's before it + // uses the selection. + selection->mLimits.mLclHeader = + mPreviousLedger->getLastClosedLedgerHeader().header; + selection->mLimits.mSorobanConfig = + mPreviousLedger->hasSorobanConfig() + ? &mPreviousLedger->getSorobanConfig() + : nullptr; + releaseAssert(selection->mLimits.mSorobanConfig); + + PerPhaseTransactionList txPhases; + txPhases.emplace_back(std::move(classicTxs)); + txPhases.emplace_back(std::move(sorobanTxs)); + PerPhaseTransactionList invalidTxPhases; + invalidTxPhases.resize(txPhases.size()); + + // Validity is read from the tasks' results. The context's view is the + // predicted state all the same, for anything the tasks did not cover. + LedgerHeaderHistoryEntry predicted; + predicted.header = *mPredictedHeader; + TxSetValidationContext const ctx{ + predicted, selection->mLimits.mSorobanConfig, + [this, &metrics]() { + return makePredictedLedgerView(mPreviousLedger, mPredictedHeader, + mPredicted, metrics); + }, + 1, &results}; + auto const offset = ApplyTimeOffset::fromSeconds(mAssumedCloseTimeOffset); + selection->mPhases = TxSetBuilder::selectPhases( + txPhases, app, selection->mLimits, ctx, offset, invalidTxPhases, {}, + "early", mSelectionSeed, &selection->mHadTxNotFittingLane); + if (mStop.load()) + { + return nullptr; + } + // The structure of the selection is validated when it is completed, on + // the phases as they are then (TxSetBuilder::writeSelection). + + UnorderedSet selected; + for (auto const& phase : selection->mPhases) + { + for (auto const& tx : phase) + { + selected.insert(tx.get()); + } + } + selection->mSelected = selected.size(); + selection->mLeftovers.reserve(candidates.size() - selected.size()); + for (auto const& candidate : candidates) + { + if (selected.count(candidate.mTx.get()) == 0) + { + selection->mLeftovers.push_back(candidate.mTx); + } + } + selection->mSeen = std::move(mSeen); + selection->mSelectedAt = std::chrono::steady_clock::now(); + return selection; +} + +void +EarlyTxSetJob::runTask(Application& app, + std::shared_ptr const& job, size_t task) +{ + ZoneScopedN("early tx set selection"); +#ifdef BUILD_TESTS + while (job->mHoldForTesting && job->mHoldForTesting->load()) + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } +#endif + { + std::lock_guard lock(job->mMutex); + --job->mPendingTasks; + if (job->mStop.load()) + { + if (job->mPendingTasks == 0 && job->mActiveTasks == 0) + { + job->mDone.store(true); + } + return; + } + ++job->mActiveTasks; + } + std::vector mine; + try + { + std::call_once(job->mPrepared, [&]() { + // Must not throw out of call_once. + try + { + job->prepare(app); + } + catch (std::exception const& e) + { + CLOG_WARNING(Herder, + "Early tx set selection could not read its " + "input: {}", + e.what()); + std::lock_guard lock(job->mMutex); + job->mStop.store(true); + } + }); + if (!job->mStop.load()) + { + auto& metrics = app.getMetrics(); + auto view = makePredictedLedgerView(job->mPreviousLedger, + job->mPredictedHeader, + job->mPredicted, metrics); + auto const& header = *job->mPredictedHeader; + // The frames keep a reference to the network ID: the + // application's outlives them. + Hash const& networkID = app.getNetworkID(); + auto& appConnector = app.getAppConnector(); + auto diagnostics = DiagnosticEventManager::createDisabled(); + bool const supportsSoroban = protocolVersionStartsFrom( + header.ledgerVersion, SOROBAN_PROTOCOL_VERSION); + std::optional validationLedgerSeq; + if (protocolVersionStartsFrom(header.ledgerVersion, + ProtocolVersion::V_19)) + { + validationLedgerSeq = job->mClosingLedgerSeq + 1; + } + auto const offset = job->mAssumedCloseTimeOffset; + auto const* base = job->mPayload->data(); + mine.reserve(job->mPulled.size() / job->mTasks + 1); + for (size_t i = task; i < job->mPulled.size(); i += job->mTasks) + { + if (job->mStop.load(std::memory_order_relaxed)) + { + break; + } + auto const& [at, length] = job->mPulled[i]; + TransactionEnvelope env; + if (!decodePulledTx(base + at, length, env)) + { + continue; + } + auto tx = TransactionFrameBase::makeTransactionFromWire( + networkID, env); + if (tx->isSoroban() && !supportsSoroban) + { + continue; + } + // N's own transactions are still in the mempool. + if (job->mClosingTxHashes.count(tx->getFullHash()) != 0) + { + continue; + } + TxValidationResult result; + result.mValid = + tx->checkValid(appConnector, *view, 0, offset, offset, + diagnostics, validationLedgerSeq) + ->isSuccess(); + if (result.mValid) + { + auto feeSource = view->getAccount(tx->getFeeSourceID()); + if (feeSource) + { + result.mFeeSourceBalance = + getAvailableBalance(header, feeSource.current()); + } + } + mine.push_back(Candidate{i, std::move(tx), result}); + } + } + } + catch (std::exception const& e) + { + // The ledger gets a set built the usual way. + CLOG_WARNING(Herder, "Early tx set selection failed: {}", e.what()); + mine.clear(); + std::lock_guard lock(job->mMutex); + job->mStop.store(true); + } + bool last = false; + { + std::lock_guard lock(job->mMutex); + if (!job->mStop.load()) + { + job->mTaskResults.at(task) = std::move(mine); + } + last = --job->mActiveTasks == 0 && job->mPendingTasks == 0; + } + if (!last) + { + return; + } + // Every task has left: nothing else touches the job's transactions. + std::shared_ptr selection; + if (!job->mStop.load()) + { + try + { + selection = job->select(app); + } + catch (std::exception const& e) + { + CLOG_WARNING(Herder, + "Could not select the TX set for ledger {} ahead of " + "time: {}", + job->mClosingLedgerSeq + 1, e.what()); + } + } + { + std::lock_guard lock(job->mMutex); + if (selection && !job->mStop.load()) + { + job->mFinishedAfterNs = + std::chrono::duration_cast( + selection->mSelectedAt - job->mStart) + .count(); + job->mSelection = std::move(selection); + } + } + job->mDone.store(true); +} +} diff --git a/src/herder/EarlyTxSetSelection.h b/src/herder/EarlyTxSetSelection.h new file mode 100644 index 000000000..b91352c79 --- /dev/null +++ b/src/herder/EarlyTxSetSelection.h @@ -0,0 +1,178 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#pragma once + +// The selection of the next transaction set before the current ledger has +// closed. +// +// Ledger N is going through consensus; the last closed ledger is N-1. A node +// that expects to lead ledger N+1 knows N's transaction set as soon as it +// votes for it, and with it most of what N will do to the candidates of N+1: +// which transactions are used up, which source accounts move their sequence +// number, which fees are charged. An EarlyTxSetJob selects the set for N+1 on +// that prediction, on background threads, while N is still in its ballot. +// +// The prediction is an optimization only. When N has closed, the node +// completes the selection (TxSetBuilder::completeSelection): every one of its +// transactions is validated against N as it really closed, what is not valid +// is removed, and the set is topped up with what the job did not select or +// see. Nothing here decides what is valid; it decides what is worth +// validating later. + +#include "herder/TxSetFrame.h" +#include "ledger/ImmutableLedgerView.h" +#include "util/UnorderedMap.h" +#include "util/UnorderedSet.h" +#include "xdr/Stellar-ledger.h" + +#include +#include +#include +#include +#include +#include + +namespace stellar +{ +class Application; +class MetricsRegistry; + +// What a ledger is predicted to do to one account, read off the ledger's +// transaction set alone. +struct PredictedAccountChange +{ + // The highest sequence number a transaction of the set consumes for the + // account as its source (a transaction consumes its sequence number + // whether or not it succeeds), or 0. + SequenceNumber mSeqNum{0}; + // The fees the set's transactions bid in full with the account as their + // fee source: an upper bound of what the ledger charges it. + int64_t mFees{0}; +}; +using PredictedAccountChanges = UnorderedMap; + +// Adds what `tx`, as a transaction of the ledger being predicted, does to +// its source account and to its fee source. +void predictAccountChanges(TransactionFrameBase const& tx, + PredictedAccountChanges& changes); + +// A view of the state `previous` with `changes` applied to its accounts, for +// validating transactions on one thread. It answers for accounts only, and +// throws on any other kind of entry rather than answer with the previous +// ledger's version. +std::unique_ptr +makePredictedLedgerView(ImmutableLedgerDataPtr previous, + std::shared_ptr header, + std::shared_ptr changes, + MetricsRegistry& metrics); + +// Calls `f(data, length)` for every transaction of a GET_TOP_TXS response +// payload ([count:4]([len:4][TransactionEnvelope XDR:len])*), in order. A +// truncated payload ends the list. +void forEachPulledTx(std::vector const& payload, + std::function const& f); +// A hash of a pulled transaction's bytes, to recognize a transaction that +// was pulled before without decoding it. Equal bytes hash equally within a +// process; a collision makes a new transaction look like a known one. +uint64_t hashPulledTx(uint8_t const* data, size_t length); +// Decodes one pulled transaction into `tx`; false if it does not parse. +bool decodePulledTx(uint8_t const* data, size_t length, + TransactionEnvelope& tx); + +// What an EarlyTxSetJob produces. +struct EarlyTxSetSelection +{ + // The ledger the set is for (N+1). + uint32_t mLedgerSeq{0}; + // The selection: both phases, within the limits of N-1. + std::vector mPhases; + // Per phase and surge pricing lane, whether the selection left a + // transaction out for the lane's limits. + std::vector> mHadTxNotFittingLane; + // Every other candidate the job decoded (not in N, not selected), in + // mempool order: the first candidates of the top-up. + TxFrameList mLeftovers; + // hashPulledTx of everything the job pulled, N's transactions included. + UnorderedSet mSeen; + // The limits the selection was made within (those of N-1), and what + // keeps their Soroban configuration alive. + TxSetBuildLimits mLimits; + ImmutableLedgerDataPtr mPreviousLedger; + // Randomized the order of candidates with equal fee rates. + size_t mSelectionSeed{0}; + size_t mSelected{0}; + std::chrono::steady_clock::time_point mSelectedAt; +}; + +struct EarlyTxSetJob +{ + // --- Inputs, set on the main thread before the tasks are posted. --- + // The ledger in its ballot (N), the value this node votes for and that + // value's transaction set. + uint32_t mClosingLedgerSeq{0}; + StellarValue mClosingValue; + TxSetXDRFrameConstPtr mClosingTxSet; + // The last closed ledger (N-1). + ImmutableLedgerDataPtr mPreviousLedger; + // What the main thread pulled from the mempool. + std::shared_ptr const> mPayload; + // The close time the candidates are validated for, as an apply-time + // offset from N. + uint64_t mAssumedCloseTimeOffset{0}; + size_t mSelectionSeed{0}; + size_t mTasks{1}; + std::chrono::steady_clock::time_point mStart; + + // Set by whoever gives up on the job (the main thread, or a task that + // failed); tasks stop at their next transaction and nothing is handed + // over afterwards. + std::atomic mStop{false}; + + // --- Written once, by the first task (prepare). --- + std::once_flag mPrepared; + // The pulled transactions: where each is in the payload. + std::vector> mPulled; + UnorderedSet mSeen; + UnorderedSet mClosingTxHashes; + std::shared_ptr mPredicted; + std::shared_ptr mPredictedHeader; + + // --- Guarded by mMutex. --- + std::mutex mMutex; + size_t mPendingTasks{0}; + size_t mActiveTasks{0}; + struct Candidate + { + // Position in the pull (mempool order). + size_t mIndex{0}; + TransactionFrameBasePtr mTx; + TxValidationResult mResult; + }; + // What each task decoded and validated. + std::vector> mTaskResults; + // The result, once it exists and until it is taken. + std::shared_ptr mSelection; + +#ifdef BUILD_TESTS + // While this is set and true the job's tasks wait before they do + // anything: lets a test have the ledger close, or the next job start, + // while this one is still running. + std::shared_ptr> mHoldForTesting; +#endif + + // Set when the last task has left, with or without a result. + std::atomic mDone{false}; + // Nanoseconds from mStart until the selection was handed over, or -1. + std::atomic mFinishedAfterNs{-1}; + + // The body of one of the job's mTasks background tasks. + static void runTask(Application& app, + std::shared_ptr const& job, size_t task); + + private: + void prepare(Application& app); + std::shared_ptr select(Application& app); +}; +} diff --git a/src/herder/HerderImpl.cpp b/src/herder/HerderImpl.cpp index 859b1d13f..f2e8698af 100644 --- a/src/herder/HerderImpl.cpp +++ b/src/herder/HerderImpl.cpp @@ -317,6 +317,7 @@ HerderImpl::HerderImpl(Application& app) , mLastExternalize(app.getClock().now()) , mTriggerTimer(app) , mPrepareTxSetTimer(app) + , mEarlyTxSetTimer(app) , mOutOfSyncTimer(app) , mTxSetGarbageCollectTimer(app) , mCheckForDeadNodesTimer(app) @@ -464,8 +465,7 @@ HerderImpl::computeTxFloodTargets() // Every validator of the schedule, in an order that does not change // from slot to slot (the overlay is only told when the list // changes). - std::vector everyone = - mHerderSCPDriver.getLeaderSchedule(slot); + std::vector everyone = mHerderSCPDriver.getLeaderSchedule(slot); if (everyone.empty()) { return std::nullopt; @@ -509,9 +509,8 @@ HerderImpl::computeTxFloodTargets() { auto const next = computeLeaderSchedule( lcl.hash, mHerderSCPDriver.getLeaderSchedule(slot)); - if (!next.empty() && - std::find(targets.begin(), targets.end(), next.front()) == - targets.end()) + if (!next.empty() && std::find(targets.begin(), targets.end(), + next.front()) == targets.end()) { targets.emplace_back(next.front()); } @@ -738,11 +737,11 @@ HerderImpl::checkTxAdmission(TransactionFrameBasePtr const& tx, validationLedgerSeq = lcl.header.ledgerSeq + 1; } auto diagnostics = DiagnosticEventManager::createDisabled(); - auto result = tx->checkValid( - mApp.getAppConnector(), *ledgerView, 0, 0, - getUpperBoundCloseTimeOffset(mApp, - getApplyTime(lcl.header.scpValue)), - diagnostics, validationLedgerSeq); + auto result = + tx->checkValid(mApp.getAppConnector(), *ledgerView, 0, 0, + getUpperBoundCloseTimeOffset( + mApp, getApplyTime(lcl.header.scpValue)), + diagnostics, validationLedgerSeq); if (!result->isSuccess()) { CLOG_DEBUG(Herder, "Rejecting transaction {} from {}: {}", @@ -895,6 +894,7 @@ HerderImpl::shutdown() discardPreparedTxSet(); stopSignaturePrewarm(); stopProposalPrevalidation(nullptr); + stopEarlyTxSetSelection(); mTrackingTimer.cancel(); mOutOfSyncTimer.cancel(); mTriggerTimer.cancel(); @@ -1271,7 +1271,14 @@ HerderImpl::emitEnvelope(SCPEnvelope const& envelope) broadcast(envelope); + // Before the execution is handed over: its end starts the + // pre-validation of the next ledger's candidates, unless this node + // selects that ledger's set itself. + maybeScheduleEarlyTxSetSelection(envelope); maybeStartSpeculativeApply(envelope); + // A set this node built and has not seen checked: by now its own + // execution has it, or the check runs in the background. + resolvePendingTxSetSelfCheck(); } void @@ -1306,34 +1313,54 @@ HerderImpl::maybeStartSpeculativeApply(SCPEnvelope const& envelope) return; } + startSpeculativeApply(st.slotIndex, *value, nullptr); +} + +void +HerderImpl::startSpeculativeApply(uint64_t slotIndex, Value const& value, + TxSetXDRFrameConstPtr txSet) +{ + auto const& cfg = mApp.getConfig(); + if (!cfg.EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY || + !cfg.backgroundLedgerApply()) + { + return; + } // Only the ledger right after the LCL can be executed, and only while // nothing else is being applied: the ledger apply thread executes it on // top of the state the LCL left. auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); - if (st.slotIndex != lcl.header.ledgerSeq + 1 || + if (slotIndex != lcl.header.ledgerSeq + 1 || !protocolValidatesTxSetsAtApply(lcl.header.ledgerVersion) || !isTracking() || !mLedgerManager.isSynced() || mLedgerManager.isApplying()) { return; } - if (mSpeculativeApplyValue && - mSpeculativeApplyValue->first == st.slotIndex && - mSpeculativeApplyValue->second == *value) + if (mSpeculativeApplyValue && mSpeculativeApplyValue->first == slotIndex && + mSpeculativeApplyValue->second == value) { // already handed over return; } StellarValue sv; - if (!toStellarValue(*value, sv) || isEmptyTxSetStellarValue(sv)) + if (!toStellarValue(value, sv) || isEmptyTxSetStellarValue(sv)) { // An empty-tx-set value takes no time to apply. return; } - auto txSetResult = mPendingEnvelopes.getTxSet(sv.txSetHash); - auto const* txSet = std::get_if(&txSetResult); - if (!txSet || !*txSet || (*txSet)->previousLedgerHash() != lcl.hash) + if (!txSet) + { + auto txSetResult = mPendingEnvelopes.getTxSet(sv.txSetHash); + if (auto const* received = + std::get_if(&txSetResult)) + { + txSet = *received; + } + } + if (!txSet || txSet->getContentsHash() != sv.txSetHash || + txSet->previousLedgerHash() != lcl.hash) { return; } @@ -1341,11 +1368,11 @@ HerderImpl::maybeStartSpeculativeApply(SCPEnvelope const& envelope) // Executing the ledger starts with validating its transaction set, which // verifies the signatures the prewarm has not reached itself, on all // cores; a prewarm that is still running would only compete with it. - if (mSignaturePrewarm && mSignaturePrewarm->ledgerSeq <= st.slotIndex) + if (mSignaturePrewarm && mSignaturePrewarm->ledgerSeq <= slotIndex) { stopSignaturePrewarm(); } - mSpeculativeApplyValue = std::make_pair(st.slotIndex, *value); + mSpeculativeApplyValue = std::make_pair(slotIndex, value); // A pre-validation that is still running was for a build on top of the // LCL that this node has not needed (it votes for another leader's // value): keep what it found, and leave the cores to the execution. @@ -1354,17 +1381,39 @@ HerderImpl::maybeStartSpeculativeApply(SCPEnvelope const& envelope) // leaves, to validate its candidates on while this ledger is in its // ballot. A ledger with upgrades is left out: they are applied when the // ledger commits, after that state. - LedgerManager::SpeculativeApplyExecuted onExecuted; - if (sv.upgrades.empty() && getSCP().isValidator() && + bool const prevalidate = + sv.upgrades.empty() && getSCP().isValidator() && !buildsLegacyTxSet(lcl.header.ledgerVersion) && - mayLeadAfter(st.slotIndex, sv, /*appliedTransactions=*/true)) + mayLeadAfter(slotIndex, sv, /*appliedTransactions=*/true); + // Executing a ledger starts with the full validation of its set. For a + // set this node built, that is the verdict every other node will reach + // when it applies the ledger: if it comes before the proposal (the + // leader executes a prepared set ahead of the trigger), an invalid set + // is not proposed. + bool const ownTxSet = mLastBuiltTxSet && + mLastBuiltTxSet->first == sv.txSetHash && + mLastBuiltTxSet->second == getConsensusTime(sv); + LedgerManager::SpeculativeApplyExecuted onExecuted; + if (prevalidate || ownTxSet) { - onExecuted = [this](std::shared_ptr state) { - startProposalPrevalidation(std::move(state)); + onExecuted = [this, prevalidate, ownTxSet, + ledgerSeq = static_cast(slotIndex), + txSetHash = sv.txSetHash]( + std::shared_ptr state) { + if (ownTxSet && state && state->mTxSetSkipped) + { + txSetSelfCheckFailed(txSetHash, ledgerSeq, + "this node's own execution of the " + "ledger found the set invalid"); + } + if (prevalidate) + { + startProposalPrevalidation(std::move(state)); + } }; } mLedgerManager.startSpeculativeApply( - LedgerCloseData(static_cast(st.slotIndex), *txSet, sv), + LedgerCloseData(static_cast(slotIndex), txSet, sv), std::move(onExecuted)); } @@ -1405,6 +1454,16 @@ HerderImpl::startProposalPrevalidation( { return; } + // The node first in line for the next ledger selects its set ahead of + // time instead (see EarlyTxSetJob): that job validates the same + // candidates, and the selection is validated again on the closed + // ledger, so results computed here would only serve a fallback build. + if (mEarlyTxSetPlan && mEarlyTxSetPlan->planned && + mEarlyTxSetPlan->slotIndex == state->mLedgerSeq && + mEarlyTxSetPlan->stellarValue == state->mValue) + { + return; + } stopProposalPrevalidation(nullptr); // The pre-validation verifies the same signatures, and more. if (mSignaturePrewarm && mSignaturePrewarm->ledgerSeq > state->mLedgerSeq) @@ -1461,8 +1520,11 @@ HerderImpl::startProposalPrevalidation( for (size_t task = 0; task < tasks; ++task) { mApp.postOnBackgroundThread( + // The network ID by reference to the application's: the + // transaction frames made below keep a reference to it, and + // outlive this task. [job, state, payload, windows, supportsSoroban, validationLedgerSeq, - networkID = mApp.getNetworkID(), + &networkID = mApp.getNetworkID(), &appConnector = mApp.getAppConnector(), &metrics, task, tasks]() { ZoneScopedN("proposal prevalidation"); { @@ -1644,13 +1706,19 @@ HerderImpl::expectedTxSetBuildDuration() const // Until this node has built a set of some size: what a build costs per // transaction on the slower of the machines measured (about 0.17 s for // 4,000 transactions with most candidates validated ahead of time). + return std::max(mLastTxSetBuildDuration, + expectedTxSetBuildDuration(mLastClosedTxSetSize)); +} + +std::chrono::milliseconds +HerderImpl::expectedTxSetBuildDuration(size_t txs) const +{ static std::chrono::nanoseconds const DEFAULT_COST_PER_TX{40'000}; auto const perTx = mTxSetBuildCostPerTx > std::chrono::nanoseconds::zero() ? mTxSetBuildCostPerTx : DEFAULT_COST_PER_TX; - auto const bySize = std::chrono::duration_cast( - perTx * static_cast(mLastClosedTxSetSize)); - return std::max(mLastTxSetBuildDuration, bySize); + return std::chrono::duration_cast( + perTx * static_cast(txs)); } std::chrono::milliseconds @@ -2271,6 +2339,21 @@ HerderImpl::lastClosedLedgerIncreased(bool latest, TxSetXDRFrameConstPtr txSet, { stopProposalPrevalidation(nullptr); } + mPendingTxSetSelfCheck.reset(); + // The same goes for a set selected ahead of time. + if (mEarlyTxSetPlan && + (mEarlyTxSetPlan->slotIndex != lcl.header.ledgerSeq || + !(mEarlyTxSetPlan->stellarValue == lcl.header.scpValue))) + { + if (mEarlyTxSetPlan->planned && + mEarlyTxSetPlan->slotIndex == lcl.header.ledgerSeq) + { + mApp.getMetrics() + .NewMeter({"herder", "txset", "early-discarded"}, "txset") + .Mark(); + } + stopEarlyTxSetSelection(); + } // Results computed on the previous ledger are of no use either, // whether or not this node builds on the new one. if (mValidationCache.lclHash != lcl.hash) @@ -2557,42 +2640,8 @@ HerderImpl::setupTriggerNextLedger() }, &VirtualTimer::onFailureNoop); - // Prepare early only if the trigger is still ahead. A due trigger - // fires now and builds itself; both timers would fire at the same - // instant in an unspecified order (event sequence numbers are per - // timer), so the leader could build once to prepare and again at the - // trigger. - if (getSCP().isValidator() && !triggerDue) - { - if (mHerderSCPDriver.isLocalNodeRoundLeader(nextIndex, 1)) - { - // Validate against the close time we expect at the trigger, - // not the earlier wall-clock time at which we start work. - auto closeTime = std::max( - minCandidateCt, - ConsensusTime::fromSystemTime( - mApp.getClock().system_now() + - std::chrono::duration_cast< - VirtualClock::system_time_point::duration>( - triggerTime - now), - lcl.header.ledgerVersion)); - // Build shortly before the trigger rather than right away: - // the set then holds what arrived in the meantime, and is - // still built and on its way to the peers when consensus - // starts (see makeProposalValue). This also yields out of - // ledger close before touching its read view. - mPrepareTxSetTimer.expires_at(std::max( - now, triggerTime - - prepareTxSetLeadTime( - milliseconds, expectedTxSetBuildDuration()))); - mPrepareTxSetTimer.async_wait( - [this, nextIndex, closeTime]() { - prepareTxSet(static_cast(nextIndex), - closeTime); - }, - &VirtualTimer::onFailureNoop); - } - } + scheduleTxSetPreparation(nextIndex, triggerTime, triggerDue, now, + milliseconds, minCandidateCt); } #ifdef BUILD_TESTS @@ -2600,6 +2649,45 @@ HerderImpl::setupTriggerNextLedger() #endif } +void +HerderImpl::scheduleTxSetPreparation( + uint64_t nextIndex, VirtualClock::time_point triggerTime, bool triggerDue, + VirtualClock::time_point now, std::chrono::milliseconds expectedCloseTime, + ConsensusTime minCandidateCt) +{ + // Prepare early only if the trigger is still ahead. A due trigger fires + // now and builds itself; both timers would fire at the same instant in + // an unspecified order (event sequence numbers are per timer), so the + // leader could build once to prepare and again at the trigger. + if (!getSCP().isValidator() || triggerDue || + !mHerderSCPDriver.isLocalNodeRoundLeader(nextIndex, 1)) + { + return; + } + auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); + // Validate against the close time we expect at the trigger, not the + // earlier wall-clock time at which we start work. + auto closeTime = std::max( + minCandidateCt, ConsensusTime::fromSystemTime( + mApp.getClock().system_now() + + std::chrono::duration_cast< + VirtualClock::system_time_point::duration>( + triggerTime - now), + lcl.header.ledgerVersion)); + // Build shortly before the trigger rather than right away: the set then + // holds what arrived in the meantime, and is still built and on its way + // to the peers when consensus starts (see makeProposalValue). This also + // yields out of ledger close before touching its read view. + mPrepareTxSetTimer.expires_at(std::max( + now, triggerTime - prepareTxSetLeadTime(expectedCloseTime, + expectedTxSetBuildDuration()))); + mPrepareTxSetTimer.async_wait( + [this, nextIndex, closeTime]() { + prepareTxSet(static_cast(nextIndex), closeTime); + }, + &VirtualTimer::onFailureNoop); +} + // Returns the ledger index of the oldest ledger that it is safe to delete while // still keeping all the information needed to publish checkpoints static uint32_t @@ -2756,6 +2844,223 @@ HerderImpl::prepareTxSet(uint32_t ledgerSeq, ConsensusTime closeTime) "close time {}", ledgerSeq, mPreparedTxSet->txSet->sizeTxTotal(), closeTime.toString()); + executePreparedTxSet(); +} + +void +HerderImpl::handOffTxSet(TxSetXDRFrameConstPtr const& txSet, uint32_t ledgerSeq, + bool pushToPeers) +{ + auto t = mApp.getMetrics() + .NewTimer({"herder", "txset", "build-handoff"}) + .TimeScope(); + // Legacy (pre-protocol 20) sets cannot be carried by the Rust overlay. + if (!txSet->isGeneralizedTxSet()) + { + return; + } + // Serialized straight into the IPC message, without copying the set's + // XDR first. The overlay's eager compression runs concurrently in Rust. + if (mApp.getOverlayManager().cacheTxSet( + txSet->getContentsHash(), txSet->getGeneralizedXDR(), ledgerSeq)) + { + mTxSetsStoredByOverlay.insert(txSet->getContentsHash()); + } + // Leader's push: this set is about to be proposed, so send it to the + // peers now. + if (pushToPeers) + { + pushTxSetToPeers(txSet, ledgerSeq); + } +} + +void +HerderImpl::executePreparedTxSet() +{ + if (!mPreparedTxSet || !mPreparedTxSet->txSet || + !mPreparedTxSet->applicableTxSet) + { + return; + } +#ifdef BUILD_TESTS + // The trigger proposes something else than the prepared set. + if (mProposedTxSetOverride || + mApp.getConfig().TESTING_PROPOSE_RANDOM_VALUES) + { + return; + } +#endif + auto const lcl = mLedgerManager.getLastClosedLedgerHeader(); + if (mPreparedTxSet->ledgerSeq != lcl.header.ledgerSeq + 1 || + mPreparedTxSet->previousLedgerHash != lcl.hash || + !getSCP().isValidator()) + { + return; + } + // The value makeProposalValue builds at the trigger, if nothing changes + // until then. + auto const value = makeStellarValue( + mPreparedTxSet->txSet->getContentsHash(), mPreparedTxSet->closeTime, + proposalUpgrades(lcl), mApp.getConfig().NODE_SEED); + startSpeculativeApply(mPreparedTxSet->ledgerSeq, + Value(xdr::xdr_to_opaque(value)), + mPreparedTxSet->txSet); +} + +void +HerderImpl::startTxSetSelfCheck(std::shared_ptr check, + uint32_t ledgerSeq) +{ + if (!check) + { + return; + } + if (executesVotedValues(mLedgerManager.getLastClosedLedgerHeader())) + { + // This node's own execution of the set reads it back and validates + // it in full, and its verdict is taken there. The check is kept + // until the node has emitted a statement: if the ledger was not + // handed to the apply thread with this set by then (see + // resolvePendingTxSetSelfCheck), it runs after all. + mPendingTxSetSelfCheck = std::make_pair(std::move(check), ledgerSeq); + return; + } + runTxSetSelfCheck(std::move(check), ledgerSeq); +} + +void +HerderImpl::runTxSetSelfCheck(std::shared_ptr check, + uint32_t ledgerSeq) +{ + mApp.postOnBackgroundThread( + [this, check = std::move(check), ledgerSeq, &app = mApp]() { + ZoneScopedN("tx set self-check"); + auto const failure = check->run(app); + if (!failure) + { + app.getMetrics() + .NewMeter({"herder", "txset", "self-check-passed"}, "txset") + .Mark(); + return; + } + app.postOnMainThread( + [this, check, ledgerSeq, reason = *failure]() { + txSetSelfCheckFailed(check->getTxSet()->getContentsHash(), + ledgerSeq, reason); + }, + "tx set self-check failed"); + }, + "Herder: tx set self-check"); +} + +void +HerderImpl::resolvePendingTxSetSelfCheck() +{ + if (!mPendingTxSetSelfCheck) + { + return; + } + auto pending = std::move(*mPendingTxSetSelfCheck); + mPendingTxSetSelfCheck.reset(); + auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); + if (pending.second != lcl.header.ledgerSeq + 1) + { + // For a ledger that is no longer the next one: nothing to check. + return; + } + // Handed to the apply thread with this set? + StellarValue executing; + if (mSpeculativeApplyValue && + mSpeculativeApplyValue->first == pending.second && + toStellarValue(mSpeculativeApplyValue->second, executing) && + executing.txSetHash == pending.first->getTxSet()->getContentsHash()) + { + return; + } + runTxSetSelfCheck(std::move(pending.first), pending.second); +} + +bool +HerderImpl::executesVotedValues(LedgerHeaderHistoryEntry const& lcl) const +{ + auto const& cfg = mApp.getConfig(); + return cfg.EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY && + cfg.backgroundLedgerApply() && + protocolValidatesTxSetsAtApply(lcl.header.ledgerVersion); +} + +void +HerderImpl::txSetSelfCheckFailed(Hash const& txSetHash, uint32_t ledgerSeq, + std::string const& reason) +{ + releaseAssert(threadIsMain()); + CLOG_ERROR(Herder, + "TX set {} built for ledger {} failed the builder's " + "self-check: {}", + hexAbbrev(txSetHash), ledgerSeq, reason); + CLOG_ERROR(Herder, "{}", REPORT_INTERNAL_BUG); + mApp.getMetrics() + .NewMeter({"herder", "txset", "self-check-failed"}, "txset") + .Mark(); + // No more unchecked sets from this process. + mTxSetSelfCheckFailed = true; + if (mPreparedTxSet && mPreparedTxSet->txSet && + mPreparedTxSet->txSet->getContentsHash() == txSetHash) + { + // Not proposed yet: the trigger builds again, and checks what it + // builds before using it. + discardPreparedTxSet(); + } +} + +#ifdef BUILD_TESTS +void +HerderImpl::overrideBuiltTxSetForTesting( + TxSetXDRFrameConstPtr& txSet, + std::shared_ptr& selfCheck) +{ + if (!mBuiltTxSetOverrideForTesting) + { + return; + } + txSet = mBuiltTxSetOverrideForTesting(txSet); + if (selfCheck) + { + selfCheck = selfCheck->withTxSetForTesting(txSet); + } +} +#endif + +xdr::xvector +HerderImpl::proposalUpgrades(LedgerHeaderHistoryEntry const& lcl) +{ + auto newUpgrades = emptyUpgradeSteps; + + // see if we need to include some upgrades + std::vector upgrades; + { + auto ledgerView = mApp.getLedgerManager().getLCLView(); + upgrades = mUpgrades.createUpgradesFor(lcl.header, *ledgerView, + mApp.getConfig()); + } + for (auto const& upgrade : upgrades) + { + Value v(xdr::xdr_to_opaque(upgrade)); + if (v.size() >= UpgradeType::max_size()) + { + CLOG_ERROR( + Herder, + "HerderImpl::triggerNextLedger exceeded size for upgrade " + "step (got {} ) for upgrade type {}", + v.size(), upgrade.type()); + CLOG_ERROR(Herder, "{}", REPORT_INTERNAL_BUG); + } + else + { + newUpgrades.emplace_back(v.begin(), v.end()); + } + } + return newUpgrades; } void @@ -2787,6 +3092,64 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, : legacyTxSet ? std::string("validator (legacy tx set)") : std::string("first-round leader (early)"), ledgerSeq); + // This build validates every candidate itself. + stopSignaturePrewarm(); + auto& metrics = mApp.getMetrics(); + + // How long after this node's vote for the previous ledger its set for + // this one is needed: what the start of the next early selection is + // timed by (earlyTxSetStartDelay). + if (mFirstVote && mFirstVote->first + 1 == ledgerSeq && round <= 1) + { + // (the first build of the ledger, by the first round's leader: a + // rebuild at the trigger or a later round's build comes when it + // comes) + auto const votedAt = mFirstVote->second; + mFirstVote.reset(); + mLastVoteToBuild = + std::max(std::chrono::milliseconds(1), + std::chrono::duration_cast( + mApp.getClock().now() - votedAt)); + mVoteToBuild = mVoteToBuild == std::chrono::milliseconds::zero() + ? mLastVoteToBuild + : (3 * mVoteToBuild + mLastVoteToBuild) / 4; + } + + auto finish = [&](PreparedTxSet&& prepared, bool built) { + auto const buildDuration = + std::chrono::steady_clock::now() - txSetBuildStart; + mSCPMetrics.mTxSetBuild.Update( + std::chrono::duration_cast( + buildDuration)); + if (!built) + { + // A completed selection says nothing about what a build costs, + // which is what a build's lead time has to cover. + return std::move(prepared); + } + // The next build is scheduled far enough ahead of its trigger to fit + // one like this (see prepareTxSetLeadTime). + mLastTxSetBuildDuration = + std::chrono::duration_cast( + buildDuration); + if (prepared.txSet->sizeTxTotal() >= 100) + { + mTxSetBuildCostPerTx = + std::chrono::duration_cast( + buildDuration) / + static_cast(prepared.txSet->sizeTxTotal()); + } + return std::move(prepared); + }; + + // A set this node selected while the previous ledger was in its ballot + // only has to be completed. + if (auto completed = + completeEarlyTxSet(ledgerSeq, closeTime, pushToPeers, lcl)) + { + return finish(std::move(*completed), /*built=*/false); + } + // Apply-time offset: see the comment in triggerNextLedger auto const closeTimeOffset = closeTime.toApplyTime() - getApplyTime(lcl.header.scpValue); @@ -2796,77 +3159,121 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, // Build TX set from Rust overlay's mempool (not local TransactionQueue) // The Rust overlay maintains the mempool via TX flooding PerPhaseTransactionList txPhases; + bool const supportsSoroban = protocolVersionStartsFrom( + lcl.header.ledgerVersion, SOROBAN_PROTOCOL_VERSION); + auto* const validationCache = validationCacheFor(lcl); - // Get TXs from Rust overlay - auto& overlayMgr = mApp.getOverlayManager(); - // This build validates every candidate itself. - stopSignaturePrewarm(); - auto& metrics = mApp.getMetrics(); - auto txEnvelopes = [&]() { - auto t = - metrics.NewTimer({"herder", "txset", "build-pull"}).TimeScope(); - return pullTopTransactions(maxTxSetCandidates() * 2); - }(); - auto framesTimer = - metrics.NewTimer({"herder", "txset", "build-frames"}).TimeScope(); - - CLOG_INFO(Herder, "Got {} transactions from Rust overlay mempool", - txEnvelopes.size()); + // Signature cache lookups of this build, the validation for a refill + // included. + uint64_t sigCacheHits = 0; + uint64_t sigCacheMisses = 0; + PubKeyUtils::getVerifySigCacheCounts(sigCacheHits, sigCacheMisses); - // Convert TransactionEnvelopes to TransactionFrameBasePtrs and place them - // into the phase expected by TxSetFrame. + // The pull is the capacity of the set plus a margin. If validation + // takes so much out of a phase that it ends below its capacity while + // the mempool has more to give, the build pulls again, twice as much, + // and goes on with what it has not seen yet. TxFrameList classicTxs; TxFrameList sorobanTxs; - Hash const& networkID = mApp.getNetworkID(); - bool const supportsSoroban = protocolVersionStartsFrom( - lcl.header.ledgerVersion, SOROBAN_PROTOCOL_VERSION); - for (auto const& env : txEnvelopes) - { - auto txFrame = - TransactionFrameBase::makeTransactionFromWire(networkID, env); - if (txFrame->isSoroban()) + UnorderedSet seen; + bool mempoolExhausted = false; + size_t pulled = 0; + { + std::array const capacity{ + mLedgerManager.getLastMaxTxSetSizeOps(), + supportsSoroban && + mLedgerManager.hasLastClosedSorobanNetworkConfig() + ? static_cast( + mLedgerManager.getLastClosedSorobanNetworkConfig() + .ledgerMaxTxCount()) + : size_t(0)}; + // Per phase: how many candidates were pulled, and how much of the + // phase's capacity those of them fill that are valid and the first + // of their source account (a set holds one transaction per + // account). + std::array candidates{0, 0}; + std::array usable{0, 0}; + std::array filled{0, 0}; + UnorderedSet sources; + UnorderedMap feeTotals; + for (size_t pull = 0; pull < MAX_TX_SET_PULLS; ++pull) { - if (supportsSoroban) + bool full = false; + auto const fresh = + pullNewCandidates(txSetPullCount() << pull, seen, full); + pulled += fresh.size(); + std::array freshPhases; + for (auto const& tx : fresh) { - sorobanTxs.push_back(txFrame); + freshPhases[tx->isSoroban() ? 1 : 0].push_back(tx); } - else + classicTxs.insert(classicTxs.end(), freshPhases[0].begin(), + freshPhases[0].end()); + sorobanTxs.insert(sorobanTxs.end(), freshPhases[1].begin(), + freshPhases[1].end()); + if (!full) { - CLOG_DEBUG(Herder, - "Ignoring Soroban transaction before Soroban " - "protocol support"); + mempoolExhausted = true; + break; } - } - else - { - classicTxs.push_back(txFrame); - } - } - // The mempool is fee-ordered and sequence-number-oblivious, so it can - // hand us several transactions from one source account (e.g. a chained - // pair). A tx set may only contain one tx per source account, so keep - // the lowest sequence number per account and let the others wait for a - // later ledger. - auto onePerSourceAccount = [](TxFrameList& txs) { - std::unordered_map firstBySource; - TxFrameList kept; - for (auto const& tx : txs) - { - auto [it, inserted] = - firstBySource.emplace(tx->getSourceID(), kept.size()); - if (inserted) + if (pull + 1 == MAX_TX_SET_PULLS) + { + break; + } + // The mempool may hold more. More is needed if a phase lost + // candidates (to validation, or to the one-per-account rule) + // and what is left does not fill it: validate what is new for + // the phases that are not filled yet (the selection below finds + // the results in the cache). A phase that lost none got all the + // mempool's top holds for it. A phase that lost some goes on + // as long as each pull brings it something new: a stale + // transaction of a kind the mempool holds little of costs one + // deeper pull, not all of them. + auto refillTimer = + metrics.NewTimer({"herder", "txset", "build-refill-check"}) + .TimeScope(); + auto ctx = TxSetValidationContext::forLCL(mApp); + ctx.mValidationCache = validationCache; + bool shortPhase = false; + for (size_t p = 0; p < freshPhases.size(); ++p) { - kept.push_back(tx); + if (filled[p] >= capacity[p]) + { + continue; + } + TxFrameList notValid; + auto const ok = TxSetUtils::trimInvalid( + freshPhases[p], mApp, ctx, feeTotals, + closeTimeOffset.seconds(), closeTimeOffset.seconds(), + notValid); + candidates[p] += freshPhases[p].size(); + for (auto const& tx : ok) + { + if (sources.insert(tx->getSourceID()).second) + { + ++usable[p]; + filled[p] += p == 0 ? tx->getNumOperations() : 1; + } + } + shortPhase = + shortPhase || + (usable[p] < candidates[p] && filled[p] < capacity[p] && + (pull == 0 || !freshPhases[p].empty())); } - else if (tx->getSeqNum() < kept[it->second]->getSeqNum()) + if (!shortPhase) { - kept[it->second] = tx; + break; } + metrics.NewMeter({"herder", "txset", "refill-pull"}, "pull").Mark(); } - txs = std::move(kept); - }; - onePerSourceAccount(classicTxs); - onePerSourceAccount(sorobanTxs); + } + auto framesTimer = + metrics.NewTimer({"herder", "txset", "build-frames"}).TimeScope(); + + CLOG_INFO(Herder, "Got {} transactions from Rust overlay mempool", pulled); + + // One transaction per source account. + TxSetUtils::keepOnePerSourceAccount(classicTxs, sorobanTxs); txPhases.emplace_back(std::move(classicTxs)); if (supportsSoroban) @@ -2882,9 +3289,15 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, invalidTxPhases.resize(txPhases.size()); framesTimer.Stop(); - uint64_t sigCacheHits = 0; - uint64_t sigCacheMisses = 0; - PubKeyUtils::getVerifySigCacheCounts(sigCacheHits, sigCacheMisses); + // The builder's check of its own set does not hold up the set: it runs + // in the background (startTxSetSelfCheck), unless one has failed before. + std::shared_ptr selfCheck; + // Only where every node validates the set when it applies the ledger: + // before that protocol the followers validate before they vote, and + // this node records its own set as valid for them (cacheValidTxSet). + bool const deferSelfCheck = + !legacyTxSet && !mTxSetSelfCheckFailed && + protocolValidatesTxSetsAtApply(lcl.header.ledgerVersion); std::tie(proposedSet, applicableProposedSet) = makeTxSetFromTransactions( txPhases, mApp, closeTimeOffset, invalidTxPhases #ifdef BUILD_TESTS @@ -2893,36 +3306,35 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, #endif , [&](TxSetXDRFrameConstPtr const& txSet) { - auto t = metrics.NewTimer({"herder", "txset", "build-handoff"}) - .TimeScope(); - // Hand the final XDR to the overlay before the builder's remaining - // validation. Its eager compression runs concurrently in Rust. - if (txSet->isGeneralizedTxSet()) - { - // Serialized straight into the IPC message, without copying - // the set's XDR first. - if (overlayMgr.cacheTxSet(txSet->getContentsHash(), - txSet->getGeneralizedXDR(), - ledgerSeq)) - { - mTxSetsStoredByOverlay.insert(txSet->getContentsHash()); - } - // Leader's push: this set is about to be proposed, so send - // it to the peers now, while the builder's remaining - // validation and the proposal's persistence still run. If - // that validation fails nothing is proposed, and peers drop - // the unused set. - if (pushToPeers) - { - pushTxSetToPeers(txSet, ledgerSeq); - } - } + // Hand the final XDR to the overlay, and with `pushToPeers` to + // the peers, before the builder's check of the set. If that + // check fails before the proposal, nothing is proposed, and + // peers drop the unused set. + handOffTxSet(txSet, ledgerSeq, pushToPeers); }, - targets, validationCacheFor(lcl)); + targets, validationCache, deferSelfCheck ? &selfCheck : nullptr); + if (applicableProposedSet) + { + recordTxSetFill(*applicableProposedSet, mempoolExhausted); + } +#ifdef BUILD_TESTS + overrideBuiltTxSetForTesting(proposedSet, selfCheck); +#endif + mLastBuiltTxSet = std::make_pair(proposedSet->getContentsHash(), closeTime); + startTxSetSelfCheck(selfCheck, ledgerSeq); markSignatureCacheLookups(mApp, "build", sigCacheHits, sigCacheMisses); CLOG_INFO(Herder, "Proposed TX set has {} transactions", proposedSet->sizeTxTotal()); + return finish(PreparedTxSet{lcl.hash, ledgerSeq, closeTime, proposedSet, + std::move(applicableProposedSet), + invalidTxsToRemove(invalidTxPhases)}, + /*built=*/true); +} + +std::vector +HerderImpl::invalidTxsToRemove(PerPhaseTransactionList const& invalidTxs) +{ // The mempool does no stateful validation, so it would keep handing us the // transactions that just failed validation (stale sequence number, can't // pay fee, expired, ...) on every nomination, crowding out valid ones. @@ -2930,10 +3342,10 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, // a *future* sequence number: those are chained behind a pending // transaction from the same account and become valid once it applies. std::vector invalidTxHashes; - if (!invalidTxPhases.empty()) + if (!invalidTxs.empty()) { auto ledgerView = mLedgerManager.getLCLView(); - for (auto const& phase : invalidTxPhases) + for (auto const& phase : invalidTxs) { for (auto const& tx : phase) { @@ -2952,28 +3364,565 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, } } } + return invalidTxHashes; +} + +size_t +HerderImpl::txSetPullCount() const +{ + size_t const capacity = maxTxSetCandidates(); + size_t const margin = + (capacity * mApp.getConfig().EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT + + 99) / + 100; + return std::max(1, capacity + margin); +} - auto const buildDuration = - std::chrono::steady_clock::now() - txSetBuildStart; - mSCPMetrics.mTxSetBuild.Update( - std::chrono::duration_cast(buildDuration)); - // The next build is scheduled far enough ahead of its trigger to fit one - // like this (see prepareTxSetLeadTime). - mLastTxSetBuildDuration = - std::chrono::duration_cast(buildDuration); - if (proposedSet->sizeTxTotal() >= 100) +TxFrameList +HerderImpl::pullNewCandidates(size_t count, UnorderedSet& seen, + bool& full) +{ + auto& metrics = mApp.getMetrics(); + auto const payload = [&]() { + auto t = + metrics.NewTimer({"herder", "txset", "build-pull"}).TimeScope(); + return pullTopTransactionsPayload(count); + }(); + auto framesTimer = + metrics.NewTimer({"herder", "txset", "build-frames"}).TimeScope(); + uint32_t pulled = 0; + if (payload.size() >= 4) { - mTxSetBuildCostPerTx = - std::chrono::duration_cast( - buildDuration) / - static_cast(proposedSet->sizeTxTotal()); + std::memcpy(&pulled, payload.data(), 4); + } + full = pulled >= count; + TxFrameList fresh; + Hash const& networkID = mApp.getNetworkID(); + bool const supportsSoroban = protocolVersionStartsFrom( + mLedgerManager.getLastClosedLedgerHeader().header.ledgerVersion, + SOROBAN_PROTOCOL_VERSION); + forEachPulledTx(payload, [&](uint8_t const* data, size_t length) { + if (!seen.insert(hashPulledTx(data, length)).second) + { + return; + } + TransactionEnvelope env; + if (!decodePulledTx(data, length, env)) + { + CLOG_WARNING(Herder, "Failed to parse a pulled transaction"); + return; + } + auto tx = TransactionFrameBase::makeTransactionFromWire(networkID, env); + if (tx->isSoroban() && !supportsSoroban) + { + CLOG_DEBUG(Herder, "Ignoring Soroban transaction before Soroban " + "protocol support"); + return; + } + fresh.push_back(std::move(tx)); + }); + return fresh; +} + +void +HerderImpl::recordTxSetFill(ApplicableTxSetFrame const& txSet, + bool mempoolExhausted) +{ + auto& metrics = mApp.getMetrics(); + auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); + // A phase below its capacity although the mempool had more to give: + // the build stopped looking (its pulls are bounded), or what it found + // did not fit. A phase without transactions is left out: the mempool's + // top held no candidate of its kind, whatever else the mempool holds. + auto record = [&](char const* fill, char const* isShort, size_t size, + size_t capacity) { + if (capacity == 0) + { + return; + } + metrics.NewHistogram({"herder", "txset", fill}) + .Update(static_cast( + std::min(100, size * 100 / capacity))); + if (size != 0 && size < capacity && !mempoolExhausted) + { + metrics.NewMeter({"herder", "txset", isShort}, "txset").Mark(); + } + }; + record("fill-classic", "short-classic", txSet.sizeOp(TxSetPhase::CLASSIC), + mLedgerManager.getLastMaxTxSetSizeOps()); + if (txSet.numPhases() > static_cast(TxSetPhase::SOROBAN) && + protocolVersionStartsFrom(lcl.header.ledgerVersion, + SOROBAN_PROTOCOL_VERSION) && + mLedgerManager.hasLastClosedSorobanNetworkConfig()) + { + record("fill-soroban", "short-soroban", + txSet.sizeTx(TxSetPhase::SOROBAN), + mLedgerManager.getLastClosedSorobanNetworkConfig() + .ledgerMaxTxCount()); + } +} + +std::chrono::milliseconds +HerderImpl::earlyTxSetStartDelay() const +{ + using std::chrono::milliseconds; +#ifdef BUILD_TESTS + if (mEarlyTxSetStartDelayForTesting) + { + return *mEarlyTxSetStartDelayForTesting; + } +#endif + if (mVoteToBuild == milliseconds::zero() || + mEarlyTxSetJobDuration == milliseconds::zero()) + { + // Nothing measured yet: start at once. + return milliseconds::zero(); + } + // The earlier of what the build did lately and what it did last time, + // less a selection that takes as long as the slower of the two, with a + // quarter to spare, and 30 ms for the hand-over. + auto const untilBuild = std::min(mVoteToBuild, mLastVoteToBuild); + auto const job = + std::max(mEarlyTxSetJobDuration, mLastEarlyTxSetJobDuration); + auto const delay = untilBuild - job - job / 4 - milliseconds(30); + return std::max(milliseconds::zero(), delay); +} + +void +HerderImpl::stopEarlyTxSetSelection() +{ + mEarlyTxSetTimer.cancel(); + if (mEarlyTxSet) + { + std::lock_guard lock(mEarlyTxSet->mMutex); + mEarlyTxSet->mStop.store(true); + } + mEarlyTxSet.reset(); + mEarlyTxSetPlan.reset(); +} + +void +HerderImpl::maybeScheduleEarlyTxSetSelection(SCPEnvelope const& envelope) +{ + auto const& cfg = mApp.getConfig(); + if (!cfg.EXPERIMENTAL_EARLY_TX_SET_SELECTION || !getSCP().isValidator()) + { + return; + } +#ifdef BUILD_TESTS + // The bucket list isn't populated in this mode: there is no snapshot of + // the last closed ledger to read from other threads. + if (cfg.MODE_USES_IN_MEMORY_LEDGER) + { + return; + } +#endif + // The value this node is working on: what it votes to prepare, or to + // commit. + auto const& st = envelope.statement; + Value const* value = nullptr; + switch (st.pledges.type()) + { + case SCP_ST_PREPARE: + value = &st.pledges.prepare().ballot.value; + break; + case SCP_ST_CONFIRM: + value = &st.pledges.confirm().ballot.value; + break; + default: + return; + } + auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); + if (st.slotIndex != lcl.header.ledgerSeq + 1 || + !protocolValidatesTxSetsAtApply(lcl.header.ledgerVersion) || + buildsLegacyTxSet(lcl.header.ledgerVersion) || !isTracking() || + !mLedgerManager.isSynced()) + { + return; + } + if (!mFirstVote || mFirstVote->first != st.slotIndex) + { + mFirstVote = std::make_pair(st.slotIndex, mApp.getClock().now()); + } + if (mEarlyTxSetPlan && mEarlyTxSetPlan->slotIndex == st.slotIndex && + mEarlyTxSetPlan->value == *value) + { + // Looked at already. + return; + } + // The first vote for this ledger, or a vote for another value: whatever + // was planned or running was for a value this node no longer votes for. + stopEarlyTxSetSelection(); + EarlyTxSetPlan plan; + plan.slotIndex = st.slotIndex; + plan.value = *value; + mEarlyTxSetPlan = plan; + + // No selection on a ledger with upgrades: they change what the next set + // is validated against and the limits it is selected within. None after + // an empty set either: nothing is known about the next ledger then that + // a build would not know. + StellarValue sv; + if (!toStellarValue(*value, sv) || isEmptyTxSetStellarValue(sv) || + !sv.upgrades.empty()) + { + return; + } + auto txSetResult = mPendingEnvelopes.getTxSet(sv.txSetHash); + auto const* received = std::get_if(&txSetResult); + if (!received || !*received) + { + // Not in hand yet: look again at the next statement for the value. + mEarlyTxSetPlan.reset(); + return; + } + if (!(*received)->isGeneralizedTxSet() || + (*received)->previousLedgerHash() != lcl.hash || + (*received)->sizeTxTotal() == 0) + { + return; + } + // Only the node first in line for the next ledger. + auto const schedule = mHerderSCPDriver.getExpectedLeaderSchedule( + st.slotIndex + 1, sv, /*appliedTransactions=*/true); + if (schedule.empty() || !(schedule[0] == getSCP().getLocalNodeID())) + { + return; + } + mEarlyTxSetPlan->stellarValue = sv; + mEarlyTxSetPlan->planned = true; + mApp.getMetrics() + .NewMeter({"herder", "txset", "early-planned"}, "txset") + .Mark(); + // Through the timer even without a delay: this is called while SCP + // emits an envelope, and the start pulls the mempool. + mEarlyTxSetTimer.expires_from_now(earlyTxSetStartDelay()); + mEarlyTxSetTimer.async_wait( + [this, slotIndex = st.slotIndex, value = *value, txSet = *received]() { + startEarlyTxSetSelection(slotIndex, value, txSet); + }, + &VirtualTimer::onFailureNoop); +} + +void +HerderImpl::startEarlyTxSetSelection(uint64_t slotIndex, Value const& value, + TxSetXDRFrameConstPtr const& txSet) +{ + ZoneScoped; + releaseAssert(threadIsMain()); + if (mApp.isStopping()) + { + return; + } + auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); + // Only while the ledger is still the next one to close and this node + // still votes for the value. + if (!mEarlyTxSetPlan || !mEarlyTxSetPlan->planned || mEarlyTxSet || + mEarlyTxSetPlan->slotIndex != slotIndex || + !(mEarlyTxSetPlan->value == value) || + slotIndex != lcl.header.ledgerSeq + 1 || + txSet->previousLedgerHash() != lcl.hash || !isTracking()) + { + return; + } + auto& metrics = mApp.getMetrics(); + auto job = std::make_shared(); + job->mStart = std::chrono::steady_clock::now(); + job->mClosingLedgerSeq = static_cast(slotIndex); + job->mClosingValue = mEarlyTxSetPlan->stellarValue; + job->mClosingTxSet = txSet; + job->mPreviousLedger = mLedgerManager.getLastClosedLedgerState(); + if (job->mPreviousLedger->getLastClosedLedgerHeader().hash != lcl.hash) + { + return; + } + // The closing ledger's transactions are still in the mempool: look past + // that many more. + job->mPayload = [&]() { + auto timer = + metrics.NewTimer({"herder", "txset", "early-pull"}).TimeScope(); + return std::make_shared const>( + pullTopTransactionsPayload(txSetPullCount() + + txSet->sizeTxTotal())); + }(); + { + // The candidates are validated for the close time the next trigger + // is expected to have: the target after this ledger's close time, + // or now if that has passed. The completion validates them again + // for the close time the set really gets. + auto const version = lcl.header.ledgerVersion; + auto const closingCloseTime = getConsensusTime(job->mClosingValue); + auto const expectedTrigger = std::max( + closingCloseTime.toSystemTime() + + mLedgerManager.getExpectedLedgerCloseTime(), + mApp.getClock().system_now()); + auto const assumedCloseTime = + std::max(ConsensusTime::fromSystemTime(expectedTrigger, version), + closingCloseTime.next(version)); + job->mAssumedCloseTimeOffset = + (assumedCloseTime.toApplyTime() - getApplyTime(job->mClosingValue)) + .seconds(); + } + { + // What orders candidates with equal fee rates must not be + // predictable by who submits them. The global random engine is not + // used: whether a job runs depends on timing, and a draw here would + // shift every later draw of this node. A hash keyed with a secret + // of this node is as unpredictable to others. + if (!mTieBreakSecret) + { + mTieBreakSecret = + sha256(mApp.getConfig().NODE_SEED.getStrKeySeed().value + + " tx set tie-break"); + } + SHA256 hasher; + hasher.add(*mTieBreakSecret); + hasher.add(xdr::xdr_to_opaque(job->mClosingLedgerSeq, + job->mClosingValue.txSetHash)); + auto const digest = hasher.finish(); + static_assert(sizeof(size_t) <= sizeof(digest)); + std::memcpy(&job->mSelectionSeed, digest.data(), sizeof(size_t)); + } + // As many background tasks as the batch executor would use, but leave + // worker threads for long background jobs such as bucket merges. + size_t const workers = static_cast(mApp.getConfig().WORKER_THREADS); + job->mTasks = std::max( + 1, std::min(mApp.getBatchExecutor().preferredTaskCount(), + workers > 2 ? workers - 2 : 1)); + job->mPendingTasks = job->mTasks; + job->mTaskResults.resize(job->mTasks); +#ifdef BUILD_TESTS + job->mHoldForTesting = mEarlyTxSetHoldForTesting; +#endif + mEarlyTxSet = job; + metrics.NewMeter({"herder", "txset", "early-started"}, "txset").Mark(); + CLOG_DEBUG(Herder, + "Selecting TX set for ledger {} early, while ledger {} is in " + "its ballot: {} tasks", + slotIndex + 1, slotIndex, job->mTasks); + for (size_t task = 0; task < job->mTasks; ++task) + { + mApp.postOnBackgroundThread( + [job, task, &app = mApp]() { + EarlyTxSetJob::runTask(app, job, task); + }, + "Herder: early tx set selection"); + } +} + +std::optional +HerderImpl::completeEarlyTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, + bool pushToPeers, + LedgerHeaderHistoryEntry const& lcl) +{ + ZoneScoped; + releaseAssert(threadIsMain()); + auto& metrics = mApp.getMetrics(); + auto const plan = mEarlyTxSetPlan; + auto const job = mEarlyTxSet; + // Whatever comes of it, the plan ends here: it was for this build. + stopEarlyTxSetSelection(); + if (!plan || !plan->planned || mLedgerManager.isApplying()) + { + return std::nullopt; + } + auto mark = [&](char const* name) { + metrics.NewMeter({"herder", "txset", name}, "txset").Mark(); + }; + auto discard = [&](char const* why) { + mark("early-discarded"); + CLOG_INFO(Herder, + "Not using the TX set selected early for ledger {}: {}", + ledgerSeq, why); + return std::nullopt; + }; + // The selection was made for one ledger closing with one value. + if (plan->slotIndex != lcl.header.ledgerSeq || + ledgerSeq != lcl.header.ledgerSeq + 1 || + !(plan->stellarValue == lcl.header.scpValue) || + !lcl.header.scpValue.upgrades.empty() || + buildsLegacyTxSet(lcl.header.ledgerVersion)) + { + return discard("the ledger did not close with the value it is for"); + } + if (!job) + { + // The ledger closed before the selection was due to start. + mark("early-not-started"); + return std::nullopt; + } + std::shared_ptr early; + { + std::lock_guard lock(job->mMutex); + early = std::move(job->mSelection); + } + if (!early) + { + // Not finished (or failed): the usual build. Nobody waits for the + // job. It needed more time than it had: the next one starts earlier + // by half of that again (earlyTxSetStartDelay). + mark("early-not-ready"); + if (!job->mDone.load()) + { + auto const had = + std::chrono::duration_cast( + std::chrono::steady_clock::now() - job->mStart); + mLastEarlyTxSetJobDuration = + std::min(std::chrono::milliseconds(2000), + std::max(mLastEarlyTxSetJobDuration, had + had / 2)); + mEarlyTxSetJobDuration = + std::max(mEarlyTxSetJobDuration, mLastEarlyTxSetJobDuration); + } + return std::nullopt; + } + { + auto const took = + std::chrono::nanoseconds(job->mFinishedAfterNs.load()); + metrics.NewTimer({"herder", "txset", "early-job"}).Update(took); + mLastEarlyTxSetJobDuration = std::max( + std::chrono::milliseconds(1), + std::chrono::duration_cast(took)); + mEarlyTxSetJobDuration = + mEarlyTxSetJobDuration == std::chrono::milliseconds::zero() + ? mLastEarlyTxSetJobDuration + : (3 * mEarlyTxSetJobDuration + mLastEarlyTxSetJobDuration) / 4; + } + if (early->mLedgerSeq != ledgerSeq) + { + return discard("it is not for this ledger"); + } + // Nothing but an upgrade changes the limits a set is selected within, + // and the closed value had none. Checked all the same. + auto const limits = TxSetBuildLimits::forLCL(mApp); + if (!limits.sameAs(early->mLimits)) + { + return discard("the limits of the ledger changed"); } + + auto timer = + metrics.NewTimer({"herder", "txset", "early-complete"}).TimeScope(); + auto const closeTimeOffset = + closeTime.toApplyTime() - getApplyTime(lcl.header.scpValue); + // The closed ledger, read as a build reads it. No validation cache: + // every transaction of the selection is validated here and now. + auto ctx = TxSetValidationContext::forLCL(mApp); + PerPhaseTransactionList invalidTxPhases; + invalidTxPhases.resize(early->mPhases.size()); + TxSetCompletionStats stats; + + // Further candidates for the top-up: what the job decoded and did not + // select, then what the mempool holds that the job did not see, in + // pulls of growing size while they come back full. + auto seen = std::move(early->mSeen); + bool leftoversGiven = false; + bool mempoolExhausted = false; + size_t pulls = 0; + TxSetCandidateSource const moreCandidates = + [&]() -> std::optional { + if (!leftoversGiven) + { + leftoversGiven = true; + return std::make_optional( + TxSetCandidateBatch{std::move(early->mLeftovers), false}); + } + // One pull per call, twice as much each time, handed over also when + // it holds nothing that was not seen before: a pull full of known + // transactions says nothing about what lies behind them, and the + // completion decides whether a phase is worth a deeper one. + if (mempoolExhausted || pulls >= MAX_TX_SET_PULLS) + { + return std::nullopt; + } + bool full = false; + auto fresh = pullNewCandidates(txSetPullCount() << pulls, seen, full); + if (pulls > 0) + { + metrics.NewMeter({"herder", "txset", "refill-pull"}, "pull").Mark(); + } + ++pulls; + mempoolExhausted = !full; + return std::make_optional(TxSetCandidateBatch{std::move(fresh), true}); + }; + + TxSetXDRFrameConstPtr txSet; + ApplicableTxSetFrameConstPtr applicableTxSet; + std::shared_ptr selfCheck; + try + { +#ifdef BUILD_TESTS + if (mSkipEarlyTxSetFinalPassForTesting) + { + // What the node would propose if it trusted the selection. + std::tie(txSet, applicableTxSet) = TxSetBuilder::writeSelection( + early->mPhases, mApp, ctx, closeTimeOffset, + [&](TxSetXDRFrameConstPtr const& written) { + handOffTxSet(written, ledgerSeq, pushToPeers); + }, + &selfCheck); + } + else +#endif + { + auto const phases = TxSetBuilder::completeSelection( + early->mPhases, early->mHadTxNotFittingLane, moreCandidates, + mApp, limits, ctx, closeTimeOffset, invalidTxPhases, stats, + early->mSelectionSeed); + // As in a build: the structure of the selection is validated + // before the set is written and handed over, the written set + // in the background unless such a check has failed before. + std::tie(txSet, applicableTxSet) = TxSetBuilder::writeSelection( + phases, mApp, ctx, closeTimeOffset, + [&](TxSetXDRFrameConstPtr const& written) { + handOffTxSet(written, ledgerSeq, pushToPeers); + }, + mTxSetSelfCheckFailed ? nullptr : &selfCheck); + } + } + catch (std::exception const& e) + { + // The completion made something that is not a valid set. Nothing of + // it is proposed: the build that follows starts from the mempool, + // and from now on this node checks every set in line. + CLOG_ERROR(Herder, + "Completing the TX set selected early for ledger {} " + "failed: {}", + ledgerSeq, e.what()); + CLOG_ERROR(Herder, "{}", REPORT_INTERNAL_BUG); + mark("early-failed"); + mTxSetSelfCheckFailed = true; + return std::nullopt; + } + timer.Stop(); + // (no pull: every phase was full with what the job had left) + recordTxSetFill(*applicableTxSet, mempoolExhausted || pulls == 0); +#ifdef BUILD_TESTS + overrideBuiltTxSetForTesting(txSet, selfCheck); +#endif + mLastBuiltTxSet = std::make_pair(txSet->getContentsHash(), closeTime); + startTxSetSelfCheck(selfCheck, ledgerSeq); + // The validation results of an earlier build on this ledger stay + // available to a later one (a rebuild at the trigger, a later round's). + validationCacheFor(lcl); + + mark("early-completed"); + metrics.NewMeter({"herder", "txset", "early-removed"}, "tx") + .Mark(stats.mRemoved); + metrics.NewMeter({"herder", "txset", "early-added"}, "tx") + .Mark(stats.mAdded); + metrics.NewTimer({"herder", "txset", "early-lead"}) + .Update(std::chrono::steady_clock::now() - early->mSelectedAt); + CLOG_INFO(Herder, + "Completed the TX set selected early for ledger {}: {} " + "selected, {} not valid on ledger {}, {} of {} further " + "candidates added ({} pulls), {} transactions", + ledgerSeq, stats.mSelected, stats.mRemoved, lcl.header.ledgerSeq, + stats.mAdded, stats.mCandidates, pulls, txSet->sizeTxTotal()); return PreparedTxSet{lcl.hash, ledgerSeq, closeTime, - proposedSet, - std::move(applicableProposedSet), - std::move(invalidTxHashes)}; + txSet, + std::move(applicableTxSet), + invalidTxsToRemove(invalidTxPhases)}; } // called to take a position during the next round @@ -3156,6 +4105,10 @@ std::vector HerderImpl::pullTopTransactionsPayload(size_t count) { #ifdef BUILD_TESTS + if (mGetTopTransactionsPayloadForTesting) + { + return mGetTopTransactionsPayloadForTesting(count); + } if (mGetTopTransactionsForTesting) { // Encode the test mempool's answer as the overlay does. @@ -3475,32 +4428,7 @@ HerderImpl::makeProposalValue(uint32_t ledgerSeqToTrigger) return nullptr; } - auto newUpgrades = emptyUpgradeSteps; - - // see if we need to include some upgrades - std::vector upgrades; - { - auto ledgerView = mApp.getLedgerManager().getLCLView(); - upgrades = mUpgrades.createUpgradesFor(lcl.header, *ledgerView, - mApp.getConfig()); - } - for (auto const& upgrade : upgrades) - { - Value v(xdr::xdr_to_opaque(upgrade)); - if (v.size() >= UpgradeType::max_size()) - { - CLOG_ERROR( - Herder, - "HerderImpl::triggerNextLedger exceeded size for upgrade " - "step (got {} ) for upgrade type {}", - v.size(), upgrade.type()); - CLOG_ERROR(Herder, "{}", REPORT_INTERNAL_BUG); - } - else - { - newUpgrades.emplace_back(v.begin(), v.end()); - } - } + auto const newUpgrades = proposalUpgrades(lcl); #ifdef BUILD_TESTS if (mApp.getConfig().TESTING_PROPOSE_RANDOM_VALUES && diff --git a/src/herder/HerderImpl.h b/src/herder/HerderImpl.h index c4f1b9369..7c2beea97 100644 --- a/src/herder/HerderImpl.h +++ b/src/herder/HerderImpl.h @@ -4,6 +4,7 @@ #pragma once +#include "herder/EarlyTxSetSelection.h" #include "herder/Herder.h" #include "herder/HerderSCPDriver.h" #include "herder/LedgerCloseData.h" @@ -152,6 +153,13 @@ class HerderImpl : public Herder // apply thread for speculative execution (see // LedgerManager::startSpeculativeApply), unless it was already. void maybeStartSpeculativeApply(SCPEnvelope const& envelope); + // Hands the ledger `slotIndex` with `value` to the apply thread for + // speculative execution if it is the ledger after the LCL, nothing is + // being applied and it was not handed over already. `txSet` is the + // value's transaction set if the caller holds it; otherwise it is looked + // up among the sets this node has received. + void startSpeculativeApply(uint64_t slotIndex, Value const& value, + TxSetXDRFrameConstPtr txSet); #ifdef BUILD_TESTS TxSubmitStatus recvTransaction(TransactionFrameBasePtr tx, @@ -188,8 +196,30 @@ class HerderImpl : public Herder // still run through their production paths. std::function(size_t)> mGetTopTransactionsForTesting; + // The same for callers that take the pull as the overlay encodes it + // (pullTopTransactionsPayload); when not set, the envelopes of + // mGetTopTransactionsForTesting are encoded. + std::function(size_t)> + mGetTopTransactionsPayloadForTesting; friend class LeaderBuildTestAccess; friend class ProposalPrevalidationTestAccess; + friend class EarlyTxSetTestAccess; + // When set, replaces the delay between this node's vote for a ledger + // and the start of its early selection for the next one + // (earlyTxSetStartDelay). + std::optional mEarlyTxSetStartDelayForTesting; + // When set, the validity pass over an early selection on the closed + // ledger is skipped: for the tests that show what that pass is for. + bool mSkipEarlyTxSetFinalPassForTesting{false}; + // Handed to every early selection job that starts (see + // EarlyTxSetJob::mHoldForTesting). + std::shared_ptr> mEarlyTxSetHoldForTesting; + + // When set, a set this node just built as a leader is replaced by what + // this returns, for the proposal and for the builder's self-check alike: + // lets a test make the node build a broken set. + std::function + mBuiltTxSetOverrideForTesting; std::optional mMaxClassicTxSize; std::optional mMaxTxSizeOverride; @@ -345,6 +375,8 @@ class HerderImpl : public Herder // rose (every node's first term under load starts that way), and a lead // time sized for it would make the first ledger of the term late. std::chrono::milliseconds expectedTxSetBuildDuration() const; + // What a build of a set of `txs` transactions costs at this node's rate. + std::chrono::milliseconds expectedTxSetBuildDuration(size_t txs) const; std::optional mPreparedTxSet; // `pushToPeers`: this is the set the node is about to propose as a round // leader, so the overlay sends it to every peer as soon as its XDR exists @@ -359,6 +391,82 @@ class HerderImpl : public Herder void prepareTxSet(uint32_t ledgerSeq, ConsensusTime closeTime); void discardPreparedTxSet(); ValueWrapperPtr makeProposalValue(uint32_t ledgerSeq); + // The upgrades a value proposed now on top of `lcl` carries. + xdr::xvector + proposalUpgrades(LedgerHeaderHistoryEntry const& lcl); + // Hands a set this node built to the overlay, which keeps it for peers + // that ask and, with `pushToPeers`, sends it to every peer at once. + void handOffTxSet(TxSetXDRFrameConstPtr const& txSet, uint32_t ledgerSeq, + bool pushToPeers); + + // The leader executes its own set as soon as it can. + // + // A prepared set is proposed unchanged at the trigger, with the close + // time it was prepared for. So the value this node will propose is known + // as soon as the set is: the node hands the ledger to the apply thread + // right then (startSpeculativeApply) instead of when it emits its first + // ballot statement at the trigger, and the execution runs beside the + // set's dissemination and the wait for the trigger. If the trigger ends + // up proposing another value (the prepared set was dropped, the upgrades + // to propose changed), the statement it emits replaces the execution, as + // any change of vote does. Called whenever mPreparedTxSet has been set. + void executePreparedTxSet(); + + // The builder's self-check (see TxSetSelfCheck) of a set built by this + // node as a leader does not hold up the push of the set or its proposal. + // It re-reads the written set and validates the structure of what it + // read. A node that executes its own value speculatively does exactly + // that, and more, on its apply thread as soon as it proposes (or as soon + // as the set is prepared, see executePreparedTxSet), and the verdict of + // that execution on a set this node built takes the same path as a + // failed self-check (see startSpeculativeApply): there the check would + // be done twice, so it is left to the execution. Otherwise it runs on a + // background thread. Either way, if it fails the set may already be + // proposed: see txSetSelfCheckFailed. + void startTxSetSelfCheck(std::shared_ptr check, + uint32_t ledgerSeq); + // Whether this node hands the ledger after `lcl` to its apply thread + // when it votes for a value (see maybeStartSpeculativeApply). + bool executesVotedValues(LedgerHeaderHistoryEntry const& lcl) const; + // Runs the check on a background thread. + void runTxSetSelfCheck(std::shared_ptr check, + uint32_t ledgerSeq); + // A check that was left to the node's own execution of the set, and the + // ledger the set is for, until it is known whether that execution + // happens. Called when the node has emitted a statement: the check is + // dropped if the apply thread was handed the ledger with this set, and + // run in the background otherwise (the node was not tracking, was + // applying, or voted for another value). + void resolvePendingTxSetSelfCheck(); + std::optional, uint32_t>> + mPendingTxSetSelfCheck; + // Called on the main thread when a self-check failed, or when this + // node's own execution of a set it built found the set invalid. The + // failure is logged and counted. A set that has not been proposed yet is + // dropped, + // so that the trigger builds another one. A proposed set cannot be + // recalled; if it is invalid, every node finds that at apply and closes + // the ledger without applying it. From now on this node checks every set + // before it uses it (mTxSetSelfCheckFailed), where a failure throws. + void txSetSelfCheckFailed(Hash const& txSetHash, uint32_t ledgerSeq, + std::string const& reason); + bool mTxSetSelfCheckFailed{false}; + // The last set this node built itself (by a build, or by completing a + // set it selected early), and the close time it built it for. When the + // node's own apply thread executes that set with that close time and + // finds it invalid, that is a verdict on this node's builder (see + // startSpeculativeApply). With another close time it is not: another + // leader may propose this node's set for a time it was not built for. + std::optional> mLastBuiltTxSet; + // Keys the tie-break seed of early selections; derived from the node + // seed on first use. + std::optional mTieBreakSecret; +#ifdef BUILD_TESTS + // Applies mBuiltTxSetOverrideForTesting to a set that was just built. + void overrideBuiltTxSetForTesting( + TxSetXDRFrameConstPtr& txSet, + std::shared_ptr& selfCheck); +#endif // Whether a ledger closing on an LCL with this protocol uses a legacy // (non-generalized) tx set, which every validator builds locally. static bool buildsLegacyTxSet(uint32_t lclLedgerVersion); @@ -429,6 +537,96 @@ class HerderImpl : public Herder // Nanoseconds from `start` until the last task finished, or -1. std::atomic finishedAfterNs{-1}; }; + // The part of setupTriggerNextLedger that arms the timer for the + // first-round leader's build a lead time ahead of the trigger at + // `triggerTime`; nothing if the trigger is due (it builds itself). + void scheduleTxSetPreparation(uint64_t nextIndex, + VirtualClock::time_point triggerTime, + bool triggerDue, VirtualClock::time_point now, + std::chrono::milliseconds expectedCloseTime, + ConsensusTime minCandidateCt); + // Early selection of the next transaction set (see + // herder/EarlyTxSetSelection.h). + // + // When this node votes for a value for ledger N and expects to lead + // round 1 of N+1 if N closes with it, it plans a background selection of + // the set for N+1 (maybeScheduleEarlyTxSetSelection), started as late + // as it can be and still finish before the set is needed + // (earlyTxSetStartDelay): the candidates of N+1 arrive while N is in its + // ballot. The selection is made on N-1's state and a prediction of what + // N's set does to it. + // + // The build of N+1 (buildTxSet) uses the selection if the job has + // finished and N closed with exactly the value the job was for + // (completeEarlyTxSet): every transaction of the selection is validated + // against N as it closed, what is not valid is removed, and the set is + // topped up from what the job left and from the mempool. Otherwise the + // set is built as always. + struct EarlyTxSetPlan + { + // The ledger in its ballot (N) and the value this node votes for. + uint64_t slotIndex{0}; + Value value; + StellarValue stellarValue; + // False if the vote was looked at and no selection is to be made + // for it (this node is not first in line, the value has upgrades, + // ...). + bool planned{false}; + }; + std::optional mEarlyTxSetPlan; + // The running or finished selection of the plan. + std::shared_ptr mEarlyTxSet; + // Called with every statement this node emits. + void maybeScheduleEarlyTxSetSelection(SCPEnvelope const& envelope); + void startEarlyTxSetSelection(uint64_t slotIndex, Value const& value, + TxSetXDRFrameConstPtr const& txSet); + // Drops the plan, its timer and its job. + void stopEarlyTxSetSelection(); + // How long after its vote for a ledger this node starts the selection + // for the next one: so that it is expected to finish a little before + // the build that uses it. Zero until the node has measured both (how + // long after its vote it builds, and how long a selection takes). + std::chrono::milliseconds earlyTxSetStartDelay() const; + // When this node first voted for a value of the ledger after the LCL. + std::optional> mFirstVote; + // From that vote to the start of the build of the ledger after: a + // running average and the last sample; zero before the first. + std::chrono::milliseconds mVoteToBuild{0}; + std::chrono::milliseconds mLastVoteToBuild{0}; + // How long a selection job takes (pull to result), likewise. + std::chrono::milliseconds mEarlyTxSetJobDuration{0}; + std::chrono::milliseconds mLastEarlyTxSetJobDuration{0}; + // Completes the early selection for `ledgerSeq` on top of `lcl`, as + // buildTxSet builds a set; nullopt if there is none that can be used + // (the caller then builds). Always ends the plan. + std::optional + completeEarlyTxSet(uint32_t ledgerSeq, ConsensusTime closeTime, + bool pushToPeers, LedgerHeaderHistoryEntry const& lcl); + + // How many transactions one pull of the mempool asks for: the capacity + // of a set plus EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT. A set that is + // short after validation pulls again, twice as much each time, at most + // MAX_TX_SET_PULLS times. + size_t txSetPullCount() const; + static constexpr size_t MAX_TX_SET_PULLS{3}; + // Pulls the top `count` transactions of the mempool and returns those + // not in `seen` (by hashPulledTx; they are added to it), decoded, in + // mempool order. `full` tells whether the mempool had `count` + // transactions to give, i.e. may hold more. + TxFrameList pullNewCandidates(size_t count, UnorderedSet& seen, + bool& full); + // The transactions of `invalidTxs` (found invalid on the LCL by a build + // or a completion) to remove from the mempool: all but those with a + // future sequence number, which wait for a pending transaction of their + // account. + std::vector + invalidTxsToRemove(PerPhaseTransactionList const& invalidTxs); + // Records a set this node is about to use: its size against the + // capacity of the phases, and whether it is short although the mempool + // had more to give. + void recordTxSetFill(ApplicableTxSetFrame const& txSet, + bool mempoolExhausted); + std::shared_ptr mProposalPrevalidation; // Whether the local node would lead round 1 or 2 of the ledger after // `slotIndex` if `slotIndex` closes with `value`. @@ -548,6 +746,8 @@ class HerderImpl : public Herder VirtualTimer mTriggerTimer; VirtualTimer mPrepareTxSetTimer; + // Fires when the planned early selection is to start (mEarlyTxSetPlan). + VirtualTimer mEarlyTxSetTimer; // What updateTxFloodTargets last told the overlay (nullopt = everyone, // which is also what the overlay starts with). diff --git a/src/herder/ParallelTxSetBuilder.cpp b/src/herder/ParallelTxSetBuilder.cpp index bbb9d552e..9f91b0313 100644 --- a/src/herder/ParallelTxSetBuilder.cpp +++ b/src/herder/ParallelTxSetBuilder.cpp @@ -705,7 +705,8 @@ buildSurgePricedParallelSorobanPhase( TxFrameList const& txFrames, Config const& cfg, SorobanNetworkConfig const& sorobanCfg, std::shared_ptr laneConfig, - std::vector& hadTxNotFittingLane, uint32_t ledgerVersion) + std::vector& hadTxNotFittingLane, uint32_t ledgerVersion, + std::optional seed) { ZoneScoped; // We prefer the transaction sets that are well utilized, but we also want @@ -720,9 +721,11 @@ buildSurgePricedParallelSorobanPhase( // Sort transactions in decreasing inclusion fee order. Look up each // transaction's fee bid and operation count once, not per comparison. - TxFeeComparator txComparator( - /* isGreater */ true, - stellar::rand_uniform(0, std::numeric_limits::max())); + size_t const tieBreakSeed = + seed ? *seed + : stellar::rand_uniform( + 0, std::numeric_limits::max()); + TxFeeComparator txComparator(/* isGreater */ true, tieBreakSeed); std::vector> feeBids; feeBids.reserve(txFrames.size()); for (auto const& tx : txFrames) @@ -809,4 +812,202 @@ buildSurgePricedParallelSorobanPhase( return std::move(bestResult.mStages); } +ParallelPhaseExtender::ParallelPhaseExtender( + TxStageFrameList stages, SorobanNetworkConfig const& sorobanCfg, + Resource const& laneLimit, uint32_t ledgerVersion) + : mStages(std::move(stages)) + , mLaneLimit(laneLimit) + , mTotal(Resource::makeEmptySoroban()) + , mLedgerVersion(ledgerVersion) + , mMaxClusters(sorobanCfg.ledgerMaxDependentTxClusters()) + , mMaxInstructions(sorobanCfg.ledgerMaxInstructions()) +{ + ZoneScoped; + mState.resize(mStages.size()); + for (size_t s = 0; s < mStages.size(); ++s) + { + auto& state = mState[s]; + auto const& stage = mStages[s]; + state.mClusterInstructions.resize(stage.size(), 0); + for (size_t c = 0; c < stage.size(); ++c) + { + for (auto const& tx : stage[c]) + { + state.mClusterInstructions[c] += + tx->sorobanResources().instructions; + index(state, static_cast(c), *tx); + mTotal += tx->getResources(/* useByteLimitInClassic */ false, + mLedgerVersion); + ++mTxCount; + } + state.mMaxClusterInstructions = std::max( + state.mMaxClusterInstructions, state.mClusterInstructions[c]); + } + mSequentialInstructions += state.mMaxClusterInstructions; + } +} + +void +ParallelPhaseExtender::index(StageState& stage, uint32_t cluster, + TransactionFrameBase const& tx) +{ + std::hash keyHasher; + auto const& footprint = tx.sorobanResources().footprint; + for (auto const& key : footprint.readWrite) + { + stage.mWriters[keyHasher(key)] = cluster; + } + for (auto const& key : footprint.readOnly) + { + auto& readers = stage.mReaders[keyHasher(key)]; + if (std::find(readers.begin(), readers.end(), cluster) == readers.end()) + { + readers.push_back(cluster); + } + } +} + +std::optional> +ParallelPhaseExtender::conflictingCluster(StageState const& stage, + TransactionFrameBase const& tx) const +{ + // Keys are compared by hash, as the builder of a phase does: a collision + // makes a conflict of two transactions that have none, which only costs + // the transaction its place. + std::hash keyHasher; + std::optional conflict; + // Returns false once a second cluster conflicts. + auto add = [&conflict](uint32_t cluster) { + if (conflict && *conflict != cluster) + { + return false; + } + conflict = cluster; + return true; + }; + auto const& footprint = tx.sorobanResources().footprint; + for (auto const& key : footprint.readWrite) + { + auto const h = keyHasher(key); + if (auto it = stage.mWriters.find(h); it != stage.mWriters.end()) + { + if (!add(it->second)) + { + return std::nullopt; + } + } + if (auto it = stage.mReaders.find(h); it != stage.mReaders.end()) + { + for (auto cluster : it->second) + { + if (!add(cluster)) + { + return std::nullopt; + } + } + } + } + for (auto const& key : footprint.readOnly) + { + if (auto it = stage.mWriters.find(keyHasher(key)); + it != stage.mWriters.end()) + { + if (!add(it->second)) + { + return std::nullopt; + } + } + } + return std::make_optional(conflict); +} + +bool +ParallelPhaseExtender::withinInstructionLimit( + StageState const& stage, uint64_t clusterInstructions) const +{ + // What TxSetPhaseFrame::checkValidSoroban checks: stages run one after + // another, each as long as its largest cluster. + auto const stageMax = + std::max(stage.mMaxClusterInstructions, clusterInstructions); + return mSequentialInstructions - stage.mMaxClusterInstructions + stageMax <= + mMaxInstructions; +} + +bool +ParallelPhaseExtender::tryAdd(TransactionFrameBasePtr const& tx) +{ + releaseAssert(tx->isSoroban()); + auto const resources = + tx->getResources(/* useByteLimitInClassic */ false, mLedgerVersion); + if (!mTotal.canAdd(resources) || anyGreater(mTotal + resources, mLaneLimit)) + { + return false; + } + uint64_t const instructions = tx->sorobanResources().instructions; + if (mStages.empty()) + { + // The first transaction of the phase: one stage, one cluster, if + // that is allowed. (No stage is left behind without a + // transaction.) + if (mMaxClusters == 0 || instructions > mMaxInstructions) + { + return false; + } + mStages.emplace_back(); + mState.emplace_back(); + } + for (size_t s = 0; s < mStages.size(); ++s) + { + auto& state = mState[s]; + auto& stage = mStages[s]; + auto const conflict = conflictingCluster(state, *tx); + if (!conflict) + { + // It would join two clusters of this stage. + continue; + } + std::optional cluster = *conflict; + if (!cluster) + { + // Free to go anywhere: the emptiest cluster, counting one that + // does not exist yet. + if (stage.size() < mMaxClusters) + { + cluster = static_cast(stage.size()); + } + else + { + auto const least = + std::min_element(state.mClusterInstructions.begin(), + state.mClusterInstructions.end()); + cluster = static_cast( + least - state.mClusterInstructions.begin()); + } + } + bool const isNew = *cluster == stage.size(); + uint64_t const clusterInstructions = + (isNew ? 0 : state.mClusterInstructions[*cluster]) + instructions; + if (!withinInstructionLimit(state, clusterInstructions)) + { + continue; + } + if (isNew) + { + stage.emplace_back(); + state.mClusterInstructions.push_back(0); + } + stage[*cluster].push_back(tx); + state.mClusterInstructions[*cluster] = clusterInstructions; + mSequentialInstructions -= state.mMaxClusterInstructions; + state.mMaxClusterInstructions = + std::max(state.mMaxClusterInstructions, clusterInstructions); + mSequentialInstructions += state.mMaxClusterInstructions; + index(state, *cluster, *tx); + mTotal += resources; + ++mTxCount; + return true; + } + return false; +} + } // namespace stellar diff --git a/src/herder/ParallelTxSetBuilder.h b/src/herder/ParallelTxSetBuilder.h index 9ea3feabc..bdc7f20ab 100644 --- a/src/herder/ParallelTxSetBuilder.h +++ b/src/herder/ParallelTxSetBuilder.h @@ -24,6 +24,87 @@ TxStageFrameList buildSurgePricedParallelSorobanPhase( TxFrameList const& txFrames, Config const& cfg, SorobanNetworkConfig const& sorobanCfg, std::shared_ptr laneConfig, - std::vector& hadTxNotFittingLane, uint32_t ledgerVersion); + std::vector& hadTxNotFittingLane, uint32_t ledgerVersion, + // Randomizes the order of transactions with equal fee rates. Drawn from + // the global random engine (main thread only) unless provided. + std::optional seed = std::nullopt); + +// Adds transactions to a parallel Soroban phase that has been selected +// already, without selecting the phase again. +// +// A leader selects the set for the next ledger before the current one has +// closed, and completes it at the close: what turned out invalid is removed +// (a phase stays valid when transactions are taken out of it), and what +// arrived since is added. Selecting the whole phase again +// (`buildSurgePricedParallelSorobanPhase`) costs as much as the first time; +// this adds one transaction at a time to the stages that exist. +// +// A transaction goes into the first stage where it conflicts with at most +// one cluster: into that cluster, or, without a conflict, into the cluster +// with the fewest instructions (a new one while the stage has fewer than +// `ledgerMaxDependentTxClusters`). It is only added if the result is within +// the limits a set is validated against: the lane's resources, and the +// ledger's instruction limit for the sum over the stages of each stage's +// largest cluster. Clusters are never merged and stages are never added (a +// phase without stages gets one): a transaction that would need either is not +// added. The result is checked like any selection +// (`TxSetBuilder::checkSelection`) by whoever uses it. +class ParallelPhaseExtender +{ + public: + // `stages` is the phase so far; it may be empty. `laneLimit` is the + // limit of the phase's (only) surge pricing lane. + ParallelPhaseExtender(TxStageFrameList stages, + SorobanNetworkConfig const& sorobanCfg, + Resource const& laneLimit, uint32_t ledgerVersion); + + // Adds `tx` if it fits; returns whether it did. + bool tryAdd(TransactionFrameBasePtr const& tx); + + size_t + sizeTx() const + { + return mTxCount; + } + + TxStageFrameList const& + getStages() const + { + return mStages; + } + + private: + struct StageState + { + std::vector mClusterInstructions; + uint64_t mMaxClusterInstructions{0}; + // Hash of a footprint key -> the cluster that writes it. One cluster + // at most: two clusters of a stage never share a written key. + std::unordered_map mWriters; + // Hash of a footprint key -> the clusters that only read it. + std::unordered_map> mReaders; + }; + + void index(StageState& stage, uint32_t cluster, + TransactionFrameBase const& tx); + // The clusters of `stage` that `tx` conflicts with: none, one, or (as + // nullopt) more than one. + std::optional> + conflictingCluster(StageState const& stage, + TransactionFrameBase const& tx) const; + bool withinInstructionLimit(StageState const& stage, + uint64_t clusterInstructions) const; + + TxStageFrameList mStages; + std::vector mState; + Resource mLaneLimit; + Resource mTotal; + uint32_t const mLedgerVersion; + uint32_t const mMaxClusters; + uint64_t const mMaxInstructions; + // Sum over the stages of their largest cluster's instructions. + uint64_t mSequentialInstructions{0}; + size_t mTxCount{0}; +}; } // namespace stellar diff --git a/src/herder/SurgePricingUtils.cpp b/src/herder/SurgePricingUtils.cpp index edec65d71..2b8c83920 100644 --- a/src/herder/SurgePricingUtils.cpp +++ b/src/herder/SurgePricingUtils.cpp @@ -146,13 +146,17 @@ std::vector SurgePricingPriorityQueue::getMostTopTxsWithinLimits( std::vector const& txs, std::shared_ptr laneConfig, - std::vector& hadTxNotFittingLane, uint32_t ledgerVersion) + std::vector& hadTxNotFittingLane, uint32_t ledgerVersion, + std::optional seed) { ZoneScoped; - SurgePricingPriorityQueue queue( - /* isHighestPriority */ true, laneConfig, - stellar::rand_uniform(0, std::numeric_limits::max())); + size_t const tieBreakSeed = + seed ? *seed + : stellar::rand_uniform( + 0, std::numeric_limits::max()); + SurgePricingPriorityQueue queue(/* isHighestPriority */ true, laneConfig, + tieBreakSeed); bool allFit = true; auto res = queue.countTxsResources(txs, ledgerVersion); diff --git a/src/herder/SurgePricingUtils.h b/src/herder/SurgePricingUtils.h index f6e7fb3b6..95ca192df 100644 --- a/src/herder/SurgePricingUtils.h +++ b/src/herder/SurgePricingUtils.h @@ -4,6 +4,7 @@ #pragma once +#include #include #include "transactions/TransactionFrameBase.h" @@ -153,10 +154,14 @@ class SurgePricingPriorityQueue // `hadTxNotFittingLane` is an output parameter that for every lane will // identify whether there was a transaction that didn't fit into that lane's // limit. + // `seed` randomizes the order of transactions with equal fee rates. It + // is drawn from the global random engine (main thread only) unless the + // caller provides one, which a caller on another thread has to. static std::vector getMostTopTxsWithinLimits( std::vector const& txs, std::shared_ptr laneConfig, - std::vector& hadTxNotFittingLane, uint32_t ledgerVersion); + std::vector& hadTxNotFittingLane, uint32_t ledgerVersion, + std::optional seed = std::nullopt); // Returns total amount of resources in all the transactions in this queue. Resource totalResources() const; diff --git a/src/herder/TxSetFrame.cpp b/src/herder/TxSetFrame.cpp index de090b70c..cc93a9933 100644 --- a/src/herder/TxSetFrame.cpp +++ b/src/herder/TxSetFrame.cpp @@ -31,6 +31,7 @@ #include #include +#include #include #include #include @@ -496,31 +497,27 @@ computeLaneBaseFee(TxSetPhase phase, LedgerHeader const& ledgerHeader, } std::shared_ptr -createSurgePricingLangeConfig(TxSetPhase phase, Application& app) +createSurgePricingLaneConfig(TxSetPhase phase, Config const& cfg, + TxSetBuildLimits const& limits) { ZoneScoped; - releaseAssert(threadIsMain()); - releaseAssert(!app.getLedgerManager().isApplying()); - auto const& lclHeader = - app.getLedgerManager().getLastClosedLedgerHeader().header; + auto const& lclHeader = limits.mLclHeader; std::vector hadTxNotFittingLane; std::shared_ptr surgePricingLaneConfig; if (phase == TxSetPhase::CLASSIC) { - auto maxOps = Resource( - {static_cast( - app.getLedgerManager().getLastMaxTxSetSizeOps()), - static_cast(app.getConfig().getClassicByteAllowance())}); + auto maxOps = + Resource({limits.maxTxSetSizeOps(), + static_cast(cfg.getClassicByteAllowance())}); std::optional dexOpsLimit; - if (app.getConfig().MAX_DEX_TX_OPERATIONS_IN_TX_SET) + if (cfg.MAX_DEX_TX_OPERATIONS_IN_TX_SET) { // DEX operations limit implies that DEX transactions should // compete with each other in in a separate fee lane, which // is only possible with generalized tx set. - dexOpsLimit = - Resource({*app.getConfig().MAX_DEX_TX_OPERATIONS_IN_TX_SET, - MAX_CLASSIC_BYTE_ALLOWANCE}); + dexOpsLimit = Resource({*cfg.MAX_DEX_TX_OPERATIONS_IN_TX_SET, + MAX_CLASSIC_BYTE_ALLOWANCE}); } surgePricingLaneConfig = @@ -529,9 +526,9 @@ createSurgePricingLangeConfig(TxSetPhase phase, Application& app) else { releaseAssert(phase == TxSetPhase::SOROBAN); + releaseAssert(limits.mSorobanConfig); - auto limits = app.getLedgerManager().maxLedgerResources( - /* isSoroban */ true); + auto limitsRes = limits.mSorobanConfig->maxLedgerResources(); // When building Soroban tx sets with parallel execution support, // instructions are accounted for by the build logic, not by the surge // pricing config, so we need to relax the instruction limit in surge @@ -539,17 +536,17 @@ createSurgePricingLangeConfig(TxSetPhase phase, Application& app) if (protocolVersionStartsFrom(lclHeader.ledgerVersion, PARALLEL_SOROBAN_PHASE_PROTOCOL_VERSION)) { - limits.setVal(Resource::Type::INSTRUCTIONS, - std::numeric_limits::max()); + limitsRes.setVal(Resource::Type::INSTRUCTIONS, + std::numeric_limits::max()); } - auto byteLimit = std::min( - static_cast(app.getConfig().getSorobanByteAllowance()), - limits.getVal(Resource::Type::TX_BYTE_SIZE)); - limits.setVal(Resource::Type::TX_BYTE_SIZE, byteLimit); + auto byteLimit = + std::min(static_cast(cfg.getSorobanByteAllowance()), + limitsRes.getVal(Resource::Type::TX_BYTE_SIZE)); + limitsRes.setVal(Resource::Type::TX_BYTE_SIZE, byteLimit); surgePricingLaneConfig = - std::make_shared(limits); + std::make_shared(limitsRes); } return surgePricingLaneConfig; } @@ -558,16 +555,91 @@ TxFrameList buildSurgePricedSequentialPhase( TxFrameList const& txs, std::shared_ptr surgePricingLaneConfig, - std::vector& hadTxNotFittingLane, uint32_t ledgerVersion) + std::vector& hadTxNotFittingLane, uint32_t ledgerVersion, + std::optional seed) { ZoneScoped; return SurgePricingPriorityQueue::getMostTopTxsWithinLimits( - txs, surgePricingLaneConfig, hadTxNotFittingLane, ledgerVersion); + txs, surgePricingLaneConfig, hadTxNotFittingLane, ledgerVersion, seed); } +// The base fee every transaction of a selected phase is included at: per +// surge pricing lane, the ledger's base fee, or the lowest fee rate included +// if transactions of the lane (or of the generic lane) were left out. +std::shared_ptr +computeInclusionFeeMap( + TxSetPhase phase, + std::variant const& includedTxs, + SurgePricingLaneConfig const& surgePricingLaneConfig, + LedgerHeader const& lclHeader, std::vector const& hadTxNotFittingLane) +{ + auto visitIncludedTxs = + [&includedTxs]( + std::function visitor) { + std::visit( + [&visitor](auto const& txs) { + using T = std::decay_t; + if constexpr (std::is_same_v) + { + for (auto const& tx : txs) + { + visitor(tx); + } + } + else if constexpr (std::is_same_v) + { + for (auto const& stage : txs) + { + for (auto const& thread : stage) + { + for (auto const& tx : thread) + { + visitor(tx); + } + } + } + } + else + { + // This can't be just `false` as if an assertion is not + // dependent on template argument, it will be + // unconditionally triggered. + static_assert(!std::is_same_v, + "Non-exhaustive visitor"); + } + }, + includedTxs); + }; + + std::vector lowestLaneFee; + size_t laneCount = surgePricingLaneConfig.getLaneLimits().size(); + releaseAssert(hadTxNotFittingLane.size() == laneCount); + lowestLaneFee.resize(laneCount, std::numeric_limits::max()); + visitIncludedTxs( + [&lowestLaneFee, &surgePricingLaneConfig, &lclHeader](auto const& tx) { + size_t lane = surgePricingLaneConfig.getLane(*tx); + auto perOpFee = computePerOpFee(*tx, lclHeader.ledgerVersion); + lowestLaneFee[lane] = std::min(lowestLaneFee[lane], perOpFee); + }); + auto laneBaseFee = + computeLaneBaseFee(phase, lclHeader, surgePricingLaneConfig, + lowestLaneFee, hadTxNotFittingLane); + auto inclusionFeeMapPtr = std::make_shared(); + auto& inclusionFeeMap = *inclusionFeeMapPtr; + visitIncludedTxs([&inclusionFeeMap, &laneBaseFee, + &surgePricingLaneConfig](auto const& tx) { + inclusionFeeMap[tx] = laneBaseFee[surgePricingLaneConfig.getLane(*tx)]; + }); + return inclusionFeeMapPtr; +} + +// `hadTxNotFittingLaneOut`, if given, receives per surge pricing lane of the +// phase whether a transaction was left out for the lane's limits. std::pair, std::shared_ptr> -applySurgePricing(TxSetPhase phase, TxFrameList const& txs, Application& app +applySurgePricing(TxSetPhase phase, TxFrameList const& txs, Config const& cfg, + TxSetBuildLimits const& limits, std::optional seed, + std::vector* hadTxNotFittingLaneOut #ifdef BUILD_TESTS , bool enforceTxsApplyOrder, @@ -576,10 +648,10 @@ applySurgePricing(TxSetPhase phase, TxFrameList const& txs, Application& app ) { ZoneScoped; - auto surgePricingLaneConfig = createSurgePricingLangeConfig(phase, app); + auto surgePricingLaneConfig = + createSurgePricingLaneConfig(phase, cfg, limits); std::vector hadTxNotFittingLane; - uint32_t ledgerVersion = - app.getLedgerManager().getLastClosedLedgerHeader().header.ledgerVersion; + uint32_t ledgerVersion = limits.mLclHeader.ledgerVersion; bool isParallelSoroban = phase == TxSetPhase::SOROBAN && protocolVersionStartsFrom(ledgerVersion, @@ -623,10 +695,10 @@ applySurgePricing(TxSetPhase phase, TxFrameList const& txs, Application& app else { #endif + releaseAssert(limits.mSorobanConfig); includedTxs = buildSurgePricedParallelSorobanPhase( - txs, app.getConfig(), - app.getLedgerManager().getLastClosedSorobanNetworkConfig(), - surgePricingLaneConfig, hadTxNotFittingLane, ledgerVersion); + txs, cfg, *limits.mSorobanConfig, surgePricingLaneConfig, + hadTxNotFittingLane, ledgerVersion, seed); #ifdef BUILD_TESTS } #endif @@ -634,69 +706,17 @@ applySurgePricing(TxSetPhase phase, TxFrameList const& txs, Application& app else { includedTxs = buildSurgePricedSequentialPhase( - txs, surgePricingLaneConfig, hadTxNotFittingLane, ledgerVersion); + txs, surgePricingLaneConfig, hadTxNotFittingLane, ledgerVersion, + seed); } - auto visitIncludedTxs = - [&includedTxs]( - std::function visitor) { - std::visit( - [&visitor](auto const& txs) { - using T = std::decay_t; - if constexpr (std::is_same_v) - { - for (auto const& tx : txs) - { - visitor(tx); - } - } - else if constexpr (std::is_same_v) - { - for (auto const& stage : txs) - { - for (auto const& thread : stage) - { - for (auto const& tx : thread) - { - visitor(tx); - } - } - } - } - else - { - // This can't be just `false` as if an assertion is not - // dependent on template argument, it will be - // unconditionally triggered. - static_assert(!std::is_same_v, - "Non-exhaustive visitor"); - } - }, - includedTxs); - }; - - std::vector lowestLaneFee; - auto const& lclHeader = - app.getLedgerManager().getLastClosedLedgerHeader().header; - - size_t laneCount = surgePricingLaneConfig->getLaneLimits().size(); - lowestLaneFee.resize(laneCount, std::numeric_limits::max()); - visitIncludedTxs( - [&lowestLaneFee, &surgePricingLaneConfig, &lclHeader](auto const& tx) { - size_t lane = surgePricingLaneConfig->getLane(*tx); - auto perOpFee = computePerOpFee(*tx, lclHeader.ledgerVersion); - lowestLaneFee[lane] = std::min(lowestLaneFee[lane], perOpFee); - }); - auto laneBaseFee = - computeLaneBaseFee(phase, lclHeader, *surgePricingLaneConfig, - lowestLaneFee, hadTxNotFittingLane); - auto inclusionFeeMapPtr = std::make_shared(); - auto& inclusionFeeMap = *inclusionFeeMapPtr; - visitIncludedTxs([&inclusionFeeMap, &laneBaseFee, - &surgePricingLaneConfig](auto const& tx) { - inclusionFeeMap[tx] = laneBaseFee[surgePricingLaneConfig->getLane(*tx)]; - }); - + auto inclusionFeeMapPtr = + computeInclusionFeeMap(phase, includedTxs, *surgePricingLaneConfig, + limits.mLclHeader, hadTxNotFittingLane); + if (hadTxNotFittingLaneOut) + { + *hadTxNotFittingLaneOut = hadTxNotFittingLane; + } return std::make_pair(includedTxs, inclusionFeeMapPtr); } @@ -882,16 +902,264 @@ makeTxSetFromTransactions( , TxSetReadyCallback const& onTxSetReady, std::vector const& validTxTargets, - TxValidationCache* validationCache) + TxValidationCache* validationCache, + std::shared_ptr* deferredSelfCheck) +{ + releaseAssert(threadIsMain()); + releaseAssert(!app.getLedgerManager().isApplying()); + releaseAssert(txPhases.size() == invalidTxs.size()); + + // Everything is read from the LCL: validity through the context, the + // limits of the set from its header and Soroban configuration. + auto ctx = TxSetValidationContext::forLCL(app); + ctx.mValidationCache = validationCache; + auto const validatedPhases = TxSetBuilder::selectPhases( + txPhases, app, TxSetBuildLimits::forLCL(app), ctx, closeTimeOffset, + invalidTxs, validTxTargets, "build", std::nullopt, nullptr +#ifdef BUILD_TESTS + , + enforceTxsApplyOrder, parallelSorobanOrder, + disableTxValidationForLegacyScenario +#endif + ); + + return TxSetBuilder::writeSelection(validatedPhases, app, ctx, + closeTimeOffset, onTxSetReady, + deferredSelfCheck +#ifdef BUILD_TESTS + , + enforceTxsApplyOrder +#endif + ); +} + +std::pair +TxSetBuilder::writeSelection( + std::vector const& validatedPhases, Application& app, + TxSetValidationContext const& ctx, ApplyTimeOffset closeTimeOffset, + TxSetReadyCallback const& onTxSetReady, + std::shared_ptr* deferredSelfCheck +#ifdef BUILD_TESTS + , + bool enforceTxsApplyOrder +#endif +) { releaseAssert(threadIsMain()); releaseAssert(!app.getLedgerManager().isApplying()); + auto const& lclHeader = app.getLedgerManager().getLastClosedLedgerHeader(); + bool const selfCheckIsDeferred = deferredSelfCheck +#ifdef BUILD_TESTS + && !enforceTxsApplyOrder +#endif + ; + // The check of the written set comes after the set is in use. What the + // selection itself can get wrong (two transactions of one account, fees, + // limits, conflicting clusters) is checked before anything leaves this + // function, as it always was: on another thread while this one writes + // the set (both only read the selection), and waited for before the set + // is handed to anyone. + std::future selectionCheck; + if (selfCheckIsDeferred) + { + bool const isGeneralized = protocolVersionStartsFrom( + lclHeader.header.ledgerVersion, SOROBAN_PROTOCOL_VERSION); + // The two threads read the same frames. That is safe because the + // check reads no lazily computed member of a frame, and the write + // only reads full hashes, which every selected frame computed when + // it was validated. + auto check = [&validatedPhases, &app, &ctx, isGeneralized, + closeTimeOffset]() { + return TxSetBuilder::checkSelection(validatedPhases, app, ctx, + isGeneralized, closeTimeOffset); + }; + try + { + selectionCheck = std::async(std::launch::async, check); + } + catch (std::system_error const&) + { + // No thread to be had: check here. + std::promise done; + done.set_value(check()); + selectionCheck = done.get_future(); + } + } + auto xdrTimer = + app.getMetrics().NewTimer({"herder", "txset", "build-xdr"}).TimeScope(); + // Preliminary applicable frame - we don't know the contents hash yet, but + // we also don't return this. + std::unique_ptr preliminaryApplicableTxSet( + new ApplicableTxSetFrame(app, lclHeader, validatedPhases, + std::nullopt)); + + // Do the roundtrip through XDR to ensure we never build an incorrect tx set + // for nomination. + auto outputTxSet = preliminaryApplicableTxSet->toWireTxSetFrame(); +#ifdef BUILD_TESTS + if (enforceTxsApplyOrder) + { + // Fill in the contents hash if we're skipping the normal roundtrip + // and validation flow. + preliminaryApplicableTxSet->mContentsHash = + outputTxSet->getContentsHash(); + return std::make_pair(outputTxSet, + std::move(preliminaryApplicableTxSet)); + } +#endif + + xdrTimer.Stop(); + if (selectionCheck.valid()) + { + auto const result = selectionCheck.get(); + if (result != TxSetValidationResult::VALID) + { + throw std::runtime_error(fmt::format( + FMT_STRING( + "Created invalid tx set frame, validation result: {:s}"), + toString(result))); + } + } + // The final XDR is available now. Start proposal encoding while the + // roundtrip and final validity checks below run; only the local builder + // supplies this callback. The callback must not nominate the set. + if (onTxSetReady) + { + onTxSetReady(outputTxSet); + } + + if (deferredSelfCheck) + { + // What is cheap to check of the written set is checked here: the + // shape of its XDR (no empty component, stage or cluster, ...), and + // that it holds as many transactions as the selection. + if (outputTxSet->isGeneralizedTxSet() && + !validateTxSetXDRStructure(outputTxSet->getGeneralizedXDR(), + lclHeader.header.ledgerVersion)) + { + throw std::runtime_error( + "Couldn't prepare created tx set frame for apply"); + } + size_t selected = 0; + for (auto const& phase : validatedPhases) + { + selected += phase.sizeTx(); + } + if (outputTxSet->sizeTxTotal() != selected) + { + throw std::runtime_error("Created invalid tx set frame - shape is " + "mismatched after writing."); + } + // The caller runs the check (see TxSetSelfCheck) and goes on with + // the set as selected, which is what the check compares the XDR + // with. + *deferredSelfCheck = TxSetBuilder::makeSelfCheck( + app, outputTxSet, lclHeader, validatedPhases, closeTimeOffset); + preliminaryApplicableTxSet->mContentsHash = + outputTxSet->getContentsHash(); + return std::make_pair(outputTxSet, + std::move(preliminaryApplicableTxSet)); + } + + std::vector phaseSizes; + for (auto const& phase : validatedPhases) + { + phaseSizes.push_back(phase.sizeTx()); + } + ApplicableTxSetFrameConstPtr outputApplicableTxSet; + // We already trimmed invalid transactions in an earlier call to + // `trimInvalid`, so the check skips transaction validation. + auto const failure = TxSetBuilder::checkBuiltTxSet( + app, *outputTxSet, TxSetValidationContext::forLCL(app), phaseSizes, + closeTimeOffset.seconds(), &outputApplicableTxSet); + if (failure) + { + throw std::runtime_error(*failure); + } + + return std::make_pair(outputTxSet, std::move(outputApplicableTxSet)); +} + +uint32_t +TxSetBuildLimits::maxTxSetSizeOps() const +{ + // As LedgerManager::getLastMaxTxSetSizeOps. + auto const n = mLclHeader.maxTxSetSize; + return protocolVersionStartsFrom(mLclHeader.ledgerVersion, + ProtocolVersion::V_11) + ? n + : (n * MAX_OPS_PER_TX); +} + +bool +TxSetBuildLimits::sameAs(TxSetBuildLimits const& other) const +{ + auto const& a = mLclHeader; + auto const& b = other.mLclHeader; + if (a.ledgerVersion != b.ledgerVersion || a.baseFee != b.baseFee || + a.baseReserve != b.baseReserve || a.maxTxSetSize != b.maxTxSetSize) + { + return false; + } + if ((mSorobanConfig == nullptr) != (other.mSorobanConfig == nullptr)) + { + return false; + } + if (!mSorobanConfig) + { + return true; + } + auto const& s = *mSorobanConfig; + auto const& t = *other.mSorobanConfig; + return s.maxLedgerResources() == t.maxLedgerResources() && + s.ledgerMaxInstructions() == t.ledgerMaxInstructions() && + s.ledgerMaxDependentTxClusters() == + t.ledgerMaxDependentTxClusters() && + s.ledgerMaxTxCount() == t.ledgerMaxTxCount(); +} + +TxSetBuildLimits +TxSetBuildLimits::forLCL(Application& app) +{ + releaseAssert(threadIsMain()); + auto& lm = app.getLedgerManager(); + TxSetBuildLimits limits; + limits.mLclHeader = lm.getLastClosedLedgerHeader().header; + if (protocolVersionStartsFrom(limits.mLclHeader.ledgerVersion, + SOROBAN_PROTOCOL_VERSION)) + { + limits.mSorobanConfig = &lm.getLastClosedSorobanNetworkConfig(); + } + return limits; +} + +std::vector +TxSetBuilder::selectPhases( + PerPhaseTransactionList const& txPhases, Application& app, + TxSetBuildLimits const& limits, TxSetValidationContext const& ctx, + ApplyTimeOffset closeTimeOffset, PerPhaseTransactionList& invalidTxs, + std::vector const& validTxTargets, std::string const& timerPrefix, + std::optional tieBreakSeed, + std::vector>* hadTxNotFittingLane +#ifdef BUILD_TESTS + , + bool enforceTxsApplyOrder, + txtest::ParallelSorobanOrder const& parallelSorobanOrder, + bool disableTxValidationForLegacyScenario +#endif +) +{ releaseAssert(txPhases.size() == invalidTxs.size()); + if (hadTxNotFittingLane) + { + hadTxNotFittingLane->assign(txPhases.size(), {}); + } releaseAssert(validTxTargets.empty() || validTxTargets.size() == txPhases.size()); releaseAssert(txPhases.size() <= static_cast(TxSetPhase::PHASE_COUNT)); + auto const& cfg = app.getConfig(); std::vector validatedPhases; UnorderedMap accountFeeMap; for (size_t i = 0; i < txPhases.size(); ++i) @@ -911,38 +1179,38 @@ makeTxSetFromTransactions( #ifdef BUILD_TESTS if (disableTxValidationForLegacyScenario) { - REQUIRE(protocolVersionIsBefore(app.getLedgerManager() - .getLastClosedLedgerHeader() - .header.ledgerVersion, + REQUIRE(protocolVersionIsBefore(limits.mLclHeader.ledgerVersion, ProtocolVersion::V_20)); validatedTxs = phaseTxs; } else { #endif - auto trimTimer = app.getMetrics() - .NewTimer({"herder", "txset", "build-trim"}) - .TimeScope(); + auto trimTimer = + app.getMetrics() + .NewTimer({"herder", "txset", timerPrefix + "-trim"}) + .TimeScope(); if (validTxTargets.empty()) { - validatedTxs = TxSetUtils::trimInvalid( - phaseTxs, app, accountFeeMap, closeTimeOffset.seconds(), - closeTimeOffset.seconds(), invalid, validationCache); + validatedTxs = + TxSetUtils::trimInvalid(phaseTxs, app, ctx, accountFeeMap, + closeTimeOffset.seconds(), + closeTimeOffset.seconds(), invalid); } else { // Transactions of limited lanes (DEX) can't fill the phase, // so only generic lane ones count toward the target. - auto laneConfig = createSurgePricingLangeConfig( - static_cast(i), app); + auto laneConfig = createSurgePricingLaneConfig( + static_cast(i), cfg, limits); validatedTxs = TxSetUtils::trimInvalidPrefix( phaseTxs, validTxTargets[i], [&laneConfig](TransactionFrameBase const& tx) { return laneConfig->getLane(tx) == SurgePricingPriorityQueue::GENERIC_LANE; }, - app, accountFeeMap, closeTimeOffset.seconds(), - closeTimeOffset.seconds(), invalid, validationCache); + app, ctx, accountFeeMap, closeTimeOffset.seconds(), + closeTimeOffset.seconds(), invalid); } #ifdef BUILD_TESTS // In tests we shouldn't be really trying to close ledgers with @@ -963,16 +1231,19 @@ makeTxSetFromTransactions( } #endif auto phaseType = static_cast(i); - auto selectTimer = app.getMetrics() - .NewTimer({"herder", "txset", "build-select"}) - .TimeScope(); - auto [includedTxs, inclusionFeeMapBinding] = - applySurgePricing(phaseType, validatedTxs, app + auto selectTimer = + app.getMetrics() + .NewTimer({"herder", "txset", timerPrefix + "-select"}) + .TimeScope(); + auto [includedTxs, inclusionFeeMapBinding] = applySurgePricing( + phaseType, validatedTxs, cfg, limits, + tieBreakSeed ? std::make_optional(*tieBreakSeed + i) : std::nullopt, + hadTxNotFittingLane ? &(*hadTxNotFittingLane)[i] : nullptr #ifdef BUILD_TESTS - , - enforceTxsApplyOrder, parallelSorobanOrder + , + enforceTxsApplyOrder, parallelSorobanOrder #endif - ); + ); auto inclusionFeeMap = inclusionFeeMapBinding; std::visit( [&validatedPhases, phaseType, inclusionFeeMap](auto&& txs) { @@ -998,88 +1269,526 @@ makeTxSetFromTransactions( }, includedTxs); } + return validatedPhases; +} - auto const& lclHeader = app.getLedgerManager().getLastClosedLedgerHeader(); - auto xdrTimer = - app.getMetrics().NewTimer({"herder", "txset", "build-xdr"}).TimeScope(); - // Preliminary applicable frame - we don't know the contents hash yet, but - // we also don't return this. - std::unique_ptr preliminaryApplicableTxSet( - new ApplicableTxSetFrame(app, lclHeader, validatedPhases, - std::nullopt)); +TxSetValidationResult +TxSetBuilder::checkSelection(std::vector const& phases, + Application& app, + TxSetValidationContext const& ctx, + bool isGeneralized, + ApplyTimeOffset closeTimeOffset) +{ + ZoneScoped; + auto timer = app.getMetrics() + .NewTimer({"herder", "txset", "build-selection-check"}) + .TimeScope(); + // The transactions were validated by the selection. + return ApplicableTxSetFrame::checkPhasesValid( + app, ctx, phases, isGeneralized, closeTimeOffset.seconds(), + closeTimeOffset.seconds(), /* txsAreValidated */ true); +} - // Do the roundtrip through XDR to ensure we never build an incorrect tx set - // for nomination. - auto outputTxSet = preliminaryApplicableTxSet->toWireTxSetFrame(); -#ifdef BUILD_TESTS - if (enforceTxsApplyOrder) +namespace +{ +// Removes `gone` from a phase's transactions, and the clusters and stages +// that are left empty with them. +template +TxStageFrameList +withoutTxs(TxStageFrameList const& stages, Gone const& gone) +{ + TxStageFrameList res; + for (auto const& stage : stages) { - // Fill in the contents hash if we're skipping the normal roundtrip - // and validation flow. - preliminaryApplicableTxSet->mContentsHash = - outputTxSet->getContentsHash(); - return std::make_pair(outputTxSet, - std::move(preliminaryApplicableTxSet)); + TxStageFrame keptStage; + for (auto const& cluster : stage) + { + TxClusterFrame keptCluster; + for (auto const& tx : cluster) + { + if (gone.count(tx.get()) == 0) + { + keptCluster.push_back(tx); + } + } + if (!keptCluster.empty()) + { + keptStage.emplace_back(std::move(keptCluster)); + } + } + if (!keptStage.empty()) + { + res.emplace_back(std::move(keptStage)); + } } + return res; +} +} + +std::vector +TxSetBuilder::completeSelection( + std::vector const& selected, + std::vector> const& hadTxNotFittingLane, + TxSetCandidateSource const& moreCandidates, Application& app, + TxSetBuildLimits const& limits, TxSetValidationContext const& ctx, + ApplyTimeOffset closeTimeOffset, PerPhaseTransactionList& invalidTxs, + TxSetCompletionStats& stats, std::optional tieBreakSeed) +{ + ZoneScoped; + size_t const phaseCount = selected.size(); + releaseAssert(phaseCount == static_cast(TxSetPhase::PHASE_COUNT)); + releaseAssert(hadTxNotFittingLane.size() == phaseCount); + releaseAssert(invalidTxs.size() == phaseCount); + releaseAssert(limits.mSorobanConfig); + auto& metrics = app.getMetrics(); + auto const& cfg = app.getConfig(); + auto const offset = closeTimeOffset.seconds(); + auto const ledgerVersion = limits.mLclHeader.ledgerVersion; + + struct PhaseState + { + std::shared_ptr mLaneConfig; + bool mParallel{false}; + // What is selected so far, in one of the two shapes. + TxFrameList mSequential; + TxStageFrameList mStages; + std::optional mExtender; + // What the selection reported, per surge pricing lane. + std::vector mHadTxNotFittingLane; + // The same for the completion: a valid candidate was left out. + std::vector mLeftOutLanes; + bool mLeftOut{false}; + // A sequential phase was selected again. + bool mReselected{false}; + size_t mRemoved{0}; + size_t mAdded{0}; + // Candidates were looked at for this phase during the completion. + bool mExamined{false}; + // The phase lost transactions or candidates: to the validity pass, + // or because a candidate was invalid or of an account that is in + // the set already. (Not: a valid candidate that did not fit.) + bool mLost{false}; + // Batches from the mempool the phase has looked at. + size_t mMempoolBatches{0}; + // Nothing more is looked for (see the top-up below). + bool mFull{false}; + bool mStarved{false}; + }; + std::vector states(phaseCount); + // One fee-source total for the whole set, as in a build. + UnorderedMap accountFeeMap; + // One transaction per source account, across the phases. + UnorderedSet sources; + + auto atCapacity = [&](size_t p) { + auto const& state = states[p]; + auto const limit = state.mLaneConfig->getLaneLimits().at( + SurgePricingPriorityQueue::GENERIC_LANE); + int64_t ops = 0; + if (state.mExtender) + { + ops = static_cast(state.mExtender->sizeTx()); + } + else if (state.mParallel) + { + for (auto const& stage : state.mStages) + { + for (auto const& cluster : stage) + { + ops += static_cast(cluster.size()); + } + } + } + else + { + ops = static_cast(countOps(state.mSequential)); + } + return ops >= limit.getVal(Resource::Type::OPERATIONS); + }; + + // 1. The validity pass. Whatever the selection was made on, every one of + // its transactions is validated here against the ledger the set goes on + // top of, by the function that validates a set a node receives; what + // fails leaves the selection. + { + auto timer = metrics.NewTimer({"herder", "txset", "early-final-pass"}) + .TimeScope(); + for (size_t p = 0; p < phaseCount; ++p) + { + auto const& phase = selected[p]; + auto& state = states[p]; + state.mLaneConfig = createSurgePricingLaneConfig( + static_cast(p), cfg, limits); + state.mParallel = phase.isParallel(); + state.mHadTxNotFittingLane = hadTxNotFittingLane[p]; + releaseAssert(state.mHadTxNotFittingLane.size() == + state.mLaneConfig->getLaneLimits().size()); + state.mLeftOutLanes.assign(state.mHadTxNotFittingLane.size(), + false); + stats.mSelected += phase.sizeTx(); + auto invalid = TxSetUtils::getInvalidTxListWithErrors( + phase, app, ctx, accountFeeMap, offset, offset) + .first; + UnorderedSet gone; + for (auto const& tx : invalid) + { + gone.insert(tx.get()); + } + state.mRemoved = gone.size(); + stats.mRemoved += gone.size(); + if (state.mParallel) + { + state.mStages = withoutTxs(phase.getParallelStages(), gone); + } + else + { + for (auto const& tx : phase.getSequentialTxs()) + { + if (gone.count(tx.get()) == 0) + { + state.mSequential.push_back(tx); + } + } + } + for (auto const& tx : phase) + { + if (gone.count(tx.get()) == 0) + { + sources.insert(tx->getSourceID()); + } + } + invalidTxs[p] = std::move(invalid); + // Only a phase at its capacity is full from the start. That the + // selection left a transaction out says little: one that did + // not fit where it had to go (a cluster, a lane of its own) + // leaves room for others. + state.mFull = atCapacity(p); + } + } + + // 2. The top-up: while a phase has room, take further candidates, each + // validated like the selection's own. + // + // The rule is a build's: a build looks at the mempool's top (the + // capacity plus the margin) and looks deeper only for a phase that is + // short and lost candidates. So a phase stops asking when + // - it is at its capacity; or + // - it has seen the mempool's top once and lost nothing (no + // transaction removed by the validity pass, no candidate that was + // invalid or of an account already in the set): it got all that the + // top holds for it; or + // - it lost something, went on to a deeper pull, and that pull held + // nothing new of its kind. + // That valid candidates did not fit closes nothing: a transaction that + // could not go where it had to (a full cluster, a lane of its own) + // leaves room for others, whatever their number. + for (size_t p = 0; p < phaseCount; ++p) + { + states[p].mLost = states[p].mRemoved != 0; + } + // Selects a sequential phase again, from what it holds and `extra`; + // returns how many of `extra` are in. + auto reselect = [&](size_t p, TxFrameList const& extra) { + auto& state = states[p]; + TxFrameList all = state.mSequential; + all.insert(all.end(), extra.begin(), extra.end()); + std::vector flags; + auto [included, feeMap] = applySurgePricing( + static_cast(p), all, cfg, limits, + tieBreakSeed ? std::make_optional(*tieBreakSeed + p) : std::nullopt, + &flags +#ifdef BUILD_TESTS + , + false, {} #endif + ); + state.mSequential = std::get(included); + state.mReselected = true; + releaseAssert(flags.size() == state.mLeftOutLanes.size()); + for (size_t lane = 0; lane < flags.size(); ++lane) + { + state.mLeftOutLanes[lane] = + state.mLeftOutLanes[lane] || flags[lane]; + } + UnorderedSet in; + for (auto const& tx : state.mSequential) + { + in.insert(tx.get()); + } + size_t added = 0; + for (auto const& tx : extra) + { + if (in.count(tx.get()) != 0) + { + sources.insert(tx->getSourceID()); + ++added; + } + } + return added; + }; + auto wantsMore = [&]() { + return std::any_of( + states.begin(), states.end(), + [](PhaseState const& s) { return !s.mFull && !s.mStarved; }); + }; + { + auto timer = + metrics.NewTimer({"herder", "txset", "early-top-up"}).TimeScope(); + while (wantsMore()) + { + auto batch = moreCandidates(); + if (!batch) + { + break; + } + stats.mCandidates += batch->mTxs.size(); + TxFrameList classicTxs; + TxFrameList sorobanTxs; + std::vector offered(phaseCount, 0); + for (auto const& tx : batch->mTxs) + { + size_t const p = tx->isSoroban() ? 1 : 0; + if (states[p].mFull) + { + continue; + } + ++offered[p]; + if (sources.count(tx->getSourceID()) != 0) + { + continue; + } + (p == 1 ? sorobanTxs : classicTxs).push_back(tx); + } + TxSetUtils::keepOnePerSourceAccount(classicTxs, sorobanTxs); + for (size_t p = 0; p < phaseCount; ++p) + { + auto& state = states[p]; + if (state.mFull) + { + continue; + } + // The selection's own verdict on what it left out is void + // from here on: the candidates are judged on this ledger. + state.mExamined = true; + auto const& candidates = p == 1 ? sorobanTxs : classicTxs; + size_t valid = 0; + size_t added = 0; + if (!candidates.empty()) + { + TxFrameList invalid; + auto const validTxs = TxSetUtils::trimInvalid( + candidates, app, ctx, accountFeeMap, offset, offset, + invalid); + invalidTxs[p].insert(invalidTxs[p].end(), invalid.begin(), + invalid.end()); + valid = validTxs.size(); + if (valid != 0 && state.mParallel) + { + if (!state.mExtender) + { + state.mExtender.emplace( + std::move(state.mStages), + *limits.mSorobanConfig, + state.mLaneConfig->getLaneLimits().at( + SurgePricingPriorityQueue::GENERIC_LANE), + ledgerVersion); + } + for (auto const& tx : validTxs) + { + if (state.mExtender->tryAdd(tx)) + { + sources.insert(tx->getSourceID()); + ++added; + } + } + } + else if (valid != 0) + { + // A sequential phase is cheap to select: again, by + // the standard surge pricing. + added = reselect(p, validTxs); + } + } + state.mAdded += added; + // A valid candidate that is not in: the phase's + // transactions competed for its room. + state.mLeftOut = state.mLeftOut || added < valid; + state.mLost = state.mLost || valid < offered[p]; + if (atCapacity(p)) + { + state.mFull = true; + } + else if (batch->mFromMempool) + { + ++state.mMempoolBatches; + state.mStarved = + !state.mLost || + (state.mMempoolBatches > 1 && offered[p] == 0); + } + } + } + } - xdrTimer.Stop(); - // The final XDR is available now. Start proposal encoding while the - // roundtrip and final validity checks below run; only the local builder - // supplies this callback. The callback must not nominate the set. - if (onTxSetReady) + // 3. The phases as they are now. Base fees by the rule of a build: + // transactions of a lane compete on fees only if a valid one was left + // out. For a phase that looked at candidates here, that is what + // happened here; otherwise it is what the selection reported. + std::vector res; + for (size_t p = 0; p < phaseCount; ++p) { - onTxSetReady(outputTxSet); + auto& state = states[p]; + auto const phaseType = static_cast(p); + stats.mAdded += state.mAdded; + std::vector flags = state.mHadTxNotFittingLane; + if (state.mExamined || state.mRemoved != 0) + { + flags = state.mLeftOutLanes; + if (state.mParallel) + { + flags.at(SurgePricingPriorityQueue::GENERIC_LANE) = + state.mLeftOut; + } + } + if (state.mRemoved == 0 && state.mAdded == 0 && !state.mReselected && + flags == state.mHadTxNotFittingLane) + { + // Untouched. + res.emplace_back(selected[p]); + continue; + } + if (state.mParallel) + { + auto stages = state.mExtender ? state.mExtender->getStages() + : std::move(state.mStages); + std::variant included = stages; + auto feeMap = + computeInclusionFeeMap(phaseType, included, *state.mLaneConfig, + limits.mLclHeader, flags); + res.emplace_back( + TxSetPhaseFrame(phaseType, std::move(stages), feeMap)); + } + else + { + if (!state.mReselected) + { + // Transactions were removed and none added: select what is + // left. + reselect(p, {}); + flags = state.mLeftOutLanes; + } + std::variant included = + state.mSequential; + auto feeMap = + computeInclusionFeeMap(phaseType, included, *state.mLaneConfig, + limits.mLclHeader, flags); + res.emplace_back( + TxSetPhaseFrame(phaseType, state.mSequential, feeMap)); + } } + return res; +} +std::shared_ptr +TxSetBuilder::makeSelfCheck(Application& app, + TxSetXDRFrameConstPtr const& txSet, + LedgerHeaderHistoryEntry const& lcl, + std::vector const& phases, + ApplyTimeOffset closeTimeOffset) +{ + releaseAssert(threadIsMain()); + std::shared_ptr check(new TxSetSelfCheck()); + check->mTxSet = txSet; + check->mLcl = lcl; + if (protocolVersionStartsFrom(lcl.header.ledgerVersion, + SOROBAN_PROTOCOL_VERSION)) + { + check->mSorobanConfig = std::make_shared( + app.getLedgerManager().getLastClosedSorobanNetworkConfig()); + } + for (auto const& phase : phases) + { + check->mPhaseSizes.push_back(phase.sizeTx()); + } + check->mCloseTimeOffset = closeTimeOffset.seconds(); + return check; +} + +std::optional +TxSetBuilder::checkBuiltTxSet(Application& app, TxSetXDRFrame const& txSet, + TxSetValidationContext const& ctx, + std::vector const& phaseSizes, + uint64_t closeTimeOffset, + ApplicableTxSetFrameConstPtr* roundTripped) +{ + ZoneScoped; auto roundtripTimer = app.getMetrics() .NewTimer({"herder", "txset", "build-roundtrip"}) .TimeScope(); - ApplicableTxSetFrameConstPtr outputApplicableTxSet = - outputTxSet->prepareForApply(app, lclHeader.header); + ApplicableTxSetFrameConstPtr applicable = + txSet.prepareForApply(app, ctx.mLcl.header); roundtripTimer.Stop(); - if (!outputApplicableTxSet) + if (!applicable) { - throw std::runtime_error( - "Couldn't prepare created tx set frame for apply"); + return "Couldn't prepare created tx set frame for apply"; } // Make sure no transactions were lost during the roundtrip and the output // tx set is valid. - bool valid = preliminaryApplicableTxSet->numPhases() == - outputApplicableTxSet->numPhases(); + bool valid = phaseSizes.size() == applicable->numPhases(); if (valid) { - for (size_t i = 0; i < preliminaryApplicableTxSet->numPhases(); ++i) + for (size_t i = 0; i < phaseSizes.size(); ++i) { - valid = valid && preliminaryApplicableTxSet->sizeTx( - static_cast(i)) == - outputApplicableTxSet->sizeTx( - static_cast(i)); + valid = valid && phaseSizes[i] == + applicable->sizeTx(static_cast(i)); } } if (!valid) { - throw std::runtime_error("Created invalid tx set frame - shape is " - "mismatched after roundtrip."); + return "Created invalid tx set frame - shape is mismatched after " + "roundtrip."; } - // We already trimmed invalid transactions in an earlier call to - // `trimInvalid`, so skip transaction validation here auto finalTimer = app.getMetrics() .NewTimer({"herder", "txset", "build-final-check"}) .TimeScope(); - auto validationResult = outputApplicableTxSet->checkValidInternalWithResult( - app, closeTimeOffset.seconds(), closeTimeOffset.seconds(), true); + auto validationResult = applicable->checkValidInternalWithResult( + app, ctx, closeTimeOffset, closeTimeOffset, true); finalTimer.Stop(); if (validationResult != TxSetValidationResult::VALID) { - throw std::runtime_error(fmt::format( + return fmt::format( FMT_STRING("Created invalid tx set frame, validation result: {:s}"), - toString(validationResult))); + toString(validationResult)); + } + if (roundTripped) + { + *roundTripped = std::move(applicable); } + return std::nullopt; +} - return std::make_pair(outputTxSet, std::move(outputApplicableTxSet)); +std::optional +TxSetSelfCheck::run(Application& app) const +{ + // The check does not validate transactions, so it never asks for a view + // of the ledger. + TxSetValidationContext const ctx{ + mLcl, mSorobanConfig.get(), + []() -> std::unique_ptr { + throw std::runtime_error( + "the tx set self-check does not read the ledger"); + }, + 1}; + try + { + return TxSetBuilder::checkBuiltTxSet(app, *mTxSet, ctx, mPhaseSizes, + mCloseTimeOffset, nullptr); + } + catch (std::exception const& e) + { + return fmt::format(FMT_STRING("tx set self-check threw: {}"), e.what()); + } } TxSetXDRFrameConstPtr @@ -2458,7 +3167,20 @@ ApplicableTxSetFrame::checkValidInternalWithResult( releaseAssert(mPhases.size() == 1); } - bool enforceUniqueSourceAccount = needGeneralizedTxSet; + return checkPhasesValid(app, ctx, mPhases, isGeneralizedTxSet(), + lowerBoundCloseTimeOffset, + upperBoundCloseTimeOffset, txsAreValidated); +} + +TxSetValidationResult +ApplicableTxSetFrame::checkPhasesValid( + Application& app, TxSetValidationContext const& ctx, + std::vector const& phases, bool isGeneralized, + uint64_t lowerBoundCloseTimeOffset, uint64_t upperBoundCloseTimeOffset, + bool txsAreValidated) +{ + auto const& lcl = ctx.mLcl; + bool enforceUniqueSourceAccount = isGeneralized; #ifdef BUILD_TESTS if (app.getRunInOverlayOnlyMode()) { @@ -2470,7 +3192,7 @@ ApplicableTxSetFrame::checkValidInternalWithResult( // Ensure the tx set does not contain multiple txs per source // account std::unordered_set seenAccounts; - for (auto const& phase : mPhases) + for (auto const& phase : phases) { for (auto const& tx : phase) { @@ -2489,7 +3211,7 @@ ApplicableTxSetFrame::checkValidInternalWithResult( bool useCrossPhaseFeeMap = protocolVersionStartsFrom( lcl.header.ledgerVersion, ProtocolVersion::V_26); UnorderedMap accountFeeMap; - for (auto const& phase : mPhases) + for (auto const& phase : phases) { if (!useCrossPhaseFeeMap) { diff --git a/src/herder/TxSetFrame.h b/src/herder/TxSetFrame.h index ef074dced..888ad9e31 100644 --- a/src/herder/TxSetFrame.h +++ b/src/herder/TxSetFrame.h @@ -27,6 +27,8 @@ class AbstractLedgerView; class SorobanNetworkConfig; class TxSetXDRFrame; class ApplicableTxSetFrame; +class TxSetBuilder; +class TxSetSelfCheck; using TxSetXDRFrameConstPtr = std::shared_ptr; using TxSetReadyCallback = std::function; using ApplicableTxSetFrameConstPtr = @@ -152,6 +154,30 @@ struct TxSetValidationContext static TxSetValidationContext forLCL(Application& app); }; +// The limits a transaction set is selected within: those of the ledger it +// goes on top of. Selection (surge pricing, the Soroban stages) only reads +// the ledger through this, so it can run for a ledger that is not the main +// thread's LCL, and on another thread. +struct TxSetBuildLimits +{ + // Header of the ledger the set applies on top of. Selection reads its + // protocol version, base fee and maximum tx set size; its hashes are not + // used. + LedgerHeader mLclHeader; + // Soroban network configuration of that ledger; null before Soroban. + // Whoever makes the limits keeps it alive. + SorobanNetworkConfig const* mSorobanConfig{nullptr}; + + // Capacity of the classic phase, in operations. + uint32_t maxTxSetSizeOps() const; + // Whether a set selected within `other` is selected within these limits + // too: everything selection reads is equal. + bool sameAs(TxSetBuildLimits const& other) const; + + // The limits of the LCL as seen by the main thread. Main thread only. + static TxSetBuildLimits forLCL(Application& app); +}; + using TxFrameList = std::vector; using PerPhaseTransactionList = std::vector; @@ -215,7 +241,16 @@ makeTxSetFromTransactions( std::vector const& validTxTargets = {}, // Results of earlier validations against the LCL (see // `TxValidationCache`); extended with this build's. - TxValidationCache* validationCache = nullptr); + TxValidationCache* validationCache = nullptr, + // The builder checks every set it makes: it reads the XDR back and + // validates the structure of the result (`TxSetSelfCheck`). Without + // `deferredSelfCheck` it does so before returning, and throws if the + // check fails. With it, only the structure of the selection is validated + // before returning (`TxSetBuilder::checkSelection`; a failure throws); + // the check of the written set is not run: it is returned here for the + // caller to run (on any thread), so that it does not hold up the use of + // the set. + std::shared_ptr* deferredSelfCheck = nullptr); #ifdef BUILD_TESTS std::pair @@ -480,7 +515,9 @@ class TxSetPhaseFrame , TxSetReadyCallback const& onTxSetReady, std::vector const& validTxTargets, - TxValidationCache* validationCache); + TxValidationCache* validationCache, + std::shared_ptr* deferredSelfCheck); + friend class TxSetBuilder; #ifdef BUILD_TESTS friend std::pair makeTxSetFromTransactions( @@ -674,7 +711,9 @@ class ApplicableTxSetFrame , TxSetReadyCallback const& onTxSetReady, std::vector const& validTxTargets, - TxValidationCache* validationCache); + TxValidationCache* validationCache, + std::shared_ptr* deferredSelfCheck); + friend class TxSetBuilder; #ifdef BUILD_TESTS friend std::pair makeTxSetFromTransactions( @@ -695,6 +734,17 @@ class ApplicableTxSetFrame ApplicableTxSetFrame(ApplicableTxSetFrame const&) = default; ApplicableTxSetFrame(ApplicableTxSetFrame&&) = default; + // Everything `checkValidInternalWithResult` checks about the phases of a + // set, i.e. all of it but the set's parent and kind: one transaction per + // source account across the phases, and each phase's fees, limits and + // (unless `txsAreValidated`) transactions. Used on its own for a + // selection that has no parent yet. + static TxSetValidationResult + checkPhasesValid(Application& app, TxSetValidationContext const& ctx, + std::vector const& phases, + bool isGeneralized, uint64_t lowerBoundCloseTimeOffset, + uint64_t upperBoundCloseTimeOffset, bool txsAreValidated); + void toXDR(TransactionSet& set) const; void toXDR(GeneralizedTransactionSet& generalizedTxSet) const; @@ -716,4 +766,205 @@ class ApplicableTxSetFrame std::optional mContentsHash; }; +// The check a builder makes of a set it just built, before the set is relied +// on: the XDR is read back into an applicable set, which must have the shape +// of the selection it was written from and pass the structural validation of +// a set on top of the ledger it was built for (limits, fees, order; the +// transactions themselves were validated by the selection). +// +// It holds copies of everything it reads, so it can run on any thread while +// the main thread goes on with the set. +class TxSetSelfCheck +{ + public: + // Runs the check. Returns why the set must not be used, or nullopt if it + // passed. `app` is only read for the network ID and the configuration. + std::optional run(Application& app) const; + + TxSetXDRFrameConstPtr const& + getTxSet() const + { + return mTxSet; + } + +#ifdef BUILD_TESTS + // The same check of another set: for tests that make a builder produce + // a broken one. + std::shared_ptr + withTxSetForTesting(TxSetXDRFrameConstPtr const& txSet) const + { + std::shared_ptr res(new TxSetSelfCheck(*this)); + res->mTxSet = txSet; + return res; + } +#endif + + private: + friend class TxSetBuilder; + TxSetSelfCheck() = default; + + TxSetXDRFrameConstPtr mTxSet; + // The ledger the set was built on top of. + LedgerHeaderHistoryEntry mLcl; + std::shared_ptr mSorobanConfig; + // Number of transactions per phase of the selection. + std::vector mPhaseSizes; + uint64_t mCloseTimeOffset{0}; +}; + +// What the completion of a selection that was made ahead of time did (see +// `TxSetBuilder::completeSelection`). +struct TxSetCompletionStats +{ + // Transactions of the selection as it came. + size_t mSelected{0}; + // Of them, removed because they are not valid on the closed ledger. + size_t mRemoved{0}; + // Further candidates that were looked at, and those of them added. + size_t mCandidates{0}; + size_t mAdded{0}; +}; + +// Where the completion of a selection gets further candidates from: each call +// returns the next batch (decoded, in mempool order, none handed out before), +// or nullopt when there is nothing more to get. +struct TxSetCandidateBatch +{ + TxFrameList mTxs; + // The batch is what a pull of the mempool's top held that was not seen + // before (possibly nothing). + bool mFromMempool{false}; +}; +using TxSetCandidateSource = + std::function()>; + +// The steps `makeTxSetFromTransactions` is made of, for callers that run them +// apart from each other. +class TxSetBuilder +{ + public: + // Trims the invalid transactions of every phase and applies surge + // pricing, as `makeTxSetFromTransactions` does, but against an arbitrary + // ledger: validity is read through `ctx`, limits from `limits`. Nothing + // of the main thread's or of ledger apply's state is read, so this may + // run on any thread (`ctx` then has to ask for a single validation task). + // `timerPrefix` names the stage timers + // (`herder.txset.-trim` and `-select`). + static std::vector selectPhases( + PerPhaseTransactionList const& txPhases, Application& app, + TxSetBuildLimits const& limits, TxSetValidationContext const& ctx, + ApplyTimeOffset closeTimeOffset, PerPhaseTransactionList& invalidTxs, + std::vector const& validTxTargets, + std::string const& timerPrefix, + // Randomizes the order of transactions with equal fee + // rates. Drawn from the global random engine (main thread + // only) unless provided. + std::optional tieBreakSeed, + // If given, receives for every phase, per surge pricing lane, + // whether a transaction was left out for the lane's limits. + std::vector>* hadTxNotFittingLane = nullptr +#ifdef BUILD_TESTS + , + bool enforceTxsApplyOrder = false, + txtest::ParallelSorobanOrder const& parallelSorobanOrder = {}, + bool disableTxValidationForLegacyScenario = false +#endif + ); + + // The structural validation of a selection (ApplicableTxSetFrame's, less + // the parent hash): what a set made of `phases` would be rejected for, + // whatever its parent and its bytes. Cheap next to the check of the + // written set (TxSetSelfCheck): it reads the frames the selection + // already holds. Any thread. + static TxSetValidationResult + checkSelection(std::vector const& phases, Application& app, + TxSetValidationContext const& ctx, bool isGeneralized, + ApplyTimeOffset closeTimeOffset); + + // Completes a selection that was made before the ledger it goes on top + // of had closed, now that it has: `ctx` and `limits` describe that + // ledger (the LCL). + // + // 1. Every transaction of `selected` is validated against `ctx`, with + // the close time the set gets, by the function that validates the + // transactions of a set a node receives + // (`TxSetUtils::getInvalidTxListWithErrors`); those that fail are + // removed (and returned in `invalidTxs`). Nothing the selection was + // made on is relied upon: after this step every transaction left is + // valid on the LCL. + // 2. While a phase is below its capacity, further candidates are taken + // from `moreCandidates`, validated the same way, kept to one + // transaction per source account against what is selected, and added: + // to a parallel Soroban phase one by one (`ParallelPhaseExtender`), to + // a sequential phase by selecting it again. + // 3. The base fees are set by the rule of a build. + // + // `hadTxNotFittingLane` is what the selection reported + // (`selectPhases`). The result has not been checked for its structure: + // `writeSelection` does that. Generalized sets only. Main thread. + // + // When a phase stops asking for candidates is described at the top-up + // in the implementation: at its capacity; when valid candidates could + // not be added; after one look at the mempool's top, unless it lost + // candidates and that top was dense in its kind. + static std::vector completeSelection( + std::vector const& selected, + std::vector> const& hadTxNotFittingLane, + TxSetCandidateSource const& moreCandidates, Application& app, + TxSetBuildLimits const& limits, TxSetValidationContext const& ctx, + ApplyTimeOffset closeTimeOffset, PerPhaseTransactionList& invalidTxs, + TxSetCompletionStats& stats, std::optional tieBreakSeed); + + // The second half of `makeTxSetFromTransactions`: writes the set of a + // selection for the ledger after the LCL (which `ctx` describes) and + // checks it. With `deferredSelfCheck` the structure of the selection is + // validated before anything is written and the check of the written set + // is returned instead of run; without, the written set is read back and + // checked here. Throws if a check fails. Main thread. + static std::pair + writeSelection(std::vector const& phases, Application& app, + TxSetValidationContext const& ctx, + ApplyTimeOffset closeTimeOffset, + TxSetReadyCallback const& onTxSetReady, + std::shared_ptr* deferredSelfCheck +#ifdef BUILD_TESTS + , + bool enforceTxsApplyOrder = false +#endif + ); + + private: + friend std::pair + makeTxSetFromTransactions( + PerPhaseTransactionList const& txPhases, Application& app, + ApplyTimeOffset closeTimeOffset, + PerPhaseTransactionList& invalidTxsPerPhase +#ifdef BUILD_TESTS + , + bool enforceTxsApplyOrder, + txtest::ParallelSorobanOrder const& parallelSorobanOrder, + bool disableTxValidationForLegacyScenario +#endif + , + TxSetReadyCallback const& onTxSetReady, + std::vector const& validTxTargets, + TxValidationCache* validationCache, + std::shared_ptr* deferredSelfCheck); + friend class TxSetSelfCheck; + + static std::shared_ptr + makeSelfCheck(Application& app, TxSetXDRFrameConstPtr const& txSet, + LedgerHeaderHistoryEntry const& lcl, + std::vector const& phases, + ApplyTimeOffset closeTimeOffset); + // The check itself. `roundTripped`, if given, receives the applicable + // set read back from the XDR. + static std::optional + checkBuiltTxSet(Application& app, TxSetXDRFrame const& txSet, + TxSetValidationContext const& ctx, + std::vector const& phaseSizes, + uint64_t closeTimeOffset, + ApplicableTxSetFrameConstPtr* roundTripped); +}; + } // namespace stellar diff --git a/src/herder/TxSetUtils.cpp b/src/herder/TxSetUtils.cpp index 00483cd04..55645e368 100644 --- a/src/herder/TxSetUtils.cpp +++ b/src/herder/TxSetUtils.cpp @@ -383,6 +383,18 @@ TxSetUtils::trimInvalid(TxFrameList const& txs, Application& app, releaseAssert(threadIsMain()); auto ctx = TxSetValidationContext::forLCL(app); ctx.mValidationCache = validationCache; + return trimInvalid(txs, app, ctx, accountFeeMap, lowerBoundCloseTimeOffset, + upperBoundCloseTimeOffset, invalidTxs); +} + +TxFrameList +TxSetUtils::trimInvalid(TxFrameList const& txs, Application& app, + TxSetValidationContext const& ctx, + UnorderedMap& accountFeeMap, + uint64_t lowerBoundCloseTimeOffset, + uint64_t upperBoundCloseTimeOffset, + TxFrameList& invalidTxs) +{ invalidTxs = getInvalidTxListWithErrors(txs, app, ctx, accountFeeMap, lowerBoundCloseTimeOffset, upperBoundCloseTimeOffset) @@ -397,6 +409,23 @@ TxSetUtils::trimInvalidPrefix( Application& app, UnorderedMap& accountFeeMap, uint64_t lowerBoundCloseTimeOffset, uint64_t upperBoundCloseTimeOffset, TxFrameList& invalidTxs, TxValidationCache* validationCache) +{ + releaseAssert(threadIsMain()); + auto ctx = TxSetValidationContext::forLCL(app); + ctx.mValidationCache = validationCache; + return trimInvalidPrefix(txs, targetValidTxs, countsTowardTarget, app, ctx, + accountFeeMap, lowerBoundCloseTimeOffset, + upperBoundCloseTimeOffset, invalidTxs); +} + +TxFrameList +TxSetUtils::trimInvalidPrefix( + TxFrameList const& txs, size_t targetValidTxs, + std::function const& countsTowardTarget, + Application& app, TxSetValidationContext const& ctx, + UnorderedMap& accountFeeMap, + uint64_t lowerBoundCloseTimeOffset, uint64_t upperBoundCloseTimeOffset, + TxFrameList& invalidTxs) { ZoneScoped; TxFrameList validTxs; @@ -420,8 +449,8 @@ TxSetUtils::trimInvalidPrefix( validated += count; TxFrameList batchInvalid; for (auto const& tx : trimInvalid( - batch, app, accountFeeMap, lowerBoundCloseTimeOffset, - upperBoundCloseTimeOffset, batchInvalid, validationCache)) + batch, app, ctx, accountFeeMap, lowerBoundCloseTimeOffset, + upperBoundCloseTimeOffset, batchInvalid)) { if (!countsTowardTarget || countsTowardTarget(*tx)) { @@ -435,4 +464,77 @@ TxSetUtils::trimInvalidPrefix( return validTxs; } +// The mempool is fee-ordered and sequence-number-oblivious, so it can hand +// out several transactions from one source account (e.g. a chained pair). A +// tx set may only contain one tx per source account, so keep the lowest +// sequence number per account and let the others wait for a later ledger. +void +TxSetUtils::keepOnePerSourceAccount(TxFrameList& txs) +{ + std::unordered_map firstBySource; + TxFrameList kept; + for (auto const& tx : txs) + { + auto [it, inserted] = + firstBySource.emplace(tx->getSourceID(), kept.size()); + if (inserted) + { + kept.push_back(tx); + } + else if (tx->getSeqNum() < kept[it->second]->getSeqNum()) + { + kept[it->second] = tx; + } + } + txs = std::move(kept); +} + +// One transaction per source account holds across the phases of a set, not +// only within each: of an account's classic and Soroban candidates, the one +// with the lower sequence number stays (the classic one if they are equal). +void +TxSetUtils::keepOnePerSourceAccount(TxFrameList& classicTxs, + TxFrameList& sorobanTxs) +{ + keepOnePerSourceAccount(classicTxs); + keepOnePerSourceAccount(sorobanTxs); + if (classicTxs.empty() || sorobanTxs.empty()) + { + return; + } + std::unordered_map classicSeq; + for (auto const& tx : classicTxs) + { + classicSeq.emplace(tx->getSourceID(), tx->getSeqNum()); + } + std::unordered_set dropFromClassic; + TxFrameList keptSoroban; + for (auto const& tx : sorobanTxs) + { + auto it = classicSeq.find(tx->getSourceID()); + if (it == classicSeq.end()) + { + keptSoroban.push_back(tx); + } + else if (tx->getSeqNum() < it->second) + { + dropFromClassic.insert(it->first); + keptSoroban.push_back(tx); + } + } + sorobanTxs = std::move(keptSoroban); + if (!dropFromClassic.empty()) + { + TxFrameList keptClassic; + for (auto const& tx : classicTxs) + { + if (dropFromClassic.count(tx->getSourceID()) == 0) + { + keptClassic.push_back(tx); + } + } + classicTxs = std::move(keptClassic); + } +} + } // namespace stellar diff --git a/src/herder/TxSetUtils.h b/src/herder/TxSetUtils.h index 09ef4dc7c..960c5b318 100644 --- a/src/herder/TxSetUtils.h +++ b/src/herder/TxSetUtils.h @@ -73,6 +73,14 @@ class TxSetUtils uint64_t lowerBoundCloseTimeOffset, uint64_t upperBoundCloseTimeOffset, TxFrameList& invalidTxs, TxValidationCache* validationCache = nullptr); + // Same, against the ledger state described by `ctx` (which also carries + // the validation cache, if any) instead of the main thread's LCL. + static TxFrameList + trimInvalid(TxFrameList const& txs, Application& app, + TxSetValidationContext const& ctx, + UnorderedMap& accountFeeMap, + uint64_t lowerBoundCloseTimeOffset, + uint64_t upperBoundCloseTimeOffset, TxFrameList& invalidTxs); // Like `trimInvalid`, for candidates in priority order when only the // first `targetValidTxs` valid ones that `countsTowardTarget` accepts are @@ -88,5 +96,26 @@ class TxSetUtils Application& app, UnorderedMap& accountFeeMap, uint64_t lowerBoundCloseTimeOffset, uint64_t upperBoundCloseTimeOffset, TxFrameList& invalidTxs, TxValidationCache* validationCache = nullptr); + // Same, against the ledger state described by `ctx`. + static TxFrameList + trimInvalidPrefix(TxFrameList const& txs, size_t targetValidTxs, + std::function const& + countsTowardTarget, + Application& app, TxSetValidationContext const& ctx, + UnorderedMap& accountFeeMap, + uint64_t lowerBoundCloseTimeOffset, + uint64_t upperBoundCloseTimeOffset, + TxFrameList& invalidTxs); + + // The mempool is fee-ordered and sequence-number-oblivious, so it can + // hand out several transactions of one source account (e.g. a chained + // pair). A tx set may hold one transaction per source account: keeps + // the one with the lowest sequence number, in the place of the first. + static void keepOnePerSourceAccount(TxFrameList& txs); + // The same across the two phases of a set: of an account's classic and + // Soroban candidates, the one with the lower sequence number stays (the + // classic one if they are equal). + static void keepOnePerSourceAccount(TxFrameList& classicTxs, + TxFrameList& sorobanTxs); }; // class TxSetUtils } // namespace stellar diff --git a/src/herder/test/EarlyTxSetTests.cpp b/src/herder/test/EarlyTxSetTests.cpp new file mode 100644 index 000000000..337589127 --- /dev/null +++ b/src/herder/test/EarlyTxSetTests.cpp @@ -0,0 +1,2673 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +// Tests of the selection of the next transaction set before the current +// ledger has closed (herder/EarlyTxSetSelection.h), of its completion on the +// closed ledger (TxSetBuilder::completeSelection, HerderImpl::buildTxSet), and +// of the pull of a build (capacity plus a margin, refilled when validation +// leaves the set short). +// +// Two properties are what these tests are for: +// a. whatever the overlap between ledger N and the candidates of N+1, the +// set a node proposes for N+1 passes the validation every node applies +// to it, against N as it really closed; +// b. the set is as full as the mempool allows. + +#include "crypto/SHA.h" +#include "herder/EarlyTxSetSelection.h" +#include "herder/Herder.h" +#include "herder/HerderImpl.h" +#include "herder/HerderUtils.h" +#include "herder/LedgerCloseData.h" +#include "herder/ParallelTxSetBuilder.h" +#include "herder/TxSetFrame.h" +#include "herder/Upgrades.h" +#include "herder/test/TestTxSetUtils.h" +#include "ledger/ImmutableLedgerView.h" +#include "ledger/LedgerManager.h" +#include "ledger/test/LedgerTestUtils.h" +#include "main/Application.h" +#include "medida/histogram.h" +#include "medida/meter.h" +#include "simulation/ApplyLoad.h" +#include "simulation/Simulation.h" +#include "simulation/Topologies.h" +#include "test/Catch2.h" +#include "test/TestAccount.h" +#include "test/TestUtils.h" +#include "test/TxTests.h" +#include "test/test.h" +#include "transactions/SignatureUtils.h" +#include "transactions/TransactionUtils.h" +#include "util/BatchExecutor.h" +#include "util/Math.h" +#include "util/MetricsRegistry.h" +#include "util/ProtocolVersion.h" +#include "util/XDROperators.h" +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace stellar; +using namespace stellar::txtest; + +namespace stellar +{ +class EarlyTxSetTestAccess +{ + public: + // What the herder does with a statement this node emits. + static void + voted(HerderImpl& herder, SCPEnvelope const& envelope) + { + herder.maybeScheduleEarlyTxSetSelection(envelope); + } + + static bool + hasPlan(HerderImpl& herder) + { + return herder.mEarlyTxSetPlan.has_value(); + } + + static bool + planned(HerderImpl& herder) + { + return herder.mEarlyTxSetPlan && herder.mEarlyTxSetPlan->planned; + } + + static std::shared_ptr + job(HerderImpl& herder) + { + return herder.mEarlyTxSet; + } + + static auto + build(HerderImpl& herder, uint32_t ledgerSeq, ConsensusTime closeTime) + { + return herder.buildTxSet(ledgerSeq, closeTime, false); + } + + static size_t + pullCount(HerderImpl& herder) + { + return herder.txSetPullCount(); + } + + static void + setStartDelay(HerderImpl& herder, + std::optional delay) + { + herder.mEarlyTxSetStartDelayForTesting = delay; + } + + static std::chrono::milliseconds + startDelay(HerderImpl& herder) + { + return herder.earlyTxSetStartDelay(); + } + + static void + setTimings(HerderImpl& herder, std::chrono::milliseconds voteToBuild, + std::chrono::milliseconds lastVoteToBuild, + std::chrono::milliseconds job, std::chrono::milliseconds lastJob) + { + herder.mVoteToBuild = voteToBuild; + herder.mLastVoteToBuild = lastVoteToBuild; + herder.mEarlyTxSetJobDuration = job; + herder.mLastEarlyTxSetJobDuration = lastJob; + } + + // What the node takes a selection job to last (running average, last). + static std::pair + jobDuration(HerderImpl& herder) + { + return {herder.mEarlyTxSetJobDuration, + herder.mLastEarlyTxSetJobDuration}; + } + + static void + skipFinalPass(HerderImpl& herder, bool skip) + { + herder.mSkipEarlyTxSetFinalPassForTesting = skip; + } + + static void + hold(HerderImpl& herder, std::shared_ptr> hold) + { + herder.mEarlyTxSetHoldForTesting = std::move(hold); + } + + static bool + selfCheckLatched(HerderImpl& herder) + { + return herder.mTxSetSelfCheckFailed; + } +}; +} + +namespace +{ +using Access = EarlyTxSetTestAccess; + +uint32_t const GATE_PROTOCOL = + static_cast(APPLY_TIME_TX_SET_VALIDATION_PROTOCOL_VERSION); + +int64_t +txsetMeter(Application& app, std::string const& name, + std::string const& unit = "txset") +{ + return static_cast( + app.getMetrics().NewMeter({"herder", "txset", name}, unit).count()); +} + +int64_t +applyCounter(Application& app, std::string const& name) +{ + return app.getMetrics().NewCounter({"ledger", "apply", name}).count(); +} + +std::set +hashesOf(std::vector const& txs) +{ + std::set res; + for (auto const& tx : txs) + { + res.insert(tx->getFullHash()); + } + return res; +} + +std::set +hashesOf(ApplicableTxSetFrame const& txSet) +{ + std::set res; + for (auto const& phase : txSet.getPhases()) + { + for (auto const& tx : phase) + { + res.insert(tx->getFullHash()); + } + } + return res; +} + +// What a node got when it asked for its next set. +struct Built +{ + TxSetXDRFrameConstPtr mTxSet; + ApplicableTxSetFrameConstPtr mApplicable; + // The set is a completed early selection, not a build. + bool mEarly{false}; + std::vector mInvalid; + + std::set + hashes() const + { + return hashesOf(*mApplicable); + } + + bool + holds(TransactionFrameBasePtr const& tx) const + { + return hashes().count(tx->getFullHash()) != 0; + } +}; + +// A single validator (so it leads every ledger) with funded accounts. Its +// mempool is `mMempool`, pulled in that order; the test plays the network: +// it has the node vote for a set, closes the ledger with a set, and asks the +// node for its next set. +struct EarlyFixture +{ + VirtualClock mClock; + Config mCfg; + Application::pointer mApp; + std::vector mAccounts; + std::vector mMempool; + // What each pull of the mempool asked for. + std::vector mPulls; + + explicit EarlyFixture(size_t accounts = 8, + std::optional maxTxSetSize = std::nullopt, + std::function tweak = nullptr) + : mCfg(getTestConfig(0, Config::TESTDB_BUCKET_DB_PERSISTENT)) + { + mCfg.LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; + mCfg.TESTING_UPGRADE_LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; + if (maxTxSetSize) + { + mCfg.TESTING_UPGRADE_MAX_TX_SET_SIZE = *maxTxSetSize; + } + // The tests close ledgers themselves. + mCfg.EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY = false; + if (tweak) + { + tweak(mCfg); + } + REQUIRE(mCfg.backgroundLedgerApply()); + mApp = createTestApplication(mClock, mCfg); + auto root = mApp->getRoot(); + auto const balance = + mApp->getLedgerManager().getLastMinBalance(2) + 1'000'000'000; + for (size_t i = 0; i < accounts; ++i) + { + mAccounts.emplace_back( + root->create(fmt::format("account-{}", i), balance)); + } + // An early selection starts as soon as the node has voted. + Access::setStartDelay(herder(), std::chrono::milliseconds(0)); + herder().mGetTopTransactionsForTesting = [this](size_t count) { + mPulls.push_back(count); + std::vector envelopes; + for (size_t i = 0; i < mMempool.size() && i < count; ++i) + { + envelopes.push_back(mMempool[i]->getEnvelope()); + } + return envelopes; + }; + } + + ~EarlyFixture() + { + herder().mGetTopTransactionsForTesting = nullptr; + } + + HerderImpl& + herder() + { + return static_cast(mApp->getHerder()); + } + + LedgerManager& + lm() + { + return mApp->getLedgerManager(); + } + + TestAccount& + acc(size_t i) + { + return mAccounts.at(i); + } + + uint32_t + nextSeq() + { + return lm().getLastClosedLedgerNum() + 1; + } + + ConsensusTime + nextCloseTime() + { + return ConsensusTime::fromApplyTime(ApplyTime::fromTimePoint( + lm().getLastClosedLedgerHeader().header.scpValue.closeTime + 1)); + } + + // The sequence number and balance of an account on the LCL. + SequenceNumber + seqOf(TestAccount& account) + { + auto entry = lm().getLCLView()->getAccount(account.getPublicKey()); + REQUIRE(entry); + return entry.current().data.account().seqNum; + } + + bool + exists(TestAccount& account) + { + return static_cast( + lm().getLCLView()->getAccount(account.getPublicKey())); + } + + // A payment of `account` to the root account with the `n`-th sequence + // number after the account's current one. + TransactionFrameBasePtr + pay(TestAccount& account, int n, int64_t amount = 1, uint32_t fee = 100) + { + return transactionFromOperations( + *mApp, account.getSecretKey(), seqOf(account) + n, + {payment(mApp->getRoot()->getPublicKey(), amount)}, fee); + } + + TransactionFrameBasePtr + tx(TestAccount& account, int n, std::vector const& ops, + uint32_t fee = 100) + { + return transactionFromOperations( + *mApp, account.getSecretKey(), seqOf(account) + n, ops, + fee * static_cast(ops.size())); + } + + // A Soroban transaction of `account` that conflicts with no other + // (`seed` picks the entry it writes). + TransactionFrameBasePtr + upload(TestAccount& account, int n, uint64_t seed, + uint32_t inclusionFee = 100, uint32_t instructions = 1'000'000) + { + SorobanResources resources; + resources.instructions = instructions; + resources.diskReadBytes = 1000; + resources.writeBytes = 2000; + return createUploadWasmTx(*mApp, account, inclusionFee, + DEFAULT_TEST_RESOURCE_FEE, resources, + std::nullopt, 0, std::nullopt, + seqOf(account) + n, std::make_optional(seed)); + } + + // A set of classic transactions on top of the LCL, not validated. + TxSetXDRFrameConstPtr + setOf(std::vector const& txs) + { + auto res = testtxset::makeNonValidatedGeneralizedTxSet( + {{std::make_pair(std::nullopt, txs)}, {}}, *mApp, + lm().getLastClosedLedgerHeader().hash); + REQUIRE(res.first); + return res.first; + } + + StellarValue + valueFor(TxSetXDRFrameConstPtr const& txSet, + xdr::xvector const& upgrades = emptyUpgradeSteps, + std::optional closeTime = std::nullopt) + { + return herder().makeStellarValue( + txSet->getContentsHash(), closeTime ? *closeTime : nextCloseTime(), + upgrades, mCfg.NODE_SEED); + } + + // Cranks until `done`, giving background threads time to post to the + // main thread. + template + void + crankUntil(F&& done) + { + auto const deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (!done()) + { + REQUIRE(std::chrono::steady_clock::now() < deadline); + mClock.crank(false); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + } + + void + crankFor(std::chrono::milliseconds duration) + { + auto const deadline = std::chrono::steady_clock::now() + duration; + while (std::chrono::steady_clock::now() < deadline) + { + mClock.crank(false); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + } + + // The node emits a vote for `closing` as the next ledger. + void + vote(TxSetXDRFrameConstPtr const& closing, + xdr::xvector const& upgrades = emptyUpgradeSteps) + { + // By the time a node votes, SCP has asked for the slot's leaders, + // which brings the tracked quorum (the node set) up to date. + REQUIRE(herder().getHerderSCPDriver().getLeaderSchedule(nextSeq()) == + std::vector{mCfg.NODE_SEED.getPublicKey()}); + herder().getPendingEnvelopes().putTxSet(closing->getContentsHash(), + nextSeq(), closing); + SCPEnvelope envelope; + envelope.statement.slotIndex = nextSeq(); + envelope.statement.pledges.type(SCP_ST_PREPARE); + envelope.statement.pledges.prepare().ballot.counter = 1; + envelope.statement.pledges.prepare().ballot.value = + xdr::xdr_to_opaque(valueFor(closing, upgrades)); + Access::voted(herder(), envelope); + } + + // Waits until the selection the vote planned has started. + std::shared_ptr + waitStarted() + { + REQUIRE(Access::planned(herder())); + crankUntil([&]() { return Access::job(herder()) != nullptr; }); + return Access::job(herder()); + } + + // Waits until it has finished; returns whether it selected a set. + bool + waitSelected() + { + auto job = waitStarted(); + crankUntil([&]() { return job->mDone.load(); }); + std::lock_guard lock(job->mMutex); + return job->mSelection != nullptr; + } + + // The node votes for `closing` and selects the set of the ledger after. + void + voteAndSelect(TxSetXDRFrameConstPtr const& closing) + { + vote(closing); + REQUIRE(waitSelected()); + } + + // The next ledger closes with `closing`. The overlay drops a closed + // set's transactions from the mempool; with `keepInMempool` it has not + // got to it yet. + void + close(TxSetXDRFrameConstPtr const& closing, bool keepInMempool = false, + xdr::xvector const& upgrades = emptyUpgradeSteps, + std::optional closeTime = std::nullopt) + { + auto const seq = nextSeq(); + mApp->getHerder().externalizeValue( + closing, seq, closeTime ? *closeTime : nextCloseTime(), upgrades); + REQUIRE(lm().getLastClosedLedgerNum() == seq); + if (!keepInMempool) + { + std::set closed; + for (auto const& hash : closing->getTransactionFullHashes()) + { + closed.insert(hash); + } + std::vector left; + for (auto const& tx : mMempool) + { + if (closed.count(tx->getFullHash()) == 0) + { + left.push_back(tx); + } + } + mMempool = std::move(left); + } + } + + // The node's set for the next ledger. + Built + build() + { + auto const completed = txsetMeter(*mApp, "early-completed"); + auto prepared = Access::build(herder(), nextSeq(), nextCloseTime()); + Built built; + built.mTxSet = prepared.txSet; + built.mApplicable = std::move(prepared.applicableTxSet); + built.mInvalid = prepared.invalidTxHashes; + built.mEarly = txsetMeter(*mApp, "early-completed") == completed + 1; + REQUIRE(built.mTxSet); + REQUIRE(built.mApplicable); + return built; + } + + // Property a: the set passes the validation every node applies to a + // set, in full (structure and every transaction), on the LCL. + TxSetValidationResult + validate(Built const& built) + { + // As a node that receives the set: read from its bytes. (build() + // asks for a close time one second after the LCL's.) + auto received = TxSetXDRFrame::makeFromWire( + GeneralizedTransactionSet(built.mTxSet->getGeneralizedXDR())); + REQUIRE(received->getContentsHash() == built.mTxSet->getContentsHash()); + auto applicable = received->prepareForApply( + *mApp, lm().getLastClosedLedgerHeader().header); + REQUIRE(applicable); + REQUIRE(hashesOf(*applicable) == built.hashes()); + auto const result = applicable->checkValidWithResult(*mApp, 1, 1); + // The set the node keeps says the same. + REQUIRE(built.mApplicable->checkValidWithResult(*mApp, 1, 1) == result); + return result; + } + + // The set a build from scratch selects out of `txs` on the LCL. + Built + buildFrom(std::vector const& txs) + { + auto mempool = mMempool; + mMempool = txs; + auto built = build(); + mMempool = std::move(mempool); + REQUIRE(!built.mEarly); + return built; + } + + // Closes the next ledger with a set this node proposed and checks that + // it was applied: the set was not skipped as invalid, and every one of + // its transactions consumed its sequence number. + void + closeAndCheckApplied(Built const& built) + { + REQUIRE(validate(built) == TxSetValidationResult::VALID); + auto const skipped = applyCounter(*mApp, "txset-skipped"); + std::vector> expected; + for (auto const& phase : built.mApplicable->getPhases()) + { + for (auto const& tx : phase) + { + expected.emplace_back(tx->getSourceID(), tx->getSeqNum()); + } + } + close(built.mTxSet); + REQUIRE(applyCounter(*mApp, "txset-skipped") == skipped); + auto view = lm().getLCLView(); + for (auto const& [account, seq] : expected) + { + auto entry = view->getAccount(account); + // (an account that merged itself away is gone) + if (entry) + { + REQUIRE(entry.current().data.account().seqNum >= seq); + } + } + } +}; + +} + +// --------------------------------------------------------------------------- +// The mechanism. +// --------------------------------------------------------------------------- + +TEST_CASE("the next tx set is selected while the current ledger is in its " + "ballot", + "[herder][txset][early-txset]") +{ + EarlyFixture f; + auto& app = *f.mApp; + auto& herder = f.herder(); + auto closingA = f.pay(f.acc(0), 1); + auto nextA = f.pay(f.acc(0), 2); + auto plainB = f.pay(f.acc(1), 1); + auto closing = f.setOf({closingA}); + // The closing ledger's transaction is still in the mempool. + f.mMempool = {closingA, nextA, plainB}; + + auto const planned = txsetMeter(app, "early-planned"); + auto const started = txsetMeter(app, "early-started"); + f.vote(closing); + REQUIRE(Access::planned(herder)); + REQUIRE(txsetMeter(app, "early-planned") == planned + 1); + // Nothing is pulled before the selection starts. + REQUIRE(f.mPulls.empty()); + REQUIRE(f.waitSelected()); + REQUIRE(txsetMeter(app, "early-started") == started + 1); + // One pull: the capacity of a set plus the margin, plus the closing + // ledger's transactions. + REQUIRE(f.mPulls == std::vector{Access::pullCount(herder) + 1}); + + { + auto job = Access::job(herder); + std::lock_guard lock(job->mMutex); + auto const& selection = *job->mSelection; + REQUIRE(selection.mLedgerSeq == f.nextSeq() + 1); + std::set selected; + for (auto const& phase : selection.mPhases) + { + for (auto const& tx : phase) + { + selected.insert(tx->getFullHash()); + } + } + // A's next payment is only valid once the closing ledger has + // applied: the selection is made on what that ledger will do. + REQUIRE(selected == hashesOf({nextA, plainB})); + REQUIRE(selection.mLeftovers.empty()); + } + + SECTION("the ledger closes with the value: the selection is completed") + { + f.close(closing); + f.mPulls.clear(); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({nextA, plainB})); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + REQUIRE(built.mTxSet->previousLedgerHash() == + f.lm().getLastClosedLedgerHeader().hash); + REQUIRE(built.mInvalid.empty()); + // The set had room: the mempool was asked once for what arrived + // since (nothing). + REQUIRE(f.mPulls == std::vector{Access::pullCount(herder)}); + // The same set a build from scratch selects. + REQUIRE(f.buildFrom({nextA, plainB}).hashes() == built.hashes()); + f.closeAndCheckApplied(built); + // The plan is used up. + REQUIRE(!Access::hasPlan(herder)); + REQUIRE(!Access::job(herder)); + } + SECTION("a second build on the same ledger is a build") + { + f.close(closing); + REQUIRE(f.build().mEarly); + auto const again = f.build(); + REQUIRE(!again.mEarly); + REQUIRE(again.hashes() == hashesOf({nextA, plainB})); + } + SECTION("statements for the same value change nothing") + { + auto job = Access::job(herder); + f.vote(closing); + REQUIRE(Access::job(herder) == job); + REQUIRE(txsetMeter(app, "early-planned") == planned + 1); + } +} + +TEST_CASE("when a tx set is selected early", "[herder][txset][early-txset]") +{ + SECTION("not when the setting is off") + { + EarlyFixture f(2, std::nullopt, [](Config& cfg) { + cfg.EXPERIMENTAL_EARLY_TX_SET_SELECTION = false; + }); + auto closingA = f.pay(f.acc(0), 1); + auto nextA = f.pay(f.acc(0), 2); + f.mMempool = {closingA, nextA}; + auto closing = f.setOf({closingA}); + f.vote(closing); + REQUIRE(!Access::hasPlan(f.herder())); + f.close(closing); + auto const built = f.build(); + REQUIRE(!built.mEarly); + REQUIRE(built.hashes() == hashesOf({nextA})); + } + SECTION("not for a value with upgrades") + { + EarlyFixture f(2); + auto closingA = f.pay(f.acc(0), 1); + auto nextA = f.pay(f.acc(0), 2); + f.mMempool = {closingA, nextA}; + auto closing = f.setOf({closingA}); + xdr::xvector upgrades; + LedgerUpgrade upgrade(LEDGER_UPGRADE_BASE_FEE); + upgrade.newBaseFee() = 150; + upgrades.emplace_back(LedgerTestUtils::toUpgradeType(upgrade)); + f.vote(closing, upgrades); + // Looked at, and nothing planned. + REQUIRE(Access::hasPlan(f.herder())); + REQUIRE(!Access::planned(f.herder())); + f.crankFor(std::chrono::milliseconds(20)); + REQUIRE(!Access::job(f.herder())); + REQUIRE(f.mPulls.empty()); + } + SECTION("not for an empty set") + { + EarlyFixture f(2); + f.mMempool = {f.pay(f.acc(0), 1)}; + auto empty = + TxSetXDRFrame::makeEmpty(f.lm().getLastClosedLedgerHeader()); + f.vote(empty); + REQUIRE(!Access::planned(f.herder())); + } + SECTION("not for a ledger that is not the next one") + { + EarlyFixture f(2); + auto closingA = f.pay(f.acc(0), 1); + auto closing = f.setOf({closingA}); + SCPEnvelope envelope; + envelope.statement.slotIndex = f.nextSeq() + 1; + envelope.statement.pledges.type(SCP_ST_PREPARE); + envelope.statement.pledges.prepare().ballot.counter = 1; + envelope.statement.pledges.prepare().ballot.value = + xdr::xdr_to_opaque(f.valueFor(closing)); + Access::voted(f.herder(), envelope); + REQUIRE(!Access::hasPlan(f.herder())); + } + SECTION("the start is timed to end shortly before the set is needed") + { + using std::chrono::milliseconds; + EarlyFixture f(1); + auto& herder = f.herder(); + Access::setStartDelay(herder, std::nullopt); + // Nothing measured: at once. + Access::setTimings(herder, milliseconds(0), milliseconds(0), + milliseconds(0), milliseconds(0)); + REQUIRE(Access::startDelay(herder) == milliseconds(0)); + Access::setTimings(herder, milliseconds(800), milliseconds(800), + milliseconds(0), milliseconds(0)); + REQUIRE(Access::startDelay(herder) == milliseconds(0)); + // 800 ms from the vote to the build, a selection takes 200 ms: + // 800 - 200 - 50 - 30. + Access::setTimings(herder, milliseconds(800), milliseconds(800), + milliseconds(200), milliseconds(200)); + REQUIRE(Access::startDelay(herder) == milliseconds(520)); + // The earlier of the two build times, the longer of the two jobs. + Access::setTimings(herder, milliseconds(800), milliseconds(700), + milliseconds(200), milliseconds(240)); + REQUIRE(Access::startDelay(herder) == + milliseconds(700 - 240 - 60 - 30)); + // Never negative. + Access::setTimings(herder, milliseconds(100), milliseconds(100), + milliseconds(200), milliseconds(200)); + REQUIRE(Access::startDelay(herder) == milliseconds(0)); + } + SECTION("a selection that has not started when the ledger closes") + { + EarlyFixture f(2); + auto& app = *f.mApp; + auto closingA = f.pay(f.acc(0), 1); + auto nextA = f.pay(f.acc(0), 2); + f.mMempool = {closingA, nextA}; + auto closing = f.setOf({closingA}); + // The start is due long after the close. + Access::setStartDelay(f.herder(), std::chrono::milliseconds(60'000)); + f.vote(closing); + REQUIRE(Access::planned(f.herder())); + f.close(closing); + auto const notStarted = txsetMeter(app, "early-not-started"); + auto const built = f.build(); + REQUIRE(!built.mEarly); + REQUIRE(txsetMeter(app, "early-not-started") == notStarted + 1); + REQUIRE(built.hashes() == hashesOf({nextA})); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + } +} + +// --------------------------------------------------------------------------- +// Property a: no invalid set, whatever the overlap between N and N+1. +// --------------------------------------------------------------------------- + +TEST_CASE("a set selected early is valid on the closed ledger whatever the " + "previous set held", + "[herder][txset][early-txset]") +{ + EarlyFixture f(8); + auto& herder = f.herder(); + auto& a = f.acc(0); + auto& b = f.acc(1); + auto& c = f.acc(2); + auto& d = f.acc(3); + + SECTION("the candidates are exactly the previous set's transactions") + { + std::vector txs{f.pay(a, 1), f.pay(b, 1), + f.pay(c, 1)}; + auto closing = f.setOf(txs); + f.mMempool = txs; + f.voteAndSelect(closing); + SECTION("the overlay has dropped them at the close") + { + f.close(closing); + } + SECTION("the overlay still holds them at the close") + { + f.close(closing, /*keepInMempool=*/true); + } + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes().empty()); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + } + SECTION("the same accounts with their next sequence numbers") + { + std::vector first{f.pay(a, 1), f.pay(b, 1), + f.pay(c, 1)}; + std::vector second{f.pay(a, 2), f.pay(b, 2), + f.pay(c, 2)}; + auto closing = f.setOf(first); + f.mMempool = first; + f.mMempool.insert(f.mMempool.end(), second.begin(), second.end()); + f.voteAndSelect(closing); + f.close(closing, /*keepInMempool=*/true); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf(second)); + f.closeAndCheckApplied(built); + REQUIRE(f.seqOf(a) == second[0]->getSeqNum()); + } + SECTION("a transaction of the previous set fails when it is applied") + { + // The payment is more than the account has: the transaction is + // valid, fails, and consumes its sequence number all the same. + auto failing = f.pay(a, 1, a.getBalance() * 2); + auto nextA = f.pay(a, 2); + auto closing = f.setOf({failing}); + f.mMempool = {failing, nextA}; + f.voteAndSelect(closing); + auto const balance = a.getBalance(); + f.close(closing); + // It failed: only the fee is gone. + REQUIRE(a.getBalance() == balance - 100); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({nextA})); + f.closeAndCheckApplied(built); + } + SECTION("another transaction with a sequence number the previous set " + "used") + { + auto closingA = f.pay(a, 1, 1); + auto sameSeq = f.pay(a, 1, 2); + auto plainB = f.pay(b, 1); + auto closing = f.setOf({closingA}); + f.mMempool = {closingA, sameSeq, plainB}; + f.voteAndSelect(closing); + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({plainB})); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + } + SECTION("the previous set drains an account") + { + // Everything but the reserve goes: A cannot pay another fee. The + // selection cannot know (it predicts fees, not payments). + auto const spare = a.getAvailableBalance() - 100; + auto drain = f.pay(a, 1, spare); + auto nextA = f.pay(a, 2); + auto plainB = f.pay(b, 1); + auto closing = f.setOf({drain}); + f.mMempool = {drain, nextA, plainB}; + f.voteAndSelect(closing); + { + auto job = Access::job(herder); + std::lock_guard lock(job->mMutex); + REQUIRE(job->mSelection->mSelected == 2); + } + f.close(closing); + REQUIRE(a.getAvailableBalance() == 0); + SECTION("the validity pass removes its next transaction") + { + auto const removed = txsetMeter(*f.mApp, "early-removed", "tx"); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({plainB})); + REQUIRE(txsetMeter(*f.mApp, "early-removed", "tx") == removed + 1); + // It could not pay its fee: out of the mempool. + REQUIRE(built.mInvalid == std::vector{nextA->getFullHash()}); + f.closeAndCheckApplied(built); + } + SECTION("without the validity pass the set would be invalid") + { + Access::skipFinalPass(herder, true); + auto const built = f.build(); + Access::skipFinalPass(herder, false); + REQUIRE(built.mEarly); + REQUIRE(built.holds(nextA)); + REQUIRE(f.validate(built) != TxSetValidationResult::VALID); + } + } + SECTION("the previous set merges an account away") + { + auto merge = f.tx(a, 1, {accountMerge(b.getPublicKey())}); + auto nextA = f.pay(a, 2); + auto plainC = f.pay(c, 1); + auto closing = f.setOf({merge}); + f.mMempool = {merge, nextA, plainC}; + f.voteAndSelect(closing); + f.close(closing); + REQUIRE(!f.exists(a)); + SECTION("the validity pass removes its next transaction") + { + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({plainC})); + f.closeAndCheckApplied(built); + } + SECTION("without the validity pass the set would be invalid") + { + Access::skipFinalPass(herder, true); + auto const built = f.build(); + Access::skipFinalPass(herder, false); + REQUIRE(built.holds(nextA)); + REQUIRE(f.validate(built) != TxSetValidationResult::VALID); + } + } + SECTION("the previous set changes an account's signers") + { + // The master key loses its weight to another key: what A signed + // with it no longer counts. + auto other = SecretKey::pseudoRandomForTesting(); + auto rekey = f.tx(a, 1, + {setOptions(setSigner(makeSigner(other, 1))), + setOptions(setMasterWeight(0))}); + auto nextA = f.pay(a, 2); + auto plainC = f.pay(c, 1); + auto closing = f.setOf({rekey}); + f.mMempool = {rekey, nextA, plainC}; + f.voteAndSelect(closing); + f.close(closing); + REQUIRE(f.seqOf(a) == rekey->getSeqNum()); + SECTION("the validity pass removes what the old key signed") + { + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({plainC})); + f.closeAndCheckApplied(built); + } + SECTION("without the validity pass the set would be invalid") + { + Access::skipFinalPass(herder, true); + auto const built = f.build(); + Access::skipFinalPass(herder, false); + REQUIRE(built.holds(nextA)); + REQUIRE(f.validate(built) != TxSetValidationResult::VALID); + } + } + SECTION("fee bumps") + { + // D pays for A's transactions. In the previous set it also sends + // away all it has but one more fee bump's worth. + auto innerClosing = f.pay(a, 1); + auto bumpClosing = feeBump(*f.mApp, d, innerClosing, 200); + auto const fees = bumpClosing->getFullFee(); + auto drainD = f.pay(d, 1, d.getAvailableBalance() - 100 - 2 * fees); + auto closing = f.setOf({bumpClosing, drainD}); + // Two fee bumps by D for the next ledger: it can pay for one. + auto bumpNextA = feeBump(*f.mApp, d, f.pay(a, 2), 200); + auto bumpB = feeBump(*f.mApp, d, f.pay(b, 1), 200); + auto plainC = f.pay(c, 1); + f.mMempool = {bumpClosing, drainD, bumpNextA, bumpB, plainC}; + f.voteAndSelect(closing); + f.close(closing); + REQUIRE(f.seqOf(a) == innerClosing->getSeqNum()); + REQUIRE(d.getAvailableBalance() == fees); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.holds(plainC)); + // One of D's two, whichever comes first, and not the other. + REQUIRE(built.holds(bumpNextA)); + REQUIRE(!built.holds(bumpB)); + f.closeAndCheckApplied(built); + } + SECTION("the previous set's own transactions were not applied") + { + // The set this node voted for is not valid (a sequence number too + // far ahead): the ledger closes with it and applies nothing. What + // the selection predicted of it is wrong for every account. + auto closingA = f.pay(a, 1); + auto bad = f.pay(c, 5); + auto nextA = f.pay(a, 2); + auto plainB = f.pay(b, 1); + auto closing = f.setOf({closingA, bad}); + f.mMempool = {closingA, bad, nextA, plainB}; + f.voteAndSelect(closing); + auto const skipped = applyCounter(*f.mApp, "txset-skipped"); + auto const seqA = f.seqOf(a); + f.close(closing); + REQUIRE(applyCounter(*f.mApp, "txset-skipped") == skipped + 1); + REQUIRE(f.seqOf(a) == seqA); + auto const built = f.build(); + REQUIRE(built.mEarly); + // A's second payment waits for its first. + REQUIRE(built.hashes() == hashesOf({plainB})); + f.closeAndCheckApplied(built); + } +} + +TEST_CASE("a set selected early is valid for any overlap with the previous " + "set", + "[herder][txset][early-txset]") +{ + // Per account, one of: nothing in the previous set; its next transaction + // in the previous set; and as candidates any of: the transaction after + // that, another one with the sequence number the previous set uses, one + // far ahead. In random order, for a random share of the accounts. + size_t const ACCOUNTS = 24; + EarlyFixture f(ACCOUNTS); + auto const seed = GENERATE(1u, 2u, 3u, 4u, 5u, 6u); + auto const overlapPercent = GENERATE(0, 30, 70, 100); + CAPTURE(seed, overlapPercent); + std::mt19937 gen(seed * 1000 + static_cast(overlapPercent)); + auto chance = [&](int percent) { + return stellar::uniform_int_distribution(0, 99)(gen) < percent; + }; + + std::vector previous; + std::vector candidates; + // What must be in the next set, and what must not. + std::vector valid; + std::vector invalid; + for (size_t i = 0; i < ACCOUNTS; ++i) + { + auto& account = f.acc(i); + bool const inPrevious = chance(overlapPercent); + if (inPrevious) + { + previous.push_back(f.pay(account, 1, 1)); + if (chance(60)) + { + valid.push_back(f.pay(account, 2)); + candidates.push_back(valid.back()); + } + if (chance(30)) + { + invalid.push_back(f.pay(account, 1, 7)); + candidates.push_back(invalid.back()); + } + } + else if (chance(70)) + { + valid.push_back(f.pay(account, 1)); + candidates.push_back(valid.back()); + } + if (chance(20)) + { + invalid.push_back(f.pay(account, 9)); + candidates.push_back(invalid.back()); + } + } + if (previous.empty()) + { + // (an empty set plans no selection) + return; + } + f.mMempool = previous; + f.mMempool.insert(f.mMempool.end(), candidates.begin(), candidates.end()); + stellar::shuffle(f.mMempool.begin(), f.mMempool.end(), gen); + + auto closing = f.setOf(previous); + f.voteAndSelect(closing); + f.close(closing, /*keepInMempool=*/chance(50)); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + auto const hashes = built.hashes(); + for (auto const& tx : previous) + { + REQUIRE(hashes.count(tx->getFullHash()) == 0); + } + for (auto const& tx : invalid) + { + REQUIRE(hashes.count(tx->getFullHash()) == 0); + } + // Nothing is left behind that a build would take: it is the set a build + // from scratch selects on the closed ledger once the closed set's + // transactions have left the mempool. (A candidate that reuses a + // sequence number shadows the account's next one in both: one + // transaction per account, the lowest sequence number first.) + std::vector stillPending; + { + auto const closed = hashesOf(previous); + for (auto const& tx : f.mMempool) + { + if (closed.count(tx->getFullHash()) == 0) + { + stillPending.push_back(tx); + } + } + } + REQUIRE(hashes == f.buildFrom(stillPending).hashes()); + std::set validHashes = hashesOf(valid); + for (auto const& hash : hashes) + { + REQUIRE(validHashes.count(hash) == 1); + } + f.closeAndCheckApplied(built); +} + +TEST_CASE("a set selected early is not used when the ledger closes " + "otherwise", + "[herder][txset][early-txset]") +{ + EarlyFixture f(4); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = f.acc(0); + auto& b = f.acc(1); + auto closingA = f.pay(a, 1); + auto nextA = f.pay(a, 2); + auto plainB = f.pay(b, 1); + auto closing = f.setOf({closingA}); + f.mMempool = {closingA, nextA, plainB}; + f.voteAndSelect(closing); + auto const discarded = txsetMeter(app, "early-discarded"); + + SECTION("another set closes the ledger") + { + f.close(f.setOf({plainB})); + REQUIRE(txsetMeter(app, "early-discarded") == discarded + 1); + REQUIRE(!Access::hasPlan(herder)); + auto const built = f.build(); + REQUIRE(!built.mEarly); + // A's first payment is still to come. + REQUIRE(built.hashes() == hashesOf({closingA})); + f.closeAndCheckApplied(built); + } + SECTION("the same set closes the ledger with another close time") + { + auto const later = + ConsensusTime::fromApplyTime(ApplyTime::fromTimePoint( + f.lm().getLastClosedLedgerHeader().header.scpValue.closeTime + + 3)); + f.close(closing, false, emptyUpgradeSteps, later); + REQUIRE(txsetMeter(app, "early-discarded") == discarded + 1); + auto const built = f.build(); + REQUIRE(!built.mEarly); + REQUIRE(built.hashes() == hashesOf({nextA, plainB})); + } + SECTION("the ledger closes with an upgrade that lowers the limits") + { + xdr::xvector upgrades; + LedgerUpgrade upgrade(LEDGER_UPGRADE_MAX_TX_SET_SIZE); + upgrade.newMaxTxSetSize() = 1; + upgrades.emplace_back(LedgerTestUtils::toUpgradeType(upgrade)); + f.close(closing, false, upgrades); + REQUIRE(f.lm().getLastClosedLedgerHeader().header.maxTxSetSize == 1); + REQUIRE(txsetMeter(app, "early-discarded") == discarded + 1); + auto const built = f.build(); + REQUIRE(!built.mEarly); + // The selection held two transactions; the ledger takes one. + REQUIRE(built.hashes().size() == 1); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + f.closeAndCheckApplied(built); + } + SECTION("the limits of the closed ledger are not the ones it assumed") + { + // Cannot happen without an upgrade; checked all the same. + { + auto job = Access::job(herder); + std::lock_guard lock(job->mMutex); + job->mSelection->mLimits.mLclHeader.maxTxSetSize += 1; + } + f.close(closing); + auto const built = f.build(); + REQUIRE(!built.mEarly); + REQUIRE(txsetMeter(app, "early-discarded") == discarded + 1); + REQUIRE(built.hashes() == hashesOf({nextA, plainB})); + } + SECTION("the node votes for another value before the ledger closes") + { + // The round's leader timed out; the next one proposes another set. + auto first = Access::job(herder); + auto other = f.setOf({plainB}); + f.vote(other); + // The first selection is abandoned ... + REQUIRE(first->mStop.load()); + REQUIRE(Access::job(herder) != first); + // ... and one for the new value made. + REQUIRE(f.waitSelected()); + SECTION("the ledger closes with the new value") + { + f.close(other); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({closingA})); + f.closeAndCheckApplied(built); + } + SECTION("the ledger closes with the first value after all") + { + f.close(closing); + REQUIRE(txsetMeter(app, "early-discarded") == discarded + 1); + auto const built = f.build(); + REQUIRE(!built.mEarly); + REQUIRE(built.hashes() == hashesOf({nextA, plainB})); + f.closeAndCheckApplied(built); + } + } +} + +TEST_CASE("a selection that is still running when it is needed", + "[herder][txset][early-txset]") +{ + EarlyFixture f(4); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = f.acc(0); + auto& b = f.acc(1); + auto closingA = f.pay(a, 1); + auto nextA = f.pay(a, 2); + auto afterNextA = f.pay(a, 3); + auto plainB = f.pay(b, 1); + auto closing = f.setOf({closingA}); + f.mMempool = {closingA, nextA, plainB}; + + // The job's tasks wait until the test lets them go. + auto hold = std::make_shared>(true); + Access::hold(herder, hold); + f.vote(closing); + auto first = f.waitStarted(); + REQUIRE(!first->mDone.load()); + REQUIRE(Access::jobDuration(herder).first == std::chrono::milliseconds(0)); + f.close(closing); + f.crankFor(std::chrono::milliseconds(20)); + auto const notReady = txsetMeter(app, "early-not-ready"); + auto const built = f.build(); + // The ledger gets a set built the usual way; nobody waits for the job. + REQUIRE(!built.mEarly); + REQUIRE(txsetMeter(app, "early-not-ready") == notReady + 1); + // The job had 20 ms and more and was not done: the node now takes a + // selection to last half as long again, so the next one starts earlier. + REQUIRE(Access::jobDuration(herder).first >= std::chrono::milliseconds(30)); + REQUIRE(Access::jobDuration(herder).second >= + std::chrono::milliseconds(30)); + REQUIRE(built.hashes() == hashesOf({nextA, plainB})); + REQUIRE(first->mStop.load()); + REQUIRE(!Access::job(herder)); + + SECTION("it finishes after the build and hands nothing over") + { + hold->store(false); + f.crankUntil([&]() { return first->mDone.load(); }); + std::lock_guard lock(first->mMutex); + REQUIRE(!first->mSelection); + } + SECTION("it is still running when the next selection starts") + { + // The next ledger is in its ballot, with this node's set. + f.mMempool = {nextA, plainB, afterNextA}; + Access::hold(herder, nullptr); + f.vote(built.mTxSet); + REQUIRE(f.waitSelected()); + auto second = Access::job(herder); + REQUIRE(second != first); + // Only now does the first job get to run. + hold->store(false); + f.crankUntil([&]() { return first->mDone.load(); }); + { + std::lock_guard lock(first->mMutex); + REQUIRE(!first->mSelection); + } + REQUIRE(Access::job(herder) == second); + f.closeAndCheckApplied(built); + auto const next = f.build(); + REQUIRE(next.mEarly); + REQUIRE(next.hashes() == hashesOf({afterNextA})); + f.closeAndCheckApplied(next); + } + hold->store(false); +} + +TEST_CASE("the application shuts down while a selection runs", + "[herder][txset][early-txset]") +{ + auto f = std::make_unique(4); + auto closingA = f->pay(f->acc(0), 1); + f->mMempool = {closingA, f->pay(f->acc(0), 2), f->pay(f->acc(1), 1)}; + auto hold = std::make_shared>(true); + Access::hold(f->herder(), hold); + f->vote(f->setOf({closingA})); + auto job = f->waitStarted(); + REQUIRE(!job->mDone.load()); + // The tasks are let go while the application is going down: they find + // the job stopped, or finish it; either way nothing is left running + // when the application is gone. + std::thread release([hold]() { + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + hold->store(false); + }); + f.reset(); + release.join(); + REQUIRE(job->mDone.load()); +} + +TEST_CASE("candidates whose validity depends on the close time in a set " + "selected early", + "[herder][txset][early-txset]") +{ + EarlyFixture f(4); + auto& app = *f.mApp; + auto& a = f.acc(0); + auto& b = f.acc(1); + auto& c = f.acc(2); + auto closingA = f.pay(a, 1); + auto closing = f.setOf({closingA}); + auto const lclCloseTime = + f.lm().getLastClosedLedgerHeader().header.scpValue.closeTime; + auto timeBounded = [&](TestAccount& account, TimePoint maxTime) { + PreconditionsV2 cond; + cond.timeBounds.activate().minTime = 0; + cond.timeBounds.activate().maxTime = maxTime; + return TransactionFrameBasePtr(transactionFromOperationsV1( + app, account.getSecretKey(), f.seqOf(account) + 1, + {payment(app.getRoot()->getPublicKey(), 1)}, 100, cond)); + }; + // The closing ledger gets close time +1 and the next +2 (seconds after + // the LCL's). B's transaction expires before the next ledger, C's lasts. + auto expiring = timeBounded(b, lclCloseTime + 1); + auto lasting = timeBounded(c, lclCloseTime + 1000); + REQUIRE(txValidityDependsOnCloseTime(expiring->getEnvelope())); + f.mMempool = {closingA, expiring, lasting}; + f.voteAndSelect(closing); + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + // Whatever close time the selection assumed, the completed set is + // validated for the one it gets. + REQUIRE(built.hashes() == hashesOf({lasting})); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + f.closeAndCheckApplied(built); +} + +// --------------------------------------------------------------------------- +// Property b: full sets. +// --------------------------------------------------------------------------- + +TEST_CASE("a set selected early is topped up when the ledger has closed", + "[herder][txset][early-txset]") +{ + EarlyFixture f(10, 6); + auto& app = *f.mApp; + auto& herder = f.herder(); + REQUIRE(f.lm().getLastMaxTxSetSizeOps() == 6); + auto closingA = f.pay(f.acc(0), 1); + auto closing = f.setOf({closingA}); + std::vector early{f.pay(f.acc(1), 1), + f.pay(f.acc(2), 1)}; + f.mMempool = {closingA, early[0], early[1]}; + f.voteAndSelect(closing); + auto const added = txsetMeter(app, "early-added", "tx"); + + SECTION("with what arrived after the selection's pull") + { + std::vector late{ + f.pay(f.acc(3), 1), f.pay(f.acc(4), 1), f.pay(f.acc(0), 2)}; + f.mMempool.insert(f.mMempool.end(), late.begin(), late.end()); + f.close(closing); + f.mPulls.clear(); + auto const built = f.build(); + REQUIRE(built.mEarly); + auto all = early; + all.insert(all.end(), late.begin(), late.end()); + REQUIRE(built.hashes() == hashesOf(all)); + REQUIRE(txsetMeter(app, "early-added", "tx") == added + 3); + REQUIRE(f.mPulls == std::vector{Access::pullCount(herder)}); + f.closeAndCheckApplied(built); + } + SECTION("up to its capacity, and no further") + { + std::vector late; + for (size_t i = 3; i < 10; ++i) + { + late.push_back(f.pay(f.acc(i), 1)); + } + f.mMempool.insert(f.mMempool.end(), late.begin(), late.end()); + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes().size() == 6); + REQUIRE(built.holds(early[0])); + REQUIRE(built.holds(early[1])); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + f.closeAndCheckApplied(built); + } + SECTION("a new transaction of an account that is in the set already " + "waits") + { + auto second = f.pay(f.acc(1), 2); + auto plain = f.pay(f.acc(5), 1); + f.mMempool.push_back(second); + f.mMempool.push_back(plain); + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({early[0], early[1], plain})); + f.closeAndCheckApplied(built); + } + SECTION("a full phase asks for nothing more") + { + // Another selection, with more candidates than the set takes. + f.close(closing); + f.closeAndCheckApplied(f.build()); + auto closingB = f.pay(f.acc(1), 1); + auto closing2 = f.setOf({closingB}); + f.mMempool = {closingB}; + for (size_t i = 2; i < 10; ++i) + { + f.mMempool.push_back(f.pay(f.acc(i), 1)); + } + f.voteAndSelect(closing2); + f.close(closing2); + f.mPulls.clear(); + auto const added = txsetMeter(app, "early-added", "tx"); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes().size() == 6); + REQUIRE(txsetMeter(app, "early-added", "tx") == added); + // One look at the mempool's top, for the Soroban phase, which has + // room and never had a candidate: nothing there for it, so no + // deeper pull. + REQUIRE(f.mPulls == std::vector{Access::pullCount(herder)}); + f.closeAndCheckApplied(built); + } +} + +TEST_CASE("a set selected early is refilled when transactions of it turn " + "out invalid", + "[herder][txset][early-txset]") +{ + // A set takes 10 transactions; the mempool holds 40 candidates beside + // the closing ledger's. The closing ledger drains the accounts of some + // of them, which only the closed ledger shows. + size_t const CAPACITY = 10; + size_t const CANDIDATES = 40; + EarlyFixture f(CANDIDATES + 1, CAPACITY); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto const drainedPercent = GENERATE(0, 10, 30, 60, 100); + CAPTURE(drainedPercent); + + // Candidate i comes from account i; the closing set drains the first + // `drained` of those accounts. + size_t const drained = CANDIDATES * drainedPercent / 100; + std::vector previous; + std::vector candidates; + for (size_t i = 0; i < CANDIDATES; ++i) + { + auto& account = f.acc(i); + if (i < drained) + { + previous.push_back( + f.pay(account, 1, account.getAvailableBalance() - 100)); + candidates.push_back(f.pay(account, 2)); + } + else + { + candidates.push_back(f.pay(account, 1)); + } + } + if (previous.empty()) + { + previous.push_back(f.pay(f.acc(CANDIDATES), 1)); + } + f.mMempool = previous; + f.mMempool.insert(f.mMempool.end(), candidates.begin(), candidates.end()); + auto closing = f.setOf(previous); + f.voteAndSelect(closing); + f.close(closing); + + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + auto const hashes = built.hashes(); + for (size_t i = 0; i < drained; ++i) + { + REQUIRE(hashes.count(candidates[i]->getFullHash()) == 0); + } + // Full whenever enough valid candidates exist. + size_t const validCandidates = CANDIDATES - drained; + REQUIRE(hashes.size() == std::min(CAPACITY, validCandidates)); + auto& fill = + app.getMetrics().NewHistogram({"herder", "txset", "fill-classic"}); + REQUIRE(fill.count() >= 1); + f.closeAndCheckApplied(built); +} + +TEST_CASE("a build pulls the capacity of a set plus a margin and refills", + "[herder][txset][early-txset]") +{ + size_t const CAPACITY = 20; + auto const marginPercent = GENERATE(0u, 10u, 50u); + CAPTURE(marginPercent); + size_t const ACCOUNTS = 120; + EarlyFixture f(ACCOUNTS, CAPACITY, [&](Config& cfg) { + cfg.EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT = marginPercent; + // Builds only. + cfg.EXPERIMENTAL_EARLY_TX_SET_SELECTION = false; + }); + auto& app = *f.mApp; + auto& herder = f.herder(); + // The capacity of both phases together, and the margin on top. + size_t const total = + CAPACITY + + f.lm().getLastClosedSorobanNetworkConfig().ledgerMaxTxCount(); + size_t const pull = total + (total * marginPercent + 99) / 100; + REQUIRE(Access::pullCount(herder) == pull); + + SECTION("one pull when the mempool holds less than it asks for") + { + for (size_t i = 0; i < 30; ++i) + { + f.mMempool.push_back(f.pay(f.acc(i), 1)); + } + auto const built = f.build(); + REQUIRE(f.mPulls == std::vector{pull}); + REQUIRE(built.hashes().size() == CAPACITY); + } + SECTION("candidates lost to the one-per-account rule are refilled too") + { + // The top of the mempool is one account's transactions with one + // sequence number, each valid by itself; a set takes one of them. + for (size_t i = 0; i < pull; ++i) + { + f.mMempool.push_back( + f.pay(f.acc(0), 1, static_cast(i + 1))); + } + for (size_t i = 1; i < 60; ++i) + { + f.mMempool.push_back(f.pay(f.acc(i), 1)); + } + auto const built = f.build(); + REQUIRE(f.mPulls.size() == 2); + REQUIRE(built.hashes().size() == CAPACITY); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + } + SECTION("X% of the candidates are invalid") + { + // The mempool holds more than three pulls can reach; every k-th + // candidate is invalid (a sequence number of the past). + auto const invalidPercent = GENERATE(0, 5, 20, 50, 90); + CAPTURE(invalidPercent); + auto const refills = txsetMeter(app, "refill-pull", "pull"); + std::mt19937 gen(42); + size_t valid = 0; + std::vector spent; + // One spent sequence number per account to make invalid ones from. + for (size_t i = 0; i < ACCOUNTS; ++i) + { + bool const bad = stellar::uniform_int_distribution(0, 99)( + gen) < invalidPercent; + if (bad) + { + f.mMempool.push_back(f.pay(f.acc(i), 0)); + } + else + { + f.mMempool.push_back(f.pay(f.acc(i), 1)); + ++valid; + } + } + auto const built = f.build(); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + // Count the valid candidates within reach of the pulls made. + size_t const reached = std::min(f.mPulls.back(), ACCOUNTS); + size_t validReached = 0; + for (size_t i = 0; i < reached; ++i) + { + if (f.mMempool[i]->getSeqNum() == f.seqOf(f.acc(i)) + 1) + { + ++validReached; + } + } + REQUIRE(built.hashes().size() == + std::min(CAPACITY, validReached)); + // Each pull asks for twice the one before. + for (size_t i = 0; i < f.mPulls.size(); ++i) + { + REQUIRE(f.mPulls[i] == pull << i); + } + REQUIRE(f.mPulls.size() <= 3); + REQUIRE(txsetMeter(app, "refill-pull", "pull") == + refills + static_cast(f.mPulls.size()) - 1); + // The set is full whenever the mempool holds enough valid + // transactions within three pulls. + if (valid >= CAPACITY && ACCOUNTS <= (pull << 2)) + { + REQUIRE(built.hashes().size() == CAPACITY); + } + } +} + +TEST_CASE("a phase that lost candidates is refilled from below a mempool top " + "of the other phase", + "[herder][txset][early-txset]") +{ + // The mempool has one fee order for both phases. A phase that lost + // candidates looks deeper also when the top holds little of its kind: + // what it needs may lie below the other phase's transactions. + size_t const CAPACITY = 10; + size_t const SOROBAN_CAPACITY = 10; + SECTION("a completion") + { + EarlyFixture f(60, CAPACITY); + auto& app = *f.mApp; + auto& herder = f.herder(); + overrideSorobanNetworkConfigForTest(app); + modifySorobanNetworkConfig(app, [&](SorobanNetworkConfig& cfg) { + cfg.mLedgerMaxTxCount = SOROBAN_CAPACITY; + }); + size_t const pull = Access::pullCount(herder); + REQUIRE(pull >= CAPACITY + SOROBAN_CAPACITY); + // The closing set drains accounts 0 and 1. The selection holds + // their next payments and two more: four classic candidates, all + // there are. + std::vector previous; + std::vector candidates; + for (size_t i = 0; i < 2; ++i) + { + auto& account = f.acc(i); + previous.push_back( + f.pay(account, 1, account.getAvailableBalance() - 100)); + candidates.push_back(f.pay(account, 2)); + } + candidates.push_back(f.pay(f.acc(2), 1)); + candidates.push_back(f.pay(f.acc(3), 1)); + f.mMempool = previous; + f.mMempool.insert(f.mMempool.end(), candidates.begin(), + candidates.end()); + auto closing = f.setOf(previous); + f.voteAndSelect(closing); + f.close(closing); + // During the ballot: more Soroban transactions than a pull holds, + // and below them classic ones. + for (size_t i = 0; i < pull + 4; ++i) + { + f.mMempool.push_back(f.upload(f.acc(4 + i), 1, 100 + i)); + } + std::vector below; + for (size_t i = 0; i < 8; ++i) + { + below.push_back(f.pay(f.acc(4 + pull + 4 + i), 1)); + } + f.mMempool.insert(f.mMempool.end(), below.begin(), below.end()); + auto const refills = txsetMeter(app, "refill-pull", "pull"); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + REQUIRE(!built.holds(candidates[0])); + REQUIRE(!built.holds(candidates[1])); + REQUIRE(built.holds(candidates[2])); + REQUIRE(built.holds(candidates[3])); + // The two it lost are made up for from below. + REQUIRE(txsetMeter(app, "refill-pull", "pull") > refills); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::CLASSIC) > 2); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::SOROBAN) == + SOROBAN_CAPACITY); + f.closeAndCheckApplied(built); + } + SECTION("a build") + { + EarlyFixture f(120, CAPACITY, [&](Config& cfg) { + cfg.EXPERIMENTAL_EARLY_TX_SET_SELECTION = false; + }); + auto& app = *f.mApp; + auto& herder = f.herder(); + overrideSorobanNetworkConfigForTest(app); + modifySorobanNetworkConfig(app, [&](SorobanNetworkConfig& cfg) { + cfg.mLedgerMaxTxCount = SOROBAN_CAPACITY; + }); + size_t const pull = Access::pullCount(herder); + // The top: six classic candidates, four of them with a spent + // sequence number, then Soroban transactions to the end of the + // pull. Below: valid classic ones. + for (size_t i = 0; i < 6; ++i) + { + f.mMempool.push_back(f.pay(f.acc(i), i < 4 ? 0 : 1)); + } + for (size_t i = 6; i < pull; ++i) + { + f.mMempool.push_back(f.upload(f.acc(i), 1, 100 + i)); + } + for (size_t i = 0; i < 12; ++i) + { + f.mMempool.push_back(f.pay(f.acc(pull + i), 1)); + } + auto const built = f.build(); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + REQUIRE(f.mPulls.size() == 2); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::CLASSIC) == CAPACITY); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::SOROBAN) == + SOROBAN_CAPACITY); + } +} + +TEST_CASE("what the top-up may not add to a set selected early", + "[herder][txset][early-txset]") +{ + EarlyFixture f(8); + auto& app = *f.mApp; + auto& a = f.acc(0); + auto& b = f.acc(1); + auto& c = f.acc(2); + auto& d = f.acc(3); + auto closingA = f.pay(a, 1); + auto closing = f.setOf({closingA}); + + SECTION("a transaction its fee source cannot pay for on top of the " + "selected ones") + { + // D pays the fees of other accounts' transactions. The closing + // ledger leaves it with what two fee bumps cost (the selection + // cannot know: it predicts fees, not payments). One bump by D is + // selected; two more arrive later. D can pay for one of them. + auto bumpB = feeBump(app, d, f.pay(b, 1), 200); + auto const fees = bumpB->getFullFee(); + auto drainD = f.pay(d, 1, d.getAvailableBalance() - 100 - 2 * fees); + auto closing2 = f.setOf({closingA, drainD}); + auto bumpLate1 = feeBump(app, d, f.pay(c, 1), 200); + auto bumpLate2 = feeBump(app, d, f.pay(f.acc(4), 1), 200); + auto plain = f.pay(f.acc(5), 1); + f.mMempool = {closingA, drainD, bumpB}; + f.voteAndSelect(closing2); + f.close(closing2); + REQUIRE(d.getAvailableBalance() == 2 * fees); + f.mMempool.push_back(bumpLate1); + f.mMempool.push_back(bumpLate2); + f.mMempool.push_back(plain); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.holds(bumpB)); + REQUIRE(built.holds(plain)); + // The selected bump counts against D's balance when the late ones + // are validated: one more fits, the next does not. + REQUIRE(built.holds(bumpLate1)); + REQUIRE(!built.holds(bumpLate2)); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + f.closeAndCheckApplied(built); + } + SECTION("a second transaction of an account that has one in the set") + { + // Another transaction of B with the sequence number of the selected + // one: each is valid on the closed ledger, together they are not a + // set. + auto selectedB = f.pay(b, 1, 1); + auto otherB = f.pay(b, 1, 2); + auto plain = f.pay(c, 1); + f.mMempool = {closingA, selectedB}; + f.voteAndSelect(closing); + f.close(closing); + f.mMempool.push_back(otherB); + f.mMempool.push_back(plain); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({selectedB, plain})); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + REQUIRE(!Access::selfCheckLatched(f.herder())); + f.closeAndCheckApplied(built); + } +} + +TEST_CASE("the base fee of a set selected early follows what was left out", + "[herder][txset][early-txset]") +{ + // A set takes 4 transactions. Fees: 500, 400, 300, 200 selected; 150 + // left over. + EarlyFixture f(8, 4); + auto& app = *f.mApp; + auto closingA = f.pay(f.acc(0), 1); + auto closing = f.setOf({closingA}); + auto t500 = f.pay(f.acc(1), 1, 1, 500); + auto t400 = f.pay(f.acc(2), 1, 1, 400); + auto t300 = f.pay(f.acc(3), 1, 1, 300); + auto t150 = f.pay(f.acc(5), 1, 1, 150); + auto baseFeeOf = [&](Built const& built, + TransactionFrameBasePtr const& tx) { + for (auto const& candidate : + built.mApplicable->getPhase(TxSetPhase::CLASSIC)) + { + if (candidate->getFullHash() == tx->getFullHash()) + { + auto fee = built.mApplicable->getTxBaseFee(candidate); + REQUIRE(fee); + return *fee; + } + } + REQUIRE(false); + return int64_t(0); + }; + auto const ledgerBaseFee = + f.lm().getLastClosedLedgerHeader().header.baseFee; + + SECTION("untouched: the lowest fee selected, because one was left out") + { + auto t200 = f.pay(f.acc(4), 1, 1, 200); + f.mMempool = {closingA, t500, t400, t300, t200, t150}; + f.voteAndSelect(closing); + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({t500, t400, t300, t200})); + REQUIRE(baseFeeOf(built, t500) == 200); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + } + SECTION("one removed and replaced by the left-over: its fee, nothing " + "is left out any more") + { + // The account of the 200 transaction is drained by the closing + // ledger. + auto& drained = f.acc(4); + auto drain = f.pay(drained, 1, drained.getAvailableBalance() - 100); + auto t200 = f.pay(drained, 2, 1, 200); + auto closing2 = f.setOf({drain}); + f.mMempool = {drain, t500, t400, t300, t200, t150}; + f.voteAndSelect(closing2); + f.close(closing2); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({t500, t400, t300, t150})); + // Everything valid is in: nobody competed for the room. + REQUIRE(baseFeeOf(built, t500) == ledgerBaseFee); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + // The same as a build from scratch. + auto const fresh = f.buildFrom({t500, t400, t300, t200, t150}); + REQUIRE(fresh.hashes() == built.hashes()); + REQUIRE(baseFeeOf(fresh, t500) == ledgerBaseFee); + f.closeAndCheckApplied(built); + } + SECTION("more arrive than fit: the lowest fee that got in") + { + f.mMempool = {closingA, t500, t400}; + f.voteAndSelect(closing); + f.close(closing); + auto t350 = f.pay(f.acc(4), 1, 1, 350); + auto t250 = f.pay(f.acc(6), 1, 1, 250); + f.mMempool = {t500, t400, t350, t300, t250, t150}; + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({t500, t400, t350, t300})); + REQUIRE(baseFeeOf(built, t500) == 300); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + f.closeAndCheckApplied(built); + } +} + +// --------------------------------------------------------------------------- +// Soroban: the parallel phase is extended in place. +// --------------------------------------------------------------------------- + +TEST_CASE("a parallel Soroban phase is extended without selecting it again", + "[herder][txset][early-txset][soroban]") +{ + EarlyFixture f(12); + auto& app = *f.mApp; + // Limits under which the transactions of this test are valid. + overrideSorobanNetworkConfigForTest(app); + auto const lcl = f.lm().getLastClosedLedgerHeader(); + auto const version = lcl.header.ledgerVersion; + SorobanNetworkConfig config = f.lm().getLastClosedSorobanNetworkConfig(); + // No limit but the ones each section sets. + auto laneLimit = config.maxLedgerResources(); + for (auto type : + {Resource::Type::OPERATIONS, Resource::Type::INSTRUCTIONS, + Resource::Type::TX_BYTE_SIZE, Resource::Type::DISK_READ_BYTES, + Resource::Type::WRITE_BYTES, Resource::Type::READ_LEDGER_ENTRIES, + Resource::Type::WRITE_LEDGER_ENTRIES}) + { + laneLimit.setVal(type, std::numeric_limits::max() / 4); + } + + // The stages an extender made, as a set: validated like any set. + auto validity = [&](TxStageFrameList const& stages, + SorobanNetworkConfig const& cfg) { + auto set = testtxset::makeNonValidatedGeneralizedTxSet( + {}, std::make_optional(lcl.header.baseFee), stages, app, + lcl.hash); + REQUIRE(set.second); + TxSetValidationContext ctx = TxSetValidationContext::forLCL(app); + ctx.mSorobanConfig = &cfg; + return set.second->checkValidWithResult(app, ctx, 0, 0); + }; + auto count = [](TxStageFrameList const& stages) { + size_t n = 0; + for (auto const& stage : stages) + { + for (auto const& cluster : stage) + { + n += cluster.size(); + } + } + return n; + }; + + SECTION("independent transactions spread over the clusters") + { + config.mLedgerMaxDependentTxClusters = 4; + config.mLedgerMaxInstructions = 3'000'000; + ParallelPhaseExtender extender({}, config, laneLimit, version); + // 4 clusters of 3,000,000 instructions take 12 transactions of + // 1,000,000. + for (size_t i = 0; i < 12; ++i) + { + REQUIRE(extender.tryAdd(f.upload(f.acc(i), 1, 100 + i))); + } + REQUIRE(extender.sizeTx() == 12); + REQUIRE(extender.getStages().size() == 1); + REQUIRE(extender.getStages()[0].size() == 4); + for (auto const& cluster : extender.getStages()[0]) + { + REQUIRE(cluster.size() == 3); + } + REQUIRE(validity(extender.getStages(), config) == + TxSetValidationResult::VALID); + // The thirteenth does not fit the instruction limit. + REQUIRE(!extender.tryAdd(f.upload(f.acc(0), 2, 999))); + REQUIRE(extender.sizeTx() == 12); + } + SECTION("a transaction joins the cluster it conflicts with") + { + config.mLedgerMaxDependentTxClusters = 4; + config.mLedgerMaxInstructions = 10'000'000; + ParallelPhaseExtender extender({}, config, laneLimit, version); + // Same seed: the same entry written. + REQUIRE(extender.tryAdd(f.upload(f.acc(0), 1, 7))); + REQUIRE(extender.tryAdd(f.upload(f.acc(1), 1, 8))); + REQUIRE(extender.tryAdd(f.upload(f.acc(2), 1, 7))); + auto const& stage = extender.getStages().at(0); + REQUIRE(stage.size() == 2); + REQUIRE(stage[0].size() == 2); + REQUIRE(stage[1].size() == 1); + REQUIRE(validity(extender.getStages(), config) == + TxSetValidationResult::VALID); + } + SECTION("a transaction that would join two clusters is left out") + { + config.mLedgerMaxDependentTxClusters = 4; + config.mLedgerMaxInstructions = 10'000'000; + auto first = f.upload(f.acc(0), 1, 7); + auto second = f.upload(f.acc(1), 1, 8); + // Writes what the two others write. + SorobanResources resources; + resources.instructions = 1'000'000; + resources.diskReadBytes = 1000; + resources.writeBytes = 4000; + resources.footprint.readWrite = + first->sorobanResources().footprint.readWrite; + resources.footprint.readWrite.push_back( + second->sorobanResources().footprint.readWrite.at(0)); + auto both = createUploadWasmTx(app, f.acc(2), 100, + DEFAULT_TEST_RESOURCE_FEE, resources); + ParallelPhaseExtender extender({}, config, laneLimit, version); + REQUIRE(extender.tryAdd(first)); + REQUIRE(extender.tryAdd(second)); + REQUIRE(!extender.tryAdd(both)); + REQUIRE(extender.sizeTx() == 2); + // Had it been added to either cluster, the phase would not be + // valid. + auto broken = extender.getStages(); + broken[0][0].push_back(both); + REQUIRE(validity(broken, config) == + TxSetValidationResult::TX_ORDERING_INVALID); + } + SECTION("a reader conflicts with the writer, not with other readers") + { + config.mLedgerMaxDependentTxClusters = 4; + config.mLedgerMaxInstructions = 10'000'000; + auto writer = f.upload(f.acc(0), 1, 7); + auto const key = writer->sorobanResources().footprint.readWrite.at(0); + auto reader = [&](TestAccount& account, uint64_t seed) { + SorobanResources resources; + resources.instructions = 1'000'000; + resources.diskReadBytes = 1000; + resources.writeBytes = 2000; + auto own = f.upload(account, 1, seed); + resources.footprint.readWrite = + own->sorobanResources().footprint.readWrite; + resources.footprint.readOnly = {key}; + return createUploadWasmTx(app, account, 100, + DEFAULT_TEST_RESOURCE_FEE, resources, + std::nullopt, 0, std::nullopt, + std::nullopt, std::make_optional(seed)); + }; + SECTION("readers first") + { + ParallelPhaseExtender extender({}, config, laneLimit, version); + REQUIRE(extender.tryAdd(reader(f.acc(1), 11))); + REQUIRE(extender.tryAdd(reader(f.acc(2), 12))); + // Two readers, two clusters: the writer would join both. + REQUIRE(extender.getStages().at(0).size() == 2); + REQUIRE(!extender.tryAdd(writer)); + REQUIRE(validity(extender.getStages(), config) == + TxSetValidationResult::VALID); + } + SECTION("the writer first") + { + ParallelPhaseExtender extender({}, config, laneLimit, version); + REQUIRE(extender.tryAdd(writer)); + REQUIRE(extender.tryAdd(reader(f.acc(1), 11))); + REQUIRE(extender.tryAdd(reader(f.acc(2), 12))); + // Both readers are with the writer. + REQUIRE(extender.getStages().at(0).size() == 1); + REQUIRE(extender.getStages()[0][0].size() == 3); + REQUIRE(validity(extender.getStages(), config) == + TxSetValidationResult::VALID); + } + } + SECTION("the lane's limits hold") + { + config.mLedgerMaxDependentTxClusters = 4; + config.mLedgerMaxInstructions = 100'000'000; + auto limit = laneLimit; + // Three transactions. + limit.setVal(Resource::Type::OPERATIONS, 3); + ParallelPhaseExtender extender({}, config, limit, version); + REQUIRE(extender.tryAdd(f.upload(f.acc(0), 1, 1))); + REQUIRE(extender.tryAdd(f.upload(f.acc(1), 1, 2))); + REQUIRE(extender.tryAdd(f.upload(f.acc(2), 1, 3))); + REQUIRE(!extender.tryAdd(f.upload(f.acc(3), 1, 4))); + REQUIRE(count(extender.getStages()) == 3); + } + SECTION("a transaction joins the cluster of an existing phase it " + "conflicts with") + { + config.mLedgerMaxDependentTxClusters = 4; + config.mLedgerMaxInstructions = 10'000'000; + // One stage, two clusters, as a selection made them. + TxStageFrameList stages{ + {{f.upload(f.acc(0), 1, 1)}, {f.upload(f.acc(1), 1, 2)}}}; + ParallelPhaseExtender extender(stages, config, laneLimit, version); + // Writes what the second cluster writes. + REQUIRE(extender.tryAdd(f.upload(f.acc(2), 1, 2))); + REQUIRE(extender.getStages().at(0).size() == 2); + REQUIRE(extender.getStages()[0][0].size() == 1); + REQUIRE(extender.getStages()[0][1].size() == 2); + REQUIRE(validity(extender.getStages(), config) == + TxSetValidationResult::VALID); + // In the other cluster the phase would not be valid. + auto broken = stages; + broken[0][0].push_back(f.upload(f.acc(2), 1, 2)); + REQUIRE(validity(broken, config) == + TxSetValidationResult::TX_ORDERING_INVALID); + } + SECTION("stages that exist are extended, and stay within the limit") + { + config.mLedgerMaxDependentTxClusters = 2; + config.mLedgerMaxInstructions = 4'000'000; + // Two stages of one cluster each: 2,000,000 + 1,000,000 sequential + // instructions of 4,000,000. + TxStageFrameList stages{ + {{f.upload(f.acc(0), 1, 1), f.upload(f.acc(1), 1, 2)}}, + {{f.upload(f.acc(2), 1, 3)}}}; + ParallelPhaseExtender extender(stages, config, laneLimit, version); + REQUIRE(extender.sizeTx() == 3); + // A new cluster beside the first stage's: no more sequential + // instructions. + REQUIRE(extender.tryAdd(f.upload(f.acc(3), 1, 4))); + REQUIRE(extender.tryAdd(f.upload(f.acc(4), 1, 5))); + REQUIRE(extender.getStages()[0].size() == 2); + // The first stage is at 2,000,000 in both clusters: the next goes + // where a stage can still grow. + size_t added = 0; + for (size_t i = 5; i < 12; ++i) + { + if (extender.tryAdd(f.upload(f.acc(i), 1, 10 + i))) + { + ++added; + } + } + REQUIRE(validity(extender.getStages(), config) == + TxSetValidationResult::VALID); + // 4,000,000 sequential instructions in two clusters per stage: + // 8 transactions at most. + REQUIRE(extender.sizeTx() == 8); + REQUIRE(added == 3); + } +} + +TEST_CASE("a set selected early holds Soroban transactions and is topped up " + "with them", + "[herder][txset][early-txset][soroban]") +{ + EarlyFixture f(14); + auto& app = *f.mApp; + auto& herder = f.herder(); + overrideSorobanNetworkConfigForTest(app); + modifySorobanNetworkConfig( + app, [](SorobanNetworkConfig& cfg) { cfg.mLedgerMaxTxCount = 8; }); + REQUIRE(f.lm().getLastClosedSorobanNetworkConfig().ledgerMaxTxCount() == 8); + auto closingA = f.pay(f.acc(0), 1); + auto closing = f.setOf({closingA}); + std::vector early{f.upload(f.acc(1), 1, 1), + f.upload(f.acc(2), 1, 2), + f.upload(f.acc(3), 1, 3)}; + auto classic = f.pay(f.acc(4), 1); + f.mMempool = {closingA, early[0], classic, early[1], early[2]}; + f.voteAndSelect(closing); + + SECTION("candidates that arrive later are added to the phase") + { + std::vector late{f.upload(f.acc(5), 1, 5), + f.upload(f.acc(6), 1, 6)}; + f.mMempool.insert(f.mMempool.end(), late.begin(), late.end()); + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + auto all = early; + all.insert(all.end(), late.begin(), late.end()); + all.push_back(classic); + REQUIRE(built.hashes() == hashesOf(all)); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::SOROBAN) == 5); + f.closeAndCheckApplied(built); + } + SECTION("one that conflicts with a selected one joins its cluster") + { + // Writes the entry the first selected transaction writes. + auto conflicting = f.upload(f.acc(5), 1, 1); + f.mMempool.push_back(conflicting); + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.holds(conflicting)); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::SOROBAN) == 4); + REQUIRE(!Access::selfCheckLatched(herder)); + f.closeAndCheckApplied(built); + } + SECTION("up to the phase's capacity") + { + for (size_t i = 5; i < 14; ++i) + { + f.mMempool.push_back(f.upload(f.acc(i), 1, 10 + i)); + } + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::SOROBAN) == 8); + REQUIRE(built.holds(classic)); + f.closeAndCheckApplied(built); + } + SECTION("one that the closed ledger made invalid is replaced") + { + // Account 1 is drained by the closing ledger... which is not what + // closes: another set does, with the same account's payment. + f.close(closing); + auto const built = f.build(); + REQUIRE(built.mEarly); + f.closeAndCheckApplied(built); + // Next round: the closing set drains account 5, whose Soroban + // transaction the selection picks. + auto& drainedAccount = f.acc(5); + auto drain = f.pay(drainedAccount, 1, + drainedAccount.getAvailableBalance() - 100); + auto closing2 = f.setOf({drain}); + auto doomed = f.upload(drainedAccount, 2, 50); + std::vector others; + for (size_t i = 6; i < 14; ++i) + { + others.push_back(f.upload(f.acc(i), 1, 60 + i)); + } + // The selection is full with `doomed` and seven others; one more + // is left over. + f.mMempool = {drain, doomed}; + f.mMempool.insert(f.mMempool.end(), others.begin(), others.end()); + f.voteAndSelect(closing2); + { + auto job = Access::job(herder); + std::lock_guard lock(job->mMutex); + REQUIRE(job->mSelection->mSelected == 8); + REQUIRE(job->mSelection->mLeftovers.size() == 1); + } + f.close(closing2); + f.mPulls.clear(); + auto const next = f.build(); + REQUIRE(next.mEarly); + REQUIRE(!next.holds(doomed)); + // Full again, from what the selection had left over. The classic + // phase, which never had a candidate, looks at the mempool's top + // once. + REQUIRE(next.mApplicable->sizeTx(TxSetPhase::SOROBAN) == 8); + REQUIRE(next.hashes() == hashesOf(others)); + REQUIRE(f.mPulls == std::vector{Access::pullCount(herder)}); + f.closeAndCheckApplied(next); + } +} + +TEST_CASE("a phase the selection could not fill is filled at the close", + "[herder][txset][early-txset][soroban]") +{ + EarlyFixture f(14); + auto& app = *f.mApp; + auto& herder = f.herder(); + overrideSorobanNetworkConfigForTest(app); + auto closingA = f.pay(f.acc(0), 1); + auto closing = f.setOf({closingA}); + + SECTION("a phase without a single candidate when the selection was made") + { + modifySorobanNetworkConfig( + app, [](SorobanNetworkConfig& cfg) { cfg.mLedgerMaxTxCount = 8; }); + closingA = f.pay(f.acc(0), 1); + closing = f.setOf({closingA}); + // Classic candidates only. + auto classic = f.pay(f.acc(1), 1); + f.mMempool = {closingA, classic}; + f.voteAndSelect(closing); + f.close(closing); + // Soroban transactions arrive during the ballot. + std::vector late{f.upload(f.acc(2), 1, 2), + f.upload(f.acc(3), 1, 3)}; + f.mMempool.insert(f.mMempool.end(), late.begin(), late.end()); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.hashes() == hashesOf({classic, late[0], late[1]})); + f.closeAndCheckApplied(built); + } + SECTION("a selection that left out a transaction for its cluster's " + "limit") + { + // Two clusters of 10,000,000 instructions. Three transactions of + // 5,000,000 on one entry: two fill a cluster, the third has + // nowhere to go, while the other cluster is empty. + modifySorobanNetworkConfig(app, [](SorobanNetworkConfig& cfg) { + cfg.mLedgerMaxTxCount = 8; + cfg.mLedgerMaxDependentTxClusters = 2; + cfg.mTxMaxInstructions = 10'000'000; + cfg.mLedgerMaxInstructions = 10'000'000; + }); + closingA = f.pay(f.acc(0), 1); + closing = f.setOf({closingA}); + std::vector hot{ + f.upload(f.acc(1), 1, 7, 100, 5'000'000), + f.upload(f.acc(2), 1, 7, 100, 5'000'000), + f.upload(f.acc(3), 1, 7, 100, 5'000'000)}; + f.mMempool = {closingA, hot[0], hot[1], hot[2]}; + f.voteAndSelect(closing); + { + auto job = Access::job(herder); + std::lock_guard lock(job->mMutex); + REQUIRE(job->mSelection->mSelected == 2); + REQUIRE(job->mSelection->mHadTxNotFittingLane.at(1).at(0)); + } + f.close(closing); + // Two that write other entries arrive: they have room. + std::vector late{ + f.upload(f.acc(4), 1, 8, 100, 5'000'000), + f.upload(f.acc(5), 1, 9, 100, 5'000'000)}; + f.mMempool.insert(f.mMempool.end(), late.begin(), late.end()); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.holds(late[0])); + REQUIRE(built.holds(late[1])); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::SOROBAN) == 4); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + f.closeAndCheckApplied(built); + } + SECTION("many left-overs with nowhere to go do not close the phase") + { + // As above, with ten transactions on the one entry: eight valid + // left-overs that fit nowhere. The phase is not full for that: two + // that write other entries still have the second cluster. + modifySorobanNetworkConfig(app, [](SorobanNetworkConfig& cfg) { + cfg.mLedgerMaxTxCount = 16; + cfg.mLedgerMaxDependentTxClusters = 2; + cfg.mTxMaxInstructions = 10'000'000; + cfg.mLedgerMaxInstructions = 10'000'000; + }); + closingA = f.pay(f.acc(0), 1); + closing = f.setOf({closingA}); + f.mMempool = {closingA}; + for (size_t i = 1; i <= 10; ++i) + { + f.mMempool.push_back(f.upload(f.acc(i), 1, 7, 100, 5'000'000)); + } + f.voteAndSelect(closing); + { + auto job = Access::job(herder); + std::lock_guard lock(job->mMutex); + REQUIRE(job->mSelection->mSelected == 2); + } + f.close(closing); + std::vector late{ + f.upload(f.acc(11), 1, 8, 100, 5'000'000), + f.upload(f.acc(12), 1, 9, 100, 5'000'000)}; + f.mMempool.insert(f.mMempool.end(), late.begin(), late.end()); + auto const built = f.build(); + REQUIRE(built.mEarly); + REQUIRE(built.holds(late[0])); + REQUIRE(built.holds(late[1])); + REQUIRE(built.mApplicable->sizeTx(TxSetPhase::SOROBAN) == 4); + REQUIRE(f.validate(built) == TxSetValidationResult::VALID); + f.closeAndCheckApplied(built); + } +} + +// --------------------------------------------------------------------------- +// The prediction. +// --------------------------------------------------------------------------- + +TEST_CASE("what the transaction set of a ledger predicts of the ledger", + "[herder][txset][early-txset]") +{ + EarlyFixture f(4); + auto& app = *f.mApp; + auto& a = f.acc(0); + auto& b = f.acc(1); + auto& c = f.acc(2); + auto payA = f.pay(a, 1, 5, 300); + auto bump = feeBump(app, c, f.pay(b, 1), 500); + PredictedAccountChanges changes; + predictAccountChanges(*payA, changes); + predictAccountChanges(*bump, changes); + REQUIRE(changes.size() == 3); + // A: its sequence number and its fee. + REQUIRE(changes[a.getPublicKey()].mSeqNum == payA->getSeqNum()); + REQUIRE(changes[a.getPublicKey()].mFees == 300); + // B: the sequence number of the inner transaction, no fee. + REQUIRE(changes[b.getPublicKey()].mSeqNum == f.seqOf(b) + 1); + REQUIRE(changes[b.getPublicKey()].mFees == 0); + // C: the fee bump's fee, no sequence number. + REQUIRE(changes[c.getPublicKey()].mSeqNum == 0); + REQUIRE(changes[c.getPublicKey()].mFees == bump->getFullFee()); + + auto const& lcl = f.lm().getLastClosedLedgerHeader(); + auto header = std::make_shared(lcl.header); + header->ledgerSeq += 1; + auto view = makePredictedLedgerView( + f.lm().getLastClosedLedgerState(), header, + std::make_shared(changes), + app.getMetrics()); + REQUIRE(view->getLedgerHeader().current().ledgerSeq == + lcl.header.ledgerSeq + 1); + auto lclView = f.lm().getLCLView(); + auto account = [&](AbstractLedgerView const& v, TestAccount& acc) { + auto entry = v.getAccount(acc.getPublicKey()); + REQUIRE(entry); + return entry.current().data.account(); + }; + REQUIRE(account(*view, a).seqNum == payA->getSeqNum()); + REQUIRE(account(*view, a).balance == account(*lclView, a).balance - 300); + REQUIRE(account(*view, b).seqNum == account(*lclView, b).seqNum + 1); + REQUIRE(account(*view, b).balance == account(*lclView, b).balance); + REQUIRE(account(*view, c).seqNum == account(*lclView, c).seqNum); + REQUIRE(account(*view, c).balance == + account(*lclView, c).balance - bump->getFullFee()); + // An account the set does not name is the last closed ledger's. + REQUIRE(account(*view, f.acc(3)) == account(*lclView, f.acc(3))); + // An account that does not exist does not appear. + REQUIRE( + !view->getAccount(SecretKey::pseudoRandomForTesting().getPublicKey())); + // Nothing is predicted about other entries. + LedgerKey code(CONTRACT_CODE); + REQUIRE_THROWS_AS(view->load(code), std::logic_error); +} + +// --------------------------------------------------------------------------- +// In a network. +// --------------------------------------------------------------------------- + +TEST_CASE("leaders select their next set early in a network", + "[herder][txset][early-txset][simulation]") +{ + Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); + auto simulation = Topologies::core(4, 0.75, networkID, [](int i) { + auto cfg = getTestConfig(i, Config::TESTDB_BUCKET_DB_PERSISTENT); + cfg.LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; + cfg.TESTING_UPGRADE_LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; + cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; + return cfg; + }); + auto nodes = simulation->getNodes(); + simulation->startAllNodes(); + + // Every node's mempool holds the root account's next payments: one fits + // each ledger (one transaction per source account), and the one after it + // only becomes valid when that ledger has applied. A leader that selects + // early gets the next one right only by predicting what the ledger in + // its ballot does. + auto const rootKey = nodes[0]->getRoot()->getSecretKey(); + auto const rootID = rootKey.getPublicKey(); + auto const firstSeq = nodes[0]->getRoot()->getLastSequenceNumber(); + std::vector payments; + for (int i = 1; i <= 400; ++i) + { + payments.push_back(transactionFromOperations(*nodes[0], rootKey, + firstSeq + i, + {payment(rootID, 1)}, 100) + ->getEnvelope()); + } + auto rootSeq = [&](Application& app) { + auto root = app.getLedgerManager().getLCLView()->getAccount(rootID); + REQUIRE(root); + return root.current().data.account().seqNum; + }; + for (auto const& node : nodes) + { + auto* app = node.get(); + auto& herder = static_cast(app->getHerder()); + // Virtual time runs ahead of the job's threads: start at the vote. + Access::setStartDelay(herder, std::chrono::milliseconds(0)); + herder.mGetTopTransactionsForTesting = [&, app](size_t) { + std::vector top; + auto const next = static_cast(rootSeq(*app) - firstSeq); + for (size_t i = next; i < next + 4 && i < payments.size(); ++i) + { + top.push_back(payments[i]); + } + return top; + }; + } + + auto& lm = nodes[1]->getLedgerManager(); + uint32_t const target = lm.getLastClosedLedgerNum() + 40; + simulation->crankUntil( + [&]() { return simulation->haveAllExternalized(target, 2); }, + 80 * simulation->getExpectedLedgerCloseTime(), false); + + int64_t planned = 0; + int64_t completed = 0; + int64_t notReady = 0; + int64_t failed = 0; + int64_t selfCheckFailed = 0; + for (auto const& node : nodes) + { + planned += txsetMeter(*node, "early-planned"); + completed += txsetMeter(*node, "early-completed"); + notReady += txsetMeter(*node, "early-not-ready") + + txsetMeter(*node, "early-not-started"); + failed += txsetMeter(*node, "early-failed"); + selfCheckFailed += txsetMeter(*node, "self-check-failed"); + // No leader proposed a set that apply found invalid. + REQUIRE(applyCounter(*node, "txset-skipped") == 0); + } + INFO(fmt::format("planned {}, completed {}, not ready {}", planned, + completed, notReady)); + REQUIRE(failed == 0); + REQUIRE(selfCheckFailed == 0); + REQUIRE(planned > 0); + // How often the selection is ready depends on the machine (the jobs run + // in real time, the network in virtual time). + REQUIRE(completed + notReady > 0); + // The ledgers carried the payments. + REQUIRE(rootSeq(*nodes[1]) - firstSeq >= 20); + for (auto const& node : nodes) + { + static_cast(node->getHerder()) + .mGetTopTransactionsForTesting = nullptr; + } +} + +// What the leader's path to its next set costs at perfnet's scale, with real +// SAC payments (never validated by Core before: the Rust overlay admitted +// them): a build as it is now, and the completion of a set that was selected +// while the previous ledger was in its ballot, with a share of the candidates +// arriving only after the selection's pull. +// Env: BENCH_CAP (transactions per set, default 8000), BENCH_LATE_PERCENT +// (share of a ledger's candidates that arrive after the selection's pull, +// default 30), BENCH_BACKLOG (1: the mempool holds twice the capacity at the +// build, as in a backlog; default 0), BENCH_REPS (default 2), BENCH_TASKS +// (validation tasks, default 16), BENCH_CLUSTERS (default 16). +TEST_CASE("early tx set completion costs", "[herder][.][benchmark]") +{ + auto envInt = [](char const* name, uint32_t dflt) { + auto v = std::getenv(name); + return v ? static_cast(std::stoul(v)) : dflt; + }; + uint32_t const cap = envInt("BENCH_CAP", 8000); + uint32_t const latePercent = envInt("BENCH_LATE_PERCENT", 30); + bool const backlog = envInt("BENCH_BACKLOG", 0) != 0; + uint32_t const reps = envInt("BENCH_REPS", 2); + uint32_t const tasks = envInt("BENCH_TASKS", 16); + uint32_t const clusters = envInt("BENCH_CLUSTERS", 16); + // Per repetition: a closing ledger and the next one's candidates (twice + // as many in a backlog), at 80% of the cap as on perfnet. + uint32_t const perLedger = cap * 8 / 10; + uint32_t const nextCount = backlog ? 2 * cap : perLedger; + uint32_t const perRep = perLedger + nextCount; + uint32_t const candidates = (reps + 1) * perRep; + + auto cfg = getTestConfig(0, Config::TESTDB_BUCKET_DB_PERSISTENT); + cfg.APPLY_LOAD_MODE = ApplyLoadMode::BENCHMARK_MODEL_TX; + cfg.APPLY_LOAD_MODEL_TX = ApplyLoadModelTx::SAC; + cfg.USE_CONFIG_FOR_GENESIS = true; + cfg.LEDGER_PROTOCOL_VERSION = Config::CURRENT_LEDGER_PROTOCOL_VERSION; + cfg.IGNORE_MESSAGE_LIMITS_FOR_TESTING = true; + cfg.APPLY_LOAD_MAX_SOROBAN_TX_COUNT = cap; + cfg.APPLY_LOAD_BATCH_SAC_COUNT = 1; + cfg.APPLY_LOAD_LEDGER_MAX_DEPENDENT_TX_CLUSTERS = clusters; + cfg.APPLY_LOAD_CLASSIC_TXS_PER_LEDGER = 0; + cfg.TESTING_UPGRADE_MAX_TX_SET_SIZE = 100; + // ApplyLoad makes one account per candidate it is configured for. + cfg.SOROBAN_TRANSACTION_QUEUE_SIZE_MULTIPLIER = candidates / cap + 1; + cfg.GENESIS_TEST_ACCOUNT_COUNT = + cap * cfg.SOROBAN_TRANSACTION_QUEUE_SIZE_MULTIPLIER + 10; + cfg.EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY = false; + cfg.DISABLE_TX_META_FOR_TESTING = true; + cfg.METADATA_DEBUG_LEDGERS = 0; + cfg.WORKER_THREADS = static_cast(tasks) + 2; + VirtualClock clock(VirtualClock::REAL_TIME); + auto app = createTestApplication(clock, cfg); + auto& herder = static_cast(app->getHerder()); + auto& lm = app->getLedgerManager(); + auto& metrics = app->getMetrics(); + Access::setStartDelay(herder, std::chrono::milliseconds(0)); + + std::vector envelopes; + { + ApplyLoad al(*app); + envelopes = al.generateSacPaymentEnvelopesForTesting(candidates); + } + REQUIRE(envelopes.size() == candidates); + app->getBatchExecutor().setPreferredTaskCountForTesting(tasks); + REQUIRE(lm.getLastClosedSorobanNetworkConfig().ledgerMaxTxCount() == cap); + + // The mempool: a range of the envelopes, handed over as the overlay + // encodes a pull ([count:4]([len:4][xdr])*), encoded ahead of time. + std::vector> encoded; + encoded.reserve(envelopes.size()); + for (auto const& env : envelopes) + { + encoded.emplace_back(xdr::xdr_to_opaque(env)); + } + std::vector mempool; + herder.mGetTopTransactionsPayloadForTesting = [&](size_t count) { + uint32_t const n = + static_cast(std::min(count, mempool.size())); + std::vector payload(4); + std::memcpy(payload.data(), &n, 4); + for (uint32_t i = 0; i < n; ++i) + { + auto const& bytes = encoded[mempool[i]]; + uint32_t const len = static_cast(bytes.size()); + auto const at = payload.size(); + payload.resize(at + 4 + len); + std::memcpy(payload.data() + at, &len, 4); + std::memcpy(payload.data() + at + 4, bytes.data(), len); + } + return payload; + }; + auto range = [&](size_t from, size_t count) { + std::vector res(count); + std::iota(res.begin(), res.end(), from); + return res; + }; + auto nextCloseTime = [&]() { + auto const& lcl = lm.getLastClosedLedgerHeader(); + return getConsensusTime(lcl.header.scpValue) + .next(lcl.header.ledgerVersion); + }; + auto crankUntil = [&](std::function done) { + auto const deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(120); + while (!done()) + { + REQUIRE(std::chrono::steady_clock::now() < deadline); + clock.crank(false); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + }; + std::vector const stages = { + "build-pull", "build-frames", "build-refill-check", + "build-trim", "build-select", "early-final-pass", + "early-top-up", "build-selection-check", "build-xdr", + "build-handoff", "early-complete", "build", + "early-pull", "early-select", "early-job"}; + auto clearStages = [&]() { + for (auto const& stage : stages) + { + metrics.NewTimer({"herder", "txset", stage}).Clear(); + } + }; + std::map>> samples; + auto record = [&](std::string const& what) { + for (auto const& stage : stages) + { + auto& timer = metrics.NewTimer({"herder", "txset", stage}); + if (timer.count() > 0) + { + samples[what][stage].push_back(timer.sum()); + } + } + }; + + std::cout << "=== leader's next set: cap " << cap << ", " << perLedger + << " per ledger, " << (backlog ? "backlog, " : "") << latePercent + << "% arriving after the selection's pull, " << tasks + << " validation tasks, " << clusters + << " clusters ===" << std::endl; + size_t included = 0; + size_t builtSize = 0; + for (uint32_t rep = 0; rep < reps; ++rep) + { + size_t const base = rep * perRep; + auto const closingTxs = range(base, perLedger); + auto const nextTxs = range(base + perLedger, nextCount); + size_t const earlyCount = nextCount * (100 - latePercent) / 100; + + // The closing ledger's set, built from the mempool: a plain build. + // In a backlog the mempool holds twice the cap behind it. + mempool = closingTxs; + if (backlog) + { + auto more = + range(base + perLedger, std::min(nextCount, 2 * cap)); + mempool.insert(mempool.end(), more.begin(), more.end()); + } + PubKeyUtils::clearVerifySigCache(); + clearStages(); + auto closing = Access::build(herder, lm.getLastClosedLedgerNum() + 1, + nextCloseTime()); + record("1. build, cold signature cache"); + clearStages(); + auto closing2 = Access::build(herder, lm.getLastClosedLedgerNum() + 1, + nextCloseTime()); + record("2. build, warm signature cache"); + builtSize = closing.txSet->sizeTxTotal(); + + // The node votes for it; its early selection for the ledger after + // sees the closing ledger's transactions and the early arrivals. + mempool = closingTxs; + mempool.insert(mempool.end(), nextTxs.begin(), + nextTxs.begin() + earlyCount); + auto const seq = lm.getLastClosedLedgerNum() + 1; + auto const closeTime = nextCloseTime(); + herder.getPendingEnvelopes().putTxSet(closing.txSet->getContentsHash(), + seq, closing.txSet); + SCPEnvelope envelope; + envelope.statement.slotIndex = seq; + envelope.statement.pledges.type(SCP_ST_PREPARE); + envelope.statement.pledges.prepare().ballot.counter = 1; + envelope.statement.pledges.prepare().ballot.value = xdr::xdr_to_opaque( + herder.makeStellarValue(closing.txSet->getContentsHash(), closeTime, + emptyUpgradeSteps, cfg.NODE_SEED)); + clearStages(); + Access::voted(herder, envelope); + REQUIRE(Access::planned(herder)); + crankUntil([&]() { return Access::job(herder) != nullptr; }); + auto job = Access::job(herder); + crankUntil([&]() { return job->mDone.load(); }); + samples["3. early selection (background, during the ballot)"] + ["job, wall time"] + .push_back(static_cast(job->mFinishedAfterNs) / 1e6); + record("3. early selection (background, during the ballot)"); + + // The ledger closes; the rest of the candidates has arrived. + app->getHerder().externalizeValue(closing.txSet, seq, closeTime, + emptyUpgradeSteps); + REQUIRE(lm.getLastClosedLedgerNum() == seq); + mempool = nextTxs; + clearStages(); + auto const completedBefore = txsetMeter(*app, "early-completed"); + auto next = Access::build(herder, lm.getLastClosedLedgerNum() + 1, + nextCloseTime()); + REQUIRE(txsetMeter(*app, "early-completed") == completedBefore + 1); + record("4. completion at the close"); + included = next.txSet->sizeTxTotal(); + REQUIRE(next.applicableTxSet->checkValid(*app, 1, 1)); + REQUIRE(included == std::min(cap, nextCount)); + } + std::cout << "plain build: " << builtSize + << " transactions; completed set: " << included << " transactions" + << std::endl; + for (auto& [what, byStage] : samples) + { + std::cout << what << std::endl; + for (auto& [stage, v] : byStage) + { + std::sort(v.begin(), v.end()); + std::cout << " " << stage << ": median " << v[v.size() / 2] + << " ms (min " << v.front() << ", max " << v.back() + << ", n=" << v.size() << ")" << std::endl; + } + } + herder.mGetTopTransactionsPayloadForTesting = nullptr; +} diff --git a/src/herder/test/HerderTests.cpp b/src/herder/test/HerderTests.cpp index 012af8cb1..0f915dca9 100644 --- a/src/herder/test/HerderTests.cpp +++ b/src/herder/test/HerderTests.cpp @@ -73,6 +73,15 @@ using namespace stellar::txbridge; using namespace stellar::txtest; using namespace historytestutils; +namespace stellar +{ +// LeaderBuildTestAccess::selectLocalFirstLeader (defined further down), for +// the fixtures above it: replaces the second validator of a two-validator +// quorum set by one that sorts behind the local node in the next slot's +// leader order. +void makeLocalNodeFirstLeader(HerderImpl& herder); +} + static TransactionTestFramePtr makeMultiPayment(stellar::TestAccount& destAccount, stellar::TestAccount& src, int nbOps, int64 paymentBase, uint32 extraFee, uint32 feeMult) @@ -2960,6 +2969,10 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) SECTION("validateValue txSet cached") { auto& herder = static_cast(app->getHerder()); + // Which of this fixture's two validators leads the next slot depends + // on the local node's key, i.e. on the test instance number; the + // expectations below are for the node that builds and proposes. + makeLocalNodeFirstLeader(herder); auto seq = herder.trackingConsensusLedgerIndex() + 1; auto& cache = herder.getHerderSCPDriver().getTxSetValidityCache(); @@ -5387,6 +5400,12 @@ namespace stellar class LeaderBuildTestAccess { public: + static void + checkInLine(HerderImpl& herder) + { + herder.mTxSetSelfCheckFailed = true; + } + static VirtualClock::time_point triggerAnchor(HerderImpl& herder, uint64_t slot) { @@ -5544,6 +5563,12 @@ class LeaderBuildTestAccess FAIL("Could not select a first-round local leader"); } }; + +void +makeLocalNodeFirstLeader(HerderImpl& herder) +{ + LeaderBuildTestAccess::selectLocalFirstLeader(herder); +} } TEST_CASE("trigger fallback credits elapsed time up to the first ballot", @@ -6008,10 +6033,17 @@ TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") tx->addSignature(alice); auto extra = makeTx(bob, alice); mempool = {tx->getEnvelope(), extra->getEnvelope()}; - size_t pulls = 0; - herder.mGetTopTransactionsForTesting = [&](size_t) { - ++pulls; - return mempool; + herder.mGetTopTransactionsForTesting = [&](size_t) { return mempool; }; + // Builds of a tx set since here. (Not pulls of the mempool: the leader + // starts executing a prepared set at once, and that execution pulls + // candidates for the ledger after it.) + auto const buildsBefore = + app->getMetrics().NewTimer({"herder", "txset", "build"}).count(); + auto builds = [&]() { + return app->getMetrics() + .NewTimer({"herder", "txset", "build"}) + .count() - + buildsBefore; }; // Exercise the real post-close hook, including its heartbeat reset after // advancing the fixture clock past the upgrade helper's close time. @@ -6023,7 +6055,7 @@ TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") [&]() { return LeaderBuildTestAccess::prepared(herder) != nullptr; }, std::chrono::seconds(1)); REQUIRE(clock.now() < trigger); - REQUIRE(pulls == 1); + REQUIRE(builds() == 1); auto prepared = LeaderBuildTestAccess::prepared(herder); auto const preparedCloseTime = LeaderBuildTestAccess::closeTime(herder); REQUIRE(prepared); @@ -6042,7 +6074,7 @@ TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") testutil::crankUntil( app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); - REQUIRE(pulls == 1); + REQUIRE(builds() == 1); REQUIRE(!LeaderBuildTestAccess::prepared(herder)); REQUIRE(std::get( herder.getTxSet(prepared->getContentsHash())) == prepared); @@ -6071,7 +6103,7 @@ TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") xdr::xdr_from_opaque(ballot.value, value); REQUIRE(value.txSetHash == prepared->getContentsHash()); REQUIRE(getConsensusTime(value) == preparedCloseTime); - REQUIRE(pulls == 1); + REQUIRE(builds() == 1); } SECTION("clock regression rejects the prepared future close time") { @@ -6079,7 +6111,7 @@ TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") // The fallback builds with today's valid close time, and must not // claim that our time-bound transaction is valid for that time. herder.triggerNextLedger(seq, true); - REQUIRE(pulls == 2); + REQUIRE(builds() == 2); REQUIRE(!LeaderBuildTestAccess::prepared(herder)); REQUIRE(std::get( herder.getTxSet(prepared->getContentsHash())) == nullptr); @@ -6089,17 +6121,17 @@ TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") herder.externalizeValue(TxSetXDRFrame::makeEmpty(lcl), seq, preparedCloseTime, {}, cfg.NODE_SEED); REQUIRE(LeaderBuildTestAccess::prepared(herder) != prepared); - auto const pullsAfterClose = pulls; - // A stale timer/caller cannot propose for the old slot or pull again. + auto const buildsAfterClose = builds(); + // A stale timer/caller cannot propose for the old slot or build again. herder.triggerNextLedger(seq, true); - REQUIRE(pulls == pullsAfterClose); + REQUIRE(builds() == buildsAfterClose); } SECTION("loss of sync discards preparation") { herder.lostSync(); REQUIRE(!LeaderBuildTestAccess::prepared(herder)); herder.triggerNextLedger(seq, true); - REQUIRE(pulls == 1); + REQUIRE(builds() == 1); } SECTION("shutdown cancels preparation") { @@ -6958,17 +6990,20 @@ TEST_CASE("the set prepared before the trigger is proposed unchanged", // More candidates than capacity, but they cannot fill the ledger. mempool = {bad->getEnvelope(), bad->getEnvelope(), bad->getEnvelope()}; } - size_t pulls = 0; - herder.mGetTopTransactionsForTesting = [&](size_t) { - ++pulls; - return mempool; + herder.mGetTopTransactionsForTesting = [&](size_t) { return mempool; }; + // Builds, not pulls of the mempool: the leader starts executing the + // prepared set at once, and that execution pulls candidates for the + // ledger after it. + auto builds = [&]() { + return app->getMetrics().NewTimer({"herder", "txset", "build"}).count(); }; + REQUIRE(builds() == 0); LeaderBuildTestAccess::schedule(herder); testutil::crankUntil( app, [&]() { return LeaderBuildTestAccess::prepared(herder) != nullptr; }, std::chrono::seconds(1)); - REQUIRE(pulls == 1); + REQUIRE(builds() == 1); auto const prepared = LeaderBuildTestAccess::prepared(herder); REQUIRE(prepared->sizeTxTotal() == (invalidFirst ? 0 : 1)); // It was built a short lead time before the trigger, not earlier. @@ -6982,7 +7017,7 @@ TEST_CASE("the set prepared before the trigger is proposed unchanged", app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); // No second build: the proposal names the prepared set. - REQUIRE(pulls == 1); + REQUIRE(builds() == 1); auto messages = herder.getSCP().getLatestMessagesSend(seq); REQUIRE(!messages.empty()); REQUIRE(messages.front().statement.pledges.type() == SCP_ST_PREPARE); @@ -7458,9 +7493,9 @@ TEST_CASE("leader builds validate only the top of the mempool pull", mempool.push_back(tx->getEnvelope()); hashes.push_back(tx->getFullHash()); } - size_t requested = 0; + std::vector requested; herder.mGetTopTransactionsForTesting = [&](size_t count) { - requested = count; + requested.push_back(count); return std::vector( mempool.begin(), mempool.begin() + std::min(count, mempool.size())); }; @@ -7471,10 +7506,13 @@ TEST_CASE("leader builds validate only the top of the mempool pull", getConsensusTime(lcl.header.scpValue).next(lcl.header.ledgerVersion)); herder.mGetTopTransactionsForTesting = nullptr; - // The pull is unchanged: twice the capacity of both phases. - REQUIRE( - requested == - 2 * (10 + lm.getLastClosedSorobanNetworkConfig().ledgerMaxTxCount())); + // The pull is the capacity of both phases plus the margin of 10%. Five + // of its candidates are invalid, which leaves the set short while the + // mempool has more: the build pulls again, twice as much. + size_t const capacity = + 10 + lm.getLastClosedSorobanNetworkConfig().ledgerMaxTxCount(); + size_t const pull = capacity + (capacity * 10 + 99) / 100; + REQUIRE(requested == std::vector{pull, 2 * pull}); // The invalid top candidates were replaced: the set is full, with the // best valid ones. auto applicable = txSet->prepareForApply(*app, lcl.header); @@ -7486,7 +7524,7 @@ TEST_CASE("leader builds validate only the top of the mempool pull", } REQUIRE(included == std::set(hashes.begin() + 5, hashes.begin() + 15)); - // #40 is past the validated candidates (0-22): it stays in the mempool + // #40 is past the candidates the build pulled: it stays in the mempool // until it reaches them. REQUIRE(std::set(invalid.begin(), invalid.end()) == std::set(hashes.begin(), hashes.begin() + 5)); @@ -7566,7 +7604,29 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") herder.mGetTopTransactionsForTesting = [&](size_t) { return OverlayIPC::parseTopTransactions(payload); }; + // A build takes the payload itself, and only as many transactions as + // it asks for. + std::vector ends{4}; + for (uint32_t i = 0; i < pull; ++i) + { + uint32_t len = 0; + std::memcpy(&len, payload.data() + ends.back(), 4); + ends.push_back(ends.back() + 4 + len); + } + herder.mGetTopTransactionsPayloadForTesting = [&](size_t n) { + uint32_t const given = static_cast(std::min(n, pull)); + std::vector res(payload.begin(), + payload.begin() + ends[given]); + std::memcpy(res.data(), &given, 4); + return res; + }; + // BENCH_INLINE_CHECK=1: the build checks its written set before it + // returns (as it does once a background check has failed). + if (envInt("BENCH_INLINE_CHECK", 0) != 0) + { + LeaderBuildTestAccess::checkInLine(herder); + } auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; auto const closeTime = @@ -7637,6 +7697,11 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") // against perfnet's herder.txset.validate). auto const& lastSet = LeaderBuildTestAccess::build(herder, seq, closeTime); + if (envInt("BENCH_SETTLE_MS", 0) != 0) + { + std::this_thread::sleep_for( + std::chrono::milliseconds(envInt("BENCH_SETTLE_MS", 0))); + } for (uint32_t rep = 0; rep < reps; ++rep) { auto received = TxSetXDRFrame::makeFromWire( @@ -7647,6 +7712,10 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") } auto const start = std::chrono::steady_clock::now(); auto applicable = received->prepareForApply(*app, lcl.header); + samples["received-set-prepare"].push_back( + std::chrono::duration( + std::chrono::steady_clock::now() - start) + .count()); REQUIRE(applicable); REQUIRE(applicable->checkValid(*app, 0, 0)); samples["received-set-validate"].push_back( @@ -7733,5 +7802,6 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") << ", n=" << v.size() << ")" << std::endl; } } + herder.mGetTopTransactionsPayloadForTesting = nullptr; herder.mGetTopTransactionsForTesting = nullptr; } diff --git a/src/herder/test/ProposalPrevalidationTests.cpp b/src/herder/test/ProposalPrevalidationTests.cpp index 0de8f5eea..454d4c5ea 100644 --- a/src/herder/test/ProposalPrevalidationTests.cpp +++ b/src/herder/test/ProposalPrevalidationTests.cpp @@ -8,10 +8,17 @@ // next ledger validates its candidates against it in the background // (HerderImpl::startProposalPrevalidation), and the build of the next set // takes those results instead of validating again (TxValidationCache). +// +// And of what the leader does beyond that: it executes its own set as soon +// as it is prepared (HerderImpl::executePreparedTxSet), and checks the sets +// it builds in the background (TxSetSelfCheck). The selection of the next +// set ahead of time has its own tests (EarlyTxSetTests.cpp); here it only +// serves as one of the two ways a set gets prepared. #include "crypto/SHA.h" #include "herder/Herder.h" #include "herder/HerderImpl.h" +#include "herder/HerderUtils.h" #include "herder/LeaderSchedule.h" #include "herder/LedgerCloseData.h" #include "herder/TxSetFrame.h" @@ -140,6 +147,110 @@ class ProposalPrevalidationTestAccess { return herder.mLastClosedTxSetSize; } + + // What the herder does with a statement this node emits, as far as the + // early selection of the next set goes. + static void + voted(HerderImpl& herder, SCPEnvelope const& envelope) + { + herder.maybeScheduleEarlyTxSetSelection(envelope); + } + + static bool + earlySelectionPlanned(HerderImpl& herder) + { + return herder.mEarlyTxSetPlan && herder.mEarlyTxSetPlan->planned; + } + + // The running or finished early selection job, if any. + static std::shared_ptr + earlyJob(HerderImpl& herder) + { + return herder.mEarlyTxSet; + } + + // Prepares the set for `ledgerSeq` as the first-round leader does a lead + // time before the trigger; returns whether the set is a completed early + // selection (and not a build). + static bool + prepareEarly(HerderImpl& herder, uint32_t ledgerSeq, + ConsensusTime closeTime) + { + auto& completed = herder.mApp.getMetrics().NewMeter( + {"herder", "txset", "early-completed"}, "txset"); + auto const before = completed.count(); + herder.prepareTxSet(ledgerSeq, closeTime); + return herder.mPreparedTxSet && completed.count() == before + 1; + } + + // What the first-round leader does a lead time before the trigger. + static void + prepareByBuilding(HerderImpl& herder, uint32_t ledgerSeq, + ConsensusTime closeTime) + { + herder.prepareTxSet(ledgerSeq, closeTime); + } + + static TxSetXDRFrameConstPtr + prepared(HerderImpl& herder) + { + return herder.mPreparedTxSet ? herder.mPreparedTxSet->txSet : nullptr; + } + + static std::vector + preparedInvalidTxs(HerderImpl& herder) + { + REQUIRE(herder.mPreparedTxSet); + return herder.mPreparedTxSet->invalidTxHashes; + } + + // What the node proposes when the trigger fires. + static ValueWrapperPtr + propose(HerderImpl& herder, uint32_t ledgerSeq) + { + return herder.makeProposalValue(ledgerSeq); + } + + // The (slot, value) last handed to the apply thread by the herder. + static std::optional> const& + speculativeValue(HerderImpl& herder) + { + return herder.mSpeculativeApplyValue; + } + + // What the herder does when the node emits `envelope`. + static void + emitted(HerderImpl& herder, SCPEnvelope const& envelope) + { + herder.maybeStartSpeculativeApply(envelope); + } + + // What the herder does about a set it built once it has emitted a + // statement. + static void + statementEmitted(HerderImpl& herder) + { + herder.resolvePendingTxSetSelfCheck(); + } + + static bool + selfCheckHasFailed(HerderImpl& herder) + { + return herder.mTxSetSelfCheckFailed; + } + + // Whether a build is scheduled for a lead time before the trigger. + static bool + buildScheduled(HerderImpl& herder) + { + return herder.mPrepareTxSetTimer.seq() != 0; + } + + static void + discardPrepared(HerderImpl& herder) + { + herder.discardPreparedTxSet(); + } }; } @@ -237,14 +348,26 @@ struct PrevalidationFixture std::optional mE; // What the mempool holds; the Herder pulls it in this order. std::vector mMempool; - - PrevalidationFixture() + // How often the Herder pulled it. + size_t mPulls{0}; + + // With `herderSpeculates` the Herder hands ledgers to the apply thread + // itself, where it would in production; the tests still drive the + // speculation of the ledger they close first. + explicit PrevalidationFixture( + bool herderSpeculates = false, + std::optional maxTxSetSize = std::nullopt, + uint32_t protocol = GATE_PROTOCOL) : mCfg(getTestConfig(0, Config::TESTDB_BUCKET_DB_PERSISTENT)) { - mCfg.LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; - mCfg.TESTING_UPGRADE_LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; + mCfg.LEDGER_PROTOCOL_VERSION = protocol; + mCfg.TESTING_UPGRADE_LEDGER_PROTOCOL_VERSION = protocol; + if (maxTxSetSize) + { + mCfg.TESTING_UPGRADE_MAX_TX_SET_SIZE = *maxTxSetSize; + } // The tests drive speculation themselves. - mCfg.EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY = false; + mCfg.EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY = herderSpeculates; REQUIRE(mCfg.backgroundLedgerApply()); mApp = createTestApplication(mClock, mCfg); auto root = mApp->getRoot(); @@ -255,7 +378,10 @@ struct PrevalidationFixture mC.emplace(root->create("C", balance)); mD.emplace(root->create("D", balance)); mE.emplace(root->create("E", balance)); + // An early selection starts as soon as the node has voted. + herder().mEarlyTxSetStartDelayForTesting = std::chrono::milliseconds(0); herder().mGetTopTransactionsForTesting = [this](size_t) { + ++mPulls; std::vector envelopes; for (auto const& tx : mMempool) { @@ -268,6 +394,74 @@ struct PrevalidationFixture ~PrevalidationFixture() { herder().mGetTopTransactionsForTesting = nullptr; + herder().mBuiltTxSetOverrideForTesting = nullptr; + } + + // Cranks until `done`, giving background threads time to post to the + // main thread. + template + void + crankUntil(F&& done) + { + auto const deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (!done()) + { + REQUIRE(std::chrono::steady_clock::now() < deadline); + mClock.crank(false); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + } + + void + crankFor(std::chrono::milliseconds duration) + { + auto const deadline = std::chrono::steady_clock::now() + duration; + while (std::chrono::steady_clock::now() < deadline) + { + mClock.crank(false); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + } + + // The node votes for `closing` as the next ledger and, being first in + // line for the one after, selects that one's set ahead of time: waits + // for the selection. + void + selectEarly(TxSetXDRFrameConstPtr const& closing, bool wait = true) + { + auto const value = valueFor(closing); + // By the time a node votes, SCP has asked for the slot's leaders, + // which brings the tracked quorum (the node set) up to date. + REQUIRE(herder().getHerderSCPDriver().getLeaderSchedule(nextSeq()) == + std::vector{mCfg.NODE_SEED.getPublicKey()}); + herder().getPendingEnvelopes().putTxSet(closing->getContentsHash(), + nextSeq(), closing); + SCPEnvelope vote; + vote.statement.slotIndex = nextSeq(); + vote.statement.pledges.type(SCP_ST_PREPARE); + vote.statement.pledges.prepare().ballot.counter = 1; + vote.statement.pledges.prepare().ballot.value = + xdr::xdr_to_opaque(value); + Access::voted(herder(), vote); + REQUIRE(Access::earlySelectionPlanned(herder())); + crankUntil([&]() { return Access::earlyJob(herder()) != nullptr; }); + if (!wait) + { + return; + } + auto job = Access::earlyJob(herder()); + crankUntil([&]() { return job->mDone.load(); }); + std::lock_guard lock(job->mMutex); + REQUIRE(job->mSelection); + } + + // The same, and then the ledger closes with `closing`. + void + closeWithEarlySelection(TxSetXDRFrameConstPtr const& closing) + { + selectEarly(closing); + externalize(closing); } HerderImpl& @@ -986,6 +1180,9 @@ TEST_CASE("leaders keep their term and pre-validate in a network", cfg.LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; cfg.TESTING_UPGRADE_LEDGER_PROTOCOL_VERSION = GATE_PROTOCOL; cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; + // This test is about the pre-validation, which the node first in + // line skips when it selects the next set ahead of time. + cfg.EXPERIMENTAL_EARLY_TX_SET_SELECTION = false; return cfg; }); auto nodes = simulation->getNodes(); @@ -1144,3 +1341,912 @@ TEST_CASE("leaders keep their term and pre-validate in a network", .mGetTopTransactionsForTesting = nullptr; } } + +// --------------------------------------------------------------------------- +// Early selection: the set for ledger N+1 is selected while ledger N is in +// its ballot and completed when N has closed. +// --------------------------------------------------------------------------- + +namespace +{ +int64_t +txsetMeter(Application& app, std::string const& name) +{ + return meter(app, "txset", name, "txset"); +} + +StellarValue +stellarValueOf(ValueWrapperPtr const& value) +{ + REQUIRE(value); + StellarValue sv; + REQUIRE(toStellarValue(value->getValue(), sv)); + return sv; +} +} + +TEST_CASE("the leader executes its prepared set before the trigger", + "[herder][txset][speculative-apply][early-txset]") +{ + PrevalidationFixture f(/*herderSpeculates=*/true); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& lm = f.lm(); + auto& a = *f.mA; + auto& b = *f.mB; + auto const aSeq = a.getLastSequenceNumber(); + auto const bSeq = b.getLastSequenceNumber(); + auto closingA = paymentWithSeq(app, a, aSeq + 1); + auto nextA = paymentWithSeq(app, a, aSeq + 2); + auto plainB = paymentWithSeq(app, b, bSeq + 1); + auto closing = makeUnvalidatedTxSet(app, {closingA}); + f.mMempool = {closingA, nextA, plainB}; + f.closeWithEarlySelection(closing); + f.mMempool = {nextA, plainB}; + + auto const seq = f.nextSeq(); + auto const closeTime = f.nextCloseTime(); + auto const started = applyCounter(app, "speculation-started"); + auto const hits = applyCounter(app, "speculation-hit"); + auto const misses = applyCounter(app, "speculation-miss"); + + auto afterPreparing = [&]() { + auto prepared = Access::prepared(herder); + REQUIRE(prepared); + // The ledger is with the apply thread already, although the node + // has not proposed anything yet. + REQUIRE(applyCounter(app, "speculation-started") == started + 1); + while (lm.isSpeculativelyApplying()) + { + f.mClock.crank(true); + } + REQUIRE(lm.getLastClosedLedgerNum() == seq - 1); + // It is the very value the trigger proposes ... + auto const proposal = Access::propose(herder, seq); + REQUIRE(proposal); + REQUIRE(Access::speculativeValue(herder)); + REQUIRE(Access::speculativeValue(herder)->first == seq); + REQUIRE(Access::speculativeValue(herder)->second == + proposal->getValue()); + REQUIRE(stellarValueOf(proposal).txSetHash == + prepared->getContentsHash()); + // ... so the vote the node then emits does not start it again. + SCPEnvelope vote; + vote.statement.slotIndex = seq; + vote.statement.pledges.type(SCP_ST_PREPARE); + vote.statement.pledges.prepare().ballot.counter = 1; + vote.statement.pledges.prepare().ballot.value = proposal->getValue(); + Access::emitted(herder, vote); + REQUIRE(applyCounter(app, "speculation-started") == started + 1); + + SECTION("its value externalizes: the execution is committed") + { + f.externalize(prepared); + REQUIRE(applyCounter(app, "speculation-hit") == hits + 1); + REQUIRE(applyCounter(app, "speculation-miss") == misses); + REQUIRE(a.getLastSequenceNumber() == aSeq + 2); + REQUIRE(b.getLastSequenceNumber() == bSeq + 1); + } + SECTION("another value externalizes: the execution is dropped") + { + f.externalize(makeUnvalidatedTxSet(app, {plainB})); + REQUIRE(applyCounter(app, "speculation-hit") == hits); + REQUIRE(applyCounter(app, "speculation-miss") == misses + 1); + REQUIRE(a.getLastSequenceNumber() == aSeq + 1); + REQUIRE(b.getLastSequenceNumber() == bSeq + 1); + } + SECTION("the node votes for another value: that one is executed") + { + auto other = makeUnvalidatedTxSet(app, {plainB}); + herder.getPendingEnvelopes().putTxSet(other->getContentsHash(), seq, + other); + auto const otherValue = f.valueFor(other); + vote.statement.pledges.prepare().ballot.value = + xdr::xdr_to_opaque(otherValue); + Access::emitted(herder, vote); + REQUIRE(applyCounter(app, "speculation-started") == started + 2); + while (lm.isSpeculativelyApplying()) + { + f.mClock.crank(true); + } + f.externalize(other); + REQUIRE(applyCounter(app, "speculation-hit") == hits + 1); + REQUIRE(a.getLastSequenceNumber() == aSeq + 1); + REQUIRE(b.getLastSequenceNumber() == bSeq + 1); + } + }; + + SECTION("a set selected early") + { + REQUIRE(Access::prepareEarly(herder, seq, closeTime)); + afterPreparing(); + } + SECTION("a set built before the trigger") + { + Access::prepareByBuilding(herder, seq, closeTime); + afterPreparing(); + } +} + +TEST_CASE("a prepared set is not executed early when it cannot be", + "[herder][txset][speculative-apply][early-txset]") +{ + SECTION("speculative apply is off") + { + PrevalidationFixture f(/*herderSpeculates=*/false); + auto& app = *f.mApp; + auto& a = *f.mA; + f.mMempool = {paymentWithSeq(app, a, a.getLastSequenceNumber() + 1)}; + auto const started = applyCounter(app, "speculation-started"); + Access::prepareByBuilding(f.herder(), f.nextSeq(), f.nextCloseTime()); + REQUIRE(Access::prepared(f.herder())); + REQUIRE(applyCounter(app, "speculation-started") == started); + REQUIRE(!Access::speculativeValue(f.herder())); + } + SECTION("the node is about to propose another set") + { + PrevalidationFixture f(/*herderSpeculates=*/true); + auto& app = *f.mApp; + auto& a = *f.mA; + f.mMempool = {paymentWithSeq(app, a, a.getLastSequenceNumber() + 1)}; + f.herder().mProposedTxSetOverride = + [&](LedgerHeaderHistoryEntry const& lcl) { + return TxSetXDRFrame::makeEmpty(lcl); + }; + auto const started = applyCounter(app, "speculation-started"); + Access::prepareByBuilding(f.herder(), f.nextSeq(), f.nextCloseTime()); + REQUIRE(Access::prepared(f.herder())); + REQUIRE(applyCounter(app, "speculation-started") == started); + f.herder().mProposedTxSetOverride = nullptr; + } +} + +// --------------------------------------------------------------------------- +// The builder's self-check runs beside the use of the set. +// --------------------------------------------------------------------------- + +TEST_CASE("the builder checks the set it built", "[herder][txset][early-txset]") +{ + PrevalidationFixture f; + auto& app = *f.mApp; + auto& a = *f.mA; + auto& b = *f.mB; + auto txA = paymentWithSeq(app, a, a.getLastSequenceNumber() + 1); + auto txB = paymentWithSeq(app, b, b.getLastSequenceNumber() + 1); + auto const lcl = f.lm().getLastClosedLedgerHeader(); + PerPhaseTransactionList phases{{txA, txB}, {}}; + PerPhaseTransactionList invalid(2); + + // Built without the check: the caller gets it to run. + std::shared_ptr check; + bool handedOut = false; + auto built = makeTxSetFromTransactions( + phases, app, ApplyTimeOffset::fromSeconds(1), invalid, false, {}, false, + [&](TxSetXDRFrameConstPtr const&) { handedOut = true; }, {}, nullptr, + &check); + REQUIRE(handedOut); + REQUIRE(built.first); + REQUIRE(built.second); + REQUIRE(check); + REQUIRE(check->getTxSet() == built.first); + REQUIRE(built.second->getContentsHash() == built.first->getContentsHash()); + REQUIRE(txHashes(built.second) == hashesOf({txA, txB})); + + SECTION("passes on the set that was built") + { + REQUIRE(!check->run(app)); + // the same set as a build that checks before it returns + auto checked = makeTxSetFromTransactions( + phases, app, ApplyTimeOffset::fromSeconds(1), invalid); + REQUIRE(checked.first->getContentsHash() == + built.first->getContentsHash()); + } + SECTION("runs on another thread") + { + std::optional failure{"not run"}; + std::thread t([&]() { failure = check->run(app); }); + t.join(); + REQUIRE(!failure); + } + SECTION("a set that lost a transaction") + { + auto broken = + check->withTxSetForTesting(makeUnvalidatedTxSet(app, {txA})); + auto const failure = broken->run(app); + REQUIRE(failure); + REQUIRE(failure->find("shape is mismatched") != std::string::npos); + } + SECTION("a set for another ledger") + { + // The same transactions on top of the ledger before. (Checked off + // the main thread, as the node does: on the main thread reading + // such a set back is an assertion failure.) + auto xdr = built.first->getGeneralizedXDR(); + xdr.v1TxSet().previousLedgerHash = lcl.header.previousLedgerHash; + auto other = check->withTxSetForTesting( + TxSetXDRFrame::makeFromWire(std::move(xdr))); + std::optional failure; + std::thread t([&]() { failure = other->run(app); }); + t.join(); + REQUIRE(failure); + REQUIRE(failure->find("PREVIOUS_LEDGER_HASH_MISMATCH") != + std::string::npos); + } + SECTION("a set with a base fee below the ledger's") + { + auto cheap = testtxset::makeNonValidatedGeneralizedTxSet( + {{std::make_pair(std::make_optional(1), + std::vector{txA, txB})}, + {}}, + app, lcl.hash); + REQUIRE(cheap.first); + REQUIRE(check->withTxSetForTesting(cheap.first)->run(app)); + } +} + +TEST_CASE("a set that fails its self-check after it was handed out", + "[herder][txset][early-txset]") +{ + PrevalidationFixture f; + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = *f.mA; + auto& b = *f.mB; + auto& c = *f.mC; + auto const aSeq = a.getLastSequenceNumber(); + auto const cSeq = c.getLastSequenceNumber(); + auto closingA = paymentWithSeq(app, a, aSeq + 1); + auto nextA = paymentWithSeq(app, a, aSeq + 2); + auto plainB = paymentWithSeq(app, b, b.getLastSequenceNumber() + 1); + auto closing = makeUnvalidatedTxSet(app, {closingA}); + f.mMempool = {closingA, nextA, plainB}; + f.closeWithEarlySelection(closing); + f.mMempool = {nextA, plainB}; + auto const seq = f.nextSeq(); + auto const closeTime = f.nextCloseTime(); + auto const failedBefore = txsetMeter(app, "self-check-failed"); + auto const passedBefore = txsetMeter(app, "self-check-passed"); + + // What the node "builds": a set that is not what it selected. The one + // transaction it holds is not valid either (a sequence number too far + // ahead), so ledger apply refuses it. + auto broken = makeUnvalidatedTxSet(app, {paymentWithSeq(app, c, cSeq + 5)}); + auto buildBroken = [&]() { + herder.mBuiltTxSetOverrideForTesting = + [&](TxSetXDRFrameConstPtr const&) { return broken; }; + }; + auto failed = [&]() { + return txsetMeter(app, "self-check-failed") == failedBefore + 1; + }; + + SECTION("a set that passes stays prepared") + { + REQUIRE(Access::prepareEarly(herder, seq, closeTime)); + auto prepared = Access::prepared(herder); + f.crankUntil([&]() { + return txsetMeter(app, "self-check-passed") == passedBefore + 1; + }); + REQUIRE(Access::prepared(herder) == prepared); + REQUIRE(!Access::selfCheckHasFailed(herder)); + REQUIRE(txsetMeter(app, "self-check-failed") == failedBefore); + } + SECTION("selected early, caught before the proposal") + { + buildBroken(); + REQUIRE(Access::prepareEarly(herder, seq, closeTime)); + // The set is out (prepared, pushed) before the check has run. + REQUIRE(Access::prepared(herder) == broken); + herder.mBuiltTxSetOverrideForTesting = nullptr; + f.crankUntil(failed); + // Dropped: the trigger builds another one, checked before use. + REQUIRE(!Access::prepared(herder)); + REQUIRE(Access::selfCheckHasFailed(herder)); + auto const pulls = f.mPulls; + auto const proposed = stellarValueOf(Access::propose(herder, seq)); + REQUIRE(f.mPulls == pulls + 1); + REQUIRE(proposed.txSetHash != broken->getContentsHash()); + auto txSet = herder.getTxSet(proposed.txSetHash); + auto const* frame = std::get_if(&txSet); + REQUIRE(frame); + REQUIRE(*frame); + auto applicable = (*frame)->prepareForApply( + app, f.lm().getLastClosedLedgerHeader().header); + REQUIRE(applicable); + REQUIRE(txHashes(applicable) == hashesOf({nextA, plainB})); + // That build ran its check before returning: none in the + // background. + f.crankFor(std::chrono::milliseconds(50)); + REQUIRE(txsetMeter(app, "self-check-passed") == passedBefore); + } + SECTION("built before the trigger, caught before the proposal") + { + buildBroken(); + Access::prepareByBuilding(herder, seq, closeTime); + REQUIRE(Access::prepared(herder) == broken); + herder.mBuiltTxSetOverrideForTesting = nullptr; + f.crankUntil(failed); + REQUIRE(!Access::prepared(herder)); + REQUIRE(Access::selfCheckHasFailed(herder)); + } + SECTION("caught after the proposal") + { + buildBroken(); + REQUIRE(Access::prepareEarly(herder, seq, closeTime)); + // The trigger fires before the check is back: the set is proposed. + auto const proposed = stellarValueOf(Access::propose(herder, seq)); + REQUIRE(proposed.txSetHash == broken->getContentsHash()); + herder.mBuiltTxSetOverrideForTesting = nullptr; + f.crankUntil(failed); + REQUIRE(Access::selfCheckHasFailed(herder)); + // It cannot be recalled. Every node validates it when it applies + // the ledger, this one included: the ledger closes without it. + auto const skipped = applyCounter(app, "txset-skipped"); + f.externalize(broken); + REQUIRE(applyCounter(app, "txset-skipped") == skipped + 1); + REQUIRE(c.getLastSequenceNumber() == cSeq); + // From now on a set selected early is checked in line when it is + // completed, like a build: no check in the background. + auto nextNextA = paymentWithSeq(app, a, aSeq + 3); + f.mMempool = {nextA, nextNextA, plainB}; + f.closeWithEarlySelection(makeUnvalidatedTxSet(app, {nextA})); + f.mMempool = {nextNextA, plainB}; + REQUIRE(Access::prepareEarly(herder, f.nextSeq(), f.nextCloseTime())); + REQUIRE(Access::prepared(herder)); + { + auto applicable = Access::prepared(herder)->prepareForApply( + app, f.lm().getLastClosedLedgerHeader().header); + REQUIRE(applicable); + REQUIRE(txHashes(applicable) == hashesOf({nextNextA, plainB})); + } + f.crankFor(std::chrono::milliseconds(50)); + REQUIRE(txsetMeter(app, "self-check-passed") == passedBefore); + REQUIRE(txsetMeter(app, "self-check-failed") == failedBefore + 1); + Access::discardPrepared(herder); + // The build that takes its place checks its own result. + f.mMempool = {nextNextA, plainB}; + REQUIRE(f.build() == hashesOf({nextNextA, plainB})); + } +} + +TEST_CASE("candidates of one account in both phases", + "[herder][txset][prevalidation][early-txset][soroban]") +{ + PrevalidationFixture f; + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = *f.mA; + auto& b = *f.mB; + auto const aSeq = a.getLastSequenceNumber(); + auto both = app.getRoot()->create( + "S", app.getLedgerManager().getLastMinBalance(2) + 1'000'000'000); + auto const sSeq = both.getLastSequenceNumber(); + SorobanResources resources; + resources.instructions = 1'000'000; + resources.diskReadBytes = 1000; + resources.writeBytes = 1000; + auto plainB = paymentWithSeq(app, b, b.getLastSequenceNumber() + 1); + auto classic = paymentWithSeq(app, both, sSeq + 1); + auto soroban = + createUploadWasmTx(app, both, 100, DEFAULT_TEST_RESOURCE_FEE, resources, + std::nullopt, 0, std::nullopt, sSeq + 1); + auto laterClassic = paymentWithSeq(app, both, sSeq + 2); + REQUIRE(soroban->getSeqNum() == classic->getSeqNum()); + + SECTION("a build keeps one of them") + { + SECTION("the same sequence number: the classic one") + { + f.mMempool = {soroban, classic, plainB}; + REQUIRE(f.build() == hashesOf({classic, plainB})); + } + SECTION("the Soroban one comes first in the account's sequence") + { + f.mMempool = {laterClassic, soroban, plainB}; + REQUIRE(f.build() == hashesOf({soroban, plainB})); + } + SECTION("the classic one comes first in the account's sequence") + { + auto laterSoroban = createUploadWasmTx( + app, both, 100, DEFAULT_TEST_RESOURCE_FEE, resources, + std::nullopt, 0, std::nullopt, sSeq + 2); + f.mMempool = {laterSoroban, classic, plainB}; + REQUIRE(f.build() == hashesOf({classic, plainB})); + } + } + SECTION("so does a set selected early") + { + auto closingA = paymentWithSeq(app, a, aSeq + 1); + f.mMempool = {closingA, soroban, classic, plainB}; + f.closeWithEarlySelection(makeUnvalidatedTxSet(app, {closingA})); + REQUIRE(Access::prepareEarly(herder, f.nextSeq(), f.nextCloseTime())); + auto applicable = Access::prepared(herder)->prepareForApply( + app, f.lm().getLastClosedLedgerHeader().header); + REQUIRE(applicable); + REQUIRE(txHashes(applicable) == hashesOf({classic, plainB})); + REQUIRE(applicable->checkValid(app, 1, 1)); + } +} + +TEST_CASE("the limits a tx set is selected within", + "[herder][txset][early-txset]") +{ + PrevalidationFixture f; + auto& app = *f.mApp; + auto const limits = TxSetBuildLimits::forLCL(app); + REQUIRE(limits.mSorobanConfig); + REQUIRE(limits.sameAs(limits)); + REQUIRE(limits.maxTxSetSizeOps() == f.lm().getLastMaxTxSetSizeOps()); + REQUIRE(limits.mLclHeader == f.lm().getLastClosedLedgerHeader().header); + + auto differs = [&](TxSetBuildLimits const& other) { + return !limits.sameAs(other) && !other.sameAs(limits); + }; + auto withHeader = [&](std::function change) { + auto other = limits; + change(other.mLclHeader); + return other; + }; + // What selection reads of the header ... + REQUIRE(differs(withHeader([](LedgerHeader& h) { h.ledgerVersion -= 1; }))); + REQUIRE(differs(withHeader([](LedgerHeader& h) { h.baseFee += 1; }))); + REQUIRE(differs(withHeader([](LedgerHeader& h) { h.baseReserve += 1; }))); + REQUIRE(differs(withHeader([](LedgerHeader& h) { h.maxTxSetSize += 1; }))); + // ... and what it does not: every ledger changes these. + REQUIRE(limits.sameAs(withHeader([](LedgerHeader& h) { + h.ledgerSeq += 1; + h.feePool += 100; + h.idPool += 1; + h.scpValue.closeTime += 1; + h.previousLedgerHash = sha256("another ledger"); + h.bucketListHash = sha256("another bucket list"); + h.txSetResultHash = sha256("other results"); + }))); + + // The Soroban limits of the ledger. + auto withoutSoroban = limits; + withoutSoroban.mSorobanConfig = nullptr; + REQUIRE(differs(withoutSoroban)); + auto withSoroban = [&](SorobanNetworkConfig& config, + std::function change) { + config = *limits.mSorobanConfig; + change(config); + auto other = limits; + other.mSorobanConfig = &config; + return other; + }; + SorobanNetworkConfig config = *limits.mSorobanConfig; + REQUIRE(limits.sameAs(withSoroban(config, [](SorobanNetworkConfig&) {}))); + REQUIRE(differs(withSoroban( + config, [](SorobanNetworkConfig& c) { c.mLedgerMaxTxCount += 1; }))); + REQUIRE(differs(withSoroban(config, [](SorobanNetworkConfig& c) { + c.mLedgerMaxInstructions += 1; + }))); + REQUIRE(differs(withSoroban(config, [](SorobanNetworkConfig& c) { + c.mLedgerMaxDependentTxClusters += 1; + }))); + REQUIRE(differs(withSoroban(config, [](SorobanNetworkConfig& c) { + c.mLedgerMaxTransactionsSizeBytes += 1; + }))); + REQUIRE(differs(withSoroban(config, [](SorobanNetworkConfig& c) { + c.mLedgerMaxDiskReadEntries += 1; + }))); +} + +TEST_CASE("the upgrades due are part of the value the leader executes early", + "[herder][txset][speculative-apply][early-txset]") +{ + PrevalidationFixture f(/*herderSpeculates=*/true); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = *f.mA; + f.mMempool = {paymentWithSeq(app, a, a.getLastSequenceNumber() + 1)}; + Upgrades::UpgradeParameters params; + params.mUpgradeTime = VirtualClock::system_time_point{}; + params.mBaseFee = 250; + herder.setUpgrades(params); + auto const seq = f.nextSeq(); + auto const started = applyCounter(app, "speculation-started"); + + Access::prepareByBuilding(herder, seq, f.nextCloseTime()); + REQUIRE(Access::prepared(herder)); + REQUIRE(applyCounter(app, "speculation-started") == started + 1); + while (f.lm().isSpeculativelyApplying()) + { + f.mClock.crank(true); + } + auto const proposal = Access::propose(herder, seq); + auto const proposed = stellarValueOf(proposal); + REQUIRE(proposed.upgrades.size() == 1); + REQUIRE(Access::speculativeValue(herder)); + REQUIRE(Access::speculativeValue(herder)->second == proposal->getValue()); +} + +// Executing a ledger starts with the full validation of its set: the same +// check every other node runs when it applies the ledger. When the leader's +// own execution of its prepared set is done before the trigger, an invalid +// set is not proposed. +TEST_CASE("a prepared set that the leader finds invalid when executing it " + "is not proposed", + "[herder][txset][speculative-apply][early-txset]") +{ + PrevalidationFixture f(/*herderSpeculates=*/true); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = *f.mA; + auto& b = *f.mB; + auto& c = *f.mC; + auto nextA = paymentWithSeq(app, a, a.getLastSequenceNumber() + 1); + auto plainB = paymentWithSeq(app, b, b.getLastSequenceNumber() + 1); + f.mMempool = {nextA, plainB}; + auto const seq = f.nextSeq(); + auto const failedBefore = txsetMeter(app, "self-check-failed"); + auto const passedBefore = txsetMeter(app, "self-check-passed"); + auto const skippedBefore = applyCounter(app, "txset-skipped"); + + // What the node "builds" has the shape of its selection (two classic + // transactions) and passes the structural check, but one of its + // transactions is not valid: only validating the transactions shows it. + auto invalid = makeUnvalidatedTxSet( + app, {nextA, paymentWithSeq(app, c, c.getLastSequenceNumber() + 5)}); + herder.mBuiltTxSetOverrideForTesting = [&](TxSetXDRFrameConstPtr const&) { + return invalid; + }; + Access::prepareByBuilding(herder, seq, f.nextCloseTime()); + herder.mBuiltTxSetOverrideForTesting = nullptr; + REQUIRE(Access::prepared(herder) == invalid); + + f.crankUntil([&]() { + return txsetMeter(app, "self-check-failed") == failedBefore + 1; + }); + // The verdict is the execution's: a node that executes its own value + // leaves the re-read of the written set to that execution, so no check + // ran in the background beside it. + f.crankFor(std::chrono::milliseconds(50)); + REQUIRE(txsetMeter(app, "self-check-passed") == passedBefore); + // The apply thread skipped the set, and the herder dropped it. + REQUIRE(applyCounter(app, "txset-skipped") == skippedBefore + 1); + REQUIRE(!Access::prepared(herder)); + REQUIRE(Access::selfCheckHasFailed(herder)); + REQUIRE(f.lm().getLastClosedLedgerNum() == seq - 1); + // The trigger builds a set of its own. + auto const proposed = stellarValueOf(Access::propose(herder, seq)); + REQUIRE(proposed.txSetHash != invalid->getContentsHash()); + auto txSet = herder.getTxSet(proposed.txSetHash); + auto const* frame = std::get_if(&txSet); + REQUIRE(frame); + REQUIRE(*frame); + auto applicable = (*frame)->prepareForApply( + app, f.lm().getLastClosedLedgerHeader().header); + REQUIRE(applicable); + REQUIRE(txHashes(applicable) == hashesOf({nextA, plainB})); +} + +// The check of the written set (TxSetSelfCheck) reads the set back and +// validates what it read. A node that executes the value it votes for does +// that on its apply thread, on the same bytes, as soon as it proposes: there +// the check is not run a second time in the background. +TEST_CASE("the check of a written set is left to the execution of the vote " + "where the node executes it", + "[herder][txset][speculative-apply][early-txset]") +{ + auto run = [](bool herderSpeculates) { + PrevalidationFixture f(herderSpeculates); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = *f.mA; + f.mMempool = {paymentWithSeq(app, a, a.getLastSequenceNumber() + 1)}; + auto const passedBefore = txsetMeter(app, "self-check-passed"); + auto const failedBefore = txsetMeter(app, "self-check-failed"); + auto const started = applyCounter(app, "speculation-started"); + Access::prepareByBuilding(herder, f.nextSeq(), f.nextCloseTime()); + REQUIRE(Access::prepared(herder)); + if (herderSpeculates) + { + // The apply thread has the ledger: it validates the set in + // full, starting with reading it back. + REQUIRE(applyCounter(app, "speculation-started") == started + 1); + while (f.lm().isSpeculativelyApplying()) + { + f.mClock.crank(true); + } + // Also once the node has emitted its statement: the apply + // thread was handed this very set. + Access::statementEmitted(herder); + f.crankFor(std::chrono::milliseconds(50)); + REQUIRE(txsetMeter(app, "self-check-passed") == passedBefore); + } + else + { + REQUIRE(applyCounter(app, "speculation-started") == started); + f.crankUntil([&]() { + return txsetMeter(app, "self-check-passed") == passedBefore + 1; + }); + } + REQUIRE(txsetMeter(app, "self-check-failed") == failedBefore); + REQUIRE(Access::prepared(herder)); + REQUIRE(!Access::selfCheckHasFailed(herder)); + }; + SECTION("a node that executes its vote") + { + run(true); + } + SECTION("a node that does not") + { + run(false); + } + SECTION("a node that executes its votes, but not this set") + { + // The node built a set and then emits a statement without its + // apply thread having been handed that set (here: the trigger + // proposes another one; a node that is not tracking, or is still + // applying, gets there too): the check runs in the background + // after all. + PrevalidationFixture f(/*herderSpeculates=*/true); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = *f.mA; + f.mMempool = {paymentWithSeq(app, a, a.getLastSequenceNumber() + 1)}; + auto const passedBefore = txsetMeter(app, "self-check-passed"); + herder.mProposedTxSetOverride = + [&](LedgerHeaderHistoryEntry const& lcl) { + return TxSetXDRFrame::makeEmpty(lcl); + }; + // (with the override the prepared set is not executed early) + Access::prepareByBuilding(herder, f.nextSeq(), f.nextCloseTime()); + herder.mProposedTxSetOverride = nullptr; + REQUIRE(Access::prepared(herder)); + REQUIRE(!Access::speculativeValue(herder)); + Access::statementEmitted(herder); + f.crankUntil([&]() { + return txsetMeter(app, "self-check-passed") == passedBefore + 1; + }); + // Once only. + Access::statementEmitted(herder); + f.crankFor(std::chrono::milliseconds(50)); + REQUIRE(txsetMeter(app, "self-check-passed") == passedBefore + 1); + } +} + +TEST_CASE("the application shuts down while a set is being checked or " + "selected", + "[herder][txset][prevalidation][early-txset]") +{ + auto f = std::make_unique(); + auto& app = *f->mApp; + auto& herder = f->herder(); + auto& a = *f->mA; + auto const aSeq = a.getLastSequenceNumber(); + auto closingA = paymentWithSeq(app, a, aSeq + 1); + auto nextA = paymentWithSeq(app, a, aSeq + 2); + + SECTION("the self-check of a set built before the trigger") + { + f->mMempool = {closingA}; + Access::prepareByBuilding(herder, f->nextSeq(), f->nextCloseTime()); + REQUIRE(Access::prepared(herder)); + f.reset(); + } + SECTION("the selection") + { + f->mMempool = {closingA}; + for (int i = 0; i < 400; ++i) + { + f->mMempool.push_back(nextA); + } + // The job has started; its tasks may be anywhere. + f->selectEarly(makeUnvalidatedTxSet(app, {closingA}), /*wait=*/false); + f.reset(); + } + SECTION("the self-check of a set selected early") + { + f->mMempool = {closingA, nextA}; + f->closeWithEarlySelection(makeUnvalidatedTxSet(app, {closingA})); + REQUIRE(Access::prepareEarly(herder, f->nextSeq(), f->nextCloseTime())); + f.reset(); + } +} + +// --------------------------------------------------------------------------- +// When a set selected early is used. +// --------------------------------------------------------------------------- + +TEST_CASE("the verdict of the leader on its own set", + "[herder][txset][speculative-apply][early-txset]") +{ + PrevalidationFixture f(/*herderSpeculates=*/true); + auto& app = *f.mApp; + auto& herder = f.herder(); + auto& a = *f.mA; + auto& b = *f.mB; + auto& c = *f.mC; + auto const aSeq = a.getLastSequenceNumber(); + auto const failedBefore = txsetMeter(app, "self-check-failed"); + auto const skippedBefore = applyCounter(app, "txset-skipped"); + auto vote = [&](uint32_t seq, StellarValue const& value) { + SCPEnvelope envelope; + envelope.statement.slotIndex = seq; + envelope.statement.pledges.type(SCP_ST_PREPARE); + envelope.statement.pledges.prepare().ballot.counter = 1; + envelope.statement.pledges.prepare().ballot.value = + xdr::xdr_to_opaque(value); + Access::emitted(herder, envelope); + }; + auto executed = [&]() { + while (f.lm().isSpeculativelyApplying()) + { + f.mClock.crank(true); + } + f.crankFor(std::chrono::milliseconds(20)); + }; + + SECTION("a set selected early that is invalid is dropped before the " + "proposal") + { + auto closingA = paymentWithSeq(app, a, aSeq + 1); + auto nextA = paymentWithSeq(app, a, aSeq + 2); + auto plainB = paymentWithSeq(app, b, b.getLastSequenceNumber() + 1); + f.mMempool = {closingA, nextA, plainB}; + f.closeWithEarlySelection(makeUnvalidatedTxSet(app, {closingA})); + // Same shape as the selection, one transaction not valid. + auto invalid = makeUnvalidatedTxSet( + app, + {nextA, paymentWithSeq(app, c, c.getLastSequenceNumber() + 5)}); + herder.mBuiltTxSetOverrideForTesting = + [&](TxSetXDRFrameConstPtr const&) { return invalid; }; + REQUIRE(Access::prepareEarly(herder, f.nextSeq(), f.nextCloseTime())); + herder.mBuiltTxSetOverrideForTesting = nullptr; + REQUIRE(Access::prepared(herder) == invalid); + f.crankUntil([&]() { + return txsetMeter(app, "self-check-failed") == failedBefore + 1; + }); + REQUIRE(applyCounter(app, "txset-skipped") == skippedBefore + 1); + REQUIRE(!Access::prepared(herder)); + REQUIRE(Access::selfCheckHasFailed(herder)); + } + SECTION("a verdict that comes after the proposal is recorded") + { + auto nextA = paymentWithSeq(app, a, aSeq + 1); + auto plainB = paymentWithSeq(app, b, b.getLastSequenceNumber() + 1); + f.mMempool = {nextA, plainB}; + auto invalid = makeUnvalidatedTxSet( + app, + {nextA, paymentWithSeq(app, c, c.getLastSequenceNumber() + 5)}); + herder.mBuiltTxSetOverrideForTesting = + [&](TxSetXDRFrameConstPtr const&) { return invalid; }; + auto const seq = f.nextSeq(); + Access::prepareByBuilding(herder, seq, f.nextCloseTime()); + herder.mBuiltTxSetOverrideForTesting = nullptr; + // The trigger fires before the execution has got anywhere. + auto const proposed = stellarValueOf(Access::propose(herder, seq)); + REQUIRE(proposed.txSetHash == invalid->getContentsHash()); + f.crankUntil([&]() { + return txsetMeter(app, "self-check-failed") == failedBefore + 1; + }); + REQUIRE(Access::selfCheckHasFailed(herder)); + // The ledger closes without applying the set, here as everywhere. + f.externalize(invalid); + REQUIRE(f.lm().getLastClosedLedgerNum() == seq); + REQUIRE(a.getLastSequenceNumber() == aSeq); + } + SECTION("an invalid set of another node says nothing about this node") + { + auto bad = paymentWithSeq(app, c, c.getLastSequenceNumber() + 5); + auto foreign = makeUnvalidatedTxSet(app, {bad}); + auto const seq = f.nextSeq(); + herder.getPendingEnvelopes().putTxSet(foreign->getContentsHash(), seq, + foreign); + vote(seq, f.valueFor(foreign)); + executed(); + REQUIRE(applyCounter(app, "txset-skipped") == skippedBefore + 1); + REQUIRE(txsetMeter(app, "self-check-failed") == failedBefore); + REQUIRE(!Access::selfCheckHasFailed(herder)); + } + SECTION("nor does this node's set under a close time it was not built " + "for") + { + // Valid for the close time the node built it for; another leader + // puts it forward for a later one, at which its time-bounded + // transaction has expired. + auto const builtFor = f.nextCloseTime(); + PreconditionsV2 cond; + cond.timeBounds.activate().maxTime = builtFor.toApplyTime().timePoint(); + auto bounded = transactionWithV2Precondition(app, b, 1, 100, cond); + f.mMempool = {bounded}; + auto const seq = f.nextSeq(); + Access::prepareByBuilding(herder, seq, builtFor); + auto own = Access::prepared(herder); + REQUIRE(own); + REQUIRE(own->sizeTxTotal() == 1); + executed(); + REQUIRE(applyCounter(app, "txset-skipped") == skippedBefore); + herder.getPendingEnvelopes().putTxSet(own->getContentsHash(), seq, own); + auto const later = ConsensusTime::fromApplyTime( + ApplyTime::fromTimePoint(builtFor.toApplyTime().timePoint() + 5)); + vote(seq, herder.makeStellarValue(own->getContentsHash(), later, + emptyUpgradeSteps, f.mCfg.NODE_SEED)); + executed(); + REQUIRE(applyCounter(app, "txset-skipped") == skippedBefore + 1); + REQUIRE(txsetMeter(app, "self-check-failed") == failedBefore); + REQUIRE(!Access::selfCheckHasFailed(herder)); + REQUIRE(Access::prepared(herder) == own); + } +} + +// What the selection itself can get wrong is checked before the set leaves +// the builder, also when the check of the written set comes later. +TEST_CASE("a build checks its selection before the set leaves it", + "[herder][txset][early-txset][soroban]") +{ + PrevalidationFixture f; + auto& app = *f.mApp; + auto both = app.getRoot()->create( + "S", app.getLedgerManager().getLastMinBalance(2) + 1'000'000'000); + auto const sSeq = both.getLastSequenceNumber(); + SorobanResources resources; + resources.instructions = 1'000'000; + resources.diskReadBytes = 1000; + resources.writeBytes = 1000; + TransactionFrameBasePtr classic = paymentWithSeq(app, both, sSeq + 1); + TransactionFrameBasePtr soroban = + createUploadWasmTx(app, both, 100, DEFAULT_TEST_RESOURCE_FEE, resources, + std::nullopt, 0, std::nullopt, sSeq + 1); + // Candidates as the herder must not hand them over: one account in both + // phases. + PerPhaseTransactionList phases{{classic}, {soroban}}; + PerPhaseTransactionList invalid(2); + bool handedOut = false; + auto onReady = [&](TxSetXDRFrameConstPtr const&) { handedOut = true; }; + + SECTION("with the check of the written set deferred") + { + std::shared_ptr check; + REQUIRE_THROWS_WITH( + makeTxSetFromTransactions( + phases, app, ApplyTimeOffset::fromSeconds(1), invalid, false, + {}, false, onReady, {}, nullptr, &check), + Catch::Matchers::Contains("MULTIPLE_TXS_PER_SOURCE_ACCOUNT")); + // Nothing was handed to the overlay or the peers. + REQUIRE(!handedOut); + REQUIRE(!check); + } + SECTION("with the check before returning, as always") + { + REQUIRE_THROWS_WITH( + makeTxSetFromTransactions(phases, app, + ApplyTimeOffset::fromSeconds(1), invalid, + false, {}, false, onReady), + Catch::Matchers::Contains("MULTIPLE_TXS_PER_SOURCE_ACCOUNT")); + } +} + +// Before tx sets are validated at apply, the other nodes validate a set +// before they vote for it, and the leader vouches for its own: there the +// builder checks the written set before it returns, as it always did. +TEST_CASE("the check of the written set is deferred only where sets are " + "validated at apply", + "[herder][txset][early-txset]") +{ + auto const deferred = GENERATE(false, true); + PrevalidationFixture f(/*herderSpeculates=*/false, std::nullopt, + deferred ? GATE_PROTOCOL : GATE_PROTOCOL - 1); + auto& app = *f.mApp; + auto& a = *f.mA; + REQUIRE(protocolValidatesTxSetsAtApply( + f.lm().getLastClosedLedgerHeader().header.ledgerVersion) == + deferred); + auto const passedBefore = txsetMeter(app, "self-check-passed"); + f.mMempool = {paymentWithSeq(app, a, a.getLastSequenceNumber() + 1)}; + Access::prepareByBuilding(f.herder(), f.nextSeq(), f.nextCloseTime()); + REQUIRE(Access::prepared(f.herder())); + if (deferred) + { + f.crankUntil([&]() { + return txsetMeter(app, "self-check-passed") == passedBefore + 1; + }); + } + else + { + f.crankFor(std::chrono::milliseconds(50)); + REQUIRE(txsetMeter(app, "self-check-passed") == passedBefore); + } +} diff --git a/src/ledger/LedgerManager.h b/src/ledger/LedgerManager.h index 74d18659a..ad551c68c 100644 --- a/src/ledger/LedgerManager.h +++ b/src/ledger/LedgerManager.h @@ -276,6 +276,12 @@ class LedgerManager // instead. virtual std::unique_ptr getLCLView() const = 0; + // The immutable state of the last closed ledger itself, for a caller + // that reads it from other threads, each through a view of its own, and + // needs it to stay the same ledger while the LCL moves on. Callable from + // any thread but the apply thread. + virtual ImmutableLedgerDataPtr getLastClosedLedgerState() const = 0; + // Create a copy of the canonical ledger snapshot at the current moment in // apply flow. // The returned snapshot is *not* thread-safe, so every thread used during diff --git a/src/ledger/LedgerManagerImpl.cpp b/src/ledger/LedgerManagerImpl.cpp index abf426eaa..be06f6326 100644 --- a/src/ledger/LedgerManagerImpl.cpp +++ b/src/ledger/LedgerManagerImpl.cpp @@ -2975,6 +2975,15 @@ LedgerManagerImpl::getLCLView() const return std::make_unique(copyImmutableLedgerView()); } +ImmutableLedgerDataPtr +LedgerManagerImpl::getLastClosedLedgerState() const +{ + releaseAssert(!mApp.threadIsType(Application::ThreadType::APPLY)); + SharedLockShared guard(mLastClosedLedgerStateMutex); + releaseAssert(mLastClosedLedgerState); + return mLastClosedLedgerState; +} + ApplyLedgerView LedgerManagerImpl::copyApplyLedgerView() const { diff --git a/src/ledger/LedgerManagerImpl.h b/src/ledger/LedgerManagerImpl.h index fbc81def6..fb4f59a5d 100644 --- a/src/ledger/LedgerManagerImpl.h +++ b/src/ledger/LedgerManagerImpl.h @@ -731,6 +731,7 @@ class LedgerManagerImpl : public LedgerManager public: std::unique_ptr getLCLView() const override; + ImmutableLedgerDataPtr getLastClosedLedgerState() const override; ApplyLedgerView copyApplyLedgerView() const override; void syncWithLCLView( std::unique_ptr& ledgerView) const override; diff --git a/src/main/CommandHandler.cpp b/src/main/CommandHandler.cpp index 4582b9dc7..4410eb390 100644 --- a/src/main/CommandHandler.cpp +++ b/src/main/CommandHandler.cpp @@ -1161,6 +1161,10 @@ CommandHandler::generateLoad(std::string const& params, std::string& retStr) parseOptionalParam(map, "maxfeerate"); cfg.skipLowFeeTxs = parseOptionalParamOrDefault(map, "skiplowfeetxs", false); + // Ledgers to wait, after the last submission, for the load to be + // applied; 0 keeps the mode's default. + cfg.completionTimeoutLedgers = + parseOptionalParamOrDefault(map, "completiontimeout", 0); if (cfg.isSoroban()) { diff --git a/src/main/Config.cpp b/src/main/Config.cpp index 50e994f44..7eac86487 100644 --- a/src/main/Config.cpp +++ b/src/main/Config.cpp @@ -175,6 +175,8 @@ Config::Config() : NODE_SEED(SecretKey::random()) BACKGROUND_OVERLAY_PROCESSING = true; PARALLEL_LEDGER_APPLY = true; EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY = true; + EXPERIMENTAL_EARLY_TX_SET_SELECTION = true; + EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT = 10; EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = false; DISABLE_SOROBAN_METRICS_FOR_TESTING = false; DISABLE_TX_META_FOR_TESTING = false; @@ -1218,6 +1220,15 @@ Config::processConfig(std::shared_ptr t) [&]() { EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY = readBool(item); }}, + {"EXPERIMENTAL_EARLY_TX_SET_SELECTION", + [&]() { + EXPERIMENTAL_EARLY_TX_SET_SELECTION = readBool(item); + }}, + {"EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT", + [&]() { + EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT = + readInt(item, 0, 1000); + }}, {"EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD", [&]() { EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = readBool(item); @@ -1243,9 +1254,7 @@ Config::processConfig(std::shared_ptr t) readBool(item); }}, {"TX_FLOOD_TARGET_LEADERS", - [&]() { - TX_FLOOD_TARGET_LEADERS = readInt(item); - }}, + [&]() { TX_FLOOD_TARGET_LEADERS = readInt(item); }}, {"TX_FLOOD_LEGACY_ADVERTS", [&]() { TX_FLOOD_LEGACY_ADVERTS = readBool(item); }}, {"BATCH_EXECUTOR_TASKS", @@ -2401,6 +2410,11 @@ Config::logBasicInfo() const "EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY=" "{}", EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY ? "true" : "false"); + LOG_INFO(DEFAULT_LOG, + "EXPERIMENTAL_EARLY_TX_SET_SELECTION={}, " + "EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT={}", + EXPERIMENTAL_EARLY_TX_SET_SELECTION ? "true" : "false", + EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT); } void diff --git a/src/main/Config.h b/src/main/Config.h index 623f760dd..b9fd6d0f0 100644 --- a/src/main/Config.h +++ b/src/main/Config.h @@ -540,6 +540,22 @@ class Config : public std::enable_shared_from_this // APPLY_TIME_TX_SET_VALIDATION_PROTOCOL_VERSION on. bool EXPERIMENTAL_SPECULATIVE_LEDGER_APPLY; + // The node that expects to lead the next ledger selects its transaction + // set while the current ledger is still in its ballot, from the current + // ledger's set and the last closed state, and completes it when the + // current ledger has closed: every selected transaction is validated + // against the closed ledger and the set is topped up from the mempool + // (see herder/EarlyTxSetSelection.h). Without it the set is built when + // the current ledger has closed. Only from + // APPLY_TIME_TX_SET_VALIDATION_PROTOCOL_VERSION on. + bool EXPERIMENTAL_EARLY_TX_SET_SELECTION; + + // How much more than a transaction set can hold a leader pulls from the + // mempool for a build, in percent of the set's capacity. A build whose + // set is short after validation pulls again, so the margin only decides + // how often that happens. + uint32_t EXPERIMENTAL_TX_SET_PULL_MARGIN_PERCENT; + // Allow downloading of transaction sets in parallel with SCP (experimental) bool EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD; diff --git a/src/simulation/LoadGenerator.cpp b/src/simulation/LoadGenerator.cpp index c763d764f..49d64847f 100644 --- a/src/simulation/LoadGenerator.cpp +++ b/src/simulation/LoadGenerator.cpp @@ -3,6 +3,7 @@ // of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 #include "simulation/LoadGenerator.h" +#include "crypto/KeyUtils.h" #include "database/Database.h" #include "herder/Herder.h" #include "ledger/LedgerManager.h" @@ -71,6 +72,16 @@ uint32_t const LoadGenerator::TX_SUBMIT_MAX_TRIES = 10; // After successfully submitting desired load, wait a bit to let it get into the // ledger. uint32_t const LoadGenerator::TIMEOUT_NUM_LEDGERS = 20; +uint32_t const LoadGenerator::DEFAULT_COMPLETION_TIMEOUT_NUM_LEDGERS = + LoadGenerator::TIMEOUT_NUM_LEDGERS; +// The payments of SAC_PAYMENT travel to the round leaders only, without an +// acknowledgement. A node's last few can be included much later than the +// rest (25 to 70 s were measured on a 90-validator network at 0.8 s per +// ledger, where 1 to 9 of 90 generators hit it per run): a leader dropped +// them from its build, and the origin sends them again only by its resend +// rules. They are applied in the end, so the run is complete, only late; +// the default wait of 20 ledgers reported such a run as failed. +uint32_t const LoadGenerator::SAC_PAYMENT_COMPLETION_TIMEOUT_NUM_LEDGERS = 150; // After successfully submitting desired load, wait for this many ledgers // without checking for account consistency. @@ -106,6 +117,29 @@ isMixedPregenMode(LoadGenMode mode) mode == LoadGenMode::MIXED_PREGEN_SOROSWAP_SWAP; } +LedgerKey +LoadGenerator::getNativeSACInstanceKey(Hash const& networkID) +{ + SCAddress contract(SC_ADDRESS_TYPE_CONTRACT); + contract.contractId() = getAssetContractID(networkID, makeNativeAsset()); + return makeContractInstanceKey(contract); +} + +SCAddress +LoadGenerator::getNativeSACPaymentDestination(NodeID const& node, + uint32_t ledgerNum, + uint64_t counter) +{ + // The name holds the generating node's key, so every generator of a + // network pays to destinations of its own; the ledger and the counter + // make the destinations of one generator distinct. + SCAddress dest(SC_ADDRESS_TYPE_CONTRACT); + dest.contractId() = + sha256(fmt::format("loadgen_native_sac_dest_{}_{}_{}", + KeyUtils::toStrKey(node), ledgerNum, counter)); + return dest; +} + LoadGenerator::LoadGenerator(Application& app) : mTxGenerator(app) , mApp(app) @@ -173,6 +207,14 @@ LoadGenerator::getMode(std::string const& mode) { return LoadGenMode::MIXED_PREGEN_SOROSWAP_SWAP; } + else if (mode == "sac_setup") + { + return LoadGenMode::SAC_SETUP; + } + else if (mode == "sac_payment") + { + return LoadGenMode::SAC_PAYMENT; + } else { throw std::runtime_error( @@ -345,6 +387,8 @@ LoadGenerator::reset() mSyntheticSoroswapState.reset(); mSyntheticSACDestCounter = 0; mSyntheticSoroswapSwapCounter = 0; + mNativeSACInstance.reset(); + mNativeSACDestCounter = 0; mWaitTillCompleteForLedgers = 0; mFailed = false; mStarted = false; @@ -401,8 +445,48 @@ LoadGenerator::start(GeneratedLoadConfig& cfg) cfg.txRate = 1; } + if (cfg.modeUsesNativeSAC()) + { + // These modes exist to exercise apply. In overlay only mode nothing + // is applied, so the instance would never appear and the payments + // would never complete. + if (mApp.getRunInOverlayOnlyMode()) + { + reset(); + throw std::runtime_error( + "SAC_SETUP / SAC_PAYMENT need transactions to be applied, " + "they can not run in overlay only mode"); + } + + auto instanceKey = getNativeSACInstanceKey(mApp.getNetworkID()); + bool instanceExists = static_cast( + mApp.getLedgerManager().getLCLView()->load(instanceKey)); + if (cfg.mode == LoadGenMode::SAC_SETUP) + { + // A single transaction creates the instance. If it is already + // there (e.g. another node ran the setup), there is nothing to do. + cfg.nTxs = instanceExists ? 0 : 1; + cfg.skipLowFeeTxs = false; + cfg.spikeInterval = std::chrono::seconds(0); + cfg.spikeSize = 0; + } + else if (!instanceExists) + { + reset(); + throw std::runtime_error( + "Before running MODE::SAC_PAYMENT, please run MODE::SAC_SETUP " + "to create the native asset contract first."); + } + + TxGenerator::ContractInstance instance; + instance.readOnlyKeys.emplace_back(instanceKey); + instance.contractID = instanceKey.contractData().contract; + mNativeSACInstance = instance; + } + // Setup config for soroban modes - if (cfg.isSoroban() && cfg.mode != LoadGenMode::SOROBAN_UPLOAD) + if (cfg.isSoroban() && cfg.mode != LoadGenMode::SOROBAN_UPLOAD && + !cfg.modeUsesNativeSAC()) { auto& sorobanLoadCfg = cfg.getMutSorobanConfig(); sorobanLoadCfg.nWasms = 1; @@ -715,6 +799,12 @@ GeneratedLoadConfig::getStatus() const case LoadGenMode::MIXED_PREGEN_SOROSWAP_SWAP: modeStr = "mixed_pregen_soroswap_swap"; break; + case LoadGenMode::SAC_SETUP: + modeStr = "sac_setup"; + break; + case LoadGenMode::SAC_PAYMENT: + modeStr = "sac_payment"; + break; } ret["mode"] = modeStr; @@ -947,6 +1037,19 @@ LoadGenerator::generateLoad(GeneratedLoadConfig cfg) return readTransactionFromFile(cfg); }; break; + case LoadGenMode::SAC_SETUP: + generateTx = [&]() { + return mTxGenerator.createSACTransaction( + ledgerNum, sourceAccountId, makeNativeAsset(), + cfg.maxGeneratedFeeRate); + }; + break; + case LoadGenMode::SAC_PAYMENT: + generateTx = [&]() { + return createNativeSACPaymentTransaction(ledgerNum, + sourceAccountId, cfg); + }; + break; } try @@ -1231,6 +1334,23 @@ LoadGenerator::createSyntheticSorobanTransaction(uint32_t ledgerNum, } } +std::pair +LoadGenerator::createNativeSACPaymentTransaction(uint32_t ledgerNum, + uint64_t sourceAccountId, + GeneratedLoadConfig const& cfg) +{ + releaseAssert(mNativeSACInstance); + // A payment writes only its source account and the balance entry of a + // destination no other transaction uses, so payments from different + // accounts never conflict. + auto dest = getNativeSACPaymentDestination( + mApp.getConfig().NODE_SEED.getPublicKey(), ledgerNum, + mNativeSACDestCounter++); + return mTxGenerator.invokeSACPayment(ledgerNum, sourceAccountId, dest, + *mNativeSACInstance, /* amount */ 100, + cfg.maxGeneratedFeeRate); +} + std::pair LoadGenerator::createUploadWasmTransaction(GeneratedLoadConfig const& cfg, uint32_t ledgerNum, @@ -1307,6 +1427,19 @@ LoadGenerator::checkSorobanStateSynced(Application& app, std::vector result; auto ledgerView = mApp.getLedgerManager().getLCLView(); + if (cfg.modeUsesNativeSAC()) + { + // SAC_SETUP is done once the instance is on the ledger. The payments + // do not create persistent loadgen state; their source accounts are + // checked by checkAccountSynced. + auto instanceKey = getNativeSACInstanceKey(mApp.getNetworkID()); + if (!ledgerView->load(instanceKey)) + { + result.emplace_back(instanceKey); + } + return result; + } + for (auto const& lk : mContractInstanceKeys) { if (!ledgerView->load(lk)) @@ -1382,6 +1515,18 @@ LoadGenerator::checkMinimumSorobanSuccess(GeneratedLoadConfig const& cfg) return (nSuccessful * 100) / nTxns >= cfg.getMinSorobanPercentSuccess(); } +uint32_t +LoadGenerator::completionTimeoutLedgers(GeneratedLoadConfig const& cfg) +{ + if (cfg.completionTimeoutLedgers != 0) + { + return cfg.completionTimeoutLedgers; + } + return cfg.mode == LoadGenMode::SAC_PAYMENT + ? SAC_PAYMENT_COMPLETION_TIMEOUT_NUM_LEDGERS + : DEFAULT_COMPLETION_TIMEOUT_NUM_LEDGERS; +} + void LoadGenerator::waitTillComplete(GeneratedLoadConfig cfg) { @@ -1440,7 +1585,7 @@ LoadGenerator::waitTillComplete(GeneratedLoadConfig cfg) // If we have an inconsistency, reset the timer and wait for another ledger else if (!classicIsDone || !sorobanIsDone) { - if (++mWaitTillCompleteForLedgers >= TIMEOUT_NUM_LEDGERS) + if (++mWaitTillCompleteForLedgers >= completionTimeoutLedgers(cfg)) { emitFailure(!sorobanIsDone); return; @@ -1634,6 +1779,12 @@ LoadGenerator::execute(TransactionFrameBasePtr txf, LoadGenMode mode, txm.mNativePaymentBytes.Mark(msgBytes); } break; + case LoadGenMode::SAC_SETUP: + txm.mSorobanSetupInvokeTxs.Mark(); + break; + case LoadGenMode::SAC_PAYMENT: + txm.mSorobanInvokeTxs.Mark(); + break; } txm.mTxnAttempted.Mark(); @@ -1951,7 +2102,7 @@ GeneratedLoadConfig::isSoroban() const mode == LoadGenMode::SOROBAN_UPLOAD || mode == LoadGenMode::SOROBAN_UPGRADE_SETUP || mode == LoadGenMode::SOROBAN_CREATE_UPGRADE || - mode == LoadGenMode::MIXED_CLASSIC_SOROBAN; + mode == LoadGenMode::MIXED_CLASSIC_SOROBAN || modeUsesNativeSAC(); } bool @@ -1968,7 +2119,8 @@ GeneratedLoadConfig::isLoad() const mode == LoadGenMode::SOROBAN_INVOKE || mode == LoadGenMode::SOROBAN_CREATE_UPGRADE || mode == LoadGenMode::MIXED_CLASSIC_SOROBAN || - mode == LoadGenMode::PAY_PREGENERATED || modeMixesPregen(); + mode == LoadGenMode::PAY_PREGENERATED || modeMixesPregen() || + modeUsesNativeSAC(); } bool @@ -1984,6 +2136,12 @@ GeneratedLoadConfig::modeMixesPregen() const return isMixedPregenMode(mode); } +bool +GeneratedLoadConfig::modeUsesNativeSAC() const +{ + return mode == LoadGenMode::SAC_SETUP || mode == LoadGenMode::SAC_PAYMENT; +} + bool GeneratedLoadConfig::modeSetsUpInvoke() const { diff --git a/src/simulation/LoadGenerator.h b/src/simulation/LoadGenerator.h index 97ca72ecd..e357327ab 100644 --- a/src/simulation/LoadGenerator.h +++ b/src/simulation/LoadGenerator.h @@ -54,7 +54,17 @@ enum class LoadGenMode // is skipped so nothing is actually executed. MIXED_PREGEN_SAC_PAYMENT, MIXED_PREGEN_OZ_TOKEN_TRANSFER, - MIXED_PREGEN_SOROSWAP_SWAP + MIXED_PREGEN_SOROSWAP_SWAP, + // Create the Stellar Asset Contract instance of the native asset with one + // transaction, for SAC_PAYMENT. Completes once the instance is on the + // ledger; does nothing if it is already there. + SAC_SETUP, + // SAC payments of the native asset that are really applied: the same + // transaction shape as MIXED_PREGEN_SAC_PAYMENT, but against the instance + // SAC_SETUP created, so it needs apply (not overlay only mode). Each + // payment goes to a new contract address, so no two transactions write the + // same entry. + SAC_PAYMENT }; struct GeneratedLoadConfig @@ -135,6 +145,10 @@ struct GeneratedLoadConfig // True iff mode generates SOROBAN_UPLOAD load bool modeUploads() const; + // True iff mode works on the native asset's SAC instance (SAC_SETUP and + // SAC_PAYMENT). + bool modeUsesNativeSAC() const; + bool isDone() const; bool areTxsRemaining() const; Json::Value getStatus() const; @@ -158,6 +172,10 @@ struct GeneratedLoadConfig // to `TxQueueLimiter` limiting the operation count per ledger). Otherwise, // the load generation will fail after a couple of retries. bool skipLowFeeTxs = false; + // How many ledgers a completed submission waits for its transactions to + // be applied before the run is reported as failed. 0: the default of the + // mode (see LoadGenerator::completionTimeoutLedgers). + uint32_t completionTimeoutLedgers = 0; // Path to the pre-generated transactions file for PAY_PREGENERATED mode std::filesystem::path preloadedTransactionsFile; @@ -175,10 +193,29 @@ struct GeneratedLoadConfig class LoadGenerator { public: + // How long a run waits, after its last submission, for its + // transactions to be applied, unless its config says otherwise: 20 + // ledgers, and for SAC_PAYMENT, whose last payments can be included + // late, 150. + static uint32_t const DEFAULT_COMPLETION_TIMEOUT_NUM_LEDGERS; + static uint32_t const SAC_PAYMENT_COMPLETION_TIMEOUT_NUM_LEDGERS; + // The ledgers a run of `cfg` waits. + static uint32_t completionTimeoutLedgers(GeneratedLoadConfig const& cfg); + LoadGenerator(Application& app); static LoadGenMode getMode(std::string const& mode); + // Ledger key of the native asset's SAC instance (SAC_SETUP creates it, + // SAC_PAYMENT pays through it). + static LedgerKey getNativeSACInstanceKey(Hash const& networkID); + + // Destination of the `counter`-th SAC_PAYMENT transaction that the node + // `node` generated for ledger `ledgerNum`. + static SCAddress getNativeSACPaymentDestination(NodeID const& node, + uint32_t ledgerNum, + uint64_t counter); + // Returns true if loadgen has submitted all required TXs bool isDone(GeneratedLoadConfig const& cfg) const; @@ -336,6 +373,11 @@ class LoadGenerator std::optional mSyntheticSoroswapState; // Counter for generating unique SAC payment destination addresses. uint32_t mSyntheticSACDestCounter{0}; + + // SAC_SETUP / SAC_PAYMENT: the native asset's SAC instance, and the + // number of payment destinations generated so far in this run. + std::optional mNativeSACInstance; + uint64_t mNativeSACDestCounter{0}; // Round-robin counter for Soroswap pair selection + direction. uint32_t mSyntheticSoroswapSwapCounter{0}; // Per-stream submission counters, tracked for every mode. For single- @@ -393,6 +435,13 @@ class LoadGenerator uint64_t sourceAccountId, GeneratedLoadConfig const& cfg); + // Build a SAC_PAYMENT transaction: a native asset payment through the + // real SAC instance, to a destination no other transaction uses. + std::pair + createNativeSACPaymentTransaction(uint32_t ledgerNum, + uint64_t sourceAccountId, + GeneratedLoadConfig const& cfg); + std::pair createUploadWasmTransaction(GeneratedLoadConfig const& cfg, uint32_t ledgerNum, uint64_t sourceAccountId); diff --git a/src/simulation/test/LoadGeneratorTests.cpp b/src/simulation/test/LoadGeneratorTests.cpp index 7e482fb35..ac9d17d60 100644 --- a/src/simulation/test/LoadGeneratorTests.cpp +++ b/src/simulation/test/LoadGeneratorTests.cpp @@ -7,12 +7,14 @@ #include "crypto/SecretKey.h" #include "ledger/ImmutableLedgerView.h" #include "ledger/LedgerManager.h" +#include "ledger/NetworkConfig.h" #include "main/Config.h" #include "simulation/ApplyLoad.h" #include "simulation/LoadGenerator.h" #include "simulation/Topologies.h" #include "test/Catch2.h" #include "test/test.h" +#include "transactions/TransactionUtils.h" #include "transactions/test/SorobanTxTestUtils.h" #include "util/Math.h" #include "util/MetricsRegistry.h" @@ -378,6 +380,225 @@ TEST_CASE("synthetic SAC load recycles source accounts in overlay-only mode", REQUIRE(bytes.count() - bytesBefore == nTxs * refMsgBytes); } +TEST_CASE("native SAC setup and payments are applied", "[loadgen][soroban]") +{ + // SAC_SETUP creates the native asset contract with a real transaction and + // SAC_PAYMENT pays through it, with apply on (unlike the MIXED_PREGEN_* + // modes, whose synthetic contract only exists in overlay-only mode). The + // run has more transactions than source accounts, so it also needs + // accounts to return to the pool after their transactions are applied. + uint32_t const nAccounts = 300; + uint32_t const txRate = 15; + uint32_t const nTxs = nAccounts + 100; + + Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); + Simulation::pointer simulation = Topologies::pair(networkID, [&](int i) { + auto cfg = getTestConfig(i); + cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; + cfg.ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING = true; + cfg.TESTING_UPGRADE_LEDGER_PROTOCOL_VERSION = + Config::CURRENT_LEDGER_PROTOCOL_VERSION; + cfg.GENESIS_TEST_ACCOUNT_COUNT = nAccounts; + return cfg; + }); + + simulation->startAllNodes(); + simulation->crankUntil( + [&]() { return simulation->haveAllExternalized(3, 1); }, + 10 * simulation->getExpectedLedgerCloseTime(), false); + + auto nodes = simulation->getNodes(); + auto& app = *nodes[0]; + auto& loadGen = app.getLoadGenerator(); + + // Ledger limits high enough that every submitted transaction fits in the + // next ledger. The per-transaction limits stay as perfnet's + // block_time_check leaves them: it only raises a limit that is under what + // its SAC load needs (4 footprint entries, 4 read and 2 written entries), + // so every other limit is the network's initial one. In particular a + // transaction's events may take 200 bytes at most, and both the setup + // transaction and the payments must be applied within that. + upgradeSorobanNetworkConfig( + [&](SorobanNetworkConfig& cfg) { + auto mx = std::numeric_limits::max(); + cfg.mLedgerMaxTxCount = mx; + cfg.mLedgerMaxInstructions = mx; + cfg.mLedgerMaxTransactionsSizeBytes = mx; + cfg.mLedgerMaxDiskReadEntries = mx; + cfg.mLedgerMaxDiskReadBytes = mx; + cfg.mLedgerMaxWriteLedgerEntries = mx; + cfg.mLedgerMaxWriteBytes = mx; + cfg.mTxMaxDiskReadEntries = + std::max(cfg.mTxMaxDiskReadEntries, 4); + cfg.mTxMaxWriteLedgerEntries = + std::max(cfg.mTxMaxWriteLedgerEntries, 2); + cfg.mTxMaxFootprintEntries = + std::max(cfg.mTxMaxFootprintEntries, 4); + }, + simulation); + { + auto const& sorobanCfg = + app.getLedgerManager().getLastClosedSorobanNetworkConfig(); + REQUIRE(sorobanCfg.txMaxContractEventsSizeBytes() == + MinimumSorobanNetworkConfig::TX_MAX_CONTRACT_EVENTS_SIZE_BYTES); + REQUIRE(sorobanCfg.txMaxInstructions() == + MinimumSorobanNetworkConfig::TX_MAX_INSTRUCTIONS); + } + + auto& completed = + app.getMetrics().NewMeter({"loadgen", "run", "complete"}, "run"); + auto& failed = + app.getMetrics().NewMeter({"loadgen", "run", "failed"}, "run"); + auto& attempted = + app.getMetrics().NewMeter({"loadgen", "txn", "attempted"}, "txn"); + auto applySuccess = [](Application& node) { + return node.getMetrics() + .NewCounter({"ledger", "apply-soroban", "success"}) + .count(); + }; + auto applyFailure = [](Application& node) { + return node.getMetrics() + .NewCounter({"ledger", "apply-soroban", "failure"}) + .count(); + }; + auto runToCompletion = [&](GeneratedLoadConfig cfg) { + auto const completedBefore = completed.count(); + auto const failedBefore = failed.count(); + loadGen.generateLoad(cfg); + simulation->crankUntil( + [&]() { + return completed.count() == completedBefore + 1 || + failed.count() != failedBefore; + }, + 120 * simulation->getExpectedLedgerCloseTime(), false); + REQUIRE(failed.count() == failedBefore); + REQUIRE(completed.count() == completedBefore + 1); + }; + + auto instanceKey = LoadGenerator::getNativeSACInstanceKey(networkID); + auto setupCfg = GeneratedLoadConfig::txLoad(LoadGenMode::SAC_SETUP, + nAccounts, /* nTxs */ 1, + /* txRate */ 1); + auto paymentCfg = GeneratedLoadConfig::txLoad(LoadGenMode::SAC_PAYMENT, + nAccounts, nTxs, txRate); + // Any payment that fails at apply fails the run. + paymentCfg.setMinSorobanPercentSuccess(100); + + // A node's last payments can be included late (they travel to the round + // leaders only), so the payment mode waits longer for them than other + // modes wait for theirs, unless the request says how long. + REQUIRE(LoadGenerator::completionTimeoutLedgers(setupCfg) == + LoadGenerator::DEFAULT_COMPLETION_TIMEOUT_NUM_LEDGERS); + REQUIRE(LoadGenerator::completionTimeoutLedgers(paymentCfg) == + LoadGenerator::SAC_PAYMENT_COMPLETION_TIMEOUT_NUM_LEDGERS); + REQUIRE(LoadGenerator::SAC_PAYMENT_COMPLETION_TIMEOUT_NUM_LEDGERS > + LoadGenerator::DEFAULT_COMPLETION_TIMEOUT_NUM_LEDGERS); + { + auto withTimeout = paymentCfg; + withTimeout.completionTimeoutLedgers = 7; + REQUIRE(LoadGenerator::completionTimeoutLedgers(withTimeout) == 7); + withTimeout.mode = LoadGenMode::PAY; + REQUIRE(LoadGenerator::completionTimeoutLedgers(withTimeout) == 7); + } + + // The payments need the contract. + REQUIRE(!app.getLedgerManager().getLCLView()->load(instanceKey)); + REQUIRE_THROWS_WITH( + loadGen.generateLoad(paymentCfg), + "Before running MODE::SAC_PAYMENT, please run MODE::SAC_SETUP to " + "create the native asset contract first."); + + // Neither mode can complete when nothing is applied. + app.setRunInOverlayOnlyMode(true); + REQUIRE_THROWS_WITH(loadGen.generateLoad(setupCfg), + "SAC_SETUP / SAC_PAYMENT need transactions to be " + "applied, they can not run in overlay only mode"); + REQUIRE_THROWS_WITH(loadGen.generateLoad(paymentCfg), + "SAC_SETUP / SAC_PAYMENT need transactions to be " + "applied, they can not run in overlay only mode"); + app.setRunInOverlayOnlyMode(false); + + // Setup: one applied transaction creates the instance on every node. + auto successBefore = applySuccess(app); + auto attemptedBefore = attempted.count(); + runToCompletion(setupCfg); + REQUIRE(attempted.count() - attemptedBefore == 1); + for (auto const& node : nodes) + { + REQUIRE(static_cast( + node->getLedgerManager().getLCLView()->load(instanceKey))); + REQUIRE(applySuccess(*node) == successBefore + 1); + REQUIRE(applyFailure(*node) == 0); + } + + // Setup again: the instance is there, so nothing is submitted. + attemptedBefore = attempted.count(); + runToCompletion(setupCfg); + REQUIRE(attempted.count() == attemptedBefore); + REQUIRE(applySuccess(app) == successBefore + 1); + + // Payments: every one is applied successfully on every node. + successBefore = applySuccess(app); + attemptedBefore = attempted.count(); + auto const firstLedger = + app.getLedgerManager().getLastClosedLedgerNum() + 1; + runToCompletion(paymentCfg); + auto const lastLedger = app.getLedgerManager().getLastClosedLedgerNum(); + REQUIRE(attempted.count() - attemptedBefore == nTxs); + for (auto const& node : nodes) + { + REQUIRE(applySuccess(*node) == successBefore + nTxs); + REQUIRE(applyFailure(*node) == 0); + } + + // The payments moved funds: the destination of the first payment of the + // run holds the amount paid. Its name contains the ledger the payment was + // generated for, which is one of the ledgers of the run. + auto balanceOf = [&](Application& node, + SCAddress const& dest) -> std::optional { + SCVal destVal(SCV_ADDRESS); + destVal.address() = dest; + LedgerKey balanceKey(CONTRACT_DATA); + balanceKey.contractData().contract = + instanceKey.contractData().contract; + balanceKey.contractData().key = + txtest::makeVecSCVal({makeSymbolSCVal("Balance"), destVal}); + balanceKey.contractData().durability = + ContractDataDurability::PERSISTENT; + auto entry = node.getLedgerManager().getLCLView()->load(balanceKey); + if (!entry) + { + return std::nullopt; + } + for (auto const& field : *entry.current().data.contractData().val.map()) + { + if (field.key.type() == SCV_SYMBOL && field.key.sym() == "amount") + { + REQUIRE(field.val.i128().hi == 0); + return field.val.i128().lo; + } + } + return std::nullopt; + }; + for (auto const& node : nodes) + { + uint32_t found = 0; + for (auto ledger = firstLedger; ledger <= lastLedger; ++ledger) + { + auto balance = balanceOf( + *node, LoadGenerator::getNativeSACPaymentDestination( + app.getConfig().NODE_SEED.getPublicKey(), ledger, + /* counter */ 0)); + if (balance) + { + REQUIRE(*balance == 100); + ++found; + } + } + REQUIRE(found == 1); + } +} + TEST_CASE("externalized tx set hashes and sources match transaction frames", "[loadgen][herder][txset]") { diff --git a/src/util/RandomEvictionCache.h b/src/util/RandomEvictionCache.h index b503297f7..d8ad6f9b5 100644 --- a/src/util/RandomEvictionCache.h +++ b/src/util/RandomEvictionCache.h @@ -224,6 +224,19 @@ class RandomEvictionCache : public NonMovableOrCopyable } } + // Returns a pointer to the value if the key exists and nullptr + // otherwise, like `maybeGet`, without recording the access: no counter + // is touched and the entry does not become any younger for eviction. + // That makes it a read: any number of threads may call it at the same + // time as long as none changes the cache (`put`, `maybeGet`, ...) + // meanwhile. The pointer is good until the cache is changed. + V const* + peek(K const& k) const + { + auto it = mValueMap.find(k); + return it == mValueMap.end() ? nullptr : &it->second.mValue; + } + // `get` offers basic exception safety guarantee. V& get(K const& k) diff --git a/src/util/test/CacheTests.cpp b/src/util/test/CacheTests.cpp index c051ea7d5..e3a330af4 100644 --- a/src/util/test/CacheTests.cpp +++ b/src/util/test/CacheTests.cpp @@ -101,6 +101,32 @@ TEST_CASE("RandomEvictionCache works as a cache", "[randomevictioncache]") } } +TEST_CASE("RandomEvictionCache peek reads without recording the access", + "[randomevictioncache]") +{ + RandomEvictionCache cache(4); + RandomEvictionCache const& constCache = cache; + cache.put(1, 10); + cache.put(2, 20); + auto const before = cache.getCounters(); + REQUIRE(constCache.peek(1)); + REQUIRE(*constCache.peek(1) == 10); + REQUIRE(*constCache.peek(2) == 20); + REQUIRE(constCache.peek(3) == nullptr); + // Neither hits nor misses are counted, and nothing was inserted. + auto const after = cache.getCounters(); + REQUIRE(after.mHits == before.mHits); + REQUIRE(after.mMisses == before.mMisses); + REQUIRE(after.mInserts == before.mInserts); + REQUIRE(cache.size() == 2); + // An updated value is what peek returns next. + cache.put(1, 11); + REQUIRE(*constCache.peek(1) == 11); + // maybeGet still counts. + REQUIRE(cache.maybeGet(1)); + REQUIRE(cache.getCounters().mHits == before.mHits + 1); +} + TEST_CASE("RandomEvictionCache does not thrash", "[randomevictioncachethrash][!hide]") {