diff --git a/README.md b/README.md index 0126b7350e..f6d812cfd7 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,248 @@ -
-Stellar -
-Creating equitable access to the global financial system -

Stellar Core

-
-

-Build Status -

+# stellar-core: leader schedule without SCP nomination (experimental branch) -Stellar-core is a replicated state machine that maintains a local copy of a cryptographic ledger and processes transactions against it, in consensus with a set of peers. -It implements the [Stellar Consensus Protocol](https://github.com/stellar/stellar-core/blob/master/src/scp/README.md), a _federated_ consensus protocol. -It is written in C++20 and runs on Linux, OSX and Windows. -Learn more by reading the [overview document](https://github.com/stellar/stellar-core/blob/master/docs/README.md). +**This branch is an experiment. It is not production code and must not run on a real network.** +It has no feature flag, no protocol gate, and it is not compatible with nodes that nominate. -# Documentation +Results below were measured on commit `daa965f4c` on 2026-11-10 and are one run per setting. -Documentation of the code's layout and abstractions, as well as for the -functionality available, can be found in -[`./docs`](https://github.com/stellar/stellar-core/tree/master/docs). +## What this branch is -# Installation +It is the overlay-v2 stack up to [#5459](https://github.com/stellar/stellar-core/pull/5459) +(Rust overlay over QUIC, millisecond close times, compressed tx-set responses, leader-only tx-set +construction) plus one change: **the SCP nomination phase is deleted and replaced by a leader +schedule that every node computes locally.** Draft PR: +[#5495](https://github.com/stellar/stellar-core/pull/5495). -See [Installation](./INSTALL.md) +- **Schedule.** For slot N, the nodes of the transitive quorum are sorted by + `SHA256(hash of ledger header N-1 || node id)`. The leader of round r is entry `(r-1) mod n`. +- **Rounds.** A round is a ballot counter. The round's leader builds its value and goes straight to + the ballot protocol with `(r, value)`. The other nodes vote for the leader's ballot once it passes + proposal validation. A timeout (the ballot timeout settings) hands the slot to the next leader. +- **Unchanged.** The ballot protocol's accept, confirm and commit rules, value validation, apply, + ledger state and meta. -# Contributing +The question it answers: how much of a ledger's time is nomination, and what limits block time once +nomination is gone? -See [Contributing](./CONTRIBUTING.md) +## Performance results -# Running tests +| Target block time | Load | Verdict | Block time measured (average over 30 nodes) | +|---|---|---|---| +| **2,000 ms** | 3,000 TPS | **PASS**, all four windows | p75 2,066-2,109 ms, p99 2,286-2,324 ms | +| **1,000 ms** | 3,000 TPS | **FAIL**, first window | p75 1,386 ms (limit 1,200), p99 1,599 ms | -See [running tests](./CONTRIBUTING.md#running-tests) +In short: at 2 s the network keeps up and has more slack per ledger than with nomination. At 1 s it +settles at about 1.33 s per ledger. Nothing breaks; the stages of one ledger simply add up to more +than a second. -[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/stellar/stellar-core) +### 2,000 ms target: PASS + +Load ran its full 900 s. Every node passed every window. + +| Window | p75, average (range over nodes) | p99, average (range over nodes) | +|---|---|---| +| 1 | 2,109 ms (2,086-2,128) | 2,324 ms (2,282-2,359) | +| 2 | 2,066 ms (2,058-2,076) | 2,286 ms (2,239-2,328) | +| 3 | 2,082 ms (2,067-2,091) | 2,296 ms (2,251-2,333) | +| 4 | 2,094 ms (2,080-2,105) | 2,303 ms (2,262-2,344) | + +Checks, all passed: +- **Inclusion:** every node's ledgers hold exactly the 2,880,000 offered transactions. +- **No fork**, all 30 nodes in sync, still closing ledgers after the load. +- **Error counters zero** (invalid signatures, invariant failures, publish failures, invalid values, + lost sync), no node restarts. +- **No wasted rounds:** every ledger was decided by the first leader. 0 leader timeouts, 0 ballot + timeouts, 0 empty transaction sets. + +Against the same fixture on #5459's head `ebdd73635` (with nomination), run the same day on the same +instance type: + +| 5-minute means, median node | #5459 `ebdd73635` | This branch `daa965f4c` | +|---|---:|---:| +| Verdict | PASS | PASS | +| p75 per window | 2,112 / 2,099 / 2,101 / 2,099 ms | 2,109 / 2,066 / 2,082 / 2,094 ms | +| p99 per window | 2,334 / 2,391 / 2,324 / 2,354 ms | 2,324 / 2,286 / 2,296 / 2,303 ms | +| Ledgers per 5 minutes | 147-148 | 149-150 | +| Transactions per ledger | 6,092 | 6,003 | +| Trigger to first ballot | 1.21 s | 0.88 s | +| First ballot to externalize | 0.29 s | 0.43 s | +| **Trigger to externalize** | **1.50 s** | **1.31 s** | +| Leader tx-set build | 0.28 s | 0.27 s | +| Tx-set fetch (follower) | 0.24 s | 0.26 s | +| Tx-set validation (follower) | 0.20 s | 0.19 s | +| Ledger close | 0.164 s | 0.162 s | + +- Block time is the same on both, as expected: when the network keeps up, the 2 s trigger sets the + cadence. +- The time from trigger to externalize is 0.19 s shorter. The part before the first ballot that is + not build, fetch or validation fell from about 0.48 s (nomination rounds) to about 0.15 s. +- Some of that came back in the ballot phase. Nodes now start their ballot at different times, as + each finishes validating the leader's set, so the early ones wait for a quorum of the others. + +### 1,000 ms target: FAIL + +The first 5-minute window missed on all 30 nodes, and the test stops the load at the first miss. + +| Window 1 | Value | +|---|---| +| p75, average (range over nodes) | 1,386 ms (1,375-1,399); the limit is below 1,200 ms | +| p99, average (range over nodes) | 1,599 ms (1,582-1,628); the limit is 2,000 ms | +| Ledgers per 5 minutes | 225 (1.33 s each) | +| Transactions per ledger | 3,750, the cap, on every ledger | +| Leader timeouts, ballot timeouts, empty tx sets | 0, 0, 0 | +| Error counters, in sync, no fork, still closing | all passed | + +The network was over capacity at this block time: 3,000 TPS x 1.33 s is 4,000 transactions offered +per ledger against a cap of 3,750, so the mempool backed up and every ledger started the instant +the previous one closed. Inclusion is not checked after an early stop. No baseline was run at this +setting. + +### Where a ledger's time goes + +| Stage (5-minute means, median node) | 2,000 ms target (6,000 txs) | 1,000 ms target (3,750 txs) | +|---|---:|---:| +| Leader builds the tx set | 0.27 s | 0.25 s | +| Trigger to first ballot (includes the build, the fetch and validation) | 0.88 s | 0.70 s | +| First ballot to externalize | 0.43 s | 0.41 s | +| Ledger close (150 ms / 187 ms of it is simulated apply) | 0.16 s | 0.20 s | +| **Sum** | **1.47 s** | **1.30 s** | + +Halving the transaction set took only 0.17 s off the cycle. What remains is mostly cross-region +round trips, not CPU: + +1. **Tx-set dissemination takes about 2.5 round trips.** A follower receives the proposal, requests + the set from the proposer, and the response takes two flights. Seen from us-west-2, the fetch + took 27 ms from the same region, 150 ms from us-east-1, 301 ms from eu-central-1 and 362 ms from + ap-southeast-1. Nothing is pushed, and transactions are sent in full although followers already + hold nearly all of them. +2. **Followers validate before they vote, and the ballot then needs four one-way trips.** +3. **About 70 ms pass between the leader finishing its set and the proposal reaching a peer in the + same region.** SCP state, including the tx set, is written to the database before each + broadcast. That this write accounts for most of the 70 ms is inferred, not timed. +4. **The leader's build is on the critical path.** At 1 s nothing can be built ahead of the trigger. + +## How it was measured + +Both runs use the `block_time_check` mission of perfnet, SDF's tooling for performance networks on +AWS. Each run launches a fresh network, brings it to the target settings, runs the load, judges +it, and terminates the network. + +### Network topology + +| | | +|---|---| +| Validators | 30, all validating, no watchers | +| Organizations | 10 home domains of 3 validators each, all `HIGH` quality | +| Quorum set | Generated by stellar-core from the home domains: 7 of 10 organizations, 2 of 3 validators within each | +| Regions | 4 AWS regions: eu-central-1 (8 nodes), ap-southeast-1 (8), us-east-1 (7), us-west-2 (7) | +| Placement | 9 organizations have their three validators in three different regions; one has two in us-east-1 and one in ap-southeast-1. The layout was identical in both runs | +| Connectivity | Full mesh: every node is configured with the other 29 as its only peers | +| Round-trip times seen from us-west-2 | us-east-1 about 60 ms, eu-central-1 about 147 ms, ap-southeast-1 about 180 ms | + +### Hardware and software + +| | | +|---|---| +| Instance type | c6id.2xlarge: 8 vCPU (Intel Ice Lake, with SHA extensions), 16 GiB RAM, local NVMe | +| Build | Commit `daa965f4c`, the perftests package `30.0.1-3674.daa965f4ce.noble~perf.leader.schedule.no.nomination~perftests`. All 30 nodes reported that commit in their build label | +| Overlay | The Rust overlay of this stack: QUIC, one connection per peer | +| Ledger protocol | 30, upgraded from genesis during preparation | +| Database | SQLite | + +### Node configuration + +Beyond identity, peers and home domains, every node ran with: + +``` +ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING=true +UNSAFE_QUORUM=true +FAILURE_SAFETY=0 +GENESIS_TEST_ACCOUNT_COUNT=1000000 +TARGET_PEER_CONNECTIONS=29 +MAX_ADDITIONAL_PEER_CONNECTIONS=87 +PREFERRED_PEERS_ONLY=true +FLOOD_DEMAND_BACKOFF_DELAY_MS=1000 +INVARIANT_CHECKS=[] +TESTING_IGNORE_LEDGER_TIME_UPGRADE_BOUNDS=true +TESTING_MAX_CLASSIC_BYTE_ALLOWANCE=1048576 +TESTING_MAX_SOROBAN_BYTE_ALLOWANCE=9437184 +LOADGEN_PREGENERATED_TRANSACTIONS_FILE="/var/lib/stellar-core/loadgen-txs.xdr" +OP_APPLY_SLEEP_TIME_DURATION_FOR_TESTING=[25] # [50] in the 1,000 ms run +OP_APPLY_SLEEP_TIME_WEIGHT_FOR_TESTING=[100] +``` + +`EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD` was off (the default). Catchup and automatic restart are +off, so a node that crashes or falls behind stays that way and fails the run. + +### Network settings (applied by upgrade before the load) + +| Setting | 2,000 ms run | 1,000 ms run | +|---|---:|---:| +| Target ledger close time | 2,000 ms | 1,000 ms | +| SCP timeouts (ballot and nomination, initial and increment) | 2,000 ms | 2,000 ms | +| Maximum transactions per ledger (125% of the offered load per block) | 7,500 | 3,750 | +| Maximum tx-set size in bytes | 10 MiB (1 MiB classic + 9 MiB Soroban) | same | +| Soroban limits | raised to fit the load | raised to fit the load | + +The leader timer of this branch uses the ballot timeout, so a round's leader had 2 s (4 s with its +one extension) in both runs. + +### Load + +| | | +|---|---| +| Transactions | Stellar Asset Contract (SAC) payments, pregenerated; no classic transactions | +| Rate | 3,000 TPS offered: every validator generates 100 TPS | +| Accounts | 1,000,000 genesis accounts, 33,333 per generator | +| Duration | 60 s warm-up, then 900 s measured | +| Apply | Skipped ("overlay-only" mode) and replaced by a sleep per operation, sized as a 150 ms budget per ledger at the offered load: 25 us per operation at 2,000 ms, 50 us at 1,000 ms | + +Commands (the two runs differ only in the block time): + +``` +block_time_check.py \ + --stellar-core-version 30.0.1-3674.daa965f4ce.noble~perf.leader.schedule.no.nomination~perftests \ + --block-time-ms 2000 \ # 1000 for the second run + --soroban-tps 3000 --soroban-type sac --classic-tps 0 \ + --simulate-apply-budget-ms 150 --load-duration-s 900 \ + --instance-type c6id.2xlarge --save-logs always +``` + +Defaults that matter: 10 organizations, validators as generators, `--scp-timeout-ms 2000`, +`--max-tx-set-bytes 10485760`, 1,000,000 genesis accounts, seed 0. + +### How a run is judged + +- Ledger close times are read per node from Prometheus (scraped every 10 s) and judged over + 5-minute windows after the warm-up. +- **PASS** needs, on every node in every window: p75 within [0.8 T, 1.2 T), p99 at most 2 T, and at + least one close per 2 T on average, where T is the target block time. It also needs every node's + ledgers to hold exactly the offered transactions, zero error counters, all nodes in sync, no + fork, and the network still closing ledgers afterwards. +- The load stops at the first window that misses, so a FAIL can rest on one window. + +### Limits of these numbers + +- One run per setting, and one baseline run (at 2,000 ms only). +- Apply is simulated, so these are consensus and overlay numbers, not end-to-end ledger close. +- `scp.timing.trigger-to-ballot` records 0 when a follower votes before its own trigger fires, so it + is not strictly comparable with the nomination build. Triggers were on time in the 2,000 ms run. +- The per-stage timeline and the fetch times by region come from one node's logs at 2,000 ms and + four nodes' logs at 1,000 ms. + +## Known limitations of the prototype + +- The leader schedule's node set is whatever transitive quorum each node currently knows. It + assumes a symmetric, static quorum. +- Upgrades: a value needs a quorum of nodes that armed the same upgrades; there is no stripping or + merging. A follower follows a leader's empty-tx-set ballot under the weaker ballot validation. +- Leader grinding through the header hash and fairness between organizations are out of scope. +- The leader timer extension is broader than intended, and a proposal that arrives just after a + timeout is wasted. Neither occurred in these runs. +- A build with `--disable-tests` does not compile at this commit (test headers reached from + `herder/HerderImpl.cpp`). + +## Building and the rest of the documentation + +Unchanged from stellar-core: see [INSTALL.md](./INSTALL.md), [CONTRIBUTING.md](./CONTRIBUTING.md) +and [`./docs`](./docs). The full test suite passes on `daa965f4c`, also with `--all-versions`. diff --git a/src/herder/Herder.h b/src/herder/Herder.h index 66207170ee..4492254967 100644 --- a/src/herder/Herder.h +++ b/src/herder/Herder.h @@ -200,8 +200,8 @@ class Herder virtual uint32 getMinLedgerSeqToAskPeers() const = 0; virtual uint32 getMinLedgerSeqToRemember() const = 0; - virtual bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) = 0; + virtual bool isNewerStatement(SCPStatement const& oldSt, + SCPStatement const& newSt) = 0; // Returns sequence number for most recent completed checkpoint that the // node knows about, as derived from diff --git a/src/herder/HerderImpl.cpp b/src/herder/HerderImpl.cpp index e261256430..1a66412bb7 100644 --- a/src/herder/HerderImpl.cpp +++ b/src/herder/HerderImpl.cpp @@ -999,6 +999,12 @@ HerderImpl::recvSCPEnvelope(SCPEnvelope const& envelope) mSCPMetrics.mEnvelopeReceive.Mark(); + if (envelope.statement.pledges.type() == SCP_ST_NOMINATE) + { + // There is no nomination phase (see HerderSCPDriver::getRoundLeader). + return Herder::ENVELOPE_STATUS_DISCARDED; + } + // **** first perform checks that do NOT require signature verification // this allows to fast fail messages that we'd throw away anyways @@ -1691,9 +1697,7 @@ HerderImpl::setupTriggerNextLedger() // trigger. if (getSCP().isValidator() && !triggerDue) { - auto leaders = getSCP().predictNominationLeaders( - nextIndex, xdr::xdr_to_opaque(lcl.header.scpValue), 1); - if (leaders.count(getSCP().getLocalNodeID())) + if (mHerderSCPDriver.isLocalNodeRoundLeader(nextIndex, 1)) { // Validate against the close time we expect at the trigger, // not the earlier wall-clock time at which we start work. @@ -1865,12 +1869,8 @@ HerderImpl::prepareTxSet(uint32_t ledgerSeq, ConsensusTime closeTime) { return; } - auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); if (!getSCP().isValidator() || - !getSCP() - .predictNominationLeaders( - ledgerSeq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(getSCP().getLocalNodeID())) + !mHerderSCPDriver.isLocalNodeRoundLeader(ledgerSeq, 1)) { return; } @@ -1889,18 +1889,14 @@ HerderImpl::buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime) auto const lcl = mLedgerManager.getLastClosedLedgerHeader(); releaseAssert(ledgerSeq == lcl.header.ledgerSeq + 1); releaseAssert(closeTime > getConsensusTime(lcl.header.scpValue)); - bool const activeLeader = getSCP().getNominationLeaders(ledgerSeq).count( - getSCP().getLocalNodeID()); + uint32_t const round = getSCP().getLeaderRound(ledgerSeq); + bool const activeLeader = + round != 0 && mHerderSCPDriver.isLocalNodeRoundLeader(ledgerSeq, round); bool const legacyTxSet = buildsLegacyTxSet(lcl.header.ledgerVersion); - releaseAssert(activeLeader || legacyTxSet || - getSCP() - .predictNominationLeaders( - ledgerSeq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(getSCP().getLocalNodeID())); CLOG_INFO(Herder, "Building TX set as {} for ledger {}", - activeLeader ? "active nomination leader" - : legacyTxSet ? "validator (legacy tx set)" - : "first-round nomination leader", + activeLeader ? fmt::format("round {} leader", round) + : legacyTxSet ? std::string("validator (legacy tx set)") + : std::string("first-round leader (early)"), ledgerSeq); // Apply-time offset: see the comment in triggerNextLedger auto const closeTimeOffset = @@ -2162,29 +2158,30 @@ HerderImpl::triggerNextLedger(uint32_t ledgerSeqToTrigger, if (!isCtValid) { CLOG_WARNING(Herder, - "Invalid close time selected ({}), skipping nomination", + "Invalid close time selected ({}), skipping the slot", nextCloseTime.toString()); return; } // Start the trigger-to-ballot interval only once this node is going to - // nominate. A skipped trigger must not credit the time spent waiting for + // take part. A skipped trigger must not credit the time spent waiting for // peers to the drift allowance. - mHerderSCPDriver.recordNominationTrigger(ledgerSeqToTrigger); + mHerderSCPDriver.recordTrigger(ledgerSeqToTrigger); if (!getSCP().isValidator()) { - CLOG_DEBUG(Herder, "Non-validating node, skipping nomination (SCP)."); + CLOG_DEBUG(Herder, "Non-validating node, not starting the slot (SCP)."); return; } - getHerderSCPDriver().recordSCPEvent(ledgerSeqToTrigger, true); - mHerderSCPDriver.nominate( + // There is no nomination: the leader of each round (a deterministic + // schedule seeded by the LCL hash) proposes straight into the ballot + // protocol. The value is only built if this node leads a round. + mHerderSCPDriver.startSlot( ledgerSeqToTrigger, [this, ledgerSeqToTrigger]() { - return makeNominationValue(ledgerSeqToTrigger); + return makeProposalValue(ledgerSeqToTrigger); }, - lcl.header.scpValue, /*constructValueNow=*/buildsLegacyTxSet(lcl.header.ledgerVersion)); // A follower validates the leader's set when it arrives, and most of its @@ -2194,9 +2191,10 @@ HerderImpl::triggerNextLedger(uint32_t ledgerSeqToTrigger, // so that validation finds them in the cache. Skip it if this node already // voted, i.e. validated the leader's value. if (!buildsLegacyTxSet(lcl.header.ledgerVersion) && - !getSCP() - .getNominationLeaders(ledgerSeqToTrigger) - .count(getSCP().getLocalNodeID()) && + !mHerderSCPDriver.isLocalNodeRoundLeader( + ledgerSeqToTrigger, + std::max(1, + getSCP().getLeaderRound(ledgerSeqToTrigger))) && getSCP().getLatestMessagesSend(ledgerSeqToTrigger).empty()) { prewarmSignatureCache(ledgerSeqToTrigger, {}); @@ -2444,7 +2442,7 @@ HerderImpl::buildsLegacyTxSet(uint32_t lclLedgerVersion) } ValueWrapperPtr -HerderImpl::makeNominationValue(uint32_t ledgerSeqToTrigger) +HerderImpl::makeProposalValue(uint32_t ledgerSeqToTrigger) { // A follower can be promoted long after the trigger. Recheck the ledger // state and select a fresh close time when SCP actually needs our value. @@ -2522,9 +2520,9 @@ HerderImpl::makeNominationValue(uint32_t ledgerSeqToTrigger) closeTimeOffset); #ifdef BUILD_TESTS - if (mNominatedTxSetOverride) + if (mProposedTxSetOverride) { - proposedSet = mNominatedTxSetOverride(lcl); + proposedSet = mProposedTxSetOverride(lcl); CLOG_INFO(Herder, "Nominating overridden tx set {} for slot {}", hexAbbrev(proposedSet->getContentsHash()), lcl.header.ledgerSeq + 1); @@ -2591,12 +2589,12 @@ HerderImpl::makeNominationValue(uint32_t ledgerSeqToTrigger) } #ifdef BUILD_TESTS - if (mApp.getConfig().TESTING_NOMINATE_RANDOM_VALUES && + if (mApp.getConfig().TESTING_PROPOSE_RANDOM_VALUES && getHerderSCPDriver().protocolAllowsEmptyTxSetValues()) { txSetHash = HashUtils::pseudoRandomForTesting(); CLOG_INFO(Herder, - "TESTING_NOMINATE_RANDOM_VALUES: nominating slot {} " + "TESTING_PROPOSE_RANDOM_VALUES: proposing slot {} " "with random tx-set hash {}", slotIndex, hexAbbrev(txSetHash)); } @@ -3623,9 +3621,9 @@ HerderImpl::makeStellarValue(Hash const& txSetHash, ConsensusTime closeTime, } bool -HerderImpl::isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) +HerderImpl::isNewerStatement(SCPStatement const& oldSt, + SCPStatement const& newSt) { - return getSCP().isNewerNominationOrBallotSt(oldSt, newSt); + return getSCP().isNewerStatement(oldSt, newSt); } } diff --git a/src/herder/HerderImpl.h b/src/herder/HerderImpl.h index fb29ba924b..f46ad8bf7d 100644 --- a/src/herder/HerderImpl.h +++ b/src/herder/HerderImpl.h @@ -138,7 +138,7 @@ class HerderImpl : public Herder // still run through their production paths. std::function(size_t)> mGetTopTransactionsForTesting; - friend class EarlyNominationTestAccess; + friend class LeaderBuildTestAccess; std::optional mMaxClassicTxSize; std::optional mMaxTxSizeOverride; @@ -162,7 +162,7 @@ class HerderImpl : public Herder // When set, this node nominates the tx set returned by this function // (given the LCL) instead of the one it builds. std::function - mNominatedTxSetOverride; + mProposedTxSetOverride; #endif std::vector getSCPStateForPeer(uint32 ledgerSeq) override; @@ -195,8 +195,8 @@ class HerderImpl : public Herder uint32_t getMinLedgerSeqToRemember() const override; - bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) override; + bool isNewerStatement(SCPStatement const& oldSt, + SCPStatement const& newSt) override; uint32_t getMostRecentCheckpointSeq() override; @@ -272,7 +272,7 @@ class HerderImpl : public Herder PreparedTxSet buildTxSet(uint32_t ledgerSeq, ConsensusTime closeTime); void prepareTxSet(uint32_t ledgerSeq, ConsensusTime closeTime); void discardPreparedTxSet(); - ValueWrapperPtr makeNominationValue(uint32_t ledgerSeq); + ValueWrapperPtr makeProposalValue(uint32_t ledgerSeq); // Whether a ledger closing on an LCL with this protocol uses a legacy // (non-generalized) tx set, which every validator builds locally. static bool buildsLegacyTxSet(uint32_t lclLedgerVersion); diff --git a/src/herder/HerderSCPDriver.cpp b/src/herder/HerderSCPDriver.cpp index 84a2c58820..4a04b8217f 100644 --- a/src/herder/HerderSCPDriver.cpp +++ b/src/herder/HerderSCPDriver.cpp @@ -8,6 +8,7 @@ #include "crypto/SHA.h" #include "crypto/SecretKey.h" #include "herder/HerderImpl.h" +#include "herder/LeaderSchedule.h" #include "herder/LedgerCloseData.h" #include "herder/PendingEnvelopes.h" #include "ledger/LedgerHeaderUtils.h" @@ -54,10 +55,6 @@ HerderSCPDriver::SCPMetrics::SCPMetrics(Application& app) , mValueValid(app.getMetrics().NewMeter({"scp", "value", "valid"}, "value")) , mValueInvalid( app.getMetrics().NewMeter({"scp", "value", "invalid"}, "value")) - , mCombinedCandidates(app.getMetrics().NewMeter( - {"scp", "nomination", "combinecandidates"}, "value")) - , mNominateToPrepare( - app.getMetrics().NewTimer({"scp", "timing", "nominated"})) , mPrepareToExternalize( app.getMetrics().NewTimer({"scp", "timing", "externalized"})) , mTriggerToBallot( @@ -88,13 +85,14 @@ HerderSCPDriver::HerderSCPDriver(Application& app, HerderImpl& herder, , mSCP{*this, mApp.getConfig().NODE_SEED.getPublicKey(), mApp.getConfig().NODE_IS_VALIDATOR, mApp.getConfig().QUORUM_SET} , mSCPMetrics{mApp} - , mNominateTimeout{mApp.getMetrics().NewHistogram( - {"scp", "timeout", "nominate"})} + , mLeaderTimeout{mApp.getMetrics().NewHistogram( + {"scp", "timeout", "leader"})} + , mLeaderRound{mApp.getMetrics().NewHistogram({"scp", "leader", "round"})} , mPrepareTimeout{mApp.getMetrics().NewHistogram( {"scp", "timeout", "prepare"})} , mUniqueValues{mApp.getMetrics().NewHistogram( {"scp", "slot", "values-referenced"})} - , mLedgerSeqNominating(0) + , mLedgerSeqProposing(0) , mTxSetValidCache(TXSETVALID_CACHE_SIZE) { } @@ -406,7 +404,7 @@ HerderSCPDriver::validatePastOrFutureValue( SCPDriver::ValidationLevel HerderSCPDriver::validateValueAgainstLocalState(uint64_t slotIndex, StellarValue const& b, - bool nomination) const + bool proposal) const { ZoneScoped; releaseAssert(threadIsMain()); @@ -437,13 +435,13 @@ HerderSCPDriver::validateValueAgainstLocalState(uint64_t slotIndex, return SCPDriver::kInvalidValue; } - if (nomination) + if (proposal) { - // Empty-tx-set values should only appear in balloting, and so - // are considered invalid during nomination. + // Empty-tx-set values only come out of balloting (a node + // gives up on a download): nobody proposes one. CLOG_DEBUG(Herder, "HerderSCPDriver::validateValue i: {} rejecting " - "empty-tx-set value during nomination", + "empty-tx-set value as a proposal", slotIndex); return SCPDriver::kInvalidValue; } @@ -609,14 +607,14 @@ HerderSCPDriver::deserializeAndValidateStellarValue(uint64_t slotIndex, } void -HerderSCPDriver::extractValidUpgrades(StellarValue& sv, bool nomination) const +HerderSCPDriver::extractValidUpgrades(StellarValue& sv, bool proposal) const { LedgerUpgradeType lastUpgradeType = LEDGER_UPGRADE_VERSION; LedgerUpgradeType thisUpgradeType; bool first = true; for (auto it = sv.upgrades.begin(); it != sv.upgrades.end();) { - if (!mUpgrades.isValid(*it, thisUpgradeType, nomination, mApp)) + if (!mUpgrades.isValid(*it, thisUpgradeType, proposal, mApp)) { it = sv.upgrades.erase(it); } @@ -635,7 +633,7 @@ HerderSCPDriver::extractValidUpgrades(StellarValue& sv, bool nomination) const SCPDriver::ValidationLevel HerderSCPDriver::validateValue(uint64_t slotIndex, Value const& value, - bool nomination) const + bool proposal) const { ZoneScoped; releaseAssert(threadIsMain()); @@ -648,11 +646,11 @@ HerderSCPDriver::validateValue(uint64_t slotIndex, Value const& value, } SCPDriver::ValidationLevel res = - validateValueAgainstLocalState(slotIndex, b, nomination); + validateValueAgainstLocalState(slotIndex, b, proposal); if (res != SCPDriver::kInvalidValue) { auto origSize = b.upgrades.size(); - extractValidUpgrades(b, nomination); + extractValidUpgrades(b, proposal); if (b.upgrades.size() != origSize) { CLOG_TRACE(Herder, @@ -674,27 +672,6 @@ HerderSCPDriver::validateValue(uint64_t slotIndex, Value const& value, return res; } -ValueWrapperPtr -HerderSCPDriver::extractValidValue(uint64_t slotIndex, Value const& value) -{ - ZoneScoped; - StellarValue b; - if (!deserializeAndValidateStellarValue(slotIndex, value, b)) - { - return nullptr; - } - - ValueWrapperPtr res; - if (validateValueAgainstLocalState(slotIndex, b, true) >= - SCPDriver::kStructurallyValidValue) - { - extractValidUpgrades(b, true); - res = wrapStellarValue(b); - } - - return res; -} - // value marshaling std::string @@ -776,18 +753,6 @@ void HerderSCPDriver::timerCallbackWrapper(uint64_t slotIndex, int timerID, std::function cb) { -#ifdef BUILD_TESTS - if (timerID == Slot::NOMINATION_EMIT_TIMER) - { - if (!mHerder.isTracking() || - mHerder.nextConsensusLedgerIndex() == slotIndex) - { - cb(); - } - return; - } -#endif - // reschedule timers for future slots when tracking if (mHerder.isTracking() && mHerder.nextConsensusLedgerIndex() != slotIndex) { @@ -811,11 +776,8 @@ HerderSCPDriver::timerCallbackWrapper(uint64_t slotIndex, int timerID, } else { - if (!SCPTiming.mPrepareStart) - { - // Timeout happened between nominate and first prepare - ++SCPTiming.mNominationTimeoutCount; - } + // Timed out waiting for the round leader's proposal + ++SCPTiming.mLeaderTimeoutCount; } } @@ -876,7 +838,7 @@ HerderSCPDriver::stopTimer(uint64 slotIndex, int timerID) static uint32_t const MAX_TIMEOUT_MS = (30 * 60) * 1000; std::chrono::milliseconds -HerderSCPDriver::computeTimeout(uint32 roundNumber, bool isNomination) +HerderSCPDriver::computeTimeout(uint32 roundNumber) { releaseAssertOrThrow(roundNumber > 0); @@ -891,18 +853,8 @@ HerderSCPDriver::computeTimeout(uint32 roundNumber, bool isNomination) { auto const& networkConfig = mLedgerManager.getLastClosedSorobanNetworkConfig(); - if (isNomination) - { - initialTimeoutMS = - networkConfig.nominationTimeoutInitialMilliseconds(); - incrementMS = - networkConfig.nominationTimeoutIncrementMilliseconds(); - } - else - { - initialTimeoutMS = networkConfig.ballotTimeoutInitialMilliseconds(); - incrementMS = networkConfig.ballotTimeoutIncrementMilliseconds(); - } + initialTimeoutMS = networkConfig.ballotTimeoutInitialMilliseconds(); + incrementMS = networkConfig.ballotTimeoutIncrementMilliseconds(); } auto timeoutMS = initialTimeoutMS + (roundNumber - 1) * incrementMS; @@ -913,284 +865,6 @@ HerderSCPDriver::computeTimeout(uint32 roundNumber, bool isNomination) return std::chrono::milliseconds(timeoutMS); } -#ifdef BUILD_TESTS -std::chrono::milliseconds -HerderSCPDriver::getNominationEmitDelayForTesting() const -{ - return mApp.getConfig().ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING; -} -#endif - -// returns true if l < r -// lh, rh are the hashes of l,h -static bool -compareTxSets(ApplicableTxSetFrameConstPtr const& l, - ApplicableTxSetFrameConstPtr const& r, Hash const& lh, - Hash const& rh, std::optional lEncodedSize, - std::optional rEncodedSize, LedgerHeader const& header, - Hash const& s) -{ - if (!l && !r) - { - // Do not have either tx set. Compare hashes - return lessThanXored(lh, rh, s); - } - - if (!l || !r) - { - // If one exists, choose it - return !l; - } - - auto lSize = l->size(header); - auto rSize = r->size(header); - if (lSize != rSize) - { - return lSize < rSize; - } - if (protocolVersionStartsFrom(header.ledgerVersion, - SOROBAN_PROTOCOL_VERSION)) - { - auto lBids = l->getTotalInclusionFees(); - auto rBids = r->getTotalInclusionFees(); - if (lBids != rBids) - { - return lBids < rBids; - } - } - if (protocolVersionStartsFrom(header.ledgerVersion, ProtocolVersion::V_11)) - { - auto lFee = l->getTotalFees(header); - auto rFee = r->getTotalFees(header); - if (lFee != rFee) - { - return lFee < rFee; - } - } - if (protocolVersionStartsFrom(header.ledgerVersion, - SOROBAN_PROTOCOL_VERSION)) - { - if (lEncodedSize.value() != rEncodedSize.value()) - { - // Look for the smallest encoded size. - return lEncodedSize.value() > rEncodedSize.value(); - } - } - return lessThanXored(lh, rh, s); -} - -ValueWrapperPtr -HerderSCPDriver::combineCandidates(uint64_t slotIndex, - ValueWrapperPtrSet const& candidates) -{ - ZoneScoped; - CLOG_DEBUG(Herder, "Combining {} candidates", candidates.size()); - mSCPMetrics.mCombinedCandidates.Mark(candidates.size()); - - std::map upgrades; - - std::set aggSet; - - releaseAssert(!mLedgerManager.isApplying()); - releaseAssert(threadIsMain()); - auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); - - Hash candidatesHash; - - std::vector candidateValues; - - for (auto const& c : candidates) - { - candidateValues.emplace_back(); - StellarValue& sv = candidateValues.back(); - Value const& val = c->getValue(); - uint256 const valHash = sha256(val); - - if (!toStellarValue(val, sv)) - { - throw std::runtime_error(fmt::format( - "HerderSCPDriver::combineCandidates: cannot parse candidate " - "value with hash {}", - binToHex(valHash))); - } - - candidatesHash ^= valHash; - - for (auto const& upgrade : sv.upgrades) - { - LedgerUpgrade lupgrade; - try - { - xdr::xdr_from_opaque(upgrade, lupgrade); - } - catch (...) - { - throw std::runtime_error( - fmt::format("HerderSCPDriver::combineCandidates: cannot " - "parse upgrade in candidate with hash {}", - binToHex(valHash))); - } - auto it = upgrades.find(lupgrade.type()); - if (it == upgrades.end()) - { - upgrades.emplace(std::make_pair(lupgrade.type(), lupgrade)); - } - else - { - LedgerUpgrade& clUpgrade = it->second; - switch (lupgrade.type()) - { - case LEDGER_UPGRADE_VERSION: - // pick the highest version - clUpgrade.newLedgerVersion() = - std::max(clUpgrade.newLedgerVersion(), - lupgrade.newLedgerVersion()); - break; - case LEDGER_UPGRADE_BASE_FEE: - // take the max fee - clUpgrade.newBaseFee() = - std::max(clUpgrade.newBaseFee(), lupgrade.newBaseFee()); - break; - case LEDGER_UPGRADE_MAX_TX_SET_SIZE: - // take the max tx set size - clUpgrade.newMaxTxSetSize() = - std::max(clUpgrade.newMaxTxSetSize(), - lupgrade.newMaxTxSetSize()); - break; - case LEDGER_UPGRADE_BASE_RESERVE: - // take the max base reserve - clUpgrade.newBaseReserve() = std::max( - clUpgrade.newBaseReserve(), lupgrade.newBaseReserve()); - break; - case LEDGER_UPGRADE_FLAGS: - clUpgrade.newFlags() = - std::max(clUpgrade.newFlags(), lupgrade.newFlags()); - break; - case LEDGER_UPGRADE_CONFIG: - if (clUpgrade.newConfig().contractID < - lupgrade.newConfig().contractID) - { - clUpgrade.newConfig() = lupgrade.newConfig(); - } - else if (clUpgrade.newConfig().contractID == - lupgrade.newConfig().contractID && - clUpgrade.newConfig().contentHash < - lupgrade.newConfig().contentHash) - { - clUpgrade.newConfig() = lupgrade.newConfig(); - } - break; - case LEDGER_UPGRADE_MAX_SOROBAN_TX_SET_SIZE: - clUpgrade.newMaxSorobanTxSetSize() = - std::max(clUpgrade.newMaxSorobanTxSetSize(), - lupgrade.newMaxSorobanTxSetSize()); - break; - default: - // should never get there with values that are not valid - throw std::runtime_error("invalid upgrade step"); - } - } - } - } - - StellarValue comp; - // take the txSet with the biggest size, highest xored hash that we have - { - auto highest = candidateValues.cend(); - TxSetXDRFrameConstPtr highestTxSet; - ApplicableTxSetFrameConstPtr highestApplicableTxSet; - for (auto it = candidateValues.cbegin(); it != candidateValues.cend(); - ++it) - { - auto const& sv = *it; - TxSetXDRFrameConstPtr cTxSet; - auto const cTxSetResult = mPendingEnvelopes.getTxSet(sv.txSetHash); - if (auto const* ptr = - std::get_if(&cTxSetResult)) - { - cTxSet = *ptr; - } - // else: EmptyTxSet -> cTxSet stays null, handled by existing - - // Prefer applicable tx sets. `compareTxSets` down-ranks - // non-applicable ones. Without CAP-0083 every candidate here is - // applicable (a mismatched one can't be ratified), so the - // non-applicable case doesn't arise. Under CAP-0083 a - // non-applicable candidate may be selected and is replaced with an - // empty-tx-set value during balloting. - ApplicableTxSetFrameConstPtr cApplicableTxSet = nullptr; - if (cTxSet && cTxSet->previousLedgerHash() == lcl.hash) - { - cApplicableTxSet = cTxSet->prepareForApply(mApp, lcl.header); - } - - if (highest == candidateValues.cend() || - compareTxSets(highestApplicableTxSet, cApplicableTxSet, - highest->txSetHash, sv.txSetHash, - highestTxSet ? std::make_optional( - highestTxSet->encodedSize()) - : std::nullopt, - cTxSet ? std::make_optional(cTxSet->encodedSize()) - : std::nullopt, - lcl.header, candidatesHash)) - { - highest = it; - highestTxSet = cTxSet; - highestApplicableTxSet = std::move(cApplicableTxSet); - } - } - if (highest == candidateValues.cend()) - { - throw std::runtime_error( - "No highest candidate transaction set found"); - } - comp = *highest; - } - comp.upgrades.clear(); - for (auto const& upgrade : upgrades) - { - Value v(xdr::xdr_to_opaque(upgrade.second)); - comp.upgrades.emplace_back(v.begin(), v.end()); - } - - auto res = wrapStellarValue(comp); - return res; -} - -bool -HerderSCPDriver::hasUpgrades(Value const& v) -{ - StellarValue sv; - if (!toStellarValue(v, sv)) - { - return false; - } - return !sv.upgrades.empty(); -} - -ValueWrapperPtr -HerderSCPDriver::stripAllUpgrades(Value const& v) -{ - StellarValue sv; - if (!toStellarValue(v, sv)) - { - return nullptr; - } - - // Remove all upgrades - sv.upgrades.clear(); - - // Serialize back to Value - return wrapStellarValue(sv); -} - -uint32_t -HerderSCPDriver::getUpgradeNominationTimeoutLimit() const -{ - return mUpgrades.getParameters().mNominationTimeoutLimit.value_or( - std::numeric_limits::max()); -} - std::optional HerderSCPDriver::getTxSetDownloadWaitTime(Value const& v) const { @@ -1253,17 +927,11 @@ HerderSCPDriver::valueExternalized(uint64_t slotIndex, Value const& value) logQuorumInformationAndUpdateMetrics(slotIndex - 2); } - if (mLedgerSeqNominating != 0) - { - // stop nomination - // this may or may not be the ledger that is currently externalizing - // in both cases, we want to stop nomination as: - // either we're closing the current ledger (typical case) - // or we're going to trigger catchup from history - mSCP.stopNomination(mLedgerSeqNominating); - mCurrentValue.reset(); - mLedgerSeqNominating = 0; - } + // this may or may not be the ledger the local node started: either + // way its own proposal is obsolete (the ledger is closing, or catchup + // is about to trigger) + mCurrentValue.reset(); + mLedgerSeqProposing = 0; if (!mHerder.isTracking()) { @@ -1338,22 +1006,20 @@ HerderSCPDriver::logQuorumInformationAndUpdateMetrics(uint64_t index) } void -HerderSCPDriver::nominate(uint64_t slotIndex, NominationValueSupplier makeValue, - StellarValue const& previousValue, - bool constructValueNow) +HerderSCPDriver::startSlot(uint64_t slotIndex, ProposalValueSupplier makeValue, + bool constructValueNow) { ZoneScoped; - if (mLedgerSeqNominating != slotIndex) + if (mLedgerSeqProposing != slotIndex) { mCurrentValue.reset(); } - mLedgerSeqNominating = static_cast(slotIndex); + mLedgerSeqProposing = static_cast(slotIndex); if (constructValueNow && !mCurrentValue) { auto value = makeValue(); - // Publishing the tx set may externalize this slot, which stops its - // nomination. Do not restart it. - if (mLedgerSeqNominating != slotIndex) + // Publishing the tx set may externalize this slot. Do not restart it. + if (mLedgerSeqProposing != slotIndex) { return; } @@ -1361,28 +1027,85 @@ HerderSCPDriver::nominate(uint64_t slotIndex, NominationValueSupplier makeValue, // a leader. mCurrentValue = std::move(value); } - auto prevValue = xdr::xdr_to_opaque(previousValue); - mSCP.nominate( - slotIndex, - [this, slotIndex, - makeValue = std::move(makeValue)]() -> ValueWrapperPtr { - if (mLedgerSeqNominating != slotIndex) - { - return nullptr; - } - if (!mCurrentValue) - { - auto value = makeValue(); - // Publishing the tx set may externalize a pending value. - if (mLedgerSeqNominating != slotIndex) - { - return nullptr; - } - mCurrentValue = std::move(value); - } - return mCurrentValue; - }, - prevValue); + mSCP.startSlot(slotIndex, + [this, slotIndex, + makeValue = std::move(makeValue)]() -> ValueWrapperPtr { + if (mLedgerSeqProposing != slotIndex) + { + return nullptr; + } + if (!mCurrentValue) + { + auto value = makeValue(); + // Publishing the tx set may externalize a pending + // value. + if (mLedgerSeqProposing != slotIndex) + { + return nullptr; + } + mCurrentValue = std::move(value); + } + return mCurrentValue; + }); +} + +std::vector const& +HerderSCPDriver::getLeaderSchedule(uint64_t slotIndex) +{ + static std::vector const empty; + releaseAssert(threadIsMain()); + auto const& lcl = mLedgerManager.getLastClosedLedgerHeader(); + if (slotIndex != static_cast(lcl.header.ledgerSeq) + 1) + { + // The seed is the hash of the previous ledger's header: unknown for + // any slot but the next one. + return empty; + } + // Force a pending rebuild of the transitive quorum, if any + mPendingEnvelopes.isNodeDefinitelyInQuorum(mSCP.getLocalNodeID()); + auto const& quorum = mPendingEnvelopes.getCurrentlyTrackedQuorum(); + auto& cache = mLeaderSchedule; + if (cache.mSlotIndex != slotIndex || cache.mSeed != lcl.hash || + cache.mQuorumSize != quorum.size() || + cache.mQuorumRebuildCount != mPendingEnvelopes.getQuorumRebuildCount()) + { + std::vector nodes; + nodes.reserve(quorum.size()); + for (auto const& n : quorum) + { + nodes.emplace_back(n.first); + } + cache.mSchedule = computeLeaderSchedule(lcl.hash, std::move(nodes)); + cache.mSlotIndex = slotIndex; + cache.mSeed = lcl.hash; + cache.mQuorumSize = quorum.size(); + cache.mQuorumRebuildCount = mPendingEnvelopes.getQuorumRebuildCount(); + } + return cache.mSchedule; +} + +std::optional +HerderSCPDriver::getRoundLeader(uint64 slotIndex, uint32 round) +{ + auto const& schedule = getLeaderSchedule(slotIndex); + if (schedule.empty() || round == 0) + { + return std::nullopt; + } + return schedule[(round - 1) % schedule.size()]; +} + +bool +HerderSCPDriver::isProposalPending(uint64 slotIndex, NodeID const& leader) const +{ + return mPendingEnvelopes.hasPendingEnvelopeFrom(slotIndex, leader); +} + +bool +HerderSCPDriver::isLocalNodeRoundLeader(uint64_t slotIndex, uint32_t round) +{ + auto leader = getRoundLeader(slotIndex, round); + return leader && *leader == mSCP.getLocalNodeID(); } SCPQuorumSetPtr @@ -1432,22 +1155,44 @@ HerderSCPDriver::measureAndRecordBallotBlockedOnTxSet(uint64_t slotIndex, } void -HerderSCPDriver::nominatingValue(uint64_t slotIndex, Value const& value) +HerderSCPDriver::proposedValue(uint64_t slotIndex, uint32 round, + Value const& value) { - CLOG_DEBUG(Herder, "nominatingValue i:{} v: {}", slotIndex, + CLOG_DEBUG(Herder, "proposedValue i:{} round:{} v: {}", slotIndex, round, getValueString(value)); } void -HerderSCPDriver::updatedCandidateValue(uint64_t slotIndex, Value const& value) +HerderSCPDriver::enteredRound(uint64_t slotIndex, uint32 round, bool timedOut) { + auto it = mSCPExecutionTimes.find(slotIndex); + if (it != mSCPExecutionTimes.end()) + { + it->second.mMaxRound = std::max(it->second.mMaxRound, round); + } + if (round > 1) + { + // Only read the cached schedule here: this is called from inside SCP + // state changes (including state restore), where a quorum rebuild + // (database reads) does not belong. + auto const& cache = mLeaderSchedule; + std::string leader = "unknown"; + if (cache.mSlotIndex == slotIndex && !cache.mSchedule.empty()) + { + leader = toShortString( + cache.mSchedule[(round - 2) % cache.mSchedule.size()]); + } + CLOG_INFO(Herder, "Slot {} moved to round {}{} (round {} leader: {})", + slotIndex, round, timedOut ? " after a timeout" : "", + round - 1, leader); + } } void HerderSCPDriver::startedBallotProtocol(uint64_t slotIndex, SCPBallot const& ballot) { - recordSCPEvent(slotIndex, false); + recordBallotStart(slotIndex); // Prewarm the next ledger's signatures, after SCP has finished processing // (and emitting) this step: the mempool pull waits for the overlay. StellarValue sv; @@ -1560,40 +1305,34 @@ HerderSCPDriver::getExternalizeLag(NodeID const& id) const } void -HerderSCPDriver::recordNominationTrigger(uint64_t slotIndex) +HerderSCPDriver::recordTrigger(uint64_t slotIndex) { auto& timing = mSCPExecutionTimes[slotIndex]; - if (!timing.mTriggerStart && !timing.mNominationStart) + if (!timing.mTriggerStart) { timing.mTriggerStart = mApp.getClock().now(); + if (timing.mPrepareStart) + { + // The node already voted for the round leader's proposal before + // its own trigger fired. + mSCPMetrics.mTriggerToBallot.Update(std::chrono::nanoseconds(0)); + } } } void -HerderSCPDriver::recordSCPEvent(uint64_t slotIndex, bool isNomination) +HerderSCPDriver::recordBallotStart(uint64_t slotIndex) { - auto& timing = mSCPExecutionTimes[slotIndex]; VirtualClock::time_point start = mApp.getClock().now(); - if (isNomination) - { - if (!timing.mNominationStart) - { - timing.mNominationStart = start; - } - } - else + // The first entry into ballot ends the trigger-to-ballot interval + if (!timing.mPrepareStart && timing.mTriggerStart && + start >= *timing.mTriggerStart) { - // The first entry into ballot ends the trigger-to-ballot interval - if (!timing.mPrepareStart && timing.mTriggerStart && - start >= *timing.mTriggerStart) - { - mSCPMetrics.mTriggerToBallot.Update(start - *timing.mTriggerStart); - } - timing.mPrepareStart = - std::make_optional(start); + mSCPMetrics.mTriggerToBallot.Update(start - *timing.mTriggerStart); } + timing.mPrepareStart = std::make_optional(start); } void @@ -1686,33 +1425,20 @@ HerderSCPDriver::recordSCPExecutionMetrics(uint64_t slotIndex) auto& SCPTiming = SCPTimingIt->second; - mNominateTimeout.Update(SCPTiming.mNominationTimeoutCount); + mLeaderTimeout.Update(SCPTiming.mLeaderTimeoutCount); mPrepareTimeout.Update(SCPTiming.mPrepareTimeoutCount); - - if (SCPTiming.mNominationTimeoutCount > 0) + if (SCPTiming.mMaxRound > 0) { - auto const leaders = getSCP().getNominationLeaders(slotIndex); - std::string leaderStr; - for (auto const& leader : leaders) - { - if (!leaderStr.empty()) - { - leaderStr += ", "; - } - leaderStr += toShortString(leader); - } - CLOG_INFO(Herder, - "Nomination for slot {} timed out {} time(s) with " - "the following round leaders: [{}]", - slotIndex, SCPTiming.mNominationTimeoutCount, leaderStr); + mLeaderRound.Update(SCPTiming.mMaxRound); } - // Compute nomination time - if (SCPTiming.mNominationStart && SCPTiming.mPrepareStart) + if (SCPTiming.mLeaderTimeoutCount > 0) { - recordLogTiming(*SCPTiming.mNominationStart, *SCPTiming.mPrepareStart, - mSCPMetrics.mNominateToPrepare, "Nominate", threshold, - slotIndex); + CLOG_INFO(Herder, + "Slot {} timed out {} time(s) waiting for a round leader " + "(externalized in round {})", + slotIndex, SCPTiming.mLeaderTimeoutCount, + SCPTiming.mMaxRound); } // Compute prepare time @@ -2026,81 +1752,13 @@ HerderSCPDriver::TxSetValidityKeyHash::operator()( return res; } -uint64 -HerderSCPDriver::getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool const isLocalNode) const -{ - releaseAssert(!mLedgerManager.isApplying()); - Config const& cfg = mApp.getConfig(); - bool const unsupportedProtocol = protocolVersionIsBefore( - mApp.getLedgerManager() - .getLastClosedLedgerHeader() - .header.ledgerVersion, - APPLICATION_SPECIFIC_NOMINATION_LEADER_ELECTION_PROTOCOL_VERSION); - if (unsupportedProtocol || !cfg.VALIDATOR_WEIGHT_CONFIG.has_value() || - cfg.FORCE_OLD_STYLE_LEADER_ELECTION) - { - // Fall back on old weight algorithm if any of the following are true: - // 1. The network has not yet upgraded to - // APPLICATION_SPECIFIC_NOMINATION_LEADER_ELECTION_PROTOCOL_VERSION, - // 2. The node is using manual quorum set configuration, or - // 3. The node has the FORCE_OLD_STYLE_LEADER_ELECTION flag - // set - return SCPDriver::getNodeWeight(nodeID, qset, isLocalNode); - } - - ValidatorWeightConfig const& vwc = - mApp.getConfig().VALIDATOR_WEIGHT_CONFIG.value(); - - auto entryIt = vwc.mValidatorEntries.find(nodeID); - if (entryIt == vwc.mValidatorEntries.end()) - { - // This shouldn't be possible as the validator entries should contain - // all validators in the config. For this to happen, `getNodeWeight` - // would have to be called with a non-validator `nodeID`. Throw if - // building tests, and otherwise fall back on the old algorithm. - throw std::runtime_error( - fmt::format(FMT_STRING("Validator entry not found for node {}"), - toShortString(nodeID))); - } - - ValidatorEntry const& entry = entryIt->second; - auto homeDomainSizeIt = vwc.mHomeDomainSizes.find(entry.mHomeDomain); - if (homeDomainSizeIt == vwc.mHomeDomainSizes.end()) - { - // This shouldn't be possible as the home domain sizes should contain - // all home domains in the config. For this to happen, `getNodeWeight` - // would have to be called with a non-validator, or the config parser - // would have to allow a validator without a home domain. Throw if - // building tests, and otherwise fall back on the old algorithm. - throw std::runtime_error( - fmt::format(FMT_STRING("Home domain size not found for domain {}"), - entry.mHomeDomain)); - } - - auto qualityWeightIt = vwc.mQualityWeights.find(entry.mQuality); - if (qualityWeightIt == vwc.mQualityWeights.end()) - { - // This shouldn't be possible as the quality weights should contain all - // quality levels in the config. - throw std::runtime_error( - fmt::format(FMT_STRING("Quality weight not found for quality {}"), - static_cast(entry.mQuality))); - } - - // Node's weight is its quality's weight divided by the number of nodes in - // its home domain - releaseAssert(homeDomainSizeIt->second > 0); - return qualityWeightIt->second / homeDomainSizeIt->second; -} - std::optional -HerderSCPDriver::getNominationTimeouts(uint64_t slotIndex) const +HerderSCPDriver::getLeaderTimeouts(uint64_t slotIndex) const { auto it = mSCPExecutionTimes.find(slotIndex); if (it != mSCPExecutionTimes.end()) { - return it->second.mNominationTimeoutCount; + return it->second.mLeaderTimeoutCount; } return std::nullopt; } diff --git a/src/herder/HerderSCPDriver.h b/src/herder/HerderSCPDriver.h index 047412be30..1067c0575f 100644 --- a/src/herder/HerderSCPDriver.h +++ b/src/herder/HerderSCPDriver.h @@ -34,11 +34,6 @@ class VirtualTimer; struct StellarValue; struct SCPEnvelope; -// First protocol version supporting the application-specific weight function -// for SCP leader election. -ProtocolVersion constexpr APPLICATION_SPECIFIC_NOMINATION_LEADER_ELECTION_PROTOCOL_VERSION = - ProtocolVersion::V_22; - class HerderSCPDriver : public SCPDriver { public: @@ -57,8 +52,10 @@ class HerderSCPDriver : public SCPDriver } void recordSCPExecutionMetrics(uint64_t slotIndex); - void recordNominationTrigger(uint64_t slotIndex); - void recordSCPEvent(uint64_t slotIndex, bool isNomination); + // The local trigger for slotIndex fired. + void recordTrigger(uint64_t slotIndex); + // The local node entered the ballot protocol for slotIndex. + void recordBallotStart(uint64_t slotIndex); void recordSCPExternalizeEvent(uint64_t slotIndex, NodeID const& id, bool forceUpdateSelf); @@ -72,9 +69,7 @@ class HerderSCPDriver : public SCPDriver // value validation SCPDriver::ValidationLevel validateValue(uint64_t slotIndex, Value const& value, - bool nomination) const override; - ValueWrapperPtr extractValidValue(uint64_t slotIndex, - Value const& value) override; + bool proposal) const override; // value marshaling std::string toShortString(NodeID const& pk) const override; @@ -106,45 +101,47 @@ class HerderSCPDriver : public SCPDriver std::function cb) override; void stopTimer(uint64 slotIndex, int timerID) override; - std::chrono::milliseconds computeTimeout(uint32 roundNumber, - bool isNomination) override; -#ifdef BUILD_TESTS - std::chrono::milliseconds getNominationEmitDelayForTesting() const override; -#endif + std::chrono::milliseconds computeTimeout(uint32 roundNumber) override; // hashing support Hash getHashOf(std::vector> const& vals) const override; // core SCP - ValueWrapperPtr - combineCandidates(uint64_t slotIndex, - ValueWrapperPtrSet const& candidates) override; void valueExternalized(uint64_t slotIndex, Value const& value) override; - bool hasUpgrades(Value const& v) override; - ValueWrapperPtr stripAllUpgrades(Value const& v) override; - uint32_t getUpgradeNominationTimeoutLimit() const override; + // Deterministic leader schedule: the leader of `round` for `slotIndex`. + // The schedule is a permutation of the transitive quorum seeded by the + // hash of the last closed ledger header, so it is only defined for the + // next ledger to close (LCL+1). + bool isProposalPending(uint64 slotIndex, + NodeID const& leader) const override; + std::optional getRoundLeader(uint64 slotIndex, + uint32 round) override; + // The whole schedule for slotIndex (round r is led by entry + // (r - 1) % size). Empty when the schedule is not defined for the slot. + std::vector const& getLeaderSchedule(uint64_t slotIndex); + // True if the local node leads `round` of slotIndex. + bool isLocalNodeRoundLeader(uint64_t slotIndex, uint32_t round); // Update metrics indicating that a value was replaced with an empty-tx-set // value for slotIndex. void noteEmptyTxSetValueReplaced(uint64_t slotIndex) override; - // Start nomination for slotIndex. SCP calls makeValue only when the local - // node is a round leader that needs its own value, unless - // constructValueNow is set: then the value is constructed before - // nomination starts (and reused if this node becomes a leader). - // previousValue is the value from slotIndex-1 - void nominate(uint64_t slotIndex, NominationValueSupplier makeValue, - StellarValue const& previousValue, - bool constructValueNow = false); + // Start consensus on slotIndex. SCP calls makeValue only when the local + // node leads a round and needs its own value, unless constructValueNow + // is set: then the value is constructed first (and reused if this node + // becomes a leader). + void startSlot(uint64_t slotIndex, ProposalValueSupplier makeValue, + bool constructValueNow = false); SCPQuorumSetPtr getQSet(Hash const& qSetHash) override; // listeners void ballotDidHearFromQuorum(uint64_t slotIndex, SCPBallot const& ballot) override; - void nominatingValue(uint64_t slotIndex, Value const& value) override; - void updatedCandidateValue(uint64_t slotIndex, Value const& value) override; + void proposedValue(uint64_t slotIndex, uint32 round, + Value const& value) override; + void enteredRound(uint64_t slotIndex, uint32 round, bool timedOut) override; void startedBallotProtocol(uint64_t slotIndex, SCPBallot const& ballot) override; void acceptedBallotPrepared(uint64_t slotIndex, @@ -199,13 +196,6 @@ class HerderSCPDriver : public SCPDriver getTxSetDownloadWaitTime(Value const& v) const override; std::chrono::milliseconds getTxSetDownloadTimeout() const override; - // Application-specific weight function. This function uses the quality - // levels from automatic quorum set generation to determine the weight of a - // validator. It is designed to ensure that: - // 1. Orgs of equal quality have equal chances of winning leader election. - // 2. Higher quality orgs win more frequently than lower quality orgs. - uint64 getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const override; // For caching TxSet validity. Consist of {lcl.hash, txSetHash, // lowerBoundCloseTimeOffset, upperBoundCloseTimeOffset} using TxSetValidityKey = std::tuple; @@ -219,12 +209,14 @@ class HerderSCPDriver : public SCPDriver LedgerHeaderHistoryEntry const& lcl, ApplyTimeOffset closeTimeOffset) const; - // Get the number of nomination timeouts that occurred for a given slot - std::optional getNominationTimeouts(uint64_t slotIndex) const; + // Get the number of times the local node timed out waiting for a round + // leader in a given slot + std::optional getLeaderTimeouts(uint64_t slotIndex) const; // Elapsed local time from the trigger through entry into ballot, measured - // on the steady clock. Includes construction and incomplete nomination - // rounds. Missing timing history does not establish any allowance. + // on the steady clock. Includes construction and rounds whose leader + // never showed up. Missing timing history does not establish any + // allowance. std::chrono::milliseconds getTriggerToBallotDuration(uint64_t slotIndex) const; @@ -267,15 +259,12 @@ class HerderSCPDriver : public SCPDriver medida::Meter& mValueValid; medida::Meter& mValueInvalid; - // listeners - medida::Meter& mCombinedCandidates; - - // Timers for nomination and ballot protocols - medida::Timer& mNominateToPrepare; + // Timer for the ballot protocol medida::Timer& mPrepareToExternalize; - // Local trigger to first entry into ballot (proposal construction - // plus nomination); the drift allowance of the trigger anchor + // Local trigger to first entry into ballot (proposal construction, or + // waiting for the round leader's proposal); the drift allowance of + // the trigger anchor medida::Timer& mTriggerToBallot; // Timers tracking externalize messages @@ -300,8 +289,10 @@ class HerderSCPDriver : public SCPDriver SCPMetrics mSCPMetrics; - // Nomination timeouts per ledger - medida::Histogram& mNominateTimeout; + // Timeouts waiting for a round leader, per ledger + medida::Histogram& mLeaderTimeout; + // Round in which the ledger externalized (1 = first leader) + medida::Histogram& mLeaderRound; // Prepare timeouts per ledger medida::Histogram& mPrepareTimeout; // Unique values referenced per ledger @@ -313,11 +304,12 @@ class HerderSCPDriver : public SCPDriver struct SCPTiming { std::optional mTriggerStart; - std::optional mNominationStart; std::optional mPrepareStart; - // Nomination timeouts before first prepare - int64_t mNominationTimeoutCount{0}; + // Timeouts waiting for a round leader's proposal + int64_t mLeaderTimeoutCount{0}; + // Highest round the local node entered + uint32_t mMaxRound{0}; // Prepare timeouts before externalize int64_t mPrepareTimeoutCount{0}; @@ -331,13 +323,25 @@ class HerderSCPDriver : public SCPDriver }; // Map of time points for each slot to measure key protocol metrics: - // * nomination to first prepare + // * trigger to first prepare // * first prepare to externalize std::map mSCPExecutionTimes; - uint32_t mLedgerSeqNominating; + // Slot the local node started, and the value it built for it (if any) + uint32_t mLedgerSeqProposing; ValueWrapperPtr mCurrentValue; + // Leader schedule of the next ledger to close + struct LeaderScheduleCache + { + uint64_t mSlotIndex{0}; + Hash mSeed; + size_t mQuorumSize{0}; + uint64_t mQuorumRebuildCount{0}; + std::vector mSchedule; + }; + LeaderScheduleCache mLeaderSchedule; + // timers used by SCP // indexed by slotIndex, timerID std::map>> mSCPTimers; @@ -354,7 +358,7 @@ class HerderSCPDriver : public SCPDriver SCPDriver::ValidationLevel validateValueAgainstLocalState(uint64_t slotIndex, StellarValue const& sv, - bool nomination) const; + bool proposal) const; SCPDriver::ValidationLevel validatePastOrFutureValue(uint64_t slotIndex, StellarValue const& b, @@ -379,6 +383,6 @@ class HerderSCPDriver : public SCPDriver bool deserializeAndValidateStellarValue(uint64_t slotIndex, Value const& value, StellarValue& sv) const; - void extractValidUpgrades(StellarValue& sv, bool nomination) const; + void extractValidUpgrades(StellarValue& sv, bool proposal) const; }; } diff --git a/src/herder/LeaderSchedule.cpp b/src/herder/LeaderSchedule.cpp new file mode 100644 index 0000000000..cc9599a718 --- /dev/null +++ b/src/herder/LeaderSchedule.cpp @@ -0,0 +1,38 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#include "herder/LeaderSchedule.h" +#include "crypto/SHA.h" +#include "util/XDROperators.h" +#include "xdrpp/marshal.h" +#include +#include + +namespace stellar +{ + +std::vector +computeLeaderSchedule(Hash const& seed, std::vector nodes) +{ + std::vector> keyed; + keyed.reserve(nodes.size()); + for (auto& node : nodes) + { + SHA256 hasher; + hasher.add(seed); + hasher.add(xdr::xdr_to_opaque(node)); + keyed.emplace_back(hasher.finish(), std::move(node)); + } + std::sort(keyed.begin(), keyed.end()); + keyed.erase(std::unique(keyed.begin(), keyed.end()), keyed.end()); + + std::vector res; + res.reserve(keyed.size()); + for (auto& k : keyed) + { + res.emplace_back(std::move(k.second)); + } + return res; +} +} diff --git a/src/herder/LeaderSchedule.h b/src/herder/LeaderSchedule.h new file mode 100644 index 0000000000..9042d90eaa --- /dev/null +++ b/src/herder/LeaderSchedule.h @@ -0,0 +1,26 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#pragma once + +#include "xdr/Stellar-types.h" +#include + +namespace stellar +{ + +// Deterministic leader schedule (experimental; replaces SCP nomination). +// +// Returns `nodes` (duplicates removed) ordered by SHA256(seed || nodeID), +// ties (impossible in practice) broken by node ID. Round r (1-based) of the +// slot is led by entry (r - 1) % size, so every node gets a turn before any +// node leads twice. +// +// The order depends only on `seed` and the *set* of nodes, so every node that +// agrees on both computes the same schedule: with a symmetric quorum, seeding +// with the hash of the last closed ledger header gives the whole network the +// same leader for each round of the next ledger. +std::vector computeLeaderSchedule(Hash const& seed, + std::vector nodes); +} diff --git a/src/herder/PendingEnvelopes.cpp b/src/herder/PendingEnvelopes.cpp index 6c3a3b5af5..82b35f300f 100644 --- a/src/herder/PendingEnvelopes.cpp +++ b/src/herder/PendingEnvelopes.cpp @@ -1164,6 +1164,7 @@ PendingEnvelopes::rebuildQuorumTrackerState() { // rebuild quorum information using data sources starting with the // freshest source + ++mQuorumRebuildCount; mQuorumTracker.rebuild([&](NodeID const& id) -> SCPQuorumSetPtr { SCPQuorumSetPtr res; if (id == mHerder.getSCP().getLocalNodeID()) @@ -1195,6 +1196,32 @@ PendingEnvelopes::rebuildQuorumTrackerState() }); } +bool +PendingEnvelopes::hasPendingEnvelopeFrom(uint64 slotIndex, + NodeID const& node) const +{ + auto it = mEnvelopes.find(slotIndex); + if (it == mEnvelopes.end()) + { + return false; + } + for (auto const& f : it->second.mFetchingEnvelopes) + { + if (f.first.statement.nodeID == node) + { + return true; + } + } + for (auto const& r : it->second.mReadyEnvelopes) + { + if (r->getStatement().nodeID == node) + { + return true; + } + } + return false; +} + QuorumTracker::QuorumMap const& PendingEnvelopes::getCurrentlyTrackedQuorum() const { diff --git a/src/herder/PendingEnvelopes.h b/src/herder/PendingEnvelopes.h index 75f5973085..0011b5cda1 100644 --- a/src/herder/PendingEnvelopes.h +++ b/src/herder/PendingEnvelopes.h @@ -94,6 +94,7 @@ class PendingEnvelopes RandomEvictionCache mValueSizeCache; bool mRebuildQuorum; + uint64_t mQuorumRebuildCount{0}; QuorumTracker mQuorumTracker; medida::Counter& mProcessedCount; @@ -256,6 +257,17 @@ class PendingEnvelopes bool isNodeDefinitelyInQuorum(NodeID const& node); void rebuildQuorumTrackerState(); + // True if an envelope of `node` for `slotIndex` is being fetched (waiting + // for its tx set or quorum set) or is ready but not yet processed by SCP. + bool hasPendingEnvelopeFrom(uint64 slotIndex, NodeID const& node) const; + // Incremented every time the transitive quorum is rebuilt from scratch. + // Together with the quorum's size (it only grows between rebuilds) this + // identifies its node set. + uint64_t + getQuorumRebuildCount() const + { + return mQuorumRebuildCount; + } QuorumTracker::QuorumMap const& getCurrentlyTrackedQuorum() const; // updates internal state when an envelope was successfully processed diff --git a/src/herder/test/HerderTests.cpp b/src/herder/test/HerderTests.cpp index 0fc00e6b43..f151cbe81c 100644 --- a/src/herder/test/HerderTests.cpp +++ b/src/herder/test/HerderTests.cpp @@ -2675,23 +2675,15 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) return makeTxUpgradePair(herder, txSet, closeTime, emptyUpgradeSteps); }; auto makeEnvelope = [&s](HerderImpl& herder, TxPair const& p, Hash qSetHash, - uint64_t slotIndex, bool nomination) { + uint64_t slotIndex) { // herder must want the TxSet before receiving it, so we are sending it // fake envelope auto envelope = SCPEnvelope{}; envelope.statement.slotIndex = slotIndex; - if (nomination) - { - envelope.statement.pledges.type(SCP_ST_NOMINATE); - envelope.statement.pledges.nominate().votes.push_back(p.first); - envelope.statement.pledges.nominate().quorumSetHash = qSetHash; - } - else - { - envelope.statement.pledges.type(SCP_ST_PREPARE); - envelope.statement.pledges.prepare().ballot.value = p.first; - envelope.statement.pledges.prepare().quorumSetHash = qSetHash; - } + envelope.statement.pledges.type(SCP_ST_PREPARE); + envelope.statement.pledges.prepare().ballot.counter = 1; + envelope.statement.pledges.prepare().ballot.value = p.first; + envelope.statement.pledges.prepare().quorumSetHash = qSetHash; envelope.statement.nodeID = s.getPublicKey(); herder.signEnvelope(s, envelope); return envelope; @@ -2708,156 +2700,6 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) return makeTxSetFromTransactions(txs, *app, ApplyTimeOffset{}); }; - SECTION("combineCandidates") - { - auto& herder = static_cast(app->getHerder()); - - ValueWrapperPtrSet candidates; - - auto addToCandidates = [&](TxPair const& p) { - auto envelope = makeEnvelope( - herder, p, {}, herder.trackingConsensusLedgerIndex() + 1, true); - REQUIRE(herder.recvSCPEnvelope(envelope) == - Herder::ENVELOPE_STATUS_FETCHING); - REQUIRE(herder.recvTxSet(p.second->getContentsHash(), p.second)); - auto v = herder.getHerderSCPDriver().wrapValue(p.first); - candidates.emplace(v); - }; - - struct CandidateSpec - { - int const n; - int const nbOps; - uint32 const feeMulti; - TimePoint const closeTime; - std::optional const baseFeeIncrement; - }; - - std::vector txSetHashes; - std::vector txSetSizes; - std::vector txSetOpSizes; - std::vector closeTimes; - std::vector baseFees; - - auto addCandidateThenTest = [&](CandidateSpec const& spec) { - // Create a transaction set using the given parameters, combine - // it with the given closeTime and optionally a given base fee - // increment, and make it into a StellarValue to add to the list - // of candidates so far. Keep track of the hashes and sizes and - // operation sizes of all the transaction sets, all of the close - // times, and all of the base fee upgrades that we've seen, so that - // we can compute the expected result of combining all the - // candidates so far. (We're using base fees simply as one example - // of a type of upgrade, whose expected result is the maximum of all - // candidates'.) - auto [txSet, applicableTxSet] = - makeTransactions(spec.n, spec.nbOps, spec.feeMulti); - txSetHashes.push_back(txSet->getContentsHash()); - txSetSizes.push_back(applicableTxSet->size(lcl.header)); - txSetOpSizes.push_back(applicableTxSet->sizeOpTotal()); - closeTimes.push_back(spec.closeTime); - if (spec.baseFeeIncrement) - { - auto const baseFee = - lcl.header.baseFee + *spec.baseFeeIncrement; - baseFees.push_back(baseFee); - LedgerUpgrade ledgerUpgrade; - ledgerUpgrade.type(LEDGER_UPGRADE_BASE_FEE); - ledgerUpgrade.newBaseFee() = baseFee; - Value upgrade(xdr::xdr_to_opaque(ledgerUpgrade)); - SVUpgrades upgrades; - upgrades.emplace_back(upgrade.begin(), upgrade.end()); - addToCandidates( - makeTxUpgradePair(herder, txSet, spec.closeTime, upgrades)); - } - else - { - addToCandidates(makeTxPair(herder, txSet, spec.closeTime)); - } - - // Compute the expected transaction set, close time, and upgrade - // vector resulting from combining all the candidates so far. - auto const bestTxSetIndex = std::distance( - txSetSizes.begin(), - std::max_element(txSetSizes.begin(), txSetSizes.end())); - REQUIRE(txSetSizes.size() == closeTimes.size()); - auto const expectedHash = txSetHashes[bestTxSetIndex]; - auto const expectedCloseTime = closeTimes[bestTxSetIndex]; - SVUpgrades expectedUpgradeVector; - if (!baseFees.empty()) - { - LedgerUpgrade expectedLedgerUpgrade; - expectedLedgerUpgrade.type(LEDGER_UPGRADE_BASE_FEE); - expectedLedgerUpgrade.newBaseFee() = - *std::max_element(baseFees.begin(), baseFees.end()); - Value const expectedUpgradeValue( - xdr::xdr_to_opaque(expectedLedgerUpgrade)); - expectedUpgradeVector.emplace_back(expectedUpgradeValue.begin(), - expectedUpgradeValue.end()); - } - - // Combine all the candidates seen so far, and extract the - // returned StellarValue. - ValueWrapperPtr v = - herder.getHerderSCPDriver().combineCandidates(1, candidates); - StellarValue sv; - xdr::xdr_from_opaque(v->getValue(), sv); - - // Compare the returned StellarValue's contents with the - // expected ones that we computed above. - REQUIRE(isSignedStellarValue(sv)); - REQUIRE(sv.txSetHash == expectedHash); - REQUIRE(sv.closeTime == expectedCloseTime); - REQUIRE(sv.upgrades == expectedUpgradeVector); - }; - - // Test some list of candidates, comparing the output of - // combineCandidates() and the one we compute at each step. - - std::vector const specs{ - {0, 1, 100, 10, std::nullopt}, - {10, 1, 100, 5, std::make_optional(1)}, - {5, 3, 100, 20, std::make_optional(2)}, - {7, 2, 5, 30, std::make_optional(3)}}; - - std::for_each(specs.begin(), specs.end(), addCandidateThenTest); - - auto const bestTxSetIndex = std::distance( - txSetSizes.begin(), - std::max_element(txSetSizes.begin(), txSetSizes.end())); - REQUIRE(txSetOpSizes[bestTxSetIndex] == expectedOps); - - auto txSetL = makeTransactions(maxTxSetSize, 1, 101).first; - addToCandidates(makeTxPair(herder, txSetL, 20)); - auto txSetL2 = makeTransactions(maxTxSetSize, 1, 1000).first; - addToCandidates(makeTxPair(herder, txSetL2, 20)); - auto v = herder.getHerderSCPDriver().combineCandidates(1, candidates); - StellarValue sv; - xdr::xdr_from_opaque(v->getValue(), sv); - REQUIRE(isSignedStellarValue(sv)); - REQUIRE(sv.txSetHash == txSetL2->getContentsHash()); - - if (protocolVersionStartsFrom(protocolVersion, - MS_CLOSE_TIME_PROTOCOL_VERSION)) - { - // Test that the winning candidate keeps its full ms close time - // along with the redundant whole-second close time. - auto txSetL3 = makeTransactions(maxTxSetSize, 1, 2000).first; - StellarValue msVal = herder.makeStellarValue( - txSetL3->getContentsHash(), makeConsensusTime(20, 7), - emptyUpgradeSteps, root->getSecretKey()); - addToCandidates(TxPair{xdr::xdr_to_opaque(msVal), txSetL3}); - auto v2 = - herder.getHerderSCPDriver().combineCandidates(1, candidates); - StellarValue sv2; - xdr::xdr_from_opaque(v2->getValue(), sv2); - REQUIRE(sv2.ext.v() == STELLAR_VALUE_SIGNED_MS); - REQUIRE(sv2.closeTime == 20); - REQUIRE(getConsensusTime(sv2).milliseconds() == 20007); - REQUIRE(sv2.txSetHash == txSetL3->getContentsHash()); - } - } - SECTION("validateValue signatures") { auto& herder = static_cast(app->getHerder()); @@ -2869,7 +2711,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) { // make sure that txSet0 is loaded auto p = makeTxPair(herder, txSet0, ct); - auto envelope = makeEnvelope(herder, p, {}, seq, true); + auto envelope = makeEnvelope(herder, p, {}, seq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet0->getContentsHash(), txSet0)); @@ -2887,19 +2729,19 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) } SECTION("invalid") { - auto checkInvalid = [&](StellarValue const& sv, bool nomination) { + auto checkInvalid = [&](StellarValue const& sv, bool proposal) { auto v = xdr::xdr_to_opaque(sv); - REQUIRE(scp.validateValue(seq, v, nomination) == + REQUIRE(scp.validateValue(seq, v, proposal) == SCPDriver::kInvalidValue); }; - auto testInvalidValue = [&](bool isNomination) { + auto testInvalidValue = [&](bool isProposal) { SECTION("basic value") { auto basicVal = StellarValue(txSet0->getContentsHash(), ct, emptyUpgradeSteps, STELLAR_VALUE_BASIC); - checkInvalid(basicVal, isNomination); + checkInvalid(basicVal, isProposal); } SECTION("signed value") { @@ -2911,28 +2753,28 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) SECTION("missing signature") { getLcValueSignature(sv).signature.clear(); - checkInvalid(sv, isNomination); + checkInvalid(sv, isProposal); } SECTION("wrong signature") { getLcValueSignature(sv).signature[0] ^= 1; - checkInvalid(sv, isNomination); + checkInvalid(sv, isProposal); } SECTION("wrong signature 2") { getLcValueSignature(sv).nodeID.ed25519()[0] ^= 1; - checkInvalid(sv, isNomination); + checkInvalid(sv, isProposal); } } }; - SECTION("nomination") + SECTION("proposal") { - testInvalidValue(/* isNomination */ true); + testInvalidValue(/* isProposal */ true); } SECTION("ballot") { - testInvalidValue(/* isNomination */ false); + testInvalidValue(/* isProposal */ false); } } @@ -2945,10 +2787,6 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) SCPDriver::kInvalidValue); REQUIRE(scp.validateValue(seq, v, false) == SCPDriver::kInvalidValue); - - ValueWrapperPtr extracted; - REQUIRE_NOTHROW(extracted = scp.extractValidValue(seq, v)); - REQUIRE(extracted == nullptr); }; SECTION("signed value with empty-tx-set hash") @@ -2985,13 +2823,13 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) // once the protocol allows them. This is the assertion that // catches an inverted check in deserializeAndValidateStellarValue. REQUIRE(scp.validateValue(seq, emptyTxSetValue, - /*nomination=*/false) == + /*proposal=*/false) == (allowed ? SCPDriver::kFullyValidatedValue : SCPDriver::kInvalidValue)); - // Nomination path: empty-tx-set values are rejected by design. + // Proposal path: empty-tx-set values are rejected by design. REQUIRE(scp.validateValue(seq, emptyTxSetValue, - /*nomination=*/true) == + /*proposal=*/true) == SCPDriver::kInvalidValue); } } @@ -3048,7 +2886,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) emptyUpgradeSteps, root->getSecretKey()); auto val = TxPair{xdr::xdr_to_opaque(sv), txSet}; auto const seq = herder.trackingConsensusLedgerIndex() + 1; - auto envelope = makeEnvelope(herder, val, {}, seq, true); + auto envelope = makeEnvelope(herder, val, {}, seq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet->getContentsHash(), txSet)); @@ -3122,7 +2960,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) // All hits during the whole SCP round. One of them is the validity // check that determines whether or not to replace the transaction set // with an empty one (CAP-0083). - uint64_t const expectedHits = 11; + uint64_t const expectedHits = 9; REQUIRE(cache.getCounters().mHits == expectedHits); // One miss from the initial makeTxSetFromTransactions REQUIRE(cache.getCounters().mMisses == 1); @@ -3173,13 +3011,10 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) auto p2 = makeTxPair(herder, transactions1, 10); // use current + 1 to allow for any value (old values get filtered more) auto lseq = herder.trackingConsensusLedgerIndex() + 1; - auto saneEnvelopeQ1T1 = - makeEnvelope(herder, p1, saneQSet1Hash, lseq, true); - auto saneEnvelopeQ1T2 = - makeEnvelope(herder, p2, saneQSet1Hash, lseq, true); - auto saneEnvelopeQ2T1 = - makeEnvelope(herder, p1, saneQSet2Hash, lseq, true); - auto bigEnvelope = makeEnvelope(herder, p1, bigQSetHash, lseq, true); + auto saneEnvelopeQ1T1 = makeEnvelope(herder, p1, saneQSet1Hash, lseq); + auto saneEnvelopeQ1T2 = makeEnvelope(herder, p2, saneQSet1Hash, lseq); + auto saneEnvelopeQ2T1 = makeEnvelope(herder, p1, saneQSet2Hash, lseq); + auto bigEnvelope = makeEnvelope(herder, p1, bigQSetHash, lseq); TxSetXDRFrameConstPtr malformedTxSet; if (transactions1->isGeneralizedTxSet()) @@ -3204,7 +3039,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) } auto malformedTxSetPair = makeTxPair(herder, malformedTxSet, 10); auto malformedTxSetEnvelope = - makeEnvelope(herder, malformedTxSetPair, saneQSet1Hash, lseq, true); + makeEnvelope(herder, malformedTxSetPair, saneQSet1Hash, lseq); SECTION("return FETCHING until fetched") { @@ -3357,7 +3192,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) { // make sure that txSet0 is loaded auto p = makeTxPair(herder, txSet0, ct); - auto envelope = makeEnvelope(herder, p, {}, seq, true); + auto envelope = makeEnvelope(herder, p, {}, seq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet0->getContentsHash(), txSet0)); @@ -3365,7 +3200,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) auto check = [&](StellarValue const& sv, bool expectValid) { auto v = xdr::xdr_to_opaque(sv); - auto res = scp.validateValue(seq, v, /*nomination=*/true); + auto res = scp.validateValue(seq, v, /*proposal=*/true); if (expectValid) { REQUIRE(res == SCPDriver::kFullyValidatedValue); @@ -3428,14 +3263,14 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) REQUIRE(scp.validateValue( pastSlot, xdr::xdr_to_opaque(makeLegacySignedValue(ct)), - /*nomination=*/false) == + /*proposal=*/false) == SCPDriver::kMaybeValidNotCurrentValue); // Newer-than-externalized close time for an older slot: // rejected REQUIRE(scp.validateValue( pastSlot, xdr::xdr_to_opaque(makeLegacySignedValue(ct + 60)), - /*nomination=*/false) == SCPDriver::kInvalidValue); + /*proposal=*/false) == SCPDriver::kInvalidValue); } SECTION("legacy signed value rejected for future slots") { @@ -3445,7 +3280,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) REQUIRE(scp.validateValue( seq + 1, xdr::xdr_to_opaque(makeLegacySignedValue(ct)), - /*nomination=*/false) == SCPDriver::kInvalidValue); + /*proposal=*/false) == SCPDriver::kInvalidValue); } SECTION("legacy empty-tx-set value rejected for ratifiable slots") { @@ -3455,7 +3290,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) xdr::xdr_from_opaque(legacyEmpty, esv); REQUIRE(esv.ext.v() == STELLAR_VALUE_EMPTY_TX_SET); REQUIRE(scp.validateValue(seq, legacyEmpty, - /*nomination=*/false) == + /*proposal=*/false) == SCPDriver::kInvalidValue); } SECTION("closeTime inconsistent with closeTimeMs") @@ -3576,7 +3411,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) rand_uniform(1, 999)), emptyUpgradeSteps, s); auto val = TxPair{xdr::xdr_to_opaque(msv), txSet}; - auto envelope = makeEnvelope(herder, val, {}, zSeq, true); + auto envelope = makeEnvelope(herder, val, {}, zSeq); REQUIRE(herder.recvSCPEnvelope(envelope) == Herder::ENVELOPE_STATUS_FETCHING); REQUIRE(herder.recvTxSet(txSet->getContentsHash(), txSet)); @@ -3673,7 +3508,7 @@ testSCPDriver(uint32 protocolVersion, uint32_t maxTxSetSize, size_t expectedOps) // triggers catchup herder.lostSync(); REQUIRE(scp.validateValue(seq + 1, xdr::xdr_to_opaque(msv), - /*nomination=*/false) == + /*proposal=*/false) == SCPDriver::kMaybeValidNotCurrentValue); } } @@ -3691,64 +3526,6 @@ TEST_CASE("SCP Driver", "[herder][acceptance]") } } -// Test combineCandidates handling of candidates where -// previousLedgerHash != LCL.hash -TEST_CASE("combineCandidates with mismatched previousLedgerHash candidate", - "[herder][bug]") -{ - Config cfg(getTestConfig()); - - VirtualClock clock; - auto app = createTestApplication(clock, cfg); - auto& herder = dynamic_cast(app->getHerder()); - auto& pe = herder.getPendingEnvelopes(); - auto& driver = herder.getHerderSCPDriver(); - - auto const& lcl = app->getLedgerManager().getLastClosedLedgerHeader(); - uint32_t const ver = lcl.header.ledgerVersion; - TimePoint const closeTime = lcl.header.scpValue.closeTime + 1; - uint64_t const slotIndex = lcl.header.ledgerSeq + 1; - - // Two structurally-valid empty tx sets that differ only in - // previousLedgerHash. - auto goodTxSet = TxSetXDRFrame::makeEmpty(lcl.hash, ver); // matches LCL - auto badTxSet = TxSetXDRFrame::makeEmpty(sha256("not the LCL hash"), - ver); // mismatched - - ValueWrapperPtrSet candidates; - // Register the tx set so combineCandidates' getTxSet() returns it, then add - // a candidate value referencing it. - auto addCandidate = [&](TxSetXDRFrameConstPtr const& txSet) { - pe.addTxSet(txSet->getContentsHash(), slotIndex, txSet); - StellarValue sv = herder.makeStellarValue( - txSet->getContentsHash(), makeConsensusTime(closeTime), - emptyUpgradeSteps, cfg.NODE_SEED); - candidates.emplace(driver.wrapValue(xdr::xdr_to_opaque(sv))); - }; - auto combinedTxSetHash = [&]() { - ValueWrapperPtr result = - driver.combineCandidates(slotIndex, candidates); - StellarValue sv; - xdr::xdr_from_opaque(result->getValue(), sv); - return sv.txSetHash; - }; - - SECTION("prefer applicable candidate over mismatched candidate") - { - addCandidate(goodTxSet); - addCandidate(badTxSet); - REQUIRE(combinedTxSetHash() == goodTxSet->getContentsHash()); - } - - SECTION("all candidates have mismatched previousLedgerHash") - { - // If the *only* option is a candidate with a mismatched - // previousLedgerHash, choose it. - addCandidate(badTxSet); - REQUIRE(combinedTxSetHash() == badTxSet->getContentsHash()); - } -} - TEST_CASE("SCP checkpoint", "[catchup][herder]") { auto networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); @@ -4728,11 +4505,14 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") auto networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); // A and B keep closing ledgers in real time while C is fed slot 2 by // hand. Use a longer close time so that C's slot-2 EXTERNALIZE reaches - // A and B before they close ledger 3. + // A and B before they close ledger 3. C is not connected, so it only has + // slot 2's (empty) transaction set once it built one itself, i.e. once it + // led a round: with three nodes that can take until round 3 (two leader + // timeouts, 3 s). auto confGen = [](int i) { auto cfg = getTestConfig(i); cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.ARTIFICIALLY_SET_CLOSE_TIME_FOR_TESTING = 5000; + cfg.ARTIFICIALLY_SET_CLOSE_TIME_FOR_TESTING = 10000; return cfg; }; auto simulation = std::make_shared(networkID, confGen); @@ -4796,6 +4576,19 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") ++actualTypes[env.statement.pledges.type()]; } + std::string actual; + for (auto const& [type, count] : actualTypes) + { + actual += + fmt::format("type {}: {}; ", static_cast(type), count); + } + UNSCOPED_INFO("scphistory of ledger " + << ledgerNum << " on " + << node->getConfig().toShortString( + node->getConfig().NODE_SEED.getPublicKey()) + << ": " << actual << "(LCL " + << node->getLedgerManager().getLastClosedLedgerNum() + << ")"); return actualTypes == expectedTypes; }; @@ -4829,13 +4622,29 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") std::vector BEnvs = herderB.getSCP().getLatestMessagesSend(2); // Pass A and B's messages to C + auto const tFeedC = A->getClock().now(); + UNSCOPED_INFO( + "feeding C: A LCL " + << A->getLedgerManager().getLastClosedLedgerNum() << ", C round " + << dynamic_cast(C->getHerder()).getSCP().getLeaderRound(2) + << ", C sent " + << dynamic_cast(C->getHerder()) + .getSCP() + .getLatestMessagesSend(2) + .size()); for (auto const& env : AEnvs) { - C->getHerder().recvSCPEnvelope(env); + auto st = C->getHerder().recvSCPEnvelope(env); + UNSCOPED_INFO("C status for A's message type " + << static_cast(env.statement.pledges.type()) << ": " + << static_cast(st)); } for (auto const& env : BEnvs) { - C->getHerder().recvSCPEnvelope(env); + auto st = C->getHerder().recvSCPEnvelope(env); + UNSCOPED_INFO("C status for B's message type " + << static_cast(env.statement.pledges.type()) << ": " + << static_cast(st)); } // Crank C until it is on ledger 2 @@ -4843,15 +4652,31 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") [&]() { return C->getLedgerManager().getLastClosedLedgerNum() == 2; }, 4 * simulation->getExpectedLedgerCloseTime(), false); + UNSCOPED_INFO("C reached ledger 2 after " + << std::chrono::duration_cast( + A->getClock().now() - tFeedC) + .count() + << " ms"); + // Get messages from C HerderImpl& herderC = dynamic_cast(C->getHerder()); std::vector CEnvs = herderC.getSCP().getLatestMessagesSend(2); // Pass C's messages to A and B + INFO("C sent " << CEnvs.size() << " message(s) for ledger 2; A at LCL " + << A->getLedgerManager().getLastClosedLedgerNum() + << ", B at LCL " + << B->getLedgerManager().getLastClosedLedgerNum() + << " when they are delivered"); for (auto const& env : CEnvs) { - A->getHerder().recvSCPEnvelope(env); - B->getHerder().recvSCPEnvelope(env); + INFO("C message type " + << static_cast(env.statement.pledges.type())); + auto const statusA = A->getHerder().recvSCPEnvelope(env); + auto const statusB = B->getHerder().recvSCPEnvelope(env); + UNSCOPED_INFO("delivery status A " << static_cast(statusA) + << ", B " + << static_cast(statusB)); } // Crank A and B until they're on at least ledger 3 @@ -4882,1088 +4707,6 @@ TEST_CASE("SCP message capture from previous ledger", "[herder]") REQUIRE(checkSCPHistoryEntries(C, 2, expectedTypes)); } -using Topology = std::pair, std::vector>; - -// Generate a Topology with a single org containing 3 validators of HIGH quality -static Topology -simpleThreeNode() -{ - // Generate validators - std::vector sks; - std::vector validators; - int constexpr numValidators = 3; - for (int i = 0; i < numValidators; ++i) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}", i); - entry.mHomeDomain = "A"; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - return {sks, validators}; -} - -// Generate a topology with 3 orgs of HIGH quality. Two orgs have 3 validators -// and one org has 5 validators. -static Topology -unbalancedOrgs() -{ - // Generate validators - std::vector sks; - std::vector validators; - int constexpr numValidators = 11; - for (int i = 0; i < numValidators; ++i) - { - // Orgs A and B have 3 validators each. Org C has 5 validators. - std::string org = "C"; - if (i < 3) - { - org = "A"; - } - else if (i < 6) - { - org = "B"; - } - - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}", i); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - return {sks, validators}; -} - -// Generate a tier 1-like topology. This topology has 7 HIGH quality orgs, each -// with 3 validators. -static Topology -tier1Like() -{ - std::vector sks; - std::vector validators; - int constexpr numOrgs = 7; - int constexpr validatorsPerOrg = 3; - - for (int i = 0; i < numOrgs; ++i) - { - std::string const org = fmt::format("org-{}", i); - for (int j = 0; j < validatorsPerOrg; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}-{}", i, j); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - } - - return {sks, validators}; -} - -// Generate a slightly unbalanced topology. This topology has 7 HIGH quality -// orgs, 6 of which have 3 validators and 1 has 5 validators. -static Topology -slightlyUnbalancedOrgs() -{ - std::vector sks; - std::vector validators; - int constexpr numOrgs = 7; - - for (int i = 0; i < numOrgs; ++i) - { - std::string const org = fmt::format("org-{}", i); - int const numValidators = i == 0 ? 5 : 3; - for (int j = 0; j < numValidators; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-{}-{}", i, j); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - } - - return {sks, validators}; -} - -// Returns a random quality up to `maxQuality` -static ValidatorQuality -randomQuality(ValidatorQuality maxQuality) -{ - return static_cast(rand_uniform( - static_cast(ValidatorQuality::VALIDATOR_LOW_QUALITY), - static_cast(maxQuality))); -} - -// Returns the minimum size an org of quality `q` can have -static int constexpr minOrgSize(ValidatorQuality q) -{ - switch (q) - { - case ValidatorQuality::VALIDATOR_LOW_QUALITY: - case ValidatorQuality::VALIDATOR_MED_QUALITY: - return 1; - case ValidatorQuality::VALIDATOR_HIGH_QUALITY: - case ValidatorQuality::VALIDATOR_CRITICAL_QUALITY: - return 3; - } -} - -// Generate a random topology with up to `maxValidators` validators. Ensures at -// least one org is HIGH quality. -static Topology -randomTopology(int maxValidators) -{ - int const numValidators = rand_uniform(3, maxValidators); - int constexpr minCritOrgSize = - minOrgSize(ValidatorQuality::VALIDATOR_CRITICAL_QUALITY); - - // Generate validators - int curOrg = 0; - int curOrgSize = 0; - ValidatorQuality curQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - std::vector sks(numValidators); - std::vector validators(numValidators); - for (int i = 0; i < numValidators; ++i) - { - if (curOrgSize >= minOrgSize(curQuality) && rand_flip()) - { - // Start new org - ++curOrg; - curOrgSize = 0; - curQuality = - randomQuality(numValidators - i >= minCritOrgSize - ? ValidatorQuality::VALIDATOR_CRITICAL_QUALITY - : ValidatorQuality::VALIDATOR_MED_QUALITY); - } - - std::string const org = fmt::format("org-{}", curOrg); - SecretKey const& key = sks.at(i) = SecretKey::pseudoRandomForTesting(); - - ValidatorEntry& entry = validators.at(i); - entry.mName = fmt::format("validator-{}", i); - entry.mHomeDomain = org; - entry.mQuality = curQuality; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - - ++curOrgSize; - } - - return {sks, validators}; -} - -// Expected weight of an org with quality `orgQuality` in a topology with a max -// quality of `maxQuality` and or quality counts of `orgQualityCounts`. This -// function normalizes the weight so that the highest quality has a weight of -// `1`. -static double -expectedOrgNormalizedWeight( - std::unordered_map const& orgQualityCounts, - ValidatorQuality maxQuality, ValidatorQuality orgQuality) -{ - if (orgQuality == ValidatorQuality::VALIDATOR_LOW_QUALITY) - { - return 0.0; - } - - double normalizedWeight = 1.0; - - // For each quality level higher than `orgQuality`, divide the weight by 10 - // times the number of orgs at that quality level - for (int q = static_cast(maxQuality); q > static_cast(orgQuality); - --q) - { - normalizedWeight /= - 10 * orgQualityCounts.at(static_cast(q)); - } - return normalizedWeight; -} - -// Expected weight of a validator in an org of size `orgSize` with quality -// `orgQuality`. `maxQuality` is the maximum quality present in the -// configuration. This function normalizes the weight so that the highest -// organization-level quality has a weight of `1`. -static double -expectedNormalizedWeight( - std::unordered_map const& orgQualityCounts, - ValidatorQuality maxQuality, ValidatorQuality orgQuality, int orgSize) -{ - return expectedOrgNormalizedWeight(orgQualityCounts, maxQuality, - orgQuality) / - orgSize; -} - -// Collect information about the qualities and sizes of organizations in -// `validators` and store them in `maxQuality`, `orgQualities`, `orgSizes`, and -// `orgQualityCounts`. -static void -collectOrgInfo(ValidatorQuality& maxQuality, - std::unordered_map& orgQualities, - std::unordered_map& orgSizes, - std::unordered_map& orgQualityCounts, - std::vector const& validators) -{ - maxQuality = ValidatorQuality::VALIDATOR_LOW_QUALITY; - ValidatorQuality minQuality = ValidatorQuality::VALIDATOR_CRITICAL_QUALITY; - std::unordered_map> - orgsByQuality; - for (ValidatorEntry const& validator : validators) - { - maxQuality = std::max(maxQuality, validator.mQuality); - minQuality = std::min(minQuality, validator.mQuality); - orgQualities[validator.mHomeDomain] = validator.mQuality; - ++orgSizes[validator.mHomeDomain]; - orgsByQuality[validator.mQuality].insert(validator.mHomeDomain); - } - - // Count orgs at each quality level - for (int q = static_cast(minQuality); - q <= static_cast(maxQuality); ++q) - { - orgQualityCounts[static_cast(q)] = - orgsByQuality[static_cast(q)].size(); - if (q != static_cast(minQuality)) - { - // Add virtual org covering next lower quality level - ++orgQualityCounts[static_cast(q)]; - } - } -} - -// Given a list of validators, test that the weights of the validators herder -// reports are correct -static void -testWeights(std::vector const& validators) -{ - Config cfg = getTestConfig(0); - - cfg.generateQuorumSetForTesting(validators); - - VirtualClock clock; - Application::pointer app = createTestApplication(clock, cfg); - - // Collect info about orgs - ValidatorQuality maxQuality; - std::unordered_map orgQualities; - std::unordered_map orgSizes; - std::unordered_map orgQualityCounts; - collectOrgInfo(maxQuality, orgQualities, orgSizes, orgQualityCounts, - validators); - - // Check per-validator weights - HerderImpl& herder = dynamic_cast(app->getHerder()); - std::unordered_map normalizedOrgWeights; - for (ValidatorEntry const& validator : validators) - { - uint64_t weight = herder.getHerderSCPDriver().getNodeWeight( - validator.mKey, cfg.QUORUM_SET, false); - double normalizedWeight = - static_cast(weight) / static_cast(UINT64_MAX); - normalizedOrgWeights[validator.mHomeDomain] += normalizedWeight; - - std::string const& org = validator.mHomeDomain; - REQUIRE_THAT(normalizedWeight, - Catch::Matchers::WithinAbs( - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(org), - orgSizes.at(org)), - 0.0001)); - } - - // Check per-org weights - for (auto const& [org, weight] : normalizedOrgWeights) - { - REQUIRE_THAT( - weight, Catch::Matchers::WithinAbs( - expectedOrgNormalizedWeight( - orgQualityCounts, maxQuality, orgQualities.at(org)), - 0.0001)); - } -} - -// Test that HerderSCPDriver::getNodeWeight produces weights that result in a -// fair distribution of nomination wins. -TEST_CASE("getNodeWeight", "[herder]") -{ - SECTION("3 tier 1 validators, 1 org") - { - testWeights(simpleThreeNode().second); - } - - SECTION("11 tier 1 validators, 3 unbalanced orgs") - { - testWeights(unbalancedOrgs().second); - } - - SECTION("Tier1-like topology") - { - testWeights(tier1Like().second); - } - - SECTION("Tier1-like topology with a single unbalanced org") - { - testWeights(slightlyUnbalancedOrgs().second); - } - - SECTION("Random topology") - { - // Test weights for 1000 random topologies of up to 200 validators - for (int i = 0; i < 1000; ++i) - { - testWeights(randomTopology(200).second); - } - } -} - -static Value -getRandomValue() -{ - auto h = sha256(fmt::format("value {}", getGlobalRandomEngine()())); - return xdr::xdr_to_opaque(h); -} - -// A test version of NominationProtocol that exposes `updateRoundLeaders` -class TestNominationProtocol : public NominationProtocol -{ - public: - TestNominationProtocol(Slot& slot) : NominationProtocol(slot) - { - } - - std::set const& - updateRoundLeadersForTesting( - std::optional const& previousValue = std::nullopt) - { - mPreviousValue = previousValue.value_or(getRandomValue()); - updateRoundLeaders(); - return getLeaders(); - } - - // Detect fast timeouts by examining the final round number - bool - fastTimedOut() const - { - return mRoundNumber > 0; - } -}; - -// Test nomination over `numLedgers` slots. After running, check that the win -// percentages of each node and org are within 5% of the expected win -// percentages. -static void -testWinProbabilities(std::vector const& sks, - std::vector const& validators, - int const numLedgers) -{ - REQUIRE(sks.size() == validators.size()); - - // Collect info about orgs - ValidatorQuality maxQuality; - std::unordered_map orgQualities; - std::unordered_map orgSizes; - std::unordered_map orgQualityCounts; - collectOrgInfo(maxQuality, orgQualities, orgSizes, orgQualityCounts, - validators); - - // Generate a config - Config cfg = getTestConfig(); - cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.generateQuorumSetForTesting(validators); - cfg.NODE_SEED = sks.front(); - - // Create an application - VirtualClock clock; - Application::pointer app = createTestApplication(clock, cfg); - - // Run for `numLedgers` slots, recording the number of times each - // node wins nomination - UnorderedMap publishCounts; - HerderImpl& herder = dynamic_cast(app->getHerder()); - SCP& scp = herder.getSCP(); - int fastTimeouts = 0; - for (int i = 0; i < numLedgers; ++i) - { - auto s = std::make_shared(i, scp); - TestNominationProtocol np(*s); - - std::set const& leaders = np.updateRoundLeadersForTesting(); - REQUIRE(leaders.size() == 1); - for (NodeID const& leader : leaders) - { - ++publishCounts[leader]; - } - - if (np.fastTimedOut()) - { - ++fastTimeouts; - } - } - - CLOG_INFO(Herder, "Fast Timeouts: {} ({}%)", fastTimeouts, - fastTimeouts * 100.0 / numLedgers); - - // Compute total expected normalized weight across all nodes - double totalNormalizedWeight = 0.0; - for (ValidatorEntry const& validator : validators) - { - totalNormalizedWeight += - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(validator.mHomeDomain), - orgSizes.at(validator.mHomeDomain)); - } - - // Check validator win rates - std::map orgPublishCounts; - for (ValidatorEntry const& validator : validators) - { - NodeID const& nodeID = validator.mKey; - int publishCount = publishCounts[nodeID]; - - // Compute and report node's win rate - double winRate = static_cast(publishCount) / numLedgers; - CLOG_INFO(Herder, "Node {} win rate: {} (published {} ledgers)", - cfg.toShortString(nodeID), winRate, publishCount); - - // Expected win rate is `weight / total weight` - double expectedWinRate = - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(validator.mHomeDomain), - orgSizes.at(validator.mHomeDomain)) / - totalNormalizedWeight; - - // Check that actual win rate is within .05 of expected win - // rate. - REQUIRE_THAT(winRate, - Catch::Matchers::WithinAbs(expectedWinRate, 0.05)); - - // Record org publish counts for the next set of checks - orgPublishCounts[validator.mHomeDomain] += publishCount; - } - - // Check org win rates - for (auto const& [org, count] : orgPublishCounts) - { - // Compute and report org's win rate - double winRate = static_cast(count) / numLedgers; - CLOG_INFO(Herder, "Org {} win rate: {} (published {} ledgers)", org, - winRate, count); - - // Expected win rate is `weight / total weight` - double expectedWinRate = - expectedOrgNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(org)) / - totalNormalizedWeight; - - // Check that actual win rate is within .05 of expected win - // rate. - REQUIRE_THAT(winRate, - Catch::Matchers::WithinAbs(expectedWinRate, 0.05)); - } -} - -// Test that the nomination algorithm produces a fair distribution of ledger -// publishers. -TEST_CASE("Fair nomination win rates", "[herder]") -{ - SECTION("3 tier 1 validators, 1 org") - { - auto [sks, validators] = simpleThreeNode(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("11 tier 1 validators, 3 unbalanced orgs") - { - auto [sks, validators] = unbalancedOrgs(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("Tier 1-like topology") - { - auto [sks, validators] = tier1Like(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("Tier 1-like topology with a single unbalanced org") - { - auto [sks, validators] = slightlyUnbalancedOrgs(); - testWinProbabilities(sks, validators, 10000); - } - - SECTION("Random topology") - { - for (int i = 0; i < 10; ++i) - { - auto [sks, validators] = randomTopology(50); - testWinProbabilities(sks, validators, 10000); - } - } -} - -namespace -{ -// Returns a new `Topology` with the last org in `t` replaced with a new org -// with 3 validators. Requires that the last org in `t` have 3 validators and be -// contiguous at the back of the validators vecto. -Topology -replaceOneOrg(Topology const& t) -{ - Topology t2(t); // Copy the topology - auto& [sks, validators] = t2; - REQUIRE(sks.size() == validators.size()); - - // Give the org a unique name - std::string const orgName = "org-replaced"; - - // Double check that the new org name is unique - for (ValidatorEntry const& v : validators) - { - REQUIRE(v.mHomeDomain != orgName); - } - - // Remove the last org - constexpr int validatorsPerOrg = 3; - sks.resize(sks.size() - validatorsPerOrg); - validators.resize(validators.size() - validatorsPerOrg); - - // Add new org with 3 validators - int constexpr numValidators = 3; - for (int j = 0; j < numValidators; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("validator-replaced-{}", j); - entry.mHomeDomain = orgName; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - - return {sks, validators}; -} - -// Add `orgsToAdd` new orgs to the topology `t`. Each org will have 3 -// validators. -Topology -addOrgs(int orgsToAdd, Topology const& t) -{ - Topology t2(t); // Copy the topology - auto& [sks, validators] = t2; - REQUIRE(sks.size() == validators.size()); - - // Generate new orgs - for (int i = 0; i < orgsToAdd; ++i) - { - std::string const org = fmt::format("new-org-{}", i); - int constexpr numValidators = 3; - for (int j = 0; j < numValidators; ++j) - { - SecretKey const& key = - sks.emplace_back(SecretKey::pseudoRandomForTesting()); - ValidatorEntry& entry = validators.emplace_back(); - entry.mName = fmt::format("new-validator-{}-{}", i, j); - entry.mHomeDomain = org; - entry.mQuality = ValidatorQuality::VALIDATOR_HIGH_QUALITY; - entry.mKey = key.getPublicKey(); - entry.mHasHistory = false; - } - } - return t2; -} - -// Returns `true` if the set intersection of `leaders1` and `leaders2` is not -// empty. -bool -leadersIntersect(std::set const& leaders1, - std::set const& leaders2) -{ - std::vector intersection; - std::set_intersection(leaders1.begin(), leaders1.end(), leaders2.begin(), - leaders2.end(), std::back_inserter(intersection)); - return !intersection.empty(); -} - -// Given two quorum sets consisting of validators in `validators1` and -// `validators2`, this function returns the probability that the two quorum sets -// will agree on a leader in the first round of nomination. -double -computeExpectedFirstRoundAgreementProbability( - std::vector const& validators1, - std::vector const& validators2) -{ - // Gather orgs - std::set orgs1; - std::transform(validators1.begin(), validators1.end(), - std::inserter(orgs1, orgs1.end()), - [](ValidatorEntry const& v) { return v.mHomeDomain; }); - std::set orgs2; - std::transform(validators2.begin(), validators2.end(), - std::inserter(orgs2, orgs2.end()), - [](ValidatorEntry const& v) { return v.mHomeDomain; }); - - // Compute overlap - std::vector sharedOrgs; - std::set_intersection(orgs1.begin(), orgs1.end(), orgs2.begin(), - orgs2.end(), std::back_inserter(sharedOrgs)); - - // Probability of agreement in first round is (orgs overlapping / orgs1) * - // (orgs overlapping / orgs2). That's the probability that the two sides - // will pick any overlapping org. The algorithm guarantees that if they pick - // overlapping validator, they'll pick the same validator. - double overlap = static_cast(sharedOrgs.size()); - return overlap / orgs1.size() * overlap / orgs2.size(); -} - -// Test that the nomination algorithm behaves as expected when the two quorum -// sets `qs1` and `qs2` are not equivalent. This function requires that both -// quorum sets overlap, and contain only a single quality level of validators. -// Runs simulation for `numLedgers` slots. -// NOTE: This test counts any failure to agree on a leader as a timeout. In -// practice, it's possible that one side of the split is large enough to proceed -// without the other side. In this case, the larger side might not experience a -// timeout and "drag" the other side through consensus with it. However, this -// test aims to analyze the worst case scenario where the two sides are fairly -// balanced and real-world networking conditions are in place (some nodes -// lagging, etc), such that disagreement always results in a timeout. -void -testAsymmetricTimeouts(Topology const& qs1, Topology const& qs2, - int const numLedgers) -{ - auto const& [sks1, validators1] = qs1; - auto const& [sks2, validators2] = qs2; - - REQUIRE(sks1.size() == validators1.size()); - REQUIRE(sks2.size() == validators2.size()); - - // Generate configs and nodes representing one validator with each quorum - // set - std::vector clocks(2); - std::vector apps; - for (int i = 0; i < 2; ++i) - { - Config cfg = getTestConfig(i); - cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.generateQuorumSetForTesting(i == 0 ? validators1 : validators2); - cfg.NODE_SEED = i == 0 ? sks1.back() : sks2.back(); - - auto app = apps.emplace_back(createTestApplication(clocks.at(i), cfg)); - } - - // Run the nomination algorithm for `numLedgers` slots. Simulate timeouts by - // re-running slots that don't agree on a leader until their leader - // elections overlap. Record the number of timeouts it takes for the two - // quorum sets to agree on a leader in `timeouts`, which is effectively a - // mapping from number of timeouts to the number of ledgers that experienced - // that many timeouts. - std::vector timeouts(std::max(validators1.size(), validators2.size())); - for (int i = 0; i < numLedgers; ++i) - { - Value const v = getRandomValue(); - SCP& scp1 = dynamic_cast(apps.at(0)->getHerder()).getSCP(); - SCP& scp2 = dynamic_cast(apps.at(1)->getHerder()).getSCP(); - auto s1 = std::make_shared(i, scp1); - auto s2 = std::make_shared(i, scp2); - - TestNominationProtocol np1(*s1); - TestNominationProtocol np2(*s2); - - for (int j = 0; j < timeouts.size(); ++j) - { - std::set const& leaders1 = - np1.updateRoundLeadersForTesting(v); - std::set const& leaders2 = - np2.updateRoundLeadersForTesting(v); - REQUIRE(leaders1.size() == j + 1); - REQUIRE(leaders2.size() == j + 1); - - if (leadersIntersect(leaders1, leaders2)) - { - // Agreed on a leader! Record the number of timeouts resulted. - ++timeouts.at(j); - break; - } - } - - // If leaders don't intersect after running through the loop then the - // two quorum sets have no overlap and the test is broken. - REQUIRE(leadersIntersect(np1.getLeaders(), np2.getLeaders())); - } - - // For the first round, we can easily compute the expected agreement - // probability. For subsequent rounds, we check only that the success rate - // increases over time (modulo some small epsilon). - double expectedSuccessRate = - computeExpectedFirstRoundAgreementProbability(validators1, validators2); - - // Allow for some small decrease in success rate from the theoretical value. - // We're working with probabilistic simulation here so we can't be too - // strict or the test will be flaky. - double constexpr epsilon = 0.1; - - // There's not enough data in the tail of the distribution to allow us to - // assert that the success rate is what's expected. To avoid sporadic test - // failures, we cut off `tailCutoffPoint` of the tail of the distribution - // for the purposes of asserting test values. However, the test will still - // log those success rates for manual examination. - double constexpr tailCutoffPoint = 0.05; - - int numLedgersRemaining = numLedgers; - for (int i = 0; i < timeouts.size(); ++i) - { - int const numTimeouts = timeouts.at(i); - if (numTimeouts == 0) - { - // Avoid cluttering output - continue; - } - - CLOG_INFO(Herder, "Ledgers with {} timeouts: {} ({}%)", i, numTimeouts, - static_cast(numTimeouts) * 100 / numLedgers); - - if (numLedgersRemaining > numLedgers * tailCutoffPoint) - { - // Check that success rate increases over time. Allow some epsilon - // decrease because this is a probabilistic simulation. Also stop - // checking when we're at the last `tailCutoffPoint` timeouts as the - // data is too sparse to be useful. - double successRate = - static_cast(timeouts.at(i)) / numLedgersRemaining; - REQUIRE(successRate > expectedSuccessRate - epsilon); - - // Take max of success rate and previous success rate to avoid - // accidentally accepting a declining success rate due to episilon. - expectedSuccessRate = std::max(successRate, expectedSuccessRate); - numLedgersRemaining -= numTimeouts; - } - } -} -} // namespace - -// Test timeouts with asymmetric quorums. This test serves two purposes: -// 1. It contains assertions checking for moderate (10%) deviations from the -// expected behavior of the nomination algorithm. These should detect any -// major issues/regressions with the algorithm. -// 2. It logs the distributions of timeouts for manual inspection. This is -// useful for understanding the behavior of the algorithm and for testing -// specific scenarios one might be interested in (e.g., if tier 1 disagrees -// on one org's presence in tier 1, what is the impact on nomination -// timeouts?). -// NOTE: This provides a worst-case analysis of timeouts. See the NOTE on -// `testAsymmetricTimeouts` for more details. -TEST_CASE("Asymmetric quorum timeouts", "[herder]") -{ - // Number of slots to run for - int constexpr numLedgers = 20000; - - SECTION("Tier 1-like topology with replaced org") - { - auto t = tier1Like(); - testAsymmetricTimeouts(t, replaceOneOrg(t), numLedgers); - } - - SECTION("Tier 1-like topology with 1 added org") - { - auto t = tier1Like(); - testAsymmetricTimeouts(t, addOrgs(1, t), numLedgers); - } - - SECTION("Tier 1-like topology with 3 added orgs") - { - auto t = tier1Like(); - testAsymmetricTimeouts(t, addOrgs(3, t), numLedgers); - } -} - -// Test that the nomination algorithm behaves as expected when a random -// `numUnresponsive` set of nodes in `qs` are unresponsive. Runs simulation for -// `numLedgers` slots. -static void -testUnresponsiveTimeouts(Topology const& qs, int numUnresponsive, - int const numLedgers) -{ - auto const& [sks, validators] = qs; - REQUIRE(sks.size() == validators.size()); - REQUIRE(numUnresponsive < validators.size()); - - // extract and shuffle node ids. Choose `numUnresponsive` nodes to be the - // unresponsive nodes. - std::vector nodeIDs; - std::transform(validators.begin(), validators.end(), - std::back_inserter(nodeIDs), - [](ValidatorEntry const& v) { return v.mKey; }); - stellar::shuffle(nodeIDs.begin(), nodeIDs.end(), getGlobalRandomEngine()); - std::set unresponsive(nodeIDs.begin(), - nodeIDs.begin() + numUnresponsive); - - // Collect info about orgs - ValidatorQuality maxQuality; - std::unordered_map orgQualities; - std::unordered_map orgSizes; - std::unordered_map orgQualityCounts; - collectOrgInfo(maxQuality, orgQualities, orgSizes, orgQualityCounts, - validators); - - // Compute total weight of all validators, as well as the total weight of - // unresponsive validators - double totalWeight = 0.0; - double unresponsiveWeight = 0.0; - for (ValidatorEntry const& validator : validators) - { - double normalizedWeight = - expectedNormalizedWeight(orgQualityCounts, maxQuality, - orgQualities.at(validator.mHomeDomain), - orgSizes.at(validator.mHomeDomain)); - totalWeight += normalizedWeight; - if (unresponsive.count(validator.mKey)) - { - unresponsiveWeight += normalizedWeight; - } - } - - // Compute the average weight of an unresponsive node - double avgUnresponsiveWeight = unresponsiveWeight / numUnresponsive; - - // Compute expected number of ledgers experiencing `n` timeouts where `n` is - // the index of the `timeouts` vector. This vector is a mapping from number - // of timeouts to expected number of ledgers experiencing that number of - // timeouts. - std::vector expectedTimeouts(numUnresponsive + 1); - double remainingWeight = totalWeight; - int remainingUnresponsive = numUnresponsive; - int remainingLedgers = numLedgers; - for (int i = 0; i < expectedTimeouts.size(); ++i) - { - double timeoutProb = - (avgUnresponsiveWeight * remainingUnresponsive) / remainingWeight; - // To get expected number of ledgers experiencing `i` timeouts, we take - // the probability a timeout does not occur and multiply it by the - // number of remaining ledgers. - int expectedLedgers = (1 - timeoutProb) * remainingLedgers; - expectedTimeouts.at(i) = expectedLedgers; - - // Remaining ledgers decreases by expected number of ledgers - // experiencing `i` timeouts - remainingLedgers -= expectedLedgers; - - // For `i+1` timeouts to occur, an unresponsive node must be chosen. - // Therefore, deduct the average weight of an unresponsive node from the - // total weight left in the network. - remainingWeight -= avgUnresponsiveWeight; - --remainingUnresponsive; - } - - // Generate a config - Config cfg = getTestConfig(); - cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; - cfg.generateQuorumSetForTesting(validators); - cfg.NODE_SEED = sks.front(); - - // Create an application - VirtualClock clock; - Application::pointer app = createTestApplication(clock, cfg); - - // Run for `numLedgers` slots, recording the number of times each slot timed - // out due to unresponsive nodes before successfully electing a responsive - // leader. - SCP& scp = dynamic_cast(app->getHerder()).getSCP(); - std::vector timeouts(numUnresponsive + 1); - for (int i = 0; i < numLedgers; ++i) - { - Value const v = getRandomValue(); - auto s = std::make_shared(i, scp); - - TestNominationProtocol np(*s); - for (int i = 0; i < timeouts.size(); ++i) - { - std::set const& leaders = - np.updateRoundLeadersForTesting(v); - // If leaders is a subset of unresponsive, then a timeout occurs. - if (!std::includes(unresponsive.begin(), unresponsive.end(), - leaders.begin(), leaders.end())) - { - ++timeouts.at(i); - break; - } - } - } - - // Allow for some small multiplicative increase in timeouts from the - // theoretical value. We're working with probabilistic simulation here so - // we can't be too strict or the test will be flaky. - double constexpr epsilon = 1.1; - - // There's not enough data in the tail of the distribution to allow us to - // assert that the timeout values are what's expected. To avoid sporadic - // test failures, we cut off `tailCutoffPoint` of the tail of the - // distribution for the purposes of asserting test values. However, the test - // will still log those values for manual examination. - double constexpr tailCutoffPoint = 0.05; - - // Analyze timeouts - int numLedgersRemaining = numLedgers; - for (int i = 0; i < timeouts.size(); ++i) - { - int const numTimeouts = timeouts.at(i); - int const expectedNumTimeouts = expectedTimeouts.at(i); - - if (numLedgersRemaining > numLedgers * tailCutoffPoint) - { - // Check that timeouts are less than epsilon times the expected - // value. Also stop checking when we're at the last - // `tailCutoffPoint` timeouts as the data is too sparse to be - // useful. - REQUIRE(numTimeouts < expectedNumTimeouts * epsilon); - } - CLOG_INFO(Herder, "Ledgers with {} timeouts: {} ({}%)", i, numTimeouts, - numTimeouts * 100.0 / numLedgers); - numLedgersRemaining -= numTimeouts; - } -} - -// Test timeouts for a tier 1-like topology with 1-5 unresponsive nodes. This -// test serves two purposes: -// 1. It contains assertions checking for moderate (10%) deviations from the -// expected behavior of the nomination algorithm. These should detect any -// major issues/regressions with the algorithm. -// 2. It logs the distributions of timeouts for manual inspection. This is -// useful for understanding the behavior of the algorithm and for testing -// specific scenarios one might be interested in (e.g., if 3 tier 1 nodes -// are heavily lagging, what is the impact on nomination timeouts?). -TEST_CASE("Unresponsive quorum timeouts", "[herder]") -{ - // Number of slots to run for - int constexpr numLedgers = 20000; - - auto t = tier1Like(); - for (int i = 1; i <= 5; ++i) - { - CLOG_INFO(Herder, "Simulating nomination with {} unresponsive nodes", - i); - testUnresponsiveTimeouts(t, i, numLedgers); - } -} - -TEST_CASE("nomination timeouts with partial upgrade arming", - "[herder][acceptance]") -{ - // Configure simulation to use automatic quorum set configuration so that it - // runs with the application-specific leader election algorithm, which does - // not introduce its own timeouts. - Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); - Simulation::pointer simulation = - Topologies::separateAllHighQuality(16, networkID, [&](int i) { - return getTestConfig(i, Config::TESTDB_DEFAULT); - }); - simulation->fullyConnectAllPending(); - simulation->startAllNodes(); - auto nodes = simulation->getNodes(); - REQUIRE(nodes.size() == 16); - - // Let the network run for a few ledgers normally first - auto const expectedLedgerCloseTime = - simulation->getExpectedLedgerCloseTime(); - simulation->crankUntil( - [&]() { return simulation->haveAllExternalized(5, 1); }, - 10 * expectedLedgerCloseTime, false); - - // Create an upgrade to arm on a subset of nodes. - Upgrades::UpgradeParameters scheduledUpgrades; - auto lclCloseTime = - VirtualClock::from_time_t(nodes[0] - ->getLedgerManager() - .getLastClosedLedgerHeader() - .header.scpValue.closeTime); - - // Set upgrade time to now so it's active immediately - scheduledUpgrades.mUpgradeTime = lclCloseTime; - - // Upgrade the base fee by a small amount - auto const currentFee = nodes[0]->getLedgerManager().getLastTxFee(); - scheduledUpgrades.mBaseFee = currentFee + 100; - - // Limit max timeouts per slot to 1 - constexpr uint32_t maxTimeouts = 1; - scheduledUpgrades.mNominationTimeoutLimit = maxTimeouts; - - // Reduce upgrade window to 4 minutes - constexpr std::chrono::minutes upgradeWindow(4); - scheduledUpgrades.mExpirationMinutes = upgradeWindow; - - // Number of ledgers to check timeouts during - constexpr int ledgersToRun = 20; - - // Maximum total timeout duration for the test. Worst case is that each slot - // experiences 1 timeout, which adds 1 second each. - constexpr auto maxTotalTimeoutDuration = std::chrono::seconds(ledgersToRun); - - // Ensure upgrade window is set properly so that the upgrade doesn't expire - // during the `ledgersToRun` time period - REQUIRE(upgradeWindow > - expectedLedgerCloseTime * ledgersToRun + maxTotalTimeoutDuration); - - // Arm upgrades on 10 nodes (just 1 shy of a quorum) - for (size_t i = 0; i < 10; ++i) - { - nodes[i]->getHerder().setUpgrades(scheduledUpgrades); - } - - // Track initial ledger number - auto const startLedger = - nodes[0]->getLedgerManager().getLastClosedLedgerNum(); - - // Run for `ledgersToRun` more ledgers with mixed upgrade state - auto& herder = dynamic_cast(nodes[0]->getHerder()); - HerderSCPDriver const& driver = herder.getHerderSCPDriver(); - for (int i = 1; i <= ledgersToRun; ++i) - { - uint32_t const ledger = startLedger + i; - simulation->crankUntil( - [&]() { return simulation->haveAllExternalized(ledger, 1); }, - expectedLedgerCloseTime * 2, false); - - // Should see at most `maxTimeouts` per slot, depending on the round - // leaders. - std::optional timeouts = driver.getNominationTimeouts(ledger); - REQUIRE(timeouts.has_value()); - REQUIRE(timeouts.value() <= maxTimeouts); - } - - // Helper to check whether upgrade is still active by comparing with - // `scheduledUpgrades` - std::string const upgradeJson = scheduledUpgrades.toJson(); - auto const upgradeIsActive = [&]() { - return herder.getUpgrades().getParameters().toJson() == upgradeJson; - }; - - // Verify upgrade is still active - REQUIRE(upgradeIsActive()); - - // Verify that upgrade expires properly after the window - simulation->crankUntil(std::not_fn(upgradeIsActive), upgradeWindow, false); - - // Ensure the changed fields are all reset - auto const& upgradeParams = herder.getUpgrades().getParameters(); - REQUIRE(!upgradeParams.mBaseFee.has_value()); - REQUIRE(!upgradeParams.mNominationTimeoutLimit.has_value()); - REQUIRE(!upgradeParams.mExpirationMinutes.has_value()); - - // Verify the upgrade did not go through - REQUIRE(nodes[0]->getLedgerManager().getLastTxFee() == currentFee); -} - TEST_CASE("late joining node reaches consensus", "[herder]") { auto networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); @@ -6100,7 +4843,7 @@ TEST_CASE_VERSIONS("Herder properly validates when tx set is missing", }); } -// One of four validators nominates a non-canonical empty tx set (a tx set +// One of four validators proposes a non-canonical empty tx set (a tx set // without transactions that isn't encoded as `TxSetXDRFrame::makeEmpty`, // which history can't replay) on every ledger. Before the protocol gate the // network externalizes it, after the gate it's rejected as invalid. @@ -6132,8 +4875,7 @@ TEST_CASE("network does not externalize non-canonical empty tx set", auto simulation = Topologies::core( 4, 0.75, sha256(getTestConfig().NETWORK_PASSPHRASE), makeConfig); auto nodes = simulation->getNodes(); - static_cast(nodes[0]->getHerder()) - .mNominatedTxSetOverride = + static_cast(nodes[0]->getHerder()).mProposedTxSetOverride = [&](LedgerHeaderHistoryEntry const& lcl) { return makeNonCanonicalEmptyTxSet(lcl.hash); }; @@ -6169,8 +4911,11 @@ TEST_CASE("network does not externalize non-canonical empty tx set", } else { + // Honest nodes do not vote for the bad proposal. There is no + // nomination to carry it into their ballots (where it used to be + // replaced with an empty-tx-set value): the round times out and + // the next leader's value closes the ledger. REQUIRE(externalized == 0); - REQUIRE(replaced > 0); } }; @@ -6197,7 +4942,7 @@ TEST_CASE("network externalizes empty-tx-set on missing value", "[herder][tx]") auto cfg = getTestConfig(i, Config::TESTDB_DEFAULT); cfg.EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = true; cfg.TX_SET_DOWNLOAD_TIMEOUT = std::chrono::milliseconds{0}; - cfg.TESTING_NOMINATE_RANDOM_VALUES = true; + cfg.TESTING_PROPOSE_RANDOM_VALUES = true; return cfg; }); @@ -6301,7 +5046,7 @@ TEST_CASE("SCP state restore with missing tx set", "[herder]") // The restored value's tx set is missing and nothing is fetching it, but // the value is still structurally valid - REQUIRE(driver.validateValue(slot, value, /*nomination*/ false) == + REQUIRE(driver.validateValue(slot, value, /*proposal*/ false) == SCPDriver::kStructurallyValidValue); // The peer's view of the slot: it timed out waiting for the missing tx @@ -6372,11 +5117,10 @@ triggerTimerProtocolSupported() } // Four top-tier validators over TCP on the real clock, with a 1s artificial -// apply delay and a configurable nomination-emit delay so we can actually see -// the impact of the two different timers. +// apply delay. static Simulation::pointer makeTriggerTimerSimulation( - bool forcePrepareStartTimer, std::chrono::milliseconds nominationEmitDelay, + bool forcePrepareStartTimer, std::chrono::milliseconds driftClockOffset = std::chrono::milliseconds::zero(), std::optional startingProtocol = std::nullopt) @@ -6391,8 +5135,6 @@ makeTriggerTimerSimulation( forcePrepareStartTimer; cfg.ARTIFICIALLY_DELAY_LEDGER_CLOSE_FOR_TESTING = std::chrono::milliseconds(1000); - cfg.ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING = - nominationEmitDelay; // Remember enough SCP slots that the tests can attribute every // externalized value in their measurement windows to its // proposer. @@ -6469,7 +5211,6 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") int64_t driftedNodeFallbacks{0}; int64_t maxOtherNodeFallbacks{0}; int64_t driftedLedSlots{0}; - bool sawNominationTimeout{false}; }; auto fallbackCount = [](Application::pointer const& app) { @@ -6487,11 +5228,10 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") auto runSimulation = [&](bool forcePrepareStartTimer, - std::chrono::milliseconds nominationEmitDelay, std::chrono::milliseconds triggerClockOffset = std::chrono::milliseconds::zero()) -> RunResult { - auto simulation = makeTriggerTimerSimulation( - forcePrepareStartTimer, nominationEmitDelay, triggerClockOffset); + auto simulation = makeTriggerTimerSimulation(forcePrepareStartTimer, + triggerClockOffset); auto nodes = simulation->getNodes(); std::vector fallbackCounts; @@ -6524,18 +5264,6 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") result.maxOtherNodeFallbacks = std::max(result.maxOtherNodeFallbacks, delta); } - - auto const& driver = - dynamic_cast(nodes[i]->getHerder()) - .getHerderSCPDriver(); - for (uint32_t ledger = startLedger + 1; ledger <= targetLedger; - ++ledger) - { - auto timeouts = driver.getNominationTimeouts(ledger); - result.sawNominationTimeout = - result.sawNominationTimeout || - (timeouts.has_value() && timeouts.value() > 0); - } } // Count the externalized values the drifted @@ -6580,13 +5308,10 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") return result; }; - // New timer is faster without drift. + // Without drift the new timer never falls back. (There is no nomination + // phase whose duration would separate the two anchors any more.) { - auto const nominationDelay = std::chrono::milliseconds(1000); - auto const oldTimer = runSimulation(true, nominationDelay); - auto const newTimer = runSimulation(false, nominationDelay); - - REQUIRE(newTimer.elapsed < oldTimer.elapsed); + auto const newTimer = runSimulation(false); REQUIRE(newTimer.totalFallbacks == 0); } @@ -6594,8 +5319,7 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") // One node drifting ahead falls back. { - auto const nodeAhead = runSimulation( - false, std::chrono::milliseconds::zero(), driftOffset); + auto const nodeAhead = runSimulation(false, driftOffset); REQUIRE(nodeAhead.driftedNodeFallbacks >= LEDGERS_TO_RUN - nodeAhead.driftedLedSlots - FALLBACK_SLACK); @@ -6607,8 +5331,7 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") // One node drifting behind falls back. { - auto const nodeBehind = runSimulation( - false, std::chrono::milliseconds::zero(), -driftOffset); + auto const nodeBehind = runSimulation(false, -driftOffset); REQUIRE(nodeBehind.driftedNodeFallbacks >= LEDGERS_TO_RUN - nodeBehind.driftedLedSlots - FALLBACK_SLACK); @@ -6617,122 +5340,11 @@ TEST_CASE("consensus close time trigger timer", "[herder][!hide]") REQUIRE(nodeBehind.maxOtherNodeFallbacks <= nodeBehind.driftedLedSlots + FALLBACK_SLACK); } - - // Long nomination does not cause timer fallback - { - auto const nominationDelay = std::chrono::milliseconds(5000); - auto const slowNomination = runSimulation(false, nominationDelay); - REQUIRE(slowNomination.sawNominationTimeout); - REQUIRE(slowNomination.totalFallbacks == 0); - } -} - -TEST_CASE("trigger timer switches anchor at protocol 28 upgrade", - "[herder][upgrades][!hide]") -{ - if (!triggerTimerProtocolSupported()) - { - return; - } - - auto const upgradeVersion = - static_cast(CONSENSUS_CLOSE_TIME_TRIGGER_PROTOCOL_VERSION); - - // With a delayed nomination emit, the prepare-start timer paces ledgers - // at roughly expectedClose + nominationEmitDelay (nomination happens - // before the anchor point), while the consensus-close-time timer absorbs - // the nomination delay and paces at expectedClose. This delta helps us - // measure the timer change after the upgrade. - constexpr uint32_t LEDGERS_TO_MEASURE = 8; - auto const nominationEmitDelay = std::chrono::milliseconds(1000); - - struct RunResult - { - std::chrono::milliseconds preUpgrade; - std::chrono::milliseconds postUpgrade; - }; - - auto runSimulation = [&](bool forcePrepareStartTimer) -> RunResult { - // Start the network one protocol before the trigger-timer switch. - auto simulation = makeTriggerTimerSimulation( - forcePrepareStartTimer, nominationEmitDelay, - std::chrono::milliseconds::zero(), upgradeVersion - 1); - auto nodes = simulation->getNodes(); - auto const expectedClose = simulation->getExpectedLedgerCloseTime(); - - auto lclVersion = [](Application::pointer const& node) { - return node->getLedgerManager() - .getLastClosedLedgerHeader() - .header.ledgerVersion; - }; - - // Measure the cadence on the pre-28 protocol. - simulation->crankUntil( - [&]() { return simulation->haveAllExternalized(3, 1); }, - 10 * expectedClose, false); - auto const preUpgrade = closeLedgers(simulation, LEDGERS_TO_MEASURE); - for (auto const& node : nodes) - { - REQUIRE(lclVersion(node) == upgradeVersion - 1); - } - - // Upgrade to protocol 28 - Upgrades::UpgradeParameters scheduledUpgrades; - scheduledUpgrades.mUpgradeTime = - VirtualClock::from_time_t(nodes[0] - ->getLedgerManager() - .getLastClosedLedgerHeader() - .header.scpValue.closeTime); - scheduledUpgrades.mProtocolVersion = upgradeVersion; - for (auto const& node : nodes) - { - node->getHerder().setUpgrades(scheduledUpgrades); - } - - // Crank until every node has closed the upgrade ledger, then one - // more ledger so the measured window is fully post-upgrade. - simulation->crankUntil( - [&]() { - return std::all_of(nodes.begin(), nodes.end(), - [&](Application::pointer const& node) { - return lclVersion(node) == - upgradeVersion; - }); - }, - 10 * expectedClose, false); - closeLedgers(simulation, 1); - - auto const postUpgrade = closeLedgers(simulation, LEDGERS_TO_MEASURE); - for (auto const& node : nodes) - { - REQUIRE(lclVersion(node) == upgradeVersion); - } - - return {preUpgrade, postUpgrade}; - }; - - // Expected cadence saving is nominationEmitDelay per ledger; splitting - // pass/fail at half of it tolerates scheduling noise in both directions. - auto const cadenceMargin = LEDGERS_TO_MEASURE * nominationEmitDelay / 2; - - // Crossing the boundary switches to the consensus-close-time anchor: - // post-upgrade ledgers close significantly faster. - { - auto const result = runSimulation(false); - REQUIRE(result.postUpgrade + cadenceMargin < result.preUpgrade); - } - - // The override flag keeps the prepare-start anchor after the upgrade, so - // the cadence does not improve. - { - auto const result = runSimulation(true); - REQUIRE(result.postUpgrade + cadenceMargin > result.preUpgrade); - } } namespace stellar { -class EarlyNominationTestAccess +class LeaderBuildTestAccess { public: static VirtualClock::time_point @@ -6843,13 +5455,12 @@ class EarlyNominationTestAccess SecretKey::fromSeed(sha256(fmt::format("absent-peer-{}", i))) .getPublicKey()}; scp.updateLocalQuorumSet(qset); - if (scp.predictNominationLeaders( - lcl.header.ledgerSeq + 1, - xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(scp.getLocalNodeID())) + herder.mPendingEnvelopes.addSCPQuorumSet( + scp.getLocalNode()->getQuorumSetHash(), qset); + herder.mPendingEnvelopes.forceRebuildQuorum(); + if (herder.getHerderSCPDriver().isLocalNodeRoundLeader( + lcl.header.ledgerSeq + 1, 1)) { - herder.mPendingEnvelopes.addSCPQuorumSet( - scp.getLocalNode()->getQuorumSetHash(), qset); return; } } @@ -6858,7 +5469,7 @@ class EarlyNominationTestAccess }; } -TEST_CASE("trigger fallback credits elapsed time through nomination", +TEST_CASE("trigger fallback credits elapsed time up to the first ballot", "[herder][trigger-construction]") { using namespace std::chrono; @@ -6872,23 +5483,23 @@ TEST_CASE("trigger fallback credits elapsed time through nomination", int expectedWaitMs; int expectedFallbacks; }; - auto scenario = - GENERATE(Scenario{350, 1900, 2100, 0, true, 0, 0}, - Scenario{350, 1900, 2100, 4000, true, 1800, 1}, - Scenario{350, 1900, 2100, -4000, true, 1800, 1}, - Scenario{350, 1900, 4100, 0, true, 0, 1}, - Scenario{350, 1900, 2100, 0, false, 1800, 1}, - Scenario{350, 1500, 1700, 0, true, 300, 0}, - // A slow first round need not execute a timeout callback - // before entering ballot. Its elapsed time is still real. - Scenario{100, 2000, 2250, 0, true, 0, 0}, - // Followers need no construction, but still wait on peers. - Scenario{0, 2250, 2450, 0, true, 0, 0}, - // Delayed timer callbacks must not hide elapsed time. - Scenario{100, 6500, 6700, 0, true, 0, 0}, - // Long nomination does not excuse arbitrary clock drift. - Scenario{100, 6500, 6700, 10000, true, 1800, 1}, - Scenario{100, 6500, 6700, -10000, true, 1800, 1}); + auto scenario = GENERATE( + Scenario{350, 1900, 2100, 0, true, 0, 0}, + Scenario{350, 1900, 2100, 4000, true, 1800, 1}, + Scenario{350, 1900, 2100, -4000, true, 1800, 1}, + Scenario{350, 1900, 4100, 0, true, 0, 1}, + Scenario{350, 1900, 2100, 0, false, 1800, 1}, + Scenario{350, 1500, 1700, 0, true, 300, 0}, + // A slow first round need not execute a timeout callback + // before entering ballot. Its elapsed time is still real. + Scenario{100, 2000, 2250, 0, true, 0, 0}, + // Followers need no construction, but still wait on peers. + Scenario{0, 2250, 2450, 0, true, 0, 0}, + // Delayed timer callbacks must not hide elapsed time. + Scenario{100, 6500, 6700, 0, true, 0, 0}, + // A long wait for a proposal does not excuse arbitrary clock drift. + Scenario{100, 6500, 6700, 10000, true, 1800, 1}, + Scenario{100, 6500, 6700, -10000, true, 1800, 1}); CAPTURE(scenario.buildMs, scenario.closeMs, scenario.clockOffsetMs, scenario.recordTrigger); VirtualClock clock; @@ -6907,15 +5518,14 @@ TEST_CASE("trigger fallback credits elapsed time through nomination", herder.setTrackingSCPState(slot, value, true); if (scenario.recordTrigger) { - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); } clock.setCurrentVirtualTime(origin + milliseconds(scenario.buildMs)); - driver.recordSCPEvent(slot, true); clock.setCurrentVirtualTime(origin + milliseconds(scenario.ballotMs)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); clock.setCurrentVirtualTime(origin + milliseconds(scenario.closeMs)); clock.setSystemTimeOffset(milliseconds(scenario.clockOffsetMs)); - REQUIRE(driver.getNominationTimeouts(slot).value() == 0); + REQUIRE(driver.getLeaderTimeouts(slot).value() == 0); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(scenario.recordTrigger ? scenario.ballotMs : 0)); REQUIRE(driver.getTriggerToBallotDuration(slot + 1) == @@ -6926,7 +5536,7 @@ TEST_CASE("trigger fallback credits elapsed time through nomination", auto const before = fallback.count(); auto deadline = std::max( clock.now(), - EarlyNominationTestAccess::triggerAnchor(herder, slot) + seconds(2)); + LeaderBuildTestAccess::triggerAnchor(herder, slot) + seconds(2)); REQUIRE(duration_cast(deadline - clock.now()).count() == scenario.expectedWaitMs); REQUIRE(fallback.count() - before == scenario.expectedFallbacks); @@ -6946,7 +5556,7 @@ TEST_CASE("trigger work is recorded by the proposal path", SecretKey::pseudoRandomForTesting().getPublicKey()}; auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto& driver = herder.getHerderSCPDriver(); auto const slot = app->getLedgerManager().getLastClosedLedgerNum() + 1; herder.mGetTopTransactionsForTesting = [&](size_t) { @@ -6957,19 +5567,16 @@ TEST_CASE("trigger work is recorded by the proposal path", return std::vector{}; }; herder.triggerNextLedger(slot, true); - // Until ballot starts there is no completed interval to credit. - REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds::zero()); - clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordSCPEvent(slot, false); - REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(500)); + // The round leader proposes straight into the ballot protocol: the + // interval ends when its own build does. + REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(400)); clock.setCurrentVirtualTime(clock.now() + seconds(1)); - driver.recordNominationTrigger(slot); - driver.recordSCPEvent(slot, true); - REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(500)); + driver.recordTrigger(slot); + REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(400)); herder.mGetTopTransactionsForTesting = nullptr; } -TEST_CASE("skipped nomination does not start the trigger interval", +TEST_CASE("skipped trigger does not start the trigger interval", "[herder][trigger-construction]") { using namespace std::chrono; @@ -6977,7 +5584,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", auto cfg = getTestConfig(); cfg.HTTP_PORT = 0; cfg.MANUAL_CLOSE = true; - // A second validator keeps our own nomination from externalizing. + // A second validator keeps our own proposal from externalizing. cfg.QUORUM_SET.threshold = 2; cfg.QUORUM_SET.validators = { cfg.NODE_SEED.getPublicKey(), @@ -6996,7 +5603,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", clock.setSystemTimeOffset(-(Herder::MAX_TIME_SLIP_SECONDS + seconds(10))); // Counts proposal constructions, which happen only for triggers that go - // on to nominate. + // on to start the slot. int builds = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { ++builds; @@ -7009,7 +5616,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", // Peers move the slot into ballot while we wait. None of that waiting is // trigger-to-ballot work. clock.setCurrentVirtualTime(clock.now() + milliseconds(700)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds::zero()); // With the clock back in sync, the same path builds a proposal and its @@ -7020,7 +5627,7 @@ TEST_CASE("skipped nomination does not start the trigger interval", herder.triggerNextLedger(nextSlot, true); REQUIRE(builds == 1); clock.setCurrentVirtualTime(clock.now() + milliseconds(300)); - driver.recordSCPEvent(nextSlot, false); + driver.recordBallotStart(nextSlot); REQUIRE(driver.getTriggerToBallotDuration(nextSlot) == milliseconds(300)); herder.mGetTopTransactionsForTesting = nullptr; } @@ -7037,16 +5644,14 @@ TEST_CASE("trigger work excludes time after entering ballot", auto& driver = static_cast(app->getHerder()).getHerderSCPDriver(); uint64_t const slot = 100; - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); clock.setCurrentVirtualTime(clock.now() + milliseconds(300)); - driver.recordSCPEvent(slot, true); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(100)); - driver.recordSCPEvent(slot + 1, false); + driver.recordBallotStart(slot + 1); clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordNominationTrigger(slot + 1); - driver.recordSCPEvent(slot + 1, true); + driver.recordTrigger(slot + 1); REQUIRE(driver.getTriggerToBallotDuration(slot + 1) == milliseconds::zero()); } @@ -7067,19 +5672,17 @@ TEST_CASE("trigger allowance is independent of timeout callback ordering", auto const ct = VirtualClock::to_time_t(clock.system_now()) + 1; auto const origin = VirtualClock::from_time_t(ct); clock.setCurrentVirtualTime(origin); - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(origin + milliseconds(100)); - driver.recordSCPEvent(slot, true); bool timerFired = false; - driver.setupTimer(slot, Slot::NOMINATION_TIMER, seconds(2), + driver.setupTimer(slot, Slot::LEADER_TIMER, seconds(2), [&] { timerFired = true; }); - // Both events become runnable while the main thread is occupied. Entering - // ballot before dispatching the expired timer suppresses its timeout count. + // Both events become runnable while the main thread is occupied. clock.setCurrentVirtualTime(origin + milliseconds(2500)); if (ballotBeforeTimeout) { - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); } for (int i = 0; i < 100 && !timerFired; ++i) { @@ -7089,18 +5692,16 @@ TEST_CASE("trigger allowance is independent of timeout callback ordering", REQUIRE(clock.system_now() == origin + milliseconds(2500)); if (!ballotBeforeTimeout) { - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); } - REQUIRE(driver.getNominationTimeouts(slot) == - (ballotBeforeTimeout ? 0 : 1)); + REQUIRE(driver.getLeaderTimeouts(slot) == 1); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(2500)); StellarValue value; value.closeTime = ct; herder.setTrackingSCPState(slot, value, true); clock.setCurrentVirtualTime(origin + milliseconds(2700)); - REQUIRE(EarlyNominationTestAccess::triggerAnchor(herder, slot) + - seconds(2) <= + REQUIRE(LeaderBuildTestAccess::triggerAnchor(herder, slot) + seconds(2) <= clock.now()); } @@ -7133,10 +5734,9 @@ TEST_CASE("trigger anchor keeps millisecond close times", // 600 ms later, so the allowance is exactly 600 ms. auto const origin = getConsensusTime(value).toSystemTime(); clock.setCurrentVirtualTime(origin); - driver.recordNominationTrigger(slot); - driver.recordSCPEvent(slot, true); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(origin + milliseconds(600)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(driver.getTriggerToBallotDuration(slot) == milliseconds(600)); auto metrics = app->getMetrics().GetAllMetrics(); @@ -7145,10 +5745,9 @@ TEST_CASE("trigger anchor keeps millisecond close times", auto const before = fallback.count(); auto waitAt = [&](milliseconds sinceClose) { clock.setCurrentVirtualTime(origin + sinceClose); - auto deadline = - std::max(clock.now(), - EarlyNominationTestAccess::triggerAnchor(herder, slot) + - seconds(2)); + auto deadline = std::max( + clock.now(), + LeaderBuildTestAccess::triggerAnchor(herder, slot) + seconds(2)); return duration_cast(deadline - clock.now()); }; @@ -7158,7 +5757,8 @@ TEST_CASE("trigger anchor keeps millisecond close times", REQUIRE(fallback.count() == before); // Elapsed time up to the target plus the allowance is still explained by - // nomination; one millisecond more is attributed to clock drift. + // the wait for a proposal; one millisecond more is attributed to clock + // drift. REQUIRE(waitAt(milliseconds(2600)) == milliseconds::zero()); REQUIRE(fallback.count() == before); REQUIRE(waitAt(milliseconds(2601)) == milliseconds::zero()); @@ -7180,27 +5780,31 @@ TEST_CASE("trigger-to-ballot time is recorded at first ballot entry", app->getMetrics().NewTimer({"scp", "timing", "trigger-to-ballot"}); REQUIRE(timer.count() == 0); - // Construction and nomination both count, up to the first ballot entry. + // Construction and the wait for the leader both count, up to the first + // ballot entry. uint64_t const slot = 100; - driver.recordNominationTrigger(slot); + driver.recordTrigger(slot); clock.setCurrentVirtualTime(clock.now() + milliseconds(400)); - driver.recordSCPEvent(slot, true); clock.setCurrentVirtualTime(clock.now() + milliseconds(100)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(timer.count() == 1); REQUIRE(timer.max() == Approx(500.0)); // Later ballot events of the same slot are not new intervals. clock.setCurrentVirtualTime(clock.now() + milliseconds(50)); - driver.recordSCPEvent(slot, false); + driver.recordBallotStart(slot); REQUIRE(timer.count() == 1); - // No interval without a local trigger, or when ballot started first. - driver.recordSCPEvent(slot + 1, false); - driver.recordSCPEvent(slot + 2, false); - driver.recordNominationTrigger(slot + 2); - driver.recordSCPEvent(slot + 2, true); + // No interval without a local trigger. + driver.recordBallotStart(slot + 1); REQUIRE(timer.count() == 1); + + // A trigger that fires after the node already voted for the round + // leader's proposal records a zero interval. + driver.recordBallotStart(slot + 2); + driver.recordTrigger(slot + 2); + REQUIRE(timer.count() == 2); + REQUIRE(timer.min() == Approx(0.0)); } TEST_CASE("trigger lateness and immediate triggers are recorded", @@ -7255,7 +5859,7 @@ TEST_CASE("trigger lateness and immediate triggers are recorded", clock.setCurrentVirtualTime(std::max(overdue, clock.now()) + milliseconds(500)); auto const immediateCount = immediate.count(); - EarlyNominationTestAccess::setupTrigger(herder); + LeaderBuildTestAccess::setupTrigger(herder); REQUIRE(immediate.count() == immediateCount + 1); REQUIRE(herder.getTriggerTimer().expiry_time() == clock.now()); testutil::crankUntil( @@ -7263,7 +5867,7 @@ TEST_CASE("trigger lateness and immediate triggers are recorded", REQUIRE(lm.getLastClosedLedgerNum() > lcl); } -TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") +TEST_CASE("prepare proposal before trigger", "[herder][early-proposal]") { auto const soroban = GENERATE(false, true); VirtualClock clock; @@ -7276,7 +5880,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") cfg.GENESIS_TEST_ACCOUNT_COUNT = 10; cfg.TESTING_UPGRADE_MAX_TX_SET_SIZE = 1; // Require another node to externalize, so this test can inspect the value - // we nominate without immediately closing it and scheduling another slot. + // we propose without immediately closing it and scheduling another slot. cfg.QUORUM_SET.threshold = 2; cfg.QUORUM_SET.validators = { cfg.NODE_SEED.getPublicKey(), @@ -7289,7 +5893,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") config.mLedgerMaxTxCount = 1; }); } - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); // The Soroban upgrade helper closes ledgers at its fixed test date. The // sub-second offset makes the prepared close time carry milliseconds once @@ -7298,7 +5902,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") VirtualClock::from_time_t(lcl.header.scpValue.closeTime + 1000) + std::chrono::milliseconds(375)); auto const seq = lcl.header.ledgerSeq + 1; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); std::vector mempool; auto alice = txtest::getGenesisAccount(*app, 0); @@ -7339,18 +5943,16 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") REQUIRE(trigger > clock.now()); testutil::crankUntil( app, - [&]() { - return EarlyNominationTestAccess::prepared(herder) != nullptr; - }, + [&]() { return LeaderBuildTestAccess::prepared(herder) != nullptr; }, std::chrono::seconds(1)); REQUIRE(clock.now() < trigger); REQUIRE(pulls == 1); - auto prepared = EarlyNominationTestAccess::prepared(herder); - auto const preparedCloseTime = EarlyNominationTestAccess::closeTime(herder); + auto prepared = LeaderBuildTestAccess::prepared(herder); + auto const preparedCloseTime = LeaderBuildTestAccess::closeTime(herder); REQUIRE(prepared); REQUIRE(prepared->sizeTxTotal() == 1); REQUIRE(preparedCloseTime == expectedConsensusTime); - REQUIRE(herder.getSCP().getNominationLeaders(seq).empty()); + REQUIRE(herder.getSCP().getLeaderRound(seq) == 0); REQUIRE(herder.getSCP().getLatestMessagesSend(seq).empty()); auto known = herder.getTxSet(prepared->getContentsHash()); REQUIRE(std::get(known) == nullptr); @@ -7361,13 +5963,10 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") // used to validate time bounds, not acquire the new wall-clock time. clock.setCurrentVirtualTime(trigger + std::chrono::seconds(1)); testutil::crankUntil( - app, - [&]() { - return !herder.getSCP().getNominationLeaders(seq).empty(); - }, + app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); REQUIRE(pulls == 1); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); REQUIRE(std::get( herder.getTxSet(prepared->getContentsHash())) == prepared); // The value can be checked independently of the production cache. @@ -7387,10 +5986,12 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") std::chrono::seconds(1)); auto messages = herder.getSCP().getLatestMessagesSend(seq); REQUIRE(!messages.empty()); - auto const& votes = messages.front().statement.pledges.nominate().votes; - REQUIRE(votes.size() == 1); + REQUIRE(messages.front().statement.pledges.type() == SCP_ST_PREPARE); + auto const& ballot = + messages.front().statement.pledges.prepare().ballot; + REQUIRE(ballot.counter == 1); StellarValue value; - xdr::xdr_from_opaque(votes.front(), value); + xdr::xdr_from_opaque(ballot.value, value); REQUIRE(value.txSetHash == prepared->getContentsHash()); REQUIRE(getConsensusTime(value) == preparedCloseTime); REQUIRE(pulls == 1); @@ -7402,7 +6003,7 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") // claim that our time-bound transaction is valid for that time. herder.triggerNextLedger(seq, true); REQUIRE(pulls == 2); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); REQUIRE(std::get( herder.getTxSet(prepared->getContentsHash())) == nullptr); } @@ -7410,32 +6011,32 @@ TEST_CASE("prepare nomination before trigger", "[herder][early-nomination]") { herder.externalizeValue(TxSetXDRFrame::makeEmpty(lcl), seq, preparedCloseTime, {}, cfg.NODE_SEED); - REQUIRE(EarlyNominationTestAccess::prepared(herder) != prepared); + REQUIRE(LeaderBuildTestAccess::prepared(herder) != prepared); auto const pullsAfterClose = pulls; - // A stale timer/caller cannot nominate the old slot or pull again. + // A stale timer/caller cannot propose for the old slot or pull again. herder.triggerNextLedger(seq, true); REQUIRE(pulls == pullsAfterClose); } SECTION("loss of sync discards preparation") { herder.lostSync(); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); herder.triggerNextLedger(seq, true); REQUIRE(pulls == 1); } SECTION("shutdown cancels preparation") { herder.shutdown(); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); } herder.mGetTopTransactionsForTesting = nullptr; } -TEST_CASE("only nomination leaders construct local proposals", - "[herder][early-nomination]") +TEST_CASE("only round leaders construct local proposals", + "[herder][early-proposal]") { - // A fixed, unanimous quorum gives all three nodes the same weights and - // leader ordering. Exercise each member as the local node, rather than + // A fixed, unanimous quorum gives all three nodes the same leader + // schedule. Exercise each member as the local node, rather than // hoping random peers cover every role for one fixed local priority. std::vector keys; SCPQuorumSet qset; @@ -7443,14 +6044,14 @@ TEST_CASE("only nomination leaders construct local proposals", for (int i = 0; i < 3; ++i) { keys.push_back(SecretKey::fromSeed( - sha256(fmt::format("early-nomination-node-{}", i)))); + sha256(fmt::format("early-proposal-node-{}", i)))); qset.validators.push_back(keys.back().getPublicKey()); } std::set checkedRoles; - std::optional previous; - std::optional> expectedFirst; - std::optional> expectedFirstTwo; + std::optional previous; + std::optional expectedFirst; + std::optional expectedSecond; for (int i = 0; i < 3; ++i) { VirtualClock clock; @@ -7467,36 +6068,38 @@ TEST_CASE("only nomination leaders construct local proposals", auto& scp = herder.getSCP(); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - auto const value = xdr::xdr_to_opaque(lcl.header.scpValue); + // The schedule is seeded by the hash of the last closed ledger. if (!previous) { - previous = value; + previous = lcl.hash; } - REQUIRE(value == *previous); - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + REQUIRE(lcl.hash == *previous); + auto& driver = herder.getHerderSCPDriver(); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); size_t pulls = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { ++pulls; return std::vector{}; }; - auto first = scp.predictNominationLeaders(seq, value, 1); - auto firstTwo = scp.predictNominationLeaders(seq, value, 2); - REQUIRE(first.size() == 1); - REQUIRE(firstTwo.size() == 2); + auto first = driver.getRoundLeader(seq, 1); + auto second = driver.getRoundLeader(seq, 2); + REQUIRE(first); + REQUIRE(second); + REQUIRE(*first != *second); if (!expectedFirst) { expectedFirst = first; - expectedFirstTwo = firstTwo; + expectedSecond = second; } - REQUIRE(first == *expectedFirst); - REQUIRE(firstTwo == *expectedFirstTwo); - int role = first.count(scp.getLocalNodeID()) ? 0 - : firstTwo.count(scp.getLocalNodeID()) ? 1 - : 2; + REQUIRE(*first == *expectedFirst); + REQUIRE(*second == *expectedSecond); + int role = *first == scp.getLocalNodeID() ? 0 + : *second == scp.getLocalNodeID() ? 1 + : 2; REQUIRE(checkedRoles.insert(role).second); CAPTURE(role); - EarlyNominationTestAccess::schedule(herder); - REQUIRE(EarlyNominationTestAccess::scheduled(herder) == (role == 0)); + LeaderBuildTestAccess::schedule(herder); + REQUIRE(LeaderBuildTestAccess::scheduled(herder) == (role == 0)); bool done = false; VirtualTimer stop(clock); stop.expires_from_now(std::chrono::milliseconds(1)); @@ -7506,42 +6109,42 @@ TEST_CASE("only nomination leaders construct local proposals", REQUIRE(done); REQUIRE(clock.now() < herder.getTriggerTimer().expiry_time()); REQUIRE(pulls == (role == 0 ? 1 : 0)); - REQUIRE(bool(EarlyNominationTestAccess::prepared(herder)) == - (role == 0)); - REQUIRE(scp.getNominationLeaders(seq).empty()); + REQUIRE(bool(LeaderBuildTestAccess::prepared(herder)) == (role == 0)); + REQUIRE(scp.getLeaderRound(seq) == 0); REQUIRE(scp.getLatestMessagesSend(seq).empty()); auto const trigger = herder.getTriggerTimer().expiry_time(); clock.setCurrentVirtualTime(trigger); auto const triggerSystemTime = clock.system_now(); testutil::crankUntil( - app, [&]() { return !scp.getNominationLeaders(seq).empty(); }, + app, [&]() { return scp.getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); - REQUIRE(scp.getNominationLeaders(seq) == first); + REQUIRE(scp.getLeaderRound(seq) == 1); // The first leader refreshes its empty early snapshot. Followers // pull the mempool once, only to pre-verify signatures; they do not // build or publish a set. REQUIRE(pulls == (role == 0 ? 2 : 1)); REQUIRE(scp.getLatestMessagesSend(seq).empty() == (role != 0)); - // With the first leader silent, real nomination timers eventually + // With the first leader silent, real leader timers eventually // promote each follower. It builds exactly then, using a current close // time, and its prepared/validated tx set is available for retrieval. testutil::crankUntil( app, [&]() { return !scp.getLatestMessagesSend(seq).empty(); }, std::chrono::seconds(30)); - REQUIRE(scp.getNominationLeaders(seq).count(scp.getLocalNodeID())); + REQUIRE(scp.getLeaderRound(seq) == static_cast(role + 1)); + REQUIRE(driver.isLocalNodeRoundLeader(seq, scp.getLeaderRound(seq))); REQUIRE(pulls == 2); auto messages = scp.getLatestMessagesSend(seq); - auto const& votes = messages.front().statement.pledges.nominate().votes; - REQUIRE(votes.size() == 1); + REQUIRE(messages.front().statement.pledges.type() == SCP_ST_PREPARE); + auto const& proposed = + messages.front().statement.pledges.prepare().ballot.value; StellarValue nominated; - xdr::xdr_from_opaque(votes.front(), nominated); + xdr::xdr_from_opaque(proposed, nominated); auto txSet = std::get( herder.getTxSet(nominated.txSetHash)); REQUIRE(txSet); REQUIRE(txSet->previousLedgerHash() == lcl.hash); - REQUIRE(herder.getHerderSCPDriver().validateValue(seq, votes.front(), - true) == + REQUIRE(driver.validateValue(seq, proposed, true) == SCPDriver::kFullyValidatedValue); if (role != 0) { @@ -7559,10 +6162,10 @@ TEST_CASE("only nomination leaders construct local proposals", } TEST_CASE("a follower promoted after losing sync still proposes", - "[herder][early-nomination]") + "[herder][early-proposal]") { // Unanimous three-node quorum with silent peers: the slot cannot - // externalize, so the local follower eventually loses sync. Nomination + // externalize, so the local follower eventually loses sync. Leader // timeouts must still let it propose a value of its own once promoted. std::vector keys; SCPQuorumSet qset; @@ -7570,7 +6173,7 @@ TEST_CASE("a follower promoted after losing sync still proposes", for (int i = 0; i < 3; ++i) { keys.push_back(SecretKey::fromSeed( - sha256(fmt::format("early-nomination-node-{}", i)))); + sha256(fmt::format("early-proposal-node-{}", i)))); qset.validators.push_back(keys.back().getPublicKey()); } @@ -7591,27 +6194,26 @@ TEST_CASE("a follower promoted after losing sync still proposes", auto& scp = herder.getSCP(); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - if (scp.predictNominationLeaders( - seq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(scp.getLocalNodeID())) + auto& driver = herder.getHerderSCPDriver(); + if (driver.isLocalNodeRoundLeader(seq, 1)) { // The first leader proposes at the trigger; test a follower. continue; } checkedFollower = true; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); size_t pulls = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { ++pulls; return std::vector{}; }; - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); clock.setCurrentVirtualTime(herder.getTriggerTimer().expiry_time()); testutil::crankUntil( - app, [&]() { return !scp.getNominationLeaders(seq).empty(); }, + app, [&]() { return scp.getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); - REQUIRE(!scp.getNominationLeaders(seq).empty()); + REQUIRE(scp.getLeaderRound(seq) != 0); REQUIRE(scp.getLatestMessagesSend(seq).empty()); // Only the signature prewarm pull; no build yet. REQUIRE(pulls == 1); @@ -7623,29 +6225,29 @@ TEST_CASE("a follower promoted after losing sync still proposes", testutil::crankUntil( app, [&]() { return !scp.getLatestMessagesSend(seq).empty(); }, std::chrono::seconds(30)); - REQUIRE(scp.getNominationLeaders(seq).count(scp.getLocalNodeID())); + REQUIRE(driver.isLocalNodeRoundLeader(seq, scp.getLeaderRound(seq))); REQUIRE(pulls == 2); auto messages = scp.getLatestMessagesSend(seq); REQUIRE(!messages.empty()); - auto const& votes = messages.front().statement.pledges.nominate().votes; - REQUIRE(votes.size() == 1); - REQUIRE(herder.getHerderSCPDriver().validateValue(seq, votes.front(), - true) == - SCPDriver::kFullyValidatedValue); + REQUIRE(messages.front().statement.pledges.type() == SCP_ST_PREPARE); + REQUIRE(driver.validateValue( + seq, + messages.front().statement.pledges.prepare().ballot.value, + true) == SCPDriver::kFullyValidatedValue); herder.mGetTopTransactionsForTesting = nullptr; } REQUIRE(checkedFollower); } TEST_CASE("a leader build that completes a fetch can externalize", - "[herder][early-nomination]") + "[herder][early-proposal]") { // The local leader builds the very set a peer's ballot is waiting for (an // empty set is the same on every node). Publishing it completes the fetch - // inside SCP's nomination value supplier: the peer's CONFIRM is + // inside SCP's proposal value supplier: the peer's CONFIRM is // reprocessed there and, the peer being v-blocking, externalizes the slot - // before the supplier returns. Nomination must then stop cleanly without - // voting or emitting. + // before the supplier returns. The leader must then stop cleanly without + // proposing its own value. VirtualClock clock; clock.setCurrentVirtualTime(VirtualClock::from_time_t(1000)); auto cfg = getTestConfig(); @@ -7661,7 +6263,7 @@ TEST_CASE("a leader build that completes a fetch can externalize", auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); auto& scp = herder.getSCP(); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const qset = scp.getLocalQuorumSet(); auto const qsetHash = scp.getLocalNode()->getQuorumSetHash(); std::optional peer; @@ -7679,7 +6281,7 @@ TEST_CASE("a leader build that completes a fetch can externalize", auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); herder.mGetTopTransactionsForTesting = [](size_t) { return std::vector{}; }; @@ -7725,13 +6327,13 @@ TEST_CASE("a leader build that completes a fetch can externalize", REQUIRE(closed.header.scpValue == peerValue); for (auto const& env : scp.getLatestMessagesSend(seq)) { - REQUIRE(env.statement.pledges.type() != SCP_ST_NOMINATE); + REQUIRE(env.statement.pledges.type() == SCP_ST_EXTERNALIZE); } herder.mGetTopTransactionsForTesting = nullptr; } TEST_CASE("followers pre-verify mempool signatures at the trigger", - "[herder][early-nomination]") + "[herder][early-proposal]") { // Followers no longer validate their own mempool by building a set, so // the leader's set would be validated with a cold signature cache. At the @@ -7742,7 +6344,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", for (int i = 0; i < 3; ++i) { keys.push_back(SecretKey::fromSeed( - sha256(fmt::format("early-nomination-node-{}", i)))); + sha256(fmt::format("early-proposal-node-{}", i)))); qset.validators.push_back(keys.back().getPublicKey()); } @@ -7765,9 +6367,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; bool const leader = - scp.predictNominationLeaders( - seq, xdr::xdr_to_opaque(lcl.header.scpValue), 1) - .count(scp.getLocalNodeID()) != 0; + herder.getHerderSCPDriver().isLocalNodeRoundLeader(seq, 1); if (!checkedRoles.insert(leader).second) { continue; @@ -7797,7 +6397,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", { envelopes.push_back(tx->getEnvelope()); } - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); herder.mGetTopTransactionsForTesting = [&](size_t) { return envelopes; }; @@ -7807,12 +6407,12 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", PubKeyUtils::flushVerifySigCacheCounts(hits, misses); size_t const signatures = txs.size() + 1 /* op source */ + 1 /* fee bump's inner tx */; - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); clock.setCurrentVirtualTime(herder.getTriggerTimer().expiry_time()); testutil::crankUntil( - app, [&]() { return !scp.getNominationLeaders(seq).empty(); }, + app, [&]() { return scp.getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); - REQUIRE(!scp.getNominationLeaders(seq).empty()); + REQUIRE(scp.getLeaderRound(seq) != 0); if (leader) { @@ -7825,7 +6425,7 @@ TEST_CASE("followers pre-verify mempool signatures at the trigger", } // Background tasks verify every signature once. Wait in real time - // without cranking: virtual nomination timeouts would promote this + // without cranking: virtual leader timeouts would promote this // follower, whose build verifies signatures too. uint64_t verified = 0; auto const deadline = @@ -7929,25 +6529,24 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", ballot.counter = 1; ballot.value = value; herder.getHerderSCPDriver().startedBallotProtocol(slot, ballot); - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == 0); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == 0); testutil::crankUntil( app, [&]() { - return EarlyNominationTestAccess::prewarmLedger(herder) == slot + 1; + return LeaderBuildTestAccess::prewarmLedger(herder) == slot + 1; }, std::chrono::seconds(1)); - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == slot + 1); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == slot + 1); REQUIRE(requested == - 2 * EarlyNominationTestAccess::maxTxSetCandidates(herder) + - valueTxs); - REQUIRE(EarlyNominationTestAccess::waitForPrewarm(herder)); + 2 * LeaderBuildTestAccess::maxTxSetCandidates(herder) + valueTxs); + REQUIRE(LeaderBuildTestAccess::waitForPrewarm(herder)); // N externalizes and closes; the prewarm for N + 1 is kept. The overlay // drops N's transactions from the mempool. herder.externalizeValue(valueSet, slot, closeTime, emptyUpgradeSteps, cfg.NODE_SEED); REQUIRE(lm.getLastClosedLedgerNum() == slot); - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == slot + 1); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == slot + 1); mempool.erase(mempool.begin(), mempool.begin() + valueTxs); auto const nextLcl = lm.getLastClosedLedgerHeader(); auto const nextCloseTime = makeConsensusTime(app->timeNow() + 2); @@ -7979,7 +6578,7 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", { auto [hits0, misses0] = lookups("build"); auto txSet = - EarlyNominationTestAccess::build(herder, slot + 1, nextCloseTime); + LeaderBuildTestAccess::build(herder, slot + 1, nextCloseTime); REQUIRE(txSet->sizeTxTotal() == 16); checkLookups("build", hits0, misses0, target); } @@ -8005,7 +6604,7 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", } // Validation and the build stop the prewarm and record what it did: the // 16 transactions of N's value and the next 20, each verified once. - REQUIRE(EarlyNominationTestAccess::prewarmLedger(herder) == 0); + REQUIRE(LeaderBuildTestAccess::prewarmLedger(herder) == 0); REQUIRE(prewarmTxs.count() == txs0 + valueTxs + target); REQUIRE(prewarmVerified.count() == verified0 + valueTxs + target); REQUIRE(prewarmIncomplete.count() == incomplete0); @@ -8013,7 +6612,7 @@ TEST_CASE("signature prewarm at the ballot covers the next ledger", } TEST_CASE("early underfilled proposals refresh at the trigger", - "[herder][early-nomination]") + "[herder][early-proposal]") { auto const invalidFirst = GENERATE(false, true); VirtualClock clock; @@ -8030,10 +6629,10 @@ TEST_CASE("early underfilled proposals refresh at the trigger", SecretKey::pseudoRandomForTesting().getPublicKey()}; auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto const seq = lcl.header.ledgerSeq + 1; - herder.getHerderSCPDriver().recordSCPEvent(lcl.header.ledgerSeq, false); + herder.getHerderSCPDriver().recordBallotStart(lcl.header.ledgerSeq); auto alice = txtest::getGenesisAccount(*app, 0); auto bob = txtest::getGenesisAccount(*app, 1); auto tx = alice.tx({payment(bob, 100)}); @@ -8051,24 +6650,21 @@ TEST_CASE("early underfilled proposals refresh at the trigger", ++pulls; return mempool; }; - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); testutil::crankUntil( app, - [&]() { - return EarlyNominationTestAccess::prepared(herder) != nullptr; - }, + [&]() { return LeaderBuildTestAccess::prepared(herder) != nullptr; }, std::chrono::seconds(1)); REQUIRE(pulls == 1); - REQUIRE(EarlyNominationTestAccess::prepared(herder)->sizeTxTotal() == 0); + REQUIRE(LeaderBuildTestAccess::prepared(herder)->sizeTxTotal() == 0); // Transactions arrive while we are waiting for the normal trigger. mempool = {tx->getEnvelope(), extra->getEnvelope()}; clock.setCurrentVirtualTime(herder.getTriggerTimer().expiry_time()); testutil::crankUntil( - app, - [&]() { return !herder.getSCP().getNominationLeaders(seq).empty(); }, + app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); REQUIRE(pulls == 2); - // Check the actual set cached for nomination, independently of benchmark + // Check the actual set cached for the proposal, independently of benchmark // instrumentation. Both newly arrived transactions must be present. auto expected = makeTxSetFromTransactions(TxFrameList{tx, extra}, *app, ApplyTimeOffset{}) @@ -8078,12 +6674,12 @@ TEST_CASE("early underfilled proposals refresh at the trigger", herder.getTxSet(expected->getContentsHash())); REQUIRE(nominated); REQUIRE(nominated->sizeTxTotal() == 2); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); herder.mGetTopTransactionsForTesting = nullptr; } TEST_CASE("early preparation respects manual and immediately due triggers", - "[herder][early-nomination]") + "[herder][early-proposal]") { auto const manual = GENERATE(false, true); VirtualClock clock; @@ -8098,7 +6694,7 @@ TEST_CASE("early preparation respects manual and immediately due triggers", SecretKey::pseudoRandomForTesting().getPublicKey()}; auto app = createTestApplication(clock, cfg); auto& herder = static_cast(app->getHerder()); - EarlyNominationTestAccess::selectLocalFirstLeader(herder); + LeaderBuildTestAccess::selectLocalFirstLeader(herder); auto const seq = app->getLedgerManager().getLastClosedLedgerNum() + 1; size_t pulls = 0; herder.mGetTopTransactionsForTesting = [&](size_t) { @@ -8107,29 +6703,26 @@ TEST_CASE("early preparation respects manual and immediately due triggers", }; if (manual) { - REQUIRE(!EarlyNominationTestAccess::scheduled(herder)); + REQUIRE(!LeaderBuildTestAccess::scheduled(herder)); REQUIRE(herder.getTriggerTimer().seq() == 0); herder.triggerNextLedger(seq, true); } else { - EarlyNominationTestAccess::schedule(herder); + LeaderBuildTestAccess::schedule(herder); // No previous prepare timestamp: the normal fallback triggers now. REQUIRE(herder.getTriggerTimer().expiry_time() == clock.now()); // A due trigger builds itself. Preparing too could build twice: the // two timers would fire at the same instant in either order. - REQUIRE(!EarlyNominationTestAccess::scheduled(herder)); + REQUIRE(!LeaderBuildTestAccess::scheduled(herder)); testutil::crankUntil( - app, - [&]() { - return !herder.getSCP().getNominationLeaders(seq).empty(); - }, + app, [&]() { return herder.getSCP().getLeaderRound(seq) != 0; }, std::chrono::seconds(1)); } REQUIRE(pulls == 1); - REQUIRE(!EarlyNominationTestAccess::prepared(herder)); - REQUIRE(!EarlyNominationTestAccess::scheduled(herder)); - REQUIRE(!herder.getSCP().getNominationLeaders(seq).empty()); + REQUIRE(!LeaderBuildTestAccess::prepared(herder)); + REQUIRE(!LeaderBuildTestAccess::scheduled(herder)); + REQUIRE(herder.getSCP().getLeaderRound(seq) != 0); herder.mGetTopTransactionsForTesting = nullptr; } @@ -8553,7 +7146,7 @@ TEST_CASE("leader builds validate only the top of the mempool pull", }; auto const lcl = lm.getLastClosedLedgerHeader(); - auto [txSet, invalid] = EarlyNominationTestAccess::buildWithInvalid( + auto [txSet, invalid] = LeaderBuildTestAccess::buildWithInvalid( herder, lcl.header.ledgerSeq + 1, getConsensusTime(lcl.header.scpValue).next(lcl.header.ledgerVersion)); herder.mGetTopTransactionsForTesting = nullptr; @@ -8693,8 +7286,7 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") { CALLGRIND_START_INSTRUMENTATION; } - auto txSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + auto txSet = LeaderBuildTestAccess::build(herder, seq, closeTime); if (cold) { CALLGRIND_STOP_INSTRUMENTATION; @@ -8724,7 +7316,7 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") // A follower validating the received set (cold: calibration // against perfnet's herder.txset.validate). auto const& lastSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + LeaderBuildTestAccess::build(herder, seq, closeTime); for (uint32_t rep = 0; rep < reps; ++rep) { auto received = TxSetXDRFrame::makeFromWire( @@ -8764,23 +7356,22 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") auto& buildMisses = metrics.NewMeter( {"herder", "txset", "build-sig-cache-miss"}, "lookup"); auto const lastSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + LeaderBuildTestAccess::build(herder, seq, closeTime); uint64_t misses = 0; uint64_t validateMisses = 0; for (uint32_t rep = 0; rep < reps; ++rep) { PubKeyUtils::clearVerifySigCache(); auto const durationBefore = prewarmDuration.sum(); - EarlyNominationTestAccess::prewarm(herder, seq); - REQUIRE(EarlyNominationTestAccess::waitForPrewarm(herder)); + LeaderBuildTestAccess::prewarm(herder, seq); + REQUIRE(LeaderBuildTestAccess::waitForPrewarm(herder)); for (auto const& phase : phases) { metrics.NewTimer({"herder", "txset", phase}).Clear(); } auto const missesBefore = buildMisses.count(); // The build stops the (finished) prewarm and records it. - auto txSet = - EarlyNominationTestAccess::build(herder, seq, closeTime); + auto txSet = LeaderBuildTestAccess::build(herder, seq, closeTime); REQUIRE(txSet->sizeTxTotal() == std::min(sorobanCap, pull)); misses = buildMisses.count() - missesBefore; samples["prewarm (background, before the build)"].push_back( @@ -8794,8 +7385,8 @@ TEST_CASE("leader tx set build costs", "[herder][.][benchmark]") auto received = TxSetXDRFrame::makeFromWire( GeneralizedTransactionSet(lastSet->getGeneralizedXDR())); PubKeyUtils::clearVerifySigCache(); - EarlyNominationTestAccess::prewarm(herder, seq); - REQUIRE(EarlyNominationTestAccess::waitForPrewarm(herder)); + LeaderBuildTestAccess::prewarm(herder, seq); + REQUIRE(LeaderBuildTestAccess::waitForPrewarm(herder)); uint64_t hits0 = 0, misses0 = 0, hits1 = 0, misses1 = 0; PubKeyUtils::getVerifySigCacheCounts(hits0, misses0); auto const start = std::chrono::steady_clock::now(); diff --git a/src/herder/test/LeaderRoundsTests.cpp b/src/herder/test/LeaderRoundsTests.cpp new file mode 100644 index 0000000000..ad1db120e1 --- /dev/null +++ b/src/herder/test/LeaderRoundsTests.cpp @@ -0,0 +1,221 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +// Network-level tests of consensus without nomination: the leader of each +// round comes from a deterministic schedule seeded by the hash of the last +// closed ledger header (see herder/LeaderSchedule.h). + +#include "crypto/SHA.h" +#include "herder/Herder.h" +#include "herder/HerderImpl.h" +#include "herder/LeaderSchedule.h" +#include "ledger/LedgerHeaderUtils.h" +#include "ledger/LedgerManager.h" +#include "main/Application.h" +#include "simulation/Simulation.h" +#include "simulation/Topologies.h" +#include "test/Catch2.h" +#include "test/test.h" +#include "util/Logging.h" +#include "util/MetricsRegistry.h" +#include "util/XDROperators.h" +#include +#include +#include + +using namespace stellar; + +namespace +{ +Config +acceleratedConfig(int configNum) +{ + auto cfg = getTestConfig(configNum); + cfg.ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING = true; + return cfg; +} + +struct ClosedLedger +{ + Hash mPreviousHash; + NodeID mProposer; +}; + +// Cranks until `app` closed `lastLedger`, recording for every ledger it +// closes on the way the hash of the ledger before it and the node that signed +// the externalized value (its proposer). +std::map +crankAndRecord(Simulation::pointer const& simulation, + Application::pointer const& app, uint32_t lastLedger, + std::chrono::seconds timeout) +{ + std::map res; + auto& lm = app->getLedgerManager(); + auto prev = lm.getLastClosedLedgerHeader(); + auto const deadline = std::chrono::steady_clock::now() + timeout; + while (lm.getLastClosedLedgerNum() < lastLedger) + { + REQUIRE(std::chrono::steady_clock::now() < deadline); + simulation->crankForAtMost(std::chrono::milliseconds(20), false); + auto cur = lm.getLastClosedLedgerHeader(); + if (cur.header.ledgerSeq == prev.header.ledgerSeq + 1) + { + REQUIRE(cur.header.previousLedgerHash == prev.hash); + res[cur.header.ledgerSeq] = ClosedLedger{ + prev.hash, getLcValueSignature(cur.header.scpValue).nodeID}; + } + prev = cur; + } + return res; +} + +int64_t +leaderTimeouts(Application::pointer const& app) +{ + return static_cast( + app->getMetrics().NewHistogram({"scp", "timeout", "leader"}).sum()); +} + +int64_t +ballotTimeouts(Application::pointer const& app) +{ + return static_cast( + app->getMetrics().NewHistogram({"scp", "timeout", "prepare"}).sum()); +} +} + +TEST_CASE("leader schedule drives consensus on a healthy network", + "[herder][leader-schedule][simulation]") +{ + Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); + auto simulation = Topologies::core(4, 0.75, networkID, acceleratedConfig); + simulation->startAllNodes(); + auto nodes = simulation->getNodes(); + auto nodeIDs = simulation->getNodeIDs(); + + uint32_t const lastLedger = 16; + auto closed = crankAndRecord(simulation, nodes[0], lastLedger, + std::chrono::seconds(300)); + simulation->crankUntil( + [&]() { return simulation->haveAllExternalized(lastLedger, 3); }, + std::chrono::seconds(60), false); + + int64_t timeouts = 0; + for (auto const& n : nodes) + { + timeouts += leaderTimeouts(n) + ballotTimeouts(n); + } + + // Every node computes the same schedule from the previous ledger's hash, + // and the value that externalizes is the one its round 1 leader proposed + // (unless a round timed out somewhere, e.g. on a busy test machine). + std::set proposers; + int64_t notFirstLeader = 0; + REQUIRE(closed.size() >= 10); + for (auto const& c : closed) + { + auto schedule = computeLeaderSchedule(c.second.mPreviousHash, nodeIDs); + REQUIRE(schedule.size() == 4); + REQUIRE(std::find(schedule.begin(), schedule.end(), + c.second.mProposer) != schedule.end()); + if (c.second.mProposer != schedule[0]) + { + ++notFirstLeader; + } + proposers.insert(c.second.mProposer); + } + INFO(fmt::format("ledgers {} not-first-leader {} timeouts {}", + closed.size(), notFirstLeader, timeouts)); + REQUIRE(notFirstLeader <= timeouts); + // most ledgers close in round 1 + REQUIRE(notFirstLeader * 2 < static_cast(closed.size())); + // leadership rotates + REQUIRE(proposers.size() >= 2); + + // the nodes agree on the schedule of the next ledger + auto const next = nodes[0]->getLedgerManager().getLastClosedLedgerNum() + 1; + std::optional> expected; + for (auto const& n : nodes) + { + if (n->getLedgerManager().getLastClosedLedgerNum() + 1 != next) + { + continue; + } + auto& driver = + static_cast(n->getHerder()).getHerderSCPDriver(); + auto schedule = driver.getLeaderSchedule(next); + REQUIRE(schedule.size() == 4); + if (!expected) + { + expected = schedule; + } + REQUIRE(*expected == schedule); + REQUIRE(driver.getLeaderSchedule(next + 1).empty()); + REQUIRE(driver.getLeaderSchedule(next - 1).empty()); + } +} + +TEST_CASE("leader schedule fails over when a validator is down", + "[herder][leader-schedule][simulation]") +{ + // 3 validators, threshold 2, one never started. Whenever the dead node + // leads round 1 the others time out and the round 2 leader takes over. + Hash networkID = sha256(getTestConfig().NETWORK_PASSPHRASE); + auto simulation = + std::make_shared(networkID, acceleratedConfig); + + std::vector keys; + for (int i = 0; i < 3; i++) + { + keys.push_back( + SecretKey::fromSeed(sha256("NODE_SEED_" + std::to_string(i)))); + } + SCPQuorumSet qSet; + qSet.threshold = 2; + std::vector allNodes; + for (auto const& k : keys) + { + qSet.validators.push_back(k.getPublicKey()); + allNodes.push_back(k.getPublicKey()); + } + NodeID const dead = keys[2].getPublicKey(); + + simulation->addNode(keys[0], qSet); + simulation->addNode(keys[1], qSet); + simulation->addPendingConnection(keys[0].getPublicKey(), + keys[1].getPublicKey()); + simulation->startAllNodes(); + auto nodes = simulation->getNodes(); + + uint32_t const lastLedger = 25; + auto closed = crankAndRecord(simulation, nodes[0], lastLedger, + std::chrono::seconds(600)); + simulation->crankUntil( + [&]() { return simulation->haveAllExternalized(lastLedger, 3); }, + std::chrono::seconds(60), false); + + int64_t deadLedRound1 = 0; + REQUIRE(closed.size() >= 20); + for (auto const& c : closed) + { + // the schedule covers the whole quorum set, including the dead node + auto schedule = computeLeaderSchedule(c.second.mPreviousHash, allNodes); + REQUIRE(schedule.size() == 3); + // every ledger closed anyway, with a live node's proposal + REQUIRE(c.second.mProposer != dead); + if (schedule[0] == dead) + { + ++deadLedRound1; + } + } + INFO(fmt::format("ledgers {} led by the dead node {}", closed.size(), + deadLedRound1)); + // 1/3 of the ledgers on average: (2/3)^20 that it never happens + REQUIRE(deadLedRound1 > 0); + // and each of those cost the live nodes a leader timeout + for (auto const& n : nodes) + { + REQUIRE(leaderTimeouts(n) >= deadLedRound1); + } +} diff --git a/src/herder/test/LeaderScheduleTests.cpp b/src/herder/test/LeaderScheduleTests.cpp new file mode 100644 index 0000000000..c5bc3d1496 --- /dev/null +++ b/src/herder/test/LeaderScheduleTests.cpp @@ -0,0 +1,123 @@ +// Copyright 2026 Stellar Development Foundation and contributors. Licensed +// under the Apache License, Version 2.0. See the COPYING file at the root +// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 + +#include "crypto/SHA.h" +#include "crypto/SecretKey.h" +#include "herder/LeaderSchedule.h" +#include "test/Catch2.h" +#include "util/Math.h" +#include "util/XDROperators.h" +#include +#include +#include +#include + +using namespace stellar; + +namespace +{ +std::vector +makeNodes(size_t n) +{ + std::vector nodes; + for (size_t i = 0; i < n; i++) + { + nodes.emplace_back( + SecretKey::fromSeed(sha256(fmt::format("SCHEDULE_NODE_{}", i))) + .getPublicKey()); + } + return nodes; +} +} + +TEST_CASE("leader schedule is a deterministic permutation", + "[herder][leader-schedule]") +{ + auto nodes = makeNodes(30); + Hash seed = sha256("ledger header 41"); + + auto schedule = computeLeaderSchedule(seed, nodes); + + SECTION("it is a permutation of the node set") + { + REQUIRE(schedule.size() == nodes.size()); + REQUIRE(std::set(schedule.begin(), schedule.end()) == + std::set(nodes.begin(), nodes.end())); + } + SECTION("it does not depend on the order nodes are listed in") + { + // every node enumerates its quorum in its own order + for (int i = 0; i < 20; i++) + { + auto shuffled = nodes; + stellar::shuffle(shuffled.begin(), shuffled.end(), + getGlobalRandomEngine()); + REQUIRE(computeLeaderSchedule(seed, shuffled) == schedule); + } + } + SECTION("duplicates are ignored") + { + auto dup = nodes; + dup.insert(dup.end(), nodes.begin(), nodes.begin() + 7); + REQUIRE(computeLeaderSchedule(seed, dup) == schedule); + } + SECTION("the seed changes the order") + { + auto other = computeLeaderSchedule(sha256("ledger header 42"), nodes); + REQUIRE(std::set(other.begin(), other.end()) == + std::set(nodes.begin(), nodes.end())); + REQUIRE(other != schedule); + } + SECTION("the node set changes the order only by the nodes involved") + { + // removing a node leaves the relative order of the others alone + auto fewer = nodes; + NodeID removed = fewer.back(); + fewer.pop_back(); + auto expected = schedule; + expected.erase(std::find(expected.begin(), expected.end(), removed)); + REQUIRE(computeLeaderSchedule(seed, fewer) == expected); + } + SECTION("degenerate sets") + { + REQUIRE(computeLeaderSchedule(seed, {}).empty()); + REQUIRE(computeLeaderSchedule(seed, {nodes[3]}) == + std::vector{nodes[3]}); + } +} + +TEST_CASE("leader schedule is fair across seeds", "[herder][leader-schedule]") +{ + // Each node should lead round 1 about 1/n of the time, and the node + // leading round 2 should be independent of the one leading round 1. + size_t const n = 10; + size_t const slots = 20000; + auto nodes = makeNodes(n); + std::map first; + std::map, size_t> firstTwo; + for (size_t i = 0; i < slots; i++) + { + auto schedule = + computeLeaderSchedule(sha256(fmt::format("header {}", i)), nodes); + first[schedule[0]]++; + firstTwo[{schedule[0], schedule[1]}]++; + } + REQUIRE(first.size() == n); + double const expected = static_cast(slots) / n; + for (auto const& f : first) + { + // 2000 expected, standard deviation ~42: 10% is over 4 sigma + REQUIRE(f.second > expected * 0.9); + REQUIRE(f.second < expected * 1.1); + } + // all ordered pairs of distinct nodes show up + REQUIRE(firstTwo.size() == n * (n - 1)); + double const expectedPair = static_cast(slots) / (n * (n - 1)); + for (auto const& f : firstTwo) + { + // 222 expected, standard deviation ~15 + REQUIRE(f.second > expectedPair * 0.6); + REQUIRE(f.second < expectedPair * 1.4); + } +} diff --git a/src/herder/test/PendingEnvelopesTests.cpp b/src/herder/test/PendingEnvelopesTests.cpp index 0eee4f71c6..6b8085ade0 100644 --- a/src/herder/test/PendingEnvelopesTests.cpp +++ b/src/herder/test/PendingEnvelopesTests.cpp @@ -100,6 +100,44 @@ makePublicKey(int i) auto secretKey = SecretKey::fromSeed(hash); return secretKey.getPublicKey(); } + +// The local node votes for the round 1 leader's ballot as soon as SCP sees +// it, before its own trigger. These tests feed the node a peer's statements +// and then inspect PendingEnvelopes (reference counts, fetch status), which +// such a vote would change. So they use a peer that does not lead round 1 of +// the next slot: the local node then leads it (or a node that never speaks +// does, once the peer's quorum set is known) and does not vote. +struct FollowerPeerApp +{ + // destroyed last: the application uses the clock + std::unique_ptr mClock; + SecretKey mPeer; + Application::pointer mApp; +}; + +FollowerPeerApp +createAppWithFollowerPeer(Config const& baseCfg) +{ + for (int i = 0; i < 100; ++i) + { + FollowerPeerApp res; + res.mPeer = SecretKey::pseudoRandomForTesting(); + Config cfg(baseCfg); + cfg.QUORUM_SET.validators.emplace_back(res.mPeer.getPublicKey()); + res.mClock = std::make_unique(); + res.mApp = createTestApplication(*res.mClock, cfg); + auto& herder = static_cast(res.mApp->getHerder()); + auto const seq = + res.mApp->getLedgerManager().getLastClosedLedgerNum() + 1; + auto leader = herder.getHerderSCPDriver().getRoundLeader(seq, 1); + if (leader && !(*leader == res.mPeer.getPublicKey())) + { + return res; + } + } + throw std::runtime_error( + "could not pick a peer that does not lead round 1"); +} } // namespace TEST_CASE_VERSIONS("PendingEnvelopes recvSCPEnvelope", "[herder]") @@ -108,12 +146,10 @@ TEST_CASE_VERSIONS("PendingEnvelopes recvSCPEnvelope", "[herder]") cfg.MANUAL_CLOSE = false; cfg.EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = true; - VirtualClock clock; - - auto s = SecretKey::pseudoRandomForTesting(); + auto fixture = createAppWithFollowerPeer(cfg); + auto s = fixture.mPeer; auto& pk = s.getPublicKey(); - cfg.QUORUM_SET.validators.emplace_back(s.getPublicKey()); - Application::pointer app = createTestApplication(clock, cfg); + Application::pointer app = fixture.mApp; auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); @@ -766,12 +802,10 @@ TEST_CASE("PendingEnvelopes recvSCPEnvelope without parallel tx set download", cfg.MANUAL_CLOSE = false; cfg.EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD = false; - VirtualClock clock; - - auto s = SecretKey::pseudoRandomForTesting(); + auto fixture = createAppWithFollowerPeer(cfg); + auto s = fixture.mPeer; auto& pk = s.getPublicKey(); - cfg.QUORUM_SET.validators.emplace_back(s.getPublicKey()); - Application::pointer app = createTestApplication(clock, cfg); + Application::pointer app = fixture.mApp; auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); @@ -822,11 +856,11 @@ TEST_CASE("PendingEnvelopes re-requests tx sets the overlay does not deliver", "[herder]") { Config cfg(getTestConfig()); - VirtualClock clock; - auto s = SecretKey::pseudoRandomForTesting(); + auto fixture = createAppWithFollowerPeer(cfg); + auto& clock = *fixture.mClock; + auto s = fixture.mPeer; auto& pk = s.getPublicKey(); - cfg.QUORUM_SET.validators.emplace_back(pk); - Application::pointer app = createTestApplication(clock, cfg); + Application::pointer app = fixture.mApp; auto const lcl = app->getLedgerManager().getLastClosedLedgerHeader(); auto& herder = static_cast(app->getHerder()); auto& pendingEnvelopes = herder.getPendingEnvelopes(); diff --git a/src/herder/test/QuorumTrackerTests.cpp b/src/herder/test/QuorumTrackerTests.cpp index 030f800824..04abe00560 100644 --- a/src/herder/test/QuorumTrackerTests.cpp +++ b/src/herder/test/QuorumTrackerTests.cpp @@ -78,21 +78,25 @@ testQuorumTracker() herder->recvTxSet(p.mTxSet->getContentsHash(), p.mTxSet); } }; - auto recvNom = [&](uint64 slotID, SecretKey const& k, - SCPQuorumSet const& qSet, - std::vector const& pp) { + auto recvPrepare = [&](uint64 slotID, SecretKey const& k, + SCPQuorumSet const& qSet, + std::vector const& pp) { + // A PREPARE that references every value of `pp`: the last one is the + // ballot's value, an earlier one (if any) is the prepared value. More + // values make a newer statement. + releaseAssert(!pp.empty() && pp.size() <= 2); SCPEnvelope envelope; - envelope.statement.pledges.type(SCP_ST_NOMINATE); - auto& nom = envelope.statement.pledges.nominate(); - - std::set values; - for (auto& p : pp) + envelope.statement.pledges.type(SCP_ST_PREPARE); + auto& prep = envelope.statement.pledges.prepare(); + prep.ballot.counter = static_cast(pp.size()); + prep.ballot.value = pp.back().mSignedV; + if (pp.size() > 1) { - values.insert(p.mSignedV); + prep.prepared.activate().counter = 1; + prep.prepared->value = pp.front().mSignedV; } - nom.votes.insert(nom.votes.begin(), values.begin(), values.end()); auto qSetH = sha256(xdr::xdr_to_opaque(qSet)); - nom.quorumSetHash = qSetH; + prep.quorumSetHash = qSetH; recvEnvelope(envelope, slotID, k, qSet, pp); }; auto recvExternalize = [&](uint64 slotID, SecretKey const& k, @@ -135,17 +139,17 @@ testQuorumTracker() SECTION("Receive self") { checkInQuorum({0, 1}); - recvNom(3, cfg.NODE_SEED, cfg.QUORUM_SET, {vv}); + recvPrepare(3, cfg.NODE_SEED, cfg.QUORUM_SET, {vv}); checkInQuorum({0, 1}); } SECTION("Expand 0") { checkInQuorum({0, 1}); - recvNom(3, otherKeys[0], qSet0, {vv}); + recvPrepare(3, otherKeys[0], qSet0, {vv}); checkInQuorum({0, 1, 2, 3}); SECTION("Expand 2") { - recvNom(3, otherKeys[2], qSet2, {vv}); + recvPrepare(3, otherKeys[2], qSet2, {vv}); checkInQuorum({0, 1, 2, 3, 5, 6}); SECTION("node restart") { @@ -168,13 +172,13 @@ testQuorumTracker() SECTION("Update 0's qSet") { auto vv2 = makeValue(2); - recvNom(3, otherKeys[0], qSet0b, {vv, vv2}); + recvPrepare(3, otherKeys[0], qSet0b, {vv, vv2}); checkInQuorum({0, 1, 4, 5}); } SECTION("Update 0's qSet in an old slot") { auto vv2 = makeValue(2); - recvNom(2, otherKeys[0], qSet0b, {vv, vv2}); + recvPrepare(2, otherKeys[0], qSet0b, {vv, vv2}); // nothing changes (slot 3 has precedence) checkInQuorum({0, 1, 2, 3}); } diff --git a/src/main/Config.cpp b/src/main/Config.cpp index 457e23ab9c..33e580961a 100644 --- a/src/main/Config.cpp +++ b/src/main/Config.cpp @@ -68,7 +68,6 @@ static std::unordered_set const TESTING_ONLY_OPTIONS = { "ARTIFICIALLY_SET_SURVEY_PHASE_DURATION_FOR_TESTING", "ARTIFICIALLY_DELAY_BUCKET_APPLICATION_FOR_TESTING", "ARTIFICIALLY_SLEEP_MAIN_THREAD_FOR_TESTING", - "ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING", "ARTIFICIALLY_SKIP_CONNECTION_ADJUSTMENT_FOR_TESTING", "ARTIFICIALLY_DELAY_LEDGER_CLOSE_FOR_TESTING", "SKIP_HIGH_CRITICAL_VALIDATOR_CHECKS_FOR_TESTING", @@ -155,7 +154,7 @@ Config::Config() : NODE_SEED(SecretKey::random()) TESTING_MAX_CLASSIC_BYTE_ALLOWANCE = 0; IGNORE_MESSAGE_LIMITS_FOR_TESTING = false; TESTING_IGNORE_LEDGER_TIME_UPGRADE_BOUNDS = false; - TESTING_NOMINATE_RANDOM_VALUES = false; + TESTING_PROPOSE_RANDOM_VALUES = false; ALLOW_PRIVATE_ADDRESSES_FOR_TESTING = false; #endif @@ -376,8 +375,6 @@ Config::Config() : NODE_SEED(SecretKey::random()) SKIP_HIGH_CRITICAL_VALIDATOR_CHECKS_FOR_TESTING = false; ARTIFICIALLY_SET_SYSTEM_CLOCK_OFFSET_FOR_TESTING = std::chrono::milliseconds::zero(); - ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING = - std::chrono::milliseconds::zero(); #endif #ifdef BEST_OFFER_DEBUGGING @@ -1179,8 +1176,8 @@ Config::processConfig(std::shared_ptr t) "upgrade bounds for testing"); } }}, - {"TESTING_NOMINATE_RANDOM_VALUES", - [&]() { TESTING_NOMINATE_RANDOM_VALUES = readBool(item); }}, + {"TESTING_PROPOSE_RANDOM_VALUES", + [&]() { TESTING_PROPOSE_RANDOM_VALUES = readBool(item); }}, #endif {"PEER_PORT", [&]() { PEER_PORT = readInt(item, 1); }}, @@ -1985,11 +1982,6 @@ Config::processConfig(std::shared_ptr t) ARTIFICIALLY_SET_SYSTEM_CLOCK_OFFSET_FOR_TESTING = std::chrono::milliseconds(readInt(item)); }}, - {"ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING", - [&]() { - ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING = - std::chrono::milliseconds(readInt(item)); - }}, #endif {"MAX_DEX_TX_OPERATIONS_IN_TX_SET", [&]() { diff --git a/src/main/Config.h b/src/main/Config.h index da3a797206..f3238b9a41 100644 --- a/src/main/Config.h +++ b/src/main/Config.h @@ -952,16 +952,12 @@ class Config : public std::enable_shared_from_this // When set, this node will nominate random values when it is the round // leader. This is useful for testing CAP-0083 behavior. This is a testing // only flag. - bool TESTING_NOMINATE_RANDOM_VALUES; + bool TESTING_PROPOSE_RANDOM_VALUES; // Injects a signed wall-clock offset into the node's system clock for // testing. Expressed in milliseconds. std::chrono::milliseconds ARTIFICIALLY_SET_SYSTEM_CLOCK_OFFSET_FOR_TESTING; - // Delay emission of updated nomination messages for testing nomination - // timeout behavior. Expressed in milliseconds. - std::chrono::milliseconds ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING; - // Set QUORUM_SET using automatic quorum set configuration based on // `validators`. void diff --git a/src/overlay/test/OverlayIPCTests.cpp b/src/overlay/test/OverlayIPCTests.cpp index 12a3f5c60c..e79cd21b08 100644 --- a/src/overlay/test/OverlayIPCTests.cpp +++ b/src/overlay/test/OverlayIPCTests.cpp @@ -56,18 +56,19 @@ makeMockSCPEnvelope(uint64_t slotIndex, uint32_t nodeId) { SCPEnvelope env; env.statement.slotIndex = slotIndex; - env.statement.pledges.type(SCP_ST_NOMINATE); + env.statement.pledges.type(SCP_ST_PREPARE); // Set some mock data - auto& nom = env.statement.pledges.nominate(); - nom.quorumSetHash.fill(static_cast(nodeId)); + auto& prep = env.statement.pledges.prepare(); + prep.quorumSetHash.fill(static_cast(nodeId)); + prep.ballot.counter = 1; // Value is opaque<> (xvector), not Hash Value mockValue; mockValue.resize(32); std::fill(mockValue.begin(), mockValue.end(), static_cast(slotIndex & 0xFF)); - nom.votes.push_back(mockValue); + prep.ballot.value = mockValue; return env; } @@ -404,11 +405,11 @@ TEST_CASE("Rust overlay SCP consensus", "[overlay-ipc][.]") } /** - * Test TX set building and nomination hash request. + * Test TX set building and proposal hash request. * * This test: * 1. Creates an OverlayIPC connection to a Rust overlay - * 2. Requests a nomination hash (which builds a TX set from empty mempool) + * 2. Requests a proposal hash (which builds a TX set from empty mempool) * 3. Verifies a hash is returned */ TEST_CASE("Rust overlay get top transactions", "[overlay-ipc][.]") @@ -1008,7 +1009,7 @@ TEST_CASE("Rust overlay TX included in ledger", "[overlay-ipc][.]") * - 200 tx/ledger (heavy load) * * For each rate, runs for several ledgers and measures: - * - scp.timing.nominated (time from nomination to prepare) + * - scp.timing.trigger-to-ballot (time from the trigger to the first ballot) * - scp.timing.externalized (time from prepare to externalize) * - ledger.ledger.close (total ledger close time) */ @@ -1034,8 +1035,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", std::string label; int txSubmitted; int txIncluded; - double scpNominatedMean; - double scpNominatedMax; + double scpTriggerToBallotMean; + double scpTriggerToBallotMax; double scpExternalizedMean; double scpExternalizedMax; double ledgerCloseMean; @@ -1095,8 +1096,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", // Get metrics (they accumulate across ledgers) auto& metrics = node0->getMetrics(); - auto& scpNominated = - metrics.NewTimer({"scp", "timing", "nominated"}); + auto& scpTriggerToBallot = + metrics.NewTimer({"scp", "timing", "trigger-to-ballot"}); auto& scpExternalized = metrics.NewTimer({"scp", "timing", "externalized"}); auto& ledgerClose = metrics.NewTimer({"ledger", "ledger", "close"}); @@ -1214,8 +1215,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", res.label = run.label; res.txSubmitted = txSubmitted; res.txIncluded = static_cast(txIncluded); - res.scpNominatedMean = scpNominated.mean(); - res.scpNominatedMax = scpNominated.max(); + res.scpTriggerToBallotMean = scpTriggerToBallot.mean(); + res.scpTriggerToBallotMax = scpTriggerToBallot.max(); res.scpExternalizedMean = scpExternalized.mean(); res.scpExternalizedMax = scpExternalized.max(); res.ledgerCloseMean = ledgerClose.mean(); @@ -1250,8 +1251,8 @@ TEST_CASE("Rust overlay SCP latency under TX load", { LOG_INFO(DEFAULT_LOG, "{:<20} {:>10} {:>10} {:>12.2f} {:>12.2f} {:>12.2f} {:>12.2f}", - r.label, r.txSubmitted, r.txIncluded, r.scpNominatedMean, - r.scpNominatedMax, r.scpExternalizedMean, + r.label, r.txSubmitted, r.txIncluded, r.scpTriggerToBallotMean, + r.scpTriggerToBallotMax, r.scpExternalizedMean, r.scpExternalizedMax); } @@ -1272,11 +1273,11 @@ TEST_CASE("Rust overlay SCP latency under TX load", // Allow 3x degradation from light to heavy load if (allResults.size() >= 2) { - double lightMean = allResults[0].scpNominatedMean; - double heavyMean = allResults.back().scpNominatedMean; + double lightMean = allResults[0].scpTriggerToBallotMean; + double heavyMean = allResults.back().scpTriggerToBallotMean; LOG_INFO(DEFAULT_LOG, - "SCP nominated latency ratio (heavy/light): {:.2f}x", + "SCP trigger-to-ballot latency ratio (heavy/light): {:.2f}x", heavyMean / lightMean); // Warn but don't fail if degradation is significant @@ -1400,7 +1401,8 @@ TEST_CASE("Rust overlay 15-node 2000 TPS stress test", // Get metrics from node 0 auto& metrics = nodes[0]->getMetrics(); - auto& scpNominated = metrics.NewTimer({"scp", "timing", "nominated"}); + auto& scpTriggerToBallot = + metrics.NewTimer({"scp", "timing", "trigger-to-ballot"}); auto& scpExternalized = metrics.NewTimer({"scp", "timing", "externalized"}); auto& ledgerClose = metrics.NewTimer({"ledger", "ledger", "close"}); @@ -1470,8 +1472,8 @@ TEST_CASE("Rust overlay 15-node 2000 TPS stress test", LOG_INFO(DEFAULT_LOG, " Effective TPS: {:.0f}", effectiveTps); LOG_INFO(DEFAULT_LOG, ""); LOG_INFO(DEFAULT_LOG, "SCP Timing (ms):"); - LOG_INFO(DEFAULT_LOG, " Nominated: mean={:.2f}, max={:.2f}", - scpNominated.mean(), scpNominated.max()); + LOG_INFO(DEFAULT_LOG, " TriggerToBallot: mean={:.2f}, max={:.2f}", + scpTriggerToBallot.mean(), scpTriggerToBallot.max()); LOG_INFO(DEFAULT_LOG, " Externalized: mean={:.2f}, max={:.2f}", scpExternalized.mean(), scpExternalized.max()); LOG_INFO(DEFAULT_LOG, ""); @@ -1484,10 +1486,10 @@ TEST_CASE("Rust overlay 15-node 2000 TPS stress test", // Warn if SCP latency is concerning (but don't fail - this is // informational) - if (scpNominated.max() > 5000) + if (scpTriggerToBallot.max() > 5000) { - WARN("SCP nomination max latency exceeded 5s: " << scpNominated.max() - << "ms"); + WARN("SCP trigger-to-ballot max latency exceeded 5s: " + << scpTriggerToBallot.max() << "ms"); } LOG_INFO(DEFAULT_LOG, diff --git a/src/scp/BallotProtocol.cpp b/src/scp/BallotProtocol.cpp index b69f5c2a86..01bc51c566 100644 --- a/src/scp/BallotProtocol.cpp +++ b/src/scp/BallotProtocol.cpp @@ -32,6 +32,11 @@ BallotProtocol::BallotProtocol(Slot& slot) , mPhase(SCP_PHASE_PREPARE) , mCurrentMessageLevel(0) , mTimerExpCount(0) + , mLeaderRound(0) + , mLeaderRoundsStarted(false) + , mLeaderTimerRound(0) + , mLeaderTimerExtended(false) + , mInProgressRound(false) { } @@ -247,7 +252,18 @@ BallotProtocol::processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self) } recordEnvelope(envelope); + if (!self) + { + // the statement may be the round leader's proposal + progressRound(); + } advanceSlot(statement); + if (!self) + { + // processing the statement may have moved the local node to a + // new round, or given it a value it has to stick to + progressRound(); + } return SCP::EnvelopeState::VALID; } @@ -349,28 +365,331 @@ BallotProtocol::abandonBallot(uint32 n) { ZoneScoped; CLOG_TRACE(SCP, "BallotProtocol::abandonBallot"); + if (mPhase != SCP_PHASE_PREPARE && mPhase != SCP_PHASE_CONFIRM) + { + return false; + } + if (n == 0 && mLeaderRound == UINT32_MAX) + { + // already at the "infinite" counter of a committed ballot + return false; + } + uint32 newRound = (n == 0) ? (mLeaderRound + 1) : n; + if (newRound > mLeaderRound) + { + mLeaderRound = newRound; + if (mLeaderRound != UINT32_MAX) + { + mSlot.getSCPDriver().enteredRound(mSlot.getSlotIndex(), + mLeaderRound, n == 0); + } + } + return progressRound(); +} + +bool +BallotProtocol::startLeaderRounds(ProposalValueSupplier const& makeValue) +{ + ZoneScoped; + if (mPhase != SCP_PHASE_PREPARE && mPhase != SCP_PHASE_CONFIRM) + { + return false; + } + mMakeValue = makeValue; + mLeaderRoundsStarted = true; + if (mLeaderRound == 0) + { + mLeaderRound = 1; + mSlot.getSCPDriver().enteredRound(mSlot.getSlotIndex(), mLeaderRound, + false); + } + return progressRound(); +} + +std::optional +BallotProtocol::getLeaderBallot(NodeID const& leader, uint32 round) const +{ + auto it = mLatestEnvelopes.find(leader); + if (it == mLatestEnvelopes.end()) + { + return std::nullopt; + } + auto const& pl = it->second->getStatement().pledges; + switch (pl.type()) + { + case SCP_ST_PREPARE: + if (pl.prepare().ballot.counter == round) + { + return pl.prepare().ballot; + } + break; + case SCP_ST_CONFIRM: + if (pl.confirm().ballot.counter == round) + { + return pl.confirm().ballot; + } + break; + default: + // an EXTERNALIZE statement is not tied to a round: the regular + // accept/confirm steps deal with it + break; + } + return std::nullopt; +} + +bool +BallotProtocol::progressRound() +{ + ZoneScoped; + if (mInProgressRound) + { + // the outer call loops until there is nothing left to do + return false; + } + mInProgressRound = true; bool res = false; - auto v = mSlot.getLatestCompositeCandidate(); + try + { + // A step that changes the ballot may enable another one (e.g. the + // node moved to a new round while processing its own statement). + // The ballot only ever increases, and each productive step raises + // its counter to the current round, so this ends. + while (progressRoundStep()) + { + res = true; + } + updateLeaderTimer(); + } + catch (...) + { + mInProgressRound = false; + throw; + } + mInProgressRound = false; + return res; +} - if (!v || v->getValue().empty()) +bool +BallotProtocol::progressRoundStep() +{ + if (mPhase != SCP_PHASE_PREPARE && mPhase != SCP_PHASE_CONFIRM) { - if (mCurrentBallot) + return false; + } + if (!getLocalNode()->isValidator()) + { + // only validators vote + return false; + } + + // Before the application starts the slot, the local node may still adopt + // what the round 1 leader put forward: followers do not wait for their + // own clock. + uint32 const round = std::max(mLeaderRound, 1); + uint32 const counter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + if (counter >= round) + { + return false; + } + + auto& driver = mSlot.getSCPDriver(); + uint64 const slotIndex = mSlot.getSlotIndex(); + + if (mValueOverride) + { + // The node saw this value confirmed prepared, or voted to commit it: + // it sticks to it whoever leads the round (bumpState enforces this + // for any value passed in). + if (mLeaderRound == 0) { - v = mCurrentBallot->getWValue(); + return false; } + return bumpState(mValueOverride->getValue(), round); } - if (v && !v->getValue().empty()) + + if (mPhase != SCP_PHASE_PREPARE) { - if (n == 0) + // CONFIRM has a value override + return false; + } + + if (round == UINT32_MAX) + { + // The "infinite" counter of a committed ballot (a v-blocking set + // externalized) is not a round and has no leader: just keep up with + // the value the node already votes for, if any. + if (mCurrentBallot && mLeaderRound != 0) { - res = bumpState(v->getValue(), true); + Value v = mCurrentBallot->getBallot().value; + return bumpState(v, round); } - else + return false; + } + + auto leader = driver.getRoundLeader(slotIndex, round); + if (!leader) + { + return false; + } + + if (*leader == getLocalNode()->getNodeID()) + { + if (mLeaderRound == 0) + { + // not started: the application is not ready to propose + return false; + } + // Keep the value the node already votes for unless it is not + // acceptable as a proposal (e.g. an empty-tx-set value): the nodes + // that voted for it already know (or are fetching) its transaction + // set. bumpState gives up on a transaction set that never arrived. + if (mCurrentBallot && + driver.validateValue(slotIndex, mCurrentBallot->getBallot().value, + /*proposal=*/true) != SCPDriver::kInvalidValue) { - res = bumpState(v->getValue(), n); + Value v = mCurrentBallot->getBallot().value; + driver.proposedValue(slotIndex, round, v); + return bumpState(v, round); } + if (!mLeaderRoundsStarted || !mMakeValue || mCurrentMessageLevel != 0) + { + // Building a value re-enters the slot, so it is only done from + // the top level; processEnvelope calls back once it is there. + return false; + } + auto value = mMakeValue(); + // Building the value can make pending envelopes ready and re-enter + // SCP: the slot may have moved on. + uint32 const newCounter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + if (!value || mPhase != SCP_PHASE_PREPARE || mValueOverride || + mLeaderRound != round || newCounter >= round) + { + // re-evaluate from the new state, if it changed + return newCounter != counter || mLeaderRound != round || + mValueOverride != nullptr; + } + driver.proposedValue(slotIndex, round, value->getValue()); + return bumpState(value->getValue(), round); } - return res; + + uint32 adoptRound = round; + auto leaderBallot = getLeaderBallot(*leader, round); + if (!leaderBallot && !mCurrentBallot) + { + // The node gave up on earlier rounds without ever voting. If one of + // their leaders' proposals shows up late (e.g. its transaction set + // took longer than the leader timeout to arrive), vote for it at its + // own counter rather than discard it: the nodes that were in time + // voted for it too. The node stays in its round, so the current + // leader's ballot (higher counter) still takes over when it comes. + static uint32 const MAX_LATE_ROUNDS = 8; + for (uint32 r = round - 1; + r >= 1 && round - r <= MAX_LATE_ROUNDS && !leaderBallot; --r) + { + auto pastLeader = driver.getRoundLeader(slotIndex, r); + if (pastLeader && !(*pastLeader == getLocalNode()->getNodeID())) + { + leaderBallot = getLeaderBallot(*pastLeader, r); + adoptRound = r; + } + } + } + if (!leaderBallot) + { + return false; + } + // Voting for a leader's value is the local node's endorsement of it as a + // proposal, so the stricter validation applies (as it did for nomination + // votes): e.g. it must only carry upgrades the local node armed. + // An empty-tx-set value is never built as a proposal: a leader only votes + // for one after the ballot protocol replaced a value whose transaction + // set did not arrive. Following it is a ballot-level decision, so the + // ballot validation applies. + bool const proposal = !driver.isEmptyTxSetValue(leaderBallot->value); + if (driver.validateValue(slotIndex, leaderBallot->value, proposal) < + SCPDriver::kStructurallyValidValue) + { + return false; + } + return bumpState(leaderBallot->value, adoptRound); +} + +void +BallotProtocol::updateLeaderTimer() +{ + uint32 const counter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + bool const waiting = mPhase == SCP_PHASE_PREPARE && mLeaderRound != 0 && + mLeaderRound != UINT32_MAX && counter < mLeaderRound && + getLocalNode()->isValidator(); + if (waiting) + { + if (mLeaderTimerRound != mLeaderRound) + { + // The ballot (if any) is for a round the node gave up on. + stopBallotProtocolTimer(); + mLeaderTimerRound = mLeaderRound; + mLeaderTimerExtended = false; + armLeaderTimer(mLeaderRound); + } + } + else if (mLeaderTimerRound != 0) + { + mLeaderTimerRound = 0; + mSlot.getSCPDriver().setupTimer(mSlot.getSlotIndex(), + Slot::LEADER_TIMER, + std::chrono::seconds::zero(), nullptr); + } +} + +void +BallotProtocol::armLeaderTimer(uint32 round) +{ + std::shared_ptr slot = mSlot.shared_from_this(); + mSlot.getSCPDriver().setupTimer( + mSlot.getSlotIndex(), Slot::LEADER_TIMER, + mSlot.getSCPDriver().computeTimeout(round), [slot, round]() { + slot->getBallotProtocol().leaderTimerExpired(round); + }); +} + +void +BallotProtocol::leaderTimerExpired(uint32 round) +{ + if (mLeaderTimerRound != round) + { + // stale callback + return; + } + uint32 const counter = + mCurrentBallot ? mCurrentBallot->getBallot().counter : 0; + if (mPhase != SCP_PHASE_PREPARE || mLeaderRound != round || + counter >= round) + { + mLeaderTimerRound = 0; + return; + } + if (!mLeaderTimerExtended) + { + // The leader is not silent if its statement already reached the + // application and is only waiting for its transaction set: give it + // one more timeout before moving on, rather than have the next leader + // build a competing value. + auto leader = + mSlot.getSCPDriver().getRoundLeader(mSlot.getSlotIndex(), round); + if (leader && mSlot.getSCPDriver().isProposalPending( + mSlot.getSlotIndex(), *leader)) + { + mLeaderTimerExtended = true; + armLeaderTimer(round); + return; + } + } + mLeaderTimerRound = 0; + mTimerExpCount++; + abandonBallot(0); } bool @@ -580,13 +899,27 @@ BallotProtocol::bumpToBallot(SCPBallot const& ballot, bool check) { mHeardFromQuorum = false; } + + // The ballot can also move through the regular accept/confirm steps + // (e.g. a quorum confirmed a higher ballot prepared). + if (ballot.counter > mLeaderRound) + { + mLeaderRound = ballot.counter; + // UINT32_MAX is the "infinite" counter of a committed ballot, not a + // round + if (mLeaderRound != UINT32_MAX) + { + mSlot.getSCPDriver().enteredRound(mSlot.getSlotIndex(), + mLeaderRound, false); + } + } } void BallotProtocol::startBallotProtocolTimer() { std::chrono::milliseconds timeout = mSlot.getSCPDriver().computeTimeout( - mCurrentBallot->getBallot().counter, /*isNomination=*/false); + mCurrentBallot->getBallot().counter); std::shared_ptr slot = mSlot.shared_from_this(); mSlot.getSCPDriver().setupTimer( @@ -607,6 +940,8 @@ void BallotProtocol::ballotProtocolTimerExpired() { mTimerExpCount++; + // The timer only runs for a ballot at the current round (see + // checkHeardFromQuorum), so this moves to the round after it. abandonBallot(0); } @@ -1438,7 +1773,7 @@ void BallotProtocol::throwIfValueInvalidForConfirmCommit(Value const& value) { auto validationLevel = mSlot.getSCPDriver().validateValue( - mSlot.getSlotIndex(), value, /*nomination=*/false); + mSlot.getSlotIndex(), value, /*proposal=*/false); if (validationLevel == SCPDriver::kFullyValidatedValue || validationLevel == SCPDriver::kMaybeValidNotCurrentValue) @@ -1685,7 +2020,7 @@ BallotProtocol::setConfirmCommit(SCPBallot const& c, SCPBallot const& h) emitCurrentStateStatement(); - mSlot.stopNomination(); + updateLeaderTimer(); mSlot.getSCPDriver().valueExternalized(mSlot.getSlotIndex(), mCommit->getBallot().value); @@ -1931,6 +2266,11 @@ BallotProtocol::setStateFromEnvelope(SCPEnvelopeWrapperPtr e) { mCommit = makeBallot(prep.nC, b.value); } + if (prep.nH) + { + // as when the node confirmed the ballot prepared + mValueOverride = mSlot.getSCPDriver().wrapValue(b.value); + } mPhase = SCP_PHASE_PREPARE; } break; @@ -1942,6 +2282,7 @@ BallotProtocol::setStateFromEnvelope(SCPEnvelopeWrapperPtr e) mPrepared = makeBallot(c.nPrepared, v); mHighBallot = makeBallot(c.nH, v); mCommit = makeBallot(c.nCommit, v); + mValueOverride = mSlot.getSCPDriver().wrapValue(v); mPhase = SCP_PHASE_CONFIRM; } break; @@ -2174,6 +2515,12 @@ BallotProtocol::getJsonInfo() ret["heard"] = mHeardFromQuorum; ret["ballot"] = ballotToStr(mCurrentBallot); ret["phase"] = phaseNames[mPhase]; + ret["round"] = mLeaderRound; + if (auto leader = mSlot.getSCPDriver().getRoundLeader( + mSlot.getSlotIndex(), std::max(mLeaderRound, 1))) + { + ret["leader"] = mSlot.getSCPDriver().toShortString(*leader); + } ret["state"] = getLocalState(); return ret; @@ -2387,7 +2734,10 @@ BallotProtocol::checkHeardFromQuorum() // if we transition from not heard -> heard, we start the timer mSlot.getSCPDriver().ballotDidHearFromQuorum( mSlot.getSlotIndex(), mCurrentBallot->getBallot()); - if (mPhase != SCP_PHASE_EXTERNALIZE) + // A ballot behind the current round is one the node already + // gave up on: the leader timer runs instead. + if (mPhase != SCP_PHASE_EXTERNALIZE && + mCurrentBallot->getBallot().counter >= mLeaderRound) { startBallotProtocolTimer(); } diff --git a/src/scp/BallotProtocol.h b/src/scp/BallotProtocol.h index 2ff2cb1207..e7126d1449 100644 --- a/src/scp/BallotProtocol.h +++ b/src/scp/BallotProtocol.h @@ -9,6 +9,7 @@ #include "util/GlobalChecks.h" #include #include +#include #include #include #include @@ -95,6 +96,31 @@ class BallotProtocol SCPEnvelopeWrapperPtr mLastEnvelopeEmit; // last envelope emitted by this node + // ** Leader rounds + // There is no nomination phase. A round is a ballot counter, and each + // round has one leader (SCPDriver::getRoundLeader). At counter r a node + // votes for the ballot `leader(r)` votes for at counter r, unless it + // already holds a value it must stick to (mValueOverride). + + // The round the local node is in: the highest counter it is willing to + // vote at. 0 until the node starts the slot or adopts a round 1 ballot. + // Invariant: mLeaderRound >= mCurrentBallot's counter (so it is + // UINT32_MAX once the node moved to the "infinite" counter of a committed + // ballot). + uint32 mLeaderRound; + // True once the application started the slot (startLeaderRounds). + bool mLeaderRoundsStarted; + // Builds the local node's own value when it leads a round. + ProposalValueSupplier mMakeValue; + // Round the leader timer is armed for (0 = not armed). + uint32 mLeaderTimerRound; + // True once the leader timer of mLeaderTimerRound got its one extension + // (the leader's proposal was pending in the application when it fired). + bool mLeaderTimerExtended; + // Guards against re-entering progressRound (the value supplier and + // bumpState both re-enter the slot). + bool mInProgressRound; + public: BallotProtocol(Slot& slot); @@ -106,10 +132,21 @@ class BallotProtocol bool self); void ballotProtocolTimerExpired(); - // abandon's current ballot, move to a new ballot - // at counter `n` (or, if n == 0, increment current counter) + void leaderTimerExpired(uint32 round); + // abandon's current ballot, move to round + // `n` (or, if n == 0, to the next round) bool abandonBallot(uint32 n); + // Called when the local node is ready to take part in consensus on this + // slot: enters round 1 (if not already further along). + bool startLeaderRounds(ProposalValueSupplier const& makeValue); + + uint32 + getLeaderRound() const + { + return mLeaderRound; + } + // bumps the ballot based on the local state and the value passed in: // in prepare phase, attempts to take value // otherwise, no-ops @@ -328,6 +365,21 @@ class BallotProtocol void stopBallotProtocolTimer(); void checkHeardFromQuorum(); + // Tries to bring the current ballot up to the current round: with the + // value the node must stick to, else as the round's leader (own value), + // else by adopting what the round's leader voted for. Arms the leader + // timer while the ballot is still behind the round. + // Returns true if the current ballot changed. + bool progressRound(); + // One attempt of the above; returns true if the current ballot changed. + bool progressRoundStep(); + // Returns the ballot `leader` voted for at exactly counter `round`, if + // its latest statement is at that counter. + std::optional getLeaderBallot(NodeID const& leader, + uint32 round) const; + void updateLeaderTimer(); + void armLeaderTimer(uint32 round); + SCPBallotWrapperUPtr makeBallot(SCPBallot const& b) const; SCPBallotWrapperUPtr makeBallot(uint32 c, Value const& v) const; diff --git a/src/scp/NominationProtocol.cpp b/src/scp/NominationProtocol.cpp deleted file mode 100644 index 015bd8f814..0000000000 --- a/src/scp/NominationProtocol.cpp +++ /dev/null @@ -1,926 +0,0 @@ -// Copyright 2014 Stellar Development Foundation and contributors. Licensed -// under the Apache License, Version 2.0. See the COPYING file at the root -// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 - -#include "NominationProtocol.h" - -#include "Slot.h" -#include "crypto/Hex.h" -#include "lib/json/json.h" -#include "scp/LocalNode.h" -#include "scp/QuorumSetUtils.h" -#include "util/GlobalChecks.h" -#include "util/Logging.h" -#include "util/XDROperators.h" -#include "xdrpp/marshal.h" -#include -#include -#include - -namespace stellar -{ -using namespace std::placeholders; - -NominationProtocol::NominationProtocol(Slot& slot) - : mSlot(slot), mRoundNumber(0), mNominationStarted(false), mTimerExpCount(0) -{ -} - -std::set -NominationProtocol::predictLeaders(Slot& slot, Value const& previousValue, - uint32_t rounds) -{ - NominationProtocol preview(slot); - preview.mPreviousValue = previousValue; - for (uint32_t i = 0; i < rounds; ++i) - { - ++preview.mRoundNumber; - preview.updateRoundLeaders(); - } - return preview.mRoundLeaders; -} - -bool -NominationProtocol::isNewerStatement(NodeID const& nodeID, - SCPNomination const& st) -{ - auto oldp = mLatestNominations.find(nodeID); - bool res = false; - - if (oldp == mLatestNominations.end()) - { - res = true; - } - else - { - res = isNewerStatement(oldp->second->getStatement().pledges.nominate(), - st); - } - return res; -} - -bool -NominationProtocol::isSubsetHelper(xdr::xvector const& p, - xdr::xvector const& v, bool& notEqual) -{ - bool res; - if (p.size() <= v.size()) - { - res = std::includes(v.begin(), v.end(), p.begin(), p.end()); - if (res) - { - notEqual = p.size() != v.size(); - } - else - { - notEqual = true; - } - } - else - { - notEqual = true; - res = false; - } - return res; -} - -SCPDriver::ValidationLevel -NominationProtocol::validateValue(Value const& v) -{ - ZoneScoped; - return mSlot.getSCPDriver().validateValue(mSlot.getSlotIndex(), v, true); -} - -ValueWrapperPtr -NominationProtocol::extractValidValue(Value const& value) -{ - ZoneScoped; - return mSlot.getSCPDriver().extractValidValue(mSlot.getSlotIndex(), value); -} - -bool -NominationProtocol::isNewerStatement(SCPNomination const& oldst, - SCPNomination const& st) -{ - bool res = false; - bool grows; - bool g = false; - - if (isSubsetHelper(oldst.votes, st.votes, g)) - { - grows = g; - if (isSubsetHelper(oldst.accepted, st.accepted, g)) - { - grows = grows || g; - res = grows; // true only if one of the sets grew - } - } - - return res; -} - -bool -NominationProtocol::isSane(SCPStatement const& st) -{ - auto const& nom = st.pledges.nominate(); - bool res = (nom.votes.size() + nom.accepted.size()) != 0; - - res = res && (std::adjacent_find( - nom.votes.begin(), nom.votes.end(), - [](stellar::Value const& l, stellar::Value const& r) { - return !(l < r); - }) == nom.votes.end()); - res = res && (std::adjacent_find( - nom.accepted.begin(), nom.accepted.end(), - [](stellar::Value const& l, stellar::Value const& r) { - return !(l < r); - }) == nom.accepted.end()); - - return res; -} - -// only called after a call to isNewerStatement so safe to replace the -// mLatestNomination -void -NominationProtocol::recordEnvelope(SCPEnvelopeWrapperPtr env) -{ - auto const& st = env->getStatement(); - auto oldp = mLatestNominations.find(st.nodeID); - if (oldp == mLatestNominations.end()) - { - mLatestNominations.insert(std::make_pair(st.nodeID, env)); - } - else - { - oldp->second = env; - } - mSlot.recordStatement(env->getStatement()); -} - -void -NominationProtocol::emitNomination() -{ - ZoneScoped; - SCPStatement st; - st.nodeID = mSlot.getLocalNode()->getNodeID(); - st.pledges.type(SCP_ST_NOMINATE); - auto& nom = st.pledges.nominate(); - - nom.quorumSetHash = mSlot.getLocalNode()->getQuorumSetHash(); - - for (auto const& v : mVotes) - { - nom.votes.emplace_back(v->getValue()); - } - for (auto const& a : mAccepted) - { - nom.accepted.emplace_back(a->getValue()); - } - - SCPEnvelope envelope = mSlot.createEnvelope(st); - - auto envW = mSlot.getSCPDriver().wrapEnvelope(envelope); - - if (mSlot.processEnvelope(envW, true) == SCP::EnvelopeState::VALID) - { - if (!mLastEnvelope || - isNewerStatement(mLastEnvelope->getStatement().pledges.nominate(), - st.pledges.nominate())) - { - mLastEnvelope = envW; - if (mSlot.isFullyValidated()) - { - mSlot.getSCPDriver().emitEnvelope(envelope); - } - } - } - else - { - // there is a bug in the application if it queued up - // a statement for itself that it considers invalid - throw std::runtime_error("moved to a bad state (nomination)"); - } -} - -#ifdef BUILD_TESTS -void -NominationProtocol::emitDelayedNomination() -{ - // While the deferred emit was pending, processEnvelope may already have - // emitted (and self-processed) a statement carrying the same votes and - // accepted values. Emitting it again would be rejected as not newer by - // our own processEnvelope, which emitNomination treats as a bad state. - SCPNomination nom; - for (auto const& v : mVotes) - { - nom.votes.emplace_back(v->getValue()); - } - for (auto const& a : mAccepted) - { - nom.accepted.emplace_back(a->getValue()); - } - if (!isNewerStatement(mSlot.getLocalNode()->getNodeID(), nom)) - { - CLOG_DEBUG(SCP, "NominationProtocol: delayed emit already sent"); - return; - } - emitNomination(); -} -#endif - -bool -NominationProtocol::acceptPredicate(Value const& v, SCPStatement const& st) -{ - auto const& nom = st.pledges.nominate(); - bool res; - res = (std::find(nom.accepted.begin(), nom.accepted.end(), v) != - nom.accepted.end()); - return res; -} - -void -NominationProtocol::applyAll(SCPNomination const& nom, - std::function processor) -{ - for (auto const& v : nom.votes) - { - processor(v); - } - // NB: "accepted" should be a subset of "votes", so this should no-op - for (auto const& a : nom.accepted) - { - if (std::find(nom.votes.begin(), nom.votes.end(), a) == nom.votes.end()) - { - processor(a); - } - } -} - -void -NominationProtocol::updateRoundLeaders() -{ - ZoneScoped; - SCPQuorumSet myQSet = mSlot.getLocalNode()->getQuorumSet(); - - auto localID = mSlot.getLocalNode()->getNodeID(); - normalizeQSet(myQSet, &localID); // excludes self - - // Count potential leaders: only include nodes with non-zero weight, - // as nodes with weight 0 can never be elected as round leaders. - size_t maxLeaderCount = 0; - if (mSlot.getSCPDriver().getNodeWeight(localID, myQSet, true) > 0) - { - ++maxLeaderCount; - } - LocalNode::forAllNodes(myQSet, [&](NodeID const& cur) { - if (mSlot.getSCPDriver().getNodeWeight(cur, myQSet, false) > 0) - { - ++maxLeaderCount; - } - return true; - }); - - // Cap the number of iterations to prevent infinite loops if something has - // gone wrong - int iterationsRemaining = 1000; - while (mRoundLeaders.size() < maxLeaderCount) - { - // initialize priority with value derived from self - std::set newRoundLeaders; - - newRoundLeaders.insert(localID); - uint64 topPriority = getNodePriority(localID, myQSet); - - LocalNode::forAllNodes(myQSet, [&](NodeID const& cur) { - uint64 w = getNodePriority(cur, myQSet); - if (w > topPriority) - { - topPriority = w; - newRoundLeaders.clear(); - } - if (w == topPriority && w > 0) - { - newRoundLeaders.insert(cur); - } - return true; - }); - - if (topPriority == 0) - { - // No one had priority, so all nodes would choose themselves - // resulting in a timeout. Clear newRoundLeaders, allowing the - // algorithm to fast timeout and try again. - newRoundLeaders.clear(); - } - - // expand mRoundLeaders with the newly computed leaders - auto oldSize = mRoundLeaders.size(); - mRoundLeaders.insert(newRoundLeaders.begin(), newRoundLeaders.end()); - if (oldSize != mRoundLeaders.size()) - { - if (Logging::logDebug("SCP")) - { - CLOG_DEBUG(SCP, "updateRoundLeaders: {} -> {}", oldSize, - mRoundLeaders.size()); - for (auto const& rl : mRoundLeaders) - { - CLOG_DEBUG(SCP, " leader {}", - mSlot.getSCPDriver().toShortString(rl)); - } - } - return; - } - else - { - mRoundNumber++; - CLOG_DEBUG(SCP, - "updateRoundLeaders: fast timeout (would no op) -> {}", - mRoundNumber); - } - - --iterationsRemaining; - if (iterationsRemaining <= 0) - { - throw std::runtime_error( - "updateRoundLeaders appears to be stuck in an infinite loop"); - } - } - CLOG_DEBUG(SCP, "updateRoundLeaders: nothing to do"); -} - -uint64 -NominationProtocol::hashNode(bool isPriority, NodeID const& nodeID) -{ - ZoneScoped; - dbgAssert(!mPreviousValue.empty()); - return mSlot.getSCPDriver().computeHashNode( - mSlot.getSlotIndex(), mPreviousValue, isPriority, mRoundNumber, nodeID); -} - -uint64 -NominationProtocol::hashValue(Value const& value) -{ - ZoneScoped; - dbgAssert(!mPreviousValue.empty()); - return mSlot.getSCPDriver().computeValueHash( - mSlot.getSlotIndex(), mPreviousValue, mRoundNumber, value); -} - -uint64 -NominationProtocol::getNodePriority(NodeID const& nodeID, - SCPQuorumSet const& qset) -{ - ZoneScoped; - uint64 res; - uint64 w = mSlot.getSCPDriver().getNodeWeight( - nodeID, qset, nodeID == mSlot.getLocalNode()->getNodeID()); - - // if w > 0; w is inclusive here as - // 0 <= hashNode <= UINT64_MAX - if (w > 0 && hashNode(false, nodeID) <= w) - { - res = hashNode(true, nodeID); - } - else - { - res = 0; - } - return res; -} - -ValueWrapperPtr -NominationProtocol::getNewValueFromNomination(SCPNomination const& nom) -{ - ZoneScoped; - // pick the highest value we don't have from the leader - // sorted using hashValue. - ValueWrapperPtr newVote; - uint64 newHash = 0; - bool foundValidValue = false; - - auto pickValue = [&](Value const& value) { - ValueWrapperPtr valueToNominate; - auto vl = validateValue(value); - if (vl >= SCPDriver::kStructurallyValidValue) - { - valueToNominate = mSlot.getSCPDriver().wrapValue(value); - } - else - { - valueToNominate = extractValidValue(value); - } - if (valueToNominate) - { - foundValidValue = true; - if (mVotes.find(valueToNominate) == mVotes.end()) - { - uint64 curHash = hashValue(valueToNominate->getValue()); - if (curHash >= newHash) - { - newHash = curHash; - newVote = valueToNominate; - } - } - } - }; - - for (auto const& val : nom.accepted) - { - pickValue(val); - } - - // Move on to votes if we have not found a valid accepted value - if (!foundValidValue) - { - for (auto const& val : nom.votes) - { - pickValue(val); - } - } - - return newVote; -} - -SCP::EnvelopeState -NominationProtocol::processEnvelope(SCPEnvelopeWrapperPtr envelope) -{ - ZoneScoped; - auto const& st = envelope->getStatement(); - auto const& nom = st.pledges.nominate(); - - if (!isNewerStatement(st.nodeID, nom)) - return SCP::EnvelopeState::INVALID; - - if (!isSane(st)) - { - CLOG_TRACE(SCP, "NominationProtocol: message didn't pass sanity check"); - return SCP::EnvelopeState::INVALID; - } - - recordEnvelope(envelope); - - // Ensure node doesn't vote on future slots - if (mNominationStarted) - { - // tracks if we should emit a new nomination message - bool modified = false; - bool newCandidates = false; - - // attempts to promote some of the votes to accepted - for (auto const& v : nom.votes) - { - auto vw = mSlot.getSCPDriver().wrapValue(v); - if (mAccepted.find(vw) != mAccepted.end()) - { // v is already accepted - continue; - } - if (mSlot.federatedAccept( - [&v](SCPStatement const& st) -> bool { - auto const& nom = st.pledges.nominate(); - bool res; - res = (std::find(nom.votes.begin(), nom.votes.end(), - v) != nom.votes.end()); - return res; - }, - std::bind(&NominationProtocol::acceptPredicate, v, _1), - mLatestNominations)) - { - auto vl = validateValue(v); - if (vl >= SCPDriver::kStructurallyValidValue) - { - mAccepted.emplace(vw); - mVotes.emplace(vw); - modified = true; - } - else - { - // the value made it pretty far: - // see if we can vote for a variation that - // we consider valid - auto toVote = extractValidValue(v); - if (toVote) - { - if (mVotes.emplace(toVote).second) - { - modified = true; - } - } - } - } - } - // attempts to promote accepted values to candidates - for (auto const& a : mAccepted) - { - if (mCandidates.find(a) != mCandidates.end()) - { - continue; - } - if (mSlot.federatedRatify( - std::bind(&NominationProtocol::acceptPredicate, - a->getValue(), _1), - mLatestNominations)) - { - mCandidates.emplace(a); - newCandidates = true; - // Stop the timer, as there's no need to continue nominating, - // per the whitepaper: - // "As soon as `v` has a candidate value, however, it must cease - // voting to nominate `x` for any new values `x`" - mSlot.getSCPDriver().stopTimer(mSlot.getSlotIndex(), - Slot::NOMINATION_TIMER); - } - } - - // only take round leader votes if we're still looking for - // candidates - if (mCandidates.empty() && - mRoundLeaders.find(st.nodeID) != mRoundLeaders.end()) - { - auto newVote = getNewValueFromNomination(nom); - if (newVote) - { - mVotes.emplace(newVote); - modified = true; - mSlot.getSCPDriver().nominatingValue(mSlot.getSlotIndex(), - newVote->getValue()); - } - } - - if (modified) - { - emitNomination(); - } - - if (newCandidates) - { - mLatestCompositeCandidate = mSlot.getSCPDriver().combineCandidates( - mSlot.getSlotIndex(), mCandidates); - - mSlot.getSCPDriver().updatedCandidateValue( - mSlot.getSlotIndex(), mLatestCompositeCandidate->getValue()); - - mSlot.bumpState(mLatestCompositeCandidate->getValue(), false); - } - } - return SCP::EnvelopeState::VALID; -} - -std::vector -NominationProtocol::getStatementValues(SCPStatement const& st) -{ - std::vector res; - applyAll(st.pledges.nominate(), - [&](Value const& v) { res.emplace_back(v); }); - return res; -} - -void -NominationProtocol::stripUpgrades(ValueWrapperPtr& value) const -{ - ZoneScoped; - ValueWrapperPtr const newValue = - mSlot.getSCPDriver().stripAllUpgrades(value->getValue()); - if (newValue != nullptr && newValue->getValue() != value->getValue()) - { - CLOG_WARNING(SCP, - "Due to high timeouts, stripping upgrades from " - "nominated value: {} -> {}", - mSlot.getSCP().getValueString(value->getValue()), - mSlot.getSCP().getValueString(newValue->getValue())); - value = newValue; - } -} - -// attempts to nominate a value for consensus -bool -NominationProtocol::nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout) -{ - ZoneScoped; - - // No need to continue nominating, as per the whitepaper: - // "As soon as `v` has a candidate value, however, it must cease - // voting to nominate `x` for any new values `x`" - if (!mCandidates.empty()) - { - CLOG_DEBUG(SCP, "Skip nomination round {}, already have a candidate", - mRoundNumber); - return false; - } - - CLOG_DEBUG(SCP, "NominationProtocol::nominate ({})", mRoundNumber); - - bool updated = false; - - if (timedout) - { - mTimerExpCount++; - } - - if (timedout && !mNominationStarted) - { - CLOG_DEBUG(SCP, "NominationProtocol::nominate (TIMED OUT)"); - return false; - } - - mNominationStarted = true; - - mPreviousValue = previousValue; - - mRoundNumber++; - updateRoundLeaders(); - - std::chrono::milliseconds timeout = mSlot.getSCPDriver().computeTimeout( - mRoundNumber, /*isNomination=*/true); - - // add a few more values from other leaders - for (auto const& leader : mRoundLeaders) - { - auto it = mLatestNominations.find(leader); - if (it != mLatestNominations.end()) - { - auto lnmV = getNewValueFromNomination( - it->second->getStatement().pledges.nominate()); - if (lnmV) - { - mVotes.insert(lnmV); - updated = true; - mSlot.getSCPDriver().nominatingValue(mSlot.getSlotIndex(), - lnmV->getValue()); - } - } - } - - // Check if we are a leader for this round - if (mRoundLeaders.find(mSlot.getLocalNode()->getNodeID()) != - mRoundLeaders.end()) - { - // Check whether we've exceeded the upgrade timeout limit for this slot - bool const overUpgradeTimeoutLimit = - mTimerExpCount >= - mSlot.getSCPDriver().getUpgradeNominationTimeoutLimit(); - - bool shouldVoteForValue = false; - bool shouldStripUpgrades = false; - // Add our value if we haven't added any votes yet. - if (mVotes.empty()) - { - shouldVoteForValue = true; - } - - if (overUpgradeTimeoutLimit) - { - // We've exceeded the upgrade timeout limit. First, check whether - // all of our votes have upgrades. We only want to add a new value - // with stripped upgrades if there's a chance that it will help - // alleviate timeouts due to contentious upgrades. If we're already - // voting for a value without upgrades, then adding another is - // unlikely to help, as we are likely in one of the two following - // scenarios: - // 1. The current timeout situation is unrelated to upgrades - // 2. We just happened to add a value without upgrades earlier in - // this function - // In either case, adding a new value without upgrades will not - // help, and may make things worse by putting more strain on the - // network. - bool const allVotesHaveUpgrades = std::all_of( - mVotes.begin(), mVotes.end(), [&](ValueWrapperPtr const& v) { - return mSlot.getSCPDriver().hasUpgrades(v->getValue()); - }); - - if (allVotesHaveUpgrades) - { - // All votes have upgrades, so strip upgrades from `value` and - // vote for it. - shouldStripUpgrades = true; - shouldVoteForValue = true; - } - } - - if (shouldVoteForValue) - { - auto value = makeValue(); - // Making the value can make pending envelopes ready and reenter - // SCP, potentially externalizing this slot or creating a candidate. - // Do not vote or rearm its timer after nomination has stopped. - if (!mNominationStarted || !mCandidates.empty()) - { - return false; - } - if (value) - { - if (shouldStripUpgrades) - { - stripUpgrades(value); - } - auto ins = mVotes.insert(value); - if (ins.second) - { - updated = true; - mSlot.getSCPDriver().nominatingValue(mSlot.getSlotIndex(), - value->getValue()); - } - } - } - } - - std::shared_ptr slot = mSlot.shared_from_this(); - mSlot.getSCPDriver().setupTimer( - mSlot.getSlotIndex(), Slot::NOMINATION_TIMER, timeout, - [slot, makeValue, previousValue]() { - slot->nominate(makeValue, previousValue, true); - }); - -#ifdef BUILD_TESTS - // If a nomination-emit delay is configured, defer the emit by - // arming NOMINATION_EMIT_TIMER instead of broadcasting now. A subsequent - // nominate() will replace the timer so the eventual emit reflects the - // latest state. Skip the deferral once the SCP nomination timeout has - // fired (`timedout`) so retries don't compound latency, and cancel any - // still-armed timer in that case. - if (updated) - { - auto delay = mSlot.getSCPDriver().getNominationEmitDelayForTesting(); - if (delay > std::chrono::milliseconds::zero() && !timedout) - { - mSlot.getSCPDriver().setupTimer( - mSlot.getSlotIndex(), Slot::NOMINATION_EMIT_TIMER, delay, - [slot, this]() { emitDelayedNomination(); }); - return updated; - } - if (delay > std::chrono::milliseconds::zero()) - { - mSlot.getSCPDriver().stopTimer(mSlot.getSlotIndex(), - Slot::NOMINATION_EMIT_TIMER); - } - } -#endif - - if (updated) - { - emitNomination(); - } - else - { - CLOG_DEBUG(SCP, "NominationProtocol::nominate (SKIPPED)"); - } - - return updated; -} - -void -NominationProtocol::stopNomination() -{ - mNominationStarted = false; -} - -std::set const& -NominationProtocol::getLeaders() const -{ - return mRoundLeaders; -} - -Json::Value -NominationProtocol::getJsonInfo() -{ - Json::Value ret; - ret["roundnumber"] = mRoundNumber; - ret["started"] = mNominationStarted; - - int counter = 0; - for (auto const& v : mVotes) - { - ret["X"][counter] = mSlot.getSCP().getValueString(v->getValue()); - counter++; - } - - counter = 0; - for (auto const& v : mAccepted) - { - ret["Y"][counter] = mSlot.getSCP().getValueString(v->getValue()); - counter++; - } - - counter = 0; - for (auto const& v : mCandidates) - { - ret["Z"][counter] = mSlot.getSCP().getValueString(v->getValue()); - counter++; - } - - return ret; -} - -SCP::QuorumInfoNodeState -NominationProtocol::getState(NodeID const& n, bool selfAlreadyMovedOn) -{ - auto state = SCP::QuorumInfoNodeState::AGREE; - if (n == mSlot.getLocalNode()->getNodeID()) - { - // always mark myself as AGREE. - return state; - } - - bool enoughTimeHasPassed = - mTimerExpCount >= Slot::NUM_TIMEOUTS_THRESHOLD_FOR_REPORTING || - - selfAlreadyMovedOn; - - auto it = mLatestNominations.find(n); - if (it == mLatestNominations.end()) - { - if (enoughTimeHasPassed) - { - state = SCP::QuorumInfoNodeState::MISSING; - } - else - { - // Not enough time has passed to start calling this node missing. - state = SCP::QuorumInfoNodeState::NO_INFO; - } - } - else - { - if (enoughTimeHasPassed && mLastEnvelope) - { - // Enough time has passed & we have some information on this node. - // We'll examine the information to determine the state of this - // node. - auto const& other = - it->second->getStatement().pledges.nominate().accepted; - auto const& mine = - mLastEnvelope->getStatement().pledges.nominate().accepted; - - bool notEq, isSubset = isSubsetHelper(other, mine, notEq); - if (isSubset) - { - if (notEq) - { - // If there's some values that I've accepted that - // they haven't, they are slow. - state = SCP::QuorumInfoNodeState::DELAYED; - } - } - else - { - // If they have accepted a value that I haven't, - // it's likely that I'll never accept it. - state = SCP::QuorumInfoNodeState::DISAGREE; - } - } - } - return state; -} - -void -NominationProtocol::setStateFromEnvelope(SCPEnvelopeWrapperPtr e) -{ - if (mNominationStarted) - { - throw std::runtime_error( - "Cannot set state after nomination is started"); - } - recordEnvelope(e); - auto const& nom = e->getStatement().pledges.nominate(); - for (auto const& a : nom.accepted) - { - mAccepted.emplace(mSlot.getSCPDriver().wrapValue(a)); - } - for (auto const& v : nom.votes) - { - mVotes.emplace(mSlot.getSCPDriver().wrapValue(v)); - } - - mLastEnvelope = e; -} - -bool -NominationProtocol::processCurrentState( - std::function const& f, bool forceSelf) const -{ - for (auto const& n : mLatestNominations) - { - // only return messages for self if the slot is fully validated - if (forceSelf || !(n.first == mSlot.getSCP().getLocalNodeID()) || - mSlot.isFullyValidated()) - { - if (!f(n.second->getEnvelope())) - { - return false; - } - } - } - return true; -} - -SCPEnvelope const* -NominationProtocol::getLatestMessage(NodeID const& id) const -{ - auto it = mLatestNominations.find(id); - if (it != mLatestNominations.end()) - { - return &it->second->getEnvelope(); - } - return nullptr; -} -} diff --git a/src/scp/NominationProtocol.h b/src/scp/NominationProtocol.h deleted file mode 100644 index cf0e3656e1..0000000000 --- a/src/scp/NominationProtocol.h +++ /dev/null @@ -1,152 +0,0 @@ -// Copyright 2014 Stellar Development Foundation and contributors. Licensed -// under the Apache License, Version 2.0. See the COPYING file at the root -// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0 - -#pragma once - -#include "lib/json/json-forwards.h" -#include "scp/SCP.h" -#include -#include -#include -#include -#include - -namespace stellar -{ -class NominationProtocol -{ - protected: - Slot& mSlot; - - int32 mRoundNumber; - - ValueWrapperPtrSet mVotes; // X - ValueWrapperPtrSet mAccepted; // Y - ValueWrapperPtrSet mCandidates; // Z - std::map mLatestNominations; // N - - SCPEnvelopeWrapperPtr mLastEnvelope; // last envelope emitted by this node - - // nodes from quorum set that have the highest priority this round - std::set mRoundLeaders; - - // true if 'nominate' was called - bool mNominationStarted; - - // the latest (if any) candidate value - ValueWrapperPtr mLatestCompositeCandidate; - - // the value from the previous slot - Value mPreviousValue; - - bool isNewerStatement(NodeID const& nodeID, SCPNomination const& st); - - // returns true if 'p' is a subset of 'v' - // also sets 'notEqual' if p and v differ - // note: p and v must be sorted - static bool isSubsetHelper(xdr::xvector const& p, - xdr::xvector const& v, bool& notEqual); - - SCPDriver::ValidationLevel validateValue(Value const& v); - ValueWrapperPtr extractValidValue(Value const& value); - - bool isSane(SCPStatement const& st); - - void recordEnvelope(SCPEnvelopeWrapperPtr env); - - void emitNomination(); - -#ifdef BUILD_TESTS - // Fires the deferred emit armed by nominate() when a nomination-emit - // delay is configured. Skips the emit if this node already sent a - // statement at least as recent in the meantime. - void emitDelayedNomination(); -#endif - - // returns true if v is in the accepted list from the statement - static bool acceptPredicate(Value const& v, SCPStatement const& st); - - // applies 'processor' to all values from the passed in nomination - static void applyAll(SCPNomination const& nom, - std::function processor); - - // updates the set of nodes that have priority over the others - void updateRoundLeaders(); - - // computes Gi(isPriority?P:N, prevValue, mRoundNumber, nodeID) - // from the paper - uint64 hashNode(bool isPriority, NodeID const& nodeID); - - // computes Gi(K, prevValue, mRoundNumber, value) - uint64 hashValue(Value const& value); - - uint64 getNodePriority(NodeID const& nodeID, SCPQuorumSet const& qset); - - // returns the highest value that we don't have yet, that we should - // vote for, extracted from a nomination. - // returns nullptr if no new value was found - ValueWrapperPtr getNewValueFromNomination(SCPNomination const& nom); - - public: - static bool isNewerStatement(SCPNomination const& oldst, - SCPNomination const& st); - - NominationProtocol(Slot& slot); - - // Preview the cumulative leaders for the first `rounds` nomination calls. - // Uses the same fast-forward/tie handling as live nomination, without - // starting nomination, arming timers, or emitting statements. - static std::set - predictLeaders(Slot& slot, Value const& previousValue, uint32_t rounds); - - SCP::EnvelopeState processEnvelope(SCPEnvelopeWrapperPtr envelope); - - static std::vector getStatementValues(SCPStatement const& st); - - // attempts to nominate a value for consensus - bool nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout); - - // stops the nomination protocol - void stopNomination(); - - // return the current leaders - std::set const& getLeaders() const; - - ValueWrapperPtr const& - getLatestCompositeCandidate() const - { - return mLatestCompositeCandidate; - } - - Json::Value getJsonInfo(); - - SCP::QuorumInfoNodeState getState(NodeID const& node, - bool selfAlreadyMovedOn); - - SCPEnvelope const* - getLastMessageSend() const - { - return mLastEnvelope ? &mLastEnvelope->getEnvelope() : nullptr; - } - - void setStateFromEnvelope(SCPEnvelopeWrapperPtr e); - - bool processCurrentState(std::function const& f, - bool forceSelf) const; - - // returns the latest message from a node - // or nullptr if not found - SCPEnvelope const* getLatestMessage(NodeID const& id) const; - - private: - // The number of times the timer has expired - // Used for the quorum endpoint. - uint32_t mTimerExpCount; - - // Strip any upgrades that `value` may have, modifying it in place. Does - // nothing if `value` has no upgrades. - void stripUpgrades(ValueWrapperPtr& value) const; -}; -} diff --git a/src/scp/SCP.cpp b/src/scp/SCP.cpp index 8afbc3a6b3..f8f5641f7c 100644 --- a/src/scp/SCP.cpp +++ b/src/scp/SCP.cpp @@ -34,29 +34,17 @@ SCP::receiveEnvelope(SCPEnvelopeWrapperPtr envelope) } bool -SCP::nominate(uint64 slotIndex, NominationValueSupplier const& makeValue, - Value const& previousValue) +SCP::startSlot(uint64 slotIndex, ProposalValueSupplier const& makeValue) { dbgAssert(isValidator()); - return getSlot(slotIndex, true)->nominate(makeValue, previousValue, false); + return getSlot(slotIndex, true)->startLeaderRounds(makeValue); } -std::set -SCP::predictNominationLeaders(uint64 slotIndex, Value const& previousValue, - uint32_t rounds) -{ - Slot preview(slotIndex, *this); - return NominationProtocol::predictLeaders(preview, previousValue, rounds); -} - -void -SCP::stopNomination(uint64 slotIndex) +uint32 +SCP::getLeaderRound(uint64 slotIndex) { - auto s = getSlot(slotIndex, false); - if (s) - { - s->stopNomination(); - } + auto slot = getSlot(slotIndex, false); + return slot ? slot->getLeaderRound() : 0; } void @@ -287,17 +275,6 @@ SCP::getMissingNodes(NodeID const& id, uint64 index) return ret; } -std::set -SCP::getNominationLeaders(uint64 slotIndex) -{ - auto slot = getSlot(slotIndex, false); - if (slot) - { - return slot->getNominationLeaders(); - } - return {}; -} - bool SCP::isValidator() { @@ -433,8 +410,7 @@ SCP::getLatestMessage(NodeID const& id) } bool -SCP::isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) +SCP::isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt) { if (oldSt.slotIndex != newSt.slotIndex || !(oldSt.nodeID == newSt.nodeID)) { @@ -444,7 +420,7 @@ SCP::isNewerNominationOrBallotSt(SCPStatement const& oldSt, auto slot = getSlot(oldSt.slotIndex, false); if (slot) { - return slot->isNewerNominationOrBallotSt(oldSt, newSt); + return slot->isNewerStatement(oldSt, newSt); } return false; diff --git a/src/scp/SCP.h b/src/scp/SCP.h index e49e3535de..2fe4fbd7ba 100644 --- a/src/scp/SCP.h +++ b/src/scp/SCP.h @@ -51,13 +51,16 @@ class SCP // invokes the appropriate methods EnvelopeState receiveEnvelope(SCPEnvelopeWrapperPtr envelope); - // Start nomination for slotIndex, constructing a local value only if needed - // previousValue is the value from slotIndex-1 - bool nominate(uint64 slotIndex, NominationValueSupplier const& makeValue, - Value const& previousValue); + // Starts consensus on slotIndex for the local node (its trigger fired). + // There is no nomination phase: each round (ballot counter) has one leader + // given by SCPDriver::getRoundLeader, and nodes vote for the leader's + // ballot. `makeValue` is invoked only if the local node becomes the leader + // of a round and has no value to put forward. + bool startSlot(uint64 slotIndex, ProposalValueSupplier const& makeValue); - // stops nomination for a slot - void stopNomination(uint64 slotIndex); + // The round the local node is in for slotIndex (0 if it has not started + // the slot or does not know it). + uint32 getLeaderRound(uint64 slotIndex); // Local QuorumSet interface (can be dynamically updated) void updateLocalQuorumSet(SCPQuorumSet const& qSet); @@ -90,14 +93,6 @@ class SCP // Get a list of nodes from id's quorum set that are missing in consensus std::set getMissingNodes(NodeID const& id, uint64 index = 0); - // returns the current nomination leaders for the given slot - std::set getNominationLeaders(uint64 slotIndex); - - // A read-only preview; does not create or modify a live SCP slot. - std::set predictNominationLeaders(uint64 slotIndex, - Value const& previousValue, - uint32_t rounds); - // Purges all data relative to slots that fall outside the range // [minSlotIndex, maxSlotIndex]. Either bound may be nullopt to skip // that direction. The slotToKeep slot is never purged. @@ -148,8 +143,8 @@ class SCP // or nullptr if not found SCPEnvelope const* getLatestMessage(NodeID const& id); - bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt); + // returns true if newSt is newer than oldSt (same node, same slot) + bool isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt); // returns messages that contributed to externalizing the slot // (or empty if the slot didn't externalize) diff --git a/src/scp/SCPDriver.cpp b/src/scp/SCPDriver.cpp index 9e54c07778..4f4137e518 100644 --- a/src/scp/SCPDriver.cpp +++ b/src/scp/SCPDriver.cpp @@ -10,28 +10,11 @@ #include "crypto/KeyUtils.h" #include "crypto/SecretKey.h" #include "util/GlobalChecks.h" -#include "util/numeric.h" #include "xdrpp/marshal.h" namespace stellar { -namespace -{ -uint64 -computeWeight(uint64 m, uint64 total, uint64 threshold) -{ - uint64 res; - releaseAssert(threshold <= total); - // Since threshold <= total, calculating res=m*threshold/total will always - // produce res <= m, and we do not need to handle the possibility of this - // call returning false (indicating overflow). - bool noOverflow = bigDivideUnsigned(res, m, threshold, total, ROUND_UP); - releaseAssert(noOverflow); - return res; -} -} // namespace - bool WrappedValuePtrComparator::operator()(ValueWrapperPtr const& l, ValueWrapperPtr const& r) const @@ -90,95 +73,4 @@ SCPDriver::toShortString(NodeID const& pk) const return KeyUtils::toShortString(pk); } -// values used to switch hash function between priority and neighborhood checks -static uint32 const hash_N = 1; -static uint32 const hash_P = 2; -static uint32 const hash_K = 3; - -uint64 -SCPDriver::hashHelper( - uint64 slotIndex, Value const& prev, - std::function>&)> extra) -{ - std::vector> vals; - vals.emplace_back(xdr::xdr_to_opaque(slotIndex)); - vals.emplace_back(xdr::xdr_to_opaque(prev)); - extra(vals); - Hash t = getHashOf(vals); - uint64 res = 0; - for (size_t i = 0; i < sizeof(res); i++) - { - res = (res << 8) | t[i]; - } - return res; -} - -uint64 -SCPDriver::computeHashNode(uint64 slotIndex, Value const& prev, bool isPriority, - int32_t roundNumber, NodeID const& nodeID) -{ -#ifdef BUILD_TESTS - if (mPriorityLookupForTesting) - { - return isPriority ? mPriorityLookupForTesting(nodeID) : 0; - } -#endif - return hashHelper( - slotIndex, prev, [&](std::vector>& vals) { - vals.emplace_back(xdr::xdr_to_opaque(isPriority ? hash_P : hash_N)); - vals.emplace_back(xdr::xdr_to_opaque(roundNumber)); - vals.emplace_back(xdr::xdr_to_opaque(nodeID)); - }); -} - -uint64 -SCPDriver::computeValueHash(uint64 slotIndex, Value const& prev, - int32_t roundNumber, Value const& value) -{ - return hashHelper(slotIndex, prev, - [&](std::vector>& vals) { - vals.emplace_back(xdr::xdr_to_opaque(hash_K)); - vals.emplace_back(xdr::xdr_to_opaque(roundNumber)); - vals.emplace_back(xdr::xdr_to_opaque(value)); - }); -} - -// if a validator is repeated multiple times its weight is only the -// weight of the first occurrence -uint64 -SCPDriver::getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const -{ - if (isLocalNode) - { - // local node is in all quorum sets - return UINT64_MAX; - } - - uint64 n = qset.threshold; - uint64 d = qset.innerSets.size() + qset.validators.size(); - uint64 res; - - for (auto const& qsetNode : qset.validators) - { - if (qsetNode == nodeID) - { - res = computeWeight(UINT64_MAX, d, n); - return res; - } - } - - for (auto const& q : qset.innerSets) - { - uint64 leafW = SCPDriver::getNodeWeight(nodeID, q, isLocalNode); - if (leafW) - { - res = computeWeight(leafW, d, n); - return res; - } - } - - return 0; -} - } diff --git a/src/scp/SCPDriver.h b/src/scp/SCPDriver.h index 19fcbfc5ec..a41af7d781 100644 --- a/src/scp/SCPDriver.h +++ b/src/scp/SCPDriver.h @@ -44,10 +44,10 @@ class ValueWrapper : public NonMovableOrCopyable typedef std::shared_ptr SCPQuorumSetPtr; typedef std::shared_ptr ValueWrapperPtr; -// Called only when the local node is a nomination leader and needs its own -// value. May return nullptr if the application can no longer propose for the -// slot. The callback must remain valid across nomination timeouts. -using NominationValueSupplier = std::function; +// Called only when the local node is the leader of a round and needs to build +// its own value. May return nullptr if the application can no longer propose +// for the slot. The callback must remain valid for the lifetime of the slot. +using ProposalValueSupplier = std::function; class WrappedValuePtrComparator { @@ -155,7 +155,10 @@ class SCPDriver // be validated due to parallel downloading (e.g. it's for LCL+1 but the // node is still downloading the tx set), then kStructurallyValidValue // should be used. - // validation can be *more* restrictive during nomination as needed + // validation is *more* restrictive when `proposal` is set: that mode is + // used before the local node adopts (votes for) a round leader's value, + // and for the leader's own value. It is where the application enforces + // what it is willing to propose (e.g. only upgrades it has armed). // NB: validation levels are ordered enum ValidationLevel { @@ -172,22 +175,11 @@ class SCPDriver kFullyValidatedValue = 3 }; virtual ValidationLevel - validateValue(uint64 slotIndex, Value const& value, bool nomination) const + validateValue(uint64 slotIndex, Value const& value, bool proposal) const { return kMaybeValidNotCurrentValue; } - // `extractValidValue` transforms the value, if possible to a different - // value that the local node would agree to (fully validated). - // This is used during nomination when encountering an invalid value (ie - // validateValue did not return `kFullyValidatedValue` for this value). - // returning nullptr means no valid value could be extracted - virtual ValueWrapperPtr - extractValidValue(uint64 slotIndex, Value const& value) - { - return nullptr; - } - // Helper function to craft an empty-tx-set value from a Value. virtual Value makeEmptyTxSetValueFromValue(Value const& v) const = 0; @@ -215,33 +207,23 @@ class SCPDriver virtual Hash getHashOf(std::vector> const& vals) const = 0; - // `computeHashNode` is used by the nomination protocol to - // randomize the order of messages between nodes. - virtual uint64 computeHashNode(uint64 slotIndex, Value const& prev, - bool isPriority, int32_t roundNumber, - NodeID const& nodeID); - - // `computeValueHash` is used by the nomination protocol to - // randomize the relative order between values. - virtual uint64 computeValueHash(uint64 slotIndex, Value const& prev, - int32_t roundNumber, Value const& value); - - // `combineCandidates` computes the composite value based off a list - // of candidate values. - virtual ValueWrapperPtr - combineCandidates(uint64 slotIndex, - ValueWrapperPtrSet const& candidates) = 0; - - // Checks whether `v` contains upgrades - virtual bool hasUpgrades(Value const& v) = 0; - - // `stripAllUpgrades` returns a new value with all upgrades removed - virtual ValueWrapperPtr stripAllUpgrades(Value const& v) = 0; - - // Returns the maximum number of nomination timeouts permitted per slot for - // the currently set upgrade. Defaults to the maximum uint32_t value - // (effectively unlimited) if the upgrade does not specify a limit. - virtual uint32_t getUpgradeNominationTimeoutLimit() const = 0; + // `getRoundLeader` returns the leader of round `round` (1-based, a round + // is a ballot counter) for `slotIndex`. Every node that takes part in + // consensus on the slot must compute the same answer. Returns nullopt when + // the schedule is not known for that slot (e.g. the slot is not the next + // one to close), in which case the local node neither proposes nor adopts. + virtual std::optional getRoundLeader(uint64 slotIndex, + uint32 round) = 0; + + // `isProposalPending` returns true if the application holds a statement + // of `leader` for `slotIndex` that it has not handed to SCP yet (e.g. it + // is still fetching the transaction set of its value). SCP then gives a + // round leader one more timeout before moving to the next round. + virtual bool + isProposalPending(uint64 slotIndex, NodeID const& leader) const + { + return false; + } // `setupTimer`: requests to trigger 'cb' after timeout // if cb is nullptr, the timer is cancelled @@ -253,22 +235,7 @@ class SCPDriver // `computeTimeout` computes a timeout given a round number // it should be sufficiently large such that nodes in a // quorum can exchange 4 messages - virtual std::chrono::milliseconds computeTimeout(uint32 roundNumber, - bool isNomination) = 0; - -#ifdef BUILD_TESTS - virtual std::chrono::milliseconds - getNominationEmitDelayForTesting() const - { - return std::chrono::milliseconds::zero(); - } -#endif - - // returns the weight of the node within the qset normalized between - // 0-UINT64_MAX. If `nodeID` is the local node, then set `isLocalNode` to - // `true`. - virtual uint64 getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const; + virtual std::chrono::milliseconds computeTimeout(uint32 roundNumber) = 0; // Inform about events happening within the consensus algorithm. @@ -284,21 +251,20 @@ class SCPDriver { } - // ``nominatingValue`` is called every time the local instance nominates - // a new value. + // the following methods are used for monitoring of the SCP subsystem + // most implementation don't really need to do anything with these + + // `proposedValue` is called when the local node, as the leader of + // `round`, puts `value` forward. virtual void - nominatingValue(uint64 slotIndex, Value const& value) + proposedValue(uint64 slotIndex, uint32 round, Value const& value) { } - // the following methods are used for monitoring of the SCP subsystem - // most implementation don't really need to do anything with these - - // `updatedCandidateValue` is called every time a new candidate value - // is included in the candidate set, the value passed in is - // a composite value + // `enteredRound` is called when the local node moves to `round` for + // `slotIndex`. `timedOut` is true when a timer expiry caused the move. virtual void - updatedCandidateValue(uint64 slotIndex, Value const& value) + enteredRound(uint64 slotIndex, uint32 round, bool timedOut) { } @@ -347,19 +313,5 @@ class SCPDriver measureAndRecordBallotBlockedOnTxSet(uint64 slotIndex, Value const& value) { } - -#ifdef BUILD_TESTS - std::function mPriorityLookupForTesting; - void - setPriorityLookup(std::function const& f) - { - mPriorityLookupForTesting = f; - } -#endif - - private: - uint64 - hashHelper(uint64 slotIndex, Value const& prev, - std::function>&)> extra); }; } diff --git a/src/scp/Slot.cpp b/src/scp/Slot.cpp index 4971fc2e27..37ce296ef7 100644 --- a/src/scp/Slot.cpp +++ b/src/scp/Slot.cpp @@ -24,31 +24,18 @@ Slot::Slot(uint64 slotIndex, SCP& scp) : mSlotIndex(slotIndex) , mSCP(scp) , mBallotProtocol(*this) - , mNominationProtocol(*this) , mFullyValidated(scp.getLocalNode()->isValidator()) , mGotVBlocking(false) { } -ValueWrapperPtr const& -Slot::getLatestCompositeCandidate() -{ - return mNominationProtocol.getLatestCompositeCandidate(); -} - std::vector Slot::getLatestMessagesSend() const { std::vector res; if (mFullyValidated) { - SCPEnvelope const* e; - e = mNominationProtocol.getLastMessageSend(); - if (e) - { - res.emplace_back(*e); - } - e = mBallotProtocol.getLastMessageSend(); + SCPEnvelope const* e = mBallotProtocol.getLastMessageSend(); if (e) { res.emplace_back(*e); @@ -68,12 +55,11 @@ Slot::setStateFromEnvelope(SCPEnvelopeWrapperPtr env) if (e.statement.pledges.type() == SCPStatementType::SCP_ST_NOMINATE) { - mNominationProtocol.setStateFromEnvelope(env); - } - else - { - mBallotProtocol.setStateFromEnvelope(env); + // Nomination no longer exists: ignore state saved by a build that + // still had it. + return; } + mBallotProtocol.setStateFromEnvelope(env); if (!prev) { @@ -91,53 +77,24 @@ void Slot::processCurrentState(std::function const& f, bool forceSelf) const { - mNominationProtocol.processCurrentState(f, forceSelf) && - mBallotProtocol.processCurrentState(f, forceSelf); + mBallotProtocol.processCurrentState(f, forceSelf); } SCPEnvelope const* Slot::getLatestMessage(NodeID const& id) const { - auto m = mBallotProtocol.getLatestMessage(id); - if (m == nullptr) - { - m = mNominationProtocol.getLatestMessage(id); - } - return m; + return mBallotProtocol.getLatestMessage(id); } bool -Slot::isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt) +Slot::isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt) { - bool oldNomination = - oldSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE; - bool newNomination = - newSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE; - - if (oldNomination != newNomination) + if (oldSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE || + newSt.pledges.type() == SCPStatementType::SCP_ST_NOMINATE) { return false; } - - bool replace = false; - if (oldNomination) - { - if (NominationProtocol::isNewerStatement(oldSt.pledges.nominate(), - newSt.pledges.nominate())) - { - replace = true; - } - } - else - { - if (BallotProtocol::isNewerStatement(oldSt, newSt)) - { - replace = true; - } - } - - return replace; + return BallotProtocol::isNewerStatement(oldSt, newSt); } std::vector @@ -173,12 +130,11 @@ Slot::processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self) if (st.pledges.type() == SCPStatementType::SCP_ST_NOMINATE) { - res = mNominationProtocol.processEnvelope(envelope); - } - else - { - res = mBallotProtocol.processEnvelope(envelope, self); + // There is no nomination phase: the round leader's ballot is the + // proposal. Nomination statements are dropped. + return SCP::EnvelopeState::INVALID; } + res = mBallotProtocol.processEnvelope(envelope, self); if (!prev && res == SCP::VALID) { @@ -211,12 +167,6 @@ Slot::processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self) return res; } -bool -Slot::abandonBallot() -{ - return mBallotProtocol.abandonBallot(0); -} - bool Slot::bumpState(Value const& value, bool force) { @@ -225,22 +175,15 @@ Slot::bumpState(Value const& value, bool force) } bool -Slot::nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout) +Slot::startLeaderRounds(ProposalValueSupplier const& makeValue) { - return mNominationProtocol.nominate(makeValue, previousValue, timedout); + return mBallotProtocol.startLeaderRounds(makeValue); } -void -Slot::stopNomination() +uint32 +Slot::getLeaderRound() const { - mNominationProtocol.stopNomination(); -} - -std::set -Slot::getNominationLeaders() const -{ - return mNominationProtocol.getLeaders(); + return mBallotProtocol.getLeaderRound(); } bool @@ -300,7 +243,9 @@ Slot::getStatementValues(SCPStatement const& st) std::vector res; if (st.pledges.type() == SCP_ST_NOMINATE) { - res = NominationProtocol::getStatementValues(st); + auto const& nom = st.pledges.nominate(); + res.insert(res.end(), nom.votes.begin(), nom.votes.end()); + res.insert(res.end(), nom.accepted.begin(), nom.accepted.end()); } else { @@ -378,7 +323,6 @@ Slot::getJsonInfo(bool fullKeys) } ret["validated"] = mFullyValidated; - ret["nomination"] = mNominationProtocol.getJsonInfo(); ret["ballotProtocol"] = mBallotProtocol.getJsonInfo(); return ret; @@ -387,12 +331,7 @@ Slot::getJsonInfo(bool fullKeys) SCP::QuorumInfoNodeState Slot::getState(NodeID const& node, bool selfAlreadyMovedOn) { - auto b = mBallotProtocol.getState(node, selfAlreadyMovedOn); - if (b != SCP::QuorumInfoNodeState::NO_INFO) - { - return b; - } - return mNominationProtocol.getState(node, selfAlreadyMovedOn); + return mBallotProtocol.getState(node, selfAlreadyMovedOn); } Json::Value diff --git a/src/scp/Slot.h b/src/scp/Slot.h index 3b556a904e..06c6dbfb9b 100644 --- a/src/scp/Slot.h +++ b/src/scp/Slot.h @@ -6,7 +6,6 @@ #include "BallotProtocol.h" #include "LocalNode.h" -#include "NominationProtocol.h" #include "lib/json/json-forwards.h" #include "scp/SCP.h" #include @@ -29,7 +28,6 @@ class Slot : public std::enable_shared_from_this SCP& mSCP; BallotProtocol mBallotProtocol; - NominationProtocol mNominationProtocol; // keeps track of all statements seen so far for this slot. // it is used for debugging purpose @@ -81,8 +79,6 @@ class Slot : public std::enable_shared_from_this return mBallotProtocol; } - ValueWrapperPtr const& getLatestCompositeCandidate(); - // returns the latest messages the slot emitted std::vector getLatestMessagesSend() const; @@ -99,8 +95,7 @@ class Slot : public std::enable_shared_from_this SCPEnvelope const* getLatestMessage(NodeID const& id) const; // Return true if the statement is latest for a node that sent it - bool isNewerNominationOrBallotSt(SCPStatement const& oldSt, - SCPStatement const& newSt); + bool isNewerStatement(SCPStatement const& oldSt, SCPStatement const& newSt); // returns messages that helped this slot externalize std::vector getExternalizingState() const; @@ -115,8 +110,6 @@ class Slot : public std::enable_shared_from_this SCP::EnvelopeState processEnvelope(SCPEnvelopeWrapperPtr envelope, bool self); - bool abandonBallot(); - // bumps the ballot based on the local state and the value passed in: // in prepare phase, attempts to take value // otherwise, no-ops @@ -124,14 +117,13 @@ class Slot : public std::enable_shared_from_this // the state if no value was prepared bool bumpState(Value const& value, bool force); - // attempts to nominate a value for consensus - bool nominate(NominationValueSupplier const& makeValue, - Value const& previousValue, bool timedout); - - void stopNomination(); + // Starts leader rounds for this slot (the local node is ready to take + // part in consensus on it). `makeValue` is invoked only if the local node + // becomes the leader of a round and has no value to put forward. + bool startLeaderRounds(ProposalValueSupplier const& makeValue); - // returns the current nomination leaders - std::set getNominationLeaders() const; + // The round the local node is in (0 if it has not started the slot). + uint32 getLeaderRound() const; bool isFullyValidated() const; void setFullyValidated(bool fullyValidated); @@ -196,14 +188,10 @@ class Slot : public std::enable_shared_from_this enum timerIDs { - NOMINATION_TIMER = 0, + // Runs while the local node waits for the leader of its current round + // to put a value forward. + LEADER_TIMER = 0, BALLOT_PROTOCOL_TIMER = 1, -#ifdef BUILD_TESTS - // Test-only: defers broadcast of a nomination vote by - // ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING. Used to simulate - // slow nominators without breaking the protocol. - NOMINATION_EMIT_TIMER = 2, -#endif }; // The number of times the timer has to expire before we consider the node diff --git a/src/scp/test/SCPTests.cpp b/src/scp/test/SCPTests.cpp index 8f57dda9b3..6feb5d7638 100644 --- a/src/scp/test/SCPTests.cpp +++ b/src/scp/test/SCPTests.cpp @@ -17,6 +17,10 @@ #include "xdrpp/marshal.h" #include "xdrpp/printer.h" #include +#include +#include +#include +#include // General convention in this file is that numbers in parenthesis // refer to the rule number in the related protocol from the white paper @@ -41,18 +45,15 @@ class TestSCP : public SCPDriver SCP mSCP; uint32_t mInitialBallotTimeoutMS = 1000; uint32_t mIncrementBallotTimeoutMS = 1000; - uint32_t mInitialNominationTimeoutMS = 1000; - uint32_t mIncrementNominationTimeoutMS = 1000; TestSCP(NodeID const& nodeID, SCPQuorumSet const& qSetLocal, bool isValidator = true) : mSCP(*this, nodeID, isValidator, qSetLocal) { - mPriorityLookup = [&](NodeID const& n) { - return (n == mSCP.getLocalNodeID()) ? 1000 : 1; - }; - - mHashValueCalculator = [&](Value const& v) { return 0; }; + // By default the local node leads every round: a ballot timeout then + // bumps the counter with the value the node already votes for, like + // the plain ballot protocol. + mRoundLeader = [&](uint64, uint32) { return mSCP.getLocalNodeID(); }; auto localQSet = std::make_shared(mSCP.getLocalQuorumSet()); @@ -73,11 +74,11 @@ class TestSCP : public SCPDriver SCPDriver::ValidationLevel validateValue(uint64 slotIndex, Value const& value, - bool nomination) const override + bool proposal) const override { if (mValidateValueOverride) { - return mValidateValueOverride(slotIndex, value, nomination); + return mValidateValueOverride(slotIndex, value, proposal); } // If we're tracking download wait time for this value, it's awaiting // download @@ -178,62 +179,55 @@ class TestSCP : public SCPDriver mEnvs.push_back(envelope); } - // used to test BallotProtocol and bypass nomination + // used to test BallotProtocol directly, without a round leader bool bumpState(uint64 slotIndex, Value const& v) { return mSCP.getSlot(slotIndex, true)->bumpState(v, true); } + // starts the slot the way the application does at its trigger; `value` + // is what the local node proposes if it leads a round bool - nominate(uint64 slotIndex, Value const& value, bool timedout) + startSlot(uint64 slotIndex, Value const& value) { auto wv = wrapValue(value); - return mSCP.getSlot(slotIndex, true) - ->nominate([wv]() { return wv; }, value, timedout); + return mSCP.startSlot(slotIndex, [this, wv]() { + ++mValuesBuilt; + return mMakeValueOverride ? mMakeValueOverride() : wv; + }); } - // only used by nomination protocol - ValueWrapperPtr - combineCandidates(uint64 slotIndex, - ValueWrapperPtrSet const& candidates) override + std::optional + getRoundLeader(uint64 slotIndex, uint32 round) override { - REQUIRE(candidates.size() == mExpectedCandidates.size()); - auto it1 = candidates.begin(); - auto it2 = mExpectedCandidates.end(); - for (; it1 != candidates.end() && it2 != mExpectedCandidates.end(); - it1++, it2++) - { - REQUIRE((*it1)->getValue() == *it2); - } - - REQUIRE(!mCompositeValue.empty()); - - return wrapValue(mCompositeValue); + return mRoundLeader(slotIndex, round); } - bool - hasUpgrades(Value const& v) override + void + proposedValue(uint64 slotIndex, uint32 round, Value const& value) override { - // Not implemented - releaseAssert(false); + mProposedValues.emplace_back(round, value); } - ValueWrapperPtr - stripAllUpgrades(Value const& v) override + void + enteredRound(uint64 slotIndex, uint32 round, bool timedOut) override { - // Not implemented - releaseAssert(false); + mEnteredRounds.emplace_back(round, timedOut); } - uint32_t - getUpgradeNominationTimeoutLimit() const override + bool + isProposalPending(uint64, NodeID const&) const override { - return std::numeric_limits::max(); + return mProposalPending; } - std::set mExpectedCandidates; - Value mCompositeValue; + std::function(uint64, uint32)> mRoundLeader; + bool mProposalPending{false}; + std::function mMakeValueOverride; + int mValuesBuilt{0}; + std::vector> mProposedValues; + std::vector> mEnteredRounds; Hash getHashOf(std::vector> const& vals) const override @@ -246,34 +240,6 @@ class TestSCP : public SCPDriver return hasher.finish(); } - // override the internal hashing scheme in order to make tests - // more predictable. - uint64 - computeHashNode(uint64 slotIndex, Value const& prev, bool isPriority, - int32_t roundNumber, NodeID const& nodeID) override - { - uint64 res; - if (isPriority) - { - res = mPriorityLookup(nodeID); - } - else - { - res = 0; - } - return res; - } - - // override the value hashing, to make tests more predictable. - uint64 - computeValueHash(uint64 slotIndex, Value const& prev, int32_t roundNumber, - Value const& value) override - { - return mHashValueCalculator(value); - } - - std::function mPriorityLookup; - std::function mHashValueCalculator; std::function mValidateValueOverride; @@ -293,15 +259,6 @@ class TestSCP : public SCPDriver std::map mTimers; std::chrono::milliseconds mCurrentTimerOffset{0}; - // Defers this node's nomination emits, like - // ARTIFICIALLY_DELAY_NOMINATION_EMIT_FOR_TESTING does for Herder. - std::chrono::milliseconds mNominationEmitDelay{0}; - std::chrono::milliseconds - getNominationEmitDelayForTesting() const override - { - return mNominationEmitDelay; - } - void setupTimer(uint64 slotIndex, int timerID, std::chrono::milliseconds timeout, std::function cb) override @@ -333,6 +290,34 @@ class TestSCP : public SCPDriver return mCurrentTimerOffset; } + TimerData + getLeaderTimer() + { + return mTimers[Slot::LEADER_TIMER]; + } + + bool + hasLeaderTimer() + { + return !!getLeaderTimer().mCallback; + } + + // fires a timer the way a real (one-shot) timer does + void + fireTimer(int timerID) + { + auto cb = mTimers[timerID].mCallback; + REQUIRE(cb); + mTimers[timerID].mCallback = nullptr; + cb(); + } + + uint32 + getLeaderRound(uint64 slotIndex) + { + return mSCP.getLeaderRound(slotIndex); + } + // returns true if a ballot protocol timer exists (in the past or future) bool hasBallotTimer() @@ -351,14 +336,6 @@ class TestSCP : public SCPDriver return mCurrentTimerOffset < getBallotProtocolTimer().mAbsoluteTimeout; } - Value const& - getLatestCompositeCandidate(uint64 slotIndex) - { - return mSCP.getSlot(slotIndex, true) - ->getLatestCompositeCandidate() - ->getValue(); - } - SCP::EnvelopeState receiveEnvelope(SCPEnvelope const& envelope) { @@ -393,12 +370,6 @@ class TestSCP : public SCPDriver throw std::runtime_error("not found"); } - std::set - getNominationLeaders(uint64 slotIndex) - { - return mSCP.getSlot(slotIndex, false)->getNominationLeaders(); - } - // Helper methods for empty-tx-set value testing void startDownload(Value const& v, std::chrono::milliseconds waitTime) @@ -416,21 +387,10 @@ class TestSCP : public SCPDriver static uint32_t const MAX_TIMEOUT_MS = (30 * 60) * 1000; std::chrono::milliseconds - computeTimeout(uint32 roundNumber, bool isNomination) override + computeTimeout(uint32 roundNumber) override { - int initialTimeoutMS; - int incrementMS; - - if (isNomination) - { - initialTimeoutMS = mInitialNominationTimeoutMS; - incrementMS = mIncrementNominationTimeoutMS; - } - else - { - initialTimeoutMS = mInitialBallotTimeoutMS; - incrementMS = mIncrementBallotTimeoutMS; - } + int initialTimeoutMS = mInitialBallotTimeoutMS; + int incrementMS = mIncrementBallotTimeoutMS; int timeoutMS = initialTimeoutMS + (roundNumber - 1) * incrementMS; if (timeoutMS > MAX_TIMEOUT_MS) @@ -599,15 +559,6 @@ verifyExternalize(SCPEnvelope const& actual, SecretKey const& secretKey, REQUIRE(exp.statement == actual.statement); } -void -verifyNominate(SCPEnvelope const& actual, SecretKey const& secretKey, - Hash const& qSetHash, uint64 slotIndex, std::vector votes, - std::vector accepted) -{ - auto exp = makeNominate(secretKey, qSetHash, slotIndex, votes, accepted); - REQUIRE(exp.statement == actual.statement); -} - // Simulate xValue being only structurally valid (e.g., due to an invalid tx // set) SCPDriver::ValidationLevel @@ -803,12 +754,8 @@ testTimeouts(TestSCP& scp, std::function f) { SECTION("minimum values") { - scp.mInitialNominationTimeoutMS = MinimumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INITIAL_MILLISECONDS; scp.mInitialBallotTimeoutMS = MinimumSorobanNetworkConfig::BALLOT_TIMEOUT_INITIAL_MILLISECONDS; - scp.mIncrementNominationTimeoutMS = MinimumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INCREMENT_MILLISECONDS; scp.mIncrementBallotTimeoutMS = MinimumSorobanNetworkConfig::BALLOT_TIMEOUT_INCREMENT_MILLISECONDS; f(scp); @@ -816,12 +763,8 @@ testTimeouts(TestSCP& scp, std::function f) SECTION("initial values") { - scp.mInitialNominationTimeoutMS = InitialSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INITIAL_MILLISECONDS; scp.mInitialBallotTimeoutMS = InitialSorobanNetworkConfig::BALLOT_TIMEOUT_INITIAL_MILLISECONDS; - scp.mIncrementNominationTimeoutMS = InitialSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INCREMENT_MILLISECONDS; scp.mIncrementBallotTimeoutMS = InitialSorobanNetworkConfig::BALLOT_TIMEOUT_INCREMENT_MILLISECONDS; f(scp); @@ -829,12 +772,8 @@ testTimeouts(TestSCP& scp, std::function f) SECTION("maximum values") { - scp.mInitialNominationTimeoutMS = MaximumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INITIAL_MILLISECONDS; scp.mInitialBallotTimeoutMS = MaximumSorobanNetworkConfig::BALLOT_TIMEOUT_INITIAL_MILLISECONDS; - scp.mIncrementNominationTimeoutMS = MaximumSorobanNetworkConfig:: - NOMINATION_TIMEOUT_INCREMENT_MILLISECONDS; scp.mIncrementBallotTimeoutMS = MaximumSorobanNetworkConfig::BALLOT_TIMEOUT_INCREMENT_MILLISECONDS; f(scp); @@ -2900,663 +2839,218 @@ TEST_CASE("ballot protocol core3", "[scp][ballotprotocol]") testTimeouts(scp, test); } -TEST_CASE("delayed nomination emit after an immediate emit", - "[scp][nominationprotocol]") +TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " + "structurally-valid value", + "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - // v-blocking set size 2, quorum threshold 4 SCPQuorumSet qSet; - qSet.threshold = 4; + qSet.threshold = 2; qSet.validators.push_back(v0NodeID); qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); - qSet.validators.push_back(v3NodeID); - qSet.validators.push_back(v4NodeID); + uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - scp.mNominationEmitDelay = std::chrono::milliseconds(1000); - // v0 leads the round and votes for x, but its emit is deferred - REQUIRE(scp.nominate(0, xValue, false)); - REQUIRE(scp.mEnvs.empty()); - REQUIRE(scp.mTimers.count(Slot::NOMINATION_EMIT_TIMER) == 1); - auto delayedEmit = scp.mTimers[Slot::NOMINATION_EMIT_TIMER].mCallback; - REQUIRE(delayedEmit); + // xValue stays kStructurallyValidValue throughout, with a wait time + // below the download timeout so maybeReplaceValueWithEmptyTxSet does not + // replace it with an empty-tx-set value. + scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); + scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - std::vector votes{xValue}; + SCPBallot xB1(1, xValue); - SECTION("nothing emitted meanwhile: the delayed emit sends the vote") - { - delayedEmit(); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votes, {}); - } + // v0 enters ballot protocol with xValue. + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - SECTION("an immediate emit meanwhile: the delayed emit is not repeated") - { - // A v-blocking set accepting x makes v0 accept it, and processEnvelope - // emits the new state at once while the delayed emit is still armed. - std::vector accepted{xValue}; - scp.receiveEnvelope( - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted)); - REQUIRE(scp.mEnvs.empty()); - scp.receiveEnvelope( - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted)); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votes, - accepted); + // v1, v2 vote-prepare for (1, xValue) → v0 accept-prepared (1, xValue). + REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1)) == + SCP::EnvelopeState::VALID); - // Nothing changed since: re-emitting the same statement used to be - // rejected by our own processEnvelope ("moved to a bad state"). - REQUIRE_NOTHROW(delayedEmit()); - REQUIRE(scp.mEnvs.size() == 1); - } + // v1, v2 signal accept-prepared (1, xValue). v0 would normally + // confirm-prepared here, but setConfirmPrepared stalls on + // kStructurallyValidValue so c/h stay unset on v0's side. + REQUIRE( + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1)) == + SCP::EnvelopeState::VALID); + REQUIRE( + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1)) == + SCP::EnvelopeState::VALID); + + // v1, v2 vote-to-commit (1, xValue) via nC/nH on their PREPAREs. + // federatedAccept fires via the "quorum voted-or-accepted" path v0 + // accept-commits, transitions mPhase to CONFIRM, and self-emits a CONFIRM + // with xValue. + REQUIRE(scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == + SCP::EnvelopeState::VALID); + + // Last emitted envelope should be a CONFIRM carrying xValue — proves that + // processEnvelope correctly accepts self-emitted CONFIRMs with + // kStructurallyValidValue. + auto const& lastEnv = scp.mEnvs.back(); + REQUIRE(lastEnv.statement.pledges.type() == SCP_ST_CONFIRM); + auto const& cBallot = lastEnv.statement.pledges.confirm().ballot; + REQUIRE(cBallot.value == xValue); + REQUIRE(!scp.isEmptyTxSetValue(cBallot.value)); + + // A peer CONFIRM whose value v0 considers kStructurallyValidValue is + // rejected by processEnvelope + auto const res = scp.receiveEnvelope( + makeConfirm(v1SecretKey, qSetHash, 0, 1, xB1, 1, 1)); + REQUIRE(res == SCP::EnvelopeState::INVALID); } -TEST_CASE("nomination tests core5", "[scp][nominationprotocol]") +TEST_CASE("drop tx set on download timeout", "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - // we need 5 nodes to avoid sharing various thresholds: - // v-blocking set size: 2 - // threshold: 4 = 3 + self or 4 others + // 3 node network with threshold=2 (need any 2 nodes to form quorum) SCPQuorumSet qSet; - qSet.threshold = 4; + qSet.threshold = 2; qSet.validators.push_back(v0NodeID); qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); - qSet.validators.push_back(v3NodeID); - qSet.validators.push_back(v4NodeID); uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); - REQUIRE(xValue < yValue); - REQUIRE(yValue < zValue); - - auto checkLeaders = [&](TestSCP& scp, std::set expectedLeaders) { - auto l = scp.getNominationLeaders(0); - REQUIRE(std::equal(l.begin(), l.end(), expectedLeaders.begin(), - expectedLeaders.end())); - }; + TestSCP scp(v0SecretKey.getPublicKey(), qSet); + scp.storeQuorumSet(std::make_shared(qSet)); + uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - SECTION("nomination - v0 is top") + SECTION("timeout during prepare phase") { - TestSCP scp(v0SecretKey.getPublicKey(), qSet); + // Node v0 starts ballot protocol with xValue + // Simulate that xValue is awaiting download with timeout exceeded + scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - auto test = [&](TestSCP& scp) { - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - scp.storeQuorumSet(std::make_shared(qSet)); + // Now call bumpState which should trigger + // maybeReplaceValueWithEmptyTxSet + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - SECTION("v0 starts to nominates xValue") - { - REQUIRE(scp.nominate(0, xValue, false)); + // The ballot should have an empty-tx-set value, not the original + // xValue + auto const& emittedBallot = + scp.mEnvs[0].statement.pledges.prepare().ballot; - checkLeaders(scp, {v0SecretKey.getPublicKey()}); + REQUIRE(emittedBallot.counter == 1); + REQUIRE(scp.isEmptyTxSetValue(emittedBallot.value)); - SECTION("others nominate what v0 says (x) -> prepare x") - { - std::vector votes, accepted; - votes.emplace_back(xValue); + // Verify it's the empty-tx-set value derived from xValue + Value expectedEmptyTxSetValue = + scp.makeEmptyTxSetValueFromValue(xValue); + REQUIRE(emittedBallot.value == expectedEmptyTxSetValue); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - votes, accepted); - - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom3 = - makeNominate(v3SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom4 = - makeNominate(v4SecretKey, qSetHash, 0, votes, accepted); - - // nothing happens yet - scp.receiveEnvelope(nom1); - scp.receiveEnvelope(nom2); - REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, + SCPBallot(1, expectedEmptyTxSetValue)); + } - // this causes 'x' to be accepted (quorum) - scp.receiveEnvelope(nom3); - REQUIRE(scp.mEnvs.size() == 2); + SECTION("no timeout when wait time under threshold") + { + // Node v0 starts ballot protocol with xValue + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; + SCPBallot b1(1, xValue); - accepted.emplace_back(xValue); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votes, accepted); + // Simulate that xValue is awaiting download but wait time is still low + scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); - // extra message doesn't do anything - scp.receiveEnvelope(nom4); - REQUIRE(scp.mEnvs.size() == 2); + // Try to bump state - should NOT replace with an empty-tx-set value + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 2); - SCPEnvelope acc1 = - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope acc2 = - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope acc3 = - makeNominate(v3SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope acc4 = - makeNominate(v4SecretKey, qSetHash, 0, votes, accepted); - - // nothing happens yet - scp.receiveEnvelope(acc1); - scp.receiveEnvelope(acc2); - REQUIRE(scp.mEnvs.size() == 2); + // Verify ballot still has original xValue, not an empty-tx-set value + auto const& emittedBallot = + scp.mEnvs[1].statement.pledges.prepare().ballot; + REQUIRE(emittedBallot.counter == 2); + REQUIRE(!scp.isEmptyTxSetValue(emittedBallot.value)); + REQUIRE(emittedBallot.value == xValue); + } - scp.mCompositeValue = xValue; - // this causes the node to send a prepare message (quorum) - scp.receiveEnvelope(acc3); - REQUIRE(scp.mEnvs.size() == 3); + SECTION("empty-tx-set value can be prepared and confirmed") + { + // Start with xValue and timeout to an empty-tx-set value + scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, - SCPBallot(1, xValue)); + REQUIRE(scp.bumpState(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); - scp.receiveEnvelope(acc4); - REQUIRE(scp.mEnvs.size() == 3); + Value emptyTxSetValue = scp.makeEmptyTxSetValueFromValue(xValue); + SCPBallot emptyTxSetB1(1, emptyTxSetValue); - std::vector votes2 = votes; - votes2.emplace_back(yValue); + // Verify we emitted an empty-tx-set value + REQUIRE(scp.isEmptyTxSetValue( + scp.mEnvs[0].statement.pledges.prepare().ballot.value)); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, emptyTxSetB1); - SECTION("nominate x -> accept x -> prepare (x) ; others " - "accepted y " - "-> update latest to (z=x+y)") - { - SCPEnvelope acc1_2 = makeNominate(v1SecretKey, qSetHash, - 0, votes2, votes2); - SCPEnvelope acc2_2 = makeNominate(v2SecretKey, qSetHash, - 0, votes2, votes2); - SCPEnvelope acc3_2 = makeNominate(v3SecretKey, qSetHash, - 0, votes2, votes2); - SCPEnvelope acc4_2 = makeNominate(v4SecretKey, qSetHash, - 0, votes2, votes2); - - scp.receiveEnvelope(acc1_2); - REQUIRE(scp.mEnvs.size() == 3); - - // v-blocking - scp.receiveEnvelope(acc2_2); - REQUIRE(scp.mEnvs.size() == 4); - verifyNominate(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, - votes2, votes2); - - scp.mExpectedCandidates.insert(yValue); - scp.mCompositeValue = kValue; - // this updates the composite value to use next time - // but does not prepare it - scp.receiveEnvelope(acc3_2); - REQUIRE(scp.mEnvs.size() == 4); - - REQUIRE(scp.getLatestCompositeCandidate(0) == kValue); - - scp.receiveEnvelope(acc4_2); - REQUIRE(scp.mEnvs.size() == 4); - } - SECTION("nomination - restored state") - { - TestSCP scp2(v0SecretKey.getPublicKey(), qSet); - scp2.storeQuorumSet( - std::make_shared(qSet)); - - // at this point - // votes = { x } - // accepted = { x } - - // tests if nomination proceeds like normal - // nominates x - auto nominationRestore = [&]() { - // restores from the previous state - scp2.mSCP.setStateFromEnvelope( - 0, scp2.wrapEnvelope( - makeNominate(v0SecretKey, qSetHash0, 0, - votes, accepted))); - // tries to start nomination with yValue, but picks - // xValue since it was already in the votes - REQUIRE(!scp2.nominate(0, yValue, false)); - - checkLeaders(scp2, {v0SecretKey.getPublicKey()}); - - REQUIRE(scp2.mEnvs.size() == 0); - - // other nodes vote for 'x' - scp2.receiveEnvelope(nom1); - scp2.receiveEnvelope(nom2); - REQUIRE(scp2.mEnvs.size() == 0); - // 'x' is accepted (quorum) - // but because the restored state already included - // 'x' in the accepted set, no new message is - // emitted - scp2.receiveEnvelope(nom3); - - scp2.mExpectedCandidates.emplace(xValue); - scp2.mCompositeValue = xValue; - - // other nodes not emit 'x' as accepted - scp2.receiveEnvelope(acc1); - scp2.receiveEnvelope(acc2); - REQUIRE(scp2.mEnvs.size() == 0); - - scp2.mCompositeValue = xValue; - // this causes the node to update its composite - // value to - // x - scp2.receiveEnvelope(acc3); - }; - - SECTION("ballot protocol not started") - { - nominationRestore(); - // nomination ended up starting the ballot protocol - REQUIRE(scp2.mEnvs.size() == 1); + // Other nodes also move to the empty-tx-set value + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1)); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1)); - verifyPrepare(scp2.mEnvs[0], v0SecretKey, qSetHash0, - 0, SCPBallot(1, xValue)); - } - SECTION("ballot protocol started (on value k)") - { - scp2.mSCP.setStateFromEnvelope( - 0, scp2.wrapEnvelope( - makePrepare(v0SecretKey, qSetHash0, 0, - SCPBallot(1, kValue)))); - nominationRestore(); - // nomination didn't do anything (already working on - // k) - REQUIRE(scp2.mEnvs.size() == 0); - } - } - } - SECTION("receive more messages, then v0 switches to a " - "different leader") - { - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, {kValue}, {}); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, {yValue}, {}); - - // nothing more happens - scp.receiveEnvelope(nom1); - scp.receiveEnvelope(nom2); - REQUIRE(scp.mEnvs.size() == 1); + // Should prepare the empty-tx-set value (quorum reached) + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, emptyTxSetB1, + &emptyTxSetB1); - // switch leader to v1 - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v1NodeID) ? 1000 : 1; - }; - REQUIRE(scp.nominate(0, xValue, true)); - REQUIRE(scp.mEnvs.size() == 2); + // Quorum confirms prepared empty-tx-set value + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); - std::vector votesXK; - votesXK.emplace_back(xValue); - votesXK.emplace_back(kValue); - std::sort(votesXK.begin(), votesXK.end()); + REQUIRE(scp.mEnvs.size() == 3); + verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, emptyTxSetB1, + &emptyTxSetB1, 1, 1); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votesXK, {}); - } - SECTION("select accepted value from leader") - { - REQUIRE(xValue < yValue); - REQUIRE(yValue < zValue); - REQUIRE(scp.mEnvs.size() == 1); + // Accept commit + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, + &emptyTxSetB1, 1, 1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, + &emptyTxSetB1, 1, 1)); - // Update round leader to v1 - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v1NodeID) ? 1000 : 1; - }; + REQUIRE(scp.mEnvs.size() == 4); + verifyConfirm(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, 1, emptyTxSetB1, + 1, 1); - SCPEnvelope nom1 = makeNominate(v1SecretKey, qSetHash, 0, - {yValue, zValue}, {yValue}); + // Externalize the empty-tx-set value + scp.receiveEnvelope( + makeConfirm(v1SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); + scp.receiveEnvelope( + makeConfirm(v2SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); - SECTION("receive accepted before timeout") - { - // nothing more happens, v0 is leader - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 1); - - // Update round leaders, vote for accepted value (y) - REQUIRE(scp.nominate(0, xValue, true)); - REQUIRE(scp.mEnvs.size() == 2); - } - SECTION("receive accepted after timeout") - { - REQUIRE(!scp.nominate(0, xValue, true)); - REQUIRE(scp.mEnvs.size() == 1); - - // Vote for accepted value (y) - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 2); - } - - std::vector votesXY; - votesXY.emplace_back(xValue); - votesXY.emplace_back(yValue); - - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votesXY, {}); - - SCPEnvelope nom2 = - makeNominate(v1SecretKey, qSetHash, 0, - {yValue, zValue, zzValue}, {yValue}); - scp.receiveEnvelope(nom2); - // Nothing happens, as v0 already voted for the accepted - // value (y) - REQUIRE(scp.mEnvs.size() == 2); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - votesXY, {}); - } - } - SECTION("self nominates 'x', others nominate y -> prepare y") - { - std::vector myVotes, accepted; - myVotes.emplace_back(xValue); - - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; - REQUIRE(scp.nominate(0, xValue, false)); - - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, myVotes, - accepted); - - std::vector votes; - votes.emplace_back(yValue); - - std::vector acceptedY = accepted; - - acceptedY.emplace_back(yValue); - - SECTION("others only vote for y") - { - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom3 = - makeNominate(v3SecretKey, qSetHash, 0, votes, accepted); - SCPEnvelope nom4 = - makeNominate(v4SecretKey, qSetHash, 0, votes, accepted); - - // nothing happens yet - scp.receiveEnvelope(nom1); - scp.receiveEnvelope(nom2); - scp.receiveEnvelope(nom3); - REQUIRE(scp.mEnvs.size() == 1); - - // 'y' is accepted (quorum) - scp.receiveEnvelope(nom4); - REQUIRE(scp.mEnvs.size() == 2); - myVotes.emplace_back(yValue); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - myVotes, acceptedY); - } - SECTION("others accepted y") - { - SCPEnvelope acc1 = makeNominate(v1SecretKey, qSetHash, 0, - votes, acceptedY); - SCPEnvelope acc2 = makeNominate(v2SecretKey, qSetHash, 0, - votes, acceptedY); - SCPEnvelope acc3 = makeNominate(v3SecretKey, qSetHash, 0, - votes, acceptedY); - SCPEnvelope acc4 = makeNominate(v4SecretKey, qSetHash, 0, - votes, acceptedY); - - scp.receiveEnvelope(acc1); - REQUIRE(scp.mEnvs.size() == 1); - - // this causes 'y' to be accepted (v-blocking) - scp.receiveEnvelope(acc2); - REQUIRE(scp.mEnvs.size() == 2); - - myVotes.emplace_back(yValue); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, - myVotes, acceptedY); - - scp.mExpectedCandidates.clear(); - scp.mExpectedCandidates.insert(yValue); - scp.mCompositeValue = yValue; - // this causes the node to send a prepare message (quorum) - scp.receiveEnvelope(acc3); - REQUIRE(scp.mEnvs.size() == 3); - - verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, - SCPBallot(1, yValue)); - - scp.receiveEnvelope(acc4); - REQUIRE(scp.mEnvs.size() == 3); - } - } - }; - - testTimeouts(scp, test); - } - SECTION("v1 is top node") - { - TestSCP scp(v0SecretKey.getPublicKey(), qSet); - - auto test = [&](TestSCP& scp) { - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - scp.storeQuorumSet(std::make_shared(qSet)); - - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v1NodeID) ? 1000 : 1; - }; - - std::vector votesX, votesY, votesK, votesXY, votesYK, - votesXK, emptyV; - votesX.emplace_back(xValue); - votesY.emplace_back(yValue); - votesK.emplace_back(kValue); - - votesXY.emplace_back(xValue); - votesXY.emplace_back(yValue); - - votesYK.emplace_back(yValue); - votesYK.emplace_back(kValue); - std::sort(votesYK.begin(), votesYK.end()); - - votesXK.emplace_back(xValue); - votesXK.emplace_back(kValue); - std::sort(votesXK.begin(), votesXK.end()); - - std::vector valuesHash; - valuesHash.emplace_back(xValue); - valuesHash.emplace_back(yValue); - valuesHash.emplace_back(kValue); - std::sort(valuesHash.begin(), valuesHash.end()); - - scp.mHashValueCalculator = [&](Value const& v) { - auto pos = std::find(valuesHash.begin(), valuesHash.end(), v); - if (pos == valuesHash.end()) - { - abort(); - } - return 1 + std::distance(valuesHash.begin(), pos); - }; - - SCPEnvelope nom1 = - makeNominate(v1SecretKey, qSetHash, 0, votesXY, emptyV); - SCPEnvelope nom2 = - makeNominate(v2SecretKey, qSetHash, 0, votesXK, emptyV); - - SECTION( - "value from v1 is a candidate, self should not introduce new " - "value on timeout") - { - REQUIRE(!scp.nominate(0, xValue, false)); - checkLeaders(scp, {v1SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 0); - nom1 = makeNominate(v1SecretKey, qSetHash, 0, votesX, emptyV); - nom2 = makeNominate(v2SecretKey, qSetHash, 0, votesX, emptyV); - SCPEnvelope nom3 = - makeNominate(v3SecretKey, qSetHash, 0, votesX, emptyV); - - // Receive `x` from v1, vote for it - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votesX, - emptyV); - - scp.receiveEnvelope(nom2); - scp.receiveEnvelope(nom3); - REQUIRE(scp.mEnvs.size() == 2); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, votesX, - votesX); - - SCPEnvelope acc1 = - makeNominate(v1SecretKey, qSetHash, 0, votesX, votesX); - SCPEnvelope acc2 = - makeNominate(v2SecretKey, qSetHash, 0, votesX, votesX); - SCPEnvelope acc3 = - makeNominate(v3SecretKey, qSetHash, 0, votesX, votesX); - - scp.receiveEnvelope(acc1); - scp.receiveEnvelope(acc2); - REQUIRE(scp.mEnvs.size() == 2); - - // Receive accept from quorum, ratify and generate a candidate - // value - REQUIRE(scp.mTimers.find(Slot::NOMINATION_TIMER) != - scp.mTimers.end()); - scp.mCompositeValue = xValue; - scp.mExpectedCandidates.emplace(xValue); - scp.receiveEnvelope(acc3); - REQUIRE(scp.mEnvs.size() == 3); - // Timer is cancelled - REQUIRE(scp.mTimers.find(Slot::NOMINATION_TIMER) == - scp.mTimers.end()); - - // v0 is the new leader, but we already have a candidate - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v0NodeID) ? 1000 : 1; - }; - REQUIRE(!scp.nominate(0, kValue, true)); - } - SECTION("nomination waits for v1") - { - REQUIRE(!scp.nominate(0, xValue, false)); - - checkLeaders(scp, {v1SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 0); - - SCPEnvelope nom4 = - makeNominate(v4SecretKey, qSetHash, 0, votesXK, emptyV); - - // nothing happens with non top nodes - scp.receiveEnvelope(nom2); - // (note: don't receive anything from node3 - we want to pick - // another dead node) - REQUIRE(scp.mEnvs.size() == 0); - - // v1 is leader -> nominate the first value from its message - // that's "y" - scp.receiveEnvelope(nom1); - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, votesY, - emptyV); - - scp.receiveEnvelope(nom4); - REQUIRE(scp.mEnvs.size() == 1); - - // "timeout -> pick another value from v1" - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; - - // allows to pick another leader, - // pick another dead node v3 as to force picking up - // a new value from v1 - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v3NodeID) ? 1000 : 1; - }; - - // note: value passed in here should be ignored - REQUIRE(scp.nominate(0, kValue, true)); - // picks up 'x' from v1 (as we already have 'y') - // which also happens to causes 'x' to be accepted - REQUIRE(scp.mEnvs.size() == 2); - verifyNominate(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, votesXY, - votesX); - } - SECTION("v1 dead, timeout") - { - REQUIRE(!scp.nominate(0, xValue, false)); - - REQUIRE(scp.mEnvs.size() == 0); - - scp.receiveEnvelope(nom2); - REQUIRE(scp.mEnvs.size() == 0); - - checkLeaders(scp, {v1SecretKey.getPublicKey()}); - - SECTION("v0 is new top node") - { - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v0NodeID) ? 1000 : 1; - }; - - REQUIRE(scp.nominate(0, xValue, true)); - checkLeaders(scp, {v0SecretKey.getPublicKey(), - v1SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 1); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - votesX, emptyV); - } - SECTION("v2 is new top node") - { - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v2NodeID) ? 1000 : 1; - }; - - REQUIRE(scp.nominate(0, xValue, true)); - checkLeaders(scp, {v1SecretKey.getPublicKey(), - v2SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 1); - // v2 votes for XK, but nomination only picks the highest - // value - std::vector v2Top; - v2Top.emplace_back(std::max(xValue, kValue)); - verifyNominate(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - v2Top, emptyV); - } - SECTION("v3 is new top node") - { - scp.mPriorityLookup = [&](NodeID const& n) { - return (n == v3NodeID) ? 1000 : 1; - }; - // nothing happens, we don't have any message for v3 - REQUIRE(!scp.nominate(0, xValue, true)); - checkLeaders(scp, {v1SecretKey.getPublicKey(), - v3SecretKey.getPublicKey()}); - - REQUIRE(scp.mEnvs.size() == 0); - } - } - }; + REQUIRE(scp.mEnvs.size() == 5); + verifyExternalize(scp.mEnvs[4], v0SecretKey, qSetHash0, 0, emptyTxSetB1, + 1); - testTimeouts(scp, test); + // Verify the externalized value is the empty-tx-set value + REQUIRE(scp.mExternalizedValues.size() == 1); + REQUIRE(scp.isEmptyTxSetValue(scp.mExternalizedValues[0])); + REQUIRE(scp.mExternalizedValues[0] == emptyTxSetValue); } } -TEST_CASE("nomination times out structurally-valid value into empty tx set", - "[scp][nomination]") +TEST_CASE("Proper handling of non-current ledger value", + "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); @@ -3569,48 +3063,19 @@ TEST_CASE("nomination times out structurally-valid value into empty tx set", qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); - auto const qSetHash = sha256(xdr::xdr_to_opaque(qSet)); - TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - // xValue is structurally-valid throughout — its tx set is either still in - // flight or has been downloaded-and-found-invalid. Seed a wait time past - // the download timeout so maybeReplaceValueWithEmptyTxSet triggers - // empty-tx-set replacement at bumpState time. - scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - - REQUIRE(scp.nominate(0, xValue, false)); - - auto const followerVoteNomination = - makeNominate(v1SecretKey, qSetHash, 0, {xValue}, {}); - REQUIRE(scp.receiveEnvelope(followerVoteNomination) == - SCP::EnvelopeState::VALID); - - scp.mExpectedCandidates.emplace(xValue); - scp.mCompositeValue = xValue; + // validateValue reports xValue is not for the current ledger. + scp.mValidateValueOverride = xValueNotCurrentLedgerOverride; - auto const followerAcceptedNomination = - makeNominate(v2SecretKey, qSetHash, 0, {xValue}, {xValue}); - // Quorum accept-nominated xValue → composite value flows to - // bumpState → maybeReplaceValueWithEmptyTxSet sees - // kStructurallyValidValue with the wait time past the timeout and - // substitutes an empty-tx-set value. - REQUIRE(scp.receiveEnvelope(followerAcceptedNomination) == - SCP::EnvelopeState::VALID); + REQUIRE(scp.bumpState(0, xValue)); - // The emitted ballot should carry the empty-tx-set value derived from - // xValue, not xValue itself. - auto const& lastEnv = scp.mEnvs.back(); - REQUIRE(lastEnv.statement.pledges.type() == SCP_ST_PREPARE); - auto const& ballot = lastEnv.statement.pledges.prepare().ballot; - REQUIRE(scp.isEmptyTxSetValue(ballot.value)); - REQUIRE(ballot.value == scp.makeEmptyTxSetValueFromValue(xValue)); + // Do not emit a ballot with a kMaybeValid value + REQUIRE(scp.mEnvs.size() == 0); } -TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " - "structurally-valid value", +TEST_CASE("setConfirmPrepared stalls on kStructurallyValidValue value", "[scp][ballotprotocol]") { setupValues(); @@ -3629,27 +3094,15 @@ TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - // xValue stays kStructurallyValidValue throughout, with a wait time - // below the download timeout so maybeReplaceValueWithEmptyTxSet does not - // replace it with an empty-tx-set value. + // Simulate parallel downloading scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); - scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - - SCPBallot xB1(1, xValue); - // v0 enters ballot protocol with xValue. + // v0 enters ballot protocol REQUIRE(scp.bumpState(0, xValue)); REQUIRE(scp.mEnvs.size() == 1); + SCPBallot xB1(1, xValue); - // v1, v2 vote-prepare for (1, xValue) → v0 accept-prepared (1, xValue). - REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == - SCP::EnvelopeState::VALID); - REQUIRE(scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1)) == - SCP::EnvelopeState::VALID); - - // v1, v2 signal accept-prepared (1, xValue). v0 would normally - // confirm-prepared here, but setConfirmPrepared stalls on - // kStructurallyValidValue so c/h stay unset on v0's side. + // v1 and v2 send PREPAREs with prepared — quorum confirms prepared REQUIRE( scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1)) == SCP::EnvelopeState::VALID); @@ -3657,41 +3110,72 @@ TEST_CASE("ballot protocol self-emits CONFIRM after federated accept-commit on " scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1)) == SCP::EnvelopeState::VALID); - // v1, v2 vote-to-commit (1, xValue) via nC/nH on their PREPAREs. - // federatedAccept fires via the "quorum voted-or-accepted" path v0 - // accept-commits, transitions mPhase to CONFIRM, and self-emits a CONFIRM - // with xValue. - REQUIRE(scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == - SCP::EnvelopeState::VALID); - REQUIRE(scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1, 1, 1)) == - SCP::EnvelopeState::VALID); + SECTION("commit gate stalls mCommit but mHighBallot is set") + { - // Last emitted envelope should be a CONFIRM carrying xValue — proves that - // processEnvelope correctly accepts self-emitted CONFIRMs with - // kStructurallyValidValue. - auto const& lastEnv = scp.mEnvs.back(); - REQUIRE(lastEnv.statement.pledges.type() == SCP_ST_CONFIRM); - auto const& cBallot = lastEnv.statement.pledges.confirm().ballot; - REQUIRE(cBallot.value == xValue); - REQUIRE(!scp.isEmptyTxSetValue(cBallot.value)); + // setConfirmPrepared sets mHighBallot (nH > 0) but the commit gate + // stalls mCommit (nC == 0) because xValue is kStructurallyValidValue. + // Check the latest emitted PREPARE. + REQUIRE(scp.mEnvs.size() >= 2); + auto const& lastPrep = scp.mEnvs.back().statement.pledges.prepare(); + REQUIRE(lastPrep.nH == 1); + REQUIRE(lastPrep.nC == 0); + } - // A peer CONFIRM whose value v0 considers kStructurallyValidValue is - // rejected by processEnvelope - auto const res = scp.receiveEnvelope( - makeConfirm(v1SecretKey, qSetHash, 0, 1, xB1, 1, 1)); - REQUIRE(res == SCP::EnvelopeState::INVALID); + SECTION("proceeds after value becomes validated") + { + auto envsBeforeClear = scp.mEnvs.size(); + + // Simulate tx set arrival — value becomes fully validated + scp.clearDownload(xValue); + + // Trigger advanceSlot with envelopes that confirm-prepare at a + // higher counter, so attemptConfirmPrepared finds newH > mHighBallot. + // This causes setConfirmPrepared to be called with the now-validated + // value, setting mCommit. + SCPBallot xB2(2, xValue); + REQUIRE(scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, xB2, &xB2)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, xB2, &xB2)) == + SCP::EnvelopeState::VALID); + + // setConfirmPrepared should now succeed — mCommit set, node + // progresses. Expect at least one new envelope with nC > 0 or a + // CONFIRM/EXTERNALIZE. + REQUIRE(scp.mEnvs.size() > envsBeforeClear); + bool foundC = false; + for (size_t i = envsBeforeClear; i < scp.mEnvs.size(); i++) + { + auto const& st = scp.mEnvs[i].statement; + if (st.pledges.type() == SCP_ST_PREPARE) + { + if (st.pledges.prepare().nC > 0) + { + foundC = true; + break; + } + } + else + { + // CONFIRM or EXTERNALIZE also proves we got past the stall + foundC = true; + break; + } + } + REQUIRE(foundC); + } } -TEST_CASE("drop tx set on download timeout", "[scp][ballotprotocol]") +TEST_CASE("incoming PREPARE with structurally valid prepared value is accepted", + "[scp][ballotprotocol]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); - // 3 node network with threshold=2 (need any 2 nodes to form quorum) SCPQuorumSet qSet; qSet.threshold = 2; qSet.validators.push_back(v0NodeID); @@ -3702,124 +3186,24 @@ TEST_CASE("drop tx set on download timeout", "[scp][ballotprotocol]") TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - SECTION("timeout during prepare phase") - { - // Node v0 starts ballot protocol with xValue - // Simulate that xValue is awaiting download with timeout exceeded - scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); + // v0 enters ballot protocol with yValue + REQUIRE(scp.bumpState(0, yValue)); + REQUIRE(scp.mEnvs.size() == 1); - // Now call bumpState which should trigger - // maybeReplaceValueWithEmptyTxSet - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); + // Set xValue to kStructurallyValidValue + scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; - // The ballot should have an empty-tx-set value, not the original - // xValue - auto const& emittedBallot = - scp.mEnvs[0].statement.pledges.prepare().ballot; - - REQUIRE(emittedBallot.counter == 1); - REQUIRE(scp.isEmptyTxSetValue(emittedBallot.value)); - - // Verify it's the empty-tx-set value derived from xValue - Value expectedEmptyTxSetValue = - scp.makeEmptyTxSetValueFromValue(xValue); - REQUIRE(emittedBallot.value == expectedEmptyTxSetValue); - - verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, - SCPBallot(1, expectedEmptyTxSetValue)); - } - - SECTION("no timeout when wait time under threshold") - { - // Node v0 starts ballot protocol with xValue - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); - - SCPBallot b1(1, xValue); - - // Simulate that xValue is awaiting download but wait time is still low - scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); - - // Try to bump state - should NOT replace with an empty-tx-set value - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 2); - - // Verify ballot still has original xValue, not an empty-tx-set value - auto const& emittedBallot = - scp.mEnvs[1].statement.pledges.prepare().ballot; - REQUIRE(emittedBallot.counter == 2); - REQUIRE(!scp.isEmptyTxSetValue(emittedBallot.value)); - REQUIRE(emittedBallot.value == xValue); - } - - SECTION("empty-tx-set value can be prepared and confirmed") - { - // Start with xValue and timeout to an empty-tx-set value - scp.startDownload(xValue, OVER_TX_SET_TIMEOUT); - - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); - - Value emptyTxSetValue = scp.makeEmptyTxSetValueFromValue(xValue); - SCPBallot emptyTxSetB1(1, emptyTxSetValue); - - // Verify we emitted an empty-tx-set value - REQUIRE(scp.isEmptyTxSetValue( - scp.mEnvs[0].statement.pledges.prepare().ballot.value)); - verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, emptyTxSetB1); - - // Other nodes also move to the empty-tx-set value - scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1)); - scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1)); - - // Should prepare the empty-tx-set value (quorum reached) - REQUIRE(scp.mEnvs.size() == 2); - verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, emptyTxSetB1, - &emptyTxSetB1); - - // Quorum confirms prepared empty-tx-set value - scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); - scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, &emptyTxSetB1)); - - REQUIRE(scp.mEnvs.size() == 3); - verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, emptyTxSetB1, - &emptyTxSetB1, 1, 1); - - // Accept commit - scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, emptyTxSetB1, - &emptyTxSetB1, 1, 1)); - scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, emptyTxSetB1, - &emptyTxSetB1, 1, 1)); - - REQUIRE(scp.mEnvs.size() == 4); - verifyConfirm(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, 1, emptyTxSetB1, - 1, 1); - - // Externalize the empty-tx-set value - scp.receiveEnvelope( - makeConfirm(v1SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); - scp.receiveEnvelope( - makeConfirm(v2SecretKey, qSetHash, 0, 1, emptyTxSetB1, 1, 1)); - - REQUIRE(scp.mEnvs.size() == 5); - verifyExternalize(scp.mEnvs[4], v0SecretKey, qSetHash0, 0, emptyTxSetB1, - 1); - - // Verify the externalized value is the empty-tx-set value - REQUIRE(scp.mExternalizedValues.size() == 1); - REQUIRE(scp.isEmptyTxSetValue(scp.mExternalizedValues[0])); - REQUIRE(scp.mExternalizedValues[0] == emptyTxSetValue); - } + // v1 sends PREPARE with valid ballot value but only structurally valid + // prepared value. This should be accepted. + SCPBallot yB1(1, yValue); + SCPBallot xB1(1, xValue); + REQUIRE( + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, yB1, &xB1)) == + SCP::EnvelopeState::VALID); } -TEST_CASE("Proper handling of non-current ledger value", +TEST_CASE("incoming PREPARE with non-tx-set-invalid value is dropped", "[scp][ballotprotocol]") { setupValues(); @@ -3833,180 +3217,994 @@ TEST_CASE("Proper handling of non-current ledger value", qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); + uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); + TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); - // validateValue reports xValue is not for the current ledger. - scp.mValidateValueOverride = xValueNotCurrentLedgerOverride; + // xValue is invalid for some non-tx-set reason (close time, signature, + // ...). This should NOT be accepted as kStructurallyValidValue; the + // statement should be dropped. + scp.mValidateValueOverride = xValueNonTxSetInvalidValidationOverride; - REQUIRE(scp.bumpState(0, xValue)); + SCPBallot xB1(1, xValue); - // Do not emit a ballot with a kMaybeValid value - REQUIRE(scp.mEnvs.size() == 0); + // Envelope should be rejected as invalid + REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == + SCP::EnvelopeState::INVALID); + + // Envelope was not recorded — recordEnvelope only runs for non-kInvalid. + REQUIRE(scp.mSCP.getLatestMessage(v1NodeID) == nullptr); + // No local emit triggered. + REQUIRE(scp.mEnvs.empty()); } -TEST_CASE("setConfirmPrepared stalls on kStructurallyValidValue value", - "[scp][ballotprotocol]") +// --------------------------------------------------------------------------- +// Leader rounds +// +// There is no nomination phase. A round is a ballot counter; each round has +// one leader (SCPDriver::getRoundLeader). At counter r a node votes for the +// ballot `leader(r)` votes for at counter r, unless it already has a value it +// must stick to. +// --------------------------------------------------------------------------- + +namespace +{ +SCPBallot const& +latestBallot(TestSCP& scp) +{ + REQUIRE(!scp.mEnvs.empty()); + auto const& pl = scp.mEnvs.back().statement.pledges; + REQUIRE(pl.type() == SCP_ST_PREPARE); + return pl.prepare().ballot; +} +} + +TEST_CASE("leader rounds core5", "[scp][leaderschedule]") { setupValues(); SIMULATION_CREATE_NODE(0); SIMULATION_CREATE_NODE(1); SIMULATION_CREATE_NODE(2); + SIMULATION_CREATE_NODE(3); + SIMULATION_CREATE_NODE(4); + // we need 5 nodes to avoid sharing various thresholds: + // v-blocking set size: 2 + // threshold: 4 = 3 + self or 4 others SCPQuorumSet qSet; - qSet.threshold = 2; + qSet.threshold = 4; qSet.validators.push_back(v0NodeID); qSet.validators.push_back(v1NodeID); qSet.validators.push_back(v2NodeID); + qSet.validators.push_back(v3NodeID); + qSet.validators.push_back(v4NodeID); uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); TestSCP scp(v0SecretKey.getPublicKey(), qSet); scp.storeQuorumSet(std::make_shared(qSet)); + uint256 qSetHash0 = scp.mSCP.getLocalNode()->getQuorumSetHash(); - // Simulate parallel downloading - scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); + // round r is led by schedule[(r - 1) % size] + std::vector schedule; + scp.mRoundLeader = [&](uint64, uint32 round) -> std::optional { + if (schedule.empty()) + { + return std::nullopt; + } + return schedule[(round - 1) % schedule.size()]; + }; - // v0 enters ballot protocol - REQUIRE(scp.bumpState(0, xValue)); - REQUIRE(scp.mEnvs.size() == 1); - SCPBallot xB1(1, xValue); + SCPBallot x1(1, xValue), y1(1, yValue), x2(2, xValue), y2(2, yValue), + z2(2, zValue), z3(3, zValue); - // v1 and v2 send PREPAREs with prepared — quorum confirms prepared - REQUIRE( - scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1, &xB1)) == - SCP::EnvelopeState::VALID); - REQUIRE( - scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, xB1, &xB1)) == - SCP::EnvelopeState::VALID); + SECTION("the round 1 leader proposes when the slot starts") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.startSlot(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.mValuesBuilt == 1); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(scp.mProposedValues.size() == 1); + REQUIRE(scp.mProposedValues[0] == std::make_pair(uint32(1), xValue)); + // it has a ballot at its round: the ballot timer rules apply, not the + // leader timer + REQUIRE(!scp.hasLeaderTimer()); + + SECTION("starting again changes nothing") + { + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.mEnvs.size() == 1); + REQUIRE(scp.mValuesBuilt == 1); + } + SECTION("other nodes' ballots do not move the leader") + { + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.mEnvs.size() == 1); + } + } - SECTION("commit gate stalls mCommit but mHighBallot is set") + SECTION("a follower waits for the leader and adopts its ballot") { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.mEnvs.empty()); + // followers do not build a value + REQUIRE(scp.mValuesBuilt == 0); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(scp.hasLeaderTimer()); + + // a ballot from a node that does not lead the round is recorded but + // not adopted + REQUIRE( + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, y1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.hasLeaderTimer()); - // setConfirmPrepared sets mHighBallot (nH > 0) but the commit gate - // stalls mCommit (nC == 0) because xValue is kStructurallyValidValue. - // Check the latest emitted PREPARE. - REQUIRE(scp.mEnvs.size() >= 2); - auto const& lastPrep = scp.mEnvs.back().statement.pledges.prepare(); - REQUIRE(lastPrep.nH == 1); - REQUIRE(lastPrep.nC == 0); + // the leader's ballot is adopted + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(!scp.hasLeaderTimer()); + REQUIRE(scp.mValuesBuilt == 0); + + SECTION("and externalizes with a quorum") + { + // quorum votes prepare x1 -> accept prepared + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, x1, &x1); + // quorum accepted prepared -> confirm prepared, vote commit + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + REQUIRE(scp.mEnvs.size() == 3); + verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, x1, &x1, 1, + 1); + // quorum votes commit -> accept commit + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1, &x1, 1, 1)); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, x1, &x1, 1, 1)); + scp.receiveEnvelope( + makePrepare(v3SecretKey, qSetHash, 0, x1, &x1, 1, 1)); + REQUIRE(scp.mEnvs.size() == 4); + verifyConfirm(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, 1, x1, 1, 1); + // quorum accepted commit -> externalize + scp.receiveEnvelope( + makeConfirm(v1SecretKey, qSetHash, 0, 1, x1, 1, 1)); + scp.receiveEnvelope( + makeConfirm(v2SecretKey, qSetHash, 0, 1, x1, 1, 1)); + scp.receiveEnvelope( + makeConfirm(v3SecretKey, qSetHash, 0, 1, x1, 1, 1)); + REQUIRE(scp.mEnvs.size() == 5); + verifyExternalize(scp.mEnvs[4], v0SecretKey, qSetHash0, 0, x1, 1); + REQUIRE(scp.mExternalizedValues[0] == xValue); + REQUIRE(scp.mValuesBuilt == 0); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("a later ballot of the same leader at the same counter is " + "not adopted") + { + // equivocating leader: the node already voted for x1 + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.mEnvs.size() == 1); + } } - SECTION("proceeds after value becomes validated") + SECTION("a follower adopts the round 1 leader before its own trigger") { - auto envsBeforeClear = scp.mEnvs.size(); + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.getLeaderRound(0) == 0); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.getLeaderRound(0) == 1); - // Simulate tx set arrival — value becomes fully validated - scp.clearDownload(xValue); + // its own trigger then has nothing left to do + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.mEnvs.size() == 1); + REQUIRE(scp.mValuesBuilt == 0); + } - // Trigger advanceSlot with envelopes that confirm-prepare at a - // higher counter, so attemptConfirmPrepared finds newH > mHighBallot. - // This causes setConfirmPrepared to be called with the now-validated - // value, setting mCommit. - SCPBallot xB2(2, xValue); - REQUIRE(scp.receiveEnvelope( - makePrepare(v1SecretKey, qSetHash, 0, xB2, &xB2)) == - SCP::EnvelopeState::VALID); - REQUIRE(scp.receiveEnvelope( - makePrepare(v2SecretKey, qSetHash, 0, xB2, &xB2)) == - SCP::EnvelopeState::VALID); + SECTION("before its trigger a node does not adopt later rounds' leaders") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.getLeaderRound(0) == 0); + REQUIRE(!scp.hasLeaderTimer()); + } - // setConfirmPrepared should now succeed — mCommit set, node - // progresses. Expect at least one new envelope with nC > 0 or a - // CONFIRM/EXTERNALIZE. - REQUIRE(scp.mEnvs.size() > envsBeforeClear); - bool foundC = false; - for (size_t i = envsBeforeClear; i < scp.mEnvs.size(); i++) + SECTION("a leader that is not started does not propose") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.mValuesBuilt == 0); + } + + SECTION("silent leader: the leader timer hands over to the next leader") + { + SECTION("local node leads round 2") { - auto const& st = scp.mEnvs[i].statement; - if (st.pledges.type() == SCP_ST_PREPARE) + schedule = {v1NodeID, v0NodeID, v2NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, yValue)); + REQUIRE(scp.hasLeaderTimer()); + auto timer = scp.getLeaderTimer(); + REQUIRE(timer.mAbsoluteTimeout == + std::chrono::milliseconds(scp.mInitialBallotTimeoutMS)); + + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnteredRounds.back() == + std::make_pair(uint32(2), true)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + REQUIRE(scp.mValuesBuilt == 1); + REQUIRE(!scp.hasLeaderTimer()); + + SECTION("the round 1 leader showing up late changes nothing") { - if (st.pledges.prepare().nC > 0) - { - foundC = true; - break; - } + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); } - else + SECTION("a stale leader timer callback is ignored") { - // CONFIRM or EXTERNALIZE also proves we got past the stall - foundC = true; - break; + timer.mCallback(); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 1); } } - REQUIRE(foundC); - } -} + SECTION("another node leads round 2") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); -TEST_CASE("incoming PREPARE with structurally valid prepared value is accepted", - "[scp][ballotprotocol]") -{ - setupValues(); - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); + SECTION("its ballot arrives after the timeout") + { + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.empty()); + // waiting for the round 2 leader now, with the round 2 + // timeout + REQUIRE(scp.hasLeaderTimer()); + REQUIRE( + scp.getLeaderTimer().mAbsoluteTimeout == + std::chrono::milliseconds(scp.mInitialBallotTimeoutMS + + scp.mIncrementBallotTimeoutMS)); + + SECTION("the round 2 leader's ballot is adopted") + { + // a round 2 ballot from a node that does not lead round 2 + scp.receiveEnvelope( + makePrepare(v3SecretKey, qSetHash, 0, x2)); + REQUIRE(scp.mEnvs.empty()); - SCPQuorumSet qSet; - qSet.threshold = 2; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("a late round 1 proposal still gets the node's vote") + { + // The node never voted: it votes for the round 1 leader's + // late proposal at the round 1 counter, and keeps waiting + // for the round 2 leader. + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.hasLeaderTimer()); + REQUIRE(!scp.hasBallotTimer()); - uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); + SECTION("and externalizes with a quorum at round 1") + { + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope( + makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope( + makePrepare(v4SecretKey, qSetHash, 0, x1, &x1)); + // accepted then confirmed prepared: the node votes to + // commit x at counter 1 + REQUIRE(scp.mEnvs.size() >= 2); + auto const& last = + scp.mEnvs.back().statement.pledges.prepare(); + REQUIRE(last.ballot.value == xValue); + REQUIRE(last.nC == 1); + REQUIRE(last.nH == 1); + } + SECTION("the round 2 leader's ballot takes over") + { + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, + y2); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("a second late proposal is ignored") + { + // only a node without a ballot takes a late proposal + schedule = {v1NodeID, v2NodeID, v3NodeID, v0NodeID, + v4NodeID}; + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 3); + scp.receiveEnvelope( + makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + } + } + SECTION("a proposal that is not valid is not taken late either") + { + scp.mValidateValueOverride = [&](uint64, Value const& v, + bool proposal) { + return (proposal && v == xValue) + ? SCPDriver::kInvalidValue + : SCPDriver::kFullyValidatedValue; + }; + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.empty()); + } + } + SECTION("the round 1 leader's proposal is pending: one extension") + { + // The application holds the leader's statement (e.g. it is + // fetching its transaction set): the node waits one more + // timeout, once. + scp.mProposalPending = true; + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(scp.hasLeaderTimer()); + REQUIRE(scp.getLeaderTimer().mAbsoluteTimeout == + std::chrono::milliseconds(scp.mInitialBallotTimeoutMS)); + REQUIRE(scp.mEnvs.empty()); + + SECTION("the proposal arrives during the extension") + { + scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + REQUIRE(scp.getLeaderRound(0) == 1); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("it does not: the node moves on") + { + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.hasLeaderTimer()); + REQUIRE(scp.mEnvs.empty()); + } + } + SECTION("its ballot arrived early: kept until the round starts") + { + // a single node cannot make others skip a round + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.getLeaderRound(0) == 1); - TestSCP scp(v0SecretKey.getPublicKey(), qSet); - scp.storeQuorumSet(std::make_shared(qSet)); + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + } + SECTION("then local node leads round 3 with its own value") + { + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.mEnvs.empty()); + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 3); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, z3); + REQUIRE(scp.mValuesBuilt == 1); + } + } + } - // v0 enters ballot protocol with yValue - REQUIRE(scp.bumpState(0, yValue)); - REQUIRE(scp.mEnvs.size() == 1); + SECTION("ballot timeout: the next leader takes over") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); - // Set xValue to kStructurallyValidValue - scp.mValidateValueOverride = xValueStructurallyValidValidationOverride; + // a quorum is at counter 1: the ballot timer runs + REQUIRE(!scp.hasBallotTimer()); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, y1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, y1)); + REQUIRE(scp.hasBallotTimer()); + REQUIRE(!scp.hasLeaderTimer()); - // v1 sends PREPARE with valid ballot value but only structurally valid - // prepared value. This should be accepted. - SCPBallot yB1(1, yValue); - SCPBallot xB1(1, xValue); - REQUIRE( - scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, yB1, &xB1)) == - SCP::EnvelopeState::VALID); + SECTION("to a leader whose ballot is known") + { + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 2); + // v3 and v4 voted for y at counter 1: with the leader and the + // local node that is a quorum, so (1, y) is accepted prepared + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, y2, &y1); + } + SECTION("to a leader that has not spoken yet") + { + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + // keeps its round 1 ballot, votes for nothing new + REQUIRE(scp.mEnvs.size() == 1); + REQUIRE(scp.hasLeaderTimer()); + // the ballot timer of the round it gave up on is stopped + REQUIRE(!scp.hasBallotTimer()); + + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, y2, &y1); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("to the local node, which keeps the value it voted for") + { + schedule = {v1NodeID, v0NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 2); + // x, not its own z: the transaction set of x is already out + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, x2); + REQUIRE(scp.mValuesBuilt == 0); + } + SECTION("to the local node, whose current value is not proposable") + { + schedule = {v1NodeID, v0NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.mValidateValueOverride = [&](uint64, Value const& v, + bool proposal) { + return (proposal && v == xValue) + ? SCPDriver::kInvalidValue + : SCPDriver::kFullyValidatedValue; + }; + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, z2); + REQUIRE(scp.mValuesBuilt == 1); + } + } + + SECTION("a confirmed prepared value wins over the leader's") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 2); + verifyPrepare(scp.mEnvs[1], v0SecretKey, qSetHash0, 0, x1, &x1); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, x1, &x1)); + // confirmed prepared: x is now the value to stick to + REQUIRE(scp.mEnvs.size() == 3); + verifyPrepare(scp.mEnvs[2], v0SecretKey, qSetHash0, 0, x1, &x1, 1, 1); + + // the round 2 leader votes for something else + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 3); + scp.fireTimer(Slot::BALLOT_PROTOCOL_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + REQUIRE(scp.mEnvs.size() == 4); + verifyPrepare(scp.mEnvs[3], v0SecretKey, qSetHash0, 0, x2, &x1, 1, 1); + REQUIRE(!scp.hasLeaderTimer()); + } + + SECTION("a v-blocking set ahead moves a node to that round") + { + schedule = {v1NodeID, v2NodeID, v3NodeID, v0NodeID, v4NodeID}; + // v1 and v2 (v-blocking) are at counter 3, on a value they must + // stick to; local node has no ballot at all + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.getLeaderRound(0) == 0); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.getLeaderRound(0) == 3); + // it does not vote for what they vote for: only the round's leader + // decides + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.hasLeaderTimer()); + + SECTION("where it adopts the leader") + { + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.mEnvs.size() == 1); + // with v1, v2 and the leader that is a quorum voting for z3 + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, z3, &z3); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("where it leads once it is started") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.startSlot(0, xValue)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, + SCPBallot(3, xValue)); + } + } + + SECTION("a started leader pushed to its round by a v-blocking set " + "proposes") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, xValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, z3)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, z3)); + REQUIRE(scp.getLeaderRound(0) == 3); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, + SCPBallot(3, xValue)); + REQUIRE(scp.mValuesBuilt == 1); + } + + SECTION("proposal validation gates adoption") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(!scp.startSlot(0, zValue)); + + SECTION("a value the node would not propose is not adopted") + { + // valid for the ballot protocol, but not as a proposal (e.g. it + // carries an upgrade the local node did not arm) + scp.mValidateValueOverride = [&](uint64, Value const& v, + bool proposal) { + return (proposal && v == xValue) + ? SCPDriver::kInvalidValue + : SCPDriver::kFullyValidatedValue; + }; + REQUIRE(scp.receiveEnvelope( + makePrepare(v1SecretKey, qSetHash, 0, x1)) == + SCP::EnvelopeState::VALID); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.hasLeaderTimer()); + + // the regular protocol still applies: a quorum that prepared the + // value brings the node along + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, x1, &x1)); + // v-blocking accepted prepared + REQUIRE(scp.mEnvs.empty()); + scp.receiveEnvelope(makePrepare(v3SecretKey, qSetHash, 0, x1, &x1)); + scp.receiveEnvelope(makePrepare(v4SecretKey, qSetHash, 0, x1, &x1)); + // quorum accepted prepared -> confirmed prepared + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1, &x1, 1, + 1); + } + SECTION("a value whose transaction set is still downloading is " + "adopted") + { + scp.startDownload(xValue, UNDER_TX_SET_TIMEOUT); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, x1); + } + } + + SECTION("a leader that cannot build a value lets the round time out") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + scp.mMakeValueOverride = []() { return ValueWrapperPtr(); }; + REQUIRE(!scp.startSlot(0, xValue)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.mValuesBuilt == 1); + REQUIRE(scp.hasLeaderTimer()); + scp.fireTimer(Slot::LEADER_TIMER); + REQUIRE(scp.getLeaderRound(0) == 2); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, y2)); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, y2); + } + + SECTION("building the value re-enters SCP") + { + schedule = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, v4NodeID}; + auto xW = scp.wrapValue(xValue); + + SECTION("the slot moved to a later round meanwhile") + { + scp.mMakeValueOverride = [&]() { + // envelopes that became ready while the value was built + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, z3)); + scp.receiveEnvelope(makePrepare(v2SecretKey, qSetHash, 0, z3)); + return xW; + }; + REQUIRE(scp.startSlot(0, xValue)); + // the value was built for round 1; the node is in round 3 now, + // which v2 leads: it votes for v2's ballot, not for its own value + REQUIRE(scp.getLeaderRound(0) == 3); + REQUIRE(scp.mEnvs.size() == 1); + verifyPrepare(scp.mEnvs[0], v0SecretKey, qSetHash0, 0, z3); + REQUIRE(!scp.hasLeaderTimer()); + } + SECTION("the slot externalized meanwhile") + { + scp.mMakeValueOverride = [&]() { + for (auto const& k : + {v1SecretKey, v2SecretKey, v3SecretKey, v4SecretKey}) + { + scp.receiveEnvelope(makeExternalize(k, qSetHash, 0, y1, 1)); + } + return xW; + }; + scp.startSlot(0, xValue); + REQUIRE(scp.mExternalizedValues[0] == yValue); + for (auto const& e : scp.mEnvs) + { + // never voted for its own value + auto vals = Slot::getStatementValues(e.statement); + REQUIRE(std::find(vals.begin(), vals.end(), xValue) == + vals.end()); + } + REQUIRE(!scp.hasLeaderTimer()); + } + } + + SECTION("no schedule known: neither proposes nor adopts") + { + // e.g. the slot is not the next ledger to close + REQUIRE(!scp.startSlot(0, xValue)); + scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(scp.mEnvs.empty()); + REQUIRE(scp.mValuesBuilt == 0); + } + + SECTION("non-validators never vote for a leader's ballot") + { + TestSCP watcher(v0SecretKey.getPublicKey(), qSet, false); + watcher.storeQuorumSet(std::make_shared(qSet)); + watcher.mRoundLeader = [&](uint64, uint32) { return v1NodeID; }; + watcher.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, x1)); + REQUIRE(watcher.mEnvs.empty()); + REQUIRE(watcher.getLeaderRound(0) == 0); + REQUIRE(!watcher.hasLeaderTimer()); + } + + SECTION("nomination statements are dropped") + { + schedule = {v1NodeID, v2NodeID, v0NodeID, v3NodeID, v4NodeID}; + REQUIRE(scp.receiveEnvelope( + makeNominate(v1SecretKey, qSetHash, 0, {xValue}, {})) == + SCP::EnvelopeState::INVALID); + REQUIRE(scp.mSCP.getLatestMessage(v1NodeID) == nullptr); + REQUIRE(scp.mEnvs.empty()); + } } -TEST_CASE("incoming PREPARE with non-tx-set-invalid value is dropped", - "[scp][ballotprotocol]") +namespace { - setupValues(); - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); +// A small network of TestSCP nodes on one slot, driven by a virtual clock: +// messages are delayed, reordered and (for a while) lost; some nodes are dead; +// nodes start the slot at different times. +struct LeaderRoundsModel +{ + struct Node + { + SecretKey mKey; + std::unique_ptr mSCP; + bool mAlive{true}; + bool mStarted{false}; + std::chrono::milliseconds mStartTime{0}; + size_t mEnvsSeen{0}; + Value mValue; + }; - SCPQuorumSet qSet; - qSet.threshold = 2; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); + struct Delivery + { + std::chrono::milliseconds mWhen; + uint64_t mSeq; + size_t mTo; + SCPEnvelope mEnv; + bool + operator>(Delivery const& o) const + { + return std::tie(mWhen, mSeq) > std::tie(o.mWhen, o.mSeq); + } + }; - uint256 qSetHash = sha256(xdr::xdr_to_opaque(qSet)); + stellar_default_random_engine mRng; + std::vector mNodes; + std::vector mSchedule; + std::priority_queue, std::greater<>> mQueue; + uint64_t mSeq{0}; + std::chrono::milliseconds mNow{0}; + // until then, messages are lost with probability mLossPercent and can be + // delayed up to mMaxChaosDelay + std::chrono::milliseconds mChaosEnd{0}; + uint32_t mLossPercent{0}; + std::chrono::milliseconds mMaxChaosDelay{0}; + std::chrono::milliseconds mNextRebroadcast{0}; + size_t mThreshold{0}; + + LeaderRoundsModel(uint32_t seed, size_t numNodes, size_t numDead) + : mRng(seed) + { + mThreshold = numNodes - (numNodes - 1) / 3; + releaseAssert(numNodes - numDead >= mThreshold); + SCPQuorumSet qSet; + qSet.threshold = static_cast(mThreshold); + for (size_t i = 0; i < numNodes; i++) + { + Node n; + n.mKey = SecretKey::fromSeed( + sha256(fmt::format("LEADER_MODEL_NODE_{}", i))); + qSet.validators.push_back(n.mKey.getPublicKey()); + // distinct proposals + n.mValue = xdr::xdr_to_opaque( + sha256(fmt::format("LEADER_MODEL_VALUE_{}_{}", seed, i))); + mNodes.emplace_back(std::move(n)); + } + for (auto const& n : mNodes) + { + mSchedule.emplace_back(n.mKey.getPublicKey()); + } + stellar::shuffle(mSchedule.begin(), mSchedule.end(), mRng); - TestSCP scp(v0SecretKey.getPublicKey(), qSet); - scp.storeQuorumSet(std::make_shared(qSet)); + // dead nodes are picked from the front of the schedule half of the + // time (so the first leaders are dead), at random otherwise + std::vector order(numNodes); + std::iota(order.begin(), order.end(), 0); + if (rand_uniform(0, 1, mRng) == 0) + { + std::sort(order.begin(), order.end(), [&](size_t a, size_t b) { + auto pos = [&](size_t i) { + return std::find(mSchedule.begin(), mSchedule.end(), + mNodes[i].mKey.getPublicKey()) - + mSchedule.begin(); + }; + return pos(a) < pos(b); + }); + } + else + { + stellar::shuffle(order.begin(), order.end(), mRng); + } + for (size_t i = 0; i < numDead; i++) + { + mNodes[order[i]].mAlive = false; + } - // xValue is invalid for some non-tx-set reason (close time, signature, - // ...). This should NOT be accepted as kStructurallyValidValue; the - // statement should be dropped. - scp.mValidateValueOverride = xValueNonTxSetInvalidValidationOverride; + mChaosEnd = + std::chrono::milliseconds(rand_uniform(0, 30000, mRng)); + mLossPercent = rand_uniform(0, 40, mRng); + mMaxChaosDelay = + std::chrono::milliseconds(rand_uniform(10, 4000, mRng)); + mNextRebroadcast = mChaosEnd; - SCPBallot xB1(1, xValue); + for (auto& n : mNodes) + { + n.mSCP = std::make_unique(n.mKey.getPublicKey(), qSet); + n.mSCP->mRoundLeader = [this](uint64, uint32 round) { + return mSchedule[(round - 1) % mSchedule.size()]; + }; + // most nodes start close together; some are late + n.mStartTime = std::chrono::milliseconds( + rand_uniform(0, 3, mRng) == 0 + ? rand_uniform(0, 8000, mRng) + : rand_uniform(0, 300, mRng)); + } + } - // Envelope should be rejected as invalid - REQUIRE(scp.receiveEnvelope(makePrepare(v1SecretKey, qSetHash, 0, xB1)) == - SCP::EnvelopeState::INVALID); + void + send(size_t from, SCPEnvelope const& env) + { + for (size_t to = 0; to < mNodes.size(); to++) + { + if (to == from || !mNodes[to].mAlive) + { + continue; + } + bool chaos = mNow < mChaosEnd; + if (chaos && rand_uniform(0, 99, mRng) < mLossPercent) + { + continue; + } + auto maxDelay = + chaos ? mMaxChaosDelay : std::chrono::milliseconds(150); + auto delay = std::chrono::milliseconds( + rand_uniform(1, static_cast(maxDelay.count()), mRng)); + mQueue.push(Delivery{mNow + delay, mSeq++, to, env}); + } + } - // Envelope was not recorded — recordEnvelope only runs for non-kInvalid. - REQUIRE(scp.mSCP.getLatestMessage(v1NodeID) == nullptr); - // No local emit triggered. - REQUIRE(scp.mEnvs.empty()); + // forwards whatever the nodes emitted since the last call + void + flush() + { + for (size_t i = 0; i < mNodes.size(); i++) + { + auto& n = mNodes[i]; + while (n.mEnvsSeen < n.mSCP->mEnvs.size()) + { + auto env = n.mSCP->mEnvs[n.mEnvsSeen++]; + send(i, env); + } + } + } + + bool + allExternalized() const + { + for (auto const& n : mNodes) + { + if (n.mAlive && n.mSCP->mExternalizedValues.empty()) + { + return false; + } + } + return true; + } + + // runs until every live node externalized or `limit` (virtual time); + // returns true in the first case + bool + run(std::chrono::milliseconds limit) + { + while (!allExternalized()) + { + // next event: a start, a delivery, a timer or a rebroadcast + enum class Kind + { + NONE, + START, + DELIVERY, + TIMER, + REBROADCAST + }; + Kind kind = Kind::REBROADCAST; + auto when = mNextRebroadcast; + size_t node = 0; + int timerID = 0; + for (size_t i = 0; i < mNodes.size(); i++) + { + auto& n = mNodes[i]; + if (!n.mAlive) + { + continue; + } + if (!n.mStarted && n.mStartTime < when) + { + kind = Kind::START; + when = n.mStartTime; + node = i; + } + for (auto const& t : n.mSCP->mTimers) + { + if (t.second.mCallback && t.second.mAbsoluteTimeout < when) + { + kind = Kind::TIMER; + when = t.second.mAbsoluteTimeout; + node = i; + timerID = t.first; + } + } + } + if (!mQueue.empty() && mQueue.top().mWhen < when) + { + kind = Kind::DELIVERY; + when = mQueue.top().mWhen; + } + if (when > limit) + { + return false; + } + mNow = std::max(mNow, when); + for (auto& n : mNodes) + { + n.mSCP->mCurrentTimerOffset = mNow; + } + switch (kind) + { + case Kind::START: + { + auto& n = mNodes[node]; + n.mStarted = true; + n.mSCP->startSlot(0, n.mValue); + } + break; + case Kind::TIMER: + { + auto& td = mNodes[node].mSCP->mTimers[timerID]; + auto cb = td.mCallback; + td.mCallback = nullptr; + cb(); + } + break; + case Kind::DELIVERY: + { + auto d = mQueue.top(); + mQueue.pop(); + mNodes[d.mTo].mSCP->receiveEnvelope(d.mEnv); + } + break; + case Kind::REBROADCAST: + { + // like Herder, nodes periodically re-send their latest + // statement + mNextRebroadcast = mNow + std::chrono::milliseconds(2000); + for (size_t i = 0; i < mNodes.size(); i++) + { + auto& n = mNodes[i]; + if (n.mAlive && !n.mSCP->mEnvs.empty()) + { + send(i, n.mSCP->mEnvs.back()); + } + } + } + break; + default: + releaseAssert(false); + } + flush(); + } + return true; + } + + void + checkAgreement() const + { + std::optional externalized; + for (auto const& n : mNodes) + { + auto const& ext = n.mSCP->mExternalizedValues; + if (ext.empty()) + { + continue; + } + auto const& v = ext.begin()->second; + if (!externalized) + { + externalized = v; + // it is one of the proposals + REQUIRE( + std::any_of(mNodes.begin(), mNodes.end(), + [&](Node const& o) { return o.mValue == v; })); + } + REQUIRE(*externalized == v); + } + } +}; + +void +runLeaderRoundsModel(uint32_t firstSeed, uint32_t numSeeds) +{ + for (uint32_t seed = firstSeed; seed < firstSeed + numSeeds; seed++) + { + size_t const numNodes = 4 + seed % 4; // 4..7 + size_t const maxDead = (numNodes - 1) / 3; // 1, 1, 1, 2 + size_t const numDead = (seed / 4) % (maxDead + 1); + INFO(fmt::format("seed {} nodes {} dead {}", seed, numNodes, numDead)); + LeaderRoundsModel model(seed, numNodes, numDead); + // Once messages flow again, the slot must close: the bound leaves + // room for a full turn of the schedule with growing timeouts. + bool done = model.run(model.mChaosEnd + std::chrono::minutes(10)); + // safety holds no matter what + model.checkAgreement(); + REQUIRE(done); + } +} +} + +TEST_CASE("leader rounds network model", "[scp][leaderschedule]") +{ + runLeaderRoundsModel(1, 400); +} + +TEST_CASE("leader rounds network model long", "[scp][leaderschedule][!hide]") +{ + runLeaderRoundsModel(100000, 20000); } } diff --git a/src/scp/test/SCPUnitTests.cpp b/src/scp/test/SCPUnitTests.cpp deleted file mode 100644 index 50dbd484af..0000000000 --- a/src/scp/test/SCPUnitTests.cpp +++ /dev/null @@ -1,840 +0,0 @@ -#include "scp/LocalNode.h" -#include "scp/SCP.h" -#include "scp/Slot.h" -#include "simulation/Simulation.h" -#include "test/Catch2.h" -#include "util/Logging.h" -#include "xdrpp/marshal.h" - -namespace stellar -{ -static bool -isNear(uint64 r, double target) -{ - double v = (double)r / (double)UINT64_MAX; - return (std::abs(v - target) < .01); -} - -class TestNominationSCP : public SCPDriver -{ - public: - SCP mSCP; - uint32_t mInitialNominationTimeoutMS = 1000; - uint32_t mIncrementNominationTimeoutMS = 1000; - uint32_t mInitialBallotTimeoutMS = 1000; - uint32_t mIncrementBallotTimeoutMS = 1000; - - TestNominationSCP(NodeID const& nodeID, SCPQuorumSet const& qSetLocal) - : mSCP(*this, nodeID, true, qSetLocal) - { - auto localQSet = - std::make_shared(mSCP.getLocalQuorumSet()); - storeQuorumSet(localQSet); - } - - void - signEnvelope(SCPEnvelope&) override - { - } - - void - storeQuorumSet(SCPQuorumSetPtr qSet) - { - Hash qSetHash = sha256(xdr::xdr_to_opaque(*qSet.get())); - mQuorumSets[qSetHash] = qSet; - } - - SCPDriver::ValidationLevel - validateValue(uint64 slotIndex, Value const& value, - bool nomination) const override - { - return SCPDriver::kFullyValidatedValue; - } - - SCPQuorumSetPtr - getQSet(Hash const& qSetHash) override - { - if (mQuorumSets.find(qSetHash) != mQuorumSets.end()) - { - - return mQuorumSets[qSetHash]; - } - return SCPQuorumSetPtr(); - } - - void - emitEnvelope(SCPEnvelope const& envelope) override - { - } - - ValueWrapperPtr - combineCandidates(uint64 slotIndex, - ValueWrapperPtrSet const& candidates) override - { - return nullptr; - } - - bool - hasUpgrades(Value const& v) override - { - // Not implemented - releaseAssert(false); - } - - ValueWrapperPtr - stripAllUpgrades(Value const& v) override - { - // Not implemented - releaseAssert(false); - } - - uint32_t - getUpgradeNominationTimeoutLimit() const override - { - return std::numeric_limits::max(); - } - - void - setupTimer(uint64 slotIndex, int timerID, std::chrono::milliseconds timeout, - std::function cb) override - { - } - - void - stopTimer(uint64 slotIndex, int timerID) override - { - } - - std::optional - getTxSetDownloadWaitTime(Value const& v) const override - { - return std::nullopt; - } - - std::chrono::milliseconds - getTxSetDownloadTimeout() const override - { - return std::chrono::milliseconds(100); - } - - Value - makeEmptyTxSetValueFromValue(Value const& value) const override - { - releaseAssert(false); - } - - bool - isEmptyTxSetValue(Value const& v) const override - { - releaseAssert(false); - } - - bool - isParallelTxSetDownloadEnabled() const override - { - return true; - } - - bool - protocolAllowsEmptyTxSetValues() const override - { - return true; - } - - std::map mQuorumSets; - - Value const& - getLatestCompositeCandidate(uint64 slotIndex) - { - static Value const emptyValue{}; - return emptyValue; - } - - Hash - getHashOf(std::vector> const& vals) const override - { - SHA256 hasher; - for (auto const& v : vals) - { - hasher.add(v); - } - return hasher.finish(); - } - - // Copied from HerderSCPDriver.cpp - static uint32_t const MAX_TIMEOUT_MS = (30 * 60) * 1000; - - std::chrono::milliseconds - computeTimeout(uint32 roundNumber, bool isNomination) override - { - int initialTimeoutMS; - int incrementMS; - - if (isNomination) - { - initialTimeoutMS = mInitialNominationTimeoutMS; - incrementMS = mIncrementNominationTimeoutMS; - } - else - { - initialTimeoutMS = mInitialBallotTimeoutMS; - incrementMS = mIncrementBallotTimeoutMS; - } - - int timeoutMS = initialTimeoutMS + (roundNumber - 1) * incrementMS; - if (timeoutMS > MAX_TIMEOUT_MS) - { - timeoutMS = MAX_TIMEOUT_MS; - } - return std::chrono::milliseconds(timeoutMS); - } - - bool - isEnvelopeReady(SCPEnvelope const& envelope) const override - { - // Not implemented. These tests do not use PendingEnvelopes, and so do - // not require this method. - releaseAssert(false); - } -}; - -// Deliberately repeat the first winner in round 2, so the second call must -// fast-forward to round 3. This catches predicting round 2's hash in isolation. -class LeaderPreviewTestDriver : public TestNominationSCP -{ - public: - std::vector nodes; - bool tie = false; - std::set zeroWeight; - size_t emitted = 0; - size_t timers = 0; - std::function nominationTimeout; - - LeaderPreviewTestDriver(NodeID const& local, SCPQuorumSet const& qset) - : TestNominationSCP(local, qset), nodes(qset.validators) - { - } - - uint64 - getNodeWeight(NodeID const& id, SCPQuorumSet const&, bool) const override - { - return zeroWeight.count(id) ? 0 : UINT64_MAX; - } - - uint64 - computeHashNode(uint64, Value const&, bool priority, int32 round, - NodeID const& id) override - { - if (!priority) - { - return 0; - } - auto winner = round < 3 ? nodes[0] : round == 3 ? nodes[1] : nodes[2]; - return id == winner || (tie && round < 3 && id == nodes[2]) ? 100 : 1; - } - - void - emitEnvelope(SCPEnvelope const&) override - { - ++emitted; - } - - void - setupTimer(uint64, int timerID, std::chrono::milliseconds, - std::function cb) override - { - ++timers; - if (timerID == Slot::NOMINATION_TIMER) - { - nominationTimeout = std::move(cb); - } - } -}; - -TEST_CASE("nomination leader preview has no protocol side effects", - "[scp][early-nomination]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SCPQuorumSet qset; - qset.threshold = 2; - qset.validators = {v0NodeID, v1NodeID, v2NodeID}; - // The local node is the *second* leader. Merely preparing its value must - // not cause it to vote before the actual nomination timeout. - LeaderPreviewTestDriver driver(v1NodeID, qset); - auto& scp = driver.mSCP; - Value previous{42}, value{43}; - auto slot = std::make_shared(7, scp); - auto const first = scp.predictNominationLeaders(7, previous, 1); - auto const firstTwo = scp.predictNominationLeaders(7, previous, 2); - REQUIRE(first == std::set{v0NodeID}); - REQUIRE(firstTwo == std::set{v0NodeID, v1NodeID}); - REQUIRE(scp.getKnownSlotsCount() == 0); - REQUIRE(slot->getNominationLeaders().empty()); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 0); - - size_t builds = 0; - slot->nominate( - [&]() { - ++builds; - return std::make_shared(value); - }, - previous, false); - REQUIRE(builds == 0); - REQUIRE(slot->getNominationLeaders() == first); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 1); - // Preview again after nomination has started: it still changes no state. - REQUIRE(scp.predictNominationLeaders(7, previous, 2) == firstTwo); - REQUIRE(slot->getNominationLeaders() == first); - REQUIRE(driver.timers == 1); - auto timeout = std::move(driver.nominationTimeout); - REQUIRE(timeout); - timeout(); - REQUIRE(slot->getNominationLeaders() == firstTwo); - REQUIRE(driver.emitted == 1); - REQUIRE(driver.timers == 2); - REQUIRE(builds == 1); - timeout = std::move(driver.nominationTimeout); - REQUIRE(timeout); - timeout(); - REQUIRE(builds == 1); -} - -TEST_CASE("lazy nomination follows leaders and handles unavailable values", - "[scp][early-nomination]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SCPQuorumSet qset; - qset.threshold = 3; - qset.validators = {v0NodeID, v1NodeID, v2NodeID}; - LeaderPreviewTestDriver driver(v1NodeID, qset); - auto& scp = driver.mSCP; - Value previous{42}, value{43}; - size_t builds = 0; - bool available = true; - bool stopWhileBuilding = false; - NominationValueSupplier makeValue = [&]() -> ValueWrapperPtr { - ++builds; - if (stopWhileBuilding) - { - scp.stopNomination(7); - } - return available ? std::make_shared(value) : nullptr; - }; - - SECTION("a follower votes for the leader without constructing a value") - { - SCPEnvelope envelope; - envelope.statement.nodeID = v0NodeID; - envelope.statement.slotIndex = 7; - envelope.statement.pledges.type(SCP_ST_NOMINATE); - auto& nom = envelope.statement.pledges.nominate(); - nom.quorumSetHash = sha256(xdr::xdr_to_opaque(qset)); - nom.votes.push_back(value); - REQUIRE(scp.receiveEnvelope(driver.wrapEnvelope(envelope)) == - SCP::VALID); - REQUIRE(scp.nominate(7, makeValue, previous)); - REQUIRE(builds == 0); - REQUIRE(driver.emitted == 1); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(scp.getNominationLeaders(7).count(v1NodeID)); - // Even a promoted leader needs no private value if it already follows - // a usable value from the first leader. - REQUIRE(builds == 0); - } - SECTION("an unavailable proposal is retried on the next timeout") - { - available = false; - REQUIRE(!scp.nominate(7, makeValue, previous)); - REQUIRE(builds == 0); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(builds == 1); - REQUIRE(driver.emitted == 0); - available = true; - timeout = std::move(driver.nominationTimeout); - REQUIRE(timeout); - timeout(); - REQUIRE(builds == 2); - REQUIRE(driver.emitted == 1); - } - SECTION("stopping nomination during construction cannot emit or rearm") - { - stopWhileBuilding = true; - scp.nominate(7, makeValue, previous); - REQUIRE(builds == 0); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(builds == 1); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 1); - } - SECTION("a stopped follower never constructs a proposal") - { - scp.nominate(7, makeValue, previous); - scp.stopNomination(7); - auto timeout = std::move(driver.nominationTimeout); - timeout(); - REQUIRE(builds == 0); - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 1); - } -} - -TEST_CASE("nomination leader preview preserves ties and zero weights", - "[scp][early-nomination]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SCPQuorumSet qset; - qset.threshold = 2; - qset.validators = {v0NodeID, v1NodeID, v2NodeID}; - LeaderPreviewTestDriver driver(v1NodeID, qset); - Value previous{42}; - SECTION("ties may elect more than two nodes in the first two calls") - { - driver.tie = true; - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 1) == - std::set{v0NodeID, v2NodeID}); - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 2) == - std::set{v0NodeID, v1NodeID, v2NodeID}); - } - SECTION("zero weight winners are excluded, including self") - { - driver.zeroWeight = {v0NodeID, v1NodeID}; - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 2) == - std::set{v2NodeID}); - } - SECTION("no eligible nodes") - { - driver.zeroWeight = {v0NodeID, v1NodeID, v2NodeID}; - REQUIRE(driver.mSCP.predictNominationLeaders(7, previous, 2).empty()); - } - REQUIRE(driver.emitted == 0); - REQUIRE(driver.timers == 0); - REQUIRE(driver.mSCP.getKnownSlotsCount() == 0); -} - -TEST_CASE("nomination weight", "[scp]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - SIMULATION_CREATE_NODE(5); - - SCPQuorumSet qSet; - qSet.threshold = 3; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); - qSet.validators.push_back(v3NodeID); - - TestNominationSCP const nomSCP(v0NodeID, qSet); - - uint64 result = nomSCP.getNodeWeight(v2NodeID, qSet, false); - - REQUIRE(isNear(result, .75)); - - result = nomSCP.getNodeWeight(v4NodeID, qSet, false); - REQUIRE(result == 0); - - SCPQuorumSet iQSet; - iQSet.threshold = 1; - iQSet.validators.push_back(v4NodeID); - iQSet.validators.push_back(v5NodeID); - qSet.innerSets.push_back(iQSet); - - result = nomSCP.getNodeWeight(v4NodeID, qSet, false); - - REQUIRE(isNear(result, .6 * .5)); -} - -class NominationTestHandler : public NominationProtocol -{ - public: - NominationTestHandler(Slot& s) : NominationProtocol(s) - { - } - - void - setPreviousValue(Value const& v) - { - mPreviousValue = v; - } - - void - setRoundNumber(int32 n) - { - mRoundNumber = n; - } - - void - updateRoundLeaders() - { - NominationProtocol::updateRoundLeaders(); - } - - std::set& - getRoundLeaders() - { - return mRoundLeaders; - } - - uint64 - getNodePriority(NodeID const& nodeID, SCPQuorumSet const& qset) - { - return NominationProtocol::getNodePriority(nodeID, qset); - } -}; - -// A test SCPDriver that allows specification of nodes with 0 weight. -class ZeroWeightTestNominationSCP : public TestNominationSCP -{ - public: - std::set mZeroWeightNodes; - - ZeroWeightTestNominationSCP(NodeID const& nodeID, - SCPQuorumSet const& qSetLocal, - std::set const& zeroWeightNodes) - : TestNominationSCP(nodeID, qSetLocal) - , mZeroWeightNodes(zeroWeightNodes) - { - } - - uint64 - getNodeWeight(NodeID const& nodeID, SCPQuorumSet const& qset, - bool isLocalNode) const override - { - if (mZeroWeightNodes.count(nodeID)) - { - return 0; - } - return TestNominationSCP::getNodeWeight(nodeID, qset, isLocalNode); - } -}; - -static SCPQuorumSet -makeQSet(std::vector const& nodeIDs, int threshold, int total, - int offset) -{ - SCPQuorumSet qSet; - qSet.threshold = threshold; - for (int i = 0; i < total; i++) - { - qSet.validators.push_back(nodeIDs[i + offset]); - } - return qSet; -} - -TEST_CASE("updateRoundLeaders handles zero weight nodes", "[scp]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - - // 3 nodes total: v0 (local), v1 (normal weight), v2 (zero weight). - SCPQuorumSet qSet; - qSet.threshold = 2; - qSet.validators.push_back(v0NodeID); - qSet.validators.push_back(v1NodeID); - qSet.validators.push_back(v2NodeID); - - auto runScenario = [&](std::set const& zeroWeightNodes) { - ZeroWeightTestNominationSCP nomSCP(v0NodeID, qSet, zeroWeightNodes); - - Slot slot(0, nomSCP.mSCP); - NominationTestHandler nom(slot); - - Value v; - v.emplace_back(uint8_t(42)); - nom.setPreviousValue(v); - - // Ensure that even with many more rounds than validators, - // `updateRoundLeaders` always terminates and never picks a zero-weight - // node as leader. - int const maxRounds = 20; - for (int i = 0; i < maxRounds; i++) - { - nom.setRoundNumber(i); - nom.updateRoundLeaders(); - } - - return nom.getRoundLeaders(); - }; - - SECTION("non-local zero-weight validator is excluded from round leaders") - { - // v2 simulates a LOW-quality validator with zero weight, mimicking - // HerderSCPDriver::getNodeWeight behavior for LOW-quality nodes. - auto const& leaders = runScenario({v2NodeID}); - REQUIRE(leaders.count(v0NodeID) == 1); - REQUIRE(leaders.count(v1NodeID) == 1); - REQUIRE(leaders.count(v2NodeID) == 0); - REQUIRE(leaders.size() == 2); - } - - SECTION("local zero-weight validator is excluded from round leaders") - { - auto const& leaders = runScenario({v0NodeID}); - REQUIRE(leaders.count(v0NodeID) == 0); - REQUIRE(leaders.count(v1NodeID) == 1); - REQUIRE(leaders.count(v2NodeID) == 1); - REQUIRE(leaders.size() == 2); - } -} - -// this test case display statistical information on the priority function used -// by nomination -TEST_CASE("nomination weight stats", "[scp][!hide]") -{ - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - SIMULATION_CREATE_NODE(5); - SIMULATION_CREATE_NODE(6); - - std::vector nodeIDs = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, - v4NodeID, v5NodeID, v6NodeID}; - - int const totalSlots = 1000; - int const maxRoundPerSlot = 5; // 5 -> 15 seconds - int const totalRounds = totalSlots * maxRoundPerSlot; - - auto runTests = [&](SCPQuorumSet qSet) { - std::map wins; - - TestNominationSCP nomSCP(v0NodeID, qSet); - for (int s = 0; s < totalSlots; s++) - { - Slot slot(s, nomSCP.mSCP); - - NominationTestHandler nom(slot); - - Value v; - v.emplace_back(uint8_t(s)); // anything will do as a value - - nom.setPreviousValue(v); - - for (int i = 0; i < maxRoundPerSlot; i++) - { - nom.setRoundNumber(i); - nom.updateRoundLeaders(); - auto& l = nom.getRoundLeaders(); - REQUIRE(!l.empty()); - for (auto& w : l) - { - wins[w]++; - } - } - } - return wins; - }; - - SECTION("flat quorum") - { - auto flatTest = [&](int threshold, int total) { - auto qSet = makeQSet(nodeIDs, threshold, total, 0); - - auto wins = runTests(qSet); - - for (auto& w : wins) - { - double stats = double(w.second * 100) / double(totalRounds); - CLOG_INFO(SCP, "Got {}{}", stats, - ((v0NodeID == w.first) ? " LOCAL" : "")); - } - }; - - SECTION("3 out of 5") - { - flatTest(3, 5); - } - SECTION("2 out of 3") - { - flatTest(2, 3); - } - } - SECTION("hierarchy") - { - auto qSet = makeQSet(nodeIDs, 3, 4, 0); - - auto qSetInner = makeQSet(nodeIDs, 2, 3, 4); - qSet.innerSets.emplace_back(qSetInner); - - auto wins = runTests(qSet); - - for (auto& w : wins) - { - double stats = double(w.second * 100) / double(totalRounds); - bool outer = - std::any_of(qSet.validators.begin(), qSet.validators.end(), - [&](auto const& k) { return k == w.first; }); - CLOG_INFO(SCP, "Got {} {}", stats, - ((v0NodeID == w.first) ? "LOCAL" - : (outer ? "OUTER" : "INNER"))); - } - } -} - -TEST_CASE("nomination two nodes win stats", "[scp][!hide]") -{ - int const nbRoundsForStats = 9; - SIMULATION_CREATE_NODE(0); - SIMULATION_CREATE_NODE(1); - SIMULATION_CREATE_NODE(2); - - SIMULATION_CREATE_NODE(3); - SIMULATION_CREATE_NODE(4); - SIMULATION_CREATE_NODE(5); - SIMULATION_CREATE_NODE(6); - - std::vector nodeIDs = {v0NodeID, v1NodeID, v2NodeID, v3NodeID, - v4NodeID, v5NodeID, v6NodeID}; - - int const totalIter = 10000; - - // maxRounds is the number of rounds to evaluate in a row - // the iteration is considered successful if validators could - // agree on what to nominate before maxRounds is reached - auto nominationLeaders = [&](int maxRounds, SCPQuorumSet qSetNode0, - SCPQuorumSet qSetNode1) { - TestNominationSCP nomSCP0(v0NodeID, qSetNode0); - TestNominationSCP nomSCP1(v1NodeID, qSetNode1); - - int tot = 0; - for (int g = 0; g < totalIter; g++) - { - Slot slot0(0, nomSCP0.mSCP); - NominationTestHandler nom0(slot0); - - Slot slot1(0, nomSCP1.mSCP); - NominationTestHandler nom1(slot1); - - Value v; - v.emplace_back(uint8_t(g)); - nom0.setPreviousValue(v); - nom1.setPreviousValue(v); - - bool res = true; - - bool v0Voted = false; - bool v1Voted = false; - - int r = 0; - do - { - nom0.setRoundNumber(r); - nom1.setRoundNumber(r); - nom0.updateRoundLeaders(); - nom1.updateRoundLeaders(); - - auto& l0 = nom0.getRoundLeaders(); - REQUIRE(!l0.empty()); - auto& l1 = nom1.getRoundLeaders(); - REQUIRE(!l1.empty()); - - auto updateVoted = [&](auto const& id, auto const& leaders, - bool& voted) { - if (!voted) - { - voted = std::find(leaders.begin(), leaders.end(), id) != - leaders.end(); - } - }; - - // checks if id voted (any past round, including this one) - // AND id is a leader this round - auto findNode = [](auto const& id, bool idVoted, - auto const& otherLeaders) { - bool r = (idVoted && std::find(otherLeaders.begin(), - otherLeaders.end(), - id) != otherLeaders.end()); - return r; - }; - - updateVoted(v0NodeID, l0, v0Voted); - updateVoted(v1NodeID, l1, v1Voted); - - // either both vote for v0 or both vote for v1 - res = findNode(v0NodeID, v0Voted, l1); - res = res || findNode(v1NodeID, v1Voted, l0); - } while (!res && ++r < maxRounds); - - tot += res ? 1 : 0; - } - return tot; - }; - - SECTION("flat quorum") - { - // test using the same quorum on all nodes - auto flatTest = [&](int threshold, int total) { - auto qSet = makeQSet(nodeIDs, threshold, total, 0); - - for (int maxRounds = 1; maxRounds <= nbRoundsForStats; maxRounds++) - { - int tot = nominationLeaders(maxRounds, qSet, qSet); - double stats = double(tot * 100) / double(totalIter); - CLOG_INFO(SCP, "Win rate for {} : {}", maxRounds, stats); - } - }; - - SECTION("3 out of 5") - { - flatTest(3, 5); - } - SECTION("2 out of 3") - { - flatTest(2, 3); - } - } - - SECTION("hierarchy") - { - SECTION("same qSet") - { - auto qSet = makeQSet(nodeIDs, 3, 4, 0); - - auto qSetInner = makeQSet(nodeIDs, 2, 3, 4); - qSet.innerSets.emplace_back(qSetInner); - - for (int maxRounds = 1; maxRounds <= nbRoundsForStats; maxRounds++) - { - int tot = nominationLeaders(maxRounds, qSet, qSet); - double stats = double(tot * 100) / double(totalIter); - CLOG_INFO(SCP, "Win rate for {} : {}", maxRounds, stats); - } - } - SECTION("v0 is inner node for v1") - { - auto qSet0 = makeQSet(nodeIDs, 3, 4, 0); - auto qSetInner0 = makeQSet(nodeIDs, 2, 3, 4); - qSet0.innerSets.emplace_back(qSetInner0); - - // v1's qset: we move v0 into the inner set - auto qSet1 = qSet0; - REQUIRE(qSet1.validators[0] == v0NodeID); - std::swap(qSet1.validators[0], qSet1.innerSets[0].validators[0]); - - for (int maxRounds = 1; maxRounds <= nbRoundsForStats; maxRounds++) - { - int tot = nominationLeaders(maxRounds, qSet0, qSet1); - double stats = double(tot * 100) / double(totalIter); - CLOG_INFO(SCP, "Win rate for {} : {}", maxRounds, stats); - } - } - } -} -} diff --git a/src/simulation/ApplyLoad.cpp b/src/simulation/ApplyLoad.cpp index 720f030d86..f96126f9b5 100644 --- a/src/simulation/ApplyLoad.cpp +++ b/src/simulation/ApplyLoad.cpp @@ -1050,16 +1050,13 @@ ApplyLoad::closeBenchmarkLedger(std::vector const& txs, xdr::xdr_from_opaque(wireBytes, receivedXdr); auto receivedTxSet = TxSetXDRFrame::makeFromWire(receivedXdr); - // Nomination through externalization and apply use production SCP paths. + // Proposal through externalization and apply use production SCP paths. herder.getPendingEnvelopes().putTxSet(receivedTxSet->getContentsHash(), ledgerSeq, receivedTxSet); auto& driver = herder.getHerderSCPDriver(); - driver.nominate( - ledgerSeq, - [&driver, nominatedValue]() { - return driver.wrapStellarValue(nominatedValue); - }, - lcl.header.scpValue); + driver.startSlot(ledgerSeq, [&driver, nominatedValue]() { + return driver.wrapStellarValue(nominatedValue); + }); auto const deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); diff --git a/src/test/TxTests.cpp b/src/test/TxTests.cpp index 66b40cc8c4..d4e4a68c12 100644 --- a/src/test/TxTests.cpp +++ b/src/test/TxTests.cpp @@ -80,7 +80,7 @@ captureLastClosedLedgerLcm(Application& app) // Configs whose ledger content depends on thread scheduling or on // randomized nomination cannot produce reproducible golden data. if (app.getConfig().EXPERIMENTAL_PARALLEL_TX_SET_DOWNLOAD || - app.getConfig().TESTING_NOMINATE_RANDOM_VALUES) + app.getConfig().TESTING_PROPOSE_RANDOM_VALUES) { disableLcmCapture(); }